2024-11-07

Title: Maximal Extractable Value in Decentralized Finance: Taxonomy, Detection, and Mitigation

• Authors: Huned Materwala, Shraddha M. Naik, Aya Taha, Tala Abdulrahman Abed, Davor Svetinovic
• Subjects: cs.CR, cs.CE, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.03327
• Pdf URL: https://arxiv.org/pdf/2411.03327
• Copy Paste: [[2411.03327]] Maximal Extractable Value in Decentralized Finance: Taxonomy, Detection, and Mitigation(https://arxiv.org/abs/2411.03327)
• Keywords: secure, security
• Abstract: Decentralized Finance (DeFi) leverages blockchain-enabled smart contracts to deliver automated and trustless financial services without the need for intermediaries. However, the public visibility of financial transactions on the blockchain can be exploited, as participants can reorder, insert, or remove transactions to extract value, often at the expense of others. This extracted value is known as the Maximal Extractable Value (MEV). MEV causes financial losses and consensus instability, disrupting the security, efficiency, and decentralization goals of the DeFi ecosystem. Therefore, it is crucial to analyze, detect, and mitigate MEV to safeguard DeFi. Our comprehensive survey offers a holistic view of the MEV landscape in the DeFi ecosystem. We present an in-depth understanding of MEV through a novel taxonomy of MEV transactions supported by real transaction examples. We perform a critical comparative analysis of various MEV detection approaches, evaluating their effectiveness in identifying different transaction types. Furthermore, we assess different categories of MEV mitigation strategies and discuss their limitations. We identify the challenges of current mitigation and detection approaches and discuss potential solutions. This survey provides valuable insights for researchers, developers, stakeholders, and policymakers, helping to curb and democratize MEV for a more secure and efficient DeFi ecosystem.

Title: Unlocking the Archives: Using Large Language Models to Transcribe Handwritten Historical Documents

• Authors: Mark Humphries, Lianne C. Leddy, Quinn Downton, Meredith Legace, John McConnell, Isabella Murray, Elizabeth Spence
• Subjects: cs.CV, cs.CL, cs.DL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03340
• Pdf URL: https://arxiv.org/pdf/2411.03340
• Copy Paste: [[2411.03340]] Unlocking the Archives: Using Large Language Models to Transcribe Handwritten Historical Documents(https://arxiv.org/abs/2411.03340)
• Keywords: large language model
• Abstract: This study demonstrates that Large Language Models (LLMs) can transcribe historical handwritten documents with significantly higher accuracy than specialized Handwritten Text Recognition (HTR) software, while being faster and more cost-effective. We introduce an open-source software tool called Transcription Pearl that leverages these capabilities to automatically transcribe and correct batches of handwritten documents using commercially available multimodal LLMs from OpenAI, Anthropic, and Google. In tests on a diverse corpus of 18th/19th century English language handwritten documents, LLMs achieved Character Error Rates (CER) of 5.7 to 7% and Word Error Rates (WER) of 8.9 to 15.9%, improvements of 14% and 32% respectively over specialized state-of-the-art HTR software like Transkribus. Most significantly, when LLMs were then used to correct those transcriptions as well as texts generated by conventional HTR software, they achieved near-human levels of accuracy, that is CERs as low as 1.8% and WERs of 3.5%. The LLMs also completed these tasks 50 times faster and at approximately 1/50th the cost of proprietary HTR programs. These results demonstrate that when LLMs are incorporated into software tools like Transcription Pearl, they provide an accessible, fast, and highly accurate method for mass transcription of historical handwritten documents, significantly streamlining the digitization process.

Title: What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks

• Authors: Nathalie Maria Kirch, Severin Field, Stephen Casper
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03343
• Pdf URL: https://arxiv.org/pdf/2411.03343
• Copy Paste: [[2411.03343]] What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks(https://arxiv.org/abs/2411.03343)
• Keywords: attack, large language model
• Abstract: While `jailbreaks' have been central to research on the safety and reliability of LLMs (large language models), the underlying mechanisms behind these attacks are not well understood. Some prior works have used linear methods to analyze jailbreak prompts or model refusal. Here, however, we compare linear and nonlinear methods to study the features in prompts that contribute to successful jailbreaks. We do this by probing for jailbreak success based only on the portions of the latent representations corresponding to prompt tokens. First, we introduce a dataset of 10,800 jailbreak attempts from 35 attack methods. We then show that different jailbreaking methods work via different nonlinear features in prompts. Specifically, we find that while probes can distinguish between successful and unsuccessful jailbreaking prompts with a high degree of accuracy, they often transfer poorly to held-out attack methods. We also show that nonlinear probes can be used to mechanistically jailbreak the LLM by guiding the design of adversarial latent perturbations. These mechanistic jailbreaks are able to jailbreak Gemma-7B-IT more reliably than 34 of the 35 techniques that it was trained on. Ultimately, our results suggest that jailbreaks cannot be thoroughly understood in terms of universal or linear prompt features alone.

Title: Comparing Security and Efficiency of WebAssembly and Linux Containers in Kubernetes Cloud Computing

• Authors: Jasper Alexander Wiegratz
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.03344
• Pdf URL: https://arxiv.org/pdf/2411.03344
• Copy Paste: [[2411.03344]] Comparing Security and Efficiency of WebAssembly and Linux Containers in Kubernetes Cloud Computing(https://arxiv.org/abs/2411.03344)
• Keywords: secure, security, attack
• Abstract: This study investigates the potential of WebAssembly as a more secure and efficient alternative to Linux containers for executing untrusted code in cloud computing with Kubernetes. Specifically, it evaluates the security and performance implications of this shift. Security analyses demonstrate that both Linux containers and WebAssembly have attack surfaces when executing untrusted code, but WebAssembly presents a reduced attack surface due to an additional layer of isolation. The performance analysis further reveals that while WebAssembly introduces overhead, particularly in startup times, it could be negligible in long-running computations. However, WebAssembly enhances the core principle of containerization, offering better security through isolation and platform-agnostic portability compared to Linux containers. This research demonstrates that WebAssembly is not a silver bullet for all security concerns or performance requirements in a Kubernetes environment, but typical attacks are less likely to succeed and the performance loss is relatively small.

Title: Fixing Security Vulnerabilities with AI in OSS-Fuzz

• Authors: Yuntong Zhang, Jiawei Wang, Dominic Berzin, Martin Mirchev, Dongge Liu, Abhishek Arya, Oliver Chang, Abhik Roychoudhury
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2411.03346
• Pdf URL: https://arxiv.org/pdf/2411.03346
• Copy Paste: [[2411.03346]] Fixing Security Vulnerabilities with AI in OSS-Fuzz(https://arxiv.org/abs/2411.03346)
• Keywords: security, large language model
• Abstract: Critical open source software systems undergo significant validation in the form of lengthy fuzz campaigns. The fuzz campaigns typically conduct a biased random search over the domain of program inputs, to find inputs which crash the software system. Such fuzzing is useful to enhance the security of software systems in general since even closed source software may use open source components. Hence testing open source software is of paramount importance. Currently OSS-Fuzz is the most significant and widely used infrastructure for continuous validation of open source systems. Unfortunately even though OSS-Fuzz has identified more than 10,000 vulnerabilities across 1000 or more software projects, the detected vulnerabilities may remain unpatched, as vulnerability fixing is often manual in practice. In this work, we rely on the recent progress in Large Language Model (LLM) agents for autonomous program improvement including bug fixing. We customise the well-known AutoCodeRover agent for fixing security vulnerabilities. This is because LLM agents like AutoCodeRover fix bugs from issue descriptions via code search. Instead for security patching, we rely on the test execution of the exploit input to extract code elements relevant to the fix. Our experience with OSS-Fuzz vulnerability data shows that LLM agent autonomy is useful for successful security patching, as opposed to approaches like Agentless where the control flow is fixed. More importantly our findings show that we cannot measure quality of patches by code similarity of the patch with reference codes (as in CodeBLEU scores used in VulMaster), since patches with high CodeBLEU scores still fail to pass given the given exploit input. Our findings indicate that security patch correctness needs to consider dynamic attributes like test executions as opposed to relying of standard text/code similarity metrics.

Title: Undermining Image and Text Classification Algorithms Using Adversarial Attacks

• Authors: Langalibalele Lunga, Suhas Sreehari
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03348
• Pdf URL: https://arxiv.org/pdf/2411.03348
• Copy Paste: [[2411.03348]] Undermining Image and Text Classification Algorithms Using Adversarial Attacks(https://arxiv.org/abs/2411.03348)
• Keywords: security, defense, attack, robust, generative
• Abstract: Machine learning models are prone to adversarial attacks, where inputs can be manipulated in order to cause misclassifications. While previous research has focused on techniques like Generative Adversarial Networks (GANs), there's limited exploration of GANs and Synthetic Minority Oversampling Technique (SMOTE) in text and image classification models to perform adversarial attacks. Our study addresses this gap by training various machine learning models and using GANs and SMOTE to generate additional data points aimed at attacking text classification models. Furthermore, we extend our investigation to face recognition models, training a Convolutional Neural Network(CNN) and subjecting it to adversarial attacks with fast gradient sign perturbations on key features identified by GradCAM, a technique used to highlight key image characteristics CNNs use in classification. Our experiments reveal a significant vulnerability in classification models. Specifically, we observe a 20 % decrease in accuracy for the top-performing text classification models post-attack, along with a 30 % decrease in facial recognition accuracy. This highlights the susceptibility of these models to manipulation of input data. Adversarial attacks not only compromise the security but also undermine the reliability of machine learning systems. By showcasing the impact of adversarial attacks on both text classification and face recognition models, our study underscores the urgent need for develop robust defenses against such vulnerabilities.

Title: A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness

• Authors: Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, Qi He, Yao Ma, Ming Huang, Suhang Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03350
• Pdf URL: https://arxiv.org/pdf/2411.03350
• Copy Paste: [[2411.03350]] A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness(https://arxiv.org/abs/2411.03350)
• Keywords: privacy, large language model
• Abstract: Large language models (LLM) have demonstrated emergent abilities in text generation, question answering, and reasoning, facilitating various tasks and domains. Despite their proficiency in various tasks, LLMs like LaPM 540B and Llama-3.1 405B face limitations due to large parameter sizes and computational demands, often requiring cloud API use which raises privacy concerns, limits real-time applications on edge devices, and increases fine-tuning costs. Additionally, LLMs often underperform in specialized domains such as healthcare and law due to insufficient domain-specific knowledge, necessitating specialized models. Therefore, Small Language Models (SLMs) are increasingly favored for their low inference latency, cost-effectiveness, efficient development, and easy customization and adaptability. These models are particularly well-suited for resource-limited environments and domain knowledge acquisition, addressing LLMs' challenges and proving ideal for applications that require localized data handling for privacy, minimal inference latency for efficiency, and domain knowledge acquisition through lightweight fine-tuning. The rising demand for SLMs has spurred extensive research and development. However, a comprehensive survey investigating issues related to the definition, acquisition, application, enhancement, and reliability of SLM remains lacking, prompting us to conduct a detailed survey on these topics. The definition of SLMs varies widely, thus to standardize, we propose defining SLMs by their capability to perform specialized tasks and suitability for resource-constrained settings, setting boundaries based on the minimal size for emergent abilities and the maximum size sustainable under resource constraints. For other aspects, we provide a taxonomy of relevant models/methods and develop general frameworks for each category to enhance and utilize SLMs effectively.

Title: Tabular Data Synthesis with Differential Privacy: A Survey

• Authors: Mengmeng Yang, Chi-Hung Chi, Kwok-Yan Lam, Jie Feng, Taolin Guo, Wei Ni
• Subjects: cs.CR, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2411.03351
• Pdf URL: https://arxiv.org/pdf/2411.03351
• Copy Paste: [[2411.03351]] Tabular Data Synthesis with Differential Privacy: A Survey(https://arxiv.org/abs/2411.03351)
• Keywords: privacy, protect, attack
• Abstract: Data sharing is a prerequisite for collaborative innovation, enabling organizations to leverage diverse datasets for deeper insights. In real-world applications like FinTech and Smart Manufacturing, transactional data, often in tabular form, are generated and analyzed for insight generation. However, such datasets typically contain sensitive personal/business information, raising privacy concerns and regulatory risks. Data synthesis tackles this by generating artificial datasets that preserve the statistical characteristics of real data, removing direct links to individuals. However, attackers can still infer sensitive information using background knowledge. Differential privacy offers a solution by providing provable and quantifiable privacy protection. Consequently, differentially private data synthesis has emerged as a promising approach to privacy-aware data sharing. This paper provides a comprehensive overview of existing differentially private tabular data synthesis methods, highlighting the unique challenges of each generation model for generating tabular data under differential privacy constraints. We classify the methods into statistical and deep learning-based approaches based on their generation models, discussing them in both centralized and distributed environments. We evaluate and compare those methods within each category, highlighting their strengths and weaknesses in terms of utility, privacy, and computational complexity. Additionally, we present and discuss various evaluation methods for assessing the quality of the synthesized data, identify research gaps in the field and directions for future research.

Title: LLM-based Continuous Intrusion Detection Framework for Next-Gen Networks

• Authors: Frederic Adjewa, Moez Esseghir, Leila Merghem-Boulahia
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.03354
• Pdf URL: https://arxiv.org/pdf/2411.03354
• Copy Paste: [[2411.03354]] LLM-based Continuous Intrusion Detection Framework for Next-Gen Networks(https://arxiv.org/abs/2411.03354)
• Keywords: attack, transformer
• Abstract: In this paper, we present an adaptive framework designed for the continuous detection, identification and classification of emerging attacks in network traffic. The framework employs a transformer encoder architecture, which captures hidden patterns in a bidirectional manner to differentiate between malicious and legitimate traffic. Initially, the framework focuses on the accurate detection of malicious activities, achieving a perfect recall of 100\% in distinguishing between attack and benign traffic. Subsequently, the system incrementally identifies unknown attack types by leveraging a Gaussian Mixture Model (GMM) to cluster features derived from high-dimensional BERT embeddings. This approach allows the framework to dynamically adjust its identification capabilities as new attack clusters are discovered, maintaining high detection accuracy. Even after integrating additional unknown attack clusters, the framework continues to perform at a high level, achieving 95.6\% in both classification accuracy and this http URL results demonstrate the effectiveness of the proposed framework in adapting to evolving threats while maintaining high accuracy in both detection and identification tasks. Our ultimate goal is to develop a scalable, real-time intrusion detection system that can continuously evolve with the ever-changing network threat landscape.

Title: Exploring Feature Importance and Explainability Towards Enhanced ML-Based DoS Detection in AI Systems

• Authors: Paul Badu Yakubu, Evans Owusu, Lesther Santana, Mohamed Rahouti, Abdellah Chehri, Kaiqi Xiong
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03355
• Pdf URL: https://arxiv.org/pdf/2411.03355
• Copy Paste: [[2411.03355]] Exploring Feature Importance and Explainability Towards Enhanced ML-Based DoS Detection in AI Systems(https://arxiv.org/abs/2411.03355)
• Keywords: security, attack, explainability
• Abstract: Denial of Service (DoS) attacks pose a significant threat in the realm of AI systems security, causing substantial financial losses and downtime. However, AI systems' high computational demands, dynamic behavior, and data variability make monitoring and detecting DoS attacks challenging. Nowadays, statistical and machine learning (ML)-based DoS classification and detection approaches utilize a broad range of feature selection mechanisms to select a feature subset from networking traffic datasets. Feature selection is critical in enhancing the overall model performance and attack detection accuracy while reducing the training time. In this paper, we investigate the importance of feature selection in improving ML-based detection of DoS attacks. Specifically, we explore feature contribution to the overall components in DoS traffic datasets by utilizing statistical analysis and feature engineering approaches. Our experimental findings demonstrate the usefulness of the thorough statistical analysis of DoS traffic and feature engineering in understanding the behavior of the attack and identifying the best feature selection for ML-based DoS classification and detection.

Title: Enhancing Table Representations with LLM-powered Synthetic Data Generation

• Authors: Dayu Yang, Natawut Monaikul, Amanda Ding, Bozhao Tan, Kishore Mosaliganti, Giri Iyengar
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03356
• Pdf URL: https://arxiv.org/pdf/2411.03356
• Copy Paste: [[2411.03356]] Enhancing Table Representations with LLM-powered Synthetic Data Generation(https://arxiv.org/abs/2411.03356)
• Keywords: large language model
• Abstract: In the era of data-driven decision-making, accurate table-level representations and efficient table recommendation systems are becoming increasingly crucial for improving table management, discovery, and analysis. However, existing approaches to tabular data representation often face limitations, primarily due to their focus on cell-level tasks and the lack of high-quality training data. To address these challenges, we first formulate a clear definition of table similarity in the context of data transformation activities within data-driven enterprises. This definition serves as the foundation for synthetic data generation, which require a well-defined data generation process. Building on this, we propose a novel synthetic data generation pipeline that harnesses the code generation and data manipulation capabilities of Large Language Models (LLMs) to create a large-scale synthetic dataset tailored for table-level representation learning. Through manual validation and performance comparisons on the table recommendation task, we demonstrate that the synthetic data generated by our pipeline aligns with our proposed definition of table similarity and significantly enhances table representations, leading to improved recommendation performance.

Title: PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption

• Authors: Yifan Tan, Cheng Tan, Zeyu Mi, Haibo Chen
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2411.03357
• Pdf URL: https://arxiv.org/pdf/2411.03357
• Copy Paste: [[2411.03357]] PipeLLM: Fast and Confidential Large Language Model Services with Speculative Pipelined Encryption(https://arxiv.org/abs/2411.03357)
• Keywords: security, large language model
• Abstract: Confidential computing on GPUs, like NVIDIA H100, mitigates the security risks of outsourced Large Language Models (LLMs) by implementing strong isolation and data encryption. Nonetheless, this encryption incurs a significant performance overhead, reaching up to 52.8 percent and 88.2 percent throughput drop when serving OPT-30B and OPT-66B, respectively. To address this challenge, we introduce PipeLLM, a user-transparent runtime system. PipeLLM removes the overhead by overlapping the encryption and GPU computation through pipelining - an idea inspired by the CPU instruction pipelining - thereby effectively concealing the latency increase caused by encryption. The primary technical challenge is that, unlike CPUs, the encryption module lacks prior knowledge of the specific data needing encryption until it is requested by the GPUs. To this end, we propose speculative pipelined encryption to predict the data requiring encryption by analyzing the serving patterns of LLMs. Further, we have developed an efficient, low-cost pipeline relinquishing approach for instances of incorrect predictions. Our experiments on NVIDIA H100 GPU show that compared with vanilla systems without confidential computing (e.g., vLLM, PEFT, and FlexGen), PipeLLM incurs modest overhead (less than 19.6 percent in throughput) across various LLM sizes, from 13B to 175B.

Title: SPINEX_ Symbolic Regression: Similarity-based Symbolic Regression with Explainable Neighbors Exploration

• Authors: MZ Naser, Ahmed Z Naser
• Subjects: cs.LG, stat.CO
• Abstract URL: https://arxiv.org/abs/2411.03358
• Pdf URL: https://arxiv.org/pdf/2411.03358
• Copy Paste: [[2411.03358]] SPINEX_ Symbolic Regression: Similarity-based Symbolic Regression with Explainable Neighbors Exploration(https://arxiv.org/abs/2411.03358)
• Keywords: explainability
• Abstract: This article introduces a new symbolic regression algorithm based on the SPINEX (Similarity-based Predictions with Explainable Neighbors Exploration) family. This new algorithm (SPINEX_SymbolicRegression) adopts a similarity-based approach to identifying high-merit expressions that satisfy accuracy- and structural similarity metrics. We conducted extensive benchmarking tests comparing SPINEX_SymbolicRegression to over 180 mathematical benchmarking functions from international problem sets that span randomly generated expressions and those based on real physical phenomena. Then, we evaluated the performance of the proposed algorithm in terms of accuracy, expression similarity in terms of presence operators and variables (as compared to the actual expressions), population size, and number of generations at convergence. The results indicate that SPINEX_SymbolicRegression consistently performs well and can, in some instances, outperform leading algorithms. In addition, the algorithm's explainability capabilities are highlighted through in-depth experiments.

Title: Pedestrian Volume Prediction Using a Diffusion Convolutional Gated Recurrent Unit Model

• Authors: Yiwei Dong, Tingjin Chu, Lele Zhang, Hadi Ghaderi, Hanfang Yang
• Subjects: cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2411.03360
• Pdf URL: https://arxiv.org/pdf/2411.03360
• Copy Paste: [[2411.03360]] Pedestrian Volume Prediction Using a Diffusion Convolutional Gated Recurrent Unit Model(https://arxiv.org/abs/2411.03360)
• Keywords: diffusion
• Abstract: Effective models for analysing and predicting pedestrian flow are important to ensure the safety of both pedestrians and other road users. These tools also play a key role in optimising infrastructure design and geometry and supporting the economic utility of interconnected communities. The implementation of city-wide automatic pedestrian counting systems provides researchers with invaluable data, enabling the development and training of deep learning applications that offer better insights into traffic and crowd flows. Benefiting from real-world data provided by the City of Melbourne pedestrian counting system, this study presents a pedestrian flow prediction model, as an extension of Diffusion Convolutional Grated Recurrent Unit (DCGRU) with dynamic time warping, named DCGRU-DTW. This model captures the spatial dependencies of pedestrian flow through the diffusion process and the temporal dependency captured by Gated Recurrent Unit (GRU). Through extensive numerical experiments, we demonstrate that the proposed model outperforms the classic vector autoregressive model and the original DCGRU across multiple model accuracy metrics.

Title: TDDBench: A Benchmark for Training data detection

• Authors: Zhihao Zhu, Yi Yang, Defu Lian
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03363
• Pdf URL: https://arxiv.org/pdf/2411.03363
• Copy Paste: [[2411.03363]] TDDBench: A Benchmark for Training data detection(https://arxiv.org/abs/2411.03363)
• Keywords: security, attack, membership infer
• Abstract: Training Data Detection (TDD) is a task aimed at determining whether a specific data instance is used to train a machine learning model. In the computer security literature, TDD is also referred to as Membership Inference Attack (MIA). Given its potential to assess the risks of training data breaches, ensure copyright authentication, and verify model unlearning, TDD has garnered significant attention in recent years, leading to the development of numerous methods. Despite these advancements, there is no comprehensive benchmark to thoroughly evaluate the effectiveness of TDD methods. In this work, we introduce TDDBench, which consists of 13 datasets spanning three data modalities: image, tabular, and text. We benchmark 21 different TDD methods across four detection paradigms and evaluate their performance from five perspectives: average detection performance, best detection performance, memory consumption, and computational efficiency in both time and memory. With TDDBench, researchers can identify bottlenecks and areas for improvement in TDD algorithms, while practitioners can make informed trade-offs between effectiveness and efficiency when selecting TDD algorithms for specific use cases. Our large-scale benchmarking also reveals the generally unsatisfactory performance of TDD algorithms across different datasets. To enhance accessibility and reproducibility, we open-source TDDBench for the research community.

Title: DM4Steal: Diffusion Model For Link Stealing Attack On Graph Neural Networks

• Authors: Jinyin Chen, Haonan Ma, Haibin Zheng
• Subjects: cs.CR, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03364
• Pdf URL: https://arxiv.org/pdf/2411.03364
• Copy Paste: [[2411.03364]] DM4Steal: Diffusion Model For Link Stealing Attack On Graph Neural Networks(https://arxiv.org/abs/2411.03364)
• Keywords: privacy, defense, attack, steal, diffusion
• Abstract: Graph has become increasingly integral to the advancement of recommendation systems, particularly with the fast development of graph neural network(GNN). By exploring the virtue of rich node features and link information, GNN is designed to provide personalized and accurate suggestions. Meanwhile, the privacy leakage of GNN in such contexts has also captured special attention. Prior work has revealed that a malicious user can utilize auxiliary knowledge to extract sensitive link data of the target graph, integral to recommendation systems, via the decision made by the target GNN model. This poses a significant risk to the integrity and confidentiality of data used in recommendation system. Though important, previous works on GNN's privacy leakage are still challenged in three aspects, i.e., limited stealing attack scenarios, sub-optimal attack performance, and adaptation against defense. To address these issues, we propose a diffusion model based link stealing attack, named DM4Steal. It differs previous work from three critical aspects. (i) Generality: aiming at six attack scenarios with limited auxiliary knowledge, we propose a novel training strategy for diffusion models so that DM4Steal is transferable to diverse attack scenarios. (ii) Effectiveness: benefiting from the retention of semantic structure in the diffusion model during the training process, DM4Steal is capable to learn the precise topology of the target graph through the GNN decision process. (iii) Adaptation: when GNN is defensive (e.g., DP, Dropout), DM4Steal relies on the stability that comes from sampling the score model multiple times to keep performance degradation to a minimum, thus DM4Steal implements successful adaptive attack on defensive GNN.

Title: Enhanced Real-Time Threat Detection in 5G Networks: A Self-Attention RNN Autoencoder Approach for Spectral Intrusion Analysis

• Authors: Mohammadreza Kouchaki, Minglong Zhang, Aly S. Abdalla, Guangchen Lan, Christopher G. Brinton, Vuk Marojevic
• Subjects: cs.CR, cs.AI, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.03365
• Pdf URL: https://arxiv.org/pdf/2411.03365
• Copy Paste: [[2411.03365]] Enhanced Real-Time Threat Detection in 5G Networks: A Self-Attention RNN Autoencoder Approach for Spectral Intrusion Analysis(https://arxiv.org/abs/2411.03365)
• Keywords: security, attack, transformer
• Abstract: In the rapidly evolving landscape of 5G technology, safeguarding Radio Frequency (RF) environments against sophisticated intrusions is paramount, especially in dynamic spectrum access and management. This paper presents an enhanced experimental model that integrates a self-attention mechanism with a Recurrent Neural Network (RNN)-based autoencoder for the detection of anomalous spectral activities in 5G networks at the waveform level. Our approach, grounded in time-series analysis, processes in-phase and quadrature (I/Q) samples to identify irregularities that could indicate potential jamming attacks. The model's architecture, augmented with a self-attention layer, extends the capabilities of RNN autoencoders, enabling a more nuanced understanding of temporal dependencies and contextual relationships within the RF spectrum. Utilizing a simulated 5G Radio Access Network (RAN) test-bed constructed with srsRAN 5G and Software Defined Radios (SDRs), we generated a comprehensive stream of data that reflects real-world RF spectrum conditions and attack scenarios. The model is trained to reconstruct standard signal behavior, establishing a normative baseline against which deviations, indicative of security threats, are identified. The proposed architecture is designed to balance between detection precision and computational efficiency, so the LSTM network, enriched with self-attention, continues to optimize for minimal execution latency and power consumption. Conducted on a real-world SDR-based testbed, our results demonstrate the model's improved performance and accuracy in threat detection. Keywords: self-attention, real-time intrusion detection, RNN autoencoder, Transformer architecture, LSTM, time series anomaly detection, 5G Security, spectrum access security.

Title: Personal Data Protection in AI-Native 6G Systems

• Authors: Keivan Navaie
• Subjects: cs.CR, cs.ET, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.03368
• Pdf URL: https://arxiv.org/pdf/2411.03368
• Copy Paste: [[2411.03368]] Personal Data Protection in AI-Native 6G Systems(https://arxiv.org/abs/2411.03368)
• Keywords: security, privacy, protect, fair, generative
• Abstract: As 6G evolves into an AI-native technology, the integration of artificial intelligence (AI) and Generative AI into cellular communication systems presents unparalleled opportunities for enhancing connectivity, network optimization, and personalized services. However, these advancements also introduce significant data protection challenges, as AI models increasingly depend on vast amounts of personal data for training and decision-making. In this context, ensuring compliance with stringent data protection regulations, such as the General Data Protection Regulation (GDPR), becomes critical for the design and operational integrity of 6G networks. These regulations shape key system architecture aspects, including transparency, accountability, fairness, bias mitigation, and data security. This paper identifies and examines the primary data protection risks associated with AI-driven 6G networks, focusing on the complex data flows and processing activities throughout the 6G lifecycle. By exploring these risks, we provide a comprehensive analysis of the potential privacy implications and propose effective mitigation strategies. Our findings stress the necessity of embedding privacy-by-design and privacy-by-default principles in the development of 6G standards to ensure both regulatory compliance and the protection of individual rights.

Title: Blockchain-Based Multi-Path Mobile Access Point Selection for Secure 5G VANETs

• Authors: Zhiou Zhang, Weian Guo, Li Li, Dongyang Li
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.03371
• Pdf URL: https://arxiv.org/pdf/2411.03371
• Copy Paste: [[2411.03371]] Blockchain-Based Multi-Path Mobile Access Point Selection for Secure 5G VANETs(https://arxiv.org/abs/2411.03371)
• Keywords: secure, security, attack
• Abstract: This letter presents a blockchain-based multi-path mobile access point (MAP) selection strategy for secure 5G vehicular ad-hoc networks (VANETs). The proposed method leverages blockchain technology for decentralized, transparent, and secure MAP selection, while the multi-path transmission strategy enhances network reliability and reduces communication delays. A trust-based attack detection mechanism is integrated to ensure network security. Simulation results demonstrate that the proposed algorithm reduces both handover frequency and average communication delay by over 80%, and successfully identifies and excludes more than 95% of Sybil nodes, ensuring reliable and secure communication in highly dynamic vehicular environments.

Title: Energy Price Modelling: A Comparative Evaluation of four Generations of Forecasting Methods

• Authors: Alexandru-Victor Andrei, Georg Velev, Filip-Mihai Toma, Daniel Traian Pele, Stefan Lessmann
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03372
• Pdf URL: https://arxiv.org/pdf/2411.03372
• Copy Paste: [[2411.03372]] Energy Price Modelling: A Comparative Evaluation of four Generations of Forecasting Methods(https://arxiv.org/abs/2411.03372)
• Keywords: transformer
• Abstract: Energy is a critical driver of modern economic systems. Accurate energy price forecasting plays an important role in supporting decision-making at various levels, from operational purchasing decisions at individual business organizations to policy-making. A significant body of literature has looked into energy price forecasting, investigating a wide range of methods to improve accuracy and inform these critical decisions. Given the evolving landscape of forecasting techniques, the literature lacks a thorough empirical comparison that systematically contrasts these methods. This paper provides an in-depth review of the evolution of forecasting modeling frameworks, from well-established econometric models to machine learning methods, early sequence learners such LSTMs, and more recent advancements in deep learning with transformer networks, which represent the cutting edge in forecasting. We offer a detailed review of the related literature and categorize forecasting methodologies into four model families. We also explore emerging concepts like pre-training and transfer learning, which have transformed the analysis of unstructured data and hold significant promise for time series forecasting. We address a gap in the literature by performing a comprehensive empirical analysis on these four family models, using data from the EU energy markets, we conduct a large-scale empirical study, which contrasts the forecasting accuracy of different approaches, focusing especially on alternative propositions for time series transformers.

Title: Kernel Approximation using Analog In-Memory Computing

• Authors: Julian Büchel, Giacomo Camposampiero, Athanasios Vasilopoulos, Corey Lammie, Manuel Le Gallo, Abbas Rahimi, Abu Sebastian
• Subjects: cs.LG, cs.AR
• Abstract URL: https://arxiv.org/abs/2411.03375
• Pdf URL: https://arxiv.org/pdf/2411.03375
• Copy Paste: [[2411.03375]] Kernel Approximation using Analog In-Memory Computing(https://arxiv.org/abs/2411.03375)
• Keywords: transformer
• Abstract: Kernel functions are vital ingredients of several machine learning algorithms, but often incur significant memory and computational costs. We introduce an approach to kernel approximation in machine learning algorithms suitable for mixed-signal Analog In-Memory Computing (AIMC) architectures. Analog In-Memory Kernel Approximation addresses the performance bottlenecks of conventional kernel-based methods by executing most operations in approximate kernel methods directly in memory. The IBM HERMES Project Chip, a state-of-the-art phase-change memory based AIMC chip, is utilized for the hardware demonstration of kernel approximation. Experimental results show that our method maintains high accuracy, with less than a 1% drop in kernel-based ridge classification benchmarks and within 1% accuracy on the Long Range Arena benchmark for kernelized attention in Transformer neural networks. Compared to traditional digital accelerators, our approach is estimated to deliver superior energy efficiency and lower power consumption. These findings highlight the potential of heterogeneous AIMC architectures to enhance the efficiency and scalability of machine learning applications.

Title: Quantifying Aleatoric Uncertainty of the Treatment Effect: A Novel Orthogonal Learner

• Authors: Valentyn Melnychuk, Stefan Feuerriegel, Mihaela van der Schaar
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.03387
• Pdf URL: https://arxiv.org/pdf/2411.03387
• Copy Paste: [[2411.03387]] Quantifying Aleatoric Uncertainty of the Treatment Effect: A Novel Orthogonal Learner(https://arxiv.org/abs/2411.03387)
• Keywords: robust
• Abstract: Estimating causal quantities from observational data is crucial for understanding the safety and effectiveness of medical treatments. However, to make reliable inferences, medical practitioners require not only estimating averaged causal quantities, such as the conditional average treatment effect, but also understanding the randomness of the treatment effect as a random variable. This randomness is referred to as aleatoric uncertainty and is necessary for understanding the probability of benefit from treatment or quantiles of the treatment effect. Yet, the aleatoric uncertainty of the treatment effect has received surprisingly little attention in the causal machine learning community. To fill this gap, we aim to quantify the aleatoric uncertainty of the treatment effect at the covariate-conditional level, namely, the conditional distribution of the treatment effect (CDTE). Unlike average causal quantities, the CDTE is not point identifiable without strong additional assumptions. As a remedy, we employ partial identification to obtain sharp bounds on the CDTE and thereby quantify the aleatoric uncertainty of the treatment effect. We then develop a novel, orthogonal learner for the bounds on the CDTE, which we call AU-learner. We further show that our AU-learner has several strengths in that it satisfies Neyman-orthogonality and is doubly robust. Finally, we propose a fully-parametric deep learning instantiation of our AU-learner.

Title: EVA-S3PC: Efficient, Verifiable, Accurate Secure Matrix Multiplication Protocol Assembly and Its Application in Regression

• Authors: Shizhao Peng, Tianrui Liu, Tianle Tao, Derun Zhao, Hao Sheng, Haogang Zhu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.03404
• Pdf URL: https://arxiv.org/pdf/2411.03404
• Copy Paste: [[2411.03404]] EVA-S3PC: Efficient, Verifiable, Accurate Secure Matrix Multiplication Protocol Assembly and Its Application in Regression(https://arxiv.org/abs/2411.03404)
• Keywords: secure, security, privacy
• Abstract: Efficient multi-party secure matrix multiplication is crucial for privacy-preserving machine learning, but existing mixed-protocol frameworks often face challenges in balancing security, efficiency, and accuracy. This paper presents an efficient, verifiable and accurate secure three-party computing (EVA-S3PC) framework that addresses these challenges with elementary 2-party and 3-party matrix operations based on data obfuscation techniques. We propose basic protocols for secure matrix multiplication, inversion, and hybrid multiplication, ensuring privacy and result verifiability. Experimental results demonstrate that EVA-S3PC achieves up to 14 significant decimal digits of precision in Float64 calculations, while reducing communication overhead by up to $54.8\%$ compared to state of art methods. Furthermore, 3-party regression models trained using EVA-S3PC on vertically partitioned data achieve accuracy nearly identical to plaintext training, which illustrates its potential in scalable, efficient, and accurate solution for secure collaborative modeling across domains.

Title: Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment

• Authors: Alexander Goldberg, Ihsan Ullah, Thanh Gia Hieu Khuong, Benedictus Kent Rachmat, Zhen Xu, Isabelle Guyon, Nihar B. Shah
• Subjects: cs.CL, cs.DL, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.03417
• Pdf URL: https://arxiv.org/pdf/2411.03417
• Copy Paste: [[2411.03417]] Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment(https://arxiv.org/abs/2411.03417)
• Keywords: large language model
• Abstract: Large language models (LLMs) represent a promising, but controversial, tool in aiding scientific peer review. This study evaluates the usefulness of LLMs in a conference setting as a tool for vetting paper submissions against submission standards. We conduct an experiment at the 2024 Neural Information Processing Systems (NeurIPS) conference, where 234 papers were voluntarily submitted to an "LLM-based Checklist Assistant." This assistant validates whether papers adhere to the author checklist used by NeurIPS, which includes questions to ensure compliance with research and manuscript preparation standards. Evaluation of the assistant by NeurIPS paper authors suggests that the LLM-based assistant was generally helpful in verifying checklist completion. In post-usage surveys, over 70% of authors found the assistant useful, and 70% indicate that they would revise their papers or checklist responses based on its feedback. While causal attribution to the assistant is not definitive, qualitative evidence suggests that the LLM contributed to improving some submissions. Survey responses and analysis of re-submissions indicate that authors made substantive revisions to their submissions in response to specific feedback from the LLM. The experiment also highlights common issues with LLMs: inaccuracy (20/52) and excessive strictness (14/52) were the most frequent issues flagged by authors. We also conduct experiments to understand potential gaming of the system, which reveal that the assistant could be manipulated to enhance scores through fabricated justifications, highlighting potential vulnerabilities of automated review tools.

Title: Pathway-Guided Optimization of Deep Generative Molecular Design Models for Cancer Therapy

• Authors: Alif Bin Abdul Qayyum, Susan D. Mertins, Amanda K. Paulson, Nathan M. Urban, Byung-Jun Yoon
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2411.03460
• Pdf URL: https://arxiv.org/pdf/2411.03460
• Copy Paste: [[2411.03460]] Pathway-Guided Optimization of Deep Generative Molecular Design Models for Cancer Therapy(https://arxiv.org/abs/2411.03460)
• Keywords: generative
• Abstract: The data-driven drug design problem can be formulated as an optimization task of a potentially expensive black-box objective function over a huge high-dimensional and structured molecular space. The junction tree variational autoencoder (JTVAE) has been shown to be an efficient generative model that can be used for suggesting legitimate novel drug-like small molecules with improved properties. While the performance of the generative molecular design (GMD) scheme strongly depends on the initial training data, one can improve its sampling efficiency for suggesting better molecules with enhanced properties by optimizing the latent space. In this work, we propose how mechanistic models - such as pathway models described by differential equations - can be used for effective latent space optimization(LSO) of JTVAEs and other similar models for GMD. To demonstrate the potential of our proposed approach, we show how a pharmacodynamic model, assessing the therapeutic efficacy of a drug-like small molecule by predicting how it modulates a cancer pathway, can be incorporated for effective LSO of data-driven models for GMD.

Title: Self Supervised Networks for Learning Latent Space Representations of Human Body Scans and Motions

• Authors: Emmanuel Hartman, Nicolas Charon, Martin Bauer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03475
• Pdf URL: https://arxiv.org/pdf/2411.03475
• Copy Paste: [[2411.03475]] Self Supervised Networks for Learning Latent Space Representations of Human Body Scans and Motions(https://arxiv.org/abs/2411.03475)
• Keywords: robust
• Abstract: This paper introduces self-supervised neural network models to tackle several fundamental problems in the field of 3D human body analysis and processing. First, we propose VariShaPE (Varifold Shape Parameter Estimator), a novel architecture for the retrieval of latent space representations of body shapes and poses. This network offers a fast and robust method to estimate the embedding of arbitrary unregistered meshes into the latent space. Second, we complement the estimation of latent codes with MoGeN (Motion Geometry Network) a framework that learns the geometry on the latent space itself. This is achieved by lifting the body pose parameter space into a higher dimensional Euclidean space in which body motion mini-sequences from a training set of 4D data can be approximated by simple linear interpolation. Using the SMPL latent space representation we illustrate how the combination of these network models, once trained, can be used to perform a variety of tasks with very limited computational cost. This includes operations such as motion interpolation, extrapolation and transfer as well as random shape and pose generation.

Title: Rainfall regression from C-band Synthetic Aperture Radar using Multi-Task Generative Adversarial Networks

• Authors: Aurélien Colin, Romain Husson
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03480
• Pdf URL: https://arxiv.org/pdf/2411.03480
• Copy Paste: [[2411.03480]] Rainfall regression from C-band Synthetic Aperture Radar using Multi-Task Generative Adversarial Networks(https://arxiv.org/abs/2411.03480)
• Keywords: generative
• Abstract: This paper introduces a data-driven approach to estimate precipitation rates from Synthetic Aperture Radar (SAR) at a spatial resolution of 200 meters per pixel. It addresses previous challenges related to the collocation of SAR and weather radar data, specifically the misalignment in collocations and the scarcity of rainfall examples under strong wind. To tackle these challenges, the paper proposes a multi-objective formulation, introducing patch-level components and an adversarial component. It exploits the full NEXRAD archive to look for potential co-locations with Sentinel-1 data. With additional enhancements to the training procedure and the incorporation of additional inputs, the resulting model demonstrates improved accuracy in rainfall estimates and the ability to extend its performance to scenarios up to 15 m/s.

Title: LASER: Attention with Exponential Transformation

• Authors: Sai Surya Duvvuri, Inderjit S. Dhillon
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03493
• Pdf URL: https://arxiv.org/pdf/2411.03493
• Copy Paste: [[2411.03493]] LASER: Attention with Exponential Transformation(https://arxiv.org/abs/2411.03493)
• Keywords: transformer, large language model
• Abstract: Transformers have had tremendous impact for several sequence related tasks, largely due to their ability to retrieve from any part of the sequence via softmax based dot-product attention. This mechanism plays a crucial role in Transformer's performance. We analyze the gradients backpropagated through the softmax operation in the attention mechanism and observe that these gradients can often be small. This poor gradient signal backpropagation can lead to inefficient learning of parameters preceeding the attention operations. To this end, we introduce a new attention mechanism called LASER, which we analytically show to admit a larger gradient signal. We show that LASER Attention can be implemented by making small modifications to existing attention implementations. We conduct experiments on autoregressive large language models (LLMs) with upto 2.2 billion parameters where we show upto 3.38% and an average of ~1% improvement over standard attention on downstream evaluations. Using LASER gives the following relative improvements in generalization performance across a variety of tasks (vision, text and speech): 4.67% accuracy in Vision Transformer (ViT) on Imagenet, 2.25% error rate in Conformer on the Librispeech speech-to-text and 0.93% fraction of incorrect predictions in BERT with 2.2 billion parameters.

Title: Automatic Generation of Question Hints for Mathematics Problems using Large Language Models in Educational Technology

• Authors: Junior Cedric Tonga, Benjamin Clement, Pierre-Yves Oudeyer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03495
• Pdf URL: https://arxiv.org/pdf/2411.03495
• Copy Paste: [[2411.03495]] Automatic Generation of Question Hints for Mathematics Problems using Large Language Models in Educational Technology(https://arxiv.org/abs/2411.03495)
• Keywords: large language model
• Abstract: The automatic generation of hints by Large Language Models (LLMs) within Intelligent Tutoring Systems (ITSs) has shown potential to enhance student learning. However, generating pedagogically sound hints that address student misconceptions and adhere to specific educational objectives remains challenging. This work explores using LLMs (GPT-4o and Llama-3-8B-instruct) as teachers to generate effective hints for students simulated through LLMs (GPT-3.5-turbo, Llama-3-8B-Instruct, or Mistral-7B-instruct-v0.3) tackling math exercises designed for human high-school students, and designed using cognitive science principles. We present here the study of several dimensions: 1) identifying error patterns made by simulated students on secondary-level math exercises; 2) developing various prompts for GPT-4o as a teacher and evaluating their effectiveness in generating hints that enable simulated students to self-correct; and 3) testing the best-performing prompts, based on their ability to produce relevant hints and facilitate error correction, with Llama-3-8B-Instruct as the teacher, allowing for a performance comparison with GPT-4o. The results show that model errors increase with higher temperature settings. Notably, when hints are generated by GPT-4o, the most effective prompts include prompts tailored to specific errors as well as prompts providing general hints based on common mathematical errors. Interestingly, Llama-3-8B-Instruct as a teacher showed better overall performance than GPT-4o. Also the problem-solving and response revision capabilities of the LLMs as students, particularly GPT-3.5-turbo, improved significantly after receiving hints, especially at lower temperature settings. However, models like Mistral-7B-Instruct demonstrated a decline in performance as the temperature increased.

Title: SynthSet: Generative Diffusion Model for Semantic Segmentation in Precision Agriculture

• Authors: Andrew Heschl, Mauricio Murillo, Keyhan Najafian, Farhad Maleki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03505
• Pdf URL: https://arxiv.org/pdf/2411.03505
• Copy Paste: [[2411.03505]] SynthSet: Generative Diffusion Model for Semantic Segmentation in Precision Agriculture(https://arxiv.org/abs/2411.03505)
• Keywords: diffusion, generative, segmentation
• Abstract: This paper introduces a methodology for generating synthetic annotated data to address data scarcity in semantic segmentation tasks within the precision agriculture domain. Utilizing Denoising Diffusion Probabilistic Models (DDPMs) and Generative Adversarial Networks (GANs), we propose a dual diffusion model architecture for synthesizing realistic annotated agricultural data, without any human intervention. We employ super-resolution to enhance the phenotypic characteristics of the synthesized images and their coherence with the corresponding generated masks. We showcase the utility of the proposed method for wheat head segmentation. The high quality of synthesized data underscores the effectiveness of the proposed methodology in generating image-mask pairs. Furthermore, models trained on our generated data exhibit promising performance when tested on an external, diverse dataset of real wheat fields. The results show the efficacy of the proposed methodology for addressing data scarcity for semantic segmentation tasks. Moreover, the proposed approach can be readily adapted for various segmentation tasks in precision agriculture and beyond.

Title: Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy

• Authors: Razvan-Gabriel Dumitru, Paul-Ioan Clotan, Vikas Yadav, Darius Peteleaza, Mihai Surdeanu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03513
• Pdf URL: https://arxiv.org/pdf/2411.03513
• Copy Paste: [[2411.03513]] Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy(https://arxiv.org/abs/2411.03513)
• Keywords: large language model
• Abstract: This paper introduces a novel model compression approach through dynamic layer-specific pruning in Large Language Models (LLMs), enhancing the traditional methodology established by SliceGPT. By transitioning from constant to dynamic slicing, our method leverages the newly proposed Layer Redundancy (LR) score, which assesses how much change each layer changes its input by measuring the cosine similarity of the input to the output of the layer. We use this score to prune parts of individual layers based on redundancy in such a way that the average pruned percentage for all layers is a fixed value. We conducted extensive experiments using models like Llama3-8B and Mistral-7B on multiple datasets, evaluating different slicing bases and percentages to determine optimal configurations that balance efficiency and performance. Our findings show that our dynamic slicing approach not only maintains but, in many cases, enhances model performance compared to the baseline established by constant slicing methods. For instance, in several settings, we see performance improvements of up to 5% over the SliceGPT baseline. Additionally, a perplexity decrease by as much as 7% was observed across multiple benchmarks, validating the effectiveness of our method. The code, model weights, and datasets are open-sourced at this https URL.

Title: Long Context RAG Performance of Large Language Models

• Authors: Quinn Leng, Jacob Portes, Sam Havens, Matei Zaharia, Michael Carbin
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03538
• Pdf URL: https://arxiv.org/pdf/2411.03538
• Copy Paste: [[2411.03538]] Long Context RAG Performance of Large Language Models(https://arxiv.org/abs/2411.03538)
• Keywords: large language model
• Abstract: Retrieval Augmented Generation (RAG) has emerged as a crucial technique for enhancing the accuracy of Large Language Models (LLMs) by incorporating external information. With the advent of LLMs that support increasingly longer context lengths, there is a growing interest in understanding how these models perform in RAG scenarios. Can these new long context models improve RAG performance? This paper presents a comprehensive study of the impact of increased context length on RAG performance across 20 popular open source and commercial LLMs. We ran RAG workflows while varying the total context length from 2,000 to 128,000 tokens (and 2 million tokens when possible) on three domain-specific datasets, and report key insights on the benefits and limitations of long context in RAG applications. Our findings reveal that while retrieving more documents can improve performance, only a handful of the most recent state of the art LLMs can maintain consistent accuracy at long context above 64k tokens. We also identify distinct failure modes in long context scenarios, suggesting areas for future research.

Title: Do Mice Grok? Glimpses of Hidden Progress During Overtraining in Sensory Cortex

• Authors: Tanishq Kumar, Blake Bordelon, Cengiz Pehlevan, Venkatesh N. Murthy, Samuel J. Gershman
• Subjects: cs.LG, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2411.03541
• Pdf URL: https://arxiv.org/pdf/2411.03541
• Copy Paste: [[2411.03541]] Do Mice Grok? Glimpses of Hidden Progress During Overtraining in Sensory Cortex(https://arxiv.org/abs/2411.03541)
• Keywords: robust
• Abstract: Does learning of task-relevant representations stop when behavior stops changing? Motivated by recent theoretical advances in machine learning and the intuitive observation that human experts continue to learn from practice even after mastery, we hypothesize that task-specific representation learning can continue, even when behavior plateaus. In a novel reanalysis of recently published neural data, we find evidence for such learning in posterior piriform cortex of mice following continued training on a task, long after behavior saturates at near-ceiling performance ("overtraining"). This learning is marked by an increase in decoding accuracy from piriform neural populations and improved performance on held-out generalization tests. We demonstrate that class representations in cortex continue to separate during overtraining, so that examples that were incorrectly classified at the beginning of overtraining can abruptly be correctly classified later on, despite no changes in behavior during that time. We hypothesize this hidden yet rich learning takes the form of approximate margin maximization; we validate this and other predictions in the neural data, as well as build and interpret a simple synthetic model that recapitulates these phenomena. We conclude by showing how this model of late-time feature learning implies an explanation for the empirical puzzle of overtraining reversal in animal learning, where task-specific representations are more robust to particular task changes because the learned features can be reused.

Title: Exploring the Benefits of Domain-Pretraining of Generative Large Language Models for Chemistry

• Authors: Anurag Acharya, Shivam Sharma, Robin Cosbey, Megha Subramanian, Scott Howland, Maria Glenski
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03542
• Pdf URL: https://arxiv.org/pdf/2411.03542
• Copy Paste: [[2411.03542]] Exploring the Benefits of Domain-Pretraining of Generative Large Language Models for Chemistry(https://arxiv.org/abs/2411.03542)
• Keywords: generative, large language model
• Abstract: A proliferation of Large Language Models (the GPT series, BLOOM, LLaMA, and more) are driving forward novel development of multipurpose AI for a variety of tasks, particularly natural language processing (NLP) tasks. These models demonstrate strong performance on a range of tasks; however, there has been evidence of brittleness when applied to more niche or narrow domains where hallucinations or fluent but incorrect responses reduce performance. Given the complex nature of scientific domains, it is prudent to investigate the trade-offs of leveraging off-the-shelf versus more targeted foundation models for scientific domains. In this work, we examine the benefits of in-domain pre-training for a given scientific domain, chemistry, and compare these to open-source, off-the-shelf models with zero-shot and few-shot prompting. Our results show that not only do in-domain base models perform reasonably well on in-domain tasks in a zero-shot setting but that further adaptation using instruction fine-tuning yields impressive performance on chemistry-specific tasks such as named entity recognition and molecular formula generation.

Title: Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset

• Authors: Yingzi Ma, Jiongxiao Wang, Fei Wang, Siyuan Ma, Jiazhao Li, Xiujun Li, Furong Huang, Lichao Sun, Bo Li, Yejin Choi, Muhao Chen, Chaowei Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03554
• Pdf URL: https://arxiv.org/pdf/2411.03554
• Copy Paste: [[2411.03554]] Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset(https://arxiv.org/abs/2411.03554)
• Keywords: privacy, attack, robust, membership infer
• Abstract: Machine unlearning has emerged as an effective strategy for forgetting specific information in the training data. However, with the increasing integration of visual data, privacy concerns in Vision Language Models (VLMs) remain underexplored. To address this, we introduce Facial Identity Unlearning Benchmark (FIUBench), a novel VLM unlearning benchmark designed to robustly evaluate the effectiveness of unlearning algorithms under the Right to be Forgotten setting. Specifically, we formulate the VLM unlearning task via constructing the Fictitious Facial Identity VQA dataset and apply a two-stage evaluation pipeline that is designed to precisely control the sources of information and their exposure levels. In terms of evaluation, since VLM supports various forms of ways to ask questions with the same semantic meaning, we also provide robust evaluation metrics including membership inference attacks and carefully designed adversarial privacy attacks to evaluate the performance of algorithms. Through the evaluation of four baseline VLM unlearning algorithms within FIUBench, we find that all methods remain limited in their unlearning performance, with significant trade-offs between model utility and forget quality. Furthermore, our findings also highlight the importance of privacy attacks for robust evaluations. We hope FIUBench will drive progress in developing more effective VLM unlearning algorithms.

Title: Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data

• Authors: Seunggeun Chi, Pin-Hao Huang, Enna Sachdeva, Hengbo Ma, Karthik Ramani, Kwonjoon Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03561
• Pdf URL: https://arxiv.org/pdf/2411.03561
• Copy Paste: [[2411.03561]] Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data(https://arxiv.org/abs/2411.03561)
• Keywords: diffusion
• Abstract: We study the problem of estimating the body movements of a camera wearer from egocentric videos. Current methods for ego-body pose estimation rely on temporally dense sensor data, such as IMU measurements from spatially sparse body parts like the head and hands. However, we propose that even temporally sparse observations, such as hand poses captured intermittently from egocentric videos during natural or periodic hand movements, can effectively constrain overall body motion. Naively applying diffusion models to generate full-body pose from head pose and sparse hand pose leads to suboptimal results. To overcome this, we develop a two-stage approach that decomposes the problem into temporal completion and spatial completion. First, our method employs masked autoencoders to impute hand trajectories by leveraging the spatiotemporal correlations between the head pose sequence and intermittent hand poses, providing uncertainty estimates. Subsequently, we employ conditional diffusion models to generate plausible full-body motions based on these temporally dense trajectories of the head and hands, guided by the uncertainty estimates from the imputation. The effectiveness of our method was rigorously tested and validated through comprehensive experiments conducted on various HMD setup with AMASS and Ego-Exo4D datasets.

Title: Large Language Models Orchestrating Structured Reasoning Achieve Kaggle Grandmaster Level

• Authors: Antoine Grosnit, Alexandre Maraval, James Doran, Giuseppe Paolo, Albert Thomas, Refinath Shahul Hameed Nabeezath Beevi, Jonas Gonzalez, Khyati Khandelwal, Ignacio Iacobacci, Abdelhakim Benechehab, Hamza Cherkaoui, Youssef Attia El-Hili, Kun Shao, Jianye Hao, Jun Yao, Balazs Kegl, Haitham Bou-Ammar, Jun Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03562
• Pdf URL: https://arxiv.org/pdf/2411.03562
• Copy Paste: [[2411.03562]] Large Language Models Orchestrating Structured Reasoning Achieve Kaggle Grandmaster Level(https://arxiv.org/abs/2411.03562)
• Keywords: large language model
• Abstract: We introduce Agent K v1.0, an end-to-end autonomous data science agent designed to automate, optimise, and generalise across diverse data science tasks. Fully automated, Agent K v1.0 manages the entire data science life cycle by learning from experience. It leverages a highly flexible structured reasoning framework to enable it to dynamically process memory in a nested structure, effectively learning from accumulated experience stored to handle complex reasoning tasks. It optimises long- and short-term memory by selectively storing and retrieving key information, guiding future decisions based on environmental rewards. This iterative approach allows it to refine decisions without fine-tuning or backpropagation, achieving continuous improvement through experiential learning. We evaluate our agent's apabilities using Kaggle competitions as a case study. Following a fully automated protocol, Agent K v1.0 systematically addresses complex and multimodal data science tasks, employing Bayesian optimisation for hyperparameter tuning and feature engineering. Our new evaluation framework rigorously assesses Agent K v1.0's end-to-end capabilities to generate and send submissions starting from a Kaggle competition URL. Results demonstrate that Agent K v1.0 achieves a 92.5\% success rate across tasks, spanning tabular, computer vision, NLP, and multimodal domains. When benchmarking against 5,856 human Kaggle competitors by calculating Elo-MMR scores for each, Agent K v1.0 ranks in the top 38\%, demonstrating an overall skill level comparable to Expert-level users. Notably, its Elo-MMR score falls between the first and third quartiles of scores achieved by human Grandmasters. Furthermore, our results indicate that Agent K v1.0 has reached a performance level equivalent to Kaggle Grandmaster, with a record of 6 gold, 3 silver, and 7 bronze medals, as defined by Kaggle's progression system.

Title: The American Sign Language Knowledge Graph: Infusing ASL Models with Linguistic Knowledge

• Authors: Lee Kezar, Nidhi Munikote, Zian Zeng, Zed Sehyr, Naomi Caselli, Jesse Thomason
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03568
• Pdf URL: https://arxiv.org/pdf/2411.03568
• Copy Paste: [[2411.03568]] The American Sign Language Knowledge Graph: Infusing ASL Models with Linguistic Knowledge(https://arxiv.org/abs/2411.03568)
• Keywords: explainability
• Abstract: Language models for American Sign Language (ASL) could make language technologies substantially more accessible to those who sign. To train models on tasks such as isolated sign recognition (ISR) and ASL-to-English translation, datasets provide annotated video examples of ASL signs. To facilitate the generalizability and explainability of these models, we introduce the American Sign Language Knowledge Graph (ASLKG), compiled from twelve sources of expert linguistic knowledge. We use the ASLKG to train neuro-symbolic models for 3 ASL understanding tasks, achieving accuracies of 91% on ISR, 14% for predicting the semantic features of unseen signs, and 36% for classifying the topic of Youtube-ASL videos.

Title: Towards Personalized Federated Learning via Comprehensive Knowledge Distillation

• Authors: Pengju Wang, Bochao Liu, Weijia Guo, Yong Li, Shiming Ge
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03569
• Pdf URL: https://arxiv.org/pdf/2411.03569
• Copy Paste: [[2411.03569]] Towards Personalized Federated Learning via Comprehensive Knowledge Distillation(https://arxiv.org/abs/2411.03569)
• Keywords: privacy, protect, federate
• Abstract: Federated learning is a distributed machine learning paradigm designed to protect data privacy. However, data heterogeneity across various clients results in catastrophic forgetting, where the model rapidly forgets previous knowledge while acquiring new knowledge. To address this challenge, personalized federated learning has emerged to customize a personalized model for each client. However, the inherent limitation of this mechanism is its excessive focus on personalization, potentially hindering the generalization of those models. In this paper, we present a novel personalized federated learning method that uses global and historical models as teachers and the local model as the student to facilitate comprehensive knowledge distillation. The historical model represents the local model from the last round of client training, containing historical personalized knowledge, while the global model represents the aggregated model from the last round of server aggregation, containing global generalized knowledge. By applying knowledge distillation, we effectively transfer global generalized knowledge and historical personalized knowledge to the local model, thus mitigating catastrophic forgetting and enhancing the general performance of personalized models. Extensive experimental results demonstrate the significant advantages of our method.

Title: Hybrid Attention for Robust RGB-T Pedestrian Detection in Real-World Conditions

• Authors: Arunkumar Rathinam, Leo Pauly, Abd El Rahman Shabayek, Wassim Rharbaoui, Anis Kacem, Vincent Gaudillière, Djamila Aouada
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03576
• Pdf URL: https://arxiv.org/pdf/2411.03576
• Copy Paste: [[2411.03576]] Hybrid Attention for Robust RGB-T Pedestrian Detection in Real-World Conditions(https://arxiv.org/abs/2411.03576)
• Keywords: robust
• Abstract: Multispectral pedestrian detection has gained significant attention in recent years, particularly in autonomous driving applications. To address the challenges posed by adversarial illumination conditions, the combination of thermal and visible images has demonstrated its advantages. However, existing fusion methods rely on the critical assumption that the RGB-Thermal (RGB-T) image pairs are fully overlapping. These assumptions often do not hold in real-world applications, where only partial overlap between images can occur due to sensors configuration. Moreover, sensor failure can cause loss of information in one modality. In this paper, we propose a novel module called the Hybrid Attention (HA) mechanism as our main contribution to mitigate performance degradation caused by partial overlap and sensor failure, i.e. when at least part of the scene is acquired by only one sensor. We propose an improved RGB-T fusion algorithm, robust against partial overlap and sensor failure encountered during inference in real-world applications. We also leverage a mobile-friendly backbone to cope with resource constraints in embedded systems. We conducted experiments by simulating various partial overlap and sensor failure scenarios to evaluate the performance of our proposed method. The results demonstrate that our approach outperforms state-of-the-art methods, showcasing its superiority in handling real-world challenges.

Title: From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond

• Authors: Harsha Nori, Naoto Usuyama, Nicholas King, Scott Mayer McKinney, Xavier Fernandes, Sheng Zhang, Eric Horvitz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03590
• Pdf URL: https://arxiv.org/pdf/2411.03590
• Copy Paste: [[2411.03590]] From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond(https://arxiv.org/abs/2411.03590)
• Keywords: large language model
• Abstract: Run-time steering strategies like Medprompt are valuable for guiding large language models (LLMs) to top performance on challenging tasks. Medprompt demonstrates that a general LLM can be focused to deliver state-of-the-art performance on specialized domains like medicine by using a prompt to elicit a run-time strategy involving chain of thought reasoning and ensembling. OpenAI's o1-preview model represents a new paradigm, where a model is designed to do run-time reasoning before generating final responses. We seek to understand the behavior of o1-preview on a diverse set of medical challenge problem benchmarks. Following on the Medprompt study with GPT-4, we systematically evaluate the o1-preview model across various medical benchmarks. Notably, even without prompting techniques, o1-preview largely outperforms the GPT-4 series with Medprompt. We further systematically study the efficacy of classic prompt engineering strategies, as represented by Medprompt, within the new paradigm of reasoning models. We found that few-shot prompting hinders o1's performance, suggesting that in-context learning may no longer be an effective steering approach for reasoning-native models. While ensembling remains viable, it is resource-intensive and requires careful cost-performance optimization. Our cost and accuracy analysis across run-time strategies reveals a Pareto frontier, with GPT-4o representing a more affordable option and o1-preview achieving state-of-the-art performance at higher cost. Although o1-preview offers top performance, GPT-4o with steering strategies like Medprompt retains value in specific contexts. Moreover, we note that the o1-preview model has reached near-saturation on many existing medical benchmarks, underscoring the need for new, challenging benchmarks. We close with reflections on general directions for inference-time computation with LLMs.

Title: Open-Source High-Speed Flight Surrogate Modeling Framework

• Authors: Tyler E. Korenyi-Both, Nathan J. Falkiewicz, Matthew C. Jones
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03598
• Pdf URL: https://arxiv.org/pdf/2411.03598
• Copy Paste: [[2411.03598]] Open-Source High-Speed Flight Surrogate Modeling Framework(https://arxiv.org/abs/2411.03598)
• Keywords: defense, robust
• Abstract: High-speed flight vehicles, which travel much faster than the speed of sound, are crucial for national defense and space exploration. However, accurately predicting their behavior under numerous, varied flight conditions is a challenge and often prohibitively expensive. The proposed approach involves creating smarter, more efficient machine learning models (also known as surrogate models or meta models) that can fuse data generated from a variety of fidelity levels -- to include engineering methods, simulation, wind tunnel, and flight test data -- to make more accurate predictions. These models are able to move the bulk of the computation from high performance computing (HPC) to single user machines (laptop, desktop, etc.). The project builds upon previous work but introduces code improvements and an informed perspective on the direction of the field. The new surrogate modeling framework is now modular and, by design, broadly applicable to many modeling problems. The new framework also has a more robust automatic hyperparameter tuning capability and abstracts away most of the pre- and post-processing tasks. The Gaussian process regression and deep neural network-based models included in the presented framework were able to model two datasets with high accuracy (R^2>0.99). The primary conclusion is that the framework is effective and has been delivered to the Air Force for integration into real-world projects. For future work, significant and immediate investment in continued research is crucial. The author recommends further testing and refining modeling methods that explicitly incorporate physical laws and are robust enough to handle simulation and test data from varying resolutions and sources, including coarse meshes, fine meshes, unstructured meshes, and limited experimental test points.

Title: StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding

• Authors: Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, Maosong Sun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03628
• Pdf URL: https://arxiv.org/pdf/2411.03628
• Copy Paste: [[2411.03628]] StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding(https://arxiv.org/abs/2411.03628)
• Keywords: large language model
• Abstract: The rapid development of Multimodal Large Language Models (MLLMs) has expanded their capabilities from image comprehension to video understanding. However, most of these MLLMs focus primarily on offline video comprehension, necessitating extensive processing of all video frames before any queries can be made. This presents a significant gap compared to the human ability to watch, listen, think, and respond to streaming inputs in real time, highlighting the limitations of current MLLMs. In this paper, we introduce StreamingBench, the first comprehensive benchmark designed to evaluate the streaming video understanding capabilities of MLLMs. StreamingBench assesses three core aspects of streaming video understanding: (1) real-time visual understanding, (2) omni-source understanding, and (3) contextual understanding. The benchmark consists of 18 tasks, featuring 900 videos and 4,500 human-curated QA pairs. Each video features five questions presented at different time points to simulate a continuous streaming scenario. We conduct experiments on StreamingBench with 13 open-source and proprietary MLLMs and find that even the most advanced proprietary MLLMs like Gemini 1.5 Pro and GPT-4o perform significantly below human-level streaming video understanding capabilities. We hope our work can facilitate further advancements for MLLMs, empowering them to approach human-level video comprehension and interaction in more realistic scenarios.

Title: Adaptive Stereo Depth Estimation with Multi-Spectral Images Across All Lighting Conditions

• Authors: Zihan Qin, Jialei Xu, Wenbo Zhao, Junjun Jiang, Xianming Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03638
• Pdf URL: https://arxiv.org/pdf/2411.03638
• Copy Paste: [[2411.03638]] Adaptive Stereo Depth Estimation with Multi-Spectral Images Across All Lighting Conditions(https://arxiv.org/abs/2411.03638)
• Keywords: robust
• Abstract: Depth estimation under adverse conditions remains a significant challenge. Recently, multi-spectral depth estimation, which integrates both visible light and thermal images, has shown promise in addressing this issue. However, existing algorithms struggle with precise pixel-level feature matching, limiting their ability to fully exploit geometric constraints across different spectra. To address this, we propose a novel framework incorporating stereo depth estimation to enforce accurate geometric constraints. In particular, we treat the visible light and thermal images as a stereo pair and utilize a Cross-modal Feature Matching (CFM) Module to construct a cost volume for pixel-level matching. To mitigate the effects of poor lighting on stereo matching, we introduce Degradation Masking, which leverages robust monocular thermal depth estimation in degraded regions. Our method achieves state-of-the-art (SOTA) performance on the Multi-Spectral Stereo (MS2) dataset, with qualitative evaluations demonstrating high-quality depth maps under varying lighting conditions.

Title: Deploying Multi-task Online Server with Large Language Model

• Authors: Yincen Qu, Chao Ma, Yiting Wu, Xiangying Dai, Hui Zhou, Hengyue Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03644
• Pdf URL: https://arxiv.org/pdf/2411.03644
• Copy Paste: [[2411.03644]] Deploying Multi-task Online Server with Large Language Model(https://arxiv.org/abs/2411.03644)
• Keywords: large language model
• Abstract: In the industry, numerous tasks are deployed online. Traditional approaches often tackle each task separately by its own network, which leads to excessive costs for developing and scaling models, especially in the context of large language models. Although multi-task methods can save costs through parameter sharing, they often struggle to outperform single-task methods in real-world applications. To tackle these challenges, we present a three-stage multi-task learning framework for large language models. It involves task filtering, followed by fine-tuning on high-resource tasks, and finally fine-tuning on all tasks. We conducted comprehensive experiments in single-task and multi-task settings. Our approach, exemplified on different benchmarks, demonstrates that it is able to achieve performance comparable to the single-task method while reducing up to 90.9\% of its overhead.

Title: Can Graph Neural Networks Expose Training Data Properties? An Efficient Risk Assessment Approach

• Authors: Hanyang Yuan, Jiarong Xu, Renhong Huang, Mingli Song, Chunping Wang, Yang Yang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.03663
• Pdf URL: https://arxiv.org/pdf/2411.03663
• Copy Paste: [[2411.03663]] Can Graph Neural Networks Expose Training Data Properties? An Efficient Risk Assessment Approach(https://arxiv.org/abs/2411.03663)
• Keywords: privacy, attack
• Abstract: Graph neural networks (GNNs) have attracted considerable attention due to their diverse applications. However, the scarcity and quality limitations of graph data present challenges to their training process in practical settings. To facilitate the development of effective GNNs, companies and researchers often seek external collaboration. Yet, directly sharing data raises privacy concerns, motivating data owners to train GNNs on their private graphs and share the trained models. Unfortunately, these models may still inadvertently disclose sensitive properties of their training graphs (e.g., average default rate in a transaction network), leading to severe consequences for data owners. In this work, we study graph property inference attack to identify the risk of sensitive property information leakage from shared models. Existing approaches typically train numerous shadow models for developing such attack, which is computationally intensive and impractical. To address this issue, we propose an efficient graph property inference attack by leveraging model approximation techniques. Our method only requires training a small set of models on graphs, while generating a sufficient number of approximated shadow models for attacks. To enhance diversity while reducing errors in the approximated models, we apply edit distance to quantify the diversity within a group of approximated models and introduce a theoretically guaranteed criterion to evaluate each model's error. Subsequently, we propose a novel selection mechanism to ensure that the retained approximated models achieve high diversity and low error. Extensive experiments across six real-world scenarios demonstrate our method's substantial improvement, with average increases of 2.7% in attack accuracy and 4.1% in ROC-AUC, while being 6.5$\times$ faster compared to the best baseline.

Title: Evaluating Moral Beliefs across LLMs through a Pluralistic Framework

• Authors: Xuelin Liu, Yanfei Zhu, Shucheng Zhu, Pengyuan Liu, Ying Liu, Dong Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03665
• Pdf URL: https://arxiv.org/pdf/2411.03665
• Copy Paste: [[2411.03665]] Evaluating Moral Beliefs across LLMs through a Pluralistic Framework(https://arxiv.org/abs/2411.03665)
• Keywords: large language model
• Abstract: Proper moral beliefs are fundamental for language models, yet assessing these beliefs poses a significant challenge. This study introduces a novel three-module framework to evaluate the moral beliefs of four prominent large language models. Initially, we constructed a dataset containing 472 moral choice scenarios in Chinese, derived from moral words. The decision-making process of the models in these scenarios reveals their moral principle preferences. By ranking these moral choices, we discern the varying moral beliefs held by different language models. Additionally, through moral debates, we investigate the firmness of these models to their moral choices. Our findings indicate that English language models, namely ChatGPT and Gemini, closely mirror moral decisions of the sample of Chinese university students, demonstrating strong adherence to their choices and a preference for individualistic moral beliefs. In contrast, Chinese models such as Ernie and ChatGLM lean towards collectivist moral beliefs, exhibiting ambiguity in their moral choices and debates. This study also uncovers gender bias embedded within the moral beliefs of all examined language models. Our methodology offers an innovative means to assess moral beliefs in both artificial and human intelligence, facilitating a comparison of moral values across different cultures.

Title: Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?

• Authors: Pedro R. A. S. Bassi, Wenxuan Li, Yucheng Tang, Fabian Isensee, Zifu Wang, Jieneng Chen, Yu-Cheng Chou, Yannick Kirchhoff, Maximilian Rokuss, Ziyan Huang, Jin Ye, Junjun He, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus H. Maier-Hein, Paul Jaeger, Yiwen Ye, Yutong Xie, Jianpeng Zhang, Ziyang Chen, Yong Xia, Zhaohu Xing, Lei Zhu, Yousef Sadegheih, Afshin Bozorgpour, Pratibha Kumari, Reza Azad, Dorit Merhof, Pengcheng Shi, Ting Ma, Yuxin Du, Fan Bai, Tiejun Huang, Bo Zhao, Haonan Wang, Xiaomeng Li, Hanxue Gu, Haoyu Dong, Jichen Yang, Maciej A. Mazurowski, Saumya Gupta, Linshan Wu, Jiaxin Zhuang, Hao Chen, Holger Roth, Daguang Xu, Matthew B. Blaschko, Sergio Decherchi, Andrea Cavalli, Alan L. Yuille, Zongwei Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03670
• Pdf URL: https://arxiv.org/pdf/2411.03670
• Copy Paste: [[2411.03670]] Touchstone Benchmark: Are We on the Right Way for Evaluating AI Algorithms for Medical Segmentation?(https://arxiv.org/abs/2411.03670)
• Keywords: fair, segmentation
• Abstract: How can we test AI performance? This question seems trivial, but it isn't. Standard benchmarks often have problems such as in-distribution and small-size test sets, oversimplified metrics, unfair comparisons, and short-term outcome pressure. As a consequence, good performance on standard benchmarks does not guarantee success in real-world scenarios. To address these problems, we present Touchstone, a large-scale collaborative segmentation benchmark of 9 types of abdominal organs. This benchmark is based on 5,195 training CT scans from 76 hospitals around the world and 5,903 testing CT scans from 11 additional hospitals. This diverse test set enhances the statistical significance of benchmark results and rigorously evaluates AI algorithms across various out-of-distribution scenarios. We invited 14 inventors of 19 AI algorithms to train their algorithms, while our team, as a third party, independently evaluated these algorithms on three test sets. In addition, we also evaluated pre-existing AI frameworks--which, differing from algorithms, are more flexible and can support different algorithms--including MONAI from NVIDIA, nnU-Net from DKFZ, and numerous other open-source frameworks. We are committed to expanding this benchmark to encourage more innovation of AI algorithms for the medical domain.

Title: Towards 3D Semantic Scene Completion for Autonomous Driving: A Meta-Learning Framework Empowered by Deformable Large-Kernel Attention and Mamba Model

• Authors: Yansong Qu, Zilin Huang, Zihao Sheng, Tiantian Chen, Sikai Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03672
• Pdf URL: https://arxiv.org/pdf/2411.03672
• Copy Paste: [[2411.03672]] Towards 3D Semantic Scene Completion for Autonomous Driving: A Meta-Learning Framework Empowered by Deformable Large-Kernel Attention and Mamba Model(https://arxiv.org/abs/2411.03672)
• Keywords: segmentation
• Abstract: Semantic scene completion (SSC) is essential for achieving comprehensive perception in autonomous driving systems. However, existing SSC methods often overlook the high deployment costs in real-world applications. Traditional architectures, such as 3D Convolutional Neural Networks (3D CNNs) and self-attention mechanisms, face challenges in efficiently capturing long-range dependencies within 3D voxel grids, limiting their effectiveness. To address these issues, we introduce MetaSSC, a novel meta-learning-based framework for SSC that leverages deformable convolution, large-kernel attention, and the Mamba (D-LKA-M) model. Our approach begins with a voxel-based semantic segmentation (SS) pretraining task, aimed at exploring the semantics and geometry of incomplete regions while acquiring transferable meta-knowledge. Using simulated cooperative perception datasets, we supervise the perception training of a single vehicle using aggregated sensor data from multiple nearby connected autonomous vehicles (CAVs), generating richer and more comprehensive labels. This meta-knowledge is then adapted to the target domain through a dual-phase training strategy that does not add extra model parameters, enabling efficient deployment. To further enhance the model's capability in capturing long-sequence relationships within 3D voxel grids, we integrate Mamba blocks with deformable convolution and large-kernel attention into the backbone network. Extensive experiments demonstrate that MetaSSC achieves state-of-the-art performance, significantly outperforming competing models while also reducing deployment costs.

Title: QUILL: Quotation Generation Enhancement of Large Language Models

• Authors: Jin Xiao, Bowei Zhang, Qianyu He, Jiaqing Liang, Feng Wei, Jinglei Chen, Zujie Liang, Deqing Yang, Yanghua Xiao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03675
• Pdf URL: https://arxiv.org/pdf/2411.03675
• Copy Paste: [[2411.03675]] QUILL: Quotation Generation Enhancement of Large Language Models(https://arxiv.org/abs/2411.03675)
• Keywords: large language model
• Abstract: While Large language models (LLMs) have become excellent writing assistants, they still struggle with quotation generation. This is because they either hallucinate when providing factual quotations or fail to provide quotes that exceed human expectations. To bridge the gap, we systematically study how to evaluate and improve LLMs' performance in quotation generation tasks. We first establish a holistic and automatic evaluation system for quotation generation task, which consists of five criteria each with corresponding automatic metric. To improve the LLMs' quotation generation abilities, we construct a bilingual knowledge base that is broad in scope and rich in dimensions, containing up to 32,022 quotes. Moreover, guided by our critiria, we further design a quotation-specific metric to rerank the retrieved quotations from the knowledge base. Extensive experiments show that our metrics strongly correlate with human preferences. Existing LLMs struggle to generate desired quotes, but our quotation knowledge base and reranking metric help narrow this gap. Our dataset and code are publicly available at this https URL.

Title: AMNCutter: Affinity-Attention-Guided Multi-View Normalized Cutter for Unsupervised Surgical Instrument Segmentation

• Authors: Mingyu Sheng, Jianan Fan, Dongnan Liu, Ron Kikinis, Weidong Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03695
• Pdf URL: https://arxiv.org/pdf/2411.03695
• Copy Paste: [[2411.03695]] AMNCutter: Affinity-Attention-Guided Multi-View Normalized Cutter for Unsupervised Surgical Instrument Segmentation(https://arxiv.org/abs/2411.03695)
• Keywords: robust, segmentation
• Abstract: Surgical instrument segmentation (SIS) is pivotal for robotic-assisted minimally invasive surgery, assisting surgeons by identifying surgical instruments in endoscopic video frames. Recent unsupervised surgical instrument segmentation (USIS) methods primarily rely on pseudo-labels derived from low-level features such as color and optical flow, but these methods show limited effectiveness and generalizability in complex and unseen endoscopic scenarios. In this work, we propose a label-free unsupervised model featuring a novel module named Multi-View Normalized Cutter (m-NCutter). Different from previous USIS works, our model is trained using a graph-cutting loss function that leverages patch affinities for supervision, eliminating the need for pseudo-labels. The framework adaptively determines which affinities from which levels should be prioritized. Therefore, the low- and high-level features and their affinities are effectively integrated to train a label-free unsupervised model, showing superior effectiveness and generalization ability. We conduct comprehensive experiments across multiple SIS datasets to validate our approach's state-of-the-art (SOTA) performance, robustness, and exceptional potential as a pre-trained model. Our code is released at this https URL.

Title: Graph-Based Multi-Modal Sensor Fusion for Autonomous Driving

• Authors: Depanshu Sani, Saket Anand
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.03702
• Pdf URL: https://arxiv.org/pdf/2411.03702
• Copy Paste: [[2411.03702]] Graph-Based Multi-Modal Sensor Fusion for Autonomous Driving(https://arxiv.org/abs/2411.03702)
• Keywords: robust
• Abstract: The growing demand for robust scene understanding in mobile robotics and autonomous driving has highlighted the importance of integrating multiple sensing modalities. By combining data from diverse sensors like cameras and LIDARs, fusion techniques can overcome the limitations of individual sensors, enabling a more complete and accurate perception of the environment. We introduce a novel approach to multi-modal sensor fusion, focusing on developing a graph-based state representation that supports critical decision-making processes in autonomous driving. We present a Sensor-Agnostic Graph-Aware Kalman Filter [3], the first online state estimation technique designed to fuse multi-modal graphs derived from noisy multi-sensor data. The estimated graph-based state representations serve as a foundation for advanced applications like Multi-Object Tracking (MOT), offering a comprehensive framework for enhancing the situational awareness and safety of autonomous systems. We validate the effectiveness of our proposed framework through extensive experiments conducted on both synthetic and real-world driving datasets (nuScenes). Our results showcase an improvement in MOTA and a reduction in estimated position errors (MOTP) and identity switches (IDS) for tracked objects using the SAGA-KF. Furthermore, we highlight the capability of such a framework to develop methods that can leverage heterogeneous information (like semantic objects and geometric structures) from various sensing modalities, enabling a more holistic approach to scene understanding and enhancing the safety and effectiveness of autonomous systems.

Title: 3DGS-CD: 3D Gaussian Splatting-based Change Detection for Physical Object Rearrangement

• Authors: Ziqi Lu, Jianbo Ye, John Leonard
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.03706
• Pdf URL: https://arxiv.org/pdf/2411.03706
• Copy Paste: [[2411.03706]] 3DGS-CD: 3D Gaussian Splatting-based Change Detection for Physical Object Rearrangement(https://arxiv.org/abs/2411.03706)
• Keywords: segmentation
• Abstract: We present 3DGS-CD, the first 3D Gaussian Splatting (3DGS)-based method for detecting physical object rearrangements in 3D scenes. Our approach estimates 3D object-level changes by comparing two sets of unaligned images taken at different times. Leveraging 3DGS's novel view rendering and EfficientSAM's zero-shot segmentation capabilities, we detect 2D object-level changes, which are then associated and fused across views to estimate 3D changes. Our method can detect changes in cluttered environments using sparse post-change images within as little as 18s, using as few as a single new image. It does not rely on depth input, user instructions, object classes, or object models -- An object is recognized simply if it has been re-arranged. Our approach is evaluated on both public and self-collected real-world datasets, achieving up to 14% higher accuracy and three orders of magnitude faster performance compared to the state-of-the-art radiance-field-based change detection method. This significant performance boost enables a broad range of downstream applications, where we highlight three key use cases: object reconstruction, robot workspace reset, and 3DGS model update. Our code and data will be made available at this https URL.

Title: Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extraction

• Authors: Muhammad Tayyab Khan, Lequn Chen, Ye Han Ng, Wenhe Feng, Nicholas Yew Jin Tan, Seung Ki Moon
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03707
• Pdf URL: https://arxiv.org/pdf/2411.03707
• Copy Paste: [[2411.03707]] Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extraction(https://arxiv.org/abs/2411.03707)
• Keywords: extraction
• Abstract: Geometric Dimensioning and Tolerancing (GD&T) plays a critical role in manufacturing by defining acceptable variations in part features to ensure component quality and functionality. However, extracting GD&T information from 2D engineering drawings is a time-consuming and labor-intensive task, often relying on manual efforts or semi-automated tools. To address these challenges, this study proposes an automated and computationally efficient GD&T extraction method by fine-tuning Florence-2, an open-source vision-language model (VLM). The model is trained on a dataset of 400 drawings with ground truth annotations provided by domain experts. For comparison, two state-of-the-art closed-source VLMs, GPT-4o and Claude-3.5-Sonnet, are evaluated on the same dataset. All models are assessed using precision, recall, F1-score, and hallucination metrics. Due to the computational cost and impracticality of fine-tuning large closed-source VLMs for domain-specific tasks, GPT-4o and Claude-3.5-Sonnet are evaluated in a zero-shot setting. In contrast, Florence-2, a smaller model with 0.23 billion parameters, is optimized through full-parameter fine-tuning across three distinct experiments, each utilizing datasets augmented to different levels. The results show that Florence-2 achieves a 29.95% increase in precision, a 37.75% increase in recall, a 52.40% improvement in F1-score, and a 43.15% reduction in hallucination rate compared to the best-performing closed-source model. These findings highlight the effectiveness of fine-tuning smaller, open-source VLMs like Florence-2, offering a practical and efficient solution for automated GD&T extraction to support downstream manufacturing tasks.

Title: These Maps Are Made by Propagation: Adapting Deep Stereo Networks to Road Scenarios with Decisive Disparity Diffusion

• Authors: Chuang-Wei Liu, Yikang Zhang, Qijun Chen, Ioannis Pitas, Rui Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03717
• Pdf URL: https://arxiv.org/pdf/2411.03717
• Copy Paste: [[2411.03717]] These Maps Are Made by Propagation: Adapting Deep Stereo Networks to Road Scenarios with Decisive Disparity Diffusion(https://arxiv.org/abs/2411.03717)
• Keywords: diffusion
• Abstract: Stereo matching has emerged as a cost-effective solution for road surface 3D reconstruction, garnering significant attention towards improving both computational efficiency and accuracy. This article introduces decisive disparity diffusion (D3Stereo), marking the first exploration of dense deep feature matching that adapts pre-trained deep convolutional neural networks (DCNNs) to previously unseen road scenarios. A pyramid of cost volumes is initially created using various levels of learned representations. Subsequently, a novel recursive bilateral filtering algorithm is employed to aggregate these costs. A key innovation of D3Stereo lies in its alternating decisive disparity diffusion strategy, wherein intra-scale diffusion is employed to complete sparse disparity images, while inter-scale inheritance provides valuable prior information for higher resolutions. Extensive experiments conducted on our created UDTIRI-Stereo and Stereo-Road datasets underscore the effectiveness of D3Stereo strategy in adapting pre-trained DCNNs and its superior performance compared to all other explicit programming-based algorithms designed specifically for road surface 3D reconstruction. Additional experiments conducted on the Middlebury dataset with backbone DCNNs pre-trained on the ImageNet database further validate the versatility of D3Stereo strategy in tackling general stereo matching problems.

Title: NeurIPS 2023 Competition: Privacy Preserving Federated Learning Document VQA

• Authors: Marlon Tobaben, Mohamed Ali Souibgui, Rubèn Tito, Khanh Nguyen, Raouf Kerkouche, Kangsoo Jung, Joonas Jälkö, Lei Kang, Andrey Barsky, Vincent Poulain d'Andecy, Aurélie Joseph, Aashiq Muhamed, Kevin Kuo, Virginia Smith, Yusuke Yamasaki, Takumi Fukami, Kenta Niwa, Iifan Tyou, Hiro Ishii, Rio Yokota, Ragul N, Rintu Kutum, Josep Llados, Ernest Valveny, Antti Honkela, Mario Fritz, Dimosthenis Karatzas
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03730
• Pdf URL: https://arxiv.org/pdf/2411.03730
• Copy Paste: [[2411.03730]] NeurIPS 2023 Competition: Privacy Preserving Federated Learning Document VQA(https://arxiv.org/abs/2411.03730)
• Keywords: privacy, protect, extraction, federate, generative
• Abstract: The Privacy Preserving Federated Learning Document VQA (PFL-DocVQA) competition challenged the community to develop provably private and communication-efficient solutions in a federated setting for a real-life use case: invoice processing. The competition introduced a dataset of real invoice documents, along with associated questions and answers requiring information extraction and reasoning over the document images. Thereby, it brings together researchers and expertise from the document analysis, privacy, and federated learning communities. Participants fine-tuned a pre-trained, state-of-the-art Document Visual Question Answering model provided by the organizers for this new domain, mimicking a typical federated invoice processing setup. The base model is a multi-modal generative language model, and sensitive information could be exposed through either the visual or textual input modality. Participants proposed elegant solutions to reduce communication costs while maintaining a minimum utility threshold in track 1 and to protect all information from each document provider using differential privacy in track 2. The competition served as a new testbed for developing and testing private federated learning methods, simultaneously raising awareness about privacy within the document image analysis and recognition community. Ultimately, the competition analysis provides best practices and recommendations for successfully running privacy-focused federated learning challenges in the future.

Title: Human-in-the-Loop Feature Selection Using Interpretable Kolmogorov-Arnold Network-based Double Deep Q-Network

• Authors: Md Abrar Jahin, M. F. Mridha, Nilanjan Dey
• Subjects: cs.LG, cs.HC, stat.AP
• Abstract URL: https://arxiv.org/abs/2411.03740
• Pdf URL: https://arxiv.org/pdf/2411.03740
• Copy Paste: [[2411.03740]] Human-in-the-Loop Feature Selection Using Interpretable Kolmogorov-Arnold Network-based Double Deep Q-Network(https://arxiv.org/abs/2411.03740)
• Keywords: interpretability
• Abstract: Feature selection is critical for improving the performance and interpretability of machine learning models, particularly in high-dimensional spaces where complex feature interactions can reduce accuracy and increase computational demands. Existing approaches often rely on static feature subsets or manual intervention, limiting adaptability and scalability. However, dynamic, per-instance feature selection methods and model-specific interpretability in reinforcement learning remain underexplored. This study proposes a human-in-the-loop (HITL) feature selection framework integrated into a Double Deep Q-Network (DDQN) using a Kolmogorov-Arnold Network (KAN). Our novel approach leverages simulated human feedback and stochastic distribution-based sampling, specifically Beta, to iteratively refine feature subsets per data instance, improving flexibility in feature selection. The KAN-DDQN achieved notable test accuracies of 93% on MNIST and 83% on FashionMNIST, outperforming conventional MLP-DDQN models by up to 9%. The KAN-based model provided high interpretability via symbolic representation while using 4 times fewer neurons in the hidden layer than MLPs did. Comparatively, the models without feature selection achieved test accuracies of only 58% on MNIST and 64% on FashionMNIST, highlighting significant gains with our framework. Pruning and visualization further enhanced model transparency by elucidating decision pathways. These findings present a scalable, interpretable solution for feature selection that is suitable for applications requiring real-time, adaptive decision-making with minimal human oversight.

Title: Graph Neural Networks with Coarse- and Fine-Grained Division for Mitigating Label Sparsity and Noise

• Authors: Shuangjie Li, Baoming Zhang, Jianqing Song, Gaoli Ruan, Chongjun Wang, Junyuan Xie
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03744
• Pdf URL: https://arxiv.org/pdf/2411.03744
• Copy Paste: [[2411.03744]] Graph Neural Networks with Coarse- and Fine-Grained Division for Mitigating Label Sparsity and Noise(https://arxiv.org/abs/2411.03744)
• Keywords: robust
• Abstract: Graph Neural Networks (GNNs) have gained considerable prominence in semi-supervised learning tasks in processing graph-structured data, primarily owing to their message-passing mechanism, which largely relies on the availability of clean labels. However, in real-world scenarios, labels on nodes of graphs are inevitably noisy and sparsely labeled, significantly degrading the performance of GNNs. Exploring robust GNNs for semi-supervised node classification in the presence of noisy and sparse labels remains a critical challenge. Therefore, we propose a novel \textbf{G}raph \textbf{N}eural \textbf{N}etwork with \textbf{C}oarse- and \textbf{F}ine-\textbf{G}rained \textbf{D}ivision for mitigating label sparsity and noise, namely GNN-CFGD. The key idea of GNN-CFGD is reducing the negative impact of noisy labels via coarse- and fine-grained division, along with graph reconstruction. Specifically, we first investigate the effectiveness of linking unlabeled nodes to cleanly labeled nodes, demonstrating that this approach is more effective in combating labeling noise than linking to potentially noisy labeled nodes. Based on this observation, we introduce a Gaussian Mixture Model (GMM) based on the memory effect to perform a coarse-grained division of the given labels into clean and noisy labels. Next, we propose a clean labels oriented link that connects unlabeled nodes to cleanly labeled nodes, aimed at mitigating label sparsity and promoting supervision propagation. Furthermore, to provide refined supervision for noisy labeled nodes and additional supervision for unlabeled nodes, we fine-grain the noisy labeled and unlabeled nodes into two candidate sets based on confidence, respectively. Extensive experiments on various datasets demonstrate the superior effectiveness and robustness of GNN-CFGD.

Title: Optimal Defenses Against Gradient Reconstruction Attacks

• Authors: Yuxiao Chen, Gamze Gürsoy, Qi Lei
• Subjects: cs.LG, cs.AI, cs.CR, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.03746
• Pdf URL: https://arxiv.org/pdf/2411.03746
• Copy Paste: [[2411.03746]] Optimal Defenses Against Gradient Reconstruction Attacks(https://arxiv.org/abs/2411.03746)
• Keywords: protect, defense, attack, federate
• Abstract: Federated Learning (FL) is designed to prevent data leakage through collaborative model training without centralized data storage. However, it remains vulnerable to gradient reconstruction attacks that recover original training data from shared gradients. To optimize the trade-off between data leakage and utility loss, we first derive a theoretical lower bound of reconstruction error (among all attackers) for the two standard methods: adding noise, and gradient pruning. We then customize these two defenses to be parameter- and model-specific and achieve the optimal trade-off between our obtained reconstruction lower bound and model utility. Experimental results validate that our methods outperform Gradient Noise and Gradient Pruning by protecting the training data better while also achieving better utility.

Title: Deferred Poisoning: Making the Model More Vulnerable via Hessian Singularization

• Authors: Yuhao He, Jinyu Tian, Xianwei Zheng, Li Dong, Yuanman Li, Leo Yu Zhang, Jiantao Zhou
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03752
• Pdf URL: https://arxiv.org/pdf/2411.03752
• Copy Paste: [[2411.03752]] Deferred Poisoning: Making the Model More Vulnerable via Hessian Singularization(https://arxiv.org/abs/2411.03752)
• Keywords: security, defense, attack, robust, steal
• Abstract: Recent studies have shown that deep learning models are very vulnerable to poisoning attacks. Many defense methods have been proposed to address this issue. However, traditional poisoning attacks are not as threatening as commonly believed. This is because they often cause differences in how the model performs on the training set compared to the validation set. Such inconsistency can alert defenders that their data has been poisoned, allowing them to take the necessary defensive actions. In this paper, we introduce a more threatening type of poisoning attack called the Deferred Poisoning Attack. This new attack allows the model to function normally during the training and validation phases but makes it very sensitive to evasion attacks or even natural noise. We achieve this by ensuring the poisoned model's loss function has a similar value as a normally trained model at each input sample but with a large local curvature. A similar model loss ensures that there is no obvious inconsistency between the training and validation accuracy, demonstrating high stealthiness. On the other hand, the large curvature implies that a small perturbation may cause a significant increase in model loss, leading to substantial performance degradation, which reflects a worse robustness. We fulfill this purpose by making the model have singular Hessian information at the optimal point via our proposed Singularization Regularization term. We have conducted both theoretical and empirical analyses of the proposed method and validated its effectiveness through experiments on image classification tasks. Furthermore, we have confirmed the hazards of this form of poisoning attack under more general scenarios using natural noise, offering a new perspective for research in the field of security.

Title: Symbolic regression via MDLformer-guided search: from minimizing prediction error to minimizing description length

• Authors: Zihan Yu, Jingtao Ding, Yong Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03753
• Pdf URL: https://arxiv.org/pdf/2411.03753
• Copy Paste: [[2411.03753]] Symbolic regression via MDLformer-guided search: from minimizing prediction error to minimizing description length(https://arxiv.org/abs/2411.03753)
• Keywords: robust
• Abstract: Symbolic regression, a task discovering the formula best fitting the given data, is typically based on the heuristical search. These methods usually update candidate formulas to obtain new ones with lower prediction errors iteratively. However, since formulas with similar function shapes may have completely different symbolic forms, the prediction error does not decrease monotonously as the search approaches the target formula, causing the low recovery rate of existing methods. To solve this problem, we propose a novel search objective based on the minimum description length, which reflects the distance from the target and decreases monotonically as the search approaches the correct form of the target formula. To estimate the minimum description length of any input data, we design a neural network, MDLformer, which enables robust and scalable estimation through large-scale training. With the MDLformer's output as the search objective, we implement a symbolic regression method, SR4MDL, that can effectively recover the correct mathematical form of the formula. Extensive experiments illustrate its excellent performance in recovering formulas from data. Our method successfully recovers around 50 formulas across two benchmark datasets comprising 133 problems, outperforming state-of-the-art methods by 43.92%.

Title: Number Cookbook: Number Understanding of Language Models and How to Improve It

• Authors: Haotong Yang, Yi Hu, Shijia Kang, Zhouchen Lin, Muhan Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03766
• Pdf URL: https://arxiv.org/pdf/2411.03766
• Copy Paste: [[2411.03766]] Number Cookbook: Number Understanding of Language Models and How to Improve It(https://arxiv.org/abs/2411.03766)
• Keywords: large language model
• Abstract: Large language models (LLMs) can solve an increasing number of complex reasoning tasks while making surprising mistakes in basic numerical understanding and processing (such as 9.11 > 9.9). The latter ability is essential for tackling complex arithmetic and mathematical problems and serves as a foundation for most reasoning tasks, but previous work paid little attention to it or only discussed several restricted tasks (like integer addition). In this paper, we comprehensively investigate the numerical understanding and processing ability (NUPA) of LLMs. Firstly, we introduce a benchmark covering four common numerical representations and 17 distinct numerical tasks in four major categories, resulting in 41 meaningful combinations in total. These tasks are derived from primary and secondary education curricula, encompassing nearly all everyday numerical understanding and processing scenarios, and the rules of these tasks are very simple and clear. Through the benchmark, we find that current LLMs fail frequently in many of the tasks. To study the problem, we train small models with existing and potential techniques for enhancing NUPA (such as special tokenizers, PEs, and number formats), comprehensively evaluating their effectiveness using our testbed. We also finetune practical-scale LLMs on our proposed NUPA tasks and find that 1) naive finetuning can improve NUPA a lot on many but not all tasks, and 2) surprisingly, techniques designed to enhance NUPA prove ineffective for finetuning pretrained models. We further explore the impact of chain-of-thought techniques on NUPA. Our work takes a preliminary step towards understanding and improving NUPA of LLMs. Our benchmark and code are released at this https URL.

Title: A Bayesian Approach to Data Point Selection

• Authors: Xinnuo Xu, Minyoung Kim, Royson Lee, Brais Martinez, Timothy Hospedales
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03768
• Pdf URL: https://arxiv.org/pdf/2411.03768
• Copy Paste: [[2411.03768]] A Bayesian Approach to Data Point Selection(https://arxiv.org/abs/2411.03768)
• Keywords: large language model
• Abstract: Data point selection (DPS) is becoming a critical topic in deep learning due to the ease of acquiring uncurated training data compared to the difficulty of obtaining curated or processed data. Existing approaches to DPS are predominantly based on a bi-level optimisation (BLO) formulation, which is demanding in terms of memory and computation, and exhibits some theoretical defects regarding minibatches. Thus, we propose a novel Bayesian approach to DPS. We view the DPS problem as posterior inference in a novel Bayesian model where the posterior distributions of the instance-wise weights and the main neural network parameters are inferred under a reasonable prior and likelihood model. We employ stochastic gradient Langevin MCMC sampling to learn the main network and instance-wise weights jointly, ensuring convergence even with minibatches. Our update equation is comparable to the widely used SGD and much more efficient than existing BLO-based methods. Through controlled experiments in both the vision and language domains, we present the proof-of-concept. Additionally, we demonstrate that our method scales effectively to large language models and facilitates automated per-task optimization for instruction fine-tuning datasets.

Title: Harmformer: Harmonic Networks Meet Transformers for Continuous Roto-Translation Equivariance

• Authors: Tomáš Karella, Adam Harmanec, Jan Kotera, Jan Blažek, Filip Šroubek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03794
• Pdf URL: https://arxiv.org/pdf/2411.03794
• Copy Paste: [[2411.03794]] Harmformer: Harmonic Networks Meet Transformers for Continuous Roto-Translation Equivariance(https://arxiv.org/abs/2411.03794)
• Keywords: robust, transformer
• Abstract: CNNs exhibit inherent equivariance to image translation, leading to efficient parameter and data usage, faster learning, and improved robustness. The concept of translation equivariant networks has been successfully extended to rotation transformation using group convolution for discrete rotation groups and harmonic functions for the continuous rotation group encompassing $360^\circ$. We explore the compatibility of the SA mechanism with full rotation equivariance, in contrast to previous studies that focused on discrete rotation. We introduce the Harmformer, a harmonic transformer with a convolutional stem that achieves equivariance for both translation and continuous rotation. Accompanied by an end-to-end equivariance proof, the Harmformer not only outperforms previous equivariant transformers, but also demonstrates inherent stability under any continuous rotation, even without seeing rotated samples during training.

Title: VQA$^2$:Visual Question Answering for Video Quality Assessment

• Authors: Ziheng Jia, Zicheng Zhang, Jiaying Qian, Haoning Wu, Wei Sun, Chunyi Li, Xiaohong Liu, Weisi Lin, Guangtao Zhai, Xiongkuo Min
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03795
• Pdf URL: https://arxiv.org/pdf/2411.03795
• Copy Paste: [[2411.03795]] VQA$^2$:Visual Question Answering for Video Quality Assessment(https://arxiv.org/abs/2411.03795)
• Keywords: robust
• Abstract: The advent and proliferation of large multi-modal models (LMMs) have introduced a new paradigm to video-related computer vision fields, including training and inference methods based on visual question answering (VQA). These methods enable models to handle multiple downstream tasks robustly. Video Quality Assessment (VQA), a classic field in low-level visual quality evaluation, originally focused on quantitative video quality scoring. However, driven by advances in LMMs, it is now evolving towards more comprehensive visual quality understanding tasks. Visual question answering has significantly improved low-level visual evaluation within the image domain recently. However, related work is almost nonexistent in the video domain, leaving substantial room for improvement. To address this gap, we introduce the VQA2 Instruction Dataset the first visual question answering instruction dataset entirely focuses on video quality assessment, and based on it, we propose the VQA2 series models The VQA2 Instruction Dataset consists of three stages and covers various video types, containing 157,735 instruction question-answer pairs, including both manually annotated and synthetic data. We conduct extensive experiments on both video quality scoring and video quality understanding tasks. Results demonstrate that the VQA2 series models achieve state-of-the-art (SOTA) performance in quality scoring tasks, and their performance in visual quality question answering surpasses the renowned GPT-4o. Additionally, our final model, the VQA2-Assistant, performs well across both scoring and question-answering tasks, validating its versatility.

Title: Overcoming label shift in targeted federated learning

• Authors: Edvin Listo Zec, Adam Breitholtz, Fredrik D. Johansson
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03799
• Pdf URL: https://arxiv.org/pdf/2411.03799
• Copy Paste: [[2411.03799]] Overcoming label shift in targeted federated learning(https://arxiv.org/abs/2411.03799)
• Keywords: robust, federate
• Abstract: Federated learning enables multiple actors to collaboratively train models without sharing private data. This unlocks the potential for scaling machine learning to diverse applications. Existing algorithms for this task are well-justified when clients and the intended target domain share the same distribution of features and labels, but this assumption is often violated in real-world scenarios. One common violation is label shift, where the label distributions differ across clients or between clients and the target domain, which can significantly degrade model performance. To address this problem, we propose FedPALS, a novel model aggregation scheme that adapts to label shifts by leveraging knowledge of the target label distribution at the central server. Our approach ensures unbiased updates under stochastic gradient descent, ensuring robust generalization across clients with diverse, label-shifted data. Extensive experiments on image classification demonstrate that FedPALS consistently outperforms standard baselines by aligning model aggregation with the target domain. Our findings reveal that conventional federated learning methods suffer severely in cases of extreme client sparsity, highlighting the critical need for target-aware aggregation. FedPALS offers a principled and practical solution to mitigate label distribution mismatch, ensuring models trained in federated settings can generalize effectively to label-shifted target domains.

Title: A Comparative Study of Recent Large Language Models on Generating Hospital Discharge Summaries for Lung Cancer Patients

• Authors: Yiming Li, Fang Li, Kirk Roberts, Licong Cui, Cui Tao, Hua Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03805
• Pdf URL: https://arxiv.org/pdf/2411.03805
• Copy Paste: [[2411.03805]] A Comparative Study of Recent Large Language Models on Generating Hospital Discharge Summaries for Lung Cancer Patients(https://arxiv.org/abs/2411.03805)
• Keywords: robust, large language model
• Abstract: Generating discharge summaries is a crucial yet time-consuming task in clinical practice, essential for conveying pertinent patient information and facilitating continuity of care. Recent advancements in large language models (LLMs) have significantly enhanced their capability in understanding and summarizing complex medical texts. This research aims to explore how LLMs can alleviate the burden of manual summarization, streamline workflow efficiencies, and support informed decision-making in healthcare settings. Clinical notes from a cohort of 1,099 lung cancer patients were utilized, with a subset of 50 patients for testing purposes, and 102 patients used for model fine-tuning. This study evaluates the performance of multiple LLMs, including GPT-3.5, GPT-4, GPT-4o, and LLaMA 3 8b, in generating discharge summaries. Evaluation metrics included token-level analysis (BLEU, ROUGE-1, ROUGE-2, ROUGE-L) and semantic similarity scores between model-generated summaries and physician-written gold standards. LLaMA 3 8b was further tested on clinical notes of varying lengths to examine the stability of its performance. The study found notable variations in summarization capabilities among LLMs. GPT-4o and fine-tuned LLaMA 3 demonstrated superior token-level evaluation metrics, while LLaMA 3 consistently produced concise summaries across different input lengths. Semantic similarity scores indicated GPT-4o and LLaMA 3 as leading models in capturing clinical relevance. This study contributes insights into the efficacy of LLMs for generating discharge summaries, highlighting LLaMA 3's robust performance in maintaining clarity and relevance across varying clinical contexts. These findings underscore the potential of automated summarization tools to enhance documentation precision and efficiency, ultimately improving patient care and operational capability in healthcare settings.

Title: Understanding the Effects of Human-written Paraphrases in LLM-generated Text Detection

• Authors: Hiu Ting Lau, Arkaitz Zubiaga
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03806
• Pdf URL: https://arxiv.org/pdf/2411.03806
• Copy Paste: [[2411.03806]] Understanding the Effects of Human-written Paraphrases in LLM-generated Text Detection(https://arxiv.org/abs/2411.03806)
• Keywords: watermark, large language model
• Abstract: Natural Language Generation has been rapidly developing with the advent of large language models (LLMs). While their usage has sparked significant attention from the general public, it is important for readers to be aware when a piece of text is LLM-generated. This has brought about the need for building models that enable automated LLM-generated text detection, with the aim of mitigating potential negative outcomes of such content. Existing LLM-generated detectors show competitive performances in telling apart LLM-generated and human-written text, but this performance is likely to deteriorate when paraphrased texts are considered. In this study, we devise a new data collection strategy to collect Human & LLM Paraphrase Collection (HLPC), a first-of-its-kind dataset that incorporates human-written texts and paraphrases, as well as LLM-generated texts and paraphrases. With the aim of understanding the effects of human-written paraphrases on the performance of state-of-the-art LLM-generated text detectors OpenAI RoBERTa and watermark detectors, we perform classification experiments that incorporate human-written paraphrases, watermarked and non-watermarked LLM-generated documents from GPT and OPT, and LLM-generated paraphrases from DIPPER and BART. The results show that the inclusion of human-written paraphrases has a significant impact of LLM-generated detector performance, promoting TPR@1%FPR with a possible trade-off of AUROC and accuracy.

Title: GS2Pose: Tow-stage 6D Object Pose Estimation Guided by Gaussian Splatting

• Authors: Jilan Mei, Junbo Li, Cai Meng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03807
• Pdf URL: https://arxiv.org/pdf/2411.03807
• Copy Paste: [[2411.03807]] GS2Pose: Tow-stage 6D Object Pose Estimation Guided by Gaussian Splatting(https://arxiv.org/abs/2411.03807)
• Keywords: robust
• Abstract: This paper proposes a new method for accurate and robust 6D pose estimation of novel objects, named GS2Pose. By introducing 3D Gaussian splatting, GS2Pose can utilize the reconstruction results without requiring a high-quality CAD model, which means it only requires segmented RGBD images as input. Specifically, GS2Pose employs a two-stage structure consisting of coarse estimation followed by refined estimation. In the coarse stage, a lightweight U-Net network with a polarization attention mechanism, called Pose-Net, is designed. By using the 3DGS model for supervised training, Pose-Net can generate NOCS images to compute a coarse pose. In the refinement stage, GS2Pose formulates a pose regression algorithm following the idea of reprojection or Bundle Adjustment (BA), referred to as GS-Refiner. By leveraging Lie algebra to extend 3DGS, GS-Refiner obtains a pose-differentiable rendering pipeline that refines the coarse pose by comparing the input images with the rendered images. GS-Refiner also selectively updates parameters in the 3DGS model to achieve environmental adaptation, thereby enhancing the algorithm's robustness and flexibility to illuminative variation, occlusion, and other challenging disruptive factors. GS2Pose was evaluated through experiments conducted on the LineMod dataset, where it was compared with similar algorithms, yielding highly competitive results. The code for GS2Pose will soon be released on GitHub.

Title: SA3DIP: Segment Any 3D Instance with Potential 3D Priors

• Authors: Xi Yang, Xu Gu, Xingyilang Yin, Xinbo Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03819
• Pdf URL: https://arxiv.org/pdf/2411.03819
• Copy Paste: [[2411.03819]] SA3DIP: Segment Any 3D Instance with Potential 3D Priors(https://arxiv.org/abs/2411.03819)
• Keywords: robust, fair, segmentation
• Abstract: The proliferation of 2D foundation models has sparked research into adapting them for open-world 3D instance segmentation. Recent methods introduce a paradigm that leverages superpoints as geometric primitives and incorporates 2D multi-view masks from Segment Anything model (SAM) as merging guidance, achieving outstanding zero-shot instance segmentation results. However, the limited use of 3D priors restricts the segmentation performance. Previous methods calculate the 3D superpoints solely based on estimated normal from spatial coordinates, resulting in under-segmentation for instances with similar geometry. Besides, the heavy reliance on SAM and hand-crafted algorithms in 2D space suffers from over-segmentation due to SAM's inherent part-level segmentation tendency. To address these issues, we propose SA3DIP, a novel method for Segmenting Any 3D Instances via exploiting potential 3D Priors. Specifically, on one hand, we generate complementary 3D primitives based on both geometric and textural priors, which reduces the initial errors that accumulate in subsequent procedures. On the other hand, we introduce supplemental constraints from the 3D space by using a 3D detector to guide a further merging process. Furthermore, we notice a considerable portion of low-quality ground truth annotations in ScanNetV2 benchmark, which affect the fair evaluations. Thus, we present ScanNetV2-INS with complete ground truth labels and supplement additional instances for 3D class-agnostic instance segmentation. Experimental evaluations on various 2D-3D datasets demonstrate the effectiveness and robustness of our approach. Our code and proposed ScanNetV2-INS dataset are available HERE.

Title: Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM Data Contamination

• Authors: Dingjie Song, Sicheng Lai, Shunian Chen, Lichao Sun, Benyou Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03823
• Pdf URL: https://arxiv.org/pdf/2411.03823
• Copy Paste: [[2411.03823]] Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM Data Contamination(https://arxiv.org/abs/2411.03823)
• Keywords: large language model
• Abstract: The rapid progression of multimodal large language models (MLLMs) has demonstrated superior performance on various multimodal benchmarks. However, the issue of data contamination during training creates challenges in performance evaluation and comparison. While numerous methods exist for detecting dataset contamination in large language models (LLMs), they are less effective for MLLMs due to their various modalities and multiple training phases. In this study, we introduce a multimodal data contamination detection framework, MM-Detect, designed for MLLMs. Our experimental results indicate that MM-Detect is sensitive to varying degrees of contamination and can highlight significant performance improvements due to leakage of the training set of multimodal benchmarks. Furthermore, We also explore the possibility of contamination originating from the pre-training phase of LLMs used by MLLMs and the fine-tuning phase of MLLMs, offering new insights into the stages at which contamination may be introduced.

Title: Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts

• Authors: Zhitong Gao, Bingnan Li, Mathieu Salzmann, Xuming He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03829
• Pdf URL: https://arxiv.org/pdf/2411.03829
• Copy Paste: [[2411.03829]] Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts(https://arxiv.org/abs/2411.03829)
• Keywords: robust, generative, segmentation
• Abstract: In open-world scenarios, where both novel classes and domains may exist, an ideal segmentation model should detect anomaly classes for safety and generalize to new domains. However, existing methods often struggle to distinguish between domain-level and semantic-level distribution shifts, leading to poor out-of-distribution (OOD) detection or domain generalization performance. In this work, we aim to equip the model to generalize effectively to covariate-shift regions while precisely identifying semantic-shift regions. To achieve this, we design a novel generative augmentation method to produce coherent images that incorporate both anomaly (or novel) objects and various covariate shifts at both image and object levels. Furthermore, we introduce a training strategy that recalibrates uncertainty specifically for semantic shifts and enhances the feature extractor to align features associated with domain shifts. We validate the effectiveness of our method across benchmarks featuring both semantic and domain shifts. Our method achieves state-of-the-art performance across all benchmarks for both OOD detection and domain generalization. Code is available at this https URL.

Title: An Enhancement of Haar Cascade Algorithm Applied to Face Recognition for Gate Pass Security

• Authors: Clarence A. Antipona, Romeo R. Magsino, Raymund M. Dioses, Khatalyn E. Mata
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03831
• Pdf URL: https://arxiv.org/pdf/2411.03831
• Copy Paste: [[2411.03831]] An Enhancement of Haar Cascade Algorithm Applied to Face Recognition for Gate Pass Security(https://arxiv.org/abs/2411.03831)
• Keywords: security
• Abstract: This study is focused on enhancing the Haar Cascade Algorithm to decrease the false positive and false negative rate in face matching and face detection to increase the accuracy rate even under challenging conditions. The face recognition library was implemented with Haar Cascade Algorithm in which the 128-dimensional vectors representing the unique features of a face are encoded. A subprocess was applied where the grayscale image from Haar Cascade was converted to RGB to improve the face encoding. Logical process and face filtering are also used to decrease non-face detection. The Enhanced Haar Cascade Algorithm produced a 98.39% accuracy rate (21.39% increase), 63.59% precision rate, 98.30% recall rate, and 72.23% in F1 Score. In comparison, the Haar Cascade Algorithm achieved a 46.70% to 77.00% accuracy rate, 44.15% precision rate, 98.61% recall rate, and 47.01% in F1 Score. Both algorithms used the Confusion Matrix Test with 301,950 comparisons using the same dataset of 550 images. The 98.39% accuracy rate shows a significant decrease in false positive and false negative rates in facial recognition. Face matching and face detection are more accurate in images with complex backgrounds, lighting variations, and occlusions, or even those with similar attributes.

Title: Flexible task abstractions emerge in linear networks with fast and bounded units

• Authors: Kai Sandbrink, Jan P. Bauer, Alexandra M. Proca, Andrew M. Saxe, Christopher Summerfield, Ali Hummos
• Subjects: cs.LG, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2411.03840
• Pdf URL: https://arxiv.org/pdf/2411.03840
• Copy Paste: [[2411.03840]] Flexible task abstractions emerge in linear networks with fast and bounded units(https://arxiv.org/abs/2411.03840)
• Keywords: protect
• Abstract: Animals survive in dynamic environments changing at arbitrary timescales, but such data distribution shifts are a challenge to neural networks. To adapt to change, neural systems may change a large number of parameters, which is a slow process involving forgetting past information. In contrast, animals leverage distribution changes to segment their stream of experience into tasks and associate them with internal task abstracts. Animals can then respond flexibly by selecting the appropriate task abstraction. However, how such flexible task abstractions may arise in neural systems remains unknown. Here, we analyze a linear gated network where the weights and gates are jointly optimized via gradient descent, but with neuron-like constraints on the gates including a faster timescale, nonnegativity, and bounded activity. We observe that the weights self-organize into modules specialized for tasks or sub-tasks encountered, while the gates layer forms unique representations that switch the appropriate weight modules (task abstractions). We analytically reduce the learning dynamics to an effective eigenspace, revealing a virtuous cycle: fast adapting gates drive weight specialization by protecting previous knowledge, while weight specialization in turn increases the update rate of the gating layer. Task switching in the gating layer accelerates as a function of curriculum block size and task training, mirroring key findings in cognitive neuroscience. We show that the discovered task abstractions support generalization through both task and subtask composition, and we extend our findings to a non-linear network switching between two tasks. Overall, our work offers a theory of cognitive flexibility in animals as arising from joint gradient descent on synaptic and neural gating in a neural network architecture.

Title: Attribute-Based Encryption With Payable Outsourced Decryption Using Blockchain and Responsive Zero Knowledge Proof

• Authors: Dongliang Cai, Borui Chen, Liang Zhang, Kexin Li, Haibin Kan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.03844
• Pdf URL: https://arxiv.org/pdf/2411.03844
• Copy Paste: [[2411.03844]] Attribute-Based Encryption With Payable Outsourced Decryption Using Blockchain and Responsive Zero Knowledge Proof(https://arxiv.org/abs/2411.03844)
• Keywords: protect, fair
• Abstract: Attribute-Based Encryption (ABE) is a promising solution for access control in cloud services. However, the heavy decryption overhead hinders its widespread adoption. A general approach to address this issue is to outsource decryption to decryption cloud service(DCS). Existing schemes have utilized various methods to enable users to verify outsourced results; however, they lack an effective mechanism to achieve exemptibility which enables the honest DCS to escape from wrong claims. And it is impractical to assume that the DCS will provide free services. In this paper, we propose a blockchain-based payable outsourced decryption ABE scheme that achieves both verifiability and exemptibility without adding redundant information to ABE ciphertext. We use zero-knowledge proof to verify outsourced results on blockchain and introduce an optional single-round challenge game under optimistic assumption to address the high cost of proof generation. Moreover, our system achieves fairness and decentralized outsourcing to protect the interests of all parties. Finally, we implement and evaluate our scheme on Ethereum to demonstrate its feasibility and efficiency, the gas usage in attribute numbers from 5 to 60 is 11$\times$ to 140$\times$ in the happy case and 4$\times$ to 55$\times$ in the challenge case lower than the scheme of Ge et al. (TDSC'23).

Title: A Novel Access Control and Privacy-Enhancing Approach for Models in Edge Computing

• Authors: Peihao Li
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03847
• Pdf URL: https://arxiv.org/pdf/2411.03847
• Copy Paste: [[2411.03847]] A Novel Access Control and Privacy-Enhancing Approach for Models in Edge Computing(https://arxiv.org/abs/2411.03847)
• Keywords: security, privacy, protect, attack, robust, watermark
• Abstract: With the widespread adoption of edge computing technologies and the increasing prevalence of deep learning models in these environments, the security risks and privacy threats to models and data have grown more acute. Attackers can exploit various techniques to illegally obtain models or misuse data, leading to serious issues such as intellectual property infringement and privacy breaches. Existing model access control technologies primarily rely on traditional encryption and authentication methods; however, these approaches exhibit significant limitations in terms of flexibility and adaptability in dynamic environments. Although there have been advancements in model watermarking techniques for marking model ownership, they remain limited in their ability to proactively protect intellectual property and prevent unauthorized access. To address these challenges, we propose a novel model access control method tailored for edge computing environments. This method leverages image style as a licensing mechanism, embedding style recognition into the model's operational framework to enable intrinsic access control. Consequently, models deployed on edge platforms are designed to correctly infer only on license data with specific style, rendering them ineffective on any other data. By restricting the input data to the edge model, this approach not only prevents attackers from gaining unauthorized access to the model but also enhances the privacy of data on terminal devices. We conducted extensive experiments on benchmark datasets, including MNIST, CIFAR-10, and FACESCRUB, and the results demonstrate that our method effectively prevents unauthorized access to the model while maintaining accuracy. Additionally, the model shows strong resistance against attacks such as forged licenses and fine-tuning. These results underscore the method's usability, security, and robustness.

Title: MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba

• Authors: Masakazu Yoshimura, Teruaki Hayashi, Yota Maeda
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03855
• Pdf URL: https://arxiv.org/pdf/2411.03855
• Copy Paste: [[2411.03855]] MambaPEFT: Exploring Parameter-Efficient Fine-Tuning for Mamba(https://arxiv.org/abs/2411.03855)
• Keywords: transformer
• Abstract: An ecosystem of Transformer-based models has been established by building large models with extensive data. Parameter-efficient fine-tuning (PEFT) is a crucial technology for deploying these models to downstream tasks with minimal cost while achieving effective performance. Recently, Mamba, a State Space Model (SSM)-based model, has attracted attention as a potential alternative to Transformers. While many large-scale Mamba-based models have been proposed, efficiently adapting pre-trained Mamba-based models to downstream tasks remains unexplored. In this paper, we conduct an exploratory analysis of PEFT methods for Mamba. We investigate the effectiveness of existing PEFT methods for Transformers when applied to Mamba. We also modify these methods to better align with the Mamba architecture. Additionally, we propose new Mamba-specific PEFT methods that leverage the distinctive structure of Mamba. Our experiments indicate that PEFT performs more effectively for Mamba than Transformers. Lastly, we demonstrate how to effectively combine multiple PEFT methods and provide a framework that outperforms previous works. To ensure reproducibility, we will release the code after publication.

Title: FedRISE: Rating Induced Sign Election of Gradients for Byzantine Tolerant Federated Aggregation

• Authors: Joseph Geo Benjamin, Mothilal Asokan, Mohammad Yaqub, Karthik Nandakumar
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.03861
• Pdf URL: https://arxiv.org/pdf/2411.03861
• Copy Paste: [[2411.03861]] FedRISE: Rating Induced Sign Election of Gradients for Byzantine Tolerant Federated Aggregation(https://arxiv.org/abs/2411.03861)
• Keywords: defense, attack, robust, federate
• Abstract: One of the most common defense strategies against model poisoning in federated learning is to employ a robust aggregator mechanism that makes the training more resilient. Many of the existing Byzantine robust aggregators provide theoretical guarantees and are empirically effective against certain categories of attacks. However, we observe that certain high-strength attacks can subvert the aggregator and collapse the training. In addition, most aggregators require identifying tolerant settings to converge. Impact of attacks becomes more pronounced when the number of Byzantines is near-majority, and becomes harder to evade if the attacker is omniscient with access to data, honest updates and aggregation methods. Motivated by these observations, we develop a robust aggregator called FedRISE for cross-silo FL that is consistent and less susceptible to poisoning updates by an omniscient attacker. The proposed method explicitly determines the optimal direction of each gradient through a sign-voting strategy that uses variance-reduced sparse gradients. We argue that vote weighting based on the cosine similarity of raw gradients is misleading, and we introduce a sign-based gradient valuation function that ignores the gradient magnitude. We compare our method against 8 robust aggregators under 6 poisoning attacks on 3 datasets and architectures. Our results show that existing robust aggregators collapse for at least some attacks under severe settings, while FedRISE demonstrates better robustness because of a stringent gradient inclusion formulation.

Title: ROBIN: Robust and Invisible Watermarks for Diffusion Models with Adversarial Optimization

• Authors: Huayang Huang, Yu Wu, Qian Wang
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.03862
• Pdf URL: https://arxiv.org/pdf/2411.03862
• Copy Paste: [[2411.03862]] ROBIN: Robust and Invisible Watermarks for Diffusion Models with Adversarial Optimization(https://arxiv.org/abs/2411.03862)
• Keywords: protect, robust, watermark, diffusion, generative
• Abstract: Watermarking generative content serves as a vital tool for authentication, ownership protection, and mitigation of potential misuse. Existing watermarking methods face the challenge of balancing robustness and concealment. They empirically inject a watermark that is both invisible and robust and passively achieve concealment by limiting the strength of the watermark, thus reducing the robustness. In this paper, we propose to explicitly introduce a watermark hiding process to actively achieve concealment, thus allowing the embedding of stronger watermarks. To be specific, we implant a robust watermark in an intermediate diffusion state and then guide the model to hide the watermark in the final generated image. We employ an adversarial optimization algorithm to produce the optimal hiding prompt guiding signal for each watermark. The prompt embedding is optimized to minimize artifacts in the generated image, while the watermark is optimized to achieve maximum strength. The watermark can be verified by reversing the generation process. Experiments on various diffusion models demonstrate the watermark remains verifiable even under significant image tampering and shows superior invisibility compared to other state-of-the-art robust watermarking methods.

Title: Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward

• Authors: Shashi Kumar, Iuliia Thorbecke, Sergio Burdisso, Esaú Villatoro-Tello, Manjunath K E, Kadri Hacioğlu, Pradeep Rangappa, Petr Motlicek, Aravind Ganapathiraju, Andreas Stolcke
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.03866
• Pdf URL: https://arxiv.org/pdf/2411.03866
• Copy Paste: [[2411.03866]] Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward(https://arxiv.org/abs/2411.03866)
• Keywords: robust, large language model
• Abstract: Recent research has demonstrated that training a linear connector between speech foundation encoders and large language models (LLMs) enables this architecture to achieve strong ASR capabilities. Despite the impressive results, it remains unclear whether these simple approaches are robust enough across different scenarios and speech conditions, such as domain shifts and different speech perturbations. In this paper, we address these questions by conducting various ablation experiments using a recent and widely adopted approach called SLAM-ASR. We present novel empirical findings that offer insights on how to effectively utilize the SLAM-ASR architecture across a wide range of settings. Our main findings indicate that the SLAM-ASR exhibits poor performance in cross-domain evaluation settings. Additionally, speech perturbations within in-domain data, such as changes in speed or the presence of additive noise, can significantly impact performance. Our findings offer critical insights for fine-tuning and configuring robust LLM-based ASR models, tailored to different data characteristics and computational resources.

Title: EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoning

• Authors: Kiran Purohit, Venktesh V, Raghuram Devalla, Krishna Mohan Yerragorla, Sourangshu Bhattacharya, Avishek Anand
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03877
• Pdf URL: https://arxiv.org/pdf/2411.03877
• Copy Paste: [[2411.03877]] EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoning(https://arxiv.org/abs/2411.03877)
• Keywords: robust, large language model
• Abstract: Answering reasoning-based complex questions over text and hybrid sources, including tables, is a challenging task. Recent advances in large language models (LLMs) have enabled in-context learning (ICL), allowing LLMs to acquire proficiency in a specific task using only a few demonstration samples (exemplars). A critical challenge in ICL is the selection of optimal exemplars, which can be either task-specific (static) or test-example-specific (dynamic). Static exemplars provide faster inference times and increased robustness across a distribution of test examples. In this paper, we propose an algorithm for static exemplar subset selection for complex reasoning tasks. We introduce EXPLORA, a novel exploration method designed to estimate the parameters of the scoring function, which evaluates exemplar subsets without incorporating confidence information. EXPLORA significantly reduces the number of LLM calls to ~11% of those required by state-of-the-art methods and achieves a substantial performance improvement of 12.24%. We open-source our code and data (this https URL).

Title: MEG: Medical Knowledge-Augmented Large Language Models for Question Answering

• Authors: Laura Cabello, Carmen Martin-Turrero, Uchenna Akujuobi, Anders Søgaard, Carlos Bobed
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03883
• Pdf URL: https://arxiv.org/pdf/2411.03883
• Copy Paste: [[2411.03883]] MEG: Medical Knowledge-Augmented Large Language Models for Question Answering(https://arxiv.org/abs/2411.03883)
• Keywords: robust, large language model
• Abstract: Question answering is a natural language understanding task that involves reasoning over both explicit context and unstated, relevant domain knowledge. Large language models (LLMs), which underpin most contemporary question answering systems, struggle to induce how concepts relate in specialized domains such as medicine. Existing medical LLMs are also costly to train. In this work, we present MEG, a parameter-efficient approach for medical knowledge-augmented LLMs. MEG uses a lightweight mapping network to integrate graph embeddings into the LLM, enabling it to leverage external knowledge in a cost-effective way. We evaluate our method on four popular medical multiple-choice datasets and show that LLMs greatly benefit from the factual grounding provided by knowledge graph embeddings. MEG attains an average of +10.2% accuracy over the Mistral-Instruct baseline, and +6.7% over specialized models like BioMistral. We also show results based on Llama-3. Finally, we show that MEG's performance remains robust to the choice of graph encoder.

Title: Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models

• Authors: Zhijian Zhuo, Ya Wang, Yutao Zeng, Xiaoqing Li, Xun Zhou, Jinwen Ma
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03884
• Pdf URL: https://arxiv.org/pdf/2411.03884
• Copy Paste: [[2411.03884]] Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models(https://arxiv.org/abs/2411.03884)
• Keywords: transformer, large language model
• Abstract: Transformers have found extensive applications across various domains due to the powerful fitting capabilities. This success can be partially attributed to their inherent nonlinearity. Thus, in addition to the ReLU function employed in the original transformer architecture, researchers have explored alternative modules such as GeLU and SwishGLU to enhance nonlinearity and thereby augment representational capacity. In this paper, we propose a novel category of polynomial composition activations (PolyCom), designed to optimize the dynamics of transformers. Theoretically, we provide a comprehensive mathematical analysis of PolyCom, highlighting its enhanced expressivity and efficacy relative to other activation functions. Notably, we demonstrate that networks incorporating PolyCom achieve the $\textbf{optimal approximation rate}$, indicating that PolyCom networks require minimal parameters to approximate general smooth functions in Sobolev spaces. We conduct empirical experiments on the pre-training configurations of large language models (LLMs), including both dense and sparse architectures. By substituting conventional activation functions with PolyCom, we enable LLMs to capture higher-order interactions within the data, thus improving performance metrics in terms of accuracy and convergence rates. Extensive experimental results demonstrate the effectiveness of our method, showing substantial improvements over other activation functions. Code is available at this https URL.

Title: Computational Analysis of Gender Depiction in the Comedias of Calder\'on de la Barca

• Authors: Allison Keith, Antonio Rojas Castro, Sebastian Padó
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03895
• Pdf URL: https://arxiv.org/pdf/2411.03895
• Copy Paste: [[2411.03895]] Computational Analysis of Gender Depiction in the Comedias of Calder\'on de la Barca(https://arxiv.org/abs/2411.03895)
• Keywords: explainability
• Abstract: In theatre, playwrights use the portrayal of characters to explore culturally based gender norms. In this paper, we develop quantitative methods to study gender depiction in the non-religious works (comedias) of Pedro Calderón de la Barca, a prolific Spanish 17th century author. We gather insights from a corpus of more than 100 plays by using a gender classifier and applying model explainability (attribution) methods to determine which text features are most influential in the model's decision to classify speech as 'male' or 'female', indicating the most gendered elements of dialogue in Calderón's comedias in a human accessible manner. We find that female and male characters are portrayed differently and can be identified by the gender prediction model at practically useful accuracies (up to f=0.83). Analysis reveals semantic aspects of gender portrayal, and demonstrates that the model is even useful in providing a relatively accurate scene-by-scene prediction of cross-dressing characters.

Title: Retentive Neural Quantum States: Efficient Ans\"atze for Ab Initio Quantum Chemistry

• Authors: Oliver Knitter, Dan Zhao, James Stokes, Martin Ganahl, Stefan Leichenauer, Shravan Veerapaneni
• Subjects: cs.LG, cs.CE, quant-ph
• Abstract URL: https://arxiv.org/abs/2411.03900
• Pdf URL: https://arxiv.org/pdf/2411.03900
• Copy Paste: [[2411.03900]] Retentive Neural Quantum States: Efficient Ans\"atze for Ab Initio Quantum Chemistry(https://arxiv.org/abs/2411.03900)
• Keywords: transformer
• Abstract: Neural-network quantum states (NQS) has emerged as a powerful application of quantum-inspired deep learning for variational Monte Carlo methods, offering a competitive alternative to existing techniques for identifying ground states of quantum problems. A significant advancement toward improving the practical scalability of NQS has been the incorporation of autoregressive models, most recently transformers, as variational ansatze. Transformers learn sequence information with greater expressiveness than recurrent models, but at the cost of increased time complexity with respect to sequence length. We explore the use of the retentive network (RetNet), a recurrent alternative to transformers, as an ansatz for solving electronic ground state problems in $\textit{ab initio}$ quantum chemistry. Unlike transformers, RetNets overcome this time complexity bottleneck by processing data in parallel during training, and recurrently during inference. We give a simple computational cost estimate of the RetNet and directly compare it with similar estimates for transformers, establishing a clear threshold ratio of problem-to-model size past which the RetNet's time complexity outperforms that of the transformer. Though this efficiency can comes at the expense of decreased expressiveness relative to the transformer, we overcome this gap through training strategies that leverage the autoregressive structure of the model -- namely, variational neural annealing. Our findings support the RetNet as a means of improving the time complexity of NQS without sacrificing accuracy. We provide further evidence that the ablative improvements of neural annealing extend beyond the RetNet architecture, suggesting it would serve as an effective general training strategy for autoregressive NQS.

Title: WiP: Towards a Secure SECP256K1 for Crypto Wallets: Hardware Architecture and Implementation

• Authors: Joel Poncha Lemayian, Ghyslain Gagnon, Kaiwen Zhang, Pascal Giard
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2411.03910
• Pdf URL: https://arxiv.org/pdf/2411.03910
• Copy Paste: [[2411.03910]] WiP: Towards a Secure SECP256K1 for Crypto Wallets: Hardware Architecture and Implementation(https://arxiv.org/abs/2411.03910)
• Keywords: secure, protect, attack
• Abstract: The SECP256K1 elliptic curve algorithm is fundamental in cryptocurrency wallets for generating secure public keys from private keys, thereby ensuring the protection and ownership of blockchain-based digital assets. However, the literature highlights several successful side-channel attacks on hardware wallets that exploit SECP256K1 to extract private keys. This work proposes a novel hardware architecture for SECP256K1, optimized for side-channel attack resistance and efficient resource utilization. The architecture incorporates complete addition formulas, temporary registers, and parallel processing techniques, making elliptic curve point addition and doubling operations indistinguishable. Implementation results demonstrate an average reduction of 45% in LUT usage compared to similar works, emphasizing the design's resource efficiency.

Title: RAGulator: Lightweight Out-of-Context Detectors for Grounded Text Generation

• Authors: Ian Poey, Jiajun Liu, Qishuai Zhong, Adrien Chenailler
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03920
• Pdf URL: https://arxiv.org/pdf/2411.03920
• Copy Paste: [[2411.03920]] RAGulator: Lightweight Out-of-Context Detectors for Grounded Text Generation(https://arxiv.org/abs/2411.03920)
• Keywords: generative
• Abstract: Real-time detection of out-of-context LLM outputs is crucial for enterprises looking to safely adopt RAG applications. In this work, we train lightweight models to discriminate LLM-generated text that is semantically out-of-context from retrieved text documents. We preprocess a combination of summarisation and semantic textual similarity datasets to construct training data using minimal resources. We find that DeBERTa is not only the best-performing model under this pipeline, but it is also fast and does not require additional text preprocessing or feature engineering. While emerging work demonstrates that generative LLMs can also be fine-tuned and used in complex data pipelines to achieve state-of-the-art performance, we note that speed and resource limits are important considerations for on-premise deployment.

Title: Self-supervised Representation Learning for Cell Event Recognition through Time Arrow Prediction

• Authors: Cangxiong Chen, Vinay P. Namboodiri, Julia E. Sero
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03924
• Pdf URL: https://arxiv.org/pdf/2411.03924
• Copy Paste: [[2411.03924]] Self-supervised Representation Learning for Cell Event Recognition through Time Arrow Prediction(https://arxiv.org/abs/2411.03924)
• Keywords: segmentation
• Abstract: The spatio-temporal nature of live-cell microscopy data poses challenges in the analysis of cell states which is fundamental in bioimaging. Deep-learning based segmentation or tracking methods rely on large amount of high quality annotations to work effectively. In this work, we explore an alternative solution: using feature maps obtained from self-supervised representation learning (SSRL) on time arrow prediction (TAP) for the downstream supervised task of cell event recognition. We demonstrate through extensive experiments and analysis that this approach can achieve better performance with limited annotation compared to models trained from end to end using fully supervised approach. Our analysis also provides insight into applications of the SSRL using TAP in live-cell microscopy.

Title: Act in Collusion: A Persistent Distributed Multi-Target Backdoor in Federated Learning

• Authors: Tao Liu, Wu Yang, Chen Xu, Jiguang Lv, Huanran Wang, Yuhang Zhang, Shuchun Xu, Dapeng Man
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03926
• Pdf URL: https://arxiv.org/pdf/2411.03926
• Copy Paste: [[2411.03926]] Act in Collusion: A Persistent Distributed Multi-Target Backdoor in Federated Learning(https://arxiv.org/abs/2411.03926)
• Keywords: privacy, protect, attack, federate
• Abstract: Federated learning, a novel paradigm designed to protect data privacy, is vulnerable to backdoor attacks due to its distributed nature. Current research often designs attacks based on a single attacker with a single backdoor, overlooking more realistic and complex threats in federated learning. We propose a more practical threat model for federated learning: the distributed multi-target backdoor. In this model, multiple attackers control different clients, embedding various triggers and targeting different classes, collaboratively implanting backdoors into the global model via central aggregation. Empirical validation shows that existing methods struggle to maintain the effectiveness of multiple backdoors in the global model. Our key insight is that similar backdoor triggers cause parameter conflicts and injecting new backdoors disrupts gradient directions, significantly weakening some backdoors performance. To solve this, we propose a Distributed Multi-Target Backdoor Attack (DMBA), ensuring efficiency and persistence of backdoors from different malicious clients. To avoid parameter conflicts, we design a multi-channel dispersed frequency trigger strategy to maximize trigger differences. To mitigate gradient interference, we introduce backdoor replay in local training to neutralize conflicting gradients. Extensive validation shows that 30 rounds after the attack, Attack Success Rates of three different backdoors from various clients remain above 93%. The code will be made publicly available after the review period.

Title: Interactions Across Blocks in Post-Training Quantization of Large Language Models

• Authors: Khasmamad Shabanovi, Lukas Wiest, Vladimir Golkov, Daniel Cremers, Thomas Pfeil
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03934
• Pdf URL: https://arxiv.org/pdf/2411.03934
• Copy Paste: [[2411.03934]] Interactions Across Blocks in Post-Training Quantization of Large Language Models(https://arxiv.org/abs/2411.03934)
• Keywords: transformer, large language model
• Abstract: Post-training quantization is widely employed to reduce the computational demands of neural networks. Typically, individual substructures, such as layers or blocks of layers, are quantized with the objective of minimizing quantization errors in their pre-activations by fine-tuning the corresponding weights. Deriving this local objective from the global objective of minimizing task loss involves two key simplifications: assuming substructures are mutually independent and ignoring the knowledge of subsequent substructures as well as the task loss. In this work, we assess the effects of these simplifications on weight-only quantization of large language models. We introduce two multi-block fine-tuning strategies and compare them against the baseline of fine-tuning single transformer blocks. The first captures correlations of weights across blocks by jointly optimizing multiple quantized blocks. The second incorporates knowledge of subsequent blocks by minimizing the error in downstream pre-activations rather than focusing solely on the quantized block. Our findings indicate that the effectiveness of these methods depends on the specific network model, with no impact on some models but demonstrating significant benefits for others.

Title: GUIDE-VAE: Advancing Data Generation with User Information and Pattern Dictionaries

• Authors: Kutay Bölat, Simon Tindemans
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.03936
• Pdf URL: https://arxiv.org/pdf/2411.03936
• Copy Paste: [[2411.03936]] GUIDE-VAE: Advancing Data Generation with User Information and Pattern Dictionaries(https://arxiv.org/abs/2411.03936)
• Keywords: generative
• Abstract: Generative modelling of multi-user datasets has become prominent in science and engineering. Generating a data point for a given user requires employing user information, and conventional generative models, including variational autoencoders (VAEs), often ignore that. This paper introduces GUIDE-VAE, a novel conditional generative model that leverages user embeddings to generate user-guided data. By allowing the model to benefit from shared patterns across users, GUIDE-VAE enhances performance in multi-user settings, even under significant data imbalance. In addition to integrating user information, GUIDE-VAE incorporates a pattern dictionary-based covariance composition (PDCC) to improve the realism of generated samples by capturing complex feature dependencies. While user embeddings drive performance gains, PDCC addresses common issues such as noise and over-smoothing typically seen in VAEs. The proposed GUIDE-VAE was evaluated on a multi-user smart meter dataset characterized by substantial data imbalance across users. Quantitative results show that GUIDE-VAE performs effectively in both synthetic data generation and missing record imputation tasks, while qualitative evaluations reveal that GUIDE-VAE produces more plausible and less noisy data. These results establish GUIDE-VAE as a promising tool for controlled, realistic data generation in multi-user datasets, with potential applications across various domains requiring user-informed modelling.

Title: Face Reconstruction from Face Embeddings using Adapter to a Face Foundation Model

• Authors: Hatef Otroshi Shahreza, Anjith George, Sébastien Marcel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03960
• Pdf URL: https://arxiv.org/pdf/2411.03960
• Copy Paste: [[2411.03960]] Face Reconstruction from Face Embeddings using Adapter to a Face Foundation Model(https://arxiv.org/abs/2411.03960)
• Keywords: attack
• Abstract: Face recognition systems extract embedding vectors from face images and use these embeddings to verify or identify individuals. Face reconstruction attack (also known as template inversion) refers to reconstructing face images from face embeddings and using the reconstructed face image to enter a face recognition system. In this paper, we propose to use a face foundation model to reconstruct face images from the embeddings of a blackbox face recognition model. The foundation model is trained with 42M images to generate face images from the facial embeddings of a fixed face recognition model. We propose to use an adapter to translate target embeddings into the embedding space of the foundation model. The generated images are evaluated on different face recognition models and different datasets, demonstrating the effectiveness of our method to translate embeddings of different face recognition models. We also evaluate the transferability of reconstructed face images when attacking different face recognition models. Our experimental results show that our reconstructed face images outperform previous reconstruction attacks against face recognition models.

Title: How Does A Text Preprocessing Pipeline Affect Ontology Syntactic Matching?

• Authors: Zhangcheng Qiang, Kerry Taylor, Weiqing Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03962
• Pdf URL: https://arxiv.org/pdf/2411.03962
• Copy Paste: [[2411.03962]] How Does A Text Preprocessing Pipeline Affect Ontology Syntactic Matching?(https://arxiv.org/abs/2411.03962)
• Keywords: large language model
• Abstract: The generic text preprocessing pipeline, comprising Tokenisation, Normalisation, Stop Words Removal, and Stemming/Lemmatisation, has been implemented in many ontology matching (OM) systems. However, the lack of standardisation in text preprocessing creates diversity in mapping results. In this paper, we investigate the effect of the text preprocessing pipeline on OM tasks at syntactic levels. Our experiments on 8 Ontology Alignment Evaluation Initiative (OAEI) track repositories with 49 distinct alignments indicate: (1) Tokenisation and Normalisation are currently more effective than Stop Words Removal and Stemming/Lemmatisation; and (2) The selection of Lemmatisation and Stemming is task-specific. We recommend standalone Lemmatisation or Stemming with post-hoc corrections. We find that (3) Porter Stemmer and Snowball Stemmer perform better than Lancaster Stemmer; and that (4) Part-of-Speech (POS) Tagging does not help Lemmatisation. To repair less effective Stop Words Removal and Stemming/Lemmatisation used in OM tasks, we propose a novel context-based pipeline repair approach that significantly improves matching correctness and overall matching performance. We also discuss the use of text preprocessing pipeline in the new era of large language models (LLMs).

Title: What Really is Commonsense Knowledge?

• Authors: Quyet V. Do, Junze Li, Tung-Duong Vuong, Zhaowei Wang, Yangqiu Song, Xiaojuan Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03964
• Pdf URL: https://arxiv.org/pdf/2411.03964
• Copy Paste: [[2411.03964]] What Really is Commonsense Knowledge?(https://arxiv.org/abs/2411.03964)
• Keywords: large language model
• Abstract: Commonsense datasets have been well developed in Natural Language Processing, mainly through crowdsource human annotation. However, there are debates on the genuineness of commonsense reasoning benchmarks. In specific, a significant portion of instances in some commonsense benchmarks do not concern commonsense knowledge. That problem would undermine the measurement of the true commonsense reasoning ability of evaluated models. It is also suggested that the problem originated from a blurry concept of commonsense knowledge, as distinguished from other types of knowledge. To demystify all of the above claims, in this study, we survey existing definitions of commonsense knowledge, ground into the three frameworks for defining concepts, and consolidate them into a multi-framework unified definition of commonsense knowledge (so-called consolidated definition). We then use the consolidated definition for annotations and experiments on the CommonsenseQA and CommonsenseQA 2.0 datasets to examine the above claims. Our study shows that there exists a large portion of non-commonsense-knowledge instances in the two datasets, and a large performance gap on these two subsets where Large Language Models (LLMs) perform worse on commonsense-knowledge instances.

Title: HRDecoder: High-Resolution Decoder Network for Fundus Image Lesion Segmentation

• Authors: Ziyuan Ding, Yixiong Liang, Shichao Kan, Qing Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03976
• Pdf URL: https://arxiv.org/pdf/2411.03976
• Copy Paste: [[2411.03976]] HRDecoder: High-Resolution Decoder Network for Fundus Image Lesion Segmentation(https://arxiv.org/abs/2411.03976)
• Keywords: segmentation
• Abstract: High resolution is crucial for precise segmentation in fundus images, yet handling high-resolution inputs incurs considerable GPU memory costs, with diminishing performance gains as overhead increases. To address this issue while tackling the challenge of segmenting tiny objects, recent studies have explored local-global fusion methods. These methods preserve fine details using local regions and capture long-range context information from downscaled global images. However, the necessity of multiple forward passes inevitably incurs significant computational overhead, adversely affecting inference speed. In this paper, we propose HRDecoder, a simple High-Resolution Decoder network for fundus lesion segmentation. It integrates a high-resolution representation learning module to capture fine-grained local features and a high-resolution fusion module to fuse multi-scale predictions. Our method effectively improves the overall segmentation accuracy of fundus lesions while consuming reasonable memory and computational overhead, and maintaining satisfying inference speed. Experimental results on the IDRID and DDR datasets demonstrate the effectiveness of our method. Code is available at this https URL.

Title: Customized Multiple Clustering via Multi-Modal Subspace Proxy Learning

• Authors: Jiawei Yao, Qi Qian, Juhua Hu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03978
• Pdf URL: https://arxiv.org/pdf/2411.03978
• Copy Paste: [[2411.03978]] Customized Multiple Clustering via Multi-Modal Subspace Proxy Learning(https://arxiv.org/abs/2411.03978)
• Keywords: large language model
• Abstract: Multiple clustering aims to discover various latent structures of data from different aspects. Deep multiple clustering methods have achieved remarkable performance by exploiting complex patterns and relationships in data. However, existing works struggle to flexibly adapt to diverse user-specific needs in data grouping, which may require manual understanding of each clustering. To address these limitations, we introduce Multi-Sub, a novel end-to-end multiple clustering approach that incorporates a multi-modal subspace proxy learning framework in this work. Utilizing the synergistic capabilities of CLIP and GPT-4, Multi-Sub aligns textual prompts expressing user preferences with their corresponding visual representations. This is achieved by automatically generating proxy words from large language models that act as subspace bases, thus allowing for the customized representation of data in terms specific to the user's interests. Our method consistently outperforms existing baselines across a broad set of datasets in visual multiple clustering tasks. Our code is available at this https URL.

Title: ReEdit: Multimodal Exemplar-Based Image Editing with Diffusion Models

• Authors: Ashutosh Srivastava, Tarun Ram Menta, Abhinav Java, Avadhoot Jadhav, Silky Singh, Surgan Jandial, Balaji Krishnamurthy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03982
• Pdf URL: https://arxiv.org/pdf/2411.03982
• Copy Paste: [[2411.03982]] ReEdit: Multimodal Exemplar-Based Image Editing with Diffusion Models(https://arxiv.org/abs/2411.03982)
• Keywords: diffusion
• Abstract: Modern Text-to-Image (T2I) Diffusion models have revolutionized image editing by enabling the generation of high-quality photorealistic images. While the de facto method for performing edits with T2I models is through text instructions, this approach non-trivial due to the complex many-to-many mapping between natural language and images. In this work, we address exemplar-based image editing -- the task of transferring an edit from an exemplar pair to a content image(s). We propose ReEdit, a modular and efficient end-to-end framework that captures edits in both text and image modalities while ensuring the fidelity of the edited image. We validate the effectiveness of ReEdit through extensive comparisons with state-of-the-art baselines and sensitivity analyses of key design choices. Our results demonstrate that ReEdit consistently outperforms contemporary approaches both qualitatively and quantitatively. Additionally, ReEdit boasts high practical applicability, as it does not require any task-specific optimization and is four times faster than the next best baseline.

Title: Local vs distributed representations: What is the right basis for interpretability?

• Authors: Julien Colin, Lore Goetschalckx, Thomas Fel, Victor Boutin, Jay Gopal, Thomas Serre, Nuria Oliver
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03993
• Pdf URL: https://arxiv.org/pdf/2411.03993
• Copy Paste: [[2411.03993]] Local vs distributed representations: What is the right basis for interpretability?(https://arxiv.org/abs/2411.03993)
• Keywords: interpretability
• Abstract: Much of the research on the interpretability of deep neural networks has focused on studying the visual features that maximally activate individual neurons. However, recent work has cast doubts on the usefulness of such local representations for understanding the behavior of deep neural networks because individual neurons tend to respond to multiple unrelated visual patterns, a phenomenon referred to as "superposition". A promising alternative to disentangle these complex patterns is learning sparsely distributed vector representations from entire network layers, as the resulting basis vectors seemingly encode single identifiable visual patterns consistently. Thus, one would expect the resulting code to align better with human perceivable visual patterns, but supporting evidence remains, at best, anecdotal. To fill this gap, we conducted three large-scale psychophysics experiments collected from a pool of 560 participants. Our findings provide (i) strong evidence that features obtained from sparse distributed representations are easier to interpret by human observers and (ii) that this effect is more pronounced in the deepest layers of a neural network. Complementary analyses also reveal that (iii) features derived from sparse distributed representations contribute more to the model's decision. Overall, our results highlight that distributed representations constitute a superior basis for interpretability, underscoring a need for the field to move beyond the interpretation of local neural codes in favor of sparsely distributed ones.

Title: Towards Resource-Efficient Federated Learning in Industrial IoT for Multivariate Time Series Analysis

• Authors: Alexandros Gkillas, Aris Lalos
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2411.03996
• Pdf URL: https://arxiv.org/pdf/2411.03996
• Copy Paste: [[2411.03996]] Towards Resource-Efficient Federated Learning in Industrial IoT for Multivariate Time Series Analysis(https://arxiv.org/abs/2411.03996)
• Keywords: privacy, federate
• Abstract: Anomaly and missing data constitute a thorny problem in industrial applications. In recent years, deep learning enabled anomaly detection has emerged as a critical direction, however the improved detection accuracy is achieved with the utilization of large neural networks, increasing their storage and computational cost. Moreover, the data collected in edge devices contain user privacy, introducing challenges that can be successfully addressed by the privacy-preserving distributed paradigm, known as federated learning (FL). This framework allows edge devices to train and exchange models increasing also the communication cost. Thus, to deal with the increased communication, processing and storage challenges of the FL based deep anomaly detection NN pruning is expected to have significant benefits towards reducing the processing, storage and communication complexity. With this focus, a novel compression-based optimization problem is proposed at the server-side of a FL paradigm that fusses the received local models broadcast and performs pruning generating a more compressed model. Experiments in the context of anomaly detection and missing value imputation demonstrate that the proposed FL scenario along with the proposed compressed-based method are able to achieve high compression rates (more than $99.7\%$) with negligible performance losses (less than $1.18\%$ ) as compared to the centralized solutions.

Title: Aligning Characteristic Descriptors with Images for Human-Expert-like Explainability

• Authors: Bharat Chandra Yalavarthi, Nalini Ratha
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04008
• Pdf URL: https://arxiv.org/pdf/2411.04008
• Copy Paste: [[2411.04008]] Aligning Characteristic Descriptors with Images for Human-Expert-like Explainability(https://arxiv.org/abs/2411.04008)
• Keywords: explainability
• Abstract: In mission-critical domains such as law enforcement and medical diagnosis, the ability to explain and interpret the outputs of deep learning models is crucial for ensuring user trust and supporting informed decision-making. Despite advancements in explainability, existing methods often fall short in providing explanations that mirror the depth and clarity of those given by human experts. Such expert-level explanations are essential for the dependable application of deep learning models in law enforcement and medical contexts. Additionally, we recognize that most explanations in real-world scenarios are communicated primarily through natural language. Addressing these needs, we propose a novel approach that utilizes characteristic descriptors to explain model decisions by identifying their presence in images, thereby generating expert-like explanations. Our method incorporates a concept bottleneck layer within the model architecture, which calculates the similarity between image and descriptor encodings to deliver inherent and faithful explanations. Through experiments in face recognition and chest X-ray diagnosis, we demonstrate that our approach offers a significant contrast over existing techniques, which are often limited to the use of saliency maps. We believe our approach represents a significant step toward making deep learning systems more accountable, transparent, and trustworthy in the critical domains of face recognition and medical diagnosis.

Title: $k$NN Attention Demystified: A Theoretical Exploration for Scalable Transformers

• Authors: Themistoklis Haris
• Subjects: cs.LG, cs.DS
• Abstract URL: https://arxiv.org/abs/2411.04013
• Pdf URL: https://arxiv.org/pdf/2411.04013
• Copy Paste: [[2411.04013]] $k$NN Attention Demystified: A Theoretical Exploration for Scalable Transformers(https://arxiv.org/abs/2411.04013)
• Keywords: transformer
• Abstract: Despite their power, Transformers face challenges with long sequences due to the quadratic complexity of self-attention. To address this limitation, methods like $k$-Nearest-Neighbor ($k$NN) attention have been introduced [Roy, Saffar, Vaswani, Grangier, 2021] enabling each token to attend to only its $k$ closest tokens. While $k$NN attention has shown empirical success in making Transformers more efficient, its exact approximation guarantees have not been theoretically analyzed. In this work, we establish a theoretical framework for $k$NN attention, reformulating self-attention as expectations over softmax distributions and leveraging lazy Gumbel sampling [Mussmann, Levy, Ermon, 2017] with $k$NN indices for efficient approximation. Building on this framework, we also propose novel sub-quadratic algorithms that approximate self-attention gradients by leveraging efficient sampling techniques, such as Markov Chain-based estimation. Finally, we demonstrate the practical effectiveness of these algorithms through empirical experiments, showcasing their benefits in both training and inference.

Title: Prompt Engineering Using GPT for Word-Level Code-Mixed Language Identification in Low-Resource Dravidian Languages

• Authors: Aniket Deroy, Subhankar Maity
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04025
• Pdf URL: https://arxiv.org/pdf/2411.04025
• Copy Paste: [[2411.04025]] Prompt Engineering Using GPT for Word-Level Code-Mixed Language Identification in Low-Resource Dravidian Languages(https://arxiv.org/abs/2411.04025)
• Keywords: large language model
• Abstract: Language Identification (LI) is crucial for various natural language processing tasks, serving as a foundational step in applications such as sentiment analysis, machine translation, and information retrieval. In multilingual societies like India, particularly among the youth engaging on social media, text often exhibits code-mixing, blending local languages with English at different linguistic levels. This phenomenon presents formidable challenges for LI systems, especially when languages intermingle within single words. Dravidian languages, prevalent in southern India, possess rich morphological structures yet suffer from under-representation in digital platforms, leading to the adoption of Roman or hybrid scripts for communication. This paper introduces a prompt based method for a shared task aimed at addressing word-level LI challenges in Dravidian languages. In this work, we leveraged GPT-3.5 Turbo to understand whether the large language models is able to correctly classify words into correct categories. Our findings show that the Kannada model consistently outperformed the Tamil model across most metrics, indicating a higher accuracy and reliability in identifying and categorizing Kannada language instances. In contrast, the Tamil model showed moderate performance, particularly needing improvement in precision and recall.

Title: Quantum-Safe Hybrid Key Exchanges with KEM-Based Authentication

• Authors: Christopher Battarbee, Christoph Striecks, Ludovic Perret, Sebastian Ramacher, Kevin Verhaeghe
• Subjects: cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/2411.04030
• Pdf URL: https://arxiv.org/pdf/2411.04030
• Copy Paste: [[2411.04030]] Quantum-Safe Hybrid Key Exchanges with KEM-Based Authentication(https://arxiv.org/abs/2411.04030)
• Keywords: security
• Abstract: Authenticated Key Exchange (AKE) between any two entities is one of the most important security protocols available for securing our digital networks and infrastructures. In PQCrypto 2023, Bruckner, Ramacher and Striecks proposed a novel hybrid AKE (HAKE) protocol, dubbed Muckle+, that is particularly useful in large quantum-safe networks consisting of a large number of nodes. Their protocol is hybrid in the sense that it allows key material from conventional and post-quantum primitives, as well as from quantum key distribution, to be incorporated into a single end-to-end shared key. To achieve the desired authentication properties, Muckle+ utilizes post-quantum digital signatures. However, available instantiations of such signatures schemes are not yet efficient enough compared to their post-quantum key-encapsulation mechanism (KEM) counterparts, particularly in large networks with potentially several connections in a short period of time. To mitigate this gap, we propose Muckle# that pushes the efficiency boundaries of currently known HAKE constructions. Muckle# uses post-quantum key-encapsulating mechanisms for implicit authentication inspired by recent works done in the area of Transport Layer Security (TLS) protocols, particularly, in KEMTLS (CCS'20). We port those ideas to the HAKE framework and develop novel proof techniques on the way. Due to our novel KEM-based approach, the resulting protocol has a slightly different message flow compared to prior work that we carefully align with the HAKE framework and which makes our changes to the Muckle+ non-trivial.

Title: Beemo: Benchmark of Expert-edited Machine-generated Outputs

• Authors: Ekaterina Artemova, Jason Lucas, Saranya Venkatraman, Jooyoung Lee, Sergei Tilga, Adaku Uchendu, Vladislav Mikhailov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04032
• Pdf URL: https://arxiv.org/pdf/2411.04032
• Copy Paste: [[2411.04032]] Beemo: Benchmark of Expert-edited Machine-generated Outputs(https://arxiv.org/abs/2411.04032)
• Keywords: large language model
• Abstract: The rapid proliferation of large language models (LLMs) has increased the volume of machine-generated texts (MGTs) and blurred text authorship in various domains. However, most existing MGT benchmarks include single-author texts (human-written and machine-generated). This conventional design fails to capture more practical multi-author scenarios, where the user refines the LLM response for natural flow, coherence, and factual correctness. Our paper introduces the Benchmark of Expert-edited Machine-generated Outputs (Beemo), which includes 6.5k texts written by humans, generated by ten instruction-finetuned LLMs, and edited by experts for various use cases, ranging from creative writing to summarization. Beemo additionally comprises 13.1k machine-generated and LLM-edited texts, allowing for diverse MGT detection evaluation across various edit types. We document Beemo's creation protocol and present the results of benchmarking 33 configurations of MGT detectors in different experimental setups. We find that expert-based editing evades MGT detection, while LLM-edited texts are unlikely to be recognized as human-written. Beemo and all materials are publicly available.

Title: Pseudo-labeling with Keyword Refining for Few-Supervised Video Captioning

• Authors: Ping Li, Tao Wang, Xinkui Zhao, Xianghua Xu, Mingli Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04059
• Pdf URL: https://arxiv.org/pdf/2411.04059
• Copy Paste: [[2411.04059]] Pseudo-labeling with Keyword Refining for Few-Supervised Video Captioning(https://arxiv.org/abs/2411.04059)
• Keywords: transformer
• Abstract: Video captioning generate a sentence that describes the video content. Existing methods always require a number of captions (\eg, 10 or 20) per video to train the model, which is quite costly. In this work, we explore the possibility of using only one or very few ground-truth sentences, and introduce a new task named few-supervised video captioning. Specifically, we propose a few-supervised video captioning framework that consists of lexically constrained pseudo-labeling module and keyword-refined captioning module. Unlike the random sampling in natural language processing that may cause invalid modifications (\ie, edit words), the former module guides the model to edit words using some actions (\eg, copy, replace, insert, and delete) by a pretrained token-level classifier, and then fine-tunes candidate sentences by a pretrained language model. Meanwhile, the former employs the repetition penalized sampling to encourage the model to yield concise pseudo-labeled sentences with less repetition, and selects the most relevant sentences upon a pretrained video-text model. Moreover, to keep semantic consistency between pseudo-labeled sentences and video content, we develop the transformer-based keyword refiner with the video-keyword gated fusion strategy to emphasize more on relevant words. Extensive experiments on several benchmarks demonstrate the advantages of the proposed approach in both few-supervised and fully-supervised scenarios. The code implementation is available at this https URL

Title: Security Assessment of Mobile Banking Apps in West African Economic and Monetary Union

• Authors: Alioune Diallo, Aicha War, Moustapha Awwalou Diouf, Jordan Samhi, Steven Arzt, Tegawendé F. Bissyande, Jacque Klein
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.04068
• Pdf URL: https://arxiv.org/pdf/2411.04068
• Copy Paste: [[2411.04068]] Security Assessment of Mobile Banking Apps in West African Economic and Monetary Union(https://arxiv.org/abs/2411.04068)
• Keywords: security, attack, robust
• Abstract: The West African Economic and Monetary Union (WAEMU) states, characterized by widespread smartphone usage, have witnessed banks and financial institutions introducing mobile banking applications (MBAs). These apps empower users to perform transactions such as money transfers, bill payments, and account inquiries anytime, anywhere. However, this proliferation of MBAs also raises significant security concerns. Poorly implemented security measures during app development can expose users and financial institutions to substantial financial risks through increased vulnerability to cyberattacks. Our study evaluated fifty-nine WAEMU MBAs using static analysis techniques. These MBAs were collected from the 160 banks and financial institutions of the eight WAEMU countries listed on the Central Bank of West African States (BCEAO) website. We identified security-related code issues that could be exploited by malicious actors. We investigated the issues found in the older versions to track their evolution across updates. Additionally, we identified some banks from regions such as Europe, the United States, and other developing countries and analyzed their mobile apps for a security comparison with WAEMU MBAs. Key findings include: (1) WAEMU apps exhibit security issues introduced during development, posing significant risks of exploitation; (2) Despite frequent updates, underlying security issues often persist; (3) Compared to MBAs from developed and developing countries, WAEMU apps exhibit fewer critical security issues; and (4) Apps from banks that are branches of other non-WAEMU banks often inherit security concerns from their parent apps while also introducing additional issues unique to their context. Our research underscores the need for robust security practices in WAEMU MBAs development to enhance user safety and trust in financial services.

Title: Textual Decomposition Then Sub-motion-space Scattering for Open-Vocabulary Motion Generation

• Authors: Ke Fan, Jiangning Zhang, Ran Yi, Jingyu Gong, Yabiao Wang, Yating Wang, Xin Tan, Chengjie Wang, Lizhuang Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04079
• Pdf URL: https://arxiv.org/pdf/2411.04079
• Copy Paste: [[2411.04079]] Textual Decomposition Then Sub-motion-space Scattering for Open-Vocabulary Motion Generation(https://arxiv.org/abs/2411.04079)
• Keywords: large language model
• Abstract: Text-to-motion generation is a crucial task in computer vision, which generates the target 3D motion by the given text. The existing annotated datasets are limited in scale, resulting in most existing methods overfitting to the small datasets and unable to generalize to the motions of the open domain. Some methods attempt to solve the open-vocabulary motion generation problem by aligning to the CLIP space or using the Pretrain-then-Finetuning paradigm. However, the current annotated dataset's limited scale only allows them to achieve mapping from sub-text-space to sub-motion-space, instead of mapping between full-text-space and full-motion-space (full mapping), which is the key to attaining open-vocabulary motion generation. To this end, this paper proposes to leverage the atomic motion (simple body part motions over a short time period) as an intermediate representation, and leverage two orderly coupled steps, i.e., Textual Decomposition and Sub-motion-space Scattering, to address the full mapping problem. For Textual Decomposition, we design a fine-grained description conversion algorithm, and combine it with the generalization ability of a large language model to convert any given motion text into atomic texts. Sub-motion-space Scattering learns the compositional process from atomic motions to the target motions, to make the learned sub-motion-space scattered to form the full-motion-space. For a given motion of the open domain, it transforms the extrapolation into interpolation and thereby significantly improves generalization. Our network, $DSO$-Net, combines textual $d$ecomposition and sub-motion-space $s$cattering to solve the $o$pen-vocabulary motion generation. Extensive experiments demonstrate that our DSO-Net achieves significant improvements over the state-of-the-art methods on open-vocabulary motion generation. Code is available at this https URL.

Title: Summarization of Opinionated Political Documents with Varied Perspectives

• Authors: Nicholas Deas, Kathleen McKeown
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04093
• Pdf URL: https://arxiv.org/pdf/2411.04093
• Copy Paste: [[2411.04093]] Summarization of Opinionated Political Documents with Varied Perspectives(https://arxiv.org/abs/2411.04093)
• Keywords: extraction
• Abstract: Global partisan hostility and polarization has increased, and this polarization is heightened around presidential elections. Models capable of generating accurate summaries of diverse perspectives can help reduce such polarization by exposing users to alternative perspectives. In this work, we introduce a novel dataset and task for independently summarizing each political perspective in a set of passages from opinionated news articles. For this task, we propose a framework for evaluating different dimensions of perspective summary performance. We benchmark 10 models of varying sizes and architectures through both automatic and human evaluation. While recent models like GPT-4o perform well on this task, we find that all models struggle to generate summaries faithful to the intended perspective. Our analysis of summaries focuses on how extraction behavior depends on the features of the input documents.

Title: RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models

• Authors: Maya Varma, Jean-Benoit Delbrouck, Zhihong Chen, Akshay Chaudhari, Curtis Langlotz
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04097
• Pdf URL: https://arxiv.org/pdf/2411.04097
• Copy Paste: [[2411.04097]] RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models(https://arxiv.org/abs/2411.04097)
• Keywords: robust
• Abstract: Fine-tuned vision-language models (VLMs) often capture spurious correlations between image features and textual attributes, resulting in degraded zero-shot performance at test time. Existing approaches for addressing spurious correlations (i) primarily operate at the global image-level rather than intervening directly on fine-grained image features and (ii) are predominantly designed for unimodal settings. In this work, we present RaVL, which takes a fine-grained perspective on VLM robustness by discovering and mitigating spurious correlations using local image features rather than operating at the global image level. Given a fine-tuned VLM, RaVL first discovers spurious correlations by leveraging a region-level clustering approach to identify precise image features contributing to zero-shot classification errors. Then, RaVL mitigates the identified spurious correlation with a novel region-aware loss function that enables the VLM to focus on relevant regions and ignore spurious relationships during fine-tuning. We evaluate RaVL on 654 VLMs with various model architectures, data domains, and learned spurious correlations. Our results show that RaVL accurately discovers (191% improvement over the closest baseline) and mitigates (8.2% improvement on worst-group image classification accuracy) spurious correlations. Qualitative evaluations on general-domain and medical-domain VLMs confirm our findings.

Title: Interpretable and Efficient Data-driven Discovery and Control of Distributed Systems

• Authors: Florian Wolf, Nicolò Botteghi, Urban Fasel, Andrea Manzoni
• Subjects: cs.LG, cs.CE, math.OC
• Abstract URL: https://arxiv.org/abs/2411.04098
• Pdf URL: https://arxiv.org/pdf/2411.04098
• Copy Paste: [[2411.04098]] Interpretable and Efficient Data-driven Discovery and Control of Distributed Systems(https://arxiv.org/abs/2411.04098)
• Keywords: robust, interpretability
• Abstract: Effectively controlling systems governed by Partial Differential Equations (PDEs) is crucial in several fields of Applied Sciences and Engineering. These systems usually yield significant challenges to conventional control schemes due to their nonlinear dynamics, partial observability, high-dimensionality once discretized, distributed nature, and the requirement for low-latency feedback control. Reinforcement Learning (RL), particularly Deep RL (DRL), has recently emerged as a promising control paradigm for such systems, demonstrating exceptional capabilities in managing high-dimensional, nonlinear dynamics. However, DRL faces challenges including sample inefficiency, robustness issues, and an overall lack of interpretability. To address these issues, we propose a data-efficient, interpretable, and scalable Dyna-style Model-Based RL framework for PDE control, combining the Sparse Identification of Nonlinear Dynamics with Control (SINDy-C) algorithm and an autoencoder (AE) framework for the sake of dimensionality reduction of PDE states and actions. This novel approach enables fast rollouts, reducing the need for extensive environment interactions, and provides an interpretable latent space representation of the PDE forward dynamics. We validate our method on two PDE problems describing fluid flows - namely, the 1D Burgers equation and 2D Navier-Stokes equations - comparing it against a model-free baseline, and carrying out an extensive analysis of the learned dynamics.

Title: How Transformers Solve Propositional Logic Problems: A Mechanistic Analysis

• Authors: Guan Zhe Hong, Nishanth Dikkala, Enming Luo, Cyrus Rashtchian, Rina Panigrahy
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04105
• Pdf URL: https://arxiv.org/pdf/2411.04105
• Copy Paste: [[2411.04105]] How Transformers Solve Propositional Logic Problems: A Mechanistic Analysis(https://arxiv.org/abs/2411.04105)
• Keywords: transformer, large language model
• Abstract: Large language models (LLMs) have shown amazing performance on tasks that require planning and reasoning. Motivated by this, we investigate the internal mechanisms that underpin a network's ability to perform complex logical reasoning. We first construct a synthetic propositional logic problem that serves as a concrete test-bed for network training and evaluation. Crucially, this problem demands nontrivial planning to solve, but we can train a small transformer to achieve perfect accuracy. Building on our set-up, we then pursue an understanding of precisely how a three-layer transformer, trained from scratch, solves this problem. We are able to identify certain "planning" and "reasoning" circuits in the network that necessitate cooperation between the attention blocks to implement the desired logic. To expand our findings, we then study a larger model, Mistral 7B. Using activation patching, we characterize internal components that are critical in solving our logic problem. Overall, our work systemically uncovers novel aspects of small and large transformers, and continues the study of how they plan and reason.

Title: Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?

• Authors: Daniel P. Jeong, Saurabh Garg, Zachary C. Lipton, Michael Oberst
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04118
• Pdf URL: https://arxiv.org/pdf/2411.04118
• Copy Paste: [[2411.04118]] Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?(https://arxiv.org/abs/2411.04118)
• Keywords: large language model
• Abstract: Several recent works seek to develop foundation models specifically for medical applications, adapting general-purpose large language models (LLMs) and vision-language models (VLMs) via continued pretraining on publicly available biomedical corpora. These works typically claim that such domain-adaptive pretraining (DAPT) improves performance on downstream medical tasks, such as answering medical licensing exam questions. In this paper, we compare seven public "medical" LLMs and two VLMs against their corresponding base models, arriving at a different conclusion: all medical VLMs and nearly all medical LLMs fail to consistently improve over their base models in the zero-/few-shot prompting regime for medical question-answering (QA) tasks. For instance, across the tasks and model pairs we consider in the 3-shot setting, medical LLMs only outperform their base models in 12.1% of cases, reach a (statistical) tie in 49.8% of cases, and are significantly worse than their base models in the remaining 38.2% of cases. Our conclusions are based on (i) comparing each medical model head-to-head, directly against the corresponding base model; (ii) optimizing the prompts for each model separately; and (iii) accounting for statistical uncertainty in comparisons. While these basic practices are not consistently adopted in the literature, our ablations show that they substantially impact conclusions. Our findings suggest that state-of-the-art general-domain models may already exhibit strong medical knowledge and reasoning capabilities, and offer recommendations to strengthen the conclusions of future studies.

Title: Community Forensics: Using Thousands of Generators to Train Fake Image Detectors

• Authors: Jeongsoo Park, Andrew Owens
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04125
• Pdf URL: https://arxiv.org/pdf/2411.04125
• Copy Paste: [[2411.04125]] Community Forensics: Using Thousands of Generators to Train Fake Image Detectors(https://arxiv.org/abs/2411.04125)
• Keywords: diffusion, generative
• Abstract: One of the key challenges of detecting AI-generated images is spotting images that have been created by previously unseen generative models. We argue that the limited diversity of the training data is a major obstacle to addressing this problem, and we propose a new dataset that is significantly larger and more diverse than prior work. As part of creating this dataset, we systematically download thousands of text-to-image latent diffusion models and sample images from them. We also collect images from dozens of popular open source and commercial models. The resulting dataset contains 2.7M images that have been sampled from 4803 different models. These images collectively capture a wide range of scene content, generator architectures, and image processing settings. Using this dataset, we study the generalization abilities of fake image detectors. Our experiments suggest that detection performance improves as the number of models in the training set increases, even when these models have similar architectures. We also find that detection performance improves as the diversity of the models increases, and that our trained detectors generalize better than those trained on other datasets.