2024-10-23

Title: Solution for OOD-CV UNICORN Challenge 2024 Object Detection Assistance LLM Counting Ability Improvement

• Authors: Zhouyang Chi, Qingyuan Jiang, Yang Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16287
• Pdf URL: https://arxiv.org/pdf/2410.16287
• Copy Paste: [[2410.16287]] Solution for OOD-CV UNICORN Challenge 2024 Object Detection Assistance LLM Counting Ability Improvement(https://arxiv.org/abs/2410.16287)
• Keywords: robust, large language model
• Abstract: This report provide a detailed description of the method that we explored and proposed in the ECCV OOD-CV UNICORN Challenge 2024, which focusing on the robustness of responses from large language models. The dataset of this competition are OODCA-VQA and SketchyQA. In order to test the robustness of the model. The organizer extended two variants of the dataset OODCV-Counterfactual and Sketchy-Challenging. There are several difficulties with these datasets. Firstly, the Sketchy-Challenging dataset uses some rarer item categories to test the model's generalization ability. Secondly, in the OODCV-Counterfactual dataset, the given problems often have inflection points and computational steps, requiring the model to recognize them during the inference process. In order to address this issue, we propose a simple yet effective approach called Object Detection Assistance Large Language Model(LLM) Counting Ability Improvement(ODAC), which focuses on using the object detection model to assist the LLM. To clarify, our approach contains two main blocks: (1)Object Detection Assistance. (2) Counterfactual Specific prompt. Our approach ranked second in the final test with a score of 0.86.

Title: QML-IDS: Quantum Machine Learning Intrusion Detection System

• Authors: Diego Abreu, Christian Esteve Rothenberg, Antonio Abelem
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.16308
• Pdf URL: https://arxiv.org/pdf/2410.16308
• Copy Paste: [[2410.16308]] QML-IDS: Quantum Machine Learning Intrusion Detection System(https://arxiv.org/abs/2410.16308)
• Keywords: security, attack
• Abstract: The emergence of quantum computing and related technologies presents opportunities for enhancing network security. The transition towards quantum computational power paves the way for creating strategies to mitigate the constantly advancing threats to network integrity. In response to this technological advancement, our research presents QML-IDS, a novel Intrusion Detection System~(IDS) that combines quantum and classical computing techniques. QML-IDS employs Quantum Machine Learning~(QML) methodologies to analyze network patterns and detect attack activities. Through extensive experimental tests on publicly available datasets, we show that QML-IDS is effective at attack detection and performs well in binary and multiclass classification tasks. Our findings reveal that QML-IDS outperforms classical Machine Learning methods, demonstrating the promise of quantum-enhanced cybersecurity solutions for the age of quantum utility.

Title: A Computational Harmonic Detection Algorithm to Detect Data Leakage through EM Emanation

• Authors: Md Faizul Bari, Meghna Roy Chowdhury, Shreyas Sen
• Subjects: cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.16316
• Pdf URL: https://arxiv.org/pdf/2410.16316
• Copy Paste: [[2410.16316]] A Computational Harmonic Detection Algorithm to Detect Data Leakage through EM Emanation(https://arxiv.org/abs/2410.16316)
• Keywords: security, defense
• Abstract: Unintended electromagnetic emissions from electronic devices, known as EM emanations, pose significant security risks because they can be processed to recover the source signal's information content. Defense organizations typically use metal shielding to prevent data leakage, but this approach is costly and impractical for widespread use, especially in uncontrolled environments like government facilities in the wild. This is particularly relevant for IoT devices due to their large numbers and deployment in varied environments. This gives rise to a research need for an automated emanation detection method to monitor the facilities and take prompt steps when leakage is detected. To address this, in the preliminary version of this work [1], we collected emanation data from 3 types of HDMI cables and proposed a CNN-based detection method that provided 95% accuracy up to 22.5m. However, the CNN-based method has some limitations: hardware dependency, confusion among multiple sources, and struggle at low SNR. In this extended version, we augment the initial study by collecting emanation data from IoT devices, everyday electronic devices, and cables. Data analysis reveals that each device's emanation has a unique harmonic pattern with intermodulation products, in contrast to communication signals with fixed frequency bands, spectra, and modulation patterns. Leveraging this, we propose a harmonic-based detection method by developing a computational harmonic detector. The proposed method addresses the limitations of the CNN method and provides ~100 accuracy not only for HDMI emanation (compared to 95% in the earlier CNN-based method) but also for all other tested devices/cables in different environments.

Title: A Survey on Physical Adversarial Attacks against Face Recognition Systems

• Authors: Mingsi Wang, Jiachen Zhou, Tianlin Li, Guozhu Meng, Kai Chen
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16317
• Pdf URL: https://arxiv.org/pdf/2410.16317
• Copy Paste: [[2410.16317]] A Survey on Physical Adversarial Attacks against Face Recognition Systems(https://arxiv.org/abs/2410.16317)
• Keywords: security, defense, attack
• Abstract: As Face Recognition (FR) technology becomes increasingly prevalent in finance, the military, public safety, and everyday life, security concerns have grown substantially. Physical adversarial attacks targeting FR systems in real-world settings have attracted considerable research interest due to their practicality and the severe threats they pose. However, a systematic overview focused on physical adversarial attacks against FR systems is still lacking, hindering an in-depth exploration of the challenges and future directions in this field. In this paper, we bridge this gap by comprehensively collecting and analyzing physical adversarial attack methods targeting FR systems. Specifically, we first investigate the key challenges of physical attacks on FR systems. We then categorize existing physical attacks into three categories based on the physical medium used and summarize how the research in each category has evolved to address these challenges. Furthermore, we review current defense strategies and discuss potential future research directions. Our goal is to provide a fresh, comprehensive, and deep understanding of physical adversarial attacks against FR systems, thereby inspiring relevant research in this area.

Title: Accelerating Object Detection with YOLOv4 for Real-Time Applications

• Authors: K. Senthil Kumar, K.M.B. Abdullah Safwan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16320
• Pdf URL: https://arxiv.org/pdf/2410.16320
• Copy Paste: [[2410.16320]] Accelerating Object Detection with YOLOv4 for Real-Time Applications(https://arxiv.org/abs/2410.16320)
• Keywords: security
• Abstract: Object Detection is related to Computer Vision. Object detection enables detecting instances of objects in images and videos. Due to its increased utilization in surveillance, tracking system used in security and many others applications have propelled researchers to continuously derive more efficient and competitive algorithms. However, problems emerges while implementing it in real-time because of their dynamic environment and complex algorithms used in object detection. In the last few years, Convolution Neural Network (CNN) have emerged as a powerful tool for recognizing image content and in computer vision approach for most problems. In this paper, We revived begins the brief introduction of deep learning and object detection framework like Convolutional Neural Network(CNN), You only look once - version 4 (YOLOv4). Then we focus on our proposed object detection architectures along with some modifications. The traditional model detects a small object in images. We have some modifications to the model. Our proposed method gives the correct result with accuracy.

Title: SouLLMate: An Application Enhancing Diverse Mental Health Support with Adaptive LLMs, Prompt Engineering, and RAG Techniques

• Authors: Qiming Guo, Jinwen Tang, Wenbo Sun, Haoteng Tang, Yi Shang, Wenlu Wang
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.16322
• Pdf URL: https://arxiv.org/pdf/2410.16322
• Copy Paste: [[2410.16322]] SouLLMate: An Application Enhancing Diverse Mental Health Support with Adaptive LLMs, Prompt Engineering, and RAG Techniques(https://arxiv.org/abs/2410.16322)
• Keywords: extraction
• Abstract: Mental health issues significantly impact individuals' daily lives, yet many do not receive the help they need even with available online resources. This study aims to provide diverse, accessible, stigma-free, personalized, and real-time mental health support through cutting-edge AI technologies. It makes the following contributions: (1) Conducting an extensive survey of recent mental health support methods to identify prevalent functionalities and unmet needs. (2) Introducing SouLLMate, an adaptive LLM-driven system that integrates LLM technologies, Chain, Retrieval-Augmented Generation (RAG), prompt engineering, and domain knowledge. This system offers advanced features such as Risk Detection and Proactive Guidance Dialogue, and utilizes RAG for personalized profile uploads and Conversational Information Extraction. (3) Developing novel evaluation approaches for preliminary assessments and risk detection via professionally annotated interview data and real-life suicide tendency data. (4) Proposing the Key Indicator Summarization (KIS), Proactive Questioning Strategy (PQS), and Stacked Multi-Model Reasoning (SMMR) methods to enhance model performance and usability through context-sensitive response adjustments, semantic coherence evaluations, and enhanced accuracy of long-context reasoning in language models. This study contributes to advancing mental health support technologies, potentially improving the accessibility and effectiveness of mental health care globally.

Title: CybORG++: An Enhanced Gym for the Development of Autonomous Cyber Agents

• Authors: Harry Emerson, Liz Bates, Chris Hicks, Vasilios Mavroudis
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16324
• Pdf URL: https://arxiv.org/pdf/2410.16324
• Copy Paste: [[2410.16324]] CybORG++: An Enhanced Gym for the Development of Autonomous Cyber Agents(https://arxiv.org/abs/2410.16324)
• Keywords: robust
• Abstract: CybORG++ is an advanced toolkit for reinforcement learning research focused on network defence. Building on the CAGE 2 CybORG environment, it introduces key improvements, including enhanced debugging capabilities, refined agent implementation support, and a streamlined environment that enables faster training and easier customisation. Along with addressing several software bugs from its predecessor, CybORG++ introduces MiniCAGE, a lightweight version of CAGE 2, which improves performance dramatically, up to 1000x faster execution in parallel iterations, without sacrificing accuracy or core functionality. CybORG++ serves as a robust platform for developing and evaluating defensive agents, making it a valuable resource for advancing enterprise network defence research.

Title: Synthetic Data Generation in Cybersecurity: A Comparative Analysis

• Authors: Dure Adan Ammara, Jianguo Ding, Kurt Tutschku
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.16326
• Pdf URL: https://arxiv.org/pdf/2410.16326
• Copy Paste: [[2410.16326]] Synthetic Data Generation in Cybersecurity: A Comparative Analysis(https://arxiv.org/abs/2410.16326)
• Keywords: security, generative
• Abstract: Synthetic data generation faces significant challenges in accurately replicating real data, particularly with tabular data, where achieving high fidelity and utility is critical. While numerous methods have been developed, the most effective approach for creating high-quality synthetic data for network traffic security remains to be seen. This study conducts a comprehensive comparative analysis of non-AI, conventional AI, and generative AI techniques for synthetic tabular data generation using two widely recognized cybersecurity datasets: NSL-KDD and CICIDS-2017. Particular emphasis was placed on prominent GAN models for tabular data generation, including CTGAN, CopulaGAN, GANBLR++, and CastGAN. The results indicate that GAN-based methods, particularly CTGAN and CopulaGAN, outperform non-AI and conventional AI approaches in terms of fidelity and utility. To the best of our knowledge, this research contributes to the field by offering the first comparative evaluation of these methods specifically for cybersecurity network traffic data, filling a critical gap in the literature. It also introduces mutual information for feature selection, further enhancing the quality of the generated synthetic data. These findings provide valuable guidance for researchers seeking the most suitable synthetic data generation method in cybersecurity applications.

Title: Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMs

• Authors: Rui Pu, Chaozhuo Li, Rui Ha, Zejian Chen, Litian Zhang, Zheng Liu, Lirong Qiu, Xi Zhang
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16327
• Pdf URL: https://arxiv.org/pdf/2410.16327
• Copy Paste: [[2410.16327]] Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMs(https://arxiv.org/abs/2410.16327)
• Keywords: security, protect, defense, attack, robust, large language model
• Abstract: Jailbreak attack can be used to access the vulnerabilities of Large Language Models (LLMs) by inducing LLMs to generate the harmful content. And the most common method of the attack is to construct semantically ambiguous prompts to confuse and mislead the LLMs. To access the security and reveal the intrinsic relation between the input prompt and the output for LLMs, the distribution of attention weight is introduced to analyze the underlying reasons. By using statistical analysis methods, some novel metrics are defined to better describe the distribution of attention weight, such as the Attention Intensity on Sensitive Words (Attn_SensWords), the Attention-based Contextual Dependency Score (Attn_DepScore) and Attention Dispersion Entropy (Attn_Entropy). By leveraging the distinct characteristics of these metrics, the beam search algorithm and inspired by the military strategy "Feint and Attack", an effective jailbreak attack strategy named as Attention-Based Attack (ABA) is proposed. In the ABA, nested attack prompts are employed to divert the attention distribution of the LLMs. In this manner, more harmless parts of the input can be used to attract the attention of the LLMs. In addition, motivated by ABA, an effective defense strategy called as Attention-Based Defense (ABD) is also put forward. Compared with ABA, the ABD can be used to enhance the robustness of LLMs by calibrating the attention distribution of the input prompt. Some comparative experiments have been given to demonstrate the effectiveness of ABA and ABD. Therefore, both ABA and ABD can be used to access the security of the LLMs. The comparative experiment results also give a logical explanation that the distribution of attention weight can bring great influence on the output for LLMs.

Title: The Solution for Single Object Tracking Task of Perception Test Challenge 2024

• Authors: Zhiqiang Zhong, Yang Yang, Fengqiang Wan, Henglu Wei, Xiangyang Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16329
• Pdf URL: https://arxiv.org/pdf/2410.16329
• Copy Paste: [[2410.16329]] The Solution for Single Object Tracking Task of Perception Test Challenge 2024(https://arxiv.org/abs/2410.16329)
• Keywords: robust
• Abstract: This report presents our method for Single Object Tracking (SOT), which aims to track a specified object throughout a video sequence. We employ the LoRAT method. The essence of the work lies in adapting LoRA, a technique that fine-tunes a small subset of model parameters without adding inference latency, to the domain of visual tracking. We train our model using the extensive LaSOT and GOT-10k datasets, which provide a solid foundation for robust performance. Additionally, we implement the alpha-refine technique for post-processing the bounding box outputs. Although the alpha-refine method does not yield the anticipated results, our overall approach achieves a score of 0.813, securing first place in the competition.

Title: Disambiguating Monocular Reconstruction of 3D Clothed Human with Spatial-Temporal Transformer

• Authors: Yong Deng, Baoxing Li, Xu Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16337
• Pdf URL: https://arxiv.org/pdf/2410.16337
• Copy Paste: [[2410.16337]] Disambiguating Monocular Reconstruction of 3D Clothed Human with Spatial-Temporal Transformer(https://arxiv.org/abs/2410.16337)
• Keywords: robust, transformer
• Abstract: Reconstructing 3D clothed humans from monocular camera data is highly challenging due to viewpoint limitations and image ambiguity. While implicit function-based approaches, combined with prior knowledge from parametric models, have made significant progress, there are still two notable problems. Firstly, the back details of human models are ambiguous due to viewpoint invisibility. The quality of the back details depends on the back normal map predicted by a convolutional neural network (CNN). However, the CNN lacks global information awareness for comprehending the back texture, resulting in excessively smooth back details. Secondly, a single image suffers from local ambiguity due to lighting conditions and body movement. However, implicit functions are highly sensitive to pixel variations in ambiguous regions. To address these ambiguities, we propose the Spatial-Temporal Transformer (STT) network for 3D clothed human reconstruction. A spatial transformer is employed to extract global information for normal map prediction. The establishment of global correlations facilitates the network in comprehending the holistic texture and shape of the human body. Simultaneously, to compensate for local ambiguity in images, a temporal transformer is utilized to extract temporal features from adjacent frames. The incorporation of temporal features can enhance the accuracy of input features in implicit networks. Furthermore, to obtain more accurate temporal features, joint tokens are employed to establish local correspondences between frames. Experimental results on the Adobe and MonoPerfCap datasets have shown that our method outperforms state-of-the-art methods and maintains robust generalization even under low-light outdoor conditions.

Title: Vulnerabilities in Machine Learning-Based Voice Disorder Detection Systems

• Authors: Gianpaolo Perelli, Andrea Panzino, Roberto Casula, Marco Micheletto, Giulia Orrù, Gian Luca Marcialis
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16341
• Pdf URL: https://arxiv.org/pdf/2410.16341
• Copy Paste: [[2410.16341]] Vulnerabilities in Machine Learning-Based Voice Disorder Detection Systems(https://arxiv.org/abs/2410.16341)
• Keywords: security, attack
• Abstract: The impact of voice disorders is becoming more widely acknowledged as a public health issue. Several machine learning-based classifiers with the potential to identify disorders have been used in recent studies to differentiate between normal and pathological voices and sounds. In this paper, we focus on analyzing the vulnerabilities of these systems by exploring the possibility of attacks that can reverse classification and compromise their reliability. Given the critical nature of personal health information, understanding which types of attacks are effective is a necessary first step toward improving the security of such systems. Starting from the original audios, we implement various attack methods, including adversarial, evasion, and pitching techniques, and evaluate how state-of-the-art disorder detection models respond to them. Our findings identify the most effective attack strategies, underscoring the need to address these vulnerabilities in machine-learning systems used in the healthcare domain.

Title: Exploring how deep learning decodes anomalous diffusion via Grad-CAM

• Authors: Jaeyong Bae, Yongjoo Baek, Hawoong Jeong
• Subjects: cs.LG, physics.data-an
• Abstract URL: https://arxiv.org/abs/2410.16345
• Pdf URL: https://arxiv.org/pdf/2410.16345
• Copy Paste: [[2410.16345]] Exploring how deep learning decodes anomalous diffusion via Grad-CAM(https://arxiv.org/abs/2410.16345)
• Keywords: robust, diffusion
• Abstract: While deep learning has been successfully applied to the data-driven classification of anomalous diffusion mechanisms, how the algorithm achieves the feat still remains a mystery. In this study, we use a well-known technique aimed at achieving explainable AI, namely the Gradient-weighted Class Activation Map (Grad-CAM), to investigate how deep learning (implemented by ResNets) recognizes the distinctive features of a particular anomalous diffusion model from the raw trajectory data. Our results show that Grad-CAM reveals the portions of the trajectory that hold crucial information about the underlying mechanism of anomalous diffusion, which can be utilized to enhance the robustness of the trained classifier against the measurement noise. Moreover, we observe that deep learning distills unique statistical characteristics of different diffusion mechanisms at various spatiotemporal scales, with larger-scale (smaller-scale) features identified at higher (lower) layers.

Title: Large Language Models in Computer Science Education: A Systematic Literature Review

• Authors: Nishat Raihan, Mohammed Latif Siddiq, Joanna C.S. Santos, Marcos Zampieri
• Subjects: cs.LG, cs.HC
• Abstract URL: https://arxiv.org/abs/2410.16349
• Pdf URL: https://arxiv.org/pdf/2410.16349
• Copy Paste: [[2410.16349]] Large Language Models in Computer Science Education: A Systematic Literature Review(https://arxiv.org/abs/2410.16349)
• Keywords: transformer, generative, large language model
• Abstract: Large language models (LLMs) are becoming increasingly better at a wide range of Natural Language Processing tasks (NLP), such as text generation and understanding. Recently, these models have extended their capabilities to coding tasks, bridging the gap between natural languages (NL) and programming languages (PL). Foundational models such as the Generative Pre-trained Transformer (GPT) and LLaMA series have set strong baseline performances in various NL and PL tasks. Additionally, several models have been fine-tuned specifically for code generation, showing significant improvements in code-related applications. Both foundational and fine-tuned models are increasingly used in education, helping students write, debug, and understand code. We present a comprehensive systematic literature review to examine the impact of LLMs in computer science and computer engineering education. We analyze their effectiveness in enhancing the learning experience, supporting personalized education, and aiding educators in curriculum development. We address five research questions to uncover insights into how LLMs contribute to educational outcomes, identify challenges, and suggest directions for future research.

Title: Quantum inspired factorization up to 100-bit RSA number in polynomial time

• Authors: Marco Tesoro, Ilaria Siloi, Daniel Jaschke, Giuseppe Magnifico, Simone Montangero
• Subjects: cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/2410.16355
• Pdf URL: https://arxiv.org/pdf/2410.16355
• Copy Paste: [[2410.16355]] Quantum inspired factorization up to 100-bit RSA number in polynomial time(https://arxiv.org/abs/2410.16355)
• Keywords: security, attack
• Abstract: Classical public-key cryptography standards rely on the Rivest-Shamir-Adleman (RSA) encryption protocol. The security of this protocol is based on the exponential computational complexity of the most efficient classical algorithms for factoring large semiprime numbers into their two prime components. Here, we attack RSA factorization building on Schnorr's mathematical framework where factorization translates into a combinatorial optimization problem. We solve the optimization task via tensor network methods, a quantum-inspired classical numerical technique. This tensor network Schnorr's sieving algorithm displays numerical evidence of a polynomial scaling of the resources with the bit-length of the semiprime. We factorize RSA numbers up to 100 bits encoding the optimization problem in quantum systems with up to 256 qubits. Only the high-order polynomial scaling of the required resources limits the factorization of larger numbers. Although these results do not currently undermine the security of the present communication infrastructure, they strongly highlight the urgency of implementing post-quantum cryptography or quantum key distribution.

Title: KatzBot: Revolutionizing Academic Chatbot for Enhanced Communication

• Authors: Sahil Kumar, Deepa Paikar, Kiran Sai Vutukuri, Haider Ali, Shashidhar Reddy Ainala, Aditya Murli Krishnan, Youshan Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16385
• Pdf URL: https://arxiv.org/pdf/2410.16385
• Copy Paste: [[2410.16385]] KatzBot: Revolutionizing Academic Chatbot for Enhanced Communication(https://arxiv.org/abs/2410.16385)
• Keywords: large language model
• Abstract: Effective communication within universities is crucial for addressing the diverse information needs of students, alumni, and external stakeholders. However, existing chatbot systems often fail to deliver accurate, context-specific responses, resulting in poor user experiences. In this paper, we present KatzBot, an innovative chatbot powered by KatzGPT, a custom Large Language Model (LLM) fine-tuned on domain-specific academic data. KatzGPT is trained on two university-specific datasets: 6,280 sentence-completion pairs and 7,330 question-answer pairs. KatzBot outperforms established existing open source LLMs, achieving higher accuracy and domain relevance. KatzBot offers a user-friendly interface, significantly enhancing user satisfaction in real-world applications. The source code is publicly available at \url{this https URL}.

Title: LEGO-Learn: Label-Efficient Graph Open-Set Learning

• Authors: Haoyan Xu, Kay Liu, Zhengtao Yao, Philip S. Yu, Kaize Ding, Yue Zhao
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2410.16386
• Pdf URL: https://arxiv.org/pdf/2410.16386
• Copy Paste: [[2410.16386]] LEGO-Learn: Label-Efficient Graph Open-Set Learning(https://arxiv.org/abs/2410.16386)
• Keywords: security
• Abstract: How can we train graph-based models to recognize unseen classes while keeping labeling costs low? Graph open-set learning (GOL) and out-of-distribution (OOD) detection aim to address this challenge by training models that can accurately classify known, in-distribution (ID) classes while identifying and handling previously unseen classes during inference. It is critical for high-stakes, real-world applications where models frequently encounter unexpected data, including finance, security, and healthcare. However, current GOL methods assume access to many labeled ID samples, which is unrealistic for large-scale graphs due to high annotation costs. In this paper, we propose LEGO-Learn (Label-Efficient Graph Open-set Learning), a novel framework that tackles open-set node classification on graphs within a given label budget by selecting the most informative ID nodes. LEGO-Learn employs a GNN-based filter to identify and exclude potential OOD nodes and then select highly informative ID nodes for labeling using the K-Medoids algorithm. To prevent the filter from discarding valuable ID examples, we introduce a classifier that differentiates between the C known ID classes and an additional class representing OOD nodes (hence, a C+1 classifier). This classifier uses a weighted cross-entropy loss to balance the removal of OOD nodes while retaining informative ID nodes. Experimental results on four real-world datasets demonstrate that LEGO-Learn significantly outperforms leading methods, with up to a 6.62% improvement in ID classification accuracy and a 7.49% increase in AUROC for OOD detection.

Title: Joker: Conditional 3D Head Synthesis with Extreme Facial Expressions

• Authors: Malte Prinzler, Egor Zakharov, Vanessa Sklyarova, Berna Kabadayi, Justus Thies
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2410.16395
• Pdf URL: https://arxiv.org/pdf/2410.16395
• Copy Paste: [[2410.16395]] Joker: Conditional 3D Head Synthesis with Extreme Facial Expressions(https://arxiv.org/abs/2410.16395)
• Keywords: diffusion
• Abstract: We introduce Joker, a new method for the conditional synthesis of 3D human heads with extreme expressions. Given a single reference image of a person, we synthesize a volumetric human head with the reference identity and a new expression. We offer control over the expression via a 3D morphable model (3DMM) and textual inputs. This multi-modal conditioning signal is essential since 3DMMs alone fail to define subtle emotional changes and extreme expressions, including those involving the mouth cavity and tongue articulation. Our method is built upon a 2D diffusion-based prior that generalizes well to out-of-domain samples, such as sculptures, heavy makeup, and paintings while achieving high levels of expressiveness. To improve view consistency, we propose a new 3D distillation technique that converts predictions of our 2D prior into a neural radiance field (NeRF). Both the 2D prior and our distillation technique produce state-of-the-art results, which are confirmed by our extensive evaluations. Also, to the best of our knowledge, our method is the first to achieve view-consistent extreme tongue articulation.

Title: Federated Communication-Efficient Multi-Objective Optimization

• Authors: Baris Askin, Pranay Sharma, Gauri Joshi, Carlee Joe-Wong
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.16398
• Pdf URL: https://arxiv.org/pdf/2410.16398
• Copy Paste: [[2410.16398]] Federated Communication-Efficient Multi-Objective Optimization(https://arxiv.org/abs/2410.16398)
• Keywords: federate
• Abstract: We study a federated version of multi-objective optimization (MOO), where a single model is trained to optimize multiple objective functions. MOO has been extensively studied in the centralized setting but is less explored in federated or distributed settings. We propose FedCMOO, a novel communication-efficient federated multi-objective optimization (FMOO) algorithm that improves the error convergence performance of the model compared to existing approaches. Unlike prior works, the communication cost of FedCMOO does not scale with the number of objectives, as each client sends a single aggregated gradient, obtained using randomized SVD (singular value decomposition), to the central server. We provide a convergence analysis of the proposed method for smooth non-convex objective functions under milder assumptions than in prior work. In addition, we introduce a variant of FedCMOO that allows users to specify a preference over the objectives in terms of a desired ratio of the final objective values. Through extensive experiments, we demonstrate the superiority of our proposed method over baseline approaches.

Title: Hotel Booking Cancellation Prediction Using Applied Bayesian Models

• Authors: Md Asifuzzaman Jishan, Vikas Singh, Ayan Kumar Ghosh, Md Shahabub Alam, Khan Raqib Mahmud, Bijan Paul
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16406
• Pdf URL: https://arxiv.org/pdf/2410.16406
• Copy Paste: [[2410.16406]] Hotel Booking Cancellation Prediction Using Applied Bayesian Models(https://arxiv.org/abs/2410.16406)
• Keywords: robust
• Abstract: This study applies Bayesian models to predict hotel booking cancellations, a key challenge affecting resource allocation, revenue, and customer satisfaction in the hospitality industry. Using a Kaggle dataset with 36,285 observations and 17 features, Bayesian Logistic Regression and Beta-Binomial models were implemented. The logistic model, applied to 12 features and 5,000 randomly selected observations, outperformed the Beta-Binomial model in predictive accuracy. Key predictors included the number of adults, children, stay duration, lead time, car parking space, room type, and special requests. Model evaluation using Leave-One-Out Cross-Validation (LOO-CV) confirmed strong alignment between observed and predicted outcomes, demonstrating the model's robustness. Special requests and parking availability were found to be the strongest predictors of cancellation. This Bayesian approach provides a valuable tool for improving booking management and operational efficiency in the hotel industry.

Title: On conditional diffusion models for PDE simulations

• Authors: Aliaksandra Shysheya, Cristiana Diaconu, Federico Bergamin, Paris Perdikaris, José Miguel Hernández-Lobato, Richard E. Turner, Emile Mathieu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16415
• Pdf URL: https://arxiv.org/pdf/2410.16415
• Copy Paste: [[2410.16415]] On conditional diffusion models for PDE simulations(https://arxiv.org/abs/2410.16415)
• Keywords: diffusion
• Abstract: Modelling partial differential equations (PDEs) is of crucial importance in science and engineering, and it includes tasks ranging from forecasting to inverse problems, such as data assimilation. However, most previous numerical and machine learning approaches that target forecasting cannot be applied out-of-the-box for data assimilation. Recently, diffusion models have emerged as a powerful tool for conditional generation, being able to flexibly incorporate observations without retraining. In this work, we perform a comparative study of score-based diffusion models for forecasting and assimilation of sparse observations. In particular, we focus on diffusion models that are either trained in a conditional manner, or conditioned after unconditional training. We address the shortcomings of existing models by proposing 1) an autoregressive sampling approach that significantly improves performance in forecasting, 2) a new training strategy for conditional score-based models that achieves stable performance over a range of history lengths, and 3) a hybrid model which employs flexible pre-training conditioning on initial conditions and flexible post-training conditioning to handle data assimilation. We empirically show that these modifications are crucial for successfully tackling the combination of forecasting and data assimilation, a task commonly encountered in real-world scenarios.

Title: AttentionPainter: An Efficient and Adaptive Stroke Predictor for Scene Painting

• Authors: Yizhe Tang, Yue Wang, Teng Hu, Ran Yi, Xin Tan, Lizhuang Ma, Yu-Kun Lai, Paul L. Rosin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16418
• Pdf URL: https://arxiv.org/pdf/2410.16418
• Copy Paste: [[2410.16418]] AttentionPainter: An Efficient and Adaptive Stroke Predictor for Scene Painting(https://arxiv.org/abs/2410.16418)
• Keywords: diffusion
• Abstract: Stroke-based Rendering (SBR) aims to decompose an input image into a sequence of parameterized strokes, which can be rendered into a painting that resembles the input image. Recently, Neural Painting methods that utilize deep learning and reinforcement learning models to predict the stroke sequences have been developed, but suffer from longer inference time or unstable training. To address these issues, we propose AttentionPainter, an efficient and adaptive model for single-step neural painting. First, we propose a novel scalable stroke predictor, which predicts a large number of stroke parameters within a single forward process, instead of the iterative prediction of previous Reinforcement Learning or auto-regressive methods, which makes AttentionPainter faster than previous neural painting methods. To further increase the training efficiency, we propose a Fast Stroke Stacking algorithm, which brings 13 times acceleration for training. Moreover, we propose Stroke-density Loss, which encourages the model to use small strokes for detailed information, to help improve the reconstruction quality. Finally, we propose a new stroke diffusion model for both conditional and unconditional stroke-based generation, which denoises in the stroke parameter space and facilitates stroke-based inpainting and editing applications helpful for human artists design. Extensive experiments show that AttentionPainter outperforms the state-of-the-art neural painting methods.

Title: Position: Challenges and Opportunities for Differential Privacy in the U.S. Federal Government

• Authors: Amol Khanna, Adam McCormick, Andre Nguyen, Chris Aguirre, Edward Raff
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16423
• Pdf URL: https://arxiv.org/pdf/2410.16423
• Copy Paste: [[2410.16423]] Position: Challenges and Opportunities for Differential Privacy in the U.S. Federal Government(https://arxiv.org/abs/2410.16423)
• Keywords: security, privacy
• Abstract: In this article, we seek to elucidate challenges and opportunities for differential privacy within the federal government setting, as seen by a team of differential privacy researchers, privacy lawyers, and data scientists working closely with the U.S. government. After introducing differential privacy, we highlight three significant challenges which currently restrict the use of differential privacy in the U.S. government. We then provide two examples where differential privacy can enhance the capabilities of government agencies. The first example highlights how the quantitative nature of differential privacy allows policy security officers to release multiple versions of analyses with different levels of privacy. The second example, which we believe is a novel realization, indicates that differential privacy can be used to improve staffing efficiency in classified applications. We hope that this article can serve as a nontechnical resource which can help frame future action from the differential privacy community, privacy regulators, security officers, and lawmakers.

Title: HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality

• Authors: Zhiming Hu, Guanhua Zhang, Zheming Yin, Daniel Haeufle, Syn Schmitt, Andreas Bulling
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16430
• Pdf URL: https://arxiv.org/pdf/2410.16430
• Copy Paste: [[2410.16430]] HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality(https://arxiv.org/abs/2410.16430)
• Keywords: diffusion
• Abstract: Human hand and head movements are the most pervasive input modalities in extended reality (XR) and are significant for a wide range of applications. However, prior works on hand and head modelling in XR only explored a single modality or focused on specific applications. We present HaHeAE - a novel self-supervised method for learning generalisable joint representations of hand and head movements in XR. At the core of our method is an autoencoder (AE) that uses a graph convolutional network-based semantic encoder and a diffusion-based stochastic encoder to learn the joint semantic and stochastic representations of hand-head movements. It also features a diffusion-based decoder to reconstruct the original signals. Through extensive evaluations on three public XR datasets, we show that our method 1) significantly outperforms commonly used self-supervised methods by up to 74.0% in terms of reconstruction quality and is generalisable across users, activities, and XR environments, 2) enables new applications, including interpretable hand-head cluster identification and variable hand-head movement generation, and 3) can serve as an effective feature extractor for downstream tasks. Together, these results demonstrate the effectiveness of our method and underline the potential of self-supervised methods for jointly modelling hand-head behaviours in extended reality.

Title: Fair Bilevel Neural Network (FairBiNN): On Balancing fairness and accuracy via Stackelberg Equilibrium

• Authors: Mehdi Yazdani-Jahromi, Ali Khodabandeh Yalabadi, AmirArsalan Rajabi, Aida Tayebi, Ivan Garibay, Ozlem Ozmen Garibay
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16432
• Pdf URL: https://arxiv.org/pdf/2410.16432
• Copy Paste: [[2410.16432]] Fair Bilevel Neural Network (FairBiNN): On Balancing fairness and accuracy via Stackelberg Equilibrium(https://arxiv.org/abs/2410.16432)
• Keywords: robust, fair
• Abstract: The persistent challenge of bias in machine learning models necessitates robust solutions to ensure parity and equal treatment across diverse groups, particularly in classification tasks. Current methods for mitigating bias often result in information loss and an inadequate balance between accuracy and fairness. To address this, we propose a novel methodology grounded in bilevel optimization principles. Our deep learning-based approach concurrently optimizes for both accuracy and fairness objectives, and under certain assumptions, achieving proven Pareto optimal solutions while mitigating bias in the trained model. Theoretical analysis indicates that the upper bound on the loss incurred by this method is less than or equal to the loss of the Lagrangian approach, which involves adding a regularization term to the loss function. We demonstrate the efficacy of our model primarily on tabular datasets such as UCI Adult and Heritage Health. When benchmarked against state-of-the-art fairness methods, our model exhibits superior performance, advancing fairness-aware machine learning solutions and bridging the accuracy-fairness gap. The implementation of FairBiNN is available on this https URL.

Title: Secure Computation and Trustless Data Intermediaries in Data Spaces

• Authors: Christoph Fabianek, Stephan Krenn, Thomas Loruenser, Veronika Siska
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.16442
• Pdf URL: https://arxiv.org/pdf/2410.16442
• Copy Paste: [[2410.16442]] Secure Computation and Trustless Data Intermediaries in Data Spaces(https://arxiv.org/abs/2410.16442)
• Keywords: secure, security
• Abstract: This paper explores the integration of advanced cryptographic techniques for secure computation in data spaces to enable secure and trusted data sharing, which is essential for the evolving data economy. In addition, the paper examines the role of data intermediaries, as outlined in the EU Data Governance Act, in data spaces and specifically introduces the idea of trustless intermediaries that do not have access to their users' data. Therefore, we exploit the introduced secure computation methods, i.e. Secure Multi-Party Computation (MPC) and Fully Homomorphic Encryption (FHE), and discuss the security benefits. Overall, we identify and address key challenges for integration, focusing on areas such as identity management, policy enforcement, node selection, and access control, and present solutions through real-world use cases, including air traffic management, manufacturing, and secondary data use. Furthermore, through the analysis of practical applications, this work proposes a comprehensive framework for the implementation and standardization of secure computing technologies in dynamic, trustless data environments, paving the way for future research and development of a secure and interoperable data ecosystem.

Title: Improving Neuron-level Interpretability with White-box Language Models

• Authors: Hao Bai, Yi Ma
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16443
• Pdf URL: https://arxiv.org/pdf/2410.16443
• Copy Paste: [[2410.16443]] Improving Neuron-level Interpretability with White-box Language Models(https://arxiv.org/abs/2410.16443)
• Keywords: robust, interpretability, transformer
• Abstract: Neurons in auto-regressive language models like GPT-2 can be interpreted by analyzing their activation patterns. Recent studies have shown that techniques such as dictionary learning, a form of post-hoc sparse coding, enhance this neuron-level interpretability. In our research, we are driven by the goal to fundamentally improve neural network interpretability by embedding sparse coding directly within the model architecture, rather than applying it as an afterthought. In our study, we introduce a white-box transformer-like architecture named Coding RAte TransformEr (CRATE), explicitly engineered to capture sparse, low-dimensional structures within data distributions. Our comprehensive experiments showcase significant improvements (up to 103% relative improvement) in neuron-level interpretability across a variety of evaluation metrics. Detailed investigations confirm that this enhanced interpretability is steady across different layers irrespective of the model size, underlining CRATE's robust performance in enhancing neural network interpretability. Further analysis shows that CRATE's increased interpretability comes from its enhanced ability to consistently and distinctively activate on relevant tokens. These findings point towards a promising direction for creating white-box foundation models that excel in neuron-level interpretation.

Title: Does your LLM truly unlearn? An embarrassingly simple approach to recover unlearned knowledge

• Authors: Zhiwei Zhang, Fali Wang, Xiaomin Li, Zongyu Wu, Xianfeng Tang, Hui Liu, Qi He, Wenpeng Yin, Suhang Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16454
• Pdf URL: https://arxiv.org/pdf/2410.16454
• Copy Paste: [[2410.16454]] Does your LLM truly unlearn? An embarrassingly simple approach to recover unlearned knowledge(https://arxiv.org/abs/2410.16454)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) have shown remarkable proficiency in generating text, benefiting from extensive training on vast textual corpora. However, LLMs may also acquire unwanted behaviors from the diverse and sensitive nature of their training data, which can include copyrighted and private content. Machine unlearning has been introduced as a viable solution to remove the influence of such problematic content without the need for costly and time-consuming retraining. This process aims to erase specific knowledge from LLMs while preserving as much model utility as possible. Despite the effectiveness of current unlearning methods, little attention has been given to whether existing unlearning methods for LLMs truly achieve forgetting or merely hide the knowledge, which current unlearning benchmarks fail to detect. This paper reveals that applying quantization to models that have undergone unlearning can restore the "forgotten" information. To thoroughly evaluate this phenomenon, we conduct comprehensive experiments using various quantization techniques across multiple precision levels. We find that for unlearning methods with utility constraints, the unlearned model retains an average of 21\% of the intended forgotten knowledge in full precision, which significantly increases to 83\% after 4-bit quantization. Based on our empirical findings, we provide a theoretical explanation for the observed phenomenon and propose a quantization-robust unlearning strategy to mitigate this intricate issue...

Title: To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planning

• Authors: Da JU, Song Jiang, Andrew Cohen, Aaron Foss, Sasha Mitts, Arman Zharmagambetov, Brandon Amos, Xian Li, Justine T Kao, Maryam Fazel-Zarandi, Yuandong Tian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16456
• Pdf URL: https://arxiv.org/pdf/2410.16456
• Copy Paste: [[2410.16456]] To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planning(https://arxiv.org/abs/2410.16456)
• Keywords: large language model
• Abstract: Travel planning is a challenging and time-consuming task that aims to find an itinerary which satisfies multiple, interdependent constraints regarding flights, accommodations, attractions, and other travel arrangements. In this paper, we propose To the Globe (TTG), a real-time demo system that takes natural language requests from users, translates it to symbolic form via a fine-tuned Large Language Model, and produces optimal travel itineraries with Mixed Integer Linear Programming solvers. The overall system takes ~5 seconds to reply to the user request with guaranteed itineraries. To train TTG, we develop a synthetic data pipeline that generates user requests, flight and hotel information in symbolic form without human annotations, based on the statistics of real-world datasets, and fine-tune an LLM to translate NL user requests to their symbolic form, which is sent to the symbolic solver to compute optimal itineraries. Our NL-symbolic translation achieves ~91% exact match in a backtranslation metric (i.e., whether the estimated symbolic form of generated natural language matches the groundtruth), and its returned itineraries have a ratio of 0.979 compared to the optimal cost of the ground truth user request. When evaluated by users, TTG achieves consistently high Net Promoter Scores (NPS) of 35-40% on generated itinerary.

Title: Comparative Study of Multilingual Idioms and Similes in Large Language Models

• Authors: Paria Khoshtab, Danial Namazifard, Mostafa Masoudi, Ali Akhgary, Samin Mahdizadeh Sani, Yadollah Yaghoobzadeh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16461
• Pdf URL: https://arxiv.org/pdf/2410.16461
• Copy Paste: [[2410.16461]] Comparative Study of Multilingual Idioms and Similes in Large Language Models(https://arxiv.org/abs/2410.16461)
• Keywords: large language model
• Abstract: This study addresses the gap in the literature concerning the comparative performance of LLMs in interpreting different types of figurative language across multiple languages. By evaluating LLMs using two multilingual datasets on simile and idiom interpretation, we explore the effectiveness of various prompt engineering strategies, including chain-of-thought, few-shot, and English translation prompts. We extend the language of these datasets to Persian as well by building two new evaluation sets. Our comprehensive assessment involves both closed-source (GPT-3.5, GPT-4o mini, Gemini 1.5), and open-source models (Llama 3.1, Qwen2), highlighting significant differences in performance across languages and figurative types. Our findings reveal that while prompt engineering methods are generally effective, their success varies by figurative type, language, and model. We also observe that open-source models struggle particularly with low-resource languages in similes. Additionally, idiom interpretation is nearing saturation for many languages, necessitating more challenging evaluations.

Title: In Search of the Successful Interpolation: On the Role of Sharpness in CLIP Generalization

• Authors: Alireza Abdollahpoorrostam
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16476
• Pdf URL: https://arxiv.org/pdf/2410.16476
• Copy Paste: [[2410.16476]] In Search of the Successful Interpolation: On the Role of Sharpness in CLIP Generalization(https://arxiv.org/abs/2410.16476)
• Keywords: robust
• Abstract: \textit{Zero-shot} models like CLIP are often fine-tuned on a target dataset to improve its accuracy further, but this can compromise out-of-distribution (OOD) robustness. Robust Fine-Tuning (\texttt{RFT} )~\citep{wortsman2021robust}, which interpolates between the \textit{zero-shot} and \textit{fine-tuned} models, has been proposed to address this issue. However, understanding when \texttt{RFT} actually improves OOD error remains limited. In this work, we empirically investigate the robustness of \texttt{RFT} in CLIP models, with a focus on the \textit{sharpness} of the CLIP model during interpolation. First, we demonstrate that while sharpness may not serve as a reliable indicator for predicting the generalization of modern architectures like CLIP on OOD data, this challenges the conventional belief in the generalization benefits of flat minima in foundation models. However, by examining the role of the \textit{straggler layer} phenomenon, we show that, unlike overall sharpness, the \textit{layer-wise} sharpness of \textit{straggler} layers can reliably capture the generalization performance of interpolated CLIP models on OOD data. Our extensive experiments reveal that \textit{layer-wise} sharpness correlates with generalization in OOD accuracy for \texttt{RFT}. Furthermore, we demonstrate that by inducing sparsity in the \textit{straggler} layers, we can mitigate the \textit{failure mode} phenomenon in \texttt{RFT}. To the best of our knowledge, this is the first work to study the role of sharpness in the \textit{success} of interpolation in the weight space of CLIP foundation models. Our code is available at \url{this https URL}.

Title: Identifying Sub-networks in Neural Networks via Functionally Similar Representations

• Authors: Tian Gao, Amit Dhurandhar, Karthikeyan Natesan Ramamurthy, Dennis Wei
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16484
• Pdf URL: https://arxiv.org/pdf/2410.16484
• Copy Paste: [[2410.16484]] Identifying Sub-networks in Neural Networks via Functionally Similar Representations(https://arxiv.org/abs/2410.16484)
• Keywords: interpretability
• Abstract: Mechanistic interpretability aims to provide human-understandable insights into the inner workings of neural network models by examining their internals. Existing approaches typically require significant manual effort and prior knowledge, with strategies tailored to specific tasks. In this work, we take a step toward automating the understanding of the network by investigating the existence of distinct sub-networks. Specifically, we explore a novel automated and task-agnostic approach based on the notion of functionally similar representations within neural networks, reducing the need for human intervention. Our method identifies similar and dissimilar layers in the network, revealing potential sub-components. We achieve this by proposing, for the first time to our knowledge, the use of Gromov-Wasserstein distance, which overcomes challenges posed by varying distributions and dimensionalities across intermediate representations, issues that complicate direct layer-to-layer comparisons. Through experiments on algebraic and language tasks, we observe the emergence of sub-groups within neural network layers corresponding to functional abstractions. Additionally, we find that different training strategies influence the positioning of these sub-groups. Our approach offers meaningful insights into the behavior of neural networks with minimal human and computational cost.

Title: GenGMM: Generalized Gaussian-Mixture-based Domain Adaptation Model for Semantic Segmentation

• Authors: Nazanin Moradinasab, Hassan Jafarzadeh, Donald E. Brown
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16485
• Pdf URL: https://arxiv.org/pdf/2410.16485
• Copy Paste: [[2410.16485]] GenGMM: Generalized Gaussian-Mixture-based Domain Adaptation Model for Semantic Segmentation(https://arxiv.org/abs/2410.16485)
• Keywords: segmentation
• Abstract: Domain adaptive semantic segmentation is the task of generating precise and dense predictions for an unlabeled target domain using a model trained on a labeled source domain. While significant efforts have been devoted to improving unsupervised domain adaptation for this task, it is crucial to note that many models rely on a strong assumption that the source data is entirely and accurately labeled, while the target data is unlabeled. In real-world scenarios, however, we often encounter partially or noisy labeled data in source and target domains, referred to as Generalized Domain Adaptation (GDA). In such cases, we suggest leveraging weak or unlabeled data from both domains to narrow the gap between them, resulting in effective adaptation. We introduce the Generalized Gaussian-mixture-based (GenGMM) domain adaptation model, which harnesses the underlying data distribution in both domains to refine noisy weak and pseudo labels. The experiments demonstrate the effectiveness of our approach.

Title: LLM-TS Integrator: Integrating LLM for Enhanced Time Series Modeling

• Authors: Can Chen, Gabriel Oliveira, Hossein Sharifi Noghabi, Tristan Sylvain
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16489
• Pdf URL: https://arxiv.org/pdf/2410.16489
• Copy Paste: [[2410.16489]] LLM-TS Integrator: Integrating LLM for Enhanced Time Series Modeling(https://arxiv.org/abs/2410.16489)
• Keywords: large language model
• Abstract: Time series~(TS) modeling is essential in dynamic systems like weather prediction and anomaly detection. Recent studies utilize Large Language Models (LLMs) for TS modeling, leveraging their powerful pattern recognition capabilities. These methods primarily position LLMs as the predictive backbone, often omitting the mathematical modeling within traditional TS models, such as periodicity. However, disregarding the potential of LLMs also overlooks their pattern recognition capabilities. To address this gap, we introduce \textit{LLM-TS Integrator}, a novel framework that effectively integrates the capabilities of LLMs into traditional TS modeling. Central to this integration is our \textit{mutual information} module. The core of this \textit{mutual information} module is a traditional TS model enhanced with LLM-derived insights for improved predictive abilities. This enhancement is achieved by maximizing the mutual information between traditional model's TS representations and LLM's textual representation counterparts, bridging the two modalities. Moreover, we recognize that samples vary in importance for two losses: traditional prediction and mutual information maximization. To address this variability, we introduce the \textit{sample reweighting} module to improve information utilization. This module assigns dual weights to each sample: one for prediction loss and another for mutual information loss, dynamically optimizing these weights via bi-level optimization. Our method achieves state-of-the-art or comparable performance across five mainstream TS tasks, including short-term and long-term forecasting, imputation, classification, and anomaly detection.

Title: Hacking the Fabric: Targeting Partial Reconfiguration for Fault Injection in FPGA Fabrics

• Authors: Jayeeta Chaudhuri, Hassan Nassar, Dennis R.E. Gnad, Jorg Henkel, Mehdi B. Tahoori, Krishnendu Chakrabarty
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.16497
• Pdf URL: https://arxiv.org/pdf/2410.16497
• Copy Paste: [[2410.16497]] Hacking the Fabric: Targeting Partial Reconfiguration for Fault Injection in FPGA Fabrics(https://arxiv.org/abs/2410.16497)
• Keywords: security, attack
• Abstract: FPGAs are now ubiquitous in cloud computing infrastructures and reconfigurable system-on-chip, particularly for AI acceleration. Major cloud service providers such as Amazon and Microsoft are increasingly incorporating FPGAs for specialized compute-intensive tasks within their data centers. The availability of FPGAs in cloud data centers has opened up new opportunities for users to improve application performance by implementing customizable hardware accelerators directly on the FPGA fabric. However, the virtualization and sharing of FPGA resources among multiple users open up new security risks and threats. We present a novel fault attack methodology capable of causing persistent fault injections in partial bitstreams during the process of FPGA reconfiguration. This attack leverages power-wasters and is timed to inject faults into bitstreams as they are being loaded onto the FPGA through the reconfiguration manager, without needing to remain active throughout the entire reconfiguration process. Our experiments, conducted on a Pynq FPGA setup, demonstrate the feasibility of this attack on various partial application bitstreams, such as a neural network accelerator unit and a signal processing accelerator unit.

Title: Natural Language Processing for Human Resources: A Survey

• Authors: Naoki Otani, Nikita Bhutani, Estevam Hruschka
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16498
• Pdf URL: https://arxiv.org/pdf/2410.16498
• Copy Paste: [[2410.16498]] Natural Language Processing for Human Resources: A Survey(https://arxiv.org/abs/2410.16498)
• Keywords: extraction
• Abstract: The domain of human resources (HR) includes a broad spectrum of tasks related to natural language processing (NLP) techniques. Recent breakthroughs in NLP have generated significant interest in its industrial applications in this domain and potentially alleviate challenges such as the difficulty of resource acquisition and the complexity of problems. At the same time, the HR domain can also present unique challenges that drive state-of-the-art in NLP research. To support this, we provide NLP researchers and practitioners with an overview of key HR tasks from an NLP perspective, illustrating how specific sub-tasks (e.g., skill extraction) contribute to broader objectives (e.g., job matching). Through this survey, we identify opportunities in NLP for HR and suggest directions for future exploration.

Title: SINGAPO: Single Image Controlled Generation of Articulated Parts in Object

• Authors: Jiayi Liu, Denys Iliash, Angel X. Chang, Manolis Savva, Ali Mahdavi-Amiri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16499
• Pdf URL: https://arxiv.org/pdf/2410.16499
• Copy Paste: [[2410.16499]] SINGAPO: Single Image Controlled Generation of Articulated Parts in Object(https://arxiv.org/abs/2410.16499)
• Keywords: diffusion
• Abstract: We address the challenge of creating 3D assets for household articulated objects from a single image. Prior work on articulated object creation either requires multi-view multi-state input, or only allows coarse control over the generation process. These limitations hinder the scalability and practicality for articulated object modeling. In this work, we propose a method to generate articulated objects from a single image. Observing the object in resting state from an arbitrary view, our method generates an articulated object that is visually consistent with the input image. To capture the ambiguity in part shape and motion posed by a single view of the object, we design a diffusion model that learns the plausible variations of objects in terms of geometry and kinematics. To tackle the complexity of generating structured data with attributes in multiple domains, we design a pipeline that produces articulated objects from high-level structure to geometric details in a coarse-to-fine manner, where we use a part connectivity graph and part abstraction as proxies. Our experiments show that our method outperforms the state-of-the-art in articulated object creation by a large margin in terms of the generated object realism, resemblance to the input image, and reconstruction quality.

Title: Rulebreakers Challenge: Revealing a Blind Spot in Large Language Models' Reasoning with Formal Logic

• Authors: Jason Chan, Robert Gaizauskas, Zhixue Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16502
• Pdf URL: https://arxiv.org/pdf/2410.16502
• Copy Paste: [[2410.16502]] Rulebreakers Challenge: Revealing a Blind Spot in Large Language Models' Reasoning with Formal Logic(https://arxiv.org/abs/2410.16502)
• Keywords: large language model
• Abstract: Formal logic has long been applied to natural language reasoning, but this approach can sometimes lead to conclusions that, while logically entailed, are factually inconsistent with the premises or are not typically inferred by humans. This study introduces the concept of "rulebreakers", which refers to instances where logical entailment diverges from factually acceptable inference. We present RULEBREAKERS, a novel dataset for evaluating Large Language Models' (LLMs) ability to distinguish between rulebreakers and non-rulebreakers. Focusing on modus tollens and disjunctive syllogism, we assess six state-of-the-art LLMs using RULEBREAKERS, measuring their performance in terms of token-level exact accuracy and model confidence. Our findings reveal that while most models perform poorly to moderately in recognizing rulebreakers, they demonstrate a latent ability to distinguish rulebreakers when assessed by their confidence levels. Further analysis suggests that the failure to recognize rulebreakers is potentially associated with the models' world knowledge and their attention distribution patterns. This research highlights the limitation of LLMs' reasoning capabilities, and contributes to the ongoing discussion on reasoning in LLMs.

Title: TIPS: Text-Image Pretraining with Spatial Awareness

• Authors: Kevis-Kokitsi Maninis, Kaifeng Chen, Soham Ghosh, Arjun Karpur, Koert Chen, Ye Xia, Bingyi Cao, Daniel Salz, Guangxing Han, Jan Dlabal, Dan Gnanapragasam, Mojtaba Seyedhosseini, Howard Zhou, Andre Araujo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16512
• Pdf URL: https://arxiv.org/pdf/2410.16512
• Copy Paste: [[2410.16512]] TIPS: Text-Image Pretraining with Spatial Awareness(https://arxiv.org/abs/2410.16512)
• Keywords: transformer, segmentation
• Abstract: While image-text representation learning has become very popular in recent years, existing models tend to lack spatial awareness and have limited direct applicability for dense understanding tasks. For this reason, self-supervised image-only pretraining is still the go-to method for many dense vision applications (e.g. depth estimation, semantic segmentation), despite the lack of explicit supervisory signals. In this paper, we close this gap between image-text and self-supervised learning, by proposing a novel general-purpose image-text model, which can be effectively used off-the-shelf for dense and global vision tasks. Our method, which we refer to as Text-Image Pretraining with Spatial awareness (TIPS), leverages two simple and effective insights. First, on textual supervision: we reveal that replacing noisy web image captions by synthetically generated textual descriptions boosts dense understanding performance significantly, due to a much richer signal for learning spatially aware representations. We propose an adapted training method that combines noisy and synthetic captions, resulting in improvements across both dense and global understanding tasks. Second, on the learning technique: we propose to combine contrastive image-text learning with self-supervised masked image modeling, to encourage spatial coherence, unlocking substantial enhancements for downstream applications. Building on these two ideas, we scale our model using the transformer architecture, trained on a curated set of public images. Our experiments are conducted on 8 tasks involving 16 datasets in total, demonstrating strong off-the-shelf performance on both dense and global understanding, for several image-only and image-text tasks.

Title: Scalability of memorization-based machine unlearning

• Authors: Kairan Zhao, Peter Triantafillou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16516
• Pdf URL: https://arxiv.org/pdf/2410.16516
• Copy Paste: [[2410.16516]] Scalability of memorization-based machine unlearning(https://arxiv.org/abs/2410.16516)
• Keywords: privacy
• Abstract: Machine unlearning (MUL) focuses on removing the influence of specific subsets of data (such as noisy, poisoned, or privacy-sensitive data) from pretrained models. MUL methods typically rely on specialized forms of fine-tuning. Recent research has shown that data memorization is a key characteristic defining the difficulty of MUL. As a result, novel memorization-based unlearning methods have been developed, demonstrating exceptional performance with respect to unlearning quality, while maintaining high performance for model utility. Alas, these methods depend on knowing the memorization scores of data points and computing said scores is a notoriously time-consuming process. This in turn severely limits the scalability of these solutions and their practical impact for real-world applications. In this work, we tackle these scalability challenges of state-of-the-art memorization-based MUL algorithms using a series of memorization-score proxies. We first analyze the profiles of various proxies and then evaluate the performance of state-of-the-art (memorization-based) MUL algorithms in terms of both accuracy and privacy preservation. Our empirical results show that these proxies can introduce accuracy on par with full memorization-based unlearning while dramatically improving scalability. We view this work as an important step toward scalable and efficient machine unlearning.

Title: RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Space

• Authors: Jingdi Chen, Hanhan Zhou, Yongsheng Mei, Carlee Joe-Wong, Gina Adam, Nathaniel D. Bastian, Tian Lan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16517
• Pdf URL: https://arxiv.org/pdf/2410.16517
• Copy Paste: [[2410.16517]] RGMDT: Return-Gap-Minimizing Decision Tree Extraction in Non-Euclidean Metric Space(https://arxiv.org/abs/2410.16517)
• Keywords: extraction, interpretability
• Abstract: Deep Reinforcement Learning (DRL) algorithms have achieved great success in solving many challenging tasks while their black-box nature hinders interpretability and real-world applicability, making it difficult for human experts to interpret and understand DRL policies. Existing works on interpretable reinforcement learning have shown promise in extracting decision tree (DT) based policies from DRL policies with most focus on the single-agent settings while prior attempts to introduce DT policies in multi-agent scenarios mainly focus on heuristic designs which do not provide any quantitative guarantees on the expected return. In this paper, we establish an upper bound on the return gap between the oracle expert policy and an optimal decision tree policy. This enables us to recast the DT extraction problem into a novel non-euclidean clustering problem over the local observation and action values space of each agent, with action values as cluster labels and the upper bound on the return gap as clustering loss. Both the algorithm and the upper bound are extended to multi-agent decentralized DT extractions by an iteratively-grow-DT procedure guided by an action-value function conditioned on the current DTs of other agents. Further, we propose the Return-Gap-Minimization Decision Tree (RGMDT) algorithm, which is a surprisingly simple design and is integrated with reinforcement learning through the utilization of a novel Regularized Information Maximization loss. Evaluations on tasks like D4RL show that RGMDT significantly outperforms heuristic DT-based baselines and can achieve nearly optimal returns under given DT complexity constraints (e.g., maximum number of DT nodes).

Title: Gradient-Free Supervised Learning using Spike-Timing-Dependent Plasticity for Image Recognition

• Authors: Wei Xie
• Subjects: cs.CV, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2410.16524
• Pdf URL: https://arxiv.org/pdf/2410.16524
• Copy Paste: [[2410.16524]] Gradient-Free Supervised Learning using Spike-Timing-Dependent Plasticity for Image Recognition(https://arxiv.org/abs/2410.16524)
• Keywords: robust
• Abstract: An approach to supervised learning in spiking neural networks is presented using a gradient-free method combined with spike-timing-dependent plasticity for image recognition. The proposed network architecture is scalable to multiple layers, enabling the development of more complex and deeper SNN models. The effectiveness of this method is demonstrated by its application to the MNIST dataset, showing good learning accuracy. The proposed method provides a robust and efficient alternative to the backpropagation-based method in supervised learning.

Title: Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysis

• Authors: Jonathan Brokman, Omer Hofman, Oren Rachmil, Inderjeet Singh, Rathina Sabapathy, Aishvariya Priya, Vikas Pahuja, Amit Giloni, Roman Vainshtein, Hisashi Kojima
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16527
• Pdf URL: https://arxiv.org/pdf/2410.16527
• Copy Paste: [[2410.16527]] Insights and Current Gaps in Open-Source LLM Vulnerability Scanners: A Comparative Analysis(https://arxiv.org/abs/2410.16527)
• Keywords: security, attack, large language model
• Abstract: This report presents a comparative analysis of open-source vulnerability scanners for conversational large language models (LLMs). As LLMs become integral to various applications, they also present potential attack surfaces, exposed to security risks such as information leakage and jailbreak attacks. Our study evaluates prominent scanners - Garak, Giskard, PyRIT, and CyberSecEval - that adapt red-teaming practices to expose these vulnerabilities. We detail the distinctive features and practical use of these scanners, outline unifying principles of their design and perform quantitative evaluations to compare them. These evaluations uncover significant reliability issues in detecting successful attacks, highlighting a fundamental gap for future development. Additionally, we contribute a preliminary labelled dataset, which serves as an initial step to bridge this gap. Based on the above, we provide strategic recommendations to assist organizations choose the most suitable scanner for their red-teaming needs, accounting for customizability, test suite comprehensiveness, and industry-specific use cases.

Title: No more hard prompts: SoftSRV prompting for synthetic data generation

• Authors: Giulia DeSalvo, Giulia DeSalvo, Jean-Fracois Kagy, Lazaros Karydas, Afshin Rostamizadeh, Sanjiv Kumar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16534
• Pdf URL: https://arxiv.org/pdf/2410.16534
• Copy Paste: [[2410.16534]] No more hard prompts: SoftSRV prompting for synthetic data generation(https://arxiv.org/abs/2410.16534)
• Keywords: large language model
• Abstract: We present a novel soft prompt based framework, SoftSRV, that leverages a frozen pre-trained large language model (LLM) to generate targeted synthetic text sequences. Given a sample from the target distribution, our proposed framework uses data-driven loss minimization to train a parameterized "contextual" soft prompt. This soft prompt is then used to steer the frozen LLM to generate synthetic sequences that are similar to the target distribution. We argue that SoftSRV provides a practical improvement over common hard-prompting approaches that rely on human-curated prompt-templates, which can be idiosyncratic, labor-intensive to craft, and may need to be specialized per domain. We empirically evaluate SoftSRV and hard-prompting baselines by generating synthetic data to fine-tune a small Gemma model on three different domains (coding, math, reasoning). To stress the generality of SoftSRV, we perform these evaluations without any particular specialization of the framework to each domain. We find that SoftSRV significantly improves upon hard-prompting baselines, generating data with superior fine-tuning performance and that better matches the target distribution according to the MAUVE similarity metric.

Title: A Theoretical Understanding of Chain-of-Thought: Coherent Reasoning and Error-Aware Demonstration

• Authors: Yingqian Cui, Pengfei He, Xianfeng Tang, Qi He, Chen Luo, Jiliang Tang, Yue Xing
• Subjects: cs.CL, cs.AI, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.16540
• Pdf URL: https://arxiv.org/pdf/2410.16540
• Copy Paste: [[2410.16540]] A Theoretical Understanding of Chain-of-Thought: Coherent Reasoning and Error-Aware Demonstration(https://arxiv.org/abs/2410.16540)
• Keywords: transformer, large language model
• Abstract: Few-shot Chain-of-Thought (CoT) prompting has demonstrated strong performance in improving the reasoning capabilities of large language models (LLMs). While theoretical investigations have been conducted to understand CoT, the underlying transformer used in these studies isolates the CoT reasoning process into separated in-context learning steps (Stepwise ICL). In this work, we theoretically show that, compared to Stepwise ICL, the transformer gains better error correction ability and more accurate predictions if the reasoning from earlier steps (Coherent CoT) is integrated. Given that this coherent reasoning changes the behavior of the transformer, we further investigate the sensitivity of the transformer with Coherent CoT when the demonstration examples are corrupted at the inference stage. Our theoretical results indicate that the transformer is more sensitive to errors in intermediate reasoning steps than the final outcome. Building upon this observation, we propose an improvement on CoT by incorporating both correct and incorrect reasoning paths in the demonstration. Our experiments validate the effectiveness of the proposed approach.

Title: PlaneSAM: Multimodal Plane Instance Segmentation Using the Segment Anything Model

• Authors: Zhongchen Deng, Zhechen Yang, Chi Chen, Cheng Zeng, Yan Meng, Bisheng Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16545
• Pdf URL: https://arxiv.org/pdf/2410.16545
• Copy Paste: [[2410.16545]] PlaneSAM: Multimodal Plane Instance Segmentation Using the Segment Anything Model(https://arxiv.org/abs/2410.16545)
• Keywords: segmentation
• Abstract: Plane instance segmentation from RGB-D data is a crucial research topic for many downstream tasks. However, most existing deep-learning-based methods utilize only information within the RGB bands, neglecting the important role of the depth band in plane instance segmentation. Based on EfficientSAM, a fast version of SAM, we propose a plane instance segmentation network called PlaneSAM, which can fully integrate the information of the RGB bands (spectral bands) and the D band (geometric band), thereby improving the effectiveness of plane instance segmentation in a multimodal manner. Specifically, we use a dual-complexity backbone, with primarily the simpler branch learning D-band features and primarily the more complex branch learning RGB-band features. Consequently, the backbone can effectively learn D-band feature representations even when D-band training data is limited in scale, retain the powerful RGB-band feature representations of EfficientSAM, and allow the original backbone branch to be fine-tuned for the current task. To enhance the adaptability of our PlaneSAM to the RGB-D domain, we pretrain our dual-complexity backbone using the segment anything task on large-scale RGB-D data through a self-supervised pretraining strategy based on imperfect pseudo-labels. To support the segmentation of large planes, we optimize the loss function combination ratio of EfficientSAM. In addition, Faster R-CNN is used as a plane detector, and its predicted bounding boxes are fed into our dual-complexity network as prompts, thereby enabling fully automatic plane instance segmentation. Experimental results show that the proposed PlaneSAM sets a new SOTA performance on the ScanNet dataset, and outperforms previous SOTA approaches in zero-shot transfer on the 2D-3D-S, Matterport3D, and ICL-NUIM RGB-D datasets, while only incurring a 10% increase in computational overhead compared to EfficientSAM.

Title: Can Transformers In-Context Learn Behavior of a Linear Dynamical System?

• Authors: Usman Akram, Haris Vikalo
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2410.16546
• Pdf URL: https://arxiv.org/pdf/2410.16546
• Copy Paste: [[2410.16546]] Can Transformers In-Context Learn Behavior of a Linear Dynamical System?(https://arxiv.org/abs/2410.16546)
• Keywords: robust, transformer
• Abstract: We investigate whether transformers can learn to track a random process when given observations of a related process and parameters of the dynamical system that relates them as context. More specifically, we consider a finite-dimensional state-space model described by the state transition matrix $F$, measurement matrices $h_1, \dots, h_N$, and the process and measurement noise covariance matrices $Q$ and $R$, respectively; these parameters, randomly sampled, are provided to the transformer along with the observations $y_1,\dots,y_N$ generated by the corresponding linear dynamical system. We argue that in such settings transformers learn to approximate the celebrated Kalman filter, and empirically verify this both for the task of estimating hidden states $\hat{x}_{N|1,2,3,...,N}$ as well as for one-step prediction of the $(N+1)^{st}$ observation, $\hat{y}_{N+1|1,2,3,...,N}$. A further study of the transformer's robustness reveals that its performance is retained even if the model's parameters are partially withheld. In particular, we demonstrate that the transformer remains accurate at the considered task even in the absence of state transition and noise covariance matrices, effectively emulating operations of the Dual-Kalman filter.

Title: Gradient Normalization with(out) Clipping Ensures Convergence of Nonconvex SGD under Heavy-Tailed Noise with Improved Results

• Authors: Tao Sun, Xinwang Liu, Kun Yuan
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.16561
• Pdf URL: https://arxiv.org/pdf/2410.16561
• Copy Paste: [[2410.16561]] Gradient Normalization with(out) Clipping Ensures Convergence of Nonconvex SGD under Heavy-Tailed Noise with Improved Results(https://arxiv.org/abs/2410.16561)
• Keywords: robust
• Abstract: This paper investigates Gradient Normalization Stochastic Gradient Descent without Clipping (NSGDC) and its variance reduction variant (NSGDC-VR) for nonconvex optimization under heavy-tailed noise. We present significant improvements in the theoretical results for both algorithms, including the removal of logarithmic factors from the convergence rates and the recovery of the convergence rate to match the deterministic case when the noise variance {\sigma} is zero. Additionally, we demonstrate that gradient normalization alone, assuming individual Lipschitz smoothness, is sufficient to ensure convergence of SGD under heavy-tailed noise, eliminating the need for gradient clipping. Furthermore, we introduce accelerated nonconvex algorithms that utilize second-order Lipschitz smoothness to achieve enhanced convergence rates in the presence of heavy-tailed noise. Our findings offer a deeper understanding of how gradient normalization and variance reduction techniques can be optimized for robust performance in challenging optimization scenarios.

Title: Enhancing PAC Learning of Half spaces Through Robust Optimization Techniques

• Authors: Shirmohammad Tavangari, Zahra Shakarami, Aref Yelghi, Asef Yelghi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16573
• Pdf URL: https://arxiv.org/pdf/2410.16573
• Copy Paste: [[2410.16573]] Enhancing PAC Learning of Half spaces Through Robust Optimization Techniques(https://arxiv.org/abs/2410.16573)
• Keywords: robust
• Abstract: This paper addresses the problem of PAC learning half spaces under constant malicious noise, where a fraction of the training data is adversarially corrupted. While traditional learning models assume clean data, real-world applications often face noisy environments that can significantly degrade the performance of machine learning algorithms. My study presents a novel, efficient algorithm that extends the existing theoretical frameworks to account for noise resilience in half space learning. By leveraging robust optimization techniques and advanced error-correction strategies, the proposed approach improves learning accuracy in adversarial conditions without requiring additional computational complexity. We provide a comprehensive analysis of the algorithm's performance, demonstrating its superior robustness to malicious noise when compared to existing state-of-the-art methods. Extensive theoretical evaluations are supported by empirical results that validate the algorithm's practical utility across a range of datasets and noise conditions. This work contributes to the field by offering a new, scalable solution to learning under noise, enhancing the reliability of machine learning systems in adversarial settings.

Title: Conflict-Aware Adversarial Training

• Authors: Zhiyu Xue, Haohan Wang, Yao Qin, Ramtin Pedarsani
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16579
• Pdf URL: https://arxiv.org/pdf/2410.16579
• Copy Paste: [[2410.16579]] Conflict-Aware Adversarial Training(https://arxiv.org/abs/2410.16579)
• Keywords: attack, robust
• Abstract: Adversarial training is the most effective method to obtain adversarial robustness for deep neural networks by directly involving adversarial samples in the training procedure. To obtain an accurate and robust model, the weighted-average method is applied to optimize standard loss and adversarial loss simultaneously. In this paper, we argue that the weighted-average method does not provide the best tradeoff for the standard performance and adversarial robustness. We argue that the failure of the weighted-average method is due to the conflict between the gradients derived from standard and adversarial loss, and further demonstrate such a conflict increases with attack budget theoretically and practically. To alleviate this problem, we propose a new trade-off paradigm for adversarial training with a conflict-aware factor for the convex combination of standard and adversarial loss, named \textbf{Conflict-Aware Adversarial Training~(CA-AT)}. Comprehensive experimental results show that CA-AT consistently offers a superior trade-off between standard performance and adversarial robustness under the settings of adversarial training from scratch and parameter-efficient finetuning.

Title: Dynamic Adaptive Rank Space Exploration for Efficient Sentiment Analysis with Large Language Models

• Authors: Hongcheng Ding, Fuzhen Hu, Xuanze Zhao, Zixiao Jiang, Shamsul Nahar Abdullah, Deshinta Arrova Dewi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16589
• Pdf URL: https://arxiv.org/pdf/2410.16589
• Copy Paste: [[2410.16589]] Dynamic Adaptive Rank Space Exploration for Efficient Sentiment Analysis with Large Language Models(https://arxiv.org/abs/2410.16589)
• Keywords: large language model
• Abstract: Sentiment analysis has become increasingly important for assessing public opinion and informing decision-making. Large language models (LLMs) have revolutionized this field by capturing nuanced language patterns. However, adapting LLMs to domain-specific sentiment analysis tasks remains challenging due to computational constraints and the need for optimal fine-tuning. To address these challenges, we propose a novel Dynamic Adaptive Rank Space Exploration (DARSE) framework for efficient and effective sentiment analysis using LLMs. DARSE consists of a coarse-grained greedy algorithm to identify the optimal rank range, a fine-grained exploration algorithm to refine rank selection, and a dynamic rank allocation method to determine the optimal rank combination for each LLM layer. Extensive experiments demonstrate that DARSE significantly improves sentiment analysis accuracy, achieving a 15.1% improvement in MSE and a 4.3% improvement in accuracy compared to previous work. Our framework strikes a balance between computational efficiency and model performance, making it a promising approach for sentiment analysis with LLMs.

Title: ViMGuard: A Novel Multi-Modal System for Video Misinformation Guarding

• Authors: Andrew Kan, Christopher Kan, Zaid Nabulsi
• Subjects: cs.LG, cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.16592
• Pdf URL: https://arxiv.org/pdf/2410.16592
• Copy Paste: [[2410.16592]] ViMGuard: A Novel Multi-Modal System for Video Misinformation Guarding(https://arxiv.org/abs/2410.16592)
• Keywords: large language model
• Abstract: The rise of social media and short-form video (SFV) has facilitated a breeding ground for misinformation. With the emergence of large language models, significant research has gone into curbing this misinformation problem with automatic false claim detection for text. Unfortunately, the automatic detection of misinformation in SFV is a more complex problem that remains largely unstudied. While text samples are monomodal (only containing words), SFVs comprise three different modalities: words, visuals, and non-linguistic audio. In this work, we introduce Video Masked Autoencoders for Misinformation Guarding (ViMGuard), the first deep-learning architecture capable of fact-checking an SFV through analysis of all three of its constituent modalities. ViMGuard leverages a dual-component system. First, Video and Audio Masked Autoencoders analyze the visual and non-linguistic audio elements of a video to discern its intention; specifically whether it intends to make an informative claim. If it is deemed that the SFV has informative intent, it is passed through our second component: a Retrieval Augmented Generation system that validates the factual accuracy of spoken words. In evaluation, ViMGuard outperformed three cutting-edge fact-checkers, thus setting a new standard for SFV fact-checking and marking a significant stride toward trustworthy news on social platforms. To promote further testing and iteration, VimGuard was deployed into a Chrome extension and all code was open-sourced on GitHub.

Title: Distill-SynthKG: Distilling Knowledge Graph Synthesis Workflow for Improved Coverage and Efficiency

• Authors: Prafulla Kumar Choubey, Xin Su, Man Luo, Xiangyu Peng, Caiming Xiong, Tiep Le, Shachar Rosenman, Vasudev Lal, Phil Mui, Ricky Ho, Phillip Howard, Chien-Sheng Wu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.16597
• Pdf URL: https://arxiv.org/pdf/2410.16597
• Copy Paste: [[2410.16597]] Distill-SynthKG: Distilling Knowledge Graph Synthesis Workflow for Improved Coverage and Efficiency(https://arxiv.org/abs/2410.16597)
• Keywords: extraction, large language model
• Abstract: Knowledge graphs (KGs) generated by large language models (LLMs) are becoming increasingly valuable for Retrieval-Augmented Generation (RAG) applications that require knowledge-intensive reasoning. However, existing KG extraction methods predominantly rely on prompt-based approaches, which are inefficient for processing large-scale corpora. These approaches often suffer from information loss, particularly with long documents, due to the lack of specialized design for KG construction. Additionally, there is a gap in evaluation datasets and methodologies for ontology-free KG construction. To overcome these limitations, we propose SynthKG, a multi-step, document-level ontology-free KG synthesis workflow based on LLMs. By fine-tuning a smaller LLM on the synthesized document-KG pairs, we streamline the multi-step process into a single-step KG generation approach called Distill-SynthKG, substantially reducing the number of LLM inference calls. Furthermore, we re-purpose existing question-answering datasets to establish KG evaluation datasets and introduce new evaluation metrics. Using KGs produced by Distill-SynthKG, we also design a novel graph-based retrieval framework for RAG. Experimental results demonstrate that Distill-SynthKG not only surpasses all baseline models in KG quality -- including models up to eight times larger -- but also consistently excels in retrieval and question-answering tasks. Our proposed graph retrieval framework also outperforms all KG-retrieval methods across multiple benchmark datasets. We release the SynthKG dataset and Distill-SynthKG model publicly to support further research and development.

Title: Foundation Models for Remote Sensing and Earth Observation: A Survey

• Authors: Aoran Xiao, Weihao Xuan, Junjue Wang, Jiaxing Huang, Dacheng Tao, Shijian Lu, Naoto Yokoya
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16602
• Pdf URL: https://arxiv.org/pdf/2410.16602
• Copy Paste: [[2410.16602]] Foundation Models for Remote Sensing and Earth Observation: A Survey(https://arxiv.org/abs/2410.16602)
• Keywords: large language model
• Abstract: Remote Sensing (RS) is a crucial technology for observing, monitoring, and interpreting our planet, with broad applications across geoscience, economics, humanitarian fields, etc. While artificial intelligence (AI), particularly deep learning, has achieved significant advances in RS, unique challenges persist in developing more intelligent RS systems, including the complexity of Earth's environments, diverse sensor modalities, distinctive feature patterns, varying spatial and spectral resolutions, and temporal dynamics. Meanwhile, recent breakthroughs in large Foundation Models (FMs) have expanded AI's potential across many domains due to their exceptional generalizability and zero-shot transfer capabilities. However, their success has largely been confined to natural data like images and video, with degraded performance and even failures for RS data of various non-optical modalities. This has inspired growing interest in developing Remote Sensing Foundation Models (RSFMs) to address the complex demands of Earth Observation (EO) tasks, spanning the surface, atmosphere, and oceans. This survey systematically reviews the emerging field of RSFMs. It begins with an outline of their motivation and background, followed by an introduction of their foundational concepts. It then categorizes and reviews existing RSFM studies including their datasets and technical contributions across Visual Foundation Models (VFMs), Visual-Language Models (VLMs), Large Language Models (LLMs), and beyond. In addition, we benchmark these models against publicly available datasets, discuss existing challenges, and propose future research directions in this rapidly evolving field.

Title: GALA: Graph Diffusion-based Alignment with Jigsaw for Source-free Domain Adaptation

• Authors: Junyu Luo, Yiyang Gu, Xiao Luo, Wei Ju, Zhiping Xiao, Yusheng Zhao, Jingyang Yuan, Ming Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16606
• Pdf URL: https://arxiv.org/pdf/2410.16606
• Copy Paste: [[2410.16606]] GALA: Graph Diffusion-based Alignment with Jigsaw for Source-free Domain Adaptation(https://arxiv.org/abs/2410.16606)
• Keywords: privacy, robust, diffusion, generative
• Abstract: Source-free domain adaptation is a crucial machine learning topic, as it contains numerous applications in the real world, particularly with respect to data privacy. Existing approaches predominantly focus on Euclidean data, such as images and videos, while the exploration of non-Euclidean graph data remains scarce. Recent graph neural network (GNN) approaches can suffer from serious performance decline due to domain shift and label scarcity in source-free adaptation scenarios. In this study, we propose a novel method named Graph Diffusion-based Alignment with Jigsaw (GALA), tailored for source-free graph domain adaptation. To achieve domain alignment, GALA employs a graph diffusion model to reconstruct source-style graphs from target data. Specifically, a score-based graph diffusion model is trained using source graphs to learn the generative source styles. Then, we introduce perturbations to target graphs via a stochastic differential equation instead of sampling from a prior, followed by the reverse process to reconstruct source-style graphs. We feed the source-style graphs into an off-the-shelf GNN and introduce class-specific thresholds with curriculum learning, which can generate accurate and unbiased pseudo-labels for target graphs. Moreover, we develop a simple yet effective graph-mixing strategy named graph jigsaw to combine confident graphs and unconfident graphs, which can enhance generalization capabilities and robustness via consistency learning. Extensive experiments on benchmark datasets validate the effectiveness of GALA.

Title: SoK: Dataset Copyright Auditing in Machine Learning Systems

• Authors: Linkang Du, Xuanru Zhou, Min Chen, Chusong Zhang, Zhou Su, Peng Cheng, Jiming Chen, Zhikun Zhang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16618
• Pdf URL: https://arxiv.org/pdf/2410.16618
• Copy Paste: [[2410.16618]] SoK: Dataset Copyright Auditing in Machine Learning Systems(https://arxiv.org/abs/2410.16618)
• Keywords: protect, robust, watermark
• Abstract: As the implementation of machine learning (ML) systems becomes more widespread, especially with the introduction of larger ML models, we perceive a spring demand for massive data. However, it inevitably causes infringement and misuse problems with the data, such as using unauthorized online artworks or face images to train ML models. To address this problem, many efforts have been made to audit the copyright of the model training dataset. However, existing solutions vary in auditing assumptions and capabilities, making it difficult to compare their strengths and weaknesses. In addition, robustness evaluations usually consider only part of the ML pipeline and hardly reflect the performance of algorithms in real-world ML applications. Thus, it is essential to take a practical deployment perspective on the current dataset copyright auditing tools, examining their effectiveness and limitations. Concretely, we categorize dataset copyright auditing research into two prominent strands: intrusive methods and non-intrusive methods, depending on whether they require modifications to the original dataset. Then, we break down the intrusive methods into different watermark injection options and examine the non-intrusive methods using various fingerprints. To summarize our results, we offer detailed reference tables, highlight key points, and pinpoint unresolved issues in the current literature. By combining the pipeline in ML systems and analyzing previous studies, we highlight several future directions to make auditing tools more suitable for real-world copyright protection requirements.

Title: EVC-MF: End-to-end Video Captioning Network with Multi-scale Features

• Authors: Tian-Zi Niu, Zhen-Duo Chen, Xin Luo, Xin-Shun Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16624
• Pdf URL: https://arxiv.org/pdf/2410.16624
• Copy Paste: [[2410.16624]] EVC-MF: End-to-end Video Captioning Network with Multi-scale Features(https://arxiv.org/abs/2410.16624)
• Keywords: transformer
• Abstract: Conventional approaches for video captioning leverage a variety of offline-extracted features to generate captions. Despite the availability of various offline-feature-extractors that offer diverse information from different perspectives, they have several limitations due to fixed parameters. Concretely, these extractors are solely pre-trained on image/video comprehension tasks, making them less adaptable to video caption datasets. Additionally, most of these extractors only capture features prior to the classifier of the pre-training task, ignoring a significant amount of valuable shallow information. Furthermore, employing multiple offline-features may introduce redundant information. To address these issues, we propose an end-to-end encoder-decoder-based network (EVC-MF) for video captioning, which efficiently utilizes multi-scale visual and textual features to generate video descriptions. Specifically, EVC-MF consists of three modules. Firstly, instead of relying on multiple feature extractors, we directly feed video frames into a transformer-based network to obtain multi-scale visual features and update feature extractor parameters. Secondly, we fuse the multi-scale features and input them into a masked encoder to reduce redundancy and encourage learning useful features. Finally, we utilize an enhanced transformer-based decoder, which can efficiently leverage shallow textual information, to generate video descriptions. To evaluate our proposed model, we conduct extensive experiments on benchmark datasets. The results demonstrate that EVC-MF yields competitive performance compared with the state-of-theart methods.

Title: Benchmarking Multi-Scene Fire and Smoke Detection

• Authors: Xiaoyi Han, Nan Pu, Zunlei Feng, Yijun Bei, Qifei Zhang, Lechao Cheng, Liang Xue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16631
• Pdf URL: https://arxiv.org/pdf/2410.16631
• Copy Paste: [[2410.16631]] Benchmarking Multi-Scene Fire and Smoke Detection(https://arxiv.org/abs/2410.16631)
• Keywords: robust
• Abstract: The current irregularities in existing public Fire and Smoke Detection (FSD) datasets have become a bottleneck in the advancement of FSD technology. Upon in-depth analysis, we identify the core issue as the lack of standardized dataset construction, uniform evaluation systems, and clear performance benchmarks. To address this issue and drive innovation in FSD technology, we systematically gather diverse resources from public sources to create a more comprehensive and refined FSD benchmark. Additionally, recognizing the inadequate coverage of existing dataset scenes, we strategically expand scenes, relabel, and standardize existing public FSD datasets to ensure accuracy and consistency. We aim to establish a standardized, realistic, unified, and efficient FSD research platform that mirrors real-life scenes closely. Through our efforts, we aim to provide robust support for the breakthrough and development of FSD technology. The project is available at \href{this https URL}{this https URL}.

Title: Graph-Structured Trajectory Extraction from Travelogues

• Authors: Aitaro Yamamoto, Hiroyuki Otomo, Hiroki Ouchi, Shohei Higashiyama, Hiroki Teranishi, Hiroyuki Shindo, Taro Watanabe
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16633
• Pdf URL: https://arxiv.org/pdf/2410.16633
• Copy Paste: [[2410.16633]] Graph-Structured Trajectory Extraction from Travelogues(https://arxiv.org/abs/2410.16633)
• Keywords: extraction
• Abstract: Previous studies on sequence-based extraction of human movement trajectories have an issue of inadequate trajectory representation. Specifically, a pair of locations may not be lined up in a sequence especially when one location includes the other geographically. In this study, we propose a graph representation that retains information on the geographic hierarchy as well as the temporal order of visited locations, and have constructed a benchmark dataset for graph-structured trajectory extraction. The experiments with our baselines have demonstrated that it is possible to accurately predict visited locations and the order among them, but it remains a challenge to predict the hierarchical relations.

Title: A Statistical Analysis of LLMs' Self-Evaluation Using Proverbs

• Authors: Ryosuke Sonoda, Ramya Srinivasan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16640
• Pdf URL: https://arxiv.org/pdf/2410.16640
• Copy Paste: [[2410.16640]] A Statistical Analysis of LLMs' Self-Evaluation Using Proverbs(https://arxiv.org/abs/2410.16640)
• Keywords: large language model
• Abstract: Large language models (LLMs) such as ChatGPT, GPT-4, Claude-3, and Llama are being integrated across a variety of industries. Despite this rapid proliferation, experts are calling for caution in the interpretation and adoption of LLMs, owing to numerous associated ethical concerns. Research has also uncovered shortcomings in LLMs' reasoning and logical abilities, raising questions on the potential of LLMs as evaluation tools. In this paper, we investigate LLMs' self-evaluation capabilities on a novel proverb reasoning task. We introduce a novel proverb database consisting of 300 proverb pairs that are similar in intent but different in wordings, across topics spanning gender, wisdom, and society. We propose tests to evaluate textual consistencies as well as numerical consistencies across similar proverbs, and demonstrate the effectiveness of our method and dataset in identifying failures in LLMs' self-evaluation which in turn can highlight issues related to gender stereotypes and lack of cultural understanding in LLMs.

Title: Fire and Smoke Detection with Burning Intensity Representation

• Authors: Xiaoyi Han, Yanfei Wu, Nan Pu, Zunlei Feng, Qifei Zhang, Yijun Bei, Lechao Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16642
• Pdf URL: https://arxiv.org/pdf/2410.16642
• Copy Paste: [[2410.16642]] Fire and Smoke Detection with Burning Intensity Representation(https://arxiv.org/abs/2410.16642)
• Keywords: robust, extraction
• Abstract: An effective Fire and Smoke Detection (FSD) and analysis system is of paramount importance due to the destructive potential of fire disasters. However, many existing FSD methods directly employ generic object detection techniques without considering the transparency of fire and smoke, which leads to imprecise localization and reduces detection performance. To address this issue, a new Attentive Fire and Smoke Detection Model (a-FSDM) is proposed. This model not only retains the robust feature extraction and fusion capabilities of conventional detection algorithms but also redesigns the detection head specifically for transparent targets in FSD, termed the Attentive Transparency Detection Head (ATDH). In addition, Burning Intensity (BI) is introduced as a pivotal feature for fire-related downstream risk assessments in traditional FSD methodologies. Extensive experiments on multiple FSD datasets showcase the effectiveness and versatility of the proposed FSD model. The project is available at \href{this https URL}{this https URL}.

Title: Chatting with Bots: AI, Speech Acts, and the Edge of Assertion

• Authors: Iwan Williams, Tim Bayne
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16645
• Pdf URL: https://arxiv.org/pdf/2410.16645
• Copy Paste: [[2410.16645]] Chatting with Bots: AI, Speech Acts, and the Edge of Assertion(https://arxiv.org/abs/2410.16645)
• Keywords: large language model
• Abstract: This paper addresses the question of whether large language model-powered chatbots are capable of assertion. According to what we call the Thesis of Chatbot Assertion (TCA), chatbots are the kinds of things that can assert, and at least some of the output produced by current-generation chatbots qualifies as assertion. We provide some motivation for TCA, arguing that it ought to be taken seriously and not simply dismissed. We also review recent objections to TCA, arguing that these objections are weighty. We thus confront the following dilemma: how can we do justice to both the considerations for and against TCA? We consider two influential responses to this dilemma - the first appeals to the notion of proxy-assertion; the second appeals to fictionalism - and argue that neither is satisfactory. Instead, reflecting on the ontogenesis of assertion, we argue that we need to make space for a category of proto-assertion. We then apply the category of proto-assertion to chatbots, arguing that treating chatbots as proto-assertors provides a satisfactory resolution to the dilemma of chatbot assertion.

Title: TopoDiffusionNet: A Topology-aware Diffusion Model

• Authors: Saumya Gupta, Dimitris Samaras, Chao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16646
• Pdf URL: https://arxiv.org/pdf/2410.16646
• Copy Paste: [[2410.16646]] TopoDiffusionNet: A Topology-aware Diffusion Model(https://arxiv.org/abs/2410.16646)
• Keywords: diffusion
• Abstract: Diffusion models excel at creating visually impressive images but often struggle to generate images with a specified topology. The Betti number, which represents the number of structures in an image, is a fundamental measure in topology. Yet, diffusion models fail to satisfy even this basic constraint. This limitation restricts their utility in applications requiring exact control, like robotics and environmental modeling. To address this, we propose TopoDiffusionNet (TDN), a novel approach that enforces diffusion models to maintain the desired topology. We leverage tools from topological data analysis, particularly persistent homology, to extract the topological structures within an image. We then design a topology-based objective function to guide the denoising process, preserving intended structures while suppressing noisy ones. Our experiments across four datasets demonstrate significant improvements in topological accuracy. TDN is the first to integrate topology with diffusion models, opening new avenues of research in this area.

Title: BETA: Automated Black-box Exploration for Timing Attacks in Processors

• Authors: Congcong Chen, Jinhua Cui, Jiliang Zhang
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2410.16648
• Pdf URL: https://arxiv.org/pdf/2410.16648
• Copy Paste: [[2410.16648]] BETA: Automated Black-box Exploration for Timing Attacks in Processors(https://arxiv.org/abs/2410.16648)
• Keywords: security, attack
• Abstract: Modern processor advancements have introduced security risks, particularly in the form of microarchitectural timing attacks. High-profile attacks such as Meltdown and Spectre have revealed critical flaws, compromising the entire system's security. Recent black-box automated methods have demonstrated their advantages in identifying these vulnerabilities on various commercial processors. However, they often focus on specific attack types or incorporate numerous ineffective test cases, which severely limits the detection scope and efficiency. In this paper, we present BETA, a novel black-box framework that harnesses fuzzing to efficiently uncover multifaceted timing vulnerabilities in processors. Our framework employs a two-pronged approach, enhancing both mutation space and exploration efficiency: 1) we introduce an innovative fuzzer that precisely constrains mutation direction for diverse instruction combinations, including opcode, data, address, and execution level; 2) we develop a coverage feedback mechanism based on our instruction classification to discard potentially trivial or redundant test cases. This mechanism significantly expands coverage across a broader spectrum of instruction types. We evaluate the performance and effectiveness of BETA on four processors from Intel and AMD, each featuring distinct microarchitectures. BETA has successfully detected all x86 processor vulnerabilities previously identified by recent black-box methods, as well as 8 previously undiscovered timing vulnerabilities. BETA outperforms the existing state-of-the-art black-box methods, achieving at least 3x faster detection speed.

Title: Dual-Model Defense: Safeguarding Diffusion Models from Membership Inference Attacks through Disjoint Data Splitting

• Authors: Bao Q. Tran, Viet Nguyen, Anh Tran, Toan Tran
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16657
• Pdf URL: https://arxiv.org/pdf/2410.16657
• Copy Paste: [[2410.16657]] Dual-Model Defense: Safeguarding Diffusion Models from Membership Inference Attacks through Disjoint Data Splitting(https://arxiv.org/abs/2410.16657)
• Keywords: privacy, protect, defense, attack, membership infer, diffusion
• Abstract: Diffusion models have demonstrated remarkable capabilities in image synthesis, but their recently proven vulnerability to Membership Inference Attacks (MIAs) poses a critical privacy concern. This paper introduces two novel and efficient approaches (DualMD and DistillMD) to protect diffusion models against MIAs while maintaining high utility. Both methods are based on training two separate diffusion models on disjoint subsets of the original dataset. DualMD then employs a private inference pipeline that utilizes both models. This strategy significantly reduces the risk of black-box MIAs by limiting the information any single model contains about individual training samples. The dual models can also generate "soft targets" to train a private student model in DistillMD, enhancing privacy guarantees against all types of MIAs. Extensive evaluations of DualMD and DistillMD against state-of-the-art MIAs across various datasets in white-box and black-box settings demonstrate their effectiveness in substantially reducing MIA success rates while preserving competitive image generation performance. Notably, our experiments reveal that DistillMD not only defends against MIAs but also mitigates model memorization, indicating that both vulnerabilities stem from overfitting and can be addressed simultaneously with our unified approach.

Title: Adsorb-Agent: Autonomous Identification of Stable Adsorption Configurations via Large Language Model Agent

• Authors: Janghoon Ock, Tirtha Vinchurkar, Yayati Jadhav, Amir Barati Farimani
• Subjects: cs.CL, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2410.16658
• Pdf URL: https://arxiv.org/pdf/2410.16658
• Copy Paste: [[2410.16658]] Adsorb-Agent: Autonomous Identification of Stable Adsorption Configurations via Large Language Model Agent(https://arxiv.org/abs/2410.16658)
• Keywords: large language model
• Abstract: Adsorption energy is a key reactivity descriptor in catalysis, enabling the efficient screening of potential catalysts. However, determining adsorption energy involves comparing the energies of multiple adsorbate-catalyst configurations, which is computationally demanding due to a large number of possible configurations. Current algorithmic approaches typically enumerate adsorption sites and configurations without leveraging theoretical insights to guide the initial setup. In this work, we present Adsorb-Agent, a Large Language Model (LLM) agent designed to efficiently derive system-specific stable adsorption configurations with minimal human intervention. Adsorb-Agent leverages built-in knowledge and emergent reasoning capabilities, significantly reducing the number of initial configurations required while improving accuracy in predicting the minimum adsorption energy. We demonstrate its performance using two example systems, NNH-CuPd3 (111) and NNH-Mo3Pd (111), for the Nitrogen Reduction Reaction (NRR), a sustainable alternative to the Haber-Bosch process. Adsorb-Agent outperforms conventional "heuristic" and "random" algorithms by identifying lower-energy configurations with fewer initial setups, reducing computational costs while enhancing accuracy. This highlights its potential to accelerate catalyst discovery.

Title: RKadiyala at SemEval-2024 Task 8: Black-Box Word-Level Text Boundary Detection in Partially Machine Generated Texts

• Authors: Ram Mohan Rao Kadiyala
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16659
• Pdf URL: https://arxiv.org/pdf/2410.16659
• Copy Paste: [[2410.16659]] RKadiyala at SemEval-2024 Task 8: Black-Box Word-Level Text Boundary Detection in Partially Machine Generated Texts(https://arxiv.org/abs/2410.16659)
• Keywords: generative
• Abstract: With increasing usage of generative models for text generation and widespread use of machine generated texts in various domains, being able to distinguish between human written and machine generated texts is a significant challenge. While existing models and proprietary systems focus on identifying whether given text is entirely human written or entirely machine generated, only a few systems provide insights at sentence or paragraph level at likelihood of being machine generated at a non reliable accuracy level, working well only for a set of domains and generators. This paper introduces few reliable approaches for the novel task of identifying which part of a given text is machine generated at a word level while comparing results from different approaches and methods. We present a comparison with proprietary systems , performance of our model on unseen domains' and generators' texts. The findings reveal significant improvements in detection accuracy along with comparison on other aspects of detection capabilities. Finally we discuss potential avenues for improvement and implications of our work. The proposed model is also well suited for detecting which parts of a text are machine generated in outputs of Instruct variants of many LLMs.

Title: FastAttention: Extend FlashAttention2 to NPUs and Low-resource GPUs

• Authors: Haoran Lin, Xianzhi Yu, Kang Zhao, Lu Hou, Zongyuan Zhan, Stanislav Kamenev, Han Bao, Ting Hu, Mingkai Wang, Qixin Chang, Siyue Sui, Weihao Sun, Jiaxin Hu, Jun Yao, Zekun Yin, Cheng Qian, Ying Zhang, Yinfei Pan, Yu Yang, Weiguo Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16663
• Pdf URL: https://arxiv.org/pdf/2410.16663
• Copy Paste: [[2410.16663]] FastAttention: Extend FlashAttention2 to NPUs and Low-resource GPUs(https://arxiv.org/abs/2410.16663)
• Keywords: transformer, large language model
• Abstract: FlashAttention series has been widely applied in the inference of large language models (LLMs). However, FlashAttention series only supports the high-level GPU architectures, e.g., Ampere and Hopper. At present, FlashAttention series is not easily transferrable to NPUs and low-resource GPUs. Moreover, FlashAttention series is inefficient for multi- NPUs or GPUs inference scenarios. In this work, we propose FastAttention which pioneers the adaptation of FlashAttention series for NPUs and low-resource GPUs to boost LLM inference efficiency. Specifically, we take Ascend NPUs and Volta-based GPUs as representatives for designing our FastAttention. We migrate FlashAttention series to Ascend NPUs by proposing a novel two-level tiling strategy for runtime speedup, tiling-mask strategy for memory saving and the tiling-AllReduce strategy for reducing communication overhead, respectively. Besides, we adapt FlashAttention for Volta-based GPUs by redesigning the operands layout in shared memory and introducing a simple yet effective CPU-GPU cooperative strategy for efficient memory utilization. On Ascend NPUs, our FastAttention can achieve a 10.7$\times$ speedup compared to the standard attention implementation. Llama-7B within FastAttention reaches up to 5.16$\times$ higher throughput than within the standard attention. On Volta architecture GPUs, FastAttention yields 1.43$\times$ speedup compared to its equivalents in \texttt{xformers}. Pangu-38B within FastAttention brings 1.46$\times$ end-to-end speedup using FasterTransformer. Coupled with the propose CPU-GPU cooperative strategy, FastAttention supports a maximal input length of 256K on 8 V100 GPUs. All the codes will be made available soon.

Title: SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation

• Authors: Jing-Jing Li, Valentina Pyatkin, Max Kleiman-Weiner, Liwei Jiang, Nouha Dziri, Anne G. E. Collins, Jana Schaich Borg, Maarten Sap, Yejin Choi, Sydney Levine
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.16665
• Pdf URL: https://arxiv.org/pdf/2410.16665
• Copy Paste: [[2410.16665]] SafetyAnalyst: Interpretable, transparent, and steerable LLM safety moderation(https://arxiv.org/abs/2410.16665)
• Keywords: interpretability
• Abstract: The ideal LLM content moderation system would be both structurally interpretable (so its decisions can be explained to users) and steerable (to reflect a community's values or align to safety standards). However, current systems fall short on both of these dimensions. To address this gap, we present SafetyAnalyst, a novel LLM safety moderation framework. Given a prompt, SafetyAnalyst creates a structured "harm-benefit tree," which identifies 1) the actions that could be taken if a compliant response were provided, 2) the harmful and beneficial effects of those actions (along with their likelihood, severity, and immediacy), and 3) the stakeholders that would be impacted by those effects. It then aggregates this structured representation into a harmfulness score based on a parameterized set of safety preferences, which can be transparently aligned to particular values. Using extensive harm-benefit features generated by SOTA LLMs on 19k prompts, we fine-tuned an open-weight LM to specialize in generating harm-benefit trees through symbolic knowledge distillation. On a comprehensive set of prompt safety benchmarks, we show that our system (average F1=0.75) outperforms existing LLM safety moderation systems (average F1$<$0.72) on prompt harmfulness classification, while offering the additional advantages of interpretability and steerability.

Title: CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharing

• Authors: Chen Yang, Chenyang Zhao, Quanquan Gu, Dongruo Zhou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16670
• Pdf URL: https://arxiv.org/pdf/2410.16670
• Copy Paste: [[2410.16670]] CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharing(https://arxiv.org/abs/2410.16670)
• Keywords: large language model
• Abstract: Sequential reasoning in agent systems has been significantly advanced by large language models (LLMs), yet existing approaches face limitations. Reflection-driven reasoning relies solely on knowledge in pretrained models, limiting performance in novel scenarios, while experience-assisted reasoning often depends on external experiences and lacks clear principles for selecting representative experiences. We address these limitations by proposing CoPS (Cross-Task Experience Sharing), a generalizable algorithm that enhances sequential reasoning by cross-task experience sharing and selection. In detail, CoPS leverages agents' experiences on previous tasks, selecting distribution-matched experiences via a provable pessimism-based strategy to maximize utility while minimizing risks from distribution shifts. Extensive experimental results on benchmarks like Alfworld, Webshop, and HotPotQA demonstrate that CoPS consistently outperforms state-of-the-art baselines, with superior sample efficiency suitable for resource-constrained scenarios. Theoretically, we show that the performance of our algorithm depends on both the quality of the pretrained LLM and the matching between the agent's task-dependent trial distribution and that generated by the LLM. Our work bridges the gap between existing sequential reasoning paradigms and validates the effectiveness of leveraging cross-task experiences, shedding light on the potential to improve agents' generalization and adaptability across diverse tasks. Our codes are available at $\href{this https URL}{\text{this https URL}}$.

Title: Efficient Antibody Structure Refinement Using Energy-Guided SE(3) Flow Matching

• Authors: Jiying Zhang, Zijing Liu, Shengyuan Bai, He Cao, Yu Li, Lei Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16673
• Pdf URL: https://arxiv.org/pdf/2410.16673
• Copy Paste: [[2410.16673]] Efficient Antibody Structure Refinement Using Energy-Guided SE(3) Flow Matching(https://arxiv.org/abs/2410.16673)
• Keywords: generative
• Abstract: Antibodies are proteins produced by the immune system that recognize and bind to specific antigens, and their 3D structures are crucial for understanding their binding mechanism and designing therapeutic interventions. The specificity of antibody-antigen binding predominantly depends on the complementarity-determining regions (CDR) within antibodies. Despite recent advancements in antibody structure prediction, the quality of predicted CDRs remains suboptimal. In this paper, we develop a novel antibody structure refinement method termed FlowAB based on energy-guided flow matching. FlowAB adopts the powerful deep generative method SE(3) flow matching and simultaneously incorporates important physical prior knowledge into the flow model to guide the generation process. The extensive experiments demonstrate that FlowAB can significantly improve the antibody CDR structures. It achieves new state-of-the-art performance on the antibody structure prediction task when used in conjunction with an appropriate prior model while incurring only marginal computational overhead. This advantage makes FlowAB a practical tool in antibody engineering.

Title: Methods of improving LLM training stability

• Authors: Oleg Rybakov, Mike Chrzanowski, Peter Dykas, Jinze Xue, Ben Lanir
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16682
• Pdf URL: https://arxiv.org/pdf/2410.16682
• Copy Paste: [[2410.16682]] Methods of improving LLM training stability(https://arxiv.org/abs/2410.16682)
• Keywords: transformer, large language model
• Abstract: Training stability of large language models(LLMs) is an important research topic. Reproducing training instabilities can be costly, so we use a small language model with 830M parameters and experiment with higher learning rates to force models to diverge. One of the sources of training instability is the growth of logits in attention layers. We extend the focus of the previous work and look not only at the magnitude of the logits but at all outputs of linear layers in the Transformer block. We observe that with a high learning rate the L2 norm of all linear layer outputs can grow with each training step and the model diverges. Specifically we observe that QKV, Proj and FC2 layers have the largest growth of the output magnitude. This prompts us to explore several options: 1) apply layer normalization not only after QK layers but also after Proj and FC2 layers too; 2) apply layer normalization after the QKV layer (and remove pre normalization). 3) apply QK layer normalization together with softmax capping. We show that with the last two methods we can increase learning rate by 1.5x (without model divergence) in comparison to an approach based on QK layer normalization only. Also we observe significant perplexity improvements for all three methods in comparison to the baseline model.

Title: Governing equation discovery of a complex system from snapshots

• Authors: Qunxi Zhu, Bolin Zhao, Jingdong Zhang, Peiyang Li, Wei Lin
• Subjects: cs.LG, math.DS, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2410.16694
• Pdf URL: https://arxiv.org/pdf/2410.16694
• Copy Paste: [[2410.16694]] Governing equation discovery of a complex system from snapshots(https://arxiv.org/abs/2410.16694)
• Keywords: robust, diffusion
• Abstract: Complex systems in physics, chemistry, and biology that evolve over time with inherent randomness are typically described by stochastic differential equations (SDEs). A fundamental challenge in science and engineering is to determine the governing equations of a complex system from snapshot data. Traditional equation discovery methods often rely on stringent assumptions, such as the availability of the trajectory information or time-series data, and the presumption that the underlying system is deterministic. In this work, we introduce a data-driven, simulation-free framework, called Sparse Identification of Differential Equations from Snapshots (SpIDES), that discovers the governing equations of a complex system from snapshots by utilizing the advanced machine learning techniques to perform three essential steps: probability flow reconstruction, probability density estimation, and Bayesian sparse identification. We validate the effectiveness and robustness of SpIDES by successfully identifying the governing equation of an over-damped Langevin system confined within two potential wells. By extracting interpretable drift and diffusion terms from the SDEs, our framework provides deeper insights into system dynamics, enhances predictive accuracy, and facilitates more effective strategies for managing and simulating stochastic systems.

Title: Hyperboloid GPLVM for Discovering Continuous Hierarchies via Nonparametric Estimation

• Authors: Koshi Watanabe, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16698
• Pdf URL: https://arxiv.org/pdf/2410.16698
• Copy Paste: [[2410.16698]] Hyperboloid GPLVM for Discovering Continuous Hierarchies via Nonparametric Estimation(https://arxiv.org/abs/2410.16698)
• Keywords: generative
• Abstract: Dimensionality reduction (DR) offers a useful representation of complex high-dimensional data. Recent DR methods focus on hyperbolic geometry to derive a faithful low-dimensional representation of hierarchical data. However, existing methods are based on neighbor embedding, frequently ruining the continual relation of the hierarchies. This paper presents hyperboloid Gaussian process (GP) latent variable models (hGP-LVMs) to embed high-dimensional hierarchical data with implicit continuity via nonparametric estimation. We adopt generative modeling using the GP, which brings effective hierarchical embedding and executes ill-posed hyperparameter tuning. This paper presents three variants that employ original point, sparse point, and Bayesian estimations. We establish their learning algorithms by incorporating the Riemannian optimization and active approximation scheme of GP-LVM. For Bayesian inference, we further introduce the reparameterization trick to realize Bayesian latent variable learning. In the last part of this paper, we apply hGP-LVMs to several datasets and show their ability to represent high-dimensional hierarchies in low-dimensional spaces.

Title: Graph Transformers Dream of Electric Flow

• Authors: Xiang Cheng, Lawrence Carin, Suvrit Sra
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16699
• Pdf URL: https://arxiv.org/pdf/2410.16699
• Copy Paste: [[2410.16699]] Graph Transformers Dream of Electric Flow(https://arxiv.org/abs/2410.16699)
• Keywords: transformer
• Abstract: We show theoretically and empirically that the linear Transformer, when applied to graph data, can implement algorithms that solve canonical problems such as electric flow and eigenvector decomposition. The input to the Transformer is simply the graph incidence matrix; no other explicit positional encoding information is provided. We present explicit weight configurations for implementing each such graph algorithm, and we bound the errors of the constructed Transformers by the errors of the underlying algorithms. Our theoretical findings are corroborated by experiments on synthetic data. Additionally, on a real-world molecular regression task, we observe that the linear Transformer is capable of learning a more effective positional encoding than the default one based on Laplacian eigenvectors. Our work is an initial step towards elucidating the inner-workings of the Transformer for graph data.

Title: ClimaQA: An Automated Evaluation Framework for Climate Foundation Models

• Authors: Veeramakali Vignesh Manivannan, Yasaman Jafari, Srikar Eranky, Spencer Ho, Rose Yu, Duncan Watson-Parris, Yian Ma, Leon Bergen, Taylor Berg-Kirkpatrick
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16701
• Pdf URL: https://arxiv.org/pdf/2410.16701
• Copy Paste: [[2410.16701]] ClimaQA: An Automated Evaluation Framework for Climate Foundation Models(https://arxiv.org/abs/2410.16701)
• Keywords: large language model
• Abstract: The use of foundation models in climate science has recently gained significant attention. However, a critical issue remains: the lack of a comprehensive evaluation framework capable of assessing the quality and scientific validity of model outputs. To address this issue, we develop ClimaGen (Climate QA Generator), an automated algorithmic framework that generates question-answer pairs from graduate textbooks with climate scientists in the loop. As a result, we present ClimaQA-Gold, an expert-annotated benchmark dataset alongside ClimaQA-Silver, a large-scale, comprehensive synthetic QA dataset for climate science. Finally, we develop evaluation strategies and compare different Large Language Models (LLMs) on our benchmarks. Our results offer novel insights into various approaches used to enhance climate foundation models.

Title: PLDR-LLM: Large Language Model from Power Law Decoder Representations

• Authors: Burc Gokden
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16703
• Pdf URL: https://arxiv.org/pdf/2410.16703
• Copy Paste: [[2410.16703]] PLDR-LLM: Large Language Model from Power Law Decoder Representations(https://arxiv.org/abs/2410.16703)
• Keywords: large language model
• Abstract: We present the Large Language Model from Power Law Decoder Representations (PLDR-LLM), a language model that leverages non-linear and linear transformations through Power Law Graph Attention mechanism to generate well-defined deductive and inductive outputs. We pretrain the PLDR-LLMs of varying layer sizes with a small batch size of 32 and $\sim$8B tokens from the RefinedWeb dataset, and show that they achieve competitive performance in zero-shot and few-shot settings compared to scaled dot-product LLMs of similar model size reported in the literature. We show that deductive outputs of PLDR-LLMs can be used to compare model characteristics or improve the performance by introducing the Directed Acyclic Graph (DAG) loss as a metric and regularizer. Our results indicate that the initial maximum learning rate and warm-up steps have a lasting impact on deductive outputs throughout the pretraining. We provide a detailed description of PLDR-LLM architecture, its implementation and the pretraining procedure.

Title: DI-MaskDINO: A Joint Object Detection and Instance Segmentation Model

• Authors: Zhixiong Nan, Xianghong Li, Tao Xiang, Jifeng Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16707
• Pdf URL: https://arxiv.org/pdf/2410.16707
• Copy Paste: [[2410.16707]] DI-MaskDINO: A Joint Object Detection and Instance Segmentation Model(https://arxiv.org/abs/2410.16707)
• Keywords: transformer, segmentation
• Abstract: This paper is motivated by an interesting phenomenon: the performance of object detection lags behind that of instance segmentation (i.e., performance imbalance) when investigating the intermediate results from the beginning transformer decoder layer of MaskDINO (i.e., the SOTA model for joint detection and segmentation). This phenomenon inspires us to think about a question: will the performance imbalance at the beginning layer of transformer decoder constrain the upper bound of the final performance? With this question in mind, we further conduct qualitative and quantitative pre-experiments, which validate the negative impact of detection-segmentation imbalance issue on the model performance. To address this issue, this paper proposes DI-MaskDINO model, the core idea of which is to improve the final performance by alleviating the detection-segmentation imbalance. DI-MaskDINO is implemented by configuring our proposed De-Imbalance (DI) module and Balance-Aware Tokens Optimization (BATO) module to MaskDINO. DI is responsible for generating balance-aware query, and BATO uses the balance-aware query to guide the optimization of the initial feature tokens. The balance-aware query and optimized feature tokens are respectively taken as the Query and Key&Value of transformer decoder to perform joint object detection and instance segmentation. DI-MaskDINO outperforms existing joint object detection and instance segmentation models on COCO and BDD100K benchmarks, achieving +1.2 $AP^{box}$ and +0.9 $AP^{mask}$ improvements compared to SOTA joint detection and segmentation model MaskDINO. In addition, DI-MaskDINO also obtains +1.0 $AP^{box}$ improvement compared to SOTA object detection model DINO and +3.0 $AP^{mask}$ improvement compared to SOTA segmentation model Mask2Former.

Title: Atomic Fact Decomposition Helps Attributed Question Answering

• Authors: Zhichao Yan, Jiapu Wang, Jiaoyan Chen, Xiaoli Li, Ru Li, Jeff Z.Pan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16708
• Pdf URL: https://arxiv.org/pdf/2410.16708
• Copy Paste: [[2410.16708]] Atomic Fact Decomposition Helps Attributed Question Answering(https://arxiv.org/abs/2410.16708)
• Keywords: large language model
• Abstract: Attributed Question Answering (AQA) aims to provide both a trustworthy answer and a reliable attribution report for a given question. Retrieval is a widely adopted approach, including two general paradigms: Retrieval-Then-Read (RTR) and post-hoc retrieval. Recently, Large Language Models (LLMs) have shown remarkable proficiency, prompting growing interest in AQA among researchers. However, RTR-based AQA often suffers from irrelevant knowledge and rapidly changing information, even when LLMs are adopted, while post-hoc retrieval-based AQA struggles with comprehending long-form answers with complex logic, and precisely identifying the content needing revision and preserving the original intent. To tackle these problems, this paper proposes an Atomic fact decomposition-based Retrieval and Editing (ARE) framework, which decomposes the generated long-form answers into molecular clauses and atomic facts by the instruction-tuned LLMs. Notably, the instruction-tuned LLMs are fine-tuned using a well-constructed dataset, generated from large scale Knowledge Graphs (KGs). This process involves extracting one-hop neighbors from a given set of entities and transforming the result into coherent long-form text. Subsequently, ARE leverages a search engine to retrieve evidences related to atomic facts, inputting these evidences into an LLM-based verifier to determine whether the facts require expansion for re-retrieval or editing. Furthermore, the edited facts are backtracked into the original answer, with evidence aggregated based on the relationship between molecular clauses and atomic facts. Extensive evaluations demonstrate the superior performance of our proposed method over the state-of-the-arts on several datasets, with an additionally proposed new metric $Attr_{p}$ for evaluating the precision of evidence attribution.

Title: Development of CNN Architectures using Transfer Learning Methods for Medical Image Classification

• Authors: Ganga Prasad Basyal, David Zeng, Bhaskar Pm Rimal
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16711
• Pdf URL: https://arxiv.org/pdf/2410.16711
• Copy Paste: [[2410.16711]] Development of CNN Architectures using Transfer Learning Methods for Medical Image Classification(https://arxiv.org/abs/2410.16711)
• Keywords: segmentation
• Abstract: The application of deep learning-based architecture has seen a tremendous rise in recent years. For example, medical image classification using deep learning achieved breakthrough results. Convolutional Neural Networks (CNNs) are implemented predominantly in medical image classification and segmentation. On the other hand, transfer learning has emerged as a prominent supporting tool for enhancing the efficiency and accuracy of deep learning models. This paper investigates the development of CNN architectures using transfer learning techniques in the field of medical image classification using a timeline mapping model for key image classification challenges. Our findings help make an informed decision while selecting the optimum and state-of-the-art CNN architectures.

Title: Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World

• Authors: Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David L. Donoho, Sanmi Koyejo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16713
• Pdf URL: https://arxiv.org/pdf/2410.16713
• Copy Paste: [[2410.16713]] Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World(https://arxiv.org/abs/2410.16713)
• Keywords: generative
• Abstract: The increasing presence of AI-generated content on the internet raises a critical question: What happens when generative machine learning models are pretrained on web-scale datasets containing data created by earlier models? Some authors prophesy $\textit{model collapse}$ under a "$\textit{replace}$" scenario: a sequence of models, the first trained with real data and each later one trained only on synthetic data from its preceding model. In this scenario, models successively degrade. Others see collapse as easily avoidable; in an "$\textit{accumulate}$' scenario, a sequence of models is trained, but each training uses all real and synthetic data generated so far. In this work, we deepen and extend the study of these contrasting scenarios. First, collapse versus avoidance of collapse is studied by comparing the replace and accumulate scenarios on each of three prominent generative modeling settings; we find the same contrast emerges in all three settings. Second, we study a compromise scenario; the available data remains the same as in the accumulate scenario -- but unlike $\textit{accumulate}$ and like $\textit{replace}$, each model is trained using a fixed compute budget; we demonstrate that model test loss on real data is larger than in the $\textit{accumulate}$ scenario, but apparently plateaus, unlike the divergence seen with $\textit{replace}$. Third, we study the relative importance of cardinality and proportion of real data for avoiding model collapse. Surprisingly, we find a non-trivial interaction between real and synthetic data, where the value of synthetic data for reducing test loss depends on the absolute quantity of real data. Our insights are particularly important when forecasting whether future frontier generative models will collapse or thrive, and our results open avenues for empirically and mathematically studying the context-dependent value of synthetic data.

Title: Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Models Alignment

• Authors: Mingzhi Wang, Chengdong Ma, Qizhi Chen, Linjian Meng, Yang Han, Jiancong Xiao, Zhaowei Zhang, Jing Huo, Weijie J. Su, Yaodong Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16714
• Pdf URL: https://arxiv.org/pdf/2410.16714
• Copy Paste: [[2410.16714]] Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Models Alignment(https://arxiv.org/abs/2410.16714)
• Keywords: large language model
• Abstract: Self-play methods have demonstrated remarkable success in enhancing model capabilities across various domains. In the context of Reinforcement Learning from Human Feedback (RLHF), self-play not only boosts Large Language Model (LLM) performance but also overcomes the limitations of traditional Bradley-Terry (BT) model assumptions by finding the Nash equilibrium (NE) of a preference-based, two-player constant-sum game. However, existing methods either guarantee only average-iterate convergence, incurring high storage and inference costs, or converge to the NE of a regularized game, failing to accurately reflect true human preferences. In this paper, we introduce Magnetic Preference Optimization (MPO), a novel approach capable of achieving last-iterate convergence to the NE of the original game, effectively overcoming the limitations of existing methods. Building upon Magnetic Mirror Descent (MMD), MPO attains a linear convergence rate, making it particularly suitable for fine-tuning LLMs. To ensure our algorithm is both theoretically sound and practically viable, we present a simple yet effective implementation that adapts the theoretical insights to the RLHF setting. Empirical results demonstrate that MPO can significantly enhance the performance of LLMs, highlighting the potential of self-play methods in alignment.

Title: Optimal Partial Graph Matching

• Authors: Gathika Ratnayaka, James Nichols, Qing Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16718
• Pdf URL: https://arxiv.org/pdf/2410.16718
• Copy Paste: [[2410.16718]] Optimal Partial Graph Matching(https://arxiv.org/abs/2410.16718)
• Keywords: robust
• Abstract: Partial graph matching addresses the limitations of traditional graph matching by allowing some nodes to remain unmatched, making it applicable to more complex scenarios. However, this flexibility introduces additional complexity, as both the subset of nodes to match and the optimal mapping must be determined. While recent studies have explored deep learning techniques for partial graph matching, a significant limitation remains: the absence of an optimization objective that fully captures the problem's intrinsic nature while enabling efficient solutions. In this paper, we propose a novel optimization framework for partial graph matching, inspired by optimal partial transport. Our approach formulates an objective that enables partial assignments while incorporating matching biases, using weighted total variation as the divergence function to guarantee optimal partial assignments. We employ the Hungarian algorithm to achieve efficient, exact solutions with cubic time complexity. Our contributions are threefold: (i) we introduce a robust optimization objective that balances matched and unmatched nodes; (ii) we establish a connection between partial graph matching and the linear sum assignment problem, enabling efficient solutions; (iii) we propose a deep graph matching architecture with a novel partial matching loss, providing an end-to-end solution. The empirical evaluations on standard graph matching benchmarks demonstrate the efficacy of the proposed approach.

Title: Progressive Compositionality In Text-to-Image Generative Models

• Authors: Xu Han, Linghao Jin, Xiaofeng Liu, Paul Pu Liang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16719
• Pdf URL: https://arxiv.org/pdf/2410.16719
• Copy Paste: [[2410.16719]] Progressive Compositionality In Text-to-Image Generative Models(https://arxiv.org/abs/2410.16719)
• Keywords: diffusion, generative
• Abstract: Despite the impressive text-to-image (T2I) synthesis capabilities of diffusion models, they often struggle to understand compositional relationships between objects and attributes, especially in complex settings. Existing solutions have tackled these challenges by optimizing the cross-attention mechanism or learning from the caption pairs with minimal semantic changes. However, can we generate high-quality complex contrastive images that diffusion models can directly discriminate based on visual representations? In this work, we leverage large-language models (LLMs) to compose realistic, complex scenarios and harness Visual-Question Answering (VQA) systems alongside diffusion models to automatically curate a contrastive dataset, ConPair, consisting of 15k pairs of high-quality contrastive images. These pairs feature minimal visual discrepancies and cover a wide range of attribute categories, especially complex and natural scenarios. To learn effectively from these error cases, i.e., hard negative images, we propose EvoGen, a new multi-stage curriculum for contrastive learning of diffusion models. Through extensive experiments across a wide range of compositional scenarios, we showcase the effectiveness of our proposed framework on compositional T2I benchmarks.

Title: Polyp-E: Benchmarking the Robustness of Deep Segmentation Models via Polyp Editing

• Authors: Runpu Wei, Zijin Yin, Kongming Liang, Min Min, Chengwei Pan, Gang Yu, Haonan Huang, Yan Liu, Zhanyu Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16732
• Pdf URL: https://arxiv.org/pdf/2410.16732
• Copy Paste: [[2410.16732]] Polyp-E: Benchmarking the Robustness of Deep Segmentation Models via Polyp Editing(https://arxiv.org/abs/2410.16732)
• Keywords: robust, diffusion, segmentation
• Abstract: Automatic polyp segmentation is helpful to assist clinical diagnosis and treatment. In daily clinical practice, clinicians exhibit robustness in identifying polyps with both location and size variations. It is uncertain if deep segmentation models can achieve comparable robustness in automated colonoscopic analysis. To benchmark the model robustness, we focus on evaluating the robustness of segmentation models on the polyps with various attributes (e.g. location and size) and healthy samples. Based on the Latent Diffusion Model, we perform attribute editing on real polyps and build a new dataset named Polyp-E. Our synthetic dataset boasts exceptional realism, to the extent that clinical experts find it challenging to discern them from real data. We evaluate several existing polyp segmentation models on the proposed benchmark. The results reveal most of the models are highly sensitive to attribute variations. As a novel data augmentation technique, the proposed editing pipeline can improve both in-distribution and out-of-distribution generalization ability. The code and datasets will be released.

Title: Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration

• Authors: Qintong Li, Jiahui Gao, Sheng Wang, Renjie Pi, Xueliang Zhao, Chuan Wu, Xin Jiang, Zhenguo Li, Lingpeng Kong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16736
• Pdf URL: https://arxiv.org/pdf/2410.16736
• Copy Paste: [[2410.16736]] Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration(https://arxiv.org/abs/2410.16736)
• Keywords: large language model
• Abstract: Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data, leading to impressive performance across a range of downstream applications. Current methods often rely on human-annotated data or predefined task templates to direct powerful LLMs in synthesizing task-relevant data for effective model training. However, this dependence on manually designed components may constrain the scope of generated data, potentially overlooking critical edge cases or novel scenarios that could challenge the model. In this paper, we present a novel approach, ReverseGen, designed to automatically generate effective training samples that expose the weaknesses of LLMs. Specifically, we introduce a dedicated proposer trained to produce queries that lead target models to generate unsatisfactory responses. These failure-inducing queries are then used to construct training data, helping to address the models' shortcomings and improve overall performance. Our approach is flexible and can be applied to models of various scales (3B, 7B, and 8B). We evaluate ReverseGen on three key applications (safety, honesty, and math), demonstrating that our generated data is both highly effective and diverse. Models fine-tuned with ReverseGen-generated data consistently outperform those trained on human-annotated or general model-generated data, offering a new perspective on data synthesis for task-specific LLM enhancement.

Title: Interactive Residual Domain Adaptation Networks for Partial Transfer Industrial Fault Diagnosis

• Authors: Gecheng Chen, Chengwen Luo, Jianqiang Li, Xinkai Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16737
• Pdf URL: https://arxiv.org/pdf/2410.16737
• Copy Paste: [[2410.16737]] Interactive Residual Domain Adaptation Networks for Partial Transfer Industrial Fault Diagnosis(https://arxiv.org/abs/2410.16737)
• Keywords: extraction
• Abstract: The partial domain adaptation (PDA) challenge is a prevalent issue in industrial fault diagnosis. Current PDA approaches primarily rely on adversarial learning for domain adaptation and use reweighting strategies to exclude source samples deemed outliers. However, the transferability of features diminishes from general feature extraction layers to higher task-specific layers in adversarial learning-based adaptation modules, leading to significant negative transfer in PDA settings. We term this issue the adaptation-discrimination paradox (ADP). Furthermore, reweighting strategies often suffer from unreliable pseudo-labels, compromising their effectiveness. Drawing inspiration from traditional classification settings where such partial challenge is not a concern, we propose a novel PDA framework called Interactive Residual Domain Adaptation Networks (IRDAN), which introduces domain-wise models for each domain to provide a new perspective for the PDA challenge. Each domain-wise model is equipped with a residual domain adaptation (RDA) block to mitigate the ADP problem. Additionally, we introduce a confident information flow via an interactive learning strategy, training the modules of IRDAN sequentially to avoid cross-interference. We also establish a reliable stopping criterion for selecting the best-performing model, ensuring practical usability in real-world applications. Experiments have demonstrated the superior performance of the proposed IRDAN.

Title: LLM-Assisted Red Teaming of Diffusion Models through "Failures Are Fated, But Can Be Faded"

• Authors: Som Sagar, Aditya Taparia, Ransalu Senanayake
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16738
• Pdf URL: https://arxiv.org/pdf/2410.16738
• Copy Paste: [[2410.16738]] LLM-Assisted Red Teaming of Diffusion Models through "Failures Are Fated, But Can Be Faded"(https://arxiv.org/abs/2410.16738)
• Keywords: diffusion, generative
• Abstract: In large deep neural networks that seem to perform surprisingly well on many tasks, we also observe a few failures related to accuracy, social biases, and alignment with human values, among others. Therefore, before deploying these models, it is crucial to characterize this failure landscape for engineers to debug or audit models. Nevertheless, it is infeasible to exhaustively test for all possible combinations of factors that could lead to a model's failure. In this paper, we improve the "Failures are fated, but can be faded" framework (arXiv:2406.07145)--a post-hoc method to explore and construct the failure landscape in pre-trained generative models--with a variety of deep reinforcement learning algorithms, screening tests, and LLM-based rewards and state generation. With the aid of limited human feedback, we then demonstrate how to restructure the failure landscape to be more desirable by moving away from the discovered failure modes. We empirically demonstrate the effectiveness of the proposed method on diffusion models. We also highlight the strengths and weaknesses of each algorithm in identifying failure modes.

Title: SpikMamba: When SNN meets Mamba in Event-based Human Action Recognition

• Authors: Jiaqi Chen, Yan Yang, Shizhuo Deng, Da Teng, Liyuan Pan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16746
• Pdf URL: https://arxiv.org/pdf/2410.16746
• Copy Paste: [[2410.16746]] SpikMamba: When SNN meets Mamba in Event-based Human Action Recognition(https://arxiv.org/abs/2410.16746)
• Keywords: privacy
• Abstract: Human action recognition (HAR) plays a key role in various applications such as video analysis, surveillance, autonomous driving, robotics, and healthcare. Most HAR algorithms are developed from RGB images, which capture detailed visual information. However, these algorithms raise concerns in privacy-sensitive environments due to the recording of identifiable features. Event cameras offer a promising solution by capturing scene brightness changes sparsely at the pixel level, without capturing full images. Moreover, event cameras have high dynamic ranges that can effectively handle scenarios with complex lighting conditions, such as low light or high contrast environments. However, using event cameras introduces challenges in modeling the spatially sparse and high temporal resolution event data for HAR. To address these issues, we propose the SpikMamba framework, which combines the energy efficiency of spiking neural networks and the long sequence modeling capability of Mamba to efficiently capture global features from spatially sparse and high a temporal resolution event data. Additionally, to improve the locality of modeling, a spiking window-based linear attention mechanism is used. Extensive experiments show that SpikMamba achieves remarkable recognition performance, surpassing the previous state-of-the-art by 1.45%, 7.22%, 0.15%, and 3.92% on the PAF, HARDVS, DVS128, and E-FAction datasets, respectively. The code is available at this https URL.

Title: The Scene Language: Representing Scenes with Programs, Words, and Embeddings

• Authors: Yunzhi Zhang, Zizhang Li, Matt Zhou, Shangzhe Wu, Jiajun Wu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16770
• Pdf URL: https://arxiv.org/pdf/2410.16770
• Copy Paste: [[2410.16770]] The Scene Language: Representing Scenes with Programs, Words, and Embeddings(https://arxiv.org/abs/2410.16770)
• Keywords: robust
• Abstract: We introduce the Scene Language, a visual scene representation that concisely and precisely describes the structure, semantics, and identity of visual scenes. It represents a scene with three key components: a program that specifies the hierarchical and relational structure of entities in the scene, words in natural language that summarize the semantic class of each entity, and embeddings that capture the visual identity of each entity. This representation can be inferred from pre-trained language models via a training-free inference technique, given text or image inputs. The resulting scene can be rendered into images using traditional, neural, or hybrid graphics renderers. Together, this forms a robust, automated system for high-quality 3D and 4D scene generation. Compared with existing representations like scene graphs, our proposed Scene Language generates complex scenes with higher fidelity, while explicitly modeling the scene structures to enable precise control and editing.

Title: Beyond Retrieval: Generating Narratives in Conversational Recommender Systems

• Authors: Krishna Sayana, Raghavendra Vasudeva, Yuri Vasilevski, Kun Su, Liam Hebert, Hubert Pham, Ambarish Jash, Sukhdeep Sodhi
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16780
• Pdf URL: https://arxiv.org/pdf/2410.16780
• Copy Paste: [[2410.16780]] Beyond Retrieval: Generating Narratives in Conversational Recommender Systems(https://arxiv.org/abs/2410.16780)
• Keywords: generative, large language model
• Abstract: The recent advances in Large Language Model's generation and reasoning capabilities present an opportunity to develop truly conversational recommendation systems. However, effectively integrating recommender system knowledge into LLMs for natural language generation which is tailored towards recommendation tasks remains a challenge. This paper addresses this challenge by making two key contributions. First, we introduce a new dataset (REGEN) for natural language generation tasks in conversational recommendations. REGEN (Reviews Enhanced with GEnerative Narratives) extends the Amazon Product Reviews dataset with rich user narratives, including personalized explanations of product preferences, product endorsements for recommended items, and summaries of user purchase history. REGEN is made publicly available to facilitate further research. Furthermore, we establish benchmarks using well-known generative metrics, and perform an automated evaluation of the new dataset using a rater LLM. Second, the paper introduces a fusion architecture (CF model with an LLM) which serves as a baseline for REGEN. And to the best of our knowledge, represents the first attempt to analyze the capabilities of LLMs in understanding recommender signals and generating rich narratives. We demonstrate that LLMs can effectively learn from simple fusion architectures utilizing interaction-based CF embeddings, and this can be further enhanced using the metadata and personalization data associated with items. Our experiments show that combining CF and content embeddings leads to improvements of 4-12% in key language metrics compared to using either type of embedding individually. We also provide an analysis to interpret how CF and content embeddings contribute to this new generative task.

Title: One-Step Diffusion Distillation through Score Implicit Matching

• Authors: Weijian Luo, Zemin Huang, Zhengyang Geng, J. Zico Kolter, Guo-jun Qi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16794
• Pdf URL: https://arxiv.org/pdf/2410.16794
• Copy Paste: [[2410.16794]] One-Step Diffusion Distillation through Score Implicit Matching(https://arxiv.org/abs/2410.16794)
• Keywords: diffusion, data-free, transformer, generative
• Abstract: Despite their strong performances on many generative tasks, diffusion models require a large number of sampling steps in order to generate realistic samples. This has motivated the community to develop effective methods to distill pre-trained diffusion models into more efficient models, but these methods still typically require few-step inference or perform substantially worse than the underlying model. In this paper, we present Score Implicit Matching (SIM) a new approach to distilling pre-trained diffusion models into single-step generator models, while maintaining almost the same sample generation ability as the original model as well as being data-free with no need of training samples for distillation. The method rests upon the fact that, although the traditional score-based loss is intractable to minimize for generator models, under certain conditions we can efficiently compute the gradients for a wide class of score-based divergences between a diffusion model and a generator. SIM shows strong empirical performances for one-step generators: on the CIFAR10 dataset, it achieves an FID of 2.06 for unconditional generation and 1.96 for class-conditional generation. Moreover, by applying SIM to a leading transformer-based diffusion model, we distill a single-step generator for text-to-image (T2I) generation that attains an aesthetic score of 6.42 with no performance decline over the original multi-step counterpart, clearly outperforming the other one-step generators including SDXL-TURBO of 5.33, SDXL-LIGHTNING of 5.34 and HYPER-SDXL of 5.85. We will release this industry-ready one-step transformer-based T2I generator along with this paper.

Title: Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models

• Authors: Yuheng Lu, Bingshuo Qian, Caixia Yuan, Huixing Jiang, Xiaojie Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16801
• Pdf URL: https://arxiv.org/pdf/2410.16801
• Copy Paste: [[2410.16801]] Controlled Low-Rank Adaptation with Subspace Regularization for Continued Training on Large Language Models(https://arxiv.org/abs/2410.16801)
• Keywords: large language model
• Abstract: Large language models (LLMs) exhibit remarkable capabilities in natural language processing but face catastrophic forgetting when learning new tasks, where adaptation to a new domain leads to a substantial decline in performance on previous tasks. In this paper, we propose Controlled LoRA (CLoRA), a subspace regularization method on LoRA structure. Aiming to reduce the scale of output change while introduce minimal constraint on model capacity, CLoRA imposes constraint on the direction of updating matrix null space. Experimental results on commonly used LLM finetuning tasks reveal that CLoRA significantly outperforms existing LoRA subsequent methods on both in-domain and outdomain evaluations, highlighting the superority of CLoRA as a effective parameter-efficient finetuning method with catastrophic forgetting mitigating. Further investigation for model parameters indicates that CLoRA effectively balances the trade-off between model capacity and degree of forgetting.

Title: Evaluating the Effectiveness of Attack-Agnostic Features for Morphing Attack Detection

• Authors: Laurent Colbois, Sébastien Marcel
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.16802
• Pdf URL: https://arxiv.org/pdf/2410.16802
• Copy Paste: [[2410.16802]] Evaluating the Effectiveness of Attack-Agnostic Features for Morphing Attack Detection(https://arxiv.org/abs/2410.16802)
• Keywords: attack, robust, diffusion, generative
• Abstract: Morphing attacks have diversified significantly over the past years, with new methods based on generative adversarial networks (GANs) and diffusion models posing substantial threats to face recognition systems. Recent research has demonstrated the effectiveness of features extracted from large vision models pretrained on bonafide data only (attack-agnostic features) for detecting deep generative images. Building on this, we investigate the potential of these image representations for morphing attack detection (MAD). We develop supervised detectors by training a simple binary linear SVM on the extracted features and one-class detectors by modeling the distribution of bonafide features with a Gaussian Mixture Model (GMM). Our method is evaluated across a comprehensive set of attacks and various scenarios, including generalization to unseen attacks, different source datasets, and print-scan data. Our results indicate that attack-agnostic features can effectively detect morphing attacks, outperforming traditional supervised and one-class detectors from the literature in most scenarios. Additionally, we provide insights into the strengths and limitations of each considered representation and discuss potential future research directions to further enhance the robustness and generalizability of our approach.

Title: Test-time Adversarial Defense with Opposite Adversarial Path and High Attack Time Cost

• Authors: Cheng-Han Yeh, Kuanchun Yu, Chun-Shien Lu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.16805
• Pdf URL: https://arxiv.org/pdf/2410.16805
• Copy Paste: [[2410.16805]] Test-time Adversarial Defense with Opposite Adversarial Path and High Attack Time Cost(https://arxiv.org/abs/2410.16805)
• Keywords: defense, attack, robust, diffusion
• Abstract: Deep learning models are known to be vulnerable to adversarial attacks by injecting sophisticated designed perturbations to input data. Training-time defenses still exhibit a significant performance gap between natural accuracy and robust accuracy. In this paper, we investigate a new test-time adversarial defense method via diffusion-based recovery along opposite adversarial paths (OAPs). We present a purifier that can be plugged into a pre-trained model to resist adversarial attacks. Different from prior arts, the key idea is excessive denoising or purification by integrating the opposite adversarial direction with reverse diffusion to push the input image further toward the opposite adversarial direction. For the first time, we also exemplify the pitfall of conducting AutoAttack (Rand) for diffusion-based defense methods. Through the lens of time complexity, we examine the trade-off between the effectiveness of adaptive attack and its computation complexity against our defense. Experimental evaluation along with time cost analysis verifies the effectiveness of the proposed method.

Title: Masked Clinical Modelling: A Framework for Synthetic and Augmented Survival Data Generation

• Authors: Nicholas I-Hsien Kuo, Blanca Gallego, Louisa Jorm
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16811
• Pdf URL: https://arxiv.org/pdf/2410.16811
• Copy Paste: [[2410.16811]] Masked Clinical Modelling: A Framework for Synthetic and Augmented Survival Data Generation(https://arxiv.org/abs/2410.16811)
• Keywords: secure, privacy
• Abstract: Access to real clinical data is often restricted due to privacy obligations, creating significant barriers for healthcare research. Synthetic datasets provide a promising solution, enabling secure data sharing and model development. However, most existing approaches focus on data realism rather than utility -- ensuring that models trained on synthetic data yield clinically meaningful insights comparable to those trained on real data. In this paper, we present Masked Clinical Modelling (MCM), a framework inspired by masked language modelling, designed for both data synthesis and conditional data augmentation. We evaluate this prototype on the WHAS500 dataset using Cox Proportional Hazards models, focusing on the preservation of hazard ratios as key clinical metrics. Our results show that data generated using the MCM framework improves both discrimination and calibration in survival analysis, outperforming existing methods. MCM demonstrates strong potential to support survival data analysis and broader healthcare applications.

Title: Optimizing Chain-of-Thought Reasoning: Tackling Arranging Bottleneck via Plan Augmentation

• Authors: Yuli Qiu, Jiashu Yao, Heyan Huang, Yuhang Guo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16812
• Pdf URL: https://arxiv.org/pdf/2410.16812
• Copy Paste: [[2410.16812]] Optimizing Chain-of-Thought Reasoning: Tackling Arranging Bottleneck via Plan Augmentation(https://arxiv.org/abs/2410.16812)
• Keywords: large language model
• Abstract: Multi-step reasoning ability of large language models is crucial in tasks such as math and tool utilization. Current researches predominantly focus on enhancing model performance in these multi-step reasoning tasks through fine-tuning with Chain-of-Thought (CoT) steps, yet these methods tend to be heuristic, without exploring nor resolving the bottleneck. In this study, we subdivide CoT reasoning into two parts: arranging and executing, and identify that the bottleneck of models mainly lies in arranging rather than executing. Based on this finding, we propose a plan-based training and reasoning method that guides models to generate arranging steps through abstract plans. We experiment on both math (GSM8k) and tool utilization (ToolBench) benchmarks. Results show that compared to fine-tuning directly with CoT data, our approach achieves a better performance on alleviating arranging bottleneck, particularly excelling in long-distance reasoning generalization.

Title: PerspectiveNet: Multi-View Perception for Dynamic Scene Understanding

• Authors: Vinh Nguyen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16824
• Pdf URL: https://arxiv.org/pdf/2410.16824
• Copy Paste: [[2410.16824]] PerspectiveNet: Multi-View Perception for Dynamic Scene Understanding(https://arxiv.org/abs/2410.16824)
• Keywords: extraction, large language model
• Abstract: Generating detailed descriptions from multiple cameras and viewpoints is challenging due to the complex and inconsistent nature of visual data. In this paper, we introduce PerspectiveNet, a lightweight yet efficient model for generating long descriptions across multiple camera views. Our approach utilizes a vision encoder, a compact connector module to convert visual features into a fixed-size tensor, and large language models (LLMs) to harness the strong natural language generation capabilities of LLMs. The connector module is designed with three main goals: mapping visual features onto LLM embeddings, emphasizing key information needed for description generation, and producing a fixed-size feature matrix. Additionally, we augment our solution with a secondary task, the correct frame sequence detection, enabling the model to search for the correct sequence of frames to generate descriptions. Finally, we integrate the connector module, the secondary task, the LLM, and a visual feature extraction model into a single architecture, which is trained for the Traffic Safety Description and Analysis task. This task requires generating detailed, fine-grained descriptions of events from multiple cameras and viewpoints. The resulting model is lightweight, ensuring efficient training and inference, while remaining highly effective.

Title: MPDS: A Movie Posters Dataset for Image Generation with Diffusion Model

• Authors: Meng Xu (1), Tong Zhang (1), Fuyun Wang (1), Yi Lei (1), Xin Liu (2), Zhen Cui (1) ((1) Nanjing University of Science and Technology, Nanjing, China., (2) SeetaCloud, Nanjing, China.)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16840
• Pdf URL: https://arxiv.org/pdf/2410.16840
• Copy Paste: [[2410.16840]] MPDS: A Movie Posters Dataset for Image Generation with Diffusion Model(https://arxiv.org/abs/2410.16840)
• Keywords: diffusion
• Abstract: Movie posters are vital for captivating audiences, conveying themes, and driving market competition in the film industry. While traditional designs are laborious, intelligent generation technology offers efficiency gains and design enhancements. Despite exciting progress in image generation, current models often fall short in producing satisfactory poster results. The primary issue lies in the absence of specialized poster datasets for targeted model training. In this work, we propose a Movie Posters DataSet (MPDS), tailored for text-to-image generation models to revolutionize poster production. As dedicated to posters, MPDS stands out as the first image-text pair dataset to our knowledge, composing of 373k+ image-text pairs and 8k+ actor images (covering 4k+ actors). Detailed poster descriptions, such as movie titles, genres, casts, and synopses, are meticulously organized and standardized based on public movie synopsis, also named movie-synopsis prompt. To bolster poster descriptions as well as reduce differences from movie synopsis, further, we leverage a large-scale vision-language model to automatically produce vision-perceptive prompts for each poster, then perform manual rectification and integration with movie-synopsis prompt. In addition, we introduce a prompt of poster captions to exhibit text elements in posters like actor names and movie titles. For movie poster generation, we develop a multi-condition diffusion framework that takes poster prompt, poster caption, and actor image (for personalization) as inputs, yielding excellent results through the learning of a diffusion model. Experiments demonstrate the valuable role of our proposed MPDS dataset in advancing personalized movie poster generation. MPDS is available at this https URL.

Title: Assessment of Transformer-Based Encoder-Decoder Model for Human-Like Summarization

• Authors: Sindhu Nair, Y.S. Rao, Radha Shankarmani
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16842
• Pdf URL: https://arxiv.org/pdf/2410.16842
• Copy Paste: [[2410.16842]] Assessment of Transformer-Based Encoder-Decoder Model for Human-Like Summarization(https://arxiv.org/abs/2410.16842)
• Keywords: transformer
• Abstract: In recent times, extracting valuable information from large text is making significant progress. Especially in the current era of social media, people expect quick bites of information. Automatic text summarization seeks to tackle this by slimming large texts down into more manageable summaries. This important research area can aid in decision-making by digging out salient content from large text. With the progress in deep learning models, significant work in language models has emerged. The encoder-decoder framework in deep learning has become the central approach for automatic text summarization. This work leverages transformer-based BART model for human-like summarization which is an open-ended problem with many challenges. On training and fine-tuning the encoder-decoder model, it is tested with diverse sample articles and the quality of summaries of diverse samples is assessed based on human evaluation parameters. Further, the finetuned model performance is compared with the baseline pretrained model based on evaluation metrics like ROUGE score and BERTScore. Additionally, domain adaptation of the model is required for improved performance of abstractive summarization of dialogues between interlocutors. On investigating, the above popular evaluation metrics are found to be insensitive to factual errors. Further investigation of the summaries generated by finetuned model is done using the contemporary evaluation metrics of factual consistency like WeCheck and SummaC. Empirical results on BBC News articles highlight that the gold standard summaries written by humans are more factually consistent by 17% than the abstractive summaries generated by finetuned model.

Title: Trustworthy Alignment of Retrieval-Augmented Large Language Models via Reinforcement Learning

• Authors: Zongmeng Zhang, Yufeng Shi, Jinhua Zhu, Wengang Zhou, Xiang Qi, Peng Zhang, Houqiang Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16843
• Pdf URL: https://arxiv.org/pdf/2410.16843
• Copy Paste: [[2410.16843]] Trustworthy Alignment of Retrieval-Augmented Large Language Models via Reinforcement Learning(https://arxiv.org/abs/2410.16843)
• Keywords: large language model
• Abstract: Trustworthiness is an essential prerequisite for the real-world application of large language models. In this paper, we focus on the trustworthiness of language models with respect to retrieval augmentation. Despite being supported with external evidence, retrieval-augmented generation still suffers from hallucinations, one primary cause of which is the conflict between contextual and parametric knowledge. We deem that retrieval-augmented language models have the inherent capabilities of supplying response according to both contextual and parametric knowledge. Inspired by aligning language models with human preference, we take the first step towards aligning retrieval-augmented language models to a status where it responds relying merely on the external evidence and disregards the interference of parametric knowledge. Specifically, we propose a reinforcement learning based algorithm Trustworthy-Alignment, theoretically and experimentally demonstrating large language models' capability of reaching a trustworthy status without explicit supervision on how to respond. Our work highlights the potential of large language models on exploring its intrinsic abilities by its own and expands the application scenarios of alignment from fulfilling human preference to creating trustworthy agents.

Title: Fast Graph Sharpness-Aware Minimization for Enhancing and Accelerating Few-Shot Node Classification

• Authors: Yihong Luo, Yuhan Chen, Siya Qiu, Yiwei Wang, Chen Zhang, Yan Zhou, Xiaochun Cao, Jing Tang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16845
• Pdf URL: https://arxiv.org/pdf/2410.16845
• Copy Paste: [[2410.16845]] Fast Graph Sharpness-Aware Minimization for Enhancing and Accelerating Few-Shot Node Classification(https://arxiv.org/abs/2410.16845)
• Keywords: robust
• Abstract: Graph Neural Networks (GNNs) have shown superior performance in node classification. However, GNNs perform poorly in the Few-Shot Node Classification (FSNC) task that requires robust generalization to make accurate predictions for unseen classes with limited labels. To tackle the challenge, we propose the integration of Sharpness-Aware Minimization (SAM)--a technique designed to enhance model generalization by finding a flat minimum of the loss landscape--into GNN training. The standard SAM approach, however, consists of two forward-backward steps in each training iteration, doubling the computational cost compared to the base optimizer (e.g., Adam). To mitigate this drawback, we introduce a novel algorithm, Fast Graph Sharpness-Aware Minimization (FGSAM), that integrates the rapid training of Multi-Layer Perceptrons (MLPs) with the superior performance of GNNs. Specifically, we utilize GNNs for parameter perturbation while employing MLPs to minimize the perturbed loss so that we can find a flat minimum with good generalization more efficiently. Moreover, our method reutilizes the gradient from the perturbation phase to incorporate graph topology into the minimization process at almost zero additional cost. To further enhance training efficiency, we develop FGSAM+ that executes exact perturbations periodically. Extensive experiments demonstrate that our proposed algorithm outperforms the standard SAM with lower computational costs in FSNC tasks. In particular, our FGSAM+ as a SAM variant offers a faster optimization than the base optimizer in most cases. In addition to FSNC, our proposed methods also demonstrate competitive performance in the standard node classification task for heterophilic graphs, highlighting the broad applicability. The code is available at this https URL.

Title: ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage

• Authors: Taewhoo Lee, Chanwoong Yoon, Kyochul Jang, Donghyeon Lee, Minju Song, Hyunjae Kim, Jaewoo Kang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16848
• Pdf URL: https://arxiv.org/pdf/2410.16848
• Copy Paste: [[2410.16848]] ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage(https://arxiv.org/abs/2410.16848)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLM) capable of processing extremely long texts highlight the need for a dedicated evaluation benchmark to assess their long-context capabilities. However, existing methods, like the needle-in-a-haystack test, do not effectively assess whether these models fully utilize contextual information, raising concerns about the reliability of current evaluation techniques. To thoroughly examine the effectiveness of existing benchmarks, we introduce a new metric called information coverage (IC), which quantifies the proportion of the input context necessary for answering queries. Our findings indicate that current benchmarks exhibit low IC; although the input context may be extensive, the actual usable context is often limited. To address this, we present ETHIC, a novel benchmark designed to assess LLMs' ability to leverage the entire context. Our benchmark comprises 2,648 test instances spanning four long-context tasks with high IC scores in the domains of books, debates, medicine, and law. Our evaluations reveal significant performance drops in contemporary LLMs, highlighting a critical challenge in managing long contexts. Our benchmark is available at this https URL.

Title: CK4Gen: A Knowledge Distillation Framework for Generating High-Utility Synthetic Survival Datasets in Healthcare

• Authors: Nicholas I-Hsien Kuo, Blanca Gallego, Louisa Jorm
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16872
• Pdf URL: https://arxiv.org/pdf/2410.16872
• Copy Paste: [[2410.16872]] CK4Gen: A Knowledge Distillation Framework for Generating High-Utility Synthetic Survival Datasets in Healthcare(https://arxiv.org/abs/2410.16872)
• Keywords: privacy, generative
• Abstract: Access to real clinical data is heavily restricted by privacy regulations, hindering both healthcare research and education. These constraints slow progress in developing new treatments and data-driven healthcare solutions, while also limiting students' access to real-world datasets, leaving them without essential practical skills. High-utility synthetic datasets are therefore critical for advancing research and providing meaningful training material. However, current generative models -- such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) -- produce surface-level realism at the expense of healthcare utility, blending distinct patient profiles and producing synthetic data of limited practical relevance. To overcome these limitations, we introduce CK4Gen (Cox Knowledge for Generation), a novel framework that leverages knowledge distillation from Cox Proportional Hazards (CoxPH) models to create synthetic survival datasets that preserve key clinical characteristics, including hazard ratios and survival curves. CK4Gen avoids the interpolation issues seen in VAEs and GANs by maintaining distinct patient risk profiles, ensuring realistic and reliable outputs for research and educational use. Validated across four benchmark datasets -- GBSG2, ACTG320, WHAS500, and FLChain -- CK4Gen outperforms competing techniques by better aligning real and synthetic data, enhancing survival model performance in both discrimination and calibration via data augmentation. As CK4Gen is scalable across clinical conditions, and with code to be made publicly available, future researchers can apply it to their own datasets to generate synthetic versions suitable for open sharing.

Title: Just In Time Transformers

• Authors: Ahmed Ala Eddine Benali, Massimo Cafaro, Italo Epicoco, Marco Pulimeno, Enrico Junior Schioppa
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16881
• Pdf URL: https://arxiv.org/pdf/2410.16881
• Copy Paste: [[2410.16881]] Just In Time Transformers(https://arxiv.org/abs/2410.16881)
• Keywords: transformer
• Abstract: Precise energy load forecasting in residential households is crucial for mitigating carbon emissions and enhancing energy efficiency; indeed, accurate forecasting enables utility companies and policymakers, who advocate sustainable energy practices, to optimize resource utilization. Moreover, smart meters provide valuable information by allowing for granular insights into consumption patterns. Building upon available smart meter data, our study aims to cluster consumers into distinct groups according to their energy usage behaviours, effectively capturing a diverse spectrum of consumption patterns. Next, we design JITtrans (Just In Time transformer), a novel transformer deep learning model that significantly improves energy consumption forecasting accuracy, with respect to traditional forecasting methods. Extensive experimental results validate our claims using proprietary smart meter data. Our findings highlight the potential of advanced predictive technologies to revolutionize energy management and advance sustainable power systems: the development of efficient and eco-friendly energy solutions critically depends on such technologies.

Title: Network Inversion for Training-Like Data Reconstruction

• Authors: Pirzada Suhail, Amit Sethi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16884
• Pdf URL: https://arxiv.org/pdf/2410.16884
• Copy Paste: [[2410.16884]] Network Inversion for Training-Like Data Reconstruction(https://arxiv.org/abs/2410.16884)
• Keywords: privacy
• Abstract: Machine Learning models are often trained on proprietary and private data that cannot be shared, though the trained models themselves are distributed openly assuming that sharing model weights is privacy preserving, as training data is not expected to be inferred from the model weights. In this paper, we present Training-Like Data Reconstruction (TLDR), a network inversion-based approach to reconstruct training-like data from trained models. To begin with, we introduce a comprehensive network inversion technique that learns the input space corresponding to different classes in the classifier using a single conditioned generator. While inversion may typically return random and arbitrary input images for a given output label, we modify the inversion process to incentivize the generator to reconstruct training-like data by exploiting key properties of the classifier with respect to the training data along with some prior knowledge about the images. To validate our approach, we conduct empirical evaluations on multiple standard vision classification datasets, thereby highlighting the potential privacy risks involved in sharing machine learning models.

Title: Unsupervised Time Series Anomaly Prediction with Importance-based Generative Contrastive Learning

• Authors: Kai Zhao, Zhihao Zhuang, Chenjuan Guo, Hao Miao, Yunyao Cheng, Bin Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16888
• Pdf URL: https://arxiv.org/pdf/2410.16888
• Copy Paste: [[2410.16888]] Unsupervised Time Series Anomaly Prediction with Importance-based Generative Contrastive Learning(https://arxiv.org/abs/2410.16888)
• Keywords: generative
• Abstract: Time series anomaly prediction plays an essential role in many real-world scenarios, such as environmental prevention and prompt maintenance of cyber-physical systems. However, existing time series anomaly prediction methods mainly require supervised training with plenty of manually labeled data, which are difficult to obtain in practice. Besides, unseen anomalies can occur during inference, which could differ from the labeled training data and make these models fail to predict such new anomalies. In this paper, we study a novel problem of unsupervised time series anomaly prediction. We provide a theoretical analysis and propose Importance-based Generative Contrastive Learning (IGCL) to address the aforementioned problems. IGCL distinguishes between normal and anomaly precursors, which are generated by our anomaly precursor pattern generation module. To address the efficiency issues caused by the potential complex anomaly precursor combinations, we propose a memory bank with importance-based scores to adaptively store representative anomaly precursors and generate more complicated anomaly precursors. Extensive experiments on seven benchmark datasets show our method outperforms state-of-the-art baselines on unsupervised time series anomaly prediction problems.

Title: VistaDream: Sampling multiview consistent images for single-view scene reconstruction

• Authors: Haiping Wang, Yuan Liu, Ziwei Liu, Wenping Wang, Zhen Dong, Bisheng Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16892
• Pdf URL: https://arxiv.org/pdf/2410.16892
• Copy Paste: [[2410.16892]] VistaDream: Sampling multiview consistent images for single-view scene reconstruction(https://arxiv.org/abs/2410.16892)
• Keywords: diffusion
• Abstract: In this paper, we propose VistaDream a novel framework to reconstruct a 3D scene from a single-view image. Recent diffusion models enable generating high-quality novel-view images from a single-view input image. Most existing methods only concentrate on building the consistency between the input image and the generated images while losing the consistency between the generated images. VistaDream addresses this problem by a two-stage pipeline. In the first stage, VistaDream begins with building a global coarse 3D scaffold by zooming out a little step with inpainted boundaries and an estimated depth map. Then, on this global scaffold, we use iterative diffusion-based RGB-D inpainting to generate novel-view images to inpaint the holes of the scaffold. In the second stage, we further enhance the consistency between the generated novel-view images by a novel training-free Multiview Consistency Sampling (MCS) that introduces multi-view consistency constraints in the reverse sampling process of diffusion models. Experimental results demonstrate that without training or fine-tuning existing diffusion models, VistaDream achieves consistent and high-quality novel view synthesis using just single-view images and outperforms baseline methods by a large margin. The code, videos, and interactive demos are available at this https URL.

Title: Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projections

• Authors: Marco Miani, Hrittik Roy, Søren Hauberg
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.16901
• Pdf URL: https://arxiv.org/pdf/2410.16901
• Copy Paste: [[2410.16901]] Bayes without Underfitting: Fully Correlated Deep Learning Posteriors via Alternating Projections(https://arxiv.org/abs/2410.16901)
• Keywords: transformer, generative
• Abstract: Bayesian deep learning all too often underfits so that the Bayesian prediction is less accurate than a simple point estimate. Uncertainty quantification then comes at the cost of accuracy. For linearized models, the null space of the generalized Gauss-Newton matrix corresponds to parameters that preserve the training predictions of the point estimate. We propose to build Bayesian approximations in this null space, thereby guaranteeing that the Bayesian predictive does not underfit. We suggest a matrix-free algorithm for projecting onto this null space, which scales linearly with the number of parameters and quadratically with the number of output dimensions. We further propose an approximation that only scales linearly with parameters to make the method applicable to generative models. An extensive empirical evaluation shows that the approach scales to large models, including vision transformers with 28 million parameters.

Title: Hierarchical Clustering for Conditional Diffusion in Image Generation

• Authors: Jorge da Silva Goncalves, Laura Manduchi, Moritz Vandenhirtz, Julia E. Vogt
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16910
• Pdf URL: https://arxiv.org/pdf/2410.16910
• Copy Paste: [[2410.16910]] Hierarchical Clustering for Conditional Diffusion in Image Generation(https://arxiv.org/abs/2410.16910)
• Keywords: diffusion, generative
• Abstract: Finding clusters of data points with similar characteristics and generating new cluster-specific samples can significantly enhance our understanding of complex data distributions. While clustering has been widely explored using Variational Autoencoders, these models often lack generation quality in real-world datasets. This paper addresses this gap by introducing TreeDiffusion, a deep generative model that conditions Diffusion Models on hierarchical clusters to obtain high-quality, cluster-specific generations. The proposed pipeline consists of two steps: a VAE-based clustering model that learns the hierarchical structure of the data, and a conditional diffusion model that generates realistic images for each cluster. We propose this two-stage process to ensure that the generated samples remain representative of their respective clusters and enhance image fidelity to the level of diffusion models. A key strength of our method is its ability to create images for each cluster, providing better visualization of the learned representations by the clustering model, as demonstrated through qualitative results. This method effectively addresses the generative limitations of VAE-based approaches while preserving their clustering performance. Empirically, we demonstrate that conditioning diffusion models on hierarchical clusters significantly enhances generative performance, thereby advancing the state of generative clustering models.

Title: Pyramid Vector Quantization for LLMs

• Authors: Tycho F. A. van der Ouderaa, Maximilian L. Croci, Agrin Hilmkil, James Hensman
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16926
• Pdf URL: https://arxiv.org/pdf/2410.16926
• Copy Paste: [[2410.16926]] Pyramid Vector Quantization for LLMs(https://arxiv.org/abs/2410.16926)
• Keywords: large language model
• Abstract: Recent works on compression of large language models (LLM) using quantization considered reparameterizing the architecture such that weights are distributed on the sphere. This demonstratively improves the ability to quantize by increasing the mathematical notion of coherence, resulting in fewer weight outliers without affecting the network output. In this work, we aim to further exploit this spherical geometry of the weights when performing quantization by considering Pyramid Vector Quantization (PVQ) for large language models. Arranging points evenly on the sphere is notoriously difficult, especially in high dimensions, and in case approximate solutions exists, representing points explicitly in a codebook is typically not feasible due to its additional memory cost. Instead, PVQ uses a fixed integer lattice on the sphere by projecting points onto the 1-sphere, which allows for efficient encoding and decoding without requiring an explicit codebook in memory. To obtain a practical algorithm, we propose to combine PVQ with scale quantization for which we derive theoretically optimal quantizations, under empirically verified assumptions. Further, we extend pyramid vector quantization to use Hessian information to minimize quantization error under expected feature activations, instead of only relying on weight magnitudes. Experimentally, we achieves state-of-the-art quantization performance with pareto-optimal trade-off between performance and bits per weight and bits per activation, compared to compared methods. On weight-only, we find that we can quantize a Llama-3 70B model to 3.25 bits per weight and retain 98\% accuracy on downstream tasks.

Title: xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memories

• Authors: Maurice Kraus, Felix Divo, Devendra Singh Dhami, Kristian Kersting
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16928
• Pdf URL: https://arxiv.org/pdf/2410.16928
• Copy Paste: [[2410.16928]] xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memories(https://arxiv.org/abs/2410.16928)
• Keywords: robust
• Abstract: Time series data is prevalent across numerous fields, necessitating the development of robust and accurate forecasting models. Capturing patterns both within and between temporal and multivariate components is crucial for reliable predictions. We introduce xLSTM-Mixer, a model designed to effectively integrate temporal sequences, joint time-variate information, and multiple perspectives for robust forecasting. Our approach begins with a linear forecast shared across variates, which is then refined by xLSTM blocks. These blocks serve as key elements for modeling the complex dynamics of challenging time series data. xLSTM-Mixer ultimately reconciles two distinct views to produce the final forecast. Our extensive evaluations demonstrate xLSTM-Mixer's superior long-term forecasting performance compared to recent state-of-the-art methods. A thorough model analysis provides further insights into its key components and confirms its robustness and effectiveness. This work contributes to the resurgence of recurrent models in time series forecasting.

Title: Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passes

• Authors: Bryan R. Christ, Zack Gottesman, Jonathan Kropko, Thomas Hartvigsen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16930
• Pdf URL: https://arxiv.org/pdf/2410.16930
• Copy Paste: [[2410.16930]] Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passes(https://arxiv.org/abs/2410.16930)
• Keywords: large language model
• Abstract: Math reasoning is a highly active area of Large Language Model (LLM) research because it is a hallmark of artificial intelligence. However, few works have explored how math reasoning is encoded within LLM parameters and if it is a skill that can be isolated within a model. Doing so could allow targeted intervention to improve math performance without altering non-math behavior and foster understanding of how models encode math reasoning. We introduce Math Neurosurgery (MathNeuro), a method for isolating math-specific parameters in LLMs using only forward passes. MathNeuro builds on existing work by using weights and activations to calculate parameter importance, but isolates math-specific parameters by removing those important for general language tasks. Pruning parameters MathNeuro identifies deletes a LLM's math reasoning ability without destroying its general language ability. Scaling these parameters by a small constant improves a pretrained or instruction-tuned LLM's performance by 4-17% on GSM8K while leaving non-math behavior unaltered. MathNeuro is also data efficient: most of its effectiveness holds when identifying math-specific parameters using a single sample. MathNeuro highlights the potential for future work to intervene on math-specific parameters.

Title: LIMIS: Towards Language-based Interactive Medical Image Segmentation

• Authors: Lena Heinemann, Alexander Jaus, Zdravko Marinov, Moon Kim, Maria Francesca Spadea, Jens Kleesiek, Rainer Stiefelhagen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16939
• Pdf URL: https://arxiv.org/pdf/2410.16939
• Copy Paste: [[2410.16939]] LIMIS: Towards Language-based Interactive Medical Image Segmentation(https://arxiv.org/abs/2410.16939)
• Keywords: segmentation
• Abstract: Within this work, we introduce LIMIS: The first purely language-based interactive medical image segmentation model. We achieve this by adapting Grounded SAM to the medical domain and designing a language-based model interaction strategy that allows radiologists to incorporate their knowledge into the segmentation process. LIMIS produces high-quality initial segmentation masks by leveraging medical foundation models and allows users to adapt segmentation masks using only language, opening up interactive segmentation to scenarios where physicians require using their hands for other tasks. We evaluate LIMIS on three publicly available medical datasets in terms of performance and usability with experts from the medical domain confirming its high-quality segmentation masks and its interactive usability.

Title: DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization

• Authors: Haowei Zhu, Dehua Tang, Ji Liu, Mingjie Lu, Jintu Zheng, Jinzhang Peng, Dong Li, Yu Wang, Fan Jiang, Lu Tian, Spandan Tiwari, Ashish Sirasao, Jun-Hai Yong, Bin Wang, Emad Barsoum
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16942
• Pdf URL: https://arxiv.org/pdf/2410.16942
• Copy Paste: [[2410.16942]] DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization(https://arxiv.org/abs/2410.16942)
• Keywords: diffusion
• Abstract: Diffusion models have achieved remarkable progress in the field of image generation due to their outstanding capabilities. However, these models require substantial computing resources because of the multi-step denoising process during inference. While traditional pruning methods have been employed to optimize these models, the retraining process necessitates large-scale training datasets and extensive computational costs to maintain generalization ability, making it neither convenient nor efficient. Recent studies attempt to utilize the similarity of features across adjacent denoising stages to reduce computational costs through simple and static strategies. However, these strategies cannot fully harness the potential of the similar feature patterns across adjacent timesteps. In this work, we propose a novel pruning method that derives an efficient diffusion model via a more intelligent and differentiable pruner. At the core of our approach is casting the model pruning process into a SubNet search process. Specifically, we first introduce a SuperNet based on standard diffusion via adding some backup connections built upon the similar features. We then construct a plugin pruner network and design optimization losses to identify redundant computation. Finally, our method can identify an optimal SubNet through few-step gradient optimization and a simple post-processing procedure. We conduct extensive experiments on various diffusion models including Stable Diffusion series and DiTs. Our DiP-GO approach achieves 4.4 x speedup for SD-1.5 without any loss of accuracy, significantly outperforming the previous state-of-the-art methods.

Title: Breaking ReAct Agents: Foot-in-the-Door Attack Will Get You In

• Authors: Itay Nakash, George Kour, Guy Uziel, Ateret Anaby-Tavor
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.16950
• Pdf URL: https://arxiv.org/pdf/2410.16950
• Copy Paste: [[2410.16950]] Breaking ReAct Agents: Foot-in-the-Door Attack Will Get You In(https://arxiv.org/abs/2410.16950)
• Keywords: security, attack, large language model
• Abstract: Following the advancement of large language models (LLMs), the development of LLM-based autonomous agents has become increasingly prevalent. As a result, the need to understand the security vulnerabilities of these agents has become a critical task. We examine how ReAct agents can be exploited using a straightforward yet effective method we refer to as the foot-in-the-door attack. Our experiments show that indirect prompt injection attacks, prompted by harmless and unrelated requests (such as basic calculations) can significantly increase the likelihood of the agent performing subsequent malicious actions. Our results show that once a ReAct agents thought includes a specific tool or action, the likelihood of executing this tool in the subsequent steps increases significantly, as the agent seldom re-evaluates its actions. Consequently, even random, harmless requests can establish a foot-in-the-door, allowing an attacker to embed malicious instructions into the agents thought process, making it more susceptible to harmful directives. To mitigate this vulnerability, we propose implementing a simple reflection mechanism that prompts the agent to reassess the safety of its actions during execution, which can help reduce the success of such attacks.

Title: Towards Real Zero-Shot Camouflaged Object Segmentation without Camouflaged Annotations

• Authors: Cheng Lei, Jie Fan, Xinran Li, Tianzhu Xiang, Ao Li, Ce Zhu, Le Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16953
• Pdf URL: https://arxiv.org/pdf/2410.16953
• Copy Paste: [[2410.16953]] Towards Real Zero-Shot Camouflaged Object Segmentation without Camouflaged Annotations(https://arxiv.org/abs/2410.16953)
• Keywords: robust, large language model, segmentation
• Abstract: Camouflaged Object Segmentation (COS) faces significant challenges due to the scarcity of annotated data, where meticulous pixel-level annotation is both labor-intensive and costly, primarily due to the intricate object-background boundaries. Addressing the core question, "Can COS be effectively achieved in a zero-shot manner without manual annotations for any camouflaged object?" we affirmatively respond and introduce a robust zero-shot COS framework. This framework leverages the inherent local pattern bias of COS and employs a broad semantic feature space derived from salient object segmentation (SOS) for efficient zero-shot transfer. We incorporate an Masked Image Modeling (MIM) based image encoder optimized for Parameter-Efficient Fine-Tuning (PEFT), a Multimodal Large Language Model (M-LLM), and a Multi-scale Fine-grained Alignment (MFA) mechanism. The MIM pre-trained image encoder focuses on capturing essential low-level features, while the M-LLM generates caption embeddings processed alongside these visual cues. These embeddings are precisely aligned using MFA, enabling our framework to accurately interpret and navigate complex semantic contexts. To optimize operational efficiency, we introduce a learnable codebook that represents the M-LLM during inference, significantly reducing computational overhead. Our framework demonstrates its versatility and efficacy through rigorous experimentation, achieving state-of-the-art performance in zero-shot COS with $F_{\beta}^w$ scores of 72.9\% on CAMO and 71.7\% on COD10K. By removing the M-LLM during inference, we achieve an inference speed comparable to that of traditional end-to-end models, reaching 18.1 FPS. Code: this https URL

Title: PGCS: Physical Law embedded Generative Cloud Synthesis in Remote Sensing Images

• Authors: Liying Xu, Huifang Li, Huanfeng Shen, Mingyang Lei, Tao Jiang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.16955
• Pdf URL: https://arxiv.org/pdf/2410.16955
• Copy Paste: [[2410.16955]] PGCS: Physical Law embedded Generative Cloud Synthesis in Remote Sensing Images(https://arxiv.org/abs/2410.16955)
• Keywords: extraction, generative, segmentation
• Abstract: Data quantity and quality are both critical for information extraction and analyzation in remote sensing. However, the current remote sensing datasets often fail to meet these two requirements, for which cloud is a primary factor degrading the data quantity and quality. This limitation affects the precision of results in remote sensing application, particularly those derived from data-driven techniques. In this paper, a physical law embedded generative cloud synthesis method (PGCS) is proposed to generate diverse realistic cloud images to enhance real data and promote the development of algorithms for subsequent tasks, such as cloud correction, cloud detection, and data augmentation for classification, recognition, and segmentation. The PGCS method involves two key phases: spatial synthesis and spectral synthesis. In the spatial synthesis phase, a style-based generative adversarial network is utilized to simulate the spatial characteristics, generating an infinite number of single-channel clouds. In the spectral synthesis phase, the atmospheric scattering law is embedded through a local statistics and global fitting method, converting the single-channel clouds into multi-spectral clouds. The experimental results demonstrate that PGCS achieves a high accuracy in both phases and performs better than three other existing cloud synthesis methods. Two cloud correction methods are developed from PGCS and exhibits a superior performance compared to state-of-the-art methods in the cloud correction task. Furthermore, the application of PGCS with data from various sensors was investigated and successfully extended. Code will be provided at this https URL.

Title: Learning Mathematical Rules with Large Language Models

• Authors: Antoine Gorceix, Bastien Le Chenadec, Ahmad Rammal, Nelson Vadori, Manuela Veloso
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16973
• Pdf URL: https://arxiv.org/pdf/2410.16973
• Copy Paste: [[2410.16973]] Learning Mathematical Rules with Large Language Models(https://arxiv.org/abs/2410.16973)
• Keywords: large language model
• Abstract: In this paper, we study the ability of large language models to learn specific mathematical rules such as distributivity or simplifying equations. We present an empirical analysis of their ability to generalize these rules, as well as to reuse them in the context of word problems. For this purpose, we provide a rigorous methodology to build synthetic data incorporating such rules, and perform fine-tuning of large language models on such data. Our experiments show that our model can learn and generalize these rules to some extent, as well as suitably reuse them in the context of word problems.

Title: Publishing Neural Networks in Drug Discovery Might Compromise Training Data Privacy

• Authors: Fabian P. Krüger, Johan Östman, Lewis Mervin, Igor V. Tetko, Ola Engkvist
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.16975
• Pdf URL: https://arxiv.org/pdf/2410.16975
• Copy Paste: [[2410.16975]] Publishing Neural Networks in Drug Discovery Might Compromise Training Data Privacy(https://arxiv.org/abs/2410.16975)
• Keywords: privacy, attack, membership infer
• Abstract: This study investigates the risks of exposing confidential chemical structures when machine learning models trained on these structures are made publicly available. We use membership inference attacks, a common method to assess privacy that is largely unexplored in the context of drug discovery, to examine neural networks for molecular property prediction in a black-box setting. Our results reveal significant privacy risks across all evaluated datasets and neural network architectures. Combining multiple attacks increases these risks. Molecules from minority classes, often the most valuable in drug discovery, are particularly vulnerable. We also found that representing molecules as graphs and using message-passing neural networks may mitigate these risks. We provide a framework to assess privacy risks of classification models and molecular representations. Our findings highlight the need for careful consideration when sharing neural networks trained on proprietary chemical structures, informing organisations and researchers about the trade-offs between data confidentiality and model openness.

Title: IPL: Leveraging Multimodal Large Language Models for Intelligent Product Listing

• Authors: Kang Chen, Qingheng Zhang, Chengbao Lian, Yixin Ji, Xuwei Liu, Shuguang Han, Guoqiang Wu, Fei Huang, Jufeng Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.16977
• Pdf URL: https://arxiv.org/pdf/2410.16977
• Copy Paste: [[2410.16977]] IPL: Leveraging Multimodal Large Language Models for Intelligent Product Listing(https://arxiv.org/abs/2410.16977)
• Keywords: generative, large language model
• Abstract: Unlike professional Business-to-Consumer (B2C) e-commerce platforms (e.g., Amazon), Consumer-to-Consumer (C2C) platforms (e.g., Facebook marketplace) are mainly targeting individual sellers who usually lack sufficient experience in e-commerce. Individual sellers often struggle to compose proper descriptions for selling products. With the recent advancement of Multimodal Large Language Models (MLLMs), we attempt to integrate such state-of-the-art generative AI technologies into the product listing process. To this end, we develop IPL, an Intelligent Product Listing tool tailored to generate descriptions using various product attributes such as category, brand, color, condition, etc. IPL enables users to compose product descriptions by merely uploading photos of the selling product. More importantly, it can imitate the content style of our C2C platform Xianyu. This is achieved by employing domain-specific instruction tuning on MLLMs and adopting the multi-modal Retrieval-Augmented Generation (RAG) process. A comprehensive empirical evaluation demonstrates that the underlying model of IPL significantly outperforms the base model in domain-specific tasks while producing less hallucination. IPL has been successfully deployed in our production system, where 72% of users have their published product listings based on the generated content, and those product listings are shown to have a quality score 5.6% higher than those without AI assistance.

Title: E-3DGS: Gaussian Splatting with Exposure and Motion Events

• Authors: Xiaoting Yin, Hao Shi, Yuhan Bao, Zhenshan Bing, Yiyi Liao, Kailun Yang, Kaiwei Wang
• Subjects: cs.CV, cs.RO, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.16995
• Pdf URL: https://arxiv.org/pdf/2410.16995
• Copy Paste: [[2410.16995]] E-3DGS: Gaussian Splatting with Exposure and Motion Events(https://arxiv.org/abs/2410.16995)
• Keywords: robust
• Abstract: Estimating Neural Radiance Fields (NeRFs) from images captured under optimal conditions has been extensively explored in the vision community. However, robotic applications often face challenges such as motion blur, insufficient illumination, and high computational overhead, which adversely affect downstream tasks like navigation, inspection, and scene visualization. To address these challenges, we propose E-3DGS, a novel event-based approach that partitions events into motion (from camera or object movement) and exposure (from camera exposure), using the former to handle fast-motion scenes and using the latter to reconstruct grayscale images for high-quality training and optimization of event-based 3D Gaussian Splatting (3DGS). We introduce a novel integration of 3DGS with exposure events for high-quality reconstruction of explicit scene representations. Our versatile framework can operate on motion events alone for 3D reconstruction, enhance quality using exposure events, or adopt a hybrid mode that balances quality and effectiveness by optimizing with initial exposure events followed by high-speed motion events. We also introduce EME-3D, a real-world 3D dataset with exposure events, motion events, camera calibration parameters, and sparse point clouds. Our method is faster and delivers better reconstruction quality than event-based NeRF while being more cost-effective than NeRF methods that combine event and RGB data by using a single event sensor. By combining motion and exposure events, E-3DGS sets a new benchmark for event-based 3D reconstruction with robust performance in challenging conditions and lower hardware demands. The source code and dataset will be available at this https URL.

Title: AGSENet: A Robust Road Ponding Detection Method for Proactive Traffic Safety

• Authors: Ronghui Zhang, Shangyu Yang, Dakang Lyu, Zihan Wang, Junzhou Chen, Yilong Ren, Bolin Gao, Zhihan Lv
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.16999
• Pdf URL: https://arxiv.org/pdf/2410.16999
• Copy Paste: [[2410.16999]] AGSENet: A Robust Road Ponding Detection Method for Proactive Traffic Safety(https://arxiv.org/abs/2410.16999)
• Keywords: robust
• Abstract: Road ponding, a prevalent traffic hazard, poses a serious threat to road safety by causing vehicles to lose control and leading to accidents ranging from minor fender benders to severe collisions. Existing technologies struggle to accurately identify road ponding due to complex road textures and variable ponding coloration influenced by reflection characteristics. To address this challenge, we propose a novel approach called Self-Attention-based Global Saliency-Enhanced Network (AGSENet) for proactive road ponding detection and traffic safety improvement. AGSENet incorporates saliency detection techniques through the Channel Saliency Information Focus (CSIF) and Spatial Saliency Information Enhancement (SSIE) modules. The CSIF module, integrated into the encoder, employs self-attention to highlight similar features by fusing spatial and channel information. The SSIE module, embedded in the decoder, refines edge features and reduces noise by leveraging correlations across different feature levels. To ensure accurate and reliable evaluation, we corrected significant mislabeling and missing annotations in the Puddle-1000 dataset. Additionally, we constructed the Foggy-Puddle and Night-Puddle datasets for road ponding detection in low-light and foggy conditions, respectively. Experimental results demonstrate that AGSENet outperforms existing methods, achieving IoU improvements of 2.03\%, 0.62\%, and 1.06\% on the Puddle-1000, Foggy-Puddle, and Night-Puddle datasets, respectively, setting a new state-of-the-art in this field. Finally, we verified the algorithm's reliability on edge computing devices. This work provides a valuable reference for proactive warning research in road traffic safety.

Title: Beyond Yao's Millionaires: Secure Multi-Party Computation of Non-Polynomial Functions

• Authors: Seyed Reza Hoseini Najarkolaei, Mohammad Mahdi Mojahedian, Mohammad Reza Aref
• Subjects: cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2410.17000
• Pdf URL: https://arxiv.org/pdf/2410.17000
• Copy Paste: [[2410.17000]] Beyond Yao's Millionaires: Secure Multi-Party Computation of Non-Polynomial Functions(https://arxiv.org/abs/2410.17000)
• Keywords: secure, federate
• Abstract: In this paper, we present an unconditionally secure $N$-party comparison scheme based on Shamir secret sharing, utilizing the binary representation of private inputs to determine the $\max$ without disclosing any private inputs or intermediate results. Specifically, each party holds a private number and aims to ascertain the greatest number among the $N$ available private numbers without revealing its input, assuming that there are at most $T < \frac{N}{2}$ honest-but-curious parties. The proposed scheme demonstrates a lower computational complexity compared to existing schemes that can only compare two secret numbers at a time. To the best of our knowledge, our scheme is the only information-theoretically secure method for comparing $N$ private numbers without revealing either the private inputs or any intermediate results. We demonstrate that by modifying the proposed scheme, we can compute other well-known non-polynomial functions of the inputs, including the minimum, median, and rank. Additionally, in the proposed scheme, before the final reveal phase, each party possesses a share of the result, enabling the nodes to compute any polynomial function of the comparison result. We also explore various applications of the proposed comparison scheme, including federated learning.

Title: SPVSoAP3D: A Second-order Average Pooling Approach to enhance 3D Place Recognition in Horticultural Environments

• Authors: T. Barros, C. Premebida, S. Aravecchia, C. Pradalier, U.J. Nunes
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17017
• Pdf URL: https://arxiv.org/pdf/2410.17017
• Copy Paste: [[2410.17017]] SPVSoAP3D: A Second-order Average Pooling Approach to enhance 3D Place Recognition in Horticultural Environments(https://arxiv.org/abs/2410.17017)
• Keywords: extraction, transformer
• Abstract: 3D LiDAR-based place recognition has been extensively researched in urban environments, yet it remains underexplored in agricultural settings. Unlike urban contexts, horticultural environments, characterized by their permeability to laser beams, result in sparse and overlapping LiDAR scans with suboptimal geometries. This phenomenon leads to intra- and inter-row descriptor ambiguity. In this work, we address this challenge by introducing SPVSoAP3D, a novel modeling approach that combines a voxel-based feature extraction network with an aggregation technique based on a second-order average pooling operator, complemented by a descriptor enhancement stage. Furthermore, we augment the existing HORTO-3DLM dataset by introducing two new sequences derived from horticultural environments. We evaluate the performance of SPVSoAP3D against state-of-the-art (SOTA) models, including OverlapTransformer, PointNetVLAD, and LOGG3D-Net, utilizing a cross-validation protocol on both the newly introduced sequences and the existing HORTO-3DLM dataset. The findings indicate that the average operator is more suitable for horticultural environments compared to the max operator and other first-order pooling techniques. Additionally, the results highlight the improvements brought by the descriptor enhancement stage.

Title: Exploring Forgetting in Large Language Model Pre-Training

• Authors: Chonghua Liao, Ruobing Xie, Xingwu Sun, Haowen Sun, Zhanhui Kang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17018
• Pdf URL: https://arxiv.org/pdf/2410.17018
• Copy Paste: [[2410.17018]] Exploring Forgetting in Large Language Model Pre-Training(https://arxiv.org/abs/2410.17018)
• Keywords: large language model
• Abstract: Catastrophic forgetting remains a formidable obstacle to building an omniscient model in large language models (LLMs). Despite the pioneering research on task-level forgetting in LLM fine-tuning, there is scant focus on forgetting during pre-training. We systematically explored the existence and measurement of forgetting in pre-training, questioning traditional metrics such as perplexity (PPL) and introducing new metrics to better detect entity memory retention. Based on our revised assessment of forgetting metrics, we explored low-cost, straightforward methods to mitigate forgetting during the pre-training phase. Further, we carefully analyzed the learning curves, offering insights into the dynamics of forgetting. Extensive evaluations and analyses on forgetting of pre-training could facilitate future research on LLMs.

Title: SG-FSM: A Self-Guiding Zero-Shot Prompting Paradigm for Multi-Hop Question Answering Based on Finite State Machine

• Authors: Xiaochen Wang, Junqing He, Liang Chen, Reza Haf Zhe Yang, Yiru Wang, Xiangdi Meng, Kunhao Pan, Zhifang Sui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17021
• Pdf URL: https://arxiv.org/pdf/2410.17021
• Copy Paste: [[2410.17021]] SG-FSM: A Self-Guiding Zero-Shot Prompting Paradigm for Multi-Hop Question Answering Based on Finite State Machine(https://arxiv.org/abs/2410.17021)
• Keywords: large language model
• Abstract: Large Language Models with chain-of-thought prompting, such as OpenAI-o1, have shown impressive capabilities in natural language inference tasks. However, Multi-hop Question Answering (MHQA) remains challenging for many existing models due to issues like hallucination, error propagation, and limited context length. To address these challenges and enhance LLMs' performance on MHQA, we propose the Self-Guiding prompting Finite State Machine (SG-FSM), designed to strengthen multi-hop reasoning abilities. Unlike traditional chain-of-thought methods, SG-FSM tackles MHQA by iteratively breaking down complex questions into sub-questions, correcting itself to improve accuracy. It processes one sub-question at a time, dynamically deciding the next step based on the current context and results, functioning much like an automaton. Experiments across various benchmarks demonstrate the effectiveness of our approach, outperforming strong baselines on challenging datasets such as Musique. SG-FSM reduces hallucination, enabling recovery of the correct final answer despite intermediate errors. It also improves adherence to specified output formats, simplifying evaluation significantly.

Title: DIRI: Adversarial Patient Reidentification with Large Language Models for Evaluating Clinical Text Anonymization

• Authors: John X. Morris, Thomas R. Campion, Sri Laasya Nutheti, Yifan Peng, Akhil Raj, Ramin Zabih, Curtis L. Cole
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17035
• Pdf URL: https://arxiv.org/pdf/2410.17035
• Copy Paste: [[2410.17035]] DIRI: Adversarial Patient Reidentification with Large Language Models for Evaluating Clinical Text Anonymization(https://arxiv.org/abs/2410.17035)
• Keywords: protect, large language model
• Abstract: Sharing protected health information (PHI) is critical for furthering biomedical research. Before data can be distributed, practitioners often perform deidentification to remove any PHI contained in the text. Contemporary deidentification methods are evaluated on highly saturated datasets (tools achieve near-perfect accuracy) which may not reflect the full variability or complexity of real-world clinical text and annotating them is resource intensive, which is a barrier to real-world applications. To address this gap, we developed an adversarial approach using a large language model (LLM) to re-identify the patient corresponding to a redacted clinical note and evaluated the performance with a novel De-Identification/Re-Identification (DIRI) method. Our method uses a large language model to reidentify the patient corresponding to a redacted clinical note. We demonstrate our method on medical data from Weill Cornell Medicine anonymized with three deidentification tools: rule-based Philter and two deep-learning-based models, BiLSTM-CRF and ClinicalBERT. Although ClinicalBERT was the most effective, masking all identified PII, our tool still reidentified 9% of clinical notes Our study highlights significant weaknesses in current deidentification technologies while providing a tool for iterative development and improvement.

Title: Arabic Dataset for LLM Safeguard Evaluation

• Authors: Yasser Ashraf, Yuxia Wang, Bin Gu, Preslav Nakov, Timothy Baldwin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17040
• Pdf URL: https://arxiv.org/pdf/2410.17040
• Copy Paste: [[2410.17040]] Arabic Dataset for LLM Safeguard Evaluation(https://arxiv.org/abs/2410.17040)
• Keywords: attack, large language model
• Abstract: The growing use of large language models (LLMs) has raised concerns regarding their safety. While many studies have focused on English, the safety of LLMs in Arabic, with its linguistic and cultural complexities, remains under-explored. Here, we aim to bridge this gap. In particular, we present an Arab-region-specific safety evaluation dataset consisting of 5,799 questions, including direct attacks, indirect attacks, and harmless requests with sensitive words, adapted to reflect the socio-cultural context of the Arab world. To uncover the impact of different stances in handling sensitive and controversial topics, we propose a dual-perspective evaluation framework. It assesses the LLM responses from both governmental and opposition viewpoints. Experiments over five leading Arabic-centric and multilingual LLMs reveal substantial disparities in their safety performance. This reinforces the need for culturally specific datasets to ensure the responsible deployment of LLMs.

Title: UnStar: Unlearning with Self-Taught Anti-Sample Reasoning for LLMs

• Authors: Yash Sinha, Murari Mandal, Mohan Kankanhalli
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17050
• Pdf URL: https://arxiv.org/pdf/2410.17050
• Copy Paste: [[2410.17050]] UnStar: Unlearning with Self-Taught Anti-Sample Reasoning for LLMs(https://arxiv.org/abs/2410.17050)
• Keywords: privacy, large language model
• Abstract: The key components of machine learning are data samples for training, model for learning patterns, and loss function for optimizing accuracy. Analogously, unlearning can potentially be achieved through anti-data samples (or anti-samples), unlearning method, and reversed loss function. While prior research has explored unlearning methods and reversed loss functions, the potential of anti-samples remains largely untapped. In this paper, we introduce UnSTAR: Unlearning with Self-Taught Anti-Sample Reasoning for large language models (LLMs). Our contributions are threefold; first, we propose a novel concept of anti-sample-induced unlearning; second, we generate anti-samples by leveraging misleading rationales, which help reverse learned associations and accelerate the unlearning process; and third, we enable fine-grained targeted unlearning, allowing for the selective removal of specific associations without impacting related knowledge - something not achievable by previous works. Results demonstrate that anti-samples offer an efficient, targeted unlearning strategy for LLMs, opening new avenues for privacy-preserving machine learning and model modification.

Title: On the Vulnerability of Text Sanitization

• Authors: Meng Tong, Kejiang Chen, Xiaojian Yuang, Jiayang Liu, Weiming Zhang, Nenghai Yu, Jie Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.17052
• Pdf URL: https://arxiv.org/pdf/2410.17052
• Copy Paste: [[2410.17052]] On the Vulnerability of Text Sanitization(https://arxiv.org/abs/2410.17052)
• Keywords: privacy, protect, attack
• Abstract: Text sanitization, which employs differential privacy to replace sensitive tokens with new ones, represents a significant technique for privacy protection. Typically, its performance in preserving privacy is evaluated by measuring the attack success rate (ASR) of reconstruction attacks, where attackers attempt to recover the original tokens from the sanitized ones. However, current reconstruction attacks on text sanitization are developed empirically, making it challenging to accurately assess the effectiveness of sanitization. In this paper, we aim to provide a more accurate evaluation of sanitization effectiveness. Inspired by the works of Palamidessi et al., we implement theoretically optimal reconstruction attacks targeting text sanitization. We derive their bounds on ASR as benchmarks for evaluating sanitization performance. For real-world applications, we propose two practical reconstruction attacks based on these theoretical findings. Our experimental results underscore the necessity of reassessing these overlooked risks. Notably, one of our attacks achieves a 46.4% improvement in ASR over the state-of-the-art baseline, with a privacy budget of epsilon=4.0 on the SST-2 dataset. Our code is available at: this https URL.

Title: Multi Kernel Estimation based Object Segmentation

• Authors: Haim Goldfisher, Asaf Yekutiel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17064
• Pdf URL: https://arxiv.org/pdf/2410.17064
• Copy Paste: [[2410.17064]] Multi Kernel Estimation based Object Segmentation(https://arxiv.org/abs/2410.17064)
• Keywords: segmentation
• Abstract: This paper presents a novel approach for multi-kernel estimation by enhancing the KernelGAN algorithm, which traditionally estimates a single kernel for the entire image. We introduce Multi-KernelGAN, which extends KernelGAN's capabilities by estimating two distinct kernels based on object segmentation masks. Our approach is validated through three distinct methods: texture-based patch Fast Fourier Transform (FFT) calculation, detail-based segmentation, and deep learning-based object segmentation using YOLOv8 and the Segment Anything Model (SAM). Among these methods, the combination of YOLO and SAM yields the best results for kernel estimation. Experimental results demonstrate that our multi-kernel estimation technique outperforms conventional single-kernel methods in super-resolution tasks.

Title: Neuronal Competition Groups with Supervised STDP for Spike-Based Classification

• Authors: Gaspard Goupy, Pierre Tirilly, Ioan Marius Bilasco
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17066
• Pdf URL: https://arxiv.org/pdf/2410.17066
• Copy Paste: [[2410.17066]] Neuronal Competition Groups with Supervised STDP for Spike-Based Classification(https://arxiv.org/abs/2410.17066)
• Keywords: extraction
• Abstract: Spike Timing-Dependent Plasticity (STDP) is a promising substitute to backpropagation for local training of Spiking Neural Networks (SNNs) on neuromorphic hardware. STDP allows SNNs to address classification tasks by combining unsupervised STDP for feature extraction and supervised STDP for classification. Unsupervised STDP is usually employed with Winner-Takes-All (WTA) competition to learn distinct patterns. However, WTA for supervised STDP classification faces unbalanced competition challenges. In this paper, we propose a method to effectively implement WTA competition in a spiking classification layer employing first-spike coding and supervised STDP training. We introduce the Neuronal Competition Group (NCG), an architecture that improves classification capabilities by promoting the learning of various patterns per class. An NCG is a group of neurons mapped to a specific class, implementing intra-class WTA and a novel competition regulation mechanism based on two-compartment thresholds. We incorporate our proposed architecture into spiking classification layers trained with state-of-the-art supervised STDP rules. On top of two different unsupervised feature extractors, we obtain significant accuracy improvements on image recognition datasets such as CIFAR-10 and CIFAR-100. We show that our competition regulation mechanism is crucial for ensuring balanced competition and improved class separation.

Title: Team Ryu's Submission to SIGMORPHON 2024 Shared Task on Subword Tokenization

• Authors: Zilong Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17094
• Pdf URL: https://arxiv.org/pdf/2410.17094
• Copy Paste: [[2410.17094]] Team Ryu's Submission to SIGMORPHON 2024 Shared Task on Subword Tokenization(https://arxiv.org/abs/2410.17094)
• Keywords: transformer, segmentation
• Abstract: This papers presents the submission of team Ryu to the canceled SIGMORPHON 2024 shared task on subword tokenization. My submission explores whether morphological segmentation methods can be used as a part of subword tokenizers. I adopt two approaches: the statistical segmentation method Morfessor and a transformer based sequence-to-sequence (seq2seq) segmentation model in tokenizers. The prediction results show that morphological segmentation could be as effective as commonly used subword tokenizers. Additionally, I investigate how a tokenizer's vocabulary influences the performance of language models. A tokenizer with a balanced token frequency distribution tends to work better. A balanced token vocabulary can be achieved by keeping frequent words as unique tokens.

Title: Inferentially-Private Private Information

• Authors: Shuaiqi Wang, Shuran Zheng, Zinan Lin, Giulia Fanti, Zhiwei Steven Wu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.17095
• Pdf URL: https://arxiv.org/pdf/2410.17095
• Copy Paste: [[2410.17095]] Inferentially-Private Private Information(https://arxiv.org/abs/2410.17095)
• Keywords: privacy
• Abstract: Information disclosure can compromise privacy when revealed information is correlated with private information. We consider the notion of inferential privacy, which measures privacy leakage by bounding the inferential power a Bayesian adversary can gain by observing a released signal. Our goal is to devise an inferentially-private private information structure that maximizes the informativeness of the released signal, following the Blackwell ordering principle, while adhering to inferential privacy constraints. To achieve this, we devise an efficient release mechanism that achieves the inferentially-private Blackwell optimal private information structure for the setting where the private information is binary. Additionally, we propose a programming approach to compute the optimal structure for general cases given the utility function. The design of our mechanisms builds on our geometric characterization of the Blackwell-optimal disclosure mechanisms under privacy constraints, which may be of independent interest.

Title: Masked Differential Privacy

• Authors: David Schneider, Sina Sajadmanesh, Vikash Sehwag, Saquib Sarfraz, Rainer Stiefelhagen, Lingjuan Lyu, Vivek Sharma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17098
• Pdf URL: https://arxiv.org/pdf/2410.17098
• Copy Paste: [[2410.17098]] Masked Differential Privacy(https://arxiv.org/abs/2410.17098)
• Keywords: privacy, protect
• Abstract: Privacy-preserving computer vision is an important emerging problem in machine learning and artificial intelligence. The prevalent methods tackling this problem use differential privacy or anonymization and obfuscation techniques to protect the privacy of individuals. In both cases, the utility of the trained model is sacrificed heavily in this process. In this work, we propose an effective approach called masked differential privacy (MaskDP), which allows for controlling sensitive regions where differential privacy is applied, in contrast to applying DP on the entire input. Our method operates selectively on the data and allows for defining non-sensitive spatio-temporal regions without DP application or combining differential privacy with other privacy techniques within data samples. Experiments on four challenging action recognition datasets demonstrate that our proposed techniques result in better utility-privacy trade-offs compared to standard differentially private training in the especially demanding $\epsilon<1$ regime.

Title: Human-LLM Hybrid Text Answer Aggregation for Crowd Annotations

• Authors: Jiyi Li
• Subjects: cs.CL, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17099
• Pdf URL: https://arxiv.org/pdf/2410.17099
• Copy Paste: [[2410.17099]] Human-LLM Hybrid Text Answer Aggregation for Crowd Annotations(https://arxiv.org/abs/2410.17099)
• Keywords: large language model
• Abstract: The quality is a crucial issue for crowd annotations. Answer aggregation is an important type of solution. The aggregated answers estimated from multiple crowd answers to the same instance are the eventually collected annotations, rather than the individual crowd answers themselves. Recently, the capability of Large Language Models (LLMs) on data annotation tasks has attracted interest from researchers. Most of the existing studies mainly focus on the average performance of individual crowd workers; several recent works studied the scenarios of aggregation on categorical labels and LLMs used as label creators. However, the scenario of aggregation on text answers and the role of LLMs as aggregators are not yet well-studied. In this paper, we investigate the capability of LLMs as aggregators in the scenario of close-ended crowd text answer aggregation. We propose a human-LLM hybrid text answer aggregation method with a Creator-Aggregator Multi-Stage (CAMS) crowdsourcing framework. We make the experiments based on public crowdsourcing datasets. The results show the effectiveness of our approach based on the collaboration of crowd workers and LLMs.

Title: Feature Homomorphism -- A Cryptographic Scheme For Data Verification Under Ciphertext-Only Conditions

• Authors: Huang Neng
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.17106
• Pdf URL: https://arxiv.org/pdf/2410.17106
• Copy Paste: [[2410.17106]] Feature Homomorphism -- A Cryptographic Scheme For Data Verification Under Ciphertext-Only Conditions(https://arxiv.org/abs/2410.17106)
• Keywords: privacy
• Abstract: Privacy computing involves the extensive exchange and processing of encrypted data. For the parties involved in these interactions, how to determine the consistency of exchanged data without accessing the original data, ensuring tamper resistance, non-repudiation, quality traceability, indexing, and retrieval during the use of encrypted data, which is a key topic of achieving "Data Availability versus Visibility". This paper proposes a new type of homomorphism: Feature Homomorphism, and based on this feature, introduces a cryptographic scheme for data verification under ciphertext-only conditions. The proposed scheme involves designing a group of algorithms that meet the requirements outlined in this paper, including encryption/decryption algorithms and Feature Homomorphic Algorithm. This group of algorithms not only allows for the encryption and decryption of data but also ensures that the plaintext and its corresponding ciphertext, encrypted using the specified encryption algorithm, satisfy the following property: the eigenvalue of the plaintext obtained using the Feature Homomorphic Algorithm is equal to the eigenvalue of the ciphertext obtained using the same algorithm. With this group of algorithms, it is possible to verify data consistency directly by comparing the eigenvalues of the plaintext and ciphertext without accessing the original data (i.e., under ciphertext-only conditions). This can be used for tamper resistance, non-repudiation, and quality traceability. Additionally, the eigenvalue can serve as a ciphertext index, enabling searchable encryption. This scheme completes a piece of the puzzle in homomorphic encryption. Keywords: Privacy Computing, Data Consistency, Searchable Encryption, Zero-Knowledge Proof, Feature Homomorphism

Title: Enhancing Answer Attribution for Faithful Text Generation with Large Language Models

• Authors: Juraj Vladika, Luca Mülln, Florian Matthes
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.17112
• Pdf URL: https://arxiv.org/pdf/2410.17112
• Copy Paste: [[2410.17112]] Enhancing Answer Attribution for Faithful Text Generation with Large Language Models(https://arxiv.org/abs/2410.17112)
• Keywords: large language model, segmentation
• Abstract: The increasing popularity of Large Language Models (LLMs) in recent years has changed the way users interact with and pose questions to AI-based conversational systems. An essential aspect for increasing the trustworthiness of generated LLM answers is the ability to trace the individual claims from responses back to relevant sources that support them, the process known as answer attribution. While recent work has started exploring the task of answer attribution in LLMs, some challenges still remain. In this work, we first perform a case study analyzing the effectiveness of existing answer attribution methods, with a focus on subtasks of answer segmentation and evidence retrieval. Based on the observed shortcomings, we propose new methods for producing more independent and contextualized claims for better retrieval and attribution. The new methods are evaluated and shown to improve the performance of answer attribution components. We end with a discussion and outline of future directions for the task.

Title: Security and RAS in the Computing Continuum

• Authors: Martí Alonso, David Andreu, Ramon Canal, Stefano Di Carlo, Odysseas Chatzopoulos, Cristiano Chenet, Juanjo Costa, Andreu Girones, Dimitris Gizopoulos, George Papadimitriou, Enric Morancho, Beatriz Otero, Alessandro Savino
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2410.17116
• Pdf URL: https://arxiv.org/pdf/2410.17116
• Copy Paste: [[2410.17116]] Security and RAS in the Computing Continuum(https://arxiv.org/abs/2410.17116)
• Keywords: security, attack, robust
• Abstract: Security and RAS are two non-functional requirements under focus for current systems developed for the computing continuum. Due to the increased number of interconnected computer systems across the continuum, security becomes especially pervasive at all levels, from the smallest edge device to the high-performance cloud at the other end. Similarly, RAS (Reliability, Availability, and Serviceability) ensures the robustness of a system towards hardware defects. Namely, making them reliable, with high availability and design for easy service. In this paper and as a result of the Vitamin-V EU project, the authors detail the comprehensive approach to malware and hardware attack detection; as well as, the RAS features envisioned for future systems across the computing continuum.

Title: Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards

• Authors: Alexander G. Padula, Dennis J.N.J. Soemers
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17126
• Pdf URL: https://arxiv.org/pdf/2410.17126
• Copy Paste: [[2410.17126]] Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards(https://arxiv.org/abs/2410.17126)
• Keywords: large language model
• Abstract: Proximal Policy Optimization (PPO) is commonly used in Reinforcement Learning from Human Feedback to align large language models (LLMs) with downstream tasks. This paper investigates the feasibility of using PPO for direct reinforcement learning (RL) from explicitly programmed reward signals, as opposed to indirect learning from human feedback via an intermediary reward model. We focus on tasks expressed through formal languages, such as mathematics and programming, where explicit reward functions can be programmed to automatically assess the quality of generated outputs. We apply this approach to a sentiment alignment task, a simple arithmetic task, and a more complex game synthesis task. The sentiment alignment task replicates prior research and serves to validate our experimental setup. Our results show that pure RL-based training for the two formal language tasks is challenging, with success being limited even for the simple arithmetic task. We propose a novel batch-entropy regularization term to aid exploration, although training is not yet entirely stable. Our findings suggest that direct RL training of LLMs may be more suitable for relatively minor changes, such as alignment, than for learning new tasks altogether, even if an informative reward signal can be expressed programmatically.

Title: PAPILLON: PrivAcy Preservation from Internet-based and Local Language MOdel ENsembles

• Authors: Li Siyan, Vethavikashini Chithrra Raghuram, Omar Khattab, Julia Hirschberg, Zhou Yu
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17127
• Pdf URL: https://arxiv.org/pdf/2410.17127
• Copy Paste: [[2410.17127]] PAPILLON: PrivAcy Preservation from Internet-based and Local Language MOdel ENsembles(https://arxiv.org/abs/2410.17127)
• Keywords: privacy
• Abstract: Users can divulge sensitive information to proprietary LLM providers, raising significant privacy concerns. While open-source models, hosted locally on the user's machine, alleviate some concerns, models that users can host locally are often less capable than proprietary frontier models. Toward preserving user privacy while retaining the best quality, we propose Privacy-Conscious Delegation, a novel task for chaining API-based and local models. We utilize recent public collections of user-LLM interactions to construct a natural benchmark called PUPA, which contains personally identifiable information (PII). To study potential approaches, we devise PAPILLON, a multi-stage LLM pipeline that uses prompt optimization to address a simpler version of our task. Our best pipeline maintains high response quality for 85.5% of user queries while restricting privacy leakage to only 7.5%. We still leave a large margin to the generation quality of proprietary LLMs for future work. Our data and code will be available at this https URL.

Title: Aligning Large Language Models via Self-Steering Optimization

• Authors: Hao Xiang, Bowen Yu, Hongyu Lin, Keming Lu, Yaojie Lu, Xianpei Han, Le Sun, Jingren Zhou, Junyang Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17131
• Pdf URL: https://arxiv.org/pdf/2410.17131
• Copy Paste: [[2410.17131]] Aligning Large Language Models via Self-Steering Optimization(https://arxiv.org/abs/2410.17131)
• Keywords: large language model
• Abstract: Automated alignment develops alignment systems with minimal human intervention. The key to automated alignment lies in providing learnable and accurate preference signals for preference learning without human annotation. In this paper, we introduce Self-Steering Optimization ($SSO$), an algorithm that autonomously generates high-quality preference signals based on predefined principles during iterative training, eliminating the need for manual annotation. $SSO$ maintains the accuracy of signals by ensuring a consistent gap between chosen and rejected responses while keeping them both on-policy to suit the current policy model's learning capacity. $SSO$ can benefit the online and offline training of the policy model, as well as enhance the training of reward models. We validate the effectiveness of $SSO$ with two foundation models, Qwen2 and Llama3.1, indicating that it provides accurate, on-policy preference signals throughout iterative training. Without any manual annotation or external models, $SSO$ leads to significant performance improvements across six subjective or objective benchmarks. Besides, the preference data generated by $SSO$ significantly enhanced the performance of the reward model on Rewardbench. Our work presents a scalable approach to preference optimization, paving the way for more efficient and effective automated alignment.

Title: AlphaChimp: Tracking and Behavior Recognition of Chimpanzees

• Authors: Xiaoxuan Ma, Yutang Lin, Yuan Xu, Stephan P. Kaufhold, Jack Terwilliger, Andres Meza, Yixin Zhu, Federico Rossano, Yizhou Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17136
• Pdf URL: https://arxiv.org/pdf/2410.17136
• Copy Paste: [[2410.17136]] AlphaChimp: Tracking and Behavior Recognition of Chimpanzees(https://arxiv.org/abs/2410.17136)
• Keywords: transformer
• Abstract: Understanding non-human primate behavior is crucial for improving animal welfare, modeling social behavior, and gaining insights into both distinctly human and shared behaviors. Despite recent advances in computer vision, automated analysis of primate behavior remains challenging due to the complexity of their social interactions and the lack of specialized algorithms. Existing methods often struggle with the nuanced behaviors and frequent occlusions characteristic of primate social dynamics. This study aims to develop an effective method for automated detection, tracking, and recognition of chimpanzee behaviors in video footage. Here we show that our proposed method, AlphaChimp, an end-to-end approach that simultaneously detects chimpanzee positions and estimates behavior categories from videos, significantly outperforms existing methods in behavior recognition. AlphaChimp achieves approximately 10% higher tracking accuracy and a 20% improvement in behavior recognition compared to state-of-the-art methods, particularly excelling in the recognition of social behaviors. This superior performance stems from AlphaChimp's innovative architecture, which integrates temporal feature fusion with a Transformer-based self-attention mechanism, enabling more effective capture and interpretation of complex social interactions among chimpanzees. Our approach bridges the gap between computer vision and primatology, enhancing technical capabilities and deepening our understanding of primate communication and sociality. We release our code and models and hope this will facilitate future research in animal social dynamics. This work contributes to ethology, cognitive science, and artificial intelligence, offering new perspectives on social intelligence.

Title: Towards Automated Penetration Testing: Introducing LLM Benchmark, Analysis, and Improvements

• Authors: Isamu Isozaki, Manil Shrestha, Rick Console, Edward Kim
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17141
• Pdf URL: https://arxiv.org/pdf/2410.17141
• Copy Paste: [[2410.17141]] Towards Automated Penetration Testing: Introducing LLM Benchmark, Analysis, and Improvements(https://arxiv.org/abs/2410.17141)
• Keywords: security, large language model
• Abstract: Hacking poses a significant threat to cybersecurity, inflicting billions of dollars in damages annually. To mitigate these risks, ethical hacking, or penetration testing, is employed to identify vulnerabilities in systems and networks. Recent advancements in large language models (LLMs) have shown potential across various domains, including cybersecurity. However, there is currently no comprehensive, open, end-to-end automated penetration testing benchmark to drive progress and evaluate the capabilities of these models in security contexts. This paper introduces a novel open benchmark for LLM-based automated penetration testing, addressing this critical gap. We first evaluate the performance of LLMs, including GPT-4o and Llama 3.1-405B, using the state-of-the-art PentestGPT tool. Our findings reveal that while Llama 3.1 demonstrates an edge over GPT-4o, both models currently fall short of performing fully automated, end-to-end penetration testing. Next, we advance the state-of-the-art and present ablation studies that provide insights into improving the PentestGPT tool. Our research illuminates the challenges LLMs face in each aspect of Pentesting, e.g. enumeration, exploitation, and privilege escalation. This work contributes to the growing body of knowledge on AI-assisted cybersecurity and lays the foundation for future research in automated penetration testing using large language models.

Title: Can General-Purpose Large Language Models Generalize to English-Thai Machine Translation ?

• Authors: Jirat Chiaranaipanich, Naiyarat Hanmatheekuna, Jitkapat Sawatphol, Krittamate Tiankanon, Jiramet Kinchagawat, Amrest Chinkamol, Parinthapat Pengpun, Piyalitt Ittichaiwong, Peerat Limkonchotiwat
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17145
• Pdf URL: https://arxiv.org/pdf/2410.17145
• Copy Paste: [[2410.17145]] Can General-Purpose Large Language Models Generalize to English-Thai Machine Translation ?(https://arxiv.org/abs/2410.17145)
• Keywords: large language model
• Abstract: Large language models (LLMs) perform well on common tasks but struggle with generalization in low-resource and low-computation settings. We examine this limitation by testing various LLMs and specialized translation models on English-Thai machine translation and code-switching datasets. Our findings reveal that under more strict computational constraints, such as 4-bit quantization, LLMs fail to translate effectively. In contrast, specialized models, with comparable or lower computational requirements, consistently outperform LLMs. This underscores the importance of specialized models for maintaining performance under resource constraints.

Title: Are Visual-Language Models Effective in Action Recognition? A Comparative Study

• Authors: Mahmoud Ali, Di Yang, François Brémond
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17149
• Pdf URL: https://arxiv.org/pdf/2410.17149
• Copy Paste: [[2410.17149]] Are Visual-Language Models Effective in Action Recognition? A Comparative Study(https://arxiv.org/abs/2410.17149)
• Keywords: segmentation
• Abstract: Current vision-language foundation models, such as CLIP, have recently shown significant improvement in performance across various downstream tasks. However, whether such foundation models significantly improve more complex fine-grained action recognition tasks is still an open question. To answer this question and better find out the future research direction on human behavior analysis in-the-wild, this paper provides a large-scale study and insight on current state-of-the-art vision foundation models by comparing their transfer ability onto zero-shot and frame-wise action recognition tasks. Extensive experiments are conducted on recent fine-grained, human-centric action recognition datasets (e.g., Toyota Smarthome, Penn Action, UAV-Human, TSU, Charades) including action classification and segmentation.

Title: LiNo: Advancing Recursive Residual Decomposition of Linear and Nonlinear Patterns for Robust Time Series Forecasting

• Authors: Guoqi Yu, Yaoming Li, Xiaoyu Guo, Dayu Wang, Zirui Liu, Shujun Wang, Tong Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17159
• Pdf URL: https://arxiv.org/pdf/2410.17159
• Copy Paste: [[2410.17159]] LiNo: Advancing Recursive Residual Decomposition of Linear and Nonlinear Patterns for Robust Time Series Forecasting(https://arxiv.org/abs/2410.17159)
• Keywords: robust, extraction, transformer
• Abstract: Forecasting models are pivotal in a data-driven world with vast volumes of time series data that appear as a compound of vast Linear and Nonlinear patterns. Recent deep time series forecasting models struggle to utilize seasonal and trend decomposition to separate the entangled components. Such a strategy only explicitly extracts simple linear patterns like trends, leaving the other linear modes and vast unexplored nonlinear patterns to the residual. Their flawed linear and nonlinear feature extraction models and shallow-level decomposition limit their adaptation to the diverse patterns present in real-world scenarios. Given this, we innovate Recursive Residual Decomposition by introducing explicit extraction of both linear and nonlinear patterns. This deeper-level decomposition framework, which is named LiNo, captures linear patterns using a Li block which can be a moving average kernel, and models nonlinear patterns using a No block which can be a Transformer encoder. The extraction of these two patterns is performed alternatively and recursively. To achieve the full potential of LiNo, we develop the current simple linear pattern extractor to a general learnable autoregressive model, and design a novel No block that can handle all essential nonlinear patterns. Remarkably, the proposed LiNo achieves state-of-the-art on thirteen real-world benchmarks under univariate and multivariate forecasting scenarios. Experiments show that current forecasting models can deliver more robust and precise results through this advanced Recursive Residual Decomposition. We hope this work could offer insight into designing more effective forecasting models. Code is available at this Repository: this https URL.

Title: Interchangeable Token Embeddings for Extendable Vocabulary and Alpha-Equivalence

• Authors: İlker Işık, Ramazan Gokberk Cinbis, Ebru Aydin Gol
• Subjects: cs.CL, cs.LG, cs.LO
• Abstract URL: https://arxiv.org/abs/2410.17161
• Pdf URL: https://arxiv.org/pdf/2410.17161
• Copy Paste: [[2410.17161]] Interchangeable Token Embeddings for Extendable Vocabulary and Alpha-Equivalence(https://arxiv.org/abs/2410.17161)
• Keywords: transformer
• Abstract: We propose a novel approach for learning interchangeable tokens in language models to obtain an extendable vocabulary that can generalize to new tokens. Our method is designed to address alpha-equivalence, the principle that renaming bound variables in a syntactic expression preserves semantics. This property arises in many formal languages such as temporal logics, in which all proposition symbols represent the same concept but are distinguishable from each other. To handle such tokens, we develop a dual-part embedding approach. The first part is shared across all interchangeable tokens, thereby enforcing that they represent the same core concept. The second part is randomly generated for each token, which enables distinguishability. We evaluate our method in a Transformer encoder-decoder model on two tasks: solving linear temporal logic formulae and copying with extendable vocabulary. Our method demonstrates promising generalization capabilities in addition to introducing a favorable inductive bias for alpha-equivalence.

Title: KANICE: Kolmogorov-Arnold Networks with Interactive Convolutional Elements

• Authors: Md Meftahul Ferdaus, Mahdi Abdelguerfi, Elias Ioup, David Dobson, Kendall N. Niles, Ken Pathak, Steven Sloan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.17172
• Pdf URL: https://arxiv.org/pdf/2410.17172
• Copy Paste: [[2410.17172]] KANICE: Kolmogorov-Arnold Networks with Interactive Convolutional Elements(https://arxiv.org/abs/2410.17172)
• Keywords: extraction
• Abstract: We introduce KANICE (Kolmogorov-Arnold Networks with Interactive Convolutional Elements), a novel neural architecture that combines Convolutional Neural Networks (CNNs) with Kolmogorov-Arnold Network (KAN) principles. KANICE integrates Interactive Convolutional Blocks (ICBs) and KAN linear layers into a CNN framework. This leverages KANs' universal approximation capabilities and ICBs' adaptive feature learning. KANICE captures complex, non-linear data relationships while enabling dynamic, context-dependent feature extraction based on the Kolmogorov-Arnold representation theorem. We evaluated KANICE on four datasets: MNIST, Fashion-MNIST, EMNIST, and SVHN, comparing it against standard CNNs, CNN-KAN hybrids, and ICB variants. KANICE consistently outperformed baseline models, achieving 99.35% accuracy on MNIST and 90.05% on the SVHN dataset. Furthermore, we introduce KANICE-mini, a compact variant designed for efficiency. A comprehensive ablation study demonstrates that KANICE-mini achieves comparable performance to KANICE with significantly fewer parameters. KANICE-mini reached 90.00% accuracy on SVHN with 2,337,828 parameters, compared to KANICE's 25,432,000. This study highlights the potential of KAN-based architectures in balancing performance and computational efficiency in image classification tasks. Our work contributes to research in adaptive neural networks, integrates mathematical theorems into deep learning architectures, and explores the trade-offs between model complexity and performance, advancing computer vision and pattern recognition. The source code for this paper is publicly accessible through our GitHub repository (this https URL).

Title: From Attention to Activation: Unravelling the Enigmas of Large Language Models

• Authors: Prannay Kaul, Chengcheng Ma, Ismail Elezi, Jiankang Deng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17174
• Pdf URL: https://arxiv.org/pdf/2410.17174
• Copy Paste: [[2410.17174]] From Attention to Activation: Unravelling the Enigmas of Large Language Models(https://arxiv.org/abs/2410.17174)
• Keywords: transformer, large language model
• Abstract: We study two strange phenomena in auto-regressive Transformers: (1) the dominance of the first token in attention heads; (2) the occurrence of large outlier activations in the hidden states. We find that popular large language models, such as Llama attend maximally to the first token in 98% of attention heads, a behaviour we attribute to the softmax function. To mitigate this issue, we propose a reformulation of softmax to softmax-1. Furthermore, we identify adaptive optimisers, e.g. Adam, as the primary contributor to the large outlier activations and introduce OrthoAdam, a novel optimiser that utilises orthogonal matrices to transform gradients, to address this issue. Finally, not only do our methods prevent these phenomena from occurring, but additionally, they enable Transformers to sustain their performance when quantised using basic algorithms, something that standard methods are unable to do. In summary, our methods reduce the attention proportion on the first token from 65% to 3.3%, the activation kurtosis in the hidden states from 1657 to 3.1, and perplexity penalty under 4-bit weight quantisation from 3565 to 0.3.

Title: Remote Timing Attacks on Efficient Language Model Inference

• Authors: Nicholas Carlini, Milad Nasr
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17175
• Pdf URL: https://arxiv.org/pdf/2410.17175
• Copy Paste: [[2410.17175]] Remote Timing Attacks on Efficient Language Model Inference(https://arxiv.org/abs/2410.17175)
• Keywords: defense, attack
• Abstract: Scaling up language models has significantly increased their capabilities. But larger models are slower models, and so there is now an extensive body of work (e.g., speculative sampling or parallel decoding) that improves the (average case) efficiency of language model generation. But these techniques introduce data-dependent timing characteristics. We show it is possible to exploit these timing differences to mount a timing attack. By monitoring the (encrypted) network traffic between a victim user and a remote language model, we can learn information about the content of messages by noting when responses are faster or slower. With complete black-box access, on open source systems we show how it is possible to learn the topic of a user's conversation (e.g., medical advice vs. coding assistance) with 90%+ precision, and on production systems like OpenAI's ChatGPT and Anthropic's Claude we can distinguish between specific messages or infer the user's language. We further show that an active adversary can leverage a boosting attack to recover PII placed in messages (e.g., phone numbers or credit card numbers) for open source systems. We conclude with potential defenses and directions for future work.

Title: Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing

• Authors: Kento Nishi, Maya Okawa, Rahul Ramesh, Mikail Khona, Ekdeep Singh Lubana, Hidenori Tanaka
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17194
• Pdf URL: https://arxiv.org/pdf/2410.17194
• Copy Paste: [[2410.17194]] Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing(https://arxiv.org/abs/2410.17194)
• Keywords: transformer
• Abstract: Knowledge Editing (KE) algorithms alter models' internal weights to perform targeted updates to incorrect, outdated, or otherwise unwanted factual associations. In order to better define the possibilities and limitations of these approaches, recent work has shown that applying KE can adversely affect models' factual recall accuracy and diminish their general reasoning abilities. While these studies give broad insights into the potential harms of KE algorithms, e.g., via performance evaluations on benchmarks, we argue little is understood as to why such destructive failures occur. Is it possible KE methods distort representations of concepts beyond the targeted fact, hence hampering abilities at broad? If so, what is the extent of this distortion? To take a step towards addressing such questions, we define a novel synthetic task wherein a Transformer is trained from scratch to internalize a ``structured'' knowledge graph. The structure enforces relationships between entities of the graph, such that editing a factual association has "trickling effects" on other entities in the graph (e.g., altering X's parent is Y to Z affects who X's siblings' parent is). Through evaluations of edited models and analysis of extracted representations, we show that KE inadvertently affects representations of entities beyond the targeted one, distorting relevant structures that allow a model to infer unseen knowledge about an entity. We call this phenomenon representation shattering and demonstrate that it results in degradation of factual recall and reasoning performance more broadly. To corroborate our findings in a more naturalistic setup, we perform preliminary experiments with a pretrained GPT-2-XL model and reproduce the representation shattering effect therein as well. Overall, our work yields a precise mechanistic hypothesis to explain why KE has adverse effects on model capabilities.

Title: VoiceBench: Benchmarking LLM-Based Voice Assistants

• Authors: Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, Haizhou Li
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2410.17196
• Pdf URL: https://arxiv.org/pdf/2410.17196
• Copy Paste: [[2410.17196]] VoiceBench: Benchmarking LLM-Based Voice Assistants(https://arxiv.org/abs/2410.17196)
• Keywords: large language model
• Abstract: Building on the success of large language models (LLMs), recent advancements such as GPT-4o have enabled real-time speech interactions through LLM-based voice assistants, offering a significantly improved user experience compared to traditional text-based interactions. However, the absence of benchmarks designed to evaluate these speech interaction capabilities has hindered progress of LLM-based voice assistants development. Current evaluations focus primarily on automatic speech recognition (ASR) or general knowledge evaluation with clean speeches, neglecting the more intricate, real-world scenarios that involve diverse speaker characteristics, environmental and content factors. To address this, we introduce VoiceBench, the first benchmark designed to provide a multi-faceted evaluation of LLM-based voice assistants. VoiceBench also includes both real and synthetic spoken instructions that incorporate the above three key real-world variations. Extensive experiments reveal the limitations of current LLM-based voice assistant models and offer valuable insights for future research and development in this field.

Title: Vulnerability anti-patterns in Solidity: Increasing smart contracts security by reducing false alarms

• Authors: Tommaso Oss, Carlos E. Budde
• Subjects: cs.CR, cs.CE, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.17204
• Pdf URL: https://arxiv.org/pdf/2410.17204
• Copy Paste: [[2410.17204]] Vulnerability anti-patterns in Solidity: Increasing smart contracts security by reducing false alarms(https://arxiv.org/abs/2410.17204)
• Keywords: security, attack
• Abstract: Turing completeness has made Ethereum smart contracts attractive to blockchain developers and attackers alike. To increase code security, many tools can now spot most known vulnerabilities$-$at the cost of production efficiency. Recent studies show false-positive ratios over 99% in state-of-the-art technologies: this makes them impractical for use in industry and have raised questions on the direction of academic research. In this work we show how integrating and extending current analyses is not only feasible, but also a next logical step in smart-contract security. We propose light-weight static checks on the morphology and dynamics of Solidity code, stemming from a developer-centric notion of vulnerability, that we use to verify the output of other tools, flag potential false alarms, and suggest verifications. Besides technical details we implemented an open-source prototype. For three top-10 vulnerabilities it flags 324 warnings of other tools as false-positives, in 60 verified de-duplicated smart contracts selected from the blockchain by the presence of true (and false) vulnerabilities. This amounts to a 92%- to 100%-reduction in the number of false-positives for these vulnerabilities.

Title: EPContrast: Effective Point-level Contrastive Learning for Large-scale Point Cloud Understanding

• Authors: Zhiyi Pan, Guoqing Liu, Wei Gao, Thomas H. Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.17207
• Pdf URL: https://arxiv.org/pdf/2410.17207
• Copy Paste: [[2410.17207]] EPContrast: Effective Point-level Contrastive Learning for Large-scale Point Cloud Understanding(https://arxiv.org/abs/2410.17207)
• Keywords: segmentation
• Abstract: The acquisition of inductive bias through point-level contrastive learning holds paramount significance in point cloud pre-training. However, the square growth in computational requirements with the scale of the point cloud poses a substantial impediment to the practical deployment and execution. To address this challenge, this paper proposes an Effective Point-level Contrastive Learning method for large-scale point cloud understanding dubbed \textbf{EPContrast}, which consists of AGContrast and ChannelContrast. In practice, AGContrast constructs positive and negative pairs based on asymmetric granularity embedding, while ChannelContrast imposes contrastive supervision between channel feature maps. EPContrast offers point-level contrastive loss while concurrently mitigating the computational resource burden. The efficacy of EPContrast is substantiated through comprehensive validation on S3DIS and ScanNetV2, encompassing tasks such as semantic segmentation, instance segmentation, and object detection. In addition, rich ablation experiments demonstrate remarkable bias induction capabilities under label-efficient and one-epoch training settings.

Title: Exploring Possibilities of AI-Powered Legal Assistance in Bangladesh through Large Language Modeling

• Authors: Azmine Toushik Wasi, Wahid Faisal, Mst Rafia Islam, Mahathir Mohammad Bappy
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.17210
• Pdf URL: https://arxiv.org/pdf/2410.17210
• Copy Paste: [[2410.17210]] Exploring Possibilities of AI-Powered Legal Assistance in Bangladesh through Large Language Modeling(https://arxiv.org/abs/2410.17210)
• Keywords: large language model
• Abstract: Purpose: Bangladesh's legal system struggles with major challenges like delays, complexity, high costs, and millions of unresolved cases, which deter many from pursuing legal action due to lack of knowledge or financial constraints. This research seeks to develop a specialized Large Language Model (LLM) to assist in the Bangladeshi legal system. Methods: We created UKIL-DB-EN, an English corpus of Bangladeshi legal documents, by collecting and scraping data on various legal acts. We fine-tuned the GPT-2 model on this dataset to develop GPT2-UKIL-EN, an LLM focused on providing legal assistance in English. Results: The model was rigorously evaluated using semantic assessments, including case studies supported by expert opinions. The evaluation provided promising results, demonstrating the potential for the model to assist in legal matters within Bangladesh. Conclusion: Our work represents the first structured effort toward building an AI-based legal assistant for Bangladesh. While the results are encouraging, further refinements are necessary to improve the model's accuracy, credibility, and safety. This is a significant step toward creating a legal AI capable of serving the needs of a population of 180 million.

Title: Hierarchical Upper Confidence Bounds for Constrained Online Learning

• Authors: Ali Baheri
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2410.17216
• Pdf URL: https://arxiv.org/pdf/2410.17216
• Copy Paste: [[2410.17216]] Hierarchical Upper Confidence Bounds for Constrained Online Learning(https://arxiv.org/abs/2410.17216)
• Keywords: robust
• Abstract: The multi-armed bandit (MAB) problem is a foundational framework in sequential decision-making under uncertainty, extensively studied for its applications in areas such as clinical trials, online advertising, and resource allocation. Traditional MAB formulations, however, do not adequately capture scenarios where decisions are structured hierarchically, involve multi-level constraints, or feature context-dependent action spaces. In this paper, we introduce the hierarchical constrained bandits (HCB) framework, which extends the contextual bandit problem to incorporate hierarchical decision structures and multi-level constraints. We propose the hierarchical constrained upper confidence bound (HC-UCB) algorithm, designed to address the complexities of the HCB problem by leveraging confidence bounds within a hierarchical setting. Our theoretical analysis establishes sublinear regret bounds for HC-UCB and provides high-probability guarantees for constraint satisfaction at all hierarchical levels. Furthermore, we derive a minimax lower bound on the regret for the HCB problem, demonstrating the near-optimality of our algorithm. The results are significant for real-world applications where decision-making processes are inherently hierarchical and constrained, offering a robust and efficient solution that balances exploration and exploitation across multiple levels of decision-making.

Title: Context-aware Prompt Tuning: Advancing In-Context Learning with Adversarial Methods

• Authors: Tsachi Blau, Moshe Kimhi, Yonatan Belinkov, Alexander Bronstein, Chaim Baskin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.17222
• Pdf URL: https://arxiv.org/pdf/2410.17222
• Copy Paste: [[2410.17222]] Context-aware Prompt Tuning: Advancing In-Context Learning with Adversarial Methods(https://arxiv.org/abs/2410.17222)
• Keywords: attack, large language model
• Abstract: Fine-tuning Large Language Models (LLMs) typically involves updating at least a few billions of parameters. A more parameter-efficient approach is Prompt Tuning (PT), which updates only a few learnable tokens, and differently, In-Context Learning (ICL) adapts the model to a new task by simply including examples in the input without any training. When applying optimization-based methods, such as fine-tuning and PT for few-shot learning, the model is specifically adapted to the small set of training examples, whereas ICL leaves the model unchanged. This distinction makes traditional learning methods more prone to overfitting; in contrast, ICL is less sensitive to the few-shot scenario. While ICL is not prone to overfitting, it does not fully extract the information that exists in the training examples. This work introduces Context-aware Prompt Tuning (CPT), a method inspired by ICL, PT, and adversarial attacks. We build on the ICL strategy of concatenating examples before the input, but we extend this by PT-like learning, refining the context embedding through iterative optimization to extract deeper insights from the training examples. We carefully modify specific context tokens, considering the unique structure of input and output formats. Inspired by adversarial attacks, we adjust the input based on the labels present in the context, focusing on minimizing, rather than maximizing, the loss. Moreover, we apply a projected gradient descent algorithm to keep token embeddings close to their original values, under the assumption that the user-provided data is inherently valuable. Our method has been shown to achieve superior accuracy across multiple classification tasks using various LLM models.

Title: Fine-Tuning Large Language Models to Appropriately Abstain with Semantic Entropy

• Authors: Benedict Aaron Tjandra, Muhammed Razzak, Jannik Kossen, Kunal Handa, Yarin Gal
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17234
• Pdf URL: https://arxiv.org/pdf/2410.17234
• Copy Paste: [[2410.17234]] Fine-Tuning Large Language Models to Appropriately Abstain with Semantic Entropy(https://arxiv.org/abs/2410.17234)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) are known to hallucinate, whereby they generate plausible but inaccurate text. This phenomenon poses significant risks in critical applications, such as medicine or law, necessitating robust hallucination mitigation strategies. While recent works have proposed fine-tuning methods to teach LLMs to abstain from answering questions beyond their knowledge or capabilities, these methods rely on the existence of ground-truth labels or are limited to short-form responses. To address these limitations, we propose fine-tuning using semantic entropy, an uncertainty measure derived from introspection into the model which does not require external labels. We demonstrate that our approach matches or outperforms models fine-tuned using prior work and achieves strong performance for both short and long-form generations on a range of datasets.

Title: Large Language Models Empowered Personalized Web Agents

• Authors: Hongru Cai, Yongqi Li, Wenjie Wang, Fengbin Zhu, Xiaoyu Shen, Wenjie Li, Tat-Seng Chua
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.17236
• Pdf URL: https://arxiv.org/pdf/2410.17236
• Copy Paste: [[2410.17236]] Large Language Models Empowered Personalized Web Agents(https://arxiv.org/abs/2410.17236)
• Keywords: large language model
• Abstract: Web agents have emerged as a promising direction to automate Web task completion based on user instructions, significantly enhancing user experience. Recently, Web agents have evolved from traditional agents to Large Language Models (LLMs)-based Web agents. Despite their success, existing LLM-based Web agents overlook the importance of personalized data (e.g., user profiles and historical Web behaviors) in assisting the understanding of users' personalized instructions and executing customized actions. To overcome the limitation, we first formulate the task of LLM-empowered personalized Web agents, which integrate personalized data and user instructions to personalize instruction comprehension and action execution. To address the absence of a comprehensive evaluation benchmark, we construct a Personalized Web Agent Benchmark (PersonalWAB), featuring user instructions, personalized user data, Web functions, and two evaluation paradigms across three personalized Web tasks. Moreover, we propose a Personalized User Memory-enhanced Alignment (PUMA) framework to adapt LLMs to the personalized Web agent task. PUMA utilizes a memory bank with a task-specific retrieval strategy to filter relevant historical Web behaviors. Based on the behaviors, PUMA then aligns LLMs for personalized action execution through fine-tuning and direct preference optimization. Extensive experiments validate the superiority of PUMA over existing Web agents on PersonalWAB.

Title: LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias

• Authors: Haian Jin, Hanwen Jiang, Hao Tan, Kai Zhang, Sai Bi, Tianyuan Zhang, Fujun Luan, Noah Snavely, Zexiang Xu
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.17242
• Pdf URL: https://arxiv.org/pdf/2410.17242
• Copy Paste: [[2410.17242]] LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias(https://arxiv.org/abs/2410.17242)
• Keywords: transformer
• Abstract: We propose the Large View Synthesis Model (LVSM), a novel transformer-based approach for scalable and generalizable novel view synthesis from sparse-view inputs. We introduce two architectures: (1) an encoder-decoder LVSM, which encodes input image tokens into a fixed number of 1D latent tokens, functioning as a fully learned scene representation, and decodes novel-view images from them; and (2) a decoder-only LVSM, which directly maps input images to novel-view outputs, completely eliminating intermediate scene representations. Both models bypass the 3D inductive biases used in previous methods -- from 3D representations (e.g., NeRF, 3DGS) to network designs (e.g., epipolar projections, plane sweeps) -- addressing novel view synthesis with a fully data-driven approach. While the encoder-decoder model offers faster inference due to its independent latent representation, the decoder-only LVSM achieves superior quality, scalability, and zero-shot generalization, outperforming previous state-of-the-art methods by 1.5 to 3.5 dB PSNR. Comprehensive evaluations across multiple datasets demonstrate that both LVSM variants achieve state-of-the-art novel view synthesis quality. Notably, our models surpass all previous methods even with reduced computational resources (1-2 GPUs). Please see our website for more details: this https URL .