2024-08-14

Title: Automated Schizophrenia Detection from Handwriting Samples via Transfer Learning Convolutional Neural Networks

• Authors: Rafael Castro, Ishaan Patel, Tarun Patanjali, Priya Iyer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06347
• Pdf URL: https://arxiv.org/pdf/2408.06347
• Copy Paste: [[2408.06347]] Automated Schizophrenia Detection from Handwriting Samples via Transfer Learning Convolutional Neural Networks(https://arxiv.org/abs/2408.06347)
• Keywords: secure, robust
• Abstract: Schizophrenia is a globally prevalent psychiatric disorder that severely impairs daily life. Schizophrenia is caused by dopamine imbalances in the fronto-striatal pathways of the brain, which influences fine motor control in the cerebellum. This leads to abnormalities in handwriting. The goal of this study was to develop an accurate, objective, and accessible computational method to be able to distinguish schizophrenic handwriting samples from non-schizophrenic handwriting samples. To achieve this, data from Crespo et al. (2019) was used, which contains images of handwriting samples from schizophrenic and non-schizophrenic patients. The data was preprocessed and augmented to produce a more robust model that can recognize different types of handwriting. The data was used to train several different convolutional neural networks, and the model with the base architecture of InceptionV3 performed the best, differentiating between the two types of image with a 92% accuracy rate. To make this model accessible, a secure website was developed for medical professionals to use for their patients. Such a result suggests that handwriting analysis through computational models holds promise as a non-invasive and objective method for clinicians to diagnose and monitor schizophrenia.

Title: Enhancing Ecological Monitoring with Multi-Objective Optimization: A Novel Dataset and Methodology for Segmentation Algorithms

• Authors: Sophia J. Abraham, Jin Huang, Brandon RichardWebster, Michael Milford, Jonathan D. Hauenstein, Walter Scheirer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06356
• Pdf URL: https://arxiv.org/pdf/2408.06356
• Copy Paste: [[2408.06356]] Enhancing Ecological Monitoring with Multi-Objective Optimization: A Novel Dataset and Methodology for Segmentation Algorithms(https://arxiv.org/abs/2408.06356)
• Keywords: robust, segmentation
• Abstract: We introduce a unique semantic segmentation dataset of 6,096 high-resolution aerial images capturing indigenous and invasive grass species in Bega Valley, New South Wales, Australia, designed to address the underrepresented domain of ecological data in the computer vision community. This dataset presents a challenging task due to the overlap and distribution of grass species, which is critical for advancing models in ecological and agronomical applications. Our study features a homotopy-based multi-objective fine-tuning approach that balances segmentation accuracy and contextual consistency, applicable to various models. By integrating DiceCELoss for pixel-wise classification and a smoothness loss for spatial coherence, this method evolves during training to enhance robustness against noisy data. Performance baselines are established through a case study on the Segment Anything Model (SAM), demonstrating its effectiveness. Our annotation methodology, emphasizing pen size, zoom control, and memory management, ensures high-quality dataset creation. The dataset and code will be made publicly available, aiming to drive research in computer vision, machine learning, and ecological studies, advancing environmental monitoring and sustainable development.

Title: Algorithm Research of ELMo Word Embedding and Deep Learning Multimodal Transformer in Image Description

• Authors: Xiaohan Cheng, Taiyuan Mei, Yun Zi, Qi Wang, Zijun Gao, Haowei Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06357
• Pdf URL: https://arxiv.org/pdf/2408.06357
• Copy Paste: [[2408.06357]] Algorithm Research of ELMo Word Embedding and Deep Learning Multimodal Transformer in Image Description(https://arxiv.org/abs/2408.06357)
• Keywords: extraction, transformer
• Abstract: Zero sample learning is an effective method for data deficiency. The existing embedded zero sample learning methods only use the known classes to construct the embedded space, so there is an overfitting of the known classes in the testing process. This project uses category semantic similarity measures to classify multiple tags. This enables it to incorporate unknown classes that have the same meaning as currently known classes into the vector space when it is built. At the same time, most of the existing zero sample learning algorithms directly use the depth features of medical images as input, and the feature extraction process does not consider semantic information. This project intends to take ELMo-MCT as the main task and obtain multiple visual features related to the original image through self-attention mechanism. In this paper, a large number of experiments are carried out on three zero-shot learning reference datasets, and the best harmonic average accuracy is obtained compared with the most advanced algorithms.

Title: FedRobo: Federated Learning Driven Autonomous Inter Robots Communication For Optimal Chemical Sprays

• Authors: Jannatul Ferdaus, Sameera Pisupati, Mahedi Hasan, Sathwick Paladugu
• Subjects: cs.LG, cs.CV, cs.DC, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.06382
• Pdf URL: https://arxiv.org/pdf/2408.06382
• Copy Paste: [[2408.06382]] FedRobo: Federated Learning Driven Autonomous Inter Robots Communication For Optimal Chemical Sprays(https://arxiv.org/abs/2408.06382)
• Keywords: secure, protect, robust, federate
• Abstract: Federated Learning enables robots to learn from each other's experiences without relying on centralized data collection. Each robot independently maintains a model of crop conditions and chemical spray effectiveness, which is periodically shared with other robots in the fleet. A communication protocol is designed to optimize chemical spray applications by facilitating the exchange of information about crop conditions, weather, and other critical factors. The federated learning algorithm leverages this shared data to continuously refine the chemical spray strategy, reducing waste and improving crop yields. This approach has the potential to revolutionize the agriculture industry by offering a scalable and efficient solution for crop protection. However, significant challenges remain, including the development of a secure and robust communication protocol, the design of a federated learning algorithm that effectively integrates data from multiple sources, and ensuring the safety and reliability of autonomous robots. The proposed cluster-based federated learning approach also effectively reduces the computational load on the global server and minimizes communication overhead among clients.

Title: Dilated Convolution with Learnable Spacings

• Authors: Ismail Khalfaoui-Hassani
• Subjects: cs.LG, cs.CV, cs.NE, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.06383
• Pdf URL: https://arxiv.org/pdf/2408.06383
• Copy Paste: [[2408.06383]] Dilated Convolution with Learnable Spacings(https://arxiv.org/abs/2408.06383)
• Keywords: segmentation
• Abstract: This thesis presents and evaluates the Dilated Convolution with Learnable Spacings (DCLS) method. Through various supervised learning experiments in the fields of computer vision, audio, and speech processing, the DCLS method proves to outperform both standard and advanced convolution techniques. The research is organized into several steps, starting with an analysis of the literature and existing convolution techniques that preceded the development of the DCLS method. We were particularly interested in the methods that are closely related to our own and that remain essential to capture the nuances and uniqueness of our approach. The cornerstone of our study is the introduction and application of the DCLS method to convolutional neural networks (CNNs), as well as to hybrid architectures that rely on both convolutional and visual attention approaches. DCLS is shown to be particularly effective in tasks such as classification, semantic segmentation, and object detection. Initially using bilinear interpolation, the study also explores other interpolation methods, finding that Gaussian interpolation slightly improves performance. The DCLS method is further applied to spiking neural networks (SNNs) to enable synaptic delay learning within a neural network that could eventually be transferred to so-called neuromorphic chips. The results show that the DCLS method stands out as a new state-of-the-art technique in SNN audio classification for certain benchmark tasks in this field. These tasks involve datasets with a high temporal component. In addition, we show that DCLS can significantly improve the accuracy of artificial neural networks for the multi-label audio classification task. We conclude with a discussion of the chosen experimental setup, its limitations, the limitations of our method, and our results.

Title: Evaluating Language Models on Entity Disambiguation in Tables

• Authors: Federico Belotti, Fabio Dadda, Marco Cremaschi, Roberto Avogadro, Riccardo Pozzi, Matteo Palmonari
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06423
• Pdf URL: https://arxiv.org/pdf/2408.06423
• Copy Paste: [[2408.06423]] Evaluating Language Models on Entity Disambiguation in Tables(https://arxiv.org/abs/2408.06423)
• Keywords: large language model
• Abstract: Tables are crucial containers of information, but understanding their meaning may be challenging. Indeed, recently, there has been a focus on Semantic Table Interpretation (STI), i.e., the task that involves the semantic annotation of tabular data to disambiguate their meaning. Over the years, there has been a surge in interest in data-driven approaches based on deep learning that have increasingly been combined with heuristic-based approaches. In the last period, the advent of Large Language Models (LLMs) has led to a new category of approaches for table annotation. The interest in this research field, characterised by multiple challenges, has led to a proliferation of approaches employing different techniques. However, these approaches have not been consistently evaluated on a common ground, making evaluation and comparison difficult. This work proposes an extensive evaluation of four state-of-the-art (SOTA) approaches - Alligator (formerly s-elBat), Dagobah, TURL, and TableLlama; the first two belong to the family of heuristic-based algorithms, while the others are respectively encoder-only and decoder-only LLMs. The primary objective is to measure the ability of these approaches to solve the entity disambiguation task, with the ultimate aim of charting new research paths in the field.

Title: Wavelet based inpainting detection

• Authors: Barglazan Adrian-Alin, Brad Remus Ovidiu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06429
• Pdf URL: https://arxiv.org/pdf/2408.06429
• Copy Paste: [[2408.06429]] Wavelet based inpainting detection(https://arxiv.org/abs/2408.06429)
• Keywords: robust, segmentation
• Abstract: With the advancement in image editing tools, manipulating digital images has become alarmingly easy. Inpainting, which is used to remove objects or fill in parts of an image, serves as a powerful tool for both image restoration and forgery. This paper introduces a novel approach for detecting image inpainting forgeries by combining DT-CWT with Hierarchical Feature segmentation and with noise inconsistency analysis. The DT-CWT offers several advantages for this task, including inherent shift-invariance, which makes it robust to minor manipulations during the inpainting process, and directional selectivity, which helps capture subtle artifacts introduced by inpainting in specific frequency bands and orientations. By first applying color image segmentation and then analyzing for each segment, noise inconsistency obtained via DT-CW we can identify patterns indicative of inpainting forgeries. The proposed method is evaluated on a benchmark dataset created for this purpose and is compared with existing forgery detection techniques. Our approach demonstrates superior results compared with SOTA in detecting inpainted images.

Title: HAT: History-Augmented Anchor Transformer for Online Temporal Action Localization

• Authors: Sakib Reza, Yuexi Zhang, Mohsen Moghaddam, Octavia Camps
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06437
• Pdf URL: https://arxiv.org/pdf/2408.06437
• Copy Paste: [[2408.06437]] HAT: History-Augmented Anchor Transformer for Online Temporal Action Localization(https://arxiv.org/abs/2408.06437)
• Keywords: transformer
• Abstract: Online video understanding often relies on individual frames, leading to frame-by-frame predictions. Recent advancements such as Online Temporal Action Localization (OnTAL), extend this approach to instance-level predictions. However, existing methods mainly focus on short-term context, neglecting historical information. To address this, we introduce the History-Augmented Anchor Transformer (HAT) Framework for OnTAL. By integrating historical context, our framework enhances the synergy between long-term and short-term information, improving the quality of anchor features crucial for classification and localization. We evaluate our model on both procedural egocentric (PREGO) datasets (EGTEA and EPIC) and standard non-PREGO OnTAL datasets (THUMOS and MUSES). Results show that our model outperforms state-of-the-art approaches significantly on PREGO datasets and achieves comparable or slightly superior performance on non-PREGO datasets, underscoring the importance of leveraging long-term history, especially in procedural and egocentric action scenarios. Code is available at: this https URL

Title: Multi-View Neural Differential Equations for Continuous-Time Stream Data in Long-Term Traffic Forecasting

• Authors: Zibo Liu, Zhe Jiang, Shigang Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06445
• Pdf URL: https://arxiv.org/pdf/2408.06445
• Copy Paste: [[2408.06445]] Multi-View Neural Differential Equations for Continuous-Time Stream Data in Long-Term Traffic Forecasting(https://arxiv.org/abs/2408.06445)
• Keywords: robust
• Abstract: Long-term traffic flow forecasting plays a crucial role in intelligent transportation as it allows traffic managers to adjust their decisions in advance. However, the problem is challenging due to spatio-temporal correlations and complex dynamic patterns in continuous-time stream data. Neural Differential Equations (NDEs) are among the state-of-the-art methods for learning continuous-time traffic dynamics. However, the traditional NDE models face issues in long-term traffic forecasting due to failures in capturing delayed traffic patterns, dynamic edge (location-to-location correlation) patterns, and abrupt trend patterns. To fill this gap, we propose a new NDE architecture called Multi-View Neural Differential Equations. Our model captures current states, delayed states, and trends in different state variables (views) by learning latent multiple representations within Neural Differential Equations. Extensive experiments conducted on several real-world traffic datasets demonstrate that our proposed method outperforms the state-of-the-art and achieves superior prediction accuracy for long-term forecasting and robustness with noisy or missing inputs.

Title: S-SAM: SVD-based Fine-Tuning of Segment Anything Model for Medical Image Segmentation

• Authors: Jay N. Paranjape, Shameema Sikder, S. Swaroop Vedula, Vishal M. Patel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06447
• Pdf URL: https://arxiv.org/pdf/2408.06447
• Copy Paste: [[2408.06447]] S-SAM: SVD-based Fine-Tuning of Segment Anything Model for Medical Image Segmentation(https://arxiv.org/abs/2408.06447)
• Keywords: segmentation
• Abstract: Medical image segmentation has been traditionally approached by training or fine-tuning the entire model to cater to any new modality or dataset. However, this approach often requires tuning a large number of parameters during training. With the introduction of the Segment Anything Model (SAM) for prompted segmentation of natural images, many efforts have been made towards adapting it efficiently for medical imaging, thus reducing the training time and resources. However, these methods still require expert annotations for every image in the form of point prompts or bounding box prompts during training and inference, making it tedious to employ them in practice. In this paper, we propose an adaptation technique, called S-SAM, that only trains parameters equal to 0.4% of SAM's parameters and at the same time uses simply the label names as prompts for producing precise masks. This not only makes tuning SAM more efficient than the existing adaptation methods but also removes the burden of providing expert prompts. We call this modified version S-SAM and evaluate it on five different modalities including endoscopic images, x-ray, ultrasound, CT, and histology images. Our experiments show that S-SAM outperforms state-of-the-art methods as well as existing SAM adaptation methods while tuning a significantly less number of parameters. We release the code for S-SAM at this https URL.

Title: Advanced Vision Transformers and Open-Set Learning for Robust Mosquito Classification: A Novel Approach to Entomological Studies

• Authors: Ahmed Akib Jawad Karim, Muhammad Zawad Mahmud, Riasat Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06457
• Pdf URL: https://arxiv.org/pdf/2408.06457
• Copy Paste: [[2408.06457]] Advanced Vision Transformers and Open-Set Learning for Robust Mosquito Classification: A Novel Approach to Entomological Studies(https://arxiv.org/abs/2408.06457)
• Keywords: robust, transformer
• Abstract: Mosquito-related diseases pose a significant threat to global public health, necessitating efficient and accurate mosquito classification for effective surveillance and control. This work presents an innovative approach to mosquito classification by leveraging state-of-the-art vision transformers and open-set learning techniques. A novel framework has been introduced that integrates Transformer-based deep learning models with comprehensive data augmentation and preprocessing methods, enabling robust and precise identification of ten mosquito species. The Swin Transformer model achieves the best performance for traditional closed-set learning with 99.80\% accuracy and 0.998 F1 score. The lightweight MobileViT technique attains an almost similar accuracy of 98.90\% with significantly reduced parameters and model complexities. Next, the applied deep learning models' adaptability and generalizability in a static environment have been enhanced by using new classes of data samples during the inference stage that have not been included in the training set. The proposed framework's ability to handle unseen classes like insects similar to mosquitoes, even humans, through open-set learning further enhances its practical applicability employing the OpenMax technique and Weibull distribution. The traditional CNN model, Xception, outperforms the latest transformer with higher accuracy and F1 score for open-set learning. The study's findings highlight the transformative potential of advanced deep-learning architectures in entomology, providing a strong groundwork for future research and development in mosquito surveillance and vector control. The implications of this work extend beyond mosquito classification, offering valuable insights for broader ecological and environmental monitoring applications.

Title: Evaluating Privacy Measures for Load Hiding

• Authors: Vadim Arzamasov, Klemens Böhm
• Subjects: cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2408.06460
• Pdf URL: https://arxiv.org/pdf/2408.06460
• Copy Paste: [[2408.06460]] Evaluating Privacy Measures for Load Hiding(https://arxiv.org/abs/2408.06460)
• Keywords: privacy
• Abstract: In smart grids, the use of smart meters to measure electricity consumption at a household level raises privacy concerns. To address them, researchers have designed various load hiding algorithms that manipulate the electricity consumption measured. To compare how well these algorithms preserve privacy, various privacy measures have been proposed. However, there currently is no consensus on which privacy measure is most appropriate to use. In this study, we aim to identify the most effective privacy measure(s) for load hiding algorithms. We have crafted a series of experiments to assess the effectiveness of these measures. found 20 of the 25 measures studied to be ineffective. Next, focused on the well-known "appliance usage" secret, we have designed synthetic data to find the measure that best deals with this secret. We observe that such a measure, a variant of mutual information, actually exists.

Title: Statistical Quality Comparison of the Bitstrings Generated by a Physical Unclonable Function across Xilinx, Altera and Microsemi Devices

• Authors: Jenilee Jao, Kristi Hoffman, Cheryl Reid, Ryan Thomson, Michael Thompson, Jim Plusquellic
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.06463
• Pdf URL: https://arxiv.org/pdf/2408.06463
• Copy Paste: [[2408.06463]] Statistical Quality Comparison of the Bitstrings Generated by a Physical Unclonable Function across Xilinx, Altera and Microsemi Devices(https://arxiv.org/abs/2408.06463)
• Keywords: security
• Abstract: Entropy or randomness represents a foundational security property in security-related operations, such as key generation. Key generation in turn is central to security protocols such as authentication and encryption. Physical unclonable functions (PUF) are hardware-based primitives that can serve as key generation engines in modern microelectronic devices and applications. PUFs derive entropy from manufacturing variations that exist naturally within and across otherwise identical copies of a device. However, the levels of random variations that represent entropy, which are strongly correlated to the quality of the PUF-generated bitstrings, vary from one manufacturer to another. In this paper, we evaluate entropy across a set of devices manufactured by three mainstream FPGA vendors, Xilinx, Altera and Microsemi. The devices selected for evaluation are considered low-end commercial devices to make the analysis relevant to IoT applications. The SiRF PUF is used in the evaluation, and is constructed nearly identically across the three vendor devices, setting aside minor differences that exist in certain logic element primitives used within the PUF architecture, and which have only a minor impact on our comparative analysis. The SiRF PUF uses a high-resolution time-to-digital converter (TDC) crafted from high-speed carry-chain logic embedded within each device to measure path delays in an engineered netlist of logic gates as a source of entropy. Therefore, our analysis includes an evaluation of actual path delay variation as it exists across the three device classes, as well as a statistical evaluation of the PUF-generated bitstrings. A reliablity analysis is also provided using data collected in industrial-standard temperature experiments to round out the evaluation of important statistical properties of the PUF.

Title: Kernel Sum of Squares for Data Adapted Kernel Learning of Dynamical Systems from Data: A global optimization approach

• Authors: Daniel Lengyel, Panos Parpas, Boumediene Hamzi, Houman Owhadi
• Subjects: cs.LG, math.DS, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.06465
• Pdf URL: https://arxiv.org/pdf/2408.06465
• Copy Paste: [[2408.06465]] Kernel Sum of Squares for Data Adapted Kernel Learning of Dynamical Systems from Data: A global optimization approach(https://arxiv.org/abs/2408.06465)
• Keywords: robust
• Abstract: This paper examines the application of the Kernel Sum of Squares (KSOS) method for enhancing kernel learning from data, particularly in the context of dynamical systems. Traditional kernel-based methods, despite their theoretical soundness and numerical efficiency, frequently struggle with selecting optimal base kernels and parameter tuning, especially with gradient-based methods prone to local optima. KSOS mitigates these issues by leveraging a global optimization framework with kernel-based surrogate functions, thereby achieving more reliable and precise learning of dynamical systems. Through comprehensive numerical experiments on the Logistic Map, Henon Map, and Lorentz System, KSOS is shown to consistently outperform gradient descent in minimizing the relative-$\rho$ metric and improving kernel accuracy. These results highlight KSOS's effectiveness in predicting the behavior of chaotic dynamical systems, demonstrating its capability to adapt kernels to underlying dynamics and enhance the robustness and predictive power of kernel-based approaches, making it a valuable asset for time series analysis in various scientific fields.

Title: Theorem-Carrying-Transaction: Runtime Certification to Ensure Safety for Smart Contract Transactions

• Authors: Nikolaj S. Bjørner (1), Ashley J. Chen (2), Shuo Chen (1), Yang Chen (1), Zhongxin Guo (1), Tzu-Han Hsu (3), Peng Liu (4), Nanqing Luo (4) ((1) Microsoft Research, (2) New York University Shanghai, (3) Michigan State University, (4) Pennsylvania State University)
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2408.06478
• Pdf URL: https://arxiv.org/pdf/2408.06478
• Copy Paste: [[2408.06478]] Theorem-Carrying-Transaction: Runtime Certification to Ensure Safety for Smart Contract Transactions(https://arxiv.org/abs/2408.06478)
• Keywords: secure, security
• Abstract: Security bugs and trapdoors in smart contracts have been impacting the Ethereum community since its inception. Conceptually, the 1.45-million Ethereum's contracts form a single "gigantic program" whose behaviors are determined by the complex reference-topology between the contracts. Can the Ethereum community be assured that this gigantic program conforms to its design-level safety properties, despite unforeseeable code-level intricacies? Static code verification is inadequate due to the program's gigantic scale and high polymorphism. In this paper, we present a viable technological roadmap for the community toward this ambitious goal. Our technology, called Theorem-Carrying-Transaction (TCT), combines the benefits of concrete execution and symbolic proofs. Under the TCT protocol, every transaction carries a theorem that proves its adherence to the specified properties in the invoked contracts, and the runtime system checks the theorem before executing the transaction. Once a property is specified in a contract, it can be treated confidently as an unconditional guarantee made by the contract. As case studies, we demonstrate that TCT secures token contracts without foreseeing code-level intricacies like integer overflow and reentrancy. TCT is also successfully applied to a Uniswap codebase, showcasing a complex decentralized finance (DeFi) scenario. Our prototype incurs a negligible runtime overhead, two orders of magnitude lower than a state-of-the-art approach.

Title: Cross-Lingual Conversational Speech Summarization with Large Language Models

• Authors: Max Nelson, Shannon Wotherspoon, Francis Keith, William Hartmann, Matthew Snover
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06484
• Pdf URL: https://arxiv.org/pdf/2408.06484
• Copy Paste: [[2408.06484]] Cross-Lingual Conversational Speech Summarization with Large Language Models(https://arxiv.org/abs/2408.06484)
• Keywords: large language model
• Abstract: Cross-lingual conversational speech summarization is an important problem, but suffers from a dearth of resources. While transcriptions exist for a number of languages, translated conversational speech is rare and datasets containing summaries are non-existent. We build upon the existing Fisher and Callhome Spanish-English Speech Translation corpus by supplementing the translations with summaries. The summaries are generated using GPT-4 from the reference translations and are treated as ground truth. The task is to generate similar summaries in the presence of transcription and translation errors. We build a baseline cascade-based system using open-source speech recognition and machine translation models. We test a range of LLMs for summarization and analyze the impact of transcription and translation errors. Adapting the Mistral-7B model for this task performs significantly better than off-the-shelf models and matches the performance of GPT-4.

Title: Benchmarking tree species classification from proximally-sensed laser scanning data: introducing the FOR-species20K dataset

• Authors: Stefano Puliti, Emily R. Lines, Jana Müllerová, Julian Frey, Zoe Schindler, Adrian Straker, Matthew J. Allen, Lukas Winiwarter, Nataliia Rehush, Hristina Hristova, Brent Murray, Kim Calders, Louise Terryn, Nicholas Coops, Bernhard Höfle, Samuli Junttila, Martin Krůček, Grzegorz Krok, Kamil Král, Shaun R. Levick, Linda Luck, Azim Missarov, Martin Mokroš, Harry J. F. Owen, Krzysztof Stereńczak, Timo P. Pitkänen, Nicola Puletti, Ninni Saarinen, Chris Hopkinson, Chiara Torresan, Enrico Tomelleri, Hannah Weiser, Rasmus Astrup
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06507
• Pdf URL: https://arxiv.org/pdf/2408.06507
• Copy Paste: [[2408.06507]] Benchmarking tree species classification from proximally-sensed laser scanning data: introducing the FOR-species20K dataset(https://arxiv.org/abs/2408.06507)
• Keywords: robust
• Abstract: Proximally-sensed laser scanning offers significant potential for automated forest data capture, but challenges remain in automatically identifying tree species without additional ground data. Deep learning (DL) shows promise for automation, yet progress is slowed by the lack of large, diverse, openly available labeled datasets of single tree point clouds. This has impacted the robustness of DL models and the ability to establish best practices for species classification. To overcome these challenges, the FOR-species20K benchmark dataset was created, comprising over 20,000 tree point clouds from 33 species, captured using terrestrial (TLS), mobile (MLS), and drone laser scanning (ULS) across various European forests, with some data from other regions. This dataset enables the benchmarking of DL models for tree species classification, including both point cloud-based (PointNet++, MinkNet, MLP-Mixer, DGCNNs) and multi-view image-based methods (SimpleView, DetailView, YOLOv5). 2D image-based models generally performed better (average OA = 0.77) than 3D point cloud-based models (average OA = 0.72), with consistent results across different scanning platforms and sensors. The top model, DetailView, was particularly robust, handling data imbalances well and generalizing effectively across tree sizes. The FOR-species20K dataset, available at this https URL, is a key resource for developing and benchmarking DL models for tree species classification using laser scanning data, providing a foundation for future advancements in the field.

Title: Fooling SHAP with Output Shuffling Attacks

• Authors: Jun Yuan, Aritra Dasgupta
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.06509
• Pdf URL: https://arxiv.org/pdf/2408.06509
• Copy Paste: [[2408.06509]] Fooling SHAP with Output Shuffling Attacks(https://arxiv.org/abs/2408.06509)
• Keywords: protect, attack, fair
• Abstract: Explainable AI~(XAI) methods such as SHAP can help discover feature attributions in black-box models. If the method reveals a significant attribution from a ``protected feature'' (e.g., gender, race) on the model output, the model is considered unfair. However, adversarial attacks can subvert the detection of XAI methods. Previous approaches to constructing such an adversarial model require access to underlying data distribution, which may not be possible in many practical scenarios. We relax this constraint and propose a novel family of attacks, called shuffling attacks, that are data-agnostic. The proposed attack strategies can adapt any trained machine learning model to fool Shapley value-based explanations. We prove that Shapley values cannot detect shuffling attacks. However, algorithms that estimate Shapley values, such as linear SHAP and SHAP, can detect these attacks with varying degrees of effectiveness. We demonstrate the efficacy of the attack strategies by comparing the performance of linear SHAP and SHAP using real-world datasets.

Title: Chain-of-Strategy Planning with LLMs: Aligning the Generation of Psychotherapy Dialogue with Strategy in Motivational Interviewing

• Authors: Xin Sun, Xiao Tang, Abdallah El Ali, Zhuying Li, Xiaoyu Shen, Pengjie Ren, Jan de Wit, Jiahuan Pei, Jos A.Bosch
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06527
• Pdf URL: https://arxiv.org/pdf/2408.06527
• Copy Paste: [[2408.06527]] Chain-of-Strategy Planning with LLMs: Aligning the Generation of Psychotherapy Dialogue with Strategy in Motivational Interviewing(https://arxiv.org/abs/2408.06527)
• Keywords: explainability, large language model
• Abstract: Recent advancements in large language models (LLMs) have shown promise in generating psychotherapeutic dialogues, especially in Motivational Interviewing (MI). However, how to employ strategies, a set of motivational interviewing (MI) skills, to generate therapeutic-adherent conversations with explainability is underexplored. We propose an approach called strategy-aware dialogue generation with Chain-of-Strategy (CoS) planning, which first predicts MI strategies as reasoning and utilizes these strategies to guide the subsequent dialogue generation. It brings the potential for controllable and explainable generation in psychotherapy by aligning the generated MI dialogues with therapeutic strategies. Extensive experiments including automatic and human evaluations are conducted to validate the effectiveness of the MI strategy. Our findings demonstrate the potential of LLMs in producing strategically aligned dialogues and suggest directions for practical applications in psychotherapeutic settings.

Title: HDRGS: High Dynamic Range Gaussian Splatting

• Authors: Jiahao Wu, Lu Xiao, Chao Wang, Rui Peng, Kaiqiang Xiong, Ronggang Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06543
• Pdf URL: https://arxiv.org/pdf/2408.06543
• Copy Paste: [[2408.06543]] HDRGS: High Dynamic Range Gaussian Splatting(https://arxiv.org/abs/2408.06543)
• Keywords: robust
• Abstract: Recent years have witnessed substantial advancements in the field of 3D reconstruction from 2D images, particularly following the introduction of the neural radiance field (NeRF) technique. However, reconstructing a 3D high dynamic range (HDR) radiance field, which aligns more closely with real-world conditions, from 2D multi-exposure low dynamic range (LDR) images continues to pose significant challenges. Approaches to this issue fall into two categories: grid-based and implicit-based. Implicit methods, using multi-layer perceptrons (MLP), face inefficiencies, limited solvability, and overfitting risks. Conversely, grid-based methods require significant memory and struggle with image quality and long training times. In this paper, we introduce Gaussian Splatting-a recent, high-quality, real-time 3D reconstruction technique-into this domain. We further develop the High Dynamic Range Gaussian Splatting (HDR-GS) method, designed to address the aforementioned challenges. This method enhances color dimensionality by including luminance and uses an asymmetric grid for tone-mapping, swiftly and precisely converting pixel irradiance to color. Our approach improves HDR scene recovery accuracy and integrates a novel coarse-to-fine strategy to speed up model convergence, enhancing robustness against sparse viewpoints and exposure extremes, and preventing local optima. Extensive testing confirms that our method surpasses current state-of-the-art techniques in both synthetic and real-world scenarios. Code will be released at \url{this https URL}

Title: Prioritizing Modalities: Flexible Importance Scheduling in Federated Multimodal Learning

• Authors: Jieming Bian, Lei Wang, Jie Xu
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.06549
• Pdf URL: https://arxiv.org/pdf/2408.06549
• Copy Paste: [[2408.06549]] Prioritizing Modalities: Flexible Importance Scheduling in Federated Multimodal Learning(https://arxiv.org/abs/2408.06549)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) is a distributed machine learning approach that enables devices to collaboratively train models without sharing their local data, ensuring user privacy and scalability. However, applying FL to real-world data presents challenges, particularly as most existing FL research focuses on unimodal data. Multimodal Federated Learning (MFL) has emerged to address these challenges, leveraging modality-specific encoder models to process diverse datasets. Current MFL methods often uniformly allocate computational frequencies across all modalities, which is inefficient for IoT devices with limited resources. In this paper, we propose FlexMod, a novel approach to enhance computational efficiency in MFL by adaptively allocating training resources for each modality encoder based on their importance and training requirements. We employ prototype learning to assess the quality of modality encoders, use Shapley values to quantify the importance of each modality, and adopt the Deep Deterministic Policy Gradient (DDPG) method from deep reinforcement learning to optimize the allocation of training resources. Our method prioritizes critical modalities, optimizing model performance and resource utilization. Experimental results on three real-world datasets demonstrate that our proposed method significantly improves the performance of MFL models.

Title: AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies

• Authors: Bo-Wen Zhang, Liangdong Wang, Ye Yuan, Jijie Li, Shuhao Gu, Mengdi Zhao, Xinya Wu, Guang Liu, Chengwei Wu, Hanyu Zhao, Li Du, Yiming Ju, Quanyue Ma, Yulong Ao, Yingli Zhao, Songhe Zhu, Zhou Cao, Dong Liang, Yonghua Lin, Ming Zhang, Shunfei Wang, Yanxin Zhou, Min Ye, Xuekai Chen, Xinyang Yu, Xiangjun Huang, Jian Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06567
• Pdf URL: https://arxiv.org/pdf/2408.06567
• Copy Paste: [[2408.06567]] AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies(https://arxiv.org/abs/2408.06567)
• Keywords: large language model
• Abstract: In recent years, with the rapid application of large language models across various fields, the scale of these models has gradually increased, and the resources required for their pre-training have grown exponentially. Training an LLM from scratch will cost a lot of computation resources while scaling up from a smaller model is a more efficient approach and has thus attracted significant attention. In this paper, we present AquilaMoE, a cutting-edge bilingual 8*16B Mixture of Experts (MoE) language model that has 8 experts with 16 billion parameters each and is developed using an innovative training methodology called EfficientScale. This approach optimizes performance while minimizing data requirements through a two-stage process. The first stage, termed Scale-Up, initializes the larger model with weights from a pre-trained smaller model, enabling substantial knowledge transfer and continuous pretraining with significantly less data. The second stage, Scale-Out, uses a pre-trained dense model to initialize the MoE experts, further enhancing knowledge transfer and performance. Extensive validation experiments on 1.8B and 7B models compared various initialization schemes, achieving models that maintain and reduce loss during continuous pretraining. Utilizing the optimal scheme, we successfully trained a 16B model and subsequently the 8*16B AquilaMoE model, demonstrating significant improvements in performance and training efficiency.

Title: Social Debiasing for Fair Multi-modal LLMs

• Authors: Harry Cheng, Yangyang Guo, Qingpei Guo, Ming Yang, Tian Gan, Liqiang Nie
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06569
• Pdf URL: https://arxiv.org/pdf/2408.06569
• Copy Paste: [[2408.06569]] Social Debiasing for Fair Multi-modal LLMs(https://arxiv.org/abs/2408.06569)
• Keywords: fair, large language model
• Abstract: Multi-modal Large Language Models (MLLMs) have advanced significantly, offering powerful vision-language understanding capabilities. However, these models often inherit severe social biases from their training datasets, leading to unfair predictions based on attributes like race and gender. This paper addresses the issue of social biases in MLLMs by i) Introducing a comprehensive Counterfactual dataset with Multiple Social Concepts (CMSC), which provides a more diverse and extensive training set compared to existing datasets. ii) Proposing an Anti-Stereotype Debiasing strategy (ASD). Our method works by revisiting the MLLM training process, rescaling the autoregressive loss function, and improving data sampling methods to counteract biases. Through extensive experiments on various MLLMs, our CMSC dataset and ASD method demonstrate a significant reduction in social biases while maintaining the models' original performance.

Title: SparkRA: A Retrieval-Augmented Knowledge Service System Based on Spark Large Language Model

• Authors: Dayong Wu, Jiaqi Li, Baoxin Wang, Honghong Zhao, Siyuan Xue, Yanjie Yang, Zhijun Chang, Rui Zhang, Li Qian, Bo Wang, Shijin Wang, Zhixiong Zhang, Guoping Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06574
• Pdf URL: https://arxiv.org/pdf/2408.06574
• Copy Paste: [[2408.06574]] SparkRA: A Retrieval-Augmented Knowledge Service System Based on Spark Large Language Model(https://arxiv.org/abs/2408.06574)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown remarkable achievements across various language this http URL enhance the performance of LLMs in scientific literature services, we developed the scientific literature LLM (SciLit-LLM) through pre-training and supervised fine-tuning on scientific literature, building upon the iFLYTEK Spark LLM. Furthermore, we present a knowledge service system Spark Research Assistant (SparkRA) based on our SciLit-LLM. SparkRA is accessible online and provides three primary functions: literature investigation, paper reading, and academic writing. As of July 30, 2024, SparkRA has garnered over 50,000 registered users, with a total usage count exceeding 1.3 million.

Title: CTISum: A New Benchmark Dataset For Cyber Threat Intelligence Summarization

• Authors: Wei Peng, Junmei Ding, Wei Wang, Lei Cui, Wei Cai, Zhiyu Hao, Xiaochun Yun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06576
• Pdf URL: https://arxiv.org/pdf/2408.06576
• Copy Paste: [[2408.06576]] CTISum: A New Benchmark Dataset For Cyber Threat Intelligence Summarization(https://arxiv.org/abs/2408.06576)
• Keywords: security, attack
• Abstract: Cyber Threat Intelligence (CTI) summarization task requires the system to generate concise and accurate highlights from raw intelligence data, which plays an important role in providing decision-makers with crucial information to quickly detect and respond to cyber threats in the cybersecurity domain. However, efficient techniques for summarizing CTI reports, including facts, analytical insights, attack processes, etc., have largely been unexplored, primarily due to the lack of available dataset. To this end, we present CTISum, a new benchmark for CTI summarization task. Considering the importance of attack process, a novel fine-grained subtask of attack process summarization is proposed to enable defenders to assess risk, identify security gaps, vulnerabilities, and so on. Specifically, we first design a multi-stage annotation pipeline to gather and annotate the CTI data, and then benchmark the CTISum with a collection of extractive and abstractive summarization methods. Experimental results show that current state-of-the-art models exhibit limitations when applied to CTISum, underscoring the fact that automatically producing concise summaries of CTI reports remains an open research challenge.

Title: Biomedical Event Extraction via Structure-aware Generation

• Authors: Haohan Yuan, Siu Cheung Hui, Haopeng Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06583
• Pdf URL: https://arxiv.org/pdf/2408.06583
• Copy Paste: [[2408.06583]] Biomedical Event Extraction via Structure-aware Generation(https://arxiv.org/abs/2408.06583)
• Keywords: extraction, generative, large language model
• Abstract: Biomedical Event Extraction (BEE) is a critical task that involves modeling complex relationships between fine-grained entities in biomedical text data. However, most existing BEE models rely on classification methods that neglect the label semantics and argument dependency structure within the data. To address these limitations, we propose GenBEE, a generative model enhanced with a structure-aware prefix for biomedical event extraction. GenBEE constructs event prompts that leverage knowledge distilled from large language models (LLMs), thereby incorporating both label semantics and argument dependency relationships. Additionally, GenBEE introduces a structural prefix learning module that generates structure-aware prefixes with structural prompts, enriching the generation process with structural features. Extensive experiments on three benchmark datasets demonstrate the effectiveness of GenBEE and it achieves state-of-the-art performance on the MLEE and GE11 datasets. Furthermore, our analysis shows that the structural prefixes effectively bridge the gap between structural prompts and the representation space of generative models, enabling better integration of event structural information.

Title: GeoFormer: Learning Point Cloud Completion with Tri-Plane Integrated Transformer

• Authors: Jinpeng Yu, Binbin Huang, Yuxuan Zhang, Huaxia Li, Xu Tang, Shenghua Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06596
• Pdf URL: https://arxiv.org/pdf/2408.06596
• Copy Paste: [[2408.06596]] GeoFormer: Learning Point Cloud Completion with Tri-Plane Integrated Transformer(https://arxiv.org/abs/2408.06596)
• Keywords: transformer
• Abstract: Point cloud completion aims to recover accurate global geometry and preserve fine-grained local details from partial point clouds. Conventional methods typically predict unseen points directly from 3D point cloud coordinates or use self-projected multi-view depth maps to ease this task. However, these gray-scale depth maps cannot reach multi-view consistency, consequently restricting the performance. In this paper, we introduce a GeoFormer that simultaneously enhances the global geometric structure of the points and improves the local details. Specifically, we design a CCM Feature Enhanced Point Generator to integrate image features from multi-view consistent canonical coordinate maps (CCMs) and align them with pure point features, thereby enhancing the global geometry feature. Additionally, we employ the Multi-scale Geometry-aware Upsampler module to progressively enhance local details. This is achieved through cross attention between the multi-scale features extracted from the partial input and the features derived from previously estimated points. Extensive experiments on the PCN, ShapeNet-55/34, and KITTI benchmarks demonstrate that our GeoFormer outperforms recent methods, achieving the state-of-the-art performance. Our code is available at \href{this https URL}{this https URL}.

Title: A Perspective on Large Language Models, Intelligent Machines, and Knowledge Acquisition

• Authors: Vladimir Cherkassky, Eng Hock Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06598
• Pdf URL: https://arxiv.org/pdf/2408.06598
• Copy Paste: [[2408.06598]] A Perspective on Large Language Models, Intelligent Machines, and Knowledge Acquisition(https://arxiv.org/abs/2408.06598)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are known for their remarkable ability to generate synthesized 'knowledge', such as text documents, music, images, etc. However, there is a huge gap between LLM's and human capabilities for understanding abstract concepts and reasoning. We discuss these issues in a larger philosophical context of human knowledge acquisition and the Turing test. In addition, we illustrate the limitations of LLMs by analyzing GPT-4 responses to questions ranging from science and math to common sense reasoning. These examples show that GPT-4 can often imitate human reasoning, even though it lacks understanding. However, LLM responses are synthesized from a large LLM model trained on all available data. In contrast, human understanding is based on a small number of abstract concepts. Based on this distinction, we discuss the impact of LLMs on acquisition of human knowledge and education.

Title: MV-DETR: Multi-modality indoor object detection by Multi-View DEtecton TRansformers

• Authors: Zichao Dong, Yilin Zhang, Xufeng Huang, Hang Ji, Zhan Shi, Xin Zhan, Junbo Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06604
• Pdf URL: https://arxiv.org/pdf/2408.06604
• Copy Paste: [[2408.06604]] MV-DETR: Multi-modality indoor object detection by Multi-View DEtecton TRansformers(https://arxiv.org/abs/2408.06604)
• Keywords: extraction, transformer
• Abstract: We introduce a novel MV-DETR pipeline which is effective while efficient transformer based detection method. Given input RGBD data, we notice that there are super strong pretraining weights for RGB data while less effective works for depth related data. First and foremost , we argue that geometry and texture cues are both of vital importance while could be encoded separately. Secondly, we find that visual texture feature is relatively hard to extract compared with geometry feature in 3d space. Unfortunately, single RGBD dataset with thousands of data is not enough for training an discriminating filter for visual texture feature extraction. Last but certainly not the least, we designed a lightweight VG module consists of a visual textual encoder, a geometry encoder and a VG connector. Compared with previous state of the art works like V-DETR, gains from pretrained visual encoder could be seen. Extensive experiments on ScanNetV2 dataset shows the effectiveness of our method. It is worth mentioned that our method achieve 78\% AP which create new state of the art on ScanNetv2 benchmark.

Title: CROME: Cross-Modal Adapters for Efficient Multimodal LLM

• Authors: Sayna Ebrahimi, Sercan O. Arik, Tejas Nama, Tomas Pfister
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.06610
• Pdf URL: https://arxiv.org/pdf/2408.06610
• Copy Paste: [[2408.06610]] CROME: Cross-Modal Adapters for Efficient Multimodal LLM(https://arxiv.org/abs/2408.06610)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) demonstrate remarkable image-language capabilities, but their widespread use faces challenges in cost-effective training and adaptation. Existing approaches often necessitate expensive language model retraining and limited adaptability. Additionally, the current focus on zero-shot performance improvements offers insufficient guidance for task-specific tuning. We propose CROME, an efficient vision-language instruction tuning framework. It features a novel gated cross-modal adapter that effectively combines visual and textual representations prior to input into a frozen LLM. This lightweight adapter, trained with minimal parameters, enables efficient cross-modal understanding. Notably, CROME demonstrates superior zero-shot performance on standard visual question answering and instruction-following benchmarks. Moreover, it yields fine-tuning with exceptional parameter efficiency, competing with task-specific specialist state-of-the-art methods. CROME demonstrates the potential of pre-LM alignment for building scalable, adaptable, and parameter-efficient multimodal models.

Title: ViMo: Generating Motions from Casual Videos

• Authors: Liangdong Qiu, Chengxing Yu, Yanran Li, Zhao Wang, Haibin Huang, Chongyang Ma, Di Zhang, Pengfei Wan, Xiaoguang Han
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.06614
• Pdf URL: https://arxiv.org/pdf/2408.06614
• Copy Paste: [[2408.06614]] ViMo: Generating Motions from Casual Videos(https://arxiv.org/abs/2408.06614)
• Keywords: diffusion
• Abstract: Although humans have the innate ability to imagine multiple possible actions from videos, it remains an extraordinary challenge for computers due to the intricate camera movements and montages. Most existing motion generation methods predominantly rely on manually collected motion datasets, usually tediously sourced from motion capture (Mocap) systems or Multi-View cameras, unavoidably resulting in a limited size that severely undermines their generalizability. Inspired by recent advance of diffusion models, we probe a simple and effective way to capture motions from videos and propose a novel Video-to-Motion-Generation framework (ViMo) which could leverage the immense trove of untapped video content to produce abundant and diverse 3D human motions. Distinct from prior work, our videos could be more causal, including complicated camera movements and occlusions. Striking experimental results demonstrate the proposed model could generate natural motions even for videos where rapid movements, varying perspectives, or frequent occlusions might exist. We also show this work could enable three important downstream applications, such as generating dancing motions according to arbitrary music and source video style. Extensive experimental results prove that our model offers an effective and scalable way to generate diversity and realistic motions. Code and demos will be public soon.

Title: Generalized knowledge-enhanced framework for biomedical entity and relation extraction

• Authors: Minh Nguyen, Phuong Le
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.06618
• Pdf URL: https://arxiv.org/pdf/2408.06618
• Copy Paste: [[2408.06618]] Generalized knowledge-enhanced framework for biomedical entity and relation extraction(https://arxiv.org/abs/2408.06618)
• Keywords: extraction
• Abstract: In recent years, there has been an increasing number of frameworks developed for biomedical entity and relation extraction. This research effort aims to address the accelerating growth in biomedical publications and the intricate nature of biomedical texts, which are written for mainly domain experts. To handle these challenges, we develop a novel framework that utilizes external knowledge to construct a task-independent and reusable background knowledge graph for biomedical entity and relation extraction. The design of our model is inspired by how humans learn domain-specific topics. In particular, humans often first acquire the most basic and common knowledge regarding a field to build the foundational knowledge and then use that as a basis for extending to various specialized topics. Our framework employs such common-knowledge-sharing mechanism to build a general neural-network knowledge graph that is learning transferable to different domain-specific biomedical texts effectively. Experimental evaluations demonstrate that our model, equipped with this generalized and cross-transferable knowledge base, achieves competitive performance benchmarks, including BioRelEx for binding interaction detection and ADE for Adverse Drug Effect identification.

Title: Unveiling the Flaws: A Critical Analysis of Initialization Effect on Time Series Anomaly Detection

• Authors: Alex Koran, Hadi Hojjati, Narges Armanfard
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.06620
• Pdf URL: https://arxiv.org/pdf/2408.06620
• Copy Paste: [[2408.06620]] Unveiling the Flaws: A Critical Analysis of Initialization Effect on Time Series Anomaly Detection(https://arxiv.org/abs/2408.06620)
• Keywords: robust, fair
• Abstract: Deep learning for time-series anomaly detection (TSAD) has gained significant attention over the past decade. Despite the reported improvements in several papers, the practical application of these models remains limited. Recent studies have cast doubt on these models, attributing their results to flawed evaluation techniques. However, the impact of initialization has largely been overlooked. This paper provides a critical analysis of the initialization effects on TSAD model performance. Our extensive experiments reveal that TSAD models are highly sensitive to hyperparameters such as window size, seed number, and normalization. This sensitivity often leads to significant variability in performance, which can be exploited to artificially inflate the reported efficacy of these models. We demonstrate that even minor changes in initialization parameters can result in performance variations that overshadow the claimed improvements from novel model architectures. Our findings highlight the need for rigorous evaluation protocols and transparent reporting of preprocessing steps to ensure the reliability and fairness of anomaly detection methods. This paper calls for a more cautious interpretation of TSAD advancements and encourages the development of more robust and transparent evaluation practices to advance the field and its practical applications.

Title: Towards Robust and Cost-Efficient Knowledge Unlearning for Large Language Models

• Authors: Sungmin Cha, Sungjun Cho, Dasol Hwang, Moontae Lee
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06621
• Pdf URL: https://arxiv.org/pdf/2408.06621
• Copy Paste: [[2408.06621]] Towards Robust and Cost-Efficient Knowledge Unlearning for Large Language Models(https://arxiv.org/abs/2408.06621)
• Keywords: privacy, robust, generative, large language model
• Abstract: Large Language Models (LLMs) have demonstrated strong reasoning and memorization capabilities via pretraining on massive textual corpora. However, training LLMs on human-written text entails significant risk of privacy and copyright violations, which demands an efficient machine unlearning framework to remove knowledge of sensitive data without retraining the model from scratch. While Gradient Ascent (GA) is widely used for unlearning by reducing the likelihood of generating unwanted information, the unboundedness of increasing the cross-entropy loss causes not only unstable optimization, but also catastrophic forgetting of knowledge that needs to be retained. We also discover its joint application under low-rank adaptation results in significantly suboptimal computational cost vs. generative performance trade-offs. In light of this limitation, we propose two novel techniques for robust and cost-efficient unlearning on LLMs. We first design an Inverted Hinge loss that suppresses unwanted tokens by increasing the probability of the next most likely token, thereby retaining fluency and structure in language generation. We also propose to initialize low-rank adapter weights based on Fisher-weighted low-rank approximation, which induces faster unlearning and better knowledge retention by allowing model updates to be focused on parameters that are important in generating textual data we wish to remove.

Title: DePatch: Towards Robust Adversarial Patch for Evading Person Detectors in the Real World

• Authors: Jikang Cheng, Ying Zhang, Zhongyuan Wang, Zou Qin, Chen Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06625
• Pdf URL: https://arxiv.org/pdf/2408.06625
• Copy Paste: [[2408.06625]] DePatch: Towards Robust Adversarial Patch for Evading Person Detectors in the Real World(https://arxiv.org/abs/2408.06625)
• Keywords: attack, robust
• Abstract: Recent years have seen an increasing interest in physical adversarial attacks, which aim to craft deployable patterns for deceiving deep neural networks, especially for person detectors. However, the adversarial patterns of existing patch-based attacks heavily suffer from the self-coupling issue, where a degradation, caused by physical transformations, in any small patch segment can result in a complete adversarial dysfunction, leading to poor robustness in the complex real world. Upon this observation, we introduce the Decoupled adversarial Patch (DePatch) attack to address the self-coupling issue of adversarial patches. Specifically, we divide the adversarial patch into block-wise segments, and reduce the inter-dependency among these segments through randomly erasing out some segments during the optimization. We further introduce a border shifting operation and a progressive decoupling strategy to improve the overall attack capabilities. Extensive experiments demonstrate the superior performance of our method over other physical adversarial attacks, especially in the real world.

Title: IFShip: A Large Vision-Language Model for Interpretable Fine-grained Ship Classification via Domain Knowledge-Enhanced Instruction Tuning

• Authors: Mingning Guo, Mengwei Wu, Yuxiang Shen, Haifeng Li, Chao Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06631
• Pdf URL: https://arxiv.org/pdf/2408.06631
• Copy Paste: [[2408.06631]] IFShip: A Large Vision-Language Model for Interpretable Fine-grained Ship Classification via Domain Knowledge-Enhanced Instruction Tuning(https://arxiv.org/abs/2408.06631)
• Keywords: interpretability
• Abstract: End-to-end interpretation is currently the prevailing paradigm for remote sensing fine-grained ship classification (RS-FGSC) task. However, its inference process is uninterpretable, leading to criticism as a black box model. To address this issue, we propose a large vision-language model (LVLM) named IFShip for interpretable fine-grained ship classification. Unlike traditional methods, IFShip excels in interpretability by accurately conveying the reasoning process of FGSC in natural language. Specifically, we first design a domain knowledge-enhanced Chain-of-Thought (COT) prompt generation mechanism. This mechanism is used to semi-automatically construct a task-specific instruction-following dataset named TITANIC-FGS, which emulates human-like logical decision-making. We then train the IFShip model using task instructions tuned with the TITANIC-FGS dataset. Building on IFShip, we develop an FGSC visual chatbot that redefines the FGSC problem as a step-by-step reasoning task and conveys the reasoning process in natural language. Experimental results reveal that the proposed method surpasses state-of-the-art FGSC algorithms in both classification interpretability and accuracy. Moreover, compared to LVLMs like LLaVA and MiniGPT-4, our approach demonstrates superior expertise in the FGSC task. It provides an accurate chain of reasoning when fine-grained ship types are recognizable to the human eye and offers interpretable explanations when they are not.

Title: IDRetracor: Towards Visual Forensics Against Malicious Face Swapping

• Authors: Jikang Cheng, Jiaxin Ai, Zhen Han, Chao Liang, Qin Zou, Zhongyuan Wang, Qian Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06635
• Pdf URL: https://arxiv.org/pdf/2408.06635
• Copy Paste: [[2408.06635]] IDRetracor: Towards Visual Forensics Against Malicious Face Swapping(https://arxiv.org/abs/2408.06635)
• Keywords: security
• Abstract: The face swapping technique based on deepfake methods poses significant social risks to personal identity security. While numerous deepfake detection methods have been proposed as countermeasures against malicious face swapping, they can only output binary labels (Fake/Real) for distinguishing fake content without reliable and traceable evidence. To achieve visual forensics and target face attribution, we propose a novel task named face retracing, which considers retracing the original target face from the given fake one via inverse mapping. Toward this goal, we propose an IDRetracor that can retrace arbitrary original target identities from fake faces generated by multiple face swapping methods. Specifically, we first adopt a mapping resolver to perceive the possible solution space of the original target face for the inverse mappings. Then, we propose mapping-aware convolutions to retrace the original target face from the fake one. Such convolutions contain multiple kernels that can be combined under the control of the mapping resolver to tackle different face swapping mappings dynamically. Extensive experiments demonstrate that the IDRetracor exhibits promising retracing performance from both quantitative and qualitative perspectives.

Title: Hybrid SD: Edge-Cloud Collaborative Inference for Stable Diffusion Models

• Authors: Chenqian Yan, Songwei Liu, Hongjian Liu, Xurui Peng, Xiaojian Wang, Fangming Chen, Lean Fu, Xing Mei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06646
• Pdf URL: https://arxiv.org/pdf/2408.06646
• Copy Paste: [[2408.06646]] Hybrid SD: Edge-Cloud Collaborative Inference for Stable Diffusion Models(https://arxiv.org/abs/2408.06646)
• Keywords: diffusion
• Abstract: Stable Diffusion Models (SDMs) have shown remarkable proficiency in image synthesis. However, their broad application is impeded by their large model sizes and intensive computational requirements, which typically require expensive cloud servers for deployment. On the flip side, while there are many compact models tailored for edge devices that can reduce these demands, they often compromise on semantic integrity and visual quality when compared to full-sized SDMs. To bridge this gap, we introduce Hybrid SD, an innovative, training-free SDMs inference framework designed for edge-cloud collaborative inference. Hybrid SD distributes the early steps of the diffusion process to the large models deployed on cloud servers, enhancing semantic planning. Furthermore, small efficient models deployed on edge devices can be integrated for refining visual details in the later stages. Acknowledging the diversity of edge devices with differing computational and storage capacities, we employ structural pruning to the SDMs U-Net and train a lightweight VAE. Empirical evaluations demonstrate that our compressed models achieve state-of-the-art parameter efficiency (225.8M) on edge devices with competitive image quality. Additionally, Hybrid SD reduces the cloud cost by 66% with edge-cloud collaborative inference.

Title: Bi-directional Contextual Attention for 3D Dense Captioning

• Authors: Minjung Kim, Hyung Suk Lim, Soonyoung Lee, Bumsoo Kim, Gunhee Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06662
• Pdf URL: https://arxiv.org/pdf/2408.06662
• Copy Paste: [[2408.06662]] Bi-directional Contextual Attention for 3D Dense Captioning(https://arxiv.org/abs/2408.06662)
• Keywords: transformer
• Abstract: 3D dense captioning is a task involving the localization of objects and the generation of descriptions for each object in a 3D scene. Recent approaches have attempted to incorporate contextual information by modeling relationships with object pairs or aggregating the nearest neighbor features of an object. However, the contextual information constructed in these scenarios is limited in two aspects: first, objects have multiple positional relationships that exist across the entire global scene, not only near the object itself. Second, it faces with contradicting objectives--where localization and attribute descriptions are generated better with tight localization, while descriptions involving global positional relations are generated better with contextualized features of the global scene. To overcome this challenge, we introduce BiCA, a transformer encoder-decoder pipeline that engages in 3D dense captioning for each object with Bi-directional Contextual Attention. Leveraging parallelly decoded instance queries for objects and context queries for non-object contexts, BiCA generates object-aware contexts, where the contexts relevant to each object is summarized, and context-aware objects, where the objects relevant to the summarized object-aware contexts are aggregated. This extension relieves previous methods from the contradicting objectives, enhancing both localization performance and enabling the aggregation of contextual features throughout the global scene; thus improving caption generation performance simultaneously. Extensive experiments on two of the most widely-used 3D dense captioning datasets demonstrate that our proposed method achieves a significant improvement over prior methods.

Title: Amuro & Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models

• Authors: Kaiser Sun, Mark Dredze
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06663
• Pdf URL: https://arxiv.org/pdf/2408.06663
• Copy Paste: [[2408.06663]] Amuro & Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models(https://arxiv.org/abs/2408.06663)
• Keywords: large language model
• Abstract: The development of large language models leads to the formation of a pre-train-then-align paradigm, in which the model is typically pre-trained on a large text corpus and undergoes a tuning stage to align the model with human preference or downstream tasks. In this work, we investigate the relationship between pre-training and fine-tuning by fine-tuning multiple intermediate pre-trained model checkpoints. Our results on 18 datasets suggest that i) continual pre-training improves the model in a latent way that unveils after fine-tuning; ii) with extra fine-tuning, the datasets that the model does not demonstrate capability gain much more than those that the model performs well during the pre-training stage; iii) although model benefits significantly through supervised fine-tuning, it may forget previously known domain knowledge and the tasks that are not seen during fine-tuning; iv) the model resembles high sensitivity to evaluation prompts after supervised fine-tuning, but this sensitivity can be alleviated by more pre-training.

Title: RW-NSGCN: A Robust Approach to Structural Attacks via Negative Sampling

• Authors: Shuqi He, Jun Zhuang, Ding Wang, Jun Song
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06665
• Pdf URL: https://arxiv.org/pdf/2408.06665
• Copy Paste: [[2408.06665]] RW-NSGCN: A Robust Approach to Structural Attacks via Negative Sampling(https://arxiv.org/abs/2408.06665)
• Keywords: attack, robust
• Abstract: Node classification using Graph Neural Networks (GNNs) has been widely applied in various practical scenarios, such as predicting user interests and detecting communities in social networks. However, recent studies have shown that graph-structured networks often contain potential noise and attacks, in the form of topological perturbations and weight disturbances, which can lead to decreased classification performance in GNNs. To improve the robustness of the model, we propose a novel method: Random Walk Negative Sampling Graph Convolutional Network (RW-NSGCN). Specifically, RW-NSGCN integrates the Random Walk with Restart (RWR) and PageRank (PGR) algorithms for negative sampling and employs a Determinantal Point Process (DPP)-based GCN for convolution operations. RWR leverages both global and local information to manage noise and local variations, while PGR assesses node importance to stabilize the topological structure. The DPP-based GCN ensures diversity among negative samples and aggregates their features to produce robust node embeddings, thereby improving classification performance. Experimental results demonstrate that the RW-NSGCN model effectively addresses network topology attacks and weight instability, increasing the accuracy of anomaly detection and overall stability. In terms of classification accuracy, RW-NSGCN significantly outperforms existing methods, showing greater resilience across various scenarios and effectively mitigating the impact of such vulnerabilities.

Title: Leveraging Priors via Diffusion Bridge for Time Series Generation

• Authors: Jinseong Park, Seungyun Lee, Woojin Jeong, Yujin Choi, Jaewook Lee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06672
• Pdf URL: https://arxiv.org/pdf/2408.06672
• Copy Paste: [[2408.06672]] Leveraging Priors via Diffusion Bridge for Time Series Generation(https://arxiv.org/abs/2408.06672)
• Keywords: diffusion
• Abstract: Time series generation is widely used in real-world applications such as simulation, data augmentation, and hypothesis test techniques. Recently, diffusion models have emerged as the de facto approach for time series generation, emphasizing diverse synthesis scenarios based on historical or correlated time series data streams. Since time series have unique characteristics, such as fixed time order and data scaling, standard Gaussian prior might be ill-suited for general time series generation. In this paper, we exploit the usage of diverse prior distributions for synthesis. Then, we propose TimeBridge, a framework that enables flexible synthesis by leveraging diffusion bridges to learn the transport between chosen prior and data distributions. Our model covers a wide range of scenarios in time series diffusion models, which leverages (i) data- and time-dependent priors for unconditional synthesis, and (ii) data-scale preserving synthesis with a constraint as a prior for conditional generation. Experimentally, our model achieves state-of-the-art performance in both unconditional and conditional time series generation tasks.

Title: Pragmatic inference of scalar implicature by LLMs

• Authors: Ye-eun Cho, Seong mook Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06673
• Pdf URL: https://arxiv.org/pdf/2408.06673
• Copy Paste: [[2408.06673]] Pragmatic inference of scalar implicature by LLMs(https://arxiv.org/abs/2408.06673)
• Keywords: large language model
• Abstract: This study investigates how Large Language Models (LLMs), particularly BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019), engage in pragmatic inference of scalar implicature, such as some. Two sets of experiments were conducted using cosine similarity and next sentence/token prediction as experimental methods. The results in experiment 1 showed that, both models interpret some as pragmatic implicature not all in the absence of context, aligning with human language processing. In experiment 2, in which Question Under Discussion (QUD) was presented as a contextual cue, BERT showed consistent performance regardless of types of QUDs, while GPT-2 encountered processing difficulties since a certain type of QUD required pragmatic inference for implicature. The findings revealed that, in terms of theoretical approaches, BERT inherently incorporates pragmatic implicature not all within the term some, adhering to Default model (Levinson, 2000). In contrast, GPT-2 seems to encounter processing difficulties in inferring pragmatic implicature within context, consistent with Context-driven model (Sperber and Wilson, 2002).

Title: Latin Treebanks in Review: An Evaluation of Morphological Tagging Across Time

• Authors: Marisa Hudspeth, Brendan O'Connor, Laure Thompson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06675
• Pdf URL: https://arxiv.org/pdf/2408.06675
• Copy Paste: [[2408.06675]] Latin Treebanks in Review: An Evaluation of Morphological Tagging Across Time(https://arxiv.org/abs/2408.06675)
• Keywords: robust
• Abstract: Existing Latin treebanks draw from Latin's long written tradition, spanning 17 centuries and a variety of cultures. Recent efforts have begun to harmonize these treebanks' annotations to better train and evaluate morphological taggers. However, the heterogeneity of these treebanks must be carefully considered to build effective and reliable data. In this work, we review existing Latin treebanks to identify the texts they draw from, identify their overlap, and document their coverage across time and genre. We additionally design automated conversions of their morphological feature annotations into the conventions of standard Latin grammar. From this, we build new time-period data splits that draw from the existing treebanks which we use to perform a broad cross-time analysis for POS and morphological feature tagging. We find that BERT-based taggers outperform existing taggers while also being more robust to cross-domain shifts.

Title: Case-based Explainability for Random Forest: Prototypes, Critics, Counter-factuals and Semi-factuals

• Authors: Gregory Yampolsky, Dhruv Desai, Mingshu Li, Stefano Pasquali, Dhagash Mehta
• Subjects: cs.LG, q-fin.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.06679
• Pdf URL: https://arxiv.org/pdf/2408.06679
• Copy Paste: [[2408.06679]] Case-based Explainability for Random Forest: Prototypes, Critics, Counter-factuals and Semi-factuals(https://arxiv.org/abs/2408.06679)
• Keywords: explainability
• Abstract: The explainability of black-box machine learning algorithms, commonly known as Explainable Artificial Intelligence (XAI), has become crucial for financial and other regulated industrial applications due to regulatory requirements and the need for transparency in business practices. Among the various paradigms of XAI, Explainable Case-Based Reasoning (XCBR) stands out as a pragmatic approach that elucidates the output of a model by referencing actual examples from the data used to train or test the model. Despite its potential, XCBR has been relatively underexplored for many algorithms such as tree-based models until recently. We start by observing that most XCBR methods are defined based on the distance metric learned by the algorithm. By utilizing a recently proposed technique to extract the distance metric learned by Random Forests (RFs), which is both geometry- and accuracy-preserving, we investigate various XCBR methods. These methods amount to identify special points from the training datasets, such as prototypes, critics, counter-factuals, and semi-factuals, to explain the predictions for a given query of the RF. We evaluate these special points using various evaluation metrics to assess their explanatory power and effectiveness.

Title: DC3DO: Diffusion Classifier for 3D Objects

• Authors: Nursena Koprucu, Meher Shashwat Nigam, Shicheng Xu (Luke), Biruk Abere, Gabriele Dominici, Andrew Rodriguez, Sharvaree Vadgam, Berfin Inal, Alberto Tono
• Subjects: cs.CV, cs.AI, cs.CG
• Abstract URL: https://arxiv.org/abs/2408.06693
• Pdf URL: https://arxiv.org/pdf/2408.06693
• Copy Paste: [[2408.06693]] DC3DO: Diffusion Classifier for 3D Objects(https://arxiv.org/abs/2408.06693)
• Keywords: diffusion, generative
• Abstract: Inspired by Geoffrey Hinton emphasis on generative modeling, To recognize shapes, first learn to generate them, we explore the use of 3D diffusion models for object classification. Leveraging the density estimates from these models, our approach, the Diffusion Classifier for 3D Objects (DC3DO), enables zero-shot classification of 3D shapes without additional training. On average, our method achieves a 12.5 percent improvement compared to its multiview counterparts, demonstrating superior multimodal reasoning over discriminative approaches. DC3DO employs a class-conditional diffusion model trained on ShapeNet, and we run inferences on point clouds of chairs and cars. This work highlights the potential of generative models in 3D object classification.

Title: Information Geometry and Beta Link for Optimizing Sparse Variational Student-t Processes

• Authors: Jian Xu, Delu Zeng, John Paisley
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06699
• Pdf URL: https://arxiv.org/pdf/2408.06699
• Copy Paste: [[2408.06699]] Information Geometry and Beta Link for Optimizing Sparse Variational Student-t Processes(https://arxiv.org/abs/2408.06699)
• Keywords: robust
• Abstract: Recently, a sparse version of Student-t Processes, termed sparse variational Student-t Processes, has been proposed to enhance computational efficiency and flexibility for real-world datasets using stochastic gradient descent. However, traditional gradient descent methods like Adam may not fully exploit the parameter space geometry, potentially leading to slower convergence and suboptimal performance. To mitigate these issues, we adopt natural gradient methods from information geometry for variational parameter optimization of Student-t Processes. This approach leverages the curvature and structure of the parameter space, utilizing tools such as the Fisher information matrix which is linked to the Beta function in our model. This method provides robust mathematical support for the natural gradient algorithm when using Student's t-distribution as the variational distribution. Additionally, we present a mini-batch algorithm for efficiently computing natural gradients. Experimental results across four benchmark datasets demonstrate that our method consistently accelerates convergence speed.

Title: MAIR++: Improving Multi-view Attention Inverse Rendering with Implicit Lighting Representation

• Authors: JunYong Choi, SeokYeong Lee, Haesol Park, Seung-Won Jung, Ig-Jae Kim, Junghyun Cho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06707
• Pdf URL: https://arxiv.org/pdf/2408.06707
• Copy Paste: [[2408.06707]] MAIR++: Improving Multi-view Attention Inverse Rendering with Implicit Lighting Representation(https://arxiv.org/abs/2408.06707)
• Keywords: robust
• Abstract: In this paper, we propose a scene-level inverse rendering framework that uses multi-view images to decompose the scene into geometry, SVBRDF, and 3D spatially-varying lighting. While multi-view images have been widely used for object-level inverse rendering, scene-level inverse rendering has primarily been studied using single-view images due to the lack of a dataset containing high dynamic range multi-view images with ground-truth geometry, material, and spatially-varying lighting. To improve the quality of scene-level inverse rendering, a novel framework called Multi-view Attention Inverse Rendering (MAIR) was recently introduced. MAIR performs scene-level multi-view inverse rendering by expanding the OpenRooms dataset, designing efficient pipelines to handle multi-view images, and splitting spatially-varying lighting. Although MAIR showed impressive results, its lighting representation is fixed to spherical Gaussians, which limits its ability to render images realistically. Consequently, MAIR cannot be directly used in applications such as material editing. Moreover, its multi-view aggregation networks have difficulties extracting rich features because they only focus on the mean and variance between multi-view features. In this paper, we propose its extended version, called MAIR++. MAIR++ addresses the aforementioned limitations by introducing an implicit lighting representation that accurately captures the lighting conditions of an image while facilitating realistic rendering. Furthermore, we design a directional attention-based multi-view aggregation network to infer more intricate relationships between views. Experimental results show that MAIR++ not only achieves better performance than MAIR and single-view-based methods, but also displays robust performance on unseen real-world scenes.

Title: Variational Learning of Gaussian Process Latent Variable Models through Stochastic Gradient Annealed Importance Sampling

• Authors: Jian Xu, Shian Du, Junmei Yang, Qianli Ma, Delu Zeng
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.06710
• Pdf URL: https://arxiv.org/pdf/2408.06710
• Copy Paste: [[2408.06710]] Variational Learning of Gaussian Process Latent Variable Models through Stochastic Gradient Annealed Importance Sampling(https://arxiv.org/abs/2408.06710)
• Keywords: robust
• Abstract: Gaussian Process Latent Variable Models (GPLVMs) have become increasingly popular for unsupervised tasks such as dimensionality reduction and missing data recovery due to their flexibility and non-linear nature. An importance-weighted version of the Bayesian GPLVMs has been proposed to obtain a tighter variational bound. However, this version of the approach is primarily limited to analyzing simple data structures, as the generation of an effective proposal distribution can become quite challenging in high-dimensional spaces or with complex data sets. In this work, we propose an Annealed Importance Sampling (AIS) approach to address these issues. By transforming the posterior into a sequence of intermediate distributions using annealing, we combine the strengths of Sequential Monte Carlo samplers and VI to explore a wider range of posterior distributions and gradually approach the target distribution. We further propose an efficient algorithm by reparameterizing all variables in the evidence lower bound (ELBO). Experimental results on both toy and image datasets demonstrate that our method outperforms state-of-the-art methods in terms of tighter variational bounds, higher log-likelihoods, and more robust convergence.

Title: Computation-friendly Graph Neural Network Design by Accumulating Knowledge on Large Language Models

• Authors: Jialiang Wang, Shimin Di, Hanmo Liu, Zhili Wang, Jiachuan Wang, Lei Chen, Xiaofang Zhou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06717
• Pdf URL: https://arxiv.org/pdf/2408.06717
• Copy Paste: [[2408.06717]] Computation-friendly Graph Neural Network Design by Accumulating Knowledge on Large Language Models(https://arxiv.org/abs/2408.06717)
• Keywords: large language model
• Abstract: Graph Neural Networks (GNNs), like other neural networks, have shown remarkable success but are hampered by the complexity of their architecture designs, which heavily depend on specific data and tasks. Traditionally, designing proper architectures involves trial and error, which requires intensive manual effort to optimize various components. To reduce human workload, researchers try to develop automated algorithms to design GNNs. However, both experts and automated algorithms suffer from two major issues in designing GNNs: 1) the substantial computational resources expended in repeatedly trying candidate GNN architectures until a feasible design is achieved, and 2) the intricate and prolonged processes required for humans or algorithms to accumulate knowledge of the interrelationship between graphs, GNNs, and performance. To further enhance the automation of GNN architecture design, we propose a computation-friendly way to empower Large Language Models (LLMs) with specialized knowledge in designing GNNs, thereby drastically shortening the computational overhead and development cycle of designing GNN architectures. Our framework begins by establishing a knowledge retrieval pipeline that comprehends the intercorrelations between graphs, GNNs, and performance. This pipeline converts past model design experiences into structured knowledge for LLM reference, allowing it to quickly suggest initial model proposals. Subsequently, we introduce a knowledge-driven search strategy that emulates the exploration-exploitation process of human experts, enabling quick refinement of initial proposals within a promising scope. Extensive experiments demonstrate that our framework can efficiently deliver promising (e.g., Top-5.77%) initial model proposals for unseen datasets within seconds and without any prior training and achieve outstanding search performance in a few iterations.

Title: Multilingual Models for Check-Worthy Social Media Posts Detection

• Authors: Sebastian Kula, Michal Gregor
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06737
• Pdf URL: https://arxiv.org/pdf/2408.06737
• Copy Paste: [[2408.06737]] Multilingual Models for Check-Worthy Social Media Posts Detection(https://arxiv.org/abs/2408.06737)
• Keywords: robust, transformer
• Abstract: This work presents an extensive study of transformer-based NLP models for detection of social media posts that contain verifiable factual claims and harmful claims. The study covers various activities, including dataset collection, dataset pre-processing, architecture selection, setup of settings, model training (fine-tuning), model testing, and implementation. The study includes a comprehensive analysis of different models, with a special focus on multilingual models where the same model is capable of processing social media posts in both English and in low-resource languages such as Arabic, Bulgarian, Dutch, Polish, Czech, Slovak. The results obtained from the study were validated against state-of-the-art models, and the comparison demonstrated the robustness of the proposed models. The novelty of this work lies in the development of multi-label multilingual classification models that can simultaneously detect harmful posts and posts that contain verifiable factual claims in an efficient way.

Title: DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusion

• Authors: Yujia Wu, Yiming Shi, Jiwei Wei, Chengwei Sun, Yuyang Zhou, Yang Yang, Heng Tao Shen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06740
• Pdf URL: https://arxiv.org/pdf/2408.06740
• Copy Paste: [[2408.06740]] DiffLoRA: Generating Personalized Low-Rank Adaptation Weights with Diffusion(https://arxiv.org/abs/2408.06740)
• Keywords: diffusion, generative
• Abstract: Personalized text-to-image generation has gained significant attention for its capability to generate high-fidelity portraits of specific identities conditioned on user-defined prompts. Existing methods typically involve test-time fine-tuning or instead incorporating an additional pre-trained branch. However, these approaches struggle to simultaneously address the demands of efficiency, identity fidelity, and preserving the model's original generative capabilities. In this paper, we propose DiffLoRA, a novel approach that leverages diffusion models as a hypernetwork to predict personalized low-rank adaptation (LoRA) weights based on the reference images. By integrating these LoRA weights into the text-to-image model, DiffLoRA achieves personalization during inference without further training. Additionally, we propose an identity-oriented LoRA weight construction pipeline to facilitate the training of DiffLoRA. By utilizing the dataset produced by this pipeline, our DiffLoRA consistently generates high-performance and accurate LoRA weights. Extensive evaluations demonstrate the effectiveness of our method, achieving both time efficiency and maintaining identity fidelity throughout the personalization process.

Title: Improving Synthetic Image Detection Towards Generalization: An Image Transformation Perspective

• Authors: Ouxiang Li, Jiayin Cai, Yanbin Hao, Xiaolong Jiang, Yao Hu, Fuli Feng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06741
• Pdf URL: https://arxiv.org/pdf/2408.06741
• Copy Paste: [[2408.06741]] Improving Synthetic Image Detection Towards Generalization: An Image Transformation Perspective(https://arxiv.org/abs/2408.06741)
• Keywords: generative
• Abstract: With recent generative models facilitating photo-realistic image synthesis, the proliferation of synthetic images has also engendered certain negative impacts on social platforms, thereby raising an urgent imperative to develop effective detectors. Current synthetic image detection (SID) pipelines are primarily dedicated to crafting universal artifact features, accompanied by an oversight about SID training paradigm. In this paper, we re-examine the SID problem and identify two prevalent biases in current training paradigms, i.e., weakened artifact features and overfitted artifact features. Meanwhile, we discover that the imaging mechanism of synthetic images contributes to heightened local correlations among pixels, suggesting that detectors should be equipped with local awareness. In this light, we propose SAFE, a lightweight and effective detector with three simple image transformations. Firstly, for weakened artifact features, we substitute the down-sampling operator with the crop operator in image pre-processing to help circumvent artifact distortion. Secondly, for overfitted artifact features, we include ColorJitter and RandomRotation as additional data augmentations, to help alleviate irrelevant biases from color discrepancies and semantic differences in limited training samples. Thirdly, for local awareness, we propose a patch-based random masking strategy tailored for SID, forcing the detector to focus on local regions at training. Comparative experiments are conducted on an open-world dataset, comprising synthetic images generated by 26 distinct generative models. Our pipeline achieves a new state-of-the-art performance, with remarkable improvements of 4.5% in accuracy and 2.9% in average precision against existing methods.

Title: Class-aware and Augmentation-free Contrastive Learning from Label Proportion

• Authors: Jialiang Wang, Ning Zhang, Shimin Di, Ruidong Wang, Lei Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.06743
• Pdf URL: https://arxiv.org/pdf/2408.06743
• Copy Paste: [[2408.06743]] Class-aware and Augmentation-free Contrastive Learning from Label Proportion(https://arxiv.org/abs/2408.06743)
• Keywords: privacy, robust
• Abstract: Learning from Label Proportion (LLP) is a weakly supervised learning scenario in which training data is organized into predefined bags of instances, disclosing only the class label proportions per bag. This paradigm is essential for user modeling and personalization, where user privacy is paramount, offering insights into user preferences without revealing individual data. LLP faces a unique difficulty: the misalignment between bag-level supervision and the objective of instance-level prediction, primarily due to the inherent ambiguity in label proportion matching. Previous studies have demonstrated deep representation learning can generate auxiliary signals to promote the supervision level in the image domain. However, applying these techniques to tabular data presents significant challenges: 1) they rely heavily on label-invariant augmentation to establish multi-view, which is not feasible with the heterogeneous nature of tabular datasets, and 2) tabular datasets often lack sufficient semantics for perfect class distinction, making them prone to suboptimality caused by the inherent ambiguity of label proportion matching. To address these challenges, we propose an augmentation-free contrastive framework TabLLP-BDC that introduces class-aware supervision (explicitly aware of class differences) at the instance level. Our solution features a two-stage Bag Difference Contrastive (BDC) learning mechanism that establishes robust class-aware instance-level supervision by disassembling the nuance between bag label proportions, without relying on augmentations. Concurrently, our model presents a pioneering multi-task pretraining pipeline tailored for tabular-based LLP, capturing intrinsic tabular feature correlations in alignment with label proportion distribution. Extensive experiments demonstrate that TabLLP-BDC achieves state-of-the-art performance for LLP in the tabular domain.

Title: ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation

• Authors: Jingyun Wang, Guoliang Kang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06747
• Pdf URL: https://arxiv.org/pdf/2408.06747
• Copy Paste: [[2408.06747]] ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation(https://arxiv.org/abs/2408.06747)
• Keywords: transformer, segmentation
• Abstract: Recent works utilize CLIP to perform the challenging unsupervised semantic segmentation task where only images without annotations are available. However, we observe that when adopting CLIP to such a pixel-level understanding task, unexpected bias (including class-preference bias and space-preference bias) occurs. Previous works don't explicitly model the bias, which largely constrains the segmentation performance. In this paper, we propose to explicitly model and rectify the bias existing in CLIP to facilitate the unsupervised semantic segmentation task. Specifically, we design a learnable ''Reference'' prompt to encode class-preference bias and a projection of the positional embedding in vision transformer to encode space-preference bias respectively. To avoid interference, two kinds of biases are firstly independently encoded into the Reference feature and the positional feature. Via a matrix multiplication between two features, a bias logit map is generated to explicitly represent two kinds of biases. Then we rectify the logits of CLIP via a simple element-wise subtraction. To make the rectified results smoother and more contextual, we design a mask decoder which takes the feature of CLIP and rectified logits as input and outputs a rectified segmentation mask with the help of Gumbel-Softmax operation. To make the bias modeling and rectification process meaningful and effective, a contrastive loss based on masked visual features and the text features of different classes is imposed. To further improve the segmentation, we distill the knowledge from the rectified CLIP to the advanced segmentation architecture via minimizing our designed mask-guided, feature-guided and text-guided loss terms. Extensive experiments on various benchmarks demonstrate that ReCLIP++ performs favorably against previous SOTAs. The implementation is available at: this https URL.

Title: Sumotosima: A Framework and Dataset for Classifying and Summarizing Otoscopic Images

• Authors: Eram Anwarul Khan, Anas Anwarul Haq Khan
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06755
• Pdf URL: https://arxiv.org/pdf/2408.06755
• Copy Paste: [[2408.06755]] Sumotosima: A Framework and Dataset for Classifying and Summarizing Otoscopic Images(https://arxiv.org/abs/2408.06755)
• Keywords: transformer
• Abstract: Otoscopy is a diagnostic procedure to examine the ear canal and eardrum using an otoscope. It identifies conditions like infections, foreign bodies, ear drum perforations and ear abnormalities. We propose a novel resource efficient deep learning and transformer based framework, Sumotosima (Summarizer for otoscopic images), an end-to-end pipeline for classification followed by summarization. Our framework works on combination of triplet and cross-entropy losses. Additionally, we use Knowledge Enhanced Multimodal BART whose input is fused textual and image embedding. The objective is to provide summaries that are well-suited for patients, ensuring clarity and efficiency in understanding otoscopic images. Given the lack of existing datasets, we have curated our own OCASD (Otoscopic Classification And Summary Dataset), which includes 500 images with 5 unique categories annotated with their class and summaries by Otolaryngologists. Sumotosima achieved a result of 98.03%, which is 7.00%, 3.10%, 3.01% higher than K-Nearest Neighbors, Random Forest and Support Vector Machines, respectively, in classification tasks. For summarization, Sumotosima outperformed GPT-4o and LLaVA by 88.53% and 107.57% in ROUGE scores, respectively. We have made our code and dataset publicly available at this https URL

Title: Cross-View Geolocalization and Disaster Mapping with Street-View and VHR Satellite Imagery: A Case Study of Hurricane IAN

• Authors: Hao Li, Fabian Deuser, Wenping Yina, Xuanshu Luo, Paul Walther, Gengchen Mai, Wei Huang, Martin Werner
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06761
• Pdf URL: https://arxiv.org/pdf/2408.06761
• Copy Paste: [[2408.06761]] Cross-View Geolocalization and Disaster Mapping with Street-View and VHR Satellite Imagery: A Case Study of Hurricane IAN(https://arxiv.org/abs/2408.06761)
• Keywords: transformer
• Abstract: Nature disasters play a key role in shaping human-urban infrastructure interactions. Effective and efficient response to natural disasters is essential for building resilience and a sustainable urban environment. Two types of information are usually the most necessary and difficult to gather in disaster response. The first information is about disaster damage perception, which shows how badly people think that urban infrastructure has been damaged. The second information is geolocation awareness, which means how people whereabouts are made available. In this paper, we proposed a novel disaster mapping framework, namely CVDisaster, aiming at simultaneously addressing geolocalization and damage perception estimation using cross-view Street-View Imagery (SVI) and Very High-Resolution satellite imagery. CVDisaster consists of two cross-view models, where CVDisaster-Geoloc refers to a cross-view geolocalization model based on a contrastive learning objective with a Siamese ConvNeXt image encoder, and CVDisaster-Est is a cross-view classification model based on a Couple Global Context Vision Transformer (CGCViT). Taking Hurricane IAN as a case study, we evaluate the CVDisaster framework by creating a novel cross-view dataset (CVIAN) and conducting extensive experiments. As a result, we show that CVDisaster can achieve highly competitive performance (over 80% for geolocalization and 75% for damage perception estimation) with even limited fine-tuning efforts, which largely motivates future cross-view models and applications within a broader GeoAI research community. The data and code are publicly available at: this https URL.

Title: Robust Black-box Testing of Deep Neural Networks using Co-Domain Coverage

• Authors: Aishwarya Gupta, Indranil Saha, Piyush Rai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.06766
• Pdf URL: https://arxiv.org/pdf/2408.06766
• Copy Paste: [[2408.06766]] Robust Black-box Testing of Deep Neural Networks using Co-Domain Coverage(https://arxiv.org/abs/2408.06766)
• Keywords: robust
• Abstract: Rigorous testing of machine learning models is necessary for trustworthy deployments. We present a novel black-box approach for generating test-suites for robust testing of deep neural networks (DNNs). Most existing methods create test inputs based on maximizing some "coverage" criterion/metric such as a fraction of neurons activated by the test inputs. Such approaches, however, can only analyze each neuron's behavior or each layer's output in isolation and are unable to capture their collective effect on the DNN's output, resulting in test suites that often do not capture the various failure modes of the DNN adequately. These approaches also require white-box access, i.e., access to the DNN's internals (node activations). We present a novel black-box coverage criterion called Co-Domain Coverage (CDC), which is defined as a function of the model's output and thus takes into account its end-to-end behavior. Subsequently, we develop a new fuzz testing procedure named CoDoFuzz, which uses CDC to guide the fuzzing process to generate a test suite for a DNN. We extensively compare the test suite generated by CoDoFuzz with those generated using several state-of-the-art coverage-based fuzz testing methods for the DNNs trained on six publicly available datasets. Experimental results establish the efficiency and efficacy of CoDoFuzz in generating the largest number of misclassified inputs and the inputs for which the model lacks confidence in its decision.

Title: Exploring Domain Shift on Radar-Based 3D Object Detection Amidst Diverse Environmental Conditions

• Authors: Miao Zhang, Sherif Abdulatif, Benedikt Loesch, Marco Altmann, Marius Schwarz, Bin Yang
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.06772
• Pdf URL: https://arxiv.org/pdf/2408.06772
• Copy Paste: [[2408.06772]] Exploring Domain Shift on Radar-Based 3D Object Detection Amidst Diverse Environmental Conditions(https://arxiv.org/abs/2408.06772)
• Keywords: robust
• Abstract: The rapid evolution of deep learning and its integration with autonomous driving systems have led to substantial advancements in 3D perception using multimodal sensors. Notably, radar sensors show greater robustness compared to cameras and lidar under adverse weather and varying illumination conditions. This study delves into the often-overlooked yet crucial issue of domain shift in 4D radar-based object detection, examining how varying environmental conditions, such as different weather patterns and road types, impact 3D object detection performance. Our findings highlight distinct domain shifts across various weather scenarios, revealing unique dataset sensitivities that underscore the critical role of radar point cloud generation. Additionally, we demonstrate that transitioning between different road types, especially from highways to urban settings, introduces notable domain shifts, emphasizing the necessity for diverse data collection across varied road environments. To the best of our knowledge, this is the first comprehensive analysis of domain shift effects on 4D radar-based object detection. We believe this empirical study contributes to understanding the complex nature of domain shifts in radar data and suggests paths forward for data collection strategy in the face of environmental variability.

Title: Fast-and-Frugal Text-Graph Transformers are Effective Link Predictors

• Authors: Andrei C. Coman, Christos Theodoropoulos, Marie-Francine Moens, James Henderson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06778
• Pdf URL: https://arxiv.org/pdf/2408.06778
• Copy Paste: [[2408.06778]] Fast-and-Frugal Text-Graph Transformers are Effective Link Predictors(https://arxiv.org/abs/2408.06778)
• Keywords: transformer
• Abstract: Link prediction models can benefit from incorporating textual descriptions of entities and relations, enabling fully inductive learning and flexibility in dynamic graphs. We address the challenge of also capturing rich structured information about the local neighbourhood of entities and their relations, by introducing a Transformer-based approach that effectively integrates textual descriptions with graph structure, reducing the reliance on resource-intensive text encoders. Our experiments on three challenging datasets show that our Fast-and-Frugal Text-Graph (FnF-TG) Transformers achieve superior performance compared to the previous state-of-the-art methods, while maintaining efficiency and scalability.

Title: Do Vision-Language Foundational models show Robust Visual Perception?

• Authors: Shivam Chandhok, Pranav Tandon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06781
• Pdf URL: https://arxiv.org/pdf/2408.06781
• Copy Paste: [[2408.06781]] Do Vision-Language Foundational models show Robust Visual Perception?(https://arxiv.org/abs/2408.06781)
• Keywords: robust
• Abstract: Recent advances in vision-language foundational models have enabled development of systems that can perform visual understanding and reasoning tasks. However, it is unclear if these models are robust to distribution shifts, and how their performance and generalization capabilities vary under changes in data distribution. In this project we strive to answer the question "Are vision-language foundational models robust to distribution shifts like human perception?" Specifically, we consider a diverse range of vision-language models and compare how the performance of these systems is affected by corruption based distribution shifts (such as \textit{motion blur, fog, snow, gaussian noise}) commonly found in practical real-world scenarios. We analyse the generalization capabilities qualitatively and quantitatively on zero-shot image classification task under aforementioned distribution shifts. Our code will be avaible at \url{this https URL}

Title: Unlock the Power of Frozen LLMs in Knowledge Graph Completion

• Authors: Bo Xue, Yi Xu, Yunchong Song, Yiming Pang, Yuyang Ren, Jiaxin Ding, Luoyi Fu, Xinbing Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06787
• Pdf URL: https://arxiv.org/pdf/2408.06787
• Copy Paste: [[2408.06787]] Unlock the Power of Frozen LLMs in Knowledge Graph Completion(https://arxiv.org/abs/2408.06787)
• Keywords: large language model
• Abstract: Classical knowledge graph completion (KGC) methods rely solely on structural information, struggling with the inherent sparsity of knowledge graphs (KGs). Large Language Models (LLMs) learn extensive knowledge from large corpora with powerful context modeling, which is ideal for mitigating the limitations of previous methods. Directly fine-tuning LLMs offers great capability but comes at the cost of huge time and memory consumption, while utilizing frozen LLMs yields suboptimal results. In this work, we aim to leverage LLMs for KGC effectively and efficiently. We capture the context-aware hidden states of knowledge triples by employing prompts to stimulate the intermediate layers of LLMs. We then train a data-efficient classifier on these hidden states to harness the inherent capabilities of frozen LLMs in KGC. We also generate entity descriptions with subgraph sampling on KGs, reducing the ambiguity of triplets and enriching the knowledge representation. Extensive experiments on standard benchmarks showcase the efficiency and effectiveness of our approach. We outperform classical KGC methods on most datasets and match the performance of fine-tuned LLMs. Additionally, compared to fine-tuned LLMs, we boost GPU memory efficiency by \textbf{$188\times$} and speed up training+inference by \textbf{$13.48\times$}.

Title: Visual Neural Decoding via Improved Visual-EEG Semantic Consistency

• Authors: Hongzhou Chen, Lianghua He, Yihang Liu, Longzhen Yang
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.06788
• Pdf URL: https://arxiv.org/pdf/2408.06788
• Copy Paste: [[2408.06788]] Visual Neural Decoding via Improved Visual-EEG Semantic Consistency(https://arxiv.org/abs/2408.06788)
• Keywords: robust, extraction
• Abstract: Visual neural decoding refers to the process of extracting and interpreting original visual experiences from human brain activity. Recent advances in metric learning-based EEG visual decoding methods have delivered promising results and demonstrated the feasibility of decoding novel visual categories from brain activity. However, methods that directly map EEG features to the CLIP embedding space may introduce mapping bias and cause semantic inconsistency among features, thereby degrading alignment and impairing decoding performance. To further explore the semantic consistency between visual and neural signals. In this work, we construct a joint semantic space and propose a Visual-EEG Semantic Decouple Framework that explicitly extracts the semantic-related features of these two modalities to facilitate optimal alignment. Specifically, a cross-modal information decoupling module is introduced to guide the extraction of semantic-related information from modalities. Then, by quantifying the mutual information between visual image and EEG features, we observe a strong positive correlation between the decoding performance and the magnitude of mutual information. Furthermore, inspired by the mechanisms of visual object understanding from neuroscience, we propose an intra-class geometric consistency approach during the alignment process. This strategy maps visual samples within the same class to consistent neural patterns, which further enhances the robustness and the performance of EEG visual decoding. Experiments on a large Image-EEG dataset show that our method achieves state-of-the-art results in zero-shot neural decoding tasks.

Title: Layerwise Recurrent Router for Mixture-of-Experts

• Authors: Zihan Qiu, Zeyu Huang, Shuang Cheng, Yizhi Zhou, Zili Wang, Ivan Titov, Jie Fu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06793
• Pdf URL: https://arxiv.org/pdf/2408.06793
• Copy Paste: [[2408.06793]] Layerwise Recurrent Router for Mixture-of-Experts(https://arxiv.org/abs/2408.06793)
• Keywords: large language model
• Abstract: The scaling of large language models (LLMs) has revolutionized their capabilities in various tasks, yet this growth must be matched with efficient computational strategies. The Mixture-of-Experts (MoE) architecture stands out for its ability to scale model size without significantly increasing training costs. Despite their advantages, current MoE models often display parameter inefficiency. For instance, a pre-trained MoE-based LLM with 52 billion parameters might perform comparably to a standard model with 6.7 billion parameters. Being a crucial part of MoE, current routers in different layers independently assign tokens without leveraging historical routing information, potentially leading to suboptimal token-expert combinations and the parameter inefficiency problem. To alleviate this issue, we introduce the Layerwise Recurrent Router for Mixture-of-Experts (RMoE). RMoE leverages a Gated Recurrent Unit (GRU) to establish dependencies between routing decisions across consecutive layers. Such layerwise recurrence can be efficiently parallelly computed for input tokens and introduces negotiable costs. Our extensive empirical evaluations demonstrate that RMoE-based language models consistently outperform a spectrum of baseline models. Furthermore, RMoE integrates a novel computation stage orthogonal to existing methods, allowing seamless compatibility with other MoE architectures. Our analyses attribute RMoE's gains to its effective cross-layer information sharing, which also improves expert selection and diversity. Our code is at this https URL

Title: Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuning

• Authors: Shibo Jie, Yehui Tang, Jianyuan Guo, Zhi-Hong Deng, Kai Han, Yunhe Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06798
• Pdf URL: https://arxiv.org/pdf/2408.06798
• Copy Paste: [[2408.06798]] Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuning(https://arxiv.org/abs/2408.06798)
• Keywords: transformer
• Abstract: Token compression expedites the training and inference of Vision Transformers (ViTs) by reducing the number of the redundant tokens, e.g., pruning inattentive tokens or merging similar tokens. However, when applied to downstream tasks, these approaches suffer from significant performance drop when the compression degrees are mismatched between training and inference stages, which limits the application of token compression on off-the-shelf trained models. In this paper, we propose a model arithmetic framework to decouple the compression degrees between the two stages. In advance, we additionally perform a fast parameter-efficient self-distillation stage on the pre-trained models to obtain a small plugin, called Token Compensator (ToCom), which describes the gap between models across different compression degrees. During inference, ToCom can be directly inserted into any downstream off-the-shelf models with any mismatched training and inference compression degrees to acquire universal performance improvements without further training. Experiments on over 20 downstream tasks demonstrate the effectiveness of our framework. On CIFAR100, fine-grained visual classification, and VTAB-1k, ToCom can yield up to a maximum improvement of 2.3%, 1.5%, and 2.0% in the average performance of DeiT-B, respectively. Code: this https URL

Title: Integrating Saliency Ranking and Reinforcement Learning for Enhanced Object Detection

• Authors: Matthias Bartolo, Dylan Seychell, Josef Bajada
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06803
• Pdf URL: https://arxiv.org/pdf/2408.06803
• Copy Paste: [[2408.06803]] Integrating Saliency Ranking and Reinforcement Learning for Enhanced Object Detection(https://arxiv.org/abs/2408.06803)
• Keywords: extraction
• Abstract: With the ever-growing variety of object detection approaches, this study explores a series of experiments that combine reinforcement learning (RL)-based visual attention methods with saliency ranking techniques to investigate transparent and sustainable solutions. By integrating saliency ranking for initial bounding box prediction and subsequently applying RL techniques to refine these predictions through a finite set of actions over multiple time steps, this study aims to enhance RL object detection accuracy. Presented as a series of experiments, this research investigates the use of various image feature extraction methods and explores diverse Deep Q-Network (DQN) architectural variations for deep reinforcement learning-based localisation agent training. Additionally, we focus on optimising the detection pipeline at every step by prioritising lightweight and faster models, while also incorporating the capability to classify detected objects, a feature absent in previous RL approaches. We show that by evaluating the performance of these trained agents using the Pascal VOC 2007 dataset, faster and more optimised models were developed. Notably, the best mean Average Precision (mAP) achieved in this study was 51.4, surpassing benchmarks set by RL-based single object detectors in the literature.

Title: Structure-preserving Planar Simplification for Indoor Environments

• Authors: Bishwash Khanal, Sanjay Rijal, Manish Awale, Vaghawan Ojha
• Subjects: cs.CV, cs.CG
• Abstract URL: https://arxiv.org/abs/2408.06814
• Pdf URL: https://arxiv.org/pdf/2408.06814
• Copy Paste: [[2408.06814]] Structure-preserving Planar Simplification for Indoor Environments(https://arxiv.org/abs/2408.06814)
• Keywords: robust, segmentation
• Abstract: This paper presents a novel approach for structure-preserving planar simplification of indoor scene point clouds for both simulated and real-world environments. Initially, the scene point cloud undergoes preprocessing steps, including noise reduction and Manhattan world alignment, to ensure robustness and coherence in subsequent analyses. We segment each captured scene into structured (walls-ceiling-floor) and non-structured (indoor objects) scenes. Leveraging a RANSAC algorithm, we extract primitive planes from the input point cloud, facilitating the segmentation and simplification of the structured scene. The best-fitting wall meshes are then generated from the primitives, followed by adjacent mesh merging with the vertex-translation algorithm which preserves the mesh layout. To accurately represent ceilings and floors, we employ the mesh clipping algorithm which clips the ceiling and floor meshes with respect to wall normals. In the case of indoor scenes, we apply a surface reconstruction technique to enhance the fidelity. This paper focuses on the intricate steps of the proposed scene simplification methodology, addressing complex scenarios such as multi-story and slanted walls and ceilings. We also conduct qualitative and quantitative performance comparisons against popular surface reconstruction, shape approximation, and floorplan generation approaches.

Title: Enhancing Multiview Synergy: Robust Learning by Exploiting the Wave Loss Function with Consensus and Complementarity Principles

• Authors: A. Quadir, Mushir Akhtar, M. Tanveer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.06819
• Pdf URL: https://arxiv.org/pdf/2408.06819
• Copy Paste: [[2408.06819]] Enhancing Multiview Synergy: Robust Learning by Exploiting the Wave Loss Function with Consensus and Complementarity Principles(https://arxiv.org/abs/2408.06819)
• Keywords: robust
• Abstract: Multiview learning (MvL) is an advancing domain in machine learning, leveraging multiple data perspectives to enhance model performance through view-consistency and view-discrepancy. Despite numerous successful multiview-based SVM models, existing frameworks predominantly focus on the consensus principle, often overlooking the complementarity principle. Furthermore, they exhibit limited robustness against noisy, error-prone, and view-inconsistent samples, prevalent in multiview datasets. To tackle the aforementioned limitations, this paper introduces Wave-MvSVM, a novel multiview support vector machine framework leveraging the wave loss (W-loss) function, specifically designed to harness both consensus and complementarity principles. Unlike traditional approaches that often overlook the complementary information among different views, the proposed Wave-MvSVM ensures a more comprehensive and resilient learning process by integrating both principles effectively. The W-loss function, characterized by its smoothness, asymmetry, and bounded nature, is particularly effective in mitigating the adverse effects of noisy and outlier data, thereby enhancing model stability. Theoretically, the W-loss function also exhibits a crucial classification-calibrated property, further boosting its effectiveness. Wave-MvSVM employs a between-view co-regularization term to enforce view consistency and utilizes an adaptive combination weight strategy to maximize the discriminative power of each view. The optimization problem is efficiently solved using a combination of GD and the ADMM, ensuring reliable convergence to optimal solutions. Theoretical analyses, grounded in Rademacher complexity, validate the generalization capabilities of the Wave-MvSVM model. Extensive empirical evaluations across diverse datasets demonstrate the superior performance of Wave-MvSVM in comparison to existing benchmark models.

Title: CRISP: Confidentiality, Rollback, and Integrity Storage Protection for Confidential Cloud-Native Computing

• Authors: Ardhi Putra Pratama Hartono, Andrey Brito, Christof Fetzer
• Subjects: cs.CR, cs.OS
• Abstract URL: https://arxiv.org/abs/2408.06822
• Pdf URL: https://arxiv.org/pdf/2408.06822
• Copy Paste: [[2408.06822]] CRISP: Confidentiality, Rollback, and Integrity Storage Protection for Confidential Cloud-Native Computing(https://arxiv.org/abs/2408.06822)
• Keywords: protect, attack
• Abstract: Trusted execution environments (TEEs) protect the integrity and confidentiality of running code and its associated data. Nevertheless, TEEs' integrity protection does not extend to the state saved on disk. Furthermore, modern cloud-native applications heavily rely on orchestration (e.g., through systems such as Kubernetes) and, thus, have their services frequently restarted. During restarts, attackers can revert the state of confidential services to a previous version that may aid their malicious intent. This paper presents CRISP, a rollback protection mechanism that uses an existing runtime for Intel SGX and transparently prevents rollback. Our approach can constrain the attack window to a fixed and short period or give developers the tools to avoid the vulnerability window altogether. Finally, experiments show that applying CRISP in a critical stateful cloud-native application may incur a resource increase but only a minor performance penalty.

Title: Membership Inference Attack Against Masked Image Modeling

• Authors: Zheng Li, Xinlei He, Ning Yu, Yang Zhang
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06825
• Pdf URL: https://arxiv.org/pdf/2408.06825
• Copy Paste: [[2408.06825]] Membership Inference Attack Against Masked Image Modeling(https://arxiv.org/abs/2408.06825)
• Keywords: privacy, attack, membership infer
• Abstract: Masked Image Modeling (MIM) has achieved significant success in the realm of self-supervised learning (SSL) for visual recognition. The image encoder pre-trained through MIM, involving the masking and subsequent reconstruction of input images, attains state-of-the-art performance in various downstream vision tasks. However, most existing works focus on improving the performance of this http URL this work, we take a different angle by studying the pre-training data privacy of MIM. Specifically, we propose the first membership inference attack against image encoders pre-trained by MIM, which aims to determine whether an image is part of the MIM pre-training dataset. The key design is to simulate the pre-training paradigm of MIM, i.e., image masking and subsequent reconstruction, and then obtain reconstruction errors. These reconstruction errors can serve as membership signals for achieving attack goals, as the encoder is more capable of reconstructing the input image in its training set with lower errors. Extensive evaluations are conducted on three model architectures and three benchmark datasets. Empirical results show that our attack outperforms baseline methods. Additionally, we undertake intricate ablation studies to analyze multiple factors that could influence the performance of the attack.

Title: FlatFusion: Delving into Details of Sparse Transformer-based Camera-LiDAR Fusion for Autonomous Driving

• Authors: Yutao Zhu, Xiaosong Jia, Xinyu Yang, Junchi Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06832
• Pdf URL: https://arxiv.org/pdf/2408.06832
• Copy Paste: [[2408.06832]] FlatFusion: Delving into Details of Sparse Transformer-based Camera-LiDAR Fusion for Autonomous Driving(https://arxiv.org/abs/2408.06832)
• Keywords: transformer
• Abstract: The integration of data from diverse sensor modalities (e.g., camera and LiDAR) constitutes a prevalent methodology within the ambit of autonomous driving scenarios. Recent advancements in efficient point cloud transformers have underscored the efficacy of integrating information in sparse formats. When it comes to fusion, since image patches are dense in pixel space with ambiguous depth, it necessitates additional design considerations for effective fusion. In this paper, we conduct a comprehensive exploration of design choices for Transformer-based sparse cameraLiDAR fusion. This investigation encompasses strategies for image-to-3D and LiDAR-to-2D mapping, attention neighbor grouping, single modal tokenizer, and micro-structure of Transformer. By amalgamating the most effective principles uncovered through our investigation, we introduce FlatFusion, a carefully designed framework for sparse camera-LiDAR fusion. Notably, FlatFusion significantly outperforms state-of-the-art sparse Transformer-based methods, including UniTR, CMT, and SparseFusion, achieving 73.7 NDS on the nuScenes validation set with 10.1 FPS with PyTorch.

Title: GLGait: A Global-Local Temporal Receptive Field Network for Gait Recognition in the Wild

• Authors: Guozhen Peng, Yunhong Wang, Yuwei Zhao, Shaoxiong Zhang, Annan Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06834
• Pdf URL: https://arxiv.org/pdf/2408.06834
• Copy Paste: [[2408.06834]] GLGait: A Global-Local Temporal Receptive Field Network for Gait Recognition in the Wild(https://arxiv.org/abs/2408.06834)
• Keywords: transformer
• Abstract: Gait recognition has attracted increasing attention from academia and industry as a human recognition technology from a distance in non-intrusive ways without requiring cooperation. Although advanced methods have achieved impressive success in lab scenarios, most of them perform poorly in the wild. Recently, some Convolution Neural Networks (ConvNets) based methods have been proposed to address the issue of gait recognition in the wild. However, the temporal receptive field obtained by convolution operations is limited for long gait sequences. If directly replacing convolution blocks with visual transformer blocks, the model may not enhance a local temporal receptive field, which is important for covering a complete gait cycle. To address this issue, we design a Global-Local Temporal Receptive Field Network (GLGait). GLGait employs a Global-Local Temporal Module (GLTM) to establish a global-local temporal receptive field, which mainly consists of a Pseudo Global Temporal Self-Attention (PGTA) and a temporal convolution operation. Specifically, PGTA is used to obtain a pseudo global temporal receptive field with less memory and computation complexity compared with a multi-head self-attention (MHSA). The temporal convolution operation is used to enhance the local temporal receptive field. Besides, it can also aggregate pseudo global temporal receptive field to a true holistic temporal receptive field. Furthermore, we also propose a Center-Augmented Triplet Loss (CTL) in GLGait to reduce the intra-class distance and expand the positive samples in the training stage. Extensive experiments show that our method obtains state-of-the-art results on in-the-wild datasets, $i.e.$, Gait3D and GREW. The code is available at this https URL.

Title: Dynamic and Compressive Adaptation of Transformers From Images to Videos

• Authors: Guozhen Zhang, Jingyu Liu, Shengming Cao, Xiaotong Zhao, Kevin Zhao, Kai Ma, Limin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06840
• Pdf URL: https://arxiv.org/pdf/2408.06840
• Copy Paste: [[2408.06840]] Dynamic and Compressive Adaptation of Transformers From Images to Videos(https://arxiv.org/abs/2408.06840)
• Keywords: transformer
• Abstract: Recently, the remarkable success of pre-trained Vision Transformers (ViTs) from image-text matching has sparked an interest in image-to-video adaptation. However, most current approaches retain the full forward pass for each frame, leading to a high computation overhead for processing entire videos. In this paper, we present InTI, a novel approach for compressive image-to-video adaptation using dynamic Inter-frame Token Interpolation. InTI aims to softly preserve the informative tokens without disrupting their coherent spatiotemporal structure. Specifically, each token pair at identical positions within neighbor frames is linearly aggregated into a new token, where the aggregation weights are generated by a multi-scale context-aware network. In this way, the information of neighbor frames can be adaptively compressed in a point-by-point manner, thereby effectively reducing the number of processed frames by half each time. Importantly, InTI can be seamlessly integrated with existing adaptation methods, achieving strong performance without extra-complex design. On Kinetics-400, InTI reaches a top-1 accuracy of 87.1 with a remarkable 37.5% reduction in GFLOPs compared to naive adaptation. When combined with additional temporal modules, InTI achieves a top-1 accuracy of 87.6 with a 37% reduction in GFLOPs. Similar conclusions have been verified in other common datasets.

Title: Improving WiFi CSI Fingerprinting with IQ Samples

• Authors: Junjie Wang (1), Yong Huang (1), Feiyang Zhao (1), Wenjing Wang (1), Dalong Zhang (1), Wei Wang (2) ((1) Zhengzhou University, Zhengzhou, China, (2) Huazhong University of Science and Technology, Wuhan, China)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.06848
• Pdf URL: https://arxiv.org/pdf/2408.06848
• Copy Paste: [[2408.06848]] Improving WiFi CSI Fingerprinting with IQ Samples(https://arxiv.org/abs/2408.06848)
• Keywords: security, extraction
• Abstract: Identity authentication is crucial for ensuring the information security of wireless communication. Radio frequency (RF) fingerprinting techniques provide a prom-ising supplement to cryptography-based authentication approaches but rely on dedicated equipment to capture in-phase and quadrature (IQ) samples, hindering their wide adoption. Recent advances advocate easily obtainable channel state in-formation (CSI) by commercial WiFi devices for lightweight RF fingerprinting, but they mainly focus on eliminating channel interference and cannot address the challenges of coarse granularity and information loss of CSI measurements. To overcome these challenges, we propose CSI2Q, a novel CSI fingerprinting sys-tem that achieves comparable performance to IQ-based approaches. Instead of ex-tracting fingerprints directly from raw CSI measurements, CSI2Q first transforms them into time-domain signals that share the same feature space with IQ samples. Then, the distinct advantages of an IQ fingerprinting model in feature extraction are transferred to its CSI counterpart via an auxiliary training strategy. Finally, the trained CSI fingerprinting model is used to decide which device the sample under test comes from. We evaluate CSI2Q on both synthetic and real CSI datasets. On the synthetic dataset, our system can improve the recognition accuracy from 76% to 91%. On the real dataset, CSI2Q boosts the accuracy from 67% to 82%.

Title: LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models

• Authors: Jia-Chen Zhang, Yu-Jie Xiong, He-Xi Qiu, Dong-Hai Zhu, Chun-Ming Xia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06854
• Pdf URL: https://arxiv.org/pdf/2408.06854
• Copy Paste: [[2408.06854]] LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models(https://arxiv.org/abs/2408.06854)
• Keywords: large language model
• Abstract: Fine-tuning large language models (LLMs) with high parameter efficiency for downstream tasks has become a new paradigm. Low-Rank Adaptation (LoRA) significantly reduces the number of trainable parameters for fine-tuning. Although it has demonstrated commendable performance, updating parameters within a single scale may not be the optimal choice for complex downstream this http URL this paper, we extend the LoRA to multiple scales, dubbed as LoRA$^2$. We first combine orthogonal projection theory to train a set of LoRAs in two mutually orthogonal planes. Then, we improve the importance score algorithm, which reduce parameter sensitivity score calculations by approximately 98.5\%. By pruning singular values with lower importance scores, thereby enhancing adaptability to various downstream tasks. Extensive experiments are conducted on two widely used pre-trained models to validate the effectiveness of LoRA$^2$. Results show that it significantly reduces the number of trainable parameters to just 0.72\% compared to full fine-tuning, while still delivering highly impressive performance. Even when the parameters are further reduced to 0.17M, it still achieves comparable results to the baseline with 8 times more parameters. Our code is available here: https://anonymous.4open.science/r/LoRA-2-5B4C

Title: Leveraging Language Models for Emotion and Behavior Analysis in Education

• Authors: Kaito Tanaka, Benjamin Tan, Brian Wong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06874
• Pdf URL: https://arxiv.org/pdf/2408.06874
• Copy Paste: [[2408.06874]] Leveraging Language Models for Emotion and Behavior Analysis in Education(https://arxiv.org/abs/2408.06874)
• Keywords: privacy, large language model
• Abstract: The analysis of students' emotions and behaviors is crucial for enhancing learning outcomes and personalizing educational experiences. Traditional methods often rely on intrusive visual and physiological data collection, posing privacy concerns and scalability issues. This paper proposes a novel method leveraging large language models (LLMs) and prompt engineering to analyze textual data from students. Our approach utilizes tailored prompts to guide LLMs in detecting emotional and engagement states, providing a non-intrusive and scalable solution. We conducted experiments using Qwen, ChatGPT, Claude2, and GPT-4, comparing our method against baseline models and chain-of-thought (CoT) prompting. Results demonstrate that our method significantly outperforms the baselines in both accuracy and contextual understanding. This study highlights the potential of LLMs combined with prompt engineering to offer practical and effective tools for educational emotion and behavior analysis.

Title: PBIR-NIE: Glossy Object Capture under Non-Distant Lighting

• Authors: Guangyan Cai, Fujun Luan, Miloš Hašan, Kai Zhang, Sai Bi, Zexiang Xu, Iliyan Georgiev, Shuang Zhao
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.06878
• Pdf URL: https://arxiv.org/pdf/2408.06878
• Copy Paste: [[2408.06878]] PBIR-NIE: Glossy Object Capture under Non-Distant Lighting(https://arxiv.org/abs/2408.06878)
• Keywords: robust
• Abstract: Glossy objects present a significant challenge for 3D reconstruction from multi-view input images under natural lighting. In this paper, we introduce PBIR-NIE, an inverse rendering framework designed to holistically capture the geometry, material attributes, and surrounding illumination of such objects. We propose a novel parallax-aware non-distant environment map as a lightweight and efficient lighting representation, accurately modeling the near-field background of the scene, which is commonly encountered in real-world capture setups. This feature allows our framework to accommodate complex parallax effects beyond the capabilities of standard infinite-distance environment maps. Our method optimizes an underlying signed distance field (SDF) through physics-based differentiable rendering, seamlessly connecting surface gradients between a triangle mesh and the SDF via neural implicit evolution (NIE). To address the intricacies of highly glossy BRDFs in differentiable rendering, we integrate the antithetic sampling algorithm to mitigate variance in the Monte Carlo gradient estimator. Consequently, our framework exhibits robust capabilities in handling glossy object reconstruction, showcasing superior quality in geometry, relighting, and material estimation.

Title: Voltran: Unlocking Trust and Confidentiality in Decentralized Federated Learning Aggregation

• Authors: Hao Wang, Yichen Cai, Jun Wang, Chuan Ma, Chunpeng Ge, Xiangmou Qu, Lu Zhou
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.06885
• Pdf URL: https://arxiv.org/pdf/2408.06885
• Copy Paste: [[2408.06885]] Voltran: Unlocking Trust and Confidentiality in Decentralized Federated Learning Aggregation(https://arxiv.org/abs/2408.06885)
• Keywords: robust, federate
• Abstract: The decentralized Federated Learning (FL) paradigm built upon blockchain architectures leverages distributed node clusters to replace the single server for executing FL model aggregation. This paradigm tackles the vulnerability of the centralized malicious server in vanilla FL and inherits the trustfulness and robustness offered by blockchain. However, existing blockchain-enabled schemes face challenges related to inadequate confidentiality on models and limited computational resources of blockchains to perform large-scale FL computations. In this paper, we present Voltran, an innovative hybrid platform designed to achieve trust, confidentiality, and robustness for FL based on the combination of the Trusted Execution Environment (TEE) and blockchain technology. We offload the FL aggregation computation into TEE to provide an isolated, trusted and customizable off-chain execution, and then guarantee the authenticity and verifiability of aggregation results on the blockchain. Moreover, we provide strong scalability on multiple FL scenarios by introducing a multi-SGX parallel execution strategy to amortize the large-scale FL workload. We implement a prototype of Voltran and conduct a comprehensive performance evaluation. Extensive experimental results demonstrate that Voltran incurs minimal additional overhead while guaranteeing trust, confidentiality, and authenticity, and it significantly brings a significant speed-up compared to state-of-the-art ciphertext aggregation schemes.

Title: BMFT: Achieving Fairness via Bias-based Weight Masking Fine-tuning

• Authors: Yuyang Xue, Junyu Yan, Raman Dutt, Fasih Haider, Jingshuai Liu, Steven McDonagh, Sotirios A. Tsaftaris
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06890
• Pdf URL: https://arxiv.org/pdf/2408.06890
• Copy Paste: [[2408.06890]] BMFT: Achieving Fairness via Bias-based Weight Masking Fine-tuning(https://arxiv.org/abs/2408.06890)
• Keywords: robust, fair
• Abstract: Developing models with robust group fairness properties is paramount, particularly in ethically sensitive domains such as medical diagnosis. Recent approaches to achieving fairness in machine learning require a substantial amount of training data and depend on model retraining, which may not be practical in real-world scenarios. To mitigate these challenges, we propose Bias-based Weight Masking Fine-Tuning (BMFT), a novel post-processing method that enhances the fairness of a trained model in significantly fewer epochs without requiring access to the original training data. BMFT produces a mask over model parameters, which efficiently identifies the weights contributing the most towards biased predictions. Furthermore, we propose a two-step debiasing strategy, wherein the feature extractor undergoes initial fine-tuning on the identified bias-influenced weights, succeeded by a fine-tuning phase on a reinitialised classification layer to uphold discriminative performance. Extensive experiments across four dermatological datasets and two sensitive attributes demonstrate that BMFT outperforms existing state-of-the-art (SOTA) techniques in both diagnostic accuracy and fairness metrics. Our findings underscore the efficacy and robustness of BMFT in advancing fairness across various out-of-distribution (OOD) settings. Our code is available at: this https URL

Title: Divide and Conquer: Improving Multi-Camera 3D Perception with 2D Semantic-Depth Priors and Input-Dependent Queries

• Authors: Qi Song, Qingyong Hu, Chi Zhang, Yongquan Chen, Rui Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06901
• Pdf URL: https://arxiv.org/pdf/2408.06901
• Copy Paste: [[2408.06901]] Divide and Conquer: Improving Multi-Camera 3D Perception with 2D Semantic-Depth Priors and Input-Dependent Queries(https://arxiv.org/abs/2408.06901)
• Keywords: transformer, segmentation
• Abstract: 3D perception tasks, such as 3D object detection and Bird's-Eye-View (BEV) segmentation using multi-camera images, have drawn significant attention recently. Despite the fact that accurately estimating both semantic and 3D scene layouts are crucial for this task, existing techniques often neglect the synergistic effects of semantic and depth cues, leading to the occurrence of classification and position estimation errors. Additionally, the input-independent nature of initial queries also limits the learning capacity of Transformer-based models. To tackle these challenges, we propose an input-aware Transformer framework that leverages Semantics and Depth as priors (named SDTR). Our approach involves the use of an S-D Encoder that explicitly models semantic and depth priors, thereby disentangling the learning process of object categorization and position estimation. Moreover, we introduce a Prior-guided Query Builder that incorporates the semantic prior into the initial queries of the Transformer, resulting in more effective input-aware queries. Extensive experiments on the nuScenes and Lyft benchmarks demonstrate the state-of-the-art performance of our method in both 3D object detection and BEV segmentation tasks.

Title: Re-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectives

• Authors: Zhihu Wang, Shiwan Zhao, Yu Wang, Heyuan Huang, Jiaxin Shi, Sitao Xie, Zhixing Wang, Yubo Zhang, Hongyan Li, Junchi Yan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06904
• Pdf URL: https://arxiv.org/pdf/2408.06904
• Copy Paste: [[2408.06904]] Re-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectives(https://arxiv.org/abs/2408.06904)
• Keywords: large language model
• Abstract: As large language models (LLMs) continue to scale, their enhanced performance often proves insufficient for solving domain-specific tasks. Systematically analyzing their failures and effectively enhancing their performance remain significant challenges. This paper introduces the Re-TASK framework, a novel theoretical model that Revisits LLM Tasks from cApability, Skill, Knowledge perspectives, guided by the principles of Bloom's Taxonomy and Knowledge Space Theory. The Re-TASK framework provides a systematic methodology to deepen our understanding, evaluation, and enhancement of LLMs for domain-specific tasks. It explores the interplay among an LLM's capabilities, the knowledge it processes, and the skills it applies, elucidating how these elements are interconnected and impact task performance. Our application of the Re-TASK framework reveals that many failures in domain-specific tasks can be attributed to insufficient knowledge or inadequate skill adaptation. With this insight, we propose structured strategies for enhancing LLMs through targeted knowledge injection and skill adaptation. Specifically, we identify key capability items associated with tasks and employ a deliberately designed prompting strategy to enhance task performance, thereby reducing the need for extensive fine-tuning. Alternatively, we fine-tune the LLM using capability-specific instructions, further validating the efficacy of our framework. Experimental results confirm the framework's effectiveness, demonstrating substantial improvements in both the performance and applicability of LLMs.

Title: Quantitative analysis of attack-fault trees via Markov decision processes

• Authors: Milan Lopuhaä-Zwakenberg
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.06914
• Pdf URL: https://arxiv.org/pdf/2408.06914
• Copy Paste: [[2408.06914]] Quantitative analysis of attack-fault trees via Markov decision processes(https://arxiv.org/abs/2408.06914)
• Keywords: security, attack
• Abstract: Adequate risk assessment of safety critical systems needs to take both safety and security into account, as well as their interaction. A prominent methodology for modeling safety and security are attack-fault trees (AFTs), which combine the well-established fault tree and attack tree methodologies for safety and security, respectively. AFTs can be used for quantitative analysis as well, capturing the interplay between safety and security metrics. However, existing approaches are based on modeling the AFT as a priced-timed automaton. This allows for a wide range of analyses, but Pareto analsis is still lacking, and analyses that exist are computationally expensive. In this paper, we combine safety and security analysis techniques to introduce a novel method to find the Pareto front between the metrics reliability (safety) and attack cost (security) using Markov decision processes. This gives us the full interplay between safety and security while being considerably more lightweight and faster than the automaton approach. We validate our approach on a case study of cyberattacks on an oil pipe line.

Title: Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas

• Authors: Louis Kwok, Michal Bravansky, Lewis D. Griffin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06929
• Pdf URL: https://arxiv.org/pdf/2408.06929
• Copy Paste: [[2408.06929]] Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas(https://arxiv.org/abs/2408.06929)
• Keywords: large language model
• Abstract: The success of Large Language Models (LLMs) in multicultural environments hinges on their ability to understand users' diverse cultural backgrounds. We measure this capability by having an LLM simulate human profiles representing various nationalities within the scope of a questionnaire-style psychological experiment. Specifically, we employ GPT-3.5 to reproduce reactions to persuasive news articles of 7,286 participants from 15 countries; comparing the results with a dataset of real participants sharing the same demographic traits. Our analysis shows that specifying a person's country of residence improves GPT-3.5's alignment with their responses. In contrast, using native language prompting introduces shifts that significantly reduce overall alignment, with some languages particularly impairing performance. These findings suggest that while direct nationality information enhances the model's cultural adaptability, native language cues do not reliably improve simulation fidelity and can detract from the model's effectiveness.

Title: Diagnosis extraction from unstructured Dutch echocardiogram reports using span- and document-level characteristic classification

• Authors: Bauke Arends, Melle Vessies, Dirk van Osch, Arco Teske, Pim van der Harst, René van Es, Bram van Es
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06930
• Pdf URL: https://arxiv.org/pdf/2408.06930
• Copy Paste: [[2408.06930]] Diagnosis extraction from unstructured Dutch echocardiogram reports using span- and document-level characteristic classification(https://arxiv.org/abs/2408.06930)
• Keywords: extraction
• Abstract: Clinical machine learning research and AI driven clinical decision support models rely on clinically accurate labels. Manually extracting these labels with the help of clinical specialists is often time-consuming and expensive. This study tests the feasibility of automatic span- and document-level diagnosis extraction from unstructured Dutch echocardiogram reports. We included 115,692 unstructured echocardiogram reports from the UMCU a large university hospital in the Netherlands. A randomly selected subset was manually annotated for the occurrence and severity of eleven commonly described cardiac characteristics. We developed and tested several automatic labelling techniques at both span and document levels, using weighted and macro F1-score, precision, and recall for performance evaluation. We compared the performance of span labelling against document labelling methods, which included both direct document classifiers and indirect document classifiers that rely on span classification results. The SpanCategorizer and this http URL models outperformed all other span and document classifiers, respectively. The weighted F1-score varied between characteristics, ranging from 0.60 to 0.93 in SpanCategorizer and 0.96 to 0.98 in this http URL. Direct document classification was superior to indirect document classification using span classifiers. SetFit achieved competitive document classification performance using only 10\% of the training data. Utilizing a reduced label set yielded near-perfect document classification results. We recommend using our published SpanCategorizer and this http URL models for span- and document-level diagnosis extraction from Dutch echocardiography reports. For settings with limited training data, SetFit may be a promising alternative for document classification.

Title: The advantages of context specific language models: the case of the Erasmian Language Model

• Authors: João Gonçalves, Nick Jelicic, Michele Murgia, Evert Stamhuis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06931
• Pdf URL: https://arxiv.org/pdf/2408.06931
• Copy Paste: [[2408.06931]] The advantages of context specific language models: the case of the Erasmian Language Model(https://arxiv.org/abs/2408.06931)
• Keywords: privacy
• Abstract: The current trend to improve language model performance seems to be based on scaling up with the number of parameters (e.g. the state of the art GPT4 model has approximately 1.7 trillion parameters) or the amount of training data fed into the model. However this comes at significant costs in terms of computational resources and energy costs that compromise the sustainability of AI solutions, as well as risk relating to privacy and misuse. In this paper we present the Erasmian Language Model (ELM) a small context specific, 900 million parameter model, pre-trained and fine-tuned by and for Erasmus University Rotterdam. We show how the model performs adequately in a classroom context for essay writing, and how it achieves superior performance in subjects that are part of its context. This has implications for a wide range of institutions and organizations, showing that context specific language models may be a viable alternative for resource constrained, privacy sensitive use cases.

Title: PayOff: A Regulated Central Bank Digital Currency with Private Offline Payments

• Authors: Carolin Beer, Sheila Zingg, Kari Kostiainen, Karl Wüst, Vedran Capkun, Srdjan Capkun
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.06956
• Pdf URL: https://arxiv.org/pdf/2408.06956
• Copy Paste: [[2408.06956]] PayOff: A Regulated Central Bank Digital Currency with Private Offline Payments(https://arxiv.org/abs/2408.06956)
• Keywords: secure, security, privacy, protect, robust
• Abstract: The European Central Bank is preparing for the potential issuance of a central bank digital currency (CBDC), called the digital euro. A recent regulatory proposal by the European Commission defines several requirements for the digital euro, such as support for both online and offline payments. Offline payments are expected to enable cash-like privacy, local payment settlement, and the enforcement of holding limits. While other central banks have expressed similar desired functionality, achieving such offline payments poses a novel technical challenge. We observe that none of the existing research solutions, including offline E-cash schemes, are fully compliant. Proposed solutions based on secure elements offer no guarantees in case of compromise and can therefore lead to significant payment fraud. The main contribution of this paper is PayOff, a novel CBDC design motivated by the digital euro regulation, which focuses on offline payments. We analyze the security implications of local payment settlement and identify new security objectives. PayOff protects user privacy, supports complex regulations such as holding limits, and implements safeguards to increase robustness against secure element failure. Our analysis shows that PayOff provides strong privacy and identifies residual leakages that may arise in real-world deployments. Our evaluation shows that offline payments can be fast and that the central bank can handle high payment loads with moderate computing resources. However, the main limitation of PayOff is that offline payment messages and storage requirements grow in the number of payments that the sender makes or receives without going online in between.

Title: DyG-Mamba: Continuous State Space Modeling on Dynamic Graphs

• Authors: Dongyuan Li, Shiyin Tan, Ying Zhang, Ming Jin, Shirui Pan, Manabu Okumura, Renhe Jiang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.06966
• Pdf URL: https://arxiv.org/pdf/2408.06966
• Copy Paste: [[2408.06966]] DyG-Mamba: Continuous State Space Modeling on Dynamic Graphs(https://arxiv.org/abs/2408.06966)
• Keywords: robust
• Abstract: Dynamic graph learning aims to uncover evolutionary laws in real-world systems, enabling accurate social recommendation (link prediction) or early detection of cancer cells (classification). Inspired by the success of state space models, e.g., Mamba, for efficiently capturing long-term dependencies in language modeling, we propose DyG-Mamba, a new continuous state space model (SSM) for dynamic graph learning. Specifically, we first found that using inputs as control signals for SSM is not suitable for continuous-time dynamic network data with irregular sampling intervals, resulting in models being insensitive to time information and lacking generalization properties. Drawing inspiration from the Ebbinghaus forgetting curve, which suggests that memory of past events is strongly correlated with time intervals rather than specific details of the events themselves, we directly utilize irregular time spans as control signals for SSM to achieve significant robustness and generalization. Through exhaustive experiments on 12 datasets for dynamic link prediction and dynamic node classification tasks, we found that DyG-Mamba achieves state-of-the-art performance on most of the datasets, while also demonstrating significantly improved computation and memory efficiency.

Title: Prompt-Based Segmentation at Multiple Resolutions and Lighting Conditions using Segment Anything Model 2

• Authors: Osher Rafaeli, Tal Svoray, Ariel Nahlieli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06970
• Pdf URL: https://arxiv.org/pdf/2408.06970
• Copy Paste: [[2408.06970]] Prompt-Based Segmentation at Multiple Resolutions and Lighting Conditions using Segment Anything Model 2(https://arxiv.org/abs/2408.06970)
• Keywords: robust, segmentation
• Abstract: This paper provides insight into the effectiveness of zero-shot, prompt-based, Segment Anything Model (SAM), and its updated version, SAM 2, and the non-promptable, conventional convolutional network (CNN), in segmenting solar panels, in RGB aerial imagery, across lighting conditions, spatial resolutions, and prompt strategies. SAM 2 demonstrates improvements over SAM, particularly in sub-optimal lighting conditions when prompted by points. Both SAMs, prompted by user-box, outperformed CNN, in all scenarios. Additionally, YOLOv9 prompting outperformed user points prompting. In high-resolution imagery, both in optimal and sub-optimal lighting conditions, Eff-UNet outperformed both SAM models prompted by YOLOv9 boxes, positioning Eff-UNet as the appropriate model for automatic segmentation in high-resolution data. In low-resolution data, user box prompts were found crucial to achieve a reasonable performance. This paper provides details on strengths and limitations of each model and outlines robustness of user prompted image segmentation models in inconsistent resolution and lighting conditions of remotely sensed data.

Title: SpectralGaussians: Semantic, spectral 3D Gaussian splatting for multi-spectral scene representation, visualization and analysis

• Authors: Saptarshi Neil Sinha, Holger Graf, Michael Weinmann
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.06975
• Pdf URL: https://arxiv.org/pdf/2408.06975
• Copy Paste: [[2408.06975]] SpectralGaussians: Semantic, spectral 3D Gaussian splatting for multi-spectral scene representation, visualization and analysis(https://arxiv.org/abs/2408.06975)
• Keywords: segmentation
• Abstract: We propose a novel cross-spectral rendering framework based on 3D Gaussian Splatting (3DGS) that generates realistic and semantically meaningful splats from registered multi-view spectrum and segmentation maps. This extension enhances the representation of scenes with multiple spectra, providing insights into the underlying materials and segmentation. We introduce an improved physically-based rendering approach for Gaussian splats, estimating reflectance and lights per spectra, thereby enhancing accuracy and realism. In a comprehensive quantitative and qualitative evaluation, we demonstrate the superior performance of our approach with respect to other recent learning-based spectral scene representation approaches (i.e., XNeRF and SpectralNeRF) as well as other non-spectral state-of-the-art learning-based approaches. Our work also demonstrates the potential of spectral scene understanding for precise scene editing techniques like style transfer, inpainting, and removal. Thereby, our contributions address challenges in multi-spectral scene representation, rendering, and editing, offering new possibilities for diverse applications.

Title: Low-Bitwidth Floating Point Quantization for Efficient High-Quality Diffusion Models

• Authors: Cheng Chen, Christina Giannoula, Andreas Moshovos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06995
• Pdf URL: https://arxiv.org/pdf/2408.06995
• Copy Paste: [[2408.06995]] Low-Bitwidth Floating Point Quantization for Efficient High-Quality Diffusion Models(https://arxiv.org/abs/2408.06995)
• Keywords: diffusion
• Abstract: Diffusion models are emerging models that generate images by iteratively denoising random Gaussian noise using deep neural networks. These models typically exhibit high computational and memory demands, necessitating effective post-training quantization for high-performance inference. Recent works propose low-bitwidth (e.g., 8-bit or 4-bit) quantization for diffusion models, however 4-bit integer quantization typically results in low-quality images. We observe that on several widely used hardware platforms, there is little or no difference in compute capability between floating-point and integer arithmetic operations of the same bitwidth (e.g., 8-bit or 4-bit). Therefore, we propose an effective floating-point quantization method for diffusion models that provides better image quality compared to integer quantization methods. We employ a floating-point quantization method that was effective for other processing tasks, specifically computer vision and natural language tasks, and tailor it for diffusion models by integrating weight rounding learning during the mapping of the full-precision values to the quantized values in the quantization process. We comprehensively study integer and floating-point quantization methods in state-of-the-art diffusion models. Our floating-point quantization method not only generates higher-quality images than that of integer quantization methods, but also shows no noticeable degradation compared to full-precision models (32-bit floating-point), when both weights and activations are quantized to 8-bit floating-point values, while has minimal degradation with 4-bit weights and 8-bit activations.

Title: Generative AI for automatic topic labelling

• Authors: Diego Kozlowski, Carolina Pradier, Pierre Benz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07003
• Pdf URL: https://arxiv.org/pdf/2408.07003
• Copy Paste: [[2408.07003]] Generative AI for automatic topic labelling(https://arxiv.org/abs/2408.07003)
• Keywords: generative
• Abstract: Topic Modeling has become a prominent tool for the study of scientific fields, as they allow for a large scale interpretation of research trends. Nevertheless, the output of these models is structured as a list of keywords which requires a manual interpretation for the labelling. This paper proposes to assess the reliability of three LLMs, namely flan, GPT-4o, and GPT-4 mini for topic labelling. Drawing on previous research leveraging BERTopic, we generate topics from a dataset of all the scientific articles (n=34,797) authored by all biology professors in Switzerland (n=465) between 2008 and 2020, as recorded in the Web of Science database. We assess the output of the three models both quantitatively and qualitatively and find that, first, both GPT models are capable of accurately and precisely label topics from the models' output keywords. Second, 3-word labels are preferable to grasp the complexity of research topics.

Title: Casper: Prompt Sanitization for Protecting User Privacy in Web-Based Large Language Models

• Authors: Chun Jie Chong, Chenxi Hou, Zhihao Yao, Seyed Mohammadjavad Seyed Talebi
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07004
• Pdf URL: https://arxiv.org/pdf/2408.07004
• Copy Paste: [[2408.07004]] Casper: Prompt Sanitization for Protecting User Privacy in Web-Based Large Language Models(https://arxiv.org/abs/2408.07004)
• Keywords: privacy, protect, large language model
• Abstract: Web-based Large Language Model (LLM) services have been widely adopted and have become an integral part of our Internet experience. Third-party plugins enhance the functionalities of LLM by enabling access to real-world data and services. However, the privacy consequences associated with these services and their third-party plugins are not well understood. Sensitive prompt data are stored, processed, and shared by cloud-based LLM providers and third-party plugins. In this paper, we propose Casper, a prompt sanitization technique that aims to protect user privacy by detecting and removing sensitive information from user inputs before sending them to LLM services. Casper runs entirely on the user's device as a browser extension and does not require any changes to the online LLM services. At the core of Casper is a three-layered sanitization mechanism consisting of a rule-based filter, a Machine Learning (ML)-based named entity recognizer, and a browser-based local LLM topic identifier. We evaluate Casper on a dataset of 4000 synthesized prompts and show that it can effectively filter out Personal Identifiable Information (PII) and privacy-sensitive topics with high accuracy, at 98.5% and 89.9%, respectively.

Title: Imagen 3

• Authors: Imagen-Team-Google: Jason Baldridge, Jakob Bauer, Mukul Bhutani, Nicole Brichtova, Andrew Bunner, Kelvin Chan, Yichang Chen, Sander Dieleman, Yuqing Du, Zach Eaton-Rosen, Hongliang Fei, Nando de Freitas, Yilin Gao, Evgeny Gladchenko, Sergio Gómez Colmenarejo, Mandy Guo, Alex Haig, Will Hawkins, Hexiang Hu, Huilian Huang, Tobenna Peter Igwe, Christos Kaplanis, Siavash Khodadadeh, Yelin Kim, Ksenia Konyushkova, Karol Langner, Eric Lau, Shixin Luo, Soňa Mokrá, Henna Nandwani, Yasumasa Onoe, Aäron van den Oord, Zarana Parekh, Jordi Pont-Tuset, Hang Qi, Rui Qian, Deepak Ramachandran, Poorva Rane, Abdullah Rashwan, Ali Razavi, Robert Riachi, Hansa Srinivasan, Srivatsan Srinivasan, Robin Strudel, Benigno Uria, Oliver Wang, Su Wang, Austin Waters, Chris Wolff, Auriel Wright, Zhisheng Xiao, Hao Xiong, Keyang Xu, Marc van Zee, Junlin Zhang, Katie Zhang, Wenlei Zhou, Konrad Zolna, Ola Aboubakar, Canfer Akbulut, Oscar Akerlund, Isabela Albuquerque, Nina Anderson, Marco Andreetto, Lora Aroyo, Ben Bariach, David Barker, Sherry Ben, Dana Berman, Courtney Biles, Irina Blok, Pankil Botadra, Jenny Brennan, Karla Brown, John Buckley, Rudy Bunel, Elie Bursztein, Christina Butterfield, Ben Caine, Viral Carpenter, Norman Casagrande, Ming-Wei Chang, Solomon Chang, Shamik Chaudhuri, Tony Chen, John Choi, Dmitry Churbanau, Nathan Clement, Matan Cohen, Forrester Cole, Mikhail Dektiarev, Vincent Du, Praneet Dutta, Tom Eccles, Ndidi Elue, Ashley Feden, Shlomi Fruchter, Frankie Garcia, Roopal Garg
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07009
• Pdf URL: https://arxiv.org/pdf/2408.07009
• Copy Paste: [[2408.07009]] Imagen 3(https://arxiv.org/abs/2408.07009)
• Keywords: diffusion
• Abstract: We introduce Imagen 3, a latent diffusion model that generates high quality images from text prompts. We describe our quality and responsibility evaluations. Imagen 3 is preferred over other state-of-the-art (SOTA) models at the time of evaluation. In addition, we discuss issues around safety and representation, as well as methods we used to minimize the potential harm of our models.

Title: Improved Counting under Continual Observation with Pure Differential Privacy

• Authors: Joel Daniel Andersson, Rasmus Pagh, Sahel Torkamani
• Subjects: cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2408.07021
• Pdf URL: https://arxiv.org/pdf/2408.07021
• Copy Paste: [[2408.07021]] Improved Counting under Continual Observation with Pure Differential Privacy(https://arxiv.org/abs/2408.07021)
• Keywords: privacy, protect
• Abstract: Counting under continual observation is a well-studied problem in the area of differential privacy. Given a stream of updates $x_1,x_2,\dots,x_T \in \{0,1\}$ the problem is to continuously release estimates of the prefix sums $\sum_{i=1}^t x_i$ for $t=1,\dots,T$ while protecting each input $x_i$ in the stream with differential privacy. Recently, significant leaps have been made in our understanding of this problem under $\textit{approximate}$ differential privacy, aka. $(\varepsilon,\delta)$$\textit{-differential privacy}$. However, for the classical case of $\varepsilon$-differential privacy, we are not aware of any improvement in mean squared error since the work of Honaker (TPDP 2015). In this paper we present such an improvement, reducing the mean squared error by a factor of about 4, asymptotically. The key technique is a new generalization of the binary tree mechanism that uses a $k$-ary number system with $\textit{negative digits}$ to improve the privacy-accuracy trade-off. Our mechanism improves the mean squared error over all 'optimal' $(\varepsilon,\delta)$-differentially private factorization mechanisms based on Gaussian noise whenever $\delta$ is sufficiently small. Specifically, using $k=19$ we get an asymptotic improvement over the bound given in the work by Henzinger, Upadhyay and Upadhyay (SODA 2023) when $\delta = O(T^{-0.92})$.

Title: KAN You See It? KANs and Sentinel for Effective and Explainable Crop Field Segmentation

• Authors: Daniele Rege Cambrin, Eleonora Poeta, Eliana Pastor, Tania Cerquitelli, Elena Baralis, Paolo Garza
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07040
• Pdf URL: https://arxiv.org/pdf/2408.07040
• Copy Paste: [[2408.07040]] KAN You See It? KANs and Sentinel for Effective and Explainable Crop Field Segmentation(https://arxiv.org/abs/2408.07040)
• Keywords: explainability, segmentation
• Abstract: Segmentation of crop fields is essential for enhancing agricultural productivity, monitoring crop health, and promoting sustainable practices. Deep learning models adopted for this task must ensure accurate and reliable predictions to avoid economic losses and environmental impact. The newly proposed Kolmogorov-Arnold networks (KANs) offer promising advancements in the performance of neural networks. This paper analyzes the integration of KAN layers into the U-Net architecture (U-KAN) to segment crop fields using Sentinel-2 and Sentinel-1 satellite images and provides an analysis of the performance and explainability of these networks. Our findings indicate a 2\% improvement in IoU compared to the traditional full-convolutional U-Net model in fewer GFLOPs. Furthermore, gradient-based explanation techniques show that U-KAN predictions are highly plausible and that the network has a very high ability to focus on the boundaries of cultivated areas rather than on the areas themselves. The per-channel relevance analysis also reveals that some channels are irrelevant to this task.

Title: TableGuard -- Securing Structured & Unstructured Data

• Authors: Anantha Sharma, Ajinkya Deshmukh
• Subjects: cs.CR, cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07045
• Pdf URL: https://arxiv.org/pdf/2408.07045
• Copy Paste: [[2408.07045]] TableGuard -- Securing Structured & Unstructured Data(https://arxiv.org/abs/2408.07045)
• Keywords: secure, security, privacy, protect, robust, transformer
• Abstract: With the increasing demand for data sharing across platforms and organizations, ensuring the privacy and security of sensitive information has become a critical challenge. This paper introduces "TableGuard". An innovative approach to data obfuscation tailored for relational databases. Building on the principles and techniques developed in prior work on context-sensitive obfuscation, TableGuard applies these methods to ensure that API calls return only obfuscated data, thereby safeguarding privacy when sharing data with third parties. TableGuard leverages advanced context-sensitive obfuscation techniques to replace sensitive data elements with contextually appropriate alternatives. By maintaining the relational integrity and coherence of the data, our approach mitigates the risks of cognitive dissonance and data leakage. We demonstrate the implementation of TableGuard using a BERT based transformer model, which identifies and obfuscates sensitive entities within relational tables. Our evaluation shows that TableGuard effectively balances privacy protection with data utility, minimizing information loss while ensuring that the obfuscated data remains functionally useful for downstream applications. The results highlight the importance of domain-specific obfuscation strategies and the role of context length in preserving data integrity. The implications of this research are significant for organizations that need to share data securely with external parties. TableGuard offers a robust framework for implementing privacy-preserving data sharing mechanisms, thereby contributing to the broader field of data privacy and security.

Title: Exploiting Leakage in Password Managers via Injection Attacks

• Authors: Andrés Fábrega, Armin Namavari, Rachit Agarwal, Ben Nassi, Thomas Ristenpart
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.07054
• Pdf URL: https://arxiv.org/pdf/2408.07054
• Copy Paste: [[2408.07054]] Exploiting Leakage in Password Managers via Injection Attacks(https://arxiv.org/abs/2408.07054)
• Keywords: protect, attack
• Abstract: This work explores injection attacks against password managers. In this setting, the adversary (only) controls their own application client, which they use to "inject" chosen payloads to a victim's client via, for example, sharing credentials with them. The injections are interleaved with adversarial observations of some form of protected state (such as encrypted vault exports or the network traffic received by the application servers), from which the adversary backs out confidential information. We uncover a series of general design patterns in popular password managers that lead to vulnerabilities allowing an adversary to efficiently recover passwords, URLs, usernames, and attachments. We develop general attack templates to exploit these design patterns and experimentally showcase their practical efficacy via analysis of ten distinct password manager applications. We disclosed our findings to these vendors, many of which deployed mitigations.

Title: LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs

• Authors: Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07055
• Pdf URL: https://arxiv.org/pdf/2408.07055
• Copy Paste: [[2408.07055]] LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs(https://arxiv.org/abs/2408.07055)
• Keywords: large language model
• Abstract: Current long context large language models (LLMs) can process inputs up to 100,000 tokens, yet struggle to generate outputs exceeding even a modest length of 2,000 words. Through controlled experiments, we find that the model's effective generation length is inherently bounded by the sample it has seen during supervised fine-tuning (SFT). In other words, their output limitation is due to the scarcity of long-output examples in existing SFT datasets. To address this, we introduce AgentWrite, an agent-based pipeline that decomposes ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we construct LongWriter-6k, a dataset containing 6,000 SFT data with output lengths ranging from 2k to 32k words. By incorporating this dataset into model training, we successfully scale the output length of existing models to over 10,000 words while maintaining output quality. We also develop LongBench-Write, a comprehensive benchmark for evaluating ultra-long generation capabilities. Our 9B parameter model, further improved through DPO, achieves state-of-the-art performance on this benchmark, surpassing even much larger proprietary models. In general, our work demonstrates that existing long context LLM already possesses the potential for a larger output window--all you need is data with extended output during model alignment to unlock this capability. Our code & models are at: this https URL.