2024-08-16

Title: Impact of Inaccurate Contamination Ratio on Robust Unsupervised Anomaly Detection

• Authors: Jordan F. Masakuna, DJeff Kanda Nkashama, Arian Soltani, Marc Frappier, Pierre-Martin Tardif, Froduald Kabanza
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07718
• Pdf URL: https://arxiv.org/pdf/2408.07718
• Copy Paste: [[2408.07718]] Impact of Inaccurate Contamination Ratio on Robust Unsupervised Anomaly Detection(https://arxiv.org/abs/2408.07718)
• Keywords: robust
• Abstract: Training data sets intended for unsupervised anomaly detection, typically presumed to be anomaly-free, often contain anomalies (or contamination), a challenge that significantly undermines model performance. Most robust unsupervised anomaly detection models rely on contamination ratio information to tackle contamination. However, in reality, contamination ratio may be inaccurate. We investigate on the impact of inaccurate contamination ratio information in robust unsupervised anomaly detection. We verify whether they are resilient to misinformed contamination ratios. Our investigation on 6 benchmark data sets reveals that such models are not adversely affected by exposure to misinformation. In fact, they can exhibit improved performance when provided with such inaccurate contamination ratios.

Title: Moderator: Moderating Text-to-Image Diffusion Models through Fine-grained Context-based Policies

• Authors: Peiran Wang, Qiyu Li, Longxuan Yu, Ziyao Wang, Ang Li, Haojian Jin
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.07728
• Pdf URL: https://arxiv.org/pdf/2408.07728
• Copy Paste: [[2408.07728]] Moderator: Moderating Text-to-Image Diffusion Models through Fine-grained Context-based Policies(https://arxiv.org/abs/2408.07728)
• Keywords: diffusion
• Abstract: We present Moderator, a policy-based model management system that allows administrators to specify fine-grained content moderation policies and modify the weights of a text-to-image (TTI) model to make it significantly more challenging for users to produce images that violate the policies. In contrast to existing general-purpose model editing techniques, which unlearn concepts without considering the associated contexts, Moderator allows admins to specify what content should be moderated, under which context, how it should be moderated, and why moderation is necessary. Given a set of policies, Moderator first prompts the original model to generate images that need to be moderated, then uses these self-generated images to reverse fine-tune the model to compute task vectors for moderation and finally negates the original model with the task vectors to decrease its performance in generating moderated content. We evaluated Moderator with 14 participants to play the role of admins and found they could quickly learn and author policies to pass unit tests in approximately 2.29 policy iterations. Our experiment with 32 stable diffusion users suggested that Moderator can prevent 65% of users from generating moderated content under 15 attempts and require the remaining users an average of 8.3 times more attempts to generate undesired content.

Title: Extending Network Intrusion Detection with Enhanced Particle Swarm Optimization Techniques

• Authors: Surasit Songma, Watcharakorn Netharn, Siriluck Lorpunmanee
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.07729
• Pdf URL: https://arxiv.org/pdf/2408.07729
• Copy Paste: [[2408.07729]] Extending Network Intrusion Detection with Enhanced Particle Swarm Optimization Techniques(https://arxiv.org/abs/2408.07729)
• Keywords: security, robust, segmentation
• Abstract: The present research investigates how to improve Network Intrusion Detection Systems (NIDS) by combining Machine Learning (ML) and Deep Learning (DL) techniques, addressing the growing challenge of cybersecurity threats. A thorough process for data preparation, comprising activities like cleaning, normalization, and segmentation into training and testing sets, lays the framework for model training and evaluation. The study uses the CSE-CIC-IDS 2018 and LITNET-2020 datasets to compare ML methods (Decision Trees, Random Forest, XGBoost) and DL models (CNNs, RNNs, DNNs, MLP) against key performance metrics (Accuracy, Precision, Recall, and F1-Score). The Decision Tree model performed better across all measures after being fine-tuned with Enhanced Particle Swarm Optimization (EPSO), demonstrating the model's ability to detect network breaches effectively. The findings highlight EPSO's importance in improving ML classifiers for cybersecurity, proposing a strong framework for NIDS with high precision and dependability. This extensive analysis not only contributes to the cybersecurity arena by providing a road to robust intrusion detection solutions, but it also proposes future approaches for improving ML models to combat the changing landscape of network threats.

Title: Enhancing Adversarial Attacks via Parameter Adaptive Adversarial Attack

• Authors: Zhibo Jin, Jiayu Zhang, Zhiyu Zhu, Chenyu Zhang, Jiahao Huang, Jianlong Zhou, Fang Chen
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.07733
• Pdf URL: https://arxiv.org/pdf/2408.07733
• Copy Paste: [[2408.07733]] Enhancing Adversarial Attacks via Parameter Adaptive Adversarial Attack(https://arxiv.org/abs/2408.07733)
• Keywords: attack
• Abstract: In recent times, the swift evolution of adversarial attacks has captured widespread attention, particularly concerning their transferability and other performance attributes. These techniques are primarily executed at the sample level, frequently overlooking the intrinsic parameters of models. Such neglect suggests that the perturbations introduced in adversarial samples might have the potential for further reduction. Given the essence of adversarial attacks is to impair model integrity with minimal noise on original samples, exploring avenues to maximize the utility of such perturbations is imperative. Against this backdrop, we have delved into the complexities of adversarial attack algorithms, dissecting the adversarial process into two critical phases: the Directional Supervision Process (DSP) and the Directional Optimization Process (DOP). While DSP determines the direction of updates based on the current samples and model parameters, it has been observed that existing model parameters may not always be conducive to adversarial attacks. The impact of models on adversarial efficacy is often overlooked in current research, leading to the neglect of DSP. We propose that under certain conditions, fine-tuning model parameters can significantly enhance the quality of DSP. For the first time, we propose that under certain conditions, fine-tuning model parameters can significantly improve the quality of the DSP. We provide, for the first time, rigorous mathematical definitions and proofs for these conditions, and introduce multiple methods for fine-tuning model parameters within DSP. Our extensive experiments substantiate the effectiveness of the proposed P3A method. Our code is accessible at: https://anonymous.4open.science/r/P3A-A12C/

Title: Enhancing Model Interpretability with Local Attribution over Global Exploration

• Authors: Zhiyu Zhu, Zhibo Jin, Jiayu Zhang, Huaming Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07736
• Pdf URL: https://arxiv.org/pdf/2408.07736
• Copy Paste: [[2408.07736]] Enhancing Model Interpretability with Local Attribution over Global Exploration(https://arxiv.org/abs/2408.07736)
• Keywords: interpretability
• Abstract: In the field of artificial intelligence, AI models are frequently described as `black boxes' due to the obscurity of their internal mechanisms. It has ignited research interest on model interpretability, especially in attribution methods that offers precise explanations of model decisions. Current attribution algorithms typically evaluate the importance of each parameter by exploring the sample space. A large number of intermediate states are introduced during the exploration process, which may reach the model's Out-of-Distribution (OOD) space. Such intermediate states will impact the attribution results, making it challenging to grasp the relative importance of features. In this paper, we firstly define the local space and its relevant properties, and we propose the Local Attribution (LA) algorithm that leverages these properties. The LA algorithm comprises both targeted and untargeted exploration phases, which are designed to effectively generate intermediate states for attribution that thoroughly encompass the local space. Compared to the state-of-the-art attribution methods, our approach achieves an average improvement of 38.21\% in attribution effectiveness. Extensive ablation studies in our experiments also validate the significance of each component in our algorithm. Our code is available at: this https URL

Title: Out-of-Distribution Learning with Human Feedback

• Authors: Haoyue Bai, Xuefeng Du, Katie Rainey, Shibin Parameswaran, Yixuan Li
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.07772
• Pdf URL: https://arxiv.org/pdf/2408.07772
• Copy Paste: [[2408.07772]] Out-of-Distribution Learning with Human Feedback(https://arxiv.org/abs/2408.07772)
• Keywords: robust
• Abstract: Out-of-distribution (OOD) learning often relies heavily on statistical approaches or predefined assumptions about OOD data distributions, hindering their efficacy in addressing multifaceted challenges of OOD generalization and OOD detection in real-world deployment environments. This paper presents a novel framework for OOD learning with human feedback, which can provide invaluable insights into the nature of OOD shifts and guide effective model adaptation. Our framework capitalizes on the freely available unlabeled data in the wild that captures the environmental test-time OOD distributions under both covariate and semantic shifts. To harness such data, our key idea is to selectively provide human feedback and label a small number of informative samples from the wild data distribution, which are then used to train a multi-class classifier and an OOD detector. By exploiting human feedback, we enhance the robustness and reliability of machine learning models, equipping them with the capability to handle OOD scenarios with greater precision. We provide theoretical insights on the generalization error bounds to justify our algorithm. Extensive experiments show the superiority of our method, outperforming the current state-of-the-art by a significant margin.

Title: MedTsLLM: Leveraging LLMs for Multimodal Medical Time Series Analysis

• Authors: Nimeesha Chan, Felix Parker, William Bennett, Tianyi Wu, Mung Yao Jia, James Fackler, Kimia Ghobadi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07773
• Pdf URL: https://arxiv.org/pdf/2408.07773
• Copy Paste: [[2408.07773]] MedTsLLM: Leveraging LLMs for Multimodal Medical Time Series Analysis(https://arxiv.org/abs/2408.07773)
• Keywords: large language model, segmentation
• Abstract: The complexity and heterogeneity of data in many real-world applications pose significant challenges for traditional machine learning and signal processing techniques. For instance, in medicine, effective analysis of diverse physiological signals is crucial for patient monitoring and clinical decision-making and yet highly challenging. We introduce MedTsLLM, a general multimodal large language model (LLM) framework that effectively integrates time series data and rich contextual information in the form of text to analyze physiological signals, performing three tasks with clinical relevance: semantic segmentation, boundary detection, and anomaly detection in time series. These critical tasks enable deeper analysis of physiological signals and can provide actionable insights for clinicians. We utilize a reprogramming layer to align embeddings of time series patches with a pretrained LLM's embedding space and make effective use of raw time series, in conjunction with textual context. Given the multivariate nature of medical datasets, we develop methods to handle multiple covariates. We additionally tailor the text prompt to include patient-specific information. Our model outperforms state-of-the-art baselines, including deep learning models, other LLMs, and clinical methods across multiple medical domains, specifically electrocardiograms and respiratory waveforms. MedTsLLM presents a promising step towards harnessing the power of LLMs for medical time series analysis that can elevate data-driven tools for clinicians and improve patient outcomes.

Title: NeuroPapyri: A Deep Attention Embedding Network for Handwritten Papyri Retrieval

• Authors: Giuseppe De Gregorio, Simon Perrin, Rodrigo C. G. Pena, Isabelle Marthot-Santaniello, Harold Mouchère
• Subjects: cs.CV, cs.DL
• Abstract URL: https://arxiv.org/abs/2408.07785
• Pdf URL: https://arxiv.org/pdf/2408.07785
• Copy Paste: [[2408.07785]] NeuroPapyri: A Deep Attention Embedding Network for Handwritten Papyri Retrieval(https://arxiv.org/abs/2408.07785)
• Keywords: interpretability
• Abstract: The intersection of computer vision and machine learning has emerged as a promising avenue for advancing historical research, facilitating a more profound exploration of our past. However, the application of machine learning approaches in historical palaeography is often met with criticism due to their perceived ``black box'' nature. In response to this challenge, we introduce NeuroPapyri, an innovative deep learning-based model specifically designed for the analysis of images containing ancient Greek papyri. To address concerns related to transparency and interpretability, the model incorporates an attention mechanism. This attention mechanism not only enhances the model's performance but also provides a visual representation of the image regions that significantly contribute to the decision-making process. Specifically calibrated for processing images of papyrus documents with lines of handwritten text, the model utilizes individual attention maps to inform the presence or absence of specific characters in the input image. This paper presents the NeuroPapyri model, including its architecture and training methodology. Results from the evaluation demonstrate NeuroPapyri's efficacy in document retrieval, showcasing its potential to advance the analysis of historical manuscripts.

Title: Protecting Onion Service Users Against Phishing

• Authors: Benjamin Güldenring, Volker Roth
• Subjects: cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.07787
• Pdf URL: https://arxiv.org/pdf/2408.07787
• Copy Paste: [[2408.07787]] Protecting Onion Service Users Against Phishing(https://arxiv.org/abs/2408.07787)
• Keywords: security, privacy, protect
• Abstract: Phishing websites are a common phenomenon among Tor onion services, and phishers exploit that it is tremendously difficult to distinguish phishing from authentic onion domain names. Operators of onion services devised several strategies to protect their users against phishing. But as we show in this work, none protect users against phishing without producing traces about visited services - something that particularly vulnerable users might want to avoid. In search of a solution we review prior research addressing this problem, and find that only two known approaches, hash visualization and PAKE, are capable of solving this problem. Hash visualization requires users to recognize large hash values. In order to make hash visualization more practical we design a novel mechanism called recognizer, which substantially reduces the amount of information that users must recognize. We analyze the security and privacy properties of our system formally, and report on our prototype implementation as a browser extension for the Tor web browser.

Title: Kraken: Inherently Parallel Transformers For Efficient Multi-Device Inference

• Authors: Rohan Baskar Prabhakar, Hengrui Zhang, David Wentlzaff
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.07802
• Pdf URL: https://arxiv.org/pdf/2408.07802
• Copy Paste: [[2408.07802]] Kraken: Inherently Parallel Transformers For Efficient Multi-Device Inference(https://arxiv.org/abs/2408.07802)
• Keywords: transformer
• Abstract: Large Transformer networks are increasingly used in settings where low inference latency can improve the end-user experience and enable new applications. However, autoregressive inference is resource intensive and requires parallelism for efficiency. Parallelism introduces collective communication that is both expensive and represents a phase when hardware resources are underutilized. Towards mitigating this, Kraken is an evolution of the standard Transformer architecture that is designed to complement existing tensor parallelism schemes for efficient inference on multi-device systems. By introducing a fixed degree of intra-layer model parallelism, the architecture allows collective operations to be overlapped with compute, decreasing latency and increasing hardware utilization. When trained on OpenWebText, Kraken models reach a similar perplexity as standard Transformers while also preserving their language modeling capabilities when evaluated on the SuperGLUE benchmark. Importantly, when tested on multi-GPU systems using TensorRT-LLM engines, Kraken speeds up Time To First Token by a mean of 35.6% across a range of model sizes, context lengths, and degrees of tensor parallelism.

Title: Space-scale Exploration of the Poor Reliability of Deep Learning Models: the Case of the Remote Sensing of Rooftop Photovoltaic Systems

• Authors: Gabriel Kasmi, Laurent Dubus, Yves-Marie Saint Drenan, Philippe Blanc
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07828
• Pdf URL: https://arxiv.org/pdf/2408.07828
• Copy Paste: [[2408.07828]] Space-scale Exploration of the Poor Reliability of Deep Learning Models: the Case of the Remote Sensing of Rooftop Photovoltaic Systems(https://arxiv.org/abs/2408.07828)
• Keywords: robust
• Abstract: Photovoltaic (PV) energy grows rapidly and is crucial for the decarbonization of electric systems. However, centralized registries recording the technical characteristifs of rooftop PV systems are often missing, making it difficult to accurately monitor this growth. The lack of monitoring could threaten the integration of PV energy into the grid. To avoid this situation, the remote sensing of rooftop PV systems using deep learning emerged as a promising solution. However, existing techniques are not reliable enough to be used by public authorities or transmission system operators (TSOs) to construct up-to-date statistics on the rooftop PV fleet. The lack of reliability comes from the fact that deep learning models are sensitive to distribution shifts. This work proposes a comprehensive evaluation of the effects of distribution shifts on the classification accuracy of deep learning models trained to detect rooftop PV panels on overhead imagery. We construct a benchmark to isolate the sources of distribution shift and introduce a novel methodology that leverages explainable artificial intelligence (XAI) and decomposition of the input image and model's decision in terms of scales to understand how distribution shifts affect deep learning models. Finally, based on our analysis, we introduce a data augmentation technique meant to improve the robustness of deep learning classifiers to varying acquisition conditions. We show that our proposed approach outperforms competing methods. We discuss some practical recommendations for mapping PV systems using overhead imagery and deep learning models.

Title: Language Driven Slice Discovery and Error Rectification

• Authors: Shantanu Ghosh, Chenyu Wang, Kayhan Batmanghelich
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07832
• Pdf URL: https://arxiv.org/pdf/2408.07832
• Copy Paste: [[2408.07832]] Language Driven Slice Discovery and Error Rectification(https://arxiv.org/abs/2408.07832)
• Keywords: interpretability, large language model
• Abstract: Error slice discovery associates structured patterns with model errors. Existing methods discover error slices by clustering the error-prone samples with similar patterns or assigning discrete attributes to each sample for post-hoc analysis. While these methods aim for interpretability and easier mitigation through reweighting or rebalancing, they may not capture the full complexity of error patterns due to incomplete or missing attributes. Contrary to the existing approach, this paper utilizes the reasoning capabilities of the Large Language Model (LLM) to analyze complex error patterns and generate testable hypotheses. This paper proposes LADDER: Language Driven slice Discovery and Error Rectification. It first projects the model's representation into a language-aligned feature space (\eg CLIP) to preserve semantics in the original model feature space. This ensures the accurate retrieval of sentences that highlight the model's errors. Next, the LLM utilizes the sentences and generates hypotheses to discover error slices. Finally, we mitigate the error by fine-tuning the classification head by creating a group-balanced dataset using the hypotheses. Our entire method does not require any attribute annotation, either explicitly or through external tagging models. We validate our method with \textbf{five} image classification datasets. The code is available\footnote{\url{this https URL}}

Title: ONSEP: A Novel Online Neural-Symbolic Framework for Event Prediction Based on Large Language Model

• Authors: Xuanqing Yu, Wangtao Sun, Jingwei Li, Kang Liu, Chengbao Liu, Jie Tan
• Subjects: cs.CL, cs.AI, cs.SC
• Abstract URL: https://arxiv.org/abs/2408.07840
• Pdf URL: https://arxiv.org/pdf/2408.07840
• Copy Paste: [[2408.07840]] ONSEP: A Novel Online Neural-Symbolic Framework for Event Prediction Based on Large Language Model(https://arxiv.org/abs/2408.07840)
• Keywords: large language model
• Abstract: In the realm of event prediction, temporal knowledge graph forecasting (TKGF) stands as a pivotal technique. Previous approaches face the challenges of not utilizing experience during testing and relying on a single short-term history, which limits adaptation to evolving data. In this paper, we introduce the Online Neural-Symbolic Event Prediction (ONSEP) framework, which innovates by integrating dynamic causal rule mining (DCRM) and dual history augmented generation (DHAG). DCRM dynamically constructs causal rules from real-time data, allowing for swift adaptation to new causal relationships. In parallel, DHAG merges short-term and long-term historical contexts, leveraging a bi-branch approach to enrich event prediction. Our framework demonstrates notable performance enhancements across diverse datasets, with significant Hit@k (k=1,3,10) improvements, showcasing its ability to augment large language models (LLMs) for event prediction without necessitating extensive retraining. The ONSEP framework not only advances the field of TKGF but also underscores the potential of neural-symbolic approaches in adapting to dynamic data environments.

Title: Enhancing Equitable Access to AI in Housing and Homelessness System of Care through Federated Learning

• Authors: Musa Taib, Jiajun Wu, Steve Drew, Geoffrey G. Messier
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.07845
• Pdf URL: https://arxiv.org/pdf/2408.07845
• Copy Paste: [[2408.07845]] Enhancing Equitable Access to AI in Housing and Homelessness System of Care through Federated Learning(https://arxiv.org/abs/2408.07845)
• Keywords: privacy, federate
• Abstract: The top priority of a Housing and Homelessness System of Care (HHSC) is to connect people experiencing homelessness to supportive housing. An HHSC typically consists of many agencies serving the same population. Information technology platforms differ in type and quality between agencies, so their data are usually isolated from one agency to another. Larger agencies may have sufficient data to train and test artificial intelligence (AI) tools but smaller agencies typically do not. To address this gap, we introduce a Federated Learning (FL) approach enabling all agencies to train a predictive model collaboratively without sharing their sensitive data. We demonstrate how FL can be used within an HHSC to provide all agencies equitable access to quality AI and further assist human decision-makers in the allocation of resources within HHSC. This is achieved while preserving the privacy of the people within the data by not sharing identifying information between agencies without their consent. Our experimental results using real-world HHSC data from Calgary, Alberta, demonstrate that our FL approach offers comparable performance with the idealized scenario of training the predictive model with data fully shared and linked between agencies.

Title: SER Evals: In-domain and Out-of-domain Benchmarking for Speech Emotion Recognition

• Authors: Mohamed Osman, Daniel Z. Kaplan, Tamer Nadeem
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07851
• Pdf URL: https://arxiv.org/pdf/2408.07851
• Copy Paste: [[2408.07851]] SER Evals: In-domain and Out-of-domain Benchmarking for Speech Emotion Recognition(https://arxiv.org/abs/2408.07851)
• Keywords: robust
• Abstract: Speech emotion recognition (SER) has made significant strides with the advent of powerful self-supervised learning (SSL) models. However, the generalization of these models to diverse languages and emotional expressions remains a challenge. We propose a large-scale benchmark to evaluate the robustness and adaptability of state-of-the-art SER models in both in-domain and out-of-domain settings. Our benchmark includes a diverse set of multilingual datasets, focusing on less commonly used corpora to assess generalization to new data. We employ logit adjustment to account for varying class distributions and establish a single dataset cluster for systematic evaluation. Surprisingly, we find that the Whisper model, primarily designed for automatic speech recognition, outperforms dedicated SSL models in cross-lingual SER. Our results highlight the need for more robust and generalizable SER models, and our benchmark serves as a valuable resource to drive future research in this direction.

Title: Zero Day Ransomware Detection with Pulse: Function Classification with Transformer Models and Assembly Language

• Authors: Matthew Gaber, Mohiuddin Ahmed, Helge Janicke
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.07862
• Pdf URL: https://arxiv.org/pdf/2408.07862
• Copy Paste: [[2408.07862]] Zero Day Ransomware Detection with Pulse: Function Classification with Transformer Models and Assembly Language(https://arxiv.org/abs/2408.07862)
• Keywords: transformer
• Abstract: Finding automated AI techniques to proactively defend against malware has become increasingly critical. The ability of an AI model to correctly classify novel malware is dependent on the quality of the features it is trained with and the authenticity of the features is dependent on the analysis tool. Peekaboo, a Dynamic Binary Instrumentation tool defeats evasive malware to capture its genuine behavior. The ransomware Assembly instructions captured by Peekaboo, follow Zipf's law, a principle also observed in natural languages, indicating Transformer models are particularly well suited to binary classification. We propose Pulse, a novel framework for zero day ransomware detection with Transformer models and Assembly language. Pulse, trained with the Peekaboo ransomware and benign software data, uniquely identify truly new samples with high accuracy. Pulse eliminates any familiar functionality across the test and training samples, forcing the Transformer model to detect malicious behavior based solely on context and novel Assembly instruction combinations.

Title: Words Matter: Reducing Stigma in Online Conversations about Substance Use with Large Language Models

• Authors: Layla Bouzoubaa, Elham Aghakhani, Shadi Rezapour
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07873
• Pdf URL: https://arxiv.org/pdf/2408.07873
• Copy Paste: [[2408.07873]] Words Matter: Reducing Stigma in Online Conversations about Substance Use with Large Language Models(https://arxiv.org/abs/2408.07873)
• Keywords: large language model
• Abstract: Stigma is a barrier to treatment for individuals struggling with substance use disorders (SUD), which leads to significantly lower treatment engagement rates. With only 7% of those affected receiving any form of help, societal stigma not only discourages individuals with SUD from seeking help but isolates them, hindering their recovery journey and perpetuating a cycle of shame and self-doubt. This study investigates how stigma manifests on social media, particularly Reddit, where anonymity can exacerbate discriminatory behaviors. We analyzed over 1.2 million posts, identifying 3,207 that exhibited stigmatizing language towards people who use substances (PWUS). Using Informed and Stylized LLMs, we develop a model for de-stigmatization of these expressions into empathetic language, resulting in 1,649 reformed phrase pairs. Our paper contributes to the field by proposing a computational framework for analyzing stigma and destigmatizing online content, and delving into the linguistic features that propagate stigma towards PWUS. Our work not only enhances understanding of stigma's manifestations online but also provides practical tools for fostering a more supportive digital environment for those affected by SUD. Code and data will be made publicly available upon acceptance.

Title: To Impute or Not: Recommendations for Multibiometric Fusion

• Authors: Melissa R Dale, Elliot Singer, Bengt J. Borgström, Arun Ross
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07883
• Pdf URL: https://arxiv.org/pdf/2408.07883
• Copy Paste: [[2408.07883]] To Impute or Not: Recommendations for Multibiometric Fusion(https://arxiv.org/abs/2408.07883)
• Keywords: biometric
• Abstract: Combining match scores from different biometric systems via fusion is a well-established approach to improving recognition accuracy. However, missing scores can degrade performance as well as limit the possible fusion techniques that can be applied. Imputation is a promising technique in multibiometric systems for replacing missing data. In this paper, we evaluate various score imputation approaches on three multimodal biometric score datasets, viz. NIST BSSR1, BIOCOP2008, and MIT LL Trimodal, and investigate the factors which might influence the effectiveness of imputation. Our studies reveal three key observations: (1) Imputation is preferable over not imputing missing scores, even when the fusion rule does not require complete score data. (2) Balancing the classes in the training data is crucial to mitigate negative biases in the imputation technique towards the under-represented class, even if it involves dropping a substantial number of score vectors. (3) Multivariate imputation approaches seem to be beneficial when scores between modalities are correlated, while univariate approaches seem to benefit scenarios where scores between modalities are less correlated.

Title: Instruct Large Language Models to Generate Scientific Literature Survey Step by Step

• Authors: Yuxuan Lai, Yupeng Wu, Yidan Wang, Wenpeng Hu, Chen Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07884
• Pdf URL: https://arxiv.org/pdf/2408.07884
• Copy Paste: [[2408.07884]] Instruct Large Language Models to Generate Scientific Literature Survey Step by Step(https://arxiv.org/abs/2408.07884)
• Keywords: generative, large language model
• Abstract: Abstract. Automatically generating scientific literature surveys is a valuable task that can significantly enhance research efficiency. However, the diverse and complex nature of information within a literature survey poses substantial challenges for generative models. In this paper, we design a series of prompts to systematically leverage large language models (LLMs), enabling the creation of comprehensive literature surveys through a step-by-step approach. Specifically, we design prompts to guide LLMs to sequentially generate the title, abstract, hierarchical headings, and the main content of the literature survey. We argue that this design enables the generation of the headings from a high-level perspective. During the content generation process, this design effectively harnesses relevant information while minimizing costs by restricting the length of both input and output content in LLM queries. Our implementation with Qwen-long achieved third place in the NLPCC 2024 Scientific Literature Survey Generation evaluation task, with an overall score only 0.03% lower than the second-place team. Additionally, our soft heading recall is 95.84%, the second best among the submissions. Thanks to the efficient prompt design and the low cost of the Qwen-long API, our method reduces the expense for generating each literature survey to 0.1 RMB, enhancing the practical value of our method.

Title: Fine-tuning Large Language Models with Human-inspired Learning Strategies in Medical Question Answering

• Authors: Yushi Yang, Andrew M. Bean, Robert McCraith, Adam Mahdi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07888
• Pdf URL: https://arxiv.org/pdf/2408.07888
• Copy Paste: [[2408.07888]] Fine-tuning Large Language Models with Human-inspired Learning Strategies in Medical Question Answering(https://arxiv.org/abs/2408.07888)
• Keywords: large language model
• Abstract: Training Large Language Models (LLMs) incurs substantial data-related costs, motivating the development of data-efficient training methods through optimised data ordering and selection. Human-inspired learning strategies, such as curriculum learning, offer possibilities for efficient training by organising data according to common human learning practices. Despite evidence that fine-tuning with curriculum learning improves the performance of LLMs for natural language understanding tasks, its effectiveness is typically assessed using a single model. In this work, we extend previous research by evaluating both curriculum-based and non-curriculum-based learning strategies across multiple LLMs, using human-defined and automated data labels for medical question answering. Our results indicate a moderate impact of using human-inspired learning strategies for fine-tuning LLMs, with maximum accuracy gains of 1.77% per model and 1.81% per dataset. Crucially, we demonstrate that the effectiveness of these strategies varies significantly across different model-dataset combinations, emphasising that the benefits of a specific human-inspired strategy for fine-tuning LLMs do not generalise. Additionally, we find evidence that curriculum learning using LLM-defined question difficulty outperforms human-defined difficulty, highlighting the potential of using model-generated measures for optimal curriculum design.

Title: MambaVT: Spatio-Temporal Contextual Modeling for robust RGB-T Tracking

• Authors: Simiao Lai, Chang Liu, Jiawen Zhu, Ben Kang, Yang Liu, Dong Wang, Huchuan Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07889
• Pdf URL: https://arxiv.org/pdf/2408.07889
• Copy Paste: [[2408.07889]] MambaVT: Spatio-Temporal Contextual Modeling for robust RGB-T Tracking(https://arxiv.org/abs/2408.07889)
• Keywords: robust, transformer
• Abstract: Existing RGB-T tracking algorithms have made remarkable progress by leveraging the global interaction capability and extensive pre-trained models of the Transformer architecture. Nonetheless, these methods mainly adopt imagepair appearance matching and face challenges of the intrinsic high quadratic complexity of the attention mechanism, resulting in constrained exploitation of temporal information. Inspired by the recently emerged State Space Model Mamba, renowned for its impressive long sequence modeling capabilities and linear computational complexity, this work innovatively proposes a pure Mamba-based framework (MambaVT) to fully exploit spatio-temporal contextual modeling for robust visible-thermal tracking. Specifically, we devise the long-range cross-frame integration component to globally adapt to target appearance variations, and introduce short-term historical trajectory prompts to predict the subsequent target states based on local temporal location clues. Extensive experiments show the significant potential of vision Mamba for RGB-T tracking, with MambaVT achieving state-of-the-art performance on four mainstream benchmarks while requiring lower computational costs. We aim for this work to serve as a simple yet strong baseline, stimulating future research in this field. The code and pre-trained models will be made available.

Title: Quantum-inspired Interpretable Deep Learning Architecture for Text Sentiment Analysis

• Authors: Bingyu Li, Da Zhang, Zhiyuan Zhao, Junyu Gao, Yuan Yuan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07891
• Pdf URL: https://arxiv.org/pdf/2408.07891
• Copy Paste: [[2408.07891]] Quantum-inspired Interpretable Deep Learning Architecture for Text Sentiment Analysis(https://arxiv.org/abs/2408.07891)
• Keywords: extraction, interpretability
• Abstract: Text has become the predominant form of communication on social media, embedding a wealth of emotional nuances. Consequently, the extraction of emotional information from text is of paramount importance. Despite previous research making some progress, existing text sentiment analysis models still face challenges in integrating diverse semantic information and lack interpretability. To address these issues, we propose a quantum-inspired deep learning architecture that combines fundamental principles of quantum mechanics (QM principles) with deep learning models for text sentiment analysis. Specifically, we analyze the commonalities between text representation and QM principles to design a quantum-inspired text representation method and further develop a quantum-inspired text embedding layer. Additionally, we design a feature extraction layer based on long short-term memory (LSTM) networks and self-attention mechanisms (SAMs). Finally, we calculate the text density matrix using the quantum complex numbers principle and apply 2D-convolution neural networks (CNNs) for feature condensation and dimensionality reduction. Through a series of visualization, comparative, and ablation experiments, we demonstrate that our model not only shows significant advantages in accuracy and efficiency compared to previous related models but also achieves a certain level of interpretability by integrating QM principles. Our code is available at QISA.

Title: The Nah Bandit: Modeling User Non-compliance in Recommendation Systems

• Authors: Tianyue Zhou, Jung-Hoon Cho, Cathy Wu
• Subjects: cs.LG, cs.IR, cs.MA, eess.SY
• Abstract URL: https://arxiv.org/abs/2408.07897
• Pdf URL: https://arxiv.org/pdf/2408.07897
• Copy Paste: [[2408.07897]] The Nah Bandit: Modeling User Non-compliance in Recommendation Systems(https://arxiv.org/abs/2408.07897)
• Keywords: robust
• Abstract: Recommendation systems now pervade the digital world, ranging from advertising to entertainment. However, it remains challenging to implement effective recommendation systems in the physical world, such as in mobility or health. This work focuses on a key challenge: in the physical world, it is often easy for the user to opt out of taking any recommendation if they are not to her liking, and to fall back to her baseline behavior. It is thus crucial in cyber-physical recommendation systems to operate with an interaction model that is aware of such user behavior, lest the user abandon the recommendations altogether. This paper thus introduces the Nah Bandit, a tongue-in-cheek reference to describe a Bandit problem where users can say `nah' to the recommendation and opt for their preferred option instead. As such, this problem lies in between a typical bandit setup and supervised learning. We model the user non-compliance by parameterizing an anchoring effect of recommendations on users. We then propose the Expert with Clustering (EWC) algorithm, a hierarchical approach that incorporates feedback from both recommended and non-recommended options to accelerate user preference learning. In a recommendation scenario with $N$ users, $T$ rounds per user, and $K$ clusters, EWC achieves a regret bound of $O(N\sqrt{T\log K} + NT)$, achieving superior theoretical performance in the short term compared to LinUCB algorithm. Experimental results also highlight that EWC outperforms both supervised learning and traditional contextual bandit approaches. This advancement reveals that effective use of non-compliance feedback can accelerate preference learning and improve recommendation accuracy. This work lays the foundation for future research in Nah Bandit, providing a robust framework for more effective recommendation systems.

Title: Assessing Language Models' Worldview for Fiction Generation

• Authors: Aisha Khatun, Daniel G. Brown
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07904
• Pdf URL: https://arxiv.org/pdf/2408.07904
• Copy Paste: [[2408.07904]] Assessing Language Models' Worldview for Fiction Generation(https://arxiv.org/abs/2408.07904)
• Keywords: large language model
• Abstract: The use of Large Language Models (LLMs) has become ubiquitous, with abundant applications in computational creativity. One such application is fictional story generation. Fiction is a narrative that occurs in a story world that is slightly different than ours. With LLMs becoming writing partners, we question how suitable they are to generate fiction. This study investigates the ability of LLMs to maintain a state of world essential to generate fiction. Through a series of questions to nine LLMs, we find that only two models exhibit consistent worldview, while the rest are self-conflicting. Subsequent analysis of stories generated by four models revealed a strikingly uniform narrative pattern. This uniformity across models further suggests a lack of `state' necessary for fiction. We highlight the limitations of current LLMs in fiction writing and advocate for future research to test and create story worlds for LLMs to reside in. All code, dataset, and the generated responses can be found in this https URL.

Title: KAN versus MLP on Irregular or Noisy Functions

• Authors: Chen Zeng, Jiahui Wang, Haoran Shen, Qiao Wang
• Subjects: cs.LG, cs.AI, cs.NE, math.NA
• Abstract URL: https://arxiv.org/abs/2408.07906
• Pdf URL: https://arxiv.org/pdf/2408.07906
• Copy Paste: [[2408.07906]] KAN versus MLP on Irregular or Noisy Functions(https://arxiv.org/abs/2408.07906)
• Keywords: fair
• Abstract: In this paper, we compare the performance of Kolmogorov-Arnold Networks (KAN) and Multi-Layer Perceptron (MLP) networks on irregular or noisy functions. We control the number of parameters and the size of the training samples to ensure a fair comparison. For clarity, we categorize the functions into six types: regular functions, continuous functions with local non-differentiable points, functions with jump discontinuities, functions with singularities, functions with coherent oscillations, and noisy functions. Our experimental results indicate that KAN does not always perform best. For some types of functions, MLP outperforms or performs comparably to KAN. Furthermore, increasing the size of training samples can improve performance to some extent. When noise is added to functions, the irregular features are often obscured by the noise, making it challenging for both MLP and KAN to extract these features effectively. We hope these experiments provide valuable insights for future neural network research and encourage further investigations to overcome these challenges.

Title: CEGRL-TKGR: A Causal Enhanced Graph Representation Learning Framework for Improving Temporal Knowledge Graph Extrapolation Reasoning

• Authors: Jinze Sun, Yongpan Sheng, Lirong He
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07911
• Pdf URL: https://arxiv.org/pdf/2408.07911
• Copy Paste: [[2408.07911]] CEGRL-TKGR: A Causal Enhanced Graph Representation Learning Framework for Improving Temporal Knowledge Graph Extrapolation Reasoning(https://arxiv.org/abs/2408.07911)
• Keywords: robust
• Abstract: Temporal knowledge graph reasoning (TKGR) is increasingly gaining attention for its ability to extrapolate new events from historical data, thereby enriching the inherently incomplete temporal knowledge graphs. Existing graph-based representation learning frameworks have made significant strides in developing evolving representations for both entities and relational embeddings. Despite these achievements, there's a notable tendency in these models to inadvertently learn biased data representations and mine spurious correlations, consequently failing to discern the causal relationships between events. This often leads to incorrect predictions based on these false correlations. To address this, we propose an innovative causal enhanced graph representation learning framework for TKGR (named CEGRL-TKGR). This framework introduces causal structures in graph-based representation learning to unveil the essential causal relationships between events, ultimately enhancing task performance. Specifically, we first disentangle the evolutionary representations of entities and relations in a temporal graph sequence into two distinct components, namely causal representations and confounding representations. Then, drawing on causal intervention theory, we advocate the utilization of causal representations for predictions, aiming to mitigate the effects of erroneous correlations caused by confounding features, thus achieving more robust and accurate predictions. Finally, extensive experimental results on six benchmark datasets demonstrate the superior performance of our model in the link prediction task.

Title: GridSE: Towards Practical Secure Geographic Search via Prefix Symmetric Searchable Encryption (Full Version)

• Authors: Ruoyang Guo, Jiarui Li, Shucheng Yu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.07916
• Pdf URL: https://arxiv.org/pdf/2408.07916
• Copy Paste: [[2408.07916]] GridSE: Towards Practical Secure Geographic Search via Prefix Symmetric Searchable Encryption (Full Version)(https://arxiv.org/abs/2408.07916)
• Keywords: secure, privacy
• Abstract: The proliferation of location-based services and applications has brought significant attention to data and location privacy. While general secure computation and privacy-enhancing techniques can partially address this problem, one outstanding challenge is to provide near latency-free search and compatibility with mainstream geographic search techniques, especially the Discrete Global Grid Systems (DGGS). This paper proposes a new construction, namely GridSE, for efficient and DGGS-compatible Secure Geographic Search (SGS) with both backward and forward privacy. We first formulate the notion of a semantic-secure primitive called \textit{symmetric prefix predicate encryption} (SP$^2$E), for predicting whether or not a keyword contains a given prefix, and provide a construction. Then we extend SP$^2$E for dynamic \textit{prefix symmetric searchable encryption} (pSSE), namely GridSE, which supports both backward and forward privacy. GridSE only uses lightweight primitives including cryptographic hash and XOR operations and is extremely efficient. Furthermore, we provide a generic pSSE framework that enables prefix search for traditional dynamic SSE that supports only full keyword search. Experimental results over real-world geographic databases of sizes (by the number of entries) from $10^3$ to $10^7$ and mainstream DGGS techniques show that GridSE achieves a speedup of $150\times$ - $5000\times$ on search latency and a saving of $99\%$ on communication overhead as compared to the state-of-the-art. Interestingly, even compared to plaintext search, GridSE introduces only $1.4\times$ extra computational cost and $0.9\times$ additional communication cost. Source code of our scheme is available at this https URL.

Title: MAG-SQL: Multi-Agent Generative Approach with Soft Schema Linking and Iterative Sub-SQL Refinement for Text-to-SQL

• Authors: Wenxuan Xie, Gaochen Wu, Bowen Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07930
• Pdf URL: https://arxiv.org/pdf/2408.07930
• Copy Paste: [[2408.07930]] MAG-SQL: Multi-Agent Generative Approach with Soft Schema Linking and Iterative Sub-SQL Refinement for Text-to-SQL(https://arxiv.org/abs/2408.07930)
• Keywords: generative
• Abstract: Recent In-Context Learning based methods have achieved remarkable success in Text-to-SQL task. However, there is still a large gap between the performance of these models and human performance on datasets with complex database schema and difficult questions, such as BIRD. Besides, existing work has neglected to supervise intermediate steps when solving questions iteratively with question decomposition methods, and the schema linking methods used in these works are very rudimentary. To address these issues, we propose MAG-SQL, a multi-agent generative approach with soft schema linking and iterative Sub-SQL refinement. In our framework, an entity-based method with tables' summary is used to select the columns in database, and a novel targets-conditions decomposition method is introduced to decompose those complex questions. Additionally, we build a iterative generating module which includes a Sub-SQL Generator and Sub-SQL Refiner, introducing external oversight for each step of generation. Through a series of ablation studies, the effectiveness of each agent in our framework has been demonstrated. When evaluated on the BIRD benchmark with GPT-4, MAG-SQL achieves an execution accuracy of 61.08\%, compared to the baseline accuracy of 46.35\% for vanilla GPT-4 and the baseline accuracy of 57.56\% for MAC-SQL. Besides, our approach makes similar progress on Spider.

Title: Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning

• Authors: Haofeng Liu, Erli Zhang, Junde Wu, Mingxuan Hong, Yueming Jin
• Subjects: cs.CV, cs.AI, cs.RO, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.07931
• Pdf URL: https://arxiv.org/pdf/2408.07931
• Copy Paste: [[2408.07931]] Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning(https://arxiv.org/abs/2408.07931)
• Keywords: segmentation
• Abstract: Surgical video segmentation is a critical task in computer-assisted surgery and is vital for enhancing surgical quality and patient outcomes. Recently, the Segment Anything Model 2 (SAM2) framework has shown superior advancements in image and video segmentation. However, SAM2 struggles with efficiency due to the high computational demands of processing high-resolution images and complex and long-range temporal dynamics in surgical videos. To address these challenges, we introduce Surgical SAM 2 (SurgSAM-2), an advanced model to utilize SAM2 with an Efficient Frame Pruning (EFP) mechanism, to facilitate real-time surgical video segmentation. The EFP mechanism dynamically manages the memory bank by selectively retaining only the most informative frames, reducing memory usage and computational cost while maintaining high segmentation accuracy. Our extensive experiments demonstrate that SurgSAM-2 significantly improves both efficiency and segmentation accuracy compared to the vanilla SAM2. Remarkably, SurgSAM-2 achieves a 3$\times$ FPS compared with SAM2, while also delivering state-of-the-art performance after fine-tuning with lower-resolution data. These advancements establish SurgSAM-2 as a leading model for surgical video analysis, making real-time surgical video segmentation in resource-constrained environments a feasible reality.

Title: Addressing Skewed Heterogeneity via Federated Prototype Rectification with Personalization

• Authors: Shunxin Guo, Hongsong Wang, Shuxia Lin, Zhiqiang Kou, Xin Geng
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.07966
• Pdf URL: https://arxiv.org/pdf/2408.07966
• Copy Paste: [[2408.07966]] Addressing Skewed Heterogeneity via Federated Prototype Rectification with Personalization(https://arxiv.org/abs/2408.07966)
• Keywords: privacy, federate
• Abstract: Federated learning is an efficient framework designed to facilitate collaborative model training across multiple distributed devices while preserving user data privacy. A significant challenge of federated learning is data-level heterogeneity, i.e., skewed or long-tailed distribution of private data. Although various methods have been proposed to address this challenge, most of them assume that the underlying global data is uniformly distributed across all clients. This paper investigates data-level heterogeneity federated learning with a brief review and redefines a more practical and challenging setting called Skewed Heterogeneous Federated Learning (SHFL). Accordingly, we propose a novel Federated Prototype Rectification with Personalization which consists of two parts: Federated Personalization and Federated Prototype Rectification. The former aims to construct balanced decision boundaries between dominant and minority classes based on private data, while the latter exploits both inter-class discrimination and intra-class consistency to rectify empirical prototypes. Experiments on three popular benchmarks show that the proposed approach outperforms current state-of-the-art methods and achieves balanced performance in both personalization and generalization.

Title: Predicting Lung Cancer Patient Prognosis with Large Language Models

• Authors: Danqing Hu, Bing Liu, Xiang Li, Xiaofeng Zhu, Nan Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07971
• Pdf URL: https://arxiv.org/pdf/2408.07971
• Copy Paste: [[2408.07971]] Predicting Lung Cancer Patient Prognosis with Large Language Models(https://arxiv.org/abs/2408.07971)
• Keywords: large language model
• Abstract: Prognosis prediction is crucial for determining optimal treatment plans for lung cancer patients. Traditionally, such predictions relied on models developed from retrospective patient data. Recently, large language models (LLMs) have gained attention for their ability to process and generate text based on extensive learned knowledge. In this study, we evaluate the potential of GPT-4o mini and GPT-3.5 in predicting the prognosis of lung cancer patients. We collected two prognosis datasets, i.e., survival and post-operative complication datasets, and designed multiple tasks to assess the models' performance comprehensively. Logistic regression models were also developed as baselines for comparison. The experimental results demonstrate that LLMs can achieve competitive, and in some tasks superior, performance in lung cancer prognosis prediction compared to data-driven logistic regression models despite not using additional patient data. These findings suggest that LLMs can be effective tools for prognosis prediction in lung cancer, particularly when patient data is limited or unavailable.

Title: LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning

• Authors: Jiajie Li, Garrett Skinner, Gene Yang, Brian R Quaranto, Steven D Schwaitzberg, Peter C W Kim, Jinjun Xiong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07981
• Pdf URL: https://arxiv.org/pdf/2408.07981
• Copy Paste: [[2408.07981]] LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning(https://arxiv.org/abs/2408.07981)
• Keywords: large language model
• Abstract: Multimodal large language models (LLMs) have achieved notable success across various domains, while research in the medical field has largely focused on unimodal images. Meanwhile, current general-domain multimodal models for videos still lack the capabilities to understand and engage in conversations about surgical videos. One major contributing factor is the absence of datasets in the surgical field. In this paper, we create a new dataset, Surg-QA, consisting of 102,000 surgical video-instruction pairs, the largest of its kind so far. To build such a dataset, we propose a novel two-stage question-answer generation pipeline with LLM to learn surgical knowledge in a structured manner from the publicly available surgical lecture videos. The pipeline breaks down the generation process into two stages to significantly reduce the task complexity, allowing us to use a more affordable, locally deployed open-source LLM than the premium paid LLM services. It also mitigates the risk of LLM hallucinations during question-answer generation, thereby enhancing the overall quality of the generated data. We further train LLaVA-Surg, a novel vision-language conversational assistant capable of answering open-ended questions about surgical videos, on this Surg-QA dataset, and conduct comprehensive evaluations on zero-shot surgical video question-answering tasks. We show that LLaVA-Surg significantly outperforms all previous general-domain models, demonstrating exceptional multimodal conversational skills in answering open-ended questions about surgical videos. We will release our code, model, and the instruction-tuning dataset.

Title: ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models

• Authors: Faris Hijazi (1), Somayah AlHarbi (1), Abdulaziz AlHussein (1), Harethah Abu Shairah (2), Reem AlZahrani (2), Hebah AlShamlan (1), Omar Knio (2), George Turkiyyah (2) ((1) THIQAH, (2) KAUST)
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.07983
• Pdf URL: https://arxiv.org/pdf/2408.07983
• Copy Paste: [[2408.07983]] ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models(https://arxiv.org/abs/2408.07983)
• Keywords: large language model
• Abstract: The rapid advancements in Large Language Models (LLMs) have led to significant improvements in various natural language processing tasks. However, the evaluation of LLMs' legal knowledge, particularly in non-English languages such as Arabic, remains under-explored. To address this gap, we introduce ArabLegalEval, a multitask benchmark dataset for assessing the Arabic legal knowledge of LLMs. Inspired by the MMLU and LegalBench datasets, ArabLegalEval consists of multiple tasks sourced from Saudi legal documents and synthesized questions. In this work, we aim to analyze the capabilities required to solve legal problems in Arabic and benchmark the performance of state-of-the-art LLMs. We explore the impact of in-context learning and investigate various evaluation methods. Additionally, we explore workflows for generating questions with automatic validation to enhance the dataset's quality. We benchmark multilingual and Arabic-centric LLMs, such as GPT-4 and Jais, respectively. We also share our methodology for creating the dataset and validation, which can be generalized to other domains. We hope to accelerate AI research in the Arabic Legal domain by releasing the ArabLegalEval dataset and code: this https URL

Title: IIU: Independent Inference Units for Knowledge-based Visual Question Answering

• Authors: Yili Li, Jing Yu, Keke Gai, Gang Xiong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07989
• Pdf URL: https://arxiv.org/pdf/2408.07989
• Copy Paste: [[2408.07989]] IIU: Independent Inference Units for Knowledge-based Visual Question Answering(https://arxiv.org/abs/2408.07989)
• Keywords: interpretability
• Abstract: Knowledge-based visual question answering requires external knowledge beyond visible content to answer the question correctly. One limitation of existing methods is that they focus more on modeling the inter-modal and intra-modal correlations, which entangles complex multimodal clues by implicit embeddings and lacks interpretability and generalization ability. The key challenge to solve the above problem is to separate the information and process it separately at the functional level. By reusing each processing unit, the generalization ability of the model to deal with different data can be increased. In this paper, we propose Independent Inference Units (IIU) for fine-grained multi-modal reasoning to decompose intra-modal information by the functionally independent units. Specifically, IIU processes each semantic-specific intra-modal clue by an independent inference unit, which also collects complementary information by communication from different units. To further reduce the impact of redundant information, we propose a memory update module to maintain semantic-relevant memory along with the reasoning process gradually. In comparison with existing non-pretrained multi-modal reasoning models on standard datasets, our model achieves a new state-of-the-art, enhancing performance by 3%, and surpassing basic pretrained multi-modal models. The experimental results show that our IIU model is effective in disentangling intra-modal clues as well as reasoning units to provide explainable reasoning evidence. Our code is available at this https URL.

Title: FuseChat: Knowledge Fusion of Chat Models

• Authors: Fanqi Wan, Longguang Zhong, Ziyi Yang, Ruijun Chen, Xiaojun Quan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07990
• Pdf URL: https://arxiv.org/pdf/2408.07990
• Copy Paste: [[2408.07990]] FuseChat: Knowledge Fusion of Chat Models(https://arxiv.org/abs/2408.07990)
• Keywords: large language model
• Abstract: While training large language models (LLMs) from scratch can indeed lead to models with distinct capabilities and strengths, it incurs substantial costs and may lead to redundancy in competencies. Knowledge fusion aims to integrate existing LLMs of diverse architectures and capabilities into a more potent LLM through lightweight continual training, thereby reducing the need for costly LLM development. In this work, we propose a new framework for the knowledge fusion of chat LLMs through two main stages, resulting in FuseChat. Firstly, we conduct pairwise knowledge fusion on source chat LLMs of varying structures and scales to create multiple target LLMs with identical structure and size via lightweight fine-tuning. During this process, a statistics-based token alignment approach is introduced as the cornerstone for fusing LLMs with different structures. Secondly, we merge these target LLMs within the parameter space, where we propose a novel method for determining the merging coefficients based on the magnitude of parameter updates before and after fine-tuning. We implement and validate FuseChat using six prominent chat LLMs with diverse architectures and scales, including OpenChat-3.5-7B, Starling-LM-7B-alpha, NH2-SOLAR-10.7B, InternLM2-Chat-20B, Mixtral-8x7B-Instruct, and Qwen-1.5-Chat-72B. Experimental results on two instruction-following benchmarks, AlpacaEval 2.0 and MT-Bench, demonstrate the superiority of FuseChat-7B over baselines of various sizes. Our model is even comparable to the larger Mixtral-8x7B-Instruct and approaches GPT-3.5-Turbo-1106 on MT-Bench. Our code, model weights, and data are public at \url{this https URL}.

Title: Practical Privacy-Preserving Identity Verification using Third-Party Cloud Services and FHE (Role of Data Encoding in Circuit Depth Management)

• Authors: Deep Inder Mohan, Srinivas Vivek
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.08002
• Pdf URL: https://arxiv.org/pdf/2408.08002
• Copy Paste: [[2408.08002]] Practical Privacy-Preserving Identity Verification using Third-Party Cloud Services and FHE (Role of Data Encoding in Circuit Depth Management)(https://arxiv.org/abs/2408.08002)
• Keywords: secure, privacy, biometric
• Abstract: National digital identity verification systems have played a critical role in the effective distribution of goods and services, particularly, in developing countries. Due to the cost involved in deploying and maintaining such systems, combined with a lack of in-house technical expertise, governments seek to outsource this service to third-party cloud service providers to the extent possible. This leads to increased concerns regarding the privacy of users' personal data. In this work, we propose a practical privacy-preserving digital identity (ID) verification protocol where the third-party cloud services process the identity data encrypted using a (single-key) Fully Homomorphic Encryption (FHE) scheme such as BFV. Though the role of a trusted entity such as government is not completely eliminated, our protocol does significantly reduces the computation load on such parties. A challenge in implementing a privacy-preserving ID verification protocol using FHE is to support various types of queries such as exact and/or fuzzy demographic and biometric matches including secure age comparisons. From a cryptographic engineering perspective, our main technical contribution is a user data encoding scheme that encodes demographic and biometric user data in only two BFV ciphertexts and yet facilitates us to outsource various types of ID verification queries to a third-party cloud. Our encoding scheme also ensures that the only computation done by the trusted entity is a query-agnostic "extended" decryption. This is in stark contrast with recent works that outsource all the non-arithmetic operations to a trusted server. We implement our protocol using the Microsoft SEAL FHE library and demonstrate its practicality.

Title: Leveraging Web-Crawled Data for High-Quality Fine-Tuning

• Authors: Jing Zhou, Chenglin Jiang, Wei Shen, Xiao Zhou, Xiaonan He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08003
• Pdf URL: https://arxiv.org/pdf/2408.08003
• Copy Paste: [[2408.08003]] Leveraging Web-Crawled Data for High-Quality Fine-Tuning(https://arxiv.org/abs/2408.08003)
• Keywords: large language model
• Abstract: Most large language models are fine-tuned using either expensive human-annotated data or GPT-4 generated data which cannot guarantee performance in certain domains. We argue that although the web-crawled data often has formatting errors causing semantic inaccuracies, it can still serve as a valuable source for high-quality supervised fine-tuning in specific domains without relying on advanced models like GPT-4. To this end, we create a paired training dataset automatically by aligning web-crawled data with a smaller set of high-quality data. By training a language model on this dataset, we can convert web data with irregular formats into high-quality ones. Our experiments show that training with the model-transformed data yields better results, surpassing training with only high-quality data by an average score of 9.4% in Chinese math problems. Additionally, our 7B model outperforms several open-source models larger than 32B and surpasses well-known closed-source models such as GPT-3.5, highlighting the efficacy of our approach.

Title: Causal Discovery from Time-Series Data with Short-Term Invariance-Based Convolutional Neural Networks

• Authors: Rujia Shen, Boran Wang, Chao Zhao, Yi Guan, Jingchi Jiang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08023
• Pdf URL: https://arxiv.org/pdf/2408.08023
• Copy Paste: [[2408.08023]] Causal Discovery from Time-Series Data with Short-Term Invariance-Based Convolutional Neural Networks(https://arxiv.org/abs/2408.08023)
• Keywords: generative
• Abstract: Causal discovery from time-series data aims to capture both intra-slice (contemporaneous) and inter-slice (time-lagged) causality between variables within the temporal chain, which is crucial for various scientific disciplines. Compared to causal discovery from non-time-series data, causal discovery from time-series data necessitates more serialized samples with a larger amount of observed time steps. To address the challenges, we propose a novel gradient-based causal discovery approach STIC, which focuses on \textbf{S}hort-\textbf{T}erm \textbf{I}nvariance using \textbf{C}onvolutional neural networks to uncover the causal relationships from time-series data. Specifically, STIC leverages both the short-term time and mechanism invariance of causality within each window observation, which possesses the property of independence, to enhance sample efficiency. Furthermore, we construct two causal convolution kernels, which correspond to the short-term time and mechanism invariance respectively, to estimate the window causal graph. To demonstrate the necessity of convolutional neural networks for causal discovery from time-series data, we theoretically derive the equivalence between convolution and the underlying generative principle of time-series data under the assumption that the additive noise model is identifiable. Experimental evaluations conducted on both synthetic and FMRI benchmark datasets demonstrate that our STIC outperforms baselines significantly and achieves the state-of-the-art performance, particularly when the datasets contain a limited number of observed time steps. Code is available at \url{this https URL}.

Title: The Clever Hans Effect in Unsupervised Learning

• Authors: Jacob Kauffmann, Jonas Dippel, Lukas Ruff, Wojciech Samek, Klaus-Robert Müller, Grégoire Montavon
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.08041
• Pdf URL: https://arxiv.org/pdf/2408.08041
• Copy Paste: [[2408.08041]] The Clever Hans Effect in Unsupervised Learning(https://arxiv.org/abs/2408.08041)
• Keywords: robust
• Abstract: Unsupervised learning has become an essential building block of AI systems. The representations it produces, e.g. in foundation models, are critical to a wide variety of downstream applications. It is therefore important to carefully examine unsupervised models to ensure not only that they produce accurate predictions, but also that these predictions are not "right for the wrong reasons", the so-called Clever Hans (CH) effect. Using specially developed Explainable AI techniques, we show for the first time that CH effects are widespread in unsupervised learning. Our empirical findings are enriched by theoretical insights, which interestingly point to inductive biases in the unsupervised learning machine as a primary source of CH effects. Overall, our work sheds light on unexplored risks associated with practical applications of unsupervised learning and suggests ways to make unsupervised learning more robust.

Title: DATTA: Towards Diversity Adaptive Test-Time Adaptation in Dynamic Wild World

• Authors: Chuyang Ye, Dongyan Wei, Zhendong Liu, Yuanyi Pang, Yixi Lin, Jiarong Liao, Qinting Jiang, Xianghua Fu, Qing Li, Jingyan Jiang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08056
• Pdf URL: https://arxiv.org/pdf/2408.08056
• Copy Paste: [[2408.08056]] DATTA: Towards Diversity Adaptive Test-Time Adaptation in Dynamic Wild World(https://arxiv.org/abs/2408.08056)
• Keywords: robust
• Abstract: Test-time adaptation (TTA) effectively addresses distribution shifts between training and testing data by adjusting models on test samples, which is crucial for improving model inference in real-world applications. However, traditional TTA methods typically follow a fixed pattern to address the dynamic data patterns (low-diversity or high-diversity patterns) often leading to performance degradation and consequently a decline in Quality of Experience (QoE). The primary issues we observed are:Different scenarios require different normalization methods (e.g., Instance Normalization is optimal in mixed domains but not in static domains). Model fine-tuning can potentially harm the model and waste time.Hence, it is crucial to design strategies for effectively measuring and managing distribution diversity to minimize its negative impact on model performance. Based on these observations, this paper proposes a new general method, named Diversity Adaptive Test-Time Adaptation (DATTA), aimed at improving QoE. DATTA dynamically selects the best batch normalization methods and fine-tuning strategies by leveraging the Diversity Score to differentiate between high and low diversity score batches. It features three key components: Diversity Discrimination (DD) to assess batch diversity, Diversity Adaptive Batch Normalization (DABN) to tailor normalization methods based on DD insights, and Diversity Adaptive Fine-Tuning (DAFT) to selectively fine-tune the model. Experimental results show that our method achieves up to a 21% increase in accuracy compared to state-of-the-art methodologies, indicating that our method maintains good model performance while demonstrating its robustness. Our code will be released soon.

Title: Security Challenges of Complex Space Applications: An Empirical Study

• Authors: Tomas Paulik
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.08061
• Pdf URL: https://arxiv.org/pdf/2408.08061
• Copy Paste: [[2408.08061]] Security Challenges of Complex Space Applications: An Empirical Study(https://arxiv.org/abs/2408.08061)
• Keywords: security, defense, attack
• Abstract: Software applications in the space and defense industries have their unique characteristics: They are complex in structure, mission-critical, and often targets of state-of-the-art cyber attacks sponsored by adversary nation states. These applications have typically a high number of stakeholders in their software component supply chain, data supply chain, and user base. The aforementioned factors make such software applications potentially vulnerable to bad actors, as the widely adopted DevOps tools and practices were not designed for high-complexity and high-risk environments. In this study, I investigate the security challenges of the development and management of complex space applications, which differentiate the process from the commonly used practices. My findings are based on interviews with five domain experts from the industry and are further supported by a comprehensive review of relevant publications. To illustrate the dynamics of the problem, I present and discuss an actual software supply chain structure used by Thales Alenia Space, which is one of the largest suppliers of the European Space Agency. Subsequently, I discuss the four most critical security challenges identified by the interviewed experts: Verification of software artifacts, verification of the deployed application, single point of security failure, and data tampering by trusted stakeholders. Furthermore, I present best practices which could be used to overcome each of the given challenges, and whether the interviewed experts think their organization has access to the right tools to address them. Finally, I propose future research of new DevSecOps strategies, practices, and tools which would enable better methods of software integrity verification in the space and defense industries.

Title: MambaMIM: Pre-training Mamba with State Space Token-interpolation

• Authors: Fenghe Tang, Bingkun Nian, Yingtai Li, Jie Yang, Liu Wei, S. Kevin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08070
• Pdf URL: https://arxiv.org/pdf/2408.08070
• Copy Paste: [[2408.08070]] MambaMIM: Pre-training Mamba with State Space Token-interpolation(https://arxiv.org/abs/2408.08070)
• Keywords: transformer, generative
• Abstract: Generative self-supervised learning demonstrates outstanding representation learning capabilities in both Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). However, there are currently no generative pre-training methods related to selective state space models (Mamba) that can handle long-range dependencies effectively. To address this challenge, we introduce a generative self-supervised learning method for Mamba (MambaMIM) based on Selective Structure State Space Sequence Token-interpolation (S6T), a general-purpose pre-training method for arbitrary Mamba architectures. Our method, MambaMIM, incorporates a bottom-up 3D hybrid masking strategy in the encoder to maintain masking consistency across different architectures. Additionally, S6T is employed to learn causal relationships between the masked sequence in the state space. MambaMIM can be used on any single or hybrid Mamba architectures to enhance the Mamba long-range representation capability. Extensive downstream experiments reveal the feasibility and advancement of using Mamba for pre-training medical image tasks. The code is available at: this https URL

Title: I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm

• Authors: Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, Wenhao Huang, Jiajun Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08072
• Pdf URL: https://arxiv.org/pdf/2408.08072
• Copy Paste: [[2408.08072]] I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm(https://arxiv.org/abs/2408.08072)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have achieved significant advancements, however, the common learning paradigm treats LLMs as passive information repositories, neglecting their potential for active learning and alignment. Some approaches train LLMs using their own generated synthetic data, exploring the possibility of active alignment. However, there is still a huge gap between these one-time alignment methods and the continuous automatic alignment of humans. In this paper, we introduce \textbf{I-SHEEP}, an \textbf{I}terative \textbf{S}elf-En\textbf{H}anc\textbf{E}m\textbf{E}nt \textbf{P}aradigm.This human-like paradigm enables LLMs to \textbf{continuously self-align from scratch with nothing}. Compared to the one-time alignment method Dromedary \cite{sun2023principledriven}, which refers to the first iteration in this paper, I-SHEEP can significantly enhance capacities on both Qwen and Llama models. I-SHEEP achieves a maximum relative improvement of 78.2\% in the Alpaca Eval, 24.0\% in the MT Bench, and an absolute increase of 8.88\% in the IFEval accuracy over subsequent iterations in Qwen-1.5 72B model. Additionally, I-SHEEP surpasses the base model in various standard benchmark generation tasks, achieving an average improvement of 24.77\% in code generation tasks, 12.04\% in TrivialQA, and 20.29\% in SQuAD. We also provide new insights based on the experiment results. Our codes, datasets, and models are available at \textbf{https://anonymous.4open.science/r/I-SHEEP}.

Title: Extracting Sentence Embeddings from Pretrained Transformer Models

• Authors: Lukas Stankevičius, Mantas Lukoševičius
• Subjects: cs.CL, cs.IR, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.08073
• Pdf URL: https://arxiv.org/pdf/2408.08073
• Copy Paste: [[2408.08073]] Extracting Sentence Embeddings from Pretrained Transformer Models(https://arxiv.org/abs/2408.08073)
• Keywords: extraction, transformer
• Abstract: Background/introduction: Pre-trained transformer models shine in many natural language processing tasks and therefore are expected to bear the representation of the input sentence or text meaning. These sentence-level embeddings are also important in retrieval-augmented generation. But do commonly used plain averaging or prompt templates surface it enough? Methods: Given 110M parameters BERT's hidden representations from multiple layers and multiple tokens we tried various ways to extract optimal sentence representations. We tested various token aggregation and representation post-processing techniques. We also tested multiple ways of using a general Wikitext dataset to complement BERTs sentence representations. All methods were tested on 8 Semantic Textual Similarity (STS), 6 short text clustering, and 12 classification tasks. We also evaluated our representation-shaping techniques on other static models, including random token representations. Results: Proposed representation extraction methods improved the performance on STS and clustering tasks for all models considered. Very high improvements for static token-based models, especially random embeddings for STS tasks almost reach the performance of BERT-derived representations. Conclusions: Our work shows that for multiple tasks simple baselines with representation shaping techniques reach or even outperform more complex BERT-based models or are able to contribute to their performance.

Title: Single-image coherent reconstruction of objects and humans

• Authors: Sarthak Batra, Partha P. Chakrabarti, Simon Hadfield, Armin Mustafa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08086
• Pdf URL: https://arxiv.org/pdf/2408.08086
• Copy Paste: [[2408.08086]] Single-image coherent reconstruction of objects and humans(https://arxiv.org/abs/2408.08086)
• Keywords: robust
• Abstract: Existing methods for reconstructing objects and humans from a monocular image suffer from severe mesh collisions and performance limitations for interacting occluding objects. This paper introduces a method to obtain a globally consistent 3D reconstruction of interacting objects and people from a single image. Our contributions include: 1) an optimization framework, featuring a collision loss, tailored to handle human-object and human-human interactions, ensuring spatially coherent scene reconstruction; and 2) a novel technique to robustly estimate 6 degrees of freedom (DOF) poses, specifically for heavily occluded objects, exploiting image inpainting. Notably, our proposed method operates effectively on images from real-world scenarios, without necessitating scene or object-level 3D supervision. Extensive qualitative and quantitative evaluation against existing methods demonstrates a significant reduction in collisions in the final reconstructions of scenes with multiple interacting humans and objects and a more coherent scene reconstruction.

Title: ColorMamba: Towards High-quality NIR-to-RGB Spectral Translation with Mamba

• Authors: Huiyu Zhai, Guang Jin, Xingxing Yang, Guosheng Kang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08087
• Pdf URL: https://arxiv.org/pdf/2408.08087
• Copy Paste: [[2408.08087]] ColorMamba: Towards High-quality NIR-to-RGB Spectral Translation with Mamba(https://arxiv.org/abs/2408.08087)
• Keywords: generative
• Abstract: Translating NIR to the visible spectrum is challenging due to cross-domain complexities. Current models struggle to balance a broad receptive field with computational efficiency, limiting practical use. Although the Selective Structured State Space Model, especially the improved version, Mamba, excels in generative tasks by capturing long-range dependencies with linear complexity, its default approach of converting 2D images into 1D sequences neglects local context. In this work, we propose a simple but effective backbone, dubbed ColorMamba, which first introduces Mamba into spectral translation tasks. To explore global long-range dependencies and local context for efficient spectral translation, we introduce learnable padding tokens to enhance the distinction of image boundaries and prevent potential confusion within the sequence model. Furthermore, local convolutional enhancement and agent attention are designed to improve the vanilla Mamba. Moreover, we exploit the HSV color to provide multi-scale guidance in the reconstruction process for more accurate spectral translation. Extensive experiments show that our ColorMamba achieves a 1.02 improvement in terms of PSNR compared with the state-of-the-art method. Our code is available at this https URL.

Title: KGV: Integrating Large Language Models with Knowledge Graphs for Cyber Threat Intelligence Credibility Assessment

• Authors: Zongzong Wu, Fengxiao Tang, Ming Zhao, Yufeng Li
• Subjects: cs.CR, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.08088
• Pdf URL: https://arxiv.org/pdf/2408.08088
• Copy Paste: [[2408.08088]] KGV: Integrating Large Language Models with Knowledge Graphs for Cyber Threat Intelligence Credibility Assessment(https://arxiv.org/abs/2408.08088)
• Keywords: security, protect, attack, large language model
• Abstract: Cyber threat intelligence is a critical tool that many organizations and individuals use to protect themselves from sophisticated, organized, persistent, and weaponized cyber attacks. However, few studies have focused on the quality assessment of threat intelligence provided by intelligence platforms, and this work still requires manual analysis by cybersecurity experts. In this paper, we propose a knowledge graph-based verifier, a novel Cyber Threat Intelligence (CTI) quality assessment framework that combines knowledge graphs and Large Language Models (LLMs). Our approach introduces LLMs to automatically extract OSCTI key claims to be verified and utilizes a knowledge graph consisting of paragraphs for fact-checking. This method differs from the traditional way of constructing complex knowledge graphs with entities as nodes. By constructing knowledge graphs with paragraphs as nodes and semantic similarity as edges, it effectively enhances the semantic understanding ability of the model and simplifies labeling requirements. Additionally, to fill the gap in the research field, we created and made public the first dataset for threat intelligence assessment from heterogeneous sources. To the best of our knowledge, this work is the first to create a dataset on threat intelligence reliability verification, providing a reference for future research. Experimental results show that KGV (Knowledge Graph Verifier) significantly improves the performance of LLMs in intelligence quality assessment. Compared with traditional methods, we reduce a large amount of data annotation while the model still exhibits strong reasoning capabilities. Finally, our method can achieve XXX accuracy in network threat assessment.

Title: AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agents

• Authors: Guhong Chen, Liyang Fan, Zihan Gong, Nan Xie, Zixuan Li, Ziqiang Liu, Chengming Li, Qiang Qu, Shiwen Ni, Min Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08089
• Pdf URL: https://arxiv.org/pdf/2408.08089
• Copy Paste: [[2408.08089]] AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agents(https://arxiv.org/abs/2408.08089)
• Keywords: defense, large language model
• Abstract: In this paper, we present a simulation system called AgentCourt that simulates the entire courtroom process. The judge, plaintiff's lawyer, defense lawyer, and other participants are autonomous agents driven by large language models (LLMs). Our core goal is to enable lawyer agents to learn how to argue a case, as well as improving their overall legal skills, through courtroom process simulation. To achieve this goal, we propose an adversarial evolutionary approach for the lawyer-agent. Since AgentCourt can simulate the occurrence and development of court hearings based on a knowledge base and LLM, the lawyer agents can continuously learn and accumulate experience from real court cases. The simulation experiments show that after two lawyer-agents have engaged in a thousand adversarial legal cases in AgentCourt (which can take a decade for real-world lawyers), compared to their pre-evolutionary state, the evolved lawyer agents exhibit consistent improvement in their ability to handle legal tasks. To enhance the credibility of our experimental results, we enlisted a panel of professional lawyers to evaluate our simulations. The evaluation indicates that the evolved lawyer agents exhibit notable advancements in responsiveness, as well as expertise and logical rigor. This work paves the way for advancing LLM-driven agent technology in legal scenarios. Code is available at this https URL.

Title: When Video Coding Meets Multimodal Large Language Models: A Unified Paradigm for Video Coding

• Authors: Pingping Zhang, Jinlong Li, Meng Wang, Nicu Sebe, Sam Kwong, Shiqi Wang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.08093
• Pdf URL: https://arxiv.org/pdf/2408.08093
• Copy Paste: [[2408.08093]] When Video Coding Meets Multimodal Large Language Models: A Unified Paradigm for Video Coding(https://arxiv.org/abs/2408.08093)
• Keywords: generative, large language model
• Abstract: Existing codecs are designed to eliminate intrinsic redundancies to create a compact representation for compression. However, strong external priors from Multimodal Large Language Models (MLLMs) have not been explicitly explored in video compression. Herein, we introduce a unified paradigm for Cross-Modality Video Coding (CMVC), which is a pioneering approach to explore multimodality representation and video generative models in video coding. Specifically, on the encoder side, we disentangle a video into spatial content and motion components, which are subsequently transformed into distinct modalities to achieve very compact representation by leveraging MLLMs. During decoding, previously encoded components and video generation models are leveraged to create multiple encoding-decoding modes that optimize video reconstruction quality for specific decoding requirements, including Text-Text-to-Video (TT2V) mode to ensure high-quality semantic information and Image-Text-to-Video (IT2V) mode to achieve superb perceptual consistency. In addition, we propose an efficient frame interpolation model for IT2V mode via Low-Rank Adaption (LoRA) tuning to guarantee perceptual quality, which allows the generated motion cues to behave smoothly. Experiments on benchmarks indicate that TT2V achieves effective semantic reconstruction, while IT2V exhibits competitive perceptual consistency. These results highlight potential directions for future research in video coding.

Title: Multimodal Causal Reasoning Benchmark: Challenging Vision Large Language Models to Infer Causal Links Between Siamese Images

• Authors: Zhiyuan Li, Heng Wang, Dongnan Liu, Chaoyi Zhang, Ao Ma, Jieting Long, Weidong Cai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08105
• Pdf URL: https://arxiv.org/pdf/2408.08105
• Copy Paste: [[2408.08105]] Multimodal Causal Reasoning Benchmark: Challenging Vision Large Language Models to Infer Causal Links Between Siamese Images(https://arxiv.org/abs/2408.08105)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have showcased exceptional ability in causal reasoning from textual information. However, will these causalities remain straightforward for Vision Large Language Models (VLLMs) when only visual hints are provided? Motivated by this, we propose a novel Multimodal Causal Reasoning benchmark, namely MuCR, to challenge VLLMs to infer semantic cause-and-effect relationship when solely relying on visual cues such as action, appearance, clothing, and environment. Specifically, we introduce a prompt-driven image synthesis approach to create siamese images with embedded semantic causality and visual cues, which can effectively evaluate VLLMs' causal reasoning capabilities. Additionally, we develop tailored metrics from multiple perspectives, including image-level match, phrase-level understanding, and sentence-level explanation, to comprehensively assess VLLMs' comprehension abilities. Our extensive experiments reveal that the current state-of-the-art VLLMs are not as skilled at multimodal causal reasoning as we might have hoped. Furthermore, we perform a comprehensive analysis to understand these models' shortcomings from different views and suggest directions for future research. We hope MuCR can serve as a valuable resource and foundational benchmark in multimodal causal reasoning research. The project is available at: this https URL

Title: Unsupervised Part Discovery via Dual Representation Alignment

• Authors: Jiahao Xia, Wenjian Huang, Min Xu, Jianguo Zhang, Haimin Zhang, Ziyu Sheng, Dong Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08108
• Pdf URL: https://arxiv.org/pdf/2408.08108
• Copy Paste: [[2408.08108]] Unsupervised Part Discovery via Dual Representation Alignment(https://arxiv.org/abs/2408.08108)
• Keywords: robust, transformer
• Abstract: Object parts serve as crucial intermediate representations in various downstream tasks, but part-level representation learning still has not received as much attention as other vision tasks. Previous research has established that Vision Transformer can learn instance-level attention without labels, extracting high-quality instance-level representations for boosting downstream tasks. In this paper, we achieve unsupervised part-specific attention learning using a novel paradigm and further employ the part representations to improve part discovery performance. Specifically, paired images are generated from the same image with different geometric transformations, and multiple part representations are extracted from these paired images using a novel module, named PartFormer. These part representations from the paired images are then exchanged to improve geometric transformation invariance. Subsequently, the part representations are aligned with the feature map extracted by a feature map encoder, achieving high similarity with the pixel representations of the corresponding part regions and low similarity in irrelevant regions. Finally, the geometric and semantic constraints are applied to the part representations through the intermediate results in alignment for part-specific attention learning, encouraging the PartFormer to focus locally and the part representations to explicitly include the information of the corresponding parts. Moreover, the aligned part representations can further serve as a series of reliable detectors in the testing phase, predicting pixel masks for part discovery. Extensive experiments are carried out on four widely used datasets, and our results demonstrate that the proposed method achieves competitive performance and robustness due to its part-specific attention.

Title: EXPLAIN, AGREE, LEARN: Scaling Learning for Neural Probabilistic Logic

• Authors: Victor Verreet, Lennert De Smet, Luc De Raedt, Emanuele Sansone
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08133
• Pdf URL: https://arxiv.org/pdf/2408.08133
• Copy Paste: [[2408.08133]] EXPLAIN, AGREE, LEARN: Scaling Learning for Neural Probabilistic Logic(https://arxiv.org/abs/2408.08133)
• Keywords: robust
• Abstract: Neural probabilistic logic systems follow the neuro-symbolic (NeSy) paradigm by combining the perceptive and learning capabilities of neural networks with the robustness of probabilistic logic. Learning corresponds to likelihood optimization of the neural networks. However, to obtain the likelihood exactly, expensive probabilistic logic inference is required. To scale learning to more complex systems, we therefore propose to instead optimize a sampling based objective. We prove that the objective has a bounded error with respect to the likelihood, which vanishes when increasing the sample count. Furthermore, the error vanishes faster by exploiting a new concept of sample diversity. We then develop the EXPLAIN, AGREE, LEARN (EXAL) method that uses this objective. EXPLAIN samples explanations for the data. AGREE reweighs each explanation in concordance with the neural component. LEARN uses the reweighed explanations as a signal for learning. In contrast to previous NeSy methods, EXAL can scale to larger problem sizes while retaining theoretical guarantees on the error. Experimentally, our theoretical claims are verified and EXAL outperforms recent NeSy methods when scaling up the MNIST addition and Warcraft pathfinding problems.

Title: CorrAdaptor: Adaptive Local Context Learning for Correspondence Pruning

• Authors: Wei Zhu, Yicheng Liu, Yuping He, Tangfei Liao, Kang Zheng, Xiaoqiu Xu, Tao Wang, Tong Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08134
• Pdf URL: https://arxiv.org/pdf/2408.08134
• Copy Paste: [[2408.08134]] CorrAdaptor: Adaptive Local Context Learning for Correspondence Pruning(https://arxiv.org/abs/2408.08134)
• Keywords: robust
• Abstract: In the fields of computer vision and robotics, accurate pixel-level correspondences are essential for enabling advanced tasks such as structure-from-motion and simultaneous localization and mapping. Recent correspondence pruning methods usually focus on learning local consistency through k-nearest neighbors, which makes it difficult to capture robust context for each correspondence. We propose CorrAdaptor, a novel architecture that introduces a dual-branch structure capable of adaptively adjusting local contexts through both explicit and implicit local graph learning. Specifically, the explicit branch uses KNN-based graphs tailored for initial neighborhood identification, while the implicit branch leverages a learnable matrix to softly assign neighbors and adaptively expand the local context scope, significantly enhancing the model's robustness and adaptability to complex image variations. Moreover, we design a motion injection module to integrate motion consistency into the network to suppress the impact of outliers and refine local context learning, resulting in substantial performance improvements. The experimental results on extensive correspondence-based tasks indicate that our CorrAdaptor achieves state-of-the-art performance both qualitatively and quantitatively. The code and pre-trained models are available at this https URL.

Title: Normalized AOPC: Fixing Misleading Faithfulness Metrics for Feature Attribution Explainability

• Authors: Joakim Edin, Andreas Geert Motzfeldt, Casper L. Christensen, Tuukka Ruotsalo, Lars Maaløe, Maria Maistro
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08137
• Pdf URL: https://arxiv.org/pdf/2408.08137
• Copy Paste: [[2408.08137]] Normalized AOPC: Fixing Misleading Faithfulness Metrics for Feature Attribution Explainability(https://arxiv.org/abs/2408.08137)
• Keywords: robust, explainability
• Abstract: Deep neural network predictions are notoriously difficult to interpret. Feature attribution methods aim to explain these predictions by identifying the contribution of each input feature. Faithfulness, often evaluated using the area over the perturbation curve (AOPC), reflects feature attributions' accuracy in describing the internal mechanisms of deep neural networks. However, many studies rely on AOPC to compare faithfulness across different models, which we show can lead to false conclusions about models' faithfulness. Specifically, we find that AOPC is sensitive to variations in the model, resulting in unreliable cross-model comparisons. Moreover, AOPC scores are difficult to interpret in isolation without knowing the model-specific lower and upper limits. To address these issues, we propose a normalization approach, Normalized AOPC (NAOPC), enabling consistent cross-model evaluations and more meaningful interpretation of individual scores. Our experiments demonstrate that this normalization can radically change AOPC results, questioning the conclusions of earlier studies and offering a more robust framework for assessing feature attribution faithfulness.

Title: Unlearnable Examples Detection via Iterative Filtering

• Authors: Yi Yu, Qichen Zheng, Siyuan Yang, Wenhan Yang, Jun Liu, Shijian Lu, Yap-Peng Tan, Kwok-Yan Lam, Alex Kot
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08143
• Pdf URL: https://arxiv.org/pdf/2408.08143
• Copy Paste: [[2408.08143]] Unlearnable Examples Detection via Iterative Filtering(https://arxiv.org/abs/2408.08143)
• Keywords: attack
• Abstract: Deep neural networks are proven to be vulnerable to data poisoning attacks. Recently, a specific type of data poisoning attack known as availability attacks has led to the failure of data utilization for model learning by adding imperceptible perturbations to images. Consequently, it is quite beneficial and challenging to detect poisoned samples, also known as Unlearnable Examples (UEs), from a mixed dataset. In response, we propose an Iterative Filtering approach for UEs identification. This method leverages the distinction between the inherent semantic mapping rules and shortcuts, without the need for any additional information. We verify that when training a classifier on a mixed dataset containing both UEs and clean data, the model tends to quickly adapt to the UEs compared to the clean data. Due to the accuracy gaps between training with clean/poisoned samples, we employ a model to misclassify clean samples while correctly identifying the poisoned ones. The incorporation of additional classes and iterative refinement enhances the model's ability to differentiate between clean and poisoned samples. Extensive experiments demonstrate the superiority of our method over state-of-the-art detection approaches across various attacks, datasets, and poison ratios, significantly reducing the Half Total Error Rate (HTER) compared to existing methods.

Title: MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU

• Authors: Yan Li, So-Eon Kim, Seong-Bae Park, Soyeon Caren Han
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08144
• Pdf URL: https://arxiv.org/pdf/2408.08144
• Copy Paste: [[2408.08144]] MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU(https://arxiv.org/abs/2408.08144)
• Keywords: large language model
• Abstract: Although Large Language Models(LLMs) can generate coherent and contextually relevant text, they often struggle to recognise the intent behind the human user's query. Natural Language Understanding (NLU) models, however, interpret the purpose and key information of user's input to enable responsive interactions. Existing NLU models generally map individual utterances to a dual-level semantic frame, involving sentence-level intent and word-level slot labels. However, real-life conversations primarily consist of multi-turn conversations, involving the interpretation of complex and extended dialogues. Researchers encounter challenges addressing all facets of multi-turn dialogue conversations using a unified single NLU model. This paper introduces a novel approach, MIDAS, leveraging a multi-level intent, domain, and slot knowledge distillation for multi-turn NLU. To achieve this, we construct distinct teachers for varying levels of conversation knowledge, namely, sentence-level intent detection, word-level slot filling, and conversation-level domain classification. These teachers are then fine-tuned to acquire specific knowledge of their designated levels. A multi-teacher loss is proposed to facilitate the combination of these multi-level teachers, guiding a student model in multi-turn dialogue tasks. The experimental results demonstrate the efficacy of our model in improving the overall multi-turn conversation understanding, showcasing the potential for advancements in NLU models through the incorporation of multi-level dialogue knowledge distillation techniques.

Title: KOALA: Enhancing Speculative Decoding for LLM via Multi-Layer Draft Heads with Adversarial Learning

• Authors: Kaiqi Zhang, Jing Zhao, Rui Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08146
• Pdf URL: https://arxiv.org/pdf/2408.08146
• Copy Paste: [[2408.08146]] KOALA: Enhancing Speculative Decoding for LLM via Multi-Layer Draft Heads with Adversarial Learning(https://arxiv.org/abs/2408.08146)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) exhibit high inference latency due to their autoregressive decoding nature. While the draft head in speculative decoding mitigates this issue, its full potential remains unexplored. In this paper, we introduce KOALA (K-layer Optimized Adversarial Learning Architecture), an orthogonal approach to the draft head. By transforming the conventional single-layer draft head into a multi-layer architecture and incorporating adversarial learning into the traditional supervised training, KOALA significantly improves the accuracy of the draft head in predicting subsequent tokens, thus more closely mirroring the functionality of LLMs. Although this improvement comes at the cost of slightly increased drafting overhead, KOALA substantially unlocks the draft head's potential, greatly enhancing speculative decoding. We conducted comprehensive evaluations of KOALA, including both autoregressive and non-autoregressive draft heads across various tasks, demonstrating a latency speedup ratio improvement of 0.24x-0.41x, which is 10.57%-14.09% faster than the original draft heads.

Title: Your Turn: Real-World Turning Angle Estimation for Parkinson's Disease Severity Assessment

• Authors: Qiushuo Cheng, Catherine Morgan, Arindam Sikdar, Alessandro Masullo, Alan Whone, Majid Mirmehdi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08182
• Pdf URL: https://arxiv.org/pdf/2408.08182
• Copy Paste: [[2408.08182]] Your Turn: Real-World Turning Angle Estimation for Parkinson's Disease Severity Assessment(https://arxiv.org/abs/2408.08182)
• Keywords: transformer
• Abstract: People with Parkinson's Disease (PD) often experience progressively worsening gait, including changes in how they turn around, as the disease progresses. Existing clinical rating tools are not capable of capturing hour-by-hour variations of PD symptoms, as they are confined to brief assessments within clinic settings. Measuring real-world gait turning angles continuously and passively is a component step towards using gait characteristics as sensitive indicators of disease progression in PD. This paper presents a deep learning-based approach to automatically quantify turning angles by extracting 3D skeletons from videos and calculating the rotation of hip and knee joints. We utilise state-of-the-art human pose estimation models, Fastpose and Strided Transformer, on a total of 1386 turning video clips from 24 subjects (12 people with PD and 12 healthy control volunteers), trimmed from a PD dataset of unscripted free-living videos in a home-like setting (Turn-REMAP). We also curate a turning video dataset, Turn-H3.6M, from the public Human3.6M human pose benchmark with 3D ground truth, to further validate our method. Previous gait research has primarily taken place in clinics or laboratories evaluating scripted gait outcomes, but this work focuses on real-world settings where complexities exist, such as baggy clothing and poor lighting. Due to difficulties in obtaining accurate ground truth data in a free-living setting, we quantise the angle into the nearest bin $45^\circ$ based on the manual labelling of expert clinicians. Our method achieves a turning calculation accuracy of 41.6%, a Mean Absolute Error (MAE) of 34.7°, and a weighted precision WPrec of 68.3% for Turn-REMAP. This is the first work to explore the use of single monocular camera data to quantify turns by PD patients in a home setting.

Title: Not Every Image is Worth a Thousand Words: Quantifying Originality in Stable Diffusion

• Authors: Adi Haviv, Shahar Sarfaty, Uri Hacohen, Niva Elkin-Koren, Roi Livni, Amit H Bermano
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08184
• Pdf URL: https://arxiv.org/pdf/2408.08184
• Copy Paste: [[2408.08184]] Not Every Image is Worth a Thousand Words: Quantifying Originality in Stable Diffusion(https://arxiv.org/abs/2408.08184)
• Keywords: diffusion, generative
• Abstract: This work addresses the challenge of quantifying originality in text-to-image (T2I) generative diffusion models, with a focus on copyright originality. We begin by evaluating T2I models' ability to innovate and generalize through controlled experiments, revealing that stable diffusion models can effectively recreate unseen elements with sufficiently diverse training data. Then, our key insight is that concepts and combinations of image elements the model is familiar with, and saw more during training, are more concisly represented in the model's latent space. We hence propose a method that leverages textual inversion to measure the originality of an image based on the number of tokens required for its reconstruction by the model. Our approach is inspired by legal definitions of originality and aims to assess whether a model can produce original content without relying on specific prompts or having the training data of the model. We demonstrate our method using both a pre-trained stable diffusion model and a synthetic dataset, showing a correlation between the number of tokens and image originality. This work contributes to the understanding of originality in generative models and has implications for copyright infringement cases.

Title: Towards Practical Human Motion Prediction with LiDAR Point Clouds

• Authors: Xiao Han, Yiming Ren, Yichen Yao, Yujing Sun, Yuexin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08202
• Pdf URL: https://arxiv.org/pdf/2408.08202
• Copy Paste: [[2408.08202]] Towards Practical Human Motion Prediction with LiDAR Point Clouds(https://arxiv.org/abs/2408.08202)
• Keywords: robust
• Abstract: Human motion prediction is crucial for human-centric multimedia understanding and interacting. Current methods typically rely on ground truth human poses as observed input, which is not practical for real-world scenarios where only raw visual sensor data is available. To implement these methods in practice, a pre-phrase of pose estimation is essential. However, such two-stage approaches often lead to performance degradation due to the accumulation of errors. Moreover, reducing raw visual data to sparse keypoint representations significantly diminishes the density of information, resulting in the loss of fine-grained features. In this paper, we propose \textit{LiDAR-HMP}, the first single-LiDAR-based 3D human motion prediction approach, which receives the raw LiDAR point cloud as input and forecasts future 3D human poses directly. Building upon our novel structure-aware body feature descriptor, LiDAR-HMP adaptively maps the observed motion manifold to future poses and effectively models the spatial-temporal correlations of human motions for further refinement of prediction results. Extensive experiments show that our method achieves state-of-the-art performance on two public benchmarks and demonstrates remarkable robustness and efficacy in real-world deployments.

Title: A Multi-task Adversarial Attack Against Face Authentication

• Authors: Hanrui Wang, Shuo Wang, Cunjian Chen, Massimo Tistarelli, Zhe Jin
• Subjects: cs.CV, cs.CR, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.08205
• Pdf URL: https://arxiv.org/pdf/2408.08205
• Copy Paste: [[2408.08205]] A Multi-task Adversarial Attack Against Face Authentication(https://arxiv.org/abs/2408.08205)
• Keywords: attack
• Abstract: Deep-learning-based identity management systems, such as face authentication systems, are vulnerable to adversarial attacks. However, existing attacks are typically designed for single-task purposes, which means they are tailored to exploit vulnerabilities unique to the individual target rather than being adaptable for multiple users or systems. This limitation makes them unsuitable for certain attack scenarios, such as morphing, universal, transferable, and counter attacks. In this paper, we propose a multi-task adversarial attack algorithm called MTADV that are adaptable for multiple users or systems. By interpreting these scenarios as multi-task attacks, MTADV is applicable to both single- and multi-task attacks, and feasible in the white- and gray-box settings. Furthermore, MTADV is effective against various face datasets, including LFW, CelebA, and CelebA-HQ, and can work with different deep learning models, such as FaceNet, InsightFace, and CurricularFace. Importantly, MTADV retains its feasibility as a single-task attack targeting a single user/system. To the best of our knowledge, MTADV is the first adversarial attack method that can target all of the aforementioned scenarios in one algorithm.

Title: Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models

• Authors: Javier González, Aditya V. Nori
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08210
• Pdf URL: https://arxiv.org/pdf/2408.08210
• Copy Paste: [[2408.08210]] Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models(https://arxiv.org/abs/2408.08210)
• Keywords: large language model
• Abstract: Recent advances in AI have been significantly driven by the capabilities of large language models (LLMs) to solve complex problems in ways that resemble human thinking. However, there is an ongoing debate about the extent to which LLMs are capable of actual reasoning. Central to this debate are two key probabilistic concepts that are essential for connecting causes to their effects: the probability of necessity (PN) and the probability of sufficiency (PS). This paper introduces a framework that is both theoretical and practical, aimed at assessing how effectively LLMs are able to replicate real-world reasoning mechanisms using these probabilistic measures. By viewing LLMs as abstract machines that process information through a natural language interface, we examine the conditions under which it is possible to compute suitable approximations of PN and PS. Our research marks an important step towards gaining a deeper understanding of when LLMs are capable of reasoning, as illustrated by a series of math examples.

Title: Federated Fairness Analytics: Quantifying Fairness in Federated Learning

• Authors: Oscar Dilley, Juan Marcelo Parra-Ullauri, Rasheed Hussain, Dimitra Simeonidou
• Subjects: cs.LG, cs.AI, cs.DC, cs.GT, cs.NE
• Abstract URL: https://arxiv.org/abs/2408.08214
• Pdf URL: https://arxiv.org/pdf/2408.08214
• Copy Paste: [[2408.08214]] Federated Fairness Analytics: Quantifying Fairness in Federated Learning(https://arxiv.org/abs/2408.08214)
• Keywords: privacy, federate, fair
• Abstract: Federated Learning (FL) is a privacy-enhancing technology for distributed ML. By training models locally and aggregating updates - a federation learns together, while bypassing centralised data collection. FL is increasingly popular in healthcare, finance and personal computing. However, it inherits fairness challenges from classical ML and introduces new ones, resulting from differences in data quality, client participation, communication constraints, aggregation methods and underlying hardware. Fairness remains an unresolved issue in FL and the community has identified an absence of succinct definitions and metrics to quantify fairness; to address this, we propose Federated Fairness Analytics - a methodology for measuring fairness. Our definition of fairness comprises four notions with novel, corresponding metrics. They are symptomatically defined and leverage techniques originating from XAI, cooperative game-theory and networking engineering. We tested a range of experimental settings, varying the FL approach, ML task and data settings. The results show that statistical heterogeneity and client participation affect fairness and fairness conscious approaches such as Ditto and q-FedAvg marginally improve fairness-performance trade-offs. Using our techniques, FL practitioners can uncover previously unobtainable insights into their system's fairness, at differing levels of granularity in order to address fairness challenges in FL. We have open-sourced our work at: this https URL.

Title: The Dawn of KAN in Image-to-Image (I2I) Translation: Integrating Kolmogorov-Arnold Networks with GANs for Unpaired I2I Translation

• Authors: Arpan Mahara, Naphtali D. Rishe, Liangdong Deng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08216
• Pdf URL: https://arxiv.org/pdf/2408.08216
• Copy Paste: [[2408.08216]] The Dawn of KAN in Image-to-Image (I2I) Translation: Integrating Kolmogorov-Arnold Networks with GANs for Unpaired I2I Translation(https://arxiv.org/abs/2408.08216)
• Keywords: generative
• Abstract: Image-to-Image translation in Generative Artificial Intelligence (Generative AI) has been a central focus of research, with applications spanning healthcare, remote sensing, physics, chemistry, photography, and more. Among the numerous methodologies, Generative Adversarial Networks (GANs) with contrastive learning have been particularly successful. This study aims to demonstrate that the Kolmogorov-Arnold Network (KAN) can effectively replace the Multi-layer Perceptron (MLP) method in generative AI, particularly in the subdomain of image-to-image translation, to achieve better generative quality. Our novel approach replaces the two-layer MLP with a two-layer KAN in the existing Contrastive Unpaired Image-to-Image Translation (CUT) model, developing the KAN-CUT model. This substitution favors the generation of more informative features in low-dimensional vector representations, which contrastive learning can utilize more effectively to produce high-quality images in the target domain. Extensive experiments, detailed in the results section, demonstrate the applicability of KAN in conjunction with contrastive learning and GANs in Generative AI, particularly for image-to-image translation. This work suggests that KAN could be a valuable component in the broader generative AI domain.

Title: RED-CT: A Systems Design Methodology for Using LLM-labeled Data to Train and Deploy Edge Classifiers for Computational Social Science

• Authors: David Farr, Nico Manzonelli, Iain Cruickshank, Jevin West
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2408.08217
• Pdf URL: https://arxiv.org/pdf/2408.08217
• Copy Paste: [[2408.08217]] RED-CT: A Systems Design Methodology for Using LLM-labeled Data to Train and Deploy Edge Classifiers for Computational Social Science(https://arxiv.org/abs/2408.08217)
• Keywords: security, large language model
• Abstract: Large language models (LLMs) have enhanced our ability to rapidly analyze and classify unstructured natural language data. However, concerns regarding cost, network limitations, and security constraints have posed challenges for their integration into work processes. In this study, we adopt a systems design approach to employing LLMs as imperfect data annotators for downstream supervised learning tasks, introducing novel system intervention measures aimed at improving classification performance. Our methodology outperforms LLM-generated labels in seven of eight tests, demonstrating an effective strategy for incorporating LLMs into the design and deployment of specialized, supervised learning models present in many industry use cases.

Title: Computer Vision Model Compression Techniques for Embedded Systems: A Survey

• Authors: Alexandre Lopes, Fernando Pereira dos Santos, Diulhio de Oliveira, Mauricio Schiezaro, Helio Pedrini
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08250
• Pdf URL: https://arxiv.org/pdf/2408.08250
• Copy Paste: [[2408.08250]] Computer Vision Model Compression Techniques for Embedded Systems: A Survey(https://arxiv.org/abs/2408.08250)
• Keywords: transformer
• Abstract: Deep neural networks have consistently represented the state of the art in most computer vision problems. In these scenarios, larger and more complex models have demonstrated superior performance to smaller architectures, especially when trained with plenty of representative data. With the recent adoption of Vision Transformer (ViT) based architectures and advanced Convolutional Neural Networks (CNNs), the total number of parameters of leading backbone architectures increased from 62M parameters in 2012 with AlexNet to 7B parameters in 2024 with AIM-7B. Consequently, deploying such deep architectures faces challenges in environments with processing and runtime constraints, particularly in embedded systems. This paper covers the main model compression techniques applied for computer vision tasks, enabling modern models to be used in embedded systems. We present the characteristics of compression subareas, compare different approaches, and discuss how to choose the best technique and expected variations when analyzing it on various embedded devices. We also share codes to assist researchers and new practitioners in overcoming initial implementation challenges for each subarea and present trends for Model Compression. Case studies for compression models are available at \href{this https URL}{this https URL}.

Title: Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding

• Authors: Xiner Li, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokcen Eraslan, Surag Nair, Tommaso Biancalani, Aviv Regev, Sergey Levine, Masatoshi Uehara
• Subjects: cs.LG, cs.AI, q-bio.GN, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.08252
• Pdf URL: https://arxiv.org/pdf/2408.08252
• Copy Paste: [[2408.08252]] Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding(https://arxiv.org/abs/2408.08252)
• Keywords: diffusion, generative
• Abstract: Diffusion models excel at capturing the natural design spaces of images, molecules, DNA, RNA, and protein sequences. However, rather than merely generating designs that are natural, we often aim to optimize downstream reward functions while preserving the naturalness of these design spaces. Existing methods for achieving this goal often require ``differentiable'' proxy models (\textit{e.g.}, classifier guidance or DPS) or involve computationally expensive fine-tuning of diffusion models (\textit{e.g.}, classifier-free guidance, RL-based fine-tuning). In our work, we propose a new method to address these challenges. Our algorithm is an iterative sampling method that integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future, into the standard inference procedure of pre-trained diffusion models. Notably, our approach avoids fine-tuning generative models and eliminates the need to construct differentiable models. This enables us to (1) directly utilize non-differentiable features/reward feedback, commonly used in many scientific domains, and (2) apply our method to recent discrete diffusion models in a principled way. Finally, we demonstrate the effectiveness of our algorithm across several domains, including image generation, molecule generation, and DNA/RNA sequence generation. The code is available at \href{this https URL}{this https URL}.

Title: Snuffy: Efficient Whole Slide Image Classifier

• Authors: Hossein Jafarinia, Alireza Alipanah, Danial Hamdi, Saeed Razavi, Nahal Mirzaie, Mohammad Hossein Rohban
• Subjects: cs.CV, cs.AI, cs.LG, cs.NE, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.08258
• Pdf URL: https://arxiv.org/pdf/2408.08258
• Copy Paste: [[2408.08258]] Snuffy: Efficient Whole Slide Image Classifier(https://arxiv.org/abs/2408.08258)
• Keywords: transformer
• Abstract: Whole Slide Image (WSI) classification with multiple instance learning (MIL) in digital pathology faces significant computational challenges. Current methods mostly rely on extensive self-supervised learning (SSL) for satisfactory performance, requiring long training periods and considerable computational resources. At the same time, no pre-training affects performance due to domain shifts from natural images to WSIs. We introduce \textbf{\textit{Snuffy}} architecture, a novel MIL-pooling method based on sparse transformers that mitigates performance loss with limited pre-training and enables continual few-shot pre-training as a competitive option. Our sparsity pattern is tailored for pathology and is theoretically proven to be a universal approximator with the tightest probabilistic sharp bound on the number of layers for sparse transformers, to date. We demonstrate Snuffy's effectiveness on CAMELYON16 and TCGA Lung cancer datasets, achieving superior WSI and patch-level accuracies. The code is available on \url{this https URL}.

Title: mhGPT: A Lightweight Generative Pre-Trained Transformer for Mental Health Text Analysis

• Authors: Dae-young Kim (1), Rebecca Hwa (2), Muhammad Mahbubur Rahman (1) ((1) Children's National Hospital, Washington, DC, (2) George Washington University, Washington, DC)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08261
• Pdf URL: https://arxiv.org/pdf/2408.08261
• Copy Paste: [[2408.08261]] mhGPT: A Lightweight Generative Pre-Trained Transformer for Mental Health Text Analysis(https://arxiv.org/abs/2408.08261)
• Keywords: transformer, generative
• Abstract: This paper introduces mhGPT, a lightweight generative pre-trained transformer trained on mental health-related social media and PubMed articles. Fine-tuned for specific mental health tasks, mhGPT was evaluated under limited hardware constraints and compared with state-of-the-art models like MentaLLaMA and Gemma. Despite having only 1.98 billion parameters and using just 5% of the dataset, mhGPT outperformed larger models and matched the performance of models trained on significantly more data. The key contributions include integrating diverse mental health data, creating a custom tokenizer, and optimizing a smaller architecture for low-resource settings. This research could advance AI-driven mental health care, especially in areas with limited computing power.

Title: HeightLane: BEV Heightmap guided 3D Lane Detection

• Authors: Chaesong Park, Eunbin Seo, Jongwoo Lim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08270
• Pdf URL: https://arxiv.org/pdf/2408.08270
• Copy Paste: [[2408.08270]] HeightLane: BEV Heightmap guided 3D Lane Detection(https://arxiv.org/abs/2408.08270)
• Keywords: extraction
• Abstract: Accurate 3D lane detection from monocular images presents significant challenges due to depth ambiguity and imperfect ground modeling. Previous attempts to model the ground have often used a planar ground assumption with limited degrees of freedom, making them unsuitable for complex road environments with varying slopes. Our study introduces HeightLane, an innovative method that predicts a height map from monocular images by creating anchors based on a multi-slope assumption. This approach provides a detailed and accurate representation of the ground. HeightLane employs the predicted heightmap along with a deformable attention-based spatial feature transform framework to efficiently convert 2D image features into 3D bird's eye view (BEV) features, enhancing spatial understanding and lane structure recognition. Additionally, the heightmap is used for the positional encoding of BEV features, further improving their spatial accuracy. This explicit view transformation bridges the gap between front-view perceptions and spatially accurate BEV representations, significantly improving detection performance. To address the lack of the necessary ground truth (GT) height map in the original OpenLane dataset, we leverage the Waymo dataset and accumulate its LiDAR data to generate a height map for the drivable area of each scene. The GT heightmaps are used to train the heightmap extraction module from monocular images. Extensive experiments on the OpenLane validation set show that HeightLane achieves state-of-the-art performance in terms of F-score, highlighting its potential in real-world applications.

Title: BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts

• Authors: Qizhen Zhang, Nikolas Gritsch, Dwaraknath Gnaneshwar, Simon Guo, David Cairuz, Bharat Venkitesh, Jakob Foerster, Phil Blunsom, Sebastian Ruder, Ahmet Ustun, Acyr Locatelli
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.08274
• Pdf URL: https://arxiv.org/pdf/2408.08274
• Copy Paste: [[2408.08274]] BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts(https://arxiv.org/abs/2408.08274)
• Keywords: transformer, large language model
• Abstract: The Mixture of Experts (MoE) framework has become a popular architecture for large language models due to its superior performance over dense models. However, training MoEs from scratch in a large-scale regime is prohibitively expensive. Existing methods mitigate this by pre-training multiple dense expert models independently and using them to initialize an MoE. This is done by using experts' feed-forward network (FFN) to initialize the MoE's experts while merging other parameters. However, this method limits the reuse of dense model parameters to only the FFN layers, thereby constraining the advantages when "upcycling" these models into MoEs. We propose BAM (Branch-Attend-Mix), a simple yet effective method that addresses this shortcoming. BAM makes full use of specialized dense models by not only using their FFN to initialize the MoE layers but also leveraging experts' attention parameters fully by initializing them into a soft-variant of Mixture of Attention (MoA) layers. We explore two methods for upcycling attention parameters: 1) initializing separate attention experts from dense models including all attention parameters for the best model performance; and 2) sharing key and value parameters across all experts to facilitate for better inference efficiency. To further improve efficiency, we adopt a parallel attention transformer architecture to MoEs, which allows the attention experts and FFN experts to be computed concurrently. Our experiments on seed models ranging from 590 million to 2 billion parameters demonstrate that BAM surpasses baselines in both perplexity and downstream task performance, within the same computational and data constraints.

Title: The ShareLM Collection and Plugin: Contributing Human-Model Chats for the Benefit of the Community

• Authors: Shachar Don-Yehiya, Leshem Choshen, Omri Abend
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08291
• Pdf URL: https://arxiv.org/pdf/2408.08291
• Copy Paste: [[2408.08291]] The ShareLM Collection and Plugin: Contributing Human-Model Chats for the Benefit of the Community(https://arxiv.org/abs/2408.08291)
• Keywords: large language model
• Abstract: Human-model conversations provide a window into users' real-world scenarios, behavior, and needs, and thus are a valuable resource for model development and research. While for-profit companies collect user data through the APIs of their models, using it internally to improve their own models, the open source and research community lags behind. We introduce the ShareLM collection, a unified set of human conversations with large language models, and its accompanying plugin, a Web extension for voluntarily contributing user-model conversations. Where few platforms share their chats, the ShareLM plugin adds this functionality, thus, allowing users to share conversations from most platforms. The plugin allows the user to rate their conversations, both at the conversation and the response levels, and delete conversations they prefer to keep private before they ever leave the user's local storage. We release the plugin conversations as part of the ShareLM collection, and call for more community effort in the field of open human-model data. The code, plugin, and data are available.

Title: Towards Flexible Visual Relationship Segmentation

• Authors: Fangrui Zhu, Jianwei Yang, Huaizu Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08305
• Pdf URL: https://arxiv.org/pdf/2408.08305
• Copy Paste: [[2408.08305]] Towards Flexible Visual Relationship Segmentation(https://arxiv.org/abs/2408.08305)
• Keywords: segmentation
• Abstract: Visual relationship understanding has been studied separately in human-object interaction(HOI) detection, scene graph generation(SGG), and referring relationships(RR) tasks. Given the complexity and interconnectedness of these tasks, it is crucial to have a flexible framework that can effectively address these tasks in a cohesive manner. In this work, we propose FleVRS, a single model that seamlessly integrates the above three aspects in standard and promptable visual relationship segmentation, and further possesses the capability for open-vocabulary segmentation to adapt to novel scenarios. FleVRS leverages the synergy between text and image modalities, to ground various types of relationships from images and use textual features from vision-language models to visual conceptual understanding. Empirical validation across various datasets demonstrates that our framework outperforms existing models in standard, promptable, and open-vocabulary tasks, e.g., +1.9 $mAP$ on HICO-DET, +11.4 $Acc$ on VRD, +4.7 $mAP$ on unseen HICO-DET. Our FleVRS represents a significant step towards a more intuitive, comprehensive, and scalable understanding of visual relationships.

Title: Understanding the Local Geometry of Generative Model Manifolds

• Authors: Ahmed Imtiaz Humayun, Ibtihel Amara, Candice Schumann, Golnoosh Farnadi, Negar Rostamzadeh, Mohammad Havaei
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08307
• Pdf URL: https://arxiv.org/pdf/2408.08307
• Copy Paste: [[2408.08307]] Understanding the Local Geometry of Generative Model Manifolds(https://arxiv.org/abs/2408.08307)
• Keywords: diffusion, generative
• Abstract: Deep generative models learn continuous representations of complex data manifolds using a finite number of samples during training. For a pre-trained generative model, the common way to evaluate the quality of the manifold representation learned, is by computing global metrics like Fréchet Inception Distance using a large number of generated and real samples. However, generative model performance is not uniform across the learned manifold, e.g., for \textit{foundation models} like Stable Diffusion generation performance can vary significantly based on the conditioning or initial noise vector being denoised. In this paper we study the relationship between the \textit{local geometry of the learned manifold} and downstream generation. Based on the theory of continuous piecewise-linear (CPWL) generators, we use three geometric descriptors - scaling ($\psi$), rank ($\nu$), and complexity ($\delta$) - to characterize a pre-trained generative model manifold locally. We provide quantitative and qualitative evidence showing that for a given latent, the local descriptors are correlated with generation aesthetics, artifacts, uncertainty, and even memorization. Finally we demonstrate that training a \textit{reward model} on the local geometry can allow controlling the likelihood of a generated sample under the learned distribution.

Title: ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws

• Authors: Ruihang Li, Yixuan Wei, Miaosen Zhang, Nenghai Yu, Han Hu, Houwen Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08310
• Pdf URL: https://arxiv.org/pdf/2408.08310
• Copy Paste: [[2408.08310]] ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws(https://arxiv.org/abs/2408.08310)
• Keywords: large language model
• Abstract: High-quality data is crucial for the pre-training performance of large language models. Unfortunately, existing quality filtering methods rely on a known high-quality dataset as reference, which can introduce potential bias and compromise diversity. In this paper, we propose ScalingFilter, a novel approach that evaluates text quality based on the perplexity difference between two language models trained on the same data, thereby eliminating the influence of the reference dataset in the filtering process. An theoretical analysis shows that ScalingFilter is equivalent to an inverse utilization of scaling laws. Through training models with 1.3B parameters on the same data source processed by various quality filters, we find ScalingFilter can improve zero-shot performance of pre-trained models in downstream tasks. To assess the bias introduced by quality filtering, we introduce semantic diversity, a metric of utilizing text embedding models for semantic representations. Extensive experiments reveal that semantic diversity is a reliable indicator of dataset diversity, and ScalingFilter achieves an optimal balance between downstream performance and semantic diversity.

Title: Can Large Language Models Understand Symbolic Graphics Programs?

• Authors: Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu, Tim Z. Xiao, Katherine M. Collins, Joshua B. Tenenbaum, Adrian Weller, Michael J. Black, Bernhard Schölkopf
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.08313
• Pdf URL: https://arxiv.org/pdf/2408.08313
• Copy Paste: [[2408.08313]] Can Large Language Models Understand Symbolic Graphics Programs?(https://arxiv.org/abs/2408.08313)
• Keywords: large language model
• Abstract: Assessing the capabilities of large language models (LLMs) is often challenging, in part, because it is hard to find tasks to which they have not been exposed during training. We take one step to address this challenge by turning to a new task: focusing on symbolic graphics programs, which are a popular representation for graphics content that procedurally generates visual data. LLMs have shown exciting promise towards program synthesis, but do they understand symbolic graphics programs? Unlike conventional programs, symbolic graphics programs can be translated to graphics content. Here, we characterize an LLM's understanding of symbolic programs in terms of their ability to answer questions related to the graphics content. This task is challenging as the questions are difficult to answer from the symbolic programs alone -- yet, they would be easy to answer from the corresponding graphics content as we verify through a human experiment. To understand symbolic programs, LLMs may need to possess the ability to imagine how the corresponding graphics content would look without directly accessing the rendered visual content. We use this task to evaluate LLMs by creating a large benchmark for the semantic understanding of symbolic graphics programs. This benchmark is built via program-graphics correspondence, hence requiring minimal human efforts. We evaluate current LLMs on our benchmark to elucidate a preliminary assessment of their ability to reason about visual scenes from programs. We find that this task distinguishes existing LLMs and models considered good at reasoning perform better. Lastly, we introduce Symbolic Instruction Tuning (SIT) to improve this ability. Specifically, we query GPT4-o with questions and images generated by symbolic programs. Such data are then used to finetune an LLM. We also find that SIT data can improve the general instruction following ability of LLMs.