2024-08-09

Title: Taxonomy Driven Fast Adversarial Training

• Authors: Kun Tong, Chengze Jiang, Jie Gui, Yuan Cao
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03944
• Pdf URL: https://arxiv.org/pdf/2408.03944
• Copy Paste: [[2408.03944]] Taxonomy Driven Fast Adversarial Training(https://arxiv.org/abs/2408.03944)
• Keywords: defense, attack, robust
• Abstract: Adversarial training (AT) is an effective defense method against gradient-based attacks to enhance the robustness of neural networks. Among them, single-step AT has emerged as a hotspot topic due to its simplicity and efficiency, requiring only one gradient propagation in generating adversarial examples. Nonetheless, the problem of catastrophic overfitting (CO) that causes training collapse remains poorly understood, and there exists a gap between the robust accuracy achieved through single- and multi-step AT. In this paper, we present a surprising finding that the taxonomy of adversarial examples reveals the truth of CO. Based on this conclusion, we propose taxonomy driven fast adversarial training (TDAT) which jointly optimizes learning objective, loss function, and initialization method, thereby can be regarded as a new paradigm of single-step AT. Compared with other fast AT methods, TDAT can boost the robustness of neural networks, alleviate the influence of misclassified examples, and prevent CO during the training process while requiring almost no additional computational and memory resources. Our method achieves robust accuracy improvement of $1.59\%$, $1.62\%$, $0.71\%$, and $1.26\%$ on CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet-100 datasets, when against projected gradient descent PGD10 attack with perturbation budget 8/255. Furthermore, our proposed method also achieves state-of-the-art robust accuracy against other attacks. Code is available at this https URL.

Title: Impacts of Anthropomorphizing Large Language Models in Learning Environments

• Authors: Kristina Schaaff, Marc-André Heidelmann
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.03945
• Pdf URL: https://arxiv.org/pdf/2408.03945
• Copy Paste: [[2408.03945]] Impacts of Anthropomorphizing Large Language Models in Learning Environments(https://arxiv.org/abs/2408.03945)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are increasingly being used in learning environments to support teaching-be it as learning companions or as tutors. With our contribution, we aim to discuss the implications of the anthropomorphization of LLMs in learning environments on educational theory to build a foundation for more effective learning outcomes and understand their emotional impact on learners. According to the media equation, people tend to respond to media in the same way as they would respond to another person. A study conducted by the Georgia Institute of Technology showed that chatbots can be successfully implemented in learning environments. In this study, learners in selected online courses were unable to distinguish the chatbot from a "real" teacher. As LLM-based chatbots such as OpenAI's GPT series are increasingly used in educational tools, it is important to understand how the attribution processes to LLM-based chatbots in terms of anthropomorphization affect learners' emotions.

Title: Microservice Vulnerability Analysis: A Literature Review with Empirical Insights

• Authors: Raveen Kanishka Jayalath, Hussain Ahmad, Diksha Goel, Muhammad Shuja Syed, Faheem Ullah
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.03960
• Pdf URL: https://arxiv.org/pdf/2408.03960
• Copy Paste: [[2408.03960]] Microservice Vulnerability Analysis: A Literature Review with Empirical Insights(https://arxiv.org/abs/2408.03960)
• Keywords: security, attack
• Abstract: Microservice architectures are revolutionizing both small businesses and large corporations, igniting a new era of innovation with their exceptional advantages in maintainability, reusability, and scalability. However, these benefits come with significant security challenges, as the increased complexity of service interactions, expanded attack surfaces, and intricate dependency management introduce a new array of cybersecurity vulnerabilities. While security concerns are mounting, there is a lack of comprehensive research that integrates a review of existing knowledge with empirical analysis of microservice vulnerabilities. This study aims to fill this gap by gathering, analyzing, and synthesizing existing literature on security vulnerabilities associated with microservice architectures. Through a thorough examination of 62 studies, we identify, analyze, and report 126 security vulnerabilities inherent in microservice architectures. This comprehensive analysis enables us to (i) propose a taxonomy that categorizes microservice vulnerabilities based on the distinctive features of microservice architectures; (ii) conduct an empirical analysis by performing vulnerability scans on four diverse microservice benchmark applications using three different scanning tools to validate our taxonomy; and (iii) map our taxonomy vulnerabilities with empirically identified vulnerabilities, providing an in-depth vulnerability analysis at microservice, application, and scanning tool levels. Our study offers crucial guidelines for practitioners and researchers to advance both the state-of-the-practice and the state-of-the-art in securing microservice architectures.

Title: Enhancing Output Diversity Improves Conjugate Gradient-based Adversarial Attacks

• Authors: Keiichiro Yamamura, Issa Oe, Hiroki Ishikura, Katsuki Fujisawa
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03972
• Pdf URL: https://arxiv.org/pdf/2408.03972
• Copy Paste: [[2408.03972]] Enhancing Output Diversity Improves Conjugate Gradient-based Adversarial Attacks(https://arxiv.org/abs/2408.03972)
• Keywords: attack
• Abstract: Deep neural networks are vulnerable to adversarial examples, and adversarial attacks that generate adversarial examples have been studied in this context. Existing studies imply that increasing the diversity of model outputs contributes to improving the attack performance. This study focuses on the Auto Conjugate Gradient (ACG) attack, which is inspired by the conjugate gradient method and has a high diversification performance. We hypothesized that increasing the distance between two consecutive search points would enhance the output diversity. To test our hypothesis, we propose Rescaling-ACG (ReACG), which automatically modifies the two components that significantly affect the distance between two consecutive search points, including the search direction and step size. ReACG showed higher attack performance than that of ACG, and is particularly effective for ImageNet models with several classification classes. Experimental results show that the distance between two consecutive search points enhances the output diversity and may help develop new potent attacks. The code is available at \url{this https URL}

Title: Image-to-LaTeX Converter for Mathematical Formulas and Text

• Authors: Daniil Gurgurov, Aleksey Morshnev
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04015
• Pdf URL: https://arxiv.org/pdf/2408.04015
• Copy Paste: [[2408.04015]] Image-to-LaTeX Converter for Mathematical Formulas and Text(https://arxiv.org/abs/2408.04015)
• Keywords: transformer
• Abstract: In this project, we train a vision encoder-decoder model to generate LaTeX code from images of mathematical formulas and text. Utilizing a diverse collection of image-to-LaTeX data, we build two models: a base model with a Swin Transformer encoder and a GPT-2 decoder, trained on machine-generated images, and a fine-tuned version enhanced with Low-Rank Adaptation (LoRA) trained on handwritten formulas. We then compare the BLEU performance of our specialized model on a handwritten test set with other similar models, such as Pix2Text, TexTeller, and Sumen. Through this project, we contribute open-source models for converting images to LaTeX and provide from-scratch code for building these models with distributed training and GPU optimizations.

Title: Improving Large Language Model (LLM) fidelity through context-aware grounding: A systematic approach to reliability and veracity

• Authors: Wrick Talukdar, Anjanava Biswas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04023
• Pdf URL: https://arxiv.org/pdf/2408.04023
• Copy Paste: [[2408.04023]] Improving Large Language Model (LLM) fidelity through context-aware grounding: A systematic approach to reliability and veracity(https://arxiv.org/abs/2408.04023)
• Keywords: robust, fair, interpretability, explainability, large language model
• Abstract: As Large Language Models (LLMs) become increasingly sophisticated and ubiquitous in natural language processing (NLP) applications, ensuring their robustness, trustworthiness, and alignment with human values has become a critical challenge. This paper presents a novel framework for contextual grounding in textual models, with a particular emphasis on the Context Representation stage. Our approach aims to enhance the reliability and ethical alignment of these models through a comprehensive, context-aware methodology. By explicitly capturing and representing relevant situational, cultural, and ethical contexts in a machine-readable format, we lay the foundation for anchoring a model's behavior within these contexts. Our approach leverages techniques from knowledge representation and reasoning, such as ontologies, semantic web technologies, and logic-based formalisms. We evaluate our framework on real-world textual datasets, demonstrating its effectiveness in improving model performance, fairness, and alignment with human expectations, while maintaining high accuracy. Furthermore, we discuss the other key components of the framework, including context-aware encoding, context-aware learning, interpretability and explainability, and continuous monitoring and adaptation. This research contributes to the growing body of work on responsible AI, offering a practical approach to developing more reliable, trustworthy, and ethically-aligned language models. Our findings have significant implications for the deployment of LLMs in sensitive domains such as healthcare, legal systems, and social services, where contextual understanding is paramount.

Title: Multimodal Gender Fairness in Depression Prediction: Insights on Data from the USA & China

• Authors: Joseph Cameron, Jiaee Cheong, Micol Spitale, Hatice Gunes
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.04026
• Pdf URL: https://arxiv.org/pdf/2408.04026
• Copy Paste: [[2408.04026]] Multimodal Gender Fairness in Depression Prediction: Insights on Data from the USA & China(https://arxiv.org/abs/2408.04026)
• Keywords: fair
• Abstract: Social agents and robots are increasingly being used in wellbeing settings. However, a key challenge is that these agents and robots typically rely on machine learning (ML) algorithms to detect and analyse an individual's mental wellbeing. The problem of bias and fairness in ML algorithms is becoming an increasingly greater source of concern. In concurrence, existing literature has also indicated that mental health conditions can manifest differently across genders and cultures. We hypothesise that the representation of features (acoustic, textual, and visual) and their inter-modal relations would vary among subjects from different cultures and genders, thus impacting the performance and fairness of various ML models. We present the very first evaluation of multimodal gender fairness in depression manifestation by undertaking a study on two different datasets from the USA and China. We undertake thorough statistical and ML experimentation and repeat the experiments for several different algorithms to ensure that the results are not algorithm-dependent. Our findings indicate that though there are differences between both datasets, it is not conclusive whether this is due to the difference in depression manifestation as hypothesised or other external factors such as differences in data collection methodology. Our findings further motivate a call for a more consistent and culturally aware data collection process in order to address the problem of ML bias in depression detection and to promote the development of fairer agents and robots for wellbeing.

Title: Human Speech Perception in Noise: Can Large Language Models Paraphrase to Improve It?

• Authors: Anupama Chingacham, Miaoran Zhang, Vera Demberg, Dietrich Klakow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04029
• Pdf URL: https://arxiv.org/pdf/2408.04029
• Copy Paste: [[2408.04029]] Human Speech Perception in Noise: Can Large Language Models Paraphrase to Improve It?(https://arxiv.org/abs/2408.04029)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) can generate text by transferring style attributes like formality resulting in formal or informal text. However, instructing LLMs to generate text that when spoken, is more intelligible in an acoustically difficult environment, is an under-explored topic. We conduct the first study to evaluate LLMs on a novel task of generating acoustically intelligible paraphrases for better human speech perception in noise. Our experiments in English demonstrated that with standard prompting, LLMs struggle to control the non-textual attribute, i.e., acoustic intelligibility, while efficiently capturing the desired textual attributes like semantic equivalence. To remedy this issue, we propose a simple prompting approach, prompt-and-select, which generates paraphrases by decoupling the desired textual and non-textual attributes in the text generation pipeline. Our approach resulted in a 40% relative improvement in human speech perception, by paraphrasing utterances that are highly distorted in a listening condition with babble noise at a signal-to-noise ratio (SNR) -5 dB. This study reveals the limitation of LLMs in capturing non-textual attributes, and our proposed method showcases the potential of using LLMs for better human speech perception in noise.

Title: Deep Generative Models for Subgraph Prediction

• Authors: Erfaneh Mahmoudzadeh, Parmis Naddaf, Kiarash Zahirnia, Oliver Schulte
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04053
• Pdf URL: https://arxiv.org/pdf/2408.04053
• Copy Paste: [[2408.04053]] Deep Generative Models for Subgraph Prediction(https://arxiv.org/abs/2408.04053)
• Keywords: generative
• Abstract: Graph Neural Networks (GNNs) are important across different domains, such as social network analysis and recommendation systems, due to their ability to model complex relational data. This paper introduces subgraph queries as a new task for deep graph learning. Unlike traditional graph prediction tasks that focus on individual components like link prediction or node classification, subgraph queries jointly predict the components of a target subgraph based on evidence that is represented by an observed subgraph. For instance, a subgraph query can predict a set of target links and/or node labels. To answer subgraph queries, we utilize a probabilistic deep Graph Generative Model. Specifically, we inductively train a Variational Graph Auto-Encoder (VGAE) model, augmented to represent a joint distribution over links, node features and labels. Bayesian optimization is used to tune a weighting for the relative importance of links, node features and labels in a specific domain. We describe a deterministic and a sampling-based inference method for estimating subgraph probabilities from the VGAE generative graph distribution, without retraining, in zero-shot fashion. For evaluation, we apply the inference methods on a range of subgraph queries on six benchmark datasets. We find that inference from a model achieves superior predictive performance, surpassing independent prediction baselines with improvements in AUC scores ranging from 0.06 to 0.2 points, depending on the dataset.

Title: PushPull-Net: Inhibition-driven ResNet robust to image corruptions

• Authors: Guru Swaroop Bennabhaktula, Enrique Alegre, Nicola Strisciuglio, George Azzopardi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04077
• Pdf URL: https://arxiv.org/pdf/2408.04077
• Copy Paste: [[2408.04077]] PushPull-Net: Inhibition-driven ResNet robust to image corruptions(https://arxiv.org/abs/2408.04077)
• Keywords: robust
• Abstract: We introduce a novel computational unit, termed PushPull-Conv, in the first layer of a ResNet architecture, inspired by the anti-phase inhibition phenomenon observed in the primary visual cortex. This unit redefines the traditional convolutional layer by implementing a pair of complementary filters: a trainable push kernel and its counterpart, the pull kernel. The push kernel (analogous to traditional convolution) learns to respond to specific stimuli, while the pull kernel reacts to the same stimuli but of opposite contrast. This configuration enhances stimulus selectivity and effectively inhibits response in regions lacking preferred stimuli. This effect is attributed to the push and pull kernels, which produce responses of comparable magnitude in such regions, thereby neutralizing each other. The incorporation of the PushPull-Conv into ResNets significantly increases their robustness to image corruption. Our experiments with benchmark corruption datasets show that the PushPull-Conv can be combined with other data augmentation techniques to further improve model robustness. We set a new robustness benchmark on ResNet50 achieving an $mCE$ of 49.95$\%$ on ImageNet-C when combining PRIME augmentation with PushPull inhibition.

Title: Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters

• Authors: Vasudev Shyam, Jonathan Pilault, Emily Shepperd, Quentin Anthony, Beren Millidge
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04093
• Pdf URL: https://arxiv.org/pdf/2408.04093
• Copy Paste: [[2408.04093]] Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters(https://arxiv.org/abs/2408.04093)
• Keywords: transformer
• Abstract: Self-attention is the core mathematical operation of modern transformer architectures and is also a significant computational bottleneck due to its quadratic complexity in the sequence length. In this work, we derive the scalar energy function whose gradient computes the self-attention block, thus elucidating the theoretical underpinnings of self-attention, providing a Bayesian interpretation of the operation and linking it closely with energy-based models such as Hopfield Networks. Moreover, due to this formulation, we discover that we can use efficient and optimized automatic-differentiation techniques to derive a highly efficient Tree Attention algorithm to compute the gradient of the energy and hence self-attention. Our formulation reveals that the reduction across the sequence axis can be efficiently computed in parallel through a tree reduction. Our algorithm, for parallelizing attention computation across multiple GPUs, enables cross-device decoding to be performed asymptotically faster (up to 8x faster) than alternative approaches such as Ring Attention, while also requiring significantly less communication volume and incurring 2x less peak memory. Our code is publicly available here: \url{this https URL}

Title: ArtVLM: Attribute Recognition Through Vision-Based Prefix Language Modeling

• Authors: William Y. Zhu, Keren Ye, Junjie Ke, Jiahui Yu, Leonidas Guibas, Peyman Milanfar, Feng Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04102
• Pdf URL: https://arxiv.org/pdf/2408.04102
• Copy Paste: [[2408.04102]] ArtVLM: Attribute Recognition Through Vision-Based Prefix Language Modeling(https://arxiv.org/abs/2408.04102)
• Keywords: generative
• Abstract: Recognizing and disentangling visual attributes from objects is a foundation to many computer vision applications. While large vision language representations like CLIP had largely resolved the task of zero-shot object recognition, zero-shot visual attribute recognition remains a challenge because CLIP's contrastively-learned vision-language representation cannot effectively capture object-attribute dependencies. In this paper, we target this weakness and propose a sentence generation-based retrieval formulation for attribute recognition that is novel in 1) explicitly modeling a to-be-measured and retrieved object-attribute relation as a conditional probability graph, which converts the recognition problem into a dependency-sensitive language-modeling problem, and 2) applying a large pretrained Vision-Language Model (VLM) on this reformulation and naturally distilling its knowledge of image-object-attribute relations to use towards attribute recognition. Specifically, for each attribute to be recognized on an image, we measure the visual-conditioned probability of generating a short sentence encoding the attribute's relation to objects on the image. Unlike contrastive retrieval, which measures likelihood by globally aligning elements of the sentence to the image, generative retrieval is sensitive to the order and dependency of objects and attributes in the sentence. We demonstrate through experiments that generative retrieval consistently outperforms contrastive retrieval on two visual reasoning datasets, Visual Attribute in the Wild (VAW), and our newly-proposed Visual Genome Attribute Ranking (VGARank).

Title: PaveCap: The First Multimodal Framework for Comprehensive Pavement Condition Assessment with Dense Captioning and PCI Estimation

• Authors: Blessing Agyei Kyem, Eugene Kofi Okrah Denteh, Joshua Kofi Asamoah, Armstrong Aboah
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04110
• Pdf URL: https://arxiv.org/pdf/2408.04110
• Copy Paste: [[2408.04110]] PaveCap: The First Multimodal Framework for Comprehensive Pavement Condition Assessment with Dense Captioning and PCI Estimation(https://arxiv.org/abs/2408.04110)
• Keywords: transformer, segmentation
• Abstract: This research introduces the first multimodal approach for pavement condition assessment, providing both quantitative Pavement Condition Index (PCI) predictions and qualitative descriptions. We introduce PaveCap, a novel framework for automated pavement condition assessment. The framework consists of two main parts: a Single-Shot PCI Estimation Network and a Dense Captioning Network. The PCI Estimation Network uses YOLOv8 for object detection, the Segment Anything Model (SAM) for zero-shot segmentation, and a four-layer convolutional neural network to predict PCI. The Dense Captioning Network uses a YOLOv8 backbone, a Transformer encoder-decoder architecture, and a convolutional feed-forward module to generate detailed descriptions of pavement conditions. To train and evaluate these networks, we developed a pavement dataset with bounding box annotations, textual annotations, and PCI values. The results of our PCI Estimation Network showed a strong positive correlation (0.70) between predicted and actual PCIs, demonstrating its effectiveness in automating condition assessment. Also, the Dense Captioning Network produced accurate pavement condition descriptions, evidenced by high BLEU (0.7445), GLEU (0.5893), and METEOR (0.7252) scores. Additionally, the dense captioning model handled complex scenarios well, even correcting some errors in the ground truth data. The framework developed here can greatly improve infrastructure management and decision18 making in pavement maintenance.

Title: Can Rule-Based Insights Enhance LLMs for Radiology Report Classification? Introducing the RadPrompt Methodology

• Authors: Panagiotis Fytas, Anna Breger, Ian Selby, Simon Baker, Shahab Shahipasand, Anna Korhonen
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04121
• Pdf URL: https://arxiv.org/pdf/2408.04121
• Copy Paste: [[2408.04121]] Can Rule-Based Insights Enhance LLMs for Radiology Report Classification? Introducing the RadPrompt Methodology(https://arxiv.org/abs/2408.04121)
• Keywords: robust, extraction, large language model
• Abstract: Developing imaging models capable of detecting pathologies from chest X-rays can be cost and time-prohibitive for large datasets as it requires supervision to attain state-of-the-art performance. Instead, labels extracted from radiology reports may serve as distant supervision since these are routinely generated as part of clinical practice. Despite their widespread use, current rule-based methods for label extraction rely on extensive rule sets that are limited in their robustness to syntactic variability. To alleviate these limitations, we introduce RadPert, a rule-based system that integrates an uncertainty-aware information schema with a streamlined set of rules, enhancing performance. Additionally, we have developed RadPrompt, a multi-turn prompting strategy that leverages RadPert to bolster the zero-shot predictive capabilities of large language models, achieving a statistically significant improvement in weighted average F1 score over GPT-4 Turbo. Most notably, RadPrompt surpasses both its underlying models, showcasing the synergistic potential of LLMs with rule-based models. We have evaluated our methods on two English Corpora: the MIMIC-CXR gold-standard test set and a gold-standard dataset collected from the Cambridge University Hospitals.

Title: Overcoming Brittleness in Pareto-Optimal Learning-Augmented Algorithms

• Authors: Spyros Angelopoulos, Christoph Dürr, Alex Elenter, Yanni Lefki
• Subjects: cs.LG, cs.DS
• Abstract URL: https://arxiv.org/abs/2408.04122
• Pdf URL: https://arxiv.org/pdf/2408.04122
• Copy Paste: [[2408.04122]] Overcoming Brittleness in Pareto-Optimal Learning-Augmented Algorithms(https://arxiv.org/abs/2408.04122)
• Keywords: robust
• Abstract: The study of online algorithms with machine-learned predictions has gained considerable prominence in recent years. One of the common objectives in the design and analysis of such algorithms is to attain (Pareto) optimal tradeoffs between the consistency of the algorithm, i.e., its performance assuming perfect predictions, and its robustness, i.e., the performance of the algorithm under adversarial predictions. In this work, we demonstrate that this optimization criterion can be extremely brittle, in that the performance of Pareto-optimal algorithms may degrade dramatically even in the presence of imperceptive prediction error. To remedy this drawback, we propose a new framework in which the smoothness in the performance of the algorithm is enforced by means of a user-specified profile. This allows us to regulate the performance of the algorithm as a function of the prediction error, while simultaneously maintaining the analytical notion of consistency/robustness tradeoffs, adapted to the profile setting. We apply this new approach to a well-studied online problem, namely the one-way trading problem. For this problem, we further address another limitation of the state-of-the-art Pareto-optimal algorithms, namely the fact that they are tailored to worst-case, and extremely pessimistic inputs. We propose a new Pareto-optimal algorithm that leverages any deviation from the worst-case input to its benefit, and introduce a new metric that allows us to compare any two Pareto-optimal algorithms via a dominance relation.

Title: Enhancing Healthcare through Large Language Models: A Study on Medical Question Answering

• Authors: Haoran Yu, Chang Yu, Zihan Wang, Dongxian Zou, Hao Qin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04138
• Pdf URL: https://arxiv.org/pdf/2408.04138
• Copy Paste: [[2408.04138]] Enhancing Healthcare through Large Language Models: A Study on Medical Question Answering(https://arxiv.org/abs/2408.04138)
• Keywords: robust, large language model
• Abstract: In recent years, the application of Large Language Models (LLMs) in healthcare has shown significant promise in improving the accessibility and dissemination of medical knowledge. This paper presents a detailed study of various LLMs trained on the MedQuAD medical question-answering dataset, with a focus on identifying the most effective model for providing accurate medical information. Among the models tested, the Sentence-t5 combined with Mistral 7B demonstrated superior performance, achieving a precision score of 0.762. This model's enhanced capabilities are attributed to its advanced pretraining techniques, robust architecture, and effective prompt construction methodologies. By leveraging these strengths, the Sentence-t5 + Mistral 7B model excels in understanding and generating precise medical answers. Our findings highlight the potential of integrating sophisticated LLMs in medical contexts to facilitate efficient and accurate medical knowledge retrieval, thus significantly enhancing patient education and support.

Title: UNLEARN Efficient Removal of Knowledge in Large Language Models

• Authors: Tyler Lizzo, Larry Heck
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04140
• Pdf URL: https://arxiv.org/pdf/2408.04140
• Copy Paste: [[2408.04140]] UNLEARN Efficient Removal of Knowledge in Large Language Models(https://arxiv.org/abs/2408.04140)
• Keywords: large language model
• Abstract: Given the prevalence of large language models (LLMs) and the prohibitive cost of training these models from scratch, dynamically forgetting specific knowledge e.g., private or proprietary, without retraining the model has become an important capability. This paper proposes a novel method to achieve this objective called UNLEARN. The approach builds upon subspace methods to identify and specifically target the removal of knowledge without adversely affecting other knowledge in the LLM. Results demonstrate 96% of targeted knowledge can be forgotten while maintaining performance on other knowledge within 2.5% of the original model, significantly outperforming the discriminatory abilities of the previous state-of-the-art. A dual method called LEARN is also proposed for targeted knowledge addition. Results show LEARN can match the fine-tuning accuracy of Low-Rank Adaptation (LoRA) without adversely affecting similar tasks.

Title: Integrated Dynamic Phenological Feature for Remote Sensing Image Land Cover Change Detection

• Authors: Yi Liu, Chenhao Sun, Hao Ye, Xiangying Liu, Weilong Ju
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04144
• Pdf URL: https://arxiv.org/pdf/2408.04144
• Copy Paste: [[2408.04144]] Integrated Dynamic Phenological Feature for Remote Sensing Image Land Cover Change Detection(https://arxiv.org/abs/2408.04144)
• Keywords: extraction
• Abstract: Remote sensing image change detection (CD) is essential for analyzing land surface changes over time, with a significant challenge being the differentiation of actual changes from complex scenes while filtering out pseudo-changes. A primary contributor to this challenge is the intra-class dynamic changes due to phenological characteristics in natural areas. To overcome this, we introduce the InPhea model, which integrates phenological features into a remote sensing image CD framework. The model features a detector with a differential attention module for improved feature representation of change information, coupled with high-resolution feature extraction and spatial pyramid blocks to enhance performance. Additionally, a constrainer with four constraint modules and a multi-stage contrastive learning approach is employed to aid in the model's understanding of phenological characteristics. Experiments on the HRSCD, SECD, and PSCD-Wuhan datasets reveal that InPhea outperforms other models, confirming its effectiveness in addressing phenological pseudo-changes and its overall model superiority.

Title: Decorrelating Structure via Adapters Makes Ensemble Learning Practical for Semi-supervised Learning

• Authors: Jiaqi Wu, Junbiao Pang, Qingming Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04150
• Pdf URL: https://arxiv.org/pdf/2408.04150
• Copy Paste: [[2408.04150]] Decorrelating Structure via Adapters Makes Ensemble Learning Practical for Semi-supervised Learning(https://arxiv.org/abs/2408.04150)
• Keywords: robust
• Abstract: In computer vision, traditional ensemble learning methods exhibit either a low training efficiency or the limited performance to enhance the reliability of deep neural networks. In this paper, we propose a lightweight, loss-function-free, and architecture-agnostic ensemble learning by the Decorrelating Structure via Adapters (DSA) for various visual tasks. Concretely, the proposed DSA leverages the structure-diverse adapters to decorrelate multiple prediction heads without any tailed regularization or loss. This allows DSA to be easily extensible to architecture-agnostic networks for a range of computer vision tasks. Importantly, the theoretically analysis shows that the proposed DSA has a lower bias and variance than that of the single head based method (which is adopted by most of the state of art approaches). Consequently, the DSA makes deep networks reliable and robust for the various real-world challenges, \textit{e.g.}, data corruption, and label noises. Extensive experiments combining the proposed method with FreeMatch achieved the accuracy improvements of 5.35% on CIFAR-10 dataset with 40 labeled data and 0.71% on CIFAR-100 dataset with 400 labeled data. Besides, combining the proposed method with DualPose achieved the improvements in the Percentage of Correct Keypoints (PCK) by 2.08% on the Sniffing dataset with 100 data (30 labeled data), 5.2% on the FLIC dataset with 100 data (including 50 labeled data), and 2.35% on the LSP dataset with 200 data (100 labeled data).

Title: The Data Addition Dilemma

• Authors: Judy Hanwen Shen, Inioluwa Deborah Raji, Irene Y. Chen
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.04154
• Pdf URL: https://arxiv.org/pdf/2408.04154
• Copy Paste: [[2408.04154]] The Data Addition Dilemma(https://arxiv.org/abs/2408.04154)
• Keywords: fair
• Abstract: In many machine learning for healthcare tasks, standard datasets are constructed by amassing data across many, often fundamentally dissimilar, sources. But when does adding more data help, and when does it hinder progress on desired model outcomes in real-world settings? We identify this situation as the \textit{Data Addition Dilemma}, demonstrating that adding training data in this multi-source scaling context can at times result in reduced overall accuracy, uncertain fairness outcomes, and reduced worst-subgroup performance. We find that this possibly arises from an empirically observed trade-off between model performance improvements due to data scaling and model deterioration from distribution shift. We thus establish baseline strategies for navigating this dilemma, introducing distribution shift heuristics to guide decision-making on which data sources to add in data scaling, in order to yield the expected model performance improvements. We conclude with a discussion of the required considerations for data collection and suggestions for studying data composition and scale in the age of increasingly larger models.

Title: M2EF-NNs: Multimodal Multi-instance Evidence Fusion Neural Networks for Cancer Survival Prediction

• Authors: Hui Luo, Jiashuang Huang, Hengrong Ju, Tianyi Zhou, Weiping Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04170
• Pdf URL: https://arxiv.org/pdf/2408.04170
• Copy Paste: [[2408.04170]] M2EF-NNs: Multimodal Multi-instance Evidence Fusion Neural Networks for Cancer Survival Prediction(https://arxiv.org/abs/2408.04170)
• Keywords: transformer
• Abstract: Accurate cancer survival prediction is crucial for assisting clinical doctors in formulating treatment plans. Multimodal data, including histopathological images and genomic data, offer complementary and comprehensive information that can greatly enhance the accuracy of this task. However, the current methods, despite yielding promising results, suffer from two notable limitations: they do not effectively utilize global context and disregard modal uncertainty. In this study, we put forward a neural network model called M2EF-NNs, which leverages multimodal and multi-instance evidence fusion techniques for accurate cancer survival prediction. Specifically, to capture global information in the images, we use a pre-trained Vision Transformer (ViT) model to obtain patch feature embeddings of histopathological images. Then, we introduce a multimodal attention module that uses genomic embeddings as queries and learns the co-attention mapping between genomic and histopathological images to achieve an early interaction fusion of multimodal information and better capture their correlations. Subsequently, we are the first to apply the Dempster-Shafer evidence theory (DST) to cancer survival prediction. We parameterize the distribution of class probabilities using the processed multimodal features and introduce subjective logic to estimate the uncertainty associated with different modalities. By combining with the Dempster-Shafer theory, we can dynamically adjust the weights of class probabilities after multimodal fusion to achieve trusted survival prediction. Finally, Experimental validation on the TCGA datasets confirms the significant improvements achieved by our proposed method in cancer survival prediction and enhances the reliability of the model.

Title: MultiColor: Image Colorization by Learning from Multiple Color Spaces

• Authors: Xiangcheng Du, Zhao Zhou, Yanlong Wang, Zhuoyao Wang, Yingbin Zheng, Cheng Jin
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.04172
• Pdf URL: https://arxiv.org/pdf/2408.04172
• Copy Paste: [[2408.04172]] MultiColor: Image Colorization by Learning from Multiple Color Spaces(https://arxiv.org/abs/2408.04172)
• Keywords: transformer
• Abstract: Deep networks have shown impressive performance in the image restoration tasks, such as image colorization. However, we find that previous approaches rely on the digital representation from single color model with a specific mapping function, a.k.a., color space, during the colorization pipeline. In this paper, we first investigate the modeling of different color spaces, and find each of them exhibiting distinctive characteristics with unique distribution of colors. The complementarity among multiple color spaces leads to benefits for the image colorization task. We present MultiColor, a new learning-based approach to automatically colorize grayscale images that combines clues from multiple color spaces. Specifically, we employ a set of dedicated colorization modules for individual color space. Within each module, a transformer decoder is first employed to refine color query embeddings and then a color mapper produces color channel prediction using the embeddings and semantic features. With these predicted color channels representing various color spaces, a complementary network is designed to exploit the complementarity and generate pleasing and reasonable colorized images. We conduct extensive experiments on real-world datasets, and the results demonstrate superior performance over the state-of-the-arts.

Title: wav2graph: A Framework for Supervised Learning Knowledge Graph from Speech

• Authors: Khai Le-Duc, Quy-Anh Dang, Tan-Hanh Pham, Truong-Son Hy
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.04174
• Pdf URL: https://arxiv.org/pdf/2408.04174
• Copy Paste: [[2408.04174]] wav2graph: A Framework for Supervised Learning Knowledge Graph from Speech(https://arxiv.org/abs/2408.04174)
• Keywords: large language model
• Abstract: Knowledge graphs (KGs) enhance the performance of large language models (LLMs) and search engines by providing structured, interconnected data that improves reasoning and context-awareness. However, KGs only focus on text data, thereby neglecting other modalities such as speech. In this work, we introduce wav2graph, the first framework for supervised learning knowledge graph from speech data. Our pipeline are straightforward: (1) constructing a KG based on transcribed spoken utterances and a named entity database, (2) converting KG into embedding vectors, and (3) training graph neural networks (GNNs) for node classification and link prediction tasks. Through extensive experiments conducted in inductive and transductive learning contexts using state-of-the-art GNN models, we provide baseline results and error analysis for node classification and link prediction tasks on human transcripts and automatic speech recognition (ASR) transcripts, including evaluations using both encoder-based and decoder-based node embeddings, as well as monolingual and multilingual acoustic pre-trained models. All related code, data, and models are published online.

Title: EdgeShield: A Universal and Efficient Edge Computing Framework for Robust AI

• Authors: Duo Zhong, Bojing Li, Xiang Chen, Chenchen Liu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04181
• Pdf URL: https://arxiv.org/pdf/2408.04181
• Copy Paste: [[2408.04181]] EdgeShield: A Universal and Efficient Edge Computing Framework for Robust AI(https://arxiv.org/abs/2408.04181)
• Keywords: security, defense, attack, robust
• Abstract: The increasing prevalence of adversarial attacks on Artificial Intelligence (AI) systems has created a need for innovative security measures. However, the current methods of defending against these attacks often come with a high computing cost and require back-end processing, making real-time defense challenging. Fortunately, there have been remarkable advancements in edge-computing, which make it easier to deploy neural networks on edge devices. Building upon these advancements, we propose an edge framework design to enable universal and efficient detection of adversarial attacks. This framework incorporates an attention-based adversarial detection methodology and a lightweight detection network formation, making it suitable for a wide range of neural networks and can be deployed on edge devices. To assess the effectiveness of our proposed framework, we conducted evaluations on five neural networks. The results indicate an impressive 97.43% F-score can be achieved, demonstrating the framework's proficiency in detecting adversarial attacks. Moreover, our proposed framework also exhibits significantly reduced computing complexity and cost in comparison to previous detection methods. This aspect is particularly beneficial as it ensures that the defense mechanism can be efficiently implemented in real-time on-edge devices.

Title: Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation

• Authors: Junde Wu, Jiayuan Zhu, Yunli Qi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04187
• Pdf URL: https://arxiv.org/pdf/2408.04187
• Copy Paste: [[2408.04187]] Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation(https://arxiv.org/abs/2408.04187)
• Keywords: large language model
• Abstract: We introduce a novel graph-based Retrieval-Augmented Generation (RAG) framework specifically designed for the medical domain, called \textbf{MedGraphRAG}, aimed at enhancing Large Language Model (LLM) capabilities and generating evidence-based results, thereby improving safety and reliability when handling private medical data. Our comprehensive pipeline begins with a hybrid static-semantic approach to document chunking, significantly improving context capture over traditional methods. Extracted entities are used to create a three-tier hierarchical graph structure, linking entities to foundational medical knowledge sourced from medical papers and dictionaries. These entities are then interconnected to form meta-graphs, which are merged based on semantic similarities to develop a comprehensive global graph. This structure supports precise information retrieval and response generation. The retrieval process employs a U-retrieve method to balance global awareness and indexing efficiency of the LLM. Our approach is validated through a comprehensive ablation study comparing various methods for document chunking, graph construction, and information retrieval. The results not only demonstrate that our hierarchical graph construction method consistently outperforms state-of-the-art models on multiple medical Q\&A benchmarks, but also confirms that the responses generated include source documentation, significantly enhancing the reliability of medical LLMs in practical applications. Code will be at: this https URL

Title: Listwise Reward Estimation for Offline Preference-based Reinforcement Learning

• Authors: Heewoong Choi, Sangwon Jung, Hongjoon Ahn, Taesup Moon
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04190
• Pdf URL: https://arxiv.org/pdf/2408.04190
• Copy Paste: [[2408.04190]] Listwise Reward Estimation for Offline Preference-based Reinforcement Learning(https://arxiv.org/abs/2408.04190)
• Keywords: robust
• Abstract: In Reinforcement Learning (RL), designing precise reward functions remains to be a challenge, particularly when aligning with human intent. Preference-based RL (PbRL) was introduced to address this problem by learning reward models from human feedback. However, existing PbRL methods have limitations as they often overlook the second-order preference that indicates the relative strength of preference. In this paper, we propose Listwise Reward Estimation (LiRE), a novel approach for offline PbRL that leverages second-order preference information by constructing a Ranked List of Trajectories (RLT), which can be efficiently built by using the same ternary feedback type as traditional methods. To validate the effectiveness of LiRE, we propose a new offline PbRL dataset that objectively reflects the effect of the estimated rewards. Our extensive experiments on the dataset demonstrate the superiority of LiRE, i.e., outperforming state-of-the-art baselines even with modest feedback budgets and enjoying robustness with respect to the number of feedbacks and feedback noise. Our code is available at this https URL

Title: Uncertainty-Aware Crime Prediction With Spatial Temporal Multivariate Graph Neural Networks

• Authors: Zepu Wang, Xiaobo Ma, Huajie Yang, Weimin Lvu, Peng Sun, Sharath Chandra Guntuku
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04193
• Pdf URL: https://arxiv.org/pdf/2408.04193
• Copy Paste: [[2408.04193]] Uncertainty-Aware Crime Prediction With Spatial Temporal Multivariate Graph Neural Networks(https://arxiv.org/abs/2408.04193)
• Keywords: diffusion
• Abstract: Crime forecasting is a critical component of urban analysis and essential for stabilizing society today. Unlike other time series forecasting problems, crime incidents are sparse, particularly in small regions and within specific time periods. Traditional spatial-temporal deep learning models often struggle with this sparsity, as they typically cannot effectively handle the non-Gaussian nature of crime data, which is characterized by numerous zeros and over-dispersed patterns. To address these challenges, we introduce a novel approach termed Spatial Temporal Multivariate Zero-Inflated Negative Binomial Graph Neural Networks (STMGNN-ZINB). This framework leverages diffusion and convolution networks to analyze spatial, temporal, and multivariate correlations, enabling the parameterization of probabilistic distributions of crime incidents. By incorporating a Zero-Inflated Negative Binomial model, STMGNN-ZINB effectively manages the sparse nature of crime data, enhancing prediction accuracy and the precision of confidence intervals. Our evaluation on real-world datasets confirms that STMGNN-ZINB outperforms existing models, providing a more reliable tool for predicting and understanding crime dynamics.

Title: MMREC: LLM Based Multi-Modal Recommender System

• Authors: Jiahao Tian, Jinman Zhao, Zhenkai Wang, Zhicheng Ding
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.04211
• Pdf URL: https://arxiv.org/pdf/2408.04211
• Copy Paste: [[2408.04211]] MMREC: LLM Based Multi-Modal Recommender System(https://arxiv.org/abs/2408.04211)
• Keywords: large language model
• Abstract: The importance of recommender systems is growing rapidly due to the exponential increase in the volume of content generated daily. This surge in content presents unique challenges for designing effective recommender systems. Key among these challenges is the need to effectively leverage the vast amounts of natural language data and images that represent user preferences. This paper presents a novel approach to enhancing recommender systems by leveraging Large Language Models (LLMs) and deep learning techniques. The proposed framework aims to improve the accuracy and relevance of recommendations by incorporating multi-modal information processing and by the use of unified latent space representation. The study explores the potential of LLMs to better understand and utilize natural language data in recommendation contexts, addressing the limitations of previous methods. The framework efficiently extracts and integrates text and image information through LLMs, unifying diverse modalities in a latent space to simplify the learning process for the ranking model. Experimental results demonstrate the enhanced discriminative power of the model when utilizing multi-modal information. This research contributes to the evolving field of recommender systems by showcasing the potential of LLMs and multi-modal data integration to create more personalized and contextually relevant recommendations.

Title: Attention Mechanism and Context Modeling System for Text Mining Machine Translation

• Authors: Shi Bo, Yuwei Zhang, Junming Huang, Sitong Liu, Zexi Chen, Zizheng Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04216
• Pdf URL: https://arxiv.org/pdf/2408.04216
• Copy Paste: [[2408.04216]] Attention Mechanism and Context Modeling System for Text Mining Machine Translation(https://arxiv.org/abs/2408.04216)
• Keywords: transformer
• Abstract: This paper advances a novel architectural schema anchored upon the Transformer paradigm and innovatively amalgamates the K-means categorization algorithm to augment the contextual apprehension capabilities of the schema. The transformer model performs well in machine translation tasks due to its parallel computing power and multi-head attention mechanism. However, it may encounter contextual ambiguity or ignore local features when dealing with highly complex language structures. To circumvent this constraint, this exposition incorporates the K-Means algorithm, which is used to stratify the lexis and idioms of the input textual matter, thereby facilitating superior identification and preservation of the local structure and contextual intelligence of the language. The advantage of this combination is that K-Means can automatically discover the topic or concept regions in the text, which may be directly related to translation quality. Consequently, the schema contrived herein enlists K-Means as a preparatory phase antecedent to the Transformer and recalibrates the multi-head attention weights to assist in the discrimination of lexis and idioms bearing analogous semantics or functionalities. This ensures the schema accords heightened regard to the contextual intelligence embodied by these clusters during the training phase, rather than merely focusing on locational intelligence.

Title: Simplifying Translations for Children: Iterative Simplification Considering Age of Acquisition with LLMs

• Authors: Masashi Oshika, Makoto Morishita, Tsutomu Hirao, Ryohei Sasano, Koichi Takeda
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04217
• Pdf URL: https://arxiv.org/pdf/2408.04217
• Copy Paste: [[2408.04217]] Simplifying Translations for Children: Iterative Simplification Considering Age of Acquisition with LLMs(https://arxiv.org/abs/2408.04217)
• Keywords: large language model
• Abstract: In recent years, neural machine translation (NMT) has been widely used in everyday life. However, the current NMT lacks a mechanism to adjust the difficulty level of translations to match the user's language level. Additionally, due to the bias in the training data for NMT, translations of simple source sentences are often produced with complex words. In particular, this could pose a problem for children, who may not be able to understand the meaning of the translations correctly. In this study, we propose a method that replaces words with high Age of Acquisitions (AoA) in translations with simpler words to match the translations to the user's level. We achieve this by using large language models (LLMs), providing a triple of a source sentence, a translation, and a target word to be replaced. We create a benchmark dataset using back-translation on Simple English Wikipedia. The experimental results obtained from the dataset show that our method effectively replaces high-AoA words with lower-AoA words and, moreover, can iteratively replace most of the high-AoA words while still maintaining high BLEU and COMET scores.

Title: Diffusion Guided Language Modeling

• Authors: Justin Lovelace, Varsha Kishore, Yiwei Chen, Kilian Q. Weinberger
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04220
• Pdf URL: https://arxiv.org/pdf/2408.04220
• Copy Paste: [[2408.04220]] Diffusion Guided Language Modeling(https://arxiv.org/abs/2408.04220)
• Keywords: diffusion
• Abstract: Current language models demonstrate remarkable proficiency in text generation. However, for many applications it is desirable to control attributes, such as sentiment, or toxicity, of the generated language -- ideally tailored towards each specific use case and target audience. For auto-regressive language models, existing guidance methods are prone to decoding errors that cascade during generation and degrade performance. In contrast, text diffusion models can easily be guided with, for example, a simple linear sentiment classifier -- however they do suffer from significantly higher perplexity than auto-regressive alternatives. In this paper we use a guided diffusion model to produce a latent proposal that steers an auto-regressive language model to generate text with desired properties. Our model inherits the unmatched fluency of the auto-regressive approach and the plug-and-play flexibility of diffusion. We show that it outperforms previous plug-and-play guidance methods across a wide range of benchmark data sets. Further, controlling a new attribute in our framework is reduced to training a single logistic regression classifier.

Title: Connective Viewpoints of Signal-to-Noise Diffusion Models

• Authors: Khanh Doan, Long Tung Vuong, Tuan Nguyen, Anh Tuan Bui, Quyen Tran, Thanh-Toan Do, Dinh Phung, Trung Le
• Subjects: cs.CV, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2408.04221
• Pdf URL: https://arxiv.org/pdf/2408.04221
• Copy Paste: [[2408.04221]] Connective Viewpoints of Signal-to-Noise Diffusion Models(https://arxiv.org/abs/2408.04221)
• Keywords: diffusion, generative
• Abstract: Diffusion models (DM) have become fundamental components of generative models, excelling across various domains such as image creation, audio generation, and complex data interpolation. Signal-to-Noise diffusion models constitute a diverse family covering most state-of-the-art diffusion models. While there have been several attempts to study Signal-to-Noise (S2N) diffusion models from various perspectives, there remains a need for a comprehensive study connecting different viewpoints and exploring new perspectives. In this study, we offer a comprehensive perspective on noise schedulers, examining their role through the lens of the signal-to-noise ratio (SNR) and its connections to information theory. Building upon this framework, we have developed a generalized backward equation to enhance the performance of the inference process.

Title: VideoQA in the Era of LLMs: An Empirical Study

• Authors: Junbin Xiao, Nanxin Huang, Hangyu Qin, Dongyang Li, Yicong Li, Fengbin Zhu, Zhulin Tao, Jianxing Yu, Liang Lin, Tat-Seng Chua, Angela Yao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04223
• Pdf URL: https://arxiv.org/pdf/2408.04223
• Copy Paste: [[2408.04223]] VideoQA in the Era of LLMs: An Empirical Study(https://arxiv.org/abs/2408.04223)
• Keywords: robust, interpretability, large language model
• Abstract: Video Large Language Models (Video-LLMs) are flourishing and has advanced many video-language tasks. As a golden testbed, Video Question Answering (VideoQA) plays pivotal role in Video-LLM developing. This work conducts a timely and comprehensive study of Video-LLMs' behavior in VideoQA, aiming to elucidate their success and failure modes, and provide insights towards more human-like video understanding and question answering. Our analyses demonstrate that Video-LLMs excel in VideoQA; they can correlate contextual cues and generate plausible responses to questions about varied video contents. However, models falter in handling video temporality, both in reasoning about temporal content ordering and grounding QA-relevant temporal moments. Moreover, the models behave unintuitively - they are unresponsive to adversarial video perturbations while being sensitive to simple variations of candidate answers and questions. Also, they do not necessarily generalize better. The findings demonstrate Video-LLMs' QA capability in standard condition yet highlight their severe deficiency in robustness and interpretability, suggesting the urgent need on rationales in Video-LLM developing.

Title: Cross-View Meets Diffusion: Aerial Image Synthesis with Geometry and Text Guidance

• Authors: Ahmad Arrabi, Xiaohan Zhang, Waqas Sultan, Chen Chen, Safwan Wshah
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04224
• Pdf URL: https://arxiv.org/pdf/2408.04224
• Copy Paste: [[2408.04224]] Cross-View Meets Diffusion: Aerial Image Synthesis with Geometry and Text Guidance(https://arxiv.org/abs/2408.04224)
• Keywords: diffusion, segmentation
• Abstract: Aerial imagery analysis is critical for many research fields. However, obtaining frequent high-quality aerial images is not always accessible due to its high effort and cost requirements. One solution is to use the Ground-to-Aerial (G2A) technique to synthesize aerial images from easily collectible ground images. However, G2A is rarely studied, because of its challenges, including but not limited to, the drastic view changes, occlusion, and range of visibility. In this paper, we present a novel Geometric Preserving Ground-to-Aerial (G2A) image synthesis (GPG2A) model that can generate realistic aerial images from ground images. GPG2A consists of two stages. The first stage predicts the Bird's Eye View (BEV) segmentation (referred to as the BEV layout map) from the ground image. The second stage synthesizes the aerial image from the predicted BEV layout map and text descriptions of the ground image. To train our model, we present a new multi-modal cross-view dataset, namely VIGORv2 which is built upon VIGOR with newly collected aerial images, maps, and text descriptions. Our extensive experiments illustrate that GPG2A synthesizes better geometry-preserved aerial images than existing models. We also present two applications, data augmentation for cross-view geo-localization and sketch-based region search, to further verify the effectiveness of our GPG2A. The code and data will be publicly available.

Title: LLDif: Diffusion Models for Low-light Emotion Recognition

• Authors: Zhifeng Wang, Kaihao Zhang, Ramesh Sankaranarayana
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04235
• Pdf URL: https://arxiv.org/pdf/2408.04235
• Copy Paste: [[2408.04235]] LLDif: Diffusion Models for Low-light Emotion Recognition(https://arxiv.org/abs/2408.04235)
• Keywords: diffusion, transformer
• Abstract: This paper introduces LLDif, a novel diffusion-based facial expression recognition (FER) framework tailored for extremely low-light (LL) environments. Images captured under such conditions often suffer from low brightness and significantly reduced contrast, presenting challenges to conventional methods. These challenges include poor image quality that can significantly reduce the accuracy of emotion recognition. LLDif addresses these issues with a novel two-stage training process that combines a Label-aware CLIP (LA-CLIP), an embedding prior network (PNET), and a transformer-based network adept at handling the noise of low-light images. The first stage involves LA-CLIP generating a joint embedding prior distribution (EPD) to guide the LLformer in label recovery. In the second stage, the diffusion model (DM) refines the EPD inference, ultilising the compactness of EPD for precise predictions. Experimental evaluations on various LL-FER datasets have shown that LLDif achieves competitive performance, underscoring its potential to enhance FER applications in challenging lighting conditions.

Title: Cluster-Wide Task Slowdown Detection in Cloud System

• Authors: Feiyi Chen, Yingying Zhang, Lunting Fan, Yuxuan Liang, Guansong Pang, Qingsong Wen, Shuiguang Deng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04236
• Pdf URL: https://arxiv.org/pdf/2408.04236
• Copy Paste: [[2408.04236]] Cluster-Wide Task Slowdown Detection in Cloud System(https://arxiv.org/abs/2408.04236)
• Keywords: transformer
• Abstract: Slow task detection is a critical problem in cloud operation and maintenance since it is highly related to user experience and can bring substantial liquidated damages. Most anomaly detection methods detect it from a single-task aspect. However, considering millions of concurrent tasks in large-scale cloud computing clusters, it becomes impractical and inefficient. Moreover, single-task slowdowns are very common and do not necessarily indicate a malfunction of a cluster due to its violent fluctuation nature in a virtual environment. Thus, we shift our attention to cluster-wide task slowdowns by utilizing the duration time distribution of tasks across a cluster, so that the computation complexity is not relevant to the number of tasks. The task duration time distribution often exhibits compound periodicity and local exceptional fluctuations over time. Though transformer-based methods are one of the most powerful methods to capture these time series normal variation patterns, we empirically find and theoretically explain the flaw of the standard attention mechanism in reconstructing subperiods with low amplitude when dealing with compound periodicity. To tackle these challenges, we propose SORN (i.e., Skimming Off subperiods in descending amplitude order and Reconstructing Non-slowing fluctuation), which consists of a Skimming Attention mechanism to reconstruct the compound periodicity and a Neural Optimal Transport module to distinguish cluster-wide slowdowns from other exceptional fluctuations. Furthermore, since anomalies in the training set are inevitable in a practical scenario, we propose a picky loss function, which adaptively assigns higher weights to reliable time slots in the training set. Extensive experiments demonstrate that SORN outperforms state-of-the-art methods on multiple real-world industrial datasets.

Title: Learning to Rewrite: Generalized LLM-Generated Text Detection

• Authors: Wei Hao, Ran Li, Weiliang Zhao, Junfeng Yang, Chengzhi Mao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04237
• Pdf URL: https://arxiv.org/pdf/2408.04237
• Copy Paste: [[2408.04237]] Learning to Rewrite: Generalized LLM-Generated Text Detection(https://arxiv.org/abs/2408.04237)
• Keywords: large language model
• Abstract: Large language models (LLMs) can be abused at scale to create non-factual content and spread disinformation. Detecting LLM-generated content is essential to mitigate these risks, but current classifiers often fail to generalize in open-world contexts. Prior work shows that LLMs tend to rewrite LLM-generated content less frequently, which can be used for detection and naturally generalizes to unforeseen data. However, we find that the rewriting edit distance between human and LLM content can be indistinguishable across domains, leading to detection failures. We propose training an LLM to rewrite input text, producing minimal edits for LLM-generated content and more edits for human-written text, deriving a distinguishable and generalizable edit distance difference across different domains. Experiments on text from 21 independent domains and three popular LLMs (e.g., GPT-4o, Gemini, and Llama-3) show that our classifier outperforms the state-of-the-art zero-shot classifier by up to 20.6% on AUROC score and the rewriting classifier by 9.2% on F1 score. Our work suggests that LLM can effectively detect machine-generated text if they are trained properly.

Title: Scalable Transformer for High Dimensional Multivariate Time Series Forecasting

• Authors: Xin Zhou, Weiqing Wang, Wray Buntine, Shilin Qu, Abishek Sriramulu, Weicong Tan, Christoph Bergmeir
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.04245
• Pdf URL: https://arxiv.org/pdf/2408.04245
• Copy Paste: [[2408.04245]] Scalable Transformer for High Dimensional Multivariate Time Series Forecasting(https://arxiv.org/abs/2408.04245)
• Keywords: transformer
• Abstract: Deep models for Multivariate Time Series (MTS) forecasting have recently demonstrated significant success. Channel-dependent models capture complex dependencies that channel-independent models cannot capture. However, the number of channels in real-world applications outpaces the capabilities of existing channel-dependent models, and contrary to common expectations, some models underperform the channel-independent models in handling high-dimensional data, which raises questions about the performance of channel-dependent models. To address this, our study first investigates the reasons behind the suboptimal performance of these channel-dependent models on high-dimensional MTS data. Our analysis reveals that two primary issues lie in the introduced noise from unrelated series that increases the difficulty of capturing the crucial inter-channel dependencies, and challenges in training strategies due to high-dimensional data. To address these issues, we propose STHD, the Scalable Transformer for High-Dimensional Multivariate Time Series Forecasting. STHD has three components: a) Relation Matrix Sparsity that limits the noise introduced and alleviates the memory issue; b) ReIndex applied as a training strategy to enable a more flexible batch size setting and increase the diversity of training data; and c) Transformer that handles 2-D inputs and captures channel dependencies. These components jointly enable STHD to manage the high-dimensional MTS while maintaining computational feasibility. Furthermore, experimental results show STHD's considerable improvement on three high-dimensional datasets: Crime-Chicago, Wiki-People, and Traffic. The source code and dataset are publicly available this https URL.

Title: InstantStyleGaussian: Efficient Art Style Transfer with 3D Gaussian Splatting

• Authors: Xin-Yi Yu, Jun-Xin Yu, Li-Bo Zhou, Yan Wei, Lin-Lin Ou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04249
• Pdf URL: https://arxiv.org/pdf/2408.04249
• Copy Paste: [[2408.04249]] InstantStyleGaussian: Efficient Art Style Transfer with 3D Gaussian Splatting(https://arxiv.org/abs/2408.04249)
• Keywords: diffusion
• Abstract: We present InstantStyleGaussian, an innovative 3D style transfer method based on the 3D Gaussian Splatting (3DGS) scene representation. By inputting a target style image, it quickly generates new 3D GS scenes. Our approach operates on pre-reconstructed GS scenes, combining diffusion models with an improved iterative dataset update strategy. It utilizes diffusion models to generate target style images, adds these new images to the training dataset, and uses this dataset to iteratively update and optimize the GS scenes. Extensive experimental results demonstrate that our method ensures high-quality stylized scenes while offering significant advantages in style transfer speed and consistency.

Title: Generating Fine-Grained Causality in Climate Time Series Data for Forecasting and Anomaly Detection

• Authors: Dongqi Fu, Yada Zhu, Hanghang Tong, Kommy Weldemariam, Onkar Bhardwaj, Jingrui He
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04254
• Pdf URL: https://arxiv.org/pdf/2408.04254
• Copy Paste: [[2408.04254]] Generating Fine-Grained Causality in Climate Time Series Data for Forecasting and Anomaly Detection(https://arxiv.org/abs/2408.04254)
• Keywords: generative
• Abstract: Understanding the causal interaction of time series variables can contribute to time series data analysis for many real-world applications, such as climate forecasting and extreme weather alerts. However, causal relationships are difficult to be fully observed in real-world complex settings, such as spatial-temporal data from deployed sensor networks. Therefore, to capture fine-grained causal relations among spatial-temporal variables for further a more accurate and reliable time series analysis, we first design a conceptual fine-grained causal model named TBN Granger Causality, which adds time-respecting Bayesian Networks to the previous time-lagged Neural Granger Causality to offset the instantaneous effects. Second, we propose an end-to-end deep generative model called TacSas, which discovers TBN Granger Causality in a generative manner to help forecast time series data and detect possible anomalies during the forecast. For evaluations, besides the causality discovery benchmark Lorenz-96, we also test TacSas on climate benchmark ERA5 for climate forecasting and the extreme weather benchmark of NOAA for extreme weather alerts.

Title: UHNet: An Ultra-Lightweight and High-Speed Edge Detection Network

• Authors: Fuzhang Li, Chuan Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04258
• Pdf URL: https://arxiv.org/pdf/2408.04258
• Copy Paste: [[2408.04258]] UHNet: An Ultra-Lightweight and High-Speed Edge Detection Network(https://arxiv.org/abs/2408.04258)
• Keywords: extraction, transformer
• Abstract: Edge detection is crucial in medical image processing, enabling precise extraction of structural information to support lesion identification and image analysis. Traditional edge detection models typically rely on complex Convolutional Neural Networks and Vision Transformer architectures. Due to their numerous parameters and high computational demands, these models are limited in their application on resource-constrained devices. This paper presents an ultra-lightweight edge detection model (UHNet), characterized by its minimal parameter count, rapid computation speed, negligible of pre-training costs, and commendable performance. UHNet boasts impressive performance metrics with 42.3k parameters, 166 FPS, and 0.79G FLOPs. By employing an innovative feature extraction module and optimized residual connection method, UHNet significantly reduces model complexity and computational requirements. Additionally, a lightweight feature fusion strategy is explored, enhancing detection accuracy. Experimental results on the BSDS500, NYUD, and BIPED datasets validate that UHNet achieves remarkable edge detection performance while maintaining high efficiency. This work not only provides new insights into the design of lightweight edge detection models but also demonstrates the potential and application prospects of the UHNet model in engineering applications such as medical image processing. The codes are available at this https URL

Title: EfficientRAG: Efficient Retriever for Multi-Hop Question Answering

• Authors: Ziyuan Zhuang, Zhiyang Zhang, Sitao Cheng, Fangkai Yang, Jia Liu, Shujian Huang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04259
• Pdf URL: https://arxiv.org/pdf/2408.04259
• Copy Paste: [[2408.04259]] EfficientRAG: Efficient Retriever for Multi-Hop Question Answering(https://arxiv.org/abs/2408.04259)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) methods encounter difficulties when addressing complex questions like multi-hop queries. While iterative retrieval methods improve performance by gathering additional information, current approaches often rely on multiple calls of large language models (LLMs). In this paper, we introduce EfficientRAG, an efficient retriever for multi-hop question answering. EfficientRAG iteratively generates new queries without the need for LLM calls at each iteration and filters out irrelevant information. Experimental results demonstrate that EfficientRAG surpasses existing RAG methods on three open-domain multi-hop question-answering datasets.

Title: Unveiling Hidden Visual Information: A Reconstruction Attack Against Adversarial Visual Information Hiding

• Authors: Jonggyu Jang, Hyeonsu Lyu, Seongjin Hwang, Hyun Jong Yang
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.04261
• Pdf URL: https://arxiv.org/pdf/2408.04261
• Copy Paste: [[2408.04261]] Unveiling Hidden Visual Information: A Reconstruction Attack Against Adversarial Visual Information Hiding(https://arxiv.org/abs/2408.04261)
• Keywords: security, protect, attack, generative
• Abstract: This paper investigates the security vulnerabilities of adversarial-example-based image encryption by executing data reconstruction (DR) attacks on encrypted images. A representative image encryption method is the adversarial visual information hiding (AVIH), which uses type-I adversarial example training to protect gallery datasets used in image recognition tasks. In the AVIH method, the type-I adversarial example approach creates images that appear completely different but are still recognized by machines as the original ones. Additionally, the AVIH method can restore encrypted images to their original forms using a predefined private key generative model. For the best security, assigning a unique key to each image is recommended; however, storage limitations may necessitate some images sharing the same key model. This raises a crucial security question for AVIH: How many images can safely share the same key model without being compromised by a DR attack? To address this question, we introduce a dual-strategy DR attack against the AVIH encryption method by incorporating (1) generative-adversarial loss and (2) augmented identity loss, which prevent DR from overfitting -- an issue akin to that in machine learning. Our numerical results validate this approach through image recognition and re-identification benchmarks, demonstrating that our strategy can significantly enhance the quality of reconstructed images, thereby requiring fewer key-sharing encrypted images. Our source code to reproduce our results will be available soon.

Title: CoBooM: Codebook Guided Bootstrapping for Medical Image Representation Learning

• Authors: Azad Singh, Deepak Mishra
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04262
• Pdf URL: https://arxiv.org/pdf/2408.04262
• Copy Paste: [[2408.04262]] CoBooM: Codebook Guided Bootstrapping for Medical Image Representation Learning(https://arxiv.org/abs/2408.04262)
• Keywords: robust, extraction, segmentation
• Abstract: Self-supervised learning (SSL) has emerged as a promising paradigm for medical image analysis by harnessing unannotated data. Despite their potential, the existing SSL approaches overlook the high anatomical similarity inherent in medical images. This makes it challenging for SSL methods to capture diverse semantic content in medical images consistently. This work introduces a novel and generalized solution that implicitly exploits anatomical similarities by integrating codebooks in SSL. The codebook serves as a concise and informative dictionary of visual patterns, which not only aids in capturing nuanced anatomical details but also facilitates the creation of robust and generalized feature representations. In this context, we propose CoBooM, a novel framework for self-supervised medical image learning by integrating continuous and discrete representations. The continuous component ensures the preservation of fine-grained details, while the discrete aspect facilitates coarse-grained feature extraction through the structured embedding space. To understand the effectiveness of CoBooM, we conduct a comprehensive evaluation of various medical datasets encompassing chest X-rays and fundus images. The experimental results reveal a significant performance gain in classification and segmentation tasks.

Title: Evaluating Modern Approaches in 3D Scene Reconstruction: NeRF vs Gaussian-Based Methods

• Authors: Yiming Zhou, Zixuan Zeng, Andi Chen, Xiaofan Zhou, Haowei Ni, Shiyao Zhang, Panfeng Li, Liangxi Liu, Mengyao Zheng, Xupeng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04268
• Pdf URL: https://arxiv.org/pdf/2408.04268
• Copy Paste: [[2408.04268]] Evaluating Modern Approaches in 3D Scene Reconstruction: NeRF vs Gaussian-Based Methods(https://arxiv.org/abs/2408.04268)
• Keywords: robust
• Abstract: Exploring the capabilities of Neural Radiance Fields (NeRF) and Gaussian-based methods in the context of 3D scene reconstruction, this study contrasts these modern approaches with traditional Simultaneous Localization and Mapping (SLAM) systems. Utilizing datasets such as Replica and ScanNet, we assess performance based on tracking accuracy, mapping fidelity, and view synthesis. Findings reveal that NeRF excels in view synthesis, offering unique capabilities in generating new perspectives from existing data, albeit at slower processing speeds. Conversely, Gaussian-based methods provide rapid processing and significant expressiveness but lack comprehensive scene completion. Enhanced by global optimization and loop closure techniques, newer methods like NICE-SLAM and SplaTAM not only surpass older frameworks such as ORB-SLAM2 in terms of robustness but also demonstrate superior performance in dynamic and complex environments. This comparative analysis bridges theoretical research with practical implications, shedding light on future developments in robust 3D scene reconstruction across various real-world applications.

Title: Analysis of Argument Structure Constructions in the Large Language Model BERT

• Authors: Pegah Ramezani, Achim Schilling, Patrick Krauss
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04270
• Pdf URL: https://arxiv.org/pdf/2408.04270
• Copy Paste: [[2408.04270]] Analysis of Argument Structure Constructions in the Large Language Model BERT(https://arxiv.org/abs/2408.04270)
• Keywords: large language model
• Abstract: This study investigates how BERT processes and represents Argument Structure Constructions (ASCs), extending previous LSTM analyses. Using a dataset of 2000 sentences across four ASC types (transitive, ditransitive, caused-motion, resultative), we analyzed BERT's token embeddings across 12 layers. Visualizations with MDS and t-SNE and clustering quantified by Generalized Discrimination Value (GDV) were used. Feedforward classifiers (probes) predicted construction categories from embeddings. CLS token embeddings clustered best in layers 2-4, decreased in intermediate layers, and slightly increased in final layers. DET and SUBJ embeddings showed consistent clustering in intermediate layers, VERB embeddings increased in clustering from layer 1 to 12, and OBJ embeddings peaked in layer 10. Probe accuracies indicated low construction information in layer 1, with over 90 percent accuracy from layer 2 onward, revealing latent construction information beyond GDV clustering. Fisher Discriminant Ratio (FDR) analysis of attention weights showed OBJ tokens were crucial for differentiating ASCs, followed by VERB and DET tokens. SUBJ, CLS, and SEP tokens had insignificant FDR scores. This study highlights BERT's layered processing of linguistic constructions and its differences from LSTMs. Future research will compare these findings with neuroimaging data to understand the neural correlates of ASC processing. This research underscores neural language models' potential to mirror linguistic processing in the human brain, offering insights into the computational and neural mechanisms underlying language understanding.

Title: Stability Analysis of Equivariant Convolutional Representations Through The Lens of Equivariant Multi-layered CKNs

• Authors: Soutrik Roy Chowdhury
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04277
• Pdf URL: https://arxiv.org/pdf/2408.04277
• Copy Paste: [[2408.04277]] Stability Analysis of Equivariant Convolutional Representations Through The Lens of Equivariant Multi-layered CKNs(https://arxiv.org/abs/2408.04277)
• Keywords: robust
• Abstract: In this paper we construct and theoretically analyse group equivariant convolutional kernel networks (CKNs) which are useful in understanding the geometry of (equivariant) CNNs through the lens of reproducing kernel Hilbert spaces (RKHSs). We then proceed to study the stability analysis of such equiv-CKNs under the action of diffeomorphism and draw a connection with equiv-CNNs, where the goal is to analyse the geometry of inductive biases of equiv-CNNs through the lens of reproducing kernel Hilbert spaces (RKHSs). Traditional deep learning architectures, including CNNs, trained with sophisticated optimization algorithms is vulnerable to perturbations, including `adversarial examples'. Understanding the RKHS norm of such models through CKNs is useful in designing the appropriate architecture and can be useful in designing robust equivariant representation learning models.

Title: LaDiMo: Layer-wise Distillation Inspired MoEfier

• Authors: Sungyoon Kim, Youngjun Kim, Kihyo Moon, Minsung Jang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04278
• Pdf URL: https://arxiv.org/pdf/2408.04278
• Copy Paste: [[2408.04278]] LaDiMo: Layer-wise Distillation Inspired MoEfier(https://arxiv.org/abs/2408.04278)
• Keywords: transformer, large language model
• Abstract: The advent of large language models has revolutionized natural language processing, but their increasing complexity has led to substantial training costs, resource demands, and environmental impacts. In response, sparse Mixture-of-Experts (MoE) models have emerged as a promising alternative to dense models. Since training MoE models from scratch can be prohibitively expensive, recent studies have explored leveraging knowledge from pre-trained non-MoE models. However, existing approaches have limitations, such as requiring significant hardware resources and data. We propose a novel algorithm, LaDiMo, which efficiently converts a Transformer-based non-MoE model into a MoE model with minimal additional training cost. LaDiMo consists of two stages: layer-wise expert construction and routing policy decision. By harnessing the concept of Knowledge Distillation, we compress the model and rapidly recover its performance. Furthermore, we develop an adaptive router that optimizes inference efficiency by profiling the distribution of routing weights and determining a layer-wise policy that balances accuracy and latency. We demonstrate the effectiveness of our method by converting the LLaMA2-7B model to a MoE model using only 100K tokens, reducing activated parameters by over 20% while keeping accuracy. Our approach offers a flexible and efficient solution for building and deploying MoE models.

Title: AI-Driven Chatbot for Intrusion Detection in Edge Networks: Enhancing Cybersecurity with Ethical User Consent

• Authors: Mugheez Asif, Abdul Manan, Abdul Moiz ur Rehman, Mamoona Naveed Asghar, Muhammad Umair
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04281
• Pdf URL: https://arxiv.org/pdf/2408.04281
• Copy Paste: [[2408.04281]] AI-Driven Chatbot for Intrusion Detection in Edge Networks: Enhancing Cybersecurity with Ethical User Consent(https://arxiv.org/abs/2408.04281)
• Keywords: secure, security, robust
• Abstract: In today's contemporary digital landscape, chatbots have become indispensable tools across various sectors, streamlining customer service, providing personal assistance, automating routine tasks, and offering health advice. However, their potential remains underexplored in the realm of network security, particularly for intrusion detection. To bridge this gap, we propose an architecture chatbot specifically designed to enhance security within edge networks specifically for intrusion detection. Leveraging advanced machine learning algorithms, this chatbot will monitor network traffic to identify and mitigate potential intrusions. By securing the network environment using an edge network managed by a Raspberry Pi module and ensuring ethical user consent promoting transparency and trust, this innovative solution aims to safeguard sensitive data and maintain a secure workplace, thereby addressing the growing need for robust network security measures in the digital age.

Title: LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection

• Authors: Mervat Abassy, Kareem Elozeiri, Alexander Aziz, Minh Ngoc Ta, Raj Vardhan Tomar, Bimarsha Adhikari, Saad El Dine Ahmed, Yuxia Wang, Osama Mohammed Afzal, Zhuohan Xie, Jonibek Mansurov, Ekaterina Artemova, Vladislav Mikhailov, Rui Xing, Jiahui Geng, Hasan Iqbal, Zain Muhammad Mujahid, Tarek Mahmoud, Akim Tsvigun, Alham Fikri Aji, Artem Shelmanov, Nizar Habash, Iryna Gurevych, Preslav Nakov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04284
• Pdf URL: https://arxiv.org/pdf/2408.04284
• Copy Paste: [[2408.04284]] LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection(https://arxiv.org/abs/2408.04284)
• Keywords: large language model
• Abstract: The widespread accessibility of large language models (LLMs) to the general public has significantly amplified the dissemination of machine-generated texts (MGTs). Advancements in prompt manipulation have exacerbated the difficulty in discerning the origin of a text (human-authored vs machinegenerated). This raises concerns regarding the potential misuse of MGTs, particularly within educational and academic domains. In this paper, we present $\textbf{LLM-DetectAIve}$ -- a system designed for fine-grained MGT detection. It is able to classify texts into four categories: human-written, machine-generated, machine-written machine-humanized, and human-written machine-polished. Contrary to previous MGT detectors that perform binary classification, introducing two additional categories in LLM-DetectiAIve offers insights into the varying degrees of LLM intervention during the text creation. This might be useful in some domains like education, where any LLM intervention is usually prohibited. Experiments show that LLM-DetectAIve can effectively identify the authorship of textual content, proving its usefulness in enhancing integrity in education, academia, and other domains. LLM-DetectAIve is publicly accessible at this https URL. The video describing our system is available at this https URL.

Title: EMTeC: A Corpus of Eye Movements on Machine-Generated Texts

• Authors: Lena Sophia Bolliger, Patrick Haller, Isabelle Caroline Rose Cretton, David Robert Reich, Tannon Kew, Lena Ann Jäger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04289
• Pdf URL: https://arxiv.org/pdf/2408.04289
• Copy Paste: [[2408.04289]] EMTeC: A Corpus of Eye Movements on Machine-Generated Texts(https://arxiv.org/abs/2408.04289)
• Keywords: interpretability, large language model
• Abstract: The Eye Movements on Machine-Generated Texts Corpus (EMTeC) is a naturalistic eye-movements-while-reading corpus of 107 native English speakers reading machine-generated texts. The texts are generated by three large language models using five different decoding strategies, and they fall into six different text type categories. EMTeC entails the eye movement data at all stages of pre-processing, i.e., the raw coordinate data sampled at 2000 Hz, the fixation sequences, and the reading measures. It further provides both the original and a corrected version of the fixation sequences, accounting for vertical calibration drift. Moreover, the corpus includes the language models' internals that underlie the generation of the stimulus texts: the transition scores, the attention scores, and the hidden states. The stimuli are annotated for a range of linguistic features both at text and at word level. We anticipate EMTeC to be utilized for a variety of use cases such as, but not restricted to, the investigation of reading behavior on machine-generated text and the impact of different decoding strategies; reading behavior on different text types; the development of new pre-processing, data filtering, and drift correction algorithms; the cognitive interpretability and enhancement of language models; and the assessment of the predictive power of surprisal and entropy for human reading times. The data at all stages of pre-processing, the model internals, and the code to reproduce the stimulus generation, data pre-processing and analyses can be accessed via this https URL.

Title: Are Social Sentiments Inherent in LLMs? An Empirical Study on Extraction of Inter-demographic Sentiments

• Authors: Kunitomo Tanaka, Ryohei Sasano, Koichi Takeda
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.04293
• Pdf URL: https://arxiv.org/pdf/2408.04293
• Copy Paste: [[2408.04293]] Are Social Sentiments Inherent in LLMs? An Empirical Study on Extraction of Inter-demographic Sentiments(https://arxiv.org/abs/2408.04293)
• Keywords: extraction, large language model
• Abstract: Large language models (LLMs) are supposed to acquire unconscious human knowledge and feelings, such as social common sense and biases, by training models from large amounts of text. However, it is not clear how much the sentiments of specific social groups can be captured in various LLMs. In this study, we focus on social groups defined in terms of nationality, religion, and race/ethnicity, and validate the extent to which sentiments between social groups can be captured in and extracted from LLMs. Specifically, we input questions regarding sentiments from one group to another into LLMs, apply sentiment analysis to the responses, and compare the results with social surveys. The validation results using five representative LLMs showed higher correlations with relatively small p-values for nationalities and religions, whose number of data points were relatively large. This result indicates that the LLM responses including the inter-group sentiments align well with actual social survey results.

Title: Dual-branch PolSAR Image Classification Based on GraphMAE and Local Feature Extraction

• Authors: Yuchen Wang, Ziyi Guo, Haixia Bi, Danfeng Hong, Chen Xu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04294
• Pdf URL: https://arxiv.org/pdf/2408.04294
• Copy Paste: [[2408.04294]] Dual-branch PolSAR Image Classification Based on GraphMAE and Local Feature Extraction(https://arxiv.org/abs/2408.04294)
• Keywords: extraction, generative
• Abstract: The annotation of polarimetric synthetic aperture radar (PolSAR) images is a labor-intensive and time-consuming process. Therefore, classifying PolSAR images with limited labels is a challenging task in remote sensing domain. In recent years, self-supervised learning approaches have proven effective in PolSAR image classification with sparse labels. However, we observe a lack of research on generative selfsupervised learning in the studied task. Motivated by this, we propose a dual-branch classification model based on generative self-supervised learning in this paper. The first branch is a superpixel-branch, which learns superpixel-level polarimetric representations using a generative self-supervised graph masked autoencoder. To acquire finer classification results, a convolutional neural networks-based pixel-branch is further incorporated to learn pixel-level features. Classification with fused dual-branch features is finally performed to obtain the predictions. Experimental results on the benchmark Flevoland dataset demonstrate that our approach yields promising classification results.

Title: Tackling Noisy Clients in Federated Learning with End-to-end Label Correction

• Authors: Xuefeng Jiang, Sheng Sun, Jia Li, Jingjing Xue, Runhan Li, Zhiyuan Wu, Gang Xu, Yuwei Wang, Min Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04301
• Pdf URL: https://arxiv.org/pdf/2408.04301
• Copy Paste: [[2408.04301]] Tackling Noisy Clients in Federated Learning with End-to-end Label Correction(https://arxiv.org/abs/2408.04301)
• Keywords: privacy, federate
• Abstract: Recently, federated learning (FL) has achieved wide successes for diverse privacy-sensitive applications without sacrificing the sensitive private information of clients. However, the data quality of client datasets can not be guaranteed since corresponding annotations of different clients often contain complex label noise of varying degrees, which inevitably causes the performance degradation. Intuitively, the performance degradation is dominated by clients with higher noise rates since their trained models contain more misinformation from data, thus it is necessary to devise an effective optimization scheme to mitigate the negative impacts of these noisy clients. In this work, we propose a two-stage framework FedELC to tackle this complicated label noise issue. The first stage aims to guide the detection of noisy clients with higher label noise, while the second stage aims to correct the labels of noisy clients' data via an end-to-end label correction framework which is achieved by learning possible ground-truth labels of noisy clients' datasets via back propagation. We implement sixteen related methods and evaluate five datasets with three types of complicated label noise scenarios for a comprehensive comparison. Extensive experimental results demonstrate our proposed framework achieves superior performance than its counterparts for different scenarios. Additionally, we effectively improve the data quality of detected noisy clients' local datasets with our label correction framework. The code is available at this https URL.

Title: Constructing Adversarial Examples for Vertical Federated Learning: Optimal Client Corruption through Multi-Armed Bandit

• Authors: Duanyi Yao, Songze Li, Ye Xue, Jin Liu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.04310
• Pdf URL: https://arxiv.org/pdf/2408.04310
• Copy Paste: [[2408.04310]] Constructing Adversarial Examples for Vertical Federated Learning: Optimal Client Corruption through Multi-Armed Bandit(https://arxiv.org/abs/2408.04310)
• Keywords: security, attack, federate
• Abstract: Vertical federated learning (VFL), where each participating client holds a subset of data features, has found numerous applications in finance, healthcare, and IoT systems. However, adversarial attacks, particularly through the injection of adversarial examples (AEs), pose serious challenges to the security of VFL models. In this paper, we investigate such vulnerabilities through developing a novel attack to disrupt the VFL inference process, under a practical scenario where the adversary is able to adaptively corrupt a subset of clients. We formulate the problem of finding optimal attack strategies as an online optimization problem, which is decomposed into an inner problem of adversarial example generation (AEG) and an outer problem of corruption pattern selection (CPS). Specifically, we establish the equivalence between the formulated CPS problem and a multi-armed bandit (MAB) problem, and propose the Thompson sampling with Empirical maximum reward (E-TS) algorithm for the adversary to efficiently identify the optimal subset of clients for corruption. The key idea of E-TS is to introduce an estimation of the expected maximum reward for each arm, which helps to specify a small set of competitive arms, on which the exploration for the optimal arm is performed. This significantly reduces the exploration space, which otherwise can quickly become prohibitively large as the number of clients increases. We analytically characterize the regret bound of E-TS, and empirically demonstrate its capability of efficiently revealing the optimal corruption pattern with the highest attack success rate, under various datasets of popular VFL tasks.

Title: Federated Cubic Regularized Newton Learning with Sparsification-amplified Differential Privacy

• Authors: Wei Huo, Changxin Liu, Kemi Ding, Karl Henrik Johansson, Ling Shi
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2408.04315
• Pdf URL: https://arxiv.org/pdf/2408.04315
• Copy Paste: [[2408.04315]] Federated Cubic Regularized Newton Learning with Sparsification-amplified Differential Privacy(https://arxiv.org/abs/2408.04315)
• Keywords: privacy, protect, federate
• Abstract: This paper investigates the use of the cubic-regularized Newton method within a federated learning framework while addressing two major concerns that commonly arise in federated learning: privacy leakage and communication bottleneck. We introduce a federated learning algorithm called Differentially Private Federated Cubic Regularized Newton (DP-FCRN). By leveraging second-order techniques, our algorithm achieves lower iteration complexity compared to first-order methods. We also incorporate noise perturbation during local computations to ensure privacy. Furthermore, we employ sparsification in uplink transmission, which not only reduces the communication costs but also amplifies the privacy guarantee. Specifically, this approach reduces the necessary noise intensity without compromising privacy protection. We analyze the convergence properties of our algorithm and establish the privacy guarantee. Finally, we validate the effectiveness of the proposed algorithm through experiments on a benchmark dataset.

Title: Multi-Scale and Detail-Enhanced Segment Anything Model for Salient Object Detection

• Authors: Shixuan Gao, Pingping Zhang, Tianyu Yan, Huchuan Lu
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.04326
• Pdf URL: https://arxiv.org/pdf/2408.04326
• Copy Paste: [[2408.04326]] Multi-Scale and Detail-Enhanced Segment Anything Model for Salient Object Detection(https://arxiv.org/abs/2408.04326)
• Keywords: extraction, transformer, segmentation
• Abstract: Salient Object Detection (SOD) aims to identify and segment the most prominent objects in images. Advanced SOD methods often utilize various Convolutional Neural Networks (CNN) or Transformers for deep feature extraction. However, these methods still deliver low performance and poor generalization in complex cases. Recently, Segment Anything Model (SAM) has been proposed as a visual fundamental model, which gives strong segmentation and generalization capabilities. Nonetheless, SAM requires accurate prompts of target objects, which are unavailable in SOD. Additionally, SAM lacks the utilization of multi-scale and multi-level information, as well as the incorporation of fine-grained details. To address these shortcomings, we propose a Multi-scale and Detail-enhanced SAM (MDSAM) for SOD. Specifically, we first introduce a Lightweight Multi-Scale Adapter (LMSA), which allows SAM to learn multi-scale information with very few trainable parameters. Then, we propose a Multi-Level Fusion Module (MLFM) to comprehensively utilize the multi-level information from the SAM's encoder. Finally, we propose a Detail Enhancement Module (DEM) to incorporate SAM with fine-grained details. Experimental results demonstrate the superior performance of our model on multiple SOD datasets and its strong generalization on other segmentation tasks. The source code is released at this https URL.

Title: Enhancing Journalism with AI: A Study of Contextualized Image Captioning for News Articles using LLMs and LMMs

• Authors: Aliki Anagnostopoulou, Thiago Gouvea, Daniel Sonntag
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04331
• Pdf URL: https://arxiv.org/pdf/2408.04331
• Copy Paste: [[2408.04331]] Enhancing Journalism with AI: A Study of Contextualized Image Captioning for News Articles using LLMs and LMMs(https://arxiv.org/abs/2408.04331)
• Keywords: large language model
• Abstract: Large language models (LLMs) and large multimodal models (LMMs) have significantly impacted the AI community, industry, and various economic sectors. In journalism, integrating AI poses unique challenges and opportunities, particularly in enhancing the quality and efficiency of news reporting. This study explores how LLMs and LMMs can assist journalistic practice by generating contextualised captions for images accompanying news articles. We conducted experiments using the GoodNews dataset to evaluate the ability of LMMs (BLIP-2, GPT-4v, or LLaVA) to incorporate one of two types of context: entire news articles, or extracted named entities. In addition, we compared their performance to a two-stage pipeline composed of a captioning model (BLIP-2, OFA, or ViT-GPT2) with post-hoc contextualisation with LLMs (GPT-4 or LLaMA). We assess a diversity of models, and we find that while the choice of contextualisation model is a significant factor for the two-stage pipelines, this is not the case in the LMMs, where smaller, open-source models perform well compared to proprietary, GPT-powered ones. Additionally, we found that controlling the amount of provided context enhances performance. These results highlight the limitations of a fully automated approach and underscore the necessity for an interactive, human-in-the-loop strategy.

Title: Towards Explainable Network Intrusion Detection using Large Language Models

• Authors: Paul R. B. Houssel, Priyanka Singh, Siamak Layeghy, Marius Portmann
• Subjects: cs.CR, cs.AI, cs.NI
• Abstract URL: https://arxiv.org/abs/2408.04342
• Pdf URL: https://arxiv.org/pdf/2408.04342
• Copy Paste: [[2408.04342]] Towards Explainable Network Intrusion Detection using Large Language Models(https://arxiv.org/abs/2408.04342)
• Keywords: attack, explainability, transformer, large language model
• Abstract: Large Language Models (LLMs) have revolutionised natural language processing tasks, particularly as chat agents. However, their applicability to threat detection problems remains unclear. This paper examines the feasibility of employing LLMs as a Network Intrusion Detection System (NIDS), despite their high computational requirements, primarily for the sake of explainability. Furthermore, considerable resources have been invested in developing LLMs, and they may offer utility for NIDS. Current state-of-the-art NIDS rely on artificial benchmarking datasets, resulting in skewed performance when applied to real-world networking environments. Therefore, we compare the GPT-4 and LLama3 models against traditional architectures and transformer-based models to assess their ability to detect malicious NetFlows without depending on artificially skewed datasets, but solely on their vast pre-trained acquired knowledge. Our results reveal that, although LLMs struggle with precise attack detection, they hold significant potential for a path towards explainable NIDS. Our preliminary exploration shows that LLMs are unfit for the detection of Malicious NetFlows. Most promisingly, however, these exhibit significant potential as complementary agents in NIDS, particularly in providing explanations and aiding in threat response when integrated with Retrieval Augmented Generation (RAG) and function calling capabilities.

Title: AggSS: An Aggregated Self-Supervised Approach for Class-Incremental Learning

• Authors: Jayateja Kalla, Soma Biswas
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04347
• Pdf URL: https://arxiv.org/pdf/2408.04347
• Copy Paste: [[2408.04347]] AggSS: An Aggregated Self-Supervised Approach for Class-Incremental Learning(https://arxiv.org/abs/2408.04347)
• Keywords: robust
• Abstract: This paper investigates the impact of self-supervised learning, specifically image rotations, on various class-incremental learning paradigms. Here, each image with a predefined rotation is considered as a new class for training. At inference, all image rotation predictions are aggregated for the final prediction, a strategy we term Aggregated Self-Supervision (AggSS). We observe a shift in the deep neural network's attention towards intrinsic object features as it learns through AggSS strategy. This learning approach significantly enhances class-incremental learning by promoting robust feature learning. AggSS serves as a plug-and-play module that can be seamlessly incorporated into any class-incremental learning framework, leveraging its powerful feature learning capabilities to enhance performance across various class-incremental learning approaches. Extensive experiments conducted on standard incremental learning datasets CIFAR-100 and ImageNet-Subset demonstrate the significant role of AggSS in improving performance within these paradigms.

Title: Fuzzy to Clear: Elucidating the Threat Hunter Cognitive Process and Cognitive Support Needs

• Authors: Alessandra Maciel Paz Milani, Arty Starr, Samantha Hill, Callum Curtis, Norman Anderson, David Moreno-Lumbreras, Margaret-Anne Storey
• Subjects: cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.04348
• Pdf URL: https://arxiv.org/pdf/2408.04348
• Copy Paste: [[2408.04348]] Fuzzy to Clear: Elucidating the Threat Hunter Cognitive Process and Cognitive Support Needs(https://arxiv.org/abs/2408.04348)
• Keywords: security
• Abstract: With security threats increasing in frequency and severity, it is critical that we consider the important role of threat hunters. These highly-trained security professionals learn to see, identify, and intercept security threats. Many recent works and existing tools in cybersecurity are focused on automating the threat hunting process, often overlooking the critical human element. Our study shifts this paradigm by emphasizing a human-centered approach to understanding the lived experiences of threat hunters. By observing threat hunters during hunting sessions and analyzing the rich insights they provide, we seek to advance the understanding of their cognitive processes and the tool support they need. Through an in-depth observational study of threat hunters, we introduce a model of how they build and refine their mental models during threat hunting sessions. We also present 23 themes that provide a foundation to better understand threat hunter needs and five actionable design propositions to enhance the tools that support them. Through these contributions, our work enriches the theoretical understanding of threat hunting and provides practical insights for designing more effective, human-centered cybersecurity tools.

Title: MultiViPerFrOG: A Globally Optimized Multi-Viewpoint Perception Framework for Camera Motion and Tissue Deformation

• Authors: Guido Caccianiga, Julian Nubert, Cesar Cadena, Marco Hutter, Katherine J. Kuchenbecker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04367
• Pdf URL: https://arxiv.org/pdf/2408.04367
• Copy Paste: [[2408.04367]] MultiViPerFrOG: A Globally Optimized Multi-Viewpoint Perception Framework for Camera Motion and Tissue Deformation(https://arxiv.org/abs/2408.04367)
• Keywords: robust
• Abstract: Reconstructing the 3D shape of a deformable environment from the information captured by a moving depth camera is highly relevant to surgery. The underlying challenge is the fact that simultaneously estimating camera motion and tissue deformation in a fully deformable scene is an ill-posed problem, especially from a single arbitrarily moving viewpoint. Current solutions are often organ-specific and lack the robustness required to handle large deformations. Here we propose a multi-viewpoint global optimization framework that can flexibly integrate the output of low-level perception modules (data association, depth, and relative scene flow) with kinematic and scene-modeling priors to jointly estimate multiple camera motions and absolute scene flow. We use simulated noisy data to show three practical examples that successfully constrain the convergence to a unique solution. Overall, our method shows robustness to combined noisy input measures and can process hundreds of points in a few milliseconds. MultiViPerFrOG builds a generalized learning-free scaffolding for spatio-temporal encoding that can unlock advanced surgical scene representations and will facilitate the development of the computer-assisted-surgery technologies of the future.

Title: Analyzing Consumer Reviews for Understanding Drivers of Hotels Ratings: An Indian Perspective

• Authors: Subhasis Dasgupta, Soumya Roy, Jaydip Sen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04369
• Pdf URL: https://arxiv.org/pdf/2408.04369
• Copy Paste: [[2408.04369]] Analyzing Consumer Reviews for Understanding Drivers of Hotels Ratings: An Indian Perspective(https://arxiv.org/abs/2408.04369)
• Keywords: extraction
• Abstract: In the internet era, almost every business entity is trying to have its digital footprint in digital media and other social media platforms. For these entities, word of mouse is also very important. Particularly, this is quite crucial for the hospitality sector dealing with hotels, restaurants etc. Consumers do read other consumers reviews before making final decisions. This is where it becomes very important to understand which aspects are affecting most in the minds of the consumers while giving their ratings. The current study focuses on the consumer reviews of Indian hotels to extract aspects important for final ratings. The study involves gathering data using web scraping methods, analyzing the texts using Latent Dirichlet Allocation for topic extraction and sentiment analysis for aspect-specific sentiment mapping. Finally, it incorporates Random Forest to understand the importance of the aspects in predicting the final rating of a user.

Title: Open-domain Implicit Format Control for Large Language Model Generation

• Authors: Yiqun Yao, Wenjia Ma, Xuezhi Fang, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Jing Li, Aixin Sun, Yequan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04392
• Pdf URL: https://arxiv.org/pdf/2408.04392
• Copy Paste: [[2408.04392]] Open-domain Implicit Format Control for Large Language Model Generation(https://arxiv.org/abs/2408.04392)
• Keywords: large language model
• Abstract: Controlling the format of outputs generated by large language models (LLMs) is a critical functionality in various applications. Current methods typically employ constrained decoding with rule-based automata or fine-tuning with manually crafted format instructions, both of which struggle with open-domain format requirements. To address this limitation, we introduce a novel framework for controlled generation in LLMs, leveraging user-provided, one-shot QA pairs. This study investigates LLMs' capabilities to follow open-domain, one-shot constraints and replicate the format of the example answers. We observe that this is a non-trivial problem for current LLMs. We also develop a dataset collection methodology for supervised fine-tuning that enhances the open-domain format control of LLMs without degrading output quality, as well as a benchmark on which we evaluate both the helpfulness and format correctness of LLM outputs. The resulting datasets, named OIFC-SFT, along with the related code, will be made publicly available at this https URL.

Title: Automated Educational Question Generation at Different Bloom's Skill Levels using Large Language Models: Strategies and Evaluation

• Authors: Nicy Scaria, Suma Dharani Chenna, Deepak Subramani
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04394
• Pdf URL: https://arxiv.org/pdf/2408.04394
• Copy Paste: [[2408.04394]] Automated Educational Question Generation at Different Bloom's Skill Levels using Large Language Models: Strategies and Evaluation(https://arxiv.org/abs/2408.04394)
• Keywords: large language model
• Abstract: Developing questions that are pedagogically sound, relevant, and promote learning is a challenging and time-consuming task for educators. Modern-day large language models (LLMs) generate high-quality content across multiple domains, potentially helping educators to develop high-quality questions. Automated educational question generation (AEQG) is important in scaling online education catering to a diverse student population. Past attempts at AEQG have shown limited abilities to generate questions at higher cognitive levels. In this study, we examine the ability of five state-of-the-art LLMs of different sizes to generate diverse and high-quality questions of different cognitive levels, as defined by Bloom's taxonomy. We use advanced prompting techniques with varying complexity for AEQG. We conducted expert and LLM-based evaluations to assess the linguistic and pedagogical relevance and quality of the questions. Our findings suggest that LLms can generate relevant and high-quality educational questions of different cognitive levels when prompted with adequate information, although there is a significant variance in the performance of the five LLms considered. We also show that automated evaluation is not on par with human evaluation.

Title: Evaluating the Impact of Pulse Oximetry Bias in Machine Learning under Counterfactual Thinking

• Authors: Inês Martins, João Matos, Tiago Gonçalves, Leo A. Celi, A. Ian Wong, Jaime S. Cardoso
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04396
• Pdf URL: https://arxiv.org/pdf/2408.04396
• Copy Paste: [[2408.04396]] Evaluating the Impact of Pulse Oximetry Bias in Machine Learning under Counterfactual Thinking(https://arxiv.org/abs/2408.04396)
• Keywords: fair
• Abstract: Algorithmic bias in healthcare mirrors existing data biases. However, the factors driving unfairness are not always known. Medical devices capture significant amounts of data but are prone to errors; for instance, pulse oximeters overestimate the arterial oxygen saturation of darker-skinned individuals, leading to worse outcomes. The impact of this bias in machine learning (ML) models remains unclear. This study addresses the technical challenges of quantifying the impact of medical device bias in downstream ML. Our experiments compare a "perfect world", without pulse oximetry bias, using SaO2 (blood-gas), to the "actual world", with biased measurements, using SpO2 (pulse oximetry). Under this counterfactual design, two models are trained with identical data, features, and settings, except for the method of measuring oxygen saturation: models using SaO2 are a "control" and models using SpO2 a "treatment". The blood-gas oximetry linked dataset was a suitable test-bed, containing 163,396 nearly-simultaneous SpO2 - SaO2 paired measurements, aligned with a wide array of clinical features and outcomes. We studied three classification tasks: in-hospital mortality, respiratory SOFA score in the next 24 hours, and SOFA score increase by two points. Models using SaO2 instead of SpO2 generally showed better performance. Patients with overestimation of O2 by pulse oximetry of > 3% had significant decreases in mortality prediction recall, from 0.63 to 0.59, P < 0.001. This mirrors clinical processes where biased pulse oximetry readings provide clinicians with false reassurance of patients' oxygen levels. A similar degradation happened in ML models, with pulse oximetry biases leading to more false negatives in predicting adverse outcomes.

Title: DIVE: Subgraph Disagreement for Graph Out-of-Distribution Generalization

• Authors: Xin Sun, Liang Wang, Qiang Liu, Shu Wu, Zilei Wang, Liang Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04400
• Pdf URL: https://arxiv.org/pdf/2408.04400
• Copy Paste: [[2408.04400]] DIVE: Subgraph Disagreement for Graph Out-of-Distribution Generalization(https://arxiv.org/abs/2408.04400)
• Keywords: robust
• Abstract: This paper addresses the challenge of out-of-distribution (OOD) generalization in graph machine learning, a field rapidly advancing yet grappling with the discrepancy between source and target data distributions. Traditional graph learning algorithms, based on the assumption of uniform distribution between training and test data, falter in real-world scenarios where this assumption fails, resulting in suboptimal performance. A principal factor contributing to this suboptimal performance is the inherent simplicity bias of neural networks trained through Stochastic Gradient Descent (SGD), which prefer simpler features over more complex yet equally or more predictive ones. This bias leads to a reliance on spurious correlations, adversely affecting OOD performance in various tasks such as image recognition, natural language understanding, and graph classification. Current methodologies, including subgraph-mixup and information bottleneck approaches, have achieved partial success but struggle to overcome simplicity bias, often reinforcing spurious correlations. To tackle this, we propose DIVE, training a collection of models to focus on all label-predictive subgraphs by encouraging the models to foster divergence on the subgraph mask, which circumvents the limitation of a model solely focusing on the subgraph corresponding to simple structural patterns. Specifically, we employs a regularizer to punish overlap in extracted subgraphs across models, thereby encouraging different models to concentrate on distinct structural patterns. Model selection for robust OOD performance is achieved through validation accuracy. Tested across four datasets from GOOD benchmark and one dataset from DrugOOD benchmark, our approach demonstrates significant improvement over existing methods, effectively addressing the simplicity bias and enhancing generalization in graph machine learning.

Title: Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset

• Authors: Kentaro Ozeki, Risako Ando, Takanobu Morishita, Hirohiko Abe, Koji Mineshima, Mitsuhiro Okada
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04403
• Pdf URL: https://arxiv.org/pdf/2408.04403
• Copy Paste: [[2408.04403]] Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset(https://arxiv.org/abs/2408.04403)
• Keywords: large language model
• Abstract: This paper explores the question of how accurately current large language models can perform logical reasoning in natural language, with an emphasis on whether these models exhibit reasoning biases similar to humans. Specifically, our study focuses on syllogistic reasoning, a form of deductive reasoning extensively studied in cognitive science as a natural form of human reasoning. We present a syllogism dataset called NeuBAROCO, which consists of syllogistic reasoning problems in English and Japanese. This dataset was originally designed for psychological experiments to assess human reasoning capabilities using various forms of syllogisms. Our experiments with leading large language models indicate that these models exhibit reasoning biases similar to humans, along with other error tendencies. Notably, there is significant room for improvement in reasoning problems where the relationship between premises and hypotheses is neither entailment nor contradiction. We also present experimental results and in-depth analysis using a new Chain-of-Thought prompting method, which asks LLMs to translate syllogisms into abstract logical expressions and then explain their reasoning process. Our analysis using this method suggests that the primary limitations of LLMs lie in the reasoning process itself rather than the interpretation of syllogisms.

Title: Deeploy: Enabling Energy-Efficient Deployment of Small Language Models On Heterogeneous Microcontrollers

• Authors: Moritz Scherer, Luka Macan, Victor Jung, Philip Wiese, Luca Bompani, Alessio Burrello, Francesco Conti, Luca Benini
• Subjects: cs.LG, cs.AR
• Abstract URL: https://arxiv.org/abs/2408.04413
• Pdf URL: https://arxiv.org/pdf/2408.04413
• Copy Paste: [[2408.04413]] Deeploy: Enabling Energy-Efficient Deployment of Small Language Models On Heterogeneous Microcontrollers(https://arxiv.org/abs/2408.04413)
• Keywords: transformer
• Abstract: With the rise of Embodied Foundation Models (EFMs), most notably Small Language Models (SLMs), adapting Transformers for edge applications has become a very active field of research. However, achieving end-to-end deployment of SLMs on microcontroller (MCU)-class chips without high-bandwidth off-chip main memory access is still an open challenge. In this paper, we demonstrate high-efficiency end-to-end SLM deployment on a multicore RISC-V (RV32) MCU augmented with ML instruction extensions and a hardware neural processing unit (NPU). To automate the exploration of the constrained, multi-dimensional memory vs. computation tradeoffs involved in aggressive SLM deployment on heterogeneous (multicore+NPU) resources, we introduce Deeploy, a novel Deep Neural Network (DNN) compiler, which generates highly-optimized C code requiring minimal runtime support. We demonstrate that Deeploy generates end-to-end code for executing SLMs, fully exploiting the RV32 cores' instruction extensions and the NPU: We achieve leading-edge energy and throughput of \SI{490}{\micro\joule \per Token}, at \SI{340}{Token \per \second} for an SLM trained on the TinyStories dataset, running for the first time on an MCU-class device without external memory.

Title: Enhancing Robustness of Retrieval-Augmented Language Models with In-Context Learning

• Authors: Seong-Il Park, Seung-Woo Choi, Na-Hyun Kim, Jay-Yoon Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04414
• Pdf URL: https://arxiv.org/pdf/2408.04414
• Copy Paste: [[2408.04414]] Enhancing Robustness of Retrieval-Augmented Language Models with In-Context Learning(https://arxiv.org/abs/2408.04414)
• Keywords: robust
• Abstract: Retrieval-Augmented Language Models (RALMs) have significantly improved performance in open-domain question answering (QA) by leveraging external knowledge. However, RALMs still struggle with unanswerable queries, where the retrieved contexts do not contain the correct answer, and with conflicting information, where different sources provide contradictory answers due to imperfect retrieval. This study introduces an in-context learning-based approach to enhance the reasoning capabilities of RALMs, making them more robust in imperfect retrieval scenarios. Our method incorporates Machine Reading Comprehension (MRC) demonstrations, referred to as cases, to boost the model's capabilities to identify unanswerabilities and conflicts among the retrieved contexts. Experiments on two open-domain QA datasets show that our approach increases accuracy in identifying unanswerable and conflicting scenarios without requiring additional fine-tuning. This work demonstrates that in-context learning can effectively enhance the robustness of RALMs in open-domain QA tasks.

Title: Recognizing Emotion Regulation Strategies from Human Behavior with Large Language Models

• Authors: Philipp Müller, Alexander Heimerl, Sayed Muddashir Hossain, Lea Siegel, Jan Alexandersson, Patrick Gebhard, Elisabeth André, Tanja Schneeberger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04420
• Pdf URL: https://arxiv.org/pdf/2408.04420
• Copy Paste: [[2408.04420]] Recognizing Emotion Regulation Strategies from Human Behavior with Large Language Models(https://arxiv.org/abs/2408.04420)
• Keywords: large language model
• Abstract: Human emotions are often not expressed directly, but regulated according to internal processes and social display rules. For affective computing systems, an understanding of how users regulate their emotions can be highly useful, for example to provide feedback in job interview training, or in psychotherapeutic scenarios. However, at present no method to automatically classify different emotion regulation strategies in a cross-user scenario exists. At the same time, recent studies showed that instruction-tuned Large Language Models (LLMs) can reach impressive performance across a variety of affect recognition tasks such as categorical emotion recognition or sentiment analysis. While these results are promising, it remains unclear to what extent the representational power of LLMs can be utilized in the more subtle task of classifying users' internal emotion regulation strategy. To close this gap, we make use of the recently introduced \textsc{Deep} corpus for modeling the social display of the emotion shame, where each point in time is annotated with one of seven different emotion regulation classes. We fine-tune Llama2-7B as well as the recently introduced Gemma model using Low-rank Optimization on prompts generated from different sources of information on the \textsc{Deep} corpus. These include verbal and nonverbal behavior, person factors, as well as the results of an in-depth interview after the interaction. Our results show, that a fine-tuned Llama2-7B LLM is able to classify the utilized emotion regulation strategy with high accuracy (0.84) without needing access to data from post-interaction interviews. This represents a significant improvement over previous approaches based on Bayesian Networks and highlights the importance of modeling verbal behavior in emotion regulation.

Title: Detection of Animal Movement from Weather Radar using Self-Supervised Learning

• Authors: Mubin Ul Haque, Joel Janek Dabrowski, Rebecca M. Rogers, Hazel Parry
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04424
• Pdf URL: https://arxiv.org/pdf/2408.04424
• Copy Paste: [[2408.04424]] Detection of Animal Movement from Weather Radar using Self-Supervised Learning(https://arxiv.org/abs/2408.04424)
• Keywords: security, segmentation
• Abstract: Detecting flying animals (e.g., birds, bats, and insects) using weather radar helps gain insights into animal movement and migration patterns, aids in management efforts (such as biosecurity) and enhances our understanding of the ecosystem.The conventional approach to detecting animals in weather radar involves thresholding: defining and applying thresholds for the radar variables, based on expert opinion. More recently, Deep Learning approaches have been shown to provide improved performance in detection. However, obtaining sufficient labelled weather radar data for flying animals to build learning-based models is time-consuming and labor-intensive. To address the challenge of data labelling, we propose a self-supervised learning method for detecting animal movement. In our proposed method, we pre-train our model on a large dataset with noisy labels produced by a threshold approach. The key advantage is that the pre-trained dataset size is limited only by the number of radar images available. We then fine-tune the model on a small human-labelled dataset. Our experiments on Australian weather radar data for waterbird segmentation show that the proposed method outperforms the current state-of-the art approach by 43.53% in the dice co-efficient statistic.

Title: FedAD-Bench: A Unified Benchmark for Federated Unsupervised Anomaly Detection in Tabular Data

• Authors: Ahmed Anwar, Brian Moser, Dayananda Herurkar, Federico Raue, Vinit Hegiste, Tatjana Legler, Andreas Dengel
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04442
• Pdf URL: https://arxiv.org/pdf/2408.04442
• Copy Paste: [[2408.04442]] FedAD-Bench: A Unified Benchmark for Federated Unsupervised Anomaly Detection in Tabular Data(https://arxiv.org/abs/2408.04442)
• Keywords: security, privacy, federate, fair
• Abstract: The emergence of federated learning (FL) presents a promising approach to leverage decentralized data while preserving privacy. Furthermore, the combination of FL and anomaly detection is particularly compelling because it allows for detecting rare and critical anomalies (usually also rare in locally gathered data) in sensitive data from multiple sources, such as cybersecurity and healthcare. However, benchmarking the performance of anomaly detection methods in FL environments remains an underexplored area. This paper introduces FedAD-Bench, a unified benchmark for evaluating unsupervised anomaly detection algorithms within the context of FL. We systematically analyze and compare the performance of recent deep learning anomaly detection models under federated settings, which were typically assessed solely in centralized settings. FedAD-Bench encompasses diverse datasets and metrics to provide a holistic evaluation. Through extensive experiments, we identify key challenges such as model aggregation inefficiencies and metric unreliability. We present insights into FL's regularization effects, revealing scenarios in which it outperforms centralized approaches due to its inherent ability to mitigate overfitting. Our work aims to establish a standardized benchmark to guide future research and development in federated anomaly detection, promoting reproducibility and fair comparison across studies.

Title: Random Walk Diffusion for Efficient Large-Scale Graph Generation

• Authors: Tobias Bernecker, Ghalia Rehawi, Francesco Paolo Casale, Janine Knauer-Arloth, Annalisa Marsico
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2408.04461
• Pdf URL: https://arxiv.org/pdf/2408.04461
• Copy Paste: [[2408.04461]] Random Walk Diffusion for Efficient Large-Scale Graph Generation(https://arxiv.org/abs/2408.04461)
• Keywords: diffusion
• Abstract: Graph generation addresses the problem of generating new graphs that have a data distribution similar to real-world graphs. While previous diffusion-based graph generation methods have shown promising results, they often struggle to scale to large graphs. In this work, we propose ARROW-Diff (AutoRegressive RandOm Walk Diffusion), a novel random walk-based diffusion approach for efficient large-scale graph generation. Our method encompasses two components in an iterative process of random walk sampling and graph pruning. We demonstrate that ARROW-Diff can scale to large graphs efficiently, surpassing other baseline methods in terms of both generation time and multiple graph statistics, reflecting the high quality of the generated graphs.

Title: Crowd Intelligence for Early Misinformation Prediction on Social Media

• Authors: Megha Sundriyal, Harshit Choudhary, Tanmoy Chakraborty, Md Shad Akhtar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04463
• Pdf URL: https://arxiv.org/pdf/2408.04463
• Copy Paste: [[2408.04463]] Crowd Intelligence for Early Misinformation Prediction on Social Media(https://arxiv.org/abs/2408.04463)
• Keywords: transformer
• Abstract: Misinformation spreads rapidly on social media, causing serious damage by influencing public opinion, promoting dangerous behavior, or eroding trust in reliable sources. It spreads too fast for traditional fact-checking, stressing the need for predictive methods. We introduce CROWDSHIELD, a crowd intelligence-based method for early misinformation prediction. We hypothesize that the crowd's reactions to misinformation reveal its accuracy. Furthermore, we hinge upon exaggerated assertions/claims and replies with particular positions/stances on the source post within a conversation thread. We employ Q-learning to capture the two dimensions -- stances and claims. We utilize deep Q-learning due to its proficiency in navigating complex decision spaces and effectively learning network properties. Additionally, we use a transformer-based encoder to develop a comprehensive understanding of both content and context. This multifaceted approach helps ensure the model pays attention to user interaction and stays anchored in the communication's content. We propose MIST, a manually annotated misinformation detection Twitter corpus comprising nearly 200 conversation threads with more than 14K replies. In experiments, CROWDSHIELD outperformed ten baseline systems, achieving an improvement of ~4% macro-F1 score. We conduct an ablation study and error analysis to validate our proposed model's performance. The source code and dataset are available at this https URL.

Title: What could go wrong? Discovering and describing failure modes in computer vision

• Authors: Gabriela Csurka, Tyler L. Hayes, Diane Larlus, Riccardo Volpi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04471
• Pdf URL: https://arxiv.org/pdf/2408.04471
• Copy Paste: [[2408.04471]] What could go wrong? Discovering and describing failure modes in computer vision(https://arxiv.org/abs/2408.04471)
• Keywords: interpretability, explainability, segmentation
• Abstract: Deep learning models are effective, yet brittle. Even carefully trained, their behavior tends to be hard to predict when confronted with out-of-distribution samples. In this work, our goal is to propose a simple yet effective solution to predict and describe via natural language potential failure modes of computer vision models. Given a pretrained model and a set of samples, our aim is to find sentences that accurately describe the visual conditions in which the model underperforms. In order to study this important topic and foster future research on it, we formalize the problem of Language-Based Error Explainability (LBEE) and propose a set of metrics to evaluate and compare different methods for this task. We propose solutions that operate in a joint vision-and-language embedding space, and can characterize through language descriptions model failures caused, e.g., by objects unseen during training or adverse visual conditions. We experiment with different tasks, such as classification under the presence of dataset bias and semantic segmentation in unseen environments, and show that the proposed methodology isolates nontrivial sentences associated with specific error causes. We hope our work will help practitioners better understand the behavior of models, increasing their overall safety and interpretability.

Title: Can LLMs Beat Humans in Debating? A Dynamic Multi-agent Framework for Competitive Debate

• Authors: Yiqun Zhang, Xiaocui Yang, Shi Feng, Daling Wang, Yifei Zhang, Kaisong Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04472
• Pdf URL: https://arxiv.org/pdf/2408.04472
• Copy Paste: [[2408.04472]] Can LLMs Beat Humans in Debating? A Dynamic Multi-agent Framework for Competitive Debate(https://arxiv.org/abs/2408.04472)
• Keywords: large language model
• Abstract: Competitive debate is a comprehensive and complex computational argumentation task. Large Language Models (LLMs) encounter hallucinations and lack competitiveness in this task. To address these challenges, we introduce Agent for Debate (Agent4Debate), a dynamic, multi-agent framework based on LLMs designed to enhance their capabilities in competitive debate. Drawing inspiration from human behavior in debate preparation and execution, Agent4Debate employs a collaborative architecture where four specialized agents (Searcher, Analyzer, Writer, and Reviewer) dynamically interact and cooperate. These agents work throughout the debate process, covering multiple stages from initial research and argument formulation to rebuttal and summary. To comprehensively evaluate framework performance, we construct the Chinese Debate Arena, comprising 66 carefully selected Chinese debate motions. We recruite ten experienced human debaters and collect records of 200 debates involving Agent4Debate, baseline models, and humans. The evaluation employs the Debatrix automatic scoring system and professional human reviewers based on the established Debatrix-Elo and Human-Elo ranking. Experimental results indicate that the state-of-the-art Agent4Debate exhibits capabilities comparable to those of humans. Furthermore, ablation studies demonstrate the effectiveness of each component in the agent structure.

Title: NFDI4Health workflow and service for synthetic data generation, assessment and risk management

• Authors: Sobhan Moazemi, Tim Adams, Hwei Geok NG, Lisa Kühnel, Julian Schneider, Anatol-Fiete Näher, Juliane Fluck, Holger Fröhlich
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04478
• Pdf URL: https://arxiv.org/pdf/2408.04478
• Copy Paste: [[2408.04478]] NFDI4Health workflow and service for synthetic data generation, assessment and risk management(https://arxiv.org/abs/2408.04478)
• Keywords: privacy, generative
• Abstract: Individual health data is crucial for scientific advancements, particularly in developing Artificial Intelligence (AI); however, sharing real patient information is often restricted due to privacy concerns. A promising solution to this challenge is synthetic data generation. This technique creates entirely new datasets that mimic the statistical properties of real data, while preserving confidential patient information. In this paper, we present the workflow and different services developed in the context of Germany's National Data Infrastructure project NFDI4Health. First, two state-of-the-art AI tools (namely, VAMBN and MultiNODEs) for generating synthetic health data are outlined. Further, we introduce SYNDAT (a public web-based tool) which allows users to visualize and assess the quality and risk of synthetic data provided by desired generative models. Additionally, the utility of the proposed methods and the web-based tool is showcased using data from Alzheimer's Disease Neuroimaging Initiative (ADNI) and the Center for Cancer Registry Data of the Robert Koch Institute (RKI).

Title: SegXAL: Explainable Active Learning for Semantic Segmentation in Driving Scene Scenarios

• Authors: Sriram Mandalika, Athira Nambiar
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.04482
• Pdf URL: https://arxiv.org/pdf/2408.04482
• Copy Paste: [[2408.04482]] SegXAL: Explainable Active Learning for Semantic Segmentation in Driving Scene Scenarios(https://arxiv.org/abs/2408.04482)
• Keywords: segmentation
• Abstract: Most of the sophisticated AI models utilize huge amounts of annotated data and heavy training to achieve high-end performance. However, there are certain challenges that hinder the deployment of AI models "in-the-wild" scenarios, i.e., inefficient use of unlabeled data, lack of incorporation of human expertise, and lack of interpretation of the results. To mitigate these challenges, we propose a novel Explainable Active Learning (XAL) model, XAL-based semantic segmentation model "SegXAL", that can (i) effectively utilize the unlabeled data, (ii) facilitate the "Human-in-the-loop" paradigm, and (iii) augment the model decisions in an interpretable way. In particular, we investigate the application of the SegXAL model for semantic segmentation in driving scene scenarios. The SegXAL model proposes the image regions that require labeling assistance from Oracle by dint of explainable AI (XAI) and uncertainty measures in a weakly-supervised manner. Specifically, we propose a novel Proximity-aware Explainable-AI (PAE) module and Entropy-based Uncertainty (EBU) module to get an Explainable Error Mask, which enables the machine teachers/human experts to provide intuitive reasoning behind the results and to solicit feedback to the AI system via an active learning strategy. Such a mechanism bridges the semantic gap between man and machine through collaborative intelligence, where humans and AI actively enhance each other's complementary strengths. A novel high-confidence sample selection technique based on the DICE similarity coefficient is also presented within the SegXAL framework. Extensive quantitative and qualitative analyses are carried out in the benchmarking Cityscape dataset. Results show the outperformance of our proposed SegXAL against other state-of-the-art models.

Title: Symmetric Encryption Scheme Based on Quasigroup Using Chained Mode of Operation

• Authors: Satish Kumar, Harshdeep Singh, Indivar Gupta, Ashok Ji Gupta
• Subjects: cs.CR, math.GR
• Abstract URL: https://arxiv.org/abs/2408.04490
• Pdf URL: https://arxiv.org/pdf/2408.04490
• Copy Paste: [[2408.04490]] Symmetric Encryption Scheme Based on Quasigroup Using Chained Mode of Operation(https://arxiv.org/abs/2408.04490)
• Keywords: security, attack
• Abstract: In this paper, we propose a novel construction for a symmetric encryption scheme, referred as SEBQ which is based on the structure of quasigroup. We utilize concepts of chaining like mode of operation and present a block cipher with in-built properties. We prove that SEBQ shows resistance against chosen plaintext attack (CPA) and by applying unbalanced Feistel transformation [19], it achieves security against chosen ciphertext attacks (CCA). Subsequently, we conduct an assessment of the randomness of the proposed scheme by running the NIST test suite and we analyze the impact of the initial vector, secret key and plaintext on ciphertext through an avalanche effect analysis. We also compare the results with existing schemes based on quasigroups [11,46]. Moreover, we analyze the computational complexity in terms of number of operations needed for encryption and decryption process.

Title: Towards Synergistic Deep Learning Models for Volumetric Cirrhotic Liver Segmentation in MRIs

• Authors: Vandan Gorade, Onkar Susladkar, Gorkem Durak, Elif Keles, Ertugrul Aktas, Timurhan Cebeci, Alpay Medetalibeyoglu, Daniela Ladner, Debesh Jha, Ulas Bagci
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04491
• Pdf URL: https://arxiv.org/pdf/2408.04491
• Copy Paste: [[2408.04491]] Towards Synergistic Deep Learning Models for Volumetric Cirrhotic Liver Segmentation in MRIs(https://arxiv.org/abs/2408.04491)
• Keywords: robust, segmentation
• Abstract: Liver cirrhosis, a leading cause of global mortality, requires precise segmentation of ROIs for effective disease monitoring and treatment planning. Existing segmentation models often fail to capture complex feature interactions and generalize across diverse datasets. To address these limitations, we propose a novel synergistic theory that leverages complementary latent spaces for enhanced feature interaction modeling. Our proposed architecture, nnSynergyNet3D integrates continuous and discrete latent spaces for 3D volumes and features auto-configured training. This approach captures both fine-grained and coarse features, enabling effective modeling of intricate feature interactions. We empirically validated nnSynergyNet3D on a private dataset of 628 high-resolution T1 abdominal MRI scans from 339 patients. Our model outperformed the baseline nnUNet3D by approximately 2%. Additionally, zero-shot testing on healthy liver CT scans from the public LiTS dataset demonstrated superior cross-modal generalization capabilities. These results highlight the potential of synergistic latent space models to improve segmentation accuracy and robustness, thereby enhancing clinical workflows by ensuring consistency across CT and MRI modalities.

Title: Saliency Detection in Educational Videos: Analyzing the Performance of Current Models, Identifying Limitations and Advancement Directions

• Authors: Evelyn Navarrete, Ralph Ewerth, Anett Hoppe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04515
• Pdf URL: https://arxiv.org/pdf/2408.04515
• Copy Paste: [[2408.04515]] Saliency Detection in Educational Videos: Analyzing the Performance of Current Models, Identifying Limitations and Advancement Directions(https://arxiv.org/abs/2408.04515)
• Keywords: segmentation
• Abstract: Identifying the regions of a learning resource that a learner pays attention to is crucial for assessing the material's impact and improving its design and related support systems. Saliency detection in videos addresses the automatic recognition of attention-drawing regions in single frames. In educational settings, the recognition of pertinent regions in a video's visual stream can enhance content accessibility and information retrieval tasks such as video segmentation, navigation, and summarization. Such advancements can pave the way for the development of advanced AI-assisted technologies that support learning with greater efficacy. However, this task becomes particularly challenging for educational videos due to the combination of unique characteristics such as text, voice, illustrations, animations, and more. To the best of our knowledge, there is currently no study that evaluates saliency detection approaches in educational videos. In this paper, we address this gap by evaluating four state-of-the-art saliency detection approaches for educational videos. We reproduce the original studies and explore the replication capabilities for general-purpose (non-educational) datasets. Then, we investigate the generalization capabilities of the models and evaluate their performance on educational videos. We conduct a comprehensive analysis to identify common failure scenarios and possible areas of improvement. Our experimental results show that educational videos remain a challenging context for generic video saliency detection models.

Title: Compromesso! Italian Many-Shot Jailbreaks Undermine the Safety of Large Language Models

• Authors: Fabio Pernisi, Dirk Hovy, Paul Röttger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04522
• Pdf URL: https://arxiv.org/pdf/2408.04522
• Copy Paste: [[2408.04522]] Compromesso! Italian Many-Shot Jailbreaks Undermine the Safety of Large Language Models(https://arxiv.org/abs/2408.04522)
• Keywords: large language model
• Abstract: As diverse linguistic communities and users adopt large language models (LLMs), assessing their safety across languages becomes critical. Despite ongoing efforts to make LLMs safe, they can still be made to behave unsafely with jailbreaking, a technique in which models are prompted to act outside their operational guidelines. Research on LLM safety and jailbreaking, however, has so far mostly focused on English, limiting our understanding of LLM safety in other languages. We contribute towards closing this gap by investigating the effectiveness of many-shot jailbreaking, where models are prompted with unsafe demonstrations to induce unsafe behaviour, in Italian. To enable our analysis, we create a new dataset of unsafe Italian question-answer pairs. With this dataset, we identify clear safety vulnerabilities in four families of open-weight LLMs. We find that the models exhibit unsafe behaviors even when prompted with few unsafe demonstrations, and -- more alarmingly -- that this tendency rapidly escalates with more demonstrations.

Title: Field Testing and Detection of Camera Interference for Autonomous Driving

• Authors: Ki Beom Park, Huy Kang Kim
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.04524
• Pdf URL: https://arxiv.org/pdf/2408.04524
• Copy Paste: [[2408.04524]] Field Testing and Detection of Camera Interference for Autonomous Driving(https://arxiv.org/abs/2408.04524)
• Keywords: attack
• Abstract: In recent advancements in connected and autonomous vehicles (CAVs), automotive ethernet has emerged as a critical technology for in-vehicle networks (IVNs), superseding traditional protocols like the CAN due to its superior bandwidth and data transmission capabilities. This study explores the detection of camera interference attacks (CIA) within an automotive ethernet-driven environment using a novel GRU-based IDS. Leveraging a sliding-window data preprocessing technique, our IDS effectively analyzes packet length sequences to differentiate between normal and anomalous data transmissions. Experimental evaluations conducted on a commercial car equipped with H.264 encoding and fragmentation unit-A (FU-A) demonstrated high detection accuracy, achieving an AUC of 0.9982 and a true positive rate of 0.99 with a window size of 255.

Title: AExGym: Benchmarks and Environments for Adaptive Experimentation

• Authors: Jimmy Wang, Ethan Che, Daniel R. Jiang, Hongseok Namkoong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04531
• Pdf URL: https://arxiv.org/pdf/2408.04531
• Copy Paste: [[2408.04531]] AExGym: Benchmarks and Environments for Adaptive Experimentation(https://arxiv.org/abs/2408.04531)
• Keywords: robust
• Abstract: Innovations across science and industry are evaluated using randomized trials (a.k.a. A/B tests). While simple and robust, such static designs are inefficient or infeasible for testing many hypotheses. Adaptive designs can greatly improve statistical power in theory, but they have seen limited adoption due to their fragility in practice. We present a benchmark for adaptive experimentation based on real-world datasets, highlighting prominent practical challenges to operationalizing adaptivity: non-stationarity, batched/delayed feedback, multiple outcomes and objectives, and external validity. Our benchmark aims to spur methodological development that puts practical performance (e.g., robustness) as a central concern, rather than mathematical guarantees on contrived instances. We release an open source library, AExGym, which is designed with modularity and extensibility in mind to allow experimentation practitioners to develop custom environments and algorithms.

Title: How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression

• Authors: Xingwu Chen, Lei Zhao, Difan Zou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04532
• Pdf URL: https://arxiv.org/pdf/2408.04532
• Copy Paste: [[2408.04532]] How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regression(https://arxiv.org/abs/2408.04532)
• Keywords: transformer
• Abstract: Despite the remarkable success of transformer-based models in various real-world tasks, their underlying mechanisms remain poorly understood. Recent studies have suggested that transformers can implement gradient descent as an in-context learner for linear regression problems and have developed various theoretical analyses accordingly. However, these works mostly focus on the expressive power of transformers by designing specific parameter constructions, lacking a comprehensive understanding of their inherent working mechanisms post-training. In this study, we consider a sparse linear regression problem and investigate how a trained multi-head transformer performs in-context learning. We experimentally discover that the utilization of multi-heads exhibits different patterns across layers: multiple heads are utilized and essential in the first layer, while usually only a single head is sufficient for subsequent layers. We provide a theoretical explanation for this observation: the first layer preprocesses the context data, and the following layers execute simple optimization steps based on the preprocessed context. Moreover, we demonstrate that such a preprocess-then-optimize algorithm can significantly outperform naive gradient descent and ridge regression algorithms. Further experimental results support our explanations. Our findings offer insights into the benefits of multi-head attention and contribute to understanding the more intricate mechanisms hidden within trained transformers.

Title: Bias-Aware Low-Rank Adaptation: Mitigating Catastrophic Inheritance of Large Language Models

• Authors: Yupeng Chang, Yi Chang, Yuan Wu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04556
• Pdf URL: https://arxiv.org/pdf/2408.04556
• Copy Paste: [[2408.04556]] Bias-Aware Low-Rank Adaptation: Mitigating Catastrophic Inheritance of Large Language Models(https://arxiv.org/abs/2408.04556)
• Keywords: robust, generative, large language model
• Abstract: Large language models (LLMs) have exhibited remarkable proficiency across a diverse array of natural language processing (NLP) tasks. However, adapting LLMs to downstream applications typically necessitates computationally intensive and memory-demanding fine-tuning procedures. To mitigate these burdens, parameter-efficient fine-tuning (PEFT) techniques have emerged as a promising approach to tailor LLMs with minimal computational overhead. While PEFT methods offer substantial advantages, they do not fully address the pervasive issue of bias propagation from pre-training data. In this work, we introduce Bias-Aware Low-Rank Adaptation (BA-LoRA), a novel PEFT method designed to counteract bias inheritance. BA-LoRA incorporates three distinct regularization terms: (1) consistency regularizer, (2) diversity regularizer, and (3) singular vector decomposition regularizer. These regularizers collectively aim to improve the generative models' consistency, diversity, and generalization capabilities during the fine-tuning process. Through extensive experiments on a variety of natural language understanding (NLU) and natural language generation (NLG) tasks, employing prominent LLMs such as LLaMA, Mistral, and Gemma, we demonstrate that BA-LoRA surpasses the performance of LoRA and its state-of-the-art variants. Moreover, our method effectively mitigates the deleterious effects of pre-training bias, leading to more reliable and robust model outputs. The code is available at this https URL.

Title: Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches

• Authors: Yongzhi Xu, Yonhon Ng, Yifu Wang, Inkyu Sa, Yunfei Duan, Yang Li, Pan Ji, Hongdong Li
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.04567
• Pdf URL: https://arxiv.org/pdf/2408.04567
• Copy Paste: [[2408.04567]] Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches(https://arxiv.org/abs/2408.04567)
• Keywords: diffusion
• Abstract: 3D Content Generation is at the heart of many computer graphics applications, including video gaming, film-making, virtual and augmented reality, etc. This paper proposes a novel deep-learning based approach for automatically generating interactive and playable 3D game scenes, all from the user's casual prompts such as a hand-drawn sketch. Sketch-based input offers a natural, and convenient way to convey the user's design intention in the content creation process. To circumvent the data-deficient challenge in learning (i.e. the lack of large training data of 3D scenes), our method leverages a pre-trained 2D denoising diffusion model to generate a 2D image of the scene as the conceptual guidance. In this process, we adopt the isometric projection mode to factor out unknown camera poses while obtaining the scene layout. From the generated isometric image, we use a pre-trained image understanding method to segment the image into meaningful parts, such as off-ground objects, trees, and buildings, and extract the 2D scene layout. These segments and layouts are subsequently fed into a procedural content generation (PCG) engine, such as a 3D video game engine like Unity or Unreal, to create the 3D scene. The resulting 3D scene can be seamlessly integrated into a game development environment and is readily playable. Extensive tests demonstrate that our method can efficiently generate high-quality and interactive 3D game scenes with layouts that closely follow the user's intention.

Title: Learning Fine-Grained Grounded Citations for Attributed Large Language Models

• Authors: Lei Huang, Xiaocheng Feng, Weitao Ma, Yuxuan Gu, Weihong Zhong, Xiachong Feng, Weijiang Yu, Weihua Peng, Duyu Tang, Dandan Tu, Bing Qin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04568
• Pdf URL: https://arxiv.org/pdf/2408.04568
• Copy Paste: [[2408.04568]] Learning Fine-Grained Grounded Citations for Attributed Large Language Models(https://arxiv.org/abs/2408.04568)
• Keywords: large language model
• Abstract: Despite the impressive performance on information-seeking tasks, large language models (LLMs) still struggle with hallucinations. Attributed LLMs, which augment generated text with in-line citations, have shown potential in mitigating hallucinations and improving verifiability. However, current approaches suffer from suboptimal citation quality due to their reliance on in-context learning. Furthermore, the practice of citing only coarse document identifiers makes it challenging for users to perform fine-grained verification. In this work, we introduce FRONT, a training framework designed to teach LLMs to generate Fine-Grained Grounded Citations. By grounding model outputs in fine-grained supporting quotes, these quotes guide the generation of grounded and consistent responses, not only improving citation quality but also facilitating fine-grained verification. Experiments on the ALCE benchmark demonstrate the efficacy of FRONT in generating superior grounded responses and highly supportive citations. With LLaMA-2-7B, the framework significantly outperforms all the baselines, achieving an average of 14.21% improvement in citation quality across all datasets, even surpassing ChatGPT.

Title: Mathematical Programming For Adaptive Experiments

• Authors: Ethan Che, Daniel R. Jiang, Hongseok Namkoong, Jimmy Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04570
• Pdf URL: https://arxiv.org/pdf/2408.04570
• Copy Paste: [[2408.04570]] Mathematical Programming For Adaptive Experiments(https://arxiv.org/abs/2408.04570)
• Keywords: robust
• Abstract: Adaptive experimentation can significantly improve statistical power, but standard algorithms overlook important practical issues including batched and delayed feedback, personalization, non-stationarity, multiple objectives, and constraints. To address these issues, the current algorithm design paradigm crafts tailored methods for each problem instance. Since it is infeasible to devise novel algorithms for every real-world instance, practitioners often have to resort to suboptimal approximations that do not address all of their challenges. Moving away from developing bespoke algorithms for each setting, we present a mathematical programming view of adaptive experimentation that can flexibly incorporate a wide range of objectives, constraints, and statistical procedures. By formulating a dynamic program in the batched limit, our modeling framework enables the use of scalable optimization methods (e.g., SGD and auto-differentiation) to solve for treatment allocations. We evaluate our framework on benchmarks modeled after practical challenges such as non-stationarity, personalization, multi-objectives, and constraints. Unlike bespoke algorithms such as modified variants of Thomson sampling, our mathematical programming approach provides remarkably robust performance across instances.

Title: SAM2-Adapter: Evaluating & Adapting Segment Anything 2 in Downstream Tasks: Camouflage, Shadow, Medical Image Segmentation, and More

• Authors: Tianrun Chen, Ankang Lu, Lanyun Zhu, Chaotao Ding, Chunan Yu, Deyi Ji, Zejian Li, Lingyun Sun, Papa Mao, Ying Zang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04579
• Pdf URL: https://arxiv.org/pdf/2408.04579
• Copy Paste: [[2408.04579]] SAM2-Adapter: Evaluating & Adapting Segment Anything 2 in Downstream Tasks: Camouflage, Shadow, Medical Image Segmentation, and More(https://arxiv.org/abs/2408.04579)
• Keywords: segmentation
• Abstract: The advent of large models, also known as foundation models, has significantly transformed the AI research landscape, with models like Segment Anything (SAM) achieving notable success in diverse image segmentation scenarios. Despite its advancements, SAM encountered limitations in handling some complex low-level segmentation tasks like camouflaged object and medical imaging. In response, in 2023, we introduced SAM-Adapter, which demonstrated improved performance on these challenging tasks. Now, with the release of Segment Anything 2 (SAM2), a successor with enhanced architecture and a larger training corpus, we reassess these challenges. This paper introduces SAM2-Adapter, the first adapter designed to overcome the persistent limitations observed in SAM2 and achieve new state-of-the-art (SOTA) results in specific downstream tasks including medical image segmentation, camouflaged (concealed) object detection, and shadow detection. SAM2-Adapter builds on the SAM-Adapter's strengths, offering enhanced generalizability and composability for diverse applications. We present extensive experimental results demonstrating SAM2-Adapter's effectiveness. We show the potential and encourage the research community to leverage the SAM2 model with our SAM2-Adapter for achieving superior segmentation outcomes. Code, pre-trained models, and data processing protocols are available at this http URL

Title: Quantum Key Distribution Networks -- Key Management: A Survey

• Authors: Emir Dervisevic, Amina Tankovic, Ehsan Fazel, Ramana Kompella, Peppino Fazio, Miroslav Voznak, Miralem Mehic
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.04580
• Pdf URL: https://arxiv.org/pdf/2408.04580
• Copy Paste: [[2408.04580]] Quantum Key Distribution Networks -- Key Management: A Survey(https://arxiv.org/abs/2408.04580)
• Keywords: secure
• Abstract: Secure communication makes the widespread use of telecommunication networks and services possible. With the constant progress of computing and mathematics, new cryptographic methods are being diligently developed. Quantum Key Distribution (QKD) is a promising technology that provides an Information-Theoretically Secure (ITS) solution to the secret-key agreement problem between two remote parties. QKD networks based on trusted repeaters are built to provide service to a larger number of parties at arbitrary distances. They function as an add-on technology to traditional networks, generating, managing, distributing, and supplying ITS cryptographic keys. Since key resources are limited, integrating QKD network services into critical infrastructures necessitates effective key management. As a result, this paper provides a comprehensive review of QKD network key management approaches. They are analyzed to facilitate the identification of potential strategies and accelerate the future development of QKD networks.

Title: Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustness

• Authors: Xiaojing Fan, Chunliang Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04585
• Pdf URL: https://arxiv.org/pdf/2408.04585
• Copy Paste: [[2408.04585]] Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustness(https://arxiv.org/abs/2408.04585)
• Keywords: attack, robust, transformer, large language model
• Abstract: With the increasing demand for practical applications of Large Language Models (LLMs), many attention-efficient models have been developed to balance performance and computational cost. However, the adversarial robustness of these models remains under-explored. In this work, we design a framework to investigate the trade-off between efficiency, performance, and adversarial robustness of LLMs by comparing three prominent models with varying levels of complexity and efficiency -- Transformer++, Gated Linear Attention (GLA) Transformer, and MatMul-Free LM -- utilizing the GLUE and AdvGLUE datasets. The AdvGLUE dataset extends the GLUE dataset with adversarial samples designed to challenge model robustness. Our results show that while the GLA Transformer and MatMul-Free LM achieve slightly lower accuracy on GLUE tasks, they demonstrate higher efficiency and either superior or comparative robustness on AdvGLUE tasks compared to Transformer++ across different attack levels. These findings highlight the potential of simplified architectures to achieve a compelling balance between efficiency, performance, and adversarial robustness, offering valuable insights for applications where resource constraints and resilience to adversarial attacks are critical.

Title: HiLo: A Learning Framework for Generalized Category Discovery Robust to Domain Shifts

• Authors: Hongjun Wang, Sagar Vaze, Kai Han
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04591
• Pdf URL: https://arxiv.org/pdf/2408.04591
• Copy Paste: [[2408.04591]] HiLo: A Learning Framework for Generalized Category Discovery Robust to Domain Shifts(https://arxiv.org/abs/2408.04591)
• Keywords: robust
• Abstract: Generalized Category Discovery (GCD) is a challenging task in which, given a partially labelled dataset, models must categorize all unlabelled instances, regardless of whether they come from labelled categories or from new ones. In this paper, we challenge a remaining assumption in this task: that all images share the same domain. Specifically, we introduce a new task and method to handle GCD when the unlabelled data also contains images from different domains to the labelled set. Our proposed `HiLo' networks extract High-level semantic and Low-level domain features, before minimizing the mutual information between the representations. Our intuition is that the clusterings based on domain information and semantic information should be independent. We further extend our method with a specialized domain augmentation tailored for the GCD task, as well as a curriculum learning approach. Finally, we construct a benchmark from corrupted fine-grained datasets as well as a large-scale evaluation on DomainNet with real-world domain shifts, reimplementing a number of GCD baselines in this setting. We demonstrate that HiLo outperforms SoTA category discovery models by a large margin on all evaluations.

Title: SAM 2 in Robotic Surgery: An Empirical Evaluation for Robustness and Generalization in Surgical Video Segmentation

• Authors: Jieming Yu, An Wang, Wenzhen Dong, Mengya Xu, Mobarakol Islam, Jie Wang, Long Bai, Hongliang Ren
• Subjects: cs.CV, cs.RO, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.04593
• Pdf URL: https://arxiv.org/pdf/2408.04593
• Copy Paste: [[2408.04593]] SAM 2 in Robotic Surgery: An Empirical Evaluation for Robustness and Generalization in Surgical Video Segmentation(https://arxiv.org/abs/2408.04593)
• Keywords: robust, segmentation
• Abstract: The recent Segment Anything Model (SAM) 2 has demonstrated remarkable foundational competence in semantic segmentation, with its memory mechanism and mask decoder further addressing challenges in video tracking and object occlusion, thereby achieving superior results in interactive segmentation for both images and videos. Building upon our previous empirical studies, we further explore the zero-shot segmentation performance of SAM 2 in robot-assisted surgery based on prompts, alongside its robustness against real-world corruption. For static images, we employ two forms of prompts: 1-point and bounding box, while for video sequences, the 1-point prompt is applied to the initial frame. Through extensive experimentation on the MICCAI EndoVis 2017 and EndoVis 2018 benchmarks, SAM 2, when utilizing bounding box prompts, outperforms state-of-the-art (SOTA) methods in comparative evaluations. The results with point prompts also exhibit a substantial enhancement over SAM's capabilities, nearing or even surpassing existing unprompted SOTA methodologies. Besides, SAM 2 demonstrates improved inference speed and less performance degradation against various image corruption. Although slightly unsatisfactory results remain in specific edges or regions, SAM 2's robust adaptability to 1-point prompts underscores its potential for downstream surgical tasks with limited prompt requirements.

Title: Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models

• Authors: Qirui Jiao, Daoyuan Chen, Yilun Huang, Yaliang Li, Ying Shen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04594
• Pdf URL: https://arxiv.org/pdf/2408.04594
• Copy Paste: [[2408.04594]] Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models(https://arxiv.org/abs/2408.04594)
• Keywords: robust, diffusion, large language model
• Abstract: High-performance Multimodal Large Language Models (MLLMs) rely heavily on data quality. This study introduces a novel dataset named Img-Diff, designed to enhance fine-grained image recognition in MLLMs by leveraging insights from contrastive learning and image difference captioning. By analyzing object differences between similar images, we challenge models to identify both matching and distinct components. We utilize the Stable-Diffusion-XL model and advanced image editing techniques to create pairs of similar images that highlight object replacements. Our methodology includes a Difference Area Generator for object differences identifying, followed by a Difference Captions Generator for detailed difference descriptions. The result is a relatively small but high-quality dataset of "object replacement" samples. We use the the proposed dataset to fine-tune state-of-the-art (SOTA) MLLMs such as MGM-7B, yielding comprehensive improvements of performance scores over SOTA models that trained with larger-scale datasets, in numerous image difference and Visual Question Answering tasks. For instance, our trained models notably surpass the SOTA models GPT-4V and Gemini on the MMVP benchmark. Besides, we investigate alternative methods for generating image difference data through "object removal" and conduct thorough evaluation to confirm the dataset's diversity, quality, and robustness, presenting several insights on synthesis of such contrastive dataset. To encourage further research and advance the field of multimodal data synthesis and enhancement of MLLMs' fundamental capabilities for image understanding, we release our codes and dataset at this https URL.

Title: Improving Network Interpretability via Explanation Consistency Evaluation

• Authors: Hefeng Wu, Hao Jiang, Keze Wang, Ziyi Tang, Xianghuan He, Liang Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04600
• Pdf URL: https://arxiv.org/pdf/2408.04600
• Copy Paste: [[2408.04600]] Improving Network Interpretability via Explanation Consistency Evaluation(https://arxiv.org/abs/2408.04600)
• Keywords: attack, robust, interpretability
• Abstract: While deep neural networks have achieved remarkable performance, they tend to lack transparency in prediction. The pursuit of greater interpretability in neural networks often results in a degradation of their original performance. Some works strive to improve both interpretability and performance, but they primarily depend on meticulously imposed conditions. In this paper, we propose a simple yet effective framework that acquires more explainable activation heatmaps and simultaneously increase the model performance, without the need for any extra supervision. Specifically, our concise framework introduces a new metric, i.e., explanation consistency, to reweight the training samples adaptively in model learning. The explanation consistency metric is utilized to measure the similarity between the model's visual explanations of the original samples and those of semantic-preserved adversarial samples, whose background regions are perturbed by using image adversarial attack techniques. Our framework then promotes the model learning by paying closer attention to those training samples with a high difference in explanations (i.e., low explanation consistency), for which the current model cannot provide robust interpretations. Comprehensive experimental results on various benchmarks demonstrate the superiority of our framework in multiple aspects, including higher recognition accuracy, greater data debiasing capability, stronger network robustness, and more precise localization ability on both regular networks and interpretable networks. We also provide extensive ablation studies and qualitative analyses to unveil the detailed contribution of each component.

Title: Towards High-resolution 3D Anomaly Detection via Group-Level Feature Contrastive Learning

• Authors: Hongze Zhu, Guoyang Xie, Chengbin Hou, Tao Dai, Can Gao, Jinbao Wang, Linlin Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04604
• Pdf URL: https://arxiv.org/pdf/2408.04604
• Copy Paste: [[2408.04604]] Towards High-resolution 3D Anomaly Detection via Group-Level Feature Contrastive Learning(https://arxiv.org/abs/2408.04604)
• Keywords: transformer
• Abstract: High-resolution point clouds~(HRPCD) anomaly detection~(AD) plays a critical role in precision machining and high-end equipment manufacturing. Despite considerable 3D-AD methods that have been proposed recently, they still cannot meet the requirements of the HRPCD-AD task. There are several challenges: i) It is difficult to directly capture HRPCD information due to large amounts of points at the sample level; ii) The advanced transformer-based methods usually obtain anisotropic features, leading to degradation of the representation; iii) The proportion of abnormal areas is very small, which makes it difficult to characterize. To address these challenges, we propose a novel group-level feature-based network, called Group3AD, which has a significantly efficient representation ability. First, we design an Intercluster Uniformity Network~(IUN) to present the mapping of different groups in the feature space as several clusters, and obtain a more uniform distribution between clusters representing different parts of the point clouds in the feature space. Then, an Intracluster Alignment Network~(IAN) is designed to encourage groups within the cluster to be distributed tightly in the feature space. In addition, we propose an Adaptive Group-Center Selection~(AGCS) based on geometric information to improve the pixel density of potential anomalous regions during inference. The experimental results verify the effectiveness of our proposed Group3AD, which surpasses Reg3D-AD by the margin of 5\% in terms of object-level AUROC on Real3D-AD. We provide the code and supplementary information on our website: this https URL.

Title: Enhanced Prototypical Part Network (EPPNet) For Explainable Image Classification Via Prototypes

• Authors: Bhushan Atote, Victor Sanchez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04606
• Pdf URL: https://arxiv.org/pdf/2408.04606
• Copy Paste: [[2408.04606]] Enhanced Prototypical Part Network (EPPNet) For Explainable Image Classification Via Prototypes(https://arxiv.org/abs/2408.04606)
• Keywords: explainability
• Abstract: Explainable Artificial Intelligence (xAI) has the potential to enhance the transparency and trust of AI-based systems. Although accurate predictions can be made using Deep Neural Networks (DNNs), the process used to arrive at such predictions is usually hard to explain. In terms of perceptibly human-friendly representations, such as word phrases in text or super-pixels in images, prototype-based explanations can justify a model's decision. In this work, we introduce a DNN architecture for image classification, the Enhanced Prototypical Part Network (EPPNet), which achieves strong performance while discovering relevant prototypes that can be used to explain the classification results. This is achieved by introducing a novel cluster loss that helps to discover more relevant human-understandable prototypes. We also introduce a faithfulness score to evaluate the explainability of the results based on the discovered prototypes. Our score not only accounts for the relevance of the learned prototypes but also the performance of a model. Our evaluations on the CUB-200-2011 dataset show that the EPPNet outperforms state-of-the-art xAI-based methods, in terms of both classification accuracy and explainability

Title: Better Alignment with Instruction Back-and-Forth Translation

• Authors: Thao Nguyen, Jeffrey Li, Sewoong Oh, Ludwig Schmidt, Jason Weston, Luke Zettlemoyer, Xian Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04614
• Pdf URL: https://arxiv.org/pdf/2408.04614
• Copy Paste: [[2408.04614]] Better Alignment with Instruction Back-and-Forth Translation(https://arxiv.org/abs/2408.04614)
• Keywords: large language model
• Abstract: We propose a new method, instruction back-and-forth translation, to construct high-quality synthetic data grounded in world knowledge for aligning large language models (LLMs). Given documents from a web corpus, we generate and curate synthetic instructions using the backtranslation approach proposed by Li et al.(2023a), and rewrite the responses to improve their quality further based on the initial documents. Fine-tuning with the resulting (backtranslated instruction, rewritten response) pairs yields higher win rates on AlpacaEval than using other common instruction datasets such as Humpback, ShareGPT, Open Orca, Alpaca-GPT4 and Self-instruct. We also demonstrate that rewriting the responses with an LLM outperforms direct distillation, and the two generated text distributions exhibit significant distinction in embedding space. Further analysis shows that our backtranslated instructions are of higher quality than other sources of synthetic instructions, while our responses are more diverse and complex than those obtained from distillation. Overall we find that instruction back-and-forth translation combines the best of both worlds -- making use of the information diversity and quantity found on the web, while ensuring the quality of the responses which is necessary for effective alignment.

Title: Transformer Explainer: Interactive Learning of Text-Generative Models

• Authors: Aeree Cho, Grace C. Kim, Alexander Karpekov, Alec Helbling, Zijie J. Wang, Seongmin Lee, Benjamin Hoover, Duen Horng Chau
• Subjects: cs.LG, cs.AI, cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.04619
• Pdf URL: https://arxiv.org/pdf/2408.04619
• Copy Paste: [[2408.04619]] Transformer Explainer: Interactive Learning of Text-Generative Models(https://arxiv.org/abs/2408.04619)
• Keywords: transformer, generative
• Abstract: Transformers have revolutionized machine learning, yet their inner workings remain opaque to many. We present Transformer Explainer, an interactive visualization tool designed for non-experts to learn about Transformers through the GPT-2 model. Our tool helps users understand complex Transformer concepts by integrating a model overview and enabling smooth transitions across abstraction levels of mathematical operations and model structures. It runs a live GPT-2 instance locally in the user's browser, empowering users to experiment with their own input and observe in real-time how the internal components and parameters of the Transformer work together to predict the next tokens. Our tool requires no installation or special hardware, broadening the public's education access to modern generative AI techniques. Our open-sourced tool is available at this https URL. A video demo is available at this https URL.

Title: Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics

• Authors: Ruining Li, Chuanxia Zheng, Christian Rupprecht, Andrea Vedaldi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04631
• Pdf URL: https://arxiv.org/pdf/2408.04631
• Copy Paste: [[2408.04631]] Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics(https://arxiv.org/abs/2408.04631)
• Keywords: diffusion, generative
• Abstract: We present Puppet-Master, an interactive video generative model that can serve as a motion prior for part-level dynamics. At test time, given a single image and a sparse set of motion trajectories (i.e., drags), Puppet-Master can synthesize a video depicting realistic part-level motion faithful to the given drag interactions. This is achieved by fine-tuning a large-scale pre-trained video diffusion model, for which we propose a new conditioning architecture to inject the dragging control effectively. More importantly, we introduce the all-to-first attention mechanism, a drop-in replacement for the widely adopted spatial attention modules, which significantly improves generation quality by addressing the appearance and background issues in existing models. Unlike other motion-conditioned video generators that are trained on in-the-wild videos and mostly move an entire object, Puppet-Master is learned from Objaverse-Animation-HQ, a new dataset of curated part-level motion clips. We propose a strategy to automatically filter out sub-optimal animations and augment the synthetic renderings with meaningful motion trajectories. Puppet-Master generalizes well to real images across various categories and outperforms existing methods in a zero-shot manner on a real-world benchmark. See our project page for more results: this http URL.