secure

Title: SoK: Authenticated Prefix Relations -- A Unified Perspective On Relative Time-Stamping and Append-Only Logs. (arXiv:2308.13836v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.13836
• Code URL: null
• Copy Paste: [[2308.13836]] SoK: Authenticated Prefix Relations -- A Unified Perspective On Relative Time-Stamping and Append-Only Logs(http://arxiv.org/abs/2308.13836)
• Summary:

  Secure relative timestamping and secure append-only logs are two historically mostly independent lines of research, which we show to be sides of the same coin -- the authentication of prefix relations. From this more general viewpoint, we derive several complexity criteria not yet considered in previous literature. We define transitive prefix authentication graphs, a graph class that captures all hash-based timestamping and log designs we know of. We survey existing schemes by expressing them as transitive prefix authentication graphs, which yields more compact definitions and more complete evaluations than in the existing literature.

security

Title: Exploring Human Crowd Patterns and Categorization in Video Footage for Enhanced Security and Surveillance using Computer Vision and Machine Learning. (arXiv:2308.13910v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13910
• Code URL: null
• Copy Paste: [[2308.13910]] Exploring Human Crowd Patterns and Categorization in Video Footage for Enhanced Security and Surveillance using Computer Vision and Machine Learning(http://arxiv.org/abs/2308.13910)
• Summary:

  Computer vision and machine learning have brought revolutionary shifts in perception for researchers, scientists, and the general populace. Once thought to be unattainable, these technologies have achieved the seemingly impossible. Their exceptional applications in diverse fields like security, agriculture, and education are a testament to their impact. However, the full potential of computer vision remains untapped. This paper explores computer vision's potential in security and surveillance, presenting a novel approach to track motion in videos. By categorizing motion into Arcs, Lanes, Converging/Diverging, and Random/Block motions using Motion Information Images and Blockwise dominant motion data, the paper examines different optical flow techniques, CNN models, and machine learning models. Successfully achieving its objectives with promising accuracy, the results can train anomaly-detection models, provide behavioral insights based on motion, and enhance scene comprehension.

Title: Implementing Snort Intrusion Prevention System (IPS) for Network Forensic Analysis. (arXiv:2308.13589v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.13589
• Code URL: null
• Copy Paste: [[2308.13589]] Implementing Snort Intrusion Prevention System (IPS) for Network Forensic Analysis(http://arxiv.org/abs/2308.13589)
• Summary:

  The security trade confidentiality, integrity and availability are the main pillar of the information systems as every organization emphasize of the security. From last few decades, digital data is the main asset for every digital or non-digital organization. The proliferation of easily accessible attack software on the internet has lowered the barrier for individuals without hacking skills to engage in malicious activities. An Industrial organization operates a server that (Confluence) serves as a learning platform for newly hired employees or Management training officers, thereby making it vulnerable to potential attacks using readily available internet-based software. To mitigate this risk, it is essential to implement a security system capable of detecting and preventing attacks, as well as conducting investigations. This research project aims to develop a comprehensive security system that can detect attack attempts, initiate preventive measures, and carry out investigations by analyzing attack logs. The study adopted a survey methodology and spanned a period of four months, from March 1, 2023, to June 31, 2023. The outcome of this research is a robust security system that effectively identifies attack attempts, blocks the attacker's IP address, and employs network forensic techniques for investigation purposes. The findings indicate that deploying Snort in IPS mode on PfSense enables the detection of attacks targeting e-learning servers, triggering automatic preventive measures such as IP address blocking. The alerts generated by Snort facilitate investigative actions through network forensics, allowing for accurate reporting on the detrimental effects of the attacks.

privacy

Title: Prior-guided Source-free Domain Adaptation for Human Pose Estimation. (arXiv:2308.13954v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13954
• Code URL: null
• Copy Paste: [[2308.13954]] Prior-guided Source-free Domain Adaptation for Human Pose Estimation(http://arxiv.org/abs/2308.13954)
• Summary:

  Domain adaptation methods for 2D human pose estimation typically require continuous access to the source data during adaptation, which can be challenging due to privacy, memory, or computational constraints. To address this limitation, we focus on the task of source-free domain adaptation for pose estimation, where a source model must adapt to a new target domain using only unlabeled target data. Although recent advances have introduced source-free methods for classification tasks, extending them to the regression task of pose estimation is non-trivial. In this paper, we present Prior-guided Self-training (POST), a pseudo-labeling approach that builds on the popular Mean Teacher framework to compensate for the distribution shift. POST leverages prediction-level and feature-level consistency between a student and teacher model against certain image transformations. In the absence of source data, POST utilizes a human pose prior that regularizes the adaptation process by directing the model to generate more accurate and anatomically plausible pose pseudo-labels. Despite being simple and intuitive, our framework can deliver significant performance gains compared to applying the source model directly to the target data, as demonstrated in our extensive experiments and ablation studies. In fact, our approach achieves comparable performance to recent state-of-the-art methods that use source data for adaptation.

Title: FAM: fast adaptive meta-learning. (arXiv:2308.13970v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13970
• Code URL: null
• Copy Paste: [[2308.13970]] FAM: fast adaptive meta-learning(http://arxiv.org/abs/2308.13970)
• Summary:

  In this work, we propose a fast adaptive federated meta-learning (FAM) framework for collaboratively learning a single global model, which can then be personalized locally on individual clients. Federated learning enables multiple clients to collaborate to train a model without sharing data. Clients with insufficient data or data diversity participate in federated learning to learn a model with superior performance. Nonetheless, learning suffers when data distributions diverge. There is a need to learn a global model that can be adapted using client's specific information to create personalised models on clients is required. MRI data suffers from this problem, wherein, one, due to data acquisition challenges, local data at a site is sufficient for training an accurate model and two, there is a restriction of data sharing due to privacy concerns and three, there is a need for personalization of a learnt shared global model on account of domain shift across client sites. The global model is sparse and captures the common features in the MRI. This skeleton network is grown on each client to train a personalised model by learning additional client-specific parameters from local data. Experimental results show that the personalization process at each client quickly converges using a limited number of epochs. The personalized client models outperformed the locally trained models, demonstrating the efficacy of the FAM mechanism. Additionally, the sparse parameter set to be communicated during federated learning drastically reduced communication overhead, which makes the scheme viable for networks with limited resources.

Title: Building Trust in Conversational AI: A Comprehensive Review and Solution Architecture for Explainable, Privacy-Aware Systems using LLMs and Knowledge Graph. (arXiv:2308.13534v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.13534
• Code URL: null
• Copy Paste: [[2308.13534]] Building Trust in Conversational AI: A Comprehensive Review and Solution Architecture for Explainable, Privacy-Aware Systems using LLMs and Knowledge Graph(http://arxiv.org/abs/2308.13534)
• Summary:

  Conversational AI systems have emerged as key enablers of human-like interactions across diverse sectors. Nevertheless, the balance between linguistic nuance and factual accuracy has proven elusive. In this paper, we first introduce LLMXplorer, a comprehensive tool that provides an in-depth review of over 150 Large Language Models (LLMs), elucidating their myriad implications ranging from social and ethical to regulatory, as well as their applicability across industries. Building on this foundation, we propose a novel functional architecture that seamlessly integrates the structured dynamics of Knowledge Graphs with the linguistic capabilities of LLMs. Validated using real-world AI news data, our architecture adeptly blends linguistic sophistication with factual rigour and further strengthens data security through Role-Based Access Control. This research provides insights into the evolving landscape of conversational AI, emphasizing the imperative for systems that are efficient, transparent, and trustworthy.

Title: SOK: Privacy Definitions and Classical Mechanisms in the Local Setting. (arXiv:2308.13946v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.13946
• Code URL: null
• Copy Paste: [[2308.13946]] SOK: Privacy Definitions and Classical Mechanisms in the Local Setting(http://arxiv.org/abs/2308.13946)
• Summary:

  This paper delves into the intricate landscape of privacy notions, specifically honed in on the local setting. Central to our discussion is the juxtaposition of point-wise protection and average-case protection, offering a comparative analysis that highlights the strengths and trade-offs inherent to each approach. Beyond this, we delineate between context-aware and context-free notions, examining the implications of both in diverse application scenarios. The study further differentiates between the interactive and non-interactive models, illuminating the complexities and nuances each model introduces. By systematically navigating these core themes, our goal is to provide a cohesive framework that aids researchers and practitioners in discerning the most suitable privacy notions for their specific requirements in the local setting.

Title: Machine Unlearning for Causal Inference. (arXiv:2308.13559v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13559
• Code URL: null
• Copy Paste: [[2308.13559]] Machine Unlearning for Causal Inference(http://arxiv.org/abs/2308.13559)
• Summary:

  Machine learning models play a vital role in making predictions and deriving insights from data and are being increasingly used for causal inference. To preserve user privacy, it is important to enable the model to forget some of its learning/captured information about a given user (machine unlearning). This paper introduces the concept of machine unlearning for causal inference, particularly propensity score matching and treatment effect estimation, which aims to refine and improve the performance of machine learning models for causal analysis given the above unlearning requirements. The paper presents a methodology for machine unlearning using a neural network-based propensity score model. The dataset used in the study is the Lalonde dataset, a widely used dataset for evaluating the effectiveness i.e. the treatment effect of job training programs. The methodology involves training an initial propensity score model on the original dataset and then creating forget sets by selectively removing instances, as well as matched instance pairs. based on propensity score matching. These forget sets are used to evaluate the retrained model, allowing for the elimination of unwanted associations. The actual retraining of the model is performed using the retain set. The experimental results demonstrate the effectiveness of the machine unlearning approach. The distribution and histogram analysis of propensity scores before and after unlearning provide insights into the impact of the unlearning process on the data. This study represents the first attempt to apply machine unlearning techniques to causal inference.

protect

defense

attack

Title: LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors. (arXiv:2308.13904v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.13904
• Code URL: https://github.com/meng-wenlong/lmsanitator
• Copy Paste: [[2308.13904]] LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors(http://arxiv.org/abs/2308.13904)
• Summary:

  Prompt-tuning has emerged as an attractive paradigm for deploying large-scale language models due to its strong downstream task performance and efficient multitask serving ability. Despite its wide adoption, we empirically show that prompt-tuning is vulnerable to downstream task-agnostic backdoors, which reside in the pretrained models and can affect arbitrary downstream tasks. The state-of-the-art backdoor detection approaches cannot defend against task-agnostic backdoors since they hardly converge in reversing the backdoor triggers. To address this issue, we propose LMSanitator, a novel approach for detecting and removing task-agnostic backdoors on Transformer models. Instead of directly inversing the triggers, LMSanitator aims to inverse the predefined attack vectors (pretrained models' output when the input is embedded with triggers) of the task-agnostic backdoors, which achieves much better convergence performance and backdoor detection accuracy. LMSanitator further leverages prompt-tuning's property of freezing the pretrained model to perform accurate and fast output monitoring and input purging during the inference phase. Extensive experiments on multiple language models and NLP tasks illustrate the effectiveness of LMSanitator. For instance, LMSanitator achieves 92.8% backdoor detection accuracy on 960 models and decreases the attack success rate to less than 1% in most scenarios.

Title: Mitigation Techniques for Cyber Attacks: A Systematic Mapping Study. (arXiv:2308.13587v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.13587
• Code URL: null
• Copy Paste: [[2308.13587]] Mitigation Techniques for Cyber Attacks: A Systematic Mapping Study(http://arxiv.org/abs/2308.13587)
• Summary:

  In the wake of the arrival of digital media, the Internet, the web, and online social media, a flood of new cyber security research questions have emerged. There is a lot of money being lost around the world because of cyber-attacks. As a result, cyber security has emerged as one of the world's most complex and pressing issues. Cyber security experts from both industry and academia institutions are now analyzing current cyber-attacks occurring around the world and developing various strategies to defend systems from possible cyber-threats and attacks. This paper examines recent cyber security attacks as well as the financial losses incurred as a result of the growing number of cyber-attacks. Our findings indicate that the majority of the research chosen for this study focused solely on a small number of widespread security flaws, such as malware, phishing, and denial-of-service attacks. A total of over 50 major studies that have been published in reputable academic journals and conferences have been chosen for additional examination. A taxonomy of cyber-attacks elements that is based on the context of use in various environments has also been suggested, in addition to a review of the most recent studies on countermeasures for cyber-attacks being the state of the art. Lastly, the research gaps in terms of open issues have been described in order to offer potential future directions for the researchers working in the field of cyber security.

Title: Active learning for fast and slow modeling attacks on Arbiter PUFs. (arXiv:2308.13645v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.13645
• Code URL: null
• Copy Paste: [[2308.13645]] Active learning for fast and slow modeling attacks on Arbiter PUFs(http://arxiv.org/abs/2308.13645)
• Summary:

  Modeling attacks, in which an adversary uses machine learning techniques to model a hardware-based Physically Unclonable Function (PUF) pose a great threat to the viability of these hardware security primitives. In most modeling attacks, a random subset of challenge-response-pairs (CRPs) are used as the labeled data for the machine learning algorithm. Here, for the arbiter-PUF, a delay based PUF which may be viewed as a linear threshold function with random weights (due to manufacturing imperfections), we investigate the role of active learning in Support Vector Machine (SVM) learning. We focus on challenge selection to help SVM algorithm learn ``fast'' and learn ``slow''. Our methods construct challenges rather than relying on a sample pool of challenges as in prior work. Using active learning to learn ``fast'' (less CRPs revealed, higher accuracies) may help manufacturers learn the manufactured PUFs more efficiently, or may form a more powerful attack when the attacker may query the PUF for CRPs at will. Using active learning to select challenges from which learning is ``slow'' (low accuracy despite a large number of revealed CRPs) may provide a basis for slowing down attackers who are limited to overhearing CRPs.

robust

Title: Fusion of Infrared and Visible Images based on Spatial-Channel Attentional Mechanism. (arXiv:2308.13672v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13672
• Code URL: null
• Copy Paste: [[2308.13672]] Fusion of Infrared and Visible Images based on Spatial-Channel Attentional Mechanism(http://arxiv.org/abs/2308.13672)
• Summary:

  In the study, we present AMFusionNet, an innovative approach to infrared and visible image fusion (IVIF), harnessing the power of multiple kernel sizes and attention mechanisms. By assimilating thermal details from infrared images with texture features from visible sources, our method produces images enriched with comprehensive information. Distinct from prevailing deep learning methodologies, our model encompasses a fusion mechanism powered by multiple convolutional kernels, facilitating the robust capture of a wide feature spectrum. Notably, we incorporate parallel attention mechanisms to emphasize and retain pivotal target details in the resultant images. Moreover, the integration of the multi-scale structural similarity (MS-SSIM) loss function refines network training, optimizing the model for IVIF task. Experimental results demonstrate that our method outperforms state-of-the-art algorithms in terms of quality and quantity. The performance metrics on publicly available datasets also show significant improvement

Title: Textureless Deformable Surface Reconstruction with Invisible Markers. (arXiv:2308.13678v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13678
• Code URL: null
• Copy Paste: [[2308.13678]] Textureless Deformable Surface Reconstruction with Invisible Markers(http://arxiv.org/abs/2308.13678)
• Summary:

  Reconstructing and tracking deformable surface with little or no texture has posed long-standing challenges. Fundamentally, the challenges stem from textureless surfaces lacking features for establishing cross-image correspondences. In this work, we present a novel type of markers to proactively enrich the object's surface features, and thereby ease the 3D surface reconstruction and correspondence tracking. Our markers are made of fluorescent dyes, visible only under the ultraviolet (UV) light and invisible under regular lighting condition. Leveraging the markers, we design a multi-camera system that captures surface deformation under the UV light and the visible light in a time multiplexing fashion. Under the UV light, markers on the object emerge to enrich its surface texture, allowing high-quality 3D shape reconstruction and tracking. Under the visible light, markers become invisible, allowing us to capture the object's original untouched appearance. We perform experiments on various challenging scenes, including hand gestures, facial expressions, waving cloth, and hand-object interaction. In all these cases, we demonstrate that our system is able to produce robust, high-quality 3D reconstruction and tracking.

Title: SOGDet: Semantic-Occupancy Guided Multi-view 3D Object Detection. (arXiv:2308.13794v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13794
• Code URL: null
• Copy Paste: [[2308.13794]] SOGDet: Semantic-Occupancy Guided Multi-view 3D Object Detection(http://arxiv.org/abs/2308.13794)
• Summary:

  In the field of autonomous driving, accurate and comprehensive perception of the 3D environment is crucial. Bird's Eye View (BEV) based methods have emerged as a promising solution for 3D object detection using multi-view images as input. However, existing 3D object detection methods often ignore the physical context in the environment, such as sidewalk and vegetation, resulting in sub-optimal performance. In this paper, we propose a novel approach called SOGDet (Semantic-Occupancy Guided Multi-view 3D Object Detection), that leverages a 3D semantic-occupancy branch to improve the accuracy of 3D object detection. In particular, the physical context modeled by semantic occupancy helps the detector to perceive the scenes in a more holistic view. Our SOGDet is flexible to use and can be seamlessly integrated with most existing BEV-based methods. To evaluate its effectiveness, we apply this approach to several state-of-the-art baselines and conduct extensive experiments on the exclusive nuScenes dataset. Our results show that SOGDet consistently enhance the performance of three baseline methods in terms of nuScenes Detection Score (NDS) and mean Average Precision (mAP). This indicates that the combination of 3D object detection and 3D semantic occupancy leads to a more comprehensive perception of the 3D environment, thereby aiding build more robust autonomous driving systems. The codes are available at: https://github.com/zhouqiu/SOGDet.

Title: Late Stopping: Avoiding Confidently Learning from Mislabeled Examples. (arXiv:2308.13862v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13862
• Code URL: null
• Copy Paste: [[2308.13862]] Late Stopping: Avoiding Confidently Learning from Mislabeled Examples(http://arxiv.org/abs/2308.13862)
• Summary:

  Sample selection is a prevalent method in learning with noisy labels, where small-loss data are typically considered as correctly labeled data. However, this method may not effectively identify clean hard examples with large losses, which are critical for achieving the model's close-to-optimal generalization performance. In this paper, we propose a new framework, Late Stopping, which leverages the intrinsic robust learning ability of DNNs through a prolonged training process. Specifically, Late Stopping gradually shrinks the noisy dataset by removing high-probability mislabeled examples while retaining the majority of clean hard examples in the training set throughout the learning process. We empirically observe that mislabeled and clean examples exhibit differences in the number of epochs required for them to be consistently and correctly classified, and thus high-probability mislabeled examples can be removed. Experimental results on benchmark-simulated and real-world noisy datasets demonstrate that the proposed method outperforms state-of-the-art counterparts.

Title: Image Coding for Machines with Object Region Learning. (arXiv:2308.13984v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13984
• Code URL: null
• Copy Paste: [[2308.13984]] Image Coding for Machines with Object Region Learning(http://arxiv.org/abs/2308.13984)
• Summary:

  Compression technology is essential for efficient image transmission and storage. With the rapid advances in deep learning, images are beginning to be used for image recognition as well as for human vision. For this reason, research has been conducted on image coding for image recognition, and this field is called Image Coding for Machines (ICM). There are two main approaches in ICM: the ROI-based approach and the task-loss-based approach. The former approach has the problem of requiring an ROI-map as input in addition to the input image. The latter approach has the problems of difficulty in learning the task-loss, and lack of robustness because the specific image recognition model is used to compute the loss function. To solve these problems, we propose an image compression model that learns object regions. Our model does not require additional information as input, such as an ROI-map, and does not use task-loss. Therefore, it is possible to compress images for various image recognition models. In the experiments, we demonstrate the versatility of the proposed method by using three different image recognition models and three different datasets. In addition, we verify the effectiveness of our model by comparing it with previous methods.

Title: LDL: Line Distance Functions for Panoramic Localization. (arXiv:2308.13989v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13989
• Code URL: null
• Copy Paste: [[2308.13989]] LDL: Line Distance Functions for Panoramic Localization(http://arxiv.org/abs/2308.13989)
• Summary:

  We introduce LDL, a fast and robust algorithm that localizes a panorama to a 3D map using line segments. LDL focuses on the sparse structural information of lines in the scene, which is robust to illumination changes and can potentially enable efficient computation. While previous line-based localization approaches tend to sacrifice accuracy or computation time, our method effectively observes the holistic distribution of lines within panoramic images and 3D maps. Specifically, LDL matches the distribution of lines with 2D and 3D line distance functions, which are further decomposed along principal directions of lines to increase the expressiveness. The distance functions provide coarse pose estimates by comparing the distributional information, where the poses are further optimized using conventional local feature matching. As our pipeline solely leverages line geometry and local features, it does not require costly additional training of line-specific features or correspondence matching. Nevertheless, our method demonstrates robust performance on challenging scenarios including object layout changes, illumination shifts, and large-scale scenes, while exhibiting fast pose search terminating within a matter of milliseconds. We thus expect our method to serve as a practical solution for line-based localization, and complement the well-established point-based paradigm. The code for LDL is available through the following link: https://github.com/82magnolia/panoramic-localization.

Title: An Ensemble Approach to Personalized Real Time Predictive Writing for Experts. (arXiv:2308.13576v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.13576
• Code URL: null
• Copy Paste: [[2308.13576]] An Ensemble Approach to Personalized Real Time Predictive Writing for Experts(http://arxiv.org/abs/2308.13576)
• Summary:

  Completing a sentence, phrase or word after typing few words / characters is very helpful for Intuit financial experts, while taking notes or having a live chat with users, since they need to write complex financial concepts more efficiently and accurately many times in a day. In this paper, we tie together different approaches like large language models, traditional Markov Models and char level models to create an end-to-end system to provide personalised sentence/word auto-complete suggestions to experts, under strict latency constraints. Proposed system can auto-complete sentences, phrases or words while writing with personalisation and can be trained with very less data and resources with good efficiency. Our proposed system is not only efficient and personalized but also robust as it leverages multiple machine learning techniques along with transfer learning approach to fine tune large language model with Intuit specific data. This ensures that even in cases of rare or unusual phrases, the system can provide relevant auto-complete suggestions in near real time. Survey has showed that this system saves expert note-taking time and boosts expert confidence in their communication with teammates and clients. Since enabling this predictive writing feature for QBLive experts, more than a million keystrokes have been saved based on these suggestions. We have done comparative study for our ensemble choice. Moreover this feature can be integrated with any product which has writing facility within a very short period of time.

Title: Class Binarization to NeuroEvolution for Multiclass Classification. (arXiv:2308.13876v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13876
• Code URL: https://github.com/lafengxiaoyu/neat-ensembles
• Copy Paste: [[2308.13876]] Class Binarization to NeuroEvolution for Multiclass Classification(http://arxiv.org/abs/2308.13876)
• Summary:

  Multiclass classification is a fundamental and challenging task in machine learning. The existing techniques of multiclass classification can be categorized as (i) decomposition into binary (ii) extension from binary and (iii) hierarchical classification. Decomposing multiclass classification into a set of binary classifications that can be efficiently solved by using binary classifiers, called class binarization, which is a popular technique for multiclass classification. Neuroevolution, a general and powerful technique for evolving the structure and weights of neural networks, has been successfully applied to binary classification. In this paper, we apply class binarization techniques to a neuroevolution algorithm, NeuroEvolution of Augmenting Topologies (NEAT), that is used to generate neural networks for multiclass classification. We propose a new method that applies Error-Correcting Output Codes (ECOC) to design the class binarization strategies on the neuroevolution for multiclass classification. The ECOC strategies are compared with the class binarization strategies of One-vs-One and One-vs-All on three well-known datasets Digit, Satellite, and Ecoli. We analyse their performance from four aspects of multiclass classification degradation, accuracy, evolutionary efficiency, and robustness. The results show that the NEAT with ECOC performs high accuracy with low variance. Specifically, it shows significant benefits in a flexible number of binary classifiers and strong robustness.

Title: Label Denoising through Cross-Model Agreement. (arXiv:2308.13976v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13976
• Code URL: null
• Copy Paste: [[2308.13976]] Label Denoising through Cross-Model Agreement(http://arxiv.org/abs/2308.13976)
• Summary:

  Learning from corrupted labels is very common in real-world machine-learning applications. Memorizing such noisy labels could affect the learning of the model, leading to sub-optimal performances. In this work, we propose a novel framework to learn robust machine-learning models from noisy labels. Through an empirical study, we find that different models make relatively similar predictions on clean examples, while the predictions on noisy examples vary much more across different models. Motivated by this observation, we propose \em denoising with cross-model agreement \em (DeCA) which aims to minimize the KL-divergence between the true label distributions parameterized by two machine learning models while maximizing the likelihood of data observation. We employ the proposed DeCA on both the binary label scenario and the multiple label scenario. For the binary label scenario, we select implicit feedback recommendation as the downstream task and conduct experiments with four state-of-the-art recommendation models on four datasets. For the multiple-label scenario, the downstream application is image classification on two benchmark datasets. Experimental results demonstrate that the proposed methods significantly improve the model performance compared with normal training and other denoising methods on both binary and multiple-label scenarios.

biometric

steal

extraction

Title: WellXplain: Wellness Concept Extraction and Classification in Reddit Posts for Mental Health Analysis. (arXiv:2308.13710v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.13710
• Code URL: https://github.com/drmuskangarg/wellnessdimensions
• Copy Paste: [[2308.13710]] WellXplain: Wellness Concept Extraction and Classification in Reddit Posts for Mental Health Analysis(http://arxiv.org/abs/2308.13710)
• Summary:

  During the current mental health crisis, the importance of identifying potential indicators of mental issues from social media content has surged. Overlooking the multifaceted nature of mental and social well-being can have detrimental effects on one's mental state. In traditional therapy sessions, professionals manually pinpoint the origins and outcomes of underlying mental challenges, a process both detailed and time-intensive. We introduce an approach to this intricate mental health analysis by framing the identification of wellness dimensions in Reddit content as a wellness concept extraction and categorization challenge. We've curated a unique dataset named WELLXPLAIN, comprising 3,092 entries and totaling 72,813 words. Drawing from Halbert L. Dunn's well-regarded wellness theory, our team formulated an annotation framework along with guidelines. This dataset also includes human-marked textual segments, offering clear reasoning for decisions made in the wellness concept categorization process. Our aim in publishing this dataset and analyzing initial benchmarks is to spearhead the creation of advanced language models tailored for healthcare-focused concept extraction and categorization.

membership infer

federate

Title: Federated Learning for Computer Vision. (arXiv:2308.13558v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13558
• Code URL: null
• Copy Paste: [[2308.13558]] Federated Learning for Computer Vision(http://arxiv.org/abs/2308.13558)
• Summary:

  Computer Vision (CV) is playing a significant role in transforming society by utilizing machine learning (ML) tools for a wide range of tasks. However, the need for large-scale datasets to train ML models creates challenges for centralized ML algorithms. The massive computation loads required for processing and the potential privacy risks associated with storing and processing data on central cloud servers put these algorithms under severe strain. To address these issues, federated learning (FL) has emerged as a promising solution, allowing privacy preservation by training models locally and exchanging them to improve overall performance. Additionally, the computational load is distributed across multiple clients, reducing the burden on central servers. This paper presents, to the best of the authors' knowledge, the first review discussing recent advancements of FL in CV applications, comparing them to conventional centralized training paradigms. It provides an overview of current FL applications in various CV tasks, emphasizing the advantages of FL and the challenges of implementing it in CV. To facilitate this, the paper proposes a taxonomy of FL techniques in CV, outlining their applications and security threats. It also discusses privacy concerns related to implementing blockchain in FL schemes for CV tasks and summarizes existing privacy preservation methods. Moving on, the paper identifies open research challenges and potential future research directions to further exploit the potential of FL and blockchain in CV applications.

Title: Uncovering Promises and Challenges of Federated Learning to Detect Cardiovascular Diseases: A Scoping Literature Review. (arXiv:2308.13714v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13714
• Code URL: null
• Copy Paste: [[2308.13714]] Uncovering Promises and Challenges of Federated Learning to Detect Cardiovascular Diseases: A Scoping Literature Review(http://arxiv.org/abs/2308.13714)
• Summary:

  Cardiovascular diseases (CVD) are the leading cause of death globally, and early detection can significantly improve outcomes for patients. Machine learning (ML) models can help diagnose CVDs early, but their performance is limited by the data available for model training. Privacy concerns in healthcare make it harder to acquire data to train accurate ML models. Federated learning (FL) is an emerging approach to machine learning that allows models to be trained on data from multiple sources without compromising the privacy of the individual data owners. This survey paper provides an overview of the current state-of-the-art in FL for CVD detection. We review the different FL models proposed in various papers and discuss their advantages and challenges. We also compare FL with traditional centralized learning approaches and highlight the differences in terms of model accuracy, privacy, and data distribution handling capacity. Finally, we provide a critical analysis of FL's current challenges and limitations for CVD detection and discuss potential avenues for future research. Overall, this survey paper aims to provide a comprehensive overview of the current state-of-the-art in FL for CVD detection and to highlight its potential for improving the accuracy and privacy of CVD detection models.

Title: Resource-Efficient Federated Learning for Heterogenous and Resource-Constrained Environments. (arXiv:2308.13662v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13662
• Code URL: null
• Copy Paste: [[2308.13662]] Resource-Efficient Federated Learning for Heterogenous and Resource-Constrained Environments(http://arxiv.org/abs/2308.13662)
• Summary:

  Federated Learning (FL) is a privacy-enforcing sub-domain of machine learning that brings the model to the user's device for training, avoiding the need to share personal data with a central server. While existing works address data heterogeneity, they overlook other challenges in FL, such as device heterogeneity and communication efficiency. In this paper, we propose RE-FL, a novel approach that tackles computational and communication challenges in resource-constrained devices. Our variable pruning technique optimizes resource utilization by adapting pruning to each client's computational capabilities. We also employ knowledge distillation to reduce bandwidth consumption and communication rounds. Experimental results on image classification tasks demonstrate the effectiveness of our approach in resource-constrained environments, maintaining data privacy and performance while accommodating heterogeneous model architectures.

Title: Price-Discrimination Game for Distributed Resource Management in Federated Learning. (arXiv:2308.13838v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13838
• Code URL: null
• Copy Paste: [[2308.13838]] Price-Discrimination Game for Distributed Resource Management in Federated Learning(http://arxiv.org/abs/2308.13838)
• Summary:

  In vanilla federated learning (FL) such as FedAvg, the parameter server (PS) and multiple distributed clients can form a typical buyer's market, where the number of PS/buyers of FL services is far less than the number of clients/sellers. In order to improve the performance of FL and reduce the cost of motivating clients to participate in FL, this paper proposes to differentiate the pricing for services provided by different clients rather than simply providing the same service pricing for different clients. The price is differentiated based on the performance improvements brought to FL and their heterogeneity in computing and communication capabilities. To this end, a price-discrimination game (PDG) is formulated to comprehensively address the distributed resource management problems in FL, including multi-objective trade-off, client selection, and incentive mechanism. As the PDG is a mixed-integer nonlinear programming (MINLP) problem, a distributed semi-heuristic algorithm with low computational complexity and low communication overhead is designed to solve it. The simulation result verifies the effectiveness of the proposed approach.

Title: Effectively Heterogeneous Federated Learning: A Pairing and Split Learning Based Approach. (arXiv:2308.13849v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13849
• Code URL: null
• Copy Paste: [[2308.13849]] Effectively Heterogeneous Federated Learning: A Pairing and Split Learning Based Approach(http://arxiv.org/abs/2308.13849)
• Summary:

  As a promising paradigm federated Learning (FL) is widely used in privacy-preserving machine learning, which allows distributed devices to collaboratively train a model while avoiding data transmission among clients. Despite its immense potential, the FL suffers from bottlenecks in training speed due to client heterogeneity, leading to escalated training latency and straggling server aggregation. To deal with this challenge, a novel split federated learning (SFL) framework that pairs clients with different computational resources is proposed, where clients are paired based on computing resources and communication rates among clients, meanwhile the neural network model is split into two parts at the logical level, and each client only computes the part assigned to it by using the SL to achieve forward inference and backward training. Moreover, to effectively deal with the client pairing problem, a heuristic greedy algorithm is proposed by reconstructing the optimization of training latency as a graph edge selection problem. Simulation results show the proposed method can significantly improve the FL training speed and achieve high performance both in independent identical distribution (IID) and Non-IID data distribution.

fair

Title: Muffin: A Framework Toward Multi-Dimension AI Fairness by Uniting Off-the-Shelf Models. (arXiv:2308.13730v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13730
• Code URL: null
• Copy Paste: [[2308.13730]] Muffin: A Framework Toward Multi-Dimension AI Fairness by Uniting Off-the-Shelf Models(http://arxiv.org/abs/2308.13730)
• Summary:

  Model fairness (a.k.a., bias) has become one of the most critical problems in a wide range of AI applications. An unfair model in autonomous driving may cause a traffic accident if corner cases (e.g., extreme weather) cannot be fairly regarded; or it will incur healthcare disparities if the AI model misdiagnoses a certain group of people (e.g., brown and black skin). In recent years, there have been emerging research works on addressing unfairness, and they mainly focus on a single unfair attribute, like skin tone; however, real-world data commonly have multiple attributes, among which unfairness can exist in more than one attribute, called 'multi-dimensional fairness'. In this paper, we first reveal a strong correlation between the different unfair attributes, i.e., optimizing fairness on one attribute will lead to the collapse of others. Then, we propose a novel Multi-Dimension Fairness framework, namely Muffin, which includes an automatic tool to unite off-the-shelf models to improve the fairness on multiple attributes simultaneously. Case studies on dermatology datasets with two unfair attributes show that the existing approach can achieve 21.05% fairness improvement on the first attribute while it makes the second attribute unfair by 1.85%. On the other hand, the proposed Muffin can unite multiple models to achieve simultaneously 26.32% and 20.37% fairness improvement on both attributes; meanwhile, it obtains 5.58% accuracy gain.

interpretability

Title: Planning with Logical Graph-based Language Model for Instruction Generation. (arXiv:2308.13782v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.13782
• Code URL: null
• Copy Paste: [[2308.13782]] Planning with Logical Graph-based Language Model for Instruction Generation(http://arxiv.org/abs/2308.13782)
• Summary:

  Despite the superior performance of large language models to generate natural language texts, it is hard to generate texts with correct logic according to a given task, due to the difficulties for neural models to capture implied rules from free-form texts. In this paper, we propose a novel graph-based language model, Logical-GLM, to infuse logic into language models for more valid text generation and interpretability. Specifically, we first capture information from natural language instructions and construct logical bayes graphs that generally describe domains. Next, we generate logical skeletons to guide language model training, infusing domain knowledge into language models. Finally, we alternately optimize the searching policy of graphs and language models until convergence. The experimental results show that Logical-GLM is both effective and efficient compared with traditional language models, despite using smaller-scale training data and fewer parameters. Our approach can generate instructional texts with more correct logic owing to the internalized domain knowledge. Moreover, the usage of logical graphs reflects the inner mechanism of the language models, which improves the interpretability of black-box models.

Title: DeLELSTM: Decomposition-based Linear Explainable LSTM to Capture Instantaneous and Long-term Effects in Time Series. (arXiv:2308.13797v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13797
• Code URL: null
• Copy Paste: [[2308.13797]] DeLELSTM: Decomposition-based Linear Explainable LSTM to Capture Instantaneous and Long-term Effects in Time Series(http://arxiv.org/abs/2308.13797)
• Summary:

  Time series forecasting is prevalent in various real-world applications. Despite the promising results of deep learning models in time series forecasting, especially the Recurrent Neural Networks (RNNs), the explanations of time series models, which are critical in high-stakes applications, have received little attention. In this paper, we propose a Decomposition-based Linear Explainable LSTM (DeLELSTM) to improve the interpretability of LSTM. Conventionally, the interpretability of RNNs only concentrates on the variable importance and time importance. We additionally distinguish between the instantaneous influence of new coming data and the long-term effects of historical data. Specifically, DeLELSTM consists of two components, i.e., standard LSTM and tensorized LSTM. The tensorized LSTM assigns each variable with a unique hidden state making up a matrix $\mathbf{h}_t$, and the standard LSTM models all the variables with a shared hidden state $\mathbf{H}_t$. By decomposing the $\mathbf{H}_t$ into the linear combination of past information $\mathbf{h}_{t-1}$ and the fresh information $\mathbf{h}_{t}-\mathbf{h}_{t-1}$, we can get the instantaneous influence and the long-term effect of each variable. In addition, the advantage of linear regression also makes the explanation transparent and clear. We demonstrate the effectiveness and interpretability of DeLELSTM on three empirical datasets. Extensive experiments show that the proposed method achieves competitive performance against the baseline methods and provides a reliable explanation relative to domain knowledge.

Title: Homological Convolutional Neural Networks. (arXiv:2308.13816v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13816
• Code URL: null
• Copy Paste: [[2308.13816]] Homological Convolutional Neural Networks(http://arxiv.org/abs/2308.13816)
• Summary:

  Deep learning methods have demonstrated outstanding performances on classification and regression tasks on homogeneous data types (e.g., image, audio, and text data). However, tabular data still poses a challenge with classic machine learning approaches being often computationally cheaper and equally effective than increasingly complex deep learning architectures. The challenge arises from the fact that, in tabular data, the correlation among features is weaker than the one from spatial or semantic relationships in images or natural languages, and the dependency structures need to be modeled without any prior information. In this work, we propose a novel deep learning architecture that exploits the data structural organization through topologically constrained network representations to gain spatial information from sparse tabular data. The resulting model leverages the power of convolutions and is centered on a limited number of concepts from network topology to guarantee (i) a data-centric, deterministic building pipeline; (ii) a high level of interpretability over the inference process; and (iii) an adequate room for scalability. We test our model on 18 benchmark datasets against 5 classic machine learning and 3 deep learning models demonstrating that our approach reaches state-of-the-art performances on these challenging datasets. The code to reproduce all our experiments is provided at https://github.com/FinancialComputingUCL/HomologicalCNN.

explainability

watermark

diffusion

Title: Residual Denoising Diffusion Models. (arXiv:2308.13712v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13712
• Code URL: null
• Copy Paste: [[2308.13712]] Residual Denoising Diffusion Models(http://arxiv.org/abs/2308.13712)
• Summary:

  Current diffusion-based image restoration methods feed degraded input images as conditions into the noise estimation network. However, interpreting this diffusion process is challenging since it essentially generates the target image from the noise. To establish a unified and more interpretable model for image generation and restoration, we propose residual denoising diffusion models (RDDM). In contrast to existing diffusion models (e.g., DDPM or DDIM) that focus solely on noise estimation, our RDDM predicts residuals to represent directional diffusion from the target domain to the input domain, while concurrently estimating noise to account for random perturbations in the diffusion process. The introduction of residuals allows us to redefine the forward diffusion process, wherein the target image progressively diffuses into a purely noisy image or a noise-carrying input image, thus unifying image generation and restoration. We demonstrate that our sampling process is consistent with that of DDPM and DDIM through coefficient transformation, and propose a partially path-independent generation process to better understand the reverse process. Notably, with native support for conditional inputs, our RDDM enables a generic UNet, trained with only an $\ell _1$ loss and a batch size of 1, to compete with state-of-the-art image restoration methods. We provide code and pre-trained models to encourage further exploration, application, and development of our innovative framework (https://github.com/nachifur/RDDM).

Title: DiffI2I: Efficient Diffusion Model for Image-to-Image Translation. (arXiv:2308.13767v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13767
• Code URL: null
• Copy Paste: [[2308.13767]] DiffI2I: Efficient Diffusion Model for Image-to-Image Translation(http://arxiv.org/abs/2308.13767)
• Summary:

  The Diffusion Model (DM) has emerged as the SOTA approach for image synthesis. However, the existing DM cannot perform well on some image-to-image translation (I2I) tasks. Different from image synthesis, some I2I tasks, such as super-resolution, require generating results in accordance with GT images. Traditional DMs for image synthesis require extensive iterations and large denoising models to estimate entire images, which gives their strong generative ability but also leads to artifacts and inefficiency for I2I. To tackle this challenge, we propose a simple, efficient, and powerful DM framework for I2I, called DiffI2I. Specifically, DiffI2I comprises three key components: a compact I2I prior extraction network (CPEN), a dynamic I2I transformer (DI2Iformer), and a denoising network. We train DiffI2I in two stages: pretraining and DM training. For pretraining, GT and input images are fed into CPEN$_{S1}$ to capture a compact I2I prior representation (IPR) guiding DI2Iformer. In the second stage, the DM is trained to only use the input images to estimate the same IRP as CPEN$_{S1}$. Compared to traditional DMs, the compact IPR enables DiffI2I to obtain more accurate outcomes and employ a lighter denoising network and fewer iterations. Through extensive experiments on various I2I tasks, we demonstrate that DiffI2I achieves SOTA performance while significantly reducing computational burdens.

Title: ORES: Open-vocabulary Responsible Visual Synthesis. (arXiv:2308.13785v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13785
• Code URL: https://github.com/kodenii/ores
• Copy Paste: [[2308.13785]] ORES: Open-vocabulary Responsible Visual Synthesis(http://arxiv.org/abs/2308.13785)
• Summary:

  Avoiding synthesizing specific visual concepts is an essential challenge in responsible visual synthesis. However, the visual concept that needs to be avoided for responsible visual synthesis tends to be diverse, depending on the region, context, and usage scenarios. In this work, we formalize a new task, Open-vocabulary Responsible Visual Synthesis (ORES), where the synthesis model is able to avoid forbidden visual concepts while allowing users to input any desired content. To address this problem, we present a Two-stage Intervention (TIN) framework. By introducing 1) rewriting with learnable instruction through a large-scale language model (LLM) and 2) synthesizing with prompt intervention on a diffusion synthesis model, it can effectively synthesize images avoiding any concepts but following the user's query as much as possible. To evaluate on ORES, we provide a publicly available dataset, baseline models, and benchmark. Experimental results demonstrate the effectiveness of our method in reducing risks of image generation. Our work highlights the potential of LLMs in responsible visual synthesis. Our code and dataset is public available.

Title: Unsupervised Domain Adaptation via Domain-Adaptive Diffusion. (arXiv:2308.13893v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13893
• Code URL: null
• Copy Paste: [[2308.13893]] Unsupervised Domain Adaptation via Domain-Adaptive Diffusion(http://arxiv.org/abs/2308.13893)
• Summary:

  Unsupervised Domain Adaptation (UDA) is quite challenging due to the large distribution discrepancy between the source domain and the target domain. Inspired by diffusion models which have strong capability to gradually convert data distributions across a large gap, we consider to explore the diffusion technique to handle the challenging UDA task. However, using diffusion models to convert data distribution across different domains is a non-trivial problem as the standard diffusion models generally perform conversion from the Gaussian distribution instead of from a specific domain distribution. Besides, during the conversion, the semantics of the source-domain data needs to be preserved for classification in the target domain. To tackle these problems, we propose a novel Domain-Adaptive Diffusion (DAD) module accompanied by a Mutual Learning Strategy (MLS), which can gradually convert data distribution from the source domain to the target domain while enabling the classification model to learn along the domain transition process. Consequently, our method successfully eases the challenge of UDA by decomposing the large domain gap into small ones and gradually enhancing the capacity of classification model to finally adapt to the target domain. Our method outperforms the current state-of-the-arts by a large margin on three widely used UDA datasets.

Title: Network Embedding Using Sparse Approximations of Random Walks. (arXiv:2308.13663v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13663
• Code URL: null
• Copy Paste: [[2308.13663]] Network Embedding Using Sparse Approximations of Random Walks(http://arxiv.org/abs/2308.13663)
• Summary:

  In this paper, we propose an efficient numerical implementation of Network Embedding based on commute times, using sparse approximation of a diffusion process on the network obtained by a modified version of the diffusion wavelet algorithm. The node embeddings are computed by optimizing the cross entropy loss via the stochastic gradient descent method with sampling of low-dimensional representations of green functions. We demonstrate the efficacy of this method for data clustering and multi-label classification through several examples, and compare its performance over existing methods in terms of efficiency and accuracy. Theoretical issues justifying the scheme are also discussed.

noise learning

data-free

transformer

Title: Linear Oscillation: The Aesthetics of Confusion for Vision Transformer. (arXiv:2308.13670v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13670
• Code URL: null
• Copy Paste: [[2308.13670]] Linear Oscillation: The Aesthetics of Confusion for Vision Transformer(http://arxiv.org/abs/2308.13670)
• Summary:

  Activation functions are the linchpins of deep learning, profoundly influencing both the representational capacity and training dynamics of neural networks. They shape not only the nature of representations but also optimize convergence rates and enhance generalization potential. Appreciating this critical role, we present the Linear Oscillation (LoC) activation function, defined as $f(x) = x \times \sin(\alpha x + \beta)$. Distinct from conventional activation functions which primarily introduce non-linearity, LoC seamlessly blends linear trajectories with oscillatory deviations. The nomenclature ``Linear Oscillation'' is a nod to its unique attribute of infusing linear activations with harmonious oscillations, capturing the essence of the 'Importance of Confusion'. This concept of ``controlled confusion'' within network activations is posited to foster more robust learning, particularly in contexts that necessitate discerning subtle patterns. Our empirical studies reveal that, when integrated into diverse neural architectures, the LoC activation function consistently outperforms established counterparts like ReLU and Sigmoid. The stellar performance exhibited by the avant-garde Vision Transformer model using LoC further validates its efficacy. This study illuminates the remarkable benefits of the LoC over other prominent activation functions. It champions the notion that intermittently introducing deliberate complexity or ``confusion'' during training can spur more profound and nuanced learning. This accentuates the pivotal role of judiciously selected activation functions in shaping the future of neural network training.

Title: Enhancing Landmark Detection in Cluttered Real-World Scenarios with Vision Transformers. (arXiv:2308.13671v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13671
• Code URL: null
• Copy Paste: [[2308.13671]] Enhancing Landmark Detection in Cluttered Real-World Scenarios with Vision Transformers(http://arxiv.org/abs/2308.13671)
• Summary:

  Visual place recognition tasks often encounter significant challenges in landmark detection due to the presence of irrelevant objects such as humans, cars, and trees, despite the remarkable progress achieved by previous models, especially in the context of transformers. To address this issue, we propose a novel method that effectively leverages the strengths of vision transformers. By employing a meticulous selection process, our approach identifies and isolates specific patches within the image that correspond to occluding objects. To evaluate the efficacy of our method, we created augmented datasets and conducted comprehensive testing. The results demonstrate the superior accuracy achieved by our proposed approach. This research contributes to the advancement of landmark detection in visual place recognition and shows the potential of leveraging vision transformers to overcome challenges posed by cluttered real-world scenarios.

Title: ACC-UNet: A Completely Convolutional UNet model for the 2020s. (arXiv:2308.13680v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13680
• Code URL: https://github.com/kiharalab/acc-unet
• Copy Paste: [[2308.13680]] ACC-UNet: A Completely Convolutional UNet model for the 2020s(http://arxiv.org/abs/2308.13680)
• Summary:

  This decade is marked by the introduction of Vision Transformer, a radical paradigm shift in broad computer vision. A similar trend is followed in medical imaging, UNet, one of the most influential architectures, has been redesigned with transformers. Recently, the efficacy of convolutional models in vision is being reinvestigated by seminal works such as ConvNext, which elevates a ResNet to Swin Transformer level. Deriving inspiration from this, we aim to improve a purely convolutional UNet model so that it can be on par with the transformer-based models, e.g, Swin-Unet or UCTransNet. We examined several advantages of the transformer-based UNet models, primarily long-range dependencies and cross-level skip connections. We attempted to emulate them through convolution operations and thus propose, ACC-UNet, a completely convolutional UNet model that brings the best of both worlds, the inherent inductive biases of convnets with the design decisions of transformers. ACC-UNet was evaluated on 5 different medical image segmentation benchmarks and consistently outperformed convnets, transformers, and their hybrids. Notably, ACC-UNet outperforms state-of-the-art models Swin-Unet and UCTransNet by $2.64 \pm 2.54\%$ and $0.45 \pm 1.61\%$ in terms of dice score, respectively, while using a fraction of their parameters ($59.26\%$ and $24.24\%$). Our codes are available at https://github.com/kiharalab/ACC-UNet.

Title: EventTransAct: A video transformer-based framework for Event-camera based action recognition. (arXiv:2308.13711v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13711
• Code URL: null
• Copy Paste: [[2308.13711]] EventTransAct: A video transformer-based framework for Event-camera based action recognition(http://arxiv.org/abs/2308.13711)
• Summary:

  Recognizing and comprehending human actions and gestures is a crucial perception requirement for robots to interact with humans and carry out tasks in diverse domains, including service robotics, healthcare, and manufacturing. Event cameras, with their ability to capture fast-moving objects at a high temporal resolution, offer new opportunities compared to standard action recognition in RGB videos. However, previous research on event camera action recognition has primarily focused on sensor-specific network architectures and image encoding, which may not be suitable for new sensors and limit the use of recent advancements in transformer-based architectures. In this study, we employ a computationally efficient model, namely the video transformer network (VTN), which initially acquires spatial embeddings per event-frame and then utilizes a temporal self-attention mechanism. In order to better adopt the VTN for the sparse and fine-grained nature of event data, we design Event-Contrastive Loss ($\mathcal{L}_{EC}$) and event-specific augmentations. Proposed $\mathcal{L}_{EC}$ promotes learning fine-grained spatial cues in the spatial backbone of VTN by contrasting temporally misaligned frames. We evaluate our method on real-world action recognition of N-EPIC Kitchens dataset, and achieve state-of-the-art results on both protocols - testing in seen kitchen (\textbf{74.9\%} accuracy) and testing in unseen kitchens (\textbf{42.43\% and 46.66\% Accuracy}). Our approach also takes less computation time compared to competitive prior approaches, which demonstrates the potential of our framework \textit{EventTransAct} for real-world applications of event-camera based action recognition. Project Page: \url{https://tristandb8.github.io/EventTransAct_webpage/}

Title: Devignet: High-Resolution Vignetting Removal via a Dual Aggregated Fusion Transformer With Adaptive Channel Expansion. (arXiv:2308.13739v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13739
• Code URL: null
• Copy Paste: [[2308.13739]] Devignet: High-Resolution Vignetting Removal via a Dual Aggregated Fusion Transformer With Adaptive Channel Expansion(http://arxiv.org/abs/2308.13739)
• Summary:

  Vignetting commonly occurs as a degradation in images resulting from factors such as lens design, improper lens hood usage, and limitations in camera sensors. This degradation affects image details, color accuracy, and presents challenges in computational photography. Existing vignetting removal algorithms predominantly rely on ideal physics assumptions and hand-crafted parameters, resulting in ineffective removal of irregular vignetting and suboptimal results. Moreover, the substantial lack of real-world vignetting datasets hinders the objective and comprehensive evaluation of vignetting removal. To address these challenges, we present Vigset, a pioneering dataset for vignette removal. Vigset includes 983 pairs of both vignetting and vignetting-free high-resolution ($5340\times3697$) real-world images under various conditions. In addition, We introduce DeVigNet, a novel frequency-aware Transformer architecture designed for vignetting removal. Through the Laplacian Pyramid decomposition, we propose the Dual Aggregated Fusion Transformer to handle global features and remove vignetting in the low-frequency domain. Additionally, we introduce the Adaptive Channel Expansion Module to enhance details in the high-frequency domain. The experiments demonstrate that the proposed model outperforms existing state-of-the-art methods.

Title: Unified Single-Stage Transformer Network for Efficient RGB-T Tracking. (arXiv:2308.13764v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13764
• Code URL: null
• Copy Paste: [[2308.13764]] Unified Single-Stage Transformer Network for Efficient RGB-T Tracking(http://arxiv.org/abs/2308.13764)
• Summary:

  Most existing RGB-T tracking networks extract modality features in a separate manner, which lacks interaction and mutual guidance between modalities. This limits the network's ability to adapt to the diverse dual-modality appearances of targets and the dynamic relationships between the modalities. Additionally, the three-stage fusion tracking paradigm followed by these networks significantly restricts the tracking speed. To overcome these problems, we propose a unified single-stage Transformer RGB-T tracking network, namely USTrack, which unifies the above three stages into a single ViT (Vision Transformer) backbone with a dual embedding layer through self-attention mechanism. With this structure, the network can extract fusion features of the template and search region under the mutual interaction of modalities. Simultaneously, relation modeling is performed between these features, efficiently obtaining the search region fusion features with better target-background discriminability for prediction. Furthermore, we introduce a novel feature selection mechanism based on modality reliability to mitigate the influence of invalid modalities for prediction, further improving the tracking performance. Extensive experiments on three popular RGB-T tracking benchmarks demonstrate that our method achieves new state-of-the-art performance while maintaining the fastest inference speed 84.2FPS. In particular, MPR/MSR on the short-term and long-term subsets of VTUAV dataset increased by 11.1$\%$/11.7$\%$ and 11.3$\%$/9.7$\%$.

Title: Point-Query Quadtree for Crowd Counting, Localization, and More. (arXiv:2308.13814v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13814
• Code URL: null
• Copy Paste: [[2308.13814]] Point-Query Quadtree for Crowd Counting, Localization, and More(http://arxiv.org/abs/2308.13814)
• Summary:

  We show that crowd counting can be viewed as a decomposable point querying process. This formulation enables arbitrary points as input and jointly reasons whether the points are crowd and where they locate. The querying processing, however, raises an underlying problem on the number of necessary querying points. Too few imply underestimation; too many increase computational overhead. To address this dilemma, we introduce a decomposable structure, i.e., the point-query quadtree, and propose a new counting model, termed Point quEry Transformer (PET). PET implements decomposable point querying via data-dependent quadtree splitting, where each querying point could split into four new points when necessary, thus enabling dynamic processing of sparse and dense regions. Such a querying process yields an intuitive, universal modeling of crowd as both the input and output are interpretable and steerable. We demonstrate the applications of PET on a number of crowd-related tasks, including fully-supervised crowd counting and localization, partial annotation learning, and point annotation refinement, and also report state-of-the-art performance. For the first time, we show that a single counting model can address multiple crowd-related tasks across different learning paradigms. Code is available at https://github.com/cxliu0/PET.

Title: Transfer Learning for Microstructure Segmentation with CS-UNet: A Hybrid Algorithm with Transformer and CNN Encoders. (arXiv:2308.13917v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13917
• Code URL: https://github.com/kalrfou/swint-pretrained-microscopy-models
• Copy Paste: [[2308.13917]] Transfer Learning for Microstructure Segmentation with CS-UNet: A Hybrid Algorithm with Transformer and CNN Encoders(http://arxiv.org/abs/2308.13917)
• Summary:

  Transfer learning improves the performance of deep learning models by initializing them with parameters pre-trained on larger datasets. Intuitively, transfer learning is more effective when pre-training is on the in-domain datasets. A recent study by NASA has demonstrated that the microstructure segmentation with encoder-decoder algorithms benefits more from CNN encoders pre-trained on microscopy images than from those pre-trained on natural images. However, CNN models only capture the local spatial relations in images. In recent years, attention networks such as Transformers are increasingly used in image analysis to capture the long-range relations between pixels. In this study, we compare the segmentation performance of Transformer and CNN models pre-trained on microscopy images with those pre-trained on natural images. Our result partially confirms the NASA study that the segmentation performance of out-of-distribution images (taken under different imaging and sample conditions) is significantly improved when pre-training on microscopy images. However, the performance gain for one-shot and few-shot learning is more modest with Transformers. We also find that for image segmentation, the combination of pre-trained Transformers and CNN encoders are consistently better than pre-trained CNN encoders alone. Our dataset (of about 50,000 images) combines the public portion of the NASA dataset with additional images we collected. Even with much less training data, our pre-trained models have significantly better performance for image segmentation. This result suggests that Transformers and CNN complement each other and when pre-trained on microscopy images, they are more beneficial to the downstream tasks.

Title: Fixating on Attention: Integrating Human Eye Tracking into Vision Transformers. (arXiv:2308.13969v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13969
• Code URL: null
• Copy Paste: [[2308.13969]] Fixating on Attention: Integrating Human Eye Tracking into Vision Transformers(http://arxiv.org/abs/2308.13969)
• Summary:

  Modern transformer-based models designed for computer vision have outperformed humans across a spectrum of visual tasks. However, critical tasks, such as medical image interpretation or autonomous driving, still require reliance on human judgments. This work demonstrates how human visual input, specifically fixations collected from an eye-tracking device, can be integrated into transformer models to improve accuracy across multiple driving situations and datasets. First, we establish the significance of fixation regions in left-right driving decisions, as observed in both human subjects and a Vision Transformer (ViT). By comparing the similarity between human fixation maps and ViT attention weights, we reveal the dynamics of overlap across individual heads and layers. This overlap is exploited for model pruning without compromising accuracy. Thereafter, we incorporate information from the driving scene with fixation data, employing a "joint space-fixation" (JSF) attention setup. Lastly, we propose a "fixation-attention intersection" (FAX) loss to train the ViT model to attend to the same regions that humans fixated on. We find that the ViT performance is improved in accuracy and number of training epochs when using JSF and FAX. These results hold significant implications for human-guided artificial intelligence.

Title: Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning. (arXiv:2308.13958v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.13958
• Code URL: null
• Copy Paste: [[2308.13958]] Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning(http://arxiv.org/abs/2308.13958)
• Summary:

  The use of large transformer-based models such as BERT, GPT, and T5 has led to significant advancements in natural language processing. However, these models are computationally expensive, necessitating model compression techniques that reduce their size and complexity while maintaining accuracy. This project investigates and applies knowledge distillation for BERT model compression, specifically focusing on the TinyBERT student model. We explore various techniques to improve knowledge distillation, including experimentation with loss functions, transformer layer mapping methods, and tuning the weights of attention and representation loss and evaluate our proposed techniques on a selection of downstream tasks from the GLUE benchmark. The goal of this work is to improve the efficiency and effectiveness of knowledge distillation, enabling the development of more efficient and accurate models for a range of natural language processing tasks.

Title: Translate Meanings, Not Just Words: IdiomKB's Role in Optimizing Idiomatic Translation with Language Models. (arXiv:2308.13961v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.13961
• Code URL: null
• Copy Paste: [[2308.13961]] Translate Meanings, Not Just Words: IdiomKB's Role in Optimizing Idiomatic Translation with Language Models(http://arxiv.org/abs/2308.13961)
• Summary:

  To translate well, machine translation (MT) systems and general-purposed language models (LMs) need a deep understanding of both source and target languages and cultures. Therefore, idioms, with their non-compositional nature, pose particular challenges for Transformer-based systems, as literal translations often miss the intended meaning. Traditional methods, which replace idioms using existing knowledge bases (KBs), often lack scale and context awareness. Addressing these challenges, our approach prioritizes context awareness and scalability, allowing for offline storage of idioms in a manageable KB size. This ensures efficient serving with smaller models and provides a more comprehensive understanding of idiomatic expressions. We introduce a multilingual idiom KB (IdiomKB) developed using large LMs to address this. This KB facilitates better translation by smaller models, such as BLOOMZ (7.1B), Alpaca (7B), and InstructGPT (6.7B), by retrieving idioms' figurative meanings. We present a novel, GPT-4-powered metric for human-aligned evaluation, demonstrating that IdiomKB considerably boosts model performance. Human evaluations further validate our KB's quality.

generative

Title: A Systematic Study on Quantifying Bias in GAN-Augmented Data. (arXiv:2308.13554v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13554
• Code URL: null
• Copy Paste: [[2308.13554]] A Systematic Study on Quantifying Bias in GAN-Augmented Data(http://arxiv.org/abs/2308.13554)
• Summary:

  Generative adversarial networks (GANs) have recently become a popular data augmentation technique used by machine learning practitioners. However, they have been shown to suffer from the so-called mode collapse failure mode, which makes them vulnerable to exacerbating biases on already skewed datasets, resulting in the generated data distribution being less diverse than the training distribution. To this end, we address the problem of quantifying the extent to which mode collapse occurs. This study is a systematic effort focused on the evaluation of state-of-the-art metrics that can potentially quantify biases in GAN-augmented data. We show that, while several such methods are available, there is no single metric that quantifies bias exacerbation reliably over the span of different image domains.

Title: Out-of-distribution detection using normalizing flows on the data manifold. (arXiv:2308.13792v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13792
• Code URL: null
• Copy Paste: [[2308.13792]] Out-of-distribution detection using normalizing flows on the data manifold(http://arxiv.org/abs/2308.13792)
• Summary:

  A common approach for out-of-distribution detection involves estimating an underlying data distribution, which assigns a lower likelihood value to out-of-distribution data. Normalizing flows are likelihood-based generative models providing a tractable density estimation via dimension-preserving invertible transformations. Conventional normalizing flows are prone to fail in out-of-distribution detection, because of the well-known curse of dimensionality problem of the likelihood-based models. According to the manifold hypothesis, real-world data often lie on a low-dimensional manifold. This study investigates the effect of manifold learning using normalizing flows on out-of-distribution detection. We proceed by estimating the density on a low-dimensional manifold, coupled with measuring the distance from the manifold, as criteria for out-of-distribution detection. However, individually, each of them is insufficient for this task. The extensive experimental results show that manifold learning improves the out-of-distribution detection ability of a class of likelihood-based models known as normalizing flows. This improvement is achieved without modifying the model structure or using auxiliary out-of-distribution data during training.

Title: VIDES: Virtual Interior Design via Natural Language and Visual Guidance. (arXiv:2308.13795v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13795
• Code URL: null
• Copy Paste: [[2308.13795]] VIDES: Virtual Interior Design via Natural Language and Visual Guidance(http://arxiv.org/abs/2308.13795)
• Summary:

  Interior design is crucial in creating aesthetically pleasing and functional indoor spaces. However, developing and editing interior design concepts requires significant time and expertise. We propose Virtual Interior DESign (VIDES) system in response to this challenge. Leveraging cutting-edge technology in generative AI, our system can assist users in generating and editing indoor scene concepts quickly, given user text description and visual guidance. Using both visual guidance and language as the conditional inputs significantly enhances the accuracy and coherence of the generated scenes, resulting in visually appealing designs. Through extensive experimentation, we demonstrate the effectiveness of VIDES in developing new indoor concepts, changing indoor styles, and replacing and removing interior objects. The system successfully captures the essence of users' descriptions while providing flexibility for customization. Consequently, this system can potentially reduce the entry barrier for indoor design, making it more accessible to users with limited technical skills and reducing the time required to create high-quality images. Individuals who have a background in design can now easily communicate their ideas visually and effectively present their design concepts. https://sites.google.com/view/ltnghia/research/VIDES

Title: DM-VTON: Distilled Mobile Real-time Virtual Try-On. (arXiv:2308.13798v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13798
• Code URL: null
• Copy Paste: [[2308.13798]] DM-VTON: Distilled Mobile Real-time Virtual Try-On(http://arxiv.org/abs/2308.13798)
• Summary:

  The fashion e-commerce industry has witnessed significant growth in recent years, prompting exploring image-based virtual try-on techniques to incorporate Augmented Reality (AR) experiences into online shopping platforms. However, existing research has primarily overlooked a crucial aspect - the runtime of the underlying machine-learning model. While existing methods prioritize enhancing output quality, they often disregard the execution time, which restricts their applications on a limited range of devices. To address this gap, we propose Distilled Mobile Real-time Virtual Try-On (DM-VTON), a novel virtual try-on framework designed to achieve simplicity and efficiency. Our approach is based on a knowledge distillation scheme that leverages a strong Teacher network as supervision to guide a Student network without relying on human parsing. Notably, we introduce an efficient Mobile Generative Module within the Student network, significantly reducing the runtime while ensuring high-quality output. Additionally, we propose Virtual Try-on-guided Pose for Data Synthesis to address the limited pose variation observed in training images. Experimental results show that the proposed method can achieve 40 frames per second on a single Nvidia Tesla T4 GPU and only take up 37 MB of memory while producing almost the same output quality as other state-of-the-art methods. DM-VTON stands poised to facilitate the advancement of real-time AR applications, in addition to the generation of lifelike attired human figures tailored for diverse specialized training tasks. https://sites.google.com/view/ltnghia/research/DMVTON

Title: Time-to-Pattern: Information-Theoretic Unsupervised Learning for Scalable Time Series Summarization. (arXiv:2308.13722v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13722
• Code URL: https://github.com/alirezaghods/t2p-time-to-pattern
• Copy Paste: [[2308.13722]] Time-to-Pattern: Information-Theoretic Unsupervised Learning for Scalable Time Series Summarization(http://arxiv.org/abs/2308.13722)
• Summary:

  Data summarization is the process of generating interpretable and representative subsets from a dataset. Existing time series summarization approaches often search for recurring subsequences using a set of manually devised similarity functions to summarize the data. However, such approaches are fraught with limitations stemming from an exhaustive search coupled with a heuristic definition of series similarity. Such approaches affect the diversity and comprehensiveness of the generated data summaries. To mitigate these limitations, we introduce an approach to time series summarization, called Time-to-Pattern (T2P), which aims to find a set of diverse patterns that together encode the most salient information, following the notion of minimum description length. T2P is implemented as a deep generative model that learns informative embeddings of the discrete time series on a latent space specifically designed to be interpretable. Our synthetic and real-world experiments reveal that T2P discovers informative patterns, even in noisy and complex settings. Furthermore, our results also showcase the improved performance of T2P over previous work in pattern diversity and processing scalability, which conclusively demonstrate the algorithm's effectiveness for time series summarization.

Title: SyMOT-Flow: Learning optimal transport flow for two arbitrary distributions with maximum mean discrepancy. (arXiv:2308.13815v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13815
• Code URL: null
• Copy Paste: [[2308.13815]] SyMOT-Flow: Learning optimal transport flow for two arbitrary distributions with maximum mean discrepancy(http://arxiv.org/abs/2308.13815)
• Summary:

  Finding a transformation between two unknown probability distributions from samples is crucial for modeling complex data distributions and perform tasks such as density estimation, sample generation, and statistical inference. One powerful framework for such transformations is normalizing flow, which transforms an unknown distribution into a standard normal distribution using an invertible network. In this paper, we introduce a novel model called SyMOT-Flow that trains an invertible transformation by minimizing the symmetric maximum mean discrepancy between samples from two unknown distributions, and we incorporate an optimal transport cost as regularization to obtain a short-distance and interpretable transformation. The resulted transformation leads to more stable and accurate sample generation. We establish several theoretical results for the proposed model and demonstrate its effectiveness with low-dimensional illustrative examples as well as high-dimensional generative samples obtained through the forward and reverse flows.

large language model

Title: MLLM-DataEngine: An Iterative Refinement Approach for MLLM. (arXiv:2308.13566v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13566
• Code URL: null
• Copy Paste: [[2308.13566]] MLLM-DataEngine: An Iterative Refinement Approach for MLLM(http://arxiv.org/abs/2308.13566)
• Summary:

  Despite the great advance of Multimodal Large Language Models (MLLMs) in both instruction dataset building and benchmarking, the independence of training and evaluation makes current MLLMs hard to further improve their capability under the guidance of evaluation results with a relatively low human cost. In this paper, we propose MLLM-DataEngine, a novel closed-loop system that bridges data generation, model training, and evaluation. Within each loop iteration, the MLLM-DataEngine first analyze the weakness of the model based on the evaluation results, then generate a proper incremental dataset for the next training iteration and enhance the model capability iteratively. Compared with previous data collection methods which are separate from the benchmarking, the data generated by MLLM-DataEngine shows better targeting, quality, and correctness. For targeting, we propose an Adaptive Bad-case Sampling module, which adjusts the ratio of different types of data within each incremental dataset based on the benchmarking results. For quality, we resort to GPT-4 to generate high-quality data with each given data type. For correctness, prompt design is critical for the data generation results. Rather than previous hand-crafted prompt, we propose an Interactive Prompt Optimization strategy, which optimizes the prompt with the multi-round interaction between human and GPT, and improve the correctness of generated data greatly. Through extensive experiments, we find our MLLM-DataEngine could boost the MLLM capability in a targeted and automatic manner, with only a few human participation. The MLLM-DataEngine will be released and we hope it could be a general solution for the following MLLMs building.

Title: Large Language Models in Analyzing Crash Narratives -- A Comparative Study of ChatGPT, BARD and GPT-4. (arXiv:2308.13563v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.13563
• Code URL: null
• Copy Paste: [[2308.13563]] Large Language Models in Analyzing Crash Narratives -- A Comparative Study of ChatGPT, BARD and GPT-4(http://arxiv.org/abs/2308.13563)
• Summary:

  In traffic safety research, extracting information from crash narratives using text analysis is a common practice. With recent advancements of large language models (LLM), it would be useful to know how the popular LLM interfaces perform in classifying or extracting information from crash narratives. To explore this, our study has used the three most popular publicly available LLM interfaces- ChatGPT, BARD and GPT4. This study investigated their usefulness and boundaries in extracting information and answering queries related to accidents from 100 crash narratives from Iowa and Kansas. During the investigation, their capabilities and limitations were assessed and their responses to the queries were compared. Five questions were asked related to the narratives: 1) Who is at-fault? 2) What is the manner of collision? 3) Has the crash occurred in a work-zone? 4) Did the crash involve pedestrians? and 5) What are the sequence of harmful events in the crash? For questions 1 through 4, the overall similarity among the LLMs were 70%, 35%, 96% and 89%, respectively. The similarities were higher while answering direct questions requiring binary responses and significantly lower for complex questions. To compare the responses to question 5, network diagram and centrality measures were analyzed. The network diagram from the three LLMs were not always similar although they sometimes have the same influencing events with high in-degree, out-degree and betweenness centrality. This study suggests using multiple models to extract viable information from narratives. Also, caution must be practiced while using these interfaces to obtain crucial safety related information.

Title: DARWIN Series: Domain Specific Large Language Models for Natural Science. (arXiv:2308.13565v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.13565
• Code URL: null
• Copy Paste: [[2308.13565]] DARWIN Series: Domain Specific Large Language Models for Natural Science(http://arxiv.org/abs/2308.13565)
• Summary:

  Emerging tools bring forth fresh approaches to work, and the field of natural science is no different. In natural science, traditional manual, serial, and labour-intensive work is being augmented by automated, parallel, and iterative processes driven by artificial intelligence-based experimental automation and more. To add new capabilities in natural science, enabling the acceleration and enrichment of automation of the discovery process, we present DARWIN, a series of tailored LLMs for natural science, mainly in physics, chemistry, and material science. This series relies on open-source LLM, incorporating structured and unstructured scientific knowledge from public datasets and literature. We fine-tuned the models using over 60,000 instruction data points, emphasizing factual correctness. During the fine-tuning, we introduce the Scientific Instruction Generation (SIG) model, automating instruction generation from scientific texts. This eliminates the need for manual extraction or domain-specific knowledge graphs and efficiently injects scientific knowledge into the model. We also explore multi-task training strategies, revealing interconnections between scientific tasks. DARWIN series not only achieves state-of-the-art results on various scientific tasks but also diminishes reliance on closed-source AI models. Our research showcases the ability of LLM in the scientific domain, with the overarching goal of fostering prosperity within the broader AI for science community.

Title: Text Style Transfer Evaluation Using Large Language Models. (arXiv:2308.13577v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.13577
• Code URL: null
• Copy Paste: [[2308.13577]] Text Style Transfer Evaluation Using Large Language Models(http://arxiv.org/abs/2308.13577)
• Summary:

  Text Style Transfer (TST) is challenging to evaluate because the quality of the generated text manifests itself in multiple aspects, each of which is hard to measure individually: style transfer accuracy, content preservation, and overall fluency of the text. Human evaluation is the gold standard in TST evaluation; however, it is expensive, and the results are difficult to reproduce. Numerous automated metrics are employed to assess performance in these aspects, serving as substitutes for human evaluation. However, the correlation between many of these automated metrics and human evaluations remains unclear, raising doubts about their effectiveness as reliable benchmarks. Recent advancements in Large Language Models (LLMs) have demonstrated their ability to not only match but also surpass the average human performance across a wide range of unseen tasks. This suggests that LLMs have the potential to serve as a viable alternative to human evaluation and other automated metrics. We assess the performance of different LLMs on TST evaluation by employing multiple input prompts and comparing their results. Our findings indicate that (even zero-shot) prompting correlates strongly with human evaluation and often surpasses the performance of (other) automated metrics. Additionally, we propose the ensembling of prompts and show it increases the robustness of TST evaluation.This work contributes to the ongoing efforts in evaluating LLMs on diverse tasks, which includes a discussion of failure cases and limitations.

Title: Adversarial Fine-Tuning of Language Models: An Iterative Optimisation Approach for the Generation and Detection of Problematic Content. (arXiv:2308.13768v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.13768
• Code URL: null
• Copy Paste: [[2308.13768]] Adversarial Fine-Tuning of Language Models: An Iterative Optimisation Approach for the Generation and Detection of Problematic Content(http://arxiv.org/abs/2308.13768)
• Summary:

  In this paper, we tackle the emerging challenge of unintended harmful content generation in Large Language Models (LLMs) with a novel dual-stage optimisation technique using adversarial fine-tuning. Our two-pronged approach employs an adversarial model, fine-tuned to generate potentially harmful prompts, and a judge model, iteratively optimised to discern these prompts. In this adversarial cycle, the two models seek to outperform each other in the prompting phase, generating a dataset of rich examples which are then used for fine-tuning. This iterative application of prompting and fine-tuning allows continuous refinement and improved performance. The performance of our approach is evaluated through classification accuracy on a dataset consisting of problematic prompts not detected by GPT-4, as well as a selection of contentious but unproblematic prompts. We show considerable increase in classification accuracy of the judge model on this challenging dataset as it undergoes the optimisation process. Furthermore, we show that a rudimentary model \texttt{ada} can achieve 13\% higher accuracy on the hold-out test set than GPT-4 after only a few rounds of this process, and that this fine-tuning improves performance in parallel tasks such as toxic comment identification.

Title: Solving Math Word Problem with Problem Type Classification. (arXiv:2308.13844v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.13844
• Code URL: https://github.com/zhouzihao501/nlpcc2023-shared-task3-chinesemwp
• Copy Paste: [[2308.13844]] Solving Math Word Problem with Problem Type Classification(http://arxiv.org/abs/2308.13844)
• Summary:

  Math word problems (MWPs) require analyzing text descriptions and generating mathematical equations to derive solutions. Existing works focus on solving MWPs with two types of solvers: tree-based solver and large language model (LLM) solver. However, these approaches always solve MWPs by a single solver, which will bring the following problems: (1) Single type of solver is hard to solve all types of MWPs well. (2) A single solver will result in poor performance due to over-fitting. To address these challenges, this paper utilizes multiple ensemble approaches to improve MWP-solving ability. Firstly, We propose a problem type classifier that combines the strengths of the tree-based solver and the LLM solver. This ensemble approach leverages their respective advantages and broadens the range of MWPs that can be solved. Furthermore, we also apply ensemble techniques to both tree-based solver and LLM solver to improve their performance. For the tree-based solver, we propose an ensemble learning framework based on ten-fold cross-validation and voting mechanism. In the LLM solver, we adopt self-consistency (SC) method to improve answer selection. Experimental results demonstrate the effectiveness of these ensemble approaches in enhancing MWP-solving ability. The comprehensive evaluation showcases improved performance, validating the advantages of our proposed approach. Our code is available at this url: https://github.com/zhouzihao501/NLPCC2023-Shared-Task3-ChineseMWP.

Title: Exploring Large Language Models for Knowledge Graph Completion. (arXiv:2308.13916v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.13916
• Code URL: https://github.com/yao8839836/kg-llm
• Copy Paste: [[2308.13916]] Exploring Large Language Models for Knowledge Graph Completion(http://arxiv.org/abs/2308.13916)
• Summary:

  Knowledge graphs play a vital role in numerous artificial intelligence tasks, yet they frequently face the issue of incompleteness. In this study, we explore utilizing Large Language Models (LLM) for knowledge graph completion. We consider triples in knowledge graphs as text sequences and introduce an innovative framework called Knowledge Graph LLM (KG-LLM) to model these triples. Our technique employs entity and relation descriptions of a triple as prompts and utilizes the response for predictions. Experiments on various benchmark knowledge graphs demonstrate that our method attains state-of-the-art performance in tasks such as triple classification and relation prediction. We also find that fine-tuning relatively smaller models (e.g., LLaMA-7B, ChatGLM-6B) outperforms recent ChatGPT and GPT-4.

segmentation

Title: PE-MED: Prompt Enhancement for Interactive Medical Image Segmentation. (arXiv:2308.13746v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13746
• Code URL: null
• Copy Paste: [[2308.13746]] PE-MED: Prompt Enhancement for Interactive Medical Image Segmentation(http://arxiv.org/abs/2308.13746)
• Summary:

  Interactive medical image segmentation refers to the accurate segmentation of the target of interest through interaction (e.g., click) between the user and the image. It has been widely studied in recent years as it is less dependent on abundant annotated data and more flexible than fully automated segmentation. However, current studies have not fully explored user-provided prompt information (e.g., points), including the knowledge mined in one interaction, and the relationship between multiple interactions. Thus, in this paper, we introduce a novel framework equipped with prompt enhancement, called PE-MED, for interactive medical image segmentation. First, we introduce a Self-Loop strategy to generate warm initial segmentation results based on the first prompt. It can prevent the highly unfavorable scenarios, such as encountering a blank mask as the initial input after the first interaction. Second, we propose a novel Prompt Attention Learning Module (PALM) to mine useful prompt information in one interaction, enhancing the responsiveness of the network to user clicks. Last, we build a Time Series Information Propagation (TSIP) mechanism to extract the temporal relationships between multiple interactions and increase the model stability. Comparative experiments with other state-of-the-art (SOTA) medical image segmentation algorithms show that our method exhibits better segmentation accuracy and stability.

Title: SamDSK: Combining Segment Anything Model with Domain-Specific Knowledge for Semi-Supervised Learning in Medical Image Segmentation. (arXiv:2308.13759v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13759
• Code URL: https://github.com/yizhezhang2000/samdsk
• Copy Paste: [[2308.13759]] SamDSK: Combining Segment Anything Model with Domain-Specific Knowledge for Semi-Supervised Learning in Medical Image Segmentation(http://arxiv.org/abs/2308.13759)
• Summary:

  The Segment Anything Model (SAM) exhibits a capability to segment a wide array of objects in natural images, serving as a versatile perceptual tool for various downstream image segmentation tasks. In contrast, medical image segmentation tasks often rely on domain-specific knowledge (DSK). In this paper, we propose a novel method that combines the segmentation foundation model (i.e., SAM) with domain-specific knowledge for reliable utilization of unlabeled images in building a medical image segmentation model. Our new method is iterative and consists of two main stages: (1) segmentation model training; (2) expanding the labeled set by using the trained segmentation model, an unlabeled set, SAM, and domain-specific knowledge. These two stages are repeated until no more samples are added to the labeled set. A novel optimal-matching-based method is developed for combining the SAM-generated segmentation proposals and pixel-level and image-level DSK for constructing annotations of unlabeled images in the iterative stage (2). In experiments, we demonstrate the effectiveness of our proposed method for breast cancer segmentation in ultrasound images, polyp segmentation in endoscopic images, and skin lesion segmentation in dermoscopic images. Our work initiates a new direction of semi-supervised learning for medical image segmentation: the segmentation foundation model can be harnessed as a valuable tool for label-efficient segmentation learning in medical image segmentation.

Title: Zero-Shot Edge Detection with SCESAME: Spectral Clustering-based Ensemble for Segment Anything Model Estimation. (arXiv:2308.13779v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13779
• Code URL: https://github.com/ymgw55/scesame
• Copy Paste: [[2308.13779]] Zero-Shot Edge Detection with SCESAME: Spectral Clustering-based Ensemble for Segment Anything Model Estimation(http://arxiv.org/abs/2308.13779)
• Summary:

  This paper proposes a novel zero-shot edge detection with SCESAME, which stands for Spectral Clustering-based Ensemble for Segment Anything Model Estimation, based on the recently proposed Segment Anything Model (SAM). SAM is a foundation model for segmentation tasks, and one of the interesting applications of SAM is Automatic Mask Generation (AMG), which generates zero-shot segmentation masks of an entire image. AMG can be applied to edge detection, but suffers from the problem of overdetecting edges. Edge detection with SCESAME overcomes this problem by three steps: (1) eliminating small generated masks, (2) combining masks by spectral clustering, taking into account mask positions and overlaps, and (3) removing artifacts after edge detection. We performed edge detection experiments on two datasets, BSDS500 and NYUDv2. Although our zero-shot approach is simple, the experimental results on BSDS500 showed almost identical performance to human performance and CNN-based methods from seven years ago. In the NYUDv2 experiments, it performed almost as well as recent CNN-based methods. These results indicate that our method has the potential to be a strong baseline for future zero-shot edge detection methods. Furthermore, SCESAME is not only applicable to edge detection, but also to other downstream zero-shot tasks.

Title: Beyond One-to-One: Rethinking the Referring Image Segmentation. (arXiv:2308.13853v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13853
• Code URL: https://github.com/toggle1995/ris-dmmi
• Copy Paste: [[2308.13853]] Beyond One-to-One: Rethinking the Referring Image Segmentation(http://arxiv.org/abs/2308.13853)
• Summary:

  Referring image segmentation aims to segment the target object referred by a natural language expression. However, previous methods rely on the strong assumption that one sentence must describe one target in the image, which is often not the case in real-world applications. As a result, such methods fail when the expressions refer to either no objects or multiple objects. In this paper, we address this issue from two perspectives. First, we propose a Dual Multi-Modal Interaction (DMMI) Network, which contains two decoder branches and enables information flow in two directions. In the text-to-image decoder, text embedding is utilized to query the visual feature and localize the corresponding target. Meanwhile, the image-to-text decoder is implemented to reconstruct the erased entity-phrase conditioned on the visual feature. In this way, visual features are encouraged to contain the critical semantic information about target entity, which supports the accurate segmentation in the text-to-image decoder in turn. Secondly, we collect a new challenging but realistic dataset called Ref-ZOM, which includes image-text pairs under different settings. Extensive experiments demonstrate our method achieves state-of-the-art performance on different datasets, and the Ref-ZOM-trained model performs well on various types of text inputs. Codes and datasets are available at https://github.com/toggle1995/RIS-DMMI.

Title: Towards Real Time Egocentric Segment Captioning for The Blind and Visually Impaired in RGB-D Theatre Images. (arXiv:2308.13892v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13892
• Code URL: null
• Copy Paste: [[2308.13892]] Towards Real Time Egocentric Segment Captioning for The Blind and Visually Impaired in RGB-D Theatre Images(http://arxiv.org/abs/2308.13892)
• Summary:

  In recent years, image captioning and segmentation have emerged as crucial tasks in computer vision, with applications ranging from autonomous driving to content analysis. Although multiple solutions have emerged to help blind and visually impaired people move around their environment, few are applications that help them understand and rebuild a scene in their minds through text. Most built models focus on helping users move and avoid obstacles, restricting the number of environments blind and visually impaired people can be in.

  In this paper, we will propose an approach that helps them understand their surroundings using image captioning. The particularity of our research is that we offer them descriptions with positions of regions and objects regarding them (left, right, front), as well as positional relationships between regions, while we aim to give them access to theatre plays by applying the solution to our TS-RGBD dataset.

Title: Semi-Supervised Semantic Segmentation via Marginal Contextual Information. (arXiv:2308.13900v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13900
• Code URL: https://github.com/s4mcontext/s4mc
• Copy Paste: [[2308.13900]] Semi-Supervised Semantic Segmentation via Marginal Contextual Information(http://arxiv.org/abs/2308.13900)
• Summary:

  We present a novel confidence refinement scheme that enhances pseudo-labels in semi-supervised semantic segmentation. Unlike current leading methods, which filter pixels with low-confidence predictions in isolation, our approach leverages the spatial correlation of labels in segmentation maps by grouping neighboring pixels and considering their pseudo-labels collectively. With this contextual information, our method, named S4MC, increases the amount of unlabeled data used during training while maintaining the quality of the pseudo-labels, all with negligible computational overhead. Through extensive experiments on standard benchmarks, we demonstrate that S4MC outperforms existing state-of-the-art semi-supervised learning approaches, offering a promising solution for reducing the cost of acquiring dense annotations. For example, S4MC achieves a 1.29 mIoU improvement over the prior state-of-the-art method on PASCAL VOC 12 with 366 annotated images. The code to reproduce our experiments is available at https://s4mcontext.github.io/

Title: Sparse Models for Machine Learning. (arXiv:2308.13960v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.13960
• Code URL: null
• Copy Paste: [[2308.13960]] Sparse Models for Machine Learning(http://arxiv.org/abs/2308.13960)
• Summary:

  The sparse modeling is an evident manifestation capturing the parsimony principle just described, and sparse models are widespread in statistics, physics, information sciences, neuroscience, computational mathematics, and so on. In statistics the many applications of sparse modeling span regression, classification tasks, graphical model selection, sparse M-estimators and sparse dimensionality reduction. It is also particularly effective in many statistical and machine learning areas where the primary goal is to discover predictive patterns from data which would enhance our understanding and control of underlying physical, biological, and other natural processes, beyond just building accurate outcome black-box predictors. Common examples include selecting biomarkers in biological procedures, finding relevant brain activity locations which are predictive about brain states and processes based on fMRI data, and identifying network bottlenecks best explaining end-to-end performance. Moreover, the research and applications of efficient recovery of high-dimensional sparse signals from a relatively small number of observations, which is the main focus of compressed sensing or compressive sensing, have rapidly grown and became an extremely intense area of study beyond classical signal processing. Likewise interestingly, sparse modeling is directly related to various artificial vision tasks, such as image denoising, segmentation, restoration and superresolution, object or face detection and recognition in visual scenes, and action recognition.

  In this manuscript, we provide a brief introduction of the basic theory underlying sparse representation and compressive sensing, and then discuss some methods for recovering sparse solutions to optimization problems in effective way, together with some applications of sparse recovery in a machine learning problem known as sparse dictionary learning.

Title: Enhancing Bloodstain Analysis Through AI-Based Segmentation: Leveraging Segment Anything Model for Crime Scene Investigation. (arXiv:2308.13979v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.13979
• Code URL: https://github.com/zdong104/bloodstain_analysis_ai_tool
• Copy Paste: [[2308.13979]] Enhancing Bloodstain Analysis Through AI-Based Segmentation: Leveraging Segment Anything Model for Crime Scene Investigation(http://arxiv.org/abs/2308.13979)
• Summary:

  Bloodstain pattern analysis plays a crucial role in crime scene investigations by providing valuable information through the study of unique blood patterns. Conventional image analysis methods, like Thresholding and Contrast, impose stringent requirements on the image background and is labor-intensive in the context of droplet image segmentation. The Segment Anything Model (SAM), a recently proposed method for extensive image recognition, is yet to be adequately assessed for its accuracy and efficiency on bloodstain image segmentation. This paper explores the application of pre-trained SAM and fine-tuned SAM on bloodstain image segmentation with diverse image backgrounds. Experiment results indicate that both pre-trained and fine-tuned SAM perform the bloodstain image segmentation task with satisfactory accuracy and efficiency, while fine-tuned SAM achieves an overall 2.2\% accuracy improvement than pre-trained SAM and 4.70\% acceleration in terms of speed for image recognition. Analysis of factors that influence bloodstain recognition is carried out. This research demonstrates the potential application of SAM on bloodstain image segmentation, showcasing the effectiveness of Artificial Intelligence application in criminology research. We release all code and demos at \url{https://github.com/Zdong104/Bloodstain_Analysis_Ai_Tool}