secure

Title: Secure and Trustworthy NFC-based Sensor Readout for Battery Packs in Battery Management Systems. (arXiv:2310.10653v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10653
• Code URL: null
• Copy Paste: [[2310.10653]] Secure and Trustworthy NFC-based Sensor Readout for Battery Packs in Battery Management Systems(http://arxiv.org/abs/2310.10653)
• Summary:

  Wireless Battery Management Systems (BMS) are increasingly being considered for modern applications. The ever-increasing complexity and production costs of BMS modules and wired connections resulted in a necessity for new ideas and approaches. Despite this growing trend, there is a lack of generic solutions focused on battery cells' sensor readout, where wireless communication allows for a more flexible and cost-efficient sensor installation in battery packs. Many wireless technologies, such as those that use the 2.4 GHz frequency band, suffer from interference and other limitations. In this article, we present an alternative approach to communication in BMS that relies on the use of Near Field Communication (NFC) technology for battery sensor readouts. As an answer to the rising concern over the counterfeited battery packs, we consider an authentication schema for battery pack validation. We further consider security measures for the processed and stored BMS status data. To show that a general BMS application can make use of our design, we implement a BMS demonstrator using the targeted components. We further test the demonstrator on the technical and functional level, by also performing evaluation on its performance, energy usage, and a security threat model.

Title: Managing Networked IoT Assets Using Practical and Scalable Traffic Inference. (arXiv:2310.10657v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10657
• Code URL: null
• Copy Paste: [[2310.10657]] Managing Networked IoT Assets Using Practical and Scalable Traffic Inference(http://arxiv.org/abs/2310.10657)
• Summary:

  The Internet has recently witnessed unprecedented growth of a class of connected assets called the Internet of Things (IoT). Due to relatively immature manufacturing processes and limited computing resources, IoTs have inadequate device-level security measures, exposing the Internet to various cyber risks. Prior research leveraged predictable patterns in IoT network traffic to develop inference models. However, they fall short of expectations in addressing practical challenges, preventing them from being deployed in production settings. This thesis identifies four practical challenges and develops techniques to address them which can help secure businesses and protect user privacy against growing cyber threats.

  My first contribution balances prediction gains against computing costs of traffic features for IoT traffic classification and monitoring. My second contribution addresses the challenges of measurement costs and data quality. I develop an inference method that uses stochastic and deterministic modeling to predict IoT devices in home networks from opaque and coarse-grained IPFIX flow data. Evaluations show that false positive rates can be reduced by 75% compared to related work without significantly affecting true positives. My third contribution focuses on the challenge of concept drifts by analyzing over six million flow records collected from 12 real home networks. Finally, my fourth contribution studies the resilience of machine learning models against adversarial attacks with a specific focus on decision tree-based models.

security

Title: Identity Prove Limited Information Governance Policy against cyber security persistent threats. (arXiv:2310.10654v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10654
• Code URL: null
• Copy Paste: [[2310.10654]] Identity Prove Limited Information Governance Policy against cyber security persistent threats(http://arxiv.org/abs/2310.10654)
• Summary:

  Identity Prove Limited (IDPL) is a long-founded online identity verification software provider of citizens for Banking services. IDPL applies an information governance based on the ISO/IEC 27001:2022 standard of security and within GDPR to accomplish face verification. The company has a good reputation for biometric authentication services that allow a secure, simple, sustainable online access for financial services providers on delivering security device-independent, ensuring reassurance and convenience to users. The company should ensure a right person, a real person, authenticating in real-time. The IDPL company must assume sustainable security models for the duration of day-to-day operations does not involve human intervention. The IDPL Security Operations Centre (ISOC) should continuously provide the optimum scale of system performance, utilize security procedures against new threats, ensure the optimum scale of system performance capabilities. The aim of information governance policy is to declare and to demonstrate the performance of the company on effectively and efficiently way in front of risk detection and vulnerability mitigation. The scope of this policy involves all management systems and stakeholders details, include unique identifiers of submitter and receiver. The company has in-house systems focused on all potential risks to client data and its information system assets.

Title: Enhancing Trustworthiness in ML-Based Network Intrusion Detection with Uncertainty Quantification. (arXiv:2310.10655v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10655
• Code URL: null
• Copy Paste: [[2310.10655]] Enhancing Trustworthiness in ML-Based Network Intrusion Detection with Uncertainty Quantification(http://arxiv.org/abs/2310.10655)
• Summary:

  The evolution of Internet and its related communication technologies have consistently increased the risk of cyber-attacks. In this context, a crucial role is played by Intrusion Detection Systems (IDSs), which are security devices designed to identify and mitigate attacks to modern networks. In the last decade, data-driven approaches based on Machine Learning (ML) have gained more and more popularity for executing the classification tasks required by IDSs. However, typical ML models adopted for this purpose do not properly take into account the uncertainty associated with their own prediction. This poses significant challenges, as they tend to produce misleadingly high classification scores for both misclassified inputs and inputs belonging to unknown classes (e.g. novel attacks), limiting the trustworthiness of existing ML-based solutions. In this paper we argue that ML-based IDSs should always provide accurate uncertainty quantification to avoid overconfident predictions. In fact, an uncertainty-aware classification would be beneficial to enhance closed-set classification performance, would make it possible to efficiently carry out Active Learning, and would help recognize inputs of unknown classes as truly unknowns (i.e., not belonging to any known class), unlocking open-set classification capabilities and Out-of-Distribution (OoD) detection. To verify it, we compare various ML-based methods for uncertainty quantification and for OoD detection, either specifically designed for or tailored to the domain of network intrusion detection, showing how a proper estimation of the model uncertainty can be exploited to significantly enhance the trustworthiness of ML-based IDSs. Our results also confirm that conventional ML-based approaches to network intrusion detection (e.g. based on traditional feed-forward Neural Networks) may not be appropriate and should be adopted with caution.

Title: Checking and Automating Confidentiality Theory in Isabelle/UTP. (arXiv:2310.10658v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10658
• Code URL: null
• Copy Paste: [[2310.10658]] Checking and Automating Confidentiality Theory in Isabelle/UTP(http://arxiv.org/abs/2310.10658)
• Summary:

  The severity of recent vulnerabilities discovered on modern CPUs, e.g., Spectre [1], highlights how information leakage can have devas-tating effects to the security of computer systems. At the same time, it suggests that confidentiality should be promoted as a normal part of program verification, to discover and mitigate such vulnerabili-ties early in development. The theory we propose is primarily based on Bank's theory [2], a framework for reasoning about confidentiali-ty properties formalised in the Unifying Theories of Programming (UTP) [3]. We mechanised our encoding in the current implementa-tion of UTP in the Isabelle theorem prover, Isabelle/UTP [4]. We have identified some theoretical issues in Bank's original framework. Finally, we demonstrate how our mechanisation can be used to for-mally verify of some of the examples from Bank's work.

Title: TII-SSRC-23 Dataset: Typological Exploration of Diverse Traffic Patterns for Intrusion Detection. (arXiv:2310.10661v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10661
• Code URL: null
• Copy Paste: [[2310.10661]] TII-SSRC-23 Dataset: Typological Exploration of Diverse Traffic Patterns for Intrusion Detection(http://arxiv.org/abs/2310.10661)
• Summary:

  The effectiveness of network intrusion detection systems, predominantly based on machine learning, are highly influenced by the dataset they are trained on. Ensuring an accurate reflection of the multifaceted nature of benign and malicious traffic in these datasets is essential for creating models capable of recognizing and responding to a wide array of intrusion patterns. However, existing datasets often fall short, lacking the necessary diversity and alignment with the contemporary network environment, thereby limiting the effectiveness of intrusion detection. This paper introduces TII-SSRC-23, a novel and comprehensive dataset designed to overcome these challenges. Comprising a diverse range of traffic types and subtypes, our dataset is a robust and versatile tool for the research community. Additionally, we conduct a feature importance analysis, providing vital insights into critical features for intrusion detection tasks. Through extensive experimentation, we also establish firm baselines for supervised and unsupervised intrusion detection methodologies using our dataset, further contributing to the advancement and adaptability of intrusion detection models in the rapidly changing landscape of network security. Our dataset is available at https://kaggle.com/datasets/daniaherzalla/tii-ssrc-23.

Title: Assessing the Influence of Different Types of Probing on Adversarial Decision-Making in a Deception Game. (arXiv:2310.10662v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10662
• Code URL: null
• Copy Paste: [[2310.10662]] Assessing the Influence of Different Types of Probing on Adversarial Decision-Making in a Deception Game(http://arxiv.org/abs/2310.10662)
• Summary:

  Deception, which includes leading cyber-attackers astray with false information, has shown to be an effective method of thwarting cyber-attacks. There has been little investigation of the effect of probing action costs on adversarial decision-making, despite earlier studies on deception in cybersecurity focusing primarily on variables like network size and the percentage of honeypots utilized in games. Understanding human decision-making when prompted with choices of various costs is essential in many areas such as in cyber security. In this paper, we will use a deception game (DG) to examine different costs of probing on adversarial decisions. To achieve this we utilized an IBLT model and a delayed feedback mechanism to mimic knowledge of human actions. Our results were taken from an even split of deception and no deception to compare each influence. It was concluded that probing was slightly taken less as the cost of probing increased. The proportion of attacks stayed relatively the same as the cost of probing increased. Although a constant cost led to a slight decrease in attacks. Overall, our results concluded that the different probing costs do not have an impact on the proportion of attacks whereas it had a slightly noticeable impact on the proportion of probing.

Title: Smart OMVI: Obfuscated Malware Variant Identification using a novel dataset. (arXiv:2310.10670v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10670
• Code URL: null
• Copy Paste: [[2310.10670]] Smart OMVI: Obfuscated Malware Variant Identification using a novel dataset(http://arxiv.org/abs/2310.10670)
• Summary:

  Cybersecurity has become a significant issue in the digital era as a result of the growth in everyday computer use. Cybercriminals now engage in more than virus distribution and computer hacking. Cyberwarfare has developed as a result because it has become a threat to a nation's survival. Malware analysis serves as the first line of defence against an attack and is a significant component of cybercrime. Every day, malware attacks target a large number of computer users, businesses, and governmental agencies, causing billions of dollars in losses. Malware may evade multiple AV software with a very minor, cunning tweak made by its designers, despite the fact that security experts have a variety of tools at their disposal to identify it. To address this challenge, a new dataset called the Obfuscated Malware Dataset (OMD) has been developed. This dataset comprises 40 distinct malware families having 21924 samples, and it incorporates obfuscation techniques that mimic the strategies employed by malware creators to make their malware variations different from the original samples. The purpose of this dataset is to provide a more realistic and representative environment for evaluating the effectiveness of malware analysis techniques. Different conventional machine learning algorithms including but not limited to Support Vector Machine (SVM), Random Forrest (RF), Extreme Gradient Boosting (XGBOOST) etc are applied and contrasted. The results demonstrated that XGBoost outperformed the other algorithms, achieving an accuracy of f 82%, precision of 88%, recall of 80%, and an F1-Score of 83%.

Title: Application-layer Characterization and Traffic Analysis for Encrypted QUIC Transport Protocol. (arXiv:2310.10676v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10676
• Code URL: null
• Copy Paste: [[2310.10676]] Application-layer Characterization and Traffic Analysis for Encrypted QUIC Transport Protocol(http://arxiv.org/abs/2310.10676)
• Summary:

  Quick UDP Internet Connection (QUIC) is an emerging end-to-end encrypted, transport-layer protocol, which has been increasingly adopted by popular web services to improve communication security and quality of experience (QoE) towards end-users. However, this tendency makes the traffic analysis more challenging, given the limited information in the QUIC packet header and full encryption on the payload. To address this challenge, a novel rule-based approach is proposed to estimate the application-level traffic attributes without decrypting QUIC packets. Based on the size, timing, and direction information, our proposed algorithm analyzes the associated network traffic to infer the identity of each HTTP request and response pair, as well as the multiplexing feature in each QUIC connection. The inferred HTTP attributes can be used to evaluate the QoE of application-layer services and identify the service categories for traffic classification in the encrypted QUIC connections.

Title: Security in Cryptocurrency. (arXiv:2310.10768v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10768
• Code URL: null
• Copy Paste: [[2310.10768]] Security in Cryptocurrency(http://arxiv.org/abs/2310.10768)
• Summary:

  This paper discusses the mechanisms of cryptocurrency, the idea of using security in the system, and the popularity of it. To begin, the authors provide a background on cryptocurrency and how it works. The authors understand that while most people may be familiar with the concept, they may not know how it works. Next, the authors discuss the security of cryptocurrency in-depth within the paper. The authors also provide examples of attacks on cryptocurrency systems to show the vulnerabilities within the system. Lastly, the authors discuss the popularity of the system to further express the need for security in cryptocurrency.

Title: Is there a Trojan! : Literature survey and critical evaluation of the latest ML based modern intrusion detection systems in IoT environments. (arXiv:2310.10778v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10778
• Code URL: null
• Copy Paste: [[2310.10778]] Is there a Trojan! : Literature survey and critical evaluation of the latest ML based modern intrusion detection systems in IoT environments(http://arxiv.org/abs/2310.10778)
• Summary:

  IoT as a domain has grown so much in the last few years that it rivals that of the mobile network environments in terms of data volumes as well as cybersecurity threats. The confidentiality and privacy of data within IoT environments have become very important areas of security research within the last few years. More and more security experts are interested in designing robust IDS systems to protect IoT environments as a supplement to the more traditional security methods. Given that IoT devices are resource-constrained and have a heterogeneous protocol stack, most traditional intrusion detection approaches don't work well within these schematic boundaries. This has led security researchers to innovate at the intersection of Machine Learning and IDS to solve the shortcomings of non-learning based IDS systems in the IoT ecosystem.

  Despite various ML algorithms already having high accuracy with IoT datasets, we can see a lack of sufficient production grade models. This survey paper details a comprehensive summary of the latest learning-based approaches used in IoT intrusion detection systems, and conducts a thorough critical review of these systems, potential pitfalls in ML pipelines, challenges from an ML perspective, and discusses future research scope and recommendations.

privacy

Title: VeriDIP: Verifying Ownership of Deep Neural Networks through Privacy Leakage Fingerprints. (arXiv:2310.10656v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10656
• Code URL: null
• Copy Paste: [[2310.10656]] VeriDIP: Verifying Ownership of Deep Neural Networks through Privacy Leakage Fingerprints(http://arxiv.org/abs/2310.10656)
• Summary:

  Deploying Machine Learning as a Service gives rise to model plagiarism, leading to copyright infringement. Ownership testing techniques are designed to identify model fingerprints for verifying plagiarism. However, previous works often rely on overfitting or robustness features as fingerprints, lacking theoretical guarantees and exhibiting under-performance on generalized models. In this paper, we propose a novel ownership testing method called VeriDIP, which verifies a DNN model's intellectual property. VeriDIP makes two major contributions. (1) It utilizes membership inference attacks to estimate the lower bound of privacy leakage, which reflects the fingerprint of a given model. The privacy leakage fingerprints highlight the unique patterns through which the models memorize sensitive training datasets. (2) We introduce a novel approach using less private samples to enhance the performance of ownership testing.

  Extensive experimental results confirm that VeriDIP is effective and efficient in validating the ownership of deep learning models trained on both image and tabular datasets. VeriDIP achieves comparable performance to state-of-the-art methods on image datasets while significantly reducing computation and communication costs. Enhanced VeriDIP demonstrates superior verification performance on generalized deep learning models, particularly on table-trained models. Additionally, VeriDIP exhibits similar effectiveness on utility-preserving differentially private models compared to non-differentially private baselines.

Title: Data privacy for Mobility as a Service. (arXiv:2310.10663v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10663
• Code URL: null
• Copy Paste: [[2310.10663]] Data privacy for Mobility as a Service(http://arxiv.org/abs/2310.10663)
• Summary:

  Mobility as a Service (MaaS) is revolutionizing the transportation industry by offering convenient, efficient and integrated transportation solutions. However, the extensive use of user data as well as the integration of multiple service providers raises significant privacy concerns. The objective of this survey paper is to provide a comprehensive analysis of the current state of data privacy in MaaS, in particular by discussing the associated challenges, existing solutions as well as potential future directions to ensure user privacy while maintaining the benefits of MaaS systems for society.

Title: Privacy Preservation in Artificial Intelligence and Extended Reality (AI-XR) Metaverses: A Survey. (arXiv:2310.10665v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10665
• Code URL: null
• Copy Paste: [[2310.10665]] Privacy Preservation in Artificial Intelligence and Extended Reality (AI-XR) Metaverses: A Survey(http://arxiv.org/abs/2310.10665)
• Summary:

  The metaverse is a nascent concept that envisions a virtual universe, a collaborative space where individuals can interact, create, and participate in a wide range of activities. Privacy in the metaverse is a critical concern as the concept evolves and immersive virtual experiences become more prevalent. The metaverse privacy problem refers to the challenges and concerns surrounding the privacy of personal information and data within Virtual Reality (VR) environments as the concept of a shared VR space becomes more accessible. Metaverse will harness advancements from various technologies such as Artificial Intelligence (AI), Extended Reality (XR), Mixed Reality (MR), and 5G/6G-based communication to provide personalized and immersive services to its users. Moreover, to enable more personalized experiences, the metaverse relies on the collection of fine-grained user data that leads to various privacy issues. Therefore, before the potential of the metaverse can be fully realized, privacy concerns related to personal information and data within VR environments must be addressed. This includes safeguarding users' control over their data, ensuring the security of their personal information, and protecting in-world actions and interactions from unauthorized sharing. In this paper, we explore various privacy challenges that future metaverses are expected to face, given their reliance on AI for tracking users, creating XR and MR experiences, and facilitating interactions. Moreover, we thoroughly analyze technical solutions such as differential privacy, Homomorphic Encryption (HE), and Federated Learning (FL) and discuss related sociotechnical issues regarding privacy.

Title: Locally Differentially Private Graph Embedding. (arXiv:2310.11060v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.11060
• Code URL: null
• Copy Paste: [[2310.11060]] Locally Differentially Private Graph Embedding(http://arxiv.org/abs/2310.11060)
• Summary:

  Graph embedding has been demonstrated to be a powerful tool for learning latent representations for nodes in a graph. However, despite its superior performance in various graph-based machine learning tasks, learning over graphs can raise significant privacy concerns when graph data involves sensitive information. To address this, in this paper, we investigate the problem of developing graph embedding algorithms that satisfy local differential privacy (LDP). We propose LDP-GE, a novel privacy-preserving graph embedding framework, to protect the privacy of node data. Specifically, we propose an LDP mechanism to obfuscate node data and adopt personalized PageRank as the proximity measure to learn node representations. Then, we theoretically analyze the privacy guarantees and utility of the LDP-GE framework. Extensive experiments conducted over several real-world graph datasets demonstrate that LDP-GE achieves favorable privacy-utility trade-offs and significantly outperforms existing approaches in both node classification and link prediction tasks.

protect

defense

Title: Enhancing Network Resilience through Machine Learning-powered Graph Combinatorial Optimization: Applications in Cyber Defense and Information Diffusion. (arXiv:2310.10667v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10667
• Code URL: null
• Copy Paste: [[2310.10667]] Enhancing Network Resilience through Machine Learning-powered Graph Combinatorial Optimization: Applications in Cyber Defense and Information Diffusion(http://arxiv.org/abs/2310.10667)
• Summary:

  With the burgeoning advancements of computing and network communication technologies, network infrastructures and their application environments have become increasingly complex. Due to the increased complexity, networks are more prone to hardware faults and highly susceptible to cyber-attacks. Therefore, for rapidly growing network-centric applications, network resilience is essential to minimize the impact of attacks and to ensure that the network provides an acceptable level of services during attacks, faults or disruptions. In this regard, this thesis focuses on developing effective approaches for enhancing network resilience. Existing approaches for enhancing network resilience emphasize on determining bottleneck nodes and edges in the network and designing proactive responses to safeguard the network against attacks. However, existing solutions generally consider broader application domains and possess limited applicability when applied to specific application areas such as cyber defense and information diffusion, which are highly popular application domains among cyber attackers.

  This thesis aims to design effective, efficient and scalable techniques for discovering bottleneck nodes and edges in the network to enhance network resilience in cyber defense and information diffusion application domains. We first investigate a cyber defense graph optimization problem, i.e., hardening active directory systems by discovering bottleneck edges in the network. We then study the problem of identifying bottleneck structural hole spanner nodes, which are crucial for information diffusion in the network. We transform both problems into graph-combinatorial optimization problems and design machine learning based approaches for discovering bottleneck points vital for enhancing network resilience.

Title: State Machine Frameworks for Website Fingerprinting Defenses: Maybe Not. (arXiv:2310.10789v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10789
• Code URL: null
• Copy Paste: [[2310.10789]] State Machine Frameworks for Website Fingerprinting Defenses: Maybe Not(http://arxiv.org/abs/2310.10789)
• Summary:

  Tor is an anonymity network used by millions of people every day to evade censorship and protect their browsing activity from privacy threats such as mass surveillance. Unfortunately, Tor has been shown to be vulnerable to website fingerprinting attacks, in which an adversary observes the connection between a user and the Tor network and uses features of the encrypted traffic, such as the timing and volume of packets, to identify the websites that are being visited. In response, researchers have proposed a number of defenses against website fingerprinting attacks, and a "circuit padding framework" has been added to the Tor software which supports the implementation of defenses. However, many proposed defenses are not supported by this framework, and no defenses are currently present in Tor.

  As Arti, a reimplementation of Tor in Rust, is being developed, the issue arises of whether a new state machine framework should be included or if alternative models should instead be considered for future defense implementation. We address this question by using an improved Rust-based state machine framework, Maybenot, to implement three state-of-the-art website fingerprinting defenses. Through our evaluation, we demonstrate the potential of state machine frameworks to support effective defenses, and we highlight important features that they should contain to do so. However, our evaluation also raises uncertainty about the long-term feasibility of state machine frameworks for defense implementation. We recommend enhancements to Maybenot and substantial further evaluation, along with consideration of alternative designs, before any decision is made regarding a mechanism for implementing website fingerprinting defenses in Arti.

attack

Title: Generalizability of CNN Architectures for Face Morph Presentation Attack. (arXiv:2310.11105v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.11105
• Code URL: null
• Copy Paste: [[2310.11105]] Generalizability of CNN Architectures for Face Morph Presentation Attack(http://arxiv.org/abs/2310.11105)
• Summary:

  Automatic border control systems are wide spread in modern airports worldwide. Morphing attacks on face biometrics is a serious threat that undermines the security and reliability of face recognition systems deployed in airports and border controls. Therefore, developing a robust Machine Learning (ML) system is necessary to prevent criminals crossing borders with fake identifications especially since it has been shown that security officers cannot detect morphs better than machines. In this study, we investigate the generalization power of Convolutional Neural Network (CNN) architectures against morphing attacks. The investigation utilizes 5 distinct CNNs namely ShuffleNet, DenseNet201, VGG16, EffecientNet-B0 and InceptionResNet-v2. Each CNN architecture represents a well-known family of CNN models in terms of number of parameters, architectural design and performance across various computer vision applications. To ensure robust evaluation, we employ 4 different datasets (Utrecht, London, Defacto and KurdFace) that contain a diverse range of digital face images which cover variations in ethnicity, gender, age, lighting condition and camera setting. One of the fundamental concepts of ML system design is the ability to generalize effectively to previously unseen data, hence not only we evaluate the performance of CNN models within individual datasets but also explore their performance across combined datasets and investigating each dataset in testing phase only. Experimental results on more than 8 thousand images (genuine and morph) from the 4 datasets show that InceptionResNet-v2 generalizes better to unseen data and outperforms the other 4 CNN models.

Title: Fake News in Sheep's Clothing: Robust Fake News Detection Against LLM-Empowered Style Attacks. (arXiv:2310.10830v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10830
• Code URL: null
• Copy Paste: [[2310.10830]] Fake News in Sheep's Clothing: Robust Fake News Detection Against LLM-Empowered Style Attacks(http://arxiv.org/abs/2310.10830)
• Summary:

  It is commonly perceived that online fake news and reliable news exhibit stark differences in writing styles, such as the use of sensationalist versus objective language. However, we emphasize that style-related features can also be exploited for style-based attacks. Notably, the rise of powerful Large Language Models (LLMs) has enabled malicious users to mimic the style of trustworthy news outlets at minimal cost. Our analysis reveals that LLM-camouflaged fake news content leads to substantial performance degradation of state-of-the-art text-based detectors (up to 38% decrease in F1 Score), posing a significant challenge for automated detection in online ecosystems. To address this, we introduce SheepDog, a style-agnostic fake news detector robust to news writing styles. SheepDog achieves this adaptability through LLM-empowered news reframing, which customizes each article to match different writing styles using style-oriented reframing prompts. By employing style-agnostic training, SheepDog enhances its resilience to stylistic variations by maximizing prediction consistency across these diverse reframings. Furthermore, SheepDog extracts content-focused veracity attributions from LLMs, where the news content is evaluated against a set of fact-checking rationales. These attributions provide supplementary information and potential interpretability that assist veracity prediction. On three benchmark datasets, empirical results show that SheepDog consistently yields significant improvements over competitive baselines and enhances robustness against LLM-empowered style attacks.

Title: Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks. (arXiv:2310.10844v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10844
• Code URL: null
• Copy Paste: [[2310.10844]] Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks(http://arxiv.org/abs/2310.10844)
• Summary:

  Large Language Models (LLMs) are swiftly advancing in architecture and capability, and as they integrate more deeply into complex systems, the urgency to scrutinize their security properties grows. This paper surveys research in the emerging interdisciplinary field of adversarial attacks on LLMs, a subfield of trustworthy ML, combining the perspectives of Natural Language Processing and Security. Prior work has shown that even safety-aligned LLMs (via instruction tuning and reinforcement learning through human feedback) can be susceptible to adversarial attacks, which exploit weaknesses and mislead AI systems, as evidenced by the prevalence of `jailbreak' attacks on models like ChatGPT and Bard. In this survey, we first provide an overview of large language models, describe their safety alignment, and categorize existing research based on various learning structures: textual-only attacks, multi-modal attacks, and additional attack methods specifically targeting complex systems, such as federated learning or multi-agent systems. We also offer comprehensive remarks on works that focus on the fundamental sources of vulnerabilities and potential defenses. To make this field more accessible to newcomers, we present a systematic review of existing works, a structured typology of adversarial attack concepts, and additional resources, including slides for presentations on related topics at the 62nd Annual Meeting of the Association for Computational Linguistics (ACL'24).

Title: Backdoor Attack through Machine Unlearning. (arXiv:2310.10659v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10659
• Code URL: https://github.com/seartifacts/bamu
• Copy Paste: [[2310.10659]] Backdoor Attack through Machine Unlearning(http://arxiv.org/abs/2310.10659)
• Summary:

  In recent years, the security issues of artificial intelligence have become increasingly prominent due to the rapid development of deep learning research and applications. Backdoor attack is an attack targeting the vulnerability of deep learning models, where hidden backdoors are activated by triggers embedded by the attacker, thereby outputting malicious predictions that may not align with the intended output for a given input. In this work, we propose a novel black-box backdoor attack based on machine unlearning. The attacker first augments the training set with carefully designed samples, including poison and mitigation data, to train a 'benign' model. Then, the attacker posts unlearning requests for the mitigation samples to remove the impact of relevant data on the model, gradually activating the hidden backdoor. Since backdoors are implanted during the iterative unlearning process, it significantly increases the computational overhead of existing defense methods for backdoor detection or mitigation. To address this new security threat, we propose two methods for detecting or mitigating such malicious unlearning requests. We conduct the experiment in both naive unlearning and SISA settings. Experimental results show that: 1) our attack can successfully implant backdoor into the model, and sharding increases the difficulty of attack; 2) our detection algorithms are effective in identifying the mitigation samples, while sharding reduces the effectiveness of our detection algorithms.

Title: Analysis and Detection against Network Attacks in the Overlapping Phenomenon of Behavior Attribute. (arXiv:2310.10660v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10660
• Code URL: null
• Copy Paste: [[2310.10660]] Analysis and Detection against Network Attacks in the Overlapping Phenomenon of Behavior Attribute(http://arxiv.org/abs/2310.10660)
• Summary:

  The proliferation of network attacks poses a significant threat. Researchers propose datasets for network attacks to support research in related fields. Then, many attack detection methods based on these datasets are proposed. These detection methods, whether two-classification or multi-classification, belong to single-label learning, i.e., only one label is given to each sample. However, we discover that there is a noteworthy phenomenon of behavior attribute overlap between attacks, The presentation of this phenomenon in a dataset is that there are multiple samples with the same features but different labels. In this paper, we verify the phenomenon in well-known datasets(UNSW-NB15, CCCS-CIC-AndMal-2020) and re-label these data. In addition, detecting network attacks in a multi-label manner can obtain more information, providing support for tracing the attack source and building IDS. Therefore, we propose a multi-label detection model based on deep learning, MLD-Model, in which Wasserstein-Generative-Adversarial- Network-with-Gradient-Penalty (WGAN-GP) with improved loss performs data enhancement to alleviate the class imbalance problem, and Auto-Encoder (AE) performs classifier parameter pre-training. Experimental results demonstrate that MLD-Model can achieve excellent classification performance. It can achieve F1=80.06% in UNSW-NB15 and F1=83.63% in CCCS-CIC-AndMal-2020. Especially, MLD-Model is 5.99%-7.97% higher in F1 compared with the related single-label methods.

Title: Extracting Physical Causality from Measurements to Detect and Localize False Data Injection Attacks. (arXiv:2310.10666v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10666
• Code URL: null
• Copy Paste: [[2310.10666]] Extracting Physical Causality from Measurements to Detect and Localize False Data Injection Attacks(http://arxiv.org/abs/2310.10666)
• Summary:

  False Data Injection Attack (FDIA) has become a growing concern in modern cyber-physical power systems. Most existing FDIA detection techniques project the raw measurement data into a high-dimensional latent space to separate normal and attacked samples. These approaches focus more on the statistical correlations of data values and are therefore susceptible to data distribution drifts induced by changes in system operating points or changes in FDIA types and strengths, especially for FDIA localization tasks. Causal inference, on the other hand, extracts the causality behind the coordinated fluctuations of different measurements. The causality patterns are determined by fundamental physical laws such as Ohm's Law and Kirchhoff's Law. They are sensitive to the violation of physical laws caused by FDIA, but tend to remain stable with the drift of system operating points. Leveraging this advantage, this paper proposes a joint FDIA detection and localization framework based on causal inference and the Graph Attention Network (GAT) to identify the attacked system nodes. The proposed framework consists of two levels. The lower level uses the X-learner algorithm to estimate the causality strength between measurements and generate Measurement Causality Graphs (MCGs). The upper level then applies a GAT to identify the anomaly patterns in the MCGs. Since the extracted causality patterns are intrinsically related to the measurements, it is easier for the upper level to figure out the attacked nodes than the existing FDIA localization approaches. The performance of the proposed framework is evaluated on the IEEE 39-bus system. Experimental results show that the causality-based FDIA detection and localization mechanism is highly interpretable and robust.

Title: Demystifying Poisoning Backdoor Attacks from a Statistical Perspective. (arXiv:2310.10780v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10780
• Code URL: null
• Copy Paste: [[2310.10780]] Demystifying Poisoning Backdoor Attacks from a Statistical Perspective(http://arxiv.org/abs/2310.10780)
• Summary:

  The growing dependence on machine learning in real-world applications emphasizes the importance of understanding and ensuring its safety. Backdoor attacks pose a significant security risk due to their stealthy nature and potentially serious consequences. Such attacks involve embedding triggers within a learning model with the intention of causing malicious behavior when an active trigger is present while maintaining regular functionality without it. This paper evaluates the effectiveness of any backdoor attack incorporating a constant trigger, by establishing tight lower and upper boundaries for the performance of the compromised model on both clean and backdoor test data. The developed theory answers a series of fundamental but previously underexplored problems, including (1) what are the determining factors for a backdoor attack's success, (2) what is the direction of the most effective backdoor attack, and (3) when will a human-imperceptible trigger succeed. Our derived understanding applies to both discriminative and generative models. We also demonstrate the theory by conducting experiments using benchmark datasets and state-of-the-art backdoor attack scenarios.

Title: Investigating Threats Posed by SMS Origin Spoofing to IoT Devices. (arXiv:2310.11052v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.11052
• Code URL: null
• Copy Paste: [[2310.11052]] Investigating Threats Posed by SMS Origin Spoofing to IoT Devices(http://arxiv.org/abs/2310.11052)
• Summary:

  The short message service (SMS) is a service for exchanging texts via mobile networks that has been developed not only as a means of text communication between subscribers but also as a means to remotely manage Internet of Things (IoT) devices. However, the originating number of an SMS can be spoofed. If IoT devices authenticate administrators based on the originating number of an SMS, the authentication is bypassed via SMS origin spoofing. Consequently, IoT devices are at risk of accepting commands from attackers and performing unauthorized actions. Accordingly, in this study, the specifications of major cellular IoT gateways were evaluated by focusing on remote management via SMS, and the authentication bypass hypothesis was verified. The results showed that 25 of the 32 targeted products supported SMS-based remote management, and 20 implemented authentication based on the originating number of the SMS. Furthermore, by spoofing the originating number of the SMS, one product was demonstrated to be remotely exploitable through authentication bypassing. Thus, this study revealed the threats posed by SMS origin spoofing to IoT devices and proved that SMS origin spoofing not only threatens text communication between people but also puts machine communication at risk.

Title: Fast Adversarial Label-Flipping Attack on Tabular Data. (arXiv:2310.10744v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10744
• Code URL: null
• Copy Paste: [[2310.10744]] Fast Adversarial Label-Flipping Attack on Tabular Data(http://arxiv.org/abs/2310.10744)
• Summary:

  Machine learning models are increasingly used in fields that require high reliability such as cybersecurity. However, these models remain vulnerable to various attacks, among which the adversarial label-flipping attack poses significant threats. In label-flipping attacks, the adversary maliciously flips a portion of training labels to compromise the machine learning model. This paper raises significant concerns as these attacks can camouflage a highly skewed dataset as an easily solvable classification problem, often misleading machine learning practitioners into lower defenses and miscalculations of potential risks. This concern amplifies in tabular data settings, where identifying true labels requires expertise, allowing malicious label-flipping attacks to easily slip under the radar. To demonstrate this risk is inherited in the adversary's objective, we propose FALFA (Fast Adversarial Label-Flipping Attack), a novel efficient attack for crafting adversarial labels. FALFA is based on transforming the adversary's objective and employs linear programming to reduce computational complexity. Using ten real-world tabular datasets, we demonstrate FALFA's superior attack potential, highlighting the need for robust defenses against such threats.

robust

Title: BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeys. (arXiv:2310.10765v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.10765
• Code URL: null
• Copy Paste: [[2310.10765]] BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeys(http://arxiv.org/abs/2310.10765)
• Summary:

  Rapid progress has been made in instruction-learning for image editing with natural-language instruction, as exemplified by InstructPix2Pix. In biomedicine, such methods can be applied to counterfactual image generation, which helps differentiate causal structure from spurious correlation and facilitate robust image interpretation for disease progression modeling. However, generic image-editing models are ill-suited for the biomedical domain, and counterfactual biomedical image generation is largely underexplored. In this paper, we present BiomedJourney, a novel method for counterfactual biomedical image generation by instruction-learning from multimodal patient journeys. Given a patient with two biomedical images taken at different time points, we use GPT-4 to process the corresponding imaging reports and generate a natural language description of disease progression. The resulting triples (prior image, progression description, new image) are then used to train a latent diffusion model for counterfactual biomedical image generation. Given the relative scarcity of image time series data, we introduce a two-stage curriculum that first pretrains the denoising network using the much more abundant single image-report pairs (with dummy prior image), and then continues training using the counterfactual triples. Experiments using the standard MIMIC-CXR dataset demonstrate the promise of our method. In a comprehensive battery of tests on counterfactual medical image generation, BiomedJourney substantially outperforms prior state-of-the-art methods in instruction image editing and medical image generation such as InstructPix2Pix and RoentGen. To facilitate future study in counterfactual medical generation, we plan to release our instruction-learning code and pretrained models.

Title: Domain Generalization Using Large Pretrained Models with Mixture-of-Adapters. (arXiv:2310.11031v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.11031
• Code URL: null
• Copy Paste: [[2310.11031]] Domain Generalization Using Large Pretrained Models with Mixture-of-Adapters(http://arxiv.org/abs/2310.11031)
• Summary:

  Learning a robust vision model despite large distribution shift is essential for model deployment in real-world settings. Especially, domain generalization (DG) algorithm aims to maintain the performance of a trained model on different distributions which were not seen during training. One of the most effective methods has been leveraging the already learned rich knowledge of large pretrained models. However, naively fine-tuning large models to DG tasks is often practically infeasible due to memory limitations, extensive time requirements for training, and the risk of learned knowledge deterioration. Recently, parameter-efficient fine-tuning (PEFT) methods have been proposed to reduce the high computational cost during training and efficiently adapt large models to downstream tasks. In this work, for the first time, we find that the use of adapters in PEFT methods not only reduce high computational cost during training but also serve as an effective regularizer for DG tasks. Surprisingly, a naive adapter implementation for large models achieve superior performance on common datasets. However, in situations of large distribution shifts, additional factors such as optimal amount of regularization due to the strength of distribution shifts should be considered for a sophisticated adapter implementation. To address this, we propose a mixture-of-expert based adapter fine-tuning method, dubbed as mixture-of-adapters (MoA). Specifically, we employ multiple adapters that have varying capacities, and by using learnable routers, we allocate each token to a proper adapter. By using both PEFT and MoA methods, we effectively alleviate the performance deterioration caused by distribution shifts and achieve state-of-the-art performance on diverse DG benchmarks.

Title: DORec: Decomposed Object Reconstruction Utilizing 2D Self-Supervised Features. (arXiv:2310.11092v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.11092
• Code URL: null
• Copy Paste: [[2310.11092]] DORec: Decomposed Object Reconstruction Utilizing 2D Self-Supervised Features(http://arxiv.org/abs/2310.11092)
• Summary:

  Decomposing a target object from a complex background while reconstructing is challenging. Most approaches acquire the perception for object instances through the use of manual labels, but the annotation procedure is costly. The recent advancements in 2D self-supervised learning have brought new prospects to object-aware representation, yet it remains unclear how to leverage such noisy 2D features for clean decomposition. In this paper, we propose a Decomposed Object Reconstruction (DORec) network based on neural implicit representations. Our key idea is to transfer 2D self-supervised features into masks of two levels of granularity to supervise the decomposition, including a binary mask to indicate the foreground regions and a K-cluster mask to indicate the semantically similar regions. These two masks are complementary to each other and lead to robust decomposition. Experimental results show the superiority of DORec in segmenting and reconstructing the foreground object on various datasets.

Title: SODA: Robust Training of Test-Time Data Adaptors. (arXiv:2310.11093v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11093
• Code URL: https://github.com/tmlr-group/soda
• Copy Paste: [[2310.11093]] SODA: Robust Training of Test-Time Data Adaptors(http://arxiv.org/abs/2310.11093)
• Summary:

  Adapting models deployed to test distributions can mitigate the performance degradation caused by distribution shifts. However, privacy concerns may render model parameters inaccessible. One promising approach involves utilizing zeroth-order optimization (ZOO) to train a data adaptor to adapt the test data to fit the deployed models. Nevertheless, the data adaptor trained with ZOO typically brings restricted improvements due to the potential corruption of data features caused by the data adaptor. To address this issue, we revisit ZOO in the context of test-time data adaptation. We find that the issue directly stems from the unreliable estimation of the gradients used to optimize the data adaptor, which is inherently due to the unreliable nature of the pseudo-labels assigned to the test data. Based on this observation, we propose pseudo-label-robust data adaptation (SODA) to improve the performance of data adaptation. Specifically, SODA leverages high-confidence predicted labels as reliable labels to optimize the data adaptor with ZOO for label prediction. For data with low-confidence predictions, SODA encourages the adaptor to preserve data information to mitigate data corruption. Empirical results indicate that SODA can significantly enhance the performance of deployed models in the presence of distribution shifts without requiring access to model parameters.

Title: Sparse Multi-Object Render-and-Compare. (arXiv:2310.11184v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.11184
• Code URL: null
• Copy Paste: [[2310.11184]] Sparse Multi-Object Render-and-Compare(http://arxiv.org/abs/2310.11184)
• Summary:

  Reconstructing 3D shape and pose of static objects from a single image is an essential task for various industries, including robotics, augmented reality, and digital content creation. This can be done by directly predicting 3D shape in various representations or by retrieving CAD models from a database and predicting their alignments. Directly predicting 3D shapes often produces unrealistic, overly smoothed or tessellated shapes. Retrieving CAD models ensures realistic shapes but requires robust and accurate alignment. Learning to directly predict CAD model poses from image features is challenging and inaccurate. Works, such as ROCA, compute poses from predicted normalised object coordinates which can be more accurate but are susceptible to systematic failure. SPARC demonstrates that following a ''render-and-compare'' approach where a network iteratively improves upon its own predictions achieves accurate alignments. Nevertheless, it performs individual CAD alignment for every object detected in an image. This approach is slow when applied to many objects as the time complexity increases linearly with the number of objects and can not learn inter-object relations. Introducing a new network architecture Multi-SPARC we learn to perform CAD model alignments for multiple detected objects jointly. Compared to other single-view methods we achieve state-of-the-art performance on the challenging real-world dataset ScanNet. By improving the instance alignment accuracy from 31.8% to 40.3% we perform similar to state-of-the-art multi-view methods.

Title: Intent Detection and Slot Filling for Home Assistants: Dataset and Analysis for Bangla and Sylheti. (arXiv:2310.10935v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10935
• Code URL: null
• Copy Paste: [[2310.10935]] Intent Detection and Slot Filling for Home Assistants: Dataset and Analysis for Bangla and Sylheti(http://arxiv.org/abs/2310.10935)
• Summary:

  As voice assistants cement their place in our technologically advanced society, there remains a need to cater to the diverse linguistic landscape, including colloquial forms of low-resource languages. Our study introduces the first-ever comprehensive dataset for intent detection and slot filling in formal Bangla, colloquial Bangla, and Sylheti languages, totaling 984 samples across 10 unique intents. Our analysis reveals the robustness of large language models for tackling downstream tasks with inadequate data. The GPT-3.5 model achieves an impressive F1 score of 0.94 in intent detection and 0.51 in slot filling for colloquial Bangla.

Title: A State-Vector Framework for Dataset Effects. (arXiv:2310.10955v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10955
• Code URL: https://github.com/esmatsahak/emnlp-2023_a-state-vector-framework-for-dataset-effects_repository
• Copy Paste: [[2310.10955]] A State-Vector Framework for Dataset Effects(http://arxiv.org/abs/2310.10955)
• Summary:

  The impressive success of recent deep neural network (DNN)-based systems is significantly influenced by the high-quality datasets used in training. However, the effects of the datasets, especially how they interact with each other, remain underexplored. We propose a state-vector framework to enable rigorous studies in this direction. This framework uses idealized probing test results as the bases of a vector space. This framework allows us to quantify the effects of both standalone and interacting datasets. We show that the significant effects of some commonly-used language understanding datasets are characteristic and are concentrated on a few linguistic dimensions. Additionally, we observe some ``spill-over'' effects: the datasets could impact the models along dimensions that may seem unrelated to the intended tasks. Our state-vector framework paves the way for a systematic understanding of the dataset effects, a crucial component in responsible and robust model development.

Title: VoxArabica: A Robust Dialect-Aware Arabic Speech Recognition System. (arXiv:2310.11069v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11069
• Code URL: null
• Copy Paste: [[2310.11069]] VoxArabica: A Robust Dialect-Aware Arabic Speech Recognition System(http://arxiv.org/abs/2310.11069)
• Summary:

  Arabic is a complex language with many varieties and dialects spoken by over 450 millions all around the world. Due to the linguistic diversity and variations, it is challenging to build a robust and generalized ASR system for Arabic. In this work, we address this gap by developing and demoing a system, dubbed VoxArabica, for dialect identification (DID) as well as automatic speech recognition (ASR) of Arabic. We train a wide range of models such as HuBERT (DID), Whisper, and XLS-R (ASR) in a supervised setting for Arabic DID and ASR tasks. Our DID models are trained to identify 17 different dialects in addition to MSA. We finetune our ASR models on MSA, Egyptian, Moroccan, and mixed data. Additionally, for the remaining dialects in ASR, we provide the option to choose various models such as Whisper and MMS in a zero-shot setting. We integrate these models into a single web interface with diverse features such as audio recording, file upload, model selection, and the option to raise flags for incorrect outputs. Overall, we believe VoxArabica will be useful for a wide range of audiences concerned with Arabic research. Our system is currently running at https://cdce-206-12-100-168.ngrok.io/.

Title: Robust Collaborative Filtering to Popularity Distribution Shift. (arXiv:2310.10696v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10696
• Code URL: https://github.com/anzhang314/popgo
• Copy Paste: [[2310.10696]] Robust Collaborative Filtering to Popularity Distribution Shift(http://arxiv.org/abs/2310.10696)
• Summary:

  In leading collaborative filtering (CF) models, representations of users and items are prone to learn popularity bias in the training data as shortcuts. The popularity shortcut tricks are good for in-distribution (ID) performance but poorly generalized to out-of-distribution (OOD) data, i.e., when popularity distribution of test data shifts w.r.t. the training one. To close the gap, debiasing strategies try to assess the shortcut degrees and mitigate them from the representations. However, there exist two deficiencies: (1) when measuring the shortcut degrees, most strategies only use statistical metrics on a single aspect (i.e., item frequency on item and user frequency on user aspect), failing to accommodate the compositional degree of a user-item pair; (2) when mitigating shortcuts, many strategies assume that the test distribution is known in advance. This results in low-quality debiased representations. Worse still, these strategies achieve OOD generalizability with a sacrifice on ID performance. In this work, we present a simple yet effective debiasing strategy, PopGo, which quantifies and reduces the interaction-wise popularity shortcut without any assumptions on the test data. It first learns a shortcut model, which yields a shortcut degree of a user-item pair based on their popularity representations. Then, it trains the CF model by adjusting the predictions with the interaction-wise shortcut degrees. By taking both causal- and information-theoretical looks at PopGo, we can justify why it encourages the CF model to capture the critical popularity-agnostic features while leaving the spurious popularity-relevant patterns out. We use PopGo to debias two high-performing CF models (MF, LightGCN) on four benchmark datasets. On both ID and OOD test sets, PopGo achieves significant gains over the state-of-the-art debiasing strategies (e.g., DICE, MACR).

Title: Mori-Zwanzig latent space Koopman closure for nonlinear autoencoder. (arXiv:2310.10745v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10745
• Code URL: null
• Copy Paste: [[2310.10745]] Mori-Zwanzig latent space Koopman closure for nonlinear autoencoder(http://arxiv.org/abs/2310.10745)
• Summary:

  The Koopman operator presents an attractive approach to achieve global linearization of nonlinear systems, making it a valuable method for simplifying the understanding of complex dynamics. While data-driven methodologies have exhibited promise in approximating finite Koopman operators, they grapple with various challenges, such as the judicious selection of observables, dimensionality reduction, and the ability to predict complex system behaviours accurately. This study presents a novel approach termed Mori-Zwanzig autoencoder (MZ-AE) to robustly approximate the Koopman operator in low-dimensional spaces. The proposed method leverages a nonlinear autoencoder to extract key observables for approximating a finite invariant Koopman subspace and integrates a non-Markovian correction mechanism using the Mori-Zwanzig formalism. Consequently, this approach yields a closed representation of dynamics within the latent manifold of the nonlinear autoencoder, thereby enhancing the precision and stability of the Koopman operator approximation. Demonstrations showcase the technique's ability to capture regime transitions in the flow around a circular cylinder. It also provided a low dimensional approximation for chaotic Kuramoto-Sivashinsky with promising short-term predictability and robust long-term statistical performance. By bridging the gap between data-driven techniques and the mathematical foundations of Koopman theory, MZ-AE offers a promising avenue for improved understanding and prediction of complex nonlinear dynamics.

Title: Gotta be SAFE: A New Framework for Molecular Design. (arXiv:2310.10773v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10773
• Code URL: null
• Copy Paste: [[2310.10773]] Gotta be SAFE: A New Framework for Molecular Design(http://arxiv.org/abs/2310.10773)
• Summary:

  Traditional molecular string representations, such as SMILES, often pose challenges for AI-driven molecular design due to their non-sequential depiction of molecular substructures. To address this issue, we introduce Sequential Attachment-based Fragment Embedding (SAFE), a novel line notation for chemical structures. SAFE reimagines SMILES strings as an unordered sequence of interconnected fragment blocks while maintaining full compatibility with existing SMILES parsers. It streamlines complex generative tasks, including scaffold decoration, fragment linking, polymer generation, and scaffold hopping, while facilitating autoregressive generation for fragment-constrained design, thereby eliminating the need for intricate decoding or graph-based models. We demonstrate the effectiveness of SAFE by training an 87-million-parameter GPT2-like model on a dataset containing 1.1 billion SAFE representations. Through extensive experimentation, we show that our SAFE-GPT model exhibits versatile and robust optimization performance. SAFE opens up new avenues for the rapid exploration of chemical space under various constraints, promising breakthroughs in AI-driven molecular design.

Title: Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms. (arXiv:2310.10810v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10810
• Code URL: https://github.com/abukharin3/ernie
• Copy Paste: [[2310.10810]] Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithms(http://arxiv.org/abs/2310.10810)
• Summary:

  Multi-Agent Reinforcement Learning (MARL) has shown promising results across several domains. Despite this promise, MARL policies often lack robustness and are therefore sensitive to small changes in their environment. This presents a serious concern for the real world deployment of MARL algorithms, where the testing environment may slightly differ from the training environment. In this work we show that we can gain robustness by controlling a policy's Lipschitz constant, and under mild conditions, establish the existence of a Lipschitz and close-to-optimal policy. Based on these insights, we propose a new robust MARL framework, ERNIE, that promotes the Lipschitz continuity of the policies with respect to the state observations and actions by adversarial regularization. The ERNIE framework provides robustness against noisy observations, changing transition dynamics, and malicious actions of agents. However, ERNIE's adversarial regularization may introduce some training instability. To reduce this instability, we reformulate adversarial regularization as a Stackelberg game. We demonstrate the effectiveness of the proposed framework with extensive experiments in traffic light control and particle environments. In addition, we extend ERNIE to mean-field MARL with a formulation based on distributionally robust optimization that outperforms its non-robust counterpart and is of independent interest. Our code is available at https://github.com/abukharin3/ERNIE.

Title: Proper Laplacian Representation Learning. (arXiv:2310.10833v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10833
• Code URL: null
• Copy Paste: [[2310.10833]] Proper Laplacian Representation Learning(http://arxiv.org/abs/2310.10833)
• Summary:

  The ability to learn good representations of states is essential for solving large reinforcement learning problems, where exploration, generalization, and transfer are particularly challenging. The Laplacian representation is a promising approach to address these problems by inducing intrinsic rewards for temporally-extended action discovery and reward shaping, and informative state encoding. To obtain the Laplacian representation one needs to compute the eigensystem of the graph Laplacian, which is often approximated through optimization objectives compatible with deep learning approaches. These approximations, however, depend on hyperparameters that are impossible to tune efficiently, converge to arbitrary rotations of the desired eigenvectors, and are unable to accurately recover the corresponding eigenvalues. In this paper we introduce a theoretically sound objective and corresponding optimization algorithm for approximating the Laplacian representation. Our approach naturally recovers both the true eigenvectors and eigenvalues while eliminating the hyperparameter dependence of previous approximations. We provide theoretical guarantees for our method and we show that those results translate empirically into robust learning across multiple environments.

Title: SD-PINN: Deep Learning based Spatially Dependent PDEs Recovery. (arXiv:2310.10970v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10970
• Code URL: null
• Copy Paste: [[2310.10970]] SD-PINN: Deep Learning based Spatially Dependent PDEs Recovery(http://arxiv.org/abs/2310.10970)
• Summary:

  The physics-informed neural network (PINN) is capable of recovering partial differential equation (PDE) coefficients that remain constant throughout the spatial domain directly from physical measurements. In this work, we propose a spatially dependent physics-informed neural network (SD-PINN), which enables the recovery of coefficients in spatially-dependent PDEs using a single neural network, eliminating the requirement for domain-specific physical expertise. The proposed method exhibits robustness to noise owing to the incorporation of physical constraints. It can also incorporate the low-rank assumption of the spatial variation for the PDE coefficients to recover the coefficients at locations without available measurements.

Title: Understanding Contrastive Learning via Distributionally Robust Optimization. (arXiv:2310.11048v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11048
• Code URL: https://github.com/junkangwu/ADNCE
• Copy Paste: [[2310.11048]] Understanding Contrastive Learning via Distributionally Robust Optimization(http://arxiv.org/abs/2310.11048)
• Summary:

  This study reveals the inherent tolerance of contrastive learning (CL) towards sampling bias, wherein negative samples may encompass similar semantics (\eg labels). However, existing theories fall short in providing explanations for this phenomenon. We bridge this research gap by analyzing CL through the lens of distributionally robust optimization (DRO), yielding several key insights: (1) CL essentially conducts DRO over the negative sampling distribution, thus enabling robust performance across a variety of potential distributions and demonstrating robustness to sampling bias; (2) The design of the temperature $\tau$ is not merely heuristic but acts as a Lagrange Coefficient, regulating the size of the potential distribution set; (3) A theoretical connection is established between DRO and mutual information, thus presenting fresh evidence for ``InfoNCE as an estimate of MI'' and a new estimation approach for $\phi$-divergence-based generalized mutual information. We also identify CL's potential shortcomings, including over-conservatism and sensitivity to outliers, and introduce a novel Adjusted InfoNCE loss (ADNCE) to mitigate these issues. It refines potential distribution, improving performance and accelerating convergence. Extensive experiments on various domains (image, sentence, and graphs) validate the effectiveness of the proposal. The code is available at \url{https://github.com/junkangwu/ADNCE}.

Title: Non-parametric Conditional Independence Testing for Mixed Continuous-Categorical Variables: A Novel Method and Numerical Evaluation. (arXiv:2310.11132v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11132
• Code URL: null
• Copy Paste: [[2310.11132]] Non-parametric Conditional Independence Testing for Mixed Continuous-Categorical Variables: A Novel Method and Numerical Evaluation(http://arxiv.org/abs/2310.11132)
• Summary:

  Conditional independence testing (CIT) is a common task in machine learning, e.g., for variable selection, and a main component of constraint-based causal discovery. While most current CIT approaches assume that all variables are numerical or all variables are categorical, many real-world applications involve mixed-type datasets that include numerical and categorical variables. Non-parametric CIT can be conducted using conditional mutual information (CMI) estimators combined with a local permutation scheme. Recently, two novel CMI estimators for mixed-type datasets based on k-nearest-neighbors (k-NN) have been proposed. As with any k-NN method, these estimators rely on the definition of a distance metric. One approach computes distances by a one-hot encoding of the categorical variables, essentially treating categorical variables as discrete-numerical, while the other expresses CMI by entropy terms where the categorical variables appear as conditions only. In this work, we study these estimators and propose a variation of the former approach that does not treat categorical variables as numeric. Our numerical experiments show that our variant detects dependencies more robustly across different data distributions and preprocessing types.

Title: Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control. (arXiv:2310.11138v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11138
• Code URL: null
• Copy Paste: [[2310.11138]] Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control(http://arxiv.org/abs/2310.11138)
• Summary:

  The combination of deep reinforcement learning (DRL) with ensemble methods has been proved to be highly effective in addressing complex sequential decision-making problems. This success can be primarily attributed to the utilization of multiple models, which enhances both the robustness of the policy and the accuracy of value function estimation. However, there has been limited analysis of the empirical success of current ensemble RL methods thus far. Our new analysis reveals that the sample efficiency of previous ensemble DRL algorithms may be limited by sub-policies that are not as diverse as they could be. Motivated by these findings, our study introduces a new ensemble RL algorithm, termed \textbf{T}rajectories-awar\textbf{E} \textbf{E}nsemble exploratio\textbf{N} (TEEN). The primary goal of TEEN is to maximize the expected return while promoting more diverse trajectories. Through extensive experiments, we demonstrate that TEEN not only enhances the sample diversity of the ensemble policy compared to using sub-policies alone but also improves the performance over ensemble RL algorithms. On average, TEEN outperforms the baseline ensemble DRL algorithms by 41\% in performance on the tested representative environments.

biometric

steal

extraction

Title: Knowledge Extraction and Distillation from Large-Scale Image-Text Colonoscopy Records Leveraging Large Language and Vision Models. (arXiv:2310.11173v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.11173
• Code URL: null
• Copy Paste: [[2310.11173]] Knowledge Extraction and Distillation from Large-Scale Image-Text Colonoscopy Records Leveraging Large Language and Vision Models(http://arxiv.org/abs/2310.11173)
• Summary:

  The development of artificial intelligence systems for colonoscopy analysis often necessitates expert-annotated image datasets. However, limitations in dataset size and diversity impede model performance and generalisation. Image-text colonoscopy records from routine clinical practice, comprising millions of images and text reports, serve as a valuable data source, though annotating them is labour-intensive. Here we leverage recent advancements in large language and vision models and propose EndoKED, a data mining paradigm for deep knowledge extraction and distillation. EndoKED automates the transformation of raw colonoscopy records into image datasets with pixel-level annotation. We validate EndoKED using multi-centre datasets of raw colonoscopy records (~1 million images), demonstrating its superior performance in training polyp detection and segmentation models. Furthermore, the EndoKED pre-trained vision backbone enables data-efficient and generalisable learning for optical biopsy, achieving expert-level performance in both retrospective and prospective validation.

Title: Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction. (arXiv:2310.11016v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11016
• Code URL: null
• Copy Paste: [[2310.11016]] Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction(http://arxiv.org/abs/2310.11016)
• Summary:

  Recent advances in multimodal pre-trained models have significantly improved information extraction from visually-rich documents (VrDs), in which named entity recognition (NER) is treated as a sequence-labeling task of predicting the BIO entity tags for tokens, following the typical setting of NLP. However, BIO-tagging scheme relies on the correct order of model inputs, which is not guaranteed in real-world NER on scanned VrDs where text are recognized and arranged by OCR systems. Such reading order issue hinders the accurate marking of entities by BIO-tagging scheme, making it impossible for sequence-labeling methods to predict correct named entities. To address the reading order issue, we introduce Token Path Prediction (TPP), a simple prediction head to predict entity mentions as token sequences within documents. Alternative to token classification, TPP models the document layout as a complete directed graph of tokens, and predicts token paths within the graph as entities. For better evaluation of VrD-NER systems, we also propose two revised benchmark datasets of NER on scanned documents which can reflect real-world scenarios. Experiment results demonstrate the effectiveness of our method, and suggest its potential to be a universal solution to various information extraction tasks on documents.

Title: In-Context Few-Shot Relation Extraction via Pre-Trained Language Models. (arXiv:2310.11085v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11085
• Code URL: https://github.com/oezyurty/replm
• Copy Paste: [[2310.11085]] In-Context Few-Shot Relation Extraction via Pre-Trained Language Models(http://arxiv.org/abs/2310.11085)
• Summary:

  Relation extraction aims at inferring structured human knowledge from textual documents. State-of-the-art methods based on language models commonly have two limitations: (1) they require named entities to be either given as input or infer them, which introduces additional noise, and (2) they require human annotations of documents. As a remedy, we present a novel framework for in-context few-shot relation extraction via pre-trained language models. To the best of our knowledge, we are the first to reformulate the relation extraction task as a tailored in-context few-shot learning paradigm. Thereby, we achieve crucial benefits in that we eliminate the need for both named entity recognition and human annotation of documents. Unlike existing methods based on fine-tuning, our framework is flexible in that it can be easily updated for a new set of relations without re-training. We evaluate our framework using DocRED, the largest publicly available dataset for document-level relation extraction, and demonstrate that our framework achieves state-of-the-art performance. Finally, our framework allows us to identify missing annotations, and we thus show that our framework actually performs much better than the original labels from the development set of DocRED.

Title: Gaussian processes based data augmentation and expected signature for time series classification. (arXiv:2310.10836v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10836
• Code URL: null
• Copy Paste: [[2310.10836]] Gaussian processes based data augmentation and expected signature for time series classification(http://arxiv.org/abs/2310.10836)
• Summary:

  The signature is a fundamental object that describes paths (that is, continuous functions from an interval to a Euclidean space). Likewise, the expected signature provides a statistical description of the law of stochastic processes. We propose a feature extraction model for time series built upon the expected signature. This is computed through a Gaussian processes based data augmentation. One of the main features is that an optimal feature extraction is learnt through the supervised task that uses the model.

membership infer

federate

Title: Pure Exploration in Asynchronous Federated Bandits. (arXiv:2310.11015v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11015
• Code URL: null
• Copy Paste: [[2310.11015]] Pure Exploration in Asynchronous Federated Bandits(http://arxiv.org/abs/2310.11015)
• Summary:

  We study the federated pure exploration problem of multi-armed bandits and linear bandits, where $M$ agents cooperatively identify the best arm via communicating with the central server. To enhance the robustness against latency and unavailability of agents that are common in practice, we propose the first federated asynchronous multi-armed bandit and linear bandit algorithms for pure exploration with fixed confidence. Our theoretical analysis shows the proposed algorithms achieve near-optimal sample complexities and efficient communication costs in a fully asynchronous environment. Moreover, experimental results based on synthetic and real-world data empirically elucidate the effectiveness and communication cost-efficiency of the proposed algorithms.

Title: Federated Learning with Nonvacuous Generalisation Bounds. (arXiv:2310.11203v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11203
• Code URL: null
• Copy Paste: [[2310.11203]] Federated Learning with Nonvacuous Generalisation Bounds(http://arxiv.org/abs/2310.11203)
• Summary:

  We introduce a novel strategy to train randomised predictors in federated learning, where each node of the network aims at preserving its privacy by releasing a local predictor but keeping secret its training dataset with respect to the other nodes. We then build a global randomised predictor which inherits the properties of the local private predictors in the sense of a PAC-Bayesian generalisation bound. We consider the synchronous case where all nodes share the same training objective (derived from a generalisation bound), and the asynchronous case where each node may have its own personalised training objective. We show through a series of numerical experiments that our approach achieves a comparable predictive performance to that of the batch approach where all datasets are shared across nodes. Moreover the predictors are supported by numerically nonvacuous generalisation bounds while preserving privacy for each node. We explicitly compute the increment on predictive performance and generalisation bounds between batch and federated settings, highlighting the price to pay to preserve privacy.

fair

Title: Will the Prince Get True Love's Kiss? On the Model Sensitivity to Gender Perturbation over Fairytale Texts. (arXiv:2310.10865v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10865
• Code URL: null
• Copy Paste: [[2310.10865]] Will the Prince Get True Love's Kiss? On the Model Sensitivity to Gender Perturbation over Fairytale Texts(http://arxiv.org/abs/2310.10865)
• Summary:

  Recent studies show that traditional fairytales are rife with harmful gender biases. To help mitigate these gender biases in fairytales, this work aims to assess learned biases of language models by evaluating their robustness against gender perturbations. Specifically, we focus on Question Answering (QA) tasks in fairytales. Using counterfactual data augmentation to the FairytaleQA dataset, we evaluate model robustness against swapped gender character information, and then mitigate learned biases by introducing counterfactual gender stereotypes during training time. We additionally introduce a novel approach that utilizes the massive vocabulary of language models to support text genres beyond fairytales. Our experimental results suggest that models are sensitive to gender perturbations, with significant performance drops compared to the original testing set. However, when first fine-tuned on a counterfactual training dataset, models are less sensitive to the later introduced anti-gender stereotyped text.

Title: Understanding Fairness Surrogate Functions in Algorithmic Fairness. (arXiv:2310.11211v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11211
• Code URL: null
• Copy Paste: [[2310.11211]] Understanding Fairness Surrogate Functions in Algorithmic Fairness(http://arxiv.org/abs/2310.11211)
• Summary:

  It has been observed that machine learning algorithms exhibit biased predictions against certain population groups. To mitigate such bias while achieving comparable accuracy, a promising approach is to introduce surrogate functions of the concerned fairness definition and solve a constrained optimization problem. However, an intriguing issue in previous work is that such fairness surrogate functions may yield unfair results. In this work, in order to deeply understand this issue, taking a widely used fairness definition, demographic parity as an example, we both theoretically and empirically show that there is a surrogate-fairness gap between the fairness definition and the fairness surrogate function. The "gap" directly determines whether a surrogate function is an appropriate substitute for a fairness definition. Also, the theoretical analysis and experimental results about the "gap" motivate us that the unbounded surrogate functions will be affected by the points far from the decision boundary, which is the large margin points issue investigated in this paper. To address it, we propose the general sigmoid surrogate with a rigorous and reliable fairness guarantee. Interestingly, the theory also provides insights into two important issues that deal with the large margin points as well as obtaining a more balanced dataset are beneficial to fairness. Furthermore, we elaborate a novel and general algorithm called Balanced Surrogate, which iteratively reduces the "gap" to improve fairness. Finally, we provide empirical evidence showing that our methods achieve better fairness performance in three real-world datasets.

interpretability

Title: EXMODD: An EXplanatory Multimodal Open-Domain Dialogue dataset. (arXiv:2310.10967v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10967
• Code URL: https://github.com/poplpr/exmodd
• Copy Paste: [[2310.10967]] EXMODD: An EXplanatory Multimodal Open-Domain Dialogue dataset(http://arxiv.org/abs/2310.10967)
• Summary:

  The need for high-quality data has been a key issue hindering the research of dialogue tasks. Recent studies try to build datasets through manual, web crawling, and large pre-trained models. However, man-made data is expensive and data collected from the internet often includes generic responses, meaningless statements, and toxic dialogues. Automatic data generation through large models is a cost-effective method, but for open-domain multimodal dialogue tasks, there are still three drawbacks: 1) There is currently no open-source large model that can accept multimodal input; 2) The content generated by the model lacks interpretability; 3) The generated data is usually difficult to quality control and require extensive resource to collect. To alleviate the significant human and resource expenditure in data collection, we propose a Multimodal Data Construction Framework (MDCF). MDCF designs proper prompts to spur the large-scale pre-trained language model to generate well-formed and satisfactory content. Additionally, MDCF also automatically provides explanation for a given image and its corresponding dialogue, which can provide a certain degree of interpretability and facilitate manual follow-up quality inspection. Based on this, we release an Explanatory Multimodal Open-Domain dialogue dataset (EXMODD). Experiments indicate a positive correlation between the model's ability to generate accurate understandings and high-quality responses. Our code and data can be found at https://github.com/poplpr/EXMODD.

Title: MST-GAT: A Multimodal Spatial-Temporal Graph Attention Network for Time Series Anomaly Detection. (arXiv:2310.11169v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11169
• Code URL: null
• Copy Paste: [[2310.11169]] MST-GAT: A Multimodal Spatial-Temporal Graph Attention Network for Time Series Anomaly Detection(http://arxiv.org/abs/2310.11169)
• Summary:

  Multimodal time series (MTS) anomaly detection is crucial for maintaining the safety and stability of working devices (e.g., water treatment system and spacecraft), whose data are characterized by multivariate time series with diverse modalities. Although recent deep learning methods show great potential in anomaly detection, they do not explicitly capture spatial-temporal relationships between univariate time series of different modalities, resulting in more false negatives and false positives. In this paper, we propose a multimodal spatial-temporal graph attention network (MST-GAT) to tackle this problem. MST-GAT first employs a multimodal graph attention network (M-GAT) and a temporal convolution network to capture the spatial-temporal correlation in multimodal time series. Specifically, M-GAT uses a multi-head attention module and two relational attention modules (i.e., intra- and inter-modal attention) to model modal correlations explicitly. Furthermore, MST-GAT optimizes the reconstruction and prediction modules simultaneously. Experimental results on four multimodal benchmarks demonstrate that MST-GAT outperforms the state-of-the-art baselines. Further analysis indicates that MST-GAT strengthens the interpretability of detected anomalies by locating the most anomalous univariate time series.

explainability

Title: Nebula: Self-Attention for Dynamic Malware Analysis. (arXiv:2310.10664v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10664
• Code URL: https://github.com/dtrizna/nebula
• Copy Paste: [[2310.10664]] Nebula: Self-Attention for Dynamic Malware Analysis(http://arxiv.org/abs/2310.10664)
• Summary:

  Dynamic analysis enables detecting Windows malware by executing programs in a controlled environment, and storing their actions in log reports. Previous work has started training machine learning models on such reports to perform either malware detection or malware classification. However, most of the approaches (i) have only considered convolutional and long-short term memory networks, (ii) they have been built focusing only on APIs called at runtime, without considering other relevant though heterogeneous sources of information like network and file operations, and (iii) the code and pretrained models are hardly available, hindering reproducibility of results in this research area. In this work, we overcome these limitations by presenting Nebula, a versatile, self-attention transformer-based neural architecture that can generalize across different behavior representations and formats, combining heterogeneous information from dynamic log reports. We show the efficacy of Nebula on three distinct data collections from different dynamic analysis platforms, comparing its performance with previous state-of-the-art models developed for malware detection and classification tasks. We produce an extensive ablation study that showcases how the components of Nebula influence its predictive performance, while enabling it to outperform some competing approaches at very low false positive rates. We conclude our work by inspecting the behavior of Nebula through the application of explainability methods, which highlight that Nebula correctly focuses more on portions of reports that contain malicious activities. We release our code and models at github.com/dtrizna/nebula.

watermark

Title: Watermarking LLMs with Weight Quantization. (arXiv:2310.11237v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11237
• Code URL: https://github.com/twilight92z/quantize-watermark
• Copy Paste: [[2310.11237]] Watermarking LLMs with Weight Quantization(http://arxiv.org/abs/2310.11237)
• Summary:

  Abuse of large language models reveals high risks as large language models are being deployed at an astonishing speed. It is important to protect the model weights to avoid malicious usage that violates licenses of open-source large language models. This paper proposes a novel watermarking strategy that plants watermarks in the quantization process of large language models without pre-defined triggers during inference. The watermark works when the model is used in the fp32 mode and remains hidden when the model is quantized to int8, in this way, the users can only inference the model without further supervised fine-tuning of the model. We successfully plant the watermark into open-source large language model weights including GPT-Neo and LLaMA. We hope our proposed method can provide a potential direction for protecting model weights in the era of large language model applications.

Title: Unbiased Watermark for Large Language Models. (arXiv:2310.10669v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.10669
• Code URL: null
• Copy Paste: [[2310.10669]] Unbiased Watermark for Large Language Models(http://arxiv.org/abs/2310.10669)
• Summary:

  The recent advancements in large language models (LLMs) have sparked a growing apprehension regarding the potential misuse. One approach to mitigating this risk is to incorporate watermarking techniques into LLMs, allowing for the tracking and attribution of model outputs. This study examines a crucial aspect of watermarking: how significantly watermarks impact the quality of model-generated outputs. Previous studies have suggested a trade-off between watermark strength and output quality. However, our research demonstrates that it is possible to integrate watermarks without affecting the output probability distribution with appropriate implementation. We refer to this type of watermark as an unbiased watermark. This has significant implications for the use of LLMs, as it becomes impossible for users to discern whether a service provider has incorporated watermarks or not. Furthermore, the presence of watermarks does not compromise the performance of the model in downstream tasks, ensuring that the overall utility of the language model is preserved. Our findings contribute to the ongoing discussion around responsible AI development, suggesting that unbiased watermarks can serve as an effective means of tracking and attributing model outputs without sacrificing output quality.

diffusion

Title: LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation. (arXiv:2310.10769v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.10769
• Code URL: null
• Copy Paste: [[2310.10769]] LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation(http://arxiv.org/abs/2310.10769)
• Summary:

  With the impressive progress in diffusion-based text-to-image generation, extending such powerful generative ability to text-to-video raises enormous attention. Existing methods either require large-scale text-video pairs and a large number of training resources or learn motions that are precisely aligned with template videos. It is non-trivial to balance a trade-off between the degree of generation freedom and the resource costs for video generation. In our study, we present a few-shot-based tuning framework, LAMP, which enables text-to-image diffusion model Learn A specific Motion Pattern with 8~16 videos on a single GPU. Specifically, we design a first-frame-conditioned pipeline that uses an off-the-shelf text-to-image model for content generation so that our tuned video diffusion model mainly focuses on motion learning. The well-developed text-to-image techniques can provide visually pleasing and diverse content as generation conditions, which highly improves video quality and generation freedom. To capture the features of temporal dimension, we expand the pretrained 2D convolution layers of the T2I model to our novel temporal-spatial motion learning layers and modify the attention blocks to the temporal level. Additionally, we develop an effective inference trick, shared-noise sampling, which can improve the stability of videos with computational costs. Our method can also be flexibly applied to other tasks, e.g. real-world image animation and video editing. Extensive experiments demonstrate that LAMP can effectively learn the motion pattern on limited data and generate high-quality videos. The code and models are available at https://rq-wu.github.io/projects/LAMP.

Title: 3D Structure-guided Network for Tooth Alignment in 2D Photograph. (arXiv:2310.11106v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.11106
• Code URL: null
• Copy Paste: [[2310.11106]] 3D Structure-guided Network for Tooth Alignment in 2D Photograph(http://arxiv.org/abs/2310.11106)
• Summary:

  Orthodontics focuses on rectifying misaligned teeth (i.e., malocclusions), affecting both masticatory function and aesthetics. However, orthodontic treatment often involves complex, lengthy procedures. As such, generating a 2D photograph depicting aligned teeth prior to orthodontic treatment is crucial for effective dentist-patient communication and, more importantly, for encouraging patients to accept orthodontic intervention. In this paper, we propose a 3D structure-guided tooth alignment network that takes 2D photographs as input (e.g., photos captured by smartphones) and aligns the teeth within the 2D image space to generate an orthodontic comparison photograph featuring aesthetically pleasing, aligned teeth. Notably, while the process operates within a 2D image space, our method employs 3D intra-oral scanning models collected in clinics to learn about orthodontic treatment, i.e., projecting the pre- and post-orthodontic 3D tooth structures onto 2D tooth contours, followed by a diffusion model to learn the mapping relationship. Ultimately, the aligned tooth contours are leveraged to guide the generation of a 2D photograph with aesthetically pleasing, aligned teeth and realistic textures. We evaluate our network on various facial photographs, demonstrating its exceptional performance and strong applicability within the orthodontic industry.

Title: BayesDiff: Estimating Pixel-wise Uncertainty in Diffusion via Bayesian Inference. (arXiv:2310.11142v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.11142
• Code URL: null
• Copy Paste: [[2310.11142]] BayesDiff: Estimating Pixel-wise Uncertainty in Diffusion via Bayesian Inference(http://arxiv.org/abs/2310.11142)
• Summary:

  Diffusion models have impressive image generation capability, but low-quality generations still exist, and their identification remains challenging due to the lack of a proper sample-wise metric. To address this, we propose BayesDiff, a pixel-wise uncertainty estimator for generations from diffusion models based on Bayesian inference. In particular, we derive a novel uncertainty iteration principle to characterize the uncertainty dynamics in diffusion, and leverage the last-layer Laplace approximation for efficient Bayesian inference. The estimated pixel-wise uncertainty can not only be aggregated into a sample-wise metric to filter out low-fidelity images but also aids in augmenting successful generations and rectifying artifacts in failed generations in text-to-image tasks. Extensive experiments demonstrate the efficacy of BayesDiff and its promise for practical applications.

Title: Enhancing ML model accuracy for Digital VLSI circuits using diffusion models: A study on synthetic data generation. (arXiv:2310.10691v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10691
• Code URL: null
• Copy Paste: [[2310.10691]] Enhancing ML model accuracy for Digital VLSI circuits using diffusion models: A study on synthetic data generation(http://arxiv.org/abs/2310.10691)
• Summary:

  Generative AI has seen remarkable growth over the past few years, with diffusion models being state-of-the-art for image generation. This study investigates the use of diffusion models in generating artificial data generation for electronic circuits for enhancing the accuracy of subsequent machine learning models in tasks such as performance assessment, design, and testing when training data is usually known to be very limited. We utilize simulations in the HSPICE design environment with 22nm CMOS technology nodes to obtain representative real training data for our proposed diffusion model. Our results demonstrate the close resemblance of synthetic data using diffusion model to real data. We validate the quality of generated data, and demonstrate that data augmentation certainly effective in predictive analysis of VLSI design for digital circuits.

noise learning

data-free

transformer

Title: PELA: Learning Parameter-Efficient Models with Low-Rank Approximation. (arXiv:2310.10700v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.10700
• Code URL: https://github.com/guoyang9/pela
• Copy Paste: [[2310.10700]] PELA: Learning Parameter-Efficient Models with Low-Rank Approximation(http://arxiv.org/abs/2310.10700)
• Summary:

  Applying a pre-trained large model to downstream tasks is prohibitive under resource-constrained conditions. Recent dominant approaches for addressing efficiency issues involve adding a few learnable parameters to the fixed backbone model. This strategy, however, leads to more challenges in loading large models for downstream fine-tuning with limited resources. In this paper, we propose a novel method for increasing the parameter efficiency of pre-trained models by introducing an intermediate pre-training stage. To this end, we first employ low-rank approximation to compress the original large model and then devise a feature distillation module and a weight perturbation regularization module. These modules are specifically designed to enhance the low-rank model. Concretely, we update only the low-rank model while freezing the backbone parameters during pre-training. This allows for direct and efficient utilization of the low-rank model for downstream tasks. The proposed method achieves both efficiencies in terms of required parameters and computation time while maintaining comparable results with minimal modifications to the base architecture. Specifically, when applied to three vision-only and one vision-language Transformer models, our approach often demonstrates a $\sim$0.6 point decrease in performance while reducing the original parameter size by 1/3 to 2/3.

Title: SoybeanNet: Transformer-Based Convolutional Neural Network for Soybean Pod Counting from Unmanned Aerial Vehicle (UAV) Images. (arXiv:2310.10861v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.10861
• Code URL: https://github.com/jiajiali04/soybean-pod-counting-from-uav-images
• Copy Paste: [[2310.10861]] SoybeanNet: Transformer-Based Convolutional Neural Network for Soybean Pod Counting from Unmanned Aerial Vehicle (UAV) Images(http://arxiv.org/abs/2310.10861)
• Summary:

  Soybeans are a critical source of food, protein and oil, and thus have received extensive research aimed at enhancing their yield, refining cultivation practices, and advancing soybean breeding techniques. Within this context, soybean pod counting plays an essential role in understanding and optimizing production. Despite recent advancements, the development of a robust pod-counting algorithm capable of performing effectively in real-field conditions remains a significant challenge This paper presents a pioneering work of accurate soybean pod counting utilizing unmanned aerial vehicle (UAV) images captured from actual soybean fields in Michigan, USA. Specifically, this paper presents SoybeanNet, a novel point-based counting network that harnesses powerful transformer backbones for simultaneous soybean pod counting and localization with high accuracy. In addition, a new dataset of UAV-acquired images for soybean pod counting was created and open-sourced, consisting of 113 drone images with more than 260k manually annotated soybean pods captured under natural lighting conditions. Through comprehensive evaluations, SoybeanNet demonstrated superior performance over five state-of-the-art approaches when tested on the collected images. Remarkably, SoybeanNet achieved a counting accuracy of $84.51\%$ when tested on the testing dataset, attesting to its efficacy in real-world scenarios. The publication also provides both the source code (\url{https://github.com/JiajiaLi04/Soybean-Pod-Counting-from-UAV-Images}) and the labeled soybean dataset (\url{https://www.kaggle.com/datasets/jiajiali/uav-based-soybean-pod-images}), offering a valuable resource for future research endeavors in soybean pod counting and related fields.

Title: USDC: Unified Static and Dynamic Compression for Visual Transformer. (arXiv:2310.11117v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.11117
• Code URL: null
• Copy Paste: [[2310.11117]] USDC: Unified Static and Dynamic Compression for Visual Transformer(http://arxiv.org/abs/2310.11117)
• Summary:

  Visual Transformers have achieved great success in almost all vision tasks, such as classification, detection, and so on. However, the model complexity and the inference speed of the visual transformers hinder their deployments in industrial products. Various model compression techniques focus on directly compressing the visual transformers into a smaller one while maintaining the model performance, however, the performance drops dramatically when the compression ratio is large. Furthermore, several dynamic network techniques have also been applied to dynamically compress the visual transformers to obtain input-adaptive efficient sub-structures during the inference stage, which can achieve a better trade-off between the compression ratio and the model performance. The upper bound of memory of dynamic models is not reduced in the practical deployment since the whole original visual transformer model and the additional control gating modules should be loaded onto devices together for inference. To alleviate two disadvantages of two categories of methods, we propose to unify the static compression and dynamic compression techniques jointly to obtain an input-adaptive compressed model, which can further better balance the total compression ratios and the model performances. Moreover, in practical deployment, the batch sizes of the training and inference stage are usually different, which will cause the model inference performance to be worse than the model training performance, which is not touched by all previous dynamic network papers. We propose a sub-group gates augmentation technique to solve this performance drop problem. Extensive experiments demonstrate the superiority of our method on various baseline visual transformers such as DeiT, T2T-ViT, and so on.

Title: FocDepthFormer: Transformer with LSTM for Depth Estimation from Focus. (arXiv:2310.11178v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.11178
• Code URL: null
• Copy Paste: [[2310.11178]] FocDepthFormer: Transformer with LSTM for Depth Estimation from Focus(http://arxiv.org/abs/2310.11178)
• Summary:

  Depth estimation from focal stacks is a fundamental computer vision problem that aims to infer depth from focus/defocus cues in the image stacks. Most existing methods tackle this problem by applying convolutional neural networks (CNNs) with 2D or 3D convolutions over a set of fixed stack images to learn features across images and stacks. Their performance is restricted due to the local properties of the CNNs, and they are constrained to process a fixed number of stacks consistent in train and inference, limiting the generalization to the arbitrary length of stacks. To handle the above limitations, we develop a novel Transformer-based network, FocDepthFormer, composed mainly of a Transformer with an LSTM module and a CNN decoder. The self-attention in Transformer enables learning more informative features via an implicit non-local cross reference. The LSTM module is learned to integrate the representations across the stack with arbitrary images. To directly capture the low-level features of various degrees of focus/defocus, we propose to use multi-scale convolutional kernels in an early-stage encoder. Benefiting from the design with LSTM, our FocDepthFormer can be pre-trained with abundant monocular RGB depth estimation data for visual pattern capturing, alleviating the demand for the hard-to-collect focal stack data. Extensive experiments on various focal stack benchmark datasets show that our model outperforms the state-of-the-art models on multiple metrics.

Title: BanglaNLP at BLP-2023 Task 1: Benchmarking different Transformer Models for Violence Inciting Text Detection in Bengali. (arXiv:2310.10781v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10781
• Code URL: null
• Copy Paste: [[2310.10781]] BanglaNLP at BLP-2023 Task 1: Benchmarking different Transformer Models for Violence Inciting Text Detection in Bengali(http://arxiv.org/abs/2310.10781)
• Summary:

  This paper presents the system that we have developed while solving this shared task on violence inciting text detection in Bangla. We explain both the traditional and the recent approaches that we have used to make our models learn. Our proposed system helps to classify if the given text contains any threat. We studied the impact of data augmentation when there is a limited dataset available. Our quantitative results show that finetuning a multilingual-e5-base model performed the best in our task compared to other transformer-based architectures. We obtained a macro F1 of 68.11\% in the test set and our performance in this shared task is ranked at 23 in the leaderboard.

Title: CoTFormer: More Tokens With Attention Make Up For Less Depth. (arXiv:2310.10845v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10845
• Code URL: null
• Copy Paste: [[2310.10845]] CoTFormer: More Tokens With Attention Make Up For Less Depth(http://arxiv.org/abs/2310.10845)
• Summary:

  The race to continually develop ever larger and deeper foundational models is underway. However, techniques like the Chain-of-Thought (CoT) method continue to play a pivotal role in achieving optimal downstream performance. In this work, we establish an approximate parallel between using chain-of-thought and employing a deeper transformer. Building on this insight, we introduce CoTFormer, a transformer variant that employs an implicit CoT-like mechanism to achieve capacity comparable to a deeper model. Our empirical findings demonstrate the effectiveness of CoTFormers, as they significantly outperform larger standard transformers.

Title: Enhanced Transformer Architecture for Natural Language Processing. (arXiv:2310.10930v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10930
• Code URL: null
• Copy Paste: [[2310.10930]] Enhanced Transformer Architecture for Natural Language Processing(http://arxiv.org/abs/2310.10930)
• Summary:

  Transformer is a state-of-the-art model in the field of natural language processing (NLP). Current NLP models primarily increase the number of transformers to improve processing performance. However, this technique requires a lot of training resources such as computing capacity. In this paper, a novel structure of Transformer is proposed. It is featured by full layer normalization, weighted residual connection, positional encoding exploiting reinforcement learning, and zero masked self-attention. The proposed Transformer model, which is called Enhanced Transformer, is validated by the bilingual evaluation understudy (BLEU) score obtained with the Multi30k translation dataset. As a result, the Enhanced Transformer achieves 202.96% higher BLEU score as compared to the original transformer with the translation dataset.

Title: Understanding writing style in social media with a supervised contrastively pre-trained transformer. (arXiv:2310.11081v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11081
• Code URL: null
• Copy Paste: [[2310.11081]] Understanding writing style in social media with a supervised contrastively pre-trained transformer(http://arxiv.org/abs/2310.11081)
• Summary:

  Online Social Networks serve as fertile ground for harmful behavior, ranging from hate speech to the dissemination of disinformation. Malicious actors now have unprecedented freedom to misbehave, leading to severe societal unrest and dire consequences, as exemplified by events such as the Capitol assault during the US presidential election and the Antivaxx movement during the COVID-19 pandemic. Understanding online language has become more pressing than ever. While existing works predominantly focus on content analysis, we aim to shift the focus towards understanding harmful behaviors by relating content to their respective authors. Numerous novel approaches attempt to learn the stylistic features of authors in texts, but many of these approaches are constrained by small datasets or sub-optimal training losses. To overcome these limitations, we introduce the Style Transformer for Authorship Representations (STAR), trained on a large corpus derived from public sources of 4.5 x 10^6 authored texts involving 70k heterogeneous authors. Our model leverages Supervised Contrastive Loss to teach the model to minimize the distance between texts authored by the same individual. This author pretext pre-training task yields competitive performance at zero-shot with PAN challenges on attribution and clustering. Additionally, we attain promising results on PAN verification challenges using a single dense layer, with our model serving as an embedding encoder. Finally, we present results from our test partition on Reddit. Using a support base of 8 documents of 512 tokens, we can discern authors from sets of up to 1616 authors with at least 80\% accuracy. We share our pre-trained model at huggingface (https://huggingface.co/AIDA-UPM/star) and our code is available at (https://github.com/jahuerta92/star)

Title: ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing. (arXiv:2310.11166v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11166
• Code URL: null
• Copy Paste: [[2310.11166]] ViSoBERT: A Pre-Trained Language Model for Vietnamese Social Media Text Processing(http://arxiv.org/abs/2310.11166)
• Summary:

  English and Chinese, known as resource-rich languages, have witnessed the strong development of transformer-based language models for natural language processing tasks. Although Vietnam has approximately 100M people speaking Vietnamese, several pre-trained models, e.g., PhoBERT, ViBERT, and vELECTRA, performed well on general Vietnamese NLP tasks, including POS tagging and named entity recognition. These pre-trained language models are still limited to Vietnamese social media tasks. In this paper, we present the first monolingual pre-trained language model for Vietnamese social media texts, ViSoBERT, which is pre-trained on a large-scale corpus of high-quality and diverse Vietnamese social media texts using XLM-R architecture. Moreover, we explored our pre-trained model on five important natural language downstream tasks on Vietnamese social media texts: emotion recognition, hate speech detection, sentiment analysis, spam reviews detection, and hate speech spans detection. Our experiments demonstrate that ViSoBERT, with far fewer parameters, surpasses the previous state-of-the-art models on multiple Vietnamese social media tasks. Our ViSoBERT model is available\footnote{\url{https://huggingface.co/uitnlp/visobert}} only for research purposes.

Title: Approximating Two-Layer Feedforward Networks for Efficient Transformers. (arXiv:2310.10837v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10837
• Code URL: https://github.com/robertcsordas/moe
• Copy Paste: [[2310.10837]] Approximating Two-Layer Feedforward Networks for Efficient Transformers(http://arxiv.org/abs/2310.10837)
• Summary:

  How to reduce compute and memory requirements of neural networks (NNs) without sacrificing performance? Many recent works use sparse Mixtures of Experts (MoEs) to build resource-efficient large language models (LMs). Here we introduce several novel perspectives on MoEs, presenting a general framework that unifies various methods to approximate two-layer NNs (e.g., feedforward blocks of Transformers), including product-key memories (PKMs). Leveraging insights from this framework, we propose methods to improve both MoEs and PKMs. Unlike prior work that compares MoEs with dense baselines under the compute-equal condition, our evaluation condition is parameter-equal, which is crucial to properly evaluate LMs. We show that our MoEs are competitive with the dense Transformer-XL on both the WikiText-103 and enwiki8 datasets at two different scales, while being much more resource efficient. This demonstrates that MoEs are relevant not only to extremely large LMs but also to any-scale resource-efficient LMs. Our code is public.

Title: Instilling Inductive Biases with Subnetworks. (arXiv:2310.10899v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10899
• Code URL: https://github.com/rock-z/instilling-inductiva-bias
• Copy Paste: [[2310.10899]] Instilling Inductive Biases with Subnetworks(http://arxiv.org/abs/2310.10899)
• Summary:

  Despite the recent success of artificial neural networks on a variety of tasks, we have little knowledge or control over the exact solutions these models implement. Instilling inductive biases -- preferences for some solutions over others -- into these models is one promising path toward understanding and controlling their behavior. Much work has been done to study the inherent inductive biases of models and instill different inductive biases through hand-designed architectures or carefully curated training regimens. In this work, we explore a more mechanistic approach: Subtask Induction. Our method discovers a functional subnetwork that implements a particular subtask within a trained model and uses it to instill inductive biases towards solutions utilizing that subtask. Subtask Induction is flexible and efficient, and we demonstrate its effectiveness with two experiments. First, we show that Subtask Induction significantly reduces the amount of training data required for a model to adopt a specific, generalizable solution to a modular arithmetic task. Second, we demonstrate that Subtask Induction successfully induces a human-like shape bias while increasing data efficiency for convolutional and transformer-based image classification models.

Title: Emergent Mixture-of-Experts: Can Dense Pre-trained Transformers Benefit from Emergent Modular Structures?. (arXiv:2310.10908v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10908
• Code URL: https://github.com/qiuzh20/emoe
• Copy Paste: [[2310.10908]] Emergent Mixture-of-Experts: Can Dense Pre-trained Transformers Benefit from Emergent Modular Structures?(http://arxiv.org/abs/2310.10908)
• Summary:

  Incorporating modular designs into neural networks demonstrates superior out-of-generalization, learning efficiency, etc. Existing modular neural networks are generally $\textit{explicit}$ because their modular architectures are pre-defined, and individual modules are expected to implement distinct functions. Conversely, recent works reveal that there exist $\textit{implicit}$ modular structures in standard pre-trained transformers, namely $\textit{Emergent Modularity}$. They indicate that such modular structures exhibit during the early pre-training phase and are totally spontaneous. However, most transformers are still treated as monolithic models with their modular natures underutilized. Therefore, given the excellent properties of explicit modular architecture, we explore $\textit{whether and how dense pre-trained transformers can benefit from emergent modular structures.}$ To study this question, we construct \textbf{E}mergent $\textbf{M}$ixture-$\textbf{o}$f-$\textbf{E}$xperts (EMoE). Without introducing additional parameters, EMoE can be seen as the modular counterpart of the original model and can be effortlessly incorporated into downstream tuning. Extensive experiments (we tune 1785 models) on various downstream tasks (vision and language) and models (22M to1.5B) demonstrate that EMoE effectively boosts in-domain and out-of-domain generalization abilities. Further analysis and ablation study suggest that EMoE mitigates negative knowledge transfer and is robust to various configurations. Code is available at \url{https://github.com/qiuzh20/EMoE}

Title: Heterogenous Memory Augmented Neural Networks. (arXiv:2310.10909v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10909
• Code URL: https://github.com/qiuzh20/hma
• Copy Paste: [[2310.10909]] Heterogenous Memory Augmented Neural Networks(http://arxiv.org/abs/2310.10909)
• Summary:

  It has been shown that semi-parametric methods, which combine standard neural networks with non-parametric components such as external memory modules and data retrieval, are particularly helpful in data scarcity and out-of-distribution (OOD) scenarios. However, existing semi-parametric methods mostly depend on independent raw data points - this strategy is difficult to scale up due to both high computational costs and the incapacity of current attention mechanisms with a large number of tokens. In this paper, we introduce a novel heterogeneous memory augmentation approach for neural networks which, by introducing learnable memory tokens with attention mechanism, can effectively boost performance without huge computational overhead. Our general-purpose method can be seamlessly combined with various backbones (MLP, CNN, GNN, and Transformer) in a plug-and-play manner. We extensively evaluate our approach on various image and graph-based tasks under both in-distribution (ID) and OOD conditions and show its competitive performance against task-specific state-of-the-art methods. Code is available at \url{https://github.com/qiuzh20/HMA}.

Title: Compatible Transformer for Irregularly Sampled Multivariate Time Series. (arXiv:2310.11022v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11022
• Code URL: https://github.com/mediabrain-sjtu/coformer
• Copy Paste: [[2310.11022]] Compatible Transformer for Irregularly Sampled Multivariate Time Series(http://arxiv.org/abs/2310.11022)
• Summary:

  To analyze multivariate time series, most previous methods assume regular subsampling of time series, where the interval between adjacent measurements and the number of samples remain unchanged. Practically, data collection systems could produce irregularly sampled time series due to sensor failures and interventions. However, existing methods designed for regularly sampled multivariate time series cannot directly handle irregularity owing to misalignment along both temporal and variate dimensions. To fill this gap, we propose Compatible Transformer (CoFormer), a transformer-based encoder to achieve comprehensive temporal-interaction feature learning for each individual sample in irregular multivariate time series. In CoFormer, we view each sample as a unique variate-time point and leverage intra-variate/inter-variate attentions to learn sample-wise temporal/interaction features based on intra-variate/inter-variate neighbors. With CoFormer as the core, we can analyze irregularly sampled multivariate time series for many downstream tasks, including classification and prediction. We conduct extensive experiments on 3 real-world datasets and validate that the proposed CoFormer significantly and consistently outperforms existing methods.

Title: SignGT: Signed Attention-based Graph Transformer for Graph Representation Learning. (arXiv:2310.11025v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11025
• Code URL: null
• Copy Paste: [[2310.11025]] SignGT: Signed Attention-based Graph Transformer for Graph Representation Learning(http://arxiv.org/abs/2310.11025)
• Summary:

  The emerging graph Transformers have achieved impressive performance for graph representation learning over graph neural networks (GNNs). In this work, we regard the self-attention mechanism, the core module of graph Transformers, as a two-step aggregation operation on a fully connected graph. Due to the property of generating positive attention values, the self-attention mechanism is equal to conducting a smooth operation on all nodes, preserving the low-frequency information. However, only capturing the low-frequency information is inefficient in learning complex relations of nodes on diverse graphs, such as heterophily graphs where the high-frequency information is crucial. To this end, we propose a Signed Attention-based Graph Transformer (SignGT) to adaptively capture various frequency information from the graphs. Specifically, SignGT develops a new signed self-attention mechanism (SignSA) that produces signed attention values according to the semantic relevance of node pairs. Hence, the diverse frequency information between different node pairs could be carefully preserved. Besides, SignGT proposes a structure-aware feed-forward network (SFFN) that introduces the neighborhood bias to preserve the local topology information. In this way, SignGT could learn informative node representations from both long-range dependencies and local topology information. Extensive empirical results on both node-level and graph-level tasks indicate the superiority of SignGT against state-of-the-art graph Transformers as well as advanced GNNs.

Title: Multi-omics Sampling-based Graph Transformer for Synthetic Lethality Prediction. (arXiv:2310.11082v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11082
• Code URL: null
• Copy Paste: [[2310.11082]] Multi-omics Sampling-based Graph Transformer for Synthetic Lethality Prediction(http://arxiv.org/abs/2310.11082)
• Summary:

  Synthetic lethality (SL) prediction is used to identify if the co-mutation of two genes results in cell death. The prevalent strategy is to abstract SL prediction as an edge classification task on gene nodes within SL data and achieve it through graph neural networks (GNNs). However, GNNs suffer from limitations in their message passing mechanisms, including over-smoothing and over-squashing issues. Moreover, harnessing the information of non-SL gene relationships within large-scale multi-omics data to facilitate SL prediction poses a non-trivial challenge. To tackle these issues, we propose a new multi-omics sampling-based graph transformer for SL prediction (MSGT-SL). Concretely, we introduce a shallow multi-view GNN to acquire local structural patterns from both SL and multi-omics data. Further, we input gene features that encode multi-view information into the standard self-attention to capture long-range dependencies. Notably, starting with batch genes from SL data, we adopt parallel random walk sampling across multiple omics gene graphs encompassing them. Such sampling effectively and modestly incorporates genes from omics in a structure-aware manner before using self-attention. We showcase the effectiveness of MSGT-SL on real-world SL tasks, demonstrating the empirical benefits gained from the graph transformer and multi-omics data.

generative

Title: Improving Video Deepfake Detection: A DCT-Based Approach with Patch-Level Analysis. (arXiv:2310.11204v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.11204
• Code URL: null
• Copy Paste: [[2310.11204]] Improving Video Deepfake Detection: A DCT-Based Approach with Patch-Level Analysis(http://arxiv.org/abs/2310.11204)
• Summary:

  The term deepfake refers to all those multimedia contents that were synthetically altered or created from scratch through the use of generative models. This phenomenon has become widespread due to the use of increasingly accurate and efficient architectures capable of rendering manipulated content indistinguishable from real content. In order to fight the illicit use of this powerful technology, it has become necessary to develop algorithms able to distinguish synthetic content from real ones. In this study, a new algorithm for the detection of deepfakes in digital videos is presented, focusing on the main goal of creating a fast and explainable method from a forensic perspective. To achieve this goal, the I-frames were extracted in order to provide faster computation and analysis than approaches described in literature. In addition, to identify the most discriminating regions within individual video frames, the entire frame, background, face, eyes, nose, mouth, and face frame were analyzed separately. From the Discrete Cosine Transform (DCT), the Beta components were extracted from the AC coefficients and used as input to standard classifiers (e.g., k-NN, SVM, and others) in order to identify those frequencies most discriminative for solving the task in question. Experimental results obtained on the Faceforensics++ and Celeb-DF (v2) datasets show that the eye and mouth regions are those most discriminative and able to determine the nature of the video with greater reliability than the analysis of the whole frame. The method proposed in this study is analytical, fast and does not require much computational power.

Title: Emergent AI-Assisted Discourse: Case Study of a Second Language Writer Authoring with ChatGPT. (arXiv:2310.10903v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10903
• Code URL: null
• Copy Paste: [[2310.10903]] Emergent AI-Assisted Discourse: Case Study of a Second Language Writer Authoring with ChatGPT(http://arxiv.org/abs/2310.10903)
• Summary:

  The rapid proliferation of ChatGPT has incited debates regarding its impact on human writing. Amid concerns about declining writing standards, this study investigates the role of ChatGPT in facilitating academic writing, especially among language learners. Using a case study approach, this study examines the experiences of Kailing, a doctoral student, who integrates ChatGPT throughout their academic writing process. The study employs activity theory as a lens for understanding writing with generative AI tools and data analyzed includes semi-structured interviews, writing samples, and GPT logs. Results indicate that Kailing effectively collaborates with ChatGPT across various writing stages while preserving her distinct authorial voice and agency. This underscores the potential of AI tools such as ChatGPT to enhance academic writing for language learners without overshadowing individual authenticity. This case study offers a critical exploration of how ChatGPT is utilized in the academic writing process and the preservation of a student's authentic voice when engaging with the tool.

Title: Revealing the Unwritten: Visual Investigation of Beam Search Trees to Address Language Model Prompting Challenges. (arXiv:2310.11252v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11252
• Code URL: null
• Copy Paste: [[2310.11252]] Revealing the Unwritten: Visual Investigation of Beam Search Trees to Address Language Model Prompting Challenges(http://arxiv.org/abs/2310.11252)
• Summary:

  The growing popularity of generative language models has amplified interest in interactive methods to guide model outputs. Prompt refinement is considered one of the most effective means to influence output among these methods. We identify several challenges associated with prompting large language models, categorized into data- and model-specific, linguistic, and socio-linguistic challenges. A comprehensive examination of model outputs, including runner-up candidates and their corresponding probabilities, is needed to address these issues. The beam search tree, the prevalent algorithm to sample model outputs, can inherently supply this information. Consequently, we introduce an interactive visual method for investigating the beam search tree, facilitating analysis of the decisions made by the model during generation. We quantitatively show the value of exposing the beam search tree and present five detailed analysis scenarios addressing the identified challenges. Our methodology validates existing results and offers additional insights.

Title: ACES: generating diverse programming puzzles with autotelic language models and semantic descriptors. (arXiv:2310.10692v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10692
• Code URL: null
• Copy Paste: [[2310.10692]] ACES: generating diverse programming puzzles with autotelic language models and semantic descriptors(http://arxiv.org/abs/2310.10692)
• Summary:

  Finding and selecting new and interesting problems to solve is at the heart of curiosity, science and innovation. We here study automated problem generation in the context of the open-ended space of python programming puzzles. Existing generative models often aim at modeling a reference distribution without any explicit diversity optimization. Other methods explicitly optimizing for diversity do so either in limited hand-coded representation spaces or in uninterpretable learned embedding spaces that may not align with human perceptions of interesting variations. With ACES (Autotelic Code Exploration via Semantic descriptors), we introduce a new autotelic generation method that leverages semantic descriptors produced by a large language model (LLM) to directly optimize for interesting diversity, as well as few-shot-based generation. Each puzzle is labeled along 10 dimensions, each capturing a programming skill required to solve it. ACES generates and pursues novel and feasible goals to explore that abstract semantic space, slowly discovering a diversity of solvable programming puzzles in any given run. Across a set of experiments, we show that ACES discovers a richer diversity of puzzles than existing diversity-maximizing algorithms as measured across a range of diversity metrics. We further study whether and in which conditions this diversity can translate into the successful training of puzzle solving models.

Title: From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling. (arXiv:2310.11011v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11011
• Code URL: null
• Copy Paste: [[2310.11011]] From Identifiable Causal Representations to Controllable Counterfactual Generation: A Survey on Causal Generative Modeling(http://arxiv.org/abs/2310.11011)
• Summary:

  Deep generative models have shown tremendous success in data density estimation and data generation from finite samples. While these models have shown impressive performance by learning correlations among features in the data, some fundamental shortcomings are their lack of explainability, the tendency to induce spurious correlations, and poor out-of-distribution extrapolation. In an effort to remedy such challenges, one can incorporate the theory of causality in deep generative modeling. Structural causal models (SCMs) describe data-generating processes and model complex causal relationships and mechanisms among variables in a system. Thus, SCMs can naturally be combined with deep generative models. Causal models offer several beneficial properties to deep generative models, such as distribution shift robustness, fairness, and interoperability. We provide a technical survey on causal generative modeling categorized into causal representation learning and controllable counterfactual generation methods. We focus on fundamental theory, formulations, drawbacks, datasets, metrics, and applications of causal generative models in fairness, privacy, out-of-distribution generalization, and precision medicine. We also discuss open problems and fruitful research directions for future work in the field.

Title: HGCVAE: Integrating Generative and Contrastive Learning for Heterogeneous Graph Learning. (arXiv:2310.11102v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11102
• Code URL: null
• Copy Paste: [[2310.11102]] HGCVAE: Integrating Generative and Contrastive Learning for Heterogeneous Graph Learning(http://arxiv.org/abs/2310.11102)
• Summary:

  Generative self-supervised learning (SSL) has exhibited significant potential and garnered increasing interest in graph learning. In this study, we aim to explore the problem of generative SSL in the context of heterogeneous graph learning (HGL). The previous SSL approaches for heterogeneous graphs have primarily relied on contrastive learning, necessitating the design of complex views to capture heterogeneity. However, existing generative SSL methods have not fully leveraged the capabilities of generative models to address the challenges of HGL. In this paper, we present HGCVAE, a novel contrastive variational graph auto-encoder that liberates HGL from the burden of intricate heterogeneity capturing. Instead of focusing on complicated heterogeneity, HGCVAE harnesses the full potential of generative SSL. HGCVAE innovatively consolidates contrastive learning with generative SSL, introducing several key innovations. Firstly, we employ a progressive mechanism to generate high-quality hard negative samples for contrastive learning, utilizing the power of variational inference. Additionally, we present a dynamic mask strategy to ensure effective and stable learning. Moreover, we propose an enhanced scaled cosine error as the criterion for better attribute reconstruction. As an initial step in combining generative and contrastive SSL, HGCVAE achieves remarkable results compared to various state-of-the-art baselines, confirming its superiority.

Title: Learning to Sample Better. (arXiv:2310.11232v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.11232
• Code URL: null
• Copy Paste: [[2310.11232]] Learning to Sample Better(http://arxiv.org/abs/2310.11232)
• Summary:

  These lecture notes provide an introduction to recent advances in generative modeling methods based on the dynamical transportation of measures, by means of which samples from a simple base measure are mapped to samples from a target measure of interest. Special emphasis is put on the applications of these methods to Monte-Carlo (MC) sampling techniques, such as importance sampling and Markov Chain Monte-Carlo (MCMC) schemes. In this context, it is shown how the maps can be learned variationally using data generated by MC sampling, and how they can in turn be used to improve such sampling in a positive feedback loop.

large language model

Title: Context-Aware Meta-Learning. (arXiv:2310.10971v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.10971
• Code URL: null
• Copy Paste: [[2310.10971]] Context-Aware Meta-Learning(http://arxiv.org/abs/2310.10971)
• Summary:

  Large Language Models like ChatGPT demonstrate a remarkable capacity to learn new concepts during inference without any fine-tuning. However, visual models trained to detect new objects during inference have been unable to replicate this ability, and instead either perform poorly or require meta-training and/or fine-tuning on similar objects. In this work, we propose a meta-learning algorithm that emulates Large Language Models by learning new visual concepts during inference without fine-tuning. Our approach leverages a frozen pre-trained feature extractor, and analogous to in-context learning, recasts meta-learning as sequence modeling over datapoints with known labels and a test datapoint with an unknown label. On 8 out of 11 meta-learning benchmarks, our approach -- without meta-training or fine-tuning -- exceeds or matches the state-of-the-art algorithm, P>M>F, which is meta-trained on these benchmarks.

Title: Towards Emotion-Based Synthetic Consciousness: Using LLMs to Estimate Emotion Probability Vectors. (arXiv:2310.10673v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10673
• Code URL: null
• Copy Paste: [[2310.10673]] Towards Emotion-Based Synthetic Consciousness: Using LLMs to Estimate Emotion Probability Vectors(http://arxiv.org/abs/2310.10673)
• Summary:

  This paper shows how LLMs (Large Language Models) may be used to estimate a summary of the emotional state associated with piece of text. The summary of emotional state is a dictionary of words used to describe emotion together with the probability of the word appearing after a prompt comprising the original text and an emotion eliciting tail. Through emotion analysis of Amazon product reviews we demonstrate emotion descriptors can be mapped into a PCA type space. It was hoped that text descriptions of actions to improve a current text described state could also be elicited through a tail prompt. Experiment seemed to indicate that this is not straightforward to make work. This failure put our hoped for selection of action via choosing the best predict ed outcome via comparing emotional responses out of reach for the moment.

Title: Large language models can replicate cross-cultural differences in personality. (arXiv:2310.10679v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10679
• Code URL: null
• Copy Paste: [[2310.10679]] Large language models can replicate cross-cultural differences in personality(http://arxiv.org/abs/2310.10679)
• Summary:

  We use a large-scale experiment (N=8000) to determine whether GPT-4 can replicate cross-cultural differences in the Big Five, measured using the Ten-Item Personality Inventory. We used the US and South Korea as the cultural pair, given that prior research suggests substantial personality differences between people from these two countries. We manipulated the target of the simulation (US vs. Korean), the language of the inventory (English vs. Korean), and the language model (GPT-4 vs. GPT-3.5). Our results show that GPT-4 replicated the cross-cultural differences for each factor. However, mean ratings had an upward bias and exhibited lower variation than in the human samples, as well as lower structural validity. Overall, we provide preliminary evidence that LLMs can aid cross-cultural psychological research.

Title: Large Language Model Unlearning. (arXiv:2310.10683v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10683
• Code URL: null
• Copy Paste: [[2310.10683]] Large Language Model Unlearning(http://arxiv.org/abs/2310.10683)
• Summary:

  We study how to perform unlearning, i.e. forgetting undesirable (mis)behaviors, on large language models (LLMs). We show at least three scenarios of aligning LLMs with human preferences can benefit from unlearning: (1) removing harmful responses, (2) erasing copyright-protected content as requested, and (3) eliminating hallucinations. Unlearning, as an alignment technique, has three advantages. (1) It only requires negative (e.g. harmful) examples, which are much easier and cheaper to collect (e.g. via red teaming or user reporting) than positive (e.g. helpful and often human-written) examples required in RLHF (RL from human feedback). (2) It is computationally efficient. (3) It is especially effective when we know which training samples cause the misbehavior. To the best of our knowledge, our work is among the first to explore LLM unlearning. We are also among the first to formulate the settings, goals, and evaluations in LLM unlearning. We show that if practitioners only have limited resources, and therefore the priority is to stop generating undesirable outputs rather than to try to generate desirable outputs, unlearning is particularly appealing. Despite only having negative samples, our ablation study shows that unlearning can still achieve better alignment performance than RLHF with just 2% of its computational time.

Title: Autonomous Tree-search Ability of Large Language Models. (arXiv:2310.10686v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10686
• Code URL: null
• Copy Paste: [[2310.10686]] Autonomous Tree-search Ability of Large Language Models(http://arxiv.org/abs/2310.10686)
• Summary:

  Large Language Models have excelled in remarkable reasoning capabilities with advanced prompting techniques, but they fall short on tasks that require exploration, strategic foresight, and sequential decision-making. Recent works propose to utilize external programs to define search logic, such that LLMs can perform passive tree search to solve more challenging reasoning tasks. Though impressive results have been achieved, there are several fundamental limitations of these approaches. First, passive tree searches are not efficient as they usually require multiple rounds of LLM API calls to solve one single problem. Moreover, passive search methods are not flexible since they need task-specific program designs. Then a natural question arises: can we maintain the tree-search capability of LLMs without the aid of external programs, and can still generate responses that clearly demonstrate the process of a tree-structure search? To this end, we propose a new concept called autonomous tree-search ability of LLM, which can automatically generate a response containing search trajectories for the correct answer. Concretely, we perform search trajectories using capable LLM API via a fixed system prompt, allowing them to perform autonomous tree-search (ATS) right out of the box. Experiments on 4 puzzle games demonstrate our method can achieve huge improvements. The ATS-BFS method outperforms the Chain of Thought approach by achieving an average accuracy improvement of 33%. Compared to Tree of Thoughts, it requires 65.6% or 47.7% less GPT-api cost to attain a comparable level of accuracy. Moreover, we have collected data using the ATS prompt method and fine-tuned LLaMA. This approach yield a greater improvement compared to the ones fine-tuned on CoT data. Specifically, it outperforms CoT-tuned LLaMAs by an average of 40.6% and 38.5% for LLaMA2-7B and LLaMA2-13B, respectively.

Title: A decoder-only foundation model for time-series forecasting. (arXiv:2310.10688v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10688
• Code URL: null
• Copy Paste: [[2310.10688]] A decoder-only foundation model for time-series forecasting(http://arxiv.org/abs/2310.10688)
• Summary:

  Motivated by recent advances in large language models for Natural Language Processing (NLP), we design a time-series foundation model for forecasting whose out-of-the-box zero-shot performance on a variety of public datasets comes close to the accuracy of state-of-the-art supervised forecasting models for each individual dataset. Our model is based on pretraining a patched-decoder style attention model on a large time-series corpus, and can work well across different forecasting history lengths, prediction lengths and temporal granularities.

Title: Large Language Models for In-Context Student Modeling: Synthesizing Student's Behavior in Visual Programming from One-Shot Observation. (arXiv:2310.10690v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10690
• Code URL: null
• Copy Paste: [[2310.10690]] Large Language Models for In-Context Student Modeling: Synthesizing Student's Behavior in Visual Programming from One-Shot Observation(http://arxiv.org/abs/2310.10690)
• Summary:

  Student modeling is central to many educational technologies as it enables the prediction of future learning outcomes and targeted instructional strategies. However, open-ended learning environments pose challenges for accurately modeling students due to the diverse behaviors exhibited by students and the absence of a well-defined set of learning skills. To approach these challenges, we explore the application of Large Language Models (LLMs) for in-context student modeling in open-ended learning environments. We introduce a novel framework, LLM-SS, that leverages LLMs for synthesizing student's behavior. More concretely, given a particular student's solving attempt on a reference task as observation, the goal is to synthesize the student's attempt on a target task. Our framework can be combined with different LLMs; moreover, we fine-tune LLMs using domain-specific expertise to boost their understanding of domain background and student behaviors. We evaluate several concrete methods based on LLM-SS using the StudentSyn benchmark, an existing student's attempt synthesis benchmark in visual programming. Experimental results show a significant improvement compared to baseline methods included in the StudentSyn benchmark. Furthermore, our method using the fine-tuned Llama2-70B model improves noticeably compared to using the base model and becomes on par with using the state-of-the-art GPT-4 model.

Title: Bridging Code Semantic and LLMs: Semantic Chain-of-Thought Prompting for Code Generation. (arXiv:2310.10698v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10698
• Code URL: null
• Copy Paste: [[2310.10698]] Bridging Code Semantic and LLMs: Semantic Chain-of-Thought Prompting for Code Generation(http://arxiv.org/abs/2310.10698)
• Summary:

  Large language models (LLMs) have showcased remarkable prowess in code generation. However, automated code generation is still challenging since it requires a high-level semantic mapping between natural language requirements and codes. Most existing LLMs-based approaches for code generation rely on decoder-only causal language models often treate codes merely as plain text tokens, i.e., feeding the requirements as a prompt input, and outputing code as flat sequence of tokens, potentially missing the rich semantic features inherent in source code. To bridge this gap, this paper proposes the "Semantic Chain-of-Thought" approach to intruduce semantic information of code, named SeCoT. Our motivation is that the semantic information of the source code (\eg data flow and control flow) describes more precise program execution behavior, intention and function. By guiding LLM consider and integrate semantic information, we can achieve a more granular understanding and representation of code, enhancing code generation accuracy. Meanwhile, while traditional techniques leveraging such semantic information require complex static or dynamic code analysis to obtain features such as data flow and control flow, SeCoT demonstrates that this process can be fully automated via the intrinsic capabilities of LLMs (i.e., in-context learning), while being generalizable and applicable to challenging domains. While SeCoT can be applied with different LLMs, this paper focuses on the powerful GPT-style models: ChatGPT(close-source model) and WizardCoder(open-source model). The experimental study on three popular DL benchmarks (i.e., HumanEval, HumanEval-ET and MBPP) shows that SeCoT can achieves state-of-the-art performance, greatly improving the potential for large models and code generation.

Title: Theory of Mind for Multi-Agent Collaboration via Large Language Models. (arXiv:2310.10701v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10701
• Code URL: null
• Copy Paste: [[2310.10701]] Theory of Mind for Multi-Agent Collaboration via Large Language Models(http://arxiv.org/abs/2310.10701)
• Summary:

  While Large Language Models (LLMs) have demonstrated impressive accomplishments in both reasoning and planning, their abilities in multi-agent collaborations remains largely unexplored. This study evaluates LLM-based agents in a multi-agent cooperative text game with Theory of Mind (ToM) inference tasks, comparing their performance with Multi-Agent Reinforcement Learning (MARL) and planning-based baselines. We observed evidence of emergent collaborative behaviors and high-order Theory of Mind capabilities among LLM-based agents. Our results reveal limitations in LLM-based agents' planning optimization due to systematic failures in managing long-horizon contexts and hallucination about the task state. We explore the use of explicit belief state representations to mitigate these issues, finding that it enhances task performance and the accuracy of ToM inferences for LLM-based agents.

Title: Demonstrations Are All You Need: Advancing Offensive Content Paraphrasing using In-Context Learning. (arXiv:2310.10707v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10707
• Code URL: null
• Copy Paste: [[2310.10707]] Demonstrations Are All You Need: Advancing Offensive Content Paraphrasing using In-Context Learning(http://arxiv.org/abs/2310.10707)
• Summary:

  Paraphrasing of offensive content is a better alternative to content removal and helps improve civility in a communication environment. Supervised paraphrasers; however, rely heavily on large quantities of labelled data to help preserve meaning and intent. They also retain a large portion of the offensiveness of the original content, which raises questions on their overall usability. In this paper we aim to assist practitioners in developing usable paraphrasers by exploring In-Context Learning (ICL) with large language models (LLMs), i.e., using a limited number of input-label demonstration pairs to guide the model in generating desired outputs for specific queries. Our study focuses on key factors such as -- number and order of demonstrations, exclusion of prompt instruction, and reduction in measured toxicity. We perform principled evaluation on three datasets, including our proposed Context-Aware Polite Paraphrase dataset, comprising of dialogue-style rude utterances, polite paraphrases, and additional dialogue context. We evaluate our approach using two closed source and one open source LLM. Our results reveal that ICL is comparable to supervised methods in generation quality, while being qualitatively better by 25% on human evaluation and attaining lower toxicity by 76%. Also, ICL-based paraphrasers only show a slight reduction in performance even with just 10% training data.

Title: Towards reducing hallucination in extracting information from financial reports using Large Language Models. (arXiv:2310.10760v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10760
• Code URL: null
• Copy Paste: [[2310.10760]] Towards reducing hallucination in extracting information from financial reports using Large Language Models(http://arxiv.org/abs/2310.10760)
• Summary:

  For a financial analyst, the question and answer (Q\&A) segment of the company financial report is a crucial piece of information for various analysis and investment decisions. However, extracting valuable insights from the Q\&A section has posed considerable challenges as the conventional methods such as detailed reading and note-taking lack scalability and are susceptible to human errors, and Optical Character Recognition (OCR) and similar techniques encounter difficulties in accurately processing unstructured transcript text, often missing subtle linguistic nuances that drive investor decisions. Here, we demonstrate the utilization of Large Language Models (LLMs) to efficiently and rapidly extract information from earnings report transcripts while ensuring high accuracy transforming the extraction process as well as reducing hallucination by combining retrieval-augmented generation technique as well as metadata. We evaluate the outcomes of various LLMs with and without using our proposed approach based on various objective metrics for evaluating Q\&A systems, and empirically demonstrate superiority of our method.

Title: IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models. (arXiv:2310.10873v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10873
• Code URL: null
• Copy Paste: [[2310.10873]] IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Models(http://arxiv.org/abs/2310.10873)
• Summary:

  In-context learning is a promising paradigm that utilizes in-context examples as prompts for the predictions of large language models. These prompts are crucial for achieving strong performance. However, since the prompts need to be sampled from a large volume of annotated examples, finding the right prompt may result in high annotation costs. To address this challenge, this paper introduces an influence-driven selective annotation method that aims to minimize annotation costs while improving the quality of in-context examples. The essence of our method is to select a pivotal subset from a large-scale unlabeled data pool to annotate for the subsequent sampling of prompts. Specifically, a directed graph is first constructed to represent unlabeled data. Afterward, the influence of candidate unlabeled subsets is quantified with a diffusion process. A simple yet effective greedy algorithm for unlabeled data selection is lastly introduced. It iteratively selects the data if it provides a maximum marginal gain with respect to quantified influence. Compared with previous efforts on selective annotations, our influence-driven method works in an end-to-end manner, avoids an intractable explicit balance between data diversity and representativeness, and enjoys theoretical support. Experiments confirm the superiority of the proposed method on various benchmarks, achieving better performance under lower time consumption during subset selection. The project page is available at https://skzhang1.github.io/IDEAL/.

Title: TEQ: Trainable Equivalent Transformation for Quantization of LLMs. (arXiv:2310.10944v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10944
• Code URL: https://github.com/intel/neural-compressor
• Copy Paste: [[2310.10944]] TEQ: Trainable Equivalent Transformation for Quantization of LLMs(http://arxiv.org/abs/2310.10944)
• Summary:

  As large language models (LLMs) become more prevalent, there is a growing need for new and improved quantization methods that can meet the computationalast layer demands of these modern architectures while maintaining the accuracy. In this paper, we present TEQ, a trainable equivalent transformation that preserves the FP32 precision of the model output while taking advantage of low-precision quantization, especially 3 and 4 bits weight-only quantization. The training process is lightweight, requiring only 1K steps and fewer than 0.1 percent of the original model's trainable parameters. Furthermore, the transformation does not add any computational overhead during inference. Our results are on-par with the state-of-the-art (SOTA) methods on typical LLMs. Our approach can be combined with other methods to achieve even better performance. The code is available at https://github.com/intel/neural-compressor.

Title: Semantic-Aware Contrastive Sentence Representation Learning with Large Language Models. (arXiv:2310.10962v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.10962
• Code URL: null
• Copy Paste: [[2310.10962]] Semantic-Aware Contrastive Sentence Representation Learning with Large Language Models(http://arxiv.org/abs/2310.10962)
• Summary:

  Contrastive learning has been proven to be effective in learning better sentence representations. However, to train a contrastive learning model, large numbers of labeled sentences are required to construct positive and negative pairs explicitly, such as those in natural language inference (NLI) datasets. Unfortunately, acquiring sufficient high-quality labeled data can be both time-consuming and resource-intensive, leading researchers to focus on developing methods for learning unsupervised sentence representations. As there is no clear relationship between these unstructured randomly-sampled sentences, building positive and negative pairs over them is tricky and problematic. To tackle these challenges, in this paper, we propose SemCSR, a semantic-aware contrastive sentence representation framework. By leveraging the generation and evaluation capabilities of large language models (LLMs), we can automatically construct a high-quality NLI-style corpus without any human annotation, and further incorporate the generated sentence pairs into learning a contrastive sentence representation model. Extensive experiments and comprehensive analyses demonstrate the effectiveness of our proposed framework for learning a better sentence representation with LLMs.

Title: Correction Focused Language Model Training for Speech Recognition. (arXiv:2310.11003v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11003
• Code URL: null
• Copy Paste: [[2310.11003]] Correction Focused Language Model Training for Speech Recognition(http://arxiv.org/abs/2310.11003)
• Summary:

  Language models (LMs) have been commonly adopted to boost the performance of automatic speech recognition (ASR) particularly in domain adaptation tasks. Conventional way of LM training treats all the words in corpora equally, resulting in suboptimal improvements in ASR performance. In this work, we introduce a novel correction focused LM training approach which aims to prioritize ASR fallible words. The word-level ASR fallibility score, representing the likelihood of ASR mis-recognition, is defined and shaped as a prior word distribution to guide the LM training. To enable correction focused training with text-only corpora, large language models (LLMs) are employed as fallibility score predictors and text generators through multi-task fine-tuning. Experimental results for domain adaptation tasks demonstrate the effectiveness of our proposed method. Compared with conventional LMs, correction focused training achieves up to relatively 5.5% word error rate (WER) reduction in sufficient text scenarios. In insufficient text scenarios, LM training with LLM-generated text achieves up to relatively 13% WER reduction, while correction focused training further obtains up to relatively 6% WER reduction.

Title: Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learning. (arXiv:2310.11053v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11053
• Code URL: null
• Copy Paste: [[2310.11053]] Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learning(http://arxiv.org/abs/2310.11053)
• Summary:

  Large Language Models (LLMs) have made unprecedented breakthroughs, yet their increasing integration into everyday life might raise societal risks due to generated unethical content. Despite extensive study on specific issues like bias, the intrinsic values of LLMs remain largely unexplored from a moral philosophy perspective. This work delves into ethical values utilizing Moral Foundation Theory. Moving beyond conventional discriminative evaluations with poor reliability, we propose DeNEVIL, a novel prompt generation algorithm tailored to dynamically exploit LLMs' value vulnerabilities and elicit the violation of ethics in a generative manner, revealing their underlying value inclinations. On such a basis, we construct MoralPrompt, a high-quality dataset comprising 2,397 prompts covering 500+ value principles, and then benchmark the intrinsic values across a spectrum of LLMs. We discovered that most models are essentially misaligned, necessitating further ethical value alignment. In response, we develop VILMO, an in-context alignment method that substantially enhances the value compliance of LLM outputs by learning to generate appropriate value instructions, outperforming existing competitors. Our methods are suitable for black-box and open-source models, offering a promising initial step in studying the ethical values of LLMs.

Title: Learning from Red Teaming: Gender Bias Provocation and Mitigation in Large Language Models. (arXiv:2310.11079v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11079
• Code URL: null
• Copy Paste: [[2310.11079]] Learning from Red Teaming: Gender Bias Provocation and Mitigation in Large Language Models(http://arxiv.org/abs/2310.11079)
• Summary:

  Recently, researchers have made considerable improvements in dialogue systems with the progress of large language models (LLMs) such as ChatGPT and GPT-4. These LLM-based chatbots encode the potential biases while retaining disparities that can harm humans during interactions. The traditional biases investigation methods often rely on human-written test cases. However, these test cases are usually expensive and limited. In this work, we propose a first-of-its-kind method that automatically generates test cases to detect LLMs' potential gender bias. We apply our method to three well-known LLMs and find that the generated test cases effectively identify the presence of biases. To address the biases identified, we propose a mitigation strategy that uses the generated test cases as demonstrations for in-context learning to circumvent the need for parameter fine-tuning. The experimental results show that LLMs generate fairer responses with the proposed approach.

Title: The Quo Vadis of the Relationship between Language and Large Language Models. (arXiv:2310.11146v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11146
• Code URL: null
• Copy Paste: [[2310.11146]] The Quo Vadis of the Relationship between Language and Large Language Models(http://arxiv.org/abs/2310.11146)
• Summary:

  In the field of Artificial (General) Intelligence (AI), the several recent advancements in Natural language processing (NLP) activities relying on Large Language Models (LLMs) have come to encourage the adoption of LLMs as scientific models of language. While the terminology employed for the characterization of LLMs favors their embracing as such, it is not clear that they are in a place to offer insights into the target system they seek to represent. After identifying the most important theoretical and empirical risks brought about by the adoption of scientific models that lack transparency, we discuss LLMs relating them to every scientific model's fundamental components: the object, the medium, the meaning and the user. We conclude that, at their current stage of development, LLMs hardly offer any explanations for language, and then we provide an outlook for more informative future research directions on this topic.

Title: Probing the Creativity of Large Language Models: Can models produce divergent semantic association?. (arXiv:2310.11158v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11158
• Code URL: https://github.com/dingnlab/probing_creativity
• Copy Paste: [[2310.11158]] Probing the Creativity of Large Language Models: Can models produce divergent semantic association?(http://arxiv.org/abs/2310.11158)
• Summary:

  Large language models possess remarkable capacity for processing language, but it remains unclear whether these models can further generate creative content. The present study aims to investigate the creative thinking of large language models through a cognitive perspective. We utilize the divergent association task (DAT), an objective measurement of creativity that asks models to generate unrelated words and calculates the semantic distance between them. We compare the results across different models and decoding strategies. Our findings indicate that: (1) When using the greedy search strategy, GPT-4 outperforms 96% of humans, while GPT-3.5-turbo exceeds the average human level. (2) Stochastic sampling and temperature scaling are effective to obtain higher DAT scores for models except GPT-4, but face a trade-off between creativity and stability. These results imply that advanced large language models have divergent semantic associations, which is a fundamental process underlying creativity.

Title: Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations. (arXiv:2310.11207v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11207
• Code URL: null
• Copy Paste: [[2310.11207]] Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanations(http://arxiv.org/abs/2310.11207)
• Summary:

  Large language models (LLMs) such as ChatGPT have demonstrated superior performance on a variety of natural language processing (NLP) tasks including sentiment analysis, mathematical reasoning and summarization. Furthermore, since these models are instruction-tuned on human conversations to produce "helpful" responses, they can and often will produce explanations along with the response, which we call self-explanations. For example, when analyzing the sentiment of a movie review, the model may output not only the positivity of the sentiment, but also an explanation (e.g., by listing the sentiment-laden words such as "fantastic" and "memorable" in the review). How good are these automatically generated self-explanations? In this paper, we investigate this question on the task of sentiment analysis and for feature attribution explanation, one of the most commonly studied settings in the interpretability literature (for pre-ChatGPT models). Specifically, we study different ways to elicit the self-explanations, evaluate their faithfulness on a set of evaluation metrics, and compare them to traditional explanation methods such as occlusion or LIME saliency maps. Through an extensive set of experiments, we find that ChatGPT's self-explanations perform on par with traditional ones, but are quite different from them according to various agreement metrics, meanwhile being much cheaper to produce (as they are generated along with the prediction). In addition, we identified several interesting characteristics of them, which prompt us to rethink many current model interpretability practices in the era of ChatGPT(-like) LLMs.

Title: KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models. (arXiv:2310.11220v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11220
• Code URL: https://github.com/jiho283/kg-gpt
• Copy Paste: [[2310.11220]] KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models(http://arxiv.org/abs/2310.11220)
• Summary:

  While large language models (LLMs) have made considerable advancements in understanding and generating unstructured text, their application in structured data remains underexplored. Particularly, using LLMs for complex reasoning tasks on knowledge graphs (KGs) remains largely untouched. To address this, we propose KG-GPT, a multi-purpose framework leveraging LLMs for tasks employing KGs. KG-GPT comprises three steps: Sentence Segmentation, Graph Retrieval, and Inference, each aimed at partitioning sentences, retrieving relevant graph components, and deriving logical conclusions, respectively. We evaluate KG-GPT using KG-based fact verification and KGQA benchmarks, with the model showing competitive and robust performance, even outperforming several fully-supervised models. Our work, therefore, marks a significant step in unifying structured and unstructured data processing within the realm of LLMs.

Title: Entity Matching using Large Language Models. (arXiv:2310.11244v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11244
• Code URL: null
• Copy Paste: [[2310.11244]] Entity Matching using Large Language Models(http://arxiv.org/abs/2310.11244)
• Summary:

  Entity Matching is the task of deciding whether two entity descriptions refer to the same real-world entity. Entity Matching is a central step in most data integration pipelines and an enabler for many e-commerce applications which require to match products offers from different vendors. State-of-the-art entity matching methods often rely on pre-trained language models (PLMs) such as BERT or RoBERTa. Two major drawbacks of these models for entity matching are that (i) the models require significant amounts of task-specific training data and (ii) the fine-tuned models are not robust concerning out-of-distribution entities. In this paper, we investigate using large language models (LLMs) for entity matching as a less domain-specific training data reliant and more robust alternative to PLM-based matchers. Our study covers hosted LLMs, such as GPT3.5 and GPT4, as well as open source LLMs based on Llama2 which can be run locally. We evaluate these models in a zero-shot scenario as well as a scenario where task-specific training data is available. We compare different prompt designs as well as the prompt sensitivity of the models in the zero-shot scenario. We investigate (i) the selection of in-context demonstrations, (ii) the generation of matching rules, as well as (iii) fine-tuning GPT3.5 in the second scenario using the same pool of training data across the different approaches. Our experiments show that GPT4 without any task-specific training data outperforms fine-tuned PLMs (RoBERTa and Ditto) on three out of five benchmark datasets reaching F1 scores around 90%. The experiments with in-context learning and rule generation show that all models beside of GPT4 benefit from these techniques (on average 5.9% and 2.2% F1), while GPT4 does not need such additional guidance in most cases...

segmentation

Title: IDRNet: Intervention-Driven Relation Network for Semantic Segmentation. (arXiv:2310.10755v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.10755
• Code URL: https://github.com/segmentationblwx/sssegmentation
• Copy Paste: [[2310.10755]] IDRNet: Intervention-Driven Relation Network for Semantic Segmentation(http://arxiv.org/abs/2310.10755)
• Summary:

  Co-occurrent visual patterns suggest that pixel relation modeling facilitates dense prediction tasks, which inspires the development of numerous context modeling paradigms, \emph{e.g.}, multi-scale-driven and similarity-driven context schemes. Despite the impressive results, these existing paradigms often suffer from inadequate or ineffective contextual information aggregation due to reliance on large amounts of predetermined priors. To alleviate the issues, we propose a novel \textbf{I}ntervention-\textbf{D}riven \textbf{R}elation \textbf{Net}work (\textbf{IDRNet}), which leverages a deletion diagnostics procedure to guide the modeling of contextual relations among different pixels. Specifically, we first group pixel-level representations into semantic-level representations with the guidance of pseudo labels and further improve the distinguishability of the grouped representations with a feature enhancement module. Next, a deletion diagnostics procedure is conducted to model relations of these semantic-level representations via perceiving the network outputs and the extracted relations are utilized to guide the semantic-level representations to interact with each other. Finally, the interacted representations are utilized to augment original pixel-level representations for final predictions. Extensive experiments are conducted to validate the effectiveness of IDRNet quantitatively and qualitatively. Notably, our intervention-driven context scheme brings consistent performance improvements to state-of-the-art segmentation frameworks and achieves competitive results on popular benchmark datasets, including ADE20K, COCO-Stuff, PASCAL-Context, LIP, and Cityscapes. Code is available at \url{https://github.com/SegmentationBLWX/sssegmentation}.

Title: Filling the Holes on 3D Heritage Object Surface based on Automatic Segmentation Algorithm. (arXiv:2310.10875v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.10875
• Code URL: null
• Copy Paste: [[2310.10875]] Filling the Holes on 3D Heritage Object Surface based on Automatic Segmentation Algorithm(http://arxiv.org/abs/2310.10875)
• Summary:

  Reconstructing and processing the 3D objects are popular activities in the research field of computer graphics, image processing and computer vision. The 3D objects are processed based on the methods like geometric modeling, a branch of applied mathematics and computational geometry, or the machine learning algorithms based on image processing. The computation of geometrical objects includes processing the curves and surfaces, subdivision, simplification, meshing, holes filling, reconstructing, and refining the 3D surface objects on both point cloud data and triangular mesh. While the machine learning methods are developed using deep learning models. With the support of 3D laser scan devices and Lidar techniques, the obtained dataset is close to original shape of the real objects. Besides, the photography and its application based on the modern techniques in recent years help us collect data and process the 3D models more precise. This article proposes an improved method for filling holes on the 3D object surface based on an automatic segmentation. Instead of filling the hole directly as the existing methods, we now subdivide the hole before filling it. The hole is first determined and segmented automatically based on computation of its local curvature. It is then filled on each part of the hole to match its local curvature shape. The method can work on both 3D point cloud surfaces and triangular mesh surface. Comparing to the state of the art methods, our proposed method obtained higher accuracy of the reconstructed 3D objects.

Title: Towards Training-free Open-world Segmentation via Image Prompting Foundation Models. (arXiv:2310.10912v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.10912
• Code URL: null
• Copy Paste: [[2310.10912]] Towards Training-free Open-world Segmentation via Image Prompting Foundation Models(http://arxiv.org/abs/2310.10912)
• Summary:

  The realm of computer vision has witnessed a paradigm shift with the advent of foundational models, mirroring the transformative influence of large language models in the domain of natural language processing. This paper delves into the exploration of open-world segmentation, presenting a novel approach called Image Prompt Segmentation (IPSeg) that harnesses the power of vision foundational models. At the heart of IPSeg lies the principle of a training-free paradigm, which capitalizes on image prompting techniques. IPSeg utilizes a single image containing a subjective visual concept as a flexible prompt to query vision foundation models like DINOv2 and Stable Diffusion. Our approach extracts robust features for the prompt image and input image, then matches the input representations to the prompt representations via a novel feature interaction module to generate point prompts highlighting target objects in the input image. The generated point prompts are further utilized to guide the Segment Anything Model to segment the target object in the input image. The proposed method stands out by eliminating the need for exhaustive training sessions, thereby offering a more efficient and scalable solution. Experiments on COCO, PASCAL VOC, and other datasets demonstrate IPSeg's efficacy for flexible open-world segmentation using intuitive image prompts. This work pioneers tapping foundation models for open-world understanding through visual concepts conveyed in images.

Title: MRI brain tumor segmentation using informative feature vectors and kernel dictionary learning. (arXiv:2310.10963v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.10963
• Code URL: null
• Copy Paste: [[2310.10963]] MRI brain tumor segmentation using informative feature vectors and kernel dictionary learning(http://arxiv.org/abs/2310.10963)
• Summary:

  This paper presents a method based on a kernel dictionary learning algorithm for segmenting brain tumor regions in magnetic resonance images (MRI). A set of first-order and second-order statistical feature vectors are extracted from patches of size 3 * 3 around pixels in the brain MRI scans. These feature vectors are utilized to train two kernel dictionaries separately for healthy and tumorous tissues. To enhance the efficiency of the dictionaries and reduce training time, a correlation-based sample selection technique is developed to identify the most informative and discriminative subset of feature vectors. This technique aims to improve the performance of the dictionaries by selecting a subset of feature vectors that provide valuable information for the segmentation task. Subsequently, a linear classifier is utilized to distinguish between healthy and unhealthy pixels based on the learned dictionaries. The results demonstrate that the proposed method outperforms other existing methods in terms of segmentation accuracy and significantly reduces both the time and memory required, resulting in a remarkably fast training process.

Title: NICE: Improving Panoptic Narrative Detection and Segmentation with Cascading Collaborative Learning. (arXiv:2310.10975v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.10975
• Code URL: https://github.com/mr-neko/nice
• Copy Paste: [[2310.10975]] NICE: Improving Panoptic Narrative Detection and Segmentation with Cascading Collaborative Learning(http://arxiv.org/abs/2310.10975)
• Summary:

  Panoptic Narrative Detection (PND) and Segmentation (PNS) are two challenging tasks that involve identifying and locating multiple targets in an image according to a long narrative description. In this paper, we propose a unified and effective framework called NICE that can jointly learn these two panoptic narrative recognition tasks. Existing visual grounding tasks use a two-branch paradigm, but applying this directly to PND and PNS can result in prediction conflict due to their intrinsic many-to-many alignment property. To address this, we introduce two cascading modules based on the barycenter of the mask, which are Coordinate Guided Aggregation (CGA) and Barycenter Driven Localization (BDL), responsible for segmentation and detection, respectively. By linking PNS and PND in series with the barycenter of segmentation as the anchor, our approach naturally aligns the two tasks and allows them to complement each other for improved performance. Specifically, CGA provides the barycenter as a reference for detection, reducing BDL's reliance on a large number of candidate boxes. BDL leverages its excellent properties to distinguish different instances, which improves the performance of CGA for segmentation. Extensive experiments demonstrate that NICE surpasses all existing methods by a large margin, achieving 4.1% for PND and 2.9% for PNS over the state-of-the-art. These results validate the effectiveness of our proposed collaborative learning strategy. The project of this work is made publicly available at https://github.com/Mr-Neko/NICE.

Title: Long-form Simultaneous Speech Translation: Thesis Proposal. (arXiv:2310.11141v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.11141
• Code URL: null
• Copy Paste: [[2310.11141]] Long-form Simultaneous Speech Translation: Thesis Proposal(http://arxiv.org/abs/2310.11141)
• Summary:

  Simultaneous speech translation (SST) aims to provide real-time translation of spoken language, even before the speaker finishes their sentence. Traditionally, SST has been addressed primarily by cascaded systems that decompose the task into subtasks, including speech recognition, segmentation, and machine translation. However, the advent of deep learning has sparked significant interest in end-to-end (E2E) systems. Nevertheless, a major limitation of most approaches to E2E SST reported in the current literature is that they assume that the source speech is pre-segmented into sentences, which is a significant obstacle for practical, real-world applications. This thesis proposal addresses end-to-end simultaneous speech translation, particularly in the long-form setting, i.e., without pre-segmentation. We present a survey of the latest advancements in E2E SST, assess the primary obstacles in SST and its relevance to long-form scenarios, and suggest approaches to tackle these challenges.