secure

Title: Quarks: A Secure and Decentralized Blockchain-Based Messaging Network. (arXiv:2308.04452v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04452
• Code URL: null
• Copy Paste: [[2308.04452]] Quarks: A Secure and Decentralized Blockchain-Based Messaging Network(http://arxiv.org/abs/2308.04452)
• Summary:

  In last two decades, messaging systems have gained widespread popularity both in the enterprise and consumer sectors. Many of these systems used secure protocols like end-to-end encryption to ensure strong security in one-to-one communication. However, the majority of them rely on centralized servers, which allows them to use their users' personal data. Also, it allows the government to track and regulate their citizens' activities, which poses significant threats to "digital freedom". Also, these systems have failed to achieve security attributes like confidentiality, integrity, and privacy for group communications. In this paper, we present a novel blockchain-based secure messaging system named Quarks that overcomes the security pitfalls of the existing systems and eliminates centralized control. We have analyzed our architecture with security models to demonstrate the system's reliability and usability. We have developed a Proof of Concept (PoC) of the Quarks system leveraging Distributed Ledger Technology (DLT) and conducted load testing on that. We noticed that our PoC system achieves all the desired attributes that are prevalent in a traditional centralized messaging scheme despite the limited capacity of the development and testing environment. Therefore, this assures us of the applicability of such systems in the near future if scaled up properly.

Title: Towards Immutability: A Secure and Efficient Auditing Framework for Cloud Supporting Data Integrity and File Version Control. (arXiv:2308.04453v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04453
• Code URL: null
• Copy Paste: [[2308.04453]] Towards Immutability: A Secure and Efficient Auditing Framework for Cloud Supporting Data Integrity and File Version Control(http://arxiv.org/abs/2308.04453)
• Summary:

  Although wide-scale integration of cloud services with myriad applications increases quality of services (QoS) for enterprise users, verifying the existence and manipulation of stored cloud information remains an open research problem. Decentralized blockchain-based solutions are becoming more appealing for cloud auditing environments because of the immutable nature of blockchain. However, the decentralized structure of blockchain results in considerable synchronization and communication overhead, which increases maintenance costs for cloud service providers (CSP). This paper proposes a Merkle Hash Tree based architecture named Entangled Merkle Forest to support version control and dynamic auditing of information in centralized cloud environments. We utilized a semi-trusted third-party auditor to conduct the auditing tasks with minimal privacy-preserving file metadata. To the best of our knowledge, we are the first to design a node sharing Merkle Forest to offer a cost-effective auditing framework for centralized cloud infrastructures while achieving the immutable feature of blockchain, mitigating the synchronization and performance challenges of the decentralized architectures. Our proposed scheme outperforms it's equivalent Blockchain-based schemes by ensuring time and storage efficiency with minimum overhead as evidenced by performance analysis.

Title: can-train-and-test: A Curated CAN Dataset for Automotive Intrusion Detection. (arXiv:2308.04972v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04972
• Code URL: null
• Copy Paste: [[2308.04972]] can-train-and-test: A Curated CAN Dataset for Automotive Intrusion Detection(http://arxiv.org/abs/2308.04972)
• Summary:

  When it comes to in-vehicle networks (IVNs), the controller area network -- CAN -- bus dominates the market; automobiles manufactured and sold around the world depend on the CAN bus for safety-critical communications between various components of the vehicle (e.g., the engine, the transmission, the steering column). Unfortunately, the CAN bus is inherently insecure; in fact, it completely lacks controls such as authentication, authorization, and confidentiality (i.e., encryption). Therefore, researchers have travailed to develop automotive security enhancements. The automotive intrusion detection system (IDS) is especially popular in the literature -- due to its relatively low cost in terms of money, resource utilization, and implementation effort. That said, developing and evaluating an automotive IDS is often challenging; if researchers do not have access to a test vehicle, then they are forced to depend on publicly available CAN data -- which is not without limitations. Lack of access to adequate CAN data, then, becomes a barrier to entry into automotive security research.

  We seek to lower that barrier to entry by introducing a new CAN dataset to facilitate the development and evaluation of automotive IDSs. Our dataset, dubbed can-train-and-test, provides CAN data from four different vehicles produced by two different manufacturers. The attack captures for each vehicle model are equivalent, enabling researchers to assess the ability of a given IDS to generalize to different vehicle models and even different vehicle manufacturers. Our dataset contains replayable .log files as well as labeled and unlabeled .csv files, thereby meeting a variety of development and evaluation needs. Furthermore, can-train-and-test offers nine unique attacks, ranging from denial of service (DoS) to gear spoofing to standstill...

security

Title: Enhancing Mobile Privacy and Security: A Face Skin Patch-Based Anti-Spoofing Approach. (arXiv:2308.04798v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04798
• Code URL: null
• Copy Paste: [[2308.04798]] Enhancing Mobile Privacy and Security: A Face Skin Patch-Based Anti-Spoofing Approach(http://arxiv.org/abs/2308.04798)
• Summary:

  As Facial Recognition System(FRS) is widely applied in areas such as access control and mobile payments due to its convenience and high accuracy. The security of facial recognition is also highly regarded. The Face anti-spoofing system(FAS) for face recognition is an important component used to enhance the security of face recognition systems. Traditional FAS used images containing identity information to detect spoofing traces, however there is a risk of privacy leakage during the transmission and storage of these images. Besides, the encryption and decryption of these privacy-sensitive data takes too long compared to inference time by FAS model. To address the above issues, we propose a face anti-spoofing algorithm based on facial skin patches leveraging pure facial skin patch images as input, which contain no privacy information, no encryption or decryption is needed for these images. We conduct experiments on several public datasets, the results prove that our algorithm has demonstrated superiority in both accuracy and speed.

Title: Digital Healthcare in The Metaverse: Insights into Privacy and Security. (arXiv:2308.04438v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04438
• Code URL: null
• Copy Paste: [[2308.04438]] Digital Healthcare in The Metaverse: Insights into Privacy and Security(http://arxiv.org/abs/2308.04438)
• Summary:

  In this article, we study the privacy and security aspects of the metaverse in the context of digital healthcare. Our studies include the security aspects of data collection and communications for access to the metaverse, the privacy and security threats of employing Machine Learning and Artificial Intelligence (AI/ML) algorithms for metaverse healthcare, and the privacy of social interactions among patients in the metaverse from a human-centric perspective. In this article, we aim to provide new perspectives and less-investigated solutions, which are shown to be promising mechanisms in the context of wireless communications and computer science and can be considered novel solutions to be applied to healthcare metaverse services. Topics include physical layer security (PHYSec), Semantic Metaverse Communications (SMC), Differential Privacy (DP), and Adversarial Machine Learning (AML). As a case study, we propose distributed differential privacy for the metaverse healthcare systems, where each virtual clinic perturbs its medical model vector to enhance privacy against malicious actors and curious servers. Through our experiments on the Breast Cancer Wisconsin Dataset (BCWD), we highlight the privacy-utility trade-off for different adjustable levels of privacy.

Title: Assessment of POS Owners Awareness of Cybersecurity and Insider Threats in POS Kiosks Related Financial Crimes. (arXiv:2308.04447v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04447
• Code URL: null
• Copy Paste: [[2308.04447]] Assessment of POS Owners Awareness of Cybersecurity and Insider Threats in POS Kiosks Related Financial Crimes(http://arxiv.org/abs/2308.04447)
• Summary:

  The introduction of point of sales POS technologies as a payment system was welcoming and constitutes one of the major breakthroughs in the efforts made to rejuvenate the global financial systems. However, like other information technology IT based financial systems, the POS also poses some cybersecurity security threats. The unique thing about the POS is that the main cybersecurity threats it poses to most users are not IT based which refers to unauthorized access and gaining information without the knowledge of the user. The threats are rather connected to the mode of operation of POS kiosks, particularly as experienced in most parts of Nigeria. The mode of operation exposes users cards to possible cloning.

Title: EPS: Distinguishable IQ Data Representation for Domain-Adaptation Learning of Device Fingerprints. (arXiv:2308.04467v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04467
• Code URL: null
• Copy Paste: [[2308.04467]] EPS: Distinguishable IQ Data Representation for Domain-Adaptation Learning of Device Fingerprints(http://arxiv.org/abs/2308.04467)
• Summary:

  Deep learning (DL)-based RF fingerprinting (RFFP) technology has emerged as a powerful physical-layer security mechanism, enabling device identification and authentication based on unique device-specific signatures that can be extracted from the received RF signals. However, DL-based RFFP methods face major challenges concerning their ability to adapt to domain (e.g., day/time, location, channel, etc.) changes and variability. This work proposes a novel IQ data representation and feature design, termed Double-Sided Envelope Power Spectrum or EPS, that is proven to overcome the domain adaptation problems significantly. By accurately capturing device hardware impairments while suppressing irrelevant domain information, EPS offers improved feature selection for DL models in RFFP. Experimental evaluations demonstrate its effectiveness, achieving over 99% testing accuracy in same-day/channel/location evaluations and 93% accuracy in cross-day evaluations, outperforming the traditional IQ representation. Additionally, EPS excels in cross-location evaluations, achieving a 95% accuracy. The proposed representation significantly enhances the robustness and generalizability of DL-based RFFP methods, thereby presenting a transformative solution to IQ data-based device fingerprinting.

Title: Deep Learning for Diverse Data Types Steganalysis: A Review. (arXiv:2308.04522v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04522
• Code URL: null
• Copy Paste: [[2308.04522]] Deep Learning for Diverse Data Types Steganalysis: A Review(http://arxiv.org/abs/2308.04522)
• Summary:

  Steganography and steganalysis are two interrelated aspects of the field of information security. Steganography seeks to conceal communications, whereas steganalysis is aimed to either find them or even, if possible, recover the data they contain. Steganography and steganalysis have attracted a great deal of interest, particularly from law enforcement. Steganography is often used by cybercriminals and even terrorists to avoid being captured while in possession of incriminating evidence, even encrypted, since cryptography is prohibited or restricted in many countries. Therefore, knowledge of cutting-edge techniques to uncover concealed information is crucial in exposing illegal acts. Over the last few years, a number of strong and reliable steganography and steganalysis techniques have been introduced in the literature. This review paper provides a comprehensive overview of deep learning-based steganalysis techniques used to detect hidden information within digital media. The paper covers all types of cover in steganalysis, including image, audio, and video, and discusses the most commonly used deep learning techniques. In addition, the paper explores the use of more advanced deep learning techniques, such as deep transfer learning (DTL) and deep reinforcement learning (DRL), to enhance the performance of steganalysis systems. The paper provides a systematic review of recent research in the field, including data sets and evaluation metrics used in recent studies. It also presents a detailed analysis of DTL-based steganalysis approaches and their performance on different data sets. The review concludes with a discussion on the current state of deep learning-based steganalysis, challenges, and future research directions.

Title: An Empirical Study on Using Large Language Models to Analyze Software Supply Chain Security Failures. (arXiv:2308.04898v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04898
• Code URL: null
• Copy Paste: [[2308.04898]] An Empirical Study on Using Large Language Models to Analyze Software Supply Chain Security Failures(http://arxiv.org/abs/2308.04898)
• Summary:

  As we increasingly depend on software systems, the consequences of breaches in the software supply chain become more severe. High-profile cyber attacks like those on SolarWinds and ShadowHammer have resulted in significant financial and data losses, underlining the need for stronger cybersecurity. One way to prevent future breaches is by studying past failures. However, traditional methods of analyzing these failures require manually reading and summarizing reports about them. Automated support could reduce costs and allow analysis of more failures. Natural Language Processing (NLP) techniques such as Large Language Models (LLMs) could be leveraged to assist the analysis of failures. In this study, we assessed the ability of Large Language Models (LLMs) to analyze historical software supply chain breaches. We used LLMs to replicate the manual analysis of 69 software supply chain security failures performed by members of the Cloud Native Computing Foundation (CNCF). We developed prompts for LLMs to categorize these by four dimensions: type of compromise, intent, nature, and impact. GPT 3.5s categorizations had an average accuracy of 68% and Bard had an accuracy of 58% over these dimensions. We report that LLMs effectively characterize software supply chain failures when the source articles are detailed enough for consensus among manual analysts, but cannot yet replace human analysts. Future work can improve LLM performance in this context, and study a broader range of articles and failures.

Title: Adversarial Deep Reinforcement Learning for Cyber Security in Software Defined Networks. (arXiv:2308.04909v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04909
• Code URL: null
• Copy Paste: [[2308.04909]] Adversarial Deep Reinforcement Learning for Cyber Security in Software Defined Networks(http://arxiv.org/abs/2308.04909)
• Summary:

  This paper focuses on the impact of leveraging autonomous offensive approaches in Deep Reinforcement Learning (DRL) to train more robust agents by exploring the impact of applying adversarial learning to DRL for autonomous security in Software Defined Networks (SDN). Two algorithms, Double Deep Q-Networks (DDQN) and Neural Episodic Control to Deep Q-Network (NEC2DQN or N2D), are compared. NEC2DQN was proposed in 2018 and is a new member of the deep q-network (DQN) family of algorithms. The attacker has full observability of the environment and access to a causative attack that uses state manipulation in an attempt to poison the learning process. The implementation of the attack is done under a white-box setting, in which the attacker has access to the defender's model and experiences. Two games are played; in the first game, DDQN is a defender and N2D is an attacker, and in second game, the roles are reversed. The games are played twice; first, without an active causative attack and secondly, with an active causative attack. For execution, three sets of game results are recorded in which a single set consists of 10 game runs. The before and after results are then compared in order to see if there was actually an improvement or degradation. The results show that with minute parameter changes made to the algorithms, there was growth in the attacker's role, since it is able to win games. Implementation of the adversarial learning by the introduction of the causative attack showed the algorithms are still able to defend the network according to their strengths.

Title: Adversarial ModSecurity: Countering Adversarial SQL Injections with Robust Machine Learning. (arXiv:2308.04964v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04964
• Code URL: null
• Copy Paste: [[2308.04964]] Adversarial ModSecurity: Countering Adversarial SQL Injections with Robust Machine Learning(http://arxiv.org/abs/2308.04964)
• Summary:

  ModSecurity is widely recognized as the standard open-source Web Application Firewall (WAF), maintained by the OWASP Foundation. It detects malicious requests by matching them against the Core Rule Set, identifying well-known attack patterns. Each rule in the CRS is manually assigned a weight, based on the severity of the corresponding attack, and a request is detected as malicious if the sum of the weights of the firing rules exceeds a given threshold. In this work, we show that this simple strategy is largely ineffective for detecting SQL injection (SQLi) attacks, as it tends to block many legitimate requests, while also being vulnerable to adversarial SQLi attacks, i.e., attacks intentionally manipulated to evade detection. To overcome these issues, we design a robust machine learning model, named AdvModSec, which uses the CRS rules as input features, and it is trained to detect adversarial SQLi attacks. Our experiments show that AdvModSec, being trained on the traffic directed towards the protected web services, achieves a better trade-off between detection and false positive rates, improving the detection rate of the vanilla version of ModSecurity with CRS by 21%. Moreover, our approach is able to improve its adversarial robustness against adversarial SQLi attacks by 42%, thereby taking a step forward towards building more robust and trustworthy WAFs.

Title: CERMET: Coding for Energy Reduction with Multiple Encryption Techniques -- $It's\ easy\ being\ green$. (arXiv:2308.05063v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05063
• Code URL: null
• Copy Paste: [[2308.05063]] CERMET: Coding for Energy Reduction with Multiple Encryption Techniques -- $It's\ easy\ being\ green$(http://arxiv.org/abs/2308.05063)
• Summary:

  This paper presents CERMET, an energy-efficient hardware architecture designed for hardware-constrained cryptosystems. CERMET employs a base cryptosystem in conjunction with network coding to provide both information-theoretic and computational security while reducing energy consumption per bit. This paper introduces the hardware architecture for the system and explores various optimizations to enhance its performance. The universality of the approach is demonstrated by designing the architecture to accommodate both asymmetric and symmetric cryptosystems. The analysis reveals that the benefits of this proposed approach are multifold, reducing energy per bit and area without compromising security or throughput. The optimized hardware architectures can achieve below 1 pJ/bit operations for AES-256. Furthermore, for a public key cryptosystem based on Elliptic Curve Cryptography (ECC), a remarkable 14.6X reduction in energy per bit and a 9.3X reduction in area are observed, bringing it to less than 1 nJ/bit.

privacy

Title: FaceSkin: A Privacy Preserving Facial skin patch Dataset for multi Attributes classification. (arXiv:2308.04765v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04765
• Code URL: null
• Copy Paste: [[2308.04765]] FaceSkin: A Privacy Preserving Facial skin patch Dataset for multi Attributes classification(http://arxiv.org/abs/2308.04765)
• Summary:

  Human facial skin images contain abundant textural information that can serve as valuable features for attribute classification, such as age, race, and gender. Additionally, facial skin images offer the advantages of easy collection and minimal privacy concerns. However, the availability of well-labeled human skin datasets with a sufficient number of images is limited. To address this issue, we introduce a dataset called FaceSkin, which encompasses a diverse range of ages and races. Furthermore, to broaden the application scenarios, we incorporate synthetic skin-patches obtained from 2D and 3D attack images, including printed paper, replays, and 3D masks. We evaluate the FaceSkin dataset across distinct categories and present experimental results demonstrating its effectiveness in attribute classification, as well as its potential for various downstream tasks, such as Face anti-spoofing and Age estimation.

Title: LLaMA-E: Empowering E-commerce Authoring with Multi-Aspect Instruction Following. (arXiv:2308.04913v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04913
• Code URL: null
• Copy Paste: [[2308.04913]] LLaMA-E: Empowering E-commerce Authoring with Multi-Aspect Instruction Following(http://arxiv.org/abs/2308.04913)
• Summary:

  E-commerce authoring involves creating attractive, abundant, and targeted promotional content to drive product sales. The emergence of large language models (LLMs) introduces an innovative paradigm, offering a unified solution to address various authoring tasks within this scenario. However, mainstream LLMs trained on general corpora with common sense knowledge reveal limitations in fitting complex and personalized features unique to e-commerce products and customers. Furthermore, LLMs like GPT-3.5 necessitate remote accessibility, raising concerns about safeguarding voluminous customer privacy data during transmission. This paper proposes the LLaMA-E, the unified and customized instruction-following language models focusing on diverse e-commerce authoring tasks. Specifically, the domain experts create the seed instruction set from the tasks of ads generation, query-enhanced product title rewriting, product classification, purchase intent speculation, and general Q&A. These tasks enable the models to comprehensively understand precise e-commerce authoring knowledge by interleaving features covering typical service aspects of customers, sellers, and platforms. The GPT-3.5 is introduced as a teacher model, which expands the seed instructions to form a training set for the LLaMA-E models with various scales. The experimental results show that the proposed LLaMA-E models achieve state-of-the-art results in quantitative and qualitative evaluations, also exhibiting the advantage in zero-shot scenes. To the best of our knowledge, this study is the first to serve the LLMs to specific e-commerce authoring scenarios.

Title: Global Differential Privacy for Distributed Metaverse Healthcare Systems. (arXiv:2308.04439v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04439
• Code URL: null
• Copy Paste: [[2308.04439]] Global Differential Privacy for Distributed Metaverse Healthcare Systems(http://arxiv.org/abs/2308.04439)
• Summary:

  Metaverse-enabled digital healthcare systems are expected to exploit an unprecedented amount of personal health data, while ensuring that sensitive or private information of individuals are not disclosed. Machine learning and artificial intelligence (ML/AI) techniques can be widely utilized in metaverse healthcare systems, such as virtual clinics and intelligent consultations. In such scenarios, the key challenge is that data privacy laws might not allow virtual clinics to share their medical data with other parties. Moreover, clinical AI/ML models themselves carry extensive information about the medical datasets, such that private attributes can be easily inferred by malicious actors in the metaverse (if not rigorously privatized). In this paper, inspired by the idea of "incognito mode", which has recently been developed as a promising solution to safeguard metaverse users' privacy, we propose global differential privacy for the distributed metaverse healthcare systems. In our scheme, a randomized mechanism in the format of artificial "mix-up" noise is applied to the federated clinical ML/AI models before sharing with other peers. This way, we provide an adjustable level of distributed privacy against both the malicious actors and honest-but curious metaverse servers. Our evaluations on breast cancer Wisconsin dataset (BCWD) highlight the privacy-utility trade-off (PUT) in terms of diagnosis accuracy and loss function for different levels of privacy. We also compare our private scheme with the non-private centralized setup in terms of diagnosis accuracy.

Title: Blockchain-based Optimized Client Selection and Privacy Preserved Framework for Federated Learning. (arXiv:2308.04442v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04442
• Code URL: null
• Copy Paste: [[2308.04442]] Blockchain-based Optimized Client Selection and Privacy Preserved Framework for Federated Learning(http://arxiv.org/abs/2308.04442)
• Summary:

  Federated learning is a distributed mechanism that trained large-scale neural network models with the participation of multiple clients and data remains on their devices, only sharing the local model updates. With this feature, federated learning is considered a secure solution for data privacy issues. However, the typical FL structure relies on the client-server, which leads to the single-point-of-failure (SPoF) attack, and the random selection of clients for model training compromised the model accuracy. Furthermore, adversaries try for inference attacks i.e., attack on privacy leads to gradient leakage attacks. We proposed the blockchain-based optimized client selection and privacy-preserved framework in this context. We designed the three kinds of smart contracts such as 1) registration of clients 2) forward bidding to select optimized clients for FL model training 3) payment settlement and reward smart contracts. Moreover, fully homomorphic encryption with Cheon, Kim, Kim, and Song (CKKS) method is implemented before transmitting the local model updates to the server. Finally, we evaluated our proposed method on the benchmark dataset and compared it with state-of-the-art studies. Consequently, we achieved a higher accuracy rate and privacy-preserved FL framework with decentralized nature.

Title: Anonymizing Speech: Evaluating and Designing Speaker Anonymization Techniques. (arXiv:2308.04455v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04455
• Code URL: https://github.com/deep-privacy/SA-toolkit
• Copy Paste: [[2308.04455]] Anonymizing Speech: Evaluating and Designing Speaker Anonymization Techniques(http://arxiv.org/abs/2308.04455)
• Summary:

  The growing use of voice user interfaces has led to a surge in the collection and storage of speech data. While data collection allows for the development of efficient tools powering most speech services, it also poses serious privacy issues for users as centralized storage makes private personal speech data vulnerable to cyber threats. With the increasing use of voice-based digital assistants like Amazon's Alexa, Google's Home, and Apple's Siri, and with the increasing ease with which personal speech data can be collected, the risk of malicious use of voice-cloning and speaker/gender/pathological/etc. recognition has increased.

  This thesis proposes solutions for anonymizing speech and evaluating the degree of the anonymization. In this work, anonymization refers to making personal speech data unlinkable to an identity while maintaining the usefulness (utility) of the speech signal (e.g., access to linguistic content). We start by identifying several challenges that evaluation protocols need to consider to evaluate the degree of privacy protection properly. We clarify how anonymization systems must be configured for evaluation purposes and highlight that many practical deployment configurations do not permit privacy evaluation. Furthermore, we study and examine the most common voice conversion-based anonymization system and identify its weak points before suggesting new methods to overcome some limitations. We isolate all components of the anonymization system to evaluate the degree of speaker PPI associated with each of them. Then, we propose several transformation methods for each component to reduce as much as possible speaker PPI while maintaining utility. We promote anonymization algorithms based on quantization-based transformation as an alternative to the most-used and well-known noise-based approach. Finally, we endeavor a new attack method to invert anonymization.

Title: Collaborative Learning From Distributed Data With Differentially Private Synthetic Twin Data. (arXiv:2308.04755v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04755
• Code URL: https://github.com/dpbayes/collaborative-learning-with-dp-synthetic-twin-data
• Copy Paste: [[2308.04755]] Collaborative Learning From Distributed Data With Differentially Private Synthetic Twin Data(http://arxiv.org/abs/2308.04755)
• Summary:

  Consider a setting where multiple parties holding sensitive data aim to collaboratively learn population level statistics, but pooling the sensitive data sets is not possible. We propose a framework in which each party shares a differentially private synthetic twin of their data. We study the feasibility of combining such synthetic twin data sets for collaborative learning on real-world health data from the UK Biobank. We discover that parties engaging in the collaborative learning via shared synthetic data obtain more accurate estimates of target statistics compared to using only their local data. This finding extends to the difficult case of small heterogeneous data sets. Furthermore, the more parties participate, the larger and more consistent the improvements become. Finally, we find that data sharing can especially help parties whose data contain underrepresented groups to perform better-adjusted analysis for said groups. Based on our results we conclude that sharing of synthetic twins is a viable method for enabling learning from sensitive data without violating privacy constraints even if individual data sets are small or do not represent the overall population well. The setting of distributed sensitive data is often a bottleneck in biomedical research, which our study shows can be alleviated with privacy-preserving collaborative learning methods.

Title: Differentially Private Graph Neural Network with Importance-Grained Noise Adaption. (arXiv:2308.04943v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04943
• Code URL: null
• Copy Paste: [[2308.04943]] Differentially Private Graph Neural Network with Importance-Grained Noise Adaption(http://arxiv.org/abs/2308.04943)
• Summary:

  Graph Neural Networks (GNNs) with differential privacy have been proposed to preserve graph privacy when nodes represent personal and sensitive information. However, the existing methods ignore that nodes with different importance may yield diverse privacy demands, which may lead to over-protect some nodes and decrease model utility. In this paper, we study the problem of importance-grained privacy, where nodes contain personal data that need to be kept private but are critical for training a GNN. We propose NAP-GNN, a node-importance-grained privacy-preserving GNN algorithm with privacy guarantees based on adaptive differential privacy to safeguard node information. First, we propose a Topology-based Node Importance Estimation (TNIE) method to infer unknown node importance with neighborhood and centrality awareness. Second, an adaptive private aggregation method is proposed to perturb neighborhood aggregation from node-importance-grain. Third, we propose to privately train a graph learning algorithm on perturbed aggregations in adaptive residual connection mode over multi-layers convolution for node-wise tasks. Theoretically analysis shows that NAP-GNN satisfies privacy guarantees. Empirical experiments over real-world graph datasets show that NAP-GNN achieves a better trade-off between privacy and accuracy.

protect

defense

Title: Different Mechanisms of Machine Learning and Optimization Algorithms Utilized in Intrusion Detection Systems. (arXiv:2308.04607v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04607
• Code URL: null
• Copy Paste: [[2308.04607]] Different Mechanisms of Machine Learning and Optimization Algorithms Utilized in Intrusion Detection Systems(http://arxiv.org/abs/2308.04607)
• Summary:

  Malicious software is an integral part of cybercrime defense. Due to the growing number of malicious attacks and their target sources, detecting and preventing the attack becomes more challenging due to the assault's changing behavior. The bulk of classic malware detection systems is based on statistics, analytic techniques, or machine learning. Virus signature methods are widely used to identify malware. The bulk of anti-malware systems categorizes malware using regular expressions and patterns. While antivirus software is less likely to update its databases to identify and block malware, file features must be updated to detect and prevent newly generated malware. Creating attack signatures requires practically all of a human being's work. The purpose of this study is to undertake a review of the current research on intrusion detection models and the datasets that support them. In this article, we discuss the state-of-the-art, focusing on the strategy that was devised and executed, the dataset that was utilized, the findings, and the assessment that was undertaken. Additionally, the surveyed articles undergo critical analysis and statements in order to give a thorough comparative review. Machine learning and deep learning methods, as well as new classification and feature selection methodologies, are studied and researched. Thus far, each technique has proved the capability of constructing very accurate intrusion detection models. The survey findings reveal that Clearly, the MultiTree and adaptive voting algorithms surpassed all other models in terms of persistency and performance, averaging 99.98 percent accuracy on average.

attack

Title: Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attacks. (arXiv:2308.04451v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04451
• Code URL: null
• Copy Paste: [[2308.04451]] Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attacks(http://arxiv.org/abs/2308.04451)
• Summary:

  In this work, we assess the security of AI code generators via data poisoning, i.e., an attack that injects malicious samples into the training data to generate vulnerable code. We poison the training data by injecting increasing amounts of code containing security vulnerabilities and assess the attack's success on different state-of-the-art models for code generation. Our analysis shows that AI code generators are vulnerable to even a small amount of data poisoning. Moreover, the attack does not impact the correctness of code generated by pre-trained models, making it hard to detect.

Title: Improved Activation Clipping for Universal Backdoor Mitigation and Test-Time Detection. (arXiv:2308.04617v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04617
• Code URL: https://github.com/wanghangpsu/mmac
• Copy Paste: [[2308.04617]] Improved Activation Clipping for Universal Backdoor Mitigation and Test-Time Detection(http://arxiv.org/abs/2308.04617)
• Summary:

  Deep neural networks are vulnerable to backdoor attacks (Trojans), where an attacker poisons the training set with backdoor triggers so that the neural network learns to classify test-time triggers to the attacker's designated target class. Recent work shows that backdoor poisoning induces over-fitting (abnormally large activations) in the attacked model, which motivates a general, post-training clipping method for backdoor mitigation, i.e., with bounds on internal-layer activations learned using a small set of clean samples. We devise a new such approach, choosing the activation bounds to explicitly limit classification margins. This method gives superior performance against peer methods for CIFAR-10 image classification. We also show that this method has strong robustness against adaptive attacks, X2X attacks, and on different datasets. Finally, we demonstrate a method extension for test-time detection and correction based on the output differences between the original and activation-bounded networks. The code of our method is online available.

Title: A Feature Set of Small Size for the PDF Malware Detection. (arXiv:2308.04704v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04704
• Code URL: null
• Copy Paste: [[2308.04704]] A Feature Set of Small Size for the PDF Malware Detection(http://arxiv.org/abs/2308.04704)
• Summary:

  Machine learning (ML)-based malware detection systems are becoming increasingly important as malware threats increase and get more sophisticated. PDF files are often used as vectors for phishing attacks because they are widely regarded as trustworthy data resources, and are accessible across different platforms. Therefore, researchers have developed many different PDF malware detection methods. Performance in detecting PDF malware is greatly influenced by feature selection. In this research, we propose a small features set that don't require too much domain knowledge of the PDF file. We evaluate proposed features with six different machine learning models. We report the best accuracy of 99.75% when using Random Forest model. Our proposed feature set, which consists of just 12 features, is one of the most conciseness in the field of PDF malware detection. Despite its modest size, we obtain comparable results to state-of-the-art that employ a much larger set of features.

Title: Kairos: : Practical Intrusion Detection and Investigation using Whole-system Provenance. (arXiv:2308.05034v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05034
• Code URL: https://github.com/provenanceanalytics/kairos
• Copy Paste: [[2308.05034]] Kairos: : Practical Intrusion Detection and Investigation using Whole-system Provenance(http://arxiv.org/abs/2308.05034)
• Summary:

  Provenance graphs are structured audit logs that describe the history of a system's execution. Recent studies have explored a variety of techniques to analyze provenance graphs for automated host intrusion detection, focusing particularly on advanced persistent threats. Sifting through their design documents, we identify four common dimensions that drive the development of provenance-based intrusion detection systems (PIDSes): scope (can PIDSes detect modern attacks that infiltrate across application boundaries?), attack agnosticity (can PIDSes detect novel attacks without a priori knowledge of attack characteristics?), timeliness (can PIDSes efficiently monitor host systems as they run?), and attack reconstruction (can PIDSes distill attack activity from large provenance graphs so that sysadmins can easily understand and quickly respond to system intrusion?). We present KAIROS, the first PIDS that simultaneously satisfies the desiderata in all four dimensions, whereas existing approaches sacrifice at least one and struggle to achieve comparable detection performance.

  Kairos leverages a novel graph neural network-based encoder-decoder architecture that learns the temporal evolution of a provenance graph's structural changes to quantify the degree of anomalousness for each system event. Then, based on this fine-grained information, Kairos reconstructs attack footprints, generating compact summary graphs that accurately describe malicious activity over a stream of system audit logs. Using state-of-the-art benchmark datasets, we demonstrate that Kairos outperforms previous approaches.

robust

Title: From Fake to Real (FFR): A two-stage training pipeline for mitigating spurious correlations with synthetic data. (arXiv:2308.04553v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04553
• Code URL: null
• Copy Paste: [[2308.04553]] From Fake to Real (FFR): A two-stage training pipeline for mitigating spurious correlations with synthetic data(http://arxiv.org/abs/2308.04553)
• Summary:

  Visual recognition models are prone to learning spurious correlations induced by an imbalanced training set where certain groups (\eg Females) are under-represented in certain classes (\eg Programmers). Generative models offer a promising direction in mitigating this bias by generating synthetic data for the minority samples and thus balancing the training set. However, prior work that uses these approaches overlooks that visual recognition models could often learn to differentiate between real and synthetic images and thus fail to unlearn the bias in the original dataset. In our work, we propose a novel two-stage pipeline to mitigate this issue where 1) we pre-train a model on a balanced synthetic dataset and then 2) fine-tune on the real data. Using this pipeline, we avoid training on both real and synthetic data, thus avoiding the bias between real and synthetic data. Moreover, we learn robust features against the bias in the first step that mitigate the bias in the second step. Moreover, our pipeline naturally integrates with bias mitigation methods; they can be simply applied to the fine-tuning step. As our experiments prove, our pipeline can further improve the performance of bias mitigation methods obtaining state-of-the-art performance on three large-scale datasets.

Title: GeoAdapt: Self-Supervised Test-Time Adaption in LiDAR Place Recognition Using Geometric Priors. (arXiv:2308.04638v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04638
• Code URL: null
• Copy Paste: [[2308.04638]] GeoAdapt: Self-Supervised Test-Time Adaption in LiDAR Place Recognition Using Geometric Priors(http://arxiv.org/abs/2308.04638)
• Summary:

  LiDAR place recognition approaches based on deep learning suffer a significant degradation in performance when there is a shift between the distribution of the training and testing datasets, with re-training often required to achieve top performance. However, obtaining accurate ground truth on new environments can be prohibitively expensive, especially in complex or GPS-deprived environments. To address this issue we propose GeoAdapt, which introduces a novel auxiliary classification head to generate pseudo-labels for re-training on unseen environments in a self-supervised manner. GeoAdapt uses geometric consistency as a prior to improve the robustness of our generated pseudo-labels against domain shift, improving the performance and reliability of our Test-Time Adaptation approach. Comprehensive experiments show that GeoAdapt significantly boosts place recognition performance across moderate to severe domain shifts, and is competitive with fully supervised test-time adaptation approaches. Our code will be available at https://github.com/csiro-robotics/GeoAdapt.

Title: SAfER: Layer-Level Sensitivity Assessment for Efficient and Robust Neural Network Inference. (arXiv:2308.04753v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04753
• Code URL: null
• Copy Paste: [[2308.04753]] SAfER: Layer-Level Sensitivity Assessment for Efficient and Robust Neural Network Inference(http://arxiv.org/abs/2308.04753)
• Summary:

  Deep neural networks (DNNs) demonstrate outstanding performance across most computer vision tasks. Some critical applications, such as autonomous driving or medical imaging, also require investigation into their behavior and the reasons behind the decisions they make. In this vein, DNN attribution consists in studying the relationship between the predictions of a DNN and its inputs. Attribution methods have been adapted to highlight the most relevant weights or neurons in a DNN, allowing to more efficiently select which weights or neurons can be pruned. However, a limitation of these approaches is that weights are typically compared within each layer separately, while some layers might appear as more critical than others. In this work, we propose to investigate DNN layer importance, i.e. to estimate the sensitivity of the accuracy w.r.t. perturbations applied at the layer level. To do so, we propose a novel dataset to evaluate our method as well as future works. We benchmark a number of criteria and draw conclusions regarding how to assess DNN layer importance and, consequently, how to budgetize layers for increased DNN efficiency (with applications for DNN pruning and quantization), as well as robustness to hardware failure (e.g. bit swaps).

Title: Induction Network: Audio-Visual Modality Gap-Bridging for Self-Supervised Sound Source Localization. (arXiv:2308.04767v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04767
• Code URL: https://github.com/tahy1/avin
• Copy Paste: [[2308.04767]] Induction Network: Audio-Visual Modality Gap-Bridging for Self-Supervised Sound Source Localization(http://arxiv.org/abs/2308.04767)
• Summary:

  Self-supervised sound source localization is usually challenged by the modality inconsistency. In recent studies, contrastive learning based strategies have shown promising to establish such a consistent correspondence between audio and sound sources in visual scenarios. Unfortunately, the insufficient attention to the heterogeneity influence in the different modality features still limits this scheme to be further improved, which also becomes the motivation of our work. In this study, an Induction Network is proposed to bridge the modality gap more effectively. By decoupling the gradients of visual and audio modalities, the discriminative visual representations of sound sources can be learned with the designed Induction Vector in a bootstrap manner, which also enables the audio modality to be aligned with the visual modality consistently. In addition to a visual weighted contrastive loss, an adaptive threshold selection strategy is introduced to enhance the robustness of the Induction Network. Substantial experiments conducted on SoundNet-Flickr and VGG-Sound Source datasets have demonstrated a superior performance compared to other state-of-the-art works in different challenging scenarios. The code is available at https://github.com/Tahy1/AVIN

Title: SUnAA: Sparse Unmixing using Archetypal Analysis. (arXiv:2308.04771v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04771
• Code URL: https://github.com/behnoodrasti/sunaa
• Copy Paste: [[2308.04771]] SUnAA: Sparse Unmixing using Archetypal Analysis(http://arxiv.org/abs/2308.04771)
• Summary:

  This paper introduces a new sparse unmixing technique using archetypal analysis (SUnAA). First, we design a new model based on archetypal analysis. We assume that the endmembers of interest are a convex combination of endmembers provided by a spectral library and that the number of endmembers of interest is known. Then, we propose a minimization problem. Unlike most conventional sparse unmixing methods, here the minimization problem is non-convex. We minimize the optimization objective iteratively using an active set algorithm. Our method is robust to the initialization and only requires the number of endmembers of interest. SUnAA is evaluated using two simulated datasets for which results confirm its better performance over other conventional and advanced techniques in terms of signal-to-reconstruction error. SUnAA is also applied to Cuprite dataset and the results are compared visually with the available geological map provided for this dataset. The qualitative assessment demonstrates the successful estimation of the minerals abundances and significantly improves the detection of dominant minerals compared to the conventional regression-based sparse unmixing methods. The Python implementation of SUnAA can be found at: https://github.com/BehnoodRasti/SUnAA.

Title: Learning multi-domain feature relation for visible and Long-wave Infrared image patch matching. (arXiv:2308.04880v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04880
• Code URL: null
• Copy Paste: [[2308.04880]] Learning multi-domain feature relation for visible and Long-wave Infrared image patch matching(http://arxiv.org/abs/2308.04880)
• Summary:

  Recently, learning-based algorithms have achieved promising performance on cross-spectral image patch matching, which, however, is still far from satisfactory for practical application. On the one hand, a lack of large-scale dataset with diverse scenes haunts its further improvement for learning-based algorithms, whose performances and generalization rely heavily on the dataset size and diversity. On the other hand, more emphasis has been put on feature relation in the spatial domain whereas the scale dependency between features has often been ignored, leading to performance degeneration especially when encountering significant appearance variations for cross-spectral patches. To address these issues, we publish, to be best of our knowledge, the largest visible and Long-wave Infrared (LWIR) image patch matching dataset, termed VL-CMIM, which contains 1300 pairs of strictly aligned visible and LWIR images and over 2 million patch pairs covering diverse scenes such as asteroid, field, country, build, street and water.In addition, a multi-domain feature relation learning network (MD-FRN) is proposed. Input by the features extracted from a four-branch network, both feature relations in spatial and scale domains are learned via a spatial correlation module (SCM) and multi-scale adaptive aggregation module (MSAG), respectively. To further aggregate the multi-domain relations, a deep domain interactive mechanism (DIM) is applied, where the learnt spatial-relation and scale-relation features are exchanged and further input into MSCRM and SCM. This mechanism allows our model to learn interactive cross-domain feature relations, leading to improved robustness to significant appearance changes due to different modality.

Title: GeodesicPSIM: Predicting the Quality of Static Mesh with Texture Map via Geodesic Patch Similarity. (arXiv:2308.04928v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04928
• Code URL: null
• Copy Paste: [[2308.04928]] GeodesicPSIM: Predicting the Quality of Static Mesh with Texture Map via Geodesic Patch Similarity(http://arxiv.org/abs/2308.04928)
• Summary:

  Static meshes with texture maps have attracted considerable attention in both industrial manufacturing and academic research, leading to an urgent requirement for effective and robust objective quality evaluation. However, current model-based static mesh quality metrics have obvious limitations: most of them only consider geometry information, while color information is ignored, and they have strict constraints for the meshes' geometrical topology. Other metrics, such as image-based and point-based metrics, are easily influenced by the prepossessing algorithms, e.g., projection and sampling, hampering their ability to perform at their best. In this paper, we propose Geodesic Patch Similarity (GeodesicPSIM), a novel model-based metric to accurately predict human perception quality for static meshes. After selecting a group keypoints, 1-hop geodesic patches are constructed based on both the reference and distorted meshes cleaned by an effective mesh cleaning algorithm. A two-step patch cropping algorithm and a patch texture mapping module refine the size of 1-hop geodesic patches and build the relationship between the mesh geometry and color information, resulting in the generation of 1-hop textured geodesic patches. Three types of features are extracted to quantify the distortion: patch color smoothness, patch discrete mean curvature, and patch pixel color average and variance. To the best of our knowledge, GeodesicPSIM is the first model-based metric especially designed for static meshes with texture maps. GeodesicPSIM provides state-of-the-art performance in comparison with image-based, point-based, and video-based metrics on a newly created and challenging database. We also prove the robustness of GeodesicPSIM by introducing different settings of hyperparameters. Ablation studies also exhibit the effectiveness of three proposed features and the patch cropping algorithm.

Title: An End-to-End Framework of Road User Detection, Tracking, and Prediction from Monocular Images. (arXiv:2308.05026v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05026
• Code URL: null
• Copy Paste: [[2308.05026]] An End-to-End Framework of Road User Detection, Tracking, and Prediction from Monocular Images(http://arxiv.org/abs/2308.05026)
• Summary:

  Perception that involves multi-object detection and tracking, and trajectory prediction are two major tasks of autonomous driving. However, they are currently mostly studied separately, which results in most trajectory prediction modules being developed based on ground truth trajectories without taking into account that trajectories extracted from the detection and tracking modules in real-world scenarios are noisy. These noisy trajectories can have a significant impact on the performance of the trajectory predictor and can lead to serious prediction errors. In this paper, we build an end-to-end framework for detection, tracking, and trajectory prediction called ODTP (Online Detection, Tracking and Prediction). It adopts the state-of-the-art online multi-object tracking model, QD-3DT, for perception and trains the trajectory predictor, DCENet++, directly based on the detection results without purely relying on ground truth trajectories. We evaluate the performance of ODTP on the widely used nuScenes dataset for autonomous driving. Extensive experiments show that ODPT achieves high performance end-to-end trajectory prediction. DCENet++, with the enhanced dynamic maps, predicts more accurate trajectories than its base model. It is also more robust when compared with other generative and deterministic trajectory prediction models trained on noisy detection results.

Title: Single-Sentence Reader: A Novel Approach for Addressing Answer Position Bias. (arXiv:2308.04566v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04566
• Code URL: https://github.com/sonqt/single-sentence-reader
• Copy Paste: [[2308.04566]] Single-Sentence Reader: A Novel Approach for Addressing Answer Position Bias(http://arxiv.org/abs/2308.04566)
• Summary:

  Machine Reading Comprehension (MRC) models tend to take advantage of spurious correlations (also known as dataset bias or annotation artifacts in the research community). Consequently, these models may perform the MRC task without fully comprehending the given context and question, which is undesirable since it may result in low robustness against distribution shift. This paper delves into the concept of answer-position bias, where a significant percentage of training questions have answers located solely in the first sentence of the context. We propose a Single-Sentence Reader as a new approach for addressing answer position bias in MRC. We implement this approach using six different models and thoroughly analyze their performance. Remarkably, our proposed Single-Sentence Readers achieve results that nearly match those of models trained on conventional training sets, proving their effectiveness. Our study also discusses several challenges our Single-Sentence Readers encounter and proposes a potential solution.

Title: Improving Performance in Continual Learning Tasks using Bio-Inspired Architectures. (arXiv:2308.04539v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04539
• Code URL: null
• Copy Paste: [[2308.04539]] Improving Performance in Continual Learning Tasks using Bio-Inspired Architectures(http://arxiv.org/abs/2308.04539)
• Summary:

  The ability to learn continuously from an incoming data stream without catastrophic forgetting is critical to designing intelligent systems. Many approaches to continual learning rely on stochastic gradient descent and its variants that employ global error updates, and hence need to adopt strategies such as memory buffers or replay to circumvent its stability, greed, and short-term memory limitations. To address this limitation, we have developed a biologically inspired lightweight neural network architecture that incorporates synaptic plasticity mechanisms and neuromodulation and hence learns through local error signals to enable online continual learning without stochastic gradient descent.

  Our approach leads to superior online continual learning performance on Split-MNIST, Split-CIFAR-10, and Split-CIFAR-100 datasets compared to other memory-constrained learning approaches and matches that of the state-of-the-art memory-intensive replay-based approaches. We further demonstrate the effectiveness of our approach by integrating key design concepts into other backpropagation-based continual learning algorithms, significantly improving their accuracy. Our results provide compelling evidence for the importance of incorporating biological principles into machine learning models and offer insights into how we can leverage them to design more efficient and robust systems for online continual learning.

Title: ScatterUQ: Interactive Uncertainty Visualizations for Multiclass Deep Learning Problems. (arXiv:2308.04588v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04588
• Code URL: https://github.com/mit-ll-responsible-ai/equine-webapp
• Copy Paste: [[2308.04588]] ScatterUQ: Interactive Uncertainty Visualizations for Multiclass Deep Learning Problems(http://arxiv.org/abs/2308.04588)
• Summary:

  Recently, uncertainty-aware deep learning methods for multiclass labeling problems have been developed that provide calibrated class prediction probabilities and out-of-distribution (OOD) indicators, letting machine learning (ML) consumers and engineers gauge a model's confidence in its predictions. However, this extra neural network prediction information is challenging to scalably convey visually for arbitrary data sources under multiple uncertainty contexts. To address these challenges, we present ScatterUQ, an interactive system that provides targeted visualizations to allow users to better understand model performance in context-driven uncertainty settings. ScatterUQ leverages recent advances in distance-aware neural networks, together with dimensionality reduction techniques, to construct robust, 2-D scatter plots explaining why a model predicts a test example to be (1) in-distribution and of a particular class, (2) in-distribution but unsure of the class, and (3) out-of-distribution. ML consumers and engineers can visually compare the salient features of test samples with training examples through the use of a ``hover callback'' to understand model uncertainty performance and decide follow up courses of action. We demonstrate the effectiveness of ScatterUQ to explain model uncertainty for a multiclass image classification on a distance-aware neural network trained on Fashion-MNIST and tested on Fashion-MNIST (in distribution) and MNIST digits (out of distribution), as well as a deep learning model for a cyber dataset. We quantitatively evaluate dimensionality reduction techniques to optimize our contextually driven UQ visualizations. Our results indicate that the ScatterUQ system should scale to arbitrary, multiclass datasets. Our code is available at https://github.com/mit-ll-responsible-ai/equine-webapp

Title: Machine Learning, Deep Learning and Data Preprocessing Techniques for Detection, Prediction, and Monitoring of Stress and Stress-related Mental Disorders: A Scoping Review. (arXiv:2308.04616v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04616
• Code URL: null
• Copy Paste: [[2308.04616]] Machine Learning, Deep Learning and Data Preprocessing Techniques for Detection, Prediction, and Monitoring of Stress and Stress-related Mental Disorders: A Scoping Review(http://arxiv.org/abs/2308.04616)
• Summary:

  This comprehensive review systematically evaluates Machine Learning (ML) methodologies employed in the detection, prediction, and analysis of mental stress and its consequent mental disorders (MDs). Utilizing a rigorous scoping review process, the investigation delves into the latest ML algorithms, preprocessing techniques, and data types employed in the context of stress and stress-related MDs. The findings highlight that Support Vector Machine (SVM), Neural Network (NN), and Random Forest (RF) models consistently exhibit superior accuracy and robustness among all machine learning algorithms examined. Furthermore, the review underscores that physiological parameters, such as heart rate measurements and skin response, are prevalently used as stress predictors in ML algorithms. This is attributed to their rich explanatory information concerning stress and stress-related MDs, as well as the relative ease of data acquisition. Additionally, the application of dimensionality reduction techniques, including mappings, feature selection, filtering, and noise reduction, is frequently observed as a crucial step preceding the training of ML algorithms. The synthesis of this review identifies significant research gaps and outlines future directions for the field. These encompass areas such as model interpretability, model personalization, the incorporation of naturalistic settings, and real-time processing capabilities for detection and prediction of stress and stress-related MDs.

Title: Deep Metric Learning for the Hemodynamics Inference with Electrocardiogram Signals. (arXiv:2308.04650v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04650
• Code URL: null
• Copy Paste: [[2308.04650]] Deep Metric Learning for the Hemodynamics Inference with Electrocardiogram Signals(http://arxiv.org/abs/2308.04650)
• Summary:

  Heart failure is a debilitating condition that affects millions of people worldwide and has a significant impact on their quality of life and mortality rates. An objective assessment of cardiac pressures remains an important method for the diagnosis and treatment prognostication for patients with heart failure. Although cardiac catheterization is the gold standard for estimating central hemodynamic pressures, it is an invasive procedure that carries inherent risks, making it a potentially dangerous procedure for some patients. Approaches that leverage non-invasive signals - such as electrocardiogram (ECG) - have the promise to make the routine estimation of cardiac pressures feasible in both inpatient and outpatient settings. Prior models trained to estimate intracardiac pressures (e.g., mean pulmonary capillary wedge pressure (mPCWP)) in a supervised fashion have shown good discriminatory ability but have been limited to the labeled dataset from the heart failure cohort. To address this issue and build a robust representation, we apply deep metric learning (DML) and propose a novel self-supervised DML with distance-based mining that improves the performance of a model with limited labels. We use a dataset that contains over 5.4 million ECGs without concomitant central pressure labels to pre-train a self-supervised DML model which showed improved classification of elevated mPCWP compared to self-supervised contrastive baselines. Additionally, the supervised DML model that is using ECGs with access to 8,172 mPCWP labels demonstrated significantly better performance on the mPCWP regression task compared to the supervised baseline. Moreover, our data suggest that DML yields models that are performant across patient subgroups, even when some patient subgroups are under-represented in the dataset. Our code is available at https://github.com/mandiehyewon/ssldml

biometric

steal

extraction

Title: Multi-View Fusion and Distillation for Subgrade Distresses Detection based on 3D-GPR. (arXiv:2308.04779v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04779
• Code URL: null
• Copy Paste: [[2308.04779]] Multi-View Fusion and Distillation for Subgrade Distresses Detection based on 3D-GPR(http://arxiv.org/abs/2308.04779)
• Summary:

  The application of 3D ground-penetrating radar (3D-GPR) for subgrade distress detection has gained widespread popularity. To enhance the efficiency and accuracy of detection, pioneering studies have attempted to adopt automatic detection techniques, particularly deep learning. However, existing works typically rely on traditional 1D A-scan, 2D B-scan or 3D C-scan data of the GPR, resulting in either insufficient spatial information or high computational complexity. To address these challenges, we introduce a novel methodology for the subgrade distress detection task by leveraging the multi-view information from 3D-GPR data. Moreover, we construct a real multi-view image dataset derived from the original 3D-GPR data for the detection task, which provides richer spatial information compared to A-scan and B-scan data, while reducing computational complexity compared to C-scan data. Subsequently, we develop a novel \textbf{M}ulti-\textbf{V}iew \textbf{V}usion and \textbf{D}istillation framework, \textbf{GPR-MVFD}, specifically designed to optimally utilize the multi-view GPR dataset. This framework ingeniously incorporates multi-view distillation and attention-based fusion to facilitate significant feature extraction for subgrade distresses. In addition, a self-adaptive learning mechanism is adopted to stabilize the model training and prevent performance degeneration in each branch. Extensive experiments conducted on this new GPR benchmark demonstrate the effectiveness and efficiency of our proposed framework. Our framework outperforms not only the existing GPR baselines, but also the state-of-the-art methods in the fields of multi-view learning, multi-modal learning, and knowledge distillation. We will release the constructed multi-view GPR dataset with expert-annotated labels and the source codes of the proposed framework.

Title: DialogRE^C+: An Extension of DialogRE to Investigate How Much Coreference Helps Relation Extraction in Dialogs. (arXiv:2308.04498v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04498
• Code URL: https://github.com/palm2333/dialogre_coreference
• Copy Paste: [[2308.04498]] DialogRE^C+: An Extension of DialogRE to Investigate How Much Coreference Helps Relation Extraction in Dialogs(http://arxiv.org/abs/2308.04498)
• Summary:

  Dialogue relation extraction (DRE) that identifies the relations between argument pairs in dialogue text, suffers much from the frequent occurrence of personal pronouns, or entity and speaker coreference. This work introduces a new benchmark dataset DialogRE^C+, introducing coreference resolution into the DRE scenario. With the aid of high-quality coreference knowledge, the reasoning of argument relations is expected to be enhanced. In DialogRE^C+ dataset, we manually annotate total 5,068 coreference chains over 36,369 argument mentions based on the existing DialogRE data, where four different coreference chain types namely speaker chain, person chain, location chain and organization chain are explicitly marked. We further develop 4 coreference-enhanced graph-based DRE models, which learn effective coreference representations for improving the DRE task. We also train a coreference resolution model based on our annotations and evaluate the effect of automatically extracted coreference chains demonstrating the practicality of our dataset and its potential to other domains and tasks.

Title: Ahead of the Text: Leveraging Entity Preposition for Financial Relation Extraction. (arXiv:2308.04534v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04534
• Code URL: null
• Copy Paste: [[2308.04534]] Ahead of the Text: Leveraging Entity Preposition for Financial Relation Extraction(http://arxiv.org/abs/2308.04534)
• Summary:

  In the context of the ACM KDF-SIGIR 2023 competition, we undertook an entity relation task on a dataset of financial entity relations called REFind. Our top-performing solution involved a multi-step approach. Initially, we inserted the provided entities at their corresponding locations within the text. Subsequently, we fine-tuned the transformer-based language model roberta-large for text classification by utilizing a labeled training set to predict the entity relations. Lastly, we implemented a post-processing phase to identify and handle improbable predictions generated by the model. As a result of our methodology, we achieved the 1st place ranking on the competition's public leaderboard.

Title: RadGraph2: Modeling Disease Progression in Radiology Reports via Hierarchical Information Extraction. (arXiv:2308.05046v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.05046
• Code URL: null
• Copy Paste: [[2308.05046]] RadGraph2: Modeling Disease Progression in Radiology Reports via Hierarchical Information Extraction(http://arxiv.org/abs/2308.05046)
• Summary:

  We present RadGraph2, a novel dataset for extracting information from radiology reports that focuses on capturing changes in disease state and device placement over time. We introduce a hierarchical schema that organizes entities based on their relationships and show that using this hierarchy during training improves the performance of an information extraction model. Specifically, we propose a modification to the DyGIE++ framework, resulting in our model HGIE, which outperforms previous models in entity and relation extraction tasks. We demonstrate that RadGraph2 enables models to capture a wider variety of findings and perform better at relation extraction compared to those trained on the original RadGraph dataset. Our work provides the foundation for developing automated systems that can track disease progression over time and develop information extraction models that leverage the natural hierarchy of labels in the medical domain.

membership infer

federate

Title: Backdoor Federated Learning by Poisoning Backdoor-Critical Layers. (arXiv:2308.04466v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04466
• Code URL: null
• Copy Paste: [[2308.04466]] Backdoor Federated Learning by Poisoning Backdoor-Critical Layers(http://arxiv.org/abs/2308.04466)
• Summary:

  Federated learning (FL) has been widely deployed to enable machine learning training on sensitive data across distributed devices. However, the decentralized learning paradigm and heterogeneity of FL further extend the attack surface for backdoor attacks. Existing FL attack and defense methodologies typically focus on the whole model. None of them recognizes the existence of backdoor-critical (BC) layers-a small subset of layers that dominate the model vulnerabilities. Attacking the BC layers achieves equivalent effects as attacking the whole model but at a far smaller chance of being detected by state-of-the-art (SOTA) defenses. This paper proposes a general in-situ approach that identifies and verifies BC layers from the perspective of attackers. Based on the identified BC layers, we carefully craft a new backdoor attack methodology that adaptively seeks a fundamental balance between attacking effects and stealthiness under various defense strategies. Extensive experiments show that our BC layer-aware backdoor attacks can successfully backdoor FL under seven SOTA defenses with only 10% malicious clients and outperform the latest backdoor attack methods.

Title: A Survey on Decentralized Federated Learning. (arXiv:2308.04604v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04604
• Code URL: null
• Copy Paste: [[2308.04604]] A Survey on Decentralized Federated Learning(http://arxiv.org/abs/2308.04604)
• Summary:

  In recent years, federated learning (FL) has become a very popular paradigm for training distributed, large-scale, and privacy-preserving machine learning (ML) systems. In contrast to standard ML, where data must be collected at the exact location where training is performed, FL takes advantage of the computational capabilities of millions of edge devices to collaboratively train a shared, global model without disclosing their local private data. Specifically, in a typical FL system, the central server acts only as an orchestrator; it iteratively gathers and aggregates all the local models trained by each client on its private data until convergence. Although FL undoubtedly has several benefits over traditional ML (e.g., it protects private data ownership by design), it suffers from several weaknesses. One of the most critical challenges is to overcome the centralized orchestration of the classical FL client-server architecture, which is known to be vulnerable to single-point-of-failure risks and man-in-the-middle attacks, among others. To mitigate such exposure, decentralized FL solutions have emerged where all FL clients cooperate and communicate without a central server. This survey comprehensively summarizes and reviews existing decentralized FL approaches proposed in the literature. Furthermore, it identifies emerging challenges and suggests promising research directions in this under-explored domain.

Title: Communication-Efficient Search under Fully Homomorphic Encryption for Federated Machine Learning. (arXiv:2308.04648v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04648
• Code URL: null
• Copy Paste: [[2308.04648]] Communication-Efficient Search under Fully Homomorphic Encryption for Federated Machine Learning(http://arxiv.org/abs/2308.04648)
• Summary:

  Homomorphic encryption (HE) has found extensive utilization in federated learning (FL) systems, capitalizing on its dual advantages: (i) ensuring the confidentiality of shared models contributed by participating entities, and (ii) enabling algebraic operations directly on ciphertexts representing encrypted models. Particularly, the approximate fully homomorphic encryption (FHE) scheme, known as CKKS, has emerged as the de facto encryption scheme, notably supporting decimal numbers. While recent research predominantly focuses on enhancing CKKS's encryption rate and evaluation speed in the context of FL, the search operation has been relatively disregarded due to the tendency of some applications to discard intermediate encrypted models. Yet, emerging studies emphasize the importance of managing and searching intermediate models for specific applications like large-scale scientific computing, necessitating robust data provenance and auditing support. To address this, our paper introduces an innovative approach that efficiently searches for a target encrypted value, incurring only a logarithmic number of network interactions. The proposed method capitalizes on CKKS's additive and multiplicative properties on encrypted models, propagating equality comparisons between values through a balanced binary tree structure to ultimately reach a single aggregate. A comprehensive analysis of the proposed algorithm underscores its potential to significantly broaden FL's applicability and impact.

Title: Feature Matching Data Synthesis for Non-IID Federated Learning. (arXiv:2308.04761v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04761
• Code URL: null
• Copy Paste: [[2308.04761]] Feature Matching Data Synthesis for Non-IID Federated Learning(http://arxiv.org/abs/2308.04761)
• Summary:

  Federated learning (FL) has emerged as a privacy-preserving paradigm that trains neural networks on edge devices without collecting data at a central server. However, FL encounters an inherent challenge in dealing with non-independent and identically distributed (non-IID) data among devices. To address this challenge, this paper proposes a hard feature matching data synthesis (HFMDS) method to share auxiliary data besides local models. Specifically, synthetic data are generated by learning the essential class-relevant features of real samples and discarding the redundant features, which helps to effectively tackle the non-IID issue. For better privacy preservation, we propose a hard feature augmentation method to transfer real features towards the decision boundary, with which the synthetic data not only improve the model generalization but also erase the information of real features. By integrating the proposed HFMDS method with FL, we present a novel FL framework with data augmentation to relieve data heterogeneity. The theoretical analysis highlights the effectiveness of our proposed data synthesis method in solving the non-IID challenge. Simulation results further demonstrate that our proposed HFMDS-FL algorithm outperforms the baselines in terms of accuracy, privacy preservation, and computational cost on various benchmark datasets.

Title: Tram-FL: Routing-based Model Training for Decentralized Federated Learning. (arXiv:2308.04762v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04762
• Code URL: null
• Copy Paste: [[2308.04762]] Tram-FL: Routing-based Model Training for Decentralized Federated Learning(http://arxiv.org/abs/2308.04762)
• Summary:

  In decentralized federated learning (DFL), substantial traffic from frequent inter-node communication and non-independent and identically distributed (non-IID) data challenges high-accuracy model acquisition. We propose Tram-FL, a novel DFL method, which progressively refines a global model by transferring it sequentially amongst nodes, rather than by exchanging and aggregating local models. We also introduce a dynamic model routing algorithm for optimal route selection, aimed at enhancing model precision with minimal forwarding. Our experiments using MNIST, CIFAR-10, and IMDb datasets demonstrate that Tram-FL with the proposed routing delivers high model accuracy under non-IID conditions, outperforming baselines while reducing communication costs.

fair

Title: Addressing Racial Bias in Facial Emotion Recognition. (arXiv:2308.04674v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04674
• Code URL: null
• Copy Paste: [[2308.04674]] Addressing Racial Bias in Facial Emotion Recognition(http://arxiv.org/abs/2308.04674)
• Summary:

  Fairness in deep learning models trained with high-dimensional inputs and subjective labels remains a complex and understudied area. Facial emotion recognition, a domain where datasets are often racially imbalanced, can lead to models that yield disparate outcomes across racial groups. This study focuses on analyzing racial bias by sub-sampling training sets with varied racial distributions and assessing test performance across these simulations. Our findings indicate that smaller datasets with posed faces improve on both fairness and performance metrics as the simulations approach racial balance. Notably, the F1-score increases by $27.2\%$ points, and demographic parity increases by $15.7\%$ points on average across the simulations. However, in larger datasets with greater facial variation, fairness metrics generally remain constant, suggesting that racial balance by itself is insufficient to achieve parity in test performance across different racial groups.

Title: Exploring Multilingual Text Data Distillation. (arXiv:2308.04982v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04982
• Code URL: https://github.com/harshp1802/text-dataset-distillation
• Copy Paste: [[2308.04982]] Exploring Multilingual Text Data Distillation(http://arxiv.org/abs/2308.04982)
• Summary:

  With the rise of deep learning, large datasets and complex models have become common, requiring significant computing power. To address this, data distillation has emerged as a technique to quickly train models with lower memory and time requirements. However, data distillation on text-based datasets hasn't been explored much because of the challenges rising due to its discrete nature. Additionally, existing dataset distillation methods often struggle to generalize to new architectures. In the paper, we propose several data distillation techniques for multilingual text classification datasets using language-model-based learning methods. We conduct experiments to analyze their performance in terms of classification strength, and cross-architecture generalization. Furthermore, we investigate the language-specific fairness of the data summaries generated by these methods. Our approach builds upon existing techniques, enhancing cross-architecture generalization in the text data distillation domain.

Title: Fairness Notions in DAG-based DLTs. (arXiv:2308.04831v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04831
• Code URL: null
• Copy Paste: [[2308.04831]] Fairness Notions in DAG-based DLTs(http://arxiv.org/abs/2308.04831)
• Summary:

  This paper investigates the issue of fairness in Distributed Ledger Technology (DLT), specifically focusing on the shortcomings observed in current blockchain systems due to Miner Extractable Value (MEV) phenomena and systemic centralization. We explore the potential of Directed Acyclic Graphs (DAGs) as a solution to address or mitigate these fairness concerns. Our objective is to gain a comprehensive understanding of fairness in DAG-based DLTs by examining its different aspects and measurement metrics. We aim to establish a shared knowledge base that facilitates accurate fairness assessment and allows for an evaluation of whether DAG-based DLTs offer a more equitable design. We describe the various dimensions of fairness and conduct a comparative analysis to examine how they relate to different components of DLTs. This analysis serves as a catalyst for further research, encouraging the development of cryptographic systems that promote fairness.

interpretability

Title: Building Interpretable and Reliable Open Information Retriever for New Domains Overnight. (arXiv:2308.04756v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04756
• Code URL: null
• Copy Paste: [[2308.04756]] Building Interpretable and Reliable Open Information Retriever for New Domains Overnight(http://arxiv.org/abs/2308.04756)
• Summary:

  Information retrieval (IR) or knowledge retrieval, is a critical component for many down-stream tasks such as open-domain question answering (QA). It is also very challenging, as it requires succinctness, completeness, and correctness. In recent works, dense retrieval models have achieved state-of-the-art (SOTA) performance on in-domain IR and QA benchmarks by representing queries and knowledge passages with dense vectors and learning the lexical and semantic similarity. However, using single dense vectors and end-to-end supervision are not always optimal because queries may require attention to multiple aspects and event implicit knowledge. In this work, we propose an information retrieval pipeline that uses entity/event linking model and query decomposition model to focus more accurately on different information units of the query. We show that, while being more interpretable and reliable, our proposed pipeline significantly improves passage coverages and denotation accuracies across five IR and QA benchmarks. It will be the go-to system to use for applications that need to perform IR on a new domain without much dedicated effort, because of its superior interpretability and cross-domain performance.

Title: A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systems. (arXiv:2308.04457v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04457
• Code URL: null
• Copy Paste: [[2308.04457]] A Critical Review of Physics-Informed Machine Learning Applications in Subsurface Energy Systems(http://arxiv.org/abs/2308.04457)
• Summary:

  Machine learning has emerged as a powerful tool in various fields, including computer vision, natural language processing, and speech recognition. It can unravel hidden patterns within large data sets and reveal unparalleled insights, revolutionizing many industries and disciplines. However, machine and deep learning models lack interpretability and limited domain-specific knowledge, especially in applications such as physics and engineering. Alternatively, physics-informed machine learning (PIML) techniques integrate physics principles into data-driven models. By combining deep learning with domain knowledge, PIML improves the generalization of the model, abidance by the governing physical laws, and interpretability. This paper comprehensively reviews PIML applications related to subsurface energy systems, mainly in the oil and gas industry. The review highlights the successful utilization of PIML for tasks such as seismic applications, reservoir simulation, hydrocarbons production forecasting, and intelligent decision-making in the exploration and production stages. Additionally, it demonstrates PIML's capabilities to revolutionize the oil and gas industry and other emerging areas of interest, such as carbon and hydrogen storage; and geothermal systems by providing more accurate and reliable predictions for resource management and operational efficiency.

explainability

watermark

Title: SSL-Auth: An Authentication Framework by Fragile Watermarking for Pre-trained Encoders in Self-supervised Learning. (arXiv:2308.04673v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04673
• Code URL: null
• Copy Paste: [[2308.04673]] SSL-Auth: An Authentication Framework by Fragile Watermarking for Pre-trained Encoders in Self-supervised Learning(http://arxiv.org/abs/2308.04673)
• Summary:

  Self-supervised learning (SSL) which leverages unlabeled datasets for pre-training powerful encoders has achieved significant success in recent years. These encoders are commonly used as feature extractors for various downstream tasks, requiring substantial data and computing resources for their training process. With the deployment of pre-trained encoders in commercial use, protecting the intellectual property of model owners and ensuring the trustworthiness of the models becomes crucial. Recent research has shown that encoders are threatened by backdoor attacks, adversarial attacks, etc. Therefore, a scheme to verify the integrity of pre-trained encoders is needed to protect users. In this paper, we propose SSL-Auth, the first fragile watermarking scheme for verifying the integrity of encoders without compromising model performance. Our method utilizes selected key samples as watermark information and trains a verification network to reconstruct the watermark information, thereby verifying the integrity of the encoder. By comparing the reconstruction results of the key samples, malicious modifications can be effectively detected, as altered models should not exhibit similar reconstruction performance as the original models. Extensive evaluations on various models and diverse datasets demonstrate the effectiveness and fragility of our proposed SSL-Auth.

diffusion

Title: 3D Scene Diffusion Guidance using Scene Graphs. (arXiv:2308.04468v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04468
• Code URL: null
• Copy Paste: [[2308.04468]] 3D Scene Diffusion Guidance using Scene Graphs(http://arxiv.org/abs/2308.04468)
• Summary:

  Guided synthesis of high-quality 3D scenes is a challenging task. Diffusion models have shown promise in generating diverse data, including 3D scenes. However, current methods rely directly on text embeddings for controlling the generation, limiting the incorporation of complex spatial relationships between objects. We propose a novel approach for 3D scene diffusion guidance using scene graphs. To leverage the relative spatial information the scene graphs provide, we make use of relational graph convolutional blocks within our denoising network. We show that our approach significantly improves the alignment between scene description and generated scene.

Title: IDiff-Face: Synthetic-based Face Recognition through Fizzy Identity-Conditioned Diffusion Models. (arXiv:2308.04995v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04995
• Code URL: https://github.com/fdbtrs/IDiff-Face
• Copy Paste: [[2308.04995]] IDiff-Face: Synthetic-based Face Recognition through Fizzy Identity-Conditioned Diffusion Models(http://arxiv.org/abs/2308.04995)
• Summary:

  The availability of large-scale authentic face databases has been crucial to the significant advances made in face recognition research over the past decade. However, legal and ethical concerns led to the recent retraction of many of these databases by their creators, raising questions about the continuity of future face recognition research without one of its key resources. Synthetic datasets have emerged as a promising alternative to privacy-sensitive authentic data for face recognition development. However, recent synthetic datasets that are used to train face recognition models suffer either from limitations in intra-class diversity or cross-class (identity) discrimination, leading to less optimal accuracies, far away from the accuracies achieved by models trained on authentic data. This paper targets this issue by proposing IDiff-Face, a novel approach based on conditional latent diffusion models for synthetic identity generation with realistic identity variations for face recognition training. Through extensive evaluations, our proposed synthetic-based face recognition approach pushed the limits of state-of-the-art performances, achieving, for example, 98.00% accuracy on the Labeled Faces in the Wild (LFW) benchmark, far ahead from the recent synthetic-based face recognition solutions with 95.40% and bridging the gap to authentic-based face recognition with 99.82% accuracy.

Title: Do Diffusion Models Suffer Error Propagation? Theoretical Analysis and Consistency Regularization. (arXiv:2308.05021v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.05021
• Code URL: null
• Copy Paste: [[2308.05021]] Do Diffusion Models Suffer Error Propagation? Theoretical Analysis and Consistency Regularization(http://arxiv.org/abs/2308.05021)
• Summary:

  While diffusion models have achieved promising performances in data synthesis, they might suffer error propagation because of their cascade structure, where the distributional mismatch spreads and magnifies through the chain of denoising modules. However, a strict analysis is expected since many sequential models such as Conditional Random Field (CRF) are free from error propagation. In this paper, we empirically and theoretically verify that diffusion models are indeed affected by error propagation and we then propose a regularization to address this problem. Our theoretical analysis reveals that the question can be reduced to whether every denoising module of the diffusion model is fault-tolerant. We derive insightful transition equations, indicating that the module can't recover from input errors and even propagates additional errors to the next module. Our analysis directly leads to a consistency regularization scheme for diffusion models, which explicitly reduces the distribution gap between forward and backward processes. We further introduce a bootstrapping algorithm to reduce the computation cost of the regularizer. Our experimental results on multiple image datasets show that our regularization effectively handles error propagation and significantly improves the performance of vanilla diffusion models.

Title: LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation. (arXiv:2308.05095v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05095
• Code URL: null
• Copy Paste: [[2308.05095]] LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation(http://arxiv.org/abs/2308.05095)
• Summary:

  In the text-to-image generation field, recent remarkable progress in Stable Diffusion makes it possible to generate rich kinds of novel photorealistic images. However, current models still face misalignment issues (e.g., problematic spatial relation understanding and numeration failure) in complex natural scenes, which impedes the high-faithfulness text-to-image generation. Although recent efforts have been made to improve controllability by giving fine-grained guidance (e.g., sketch and scribbles), this issue has not been fundamentally tackled since users have to provide such guidance information manually. In this work, we strive to synthesize high-fidelity images that are semantically aligned with a given textual prompt without any guidance. Toward this end, we propose a coarse-to-fine paradigm to achieve layout planning and image generation. Concretely, we first generate the coarse-grained layout conditioned on a given textual prompt via in-context learning based on Large Language Models. Afterward, we propose a fine-grained object-interaction diffusion method to synthesize high-faithfulness images conditioned on the prompt and the automatically generated layout. Extensive experiments demonstrate that our proposed method outperforms the state-of-the-art models in terms of layout and image generation. Our code and settings are available at \url{https://layoutllm-t2i.github.io}.

Title: Going Deeper with Five-point Stencil Convolutions for Reaction-Diffusion Equations. (arXiv:2308.04735v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04735
• Code URL: null
• Copy Paste: [[2308.04735]] Going Deeper with Five-point Stencil Convolutions for Reaction-Diffusion Equations(http://arxiv.org/abs/2308.04735)
• Summary:

  Physics-informed neural networks have been widely applied to partial differential equations with great success because the physics-informed loss essentially requires no observations or discretization. However, it is difficult to optimize model parameters, and these parameters must be trained for each distinct initial condition. To overcome these challenges in second-order reaction-diffusion type equations, a possible way is to use five-point stencil convolutional neural networks (FCNNs). FCNNs are trained using two consecutive snapshots, where the time step corresponds to the step size of the given snapshots. Thus, the time evolution of FCNNs depends on the time step, and the time step must satisfy its CFL condition to avoid blow-up solutions. In this work, we propose deep FCNNs that have large receptive fields to predict time evolutions with a time step larger than the threshold of the CFL condition. To evaluate our models, we consider the heat, Fisher's, and Allen-Cahn equations with diverse initial conditions. We demonstrate that deep FCNNs retain certain accuracies, in contrast to FDMs that blow up.

noise learning

data-free

transformer

Title: Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation. (arXiv:2308.04549v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04549
• Code URL: null
• Copy Paste: [[2308.04549]] Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation(http://arxiv.org/abs/2308.04549)
• Summary:

  Transformers have become the primary backbone of the computer vision community due to their impressive performance. However, the unfriendly computation cost impedes their potential in the video recognition domain. To optimize the speed-accuracy trade-off, we propose Semantic-aware Temporal Accumulation score (STA) to prune spatio-temporal tokens integrally. STA score considers two critical factors: temporal redundancy and semantic importance. The former depicts a specific region based on whether it is a new occurrence or a seen entity by aggregating token-to-token similarity in consecutive frames while the latter evaluates each token based on its contribution to the overall prediction. As a result, tokens with higher scores of STA carry more temporal redundancy as well as lower semantics thus being pruned. Based on the STA score, we are able to progressively prune the tokens without introducing any additional parameters or requiring further re-training. We directly apply the STA module to off-the-shelf ViT and VideoSwin backbones, and the empirical results on Kinetics-400 and Something-Something V2 achieve over 30% computation reduction with a negligible ~0.2% accuracy drop. The code is released at https://github.com/Mark12Ding/STA.

Title: FocalFormer3D : Focusing on Hard Instance for 3D Object Detection. (arXiv:2308.04556v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04556
• Code URL: null
• Copy Paste: [[2308.04556]] FocalFormer3D : Focusing on Hard Instance for 3D Object Detection(http://arxiv.org/abs/2308.04556)
• Summary:

  False negatives (FN) in 3D object detection, {\em e.g.}, missing predictions of pedestrians, vehicles, or other obstacles, can lead to potentially dangerous situations in autonomous driving. While being fatal, this issue is understudied in many current 3D detection methods. In this work, we propose Hard Instance Probing (HIP), a general pipeline that identifies \textit{FN} in a multi-stage manner and guides the models to focus on excavating difficult instances. For 3D object detection, we instantiate this method as FocalFormer3D, a simple yet effective detector that excels at excavating difficult objects and improving prediction recall. FocalFormer3D features a multi-stage query generation to discover hard objects and a box-level transformer decoder to efficiently distinguish objects from massive object candidates. Experimental results on the nuScenes and Waymo datasets validate the superior performance of FocalFormer3D. The advantage leads to strong performance on both detection and tracking, in both LiDAR and multi-modal settings. Notably, FocalFormer3D achieves a 70.5 mAP and 73.9 NDS on nuScenes detection benchmark, while the nuScenes tracking benchmark shows 72.1 AMOTA, both ranking 1st place on the nuScenes LiDAR leaderboard. Our code is available at \url{https://github.com/NVlabs/FocalFormer3D}.

Title: LATR: 3D Lane Detection from Monocular Images with Transformer. (arXiv:2308.04583v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04583
• Code URL: null
• Copy Paste: [[2308.04583]] LATR: 3D Lane Detection from Monocular Images with Transformer(http://arxiv.org/abs/2308.04583)
• Summary:

  3D lane detection from monocular images is a fundamental yet challenging task in autonomous driving. Recent advances primarily rely on structural 3D surrogates (e.g., bird's eye view) that are built from front-view image features and camera parameters. However, the depth ambiguity in monocular images inevitably causes misalignment between the constructed surrogate feature map and the original image, posing a great challenge for accurate lane detection. To address the above issue, we present a novel LATR model, an end-to-end 3D lane detector that uses 3D-aware front-view features without transformed view representation. Specifically, LATR detects 3D lanes via cross-attention based on query and key-value pairs, constructed using our lane-aware query generator and dynamic 3D ground positional embedding. On the one hand, each query is generated based on 2D lane-aware features and adopts a hybrid embedding to enhance the lane information. On the other hand, 3D space information is injected as positional embedding from an iteratively-updated 3D ground plane. LATR outperforms previous state-of-the-art methods on both synthetic Apollo and realistic OpenLane by large margins (e.g., 11.4 gains in terms of F1 score on OpenLane). Code will be released at https://github.com/JMoonr/LATR.

Title: Temporal DINO: A Self-supervised Video Strategy to Enhance Action Prediction. (arXiv:2308.04589v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04589
• Code URL: null
• Copy Paste: [[2308.04589]] Temporal DINO: A Self-supervised Video Strategy to Enhance Action Prediction(http://arxiv.org/abs/2308.04589)
• Summary:

  The emerging field of action prediction plays a vital role in various computer vision applications such as autonomous driving, activity analysis and human-computer interaction. Despite significant advancements, accurately predicting future actions remains a challenging problem due to high dimensionality, complex dynamics and uncertainties inherent in video data. Traditional supervised approaches require large amounts of labelled data, which is expensive and time-consuming to obtain. This paper introduces a novel self-supervised video strategy for enhancing action prediction inspired by DINO (self-distillation with no labels). The Temporal-DINO approach employs two models; a 'student' processing past frames; and a 'teacher' processing both past and future frames, enabling a broader temporal context. During training, the teacher guides the student to learn future context by only observing past frames. The strategy is evaluated on ROAD dataset for the action prediction downstream task using 3D-ResNet, Transformer, and LSTM architectures. The experimental results showcase significant improvements in prediction performance across these architectures, with our method achieving an average enhancement of 9.9% Precision Points (PP), highlighting its effectiveness in enhancing the backbones' capabilities of capturing long-term dependencies. Furthermore, our approach demonstrates efficiency regarding the pretraining dataset size and the number of epochs required. This method overcomes limitations present in other approaches, including considering various backbone architectures, addressing multiple prediction horizons, reducing reliance on hand-crafted augmentations, and streamlining the pretraining process into a single stage. These findings highlight the potential of our approach in diverse video-based tasks such as activity recognition, motion planning, and scene understanding.

Title: Which Tokens to Use? Investigating Token Reduction in Vision Transformers. (arXiv:2308.04657v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04657
• Code URL: null
• Copy Paste: [[2308.04657]] Which Tokens to Use? Investigating Token Reduction in Vision Transformers(http://arxiv.org/abs/2308.04657)
• Summary:

  Since the introduction of the Vision Transformer (ViT), researchers have sought to make ViTs more efficient by removing redundant information in the processed tokens. While different methods have been explored to achieve this goal, we still lack understanding of the resulting reduction patterns and how those patterns differ across token reduction methods and datasets. To close this gap, we set out to understand the reduction patterns of 10 different token reduction methods using four image classification datasets. By systematically comparing these methods on the different classification tasks, we find that the Top-K pruning method is a surprisingly strong baseline. Through in-depth analysis of the different methods, we determine that: the reduction patterns are generally not consistent when varying the capacity of the backbone model, the reduction patterns of pruning-based methods significantly differ from fixed radial patterns, and the reduction patterns of pruning-based methods are correlated across classification datasets. Finally we report that the similarity of reduction patterns is a moderate-to-strong proxy for model performance. Project page at https://vap.aau.dk/tokens.

Title: Self-supervised Learning of Rotation-invariant 3D Point Set Features using Transformer and its Self-distillation. (arXiv:2308.04725v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04725
• Code URL: null
• Copy Paste: [[2308.04725]] Self-supervised Learning of Rotation-invariant 3D Point Set Features using Transformer and its Self-distillation(http://arxiv.org/abs/2308.04725)
• Summary:

  Invariance against rotations of 3D objects is an important property in analyzing 3D point set data. Conventional 3D point set DNNs having rotation invariance typically obtain accurate 3D shape features via supervised learning by using labeled 3D point sets as training samples. However, due to the rapid increase in 3D point set data and the high cost of labeling, a framework to learn rotation-invariant 3D shape features from numerous unlabeled 3D point sets is required. This paper proposes a novel self-supervised learning framework for acquiring accurate and rotation-invariant 3D point set features at object-level. Our proposed lightweight DNN architecture decomposes an input 3D point set into multiple global-scale regions, called tokens, that preserve the spatial layout of partial shapes composing the 3D object. We employ a self-attention mechanism to refine the tokens and aggregate them into an expressive rotation-invariant feature per 3D point set. Our DNN is effectively trained by using pseudo-labels generated by a self-distillation framework. To facilitate the learning of accurate features, we propose to combine multi-crop and cut-mix data augmentation techniques to diversify 3D point sets for training. Through a comprehensive evaluation, we empirically demonstrate that, (1) existing rotation-invariant DNN architectures designed for supervised learning do not necessarily learn accurate 3D shape features under a self-supervised learning scenario, and (2) our proposed algorithm learns rotation-invariant 3D point set features that are more accurate than those learned by existing algorithms. Code will be available at https://github.com/takahikof/RIPT_SDMM

Title: Joint-Relation Transformer for Multi-Person Motion Prediction. (arXiv:2308.04808v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04808
• Code URL: https://github.com/mediabrain-sjtu/jrtransformer
• Copy Paste: [[2308.04808]] Joint-Relation Transformer for Multi-Person Motion Prediction(http://arxiv.org/abs/2308.04808)
• Summary:

  Multi-person motion prediction is a challenging problem due to the dependency of motion on both individual past movements and interactions with other people. Transformer-based methods have shown promising results on this task, but they miss the explicit relation representation between joints, such as skeleton structure and pairwise distance, which is crucial for accurate interaction modeling. In this paper, we propose the Joint-Relation Transformer, which utilizes relation information to enhance interaction modeling and improve future motion prediction. Our relation information contains the relative distance and the intra-/inter-person physical constraints. To fuse relation and joint information, we design a novel joint-relation fusion layer with relation-aware attention to update both features. Additionally, we supervise the relation information by forecasting future distance. Experiments show that our method achieves a 13.4% improvement of 900ms VIM on 3DPW-SoMoF/RC and 17.8%/12.0% improvement of 3s MPJPE on CMU-Mpcap/MuPoTS-3D dataset.

Title: Unsupervised Out-of-Distribution Dialect Detection with Mahalanobis Distance. (arXiv:2308.04886v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04886
• Code URL: null
• Copy Paste: [[2308.04886]] Unsupervised Out-of-Distribution Dialect Detection with Mahalanobis Distance(http://arxiv.org/abs/2308.04886)
• Summary:

  Dialect classification is used in a variety of applications, such as machine translation and speech recognition, to improve the overall performance of the system. In a real-world scenario, a deployed dialect classification model can encounter anomalous inputs that differ from the training data distribution, also called out-of-distribution (OOD) samples. Those OOD samples can lead to unexpected outputs, as dialects of those samples are unseen during model training. Out-of-distribution detection is a new research area that has received little attention in the context of dialect classification. Towards this, we proposed a simple yet effective unsupervised Mahalanobis distance feature-based method to detect out-of-distribution samples. We utilize the latent embeddings from all intermediate layers of a wav2vec 2.0 transformer-based dialect classifier model for multi-task learning. Our proposed approach outperforms other state-of-the-art OOD detection methods significantly.

Title: Feature Modulation Transformer: Cross-Refinement of Global Representation via High-Frequency Prior for Image Super-Resolution. (arXiv:2308.05022v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05022
• Code URL: https://github.com/avc2-uestc/craft-sr
• Copy Paste: [[2308.05022]] Feature Modulation Transformer: Cross-Refinement of Global Representation via High-Frequency Prior for Image Super-Resolution(http://arxiv.org/abs/2308.05022)
• Summary:

  Transformer-based methods have exhibited remarkable potential in single image super-resolution (SISR) by effectively extracting long-range dependencies. However, most of the current research in this area has prioritized the design of transformer blocks to capture global information, while overlooking the importance of incorporating high-frequency priors, which we believe could be beneficial. In our study, we conducted a series of experiments and found that transformer structures are more adept at capturing low-frequency information, but have limited capacity in constructing high-frequency representations when compared to their convolutional counterparts. Our proposed solution, the cross-refinement adaptive feature modulation transformer (CRAFT), integrates the strengths of both convolutional and transformer structures. It comprises three key components: the high-frequency enhancement residual block (HFERB) for extracting high-frequency information, the shift rectangle window attention block (SRWAB) for capturing global information, and the hybrid fusion block (HFB) for refining the global representation. Our experiments on multiple datasets demonstrate that CRAFT outperforms state-of-the-art methods by up to 0.29dB while using fewer parameters. The source code will be made available at: https://github.com/AVC2-UESTC/CRAFT-SR.git.

Title: PAT: Position-Aware Transformer for Dense Multi-Label Action Detection. (arXiv:2308.05051v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05051
• Code URL: null
• Copy Paste: [[2308.05051]] PAT: Position-Aware Transformer for Dense Multi-Label Action Detection(http://arxiv.org/abs/2308.05051)
• Summary:

  We present PAT, a transformer-based network that learns complex temporal co-occurrence action dependencies in a video by exploiting multi-scale temporal features. In existing methods, the self-attention mechanism in transformers loses the temporal positional information, which is essential for robust action detection. To address this issue, we (i) embed relative positional encoding in the self-attention mechanism and (ii) exploit multi-scale temporal relationships by designing a novel non hierarchical network, in contrast to the recent transformer-based approaches that use a hierarchical structure. We argue that joining the self-attention mechanism with multiple sub-sampling processes in the hierarchical approaches results in increased loss of positional information. We evaluate the performance of our proposed approach on two challenging dense multi-label benchmark datasets, and show that PAT improves the current state-of-the-art result by 1.1% and 0.6% mAP on the Charades and MultiTHUMOS datasets, respectively, thereby achieving the new state-of-the-art mAP at 26.5% and 44.6%, respectively. We also perform extensive ablation studies to examine the impact of the different components of our proposed network.

Title: A degree of image identification at sub-human scales could be possible with more advanced clusters. (arXiv:2308.05092v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05092
• Code URL: https://github.com/prateekjannu/imagescale2
• Copy Paste: [[2308.05092]] A degree of image identification at sub-human scales could be possible with more advanced clusters(http://arxiv.org/abs/2308.05092)
• Summary:

  The purpose of the research is to determine if currently available self-supervised learning techniques can accomplish human level comprehension of visual images using the same degree and amount of sensory input that people acquire from. Initial research on this topic solely considered data volume scaling. Here, we scale both the volume of data and the quality of the image. This scaling experiment is a self-supervised learning method that may be done without any outside financing. We find that scaling up data volume and picture resolution at the same time enables human-level item detection performance at sub-human sizes.We run a scaling experiment with vision transformers trained on up to 200000 images up to 256 ppi.

Title: A Bipartite Graph is All We Need for Enhancing Emotional Reasoning with Commonsense Knowledge. (arXiv:2308.04811v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04811
• Code URL: https://github.com/stevekgyang/bhg
• Copy Paste: [[2308.04811]] A Bipartite Graph is All We Need for Enhancing Emotional Reasoning with Commonsense Knowledge(http://arxiv.org/abs/2308.04811)
• Summary:

  The context-aware emotional reasoning ability of AI systems, especially in conversations, is of vital importance in applications such as online opinion mining from social media and empathetic dialogue systems. Due to the implicit nature of conveying emotions in many scenarios, commonsense knowledge is widely utilized to enrich utterance semantics and enhance conversation modeling. However, most previous knowledge infusion methods perform empirical knowledge filtering and design highly customized architectures for knowledge interaction with the utterances, which can discard useful knowledge aspects and limit their generalizability to different knowledge sources. Based on these observations, we propose a Bipartite Heterogeneous Graph (BHG) method for enhancing emotional reasoning with commonsense knowledge. In BHG, the extracted context-aware utterance representations and knowledge representations are modeled as heterogeneous nodes. Two more knowledge aggregation node types are proposed to perform automatic knowledge filtering and interaction. BHG-based knowledge infusion can be directly generalized to multi-type and multi-grained knowledge sources. In addition, we propose a Multi-dimensional Heterogeneous Graph Transformer (MHGT) to perform graph reasoning, which can retain unchanged feature spaces and unequal dimensions for heterogeneous node types during inference to prevent unnecessary loss of information. Experiments show that BHG-based methods significantly outperform state-of-the-art knowledge infusion methods and show generalized knowledge infusion ability with higher efficiency. Further analysis proves that previous empirical knowledge filtering methods do not guarantee to provide the most useful knowledge information. Our code is available at: https://github.com/SteveKGYang/BHG.

Title: Performance Analysis of Transformer Based Models (BERT, ALBERT and RoBERTa) in Fake News Detection. (arXiv:2308.04950v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04950
• Code URL: https://github.com/shafna81/fakenewsdetection
• Copy Paste: [[2308.04950]] Performance Analysis of Transformer Based Models (BERT, ALBERT and RoBERTa) in Fake News Detection(http://arxiv.org/abs/2308.04950)
• Summary:

  Fake news is fake material in a news media format but is not processed properly by news agencies. The fake material can provoke or defame significant entities or individuals or potentially even for the personal interests of the creators, causing problems for society. Distinguishing fake news and real news is challenging due to limited of domain knowledge and time constraints. According to the survey, the top three areas most exposed to hoaxes and misinformation by residents are in Banten, DKI Jakarta and West Java. The model of transformers is referring to an approach in the field of artificial intelligence (AI) in natural language processing utilizing the deep learning architectures. Transformers exercise a powerful attention mechanism to process text in parallel and produce rich and contextual word representations. A previous study indicates a superior performance of a transformer model known as BERT over and above non transformer approach. However, some studies suggest the performance can be improved with the use of improved BERT models known as ALBERT and RoBERTa. However, the modified BERT models are not well explored for detecting fake news in Bahasa Indonesia. In this research, we explore those transformer models and found that ALBERT outperformed other models with 87.6% accuracy, 86.9% precision, 86.9% F1-score, and 174.5 run-time (s/epoch) respectively. Source code available at: https://github.com/Shafna81/fakenewsdetection.git

Title: Sparse Binary Transformers for Multivariate Time Series Modeling. (arXiv:2308.04637v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04637
• Code URL: null
• Copy Paste: [[2308.04637]] Sparse Binary Transformers for Multivariate Time Series Modeling(http://arxiv.org/abs/2308.04637)
• Summary:

  Compressed Neural Networks have the potential to enable deep learning across new applications and smaller computational environments. However, understanding the range of learning tasks in which such models can succeed is not well studied. In this work, we apply sparse and binary-weighted Transformers to multivariate time series problems, showing that the lightweight models achieve accuracy comparable to that of dense floating-point Transformers of the same structure. Our model achieves favorable results across three time series learning tasks: classification, anomaly detection, and single-step forecasting. Additionally, to reduce the computational complexity of the attention mechanism, we apply two modifications, which show little to no decline in model performance: 1) in the classification task, we apply a fixed mask to the query, key, and value activations, and 2) for forecasting and anomaly detection, which rely on predicting outputs at a single point in time, we propose an attention mask to allow computation only at the current time step. Together, each compression technique and attention modification substantially reduces the number of non-zero operations necessary in the Transformer. We measure the computational savings of our approach over a range of metrics including parameter count, bit size, and floating point operation (FLOPs) count, showing up to a 53x reduction in storage size and up to 10.5x reduction in FLOPs.

Title: Efficient Bayesian Optimization with Deep Kernel Learning and Transformer Pre-trained on Multiple Heterogeneous Datasets. (arXiv:2308.04660v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04660
• Code URL: null
• Copy Paste: [[2308.04660]] Efficient Bayesian Optimization with Deep Kernel Learning and Transformer Pre-trained on Multiple Heterogeneous Datasets(http://arxiv.org/abs/2308.04660)
• Summary:

  Bayesian optimization (BO) is widely adopted in black-box optimization problems and it relies on a surrogate model to approximate the black-box response function. With the increasing number of black-box optimization tasks solved and even more to solve, the ability to learn from multiple prior tasks to jointly pre-train a surrogate model is long-awaited to further boost optimization efficiency. In this paper, we propose a simple approach to pre-train a surrogate, which is a Gaussian process (GP) with a kernel defined on deep features learned from a Transformer-based encoder, using datasets from prior tasks with possibly heterogeneous input spaces. In addition, we provide a simple yet effective mix-up initialization strategy for input tokens corresponding to unseen input variables and therefore accelerate new tasks' convergence. Experiments on both synthetic and real benchmark problems demonstrate the effectiveness of our proposed pre-training and transfer BO strategy over existing methods.

Title: PETformer: Long-term Time Series Forecasting via Placeholder-enhanced Transformer. (arXiv:2308.04791v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04791
• Code URL: null
• Copy Paste: [[2308.04791]] PETformer: Long-term Time Series Forecasting via Placeholder-enhanced Transformer(http://arxiv.org/abs/2308.04791)
• Summary:

  Recently, Transformer-based models have shown remarkable performance in long-term time series forecasting (LTSF) tasks due to their ability to model long-term dependencies. However, the validity of Transformers for LTSF tasks remains debatable, particularly since recent work has shown that simple linear models can outperform numerous Transformer-based approaches. This suggests that there are limitations to the application of Transformer in LTSF. Therefore, this paper investigates three key issues when applying Transformer to LTSF: temporal continuity, information density, and multi-channel relationships. Accordingly, we propose three innovative solutions, including Placeholder Enhancement Technique (PET), Long Sub-sequence Division (LSD), and Multi-channel Separation and Interaction (MSI), which together form a novel model called PETformer. These three key designs introduce prior biases suitable for LTSF tasks. Extensive experiments have demonstrated that PETformer achieves state-of-the-art (SOTA) performance on eight commonly used public datasets for LTSF, outperforming all other models currently available. This demonstrates that Transformer still possesses powerful capabilities in LTSF.

generative

Title: GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization. (arXiv:2308.04699v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04699
• Code URL: https://github.com/ffhibnese/gifd
• Copy Paste: [[2308.04699]] GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization(http://arxiv.org/abs/2308.04699)
• Summary:

  Federated Learning (FL) has recently emerged as a promising distributed machine learning framework to preserve clients' privacy, by allowing multiple clients to upload the gradients calculated from their local data to a central server. Recent studies find that the exchanged gradients also take the risk of privacy leakage, e.g., an attacker can invert the shared gradients and recover sensitive data against an FL system by leveraging pre-trained generative adversarial networks (GAN) as prior knowledge. However, performing gradient inversion attacks in the latent space of the GAN model limits their expression ability and generalizability. To tackle these challenges, we propose \textbf{G}radient \textbf{I}nversion over \textbf{F}eature \textbf{D}omains (GIFD), which disassembles the GAN model and searches the feature domains of the intermediate layers. Instead of optimizing only over the initial latent code, we progressively change the optimized layer, from the initial latent space to intermediate layers closer to the output images. In addition, we design a regularizer to avoid unreal image generation by adding a small ${l_1}$ ball constraint to the searching range. We also extend GIFD to the out-of-distribution (OOD) setting, which weakens the assumption that the training sets of GANs and FL tasks obey the same data distribution. Extensive experiments demonstrate that our method can achieve pixel-level reconstruction and is superior to the existing methods. Notably, GIFD also shows great generalizability under different defense strategy settings and batch sizes.

Title: Benchmarking LLM powered Chatbots: Methods and Metrics. (arXiv:2308.04624v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04624
• Code URL: null
• Copy Paste: [[2308.04624]] Benchmarking LLM powered Chatbots: Methods and Metrics(http://arxiv.org/abs/2308.04624)
• Summary:

  Autonomous conversational agents, i.e. chatbots, are becoming an increasingly common mechanism for enterprises to provide support to customers and partners. In order to rate chatbots, especially ones powered by Generative AI tools like Large Language Models (LLMs) we need to be able to accurately assess their performance. This is where chatbot benchmarking becomes important. In this paper, we propose the use of a novel benchmark that we call the E2E (End to End) benchmark, and show how the E2E benchmark can be used to evaluate accuracy and usefulness of the answers provided by chatbots, especially ones powered by LLMs. We evaluate an example chatbot at different levels of sophistication based on both our E2E benchmark, as well as other available metrics commonly used in the state of art, and observe that the proposed benchmark show better results compared to others. In addition, while some metrics proved to be unpredictable, the metric associated with the E2E benchmark, which uses cosine similarity performed well in evaluating chatbots. The performance of our best models shows that there are several benefits of using the cosine similarity score as a metric in the E2E benchmark.

Title: VulLibGen: Identifying Vulnerable Third-Party Libraries via Generative Pre-Trained Model. (arXiv:2308.04662v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.04662
• Code URL: null
• Copy Paste: [[2308.04662]] VulLibGen: Identifying Vulnerable Third-Party Libraries via Generative Pre-Trained Model(http://arxiv.org/abs/2308.04662)
• Summary:

  To avoid potential risks posed by vulnerabilities in third-party libraries, security researchers maintain vulnerability databases (e.g., NVD) containing vulnerability reports, each of which records the description of a vulnerability and the name list of libraries affected by the vulnerability (a.k.a. vulnerable libraries). However, recent studies on about 200,000 vulnerability reports in NVD show that 53.3% of these reports do not include the name list of vulnerable libraries, and 59.82% of the included name lists of vulnerable libraries are incomplete or incorrect.

  To address the preceding issue, in this paper, we propose the first generative approach named VulLibGen to generate the name list of vulnerable libraries (out of all the existing libraries) for the given vulnerability by utilizing recent enormous advances in Large Language Models (LLMs), in order to achieve high accuracy. VulLibGen takes only the description of a vulnerability as input and achieves high identification accuracy based on LLMs' prior knowledge of all the existing libraries. VulLibGen also includes the input augmentation technique to help identify zero-shot vulnerable libraries (those not occurring during training) and the post-processing technique to help address VulLibGen's hallucinations. We evaluate VulLibGen using three state-of-the-art/practice approaches (LightXML, Chronos, and VulLibMiner) that identify vulnerable libraries on an open-source dataset (VulLib). Our evaluation results show that VulLibGen can accurately identify vulnerable libraries with an average F1 score of 0.626 while the state-of-the-art/practice approaches achieve only 0.561. The post-processing technique helps VulLibGen achieve an average improvement of F1@1 by 9.3%. The input augmentation technique helps VulLibGen achieve an average improvement of F1@1 by 39% in identifying zero-shot libraries.

Title: Getting from Generative AI to Trustworthy AI: What LLMs might learn from Cyc. (arXiv:2308.04445v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04445
• Code URL: null
• Copy Paste: [[2308.04445]] Getting from Generative AI to Trustworthy AI: What LLMs might learn from Cyc(http://arxiv.org/abs/2308.04445)
• Summary:

  Generative AI, the most popular current approach to AI, consists of large language models (LLMs) that are trained to produce outputs that are plausible, but not necessarily correct. Although their abilities are often uncanny, they are lacking in aspects of reasoning, leading LLMs to be less than completely trustworthy. Furthermore, their results tend to be both unpredictable and uninterpretable.

  We lay out 16 desiderata for future AI, and discuss an alternative approach to AI which could theoretically address many of the limitations associated with current approaches: AI educated with curated pieces of explicit knowledge and rules of thumb, enabling an inference engine to automatically deduce the logical entailments of all that knowledge. Even long arguments produced this way can be both trustworthy and interpretable, since the full step-by-step line of reasoning is always available, and for each step the provenance of the knowledge used can be documented and audited. There is however a catch: if the logical language is expressive enough to fully represent the meaning of anything we can say in English, then the inference engine runs much too slowly. That's why symbolic AI systems typically settle for some fast but much less expressive logic, such as knowledge graphs. We describe how one AI system, Cyc, has developed ways to overcome that tradeoff and is able to reason in higher order logic in real time.

  We suggest that any trustworthy general AI will need to hybridize the approaches, the LLM approach and more formal approach, and lay out a path to realizing that dream.

Title: Generative Perturbation Analysis for Probabilistic Black-Box Anomaly Attribution. (arXiv:2308.04708v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.04708
• Code URL: https://github.com/idesan/gpa
• Copy Paste: [[2308.04708]] Generative Perturbation Analysis for Probabilistic Black-Box Anomaly Attribution(http://arxiv.org/abs/2308.04708)
• Summary:

  We address the task of probabilistic anomaly attribution in the black-box regression setting, where the goal is to compute the probability distribution of the attribution score of each input variable, given an observed anomaly. The training dataset is assumed to be unavailable. This task differs from the standard XAI (explainable AI) scenario, since we wish to explain the anomalous deviation from a black-box prediction rather than the black-box model itself.

  We begin by showing that mainstream model-agnostic explanation methods, such as the Shapley values, are not suitable for this task because of their ``deviation-agnostic property.'' We then propose a novel framework for probabilistic anomaly attribution that allows us to not only compute attribution scores as the predictive mean but also quantify the uncertainty of those scores. This is done by considering a generative process for perturbations that counter-factually bring the observed anomalous observation back to normalcy. We introduce a variational Bayes algorithm for deriving the distributions of per variable attribution scores. To the best of our knowledge, this is the first probabilistic anomaly attribution framework that is free from being deviation-agnostic.

large language model

Title: Shepherd: A Critic for Language Model Generation. (arXiv:2308.04592v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04592
• Code URL: https://github.com/facebookresearch/shepherd
• Copy Paste: [[2308.04592]] Shepherd: A Critic for Language Model Generation(http://arxiv.org/abs/2308.04592)
• Summary:

  As large language models improve, there is increasing interest in techniques that leverage these models' capabilities to refine their own outputs. In this work, we introduce Shepherd, a language model specifically tuned to critique responses and suggest refinements, extending beyond the capabilities of an untuned model to identify diverse errors and provide suggestions to remedy them. At the core of our approach is a high quality feedback dataset, which we curate from community feedback and human annotations. Even though Shepherd is small (7B parameters), its critiques are either equivalent or preferred to those from established models including ChatGPT. Using GPT-4 for evaluation, Shepherd reaches an average win-rate of 53-87% compared to competitive alternatives. In human evaluation, Shepherd strictly outperforms other models and on average closely ties with ChatGPT.

Title: Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA. (arXiv:2308.04679v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04679
• Code URL: null
• Copy Paste: [[2308.04679]] Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA(http://arxiv.org/abs/2308.04679)
• Summary:

  Large Language Models (LLMs) have shown outstanding performance across wide range of downstream tasks. This competency is attributed to their substantial parameter size and pre-training on extensive corpus. Moreover, LLMs have exhibited enhanced reasoning capabilities in tackling complex reasoning tasks, owing to the utilization of a method named ``Chain-of-Thought (CoT) prompting''. This method is designed to generate intermediate reasoning steps that guide the inference of the final answer. However, it is essential to highlight that these advanced reasoning abilities appear to emerge in models with a minimum of 10 billion parameters, thereby limiting its efficacy in situations where computational resources are constrained. In this paper, we investigate the possibility of transferring the reasoning capabilities of LLMs to smaller models via knowledge distillation. Specifically, we propose Sci-CoT, a two-stage framework that separates the processes of generating rationales and inferring answers. This method enables a more efficient use of rationales during the answer inference stage, leading to improved performance on scientific question-answering tasks. Utilizing Sci-CoT, our 80-million parameter model is able to exceed the performance of BLOOM-176B in the ARC-Easy dataset under the few shot setting.

Title: A Comparative Study of Open-Source Large Language Models, GPT-4 and Claude 2: Multiple-Choice Test Taking in Nephrology. (arXiv:2308.04709v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04709
• Code URL: null
• Copy Paste: [[2308.04709]] A Comparative Study of Open-Source Large Language Models, GPT-4 and Claude 2: Multiple-Choice Test Taking in Nephrology(http://arxiv.org/abs/2308.04709)
• Summary:

  In recent years, there have been significant breakthroughs in the field of natural language processing, particularly with the development of large language models (LLMs). These LLMs have showcased remarkable capabilities on various benchmarks. In the healthcare field, the exact role LLMs and other future AI models will play remains unclear. There is a potential for these models in the future to be used as part of adaptive physician training, medical co-pilot applications, and digital patient interaction scenarios. The ability of AI models to participate in medical training and patient care will depend in part on their mastery of the knowledge content of specific medical fields. This study investigated the medical knowledge capability of LLMs, specifically in the context of internal medicine subspecialty multiple-choice test-taking ability. We compared the performance of several open-source LLMs (Koala 7B, Falcon 7B, Stable-Vicuna 13B, and Orca Mini 13B), to GPT-4 and Claude 2 on multiple-choice questions in the field of Nephrology. Nephrology was chosen as an example of a particularly conceptually complex subspecialty field within internal medicine. The study was conducted to evaluate the ability of LLM models to provide correct answers to nephSAP (Nephrology Self-Assessment Program) multiple-choice questions. The overall success of open-sourced LLMs in answering the 858 nephSAP multiple-choice questions correctly was 17.1% - 25.5%. In contrast, Claude 2 answered 54.4% of the questions correctly, whereas GPT-4 achieved a score of 73.3%. We show that current widely used open-sourced LLMs do poorly in their ability for zero-shot reasoning when compared to GPT-4 and Claude 2. The findings of this study potentially have significant implications for the future of subspecialty medical training and patient care.

Title: ADMUS: A Progressive Question Answering Framework Adaptable to Multiple Knowledge Sources. (arXiv:2308.04800v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04800
• Code URL: null
• Copy Paste: [[2308.04800]] ADMUS: A Progressive Question Answering Framework Adaptable to Multiple Knowledge Sources(http://arxiv.org/abs/2308.04800)
• Summary:

  With the introduction of deep learning models, semantic parsingbased knowledge base question answering (KBQA) systems have achieved high performance in handling complex questions. However, most existing approaches primarily focus on enhancing the model's effectiveness on individual benchmark datasets, disregarding the high costs of adapting the system to disparate datasets in real-world scenarios (e.g., multi-tenant platform). Therefore, we present ADMUS, a progressive knowledge base question answering framework designed to accommodate a wide variety of datasets, including multiple languages, diverse backbone knowledge bases, and disparate question answering datasets. To accomplish the purpose, we decouple the architecture of conventional KBQA systems and propose this dataset-independent framework. Our framework supports the seamless integration of new datasets with minimal effort, only requiring creating a dataset-related micro-service at a negligible cost. To enhance the usability of ADUMS, we design a progressive framework consisting of three stages, ranges from executing exact queries, generating approximate queries and retrieving open-domain knowledge referring from large language models. An online demonstration of ADUMS is available at: https://answer.gstore.cn/pc/index.html

Title: CLEVA: Chinese Language Models EVAluation Platform. (arXiv:2308.04813v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04813
• Code URL: null
• Copy Paste: [[2308.04813]] CLEVA: Chinese Language Models EVAluation Platform(http://arxiv.org/abs/2308.04813)
• Summary:

  With the continuous emergence of Chinese Large Language Models (LLMs), how to evaluate a model's capabilities has become an increasingly significant issue. The absence of a comprehensive Chinese benchmark that thoroughly assesses a model's performance, the unstandardized and incomparable prompting procedure, and the prevalent risk of contamination pose major challenges in the current evaluation of Chinese LLMs. We present CLEVA, a user-friendly platform crafted to holistically evaluate Chinese LLMs. Our platform employs a standardized workflow to assess LLMs' performance across various dimensions, regularly updating a competitive leaderboard. To alleviate contamination, CLEVA curates a significant proportion of new data and develops a sampling strategy that guarantees a unique subset for each leaderboard round. Empowered by an easy-to-use interface that requires just a few mouse clicks and a model API, users can conduct a thorough evaluation with minimal coding. Large-scale experiments featuring 23 influential Chinese LLMs have validated CLEVA's efficacy.

Title: Emotion-Conditioned Text Generation through Automatic Prompt Optimization. (arXiv:2308.04857v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04857
• Code URL: null
• Copy Paste: [[2308.04857]] Emotion-Conditioned Text Generation through Automatic Prompt Optimization(http://arxiv.org/abs/2308.04857)
• Summary:

  Conditional natural language generation methods often require either expensive fine-tuning or training a large language model from scratch. Both are unlikely to lead to good results without a substantial amount of data and computational resources. Prompt learning without changing the parameters of a large language model presents a promising alternative. It is a cost-effective approach, while still achieving competitive results. While this procedure is now established for zero- and few-shot text classification and structured prediction, it has received limited attention in conditional text generation. We present the first automatic prompt optimization approach for emotion-conditioned text generation with instruction-fine-tuned models. Our method uses an iterative optimization procedure that changes the prompt by adding, removing, or replacing tokens. As objective function, we only require a text classifier that measures the realization of the conditional variable in the generated text. We evaluate the method on emotion-conditioned text generation with a focus on event reports and compare it to manually designed prompts that also act as the seed for the optimization procedure. The optimized prompts achieve 0.75 macro-average F1 to fulfill the emotion condition in contrast to manually designed seed prompts with only 0.22 macro-average F1.

Title: LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking. (arXiv:2308.04945v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04945
• Code URL: https://github.com/qcri/llmebench
• Copy Paste: [[2308.04945]] LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking(http://arxiv.org/abs/2308.04945)
• Summary:

  The recent development and success of Large Language Models (LLMs) necessitate an evaluation of their performance across diverse NLP tasks in different languages. Although several frameworks have been developed and made publicly available, their customization capabilities for specific tasks and datasets are often complex for different users. In this study, we introduce the LLMeBench framework. Initially developed to evaluate Arabic NLP tasks using OpenAI's GPT and BLOOM models; it can be seamlessly customized for any NLP task and model, regardless of language. The framework also features zero- and few-shot learning settings. A new custom dataset can be added in less than 10 minutes, and users can use their own model API keys to evaluate the task at hand. The developed framework has been already tested on 31 unique NLP tasks using 53 publicly available datasets within 90 experimental setups, involving approximately 296K data points. We plan to open-source the framework for the community (https://github.com/qcri/LLMeBench/). A video demonstrating the framework is available online (https://youtu.be/FkQn4UjYA0s).

Title: Extrapolating Large Language Models to Non-English by Aligning Languages. (arXiv:2308.04948v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04948
• Code URL: null
• Copy Paste: [[2308.04948]] Extrapolating Large Language Models to Non-English by Aligning Languages(http://arxiv.org/abs/2308.04948)
• Summary:

  Due to the unbalanced training data distribution, the language ability of large language models (LLMs) is often biased towards English. In this paper, we propose to empower pre-trained LLMs on non-English languages by building semantic alignment across languages. We perform instruction-tuning on LLaMA with both translation task data and cross-lingual general task data to obtain cross-lingual models (x-LLaMA). Experiment results on cross-lingual benchmark XQUAD and MLQA show that x-LLaMA models outperform the English instruction-tuned counterpart (Alpaca) by 42.50% on average on six non-English languages. Further experiments on Chinese benchmark C-Eval show that x-LLaMA achieves significant improvement on Chinese humanities tasks, outperforming Alpaca by 8.2%. We also discover that incorporating non-English text on the target side of translation data is particularly effective for boosting non-English ability. Besides, we find that semantic alignment within LLM can be further strengthened as translation task data scales up and we present the formulation of the underlying scaling law. Evaluation results on translation dataset Flores-101 show that \method outperforms previous LLaMA-based models in all evaluated directions. Code and data will be available at: https://github.com/OwenNJU/x-LLM.

segmentation

Title: Large-Scale Multi-Hypotheses Cell Tracking Using Ultrametric Contours Maps. (arXiv:2308.04526v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04526
• Code URL: https://github.com/royerlab/ultrack
• Copy Paste: [[2308.04526]] Large-Scale Multi-Hypotheses Cell Tracking Using Ultrametric Contours Maps(http://arxiv.org/abs/2308.04526)
• Summary:

  In this work, we describe a method for large-scale 3D cell-tracking through a segmentation selection approach. The proposed method is effective at tracking cells across large microscopy datasets on two fronts: (i) It can solve problems containing millions of segmentation instances in terabyte-scale 3D+t datasets; (ii) It achieves competitive results with or without deep learning, which requires 3D annotated data, that is scarce in the fluorescence microscopy field. The proposed method computes cell tracks and segments using a hierarchy of segmentation hypotheses and selects disjoint segments by maximizing the overlap between adjacent frames. We show that this method achieves state-of-the-art results in 3D images from the cell tracking challenge and has a faster integer linear programming formulation. Moreover, our framework is flexible and supports segmentations from off-the-shelf cell segmentation models and can combine them into an ensemble that improves tracking. The code is available https://github.com/royerlab/ultrack.

Title: Unsupervised Camouflaged Object Segmentation as Domain Adaptation. (arXiv:2308.04528v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04528
• Code URL: https://github.com/Jun-Pu/UCOS-DA
• Copy Paste: [[2308.04528]] Unsupervised Camouflaged Object Segmentation as Domain Adaptation(http://arxiv.org/abs/2308.04528)
• Summary:

  Deep learning for unsupervised image segmentation remains challenging due to the absence of human labels. The common idea is to train a segmentation head, with the supervision of pixel-wise pseudo-labels generated based on the representation of self-supervised backbones. By doing so, the model performance depends much on the distance between the distributions of target datasets and the pre-training dataset (e.g., ImageNet). In this work, we investigate a new task, namely unsupervised camouflaged object segmentation (UCOS), where the target objects own a common rarely-seen attribute, i.e., camouflage. Unsurprisingly, we find that the state-of-the-art unsupervised models struggle in adapting UCOS, due to the domain gap between the properties of generic and camouflaged objects. To this end, we formulate the UCOS as a source-free unsupervised domain adaptation task (UCOS-DA), where both source labels and target labels are absent during the whole model training process. Specifically, we define a source model consisting of self-supervised vision transformers pre-trained on ImageNet. On the other hand, the target domain includes a simple linear layer (i.e., our target model) and unlabeled camouflaged objects. We then design a pipeline for foreground-background-contrastive self-adversarial domain adaptation, to achieve robust UCOS. As a result, our baseline model achieves superior segmentation performance when compared with competing unsupervised models on the UCOS benchmark, with the training set which's scale is only one tenth of the supervised COS counterpart.

Title: YUDO: YOLO for Uniform Directed Object Detection. (arXiv:2308.04542v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04542
• Code URL: https://github.com/djordjened92/yudo
• Copy Paste: [[2308.04542]] YUDO: YOLO for Uniform Directed Object Detection(http://arxiv.org/abs/2308.04542)
• Summary:

  This paper presents an efficient way of detecting directed objects by predicting their center coordinates and direction angle. Since the objects are of uniform size, the proposed model works without predicting the object's width and height. The dataset used for this problem is presented in Honeybee Segmentation and Tracking Datasets project. One of the contributions of this work is an examination of the ability of the standard real-time object detection architecture like YoloV7 to be customized for position and direction detection. A very efficient, tiny version of the architecture is used in this approach. Moreover, only one of three detection heads without anchors is sufficient for this task. We also introduce the extended Skew Intersection over Union (SkewIoU) calculation for rotated boxes - directed IoU (DirIoU), which includes an absolute angle difference. DirIoU is used both in the matching procedure of target and predicted bounding boxes for mAP calculation, and in the NMS filtering procedure. The code and models are available at https://github.com/djordjened92/yudo.

Title: 1st Place Solution for CVPR2023 BURST Long Tail and Open World Challenges. (arXiv:2308.04598v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04598
• Code URL: null
• Copy Paste: [[2308.04598]] 1st Place Solution for CVPR2023 BURST Long Tail and Open World Challenges(http://arxiv.org/abs/2308.04598)
• Summary:

  Currently, Video Instance Segmentation (VIS) aims at segmenting and categorizing objects in videos from a closed set of training categories that contain only a few dozen of categories, lacking the ability to handle diverse objects in real-world videos. As TAO and BURST datasets release, we have the opportunity to research VIS in long-tailed and open-world scenarios. Traditional VIS methods are evaluated on benchmarks limited to a small number of common classes, But practical applications require trackers that go beyond these common classes, detecting and tracking rare and even never-before-seen objects. Inspired by the latest MOT paper for the long tail task (Tracking Every Thing in the Wild, Siyuan Li et), for the BURST long tail challenge, we train our model on a combination of LVISv0.5 and the COCO dataset using repeat factor sampling. First, train the detector with segmentation and CEM on LVISv0.5 + COCO dataset. And then, train the instance appearance similarity head on the TAO dataset. at last, our method (LeTracker) gets 14.9 HOTAall in the BURST test set, ranking 1st in the benchmark. for the open-world challenges, we only use 64 classes (Intersection classes of BURST Train subset and COCO dataset, without LVIS dataset) annotations data training, and testing on BURST test set data and get 61.4 OWTAall, ranking 1st in the benchmark. Our code will be released to facilitate future research.

Title: Continual Road-Scene Semantic Segmentation via Feature-Aligned Symmetric Multi-Modal Network. (arXiv:2308.04702v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04702
• Code URL: null
• Copy Paste: [[2308.04702]] Continual Road-Scene Semantic Segmentation via Feature-Aligned Symmetric Multi-Modal Network(http://arxiv.org/abs/2308.04702)
• Summary:

  State-of-the-art multimodal semantic segmentation approaches combining LiDAR and color data are usually designed on top of asymmetric information-sharing schemes and assume that both modalities are always available. Regrettably, this strong assumption may not hold in real-world scenarios, where sensors are prone to failure or can face adverse conditions (night-time, rain, fog, etc.) that make the acquired information unreliable. Moreover, these architectures tend to fail in continual learning scenarios. In this work, we re-frame the task of multimodal semantic segmentation by enforcing a tightly-coupled feature representation and a symmetric information-sharing scheme, which allows our approach to work even when one of the input modalities is missing. This makes our model reliable even in safety-critical settings, as is the case of autonomous driving. We evaluate our approach on the SemanticKITTI dataset, comparing it with our closest competitor. We also introduce an ad-hoc continual learning scheme and show results in a class-incremental continual learning scenario that prove the effectiveness of the approach also in this setting.

Title: MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation. (arXiv:2308.04829v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04829
• Code URL: null
• Copy Paste: [[2308.04829]] MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation(http://arxiv.org/abs/2308.04829)
• Summary:

  Recently, semantic segmentation models trained with image-level text supervision have shown promising results in challenging open-world scenarios. However, these models still face difficulties in learning fine-grained semantic alignment at the pixel level and predicting accurate object masks. To address this issue, we propose MixReorg, a novel and straightforward pre-training paradigm for semantic segmentation that enhances a model's ability to reorganize patches mixed across images, exploring both local visual relevance and global semantic coherence. Our approach involves generating fine-grained patch-text pairs data by mixing image patches while preserving the correspondence between patches and text. The model is then trained to minimize the segmentation loss of the mixed images and the two contrastive losses of the original and restored features. With MixReorg as a mask learner, conventional text-supervised semantic segmentation models can achieve highly generalizable pixel-semantic alignment ability, which is crucial for open-world segmentation. After training with large-scale image-text data, MixReorg models can be applied directly to segment visual objects of arbitrary categories, without the need for further fine-tuning. Our proposed framework demonstrates strong performance on popular zero-shot semantic segmentation benchmarks, outperforming GroupViT by significant margins of 5.0%, 6.2%, 2.5%, and 3.4% mIoU on PASCAL VOC2012, PASCAL Context, MS COCO, and ADE20K, respectively.

Title: SLPT: Selective Labeling Meets Prompt Tuning on Label-Limited Lesion Segmentation. (arXiv:2308.04911v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04911
• Code URL: null
• Copy Paste: [[2308.04911]] SLPT: Selective Labeling Meets Prompt Tuning on Label-Limited Lesion Segmentation(http://arxiv.org/abs/2308.04911)
• Summary:

  Medical image analysis using deep learning is often challenged by limited labeled data and high annotation costs. Fine-tuning the entire network in label-limited scenarios can lead to overfitting and suboptimal performance. Recently, prompt tuning has emerged as a more promising technique that introduces a few additional tunable parameters as prompts to a task-agnostic pre-trained model, and updates only these parameters using supervision from limited labeled data while keeping the pre-trained model unchanged. However, previous work has overlooked the importance of selective labeling in downstream tasks, which aims to select the most valuable downstream samples for annotation to achieve the best performance with minimum annotation cost. To address this, we propose a framework that combines selective labeling with prompt tuning (SLPT) to boost performance in limited labels. Specifically, we introduce a feature-aware prompt updater to guide prompt tuning and a TandEm Selective LAbeling (TESLA) strategy. TESLA includes unsupervised diversity selection and supervised selection using prompt-based uncertainty. In addition, we propose a diversified visual prompt tuning strategy to provide multi-prompt-based discrepant predictions for TESLA. We evaluate our method on liver tumor segmentation and achieve state-of-the-art performance, outperforming traditional fine-tuning with only 6% of tunable parameters, also achieving 94% of full-data performance by labeling only 5% of the data.

Title: Branches Mutual Promotion for End-to-End Weakly Supervised Semantic Segmentation. (arXiv:2308.04949v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04949
• Code URL: null
• Copy Paste: [[2308.04949]] Branches Mutual Promotion for End-to-End Weakly Supervised Semantic Segmentation(http://arxiv.org/abs/2308.04949)
• Summary:

  End-to-end weakly supervised semantic segmentation aims at optimizing a segmentation model in a single-stage training process based on only image annotations. Existing methods adopt an online-trained classification branch to provide pseudo annotations for supervising the segmentation branch. However, this strategy makes the classification branch dominate the whole concurrent training process, hindering these two branches from assisting each other. In our work, we treat these two branches equally by viewing them as diverse ways to generate the segmentation map, and add interactions on both their supervision and operation to achieve mutual promotion. For this purpose, a bidirectional supervision mechanism is elaborated to force the consistency between the outputs of these two branches. Thus, the segmentation branch can also give feedback to the classification branch to enhance the quality of localization seeds. Moreover, our method also designs interaction operations between these two branches to exchange their knowledge to assist each other. Experiments indicate our work outperforms existing end-to-end weakly supervised segmentation methods.

Title: Prototypical Kernel Learning and Open-set Foreground Perception for Generalized Few-shot Semantic Segmentation. (arXiv:2308.04952v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.04952
• Code URL: null
• Copy Paste: [[2308.04952]] Prototypical Kernel Learning and Open-set Foreground Perception for Generalized Few-shot Semantic Segmentation(http://arxiv.org/abs/2308.04952)
• Summary:

  Generalized Few-shot Semantic Segmentation (GFSS) extends Few-shot Semantic Segmentation (FSS) to simultaneously segment unseen classes and seen classes during evaluation. Previous works leverage additional branch or prototypical aggregation to eliminate the constrained setting of FSS. However, representation division and embedding prejudice, which heavily results in poor performance of GFSS, have not been synthetical considered. We address the aforementioned problems by jointing the prototypical kernel learning and open-set foreground perception. Specifically, a group of learnable kernels is proposed to perform segmentation with each kernel in charge of a stuff class. Then, we explore to merge the prototypical learning to the update of base-class kernels, which is consistent with the prototype knowledge aggregation of few-shot novel classes. In addition, a foreground contextual perception module cooperating with conditional bias based inference is adopted to perform class-agnostic as well as open-set foreground detection, thus to mitigate the embedding prejudice and prevent novel targets from being misclassified as background. Moreover, we also adjust our method to the Class Incremental Few-shot Semantic Segmentation (CIFSS) which takes the knowledge of novel classes in a incremental stream. Extensive experiments on PASCAL-5i and COCO-20i datasets demonstrate that our method performs better than previous state-of-the-art.

Title: Volumetric Fast Fourier Convolution for Detecting Ink on the Carbonized Herculaneum Papyri. (arXiv:2308.05070v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05070
• Code URL: https://github.com/aimagelab/vffc
• Copy Paste: [[2308.05070]] Volumetric Fast Fourier Convolution for Detecting Ink on the Carbonized Herculaneum Papyri(http://arxiv.org/abs/2308.05070)
• Summary:

  Recent advancements in Digital Document Restoration (DDR) have led to significant breakthroughs in analyzing highly damaged written artifacts. Among those, there has been an increasing interest in applying Artificial Intelligence techniques for virtually unwrapping and automatically detecting ink on the Herculaneum papyri collection. This collection consists of carbonized scrolls and fragments of documents, which have been digitized via X-ray tomography to allow the development of ad-hoc deep learning-based DDR solutions. In this work, we propose a modification of the Fast Fourier Convolution operator for volumetric data and apply it in a segmentation architecture for ink detection on the challenging Herculaneum papyri, demonstrating its suitability via deep experimental analysis. To encourage the research on this task and the application of the proposed operator to other tasks involving volumetric data, we will release our implementation (https://github.com/aimagelab/vffc)

Title: Scene-Generalizable Interactive Segmentation of Radiance Fields. (arXiv:2308.05104v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05104
• Code URL: null
• Copy Paste: [[2308.05104]] Scene-Generalizable Interactive Segmentation of Radiance Fields(http://arxiv.org/abs/2308.05104)
• Summary:

  Existing methods for interactive segmentation in radiance fields entail scene-specific optimization and thus cannot generalize across different scenes, which greatly limits their applicability. In this work we make the first attempt at Scene-Generalizable Interactive Segmentation in Radiance Fields (SGISRF) and propose a novel SGISRF method, which can perform 3D object segmentation for novel (unseen) scenes represented by radiance fields, guided by only a few interactive user clicks in a given set of multi-view 2D images. In particular, the proposed SGISRF focuses on addressing three crucial challenges with three specially designed techniques. First, we devise the Cross-Dimension Guidance Propagation to encode the scarce 2D user clicks into informative 3D guidance representations. Second, the Uncertainty-Eliminated 3D Segmentation module is designed to achieve efficient yet effective 3D segmentation. Third, Concealment-Revealed Supervised Learning scheme is proposed to reveal and correct the concealed 3D segmentation errors resulted from the supervision in 2D space with only 2D mask annotations. Extensive experiments on two real-world challenging benchmarks covering diverse scenes demonstrate 1) effectiveness and scene-generalizability of the proposed method, 2) favorable performance compared to classical method requiring scene-specific optimization.

Title: A Comparative Study of Sentence Embedding Models for Assessing Semantic Variation. (arXiv:2308.04625v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04625
• Code URL: null
• Copy Paste: [[2308.04625]] A Comparative Study of Sentence Embedding Models for Assessing Semantic Variation(http://arxiv.org/abs/2308.04625)
• Summary:

  Analyzing the pattern of semantic variation in long real-world texts such as books or transcripts is interesting from the stylistic, cognitive, and linguistic perspectives. It is also useful for applications such as text segmentation, document summarization, and detection of semantic novelty. The recent emergence of several vector-space methods for sentence embedding has made such analysis feasible. However, this raises the issue of how consistent and meaningful the semantic representations produced by various methods are in themselves. In this paper, we compare several recent sentence embedding methods via time-series of semantic similarity between successive sentences and matrices of pairwise sentence similarity for multiple books of literature. In contrast to previous work using target tasks and curated datasets to compare sentence embedding methods, our approach provides an evaluation of the methods 'in the wild'. We find that most of the sentence embedding methods considered do infer highly correlated patterns of semantic similarity in a given document, but show interesting differences.

Title: Automatically measuring speech fluency in people with aphasia: first achievements using read-speech data. (arXiv:2308.04763v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.04763
• Code URL: null
• Copy Paste: [[2308.04763]] Automatically measuring speech fluency in people with aphasia: first achievements using read-speech data(http://arxiv.org/abs/2308.04763)
• Summary:

  Background: Speech and language pathologists (SLPs) often relyon judgements of speech fluency for diagnosing or monitoringpatients with aphasia. However, such subjective methods havebeen criticised for their lack of reliability and their clinical cost interms of time. Aims: This study aims at assessing the relevance of a signalprocessingalgorithm, initially developed in the field of language acquisition, for the automatic measurement of speech fluency in people with aphasia (PWA). Methods & Procedures: Twenty-nine PWA and five control participantswere recruited via non-profit organizations and SLP networks. All participants were recorded while reading out loud a set ofsentences taken from the French version of the Boston Diagnostic Aphasia Examination. Three trained SLPs assessed the fluency of each sentence on a five-point qualitative scale. A forward-backward divergence segmentation and a clustering algorithm were used to compute, for each sentence, four automatic predictors of speech fluency: pseudo-syllable rate, speech ratio, rate of silent breaks, and standard deviation of pseudo-syllable length. The four predictors were finally combined into multivariate regression models (a multiplelinear regression - MLR, and two non-linear models) to predict the average SLP ratings of speech fluency, using a leave-one speaker-out validation scheme. Outcomes & Results: All models achieved accurate predictions of speech fluency ratings, with average root-mean-square errors as low as 0.5. The MLR yielded a correlation coefficient of 0.87 with reference ratings at the sentence level, and of 0.93 when aggregating the data for each participant. The inclusion of an additional predictor sensitive to repetitions improved further the predictions with a correlation coefficient of 0.91 at the sentence level, and of 0.96 at the participant level. Conclusions: The algorithms used in this study can constitute a cost-effective and reliable tool for the assessment of the speech fluency of patients with aphasia in read-aloud tasks. Perspectives for the assessment of spontaneous speech are discussed.