secure

Title: A Proposal for a Lean and Functional Delivery versus Payment across two Blockchains. (arXiv:2311.05966v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05966
• Code URL: null
• Copy Paste: [[2311.05966]] A Proposal for a Lean and Functional Delivery versus Payment across two Blockchains(http://arxiv.org/abs/2311.05966)
• Summary:

  We propose a lean and functional transaction scheme to establish a secure delivery-versus-payment across two blockchains, where a) no intermediary is required and b) the operator of the payment chain/payment system has a small overhead and does not need to store state. The main idea comes with two requirements: First, the payment chain operator hosts a stateless decryption service that allows the decrypt of messages with his secret key. Second a "Payment Contract" is deployed on the payment chain that implements a function transferAndDecrypt(uint id, address from, address to, string keyEncryptedSuccess, string keyEncryptedFail) that processes the (trigger-based) payment and emits the decrypted key depending on success or failure of the transaction. The respective key can then trigger an associated transaction, e.g. claiming delivery by the buyer or re-claiming the locked asset by the seller.

Title: Deep Learning meets Blockchain for Automated and Secure Access Control. (arXiv:2311.06236v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.06236
• Code URL: null
• Copy Paste: [[2311.06236]] Deep Learning meets Blockchain for Automated and Secure Access Control(http://arxiv.org/abs/2311.06236)
• Summary:

  Access control is a critical component of computer security, governing access to system resources. However, designing policies and roles in traditional access control can be challenging and difficult to maintain in dynamic and complex systems, which is particularly problematic for organizations with numerous resources. Furthermore, traditional methods suffer from issues such as third-party involvement, inefficiency, and privacy gaps, making transparent and dynamic access control an ongoing research problem. Moreover detecting malicious activities and identifying users who are not behaving appropriately can present notable difficulties. To address these challenges, we propose DLACB, a Deep Learning Based Access Control Using Blockchain, as a solution to decentralized access control. DLACB uses blockchain to provide transparency, traceability, and reliability in various domains such as medicine, finance, and government while taking advantage of deep learning to not rely on predefined policies and eventually automate access control. With the integration of blockchain and deep learning for access control, DLACB can provide a general framework applicable to various domains, enabling transparent and reliable logging of all transactions. As all data is recorded on the blockchain, we have the capability to identify malicious activities. We store a list of malicious activities in the storage system and employ a verification algorithm to cross-reference it with the blockchain. We conduct measurements and comparisons of the smart contract processing time for the deployed access control system in contrast to traditional access control methods, determining the time overhead involved. The processing time of DLBAC demonstrates remarkable stability when exposed to increased request volumes.

Title: Scale-MIA: A Scalable Model Inversion Attack against Secure Federated Learning via Latent Space Reconstruction. (arXiv:2311.05808v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05808
• Code URL: null
• Copy Paste: [[2311.05808]] Scale-MIA: A Scalable Model Inversion Attack against Secure Federated Learning via Latent Space Reconstruction(http://arxiv.org/abs/2311.05808)
• Summary:

  Federated learning is known for its capability to safeguard participants' data privacy. However, recently emerged model inversion attacks (MIAs) have shown that a malicious parameter server can reconstruct individual users' local data samples through model updates. The state-of-the-art attacks either rely on computation-intensive search-based optimization processes to recover each input batch, making scaling difficult, or they involve the malicious parameter server adding extra modules before the global model architecture, rendering the attacks too conspicuous and easily detectable.

  To overcome these limitations, we propose Scale-MIA, a novel MIA capable of efficiently and accurately recovering training samples of clients from the aggregated updates, even when the system is under the protection of a robust secure aggregation protocol. Unlike existing approaches treating models as black boxes, Scale-MIA recognizes the importance of the intricate architecture and inner workings of machine learning models. It identifies the latent space as the critical layer for breaching privacy and decomposes the complex recovery task into an innovative two-step process to reduce computation complexity. The first step involves reconstructing the latent space representations (LSRs) from the aggregated model updates using a closed-form inversion mechanism, leveraging specially crafted adversarial linear layers. In the second step, the whole input batches are recovered from the LSRs by feeding them into a fine-tuned generative decoder.

  We implemented Scale-MIA on multiple commonly used machine learning models and conducted comprehensive experiments across various settings. The results demonstrate that Scale-MIA achieves excellent recovery performance on different datasets, exhibiting high reconstruction rates, accuracy, and attack efficiency on a larger scale compared to state-of-the-art MIAs.

Title: 1-Lipschitz Neural Networks are more expressive with N-Activations. (arXiv:2311.06103v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.06103
• Code URL: https://github.com/berndprach/nactivation
• Copy Paste: [[2311.06103]] 1-Lipschitz Neural Networks are more expressive with N-Activations(http://arxiv.org/abs/2311.06103)
• Summary:

  A crucial property for achieving secure, trustworthy and interpretable deep learning systems is their robustness: small changes to a system's inputs should not result in large changes to its outputs. Mathematically, this means one strives for networks with a small Lipschitz constant. Several recent works have focused on how to construct such Lipschitz networks, typically by imposing constraints on the weight matrices. In this work, we study an orthogonal aspect, namely the role of the activation function. We show that commonly used activation functions, such as MaxMin, as well as all piece-wise linear ones with two segments unnecessarily restrict the class of representable functions, even in the simplest one-dimensional setting. We furthermore introduce the new N-activation function that is provably more expressive than currently popular activation functions. We provide code at https://github.com/berndprach/NActivation.

security

Title: A high throughput Intrusion Detection System (IDS) to enhance the security of data transmission among research centers. (arXiv:2311.06082v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.06082
• Code URL: null
• Copy Paste: [[2311.06082]] A high throughput Intrusion Detection System (IDS) to enhance the security of data transmission among research centers(http://arxiv.org/abs/2311.06082)
• Summary:

  Data breaches and cyberattacks represent a severe problem in higher education institutions and universities that can result in illegal access to sensitive information and data loss. To enhance the security of data transmission, Intrusion Prevention Systems (IPS, i.e., firewalls) and Intrusion Detection Systems (IDS, i.e., packet sniffers) are used to detect potential threats in the exchanged data. IPSs and IDSs are usually designed as software programs running on a server machine. However, when the speed of exchanged data is too high, this solution can become unreliable. In this case, IPSs and IDSs designed on a real hardware platform, such as ASICs and FPGAs, represent a more reliable solution. This paper presents a packet sniffer that was designed using a commercial FPGA development board. The system can support a data throughput of 10 Gbit/s with preliminary results showing that the speed of data transmission can be reliably extended to 100 Gbit/s. The designed system is highly configurable by the user and can enhance the data protection of information transmitted using the Ethernet protocol. It is particularly suited for the security of universities and research centers, where point-to-point network connections are dominant and large amount of sensitive data are shared among different hosts.

Title: Triad: Trusted Timestamps in Untrusted Environments. (arXiv:2311.06156v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.06156
• Code URL: null
• Copy Paste: [[2311.06156]] Triad: Trusted Timestamps in Untrusted Environments(http://arxiv.org/abs/2311.06156)
• Summary:

  We aim to provide trusted time measurement mechanisms to applications and cloud infrastructure deployed in environments that could harbor potential adversaries, including the hardware infrastructure provider. Despite Trusted Execution Environments (TEEs) providing multiple security functionalities, timestamps from the Operating System are not covered. Nevertheless, some services require time for validating permissions or ordering events. To address that need, we introduce Triad, a trusted timestamp dispatcher of time readings. The solution provides trusted timestamps enforced by mutually supportive enclave-based clock servers that create a continuous trusted timeline. We leverage enclave properties such as forced exits and CPU-based counters to mitigate attacks on the server's timestamp counters. Triad produces trusted, confidential, monotonically-increasing timestamps with bounded error and desirable, non-trivial properties. Our implementation relies on Intel SGX and SCONE, allowing transparent usage. We evaluate Triad's error and behavior in multiple dimensions.

privacy

Title: Learning Human Action Recognition Representations Without Real Humans. (arXiv:2311.06231v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06231
• Code URL: https://github.com/howardzh01/ppma
• Copy Paste: [[2311.06231]] Learning Human Action Recognition Representations Without Real Humans(http://arxiv.org/abs/2311.06231)
• Summary:

  Pre-training on massive video datasets has become essential to achieve high action recognition performance on smaller downstream datasets. However, most large-scale video datasets contain images of people and hence are accompanied with issues related to privacy, ethics, and data protection, often preventing them from being publicly shared for reproducible research. Existing work has attempted to alleviate these problems by blurring faces, downsampling videos, or training on synthetic data. On the other hand, analysis on the transferability of privacy-preserving pre-trained models to downstream tasks has been limited. In this work, we study this problem by first asking the question: can we pre-train models for human action recognition with data that does not include real humans? To this end, we present, for the first time, a benchmark that leverages real-world videos with humans removed and synthetic data containing virtual humans to pre-train a model. We then evaluate the transferability of the representation learned on this data to a diverse set of downstream action recognition benchmarks. Furthermore, we propose a novel pre-training strategy, called Privacy-Preserving MAE-Align, to effectively combine synthetic data and human-removed real data. Our approach outperforms previous baselines by up to 5% and closes the performance gap between human and no-human action recognition representations on downstream tasks, for both linear probing and fine-tuning. Our benchmark, code, and models are available at https://github.com/howardzh01/PPMA .

Title: Does Differential Privacy Prevent Backdoor Attacks in Practice?. (arXiv:2311.06227v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.06227
• Code URL: null
• Copy Paste: [[2311.06227]] Does Differential Privacy Prevent Backdoor Attacks in Practice?(http://arxiv.org/abs/2311.06227)
• Summary:

  Differential Privacy (DP) was originally developed to protect privacy. However, it has recently been utilized to secure machine learning (ML) models from poisoning attacks, with DP-SGD receiving substantial attention. Nevertheless, a thorough investigation is required to assess the effectiveness of different DP techniques in preventing backdoor attacks in practice. In this paper, we investigate the effectiveness of DP-SGD and, for the first time in literature, examine PATE in the context of backdoor attacks. We also explore the role of different components of DP algorithms in defending against backdoor attacks and will show that PATE is effective against these attacks due to the bagging structure of the teacher models it employs. Our experiments reveal that hyperparameters and the number of backdoors in the training dataset impact the success of DP algorithms. Additionally, we propose Label-DP as a faster and more accurate alternative to DP-SGD and PATE. We conclude that while Label-DP algorithms generally offer weaker privacy protection, accurate hyper-parameter tuning can make them more effective than DP methods in defending against backdoor attacks while maintaining model accuracy.

Title: The Paradox of Noise: An Empirical Study of Noise-Infusion Mechanisms to Improve Generalization, Stability, and Privacy in Federated Learning. (arXiv:2311.05790v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05790
• Code URL: null
• Copy Paste: [[2311.05790]] The Paradox of Noise: An Empirical Study of Noise-Infusion Mechanisms to Improve Generalization, Stability, and Privacy in Federated Learning(http://arxiv.org/abs/2311.05790)
• Summary:

  In a data-centric era, concerns regarding privacy and ethical data handling grow as machine learning relies more on personal information. This empirical study investigates the privacy, generalization, and stability of deep learning models in the presence of additive noise in federated learning frameworks. Our main objective is to provide strategies to measure the generalization, stability, and privacy-preserving capabilities of these models and further improve them. To this end, five noise infusion mechanisms at varying noise levels within centralized and federated learning settings are explored. As model complexity is a key component of the generalization and stability of deep learning models during training and evaluation, a comparative analysis of three Convolutional Neural Network (CNN) architectures is provided. The paper introduces Signal-to-Noise Ratio (SNR) as a quantitative measure of the trade-off between privacy and training accuracy of noise-infused models, aiming to find the noise level that yields optimal privacy and accuracy. Moreover, the Price of Stability and Price of Anarchy are defined in the context of privacy-preserving deep learning, contributing to the systematic investigation of the noise infusion strategies to enhance privacy without compromising performance. Our research sheds light on the delicate balance between these critical factors, fostering a deeper understanding of the implications of noise-based regularization in machine learning. By leveraging noise as a tool for regularization and privacy enhancement, we aim to contribute to the development of robust, privacy-aware algorithms, ensuring that AI-driven solutions prioritize both utility and privacy.

protect

Title: Chatbots Are Not Reliable Text Annotators. (arXiv:2311.05769v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05769
• Code URL: https://github.com/centre-for-humanities-computing/llm-tweet-classification
• Copy Paste: [[2311.05769]] Chatbots Are Not Reliable Text Annotators(http://arxiv.org/abs/2311.05769)
• Summary:

  Recent research highlights the significant potential of ChatGPT for text annotation in social science research. However, ChatGPT is a closed-source product which has major drawbacks with regards to transparency, reproducibility, cost, and data protection. Recent advances in open-source (OS) large language models (LLMs) offer alternatives which remedy these challenges. This means that it is important to evaluate the performance of OS LLMs relative to ChatGPT and standard approaches to supervised machine learning classification. We conduct a systematic comparative evaluation of the performance of a range of OS LLM models alongside ChatGPT, using both zero- and few-shot learning as well as generic and custom prompts, with results compared to more traditional supervised classification models. Using a new dataset of Tweets from US news media, and focusing on simple binary text annotation tasks for standard social science concepts, we find significant variation in the performance of ChatGPT and OS models across the tasks, and that supervised classifiers consistently outperform both. Given the unreliable performance of ChatGPT and the significant challenges it poses to Open Science we advise against using ChatGPT for substantive text annotation tasks in social science research.

defense

Title: A Last-Level Defense for Application Integrity and Confidentiality. (arXiv:2311.06154v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.06154
• Code URL: null
• Copy Paste: [[2311.06154]] A Last-Level Defense for Application Integrity and Confidentiality(http://arxiv.org/abs/2311.06154)
• Summary:

  Our objective is to protect the integrity and confidentiality of applications operating in untrusted environments. Trusted Execution Environments (TEEs) are not a panacea. Hardware TEEs fail to protect applications against Sybil, Fork and Rollback Attacks and, consequently, fail to preserve the consistency and integrity of applications. We introduce a novel system, LLD, that enforces the integrity and consistency of applications in a transparent and scalable fashion. Our solution augments TEEs with instantiation control and rollback protection. Instantiation control, enforced with TEE-supported leases, mitigates Sybil/Fork Attacks without incurring the high costs of solving crypto-puzzles. Our rollback detection mechanism does not need excessive replication, nor does it sacrifice durability. We show that implementing these functionalities in the LLD runtime automatically protects applications and services such as a popular DBMS.

attack

Title: Robust Adversarial Attacks Detection for Deep Learning based Relative Pose Estimation for Space Rendezvous. (arXiv:2311.05992v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05992
• Code URL: null
• Copy Paste: [[2311.05992]] Robust Adversarial Attacks Detection for Deep Learning based Relative Pose Estimation for Space Rendezvous(http://arxiv.org/abs/2311.05992)
• Summary:

  Research on developing deep learning techniques for autonomous spacecraft relative navigation challenges is continuously growing in recent years. Adopting those techniques offers enhanced performance. However, such approaches also introduce heightened apprehensions regarding the trustability and security of such deep learning methods through their susceptibility to adversarial attacks. In this work, we propose a novel approach for adversarial attack detection for deep neural network-based relative pose estimation schemes based on the explainability concept. We develop for an orbital rendezvous scenario an innovative relative pose estimation technique adopting our proposed Convolutional Neural Network (CNN), which takes an image from the chaser's onboard camera and outputs accurately the target's relative position and rotation. We perturb seamlessly the input images using adversarial attacks that are generated by the Fast Gradient Sign Method (FGSM). The adversarial attack detector is then built based on a Long Short Term Memory (LSTM) network which takes the explainability measure namely SHapley Value from the CNN-based pose estimator and flags the detection of adversarial attacks when acting. Simulation results show that the proposed adversarial attack detector achieves a detection accuracy of 99.21%. Both the deep relative pose estimator and adversarial attack detector are then tested on real data captured from our laboratory-designed setup. The experimental results from our laboratory-designed setup demonstrate that the proposed adversarial attack detector achieves an average detection accuracy of 96.29%.

Title: Fight Fire with Fire: Combating Adversarial Patch Attacks using Pattern-randomized Defensive Patches. (arXiv:2311.06122v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06122
• Code URL: null
• Copy Paste: [[2311.06122]] Fight Fire with Fire: Combating Adversarial Patch Attacks using Pattern-randomized Defensive Patches(http://arxiv.org/abs/2311.06122)
• Summary:

  Object detection has found extensive applications in various tasks, but it is also susceptible to adversarial patch attacks. Existing defense methods often necessitate modifications to the target model or result in unacceptable time overhead. In this paper, we adopt a counterattack approach, following the principle of "fight fire with fire," and propose a novel and general methodology for defending adversarial attacks. We utilize an active defense strategy by injecting two types of defensive patches, canary and woodpecker, into the input to proactively probe or weaken potential adversarial patches without altering the target model. Moreover, inspired by randomization techniques employed in software security, we employ randomized canary and woodpecker injection patterns to defend against defense-aware attacks. The effectiveness and practicality of the proposed method are demonstrated through comprehensive experiments. The results illustrate that canary and woodpecker achieve high performance, even when confronted with unknown attack methods, while incurring limited time overhead. Furthermore, our method also exhibits sufficient robustness against defense-aware attacks, as evidenced by adaptive attack experiments.

Title: Fake Alignment: Are LLMs Really Aligned Well?. (arXiv:2311.05915v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05915
• Code URL: null
• Copy Paste: [[2311.05915]] Fake Alignment: Are LLMs Really Aligned Well?(http://arxiv.org/abs/2311.05915)
• Summary:

  The growing awareness of safety concerns in large language models (LLMs) has sparked considerable interest in the evaluation of safety within current research endeavors. This study investigates an interesting issue pertaining to the evaluation of LLMs, namely the substantial discrepancy in performance between multiple-choice questions and open-ended questions. Inspired by research on jailbreak attack patterns, we argue this is caused by mismatched generalization. That is, the LLM does not have a comprehensive understanding of the complex concept of safety. Instead, it only remembers what to answer for open-ended safety questions, which makes it unable to solve other forms of safety tests. We refer to this phenomenon as fake alignment and construct a comparative benchmark to empirically verify its existence in LLMs. Such fake alignment renders previous evaluation protocols unreliable. To address this, we introduce the FAEF framework and two novel metrics\textemdash Consistency Score (CS) and Consistent Safety Score (CSS), which jointly assess two complementary forms of evaluation to quantify fake alignment and obtain corrected performance estimates. Applying FAEF to 14 widely-used LLMs reveals several models with purported safety are poorly aligned in practice. Our work highlights potential limitations in prevailing alignment methodologies.

Title: Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration. (arXiv:2311.06062v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.06062
• Code URL: null
• Copy Paste: [[2311.06062]] Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration(http://arxiv.org/abs/2311.06062)
• Summary:

  Membership Inference Attacks (MIA) aim to infer whether a target data record has been utilized for model training or not. Prior attempts have quantified the privacy risks of language models (LMs) via MIAs, but there is still no consensus on whether existing MIA algorithms can cause remarkable privacy leakage on practical Large Language Models (LLMs). Existing MIAs designed for LMs can be classified into two categories: reference-free and reference-based attacks. They are both based on the hypothesis that training records consistently strike a higher probability of being sampled. Nevertheless, this hypothesis heavily relies on the overfitting of target models, which will be mitigated by multiple regularization methods and the generalization of LLMs. The reference-based attack seems to achieve promising effectiveness in LLMs, which measures a more reliable membership signal by comparing the probability discrepancy between the target model and the reference model. However, the performance of reference-based attack is highly dependent on a reference dataset that closely resembles the training dataset, which is usually inaccessible in the practical scenario. Overall, existing MIAs are unable to effectively unveil privacy leakage over practical fine-tuned LLMs that are overfitting-free and private. We propose a Membership Inference Attack based on Self-calibrated Probabilistic Variation (SPV-MIA). Specifically, since memorization in LLMs is inevitable during the training process and occurs before overfitting, we introduce a more reliable membership signal, probabilistic variation, which is based on memorization rather than overfitting. Furthermore, we introduce a self-prompt approach, which constructs the dataset to fine-tune the reference model by prompting the target LLM itself. In this manner, the adversary can collect a dataset with a similar distribution from public APIs.

Title: Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming in the Wild. (arXiv:2311.06237v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.06237
• Code URL: null
• Copy Paste: [[2311.06237]] Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming in the Wild(http://arxiv.org/abs/2311.06237)
• Summary:

  Engaging in the deliberate generation of abnormal outputs from large language models (LLMs) by attacking them is a novel human activity. This paper presents a thorough exposition of how and why people perform such attacks. Using a formal qualitative methodology, we interviewed dozens of practitioners from a broad range of backgrounds, all contributors to this novel work of attempting to cause LLMs to fail. We relate and connect this activity between its practitioners' motivations and goals; the strategies and techniques they deploy; and the crucial role the community plays. As a result, this paper presents a grounded theory of how and why people attack large language models: LLM red teaming in the wild.

Title: Honest Score Client Selection Scheme: Preventing Federated Learning Label Flipping Attacks in Non-IID Scenarios. (arXiv:2311.05826v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05826
• Code URL: null
• Copy Paste: [[2311.05826]] Honest Score Client Selection Scheme: Preventing Federated Learning Label Flipping Attacks in Non-IID Scenarios(http://arxiv.org/abs/2311.05826)
• Summary:

  Federated Learning (FL) is a promising technology that enables multiple actors to build a joint model without sharing their raw data. The distributed nature makes FL vulnerable to various poisoning attacks, including model poisoning attacks and data poisoning attacks. Today, many byzantine-resilient FL methods have been introduced to mitigate the model poisoning attack, while the effectiveness when defending against data poisoning attacks still remains unclear. In this paper, we focus on the most representative data poisoning attack - "label flipping attack" and monitor its effectiveness when attacking the existing FL methods. The results show that the existing FL methods perform similarly in Independent and identically distributed (IID) settings but fail to maintain the model robustness in Non-IID settings. To mitigate the weaknesses of existing FL methods in Non-IID scenarios, we introduce the Honest Score Client Selection (HSCS) scheme and the corresponding HSCSFL framework. In the HSCSFL, The server collects a clean dataset for evaluation. Under each iteration, the server collects the gradients from clients and then perform HSCS to select aggregation candidates. The server first evaluates the performance of each class of the global model and generates the corresponding risk vector to indicate which class could be potentially attacked. Similarly, the server evaluates the client's model and records the performance of each class as the accuracy vector. The dot product of each client's accuracy vector and global risk vector is generated as the client's host score; only the top p\% host score clients are included in the following aggregation. Finally, server aggregates the gradients and uses the outcome to update the global model. The comprehensive experimental results show our HSCSFL effectively enhances the FL robustness and defends against the "label flipping attack."

Title: KRATT: QBF-Assisted Removal and Structural Analysis Attack Against Logic Locking. (arXiv:2311.05982v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05982
• Code URL: null
• Copy Paste: [[2311.05982]] KRATT: QBF-Assisted Removal and Structural Analysis Attack Against Logic Locking(http://arxiv.org/abs/2311.05982)
• Summary:

  This paper introduces KRATT, a removal and structural analysis attack against state-of-the-art logic locking techniques, such as single and double flip locking techniques (SFLTs and DFLTs). KRATT utilizes powerful quantified Boolean formulas (QBFs), which have not found widespread use in hardware security, to find the secret key of SFLTs for the first time. It can handle locked circuits under both oracle-less (OL) and oracle-guided (OG) threat models. It modifies the locked circuit and uses a prominent OL attack to make a strong guess under the OL threat model. It uses a structural analysis technique to identify promising protected input patterns and explores them using the oracle under the OG model. Experimental results on ISCAS'85, ITC'99, and HeLLO: CTF'22 benchmarks show that KRATT can break SFLTs using a QBF formulation in less than a minute, can decipher a large number of key inputs of SFLTs and DFLTs with high accuracy under the OL threat model, and can easily find the secret key of DFLTs under the OG threat model. It is shown that KRATT outperforms publicly available OL and OG attacks in terms of solution quality and run-time.

robust

Title: OmniVec: Learning robust representations with cross modal sharing. (arXiv:2311.05709v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05709
• Code URL: null
• Copy Paste: [[2311.05709]] OmniVec: Learning robust representations with cross modal sharing(http://arxiv.org/abs/2311.05709)
• Summary:

  Majority of research in learning based methods has been towards designing and training networks for specific tasks. However, many of the learning based tasks, across modalities, share commonalities and could be potentially tackled in a joint framework. We present an approach in such direction, to learn multiple tasks, in multiple modalities, with a unified architecture. The proposed network is composed of task specific encoders, a common trunk in the middle, followed by task specific prediction heads. We first pre-train it by self-supervised masked training, followed by sequential training for the different tasks. We train the network on all major modalities, e.g.\ visual, audio, text and 3D, and report results on $22$ diverse and challenging public benchmarks. We demonstrate empirically that, using a joint network to train across modalities leads to meaningful information sharing and this allows us to achieve state-of-the-art results on most of the benchmarks. We also show generalization of the trained network on cross-modal tasks as well as unseen datasets and tasks.

Title: Tamil-Llama: A New Tamil Language Model Based on Llama 2. (arXiv:2311.05845v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05845
• Code URL: https://github.com/abhinand5/tamil-llama
• Copy Paste: [[2311.05845]] Tamil-Llama: A New Tamil Language Model Based on Llama 2(http://arxiv.org/abs/2311.05845)
• Summary:

  Language modeling has witnessed remarkable advancements in recent years, with Large Language Models (LLMs) like ChatGPT setting unparalleled benchmarks in human-like text generation. However, a prevailing limitation is the underrepresentation of languages like Tamil in these cutting-edge models, leading to suboptimal performance in diverse linguistic contexts. This paper addresses this lacuna, enhancing the open-source LLaMA model with an addition of 16,000 Tamil tokens, aiming to achieve superior text generation and comprehension in the Tamil language. We strategically employ the LoRA methodology for efficient model training on a comprehensive Tamil corpus, ensuring computational feasibility and model robustness. Moreover, we introduce a Tamil-translated version of the Alpaca dataset and a subset of the OpenOrca dataset tailored for instruction fine-tuning. Our results showcase significant performance improvements in Tamil text generation, with potential implications for the broader landscape of LLMs in Indian languages. We further underscore our commitment to open research by making our models, datasets, and code publicly accessible, fostering further innovations in language modeling.

Title: Robust Constant-Time Cryptography. (arXiv:2311.05831v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05831
• Code URL: null
• Copy Paste: [[2311.05831]] Robust Constant-Time Cryptography(http://arxiv.org/abs/2311.05831)
• Summary:

  The constant-time property is considered the security standard for cryptographic code. Code following the constant-time discipline is free from secret-dependent branches and memory accesses, and thus avoids leaking secrets through cache and timing side-channels. The constant-time property makes a number of implicit assumptions that are fundamentally at odds with the reality of cryptographic code. Constant-time is not robust. The first issue with constant-time is that it is a whole-program property: It relies on the entirety of the code base being constant-time. But, cryptographic developers do not generally write whole programs; rather, they provide libraries and specific algorithms for other application developers to use. As such, developers of security libraries must maintain their security guarantees even when their code is operating within (potentially untrusted) application contexts. Constant-time requires memory safety. The whole-program nature of constant-time also leads to a second issue: constant-time requires memory safety of all the running code. Any memory safety bugs, whether in the library or the application, will wend their way back to side-channel leaks of secrets if not direct disclosure. And although cryptographic libraries should (and are) written to be memory-safe, it is unfortunately unrealistic to expect the same from every application that uses each library. We formalize robust constant-time and build a RobustIsoCrypt compiler that transforms the library code and protects the secrets even when they are linked with untrusted code. Our evaluation with SUPERCOP benchmarking framework shows that the performance overhead is less than five percent on average.

Title: Enhancing Instance-Level Image Classification with Set-Level Labels. (arXiv:2311.05659v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05659
• Code URL: null
• Copy Paste: [[2311.05659]] Enhancing Instance-Level Image Classification with Set-Level Labels(http://arxiv.org/abs/2311.05659)
• Summary:

  Instance-level image classification tasks have traditionally relied on single-instance labels to train models, e.g., few-shot learning and transfer learning. However, set-level coarse-grained labels that capture relationships among instances can provide richer information in real-world scenarios. In this paper, we present a novel approach to enhance instance-level image classification by leveraging set-level labels. We provide a theoretical analysis of the proposed method, including recognition conditions for fast excess risk rate, shedding light on the theoretical foundations of our approach. We conducted experiments on two distinct categories of datasets: natural image datasets and histopathology image datasets. Our experimental results demonstrate the effectiveness of our approach, showcasing improved classification performance compared to traditional single-instance label-based methods. Notably, our algorithm achieves 13% improvement in classification accuracy compared to the strongest baseline on the histopathology image classification benchmarks. Importantly, our experimental findings align with the theoretical analysis, reinforcing the robustness and reliability of our proposed method. This work bridges the gap between instance-level and set-level image classification, offering a promising avenue for advancing the capabilities of image classification models with set-level coarse-grained labels.

Title: Towards stable real-world equation discovery with assessing differentiating quality influence. (arXiv:2311.05787v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05787
• Code URL: null
• Copy Paste: [[2311.05787]] Towards stable real-world equation discovery with assessing differentiating quality influence(http://arxiv.org/abs/2311.05787)
• Summary:

  This paper explores the critical role of differentiation approaches for data-driven differential equation discovery. Accurate derivatives of the input data are essential for reliable algorithmic operation, particularly in real-world scenarios where measurement quality is inevitably compromised. We propose alternatives to the commonly used finite differences-based method, notorious for its instability in the presence of noise, which can exacerbate random errors in the data. Our analysis covers four distinct methods: Savitzky-Golay filtering, spectral differentiation, smoothing based on artificial neural networks, and the regularization of derivative variation. We evaluate these methods in terms of applicability to problems, similar to the real ones, and their ability to ensure the convergence of equation discovery algorithms, providing valuable insights for robust modeling of real-world processes.

Title: Low-Multi-Rank High-Order Bayesian Robust Tensor Factorization. (arXiv:2311.05888v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05888
• Code URL: null
• Copy Paste: [[2311.05888]] Low-Multi-Rank High-Order Bayesian Robust Tensor Factorization(http://arxiv.org/abs/2311.05888)
• Summary:

  The recently proposed tensor robust principal component analysis (TRPCA) methods based on tensor singular value decomposition (t-SVD) have achieved numerous successes in many fields. However, most of these methods are only applicable to third-order tensors, whereas the data obtained in practice are often of higher order, such as fourth-order color videos, fourth-order hyperspectral videos, and fifth-order light-field images. Additionally, in the t-SVD framework, the multi-rank of a tensor can describe more fine-grained low-rank structure in the tensor compared with the tubal rank. However, determining the multi-rank of a tensor is a much more difficult problem than determining the tubal rank. Moreover, most of the existing TRPCA methods do not explicitly model the noises except the sparse noise, which may compromise the accuracy of estimating the low-rank tensor. In this work, we propose a novel high-order TRPCA method, named as Low-Multi-rank High-order Bayesian Robust Tensor Factorization (LMH-BRTF), within the Bayesian framework. Specifically, we decompose the observed corrupted tensor into three parts, i.e., the low-rank component, the sparse component, and the noise component. By constructing a low-rank model for the low-rank component based on the order-$d$ t-SVD and introducing a proper prior for the model, LMH-BRTF can automatically determine the tensor multi-rank. Meanwhile, benefiting from the explicit modeling of both the sparse and noise components, the proposed method can leverage information from the noises more effectivly, leading to an improved performance of TRPCA. Then, an efficient variational inference algorithm is established for parameters estimation. Empirical studies on synthetic and real-world datasets demonstrate the effectiveness of the proposed method in terms of both qualitative and quantitative results.

Title: Doubly Robust Structure Identification from Temporal Data. (arXiv:2311.06012v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.06012
• Code URL: null
• Copy Paste: [[2311.06012]] Doubly Robust Structure Identification from Temporal Data(http://arxiv.org/abs/2311.06012)
• Summary:

  Learning the causes of time-series data is a fundamental task in many applications, spanning from finance to earth sciences or bio-medical applications. Common approaches for this task are based on vector auto-regression, and they do not take into account unknown confounding between potential causes. However, in settings with many potential causes and noisy data, these approaches may be substantially biased. Furthermore, potential causes may be correlated in practical applications. Moreover, existing algorithms often do not work with cyclic data. To address these challenges, we propose a new doubly robust method for Structure Identification from Temporal Data ( SITD ). We provide theoretical guarantees, showing that our method asymptotically recovers the true underlying causal structure. Our analysis extends to cases where the potential causes have cycles and they may be confounded. We further perform extensive experiments to showcase the superior performance of our method.

Title: Distributionally Robust Skeleton Learning of Discrete Bayesian Networks. (arXiv:2311.06117v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.06117
• Code URL: null
• Copy Paste: [[2311.06117]] Distributionally Robust Skeleton Learning of Discrete Bayesian Networks(http://arxiv.org/abs/2311.06117)
• Summary:

  We consider the problem of learning the exact skeleton of general discrete Bayesian networks from potentially corrupted data. Building on distributionally robust optimization and a regression approach, we propose to optimize the most adverse risk over a family of distributions within bounded Wasserstein distance or KL divergence to the empirical distribution. The worst-case risk accounts for the effect of outliers. The proposed approach applies for general categorical random variables without assuming faithfulness, an ordinal relationship or a specific form of conditional distribution. We present efficient algorithms and show the proposed methods are closely related to the standard regularized regression approach. Under mild assumptions, we derive non-asymptotic guarantees for successful structure learning with logarithmic sample complexities for bounded-degree graphs. Numerical study on synthetic and real datasets validates the effectiveness of our method. Code is available at https://github.com/DanielLeee/drslbn.

biometric

Title: Whole-body Detection, Recognition and Identification at Altitude and Range. (arXiv:2311.05725v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05725
• Code URL: null
• Copy Paste: [[2311.05725]] Whole-body Detection, Recognition and Identification at Altitude and Range(http://arxiv.org/abs/2311.05725)
• Summary:

  In this paper, we address the challenging task of whole-body biometric detection, recognition, and identification at distances of up to 500m and large pitch angles of up to 50 degree. We propose an end-to-end system evaluated on diverse datasets, including the challenging Biometric Recognition and Identification at Range (BRIAR) dataset. Our approach involves pre-training the detector on common image datasets and fine-tuning it on BRIAR's complex videos and images. After detection, we extract body images and employ a feature extractor for recognition. We conduct thorough evaluations under various conditions, such as different ranges and angles in indoor, outdoor, and aerial scenarios. Our method achieves an average F1 score of 98.29% at IoU = 0.7 and demonstrates strong performance in recognition accuracy and true acceptance rate at low false acceptance rates compared to existing models. On a test set of 100 subjects with 444 distractors, our model achieves a rank-20 recognition accuracy of 75.13% and a TAR@1%FAR of 54.09%.

Title: Keystroke Verification Challenge (KVC): Biometric and Fairness Benchmark Evaluation. (arXiv:2311.06000v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06000
• Code URL: null
• Copy Paste: [[2311.06000]] Keystroke Verification Challenge (KVC): Biometric and Fairness Benchmark Evaluation(http://arxiv.org/abs/2311.06000)
• Summary:

  Analyzing keystroke dynamics (KD) for biometric verification has several advantages: it is among the most discriminative behavioral traits; keyboards are among the most common human-computer interfaces, being the primary means for users to enter textual data; its acquisition does not require additional hardware, and its processing is relatively lightweight; and it allows for transparently recognizing subjects. However, the heterogeneity of experimental protocols and metrics, and the limited size of the databases adopted in the literature impede direct comparisons between different systems, thus representing an obstacle in the advancement of keystroke biometrics. To alleviate this aspect, we present a new experimental framework to benchmark KD-based biometric verification performance and fairness based on tweet-long sequences of variable transcript text from over 185,000 subjects, acquired through desktop and mobile keyboards, extracted from the Aalto Keystroke Databases. The framework runs on CodaLab in the form of the Keystroke Verification Challenge (KVC). Moreover, we also introduce a novel fairness metric, the Skewed Impostor Ratio (SIR), to capture inter- and intra-demographic group bias patterns in the verification scores. We demonstrate the usefulness of the proposed framework by employing two state-of-the-art keystroke verification systems, TypeNet and TypeFormer, to compare different sets of input features, achieving a less privacy-invasive system, by discarding the analysis of text content (ASCII codes of the keys pressed) in favor of extended features in the time domain. Our experiments show that this approach allows to maintain satisfactory performance.

steal

extraction

Title: DONUT-hole: DONUT Sparsification by Harnessing Knowledge and Optimizing Learning Efficiency. (arXiv:2311.05778v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05778
• Code URL: null
• Copy Paste: [[2311.05778]] DONUT-hole: DONUT Sparsification by Harnessing Knowledge and Optimizing Learning Efficiency(http://arxiv.org/abs/2311.05778)
• Summary:

  This paper introduces DONUT-hole, a sparse OCR-free visual document understanding (VDU) model that addresses the limitations of its predecessor model, dubbed DONUT. The DONUT model, leveraging a transformer architecture, overcoming the challenges of separate optical character recognition (OCR) and visual semantic understanding (VSU) components. However, its deployment in production environments and edge devices is hindered by high memory and computational demands, particularly in large-scale request services. To overcome these challenges, we propose an optimization strategy based on knowledge distillation and model pruning. Our paradigm to produce DONUT-hole, reduces the model denisty by 54\% while preserving performance. We also achieve a global representational similarity index between DONUT and DONUT-hole based on centered kernel alignment (CKA) metric of 0.79. Moreover, we evaluate the effectiveness of DONUT-hole in the document image key information extraction (KIE) task, highlighting its potential for developing more efficient VDU systems for logistic companies.

Title: Chain of Thought with Explicit Evidence Reasoning for Few-shot Relation Extraction. (arXiv:2311.05922v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05922
• Code URL: null
• Copy Paste: [[2311.05922]] Chain of Thought with Explicit Evidence Reasoning for Few-shot Relation Extraction(http://arxiv.org/abs/2311.05922)
• Summary:

  Few-shot relation extraction involves identifying the type of relationship between two specific entities within a text, using a limited number of annotated samples. A variety of solutions to this problem have emerged by applying meta-learning and neural graph techniques which typically necessitate a training process for adaptation. Recently, the strategy of in-context learning has been demonstrating notable results without the need of training. Few studies have already utilized in-context learning for zero-shot information extraction. Unfortunately, the evidence for inference is either not considered or implicitly modeled during the construction of chain-of-thought prompts. In this paper, we propose a novel approach for few-shot relation extraction using large language models, named CoT-ER, chain-of-thought with explicit evidence reasoning. In particular, CoT-ER first induces large language models to generate evidences using task-specific and concept-level knowledge. Then these evidences are explicitly incorporated into chain-of-thought prompting for relation extraction. Experimental results demonstrate that our CoT-ER approach (with 0% training data) achieves competitive performance compared to the fully-supervised (with 100% training data) state-of-the-art approach on the FewRel1.0 and FewRel2.0 datasets.

Title: Syntax-semantics interface: an algebraic model. (arXiv:2311.06189v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.06189
• Code URL: null
• Copy Paste: [[2311.06189]] Syntax-semantics interface: an algebraic model(http://arxiv.org/abs/2311.06189)
• Summary:

  We extend our formulation of Merge and Minimalism in terms of Hopf algebras to an algebraic model of a syntactic-semantic interface. We show that methods adopted in the formulation of renormalization (extraction of meaningful physical values) in theoretical physics are relevant to describe the extraction of meaning from syntactic expressions. We show how this formulation relates to computational models of semantics and we answer some recent controversies about implications for generative linguistics of the current functioning of large language models.

membership infer

federate

Title: Federated Learning Across Decentralized and Unshared Archives for Remote Sensing Image Classification. (arXiv:2311.06141v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06141
• Code URL: null
• Copy Paste: [[2311.06141]] Federated Learning Across Decentralized and Unshared Archives for Remote Sensing Image Classification(http://arxiv.org/abs/2311.06141)
• Summary:

  Federated learning (FL) enables the collaboration of multiple deep learning models to learn from decentralized data archives (i.e., clients) without accessing data on clients. Although FL offers ample opportunities in knowledge discovery from distributed image archives, it is seldom considered in remote sensing (RS). In this paper, as a first time in RS, we present a comparative study of state-of-the-art FL algorithms. To this end, we initially provide a systematic review of the FL algorithms presented in the computer vision community for image classification problems, and select several state-of-the-art FL algorithms based on their effectiveness with respect to training data heterogeneity across clients (known as non-IID data). After presenting an extensive overview of the selected algorithms, a theoretical comparison of the algorithms is conducted based on their: 1) local training complexity; 2) aggregation complexity; 3) learning efficiency; 4) communication cost; and 5) scalability in terms of number of clients. As the classification task, we consider multi-label classification (MLC) problem since RS images typically consist of multiple classes, and thus can simultaneously be associated with multi-labels. After the theoretical comparison, experimental analyses are presented to compare them under different decentralization scenarios in terms of MLC performance. Based on our comprehensive analyses, we finally derive a guideline for selecting suitable FL algorithms in RS. The code of this work will be publicly available at https://git.tu-berlin.de/rsim/FL-RS.

Title: Federated Learning with Manifold Regularization and Normalized Update Reaggregation. (arXiv:2311.05924v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05924
• Code URL: null
• Copy Paste: [[2311.05924]] Federated Learning with Manifold Regularization and Normalized Update Reaggregation(http://arxiv.org/abs/2311.05924)
• Summary:

  Federated Learning (FL) is an emerging collaborative machine learning framework where multiple clients train the global model without sharing their own datasets. In FL, the model inconsistency caused by the local data heterogeneity across clients results in the near-orthogonality of client updates, which leads to the global update norm reduction and slows down the convergence. Most previous works focus on eliminating the difference of parameters (or gradients) between the local and global models, which may fail to reflect the model inconsistency due to the complex structure of the machine learning model and the Euclidean space's limitation in meaningful geometric representations. In this paper, we propose FedMRUR by adopting the manifold model fusion scheme and a new global optimizer to alleviate the negative impacts. Concretely, FedMRUR adopts a hyperbolic graph manifold regularizer enforcing the representations of the data in the local and global models are close to each other in a low-dimensional subspace. Because the machine learning model has the graph structure, the distance in hyperbolic space can reflect the model bias better than the Euclidean distance. In this way, FedMRUR exploits the manifold structures of the representations to significantly reduce the model inconsistency. FedMRUR also aggregates the client updates norms as the global update norm, which can appropriately enlarge each client's contribution to the global update, thereby mitigating the norm reduction introduced by the near-orthogonality of client updates. Furthermore, we theoretically prove that our algorithm can achieve a linear speedup property for non-convex setting under partial client participation.Experiments demonstrate that FedMRUR can achieve a new state-of-the-art (SOTA) accuracy with less communication.

Title: Aggregation Weighting of Federated Learning via Generalization Bound Estimation. (arXiv:2311.05936v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05936
• Code URL: null
• Copy Paste: [[2311.05936]] Aggregation Weighting of Federated Learning via Generalization Bound Estimation(http://arxiv.org/abs/2311.05936)
• Summary:

  Federated Learning (FL) typically aggregates client model parameters using a weighting approach determined by sample proportions. However, this naive weighting method may lead to unfairness and degradation in model performance due to statistical heterogeneity and the inclusion of noisy data among clients. Theoretically, distributional robustness analysis has shown that the generalization performance of a learning model with respect to any shifted distribution is bounded. This motivates us to reconsider the weighting approach in federated learning. In this paper, we replace the aforementioned weighting method with a new strategy that considers the generalization bounds of each local model. Specifically, we estimate the upper and lower bounds of the second-order origin moment of the shifted distribution for the current local model, and then use these bounds disagreements as the aggregation proportions for weightings in each communication round. Experiments demonstrate that the proposed weighting strategy significantly improves the performance of several representative FL algorithms on benchmark datasets.

fair

Title: Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization. (arXiv:2311.06243v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.06243
• Code URL: null
• Copy Paste: [[2311.06243]] Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization(http://arxiv.org/abs/2311.06243)
• Summary:

  Large foundation models are becoming ubiquitous, but training them from scratch is prohibitively expensive. Thus, efficiently adapting these powerful models to downstream tasks is increasingly important. In this paper, we study a principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream task adaptation. Despite demonstrating good generalizability, OFT still uses a fairly large number of trainable parameters due to the high dimensionality of orthogonal matrices. To address this, we start by examining OFT from an information transmission perspective, and then identify a few key desiderata that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast Fourier transform algorithm enables efficient information transmission, we propose an efficient orthogonal parameterization using butterfly structures. We apply this parameterization to OFT, creating a novel parameter-efficient finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a special case, BOFT introduces a generalized orthogonal finetuning framework. Finally, we conduct an extensive empirical study of adapting large vision transformers, large language models, and text-to-image diffusion models to various downstream tasks in vision and language.

interpretability

Title: Are "Hierarchical" Visual Representations Hierarchical?. (arXiv:2311.05784v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05784
• Code URL: https://github.com/ethanlshen/hiernet
• Copy Paste: [[2311.05784]] Are "Hierarchical" Visual Representations Hierarchical?(http://arxiv.org/abs/2311.05784)
• Summary:

  Learned visual representations often capture large amounts of semantic information for accurate downstream applications. Human understanding of the world is fundamentally grounded in hierarchy. To mimic this and further improve representation capabilities, the community has explored "hierarchical" visual representations that aim at modeling the underlying hierarchy of the visual world. In this work, we set out to investigate if hierarchical visual representations truly capture the human perceived hierarchy better than standard learned representations. To this end, we create HierNet, a suite of 12 datasets spanning 3 kinds of hierarchy from the BREEDs subset of ImageNet. After extensive evaluation of Hyperbolic and Matryoshka Representations across training setups, we conclude that they do not capture hierarchy any better than the standard representations but can assist in other aspects like search efficiency and interpretability. Our benchmark and the datasets are open-sourced at https://github.com/ethanlshen/HierNet.

Title: Interpretable Graph Anomaly Detection using Gradient Attention Maps. (arXiv:2311.06153v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.06153
• Code URL: null
• Copy Paste: [[2311.06153]] Interpretable Graph Anomaly Detection using Gradient Attention Maps(http://arxiv.org/abs/2311.06153)
• Summary:

  Detecting unusual patterns in graph data is a crucial task in data mining. However, existing methods often face challenges in consistently achieving satisfactory performance and lack interpretability, which hinders our understanding of anomaly detection decisions. In this paper, we propose a novel approach to graph anomaly detection that leverages the power of interpretability to enhance performance. Specifically, our method extracts an attention map derived from gradients of graph neural networks, which serves as a basis for scoring anomalies. In addition, we conduct theoretical analysis using synthetic data to validate our method and gain insights into its decision-making process. To demonstrate the effectiveness of our method, we extensively evaluate our approach against state-of-the-art graph anomaly detection techniques. The results consistently demonstrate the superior performance of our method compared to the baselines.

Title: Greedy PIG: Adaptive Integrated Gradients. (arXiv:2311.06192v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.06192
• Code URL: null
• Copy Paste: [[2311.06192]] Greedy PIG: Adaptive Integrated Gradients(http://arxiv.org/abs/2311.06192)
• Summary:

  Deep learning has become the standard approach for most machine learning tasks. While its impact is undeniable, interpreting the predictions of deep learning models from a human perspective remains a challenge. In contrast to model training, model interpretability is harder to quantify and pose as an explicit optimization problem. Inspired by the AUC softmax information curve (AUC SIC) metric for evaluating feature attribution methods, we propose a unified discrete optimization framework for feature attribution and feature selection based on subset selection. This leads to a natural adaptive generalization of the path integrated gradients (PIG) method for feature attribution, which we call Greedy PIG. We demonstrate the success of Greedy PIG on a wide variety of tasks, including image feature attribution, graph compression/explanation, and post-hoc feature selection on tabular data. Our results show that introducing adaptivity is a powerful and versatile method for making attribution methods more powerful.

explainability

Title: Explainable artificial intelligence for Healthcare applications using Random Forest Classifier with LIME and SHAP. (arXiv:2311.05665v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05665
• Code URL: null
• Copy Paste: [[2311.05665]] Explainable artificial intelligence for Healthcare applications using Random Forest Classifier with LIME and SHAP(http://arxiv.org/abs/2311.05665)
• Summary:

  With the advances in computationally efficient artificial Intelligence (AI) techniques and their numerous applications in our everyday life, there is a pressing need to understand the computational details hidden in black box AI techniques such as most popular machine learning and deep learning techniques; through more detailed explanations. The origin of explainable AI (xAI) is coined from these challenges and recently gained more attention by the researchers by adding explainability comprehensively in traditional AI systems. This leads to develop an appropriate framework for successful applications of xAI in real life scenarios with respect to innovations, risk mitigation, ethical issues and logical values to the users. In this book chapter, an in-depth analysis of several xAI frameworks and methods including LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations) are provided. Random Forest Classifier as black box AI is used on a publicly available Diabetes symptoms dataset with LIME and SHAP for better interpretations. The results obtained are interesting in terms of transparency, valid and trustworthiness in diabetes disease prediction.

watermark

Title: Watermarking Vision-Language Pre-trained Models for Multi-modal Embedding as a Service. (arXiv:2311.05863v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05863
• Code URL: https://github.com/Pter61/vlpmarker
• Copy Paste: [[2311.05863]] Watermarking Vision-Language Pre-trained Models for Multi-modal Embedding as a Service(http://arxiv.org/abs/2311.05863)
• Summary:

  Recent advances in vision-language pre-trained models (VLPs) have significantly increased visual understanding and cross-modal analysis capabilities. Companies have emerged to provide multi-modal Embedding as a Service (EaaS) based on VLPs (e.g., CLIP-based VLPs), which cost a large amount of training data and resources for high-performance service. However, existing studies indicate that EaaS is vulnerable to model extraction attacks that induce great loss for the owners of VLPs. Protecting the intellectual property and commercial ownership of VLPs is increasingly crucial yet challenging. A major solution of watermarking model for EaaS implants a backdoor in the model by inserting verifiable trigger embeddings into texts, but it is only applicable for large language models and is unrealistic due to data and model privacy. In this paper, we propose a safe and robust backdoor-based embedding watermarking method for VLPs called VLPMarker. VLPMarker utilizes embedding orthogonal transformation to effectively inject triggers into the VLPs without interfering with the model parameters, which achieves high-quality copyright verification and minimal impact on model performance. To enhance the watermark robustness, we further propose a collaborative copyright verification strategy based on both backdoor trigger and embedding distribution, enhancing resilience against various attacks. We increase the watermark practicality via an out-of-distribution trigger selection approach, removing access to the model training data and thus making it possible for many real-world scenarios. Our extensive experiments on various datasets indicate that the proposed watermarking approach is effective and safe for verifying the copyright of VLPs for multi-modal EaaS and robust against model extraction attacks. Our code is available at https://github.com/Pter61/vlpmarker.

diffusion

Title: Diffusion Shape Prior for Wrinkle-Accurate Cloth Registration. (arXiv:2311.05828v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05828
• Code URL: null
• Copy Paste: [[2311.05828]] Diffusion Shape Prior for Wrinkle-Accurate Cloth Registration(http://arxiv.org/abs/2311.05828)
• Summary:

  Registering clothes from 4D scans with vertex-accurate correspondence is challenging, yet important for dynamic appearance modeling and physics parameter estimation from real-world data. However, previous methods either rely on texture information, which is not always reliable, or achieve only coarse-level alignment. In this work, we present a novel approach to enabling accurate surface registration of texture-less clothes with large deformation. Our key idea is to effectively leverage a shape prior learned from pre-captured clothing using diffusion models. We also propose a multi-stage guidance scheme based on learned functional maps, which stabilizes registration for large-scale deformation even when they vary significantly from training data. Using high-fidelity real captured clothes, our experiments show that the proposed approach based on diffusion models generalizes better than surface registration with VAE or PCA-based priors, outperforming both optimization-based and learning-based non-rigid registration methods for both interpolation and extrapolation tests.

Title: Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model. (arXiv:2311.06214v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06214
• Code URL: null
• Copy Paste: [[2311.06214]] Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model(http://arxiv.org/abs/2311.06214)
• Summary:

  Text-to-3D with diffusion models have achieved remarkable progress in recent years. However, existing methods either rely on score distillation-based optimization which suffer from slow inference, low diversity and Janus problems, or are feed-forward methods that generate low quality results due to the scarcity of 3D training data. In this paper, we propose Instant3D, a novel method that generates high-quality and diverse 3D assets from text prompts in a feed-forward manner. We adopt a two-stage paradigm, which first generates a sparse set of four structured and consistent views from text in one shot with a fine-tuned 2D text-to-image diffusion model, and then directly regresses the NeRF from the generated images with a novel transformer-based sparse-view reconstructor. Through extensive experiments, we demonstrate that our method can generate high-quality, diverse and Janus-free 3D assets within 20 seconds, which is two order of magnitude faster than previous optimization-based methods that can take 1 to 10 hours. Our project webpage: https://jiahao.ai/instant3d/.

Title: Diffusion Models for Earth Observation Use-cases: from cloud removal to urban change detection. (arXiv:2311.06222v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06222
• Code URL: null
• Copy Paste: [[2311.06222]] Diffusion Models for Earth Observation Use-cases: from cloud removal to urban change detection(http://arxiv.org/abs/2311.06222)
• Summary:

  The advancements in the state of the art of generative Artificial Intelligence (AI) brought by diffusion models can be highly beneficial in novel contexts involving Earth observation data. After introducing this new family of generative models, this work proposes and analyses three use cases which demonstrate the potential of diffusion-based approaches for satellite image data. Namely, we tackle cloud removal and inpainting, dataset generation for change-detection tasks, and urban replanning.

noise learning

data-free

transformer

Title: FMViT: A multiple-frequency mixing Vision Transformer. (arXiv:2311.05707v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05707
• Code URL: null
• Copy Paste: [[2311.05707]] FMViT: A multiple-frequency mixing Vision Transformer(http://arxiv.org/abs/2311.05707)
• Summary:

  The transformer model has gained widespread adoption in computer vision tasks in recent times. However, due to the quadratic time and memory complexity of self-attention, which is proportional to the number of input tokens, most existing Vision Transformers (ViTs) encounter challenges in achieving efficient performance in practical industrial deployment scenarios, such as TensorRT and CoreML, where traditional CNNs excel. Although some recent attempts have been made to design CNN-Transformer hybrid architectures to tackle this problem, their overall performance has not met expectations. To tackle these challenges, we propose an efficient hybrid ViT architecture named FMViT. This approach enhances the model's expressive power by blending high-frequency features and low-frequency features with varying frequencies, enabling it to capture both local and global information effectively. Additionally, we introduce deploy-friendly mechanisms such as Convolutional Multigroup Reparameterization (gMLP), Lightweight Multi-head Self-Attention (RLMHSA), and Convolutional Fusion Block (CFB) to further improve the model's performance and reduce computational overhead. Our experiments demonstrate that FMViT surpasses existing CNNs, ViTs, and CNNTransformer hybrid architectures in terms of latency/accuracy trade-offs for various vision tasks. On the TensorRT platform, FMViT outperforms Resnet101 by 2.5% (83.3% vs. 80.8%) in top-1 accuracy on the ImageNet dataset while maintaining similar inference latency. Moreover, FMViT achieves comparable performance with EfficientNet-B5, but with a 43% improvement in inference speed. On CoreML, FMViT outperforms MobileOne by 2.6% in top-1 accuracy on the ImageNet dataset, with inference latency comparable to MobileOne (78.5% vs. 75.9%). Our code can be found at https://github.com/tany0699/FMViT.

Title: Intelligent Cervical Spine Fracture Detection Using Deep Learning Methods. (arXiv:2311.05708v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05708
• Code URL: null
• Copy Paste: [[2311.05708]] Intelligent Cervical Spine Fracture Detection Using Deep Learning Methods(http://arxiv.org/abs/2311.05708)
• Summary:

  Cervical spine fractures constitute a critical medical emergency, with the potential for lifelong paralysis or even fatality if left untreated or undetected. Over time, these fractures can deteriorate without intervention. To address the lack of research on the practical application of deep learning techniques for the detection of spine fractures, this study leverages a dataset containing both cervical spine fractures and non-fractured computed tomography images. This paper introduces a two-stage pipeline designed to identify the presence of cervical vertebrae in each image slice and pinpoint the location of fractures. In the first stage, a multi-input network, incorporating image and image metadata, is trained. This network is based on the Global Context Vision Transformer, and its performance is benchmarked against popular deep learning image classification model. In the second stage, a YOLOv8 model is trained to detect fractures within the images, and its effectiveness is compared to YOLOv5. The obtained results indicate that the proposed algorithm significantly reduces the workload of radiologists and enhances the accuracy of fracture detection.

Title: PolyMaX: General Dense Prediction with Mask Transformer. (arXiv:2311.05770v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05770
• Code URL: https://github.com/google-research/deeplab2
• Copy Paste: [[2311.05770]] PolyMaX: General Dense Prediction with Mask Transformer(http://arxiv.org/abs/2311.05770)
• Summary:

  Dense prediction tasks, such as semantic segmentation, depth estimation, and surface normal prediction, can be easily formulated as per-pixel classification (discrete outputs) or regression (continuous outputs). This per-pixel prediction paradigm has remained popular due to the prevalence of fully convolutional networks. However, on the recent frontier of segmentation task, the community has been witnessing a shift of paradigm from per-pixel prediction to cluster-prediction with the emergence of transformer architectures, particularly the mask transformers, which directly predicts a label for a mask instead of a pixel. Despite this shift, methods based on the per-pixel prediction paradigm still dominate the benchmarks on the other dense prediction tasks that require continuous outputs, such as depth estimation and surface normal prediction. Motivated by the success of DORN and AdaBins in depth estimation, achieved by discretizing the continuous output space, we propose to generalize the cluster-prediction based method to general dense prediction tasks. This allows us to unify dense prediction tasks with the mask transformer framework. Remarkably, the resulting model PolyMaX demonstrates state-of-the-art performance on three benchmarks of NYUD-v2 dataset. We hope our simple yet effective design can inspire more research on exploiting mask transformers for more dense prediction tasks. Code and model will be made available.

Title: Vision Big Bird: Random Sparsification for Full Attention. (arXiv:2311.05988v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05988
• Code URL: null
• Copy Paste: [[2311.05988]] Vision Big Bird: Random Sparsification for Full Attention(http://arxiv.org/abs/2311.05988)
• Summary:

  Recently, Transformers have shown promising performance in various vision tasks. However, the high costs of global self-attention remain challenging for Transformers, especially for high-resolution vision tasks. Inspired by one of the most successful transformers-based models for NLP: Big Bird, we propose a novel sparse attention mechanism for Vision Transformers (ViT). Specifically, we separate the heads into three groups, the first group used convolutional neural network (CNN) to extract local features and provide positional information for the model, the second group used Random Sampling Windows (RS-Win) for sparse self-attention calculation, and the third group reduces the resolution of the keys and values by average pooling for global attention. Based on these components, ViT maintains the sparsity of self-attention while maintaining the merits of Big Bird (i.e., the model is a universal approximator of sequence functions and is Turing complete). Moreover, our results show that the positional encoding, a crucial component in ViTs, can be safely removed in our model. Experiments show that Vision Big Bird demonstrates competitive performance on common vision tasks.

Title: Dual input stream transformer for eye-tracking line assignment. (arXiv:2311.06095v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06095
• Code URL: null
• Copy Paste: [[2311.06095]] Dual input stream transformer for eye-tracking line assignment(http://arxiv.org/abs/2311.06095)
• Summary:

  We introduce a novel Dual Input Stream Transformer (DIST) for the challenging problem of assigning fixation points from eye-tracking data collected during passage reading to the line of text that the reader was actually focused on. This post-processing step is crucial for analysis of the reading data due to the presence of noise in the form of vertical drift. We evaluate DIST against nine classical approaches on a comprehensive suite of nine diverse datasets, and demonstrate DIST's superiority. By combining multiple instances of the DIST model in an ensemble we achieve an average accuracy of 98.5\% across all datasets. Our approach presents a significant step towards addressing the bottleneck of manual line assignment in reading research. Through extensive model analysis and ablation studies, we identify key factors that contribute to DIST's success, including the incorporation of line overlap features and the use of a second input stream. Through evaluation on a set of diverse datasets we demonstrate that DIST is robust to various experimental setups, making it a safe first choice for practitioners in the field.

Title: Automatic Report Generation for Histopathology images using pre-trained Vision Transformers. (arXiv:2311.06176v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06176
• Code URL: null
• Copy Paste: [[2311.06176]] Automatic Report Generation for Histopathology images using pre-trained Vision Transformers(http://arxiv.org/abs/2311.06176)
• Summary:

  Deep learning for histopathology has been successfully used for disease classification, image segmentation and more. However, combining image and text modalities using current state-of-the-art methods has been a challenge due to the high resolution of histopathology images. Automatic report generation for histopathology images is one such challenge. In this work, we show that using an existing pre-trained Vision Transformer in a two-step process of first using it to encode 4096x4096 sized patches of the Whole Slide Image (WSI) and then using it as the encoder and an LSTM decoder for report generation, we can build a fairly performant and portable report generation mechanism that takes into account the whole of the high resolution image, instead of just the patches. We are also able to use representations from an existing powerful pre-trained hierarchical vision transformer and show its usefulness in not just zero shot classification but also for report generation.

Title: Deep Natural Language Feature Learning for Interpretable Prediction. (arXiv:2311.05754v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05754
• Code URL: https://github.com/furrutiav/nllf-emnlp-2023
• Copy Paste: [[2311.05754]] Deep Natural Language Feature Learning for Interpretable Prediction(http://arxiv.org/abs/2311.05754)
• Summary:

  We propose a general method to break down a main complex task into a set of intermediary easier sub-tasks, which are formulated in natural language as binary questions related to the final target task. Our method allows for representing each example by a vector consisting of the answers to these questions. We call this representation Natural Language Learned Features (NLLF). NLLF is generated by a small transformer language model (e.g., BERT) that has been trained in a Natural Language Inference (NLI) fashion, using weak labels automatically obtained from a Large Language Model (LLM). We show that the LLM normally struggles for the main task using in-context learning, but can handle these easiest subtasks and produce useful weak labels to train a BERT. The NLI-like training of the BERT allows for tackling zero-shot inference with any binary question, and not necessarily the ones seen during the training. We show that this NLLF vector not only helps to reach better performances by enhancing any classifier, but that it can be used as input of an easy-to-interpret machine learning model like a decision tree. This decision tree is interpretable but also reaches high performances, surpassing those of a pre-trained transformer in some cases.We have successfully applied this method to two completely different tasks: detecting incoherence in students' answers to open-ended mathematics exam questions, and screening abstracts for a systematic literature review of scientific papers on climate change and agroecology.

Title: The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models. (arXiv:2311.05928v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05928
• Code URL: null
• Copy Paste: [[2311.05928]] The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Models(http://arxiv.org/abs/2311.05928)
• Summary:

  In this study, we present an investigation into the anisotropy dynamics and intrinsic dimension of embeddings in transformer architectures, focusing on the dichotomy between encoders and decoders. Our findings reveal that the anisotropy profile in transformer decoders exhibits a distinct bell-shaped curve, with the highest anisotropy concentrations in the middle layers. This pattern diverges from the more uniformly distributed anisotropy observed in encoders. In addition, we found that the intrinsic dimension of embeddings increases in the initial phases of training, indicating an expansion into higher-dimensional space. Which is then followed by a compression phase towards the end of training with dimensionality decrease, suggesting a refinement into more compact representations. Our results provide fresh insights to the understanding of encoders and decoders embedding properties.

Title: BanglaBait: Semi-Supervised Adversarial Approach for Clickbait Detection on Bangla Clickbait Dataset. (arXiv:2311.06204v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.06204
• Code URL: https://github.com/mdmotaharmahtab/banglabait
• Copy Paste: [[2311.06204]] BanglaBait: Semi-Supervised Adversarial Approach for Clickbait Detection on Bangla Clickbait Dataset(http://arxiv.org/abs/2311.06204)
• Summary:

  Intentionally luring readers to click on a particular content by exploiting their curiosity defines a title as clickbait. Although several studies focused on detecting clickbait titles in English articles, low resource language like Bangla has not been given adequate attention. To tackle clickbait titles in Bangla, we have constructed the first Bangla clickbait detection dataset containing 15,056 labeled news articles and 65,406 unlabelled news articles extracted from clickbait dense news sites. Each article has been labeled by three expert linguists and includes an article's title, body, and other metadata. By incorporating labeled and unlabelled data, we finetune a pretrained Bangla transformer model in an adversarial fashion using Semi Supervised Generative Adversarial Networks (SS GANs). The proposed model acts as a good baseline for this dataset, outperforming traditional neural network models (LSTM, GRU, CNN) and linguistic feature based models. We expect that this dataset and the detailed analysis and comparison of these clickbait detection models will provide a fundamental basis for future research into detecting clickbait titles in Bengali articles. We have released the corresponding code and dataset.

Title: Argumentation Element Annotation Modeling using XLNet. (arXiv:2311.06239v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.06239
• Code URL: null
• Copy Paste: [[2311.06239]] Argumentation Element Annotation Modeling using XLNet(http://arxiv.org/abs/2311.06239)
• Summary:

  This study demonstrates the effectiveness of XLNet, a transformer-based language model, for annotating argumentative elements in persuasive essays. XLNet's architecture incorporates a recurrent mechanism that allows it to model long-term dependencies in lengthy texts. Fine-tuned XLNet models were applied to three datasets annotated with different schemes - a proprietary dataset using the Annotations for Revisions and Reflections on Writing (ARROW) scheme, the PERSUADE corpus, and the Argument Annotated Essays (AAE) dataset. The XLNet models achieved strong performance across all datasets, even surpassing human agreement levels in some cases. This shows XLNet capably handles diverse annotation schemes and lengthy essays. Comparisons between the model outputs on different datasets also revealed insights into the relationships between the annotation tags. Overall, XLNet's strong performance on modeling argumentative structures across diverse datasets highlights its suitability for providing automated feedback on essay organization.

Title: LogShield: A Transformer-based APT Detection System Leveraging Self-Attention. (arXiv:2311.05733v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.05733
• Code URL: null
• Copy Paste: [[2311.05733]] LogShield: A Transformer-based APT Detection System Leveraging Self-Attention(http://arxiv.org/abs/2311.05733)
• Summary:

  Cyber attacks are often identified using system and network logs. There have been significant prior works that utilize provenance graphs and ML techniques to detect attacks, specifically advanced persistent threats, which are very difficult to detect. Lately, there have been studies where transformer-based language models are being used to detect various types of attacks from system logs. However, no such attempts have been made in the case of APTs. In addition, existing state-of-the-art techniques that use system provenance graphs, lack a data processing framework generalized across datasets for optimal performance. For mitigating this limitation as well as exploring the effectiveness of transformer-based language models, this paper proposes LogShield, a framework designed to detect APT attack patterns leveraging the power of self-attention in transformers. We incorporate customized embedding layers to effectively capture the context of event sequences derived from provenance graphs. While acknowledging the computational overhead associated with training transformer networks, our framework surpasses existing LSTM and Language models regarding APT detection. We integrated the model parameters and training procedure from the RoBERTa model and conducted extensive experiments on well-known APT datasets (DARPA OpTC and DARPA TC E3). Our framework achieved superior F1 scores of 98% and 95% on the two datasets respectively, surpassing the F1 scores of 96% and 94% obtained by LSTM models. Our findings suggest that LogShield's performance benefits from larger datasets and demonstrates its potential for generalization across diverse domains. These findings contribute to the advancement of APT attack detection methods and underscore the significance of transformer-based architectures in addressing security challenges in computer systems.

Title: A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning. (arXiv:2311.05877v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05877
• Code URL: https://github.com/vcherepanova/tabular-feature-selection
• Copy Paste: [[2311.05877]] A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning(http://arxiv.org/abs/2311.05877)
• Summary:

  Academic tabular benchmarks often contain small sets of curated features. In contrast, data scientists typically collect as many features as possible into their datasets, and even engineer new features from existing ones. To prevent overfitting in subsequent downstream modeling, practitioners commonly use automated feature selection methods that identify a reduced subset of informative features. Existing benchmarks for tabular feature selection consider classical downstream models, toy synthetic datasets, or do not evaluate feature selectors on the basis of downstream performance. Motivated by the increasing popularity of tabular deep learning, we construct a challenging feature selection benchmark evaluated on downstream neural networks including transformers, using real datasets and multiple methods for generating extraneous features. We also propose an input-gradient-based analogue of Lasso for neural networks that outperforms classical feature selection methods on challenging problems such as selecting from corrupted or second-order features.

Title: FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores. (arXiv:2311.05908v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05908
• Code URL: null
• Copy Paste: [[2311.05908]] FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores(http://arxiv.org/abs/2311.05908)
• Summary:

  Convolution models with long filters have demonstrated state-of-the-art reasoning abilities in many long-sequence tasks but lag behind the most optimized Transformers in wall-clock time. A major bottleneck is the Fast Fourier Transform (FFT)--which allows long convolutions to run in $O(N logN)$ time in sequence length $N$ but has poor hardware utilization. In this paper, we study how to optimize the FFT convolution. We find two key bottlenecks: the FFT does not effectively use specialized matrix multiply units, and it incurs expensive I/O between layers of the memory hierarchy. In response, we propose FlashFFTConv. FlashFFTConv uses a matrix decomposition that computes the FFT using matrix multiply units and enables kernel fusion for long sequences, reducing I/O. We also present two sparse convolution algorithms--1) partial convolutions and 2) frequency-sparse convolutions--which can be implemented simply by skipping blocks in the matrix decomposition, enabling further opportunities for memory and compute savings. FlashFFTConv speeds up exact FFT convolutions by up to 7.93$\times$ over PyTorch and achieves up to 4.4$\times$ speedup end-to-end. Given the same compute budget, FlashFFTConv allows Hyena-GPT-s to achieve 2.3 points better perplexity on the PILE and M2-BERT-base to achieve 3.3 points higher GLUE score--matching models with twice the parameter count. FlashFFTConv also achieves 96.1% accuracy on Path-512, a high-resolution vision task where no model had previously achieved better than 50%. Furthermore, partial convolutions enable longer-sequence models--yielding the first DNA model that can process the longest human genes (2.3M base pairs)--and frequency-sparse convolutions speed up pretrained models while maintaining or improving model quality.

Title: Frequency-domain MLPs are More Effective Learners in Time Series Forecasting. (arXiv:2311.06184v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.06184
• Code URL: null
• Copy Paste: [[2311.06184]] Frequency-domain MLPs are More Effective Learners in Time Series Forecasting(http://arxiv.org/abs/2311.06184)
• Summary:

  Time series forecasting has played the key role in different industrial, including finance, traffic, energy, and healthcare domains. While existing literatures have designed many sophisticated architectures based on RNNs, GNNs, or Transformers, another kind of approaches based on multi-layer perceptrons (MLPs) are proposed with simple structure, low complexity, and {superior performance}. However, most MLP-based forecasting methods suffer from the point-wise mappings and information bottleneck, which largely hinders the forecasting performance. To overcome this problem, we explore a novel direction of applying MLPs in the frequency domain for time series forecasting. We investigate the learned patterns of frequency-domain MLPs and discover their two inherent characteristic benefiting forecasting, (i) global view: frequency spectrum makes MLPs own a complete view for signals and learn global dependencies more easily, and (ii) energy compaction: frequency-domain MLPs concentrate on smaller key part of frequency components with compact signal energy. Then, we propose FreTS, a simple yet effective architecture built upon Frequency-domain MLPs for Time Series forecasting. FreTS mainly involves two stages, (i) Domain Conversion, that transforms time-domain signals into complex numbers of frequency domain; (ii) Frequency Learning, that performs our redesigned MLPs for the learning of real and imaginary part of frequency components. The above stages operated on both inter-series and intra-series scales further contribute to channel-wise and time-wise dependency learning. Extensive experiments on 13 real-world benchmarks (including 7 benchmarks for short-term forecasting and 6 benchmarks for long-term forecasting) demonstrate our consistent superiority over state-of-the-art methods.

generative

Title: Enhancing Rock Image Segmentation in Digital Rock Physics: A Fusion of Generative AI and State-of-the-Art Neural Networks. (arXiv:2311.06079v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06079
• Code URL: null
• Copy Paste: [[2311.06079]] Enhancing Rock Image Segmentation in Digital Rock Physics: A Fusion of Generative AI and State-of-the-Art Neural Networks(http://arxiv.org/abs/2311.06079)
• Summary:

  In digital rock physics, analysing microstructures from CT and SEM scans is crucial for estimating properties like porosity and pore connectivity. Traditional segmentation methods like thresholding and CNNs often fall short in accurately detailing rock microstructures and are prone to noise. U-Net improved segmentation accuracy but required many expert-annotated samples, a laborious and error-prone process due to complex pore shapes. Our study employed an advanced generative AI model, the diffusion model, to overcome these limitations. This model generated a vast dataset of CT/SEM and binary segmentation pairs from a small initial dataset. We assessed the efficacy of three neural networks: U-Net, Attention-U-net, and TransUNet, for segmenting these enhanced images. The diffusion model proved to be an effective data augmentation technique, improving the generalization and robustness of deep learning models. TransU-Net, incorporating Transformer structures, demonstrated superior segmentation accuracy and IoU metrics, outperforming both U-Net and Attention-U-net. Our research advances rock image segmentation by combining the diffusion model with cutting-edge neural networks, reducing dependency on extensive expert data and boosting segmentation accuracy and robustness. TransU-Net sets a new standard in digital rock physics, paving the way for future geoscience and engineering breakthroughs.

Title: FinGPT: Large Generative Models for a Small Language. (arXiv:2311.05640v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05640
• Code URL: null
• Copy Paste: [[2311.05640]] FinGPT: Large Generative Models for a Small Language(http://arxiv.org/abs/2311.05640)
• Summary:

  Large language models (LLMs) excel in many tasks in NLP and beyond, but most open models have very limited coverage of smaller languages and LLM work tends to focus on languages where nearly unlimited data is available for pretraining. In this work, we study the challenges of creating LLMs for Finnish, a language spoken by less than 0.1% of the world population. We compile an extensive dataset of Finnish combining web crawls, news, social media and eBooks. We pursue two approaches to pretrain models: 1) we train seven monolingual models from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the pretraining of the multilingual BLOOM model on a mix of its original training data and Finnish, resulting in a 176 billion parameter model we call BLUUMI. For model evaluation, we introduce FIN-bench, a version of BIG-bench with Finnish tasks. We also assess other model qualities such as toxicity and bias. Our models and tools are openly available at https://turkunlp.org/gpt3-finnish.

Title: Generative Explanations for Graph Neural Network: Methods and Evaluations. (arXiv:2311.05764v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05764
• Code URL: null
• Copy Paste: [[2311.05764]] Generative Explanations for Graph Neural Network: Methods and Evaluations(http://arxiv.org/abs/2311.05764)
• Summary:

  Graph Neural Networks (GNNs) achieve state-of-the-art performance in various graph-related tasks. However, the black-box nature often limits their interpretability and trustworthiness. Numerous explainability methods have been proposed to uncover the decision-making logic of GNNs, by generating underlying explanatory substructures. In this paper, we conduct a comprehensive review of the existing explanation methods for GNNs from the perspective of graph generation. Specifically, we propose a unified optimization objective for generative explanation methods, comprising two sub-objectives: Attribution and Information constraints. We further demonstrate their specific manifestations in various generative model architectures and different explanation scenarios. With the unified objective of the explanation problem, we reveal the shared characteristics and distinctions among current methods, laying the foundation for future methodological advancements. Empirical results demonstrate the advantages and limitations of different explainability approaches in terms of explanation performance, efficiency, and generalizability.

Title: Testing Dependency of Unlabeled Databases. (arXiv:2311.05874v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.05874
• Code URL: null
• Copy Paste: [[2311.05874]] Testing Dependency of Unlabeled Databases(http://arxiv.org/abs/2311.05874)
• Summary:

  In this paper, we investigate the problem of deciding whether two random databases $\mathsf{X}\in\mathcal{X}^{n\times d}$ and $\mathsf{Y}\in\mathcal{Y}^{n\times d}$ are statistically dependent or not. This is formulated as a hypothesis testing problem, where under the null hypothesis, these two databases are statistically independent, while under the alternative, there exists an unknown row permutation $\sigma$, such that $\mathsf{X}$ and $\mathsf{Y}^\sigma$, a permuted version of $\mathsf{Y}$, are statistically dependent with some known joint distribution, but have the same marginal distributions as the null. We characterize the thresholds at which optimal testing is information-theoretically impossible and possible, as a function of $n$, $d$, and some spectral properties of the generative distributions of the datasets. For example, we prove that if a certain function of the eigenvalues of the likelihood function and $d$, is below a certain threshold, as $d\to\infty$, then weak detection (performing slightly better than random guessing) is statistically impossible, no matter what the value of $n$ is. This mimics the performance of an efficient test that thresholds a centered version of the log-likelihood function of the observed matrices. We also analyze the case where $d$ is fixed, for which we derive strong (vanishing error) and weak detection lower and upper bounds.

large language model

Title: Prompt Engineering a Prompt Engineer. (arXiv:2311.05661v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05661
• Code URL: null
• Copy Paste: [[2311.05661]] Prompt Engineering a Prompt Engineer(http://arxiv.org/abs/2311.05661)
• Summary:

  Prompt engineering is a challenging yet crucial task for optimizing the performance of large language models (LLMs). It requires complex reasoning to examine the model's errors, hypothesize what is missing or misleading in the current prompt, and communicate the task with clarity. While recent works indicate that LLMs can be meta-prompted to perform automatic prompt engineering, their potentials may not be fully untapped due to the lack of sufficient guidance to elicit complex reasoning capabilities in LLMs in the meta-prompt. In this work, we investigate the problem of "prompt engineering a prompt engineer" -- constructing a meta-prompt that more effectively guides LLMs to perform automatic prompt engineering. We introduce and analyze key components, such as a step-by-step reasoning template and context specification, which lead to improved performance. In addition, inspired by common optimization concepts such as batch size, step size and momentum, we introduce their verbalized counterparts to the meta-prompt and investigate their effects. Our final method, named PE2, finds a prompt that outperforms "let's think step by step" by 6.3% on the MultiArith dataset and 3.1% on the GSM8K dataset. To demonstrate its versatility, we apply PE2 to the Instruction Induction benchmark, a suite of counterfactual tasks, and a lengthy, real-world industrial prompt. In these settings, PE2 achieves strong performance and outperforms prior automatic prompt engineering baselines. Further, we show that PE2 makes meaningful and targeted prompt edits, amends erroneous or incomplete prompts, and presents non-trivial counterfactual reasoning abilities.

Title: Long-Horizon Dialogue Understanding for Role Identification in the Game of Avalon with Large Language Models. (arXiv:2311.05720v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05720
• Code URL: null
• Copy Paste: [[2311.05720]] Long-Horizon Dialogue Understanding for Role Identification in the Game of Avalon with Large Language Models(http://arxiv.org/abs/2311.05720)
• Summary:

  Deception and persuasion play a critical role in long-horizon dialogues between multiple parties, especially when the interests, goals, and motivations of the participants are not aligned. Such complex tasks pose challenges for current Large Language Models (LLM) as deception and persuasion can easily mislead them, especially in long-horizon multi-party dialogues. To this end, we explore the game of Avalon: The Resistance, a social deduction game in which players must determine each other's hidden identities to complete their team's objective. We introduce an online testbed and a dataset containing 20 carefully collected and labeled games among human players that exhibit long-horizon deception in a cooperative-competitive setting. We discuss the capabilities of LLMs to utilize deceptive long-horizon conversations between six human players to determine each player's goal and motivation. Particularly, we discuss the multimodal integration of the chat between the players and the game's state that grounds the conversation, providing further insights into the true player identities. We find that even current state-of-the-art LLMs do not reach human performance, making our dataset a compelling benchmark to investigate the decision-making and language-processing capabilities of LLMs. Our dataset and online testbed can be found at our project website: https://sstepput.github.io/Avalon-NLU/

Title: Efficiently Adapting Pretrained Language Models To New Languages. (arXiv:2311.05741v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05741
• Code URL: null
• Copy Paste: [[2311.05741]] Efficiently Adapting Pretrained Language Models To New Languages(http://arxiv.org/abs/2311.05741)
• Summary:

  Recent large language models (LLM) exhibit sub-optimal performance on low-resource languages, as the training data of these models is usually dominated by English and other high-resource languages. Furthermore, it is challenging to train models for low-resource languages, especially from scratch, due to a lack of high quality training data. Adapting pretrained LLMs reduces the need for data in the new language while also providing cross lingual transfer capabilities. However, naively adapting to new languages leads to catastrophic forgetting and poor tokenizer efficiency. In this work, we study how to efficiently adapt any existing pretrained LLM to a new language without running into these issues. In particular, we improve the encoding efficiency of the tokenizer by adding new tokens from the target language and study the data mixing recipe to mitigate forgetting. Our experiments on adapting an English LLM to Hungarian and Thai show that our recipe can reach better performance than open source models on the target language, with minimal regressions on English.

Title: CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model. (arXiv:2311.05812v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05812
• Code URL: https://github.com/tongjifinlab/cfbenchmark
• Copy Paste: [[2311.05812]] CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model(http://arxiv.org/abs/2311.05812)
• Summary:

  Large language models (LLMs) have demonstrated great potential in the financial domain. Thus, it becomes important to assess the performance of LLMs in the financial tasks. In this work, we introduce CFBenchmark, to evaluate the performance of LLMs for Chinese financial assistant. The basic version of CFBenchmark is designed to evaluate the basic ability in Chinese financial text processing from three aspects~(\emph{i.e.} recognition, classification, and generation) including eight tasks, and includes financial texts ranging in length from 50 to over 1,800 characters. We conduct experiments on several LLMs available in the literature with CFBenchmark-Basic, and the experimental results indicate that while some LLMs show outstanding performance in specific tasks, overall, there is still significant room for improvement in basic tasks of financial text processing with existing models. In the future, we plan to explore the advanced version of CFBenchmark, aiming to further explore the extensive capabilities of language models in more profound dimensions as a financial assistant in Chinese. Our codes are released at https://github.com/TongjiFinLab/CFBenchmark.

Title: Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications. (arXiv:2311.05876v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05876
• Code URL: null
• Copy Paste: [[2311.05876]] Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications(http://arxiv.org/abs/2311.05876)
• Summary:

  Large language models (LLMs) exhibit superior performance on various natural language tasks, but they are susceptible to issues stemming from outdated data and domain-specific limitations. In order to address these challenges, researchers have pursued two primary strategies, knowledge editing and retrieval augmentation, to enhance LLMs by incorporating external information from different aspects. Nevertheless, there is still a notable absence of a comprehensive survey. In this paper, we propose a review to discuss the trends in integration of knowledge and large language models, including taxonomy of methods, benchmarks, and applications. In addition, we conduct an in-depth analysis of different methods and point out potential research directions in the future. We hope this survey offers the community quick access and a comprehensive overview of this research area, with the intention of inspiring future research endeavors.

Title: Large Language Models are Zero Shot Hypothesis Proposers. (arXiv:2311.05965v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.05965
• Code URL: null
• Copy Paste: [[2311.05965]] Large Language Models are Zero Shot Hypothesis Proposers(http://arxiv.org/abs/2311.05965)
• Summary:

  Significant scientific discoveries have driven the progress of human civilisation. The explosion of scientific literature and data has created information barriers across disciplines that have slowed the pace of scientific discovery. Large Language Models (LLMs) hold a wealth of global and interdisciplinary knowledge that promises to break down these information barriers and foster a new wave of scientific discovery. However, the potential of LLMs for scientific discovery has not been formally explored. In this paper, we start from investigating whether LLMs can propose scientific hypotheses. To this end, we construct a dataset consist of background knowledge and hypothesis pairs from biomedical literature. The dataset is divided into training, seen, and unseen test sets based on the publication date to control visibility. We subsequently evaluate the hypothesis generation capabilities of various top-tier instructed models in zero-shot, few-shot, and fine-tuning settings, including both closed and open-source LLMs. Additionally, we introduce an LLM-based multi-agent cooperative framework with different role designs and external tools to enhance the capabilities related to generating hypotheses. We also design four metrics through a comprehensive review to evaluate the generated hypotheses for both ChatGPT-based and human evaluations. Through experiments and analyses, we arrive at the following findings: 1) LLMs surprisingly generate untrained yet validated hypotheses from testing literature. 2) Increasing uncertainty facilitates candidate generation, potentially enhancing zero-shot hypothesis generation capabilities. These findings strongly support the potential of LLMs as catalysts for new scientific discoveries and guide further exploration.

Title: ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences. (arXiv:2311.06025v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.06025
• Code URL: https://github.com/synlp/chimed-gpt
• Copy Paste: [[2311.06025]] ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences(http://arxiv.org/abs/2311.06025)
• Summary:

  Recently, the increasing demand for superior medical services has highlighted the discrepancies in the medical infrastructure. With big data, especially texts, forming the foundation of medical services, there is an exigent need for effective natural language processing (NLP) solutions tailored to the healthcare domain. Conventional approaches leveraging pre-trained models present promising results in this domain and current large language models (LLMs) offer advanced foundation for medical text processing. However, most medical LLMs are trained only with supervised fine-tuning (SFT), even though it efficiently empowers LLMs to understand and respond to medical instructions but is ineffective in learning domain knowledge and aligning with human preference. Another engineering barrier that prevents current medical LLM from better text processing ability is their restricted context length (e.g., 2,048 tokens), making it hard for the LLMs to process long context, which is frequently required in the medical domain. In this work, we propose ChiMed-GPT, a new benchmark LLM designed explicitly for Chinese medical domain, with enlarged context length to 4,096 tokens and undergoes a comprehensive training regime with pre-training, SFT, and RLHF. Evaluations on real-world tasks including information extraction, question answering, and dialogue generation demonstrate ChiMed-GPT's superior performance over general domain LLMs. Furthermore, we analyze possible biases through prompting ChiMed-GPT to perform attitude scales regarding discrimination of patients, so as to contribute to further responsible development of LLMs in the medical domain. The code and model are released at https://github.com/synlp/ChiMed-GPT.

Title: Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking. (arXiv:2311.06102v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.06102
• Code URL: null
• Copy Paste: [[2311.06102]] Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking(http://arxiv.org/abs/2311.06102)
• Summary:

  Standard Full-Data classifiers in NLP demand thousands of labeled examples, which is impractical in data-limited domains. Few-shot methods offer an alternative, utilizing contrastive learning techniques that can be effective with as little as 20 examples per class. Similarly, Large Language Models (LLMs) like GPT-4 can perform effectively with just 1-5 examples per class. However, the performance-cost trade-offs of these methods remain underexplored, a critical concern for budget-limited organizations. Our work addresses this gap by studying the aforementioned approaches over the Banking77 financial intent detection dataset, including the evaluation of cutting-edge LLMs by OpenAI, Cohere, and Anthropic in a comprehensive set of few-shot scenarios. We complete the picture with two additional methods: first, a cost-effective querying method for LLMs based on retrieval-augmented generation (RAG), able to reduce operational costs multiple times compared to classic few-shot approaches, and second, a data augmentation method using GPT-4, able to improve performance in data-limited scenarios. Finally, to inspire future research, we provide a human expert's curated subset of Banking77, along with extensive error analysis.

Title: Is it indeed bigger better? The comprehensive study of claim detection LMs applied for disinformation tackling. (arXiv:2311.06121v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.06121
• Code URL: null
• Copy Paste: [[2311.06121]] Is it indeed bigger better? The comprehensive study of claim detection LMs applied for disinformation tackling(http://arxiv.org/abs/2311.06121)
• Summary:

  This study compares the performance of (1) fine-tuned models and (2) extremely large language models on the task of check-worthy claim detection. For the purpose of the comparison we composed a multilingual and multi-topical dataset comprising texts of various sources and styles. Building on this, we performed a benchmark analysis to determine the most general multilingual and multi-topical claim detector.

  We chose three state-of-the-art models in the check-worthy claim detection task and fine-tuned them. Furthermore, we selected three state-of-the-art extremely large language models without any fine-tuning. We made modifications to the models to adapt them for multilingual settings and through extensive experimentation and evaluation. We assessed the performance of all the models in terms of accuracy, recall, and F1-score in in-domain and cross-domain scenarios. Our results demonstrate that despite the technological progress in the area of natural language processing, the models fine-tuned for the task of check-worthy claim detection still outperform the zero-shot approaches in a cross-domain settings.

Title: Language Models can be Logical Solvers. (arXiv:2311.06158v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.06158
• Code URL: null
• Copy Paste: [[2311.06158]] Language Models can be Logical Solvers(http://arxiv.org/abs/2311.06158)
• Summary:

  Logical reasoning is a fundamental aspect of human intelligence and a key component of tasks like problem-solving and decision-making. Recent advancements have enabled Large Language Models (LLMs) to potentially exhibit reasoning capabilities, but complex logical reasoning remains a challenge. The state-of-the-art, solver-augmented language models, use LLMs to parse natural language logical questions into symbolic representations first and then adopt external logical solvers to take in the symbolic representations and output the answers. Despite their impressive performance, any parsing errors will inevitably result in the failure of the execution of the external logical solver and no answer to the logical questions. In this paper, we introduce LoGiPT, a novel language model that directly emulates the reasoning processes of logical solvers and bypasses the parsing errors by learning to strict adherence to solver syntax and grammar. LoGiPT is fine-tuned on a newly constructed instruction-tuning dataset derived from revealing and refining the invisible reasoning process of deductive solvers. Experimental results on two public deductive reasoning datasets demonstrate that LoGiPT outperforms state-of-the-art solver-augmented LMs and few-shot prompting methods on competitive LLMs like ChatGPT or GPT-4.

Title: Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models. (arXiv:2311.06233v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.06233
• Code URL: null
• Copy Paste: [[2311.06233]] Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models(http://arxiv.org/abs/2311.06233)
• Summary:

  We propose the Data Contamination Quiz, a simple and effective approach to detect data contamination in large language models (LLMs) and estimate the amount of it. Specifically, we frame data contamination detection as a series of multiple-choice questions. We devise a quiz format wherein three perturbed versions of each dataset instance are created. These changes only include word-level perturbations, replacing words with their contextual synonyms, ensuring both the semantic and sentence structure remain exactly the same as the original instance. Together with the original instance, these perturbed versions constitute the choices in the quiz. Given that the only distinguishing signal among these choices is the exact wording, an LLM, when tasked with identifying the original instance from the choices, opts for the original if it has memorized it in its pre-training phase--a trait intrinsic to LLMs. A dataset partition is then marked as contaminated if the LLM's performance on the quiz surpasses what random chance suggests. Our evaluation spans seven datasets and their respective splits (train and test/validation) on two state-of-the-art LLMs: GPT-4 and GPT-3.5. While lacking access to the pre-training data, our results suggest that our approach not only enhances the detection of data contamination but also provides an accurate estimation of its extent, even when the contamination signal is weak.

segmentation

Title: Efficient Segmentation with Texture in Ore Images Based on Box-supervised Approach. (arXiv:2311.05929v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.05929
• Code URL: https://github.com/mvme-hbut/oreinst
• Copy Paste: [[2311.05929]] Efficient Segmentation with Texture in Ore Images Based on Box-supervised Approach(http://arxiv.org/abs/2311.05929)
• Summary:

  Image segmentation methods have been utilized to determine the particle size distribution of crushed ores. Due to the complex working environment, high-powered computing equipment is difficult to deploy. At the same time, the ore distribution is stacked, and it is difficult to identify the complete features. To address this issue, an effective box-supervised technique with texture features is provided for ore image segmentation that can identify complete and independent ores. Firstly, a ghost feature pyramid network (Ghost-FPN) is proposed to process the features obtained from the backbone to reduce redundant semantic information and computation generated by complex networks. Then, an optimized detection head is proposed to obtain the feature to maintain accuracy. Finally, Lab color space (Lab) and local binary patterns (LBP) texture features are combined to form a fusion feature similarity-based loss function to improve accuracy while incurring no loss. Experiments on MS COCO have shown that the proposed fusion features are also worth studying on other types of datasets. Extensive experimental results demonstrate the effectiveness of the proposed method, which achieves over 50 frames per second with a small model size of 21.6 MB. Meanwhile, the method maintains a high level of accuracy compared with the state-of-the-art approaches on ore image dataset. The source code is available at \url{https://github.com/MVME-HBUT/OREINST}.

Title: U3DS$^3$: Unsupervised 3D Semantic Scene Segmentation. (arXiv:2311.06018v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06018
• Code URL: null
• Copy Paste: [[2311.06018]] U3DS$^3$: Unsupervised 3D Semantic Scene Segmentation(http://arxiv.org/abs/2311.06018)
• Summary:

  Contemporary point cloud segmentation approaches largely rely on richly annotated 3D training data. However, it is both time-consuming and challenging to obtain consistently accurate annotations for such 3D scene data. Moreover, there is still a lack of investigation into fully unsupervised scene segmentation for point clouds, especially for holistic 3D scenes. This paper presents U3DS$^3$, as a step towards completely unsupervised point cloud segmentation for any holistic 3D scenes. To achieve this, U3DS$^3$ leverages a generalized unsupervised segmentation method for both object and background across both indoor and outdoor static 3D point clouds with no requirement for model pre-training, by leveraging only the inherent information of the point cloud to achieve full 3D scene segmentation. The initial step of our proposed approach involves generating superpoints based on the geometric characteristics of each scene. Subsequently, it undergoes a learning process through a spatial clustering-based methodology, followed by iterative training using pseudo-labels generated in accordance with the cluster centroids. Moreover, by leveraging the invariance and equivariance of the volumetric representations, we apply the geometric transformation on voxelized features to provide two sets of descriptors for robust representation learning. Finally, our evaluation provides state-of-the-art results on the ScanNet and SemanticKITTI, and competitive results on the S3DIS, benchmark datasets.

Title: Diagonal Hierarchical Consistency Learning for Semi-supervised Medical Image Segmentation. (arXiv:2311.06031v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06031
• Code URL: null
• Copy Paste: [[2311.06031]] Diagonal Hierarchical Consistency Learning for Semi-supervised Medical Image Segmentation(http://arxiv.org/abs/2311.06031)
• Summary:

  Medical image segmentation, which is essential for many clinical applications, has achieved almost human-level performance via data-driven deep learning techniques. Nevertheless, its performance is predicated on the costly process of manually annotating a large amount of medical images. To this end, we propose a novel framework for robust semi-supervised medical image segmentation using diagonal hierarchical consistency (DiHC-Net). First, it is composed of multiple sub-models with identical multi-scale architecture but with distinct sub-layers, such as up-sampling and normalisation layers. Second, a novel diagonal hierarchical consistency is enforced between one model's intermediate and final prediction and other models' soft pseudo labels in a diagonal hierarchical fashion. Experimental results verify the efficacy of our simple framework, outperforming all previous approaches on public Left Atrium (LA) dataset.

Title: Lidar-based Norwegian tree species detection using deep learning. (arXiv:2311.06066v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06066
• Code URL: null
• Copy Paste: [[2311.06066]] Lidar-based Norwegian tree species detection using deep learning(http://arxiv.org/abs/2311.06066)
• Summary:

  Background: The mapping of tree species within Norwegian forests is a time-consuming process, involving forest associations relying on manual labeling by experts. The process can involve both aerial imagery, personal familiarity, or on-scene references, and remote sensing data. The state-of-the-art methods usually use high resolution aerial imagery with semantic segmentation methods. Methods: We present a deep learning based tree species classification model utilizing only lidar (Light Detection And Ranging) data. The lidar images are segmented into four classes (Norway Spruce, Scots Pine, Birch, background) with a U-Net based network. The model is trained with focal loss over partial weak labels. A major benefit of the approach is that both the lidar imagery and the base map for the labels have free and open access. Results: Our tree species classification model achieves a macro-averaged F1 score of 0.70 on an independent validation with National Forest Inventory (NFI) in-situ sample plots. That is close to, but below the performance of aerial, or aerial and lidar combined models.

Title: ASSIST: Interactive Scene Nodes for Scalable and Realistic Indoor Simulation. (arXiv:2311.06211v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06211
• Code URL: null
• Copy Paste: [[2311.06211]] ASSIST: Interactive Scene Nodes for Scalable and Realistic Indoor Simulation(http://arxiv.org/abs/2311.06211)
• Summary:

  We present ASSIST, an object-wise neural radiance field as a panoptic representation for compositional and realistic simulation. Central to our approach is a novel scene node data structure that stores the information of each object in a unified fashion, allowing online interaction in both intra- and cross-scene settings. By incorporating a differentiable neural network along with the associated bounding box and semantic features, the proposed structure guarantees user-friendly interaction on independent objects to scale up novel view simulation. Objects in the scene can be queried, added, duplicated, deleted, transformed, or swapped simply through mouse/keyboard controls or language instructions. Experiments demonstrate the efficacy of the proposed method, where scaled realistic simulation can be achieved through interactive editing and compositional rendering, with color images, depth images, and panoptic segmentation masks generated in a 3D consistent manner.

Title: Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks. (arXiv:2311.06242v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.06242
• Code URL: null
• Copy Paste: [[2311.06242]] Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks(http://arxiv.org/abs/2311.06242)
• Summary:

  We introduce Florence-2, a novel vision foundation model with a unified, prompt-based representation for a variety of computer vision and vision-language tasks. While existing large vision models excel in transfer learning, they struggle to perform a diversity of tasks with simple instructions, a capability that implies handling the complexity of various spatial hierarchy and semantic granularity. Florence-2 was designed to take text-prompt as task instructions and generate desirable results in text forms, whether it be captioning, object detection, grounding or segmentation. This multi-task learning setup demands large-scale, high-quality annotated data. To this end, we co-developed FLD-5B that consists of 5.4 billion comprehensive visual annotations on 126 million images, using an iterative strategy of automated image annotation and model refinement. We adopted a sequence-to-sequence structure to train Florence-2 to perform versatile and comprehensive vision tasks. Extensive evaluations on numerous tasks demonstrated Florence-2 to be a strong vision foundation model contender with unprecedented zero-shot and fine-tuning capabilities.