secure

Title: Okapi: A Lightweight Architecture for Secure Speculation Exploiting Locality of Memory Accesses. (arXiv:2312.08156v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.08156
• Code URL: null
• Copy Paste: [[2312.08156]] Okapi: A Lightweight Architecture for Secure Speculation Exploiting Locality of Memory Accesses(http://arxiv.org/abs/2312.08156)
• Summary:

  This paper introduces Okapi, an innovative hardware/software cross-layer architecture designed to mitigate Transient Execution Side Channel (TES) attacks, including Spectre variants, in modern computing systems. A key contribution of Okapi is a set of security features building upon each other to offer various trade-offs between performance and security. At its core, Okapi allows for speculative data accesses if the targeted memory region has already been accessed non-speculatively before in the same trust domain. It delays first-time accesses until the speculation is resolved.

  Okapi stands out for its flexibility in security implementation. For environments with less stringent security needs, Okapi's features can be deactivated to eliminate performance overhead. When activated, the hardware modifications alone provide robust protection against transient execution attacks at a thread-level granularity, including all universal read gadgets like Spectre-PHT and Spectre-BTB. This incurs an average performance overhead of only 3.6 % for the SPEC CPU2017 benchmark suite.

  On top, Okapi introduces the OkapiReset instruction for additional software-level security support. This instruction, which can be manually inserted by developers or automatically via a compiler extension, allows for fully secure speculation and for trust domain sizes smaller than a thread. While the manual insertion of OkapiReset incurs an additional 0.6 % performance overhead, the automated compiler extension approach results in a 23.1 % overhead for making a cryptographic library fully secure. With an approximate 0.4 % hardware overhead, Okapi provides a highly scalable and adaptable solution for secure speculation in state-of-the-art processor design.

Title: Secure Deep Reinforcement Learning for Dynamic Resource Allocation in Wireless MEC Networks. (arXiv:2312.08016v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.08016
• Code URL: null
• Copy Paste: [[2312.08016]] Secure Deep Reinforcement Learning for Dynamic Resource Allocation in Wireless MEC Networks(http://arxiv.org/abs/2312.08016)
• Summary:

  This paper proposes a blockchain-secured deep reinforcement learning (BC-DRL) optimization framework for {data management and} resource allocation in decentralized {wireless mobile edge computing (MEC)} networks. In our framework, {we design a low-latency reputation-based proof-of-stake (RPoS) consensus protocol to select highly reliable blockchain-enabled BSs to securely store MEC user requests and prevent data tampering attacks.} {We formulate the MEC resource allocation optimization as a constrained Markov decision process that balances minimum processing latency and denial-of-service (DoS) probability}. {We use the MEC aggregated features as the DRL input to significantly reduce the high-dimensionality input of the remaining service processing time for individual MEC requests. Our designed constrained DRL effectively attains the optimal resource allocations that are adapted to the dynamic DoS requirements. We provide extensive simulation results and analysis to} validate that our BC-DRL framework achieves higher security, reliability, and resource utilization efficiency than benchmark blockchain consensus protocols and {MEC} resource allocation algorithms.

security

Title: On the Prediction of Hardware Security Properties of HLS Designs Using Graph Neural Networks. (arXiv:2312.07594v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.07594
• Code URL: null
• Copy Paste: [[2312.07594]] On the Prediction of Hardware Security Properties of HLS Designs Using Graph Neural Networks(http://arxiv.org/abs/2312.07594)
• Summary:

  High-level synthesis (HLS) tools have provided significant productivity enhancements to the design flow of digital systems in recent years, resulting in highly-optimized circuits, in terms of area and latency. Given the evolution of hardware attacks, which can render them vulnerable, it is essential to consider security as a significant aspect of the HLS design flow. Yet the need to evaluate a huge number of functionally equivalent de-signs of the HLS design space challenges hardware security evaluation methods (e.g., fault injection - FI campaigns). In this work, we propose an evaluation methodology of hardware security properties of HLS-produced designs using state-of-the-art Graph Neural Network (GNN) approaches that achieves significant speedup and better scalability than typical evaluation methods (such as FI). We demonstrate the proposed methodology on a Double Modular Redundancy (DMR) coun-termeasure applied on an AES SBox implementation, en-hanced by diversifying the redundant modules through HLS directives. The experimental results show that GNNs can be efficiently trained to predict important hardware security met-rics concerning fault attacks (e.g., critical and detection error rates), by using regression. The proposed method predicts the fault vulnerability metrics of the HLS-based designs with high R-squared scores and achieves huge speedup compared to fault injection once the training of the GNN is completed.

Title: Securing Graph Neural Networks in MLaaS: A Comprehensive Realization of Query-based Integrity Verification. (arXiv:2312.07870v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.07870
• Code URL: null
• Copy Paste: [[2312.07870]] Securing Graph Neural Networks in MLaaS: A Comprehensive Realization of Query-based Integrity Verification(http://arxiv.org/abs/2312.07870)
• Summary:

  The deployment of Graph Neural Networks (GNNs) within Machine Learning as a Service (MLaaS) has opened up new attack surfaces and an escalation in security concerns regarding model-centric attacks. These attacks can directly manipulate the GNN model parameters during serving, causing incorrect predictions and posing substantial threats to essential GNN applications. Traditional integrity verification methods falter in this context due to the limitations imposed by MLaaS and the distinct characteristics of GNN models.

  In this research, we introduce a groundbreaking approach to protect GNN models in MLaaS from model-centric attacks. Our approach includes a comprehensive verification schema for GNN's integrity, taking into account both transductive and inductive GNNs, and accommodating varying pre-deployment knowledge of the models. We propose a query-based verification technique, fortified with innovative node fingerprint generation algorithms. To deal with advanced attackers who know our mechanisms in advance, we introduce randomized fingerprint nodes within our design. The experimental evaluation demonstrates that our method can detect five representative adversarial model-centric attacks, displaying 2 to 4 times greater efficiency compared to baselines.

Title: Segment-Based Formal Verification of WiFi Fragmentation and Power Save Mode. (arXiv:2312.07877v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.07877
• Code URL: null
• Copy Paste: [[2312.07877]] Segment-Based Formal Verification of WiFi Fragmentation and Power Save Mode(http://arxiv.org/abs/2312.07877)
• Summary:

  The IEEE 802.11 family of standards, better known as WiFi, is a widely used protocol utilized by billions of users. Previous works on WiFi formal verification have mostly focused on the four-way handshake and other security aspects. However, recent works have uncovered severe vulnerabilities in functional aspects of WiFi, which can cause information leakage for billions of devices. No formal analysis method exists able to reason on the functional aspects of the WiFi protocol. In this paper, we take the first steps in addressing this gap and present an extensive formal analysis of the functional aspects of the WiFi protocol, more specifically, the fragmentation and the power-save-mode process. To achieve this, we design a novel segment-based formal verification process and introduce a practical threat model (i.e. MAC spoofing) in Tamarin to reason about the various capabilities of the attacker. To this end, we verify 68 properties extracted from WiFi protocol specification, find 3 vulnerabilities from the verification, verify 3 known attacks, and discover 2 new issues. These vulnerabilities and issues affect 14 commercial devices out of 17 tested cases, showing the prevalence and impact of the issues. Apart from this, we show that the proposed countermeasures indeed are sufficient to address the issues. We hope our results and analysis will help vendors adopt the countermeasures and motivate further research into the verification of the functional aspects of the WiFi protocol.

Title: Ensuring End-to-End Security with Fine-grained Access Control for Connected and Autonomous Vehicles. (arXiv:2312.07898v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.07898
• Code URL: null
• Copy Paste: [[2312.07898]] Ensuring End-to-End Security with Fine-grained Access Control for Connected and Autonomous Vehicles(http://arxiv.org/abs/2312.07898)
• Summary:

  As advanced V2X applications emerge in the connected and autonomous vehicle (CAV), the data communications between in-vehicle end-devices and outside nodes increase, which make the end-to-end (E2E) security to in-vehicle end-devices as the urgent issue to be handled. However, the E2E security with fine-grained access control still remains as a challenging issue for resource-constrained end-devices since the existing security solutions require complicated key management and high resource consumption. Therefore, we propose a practical and secure vehicular communication protocol for the E2E security based on a new attribute-based encryption (ABE) scheme. In our scheme, the outsourced computation is provided for encryption, and the computation cost for decryption constantly remains small, regardless of the number of attributes. The policy privacy can be ensured by the proposed ABE to support privacy-sensitive V2X applications, and the existing identity-based signature for outsourced signing is newly reconstructed. Our scheme achieves the confidentiality, message authentication, identity anonymity, unlinkability, traceability, and reconfigurable outsourced computation, and we also show the practical feasibility of our protocol via the performance evaluation.

Title: BinGo: Identifying Security Patches in Binary Code with Graph Representation Learning. (arXiv:2312.07921v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.07921
• Code URL: null
• Copy Paste: [[2312.07921]] BinGo: Identifying Security Patches in Binary Code with Graph Representation Learning(http://arxiv.org/abs/2312.07921)
• Summary:

  A timely software update is vital to combat the increasing security vulnerabilities. However, some software vendors may secretly patch their vulnerabilities without creating CVE entries or even describing the security issue in their change log. Thus, it is critical to identify these hidden security patches and defeat potential N-day attacks. Researchers have employed various machine learning techniques to identify security patches in open-source software, leveraging the syntax and semantic features of the software changes and commit messages. However, all these solutions cannot be directly applied to the binary code, whose instructions and program flow may dramatically vary due to different compilation configurations. In this paper, we propose BinGo, a new security patch detection system for binary code. The main idea is to present the binary code as code property graphs to enable a comprehensive understanding of program flow and perform a language model over each basic block of binary code to catch the instruction semantics. BinGo consists of four phases, namely, patch data pre-processing, graph extraction, embedding generation, and graph representation learning. Due to the lack of an existing binary security patch dataset, we construct such a dataset by compiling the pre-patch and post-patch source code of the Linux kernel. Our experimental results show BinGo can achieve up to 80.77% accuracy in identifying security patches between two neighboring versions of binary code. Moreover, BinGo can effectively reduce the false positives and false negatives caused by the different compilers and optimization levels.

Title: SoK: On the Security of Non-Fungible Tokens. (arXiv:2312.08000v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.08000
• Code URL: null
• Copy Paste: [[2312.08000]] SoK: On the Security of Non-Fungible Tokens(http://arxiv.org/abs/2312.08000)
• Summary:

  Non-fungible tokens (NFTs) drive the prosperity of the Web3 ecosystem. By November 2023, the total market value of NFT projects reached approximately 16 billion USD. Accompanying the success of NFTs are various security issues, i.e., attacks and scams are prevalent in the ecosystem. While NFTs have attracted significant attentions from both industry and academia, there is a lack of understanding of kinds of NFT security issues. The discovery, in-depth analysis, and systematic categorization of these security issues are of significant importance for the prosperous development of the NFT ecosystem. To fill the gap, we performed a systematic literature review related to NFT security, and we have identified 142 incidents from 213 security reports and 18 academic papers until October 1st, 2023. Through manual analysis of the compiled security incidents, we have classified them into 12 major categories. Then we explored potential solutions and mitigation strategies. Drawing from these analyses, we established the first NFT security reference frame. Except, we extracted the characteristics of NFT security issues, i.e., the prevalence, severity, and intractability. We have indicated the gap between industry and academy for NFT security, and provide further research directions for the community. This paper, as the first SoK of NFT security, has systematically explored the security issues within the NFT ecosystem, shedding light on their root causes, real-world attacks, and potential ways to address them. Our findings will contribute to the future research of NFT security.

Title: Provable Security for the Onion Routing and Mix Network Packet Format Sphinx. (arXiv:2312.08028v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.08028
• Code URL: null
• Copy Paste: [[2312.08028]] Provable Security for the Onion Routing and Mix Network Packet Format Sphinx(http://arxiv.org/abs/2312.08028)
• Summary:

  Onion routing and mix networks are fundamental concepts to provide users with anonymous access to the Internet. Various corresponding solutions rely on the efficient Sphinx packet format. However, flaws in Sphinx's underlying proof strategy were found recently. It is thus currently unclear which guarantees Sphinx actually provides, and, even worse, there is no suitable proof strategy available. In this paper, we restore the security foundation for all these works by building a theoretical framework for Sphinx. We discover that the previously-used DDH assumption is insufficient for a security proof and show that the Gap Diffie-Hellman (GDH) assumption is required instead. We apply it to prove that a slightly adapted version of the Sphinx packet format is secure under the GDH assumption. Ours is the first work to provide a detailed, in-depth security proof for Sphinx in this manner. Our adaptations to Sphinx are necessary, as we demonstrate with an attack on sender privacy that would be possible otherwise.

Title: Recursive Augmented Fernet (RAF) Token: Alleviating the Pain of Stolen Tokens. (arXiv:2312.08086v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.08086
• Code URL: null
• Copy Paste: [[2312.08086]] Recursive Augmented Fernet (RAF) Token: Alleviating the Pain of Stolen Tokens(http://arxiv.org/abs/2312.08086)
• Summary:

  A robust authentication and authorization mechanism is imperative in modular system development, where modularity and modular thinking are pivotal. Traditional systems often employ identity modules responsible for authentication and token issuance. Tokens, representing user credentials, offer advantages such as reduced reliance on passwords, limited lifespan, and scoped access. Despite these benefits, the "bearer token" problem persists, leaving systems vulnerable to abuse if tokens are compromised. We propose a token-based authentication mechanism addressing modular systems' critical bearer token problem. The proposed mechanism includes a novel RAF (Recursive Augmented Fernet) token, a blacklist component, and a policy enforcer component. RAF tokens are one-time-use tokens, like tickets. They carry commands, and the receiver of an RAF token can issue new tokens using the received RAF token. The blacklist component guarantees an RAF token can not be approved more than once, and the policy enforcer checks the compatibility of commands carried by an RAF token. We introduce two variations of RAF tokens: User-tied RAF, offering simplicity and compatibility, and Fully-tied RAF, providing enhanced security through service-specific secret keys. We thoroughly discuss the security guarantees, technical definitions, and construction of RAF tokens backed by game-based proofs. We demonstrate a proof of concept in the context of OpenStack, involving modifications to Keystone and creating an RAFT library. The experimental results reveal minimal overhead in typical scenarios, establishing the practicality and effectiveness of RAF. Our experiments show that the RAF mechanism beats the idea of using short-life Fernet tokens while providing much better security.

Title: Security aspects in Smart Meters: Analysis and Prevention. (arXiv:2312.08101v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.08101
• Code URL: null
• Copy Paste: [[2312.08101]] Security aspects in Smart Meters: Analysis and Prevention(http://arxiv.org/abs/2312.08101)
• Summary:

  Smart meters are of the basic elements in the so-called Smart Grid. These devices, connected to the Internet, keep bidirectional communication with other devices in the Smart Grid structure to allow remote readings and maintenance. As any other device connected to a network, smart meters become vulnerable to attacks with different purposes, like stealing data or altering readings. Nowadays, it is becoming more and more popular to buy and plug-and-play smart meters, additionally to those installed by the energy providers, to directly monitor the energy consumption at home. This option inherently entails security risks that are under the responsibility of householders. In this paper, we focus on an open solution based on Smartpi 2.0 devices with two purposes. On the one hand, we propose a network configuration and different data flows to exchange data (energy readings) in the home. These flows are designed to support collaborative among the devices in order to prevent external attacks and attempts of corrupting the data. On the other hand, we check the vulnerability by performing two kind of attacks (denial of service and stealing and changing data by using a malware). We conclude that, as expected, these devices are vulnerable to these attacks, but we provide mechanisms to detect both of them and to solve, by applying cooperation techniques

Title: Towards Evaluating the Security of Wearable Devices in the Internet of Medical Things. (arXiv:2312.08160v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.08160
• Code URL: null
• Copy Paste: [[2312.08160]] Towards Evaluating the Security of Wearable Devices in the Internet of Medical Things(http://arxiv.org/abs/2312.08160)
• Summary:

  The Internet of Medical Things (IoMT) offers a promising solution to improve patient health and reduce human error. Wearable smart infusion pumps that accurately administer medication and integrate with electronic health records are an example of technology that can improve healthcare. They can even alert healthcare professionals or remote servers during operational failure, preventing distressing incidents. However, as the number of connected medical devices increases, the risk of cyber threats also increases. Wearable medication devices based on IoT attached to patients' bodies are prone to significant cyber threats. Being connected to the Internet exposes these devices to potential harm, which could disrupt or degrade device performance and harm patients. To ensure patient safety and well-being, it is crucial to establish secure data authentication for internet-connected medical devices. It is also important to note that the wearability option of such devices might downgrade the computational resources, making them more susceptible to security risks. This paper implements a security approach to a wearable infusion pump. We discuss practical challenges in implementing security-enabled devices and propose initial solutions to mitigate cyber threats.

privacy

Title: Classification with Partially Private Features. (arXiv:2312.07583v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07583
• Code URL: null
• Copy Paste: [[2312.07583]] Classification with Partially Private Features(http://arxiv.org/abs/2312.07583)
• Summary:

  In this paper, we consider differentially private classification when some features are sensitive, while the rest of the features and the label are not. We adapt the definition of differential privacy naturally to this setting. Our main contribution is a novel adaptation of AdaBoost that is not only provably differentially private, but also significantly outperforms a natural benchmark that assumes the entire data of the individual is sensitive in the experiments. As a surprising observation, we show that boosting randomly generated classifiers suffices to achieve high accuracy. Our approach easily adapts to the classical setting where all the features are sensitive, providing an alternate algorithm for differentially private linear classification with a much simpler privacy proof and comparable or higher accuracy than differentially private logistic regression on real-world datasets.

Title: Adaptive Differentially Quantized Subspace Perturbation (ADQSP): A Unified Framework for Privacy-Preserving Distributed Average Consensus. (arXiv:2312.07947v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.07947
• Code URL: null
• Copy Paste: [[2312.07947]] Adaptive Differentially Quantized Subspace Perturbation (ADQSP): A Unified Framework for Privacy-Preserving Distributed Average Consensus(http://arxiv.org/abs/2312.07947)
• Summary:

  Privacy-preserving distributed average consensus has received significant attention recently due to its wide applicability. Based on the achieved performances, existing approaches can be broadly classified into perfect accuracy-prioritized approaches such as secure multiparty computation (SMPC), and worst-case privacy-prioritized approaches such as differential privacy (DP). Methods of the first class achieve perfect output accuracy but reveal some private information, while methods from the second class provide privacy against the strongest adversary at the cost of a loss of accuracy. In this paper, we propose a general approach named adaptive differentially quantized subspace perturbation (ADQSP) which combines quantization schemes with so-called subspace perturbation. Although not relying on cryptographic primitives, the proposed approach enjoys the benefits of both accuracy-prioritized and privacy-prioritized methods and is able to unify them. More specifically, we show that by varying a single quantization parameter the proposed method can vary between SMPC-type performances and DP-type performances. Our results show the potential of exploiting traditional distributed signal processing tools for providing cryptographic guarantees. In addition to a comprehensive theoretical analysis, numerical validations are conducted to substantiate our results.

Title: On the privacy of federated Clustering: A Cryptographic View. (arXiv:2312.07992v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.07992
• Code URL: null
• Copy Paste: [[2312.07992]] On the privacy of federated Clustering: A Cryptographic View(http://arxiv.org/abs/2312.07992)
• Summary:

  The privacy concern in federated clustering has attracted considerable attention in past decades. Many privacy-preserving clustering algorithms leverage cryptographic techniques like homomorphic encryption or secure multiparty computation, to guarantee full privacy, i.e., no additional information is leaked other than the final output. However, given the iterative nature of clustering algorithms, consistently encrypting intermediate outputs, such as centroids, hampers efficiency. This paper delves into this intricate trade-off, questioning the necessity of continuous encryption in iterative algorithms. Using the federated K-means clustering as an example, we mathematically formulate the problem of reconstructing input private data from the intermediate centroids as a classical cryptographic problem called hidden subset sum problem (HSSP)-extended from an NP-complete problem called subset sum problem (SSP). Through an in-depth analysis, we show that existing lattice-based HSSP attacks fail in reconstructing the private data given the knowledge of intermediate centroids, thus it is secure to reveal them for the sake of efficiency. To the best of our knowledge, our work is the first to cast federated clustering's privacy concerns as a cryptographic problem HSSP such that a concrete and rigorous analysis can be conducted.

Title: Efficient Representation of the Activation Space in Deep Neural Networks. (arXiv:2312.08143v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.08143
• Code URL: null
• Copy Paste: [[2312.08143]] Efficient Representation of the Activation Space in Deep Neural Networks(http://arxiv.org/abs/2312.08143)
• Summary:

  The representations of the activation space of deep neural networks (DNNs) are widely utilized for tasks like natural language processing, anomaly detection and speech recognition. Due to the diverse nature of these tasks and the large size of DNNs, an efficient and task-independent representation of activations becomes crucial. Empirical p-values have been used to quantify the relative strength of an observed node activation compared to activations created by already-known inputs. Nonetheless, keeping raw data for these calculations increases memory resource consumption and raises privacy concerns. To this end, we propose a model-agnostic framework for creating representations of activations in DNNs using node-specific histograms to compute p-values of observed activations without retaining already-known inputs. Our proposed approach demonstrates promising potential when validated with multiple network architectures across various downstream tasks and compared with the kernel density estimates and brute-force empirical baselines. In addition, the framework reduces memory usage by 30% with up to 4 times faster p-value computing time while maintaining state of-the-art detection power in downstream tasks such as the detection of adversarial attacks and synthesized content. Moreover, as we do not persist raw data at inference time, we could potentially reduce susceptibility to attacks and privacy issues.

protect

Title: A Response to Glaze Purification via IMPRESS. (arXiv:2312.07731v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.07731
• Code URL: null
• Copy Paste: [[2312.07731]] A Response to Glaze Purification via IMPRESS(http://arxiv.org/abs/2312.07731)
• Summary:

  Recent work proposed a new mechanism to remove protective perturbation added by Glaze in order to again enable mimicry of art styles from images protected by Glaze. Despite promising results shown in the original paper, our own tests with the authors' code demonstrated several limitations of the proposed purification approach. The main limitations are 1) purification has a limited effect when tested on artists that are not well-known historical artists already embedded in original training data, 2) problems in evaluation metrics, and 3) collateral damage on mimicry result for clean images. We believe these limitations should be carefully considered in order to understand real world usability of the purification attack.

Title: Combining propensity score methods with variational autoencoders for generating synthetic data in presence of latent sub-groups. (arXiv:2312.07781v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07781
• Code URL: https://github.com/kianaf/latentsubgroups
• Copy Paste: [[2312.07781]] Combining propensity score methods with variational autoencoders for generating synthetic data in presence of latent sub-groups(http://arxiv.org/abs/2312.07781)
• Summary:

  In settings requiring synthetic data generation based on a clinical cohort, e.g., due to data protection regulations, heterogeneity across individuals might be a nuisance that we need to control or faithfully preserve. The sources of such heterogeneity might be known, e.g., as indicated by sub-groups labels, or might be unknown and thus reflected only in properties of distributions, such as bimodality or skewness. We investigate how such heterogeneity can be preserved and controlled when obtaining synthetic data from variational autoencoders (VAEs), i.e., a generative deep learning technique that utilizes a low-dimensional latent representation. To faithfully reproduce unknown heterogeneity reflected in marginal distributions, we propose to combine VAEs with pre-transformations. For dealing with known heterogeneity due to sub-groups, we complement VAEs with models for group membership, specifically from propensity score regression. The evaluation is performed with a realistic simulation design that features sub-groups and challenging marginal distributions. The proposed approach faithfully recovers the latter, compared to synthetic data approaches that focus purely on marginal distributions. Propensity scores add complementary information, e.g., when visualized in the latent space, and enable sampling of synthetic data with or without sub-group specific characteristics. We also illustrate the proposed approach with real data from an international stroke trial that exhibits considerable distribution differences between study sites, in addition to bimodality. These results indicate that describing heterogeneity by statistical approaches, such as propensity score regression, might be more generally useful for complementing generative deep learning for obtaining synthetic data that faithfully reflects structure from clinical cohorts.

defense

Title: Erasing Self-Supervised Learning Backdoor by Cluster Activation Masking. (arXiv:2312.07955v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07955
• Code URL: https://github.com/livxue/poisoncam
• Copy Paste: [[2312.07955]] Erasing Self-Supervised Learning Backdoor by Cluster Activation Masking(http://arxiv.org/abs/2312.07955)
• Summary:

  Researchers have recently found that Self-Supervised Learning (SSL) is vulnerable to backdoor attacks. The attacker can embed hidden SSL backdoors via a few poisoned examples in the training dataset and maliciously manipulate the behavior of downstream models. To defend against SSL backdoor attacks, a feasible route is to detect and remove the poisonous samples in the training set. However, the existing SSL backdoor defense method fails to detect the poisonous samples precisely. In this paper, we propose to erase the SSL backdoor by cluster activation masking and propose a novel PoisonCAM method. After obtaining the threat model trained on the poisoned dataset, our method can precisely detect poisonous samples based on the assumption that masking the backdoor trigger can effectively change the activation of a downstream clustering model. In experiments, our PoisonCAM achieves 96% accuracy for backdoor trigger detection compared to 3% of the state-of-the-art method on poisoned ImageNet-100. Moreover, our proposed PoisonCAM significantly improves the performance of the trained SSL model under backdoor attacks compared to the state-of-the-art method. Our code will be available at https://github.com/LivXue/PoisonCAM.

attack

Title: Towards Better Morphed Face Images without Ghosting Artifacts. (arXiv:2312.08111v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08111
• Code URL: null
• Copy Paste: [[2312.08111]] Towards Better Morphed Face Images without Ghosting Artifacts(http://arxiv.org/abs/2312.08111)
• Summary:

  Automatic generation of morphed face images often produces ghosting artifacts due to poorly aligned structures in the input images. Manual processing can mitigate these artifacts. However, this is not feasible for the generation of large datasets, which are required for training and evaluating robust morphing attack detectors. In this paper, we propose a method for automatic prevention of ghosting artifacts based on a pixel-wise alignment during morph generation. We evaluate our proposed method on state-of-the-art detectors and show that our morphs are harder to detect, particularly, when combined with style-transfer-based improvement of low-level image characteristics. Furthermore, we show that our approach does not impair the biometric quality, which is essential for high quality morphs.

Title: TapTree: Process-Tree Based Host Behavior Modeling and Threat Detection Framework via Sequential Pattern Mining. (arXiv:2312.07575v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.07575
• Code URL: null
• Copy Paste: [[2312.07575]] TapTree: Process-Tree Based Host Behavior Modeling and Threat Detection Framework via Sequential Pattern Mining(http://arxiv.org/abs/2312.07575)
• Summary:

  Audit logs containing system level events are frequently used for behavior modeling as they can provide detailed insight into cyber-threat occurrences. However, mapping low-level system events in audit logs to highlevel behaviors has been a major challenge in identifying host contextual behavior for the purpose of detecting potential cyber threats. Relying on domain expert knowledge may limit its practical implementation. This paper presents TapTree, an automated process-tree based technique to extract host behavior by compiling system events' semantic information. After extracting behaviors as system generated process trees, TapTree integrates event semantics as a representation of behaviors. To further reduce pattern matching workloads for the analyst, TapTree aggregates semantically equivalent patterns and optimizes representative behaviors. In our evaluation against a recent benchmark audit log dataset (DARPA OpTC), TapTree employs tree pattern queries and sequential pattern mining techniques to deduce the semantics of connected system events, achieving high accuracy for behavior abstraction and then Advanced Persistent Threat (APT) attack detection. Moreover, we illustrate how to update the baseline model gradually online, allowing it to adapt to new log patterns over time.

Title: Understanding Crypto-Ransomware. (arXiv:2312.07641v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.07641
• Code URL: null
• Copy Paste: [[2312.07641]] Understanding Crypto-Ransomware(http://arxiv.org/abs/2312.07641)
• Summary:

  Crypto-Ransomware has been increasing in sophistication since it first appeared in September 2013, leveraging new attack vectors, incorporating advanced encryption algorithms, and expanding the number of file types it targets. In this report, we dissect nearly 30 samples of ransomware variants that have been encountered since September 2013, revealing a trend of increasing sophistication.

Title: RAT: Reinforcement-Learning-Driven and Adaptive Testing for Vulnerability Discovery in Web Application Firewalls. (arXiv:2312.07885v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.07885
• Code URL: https://github.com/mhamouei/rat
• Copy Paste: [[2312.07885]] RAT: Reinforcement-Learning-Driven and Adaptive Testing for Vulnerability Discovery in Web Application Firewalls(http://arxiv.org/abs/2312.07885)
• Summary:

  Due to the increasing sophistication of web attacks, Web Application Firewalls (WAFs) have to be tested and updated regularly to resist the relentless flow of web attacks. In practice, using a brute-force attack to discover vulnerabilities is infeasible due to the wide variety of attack patterns. Thus, various black-box testing techniques have been proposed in the literature. However, these techniques suffer from low efficiency. This paper presents Reinforcement-Learning-Driven and Adaptive Testing (RAT), an automated black-box testing strategy to discover injection vulnerabilities in WAFs. In particular, we focus on SQL injection and Cross-site Scripting, which have been among the top ten vulnerabilities over the past decade. More specifically, RAT clusters similar attack samples together. It then utilizes a reinforcement learning technique combined with a novel adaptive search algorithm to discover almost all bypassing attack patterns efficiently. We compare RAT with three state-of-the-art methods considering their objectives. The experiments show that RAT performs 33.53% and 63.16% on average better than its counterparts in discovering the most possible bypassing payloads and reducing the number of attempts before finding the first bypassing payload when testing well-configured WAFs, respectively.

Title: Black-box Membership Inference Attacks against Fine-tuned Diffusion Models. (arXiv:2312.08207v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.08207
• Code URL: null
• Copy Paste: [[2312.08207]] Black-box Membership Inference Attacks against Fine-tuned Diffusion Models(http://arxiv.org/abs/2312.08207)
• Summary:

  With the rapid advancement of diffusion-based image-generative models, the quality of generated images has become increasingly photorealistic. Moreover, with the release of high-quality pre-trained image-generative models, a growing number of users are downloading these pre-trained models to fine-tune them with downstream datasets for various image-generation tasks. However, employing such powerful pre-trained models in downstream tasks presents significant privacy leakage risks. In this paper, we propose the first reconstruction-based membership inference attack framework, tailored for recent diffusion models, and in the more stringent black-box access setting. Considering four distinct attack scenarios and three types of attacks, this framework is capable of targeting any popular conditional generator model, achieving high precision, evidenced by an impressive AUC of $0.95$.

robust

Title: Characteristic Guidance: Non-linear Correction for DDPM at Large Guidance Scale. (arXiv:2312.07586v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07586
• Code URL: null
• Copy Paste: [[2312.07586]] Characteristic Guidance: Non-linear Correction for DDPM at Large Guidance Scale(http://arxiv.org/abs/2312.07586)
• Summary:

  Popular guidance for denoising diffusion probabilistic model (DDPM) linearly combines distinct conditional models together to provide enhanced control over samples. However, this approach overlooks nonlinear effects that become significant when guidance scale is large. To address this issue, we propose characteristic guidance, a novel method that provides non-linear correction for classifier-free guided DDPMs. Such correction forces the guided DDPMs to respect the Fokker-Planck equation of their underlying diffusion process, in a way that is first-principle, training-free, derivative-free, and compatible with existing sampling methods. Experiments show that characteristic guidance is robust to various applications, offers enhanced control over sample generation, suppresses color and exposure issues even for latent space sampling, and can handle physics problems such as the phase transitions.

Title: Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancements. (arXiv:2312.07835v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07835
• Code URL: null
• Copy Paste: [[2312.07835]] Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancements(http://arxiv.org/abs/2312.07835)
• Summary:

  In this paper, we present a novel robust framework for low-level vision tasks, including denoising, object removal, frame interpolation, and super-resolution, that does not require any external training data corpus. Our proposed approach directly learns the weights of neural modules by optimizing over the corrupted test sequence, leveraging the spatio-temporal coherence and internal statistics of videos. Furthermore, we introduce a novel spatial pyramid loss that leverages the property of spatio-temporal patch recurrence in a video across the different scales of the video. This loss enhances robustness to unstructured noise in both the spatial and temporal domains. This further results in our framework being highly robust to degradation in input frames and yields state-of-the-art results on downstream tasks such as denoising, object removal, and frame interpolation. To validate the effectiveness of our approach, we conduct qualitative and quantitative evaluations on standard video datasets such as DAVIS, UCF-101, and VIMEO90K-T.

Title: Encoder-minimal and Decoder-minimal Framework for Remote Sensing Image Dehazing. (arXiv:2312.07849v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07849
• Code URL: https://github.com/chdwyb/rshazenet
• Copy Paste: [[2312.07849]] Encoder-minimal and Decoder-minimal Framework for Remote Sensing Image Dehazing(http://arxiv.org/abs/2312.07849)
• Summary:

  Haze obscures remote sensing images, hindering valuable information extraction. To this end, we propose RSHazeNet, an encoder-minimal and decoder-minimal framework for efficient remote sensing image dehazing. Specifically, regarding the process of merging features within the same level, we develop an innovative module called intra-level transposed fusion module (ITFM). This module employs adaptive transposed self-attention to capture comprehensive context-aware information, facilitating the robust context-aware feature fusion. Meanwhile, we present a cross-level multi-view interaction module (CMIM) to enable effective interactions between features from various levels, mitigating the loss of information due to the repeated sampling operations. In addition, we propose a multi-view progressive extraction block (MPEB) that partitions the features into four distinct components and employs convolution with varying kernel sizes, groups, and dilation factors to facilitate view-progressive feature learning. Extensive experiments demonstrate the superiority of our proposed RSHazeNet. We release the source code and all pre-trained models at \url{https://github.com/chdwyb/RSHazeNet}.

Title: Generalized Deepfakes Detection with Reconstructed-Blended Images and Multi-scale Feature Reconstruction Network. (arXiv:2312.08020v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08020
• Code URL: null
• Copy Paste: [[2312.08020]] Generalized Deepfakes Detection with Reconstructed-Blended Images and Multi-scale Feature Reconstruction Network(http://arxiv.org/abs/2312.08020)
• Summary:

  The growing diversity of digital face manipulation techniques has led to an urgent need for a universal and robust detection technology to mitigate the risks posed by malicious forgeries. We present a blended-based detection approach that has robust applicability to unseen datasets. It combines a method for generating synthetic training samples, i.e., reconstructed blended images, that incorporate potential deepfake generator artifacts and a detection model, a multi-scale feature reconstruction network, for capturing the generic boundary artifacts and noise distribution anomalies brought about by digital face manipulations. Experiments demonstrated that this approach results in better performance in both cross-manipulation detection and cross-dataset detection on unseen data.

Title: Partial Symmetry Detection for 3D Geometry using Contrastive Learning with Geodesic Point Cloud Patches. (arXiv:2312.08230v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08230
• Code URL: null
• Copy Paste: [[2312.08230]] Partial Symmetry Detection for 3D Geometry using Contrastive Learning with Geodesic Point Cloud Patches(http://arxiv.org/abs/2312.08230)
• Summary:

  Symmetry detection, especially partial and extrinsic symmetry, is essential for various downstream tasks, like 3D geometry completion, segmentation, compression and structure-aware shape encoding or generation. In order to detect partial extrinsic symmetries, we propose to learn rotation, reflection, translation and scale invariant local shape features for geodesic point cloud patches via contrastive learning, which are robust across multiple classes and generalize over different datasets. We show that our approach is able to extract multiple valid solutions for this ambiguous problem. Furthermore, we introduce a novel benchmark test for partial extrinsic symmetry detection to evaluate our method. Lastly, we incorporate the detected symmetries together with a region growing algorithm to demonstrate a downstream task with the goal of computing symmetry-aware partitions of 3D shapes. To our knowledge, we are the first to propose a self-supervised data-driven method for partial extrinsic symmetry detection.

Title: Robust Few-Shot Named Entity Recognition with Boundary Discrimination and Correlation Purification. (arXiv:2312.07961v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.07961
• Code URL: https://github.com/ckgconstruction/bdcp
• Copy Paste: [[2312.07961]] Robust Few-Shot Named Entity Recognition with Boundary Discrimination and Correlation Purification(http://arxiv.org/abs/2312.07961)
• Summary:

  Few-shot named entity recognition (NER) aims to recognize novel named entities in low-resource domains utilizing existing knowledge. However, the present few-shot NER models assume that the labeled data are all clean without noise or outliers, and there are few works focusing on the robustness of the cross-domain transfer learning ability to textual adversarial attacks in Few-shot NER. In this work, we comprehensively explore and assess the robustness of few-shot NER models under textual adversarial attack scenario, and found the vulnerability of existing few-shot NER models. Furthermore, we propose a robust two-stage few-shot NER method with Boundary Discrimination and Correlation Purification (BDCP). Specifically, in the span detection stage, the entity boundary discriminative module is introduced to provide a highly distinguishing boundary representation space to detect entity spans. In the entity typing stage, the correlations between entities and contexts are purified by minimizing the interference information and facilitating correlation generalization to alleviate the perturbations caused by textual adversarial attacks. In addition, we construct adversarial examples for few-shot NER based on public datasets Few-NERD and Cross-Dataset. Comprehensive evaluations on those two groups of few-shot NER datasets containing adversarial examples demonstrate the robustness and superiority of the proposed method.

Title: CoRTEx: Contrastive Learning for Representing Terms via Explanations with Applications on Constructing Biomedical Knowledge Graphs. (arXiv:2312.08036v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.08036
• Code URL: https://github.com/yinghy18/cortex
• Copy Paste: [[2312.08036]] CoRTEx: Contrastive Learning for Representing Terms via Explanations with Applications on Constructing Biomedical Knowledge Graphs(http://arxiv.org/abs/2312.08036)
• Summary:

  Objective: Biomedical Knowledge Graphs play a pivotal role in various biomedical research domains. Concurrently, term clustering emerges as a crucial step in constructing these knowledge graphs, aiming to identify synonymous terms. Due to a lack of knowledge, previous contrastive learning models trained with Unified Medical Language System (UMLS) synonyms struggle at clustering difficult terms and do not generalize well beyond UMLS terms. In this work, we leverage the world knowledge from Large Language Models (LLMs) and propose Contrastive Learning for Representing Terms via Explanations (CoRTEx) to enhance term representation and significantly improves term clustering. Materials and Methods: The model training involves generating explanations for a cleaned subset of UMLS terms using ChatGPT. We employ contrastive learning, considering term and explanation embeddings simultaneously, and progressively introduce hard negative samples. Additionally, a ChatGPT-assisted BIRCH algorithm is designed for efficient clustering of a new ontology. Results: We established a clustering test set and a hard negative test set, where our model consistently achieves the highest F1 score. With CoRTEx embeddings and the modified BIRCH algorithm, we grouped 35,580,932 terms from the Biomedical Informatics Ontology System (BIOS) into 22,104,559 clusters with O(N) queries to ChatGPT. Case studies highlight the model's efficacy in handling challenging samples, aided by information from explanations. Conclusion: By aligning terms to their explanations, CoRTEx demonstrates superior accuracy over benchmark models and robustness beyond its training set, and it is suitable for clustering terms for large-scale biomedical ontologies.

Title: Benchmarking Distribution Shift in Tabular Data with TableShift. (arXiv:2312.07577v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07577
• Code URL: null
• Copy Paste: [[2312.07577]] Benchmarking Distribution Shift in Tabular Data with TableShift(http://arxiv.org/abs/2312.07577)
• Summary:

  Robustness to distribution shift has become a growing concern for text and image models as they transition from research subjects to deployment in the real world. However, high-quality benchmarks for distribution shift in tabular machine learning tasks are still lacking despite the widespread real-world use of tabular data and differences in the models used for tabular data in comparison to text and images. As a consequence, the robustness of tabular models to distribution shift is poorly understood. To address this issue, we introduce TableShift, a distribution shift benchmark for tabular data. TableShift contains 15 binary classification tasks in total, each with an associated shift, and includes a diverse set of data sources, prediction targets, and distribution shifts. The benchmark covers domains including finance, education, public policy, healthcare, and civic participation, and is accessible using only a few lines of Python code via the TableShift API. We conduct a large-scale study comparing several state-of-the-art tabular data models alongside robust learning and domain generalization methods on the benchmark tasks. Our study demonstrates (1) a linear trend between in-distribution (ID) and out-of-distribution (OOD) accuracy; (2) domain robustness methods can reduce shift gaps but at the cost of reduced ID accuracy; (3) a strong relationship between shift gap (difference between ID and OOD performance) and shifts in the label distribution.

  The benchmark data, Python package, model implementations, and more information about TableShift are available at https://github.com/mlfoundations/tableshift and https://tableshift.org .

Title: An Online, Adaptive and Unsupervised Regression Framework with Drift Detection for Label Scarcity Contexts. (arXiv:2312.07682v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07682
• Code URL: https://github.com/redsofa/unsupervised-online-regression
• Copy Paste: [[2312.07682]] An Online, Adaptive and Unsupervised Regression Framework with Drift Detection for Label Scarcity Contexts(http://arxiv.org/abs/2312.07682)
• Summary:

  In scenarios where obtaining real-time labels proves challenging, conventional approaches may result in sub-optimal performance. This paper presents an optimal strategy for streaming contexts with limited labeled data, introducing an adaptive technique for unsupervised regression. The proposed method leverages a sparse set of initial labels and introduces an innovative drift detection mechanism to enable dynamic model adaptations in response to evolving patterns in the data. To enhance adaptability, we integrate the ADWIN (ADaptive WINdowing) algorithm with error generalization based on Root Mean Square Error (RMSE). ADWIN facilitates real-time drift detection, while RMSE provides a robust measure of model prediction accuracy. This combination enables our multivariate method to effectively navigate the challenges of streaming data, continuously adapting to changing patterns while maintaining a high level of predictive precision. Finally, we evaluate the performance of our multivariate method across various public datasets, comparing it to non-adapting baselines. Through comprehensive assessments, we demonstrate the superior efficacy of our adaptive regression technique for tasks where obtaining labels in real-time is a significant challenge. The results underscore the method's capacity to outperform traditional approaches and highlight its potential in scenarios characterized by label scarcity and evolving data patterns.

biometric

steal

Title: BarraCUDA: Bringing Electromagnetic Side Channel Into Play to Steal the Weights of Neural Networks from NVIDIA GPUs. (arXiv:2312.07783v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.07783
• Code URL: null
• Copy Paste: [[2312.07783]] BarraCUDA: Bringing Electromagnetic Side Channel Into Play to Steal the Weights of Neural Networks from NVIDIA GPUs(http://arxiv.org/abs/2312.07783)
• Summary:

  Over the last decade, applications of neural networks have spread to cover all aspects of life. A large number of companies base their businesses on building products that use neural networks for tasks such as face recognition, machine translation, and autonomous cars. They are being used in safety and security-critical applications like high definition maps and medical wristbands, or in globally used products like Google Translate and ChatGPT. Much of the intellectual property underpinning these products is encoded in the exact configuration of the neural networks. Consequently, protecting these is of utmost priority to businesses. At the same time, many of these products need to operate under a strong threat model, in which the adversary has unfettered physical control of the product.

  Past work has demonstrated that with physical access, attackers can reverse engineer neural networks that run on scalar microcontrollers, like ARM Cortex M3. However, for performance reasons, neural networks are often implemented on highly-parallel general purpose graphics processing units (GPGPUs), and so far, attacks on these have only recovered course-grained information on the structure of the neural network, but failed to retrieve the weights and biases.

  In this work, we present BarraCUDA, a novel attack on GPGPUs that can completely extract the parameters of neural networks. BarraCUDA uses correlation electromagnetic analysis to recover the weights and biases in the convolutional layers of neural networks. We use BarraCUDA to attack the popular NVIDIA Jetson Nano device, demonstrating successful parameter extraction of neural networks in a highly parallel and noisy environment.

extraction

Title: AI-driven Structure Detection and Information Extraction from Historical Cadastral Maps (Early 19th Century Franciscean Cadastre in the Province of Styria) and Current High-resolution Satellite and Aerial Imagery for Remote Sensing. (arXiv:2312.07560v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07560
• Code URL: null
• Copy Paste: [[2312.07560]] AI-driven Structure Detection and Information Extraction from Historical Cadastral Maps (Early 19th Century Franciscean Cadastre in the Province of Styria) and Current High-resolution Satellite and Aerial Imagery for Remote Sensing(http://arxiv.org/abs/2312.07560)
• Summary:

  Cadastres from the 19th century are a complex as well as rich source for historians and archaeologists, whose use presents them with great challenges. For archaeological and historical remote sensing, we have trained several Deep Learning models, CNNs as well as Vision Transformers, to extract large-scale data from this knowledge representation. We present the principle results of our work here and we present a the demonstrator of our browser-based tool that allows researchers and public stakeholders to quickly identify spots that featured buildings in the 19th century Franciscean Cadastre. The tool not only supports scholars and fellow researchers in building a better understanding of the settlement history of the region of Styria, it also helps public administration and fellow citizens to swiftly identify areas of heightened sensibility with regard to the cultural heritage of the region.

Title: High-Order Structure Based Middle-Feature Learning for Visible-Infrared Person Re-Identification. (arXiv:2312.07853v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07853
• Code URL: https://github.com/jaulaucoeng/hos-net
• Copy Paste: [[2312.07853]] High-Order Structure Based Middle-Feature Learning for Visible-Infrared Person Re-Identification(http://arxiv.org/abs/2312.07853)
• Summary:

  Visible-infrared person re-identification (VI-ReID) aims to retrieve images of the same persons captured by visible (VIS) and infrared (IR) cameras. Existing VI-ReID methods ignore high-order structure information of features while being relatively difficult to learn a reasonable common feature space due to the large modality discrepancy between VIS and IR images. To address the above problems, we propose a novel high-order structure based middle-feature learning network (HOS-Net) for effective VI-ReID. Specifically, we first leverage a short- and long-range feature extraction (SLE) module to effectively exploit both short-range and long-range features. Then, we propose a high-order structure learning (HSL) module to successfully model the high-order relationship across different local features of each person image based on a whitened hypergraph network.This greatly alleviates model collapse and enhances feature representations. Finally, we develop a common feature space learning (CFL) module to learn a discriminative and reasonable common feature space based on middle features generated by aligning features from different modalities and ranges. In particular, a modality-range identity-center contrastive (MRIC) loss is proposed to reduce the distances between the VIS, IR, and middle features, smoothing the training process. Extensive experiments on the SYSU-MM01, RegDB, and LLCM datasets show that our HOS-Net achieves superior state-of-the-art performance. Our code is available at \url{https://github.com/Jaulaucoeng/HOS-Net}.

Title: Divide and Conquer: Hybrid Pre-training for Person Search. (arXiv:2312.07970v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07970
• Code URL: https://github.com/personsearch/pretrainps
• Copy Paste: [[2312.07970]] Divide and Conquer: Hybrid Pre-training for Person Search(http://arxiv.org/abs/2312.07970)
• Summary:

  Large-scale pre-training has proven to be an effective method for improving performance across different tasks. Current person search methods use ImageNet pre-trained models for feature extraction, yet it is not an optimal solution due to the gap between the pre-training task and person search task (as a downstream task). Therefore, in this paper, we focus on pre-training for person search, which involves detecting and re-identifying individuals simultaneously. Although labeled data for person search is scarce, datasets for two sub-tasks person detection and re-identification are relatively abundant. To this end, we propose a hybrid pre-training framework specifically designed for person search using sub-task data only. It consists of a hybrid learning paradigm that handles data with different kinds of supervisions, and an intra-task alignment module that alleviates domain discrepancy under limited resources. To the best of our knowledge, this is the first work that investigates how to support full-task pre-training using sub-task data. Extensive experiments demonstrate that our pre-trained model can achieve significant improvements across diverse protocols, such as person search method, fine-tuning data, pre-training data and model backbone. For example, our model improves ResNet50 based NAE by 10.3% relative improvement w.r.t. mAP. Our code and pre-trained models are released for plug-and-play usage to the person search community.

Title: Fine-Grained Image-Text Alignment in Medical Imaging Enables Cyclic Image-Report Generation. (arXiv:2312.08078v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08078
• Code URL: null
• Copy Paste: [[2312.08078]] Fine-Grained Image-Text Alignment in Medical Imaging Enables Cyclic Image-Report Generation(http://arxiv.org/abs/2312.08078)
• Summary:

  To address these issues, we propose a novel Adaptive patch-word Matching (AdaMatch) model to correlate chest X-ray (CXR) image regions with words in medical reports and apply it to CXR-report generation to provide explainability for the generation process. AdaMatch exploits the fine-grained relation between adaptive patches and words to provide explanations of specific image regions with corresponding words. To capture the abnormal regions of varying sizes and positions, we introduce the Adaptive Patch extraction (AdaPatch) module to acquire the adaptive patches for these regions adaptively. In order to provide explicit explainability for CXR-report generation task, we propose an AdaMatch-based bidirectional large language model for Cyclic CXR-report generation (AdaMatch-Cyclic). It employs the AdaMatch to obtain the keywords for CXR images and `keypatches' for medical reports as hints to guide CXR-report generation. Extensive experiments on two publicly available CXR datasets prove the effectiveness of our method and its superior performance to existing methods.

Title: PAD: Self-Supervised Pre-Training with Patchwise-Scale Adapter for Infrared Images. (arXiv:2312.08192v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08192
• Code URL: null
• Copy Paste: [[2312.08192]] PAD: Self-Supervised Pre-Training with Patchwise-Scale Adapter for Infrared Images(http://arxiv.org/abs/2312.08192)
• Summary:

  Self-supervised learning (SSL) for RGB images has achieved significant success, yet there is still limited research on SSL for infrared images, primarily due to three prominent challenges: 1) the lack of a suitable large-scale infrared pre-training dataset, 2) the distinctiveness of non-iconic infrared images rendering common pre-training tasks like masked image modeling (MIM) less effective, and 3) the scarcity of fine-grained textures making it particularly challenging to learn general image features. To address these issues, we construct a Multi-Scene Infrared Pre-training (MSIP) dataset comprising 178,756 images, and introduce object-sensitive random RoI cropping, an image preprocessing method, to tackle the challenge posed by non-iconic images. To alleviate the impact of weak textures on feature learning, we propose a pre-training paradigm called Pre-training with ADapter (PAD), which uses adapters to learn domain-specific features while freezing parameters pre-trained on ImageNet to retain the general feature extraction capability. This new paradigm is applicable to any transformer-based SSL method. Furthermore, to achieve more flexible coordination between pre-trained and newly-learned features in different layers and patches, a patchwise-scale adapter with dynamically learnable scale factors is introduced. Extensive experiments on three downstream tasks show that PAD, with only 1.23M pre-trainable parameters, outperforms other baseline paradigms including continual full pre-training on MSIP. Our code and dataset are available at https://github.com/casiatao/PAD.

Title: View-Dependent Octree-based Mesh Extraction in Unbounded Scenes for Procedural Synthetic Data. (arXiv:2312.08364v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08364
• Code URL: https://github.com/princeton-vl/ocmesher
• Copy Paste: [[2312.08364]] View-Dependent Octree-based Mesh Extraction in Unbounded Scenes for Procedural Synthetic Data(http://arxiv.org/abs/2312.08364)
• Summary:

  Procedural synthetic data generation has received increasing attention in computer vision. Procedural signed distance functions (SDFs) are a powerful tool for modeling large-scale detailed scenes, but existing mesh extraction methods have artifacts or performance profiles that limit their use for synthetic data. We propose OcMesher, a mesh extraction algorithm that efficiently handles high-detail unbounded scenes with perfect view-consistency, with easy export to downstream real-time engines. The main novelty of our solution is an algorithm to construct an octree based on a given SDF and multiple camera views. We performed extensive experiments, and show our solution produces better synthetic data for training and evaluation of computer vision models.

Title: Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models. (arXiv:2312.07887v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.07887
• Code URL: https://github.com/zzz47zzz/pretrained-lm-for-incremental-learning
• Copy Paste: [[2312.07887]] Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models(http://arxiv.org/abs/2312.07887)
• Summary:

  Incremental Learning (IL) has been a long-standing problem in both vision and Natural Language Processing (NLP) communities. In recent years, as Pre-trained Language Models (PLMs) have achieved remarkable progress in various NLP downstream tasks, utilizing PLMs as backbones has become a common practice in recent research of IL in NLP. Most assume that catastrophic forgetting is the biggest obstacle to achieving superior IL performance and propose various techniques to overcome this issue. However, we find that this assumption is problematic. Specifically, we revisit more than 20 methods on four classification tasks (Text Classification, Intent Classification, Relation Extraction, and Named Entity Recognition) under the two most popular IL settings (Class-Incremental and Task-Incremental) and reveal that most of them severely underestimate the inherent anti-forgetting ability of PLMs. Based on the observation, we propose a frustratingly easy method called SEQ* for IL with PLMs. The results show that SEQ* has competitive or superior performance compared to state-of-the-art (SOTA) IL methods and requires considerably less trainable parameters and training time. These findings urge us to revisit the IL with PLMs and encourage future studies to have a fundamental understanding of the catastrophic forgetting in PLMs. The data, code and scripts are publicly available at https://github.com/zzz47zzz/pretrained-lm-for-incremental-learning.

Title: SLJP: Semantic Extraction based Legal Judgment Prediction. (arXiv:2312.07979v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.07979
• Code URL: null
• Copy Paste: [[2312.07979]] SLJP: Semantic Extraction based Legal Judgment Prediction(http://arxiv.org/abs/2312.07979)
• Summary:

  Legal Judgment Prediction (LJP) is a judicial assistance system that recommends the legal components such as applicable statues, prison term and penalty term by analyzing the given input case document. Indian legal system is in the need of technical assistance such as artificial intelligence to solve the crores of pending cases in various courts for years and its being increased day to day. Most of the existing Indian models did not adequately concentrate on the semantics embedded in the fact description (FD) that impacts the decision. The proposed semantic extraction based LJP (SLJP) model provides the advantages of pretrained transformers for complex unstructured legal case document understanding and to generate embeddings. The model draws the in-depth semantics of the given FD at multiple levels i.e., chunk and case document level by following the divide and conquer approach. It creates the concise view of the given fact description using the extracted semantics as per the original court case document structure and predicts judgment using attention mechanism. We tested the model performance on two available Indian datasets Indian Legal Documents corpus (ILDC) and Indian Legal Statue Identification (ILSI) and got promising results. Also shown the highest performance and less performance degradation for increased epochs than base models on ILDC dataset.

Title: High-throughput Biomedical Relation Extraction for Semi-Structured Web Articles Empowered by Large Language Models. (arXiv:2312.08274v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.08274
• Code URL: null
• Copy Paste: [[2312.08274]] High-throughput Biomedical Relation Extraction for Semi-Structured Web Articles Empowered by Large Language Models(http://arxiv.org/abs/2312.08274)
• Summary:

  Objective: To develop a high-throughput biomedical relation extraction system that takes advantage of the large language models' (LLMs) reading comprehension ability and biomedical world knowledge in a scalable and evidential manner. Methods: We formulate the relation extraction task as a simple binary classification problem for large language models such as ChatGPT. Specifically, LLMs make the decision based on the external corpus and its world knowledge, giving the reason for the judgment to factual verification. This method is tailored for semi-structured web articles, wherein we designate the main title as the tail entity and explicitly incorporate it into the context, and the potential head entities are matched based on a biomedical thesaurus. Moreover, lengthy contents are sliced into text chunks, embedded, and retrieved with additional embedding models, ensuring compatibility with the context window size constraints of available open-source LLMs. Results: Using an open-source LLM, we extracted 304315 relation triplets of three distinct relation types from four reputable biomedical websites. To assess the efficacy of the basic pipeline employed for biomedical relation extraction, we curated a benchmark dataset annotated by a medical expert. Evaluation results indicate that the pipeline exhibits performance comparable to that of GPT-4. Case studies further illuminate challenges faced by contemporary LLMs in the context of biomedical relation extraction for semi-structured web articles. Conclusion: The proposed method has demonstrated its effectiveness in leveraging the strengths of LLMs for high-throughput biomedical relation extraction. Its adaptability is evident, as it can be seamlessly extended to diverse semi-structured biomedical websites, facilitating the extraction of various types of biomedical relations with ease.

Title: Estimation of embedding vectors in high dimensions. (arXiv:2312.07802v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07802
• Code URL: null
• Copy Paste: [[2312.07802]] Estimation of embedding vectors in high dimensions(http://arxiv.org/abs/2312.07802)
• Summary:

  Embeddings are a basic initial feature extraction step in many machine learning models, particularly in natural language processing. An embedding attempts to map data tokens to a low-dimensional space where similar tokens are mapped to vectors that are close to one another by some metric in the embedding space. A basic question is how well can such embedding be learned? To study this problem, we consider a simple probability model for discrete data where there is some "true" but unknown embedding where the correlation of random variables is related to the similarity of the embeddings. Under this model, it is shown that the embeddings can be learned by a variant of low-rank approximate message passing (AMP) method. The AMP approach enables precise predictions of the accuracy of the estimation in certain high-dimensional limits. In particular, the methodology provides insight on the relations of key parameters such as the number of samples per value, the frequency of the terms, and the strength of the embedding correlation on the probability distribution. Our theoretical findings are validated by simulations on both synthetic data and real text data.

membership infer

Title: GraphGuard: Detecting and Counteracting Training Data Misuse in Graph Neural Networks. (arXiv:2312.07861v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07861
• Code URL: https://github.com/graphguard/graphguard-proactive
• Copy Paste: [[2312.07861]] GraphGuard: Detecting and Counteracting Training Data Misuse in Graph Neural Networks(http://arxiv.org/abs/2312.07861)
• Summary:

  The emergence of Graph Neural Networks (GNNs) in graph data analysis and their deployment on Machine Learning as a Service platforms have raised critical concerns about data misuse during model training. This situation is further exacerbated due to the lack of transparency in local training processes, potentially leading to the unauthorized accumulation of large volumes of graph data, thereby infringing on the intellectual property rights of data owners. Existing methodologies often address either data misuse detection or mitigation, and are primarily designed for local GNN models rather than cloud-based MLaaS platforms. These limitations call for an effective and comprehensive solution that detects and mitigates data misuse without requiring exact training data while respecting the proprietary nature of such data. This paper introduces a pioneering approach called GraphGuard, to tackle these challenges. We propose a training-data-free method that not only detects graph data misuse but also mitigates its impact via targeted unlearning, all without relying on the original training data. Our innovative misuse detection technique employs membership inference with radioactive data, enhancing the distinguishability between member and non-member data distributions. For mitigation, we utilize synthetic graphs that emulate the characteristics previously learned by the target model, enabling effective unlearning even in the absence of exact graph data. We conduct comprehensive experiments utilizing four real-world graph datasets to demonstrate the efficacy of GraphGuard in both detection and unlearning. We show that GraphGuard attains a near-perfect detection rate of approximately 100% across these datasets with various GNN models. In addition, it performs unlearning by eliminating the impact of the unlearned graph with a marginal decrease in accuracy (less than 5%).

federate

Title: Federated Learning for Short Text Clustering. (arXiv:2312.07556v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.07556
• Code URL: null
• Copy Paste: [[2312.07556]] Federated Learning for Short Text Clustering(http://arxiv.org/abs/2312.07556)
• Summary:

  Short text clustering has been popularly studied for its significance in mining valuable insights from many short texts. In this paper, we focus on the federated short text clustering (FSTC) problem, i.e., clustering short texts that are distributed in different clients, which is a realistic problem under privacy requirements. Compared with the centralized short text clustering problem that short texts are stored on a central server, the FSTC problem has not been explored yet. To fill this gap, we propose a Federated Robust Short Text Clustering (FSTC) framework. FSTC includes two main modules, i.e., robust short text clustering module and federated cluster center aggregation module. The robust short text clustering module aims to train an effective short text clustering model with local data in each client. We innovatively combine optimal transport to generate pseudo-labels with Gaussian-uniform mixture model to ensure the reliability of the pseudo-supervised data. The federated cluster center aggregation module aims to exchange knowledge across clients without sharing local raw data in an efficient way. The server aggregates the local cluster centers from different clients and then sends the global centers back to all clients in each communication round. Our empirical studies on three short text clustering datasets demonstrate that FSTC significantly outperforms the federated short text clustering baselines.

Title: An Incentive Mechanism for Federated Learning Based on Multiple Resource Exchange. (arXiv:2312.08096v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.08096
• Code URL: null
• Copy Paste: [[2312.08096]] An Incentive Mechanism for Federated Learning Based on Multiple Resource Exchange(http://arxiv.org/abs/2312.08096)
• Summary:

  Federated Learning (FL) is a distributed machine learning paradigm that addresses privacy concerns in machine learning and still guarantees high test accuracy. However, achieving the necessary accuracy by having all clients participate in FL is impractical, given the constraints of client local computing resource. In this paper, we introduce a multi-user collaborative computing framework, categorizing users into two roles: model owners (MOs) and data owner (DOs). Without resorting to monetary incentives, an MO can encourage more DOs to join in FL by allowing the DOs to offload extra local computing tasks to the MO for execution. This exchange of "data" for "computing resources" streamlines the incentives for clients to engage more effectively in FL. We formulate the interaction between MO and DOs as an optimization problem, and the objective is to effectively utilize the communication and computing resource of the MO and DOs to minimize the time to complete an FL task. The proposed problem is a mixed integer nonlinear programming (MINLP) with high computational complexity. We first decompose it into two distinct subproblems, namely the client selection problem and the resource allocation problem to segregate the integer variables from the continuous variables. Then, an effective iterative algorithm is proposed to solve problem. Simulation results demonstrate that the proposed collaborative computing framework can achieve an accuracy of more than 95\% while minimizing the overall time to complete an FL task.

fair

interpretability

Title: Interpretable factorization of clinical questionnaires to identify latent factors of psychopathology. (arXiv:2312.07762v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07762
• Code URL: https://github.com/jefferykclam/icqf
• Copy Paste: [[2312.07762]] Interpretable factorization of clinical questionnaires to identify latent factors of psychopathology(http://arxiv.org/abs/2312.07762)
• Summary:

  Psychiatry research seeks to understand the manifestations of psychopathology in behavior, as measured in questionnaire data, by identifying a small number of latent factors that explain them. While factor analysis is the traditional tool for this purpose, the resulting factors may not be interpretable, and may also be subject to confounding variables. Moreover, missing data are common, and explicit imputation is often required. To overcome these limitations, we introduce interpretability constrained questionnaire factorization (ICQF), a non-negative matrix factorization method with regularization tailored for questionnaire data. Our method aims to promote factor interpretability and solution stability. We provide an optimization procedure with theoretical convergence guarantees, and an automated procedure to detect latent dimensionality accurately. We validate these procedures using realistic synthetic data. We demonstrate the effectiveness of our method in a widely used general-purpose questionnaire, in two independent datasets (the Healthy Brain Network and Adolescent Brain Cognitive Development studies). Specifically, we show that ICQF improves interpretability, as defined by domain experts, while preserving diagnostic information across a range of disorders, and outperforms competing methods for smaller dataset sizes. This suggests that the regularization in our method matches domain characteristics. The python implementation for ICQF is available at \url{https://github.com/jefferykclam/ICQF}.

Title: Explainable Trajectory Representation through Dictionary Learning. (arXiv:2312.08052v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.08052
• Code URL: null
• Copy Paste: [[2312.08052]] Explainable Trajectory Representation through Dictionary Learning(http://arxiv.org/abs/2312.08052)
• Summary:

  Trajectory representation learning on a network enhances our understanding of vehicular traffic patterns and benefits numerous downstream applications. Existing approaches using classic machine learning or deep learning embed trajectories as dense vectors, which lack interpretability and are inefficient to store and analyze in downstream tasks. In this paper, an explainable trajectory representation learning framework through dictionary learning is proposed. Given a collection of trajectories on a network, it extracts a compact dictionary of commonly used subpaths called "pathlets", which optimally reconstruct each trajectory by simple concatenations. The resulting representation is naturally sparse and encodes strong spatial semantics. Theoretical analysis of our proposed algorithm is conducted to provide a probabilistic bound on the estimation error of the optimal dictionary. A hierarchical dictionary learning scheme is also proposed to ensure the algorithm's scalability on large networks, leading to a multi-scale trajectory representation. Our framework is evaluated on two large-scale real-world taxi datasets. Compared to previous work, the dictionary learned by our method is more compact and has better reconstruction rate for new trajectories. We also demonstrate the promising performance of this method in downstream tasks including trip time prediction task and data compression.

Title: TERM Model: Tensor Ring Mixture Model for Density Estimation. (arXiv:2312.08075v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.08075
• Code URL: null
• Copy Paste: [[2312.08075]] TERM Model: Tensor Ring Mixture Model for Density Estimation(http://arxiv.org/abs/2312.08075)
• Summary:

  Efficient probability density estimation is a core challenge in statistical machine learning. Tensor-based probabilistic graph methods address interpretability and stability concerns encountered in neural network approaches. However, a substantial number of potential tensor permutations can lead to a tensor network with the same structure but varying expressive capabilities. In this paper, we take tensor ring decomposition for density estimator, which significantly reduces the number of permutation candidates while enhancing expressive capability compared with existing used decompositions. Additionally, a mixture model that incorporates multiple permutation candidates with adaptive weights is further designed, resulting in increased expressive flexibility and comprehensiveness. Different from the prevailing directions of tensor network structure/permutation search, our approach provides a new viewpoint inspired by ensemble learning. This approach acknowledges that suboptimal permutations can offer distinctive information besides that of optimal permutations. Experiments show the superiority of the proposed approach in estimating probability density for moderately dimensional datasets and sampling to capture intricate details.

explainability

Title: Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing. (arXiv:2312.07875v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07875
• Code URL: https://github.com/guangmingzhu/sketchesc
• Copy Paste: [[2312.07875]] Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing(http://arxiv.org/abs/2312.07875)
• Summary:

  Free-hand sketches are appealing for humans as a universal tool to depict the visual world. Humans can recognize varied sketches of a category easily by identifying the concurrence and layout of the intrinsic semantic components of the category, since humans draw free-hand sketches based a common consensus that which types of semantic components constitute each sketch category. For example, an airplane should at least have a fuselage and wings. Based on this analysis, a semantic component-level memory module is constructed and embedded in the proposed structured sketch recognition network in this paper. The memory keys representing semantic components of each sketch category can be self-learned and enhance the recognition network's explainability. Our proposed networks can deal with different situations of sketch recognition, i.e., with or without semantic components labels of strokes. Experiments on the SPG and SketchIME datasets demonstrate the memory module's flexibility and the recognition network's explainability. The code and data are available at https://github.com/GuangmingZhu/SketchESC.

watermark

Title: A Survey of Text Watermarking in the Era of Large Language Models. (arXiv:2312.07913v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.07913
• Code URL: null
• Copy Paste: [[2312.07913]] A Survey of Text Watermarking in the Era of Large Language Models(http://arxiv.org/abs/2312.07913)
• Summary:

  In recent years, significant advancements have been made in the text generation capabilities of Large Language Models (LLMs), demonstrating exceptional performance in downstream tasks such as abstract summarization, dialogue generation, and data-to-text conversion. However, their generative abilities also pose risks such as the rapid spread of fake news, infringement of datasets/LLM copyrights, and challenges to academic integrity. Text watermarking technology emerges as a potential solution. By embedding invisible yet detectable patterns in generated texts, it helps in tracking and verifying text origins, thus preventing misuse and piracy.

  This survey aims to comprehensively summarize current text watermarking technologies, covering three main aspects: (1) an overview and comparison of different text watermarking techniques; (2) evaluation methods for text watermarking algorithms, including their success rate, impact on text quality, robustness, and unforgeability; (3) potential applications of text watermarking technologys. This survey aims to help researchers thoroughly understanding the text watermarking technologies, thereby fostering further development.

Title: Towards Optimal Statistical Watermarking. (arXiv:2312.07930v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07930
• Code URL: null
• Copy Paste: [[2312.07930]] Towards Optimal Statistical Watermarking(http://arxiv.org/abs/2312.07930)
• Summary:

  We study statistical watermarking by formulating it as a hypothesis testing problem, a general framework which subsumes all previous statistical watermarking methods. Key to our formulation is a coupling of the output tokens and the rejection region, realized by pseudo-random generators in practice, that allows non-trivial trade-off between the Type I error and Type II error. We characterize the Uniformly Most Powerful (UMP) watermark in this context. In the most common scenario where the output is a sequence of $n$ tokens, we establish matching upper and lower bounds on the number of i.i.d. tokens required to guarantee small Type I and Type II errors. Our rate scales as $\Theta(h^{-1} \log (1/h))$ with respect to the average entropy per token $h$ and thus greatly improves the $O(h^{-2})$ rate in the previous works. For scenarios where the detector lacks knowledge of the model's distribution, we introduce the concept of model-agnostic watermarking and establish the minimax bounds for the resultant increase in Type II error. Moreover, we formulate the robust watermarking problem where user is allowed to perform a class of perturbation on the generated texts, and characterize the optimal type II error of robust UMP tests via a linear programming problem. To the best of our knowledge, this is the first systematic statistical treatment on the watermarking problem with near-optimal rates in the i.i.d. setting, and might be of interest for future works.

diffusion

Title: Uncertainty Visualization via Low-Dimensional Posterior Projections. (arXiv:2312.07804v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07804
• Code URL: null
• Copy Paste: [[2312.07804]] Uncertainty Visualization via Low-Dimensional Posterior Projections(http://arxiv.org/abs/2312.07804)
• Summary:

  In ill-posed inverse problems, it is commonly desirable to obtain insight into the full spectrum of plausible solutions, rather than extracting only a single reconstruction. Information about the plausible solutions and their likelihoods is encoded in the posterior distribution. However, for high-dimensional data, this distribution is challenging to visualize. In this work, we introduce a new approach for estimating and visualizing posteriors by employing energy-based models (EBMs) over low-dimensional subspaces. Specifically, we train a conditional EBM that receives an input measurement and a set of directions that span some low-dimensional subspace of solutions, and outputs the probability density function of the posterior within that space. We demonstrate the effectiveness of our method across a diverse range of datasets and image restoration problems, showcasing its strength in uncertainty quantification and visualization. As we show, our method outperforms a baseline that projects samples from a diffusion-based posterior sampler, while being orders of magnitude faster. Furthermore, it is more accurate than a baseline that assumes a Gaussian posterior.

Title: Diffusion Models Enable Zero-Shot Pose Estimation for Lower-Limb Prosthetic Users. (arXiv:2312.07854v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07854
• Code URL: null
• Copy Paste: [[2312.07854]] Diffusion Models Enable Zero-Shot Pose Estimation for Lower-Limb Prosthetic Users(http://arxiv.org/abs/2312.07854)
• Summary:

  The application of 2D markerless gait analysis has garnered increasing interest and application within clinical settings. However, its effectiveness in the realm of lower-limb amputees has remained less than optimal. In response, this study introduces an innovative zero-shot method employing image generation diffusion models to achieve markerless pose estimation for lower-limb prosthetics, presenting a promising solution to gait analysis for this specific population. Our approach demonstrates an enhancement in detecting key points on prosthetic limbs over existing methods, and enables clinicians to gain invaluable insights into the kinematics of lower-limb amputees across the gait cycle. The outcomes obtained not only serve as a proof-of-concept for the feasibility of this zero-shot approach but also underscore its potential in advancing rehabilitation through gait analysis for this unique population.

Title: SimAC: A Simple Anti-Customization Method against Text-to-Image Synthesis of Diffusion Models. (arXiv:2312.07865v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07865
• Code URL: null
• Copy Paste: [[2312.07865]] SimAC: A Simple Anti-Customization Method against Text-to-Image Synthesis of Diffusion Models(http://arxiv.org/abs/2312.07865)
• Summary:

  Despite the success of diffusion-based customization methods on visual content creation, increasing concerns have been raised about such techniques from both privacy and political perspectives. To tackle this issue, several anti-customization methods have been proposed in very recent months, predominantly grounded in adversarial attacks. Unfortunately, most of these methods adopt straightforward designs, such as end-to-end optimization with a focus on adversarially maximizing the original training loss, thereby neglecting nuanced internal properties intrinsic to the diffusion model, and even leading to ineffective optimization in some diffusion time steps. In this paper, we strive to bridge this gap by undertaking a comprehensive exploration of these inherent properties, to boost the performance of current anti-customization approaches. Two aspects of properties are investigated: 1) We examine the relationship between time step selection and the model's perception in the frequency domain of images and find that lower time steps can give much more contributions to adversarial noises. This inspires us to propose an adaptive greedy search for optimal time steps that seamlessly integrates with existing anti-customization methods. 2) We scrutinize the roles of features at different layers during denoising and devise a sophisticated feature-based optimization framework for anti-customization. Experiments on facial benchmarks demonstrate that our approach significantly increases identity disruption, thereby enhancing user privacy and security.

Title: BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics. (arXiv:2312.07937v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07937
• Code URL: null
• Copy Paste: [[2312.07937]] BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics(http://arxiv.org/abs/2312.07937)
• Summary:

  The recently emerging text-to-motion advances have spired numerous attempts for convenient and interactive human motion generation. Yet, existing methods are largely limited to generating body motions only without considering the rich two-hand motions, let alone handling various conditions like body dynamics or texts. To break the data bottleneck, we propose BOTH57M, a novel multi-modal dataset for two-hand motion generation. Our dataset includes accurate motion tracking for the human body and hands and provides pair-wised finger-level hand annotations and body descriptions. We further provide a strong baseline method, BOTH2Hands, for the novel task: generating vivid two-hand motions from both implicit body dynamics and explicit text prompts. We first warm up two parallel body-to-hand and text-to-hand diffusion models and then utilize the cross-attention transformer for motion blending. Extensive experiments and cross-validations demonstrate the effectiveness of our approach and dataset for generating convincing two-hand motions from the hybrid body-and-textual conditions. Our dataset and code will be disseminated to the community for future research.

Title: Semantic-aware Data Augmentation for Text-to-image Synthesis. (arXiv:2312.07951v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07951
• Code URL: null
• Copy Paste: [[2312.07951]] Semantic-aware Data Augmentation for Text-to-image Synthesis(http://arxiv.org/abs/2312.07951)
• Summary:

  Data augmentation has been recently leveraged as an effective regularizer in various vision-language deep neural networks. However, in text-to-image synthesis (T2Isyn), current augmentation wisdom still suffers from the semantic mismatch between augmented paired data. Even worse, semantic collapse may occur when generated images are less semantically constrained. In this paper, we develop a novel Semantic-aware Data Augmentation (SADA) framework dedicated to T2Isyn. In particular, we propose to augment texts in the semantic space via an Implicit Textual Semantic Preserving Augmentation ($ITA$), in conjunction with a specifically designed Image Semantic Regularization Loss ($L_r$) as Generated Image Semantic Conservation, to cope well with semantic mismatch and collapse. As one major contribution, we theoretically show that $ITA$ can certify better text-image consistency while $L_r$ regularizing the semantics of generated images would avoid semantic collapse and enhance image quality. Extensive experiments validate that SADA enhances text-image consistency and improves image quality significantly in T2Isyn models across various backbones. Especially, incorporating SADA during the tuning process of Stable Diffusion models also yields performance improvements.

Title: LMD: Faster Image Reconstruction with Latent Masking Diffusion. (arXiv:2312.07971v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07971
• Code URL: https://github.com/anonymouspony/lmd
• Copy Paste: [[2312.07971]] LMD: Faster Image Reconstruction with Latent Masking Diffusion(http://arxiv.org/abs/2312.07971)
• Summary:

  As a class of fruitful approaches, diffusion probabilistic models (DPMs) have shown excellent advantages in high-resolution image reconstruction. On the other hand, masked autoencoders (MAEs), as popular self-supervised vision learners, have demonstrated simpler and more effective image reconstruction and transfer capabilities on downstream tasks. However, they all require extremely high training costs, either due to inherent high temporal-dependence (i.e., excessively long diffusion steps) or due to artificially low spatial-dependence (i.e., human-formulated high mask ratio, such as 0.75). To the end, this paper presents LMD, a faster image reconstruction framework with latent masking diffusion. First, we propose to project and reconstruct images in latent space through a pre-trained variational autoencoder, which is theoretically more efficient than in the pixel-based space. Then, we combine the advantages of MAEs and DPMs to design a progressive masking diffusion model, which gradually increases the masking proportion by three different schedulers and reconstructs the latent features from simple to difficult, without sequentially performing denoising diffusion as in DPMs or using fixed high masking ratio as in MAEs, so as to alleviate the high training time-consumption predicament. Our approach allows for learning high-capacity models and accelerate their training (by 3x or more) and barely reduces the original accuracy. Inference speed in downstream tasks also significantly outperforms the previous approaches.

Title: AdapEdit: Spatio-Temporal Guided Adaptive Editing Algorithm for Text-Based Continuity-Sensitive Image Editing. (arXiv:2312.08019v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08019
• Code URL: https://github.com/anonymouspony/adap-edit
• Copy Paste: [[2312.08019]] AdapEdit: Spatio-Temporal Guided Adaptive Editing Algorithm for Text-Based Continuity-Sensitive Image Editing(http://arxiv.org/abs/2312.08019)
• Summary:

  With the great success of text-conditioned diffusion models in creative text-to-image generation, various text-driven image editing approaches have attracted the attentions of many researchers. However, previous works mainly focus on discreteness-sensitive instructions such as adding, removing or replacing specific objects, background elements or global styles (i.e., hard editing), while generally ignoring subject-binding but semantically fine-changing continuity-sensitive instructions such as actions, poses or adjectives, and so on (i.e., soft editing), which hampers generative AI from generating user-customized visual contents. To mitigate this predicament, we propose a spatio-temporal guided adaptive editing algorithm AdapEdit, which realizes adaptive image editing by introducing a soft-attention strategy to dynamically vary the guiding degree from the editing conditions to visual pixels from both temporal and spatial perspectives. Note our approach has a significant advantage in preserving model priors and does not require model training, fine-tuning, extra data, or optimization. We present our results over a wide variety of raw images and editing instructions, demonstrating competitive performance and showing it significantly outperforms the previous approaches.

Title: ClusterDDPM: An EM clustering framework with Denoising Diffusion Probabilistic Models. (arXiv:2312.08029v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.08029
• Code URL: null
• Copy Paste: [[2312.08029]] ClusterDDPM: An EM clustering framework with Denoising Diffusion Probabilistic Models(http://arxiv.org/abs/2312.08029)
• Summary:

  Variational autoencoder (VAE) and generative adversarial networks (GAN) have found widespread applications in clustering and have achieved significant success. However, the potential of these approaches may be limited due to VAE's mediocre generation capability or GAN's well-known instability during adversarial training. In contrast, denoising diffusion probabilistic models (DDPMs) represent a new and promising class of generative models that may unlock fresh dimensions in clustering. In this study, we introduce an innovative expectation-maximization (EM) framework for clustering using DDPMs. In the E-step, we aim to derive a mixture of Gaussian priors for the subsequent M-step. In the M-step, our focus lies in learning clustering-friendly latent representations for the data by employing the conditional DDPM and matching the distribution of latent representations to the mixture of Gaussian priors. We present a rigorous theoretical analysis of the optimization process in the M-step, proving that the optimizations are equivalent to maximizing the lower bound of the Q function within the vanilla EM framework under certain constraints. Comprehensive experiments validate the advantages of the proposed framework, showcasing superior performance in clustering, unsupervised conditional generation and latent representation learning.

Title: Compositional Inversion for Stable Diffusion Models. (arXiv:2312.08048v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08048
• Code URL: https://github.com/zhangxulu1996/compositional-inversion
• Copy Paste: [[2312.08048]] Compositional Inversion for Stable Diffusion Models(http://arxiv.org/abs/2312.08048)
• Summary:

  Inversion methods, such as Textual Inversion, generate personalized images by incorporating concepts of interest provided by user images. However, existing methods often suffer from overfitting issues, where the dominant presence of inverted concepts leads to the absence of other desired concepts. It stems from the fact that during inversion, the irrelevant semantics in the user images are also encoded, forcing the inverted concepts to occupy locations far from the core distribution in the embedding space. To address this issue, we propose a method that guides the inversion process towards the core distribution for compositional embeddings. Additionally, we introduce a spatial regularization approach to balance the attention on the concepts being composed. Our method is designed as a post-training approach and can be seamlessly integrated with other inversion methods. Experimental results demonstrate the effectiveness of our proposed approach in mitigating the overfitting problem and generating more diverse and balanced compositions of concepts in the synthesized images. The source code is available at https://github.com/zhangxulu1996/Compositional-Inversion.

Title: Knowledge-Aware Artifact Image Synthesis with LLM-Enhanced Prompting and Multi-Source Supervision. (arXiv:2312.08056v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08056
• Code URL: null
• Copy Paste: [[2312.08056]] Knowledge-Aware Artifact Image Synthesis with LLM-Enhanced Prompting and Multi-Source Supervision(http://arxiv.org/abs/2312.08056)
• Summary:

  Ancient artifacts are an important medium for cultural preservation and restoration. However, many physical copies of artifacts are either damaged or lost, leaving a blank space in archaeological and historical studies that calls for artifact image generation techniques. Despite the significant advancements in open-domain text-to-image synthesis, existing approaches fail to capture the important domain knowledge presented in the textual description, resulting in errors in recreated images such as incorrect shapes and patterns. In this paper, we propose a novel knowledge-aware artifact image synthesis approach that brings lost historical objects accurately into their visual forms. We use a pretrained diffusion model as backbone and introduce three key techniques to enhance the text-to-image generation framework: 1) we construct prompts with explicit archaeological knowledge elicited from large language models (LLMs); 2) we incorporate additional textual guidance to correlated historical expertise in a contrastive manner; 3) we introduce further visual-semantic constraints on edge and perceptual features that enable our model to learn more intricate visual details of the artifacts. Compared to existing approaches, our proposed model produces higher-quality artifact images that align better with the implicit details and historical knowledge contained within written documents, thus achieving significant improvements across automatic metrics and in human evaluation. Our code and data are available at https://github.com/danielwusg/artifact_diffusion.

Title: Clockwork Diffusion: Efficient Generation With Model-Step Distillation. (arXiv:2312.08128v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08128
• Code URL: null
• Copy Paste: [[2312.08128]] Clockwork Diffusion: Efficient Generation With Model-Step Distillation(http://arxiv.org/abs/2312.08128)
• Summary:

  This work aims to improve the efficiency of text-to-image diffusion models. While diffusion models use computationally expensive UNet-based denoising operations in every generation step, we identify that not all operations are equally relevant for the final output quality. In particular, we observe that UNet layers operating on high-res feature maps are relatively sensitive to small perturbations. In contrast, low-res feature maps influence the semantic layout of the final image and can often be perturbed with no noticeable change in the output. Based on this observation, we propose Clockwork Diffusion, a method that periodically reuses computation from preceding denoising steps to approximate low-res feature maps at one or more subsequent steps. For multiple baselines, and for both text-to-image generation and image editing, we demonstrate that Clockwork leads to comparable or improved perceptual scores with drastically reduced computational complexity. As an example, for Stable Diffusion v1.5 with 8 DPM++ steps we save 32% of FLOPs with negligible FID and CLIP change.

Title: Concept-centric Personalization with Large-scale Diffusion Priors. (arXiv:2312.08195v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08195
• Code URL: https://github.com/priv-creation/concept-centric-personalization
• Copy Paste: [[2312.08195]] Concept-centric Personalization with Large-scale Diffusion Priors(http://arxiv.org/abs/2312.08195)
• Summary:

  Despite large-scale diffusion models being highly capable of generating diverse open-world content, they still struggle to match the photorealism and fidelity of concept-specific generators. In this work, we present the task of customizing large-scale diffusion priors for specific concepts as concept-centric personalization. Our goal is to generate high-quality concept-centric images while maintaining the versatile controllability inherent to open-world models, enabling applications in diverse tasks such as concept-centric stylization and image translation. To tackle these challenges, we identify catastrophic forgetting of guidance prediction from diffusion priors as the fundamental issue. Consequently, we develop a guidance-decoupled personalization framework specifically designed to address this task. We propose Generalized Classifier-free Guidance (GCFG) as the foundational theory for our framework. This approach extends Classifier-free Guidance (CFG) to accommodate an arbitrary number of guidances, sourced from a variety of conditions and models. Employing GCFG enables us to separate conditional guidance into two distinct components: concept guidance for fidelity and control guidance for controllability. This division makes it feasible to train a specialized model for concept guidance, while ensuring both control and unconditional guidance remain intact. We then present a null-text Concept-centric Diffusion Model as a concept-specific generator to learn concept guidance without the need for text annotations. Code will be available at https://github.com/PRIV-Creation/Concept-centric-Personalization.

Title: Noise in the reverse process improves the approximation capabilities of diffusion models. (arXiv:2312.07851v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07851
• Code URL: null
• Copy Paste: [[2312.07851]] Noise in the reverse process improves the approximation capabilities of diffusion models(http://arxiv.org/abs/2312.07851)
• Summary:

  In Score based Generative Modeling (SGMs), the state-of-the-art in generative modeling, stochastic reverse processes are known to perform better than their deterministic counterparts. This paper delves into the heart of this phenomenon, comparing neural ordinary differential equations (ODEs) and neural stochastic differential equations (SDEs) as reverse processes. We use a control theoretic perspective by posing the approximation of the reverse process as a trajectory tracking problem. We analyze the ability of neural SDEs to approximate trajectories of the Fokker-Planck equation, revealing the advantages of stochasticity. First, neural SDEs exhibit a powerful regularizing effect, enabling $L^2$ norm trajectory approximation surpassing the Wasserstein metric approximation achieved by neural ODEs under similar conditions, even when the reference vector field or score function is not Lipschitz. Applying this result, we establish the class of distributions that can be sampled using score matching in SGMs, relaxing the Lipschitz requirement on the gradient of the data distribution in existing literature. Second, we show that this approximation property is preserved when network width is limited to the input dimension of the network. In this limited width case, the weights act as control inputs, framing our analysis as a controllability problem for neural SDEs in probability density space. This sheds light on how noise helps to steer the system towards the desired solution and illuminates the empirical success of stochasticity in generative modeling.

Title: Time Series Diffusion Method: A Denoising Diffusion Probabilistic Model for Vibration Signal Generation. (arXiv:2312.07981v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07981
• Code URL: null
• Copy Paste: [[2312.07981]] Time Series Diffusion Method: A Denoising Diffusion Probabilistic Model for Vibration Signal Generation(http://arxiv.org/abs/2312.07981)
• Summary:

  Diffusion models have demonstrated robust data generation capabilities in various research fields. In this paper, a Time Series Diffusion Method (TSDM) is proposed for vibration signal generation, leveraging the foundational principles of diffusion models. The TSDM uses an improved U-net architecture with attention block to effectively segment and extract features from one-dimensional time series data. It operates based on forward diffusion and reverse denoising processes for time-series generation. Experimental validation is conducted using single-frequency, multi-frequency datasets, and bearing fault datasets. The results show that TSDM can accurately generate the single-frequency and multi-frequency features in the time series and retain the basic frequency features for the diffusion generation results of the bearing fault series. Finally, TSDM is applied to the small sample fault diagnosis of three public bearing fault datasets, and the results show that the accuracy of small sample fault diagnosis of the three datasets is improved by 32.380%, 18.355% and 9.298% at most, respectively

Title: SPD-DDPM: Denoising Diffusion Probabilistic Models in the Symmetric Positive Definite Space. (arXiv:2312.08200v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.08200
• Code URL: https://github.com/li-yun-chen/spd-ddpm
• Copy Paste: [[2312.08200]] SPD-DDPM: Denoising Diffusion Probabilistic Models in the Symmetric Positive Definite Space(http://arxiv.org/abs/2312.08200)
• Summary:

  Symmetric positive definite~(SPD) matrices have shown important value and applications in statistics and machine learning, such as FMRI analysis and traffic prediction. Previous works on SPD matrices mostly focus on discriminative models, where predictions are made directly on $E(X|y)$, where $y$ is a vector and $X$ is an SPD matrix. However, these methods are challenging to handle for large-scale data, as they need to access and process the whole data. In this paper, inspired by denoising diffusion probabilistic model~(DDPM), we propose a novel generative model, termed SPD-DDPM, by introducing Gaussian distribution in the SPD space to estimate $E(X|y)$. Moreover, our model is able to estimate $p(X)$ unconditionally and flexibly without giving $y$. On the one hand, the model conditionally learns $p(X|y)$ and utilizes the mean of samples to obtain $E(X|y)$ as a prediction. On the other hand, the model unconditionally learns the probability distribution of the data $p(X)$ and generates samples that conform to this distribution. Furthermore, we propose a new SPD net which is much deeper than the previous networks and allows for the inclusion of conditional factors. Experiment results on toy data and real taxi data demonstrate that our models effectively fit the data distribution both unconditionally and unconditionally and provide accurate predictions.

noise learning

data-free

transformer

Title: Supervised Contrastive Learning for Fine-grained Chromosome Recognition. (arXiv:2312.07623v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07623
• Code URL: null
• Copy Paste: [[2312.07623]] Supervised Contrastive Learning for Fine-grained Chromosome Recognition(http://arxiv.org/abs/2312.07623)
• Summary:

  Chromosome recognition is an essential task in karyotyping, which plays a vital role in birth defect diagnosis and biomedical research. However, existing classification methods face significant challenges due to the inter-class similarity and intra-class variation of chromosomes. To address this issue, we propose a supervised contrastive learning strategy that is tailored to train model-agnostic deep networks for reliable chromosome classification. This method enables extracting fine-grained chromosomal embeddings in latent space. These embeddings effectively expand inter-class boundaries and reduce intra-class variations, enhancing their distinctiveness in predicting chromosome types. On top of two large-scale chromosome datasets, we comprehensively validate the power of our contrastive learning strategy in boosting cutting-edge deep networks such as Transformers and ResNets. Extensive results demonstrate that it can significantly improve models' generalization performance, with an accuracy improvement up to +4.5%. Codes and pretrained models will be released upon acceptance of this work.

Title: Pre-trained Universal Medical Image Transformer. (arXiv:2312.07630v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07630
• Code URL: https://github.com/function2-llx/pumit
• Copy Paste: [[2312.07630]] Pre-trained Universal Medical Image Transformer(http://arxiv.org/abs/2312.07630)
• Summary:

  Self-supervised learning has emerged as a viable method to leverage the abundance of unlabeled medical imaging data, addressing the challenge of labeled data scarcity in medical image analysis. In particular, masked image modeling (MIM) with visual token reconstruction has shown promising results in the general computer vision (CV) domain and serves as a candidate for medical image analysis. However, the presence of heterogeneous 2D and 3D medical images often limits the volume and diversity of training data that can be effectively used for a single model structure. In this work, we propose a spatially adaptive convolution (SAC) module, which adaptively adjusts convolution parameters based on the voxel spacing of the input images. Employing this SAC module, we build a universal visual tokenizer and a universal Vision Transformer (ViT) capable of effectively processing a wide range of medical images with various imaging modalities and spatial properties. Moreover, in order to enhance the robustness of the visual tokenizer's reconstruction objective for MIM, we suggest to generalize the discrete token output of the visual tokenizer to a probabilistic soft token. We show that the generalized soft token representation can be effectively integrated with the prior distribution regularization through a constructive interpretation. As a result, we pre-train a universal visual tokenizer followed by a universal ViT via visual token reconstruction on 55 public medical image datasets, comprising over 9 million 2D slices (including over 48,000 3D images). This represents the largest, most comprehensive, and diverse dataset for pre-training 3D medical image models to our knowledge. Experimental results on downstream medical image classification and segmentation tasks demonstrate the superior performance of our model and improved label efficiency.

Title: Memory-Efficient Reversible Spiking Neural Networks. (arXiv:2312.07922v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07922
• Code URL: https://github.com/mi804/revsnn
• Copy Paste: [[2312.07922]] Memory-Efficient Reversible Spiking Neural Networks(http://arxiv.org/abs/2312.07922)
• Summary:

  Spiking neural networks (SNNs) are potential competitors to artificial neural networks (ANNs) due to their high energy-efficiency on neuromorphic hardware. However, SNNs are unfolded over simulation time steps during the training process. Thus, SNNs require much more memory than ANNs, which impedes the training of deeper SNN models. In this paper, we propose the reversible spiking neural network to reduce the memory cost of intermediate activations and membrane potentials during training. Firstly, we extend the reversible architecture along temporal dimension and propose the reversible spiking block, which can reconstruct the computational graph and recompute all intermediate variables in forward pass with a reverse process. On this basis, we adopt the state-of-the-art SNN models to the reversible variants, namely reversible spiking ResNet (RevSResNet) and reversible spiking transformer (RevSFormer). Through experiments on static and neuromorphic datasets, we demonstrate that the memory cost per image of our reversible SNNs does not increase with the network depth. On CIFAR10 and CIFAR100 datasets, our RevSResNet37 and RevSFormer-4-384 achieve comparable accuracies and consume 3.79x and 3.00x lower GPU memory per image than their counterparts with roughly identical model complexity and parameters. We believe that this work can unleash the memory constraints in SNN training and pave the way for training extremely large and deep SNNs. The code is available at https://github.com/mi804/RevSNN.git.

Title: Mono3DVG: 3D Visual Grounding in Monocular Images. (arXiv:2312.08022v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08022
• Code URL: https://github.com/zhanyang-nwpu/mono3dvg
• Copy Paste: [[2312.08022]] Mono3DVG: 3D Visual Grounding in Monocular Images(http://arxiv.org/abs/2312.08022)
• Summary:

  We introduce a novel task of 3D visual grounding in monocular RGB images using language descriptions with both appearance and geometry information. Specifically, we build a large-scale dataset, Mono3DRefer, which contains 3D object targets with their corresponding geometric text descriptions, generated by ChatGPT and refined manually. To foster this task, we propose Mono3DVG-TR, an end-to-end transformer-based network, which takes advantage of both the appearance and geometry information in text embeddings for multi-modal learning and 3D object localization. Depth predictor is designed to explicitly learn geometry features. The dual text-guided adapter is proposed to refine multiscale visual and geometry features of the referred object. Based on depth-text-visual stacking attention, the decoder fuses object-level geometric cues and visual appearance into a learnable query. Comprehensive benchmarks and some insightful analyses are provided for Mono3DVG. Extensive comparisons and ablation studies show that our method significantly outperforms all baselines. The dataset and code will be publicly available at: https://github.com/ZhanYang-nwpu/Mono3DVG.

Title: Efficient Multi-Object Pose Estimation using Multi-Resolution Deformable Attention and Query Aggregation. (arXiv:2312.08268v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08268
• Code URL: null
• Copy Paste: [[2312.08268]] Efficient Multi-Object Pose Estimation using Multi-Resolution Deformable Attention and Query Aggregation(http://arxiv.org/abs/2312.08268)
• Summary:

  Object pose estimation is a long-standing problem in computer vision. Recently, attention-based vision transformer models have achieved state-of-the-art results in many computer vision applications. Exploiting the permutation-invariant nature of the attention mechanism, a family of vision transformer models formulate multi-object pose estimation as a set prediction problem. However, existing vision transformer models for multi-object pose estimation rely exclusively on the attention mechanism. Convolutional neural networks, on the other hand, hard-wire various inductive biases into their architecture. In this paper, we investigate incorporating inductive biases in vision transformer models for multi-object pose estimation, which facilitates learning long-range dependencies while circumventing the costly global attention. In particular, we use multi-resolution deformable attention, where the attention operation is performed only between a few deformed reference points. Furthermore, we propose a query aggregation mechanism that enables increasing the number of object queries without increasing the computational complexity. We evaluate the proposed model on the challenging YCB-Video dataset and report state-of-the-art results.

Title: VQ-HPS: Human Pose and Shape Estimation in a Vector-Quantized Latent Space. (arXiv:2312.08291v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08291
• Code URL: null
• Copy Paste: [[2312.08291]] VQ-HPS: Human Pose and Shape Estimation in a Vector-Quantized Latent Space(http://arxiv.org/abs/2312.08291)
• Summary:

  Human Pose and Shape Estimation (HPSE) from RGB images can be broadly categorized into two main groups: parametric and non-parametric approaches. Parametric techniques leverage a low-dimensional statistical body model for realistic results, whereas recent non-parametric methods achieve higher precision by directly regressing the 3D coordinates of the human body. Despite their strengths, both approaches face limitations: the parameters of statistical body models pose challenges as regression targets, and predicting 3D coordinates introduces computational complexities and issues related to smoothness. In this work, we take a novel approach to address the HPSE problem. We introduce a unique method involving a low-dimensional discrete latent representation of the human mesh, framing HPSE as a classification task. Instead of predicting body model parameters or 3D vertex coordinates, our focus is on forecasting the proposed discrete latent representation, which can be decoded into a registered human mesh. This innovative paradigm offers two key advantages: firstly, predicting a low-dimensional discrete representation confines our predictions to the space of anthropomorphic poses and shapes; secondly, by framing the problem as a classification task, we can harness the discriminative power inherent in neural networks. Our proposed model, VQ-HPS, a transformer-based architecture, forecasts the discrete latent representation of the mesh, trained through minimizing a cross-entropy loss. Our results demonstrate that VQ-HPS outperforms the current state-of-the-art non-parametric approaches while yielding results as realistic as those produced by parametric methods. This highlights the significant potential of the classification approach for HPSE.

Title: FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects. (arXiv:2312.08344v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08344
• Code URL: null
• Copy Paste: [[2312.08344]] FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects(http://arxiv.org/abs/2312.08344)
• Summary:

  We present FoundationPose, a unified foundation model for 6D object pose estimation and tracking, supporting both model-based and model-free setups. Our approach can be instantly applied at test-time to a novel object without fine-tuning, as long as its CAD model is given, or a small number of reference images are captured. We bridge the gap between these two setups with a neural implicit representation that allows for effective novel view synthesis, keeping the downstream pose estimation modules invariant under the same unified framework. Strong generalizability is achieved via large-scale synthetic training, aided by a large language model (LLM), a novel transformer-based architecture, and contrastive learning formulation. Extensive evaluation on multiple public datasets involving challenging scenarios and objects indicate our unified approach outperforms existing methods specialized for each task by a large margin. In addition, it even achieves comparable results to instance-level methods despite the reduced assumptions. Project page: https://nvlabs.github.io/FoundationPose/

Title: PTT: Point-Trajectory Transformer for Efficient Temporal 3D Object Detection. (arXiv:2312.08371v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08371
• Code URL: null
• Copy Paste: [[2312.08371]] PTT: Point-Trajectory Transformer for Efficient Temporal 3D Object Detection(http://arxiv.org/abs/2312.08371)
• Summary:

  Recent temporal LiDAR-based 3D object detectors achieve promising performance based on the two-stage proposal-based approach. They generate 3D box candidates from the first-stage dense detector, followed by different temporal aggregation methods. However, these approaches require per-frame objects or whole point clouds, posing challenges related to memory bank utilization. Moreover, point clouds and trajectory features are combined solely based on concatenation, which may neglect effective interactions between them. In this paper, we propose a point-trajectory transformer with long short-term memory for efficient temporal 3D object detection. To this end, we only utilize point clouds of current-frame objects and their historical trajectories as input to minimize the memory bank storage requirement. Furthermore, we introduce modules to encode trajectory features, focusing on long short-term and future-aware perspectives, and then effectively aggregate them with point cloud features. We conduct extensive experiments on the large-scale Waymo dataset to demonstrate that our approach performs well against state-of-the-art methods. Code and models will be made publicly available at https://github.com/kuanchihhuang/PTT.

Title: Sentiment analysis in Tourism: Fine-tuning BERT or sentence embeddings concatenation?. (arXiv:2312.07797v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.07797
• Code URL: null
• Copy Paste: [[2312.07797]] Sentiment analysis in Tourism: Fine-tuning BERT or sentence embeddings concatenation?(http://arxiv.org/abs/2312.07797)
• Summary:

  Undoubtedly that the Bidirectional Encoder representations from Transformers is the most powerful technique in making Natural Language Processing tasks such as Named Entity Recognition, Question & Answers or Sentiment Analysis, however, the use of traditional techniques remains a major potential for the improvement of recent models, in particular word tokenization techniques and embeddings, but also the improvement of neural network architectures which are now the core of each architecture. recent. In this paper, we conduct a comparative study between Fine-Tuning the Bidirectional Encoder Representations from Transformers and a method of concatenating two embeddings to boost the performance of a stacked Bidirectional Long Short-Term Memory-Bidirectional Gated Recurrent Units model; these two approaches are applied in the context of sentiment analysis of shopping places in Morocco. A search for the best learning rate was made at the level of the two approaches, and a comparison of the best optimizers was made for each sentence embedding combination with regard to the second approach.

Title: SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention. (arXiv:2312.07987v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07987
• Code URL: https://github.com/robertcsordas/moe_attention
• Copy Paste: [[2312.07987]] SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention(http://arxiv.org/abs/2312.07987)
• Summary:

  The costly self-attention layers in modern Transformers require memory and compute quadratic in sequence length. Existing approximation methods usually underperform and fail to obtain significant speedups in practice. Here we present SwitchHead - a novel method that reduces both compute and memory requirements and achieves wall-clock speedup, while matching the language modeling performance of baseline Transformers with the same parameter budget. SwitchHead uses Mixture-of-Experts (MoE) layers for the value and output projections and requires 4 to 8 times fewer attention matrices than standard Transformers. Our novel attention can also be combined with MoE MLP layers, resulting in an efficient fully-MoE "SwitchHead" Transformer model. Our code is public.

Title: Real-time Network Intrusion Detection via Decision Transformers. (arXiv:2312.07696v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.07696
• Code URL: null
• Copy Paste: [[2312.07696]] Real-time Network Intrusion Detection via Decision Transformers(http://arxiv.org/abs/2312.07696)
• Summary:

  Many cybersecurity problems that require real-time decision-making based on temporal observations can be abstracted as a sequence modeling problem, e.g., network intrusion detection from a sequence of arriving packets. Existing approaches like reinforcement learning may not be suitable for such cybersecurity decision problems, since the Markovian property may not necessarily hold and the underlying network states are often not observable. In this paper, we cast the problem of real-time network intrusion detection as casual sequence modeling and draw upon the power of the transformer architecture for real-time decision-making. By conditioning a causal decision transformer on past trajectories, consisting of the rewards, network packets, and detection decisions, our proposed framework will generate future detection decisions to achieve the desired return. It enables decision transformers to be applied to real-time network intrusion detection, as well as a novel tradeoff between the accuracy and timeliness of detection. The proposed solution is evaluated on public network intrusion detection datasets and outperforms several baseline algorithms using reinforcement learning and sequence modeling, in terms of detection accuracy and timeliness.

Title: Hierarchical Classification of Financial Transactions Through Context-Fusion of Transformer-based Embeddings and Taxonomy-aware Attention Layer. (arXiv:2312.07730v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07730
• Code URL: null
• Copy Paste: [[2312.07730]] Hierarchical Classification of Financial Transactions Through Context-Fusion of Transformer-based Embeddings and Taxonomy-aware Attention Layer(http://arxiv.org/abs/2312.07730)
• Summary:

  This work proposes the Two-headed DragoNet, a Transformer-based model for hierarchical multi-label classification of financial transactions. Our model is based on a stack of Transformers encoder layers that generate contextual embeddings from two short textual descriptors (merchant name and business activity), followed by a Context Fusion layer and two output heads that classify transactions according to a hierarchical two-level taxonomy (macro and micro categories). Finally, our proposed Taxonomy-aware Attention Layer corrects predictions that break categorical hierarchy rules defined in the given taxonomy. Our proposal outperforms classical machine learning methods in experiments of macro-category classification by achieving an F1-score of 93\% on a card dataset and 95% on a current account dataset.

Title: Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approach. (arXiv:2312.07795v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07795
• Code URL: https://github.com/xingshuaihuang/dtlight
• Copy Paste: [[2312.07795]] Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approach(http://arxiv.org/abs/2312.07795)
• Summary:

  Efficient traffic signal control is critical for reducing traffic congestion and improving overall transportation efficiency. The dynamic nature of traffic flow has prompted researchers to explore Reinforcement Learning (RL) for traffic signal control (TSC). Compared with traditional methods, RL-based solutions have shown preferable performance. However, the application of RL-based traffic signal controllers in the real world is limited by the low sample efficiency and high computational requirements of these solutions. In this work, we propose DTLight, a simple yet powerful lightweight Decision Transformer-based TSC method that can learn policy from easily accessible offline datasets. DTLight novelly leverages knowledge distillation to learn a lightweight controller from a well-trained larger teacher model to reduce implementation computation. Additionally, it integrates adapter modules to mitigate the expenses associated with fine-tuning, which makes DTLight practical for online adaptation with minimal computation and only a few fine-tuning steps during real deployment. Moreover, DTLight is further enhanced to be more applicable to real-world TSC problems. Extensive experiments on synthetic and real-world scenarios show that DTLight pre-trained purely on offline datasets can outperform state-of-the-art online RL-based methods in most scenarios. Experiment results also show that online fine-tuning further improves the performance of DTLight by up to 42.6% over the best online RL baseline methods. In this work, we also introduce Datasets specifically designed for TSC with offline RL (referred to as DTRL). Our datasets and code are publicly available.

Title: Invariant Graph Transformer. (arXiv:2312.07859v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07859
• Code URL: null
• Copy Paste: [[2312.07859]] Invariant Graph Transformer(http://arxiv.org/abs/2312.07859)
• Summary:

  Rationale discovery is defined as finding a subset of the input data that maximally supports the prediction of downstream tasks. In graph machine learning context, graph rationale is defined to locate the critical subgraph in the given graph topology, which fundamentally determines the prediction results. In contrast to the rationale subgraph, the remaining subgraph is named the environment subgraph. Graph rationalization can enhance the model performance as the mapping between the graph rationale and prediction label is viewed as invariant, by assumption. To ensure the discriminative power of the extracted rationale subgraphs, a key technique named "intervention" is applied. The core idea of intervention is that given any changing environment subgraphs, the semantics from the rationale subgraph is invariant, which guarantees the correct prediction result. However, most, if not all, of the existing rationalization works on graph data develop their intervention strategies on the graph level, which is coarse-grained. In this paper, we propose well-tailored intervention strategies on graph data. Our idea is driven by the development of Transformer models, whose self-attention module provides rich interactions between input nodes. Based on the self-attention module, our proposed invariant graph Transformer (IGT) can achieve fine-grained, more specifically, node-level and virtual node-level intervention. Our comprehensive experiments involve 7 real-world datasets, and the proposed IGT shows significant performance advantages compared to 13 baseline methods.

generative

Title: Understanding (Un)Intended Memorization in Text-to-Image Generative Models. (arXiv:2312.07550v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07550
• Code URL: null
• Copy Paste: [[2312.07550]] Understanding (Un)Intended Memorization in Text-to-Image Generative Models(http://arxiv.org/abs/2312.07550)
• Summary:

  Multimodal machine learning, especially text-to-image models like Stable Diffusion and DALL-E 3, has gained significance for transforming text into detailed images.

  Despite their growing use and remarkable generative capabilities, there is a pressing need for a detailed examination of these models' behavior, particularly with respect to memorization. Historically, memorization in machine learning has been context-dependent, with diverse definitions emerging from classification tasks to complex models like Large Language Models (LLMs) and Diffusion models. Yet, a definitive concept of memorization that aligns with the intricacies of text-to-image synthesis remains elusive. This understanding is vital as memorization poses privacy risks yet is essential for meeting user expectations, especially when generating representations of underrepresented entities. In this paper, we introduce a specialized definition of memorization tailored to text-to-image models, categorizing it into three distinct types according to user expectations. We closely examine the subtle distinctions between intended and unintended memorization, emphasizing the importance of balancing user privacy with the generative quality of the model outputs. Using the Stable Diffusion model, we offer examples to validate our memorization definitions and clarify their application.

Title: A Foundational Multimodal Vision Language AI Assistant for Human Pathology. (arXiv:2312.07814v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07814
• Code URL: null
• Copy Paste: [[2312.07814]] A Foundational Multimodal Vision Language AI Assistant for Human Pathology(http://arxiv.org/abs/2312.07814)
• Summary:

  The field of computational pathology has witnessed remarkable progress in the development of both task-specific predictive models and task-agnostic self-supervised vision encoders. However, despite the explosive growth of generative artificial intelligence (AI), there has been limited study on building general purpose, multimodal AI assistants tailored to pathology. Here we present PathChat, a vision-language generalist AI assistant for human pathology using an in-house developed foundational vision encoder pretrained on 100 million histology images from over 100,000 patient cases and 1.18 million pathology image-caption pairs. The vision encoder is then combined with a pretrained large language model and the whole system is finetuned on over 250,000 diverse disease agnostic visual language instructions. We compare PathChat against several multimodal vision language AI assistants as well as GPT4V, which powers the commercially available multimodal general purpose AI assistant ChatGPT-4. When relevant clinical context is provided with the histology image, PathChat achieved a diagnostic accuracy of 87% on multiple-choice questions based on publicly available cases of diverse tissue origins and disease models. Additionally, using open-ended questions and human expert evaluation, we found that overall PathChat produced more accurate and pathologist-preferable responses to diverse queries related to pathology. As an interactive and general vision language AI assistant that can flexibly handle both visual and natural language inputs, PathChat can potentially find impactful applications in pathology education, research, and human-in-the-loop clinical decision making.

Title: Stable Rivers: A Case Study in the Application of Text-to-Image Generative Models for Earth Sciences. (arXiv:2312.07833v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07833
• Code URL: null
• Copy Paste: [[2312.07833]] Stable Rivers: A Case Study in the Application of Text-to-Image Generative Models for Earth Sciences(http://arxiv.org/abs/2312.07833)
• Summary:

  Text-to-image (TTI) generative models can be used to generate photorealistic images from a given text-string input. These models offer great potential to mitigate challenges to the uptake of machine learning in the earth sciences. However, the rapid increase in their use has raised questions about fairness and biases, with most research to-date focusing on social and cultural areas rather than domain-specific considerations. We conducted a case study for the earth sciences, focusing on the field of fluvial geomorphology, where we evaluated subject-area specific biases in the training data and downstream model performance of Stable Diffusion (v1.5). In addition to perpetuating Western biases, we found that the training data over-represented scenic locations, such as famous rivers and waterfalls, and showed serious under- and over-representation of many morphological and environmental terms. Despite biased training data, we found that with careful prompting, the Stable Diffusion model was able to generate photorealistic synthetic river images reproducing many important environmental and morphological characteristics. Furthermore, conditional control techniques, such as the use of condition maps with ControlNet were effective for providing additional constraints on output images. Despite great potential for the use of TTI models in the earth sciences field, we advocate for caution in sensitive applications, and advocate for domain-specific reviews of training data and image generation biases to mitigate perpetuation of existing biases.

Title: 3DGEN: A GAN-based approach for generating novel 3D models from image data. (arXiv:2312.08094v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08094
• Code URL: null
• Copy Paste: [[2312.08094]] 3DGEN: A GAN-based approach for generating novel 3D models from image data(http://arxiv.org/abs/2312.08094)
• Summary:

  The recent advances in text and image synthesis show a great promise for the future of generative models in creative fields. However, a less explored area is the one of 3D model generation, with a lot of potential applications to game design, video production, and physical product design. In our paper, we present 3DGEN, a model that leverages the recent work on both Neural Radiance Fields for object reconstruction and GAN-based image generation. We show that the proposed architecture can generate plausible meshes for objects of the same category as the training images and compare the resulting meshes with the state-of-the-art baselines, leading to visible uplifts in generation quality.

Title: A Compact and Semantic Latent Space for Disentangled and Controllable Image Editing. (arXiv:2312.08256v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08256
• Code URL: null
• Copy Paste: [[2312.08256]] A Compact and Semantic Latent Space for Disentangled and Controllable Image Editing(http://arxiv.org/abs/2312.08256)
• Summary:

  Recent advances in the field of generative models and in particular generative adversarial networks (GANs) have lead to substantial progress for controlled image editing, especially compared with the pre-deep learning era. Despite their powerful ability to apply realistic modifications to an image, these methods often lack properties like disentanglement (the capacity to edit attributes independently). In this paper, we propose an auto-encoder which re-organizes the latent space of StyleGAN, so that each attribute which we wish to edit corresponds to an axis of the new latent space, and furthermore that the latent axes are decorrelated, encouraging disentanglement. We work in a compressed version of the latent space, using Principal Component Analysis, meaning that the parameter complexity of our autoencoder is reduced, leading to short training times ($\sim$ 45 mins). Qualitative and quantitative results demonstrate the editing capabilities of our approach, with greater disentanglement than competing methods, while maintaining fidelity to the original image with respect to identity. Our autoencoder architecture simple and straightforward, facilitating implementation.

Title: PaperQA: Retrieval-Augmented Generative Agent for Scientific Research. (arXiv:2312.07559v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.07559
• Code URL: null
• Copy Paste: [[2312.07559]] PaperQA: Retrieval-Augmented Generative Agent for Scientific Research(http://arxiv.org/abs/2312.07559)
• Summary:

  Large Language Models (LLMs) generalize well across language tasks, but suffer from hallucinations and uninterpretability, making it difficult to assess their accuracy without ground-truth. Retrieval-Augmented Generation (RAG) models have been proposed to reduce hallucinations and provide provenance for how an answer was generated. Applying such models to the scientific literature may enable large-scale, systematic processing of scientific knowledge. We present PaperQA, a RAG agent for answering questions over the scientific literature. PaperQA is an agent that performs information retrieval across full-text scientific articles, assesses the relevance of sources and passages, and uses RAG to provide answers. Viewing this agent as a question answering model, we find it exceeds performance of existing LLMs and LLM agents on current science QA benchmarks. To push the field closer to how humans perform research on scientific literature, we also introduce LitQA, a more complex benchmark that requires retrieval and synthesis of information from full-text scientific papers across the literature. Finally, we demonstrate PaperQA's matches expert human researchers on LitQA.

Title: Evaluating ChatGPT as a Question Answering System: A Comprehensive Analysis and Comparison with Existing Models. (arXiv:2312.07592v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.07592
• Code URL: null
• Copy Paste: [[2312.07592]] Evaluating ChatGPT as a Question Answering System: A Comprehensive Analysis and Comparison with Existing Models(http://arxiv.org/abs/2312.07592)
• Summary:

  In the current era, a multitude of language models has emerged to cater to user inquiries. Notably, the GPT-3.5 Turbo language model has gained substantial attention as the underlying technology for ChatGPT. Leveraging extensive parameters, this model adeptly responds to a wide range of questions. However, due to its reliance on internal knowledge, the accuracy of responses may not be absolute. This article scrutinizes ChatGPT as a Question Answering System (QAS), comparing its performance to other existing QASs. The primary focus is on evaluating ChatGPT's proficiency in extracting responses from provided paragraphs, a core QAS capability. Additionally, performance comparisons are made in scenarios without a surrounding passage. Multiple experiments, exploring response hallucination and considering question complexity, were conducted on ChatGPT. Evaluation employed well-known Question Answering (QA) datasets, including SQuAD, NewsQA, and PersianQuAD, across English and Persian languages. Metrics such as F-score, exact match, and accuracy were employed in the assessment. The study reveals that, while ChatGPT demonstrates competence as a generative model, it is less effective in question answering compared to task-specific models. Providing context improves its performance, and prompt engineering enhances precision, particularly for questions lacking explicit answers in provided paragraphs. ChatGPT excels at simpler factual questions compared to "how" and "why" question types. The evaluation highlights occurrences of hallucinations, where ChatGPT provides responses to questions without available answers in the provided context.

Title: Synthetic Data: Can We Trust Statistical Estimators?. (arXiv:2312.07837v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07837
• Code URL: https://github.com/syndara-lab/inferential-utility-workshop
• Copy Paste: [[2312.07837]] Synthetic Data: Can We Trust Statistical Estimators?(http://arxiv.org/abs/2312.07837)
• Summary:

  The increasing interest in data sharing makes synthetic data appealing. However, the analysis of synthetic data raises a unique set of methodological challenges. In this work, we highlight the importance of inferential utility and provide empirical evidence against naive inference from synthetic data (that handles these as if they were really observed). We argue that the rate of false-positive findings (type 1 error) will be unacceptably high, even when the estimates are unbiased. One of the reasons is the underestimation of the true standard error, which may even progressively increase with larger sample sizes due to slower convergence. This is especially problematic for deep generative models. Before publishing synthetic data, it is essential to develop statistical inference tools for such data.

large language model

Title: CoIE: Chain-of-Instruct Editing for Multi-Attribute Face Manipulation. (arXiv:2312.07879v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07879
• Code URL: null
• Copy Paste: [[2312.07879]] CoIE: Chain-of-Instruct Editing for Multi-Attribute Face Manipulation(http://arxiv.org/abs/2312.07879)
• Summary:

  Current text-to-image editing models often encounter challenges with smoothly manipulating multiple attributes using a single instruction. Taking inspiration from the Chain-of-Thought prompting technique utilized in language models, we present an innovative concept known as Chain-of-Instruct Editing (CoIE), which enhances the capabilities of these models through step-by-step editing using a series of instructions. In particular, in the context of face manipulation, we leverage the contextual learning abilities of a pretrained Large Language Model (LLM), such as GPT-4, to generate a sequence of instructions from the original input, utilizing a purpose-designed 1-shot template. To further improve the precision of each editing step, we conduct fine-tuning on the editing models using our self-constructed instruction-guided face editing dataset, Instruct-CelebA. And additionally, we incorporate a super-resolution module to mitigate the adverse effects of editability and quality degradation. Experimental results across various challenging cases confirm the significant boost in multi-attribute facial image manipulation using chain-of-instruct editing. This is evident in enhanced editing success rates, measured by CLIPSim and Coverage metrics, improved by 17.86% and 85.45% respectively, and heightened controllability indicated by Preserve L1 and Quality metrics, improved by 11.58% and 4.93% respectively.

Title: Chat-3D v2: Bridging 3D Scene and Large Language Models with Object Identifiers. (arXiv:2312.08168v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08168
• Code URL: https://github.com/chat-3d/chat-3d-v2
• Copy Paste: [[2312.08168]] Chat-3D v2: Bridging 3D Scene and Large Language Models with Object Identifiers(http://arxiv.org/abs/2312.08168)
• Summary:

  Recent research has evidenced the significant potentials of Large Language Models (LLMs) in handling challenging tasks within 3D scenes. However, current models are constrained to addressing object-centric tasks, where each question-answer pair focuses solely on an individual object. In real-world applications, users may pose queries involving multiple objects or expect for answers that precisely reference various objects. We introduce the use of object identifiers to freely reference objects during a conversation. While this solution appears straightforward, it presents two main challenges: 1) How to establish a reliable one-to-one correspondence between each object and its identifier? 2) How to incorporate complex spatial relationships among dozens of objects into the embedding space of the LLM? To address these challenges, we propose a two-stage alignment method, which involves learning an attribute-aware token and a relation-aware token for each object. These tokens capture the object's attributes and spatial relationships with surrounding objects in the 3D scene. Once the alignment is established, we can fine-tune our model on various downstream tasks using instruction tuning. Experiments conducted on traditional datasets like ScanQA, ScanRefer, and Nr3D/Sr3D showcase the effectiveness of our proposed method. Additionally, we create a 3D scene captioning dataset annotated with rich object identifiers, with the assistant of GPT-4. This dataset aims to further explore the capability of object identifiers in effective object referencing and precise scene understanding.

Title: LD-SDM: Language-Driven Hierarchical Species Distribution Modeling. (arXiv:2312.08334v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08334
• Code URL: null
• Copy Paste: [[2312.08334]] LD-SDM: Language-Driven Hierarchical Species Distribution Modeling(http://arxiv.org/abs/2312.08334)
• Summary:

  We focus on the problem of species distribution modeling using global-scale presence-only data. Most previous studies have mapped the range of a given species using geographical and environmental features alone. To capture a stronger implicit relationship between species, we encode the taxonomic hierarchy of species using a large language model. This enables range mapping for any taxonomic rank and unseen species without additional supervision. Further, we propose a novel proximity-aware evaluation metric that enables evaluating species distribution models using any pixel-level representation of ground-truth species range map. The proposed metric penalizes the predictions of a model based on its proximity to the ground truth. We describe the effectiveness of our model by systematically evaluating on the task of species range prediction, zero-shot prediction and geo-feature regression against the state-of-the-art. Results show our model outperforms the strong baselines when trained with a variety of multi-label learning losses.

Title: Large Language Models for Intent-Driven Session Recommendations. (arXiv:2312.07552v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.07552
• Code URL: https://github.com/llm4sr/po4isr
• Copy Paste: [[2312.07552]] Large Language Models for Intent-Driven Session Recommendations(http://arxiv.org/abs/2312.07552)
• Summary:

  Intent-aware session recommendation (ISR) is pivotal in discerning user intents within sessions for precise predictions. Traditional approaches, however, face limitations due to their presumption of a uniform number of intents across all sessions. This assumption overlooks the dynamic nature of user sessions, where the number and type of intentions can significantly vary. In addition, these methods typically operate in latent spaces, thus hinder the model's transparency.Addressing these challenges, we introduce a novel ISR approach, utilizing the advanced reasoning capabilities of large language models (LLMs). First, this approach begins by generating an initial prompt that guides LLMs to predict the next item in a session, based on the varied intents manifested in user sessions. Then, to refine this process, we introduce an innovative prompt optimization mechanism that iteratively self-reflects and adjusts prompts. Furthermore, our prompt selection module, built upon the LLMs' broad adaptability, swiftly selects the most optimized prompts across diverse domains. This new paradigm empowers LLMs to discern diverse user intents at a semantic level, leading to more accurate and interpretable session recommendations. Our extensive experiments on three real-world datasets demonstrate the effectiveness of our method, marking a significant advancement in ISR systems.

Title: Can LLM find the green circle? Investigation and Human-guided tool manipulation for compositional generalization. (arXiv:2312.07763v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.07763
• Code URL: null
• Copy Paste: [[2312.07763]] Can LLM find the green circle? Investigation and Human-guided tool manipulation for compositional generalization(http://arxiv.org/abs/2312.07763)
• Summary:

  The meaning of complex phrases in natural language is composed of their individual components. The task of compositional generalization evaluates a model's ability to understand new combinations of components. Previous studies trained smaller, task-specific models, which exhibited poor generalization. While large language models (LLMs) exhibit impressive generalization abilities on many tasks through in-context learning (ICL), their potential for compositional generalization remains unexplored. In this paper, we first empirically investigate prevailing ICL methods in compositional generalization. We find that they struggle with complex compositional questions due to cumulative errors in long reasoning steps and intricate logic required for tool-making. Consequently, we propose a human-guided tool manipulation framework (HTM) that generates tools for sub-questions and integrates multiple tools. Our method enhances the effectiveness of tool creation and usage with minimal human effort. Experiments show that our method achieves state-of-the-art performance on two compositional generalization benchmarks and outperforms existing methods on the most challenging test split by 70%.

Title: Native Language Identification with Large Language Models. (arXiv:2312.07819v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.07819
• Code URL: null
• Copy Paste: [[2312.07819]] Native Language Identification with Large Language Models(http://arxiv.org/abs/2312.07819)
• Summary:

  We present the first experiments on Native Language Identification (NLI) using LLMs such as GPT-4. NLI is the task of predicting a writer's first language by analyzing their writings in a second language, and is used in second language acquisition and forensic linguistics. Our results show that GPT models are proficient at NLI classification, with GPT-4 setting a new performance record of 91.7% on the benchmark TOEFL11 test set in a zero-shot setting. We also show that unlike previous fully-supervised settings, LLMs can perform NLI without being limited to a set of known classes, which has practical implications for real-world applications. Finally, we also show that LLMs can provide justification for their choices, providing reasoning based on spelling errors, syntactic patterns, and usage of directly translated linguistic patterns.

Title: Finetuning an LLM on Contextual Knowledge of Classics for Q&A. (arXiv:2312.07848v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.07848
• Code URL: null
• Copy Paste: [[2312.07848]] Finetuning an LLM on Contextual Knowledge of Classics for Q&A(http://arxiv.org/abs/2312.07848)
• Summary:

  The open-source publishing of large language models (LLMs) has created many possibilities for how anyone who understands language and has access to a computer can interact with significant tools of artificial intelligence, particularly in the context of learning and knowledge dissemination. However, the utility of these models in specialized fields like Classics is still largely unexplored. This project is an attempt to merge the knowledge of Classics with the capabilities of artificial intelligence by finetuning an LLM to cater to the specific needs of learners and professionals. The goal of this project is to develop an LLM that not only reproduces contextual knowledge accurately but also exhibits a consistent "personality" - and, indeed, has consistent propriety - to appeal to a diverse audience who possess differing levels of knowledge. A significant portion of this project was dedicated to refining the dataset, following the principle of "garbage in, garbage out," to ensure the model generates relevant, useful, and creative responses when given a prompt (a statement, question, or single word). After training and evaluation, my model's ability to handle a vast array of different types of inputs and prompting exceeded expectations for a 355M parameter model, though its occasional hallucinations (especially when set with a high temperature), particularly in its assertions about historical events or its own identity, make it seem somewhat capricious and more work in the form of continuous finetuning will be undertaken.

Title: CBQ: Cross-Block Quantization for Large Language Models. (arXiv:2312.07950v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07950
• Code URL: null
• Copy Paste: [[2312.07950]] CBQ: Cross-Block Quantization for Large Language Models(http://arxiv.org/abs/2312.07950)
• Summary:

  Post-training quantization (PTQ) has driven attention to producing efficient large language models (LLMs) with ultra-low costs. Since hand-craft quantization parameters lead to low performance in low-bit quantization, recent methods optimize the quantization parameters through block-wise reconstruction between the floating-point and quantized models. However, these methods suffer from two challenges: accumulated errors from independent one-by-one block quantization and reconstruction difficulties from extreme weight and activation outliers. To address these two challenges, we propose CBQ, a cross-block reconstruction-based PTQ method for LLMs. To reduce error accumulation, we introduce a cross-block dependency with the aid of a homologous reconstruction scheme to build the long-range dependency between adjacent multi-blocks with overlapping. To reduce reconstruction difficulty, we design a coarse-to-fine pre-processing (CFP) to truncate weight outliers and dynamically scale activation outliers before optimization, and an adaptive rounding scheme, called LoRA-Rounding, with two low-rank learnable matrixes to further rectify weight quantization errors. Extensive experiments demonstrate that: (1) CBQ pushes both activation and weight quantization to low-bit settings W4A4, W4A8, and W2A16. (2) CBQ achieves better performance than the existing state-of-the-art methods on various LLMs and benchmark datasets.

Title: Helping Language Models Learn More: Multi-dimensional Task Prompt for Few-shot Tuning. (arXiv:2312.08027v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.08027
• Code URL: null
• Copy Paste: [[2312.08027]] Helping Language Models Learn More: Multi-dimensional Task Prompt for Few-shot Tuning(http://arxiv.org/abs/2312.08027)
• Summary:

  Large language models (LLMs) can be used as accessible and intelligent chatbots by constructing natural language queries and directly inputting the prompt into the large language model. However, different prompt' constructions often lead to uncertainty in the answers and thus make it hard to utilize the specific knowledge of LLMs (like ChatGPT). To alleviate this, we use an interpretable structure to explain the prompt learning principle in LLMs, which certificates that the effectiveness of language models is determined by position changes of the task's related tokens. Therefore, we propose MTPrompt, a multi-dimensional task prompt learning method consisting based on task-related object, summary, and task description information. By automatically building and searching for appropriate prompts, our proposed MTPrompt achieves the best results on few-shot samples setting and five different datasets. In addition, we demonstrate the effectiveness and stability of our method in different experimental settings and ablation experiments. In interaction with large language models, embedding more task-related information into prompts will make it easier to stimulate knowledge embedded in large language models.

Title: Conceptualizing Suicidal Behavior: Utilizing Explanations of Predicted Outcomes to Analyze Longitudinal Social Media Data. (arXiv:2312.08299v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.08299
• Code URL: https://github.com/fit-suicide-prevention-research/token-attribution-analysis
• Copy Paste: [[2312.08299]] Conceptualizing Suicidal Behavior: Utilizing Explanations of Predicted Outcomes to Analyze Longitudinal Social Media Data(http://arxiv.org/abs/2312.08299)
• Summary:

  The COVID-19 pandemic has escalated mental health crises worldwide, with social isolation and economic instability contributing to a rise in suicidal behavior. Suicide can result from social factors such as shame, abuse, abandonment, and mental health conditions like depression, Post-Traumatic Stress Disorder (PTSD), Attention-Deficit/Hyperactivity Disorder (ADHD), anxiety disorders, and bipolar disorders. As these conditions develop, signs of suicidal ideation may manifest in social media interactions. Analyzing social media data using artificial intelligence (AI) techniques can help identify patterns of suicidal behavior, providing invaluable insights for suicide prevention agencies, professionals, and broader community awareness initiatives. Machine learning algorithms for this purpose require large volumes of accurately labeled data. Previous research has not fully explored the potential of incorporating explanations in analyzing and labeling longitudinal social media data. In this study, we employed a model explanation method, Layer Integrated Gradients, on top of a fine-tuned state-of-the-art language model, to assign each token from Reddit users' posts an attribution score for predicting suicidal ideation. By extracting and analyzing attributions of tokens from the data, we propose a methodology for preliminary screening of social media posts for suicidal ideation without using large language models during inference.

Title: Efficient Toxic Content Detection by Bootstrapping and Distilling Large Language Models. (arXiv:2312.08303v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.08303
• Code URL: null
• Copy Paste: [[2312.08303]] Efficient Toxic Content Detection by Bootstrapping and Distilling Large Language Models(http://arxiv.org/abs/2312.08303)
• Summary:

  Toxic content detection is crucial for online services to remove inappropriate content that violates community standards. To automate the detection process, prior works have proposed varieties of machine learning (ML) approaches to train Language Models (LMs) for toxic content detection. However, both their accuracy and transferability across datasets are limited. Recently, Large Language Models (LLMs) have shown promise in toxic content detection due to their superior zero-shot and few-shot in-context learning ability as well as broad transferability on ML tasks. However, efficiently designing prompts for LLMs remains challenging. Moreover, the high run-time cost of LLMs may hinder their deployments in production. To address these challenges, in this work, we propose BD-LLM, a novel and efficient approach to Bootstrapping and Distilling LLMs for toxic content detection. Specifically, we design a novel prompting method named Decision-Tree-of-Thought (DToT) to bootstrap LLMs' detection performance and extract high-quality rationales. DToT can automatically select more fine-grained context to re-prompt LLMs when their responses lack confidence. Additionally, we use the rationales extracted via DToT to fine-tune student LMs. Our experimental results on various datasets demonstrate that DToT can improve the accuracy of LLMs by up to 4.6%. Furthermore, student LMs fine-tuned with rationales extracted via DToT outperform baselines on all datasets with up to 16.9\% accuracy improvement, while being more than 60x smaller than conventional LLMs. Finally, we observe that student LMs fine-tuned with rationales exhibit better cross-dataset transferability.

Title: Distributed Inference and Fine-tuning of Large Language Models Over The Internet. (arXiv:2312.08361v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.08361
• Code URL: null
• Copy Paste: [[2312.08361]] Distributed Inference and Fine-tuning of Large Language Models Over The Internet(http://arxiv.org/abs/2312.08361)
• Summary:

  Large language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion parameters. However, using these 50B+ models requires high-end hardware, making them inaccessible to most researchers. In this work, we investigate methods for cost-efficient inference and fine-tuning of LLMs, comparing local and distributed strategies. We observe that a large enough model (50B+) can run efficiently even on geodistributed devices in a consumer-grade network. This could allow running LLM efficiently by pooling together idle compute resources of multiple research groups and volunteers. We address two open problems: (1) how to perform inference and fine-tuning reliably if any device can disconnect abruptly and (2) how to partition LLMs between devices with uneven hardware, joining and leaving at will. In order to do that, we develop special fault-tolerant inference algorithms and load-balancing protocols that automatically assign devices to maximize the total system throughput. We showcase these algorithms in Petals - a decentralized system that runs Llama 2 (70B) and BLOOM (176B) over the Internet up to 10x faster than offloading for interactive generation. We evaluate the performance of our system in simulated conditions and a real-world setup spanning two continents.

Title: An Invitation to Deep Reinforcement Learning. (arXiv:2312.08365v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.08365
• Code URL: null
• Copy Paste: [[2312.08365]] An Invitation to Deep Reinforcement Learning(http://arxiv.org/abs/2312.08365)
• Summary:

  Training a deep neural network to maximize a target objective has become the standard recipe for successful machine learning over the last decade. These networks can be optimized with supervised learning, if the target objective is differentiable. For many interesting problems, this is however not the case. Common objectives like intersection over union (IoU), bilingual evaluation understudy (BLEU) score or rewards cannot be optimized with supervised learning. A common workaround is to define differentiable surrogate losses, leading to suboptimal solutions with respect to the actual objective. Reinforcement learning (RL) has emerged as a promising alternative for optimizing deep neural networks to maximize non-differentiable objectives in recent years. Examples include aligning large language models via human feedback, code generation, object detection or control problems. This makes RL techniques relevant to the larger machine learning audience. The subject is, however, time intensive to approach due to the large range of methods, as well as the often very theoretical presentation. In this introduction, we take an alternative approach, different from classic reinforcement learning textbooks. Rather than focusing on tabular problems, we introduce reinforcement learning as a generalization of supervised learning, which we first apply to non-differentiable objectives and later to temporal problems. Assuming only basic knowledge of supervised learning, the reader will be able to understand state-of-the-art deep RL algorithms like proximal policy optimization (PPO) after reading this tutorial.

segmentation

Title: DFGET: Displacement-Field Assisted Graph Energy Transmitter for Gland Instance Segmentation. (arXiv:2312.07584v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07584
• Code URL: null
• Copy Paste: [[2312.07584]] DFGET: Displacement-Field Assisted Graph Energy Transmitter for Gland Instance Segmentation(http://arxiv.org/abs/2312.07584)
• Summary:

  Gland instance segmentation is an essential but challenging task in the diagnosis and treatment of adenocarcinoma. The existing models usually achieve gland instance segmentation through multi-task learning and boundary loss constraint. However, how to deal with the problems of gland adhesion and inaccurate boundary in segmenting the complex samples remains a challenge. In this work, we propose a displacement-field assisted graph energy transmitter (DFGET) framework to solve these problems. Specifically, a novel message passing manner based on anisotropic diffusion is developed to update the node features, which can distinguish the isomorphic graphs and improve the expressivity of graph nodes for complex samples. Using such graph framework, the gland semantic segmentation map and the displacement field (DF) of the graph nodes are estimated with two graph network branches. With the constraint of DF, a graph cluster module based on diffusion theory is presented to improve the intra-class feature consistency and inter-class feature discrepancy, as well as to separate the adherent glands from the semantic segmentation maps. Extensive comparison and ablation experiments on the GlaS dataset demonstrate the superiority of DFGET and effectiveness of the proposed anisotropic message passing manner and clustering method. Compared to the best comparative model, DFGET increases the object-Dice and object-F1 score by 2.5% and 3.4% respectively, while decreases the object-HD by 32.4%, achieving state-of-the-art performance.

Title: Go beyond End-to-End Training: Boosting Greedy Local Learning with Context Supply. (arXiv:2312.07636v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.07636
• Code URL: https://github.com/tab-ct/contsup
• Copy Paste: [[2312.07636]] Go beyond End-to-End Training: Boosting Greedy Local Learning with Context Supply(http://arxiv.org/abs/2312.07636)
• Summary:

  Traditional end-to-end (E2E) training of deep networks necessitates storing intermediate activations for back-propagation, resulting in a large memory footprint on GPUs and restricted model parallelization. As an alternative, greedy local learning partitions the network into gradient-isolated modules and trains supervisely based on local preliminary losses, thereby providing asynchronous and parallel training methods that substantially reduce memory cost. However, empirical experiments reveal that as the number of segmentations of the gradient-isolated module increases, the performance of the local learning scheme degrades substantially, severely limiting its expansibility. To avoid this issue, we theoretically analyze the greedy local learning from the standpoint of information theory and propose a ContSup scheme, which incorporates context supply between isolated modules to compensate for information loss. Experiments on benchmark datasets (i.e. CIFAR, SVHN, STL-10) achieve SOTA results and indicate that our proposed method can significantly improve the performance of greedy local learning with minimal memory and computational overhead, allowing for the boost of the number of isolated modules. Our codes are available at https://github.com/Tab-ct/ContSup.

Title: CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor. (arXiv:2312.07661v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07661
• Code URL: null
• Copy Paste: [[2312.07661]] CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor(http://arxiv.org/abs/2312.07661)
• Summary:

  Existing open-vocabulary image segmentation methods require a fine-tuning step on mask annotations and/or image-text datasets. Mask labels are labor-intensive, which limits the number of categories in segmentation datasets. As a result, the open-vocabulary capacity of pre-trained VLMs is severely reduced after fine-tuning. However, without fine-tuning, VLMs trained under weak image-text supervision tend to make suboptimal mask predictions when there are text queries referring to non-existing concepts in the image. To alleviate these issues, we introduce a novel recurrent framework that progressively filters out irrelevant texts and enhances mask quality without training efforts. The recurrent unit is a two-stage segmenter built upon a VLM with frozen weights. Thus, our model retains the VLM's broad vocabulary space and strengthens its segmentation capability. Experimental results show that our method outperforms not only the training-free counterparts, but also those fine-tuned with millions of additional data samples, and sets new state-of-the-art records for both zero-shot semantic and referring image segmentation tasks. Specifically, we improve the current record by 28.8, 16.0, and 6.9 mIoU on Pascal VOC, COCO Object, and Pascal Context.

Title: Automated Behavioral Analysis Using Instance Segmentation. (arXiv:2312.07723v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07723
• Code URL: https://github.com/cplab/annolid
• Copy Paste: [[2312.07723]] Automated Behavioral Analysis Using Instance Segmentation(http://arxiv.org/abs/2312.07723)
• Summary:

  Animal behavior analysis plays a crucial role in various fields, such as life science and biomedical research. However, the scarcity of available data and the high cost associated with obtaining a large number of labeled datasets pose significant challenges. In this research, we propose a novel approach that leverages instance segmentation-based transfer learning to address these issues. By capitalizing on fine-tuning the classification head of the instance segmentation network, we enable the tracking of multiple animals and facilitate behavior analysis in laboratory-recorded videos. To demonstrate the effectiveness of our method, we conducted a series of experiments, revealing that our approach achieves exceptional performance levels, comparable to human capabilities, across a diverse range of animal behavior analysis tasks. Moreover, we emphasize the practicality of our solution, as it requires only a small number of labeled images for training. To facilitate the adoption and further development of our method, we have developed an open-source implementation named Annolid (An annotation and instance segmentation-based multiple animal tracking and behavior analysis package). The codebase is publicly available on GitHub at https://github.com/cplab/annolid. This resource serves as a valuable asset for researchers and practitioners interested in advancing animal behavior analysis through state-of-the-art techniques.

Title: Data-Dependent Higher-Order Clique Selection for Artery-Vein Segmentation by Energy Minimization. (arXiv:2312.07860v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07860
• Code URL: null
• Copy Paste: [[2312.07860]] Data-Dependent Higher-Order Clique Selection for Artery-Vein Segmentation by Energy Minimization(http://arxiv.org/abs/2312.07860)
• Summary:

  We propose a novel segmentation method based on energy minimization of higher-order potentials. We introduce higher-order terms into the energy to incorporate prior knowledge on the shape of the segments. The terms encourage certain sets of pixels to be entirely in one segment or the other. The sets can for instance be smooth curves in order to help delineate pulmonary vessels, which are known to run in almost straight lines. The higher-order terms can be converted to submodular first-order terms by adding auxiliary variables, which can then be globally minimized using graph cuts. We also determine the weight of these terms, or the degree of the aforementioned encouragement, in a principled way by learning from training data with the ground truth. We demonstrate the effectiveness of the method in a real-world application in fully-automatic pulmonary artery-vein segmentation in CT images.

Title: Polar-Doc: One-Stage Document Dewarping with Multi-Scope Constraints under Polar Representation. (arXiv:2312.07925v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07925
• Code URL: null
• Copy Paste: [[2312.07925]] Polar-Doc: One-Stage Document Dewarping with Multi-Scope Constraints under Polar Representation(http://arxiv.org/abs/2312.07925)
• Summary:

  Document dewarping, aiming to eliminate geometric deformation in photographed documents to benefit text recognition, has made great progress in recent years but is still far from being solved. While Cartesian coordinates are typically leveraged by state-of-the-art approaches to learn a group of deformation control points, such representation is not efficient for dewarping model to learn the deformation information. In this work, we explore Polar coordinates representation for each point in document dewarping, namely Polar-Doc. In contrast to most current works adopting a two-stage pipeline typically, Polar representation enables a unified point regression framework for both segmentation and dewarping network in one single stage. Such unification makes the whole model more efficient to learn under an end-to-end optimization pipeline, and also obtains a compact representation. Furthermore, we propose a novel multi-scope Polar-Doc-IOU loss to constrain the relationship among control points as a grid-based regularization under the Polar representation. Visual comparisons and quantitative experiments on two benchmarks show that, with much fewer parameters than the other mainstream counterparts, our one-stage model with multi-scope constraints achieves new state-of-the-art performance on both pixel alignment metrics and OCR metrics. Source codes will be available at \url{*****}.

Title: Comparing YOLOv8 and Mask RCNN for object segmentation in complex orchard environments. (arXiv:2312.07935v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07935
• Code URL: null
• Copy Paste: [[2312.07935]] Comparing YOLOv8 and Mask RCNN for object segmentation in complex orchard environments(http://arxiv.org/abs/2312.07935)
• Summary:

  Instance segmentation, an important image processing operation for automation in agriculture, is used to precisely delineate individual objects of interest within images, which provides foundational information for various automated or robotic tasks such as selective harvesting and precision pruning. This study compares the one-stage YOLOv8 and the two-stage Mask R-CNN machine learning models for instance segmentation under varying orchard conditions across two datasets. Dataset 1, collected in dormant season, includes images of dormant apple trees, which were used to train multi-object segmentation models delineating tree branches and trunks. Dataset 2, collected in the early growing season, includes images of apple tree canopies with green foliage and immature (green) apples (also called fruitlet), which were used to train single-object segmentation models delineating only immature green apples. The results showed that YOLOv8 performed better than Mask R-CNN, achieving good precision and near-perfect recall across both datasets at a confidence threshold of 0.5. Specifically, for Dataset 1, YOLOv8 achieved a precision of 0.90 and a recall of 0.95 for all classes. In comparison, Mask R-CNN demonstrated a precision of 0.81 and a recall of 0.81 for the same dataset. With Dataset 2, YOLOv8 achieved a precision of 0.93 and a recall of 0.97. Mask R-CNN, in this single-class scenario, achieved a precision of 0.85 and a recall of 0.88. Additionally, the inference times for YOLOv8 were 10.9 ms for multi-class segmentation (Dataset 1) and 7.8 ms for single-class segmentation (Dataset 2), compared to 15.6 ms and 12.8 ms achieved by Mask R-CNN's, respectively.

Title: ASLseg: Adapting SAM in the Loop for Semi-supervised Liver Tumor Segmentation. (arXiv:2312.07969v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.07969
• Code URL: null
• Copy Paste: [[2312.07969]] ASLseg: Adapting SAM in the Loop for Semi-supervised Liver Tumor Segmentation(http://arxiv.org/abs/2312.07969)
• Summary:

  Liver tumor segmentation is essential for computer-aided diagnosis, surgical planning, and prognosis evaluation. However, obtaining and maintaining a large-scale dataset with dense annotations is challenging. Semi-Supervised Learning (SSL) is a common technique to address these challenges. Recently, Segment Anything Model (SAM) has shown promising performance in some medical image segmentation tasks, but it performs poorly for liver tumor segmentation. In this paper, we propose a novel semi-supervised framework, named ASLseg, which can effectively adapt the SAM to the SSL setting and combine both domain-specific and general knowledge of liver tumors. Specifically, the segmentation model trained with a specific SSL paradigm provides the generated pseudo-labels as prompts to the fine-tuned SAM. An adaptation network is then used to refine the SAM-predictions and generate higher-quality pseudo-labels. Finally, the reliable pseudo-labels are selected to expand the labeled set for iterative training. Extensive experiments on the LiTS dataset demonstrate overwhelming performance of our ASLseg.

Title: Unveiling Parts Beyond Objects:Towards Finer-Granularity Referring Expression Segmentation. (arXiv:2312.08007v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08007
• Code URL: null
• Copy Paste: [[2312.08007]] Unveiling Parts Beyond Objects:Towards Finer-Granularity Referring Expression Segmentation(http://arxiv.org/abs/2312.08007)
• Summary:

  Referring expression segmentation (RES) aims at segmenting the foreground masks of the entities that match the descriptive natural language expression. Previous datasets and methods for classic RES task heavily rely on the prior assumption that one expression must refer to object-level targets. In this paper, we take a step further to finer-grained part-level RES task. To promote the object-level RES task towards finer-grained vision-language understanding, we put forward a new multi-granularity referring expression segmentation (MRES) task and construct an evaluation benchmark called RefCOCOm by manual annotations. By employing our automatic model-assisted data engine, we build the largest visual grounding dataset namely MRES-32M, which comprises over 32.2M high-quality masks and captions on the provided 1M images. Besides, a simple yet strong model named UniRES is designed to accomplish the unified object-level and part-level grounding task. Extensive experiments on our RefCOCOm for MRES and three datasets (i.e., RefCOCO(+/g) for classic RES task demonstrate the superiority of our method over previous state-of-the-art methods. To foster future research into fine-grained visual grounding, our benchmark RefCOCOm, the MRES-32M dataset and model UniRES will be publicly available at https://github.com/Rubics-Xuan/MRES

Title: Advanced Image Segmentation Techniques for Neural Activity Detection via C-fos Immediate Early Gene Expression. (arXiv:2312.08177v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08177
• Code URL: https://github.com/dystopians/cfoscraft
• Copy Paste: [[2312.08177]] Advanced Image Segmentation Techniques for Neural Activity Detection via C-fos Immediate Early Gene Expression(http://arxiv.org/abs/2312.08177)
• Summary:

  This paper investigates the application of advanced image segmentation techniques to analyze C-fos immediate early gene expression, a crucial marker for neural activity. Due to the complexity and high variability of neural circuits, accurate segmentation of C-fos images is paramount for the development of new insights into neural function. Amidst this backdrop, this research aims to improve accuracy and minimize manual intervention in C-fos image segmentation by leveraging the capabilities of CNNs and the Unet model. We describe the development of a novel workflow for the segmentation process involving Convolutional Neural Networks (CNNs) and the Unet model, demonstrating their efficiency in various image segmentation tasks. Our workflow incorporates pre-processing steps such as cropping, image feature extraction, and clustering for the training dataset selection. We used an AutoEncoder model to extract features and implement constrained clustering to identify similarities and differences in image types. Additionally, we utilized manual and automatic labeling approaches to enhance the performance of our model. We demonstrated the effectiveness of our method in distinguishing areas with significant C-fos expression from normal tissue areas. Lastly, we implemented a modified Unet network for the detection of C-fos expressions. This research contributes to the development of more efficient and automated image segmentation methods, advancing the understanding of neural function in neuroscience research.

Title: Beyond the Label Itself: Latent Labels Enhance Semi-supervised Point Cloud Panoptic Segmentation. (arXiv:2312.08234v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08234
• Code URL: null
• Copy Paste: [[2312.08234]] Beyond the Label Itself: Latent Labels Enhance Semi-supervised Point Cloud Panoptic Segmentation(http://arxiv.org/abs/2312.08234)
• Summary:

  As the exorbitant expense of labeling autopilot datasets and the growing trend of utilizing unlabeled data, semi-supervised segmentation on point clouds becomes increasingly imperative. Intuitively, finding out more ``unspoken words'' (i.e., latent instance information) beyond the label itself should be helpful to improve performance. In this paper, we discover two types of latent labels behind the displayed label embedded in LiDAR and image data. First, in the LiDAR Branch, we propose a novel augmentation, Cylinder-Mix, which is able to augment more yet reliable samples for training. Second, in the Image Branch, we propose the Instance Position-scale Learning (IPSL) Module to learn and fuse the information of instance position and scale, which is from a 2D pre-trained detector and a type of latent label obtained from 3D to 2D projection. Finally, the two latent labels are embedded into the multi-modal panoptic segmentation network. The ablation of the IPSL module demonstrates its robust adaptability, and the experiments evaluated on SemanticKITTI and nuScenes demonstrate that our model outperforms the state-of-the-art method, LaserMix.

Title: PnPNet: Pull-and-Push Networks for Volumetric Segmentation with Boundary Confusion. (arXiv:2312.08323v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08323
• Code URL: null
• Copy Paste: [[2312.08323]] PnPNet: Pull-and-Push Networks for Volumetric Segmentation with Boundary Confusion(http://arxiv.org/abs/2312.08323)
• Summary:

  Precise boundary segmentation of volumetric images is a critical task for image-guided diagnosis and computer-assisted intervention, especially for boundary confusion in clinical practice. However, U-shape networks cannot effectively resolve this challenge due to the lack of boundary shape constraints. Besides, existing methods of refining boundaries overemphasize the slender structure, which results in the overfitting phenomenon due to networks' limited abilities to model tiny objects. In this paper, we reconceptualize the mechanism of boundary generation by encompassing the interaction dynamics with adjacent regions. Moreover, we propose a unified network termed PnPNet to model shape characteristics of the confused boundary region. Core ingredients of PnPNet contain the pushing and pulling branches. Specifically, based on diffusion theory, we devise the semantic difference module (SDM) from the pushing branch to squeeze the boundary region. Explicit and implicit differential information inside SDM significantly boost representation abilities for inter-class boundaries. Additionally, motivated by the K-means algorithm, the class clustering module (CCM) from the pulling branch is introduced to stretch the intersected boundary region. Thus, pushing and pulling branches will shrink and enlarge the boundary uncertainty respectively. They furnish two adversarial forces to promote models to output a more precise delineation of boundaries. We carry out experiments on three challenging public datasets and one in-house dataset, containing three types of boundary confusion in model predictions. Experimental results demonstrate the superiority of PnPNet over other segmentation networks, especially on evaluation metrics of HD and ASSD. Besides, pushing and pulling branches can serve as plug-and-play modules to enhance classic U-shape baseline models. Codes are available.

Title: See, Say, and Segment: Teaching LMMs to Overcome False Premises. (arXiv:2312.08366v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08366
• Code URL: null
• Copy Paste: [[2312.08366]] See, Say, and Segment: Teaching LMMs to Overcome False Premises(http://arxiv.org/abs/2312.08366)
• Summary:

  Current open-source Large Multimodal Models (LMMs) excel at tasks such as open-vocabulary language grounding and segmentation but can suffer under false premises when queries imply the existence of something that is not actually present in the image. We observe that existing methods that fine-tune an LMM to segment images significantly degrade their ability to reliably determine ("see") if an object is present and to interact naturally with humans ("say"), a form of catastrophic forgetting. In this work, we propose a cascading and joint training approach for LMMs to solve this task, avoiding catastrophic forgetting of previous skills. Our resulting model can "see" by detecting whether objects are present in an image, "say" by telling the user if they are not, proposing alternative queries or correcting semantic errors in the query, and finally "segment" by outputting the mask of the desired objects if they exist. Additionally, we introduce a novel False Premise Correction benchmark dataset, an extension of existing RefCOCO(+/g) referring segmentation datasets (which we call FP-RefCOCO(+/g)). The results show that our method not only detects false premises up to 55% better than existing approaches, but under false premise conditions produces relative cIOU improvements of more than 31% over baselines, and produces natural language feedback judged helpful up to 67% of the time.

Title: SAM-guided Graph Cut for 3D Instance Segmentation. (arXiv:2312.08372v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.08372
• Code URL: null
• Copy Paste: [[2312.08372]] SAM-guided Graph Cut for 3D Instance Segmentation(http://arxiv.org/abs/2312.08372)
• Summary:

  This paper addresses the challenge of 3D instance segmentation by simultaneously leveraging 3D geometric and multi-view image information. Many previous works have applied deep learning techniques to 3D point clouds for instance segmentation. However, these methods often failed to generalize to various types of scenes due to the scarcity and low-diversity of labeled 3D point cloud data. Some recent works have attempted to lift 2D instance segmentations to 3D within a bottom-up framework. The inconsistency in 2D instance segmentations among views can substantially degrade the performance of 3D segmentation. In this work, we introduce a novel 3D-to-2D query framework to effectively exploit 2D segmentation models for 3D instance segmentation. Specifically, we pre-segment the scene into several superpoints in 3D, formulating the task into a graph cut problem. The superpoint graph is constructed based on 2D segmentation models, where node features are obtained from multi-view image features and edge weights are computed based on multi-view segmentation results, enabling the better generalization ability. To process the graph, we train a graph neural network using pseudo 3D labels from 2D segmentation models. Experimental results on the ScanNet, ScanNet++ and KITTI-360 datasets demonstrate that our method achieves robust segmentation performance and can generalize across different types of scenes. Our project page is available at https://zju3dv.github.io/sam_graph.

Title: Arabic Handwritten Text Line Dataset. (arXiv:2312.07573v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.07573
• Code URL: null
• Copy Paste: [[2312.07573]] Arabic Handwritten Text Line Dataset(http://arxiv.org/abs/2312.07573)
• Summary:

  Segmentation of Arabic manuscripts into lines of text and words is an important step to make recognition systems more efficient and accurate. The problem of segmentation into text lines is solved since there are carefully annotated dataset dedicated to this task. However, To the best of our knowledge, there are no dataset annotating the word position of Arabic texts. In this paper, we present a new dataset specifically designed for historical Arabic script in which we annotate position in word level.