secure

Title: HoneyEVSE: An Honeypot to emulate Electric Vehicle Supply Equipments. (arXiv:2309.06077v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06077
• Code URL: null
• Copy Paste: [[2309.06077]] HoneyEVSE: An Honeypot to emulate Electric Vehicle Supply Equipments(http://arxiv.org/abs/2309.06077)
• Summary:

  To fight climate change, new "green" technology are emerging, most of them using electricity as a power source. Among the solutions, Electric Vehicles (EVs) represent a central asset in the future transport system. EVs require a complex infrastructure to enable the so-called Vehicle-to-Grid (V2G) paradigm to manage the charging process between the smart grid and the EV. In this paradigm, the Electric Vehicle Supply Equipment (EVSE), or charging station, is the end device that authenticates the vehicle and delivers the power to charge it. However, since an EVSE is publicly exposed and connected to the Internet, recent works show how an attacker with physical tampering and remote access can target an EVSE, exposing the security of the entire infrastructure and the final user. For this reason, it is important to develop novel strategies to secure such infrastructures. In this paper we present HoneyEVSE, the first honeypot conceived to simulate an EVSE. HoneyEVSE can simulate with high fidelity the EV charging process and, at the same time, enables a user to interact with it through a dashboard. Furthermore, based on other charging columns exposed on the Internet, we emulate the login and device information pages to increase user engagement. We exposed HoneyEVSE for 30 days to the Internet to assess its capability and measured the interaction received with its Shodan Honeyscore. Results show that HoneyEVSE can successfully evade the Shodan honeyscore metric while attracting a high number of interactions on the exposed services.

security

Title: Quality-Agnostic Deepfake Detection with Intra-model Collaborative Learning. (arXiv:2309.05911v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05911
• Code URL: https://bitbucket.org/deepfake-project/qad-iccv23
• Copy Paste: [[2309.05911]] Quality-Agnostic Deepfake Detection with Intra-model Collaborative Learning(http://arxiv.org/abs/2309.05911)
• Summary:

  Deepfake has recently raised a plethora of societal concerns over its possible security threats and dissemination of fake information. Much research on deepfake detection has been undertaken. However, detecting low quality as well as simultaneously detecting different qualities of deepfakes still remains a grave challenge. Most SOTA approaches are limited by using a single specific model for detecting certain deepfake video quality type. When constructing multiple models with prior information about video quality, this kind of strategy incurs significant computational cost, as well as model and training data overhead. Further, it cannot be scalable and practical to deploy in real-world settings. In this work, we propose a universal intra-model collaborative learning framework to enable the effective and simultaneous detection of different quality of deepfakes. That is, our approach is the quality-agnostic deepfake detection method, dubbed QAD . In particular, by observing the upper bound of general error expectation, we maximize the dependency between intermediate representations of images from different quality levels via Hilbert-Schmidt Independence Criterion. In addition, an Adversarial Weight Perturbation module is carefully devised to enable the model to be more robust against image corruption while boosting the overall model's performance. Extensive experiments over seven popular deepfake datasets demonstrate the superiority of our QAD model over prior SOTA benchmarks.

Title: Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testing. (arXiv:2309.05679v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.05679
• Code URL: https://github.com/jenniferho97/xai-trend-test
• Copy Paste: [[2309.05679]] Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testing(http://arxiv.org/abs/2309.05679)
• Summary:

  While enjoying the great achievements brought by deep learning (DL), people are also worried about the decision made by DL models, since the high degree of non-linearity of DL models makes the decision extremely difficult to understand. Consequently, attacks such as adversarial attacks are easy to carry out, but difficult to detect and explain, which has led to a boom in the research on local explanation methods for explaining model decisions. In this paper, we evaluate the faithfulness of explanation methods and find that traditional tests on faithfulness encounter the random dominance problem, \ie, the random selection performs the best, especially for complex data. To further solve this problem, we propose three trend-based faithfulness tests and empirically demonstrate that the new trend tests can better assess faithfulness than traditional tests on image, natural language and security tasks. We implement the assessment system and evaluate ten popular explanation methods. Benefiting from the trend tests, we successfully assess the explanation methods on complex data for the first time, bringing unprecedented discoveries and inspiring future research. Downstream tasks also greatly benefit from the tests. For example, model debugging equipped with faithful explanation methods performs much better for detecting and correcting accuracy and security problems.

Title: REVERSIM: A Game-Based Approach to Accessing Large Populations for Studying Human Aspects in Hardware Reverse Engineering. (arXiv:2309.05740v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.05740
• Code URL: null
• Copy Paste: [[2309.05740]] REVERSIM: A Game-Based Approach to Accessing Large Populations for Studying Human Aspects in Hardware Reverse Engineering(http://arxiv.org/abs/2309.05740)
• Summary:

  Hardware Reverse Engineering (HRE) is a technique for analyzing Integrated Circuits (ICs). Experts employ HRE for various security-critical tasks, such as design verification or the detection of intellectual property violations. However, HRE also enables threat actors to subvert the security of an IC. Previous studies have shown that analysts rely heavily on their cognitive abilities to perform HRE as no fully automated solutions exist. Therefore, conducting controlled experimental studies to assess the cognitive processes involved in HRE could open new avenues for hardware protection. However, researchers have faced the methodological challenge that HRE experts are largely unavailable for such empirical research. To address this scarcity, we have developed REVERSIM, a game-based simulation that mimics realistic HRE subprocesses and is specifically designed to require no prior knowledge. To support these claims, we conducted two empirical studies: First, we performed semi-structured interviews with 14 professionals and researchers from the HRE domain, who attested to the comparability of REVERSIM to real-world HRE problems. Second, we conducted a user study involving 89 non-expert participants, demonstrating that participants could engage in the simulation without prior knowledge in HRE or related domains. Finally, we outline several research directions for experiments with REVERSIM, highlighting its potential in advancing HRE research.

Title: Systemization of Knowledge (SoK)- Cross Impact of Transfer Learning in Cybersecurity: Offensive, Defensive and Threat Intelligence Perspectives. (arXiv:2309.05889v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.05889
• Code URL: null
• Copy Paste: [[2309.05889]] Systemization of Knowledge (SoK)- Cross Impact of Transfer Learning in Cybersecurity: Offensive, Defensive and Threat Intelligence Perspectives(http://arxiv.org/abs/2309.05889)
• Summary:

  Recent literature highlights a significant cross-impact between transfer learning and cybersecurity. Many studies have been conducted on using transfer learning to enhance security, leading to various applications in different cybersecurity tasks. However, previous research is focused on specific areas of cybersecurity. This paper presents a comprehensive survey of transfer learning applications in cybersecurity by covering a wide range of domains, identifying current trends, and shedding light on under-explored areas. The survey highlights the significance of transfer learning in addressing critical issues in cybersecurity, such as improving detection accuracy, reducing training time, handling data imbalance, and enhancing privacy preservation. Additional insights are provided on the common problems solved using transfer learning, such as the lack of labeled data, different data distributions, and privacy concerns. The paper identifies future research directions and challenges that require community attention, including the need for privacy-preserving models, automatic tools for knowledge transfer, metrics for measuring domain relatedness, and enhanced privacy preservation mechanisms. The insights and roadmap presented in this paper will guide researchers in further advancing transfer learning in cybersecurity, fostering the development of robust and efficient cybersecurity systems to counter emerging threats and protect sensitive information. To the best of our knowledge, this paper is the first of its kind to present a comprehensive taxonomy of all areas of cybersecurity that benefited from transfer learning and propose a detailed future roadmap to shape the possible research direction in this area.

Title: Behind The Wings: The Case of Reverse Engineering and Drone Hijacking in DJI Enhanced Wi-Fi Protocol. (arXiv:2309.05913v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.05913
• Code URL: null
• Copy Paste: [[2309.05913]] Behind The Wings: The Case of Reverse Engineering and Drone Hijacking in DJI Enhanced Wi-Fi Protocol(http://arxiv.org/abs/2309.05913)
• Summary:

  This research paper entails an examination of the Enhanced Wi-Fi protocol, focusing on its control command reverse-engineering analysis and subsequent demonstration of a hijacking attack. Our investigation discovered vulnerabilities in the Enhanced Wi-Fi control commands, rendering them susceptible to hijacking attacks. Notably, the study established that even readily available and cost-effective commercial off-the-shelf Wi-Fi routers could be leveraged as effective tools for executing such attacks. To illustrate this vulnerability, a proof-of-concept remote hijacking attack was carried out on a DJI Mini SE drone, whereby we intercepted the control commands to manipulate the drone's flight trajectory. The findings of this research emphasize the critical necessity of implementing robust security measures to safeguard unmanned aerial vehicles against potential hijacking threats. Considering that civilian drones are now used as war weapons, the study underscores the urgent need for further exploration and advancement in the domain of civilian drone security.

Title: Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Review. (arXiv:2309.06055v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06055
• Code URL: null
• Copy Paste: [[2309.06055]] Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Review(http://arxiv.org/abs/2309.06055)
• Summary:

  Deep Neural Networks (DNNs) have led to unprecedented progress in various natural language processing (NLP) tasks. Owing to limited data and computation resources, using third-party data and models has become a new paradigm for adapting various tasks. However, research shows that it has some potential security vulnerabilities because attackers can manipulate the training process and data source. Such a way can set specific triggers, making the model exhibit expected behaviors that have little inferior influence on the model's performance for primitive tasks, called backdoor attacks. Hence, it could have dire consequences, especially considering that the backdoor attack surfaces are broad.

  To get a precise grasp and understanding of this problem, a systematic and comprehensive review is required to confront various security challenges from different phases and attack purposes. Additionally, there is a dearth of analysis and comparison of the various emerging backdoor countermeasures in this situation.In this paper, we conduct a timely review of backdoor attacks and countermeasures to sound the red alarm for the NLP security community. According to the affected stage of the machine learning pipeline, the attack surfaces are recognized to be wide and then formalized into three categorizations: attacking pre-trained model with fine-tuning (APMF) or prompt-tuning (APMP), and attacking final model with training (AFMT), where AFMT can be subdivided into different attack aims. Thus, attacks under each categorization are combed. The countermeasures are categorized into two general classes: sample inspection and model inspection. Overall, the research on the defense side is far behind the attack side, and there is no single defense that can prevent all types of backdoor attacks. An attacker can intelligently bypass existing defenses with a more invisible attack. ......

Title: A Machine Learning Framework to Deconstruct the Primary Drivers for Electricity Market Price Events. (arXiv:2309.06082v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06082
• Code URL: null
• Copy Paste: [[2309.06082]] A Machine Learning Framework to Deconstruct the Primary Drivers for Electricity Market Price Events(http://arxiv.org/abs/2309.06082)
• Summary:

  Power grids are moving towards 100% renewable energy source bulk power grids, and the overall dynamics of power system operations and electricity markets are changing. The electricity markets are not only dispatching resources economically but also taking into account various controllable actions like renewable curtailment, transmission congestion mitigation, and energy storage optimization to ensure grid reliability. As a result, price formations in electricity markets have become quite complex. Traditional root cause analysis and statistical approaches are rendered inapplicable to analyze and infer the main drivers behind price formation in the modern grid and markets with variable renewable energy (VRE). In this paper, we propose a machine learning-based analysis framework to deconstruct the primary drivers for price spike events in modern electricity markets with high renewable energy. The outcomes can be utilized for various critical aspects of market design, renewable dispatch and curtailment, operations, and cyber-security applications. The framework can be applied to any ISO or market data; however, in this paper, it is applied to open-source publicly available datasets from California Independent System Operator (CAISO) and ISO New England (ISO-NE).

privacy

Title: Generalized Rainbow Differential Privacy. (arXiv:2309.05871v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.05871
• Code URL: null
• Copy Paste: [[2309.05871]] Generalized Rainbow Differential Privacy(http://arxiv.org/abs/2309.05871)
• Summary:

  We study a new framework for designing differentially private (DP) mechanisms via randomized graph colorings, called rainbow differential privacy. In this framework, datasets are nodes in a graph, and two neighboring datasets are connected by an edge. Each dataset in the graph has a preferential ordering for the possible outputs of the mechanism, and these orderings are called rainbows. Different rainbows partition the graph of connected datasets into different regions. We show that if a DP mechanism at the boundary of such regions is fixed and it behaves identically for all same-rainbow boundary datasets, then a unique optimal $(\epsilon,\delta)$-DP mechanism exists (as long as the boundary condition is valid) and can be expressed in closed-form. Our proof technique is based on an interesting relationship between dominance ordering and DP, which applies to any finite number of colors and for $(\epsilon,\delta)$-DP, improving upon previous results that only apply to at most three colors and for $\epsilon$-DP. We justify the homogeneous boundary condition assumption by giving an example with non-homogeneous boundary condition, for which there exists no optimal DP mechanism.

Title: Concurrent Composition for Interactive Differential Privacy with Adaptive Privacy-Loss Parameters. (arXiv:2309.05901v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.05901
• Code URL: null
• Copy Paste: [[2309.05901]] Concurrent Composition for Interactive Differential Privacy with Adaptive Privacy-Loss Parameters(http://arxiv.org/abs/2309.05901)
• Summary:

  In this paper, we study the concurrent composition of interactive mechanisms with adaptively chosen privacy-loss parameters. In this setting, the adversary can interleave queries to existing interactive mechanisms, as well as create new ones. We prove that every valid privacy filter and odometer for noninteractive mechanisms extends to the concurrent composition of interactive mechanisms if privacy loss is measured using $(\epsilon, \delta)$-DP, $f$-DP, or R\'enyi DP of fixed order. Our results offer strong theoretical foundations for enabling full adaptivity in composing differentially private interactive mechanisms, showing that concurrency does not affect the privacy guarantees. We also provide an implementation for users to deploy in practice.

Title: Verifiable Fairness: Privacy-preserving Computation of Fairness for Machine Learning Systems. (arXiv:2309.06061v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06061
• Code URL: null
• Copy Paste: [[2309.06061]] Verifiable Fairness: Privacy-preserving Computation of Fairness for Machine Learning Systems(http://arxiv.org/abs/2309.06061)
• Summary:

  Fair machine learning is a thriving and vibrant research topic. In this paper, we propose Fairness as a Service (FaaS), a secure, verifiable and privacy-preserving protocol to computes and verify the fairness of any machine learning (ML) model. In the deisgn of FaaS, the data and outcomes are represented through cryptograms to ensure privacy. Also, zero knowledge proofs guarantee the well-formedness of the cryptograms and underlying data. FaaS is model--agnostic and can support various fairness metrics; hence, it can be used as a service to audit the fairness of any ML model. Our solution requires no trusted third party or private channels for the computation of the fairness metric. The security guarantees and commitments are implemented in a way that every step is securely transparent and verifiable from the start to the end of the process. The cryptograms of all input data are publicly available for everyone, e.g., auditors, social activists and experts, to verify the correctness of the process. We implemented FaaS to investigate performance and demonstrate the successful use of FaaS for a publicly available data set with thousands of entries.

Title: Systematic Evaluation of Geolocation Privacy Mechanisms. (arXiv:2309.06263v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06263
• Code URL: null
• Copy Paste: [[2309.06263]] Systematic Evaluation of Geolocation Privacy Mechanisms(http://arxiv.org/abs/2309.06263)
• Summary:

  Location data privacy has become a serious concern for users as Location Based Services (LBSs) have become an important part of their life. It is possible for malicious parties having access to geolocation data to learn sensitive information about the user such as religion or political views. Location Privacy Preserving Mechanisms (LPPMs) have been proposed by previous works to ensure the privacy of the shared data while allowing the users to use LBSs. But there is no clear view of which mechanism to use according to the scenario in which the user makes use of a LBS. The scenario is the way the user is using a LBS (frequency of reports, number of reports). In this paper, we study the sensitivity of LPPMs on the scenario on which they are used. We propose a framework to systematically evaluate LPPMs by considering an exhaustive combination of LPPMs, attacks and metrics. Using our framework we compare a selection of LPPMs including an improved mechanism that we introduce. By evaluating over a variety of scenarios, we find that the efficacy (privacy, utility, and robustness) of the studied mechanisms is dependent on the scenario: for example the privacy of Planar Laplace geo-indistinguishability is greatly reduced in a continuous scenario. We show that the scenario is essential to consider when choosing an obfuscation mechanism for a given application.

protect

Title: CToMP: A Cycle-task-oriented Memory Protection Scheme for Unmanned Systems. (arXiv:2309.05978v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.05978
• Code URL: null
• Copy Paste: [[2309.05978]] CToMP: A Cycle-task-oriented Memory Protection Scheme for Unmanned Systems(http://arxiv.org/abs/2309.05978)
• Summary:

  Memory corruption attacks (MCAs) refer to malicious behaviors of system intruders that modify the contents of a memory location to disrupt the normal operation of computing systems, causing leakage of sensitive data or perturbations to ongoing processes. Unlike general-purpose systems, unmanned systems cannot deploy complete security protection schemes, due to their limitations in size, cost and performance. MCAs in unmanned systems are particularly difficult to defend against. Furthermore, MCAs have diverse and unpredictable attack interfaces in unmanned systems, severely impacting digital and physical sectors. In this paper, we first generalize, model and taxonomize MCAs found in unmanned systems currently, laying the foundation for designing a portable and general defense approach. According to different attack mechanisms, we found that MCAs are mainly categorized into two types--return2libc and return2shellcode. To tackle return2libc attacks, we model the erratic operation of unmanned systems with cycles and then propose a cycle-task-oriented memory protection (CToMP) approach to protect control flows from tampering. To defend against return2shellcode attacks, we introduce a secure process stack with a randomized memory address by leveraging the memory pool to prevent Shellcode from being executed. Moreover, we discuss the mechanism by which CToMP resists the ROP attack, a novel variant of return2libc attacks. Finally, we implement CToMP on CUAV V5+ with Ardupilot and Crazyflie. The evaluation and security analysis results demonstrate that the proposed approach CToMP is resilient to various MCAs in unmanned systems with low footprints and system overhead.

defense

attack

Title: Generalized Attacks on Face Verification Systems. (arXiv:2309.05879v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.05879
• Code URL: null
• Copy Paste: [[2309.05879]] Generalized Attacks on Face Verification Systems(http://arxiv.org/abs/2309.05879)
• Summary:

  Face verification (FV) using deep neural network models has made tremendous progress in recent years, surpassing human accuracy and seeing deployment in various applications such as border control and smartphone unlocking. However, FV systems are vulnerable to Adversarial Attacks, which manipulate input images to deceive these systems in ways usually unnoticeable to humans. This paper provides an in-depth study of attacks on FV systems. We introduce the DodgePersonation Attack that formulates the creation of face images that impersonate a set of given identities while avoiding being identified as any of the identities in a separate, disjoint set. A taxonomy is proposed to provide a unified view of different types of Adversarial Attacks against FV systems, including Dodging Attacks, Impersonation Attacks, and Master Face Attacks. Finally, we propose the ''One Face to Rule Them All'' Attack which implements the DodgePersonation Attack with state-of-the-art performance on a well-known scenario (Master Face Attack) and which can also be used for the new scenarios introduced in this paper. While the state-of-the-art Master Face Attack can produce a set of 9 images to cover 43.82% of the identities in their test database, with 9 images our attack can cover 57.27% to 58.5% of these identifies while giving the attacker the choice of the identity to use to create the impersonation. Moreover, the 9 generated attack images appear identical to a casual observer.

Title: Adversarial Attacks Assessment of Salient Object Detection via Symbolic Learning. (arXiv:2309.05900v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05900
• Code URL: null
• Copy Paste: [[2309.05900]] Adversarial Attacks Assessment of Salient Object Detection via Symbolic Learning(http://arxiv.org/abs/2309.05900)
• Summary:

  Machine learning is at the center of mainstream technology and outperforms classical approaches to handcrafted feature design. Aside from its learning process for artificial feature extraction, it has an end-to-end paradigm from input to output, reaching outstandingly accurate results. However, security concerns about its robustness to malicious and imperceptible perturbations have drawn attention since its prediction can be changed entirely. Salient object detection is a research area where deep convolutional neural networks have proven effective but whose trustworthiness represents a significant issue requiring analysis and solutions to hackers' attacks. Brain programming is a kind of symbolic learning in the vein of good old-fashioned artificial intelligence. This work provides evidence that symbolic learning robustness is crucial in designing reliable visual attention systems since it can withstand even the most intense perturbations. We test this evolutionary computation methodology against several adversarial attacks and noise perturbations using standard databases and a real-world problem of a shorebird called the Snowy Plover portraying a visual attention task. We compare our methodology with five different deep learning approaches, proving that they do not match the symbolic paradigm regarding robustness. All neural networks suffer significant performance losses, while brain programming stands its ground and remains unaffected. Also, by studying the Snowy Plover, we remark on the importance of security in surveillance activities regarding wildlife protection and conservation.

Title: Exploring Non-additive Randomness on ViT against Query-Based Black-Box Attacks. (arXiv:2309.06438v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06438
• Code URL: null
• Copy Paste: [[2309.06438]] Exploring Non-additive Randomness on ViT against Query-Based Black-Box Attacks(http://arxiv.org/abs/2309.06438)
• Summary:

  Deep Neural Networks can be easily fooled by small and imperceptible perturbations. The query-based black-box attack (QBBA) is able to create the perturbations using model output probabilities of image queries requiring no access to the underlying models. QBBA poses realistic threats to real-world applications. Recently, various types of robustness have been explored to defend against QBBA. In this work, we first taxonomize the stochastic defense strategies against QBBA. Following our taxonomy, we propose to explore non-additive randomness in models to defend against QBBA. Specifically, we focus on underexplored Vision Transformers based on their flexible architectures. Extensive experiments show that the proposed defense approach achieves effective defense, without much sacrifice in performance.

Title: Random Segmentation: New Traffic Obfuscation against Packet-Size-Based Side-Channel Attacks. (arXiv:2309.05941v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.05941
• Code URL: null
• Copy Paste: [[2309.05941]] Random Segmentation: New Traffic Obfuscation against Packet-Size-Based Side-Channel Attacks(http://arxiv.org/abs/2309.05941)
• Summary:

  Despite encryption, the packet size is still visible, enabling observers to infer private information in the Internet of Things (IoT) environment (e.g., IoT device identification). Packet padding obfuscates packet-length characteristics with a high data overhead because it relies on adding noise to the data. This paper proposes a more data-efficient approach that randomizes packet sizes without adding noise. We achieve this by splitting large TCP segments into random-sized chunks; hence, the packet length distribution is obfuscated without adding noise data. Our client-server implementation using TCP sockets demonstrates the feasibility of our approach at the application level. We realize our packet size control by adjusting two local socket-programming parameters. First, we enable the TCP_NODELAY option to send out each packet with our specified length. Second, we downsize the sending buffer to prevent the sender from pushing out more data than can be received, which could disable our control of the packet sizes. We simulate our defense on a network trace of four IoT devices and show a reduction in device classification accuracy from 98% to 63%, close to random guessing. Meanwhile, the real-world data transmission experiments show that the added latency is reasonable, less than 21%, while the added packet header overhead is only about 5%.

Title: Unveiling Signle-Bit-Flip Attacks on DNN Executables. (arXiv:2309.06223v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.06223
• Code URL: null
• Copy Paste: [[2309.06223]] Unveiling Signle-Bit-Flip Attacks on DNN Executables(http://arxiv.org/abs/2309.06223)
• Summary:

  Recent research has shown that bit-flip attacks (BFAs) can manipulate deep neural networks (DNNs) via DRAM Rowhammer exploitations. Existing attacks are primarily launched over high-level DNN frameworks like PyTorch and flip bits in model weight files. Nevertheless, DNNs are frequently compiled into low-level executables by deep learning (DL) compilers to fully leverage low-level hardware primitives. The compiled code is usually high-speed and manifests dramatically distinct execution paradigms from high-level DNN frameworks.

  In this paper, we launch the first systematic study on the attack surface of BFA specifically for DNN executables compiled by DL compilers. We design an automated search tool to identify vulnerable bits in DNN executables and identify practical attack vectors that exploit the model structure in DNN executables with BFAs (whereas prior works make likely strong assumptions to attack model weights). DNN executables appear more "opaque" than models in high-level DNN frameworks. Nevertheless, we find that DNN executables contain extensive, severe (e.g., single-bit flip), and transferrable attack surfaces that are not present in high-level DNN models and can be exploited to deplete full model intelligence and control output labels. Our finding calls for incorporating security mechanisms in future DNN compilation toolchains.

Title: Using Reed-Muller Codes for Classification with Rejection and Recovery. (arXiv:2309.06359v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06359
• Code URL: https://github.com/dfenth/rmaggnet
• Copy Paste: [[2309.06359]] Using Reed-Muller Codes for Classification with Rejection and Recovery(http://arxiv.org/abs/2309.06359)
• Summary:

  When deploying classifiers in the real world, users expect them to respond to inputs appropriately. However, traditional classifiers are not equipped to handle inputs which lie far from the distribution they were trained on. Malicious actors can exploit this defect by making adversarial perturbations designed to cause the classifier to give an incorrect output. Classification-with-rejection methods attempt to solve this problem by allowing networks to refuse to classify an input in which they have low confidence. This works well for strongly adversarial examples, but also leads to the rejection of weakly perturbed images, which intuitively could be correctly classified. To address these issues, we propose Reed-Muller Aggregation Networks (RMAggNet), a classifier inspired by Reed-Muller error-correction codes which can correct and reject inputs. This paper shows that RMAggNet can minimise incorrectness while maintaining good correctness over multiple adversarial attacks at different perturbation budgets by leveraging the ability to correct errors in the classification process. This provides an alternative classification-with-rejection method which can reduce the amount of additional processing in situations where a small number of incorrect classifications are permissible.

robust

Title: TransferDoc: A Self-Supervised Transferable Document Representation Learning Model Unifying Vision and Language. (arXiv:2309.05756v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05756
• Code URL: null
• Copy Paste: [[2309.05756]] TransferDoc: A Self-Supervised Transferable Document Representation Learning Model Unifying Vision and Language(http://arxiv.org/abs/2309.05756)
• Summary:

  The field of visual document understanding has witnessed a rapid growth in emerging challenges and powerful multi-modal strategies. However, they rely on an extensive amount of document data to learn their pretext objectives in a ``pre-train-then-fine-tune'' paradigm and thus, suffer a significant performance drop in real-world online industrial settings. One major reason is the over-reliance on OCR engines to extract local positional information within a document page. Therefore, this hinders the model's generalizability, flexibility and robustness due to the lack of capturing global information within a document image. We introduce TransferDoc, a cross-modal transformer-based architecture pre-trained in a self-supervised fashion using three novel pretext objectives. TransferDoc learns richer semantic concepts by unifying language and visual representations, which enables the production of more transferable models. Besides, two novel downstream tasks have been introduced for a ``closer-to-real'' industrial evaluation scenario where TransferDoc outperforms other state-of-the-art approaches.

Title: A new meteor detection application robust to camera movements. (arXiv:2309.06027v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06027
• Code URL: null
• Copy Paste: [[2309.06027]] A new meteor detection application robust to camera movements(http://arxiv.org/abs/2309.06027)
• Summary:

  This article presents a new tool for the automatic detection of meteors. Fast Meteor Detection Toolbox (FMDT) is able to detect meteor sightings by analyzing videos acquired by cameras onboard weather balloons or within airplane with stabilization. The challenge consists in designing a processing chain composed of simple algorithms, that are robust to the high fluctuation of the videos and that satisfy the constraints on power consumption (10 W) and real-time processing (25 frames per second).

Title: Certified Robust Models with Slack Control and Large Lipschitz Constants. (arXiv:2309.06166v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06166
• Code URL: https://github.com/mlosch/cll
• Copy Paste: [[2309.06166]] Certified Robust Models with Slack Control and Large Lipschitz Constants(http://arxiv.org/abs/2309.06166)
• Summary:

  Despite recent success, state-of-the-art learning-based models remain highly vulnerable to input changes such as adversarial examples. In order to obtain certifiable robustness against such perturbations, recent work considers Lipschitz-based regularizers or constraints while at the same time increasing prediction margin. Unfortunately, this comes at the cost of significantly decreased accuracy. In this paper, we propose a Calibrated Lipschitz-Margin Loss (CLL) that addresses this issue and improves certified robustness by tackling two problems: Firstly, commonly used margin losses do not adjust the penalties to the shrinking output distribution; caused by minimizing the Lipschitz constant $K$. Secondly, and most importantly, we observe that minimization of $K$ can lead to overly smooth decision functions. This limits the model's complexity and thus reduces accuracy. Our CLL addresses these issues by explicitly calibrating the loss w.r.t. margin and Lipschitz constant, thereby establishing full control over slack and improving robustness certificates even with larger Lipschitz constants. On CIFAR-10, CIFAR-100 and Tiny-ImageNet, our models consistently outperform losses that leave the constant unattended. On CIFAR-100 and Tiny-ImageNet, CLL improves upon state-of-the-art deterministic $L_2$ robust accuracies. In contrast to current trends, we unlock potential of much smaller models without $K=1$ constraints.

Title: Modality Unifying Network for Visible-Infrared Person Re-Identification. (arXiv:2309.06262v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06262
• Code URL: null
• Copy Paste: [[2309.06262]] Modality Unifying Network for Visible-Infrared Person Re-Identification(http://arxiv.org/abs/2309.06262)
• Summary:

  Visible-infrared person re-identification (VI-ReID) is a challenging task due to large cross-modality discrepancies and intra-class variations. Existing methods mainly focus on learning modality-shared representations by embedding different modalities into the same feature space. As a result, the learned feature emphasizes the common patterns across modalities while suppressing modality-specific and identity-aware information that is valuable for Re-ID. To address these issues, we propose a novel Modality Unifying Network (MUN) to explore a robust auxiliary modality for VI-ReID. First, the auxiliary modality is generated by combining the proposed cross-modality learner and intra-modality learner, which can dynamically model the modality-specific and modality-shared representations to alleviate both cross-modality and intra-modality variations. Second, by aligning identity centres across the three modalities, an identity alignment loss function is proposed to discover the discriminative feature representations. Third, a modality alignment loss is introduced to consistently reduce the distribution distance of visible and infrared images by modality prototype modeling. Extensive experiments on multiple public datasets demonstrate that the proposed method surpasses the current state-of-the-art methods by a significant margin.

Title: Jersey Number Recognition using Keyframe Identification from Low-Resolution Broadcast Videos. (arXiv:2309.06285v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06285
• Code URL: null
• Copy Paste: [[2309.06285]] Jersey Number Recognition using Keyframe Identification from Low-Resolution Broadcast Videos(http://arxiv.org/abs/2309.06285)
• Summary:

  Player identification is a crucial component in vision-driven soccer analytics, enabling various downstream tasks such as player assessment, in-game analysis, and broadcast production. However, automatically detecting jersey numbers from player tracklets in videos presents challenges due to motion blur, low resolution, distortions, and occlusions. Existing methods, utilizing Spatial Transformer Networks, CNNs, and Vision Transformers, have shown success in image data but struggle with real-world video data, where jersey numbers are not visible in most of the frames. Hence, identifying frames that contain the jersey number is a key sub-problem to tackle. To address these issues, we propose a robust keyframe identification module that extracts frames containing essential high-level information about the jersey number. A spatio-temporal network is then employed to model spatial and temporal context and predict the probabilities of jersey numbers in the video. Additionally, we adopt a multi-task loss function to predict the probability distribution of each digit separately. Extensive evaluations on the SoccerNet dataset demonstrate that incorporating our proposed keyframe identification module results in a significant 37.81% and 37.70% increase in the accuracies of 2 different test sets with domain gaps. These results highlight the effectiveness and importance of our approach in tackling the challenges of automatic jersey number detection in sports videos.

Title: Generative Data Augmentation using LLMs improves Distributional Robustness in Question Answering. (arXiv:2309.06358v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06358
• Code URL: null
• Copy Paste: [[2309.06358]] Generative Data Augmentation using LLMs improves Distributional Robustness in Question Answering(http://arxiv.org/abs/2309.06358)
• Summary:

  Robustness in Natural Language Processing continues to be a pertinent issue, where state of the art models under-perform under naturally shifted distributions. In the context of Question Answering, work on domain adaptation methods continues to be a growing body of research. However, very little attention has been given to the notion of domain generalization under natural distribution shifts, where the target domain is unknown. With drastic improvements in the quality and access to generative models, we answer the question: How do generated datasets influence the performance of QA models under natural distribution shifts? We perform experiments on 4 different datasets under varying amounts of distribution shift, and analyze how "in-the-wild" generation can help achieve domain generalization. We take a two-step generation approach, generating both contexts and QA pairs to augment existing datasets. Through our experiments, we demonstrate how augmenting reading comprehension datasets with generated data leads to better robustness towards natural distribution shifts.

Title: Down the Toxicity Rabbit Hole: Investigating PaLM 2 Guardrails. (arXiv:2309.06415v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06415
• Code URL: null
• Copy Paste: [[2309.06415]] Down the Toxicity Rabbit Hole: Investigating PaLM 2 Guardrails(http://arxiv.org/abs/2309.06415)
• Summary:

  This paper conducts a robustness audit of the safety feedback of PaLM 2 through a novel toxicity rabbit hole framework introduced here. Starting with a stereotype, the framework instructs PaLM 2 to generate more toxic content than the stereotype. Every subsequent iteration it continues instructing PaLM 2 to generate more toxic content than the previous iteration until PaLM 2 safety guardrails throw a safety violation. Our experiments uncover highly disturbing antisemitic, Islamophobic, racist, homophobic, and misogynistic (to list a few) generated content that PaLM 2 safety guardrails do not evaluate as highly unsafe.

Title: Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals. (arXiv:2309.05927v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.05927
• Code URL: null
• Copy Paste: [[2309.05927]] Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals(http://arxiv.org/abs/2309.05927)
• Summary:

  Leveraging multimodal information from biosignals is vital for building a comprehensive representation of people's physical and mental states. However, multimodal biosignals often exhibit substantial distributional shifts between pretraining and inference datasets, stemming from changes in task specification or variations in modality compositions. To achieve effective pretraining in the presence of potential distributional shifts, we propose a frequency-aware masked autoencoder ($\texttt{bio}$FAME) that learns to parameterize the representation of biosignals in the frequency space. $\texttt{bio}$FAME incorporates a frequency-aware transformer, which leverages a fixed-size Fourier-based operator for global token mixing, independent of the length and sampling rate of inputs. To maintain the frequency components within each input channel, we further employ a frequency-maintain pretraining strategy that performs masked autoencoding in the latent space. The resulting architecture effectively utilizes multimodal information during pretraining, and can be seamlessly adapted to diverse tasks and modalities at test time, regardless of input size and order. We evaluated our approach on a diverse set of transfer experiments on unimodal time series, achieving an average of $\uparrow$5.5% improvement in classification accuracy over the previous state-of-the-art. Furthermore, we demonstrated that our architecture is robust in modality mismatch scenarios, including unpredicted modality dropout or substitution, proving its practical utility in real-world applications. Code will be available soon.

Title: Robust-MBDL: A Robust Multi-branch Deep Learning Based Model for Remaining Useful Life Prediction and Operational Condition Identification of Rotating Machines. (arXiv:2309.06157v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06157
• Code URL: null
• Copy Paste: [[2309.06157]] Robust-MBDL: A Robust Multi-branch Deep Learning Based Model for Remaining Useful Life Prediction and Operational Condition Identification of Rotating Machines(http://arxiv.org/abs/2309.06157)
• Summary:

  In this paper, a Robust Multi-branch Deep learning-based system for remaining useful life (RUL) prediction and condition operations (CO) identification of rotating machines is proposed. In particular, the proposed system comprises main components: (1) an LSTM-Autoencoder to denoise the vibration data; (2) a feature extraction to generate time-domain, frequency-domain, and time-frequency based features from the denoised data; (3) a novel and robust multi-branch deep learning network architecture to exploit the multiple features. The performance of our proposed system was evaluated and compared to the state-of-the-art systems on two benchmark datasets of XJTU-SY and PRONOSTIA. The experimental results prove that our proposed system outperforms the state-of-the-art systems and presents potential for real-life applications on bearing machines.

biometric

steal

extraction

Title: Self-supervised Extraction of Human Motion Structures via Frame-wise Discrete Features. (arXiv:2309.05972v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05972
• Code URL: null
• Copy Paste: [[2309.05972]] Self-supervised Extraction of Human Motion Structures via Frame-wise Discrete Features(http://arxiv.org/abs/2309.05972)
• Summary:

  The present paper proposes an encoder-decoder model for extracting the structures of human motions represented by frame-wise discrete features in a self-supervised manner. In the proposed method, features are extracted as codes in a motion codebook without the use of human knowledge, and the relationship between these codes can be visualized on a graph. Since the codes are expected to be temporally sparse compared to the captured frame rate and can be shared by multiple sequences, the proposed network model also addresses the need for training constraints. Specifically, the model consists of self-attention layers and a vector clustering block. The attention layers contribute to finding sparse keyframes and discrete features as motion codes, which are then extracted by vector clustering. The constraints are realized as training losses so that the same motion codes can be as contiguous as possible and can be shared by multiple sequences. In addition, we propose the use of causal self-attention as a method by which to calculate attention for long sequences consisting of numerous frames. In our experiments, the sparse structures of motion codes were used to compile a graph that facilitates visualization of the relationship between the codes and the differences between sequences. We then evaluated the effectiveness of the extracted motion codes by applying them to multiple recognition tasks and found that performance levels comparable to task-optimized methods could be achieved by linear probing.

Title: Feature Aggregation Network for Building Extraction from High-resolution Remote Sensing Images. (arXiv:2309.06017v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06017
• Code URL: null
• Copy Paste: [[2309.06017]] Feature Aggregation Network for Building Extraction from High-resolution Remote Sensing Images(http://arxiv.org/abs/2309.06017)
• Summary:

  The rapid advancement in high-resolution satellite remote sensing data acquisition, particularly those achieving submeter precision, has uncovered the potential for detailed extraction of surface architectural features. However, the diversity and complexity of surface distributions frequently lead to current methods focusing exclusively on localized information of surface features. This often results in significant intraclass variability in boundary recognition and between buildings. Therefore, the task of fine-grained extraction of surface features from high-resolution satellite imagery has emerged as a critical challenge in remote sensing image processing. In this work, we propose the Feature Aggregation Network (FANet), concentrating on extracting both global and local features, thereby enabling the refined extraction of landmark buildings from high-resolution satellite remote sensing imagery. The Pyramid Vision Transformer captures these global features, which are subsequently refined by the Feature Aggregation Module and merged into a cohesive representation by the Difference Elimination Module. In addition, to ensure a comprehensive feature map, we have incorporated the Receptive Field Block and Dual Attention Module, expanding the receptive field and intensifying attention across spatial and channel dimensions. Extensive experiments on multiple datasets have validated the outstanding capability of FANet in extracting features from high-resolution satellite images. This signifies a major breakthrough in the field of remote sensing image processing. We will release our code soon.

Title: C-RITNet: Set Infrared and Visible Image Fusion Free from Complementary Information Mining. (arXiv:2309.06118v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06118
• Code URL: null
• Copy Paste: [[2309.06118]] C-RITNet: Set Infrared and Visible Image Fusion Free from Complementary Information Mining(http://arxiv.org/abs/2309.06118)
• Summary:

  Infrared and visible image fusion (IVIF) aims to extract and integrate the complementary information in two different modalities to generate high-quality fused images with salient targets and abundant texture details. However, current image fusion methods go to great lengths to excavate complementary features, which is generally achieved through two efforts. On the one hand, the feature extraction network is expected to have excellent performance in extracting complementary information. On the other hand, complex fusion strategies are often designed to aggregate the complementary information. In other words, enabling the network to perceive and extract complementary information is extremely challenging. Complicated fusion strategies, while effective, still run the risk of losing weak edge details. To this end, this paper rethinks the IVIF outside the box, proposing a complementary-redundant information transfer network (C-RITNet). It reasonably transfers complementary information into redundant one, which integrates both the shared and complementary features from two modalities. Hence, the proposed method is able to alleviate the challenges posed by the complementary information extraction and reduce the reliance on sophisticated fusion strategies. Specifically, to skillfully sidestep aggregating complementary information in IVIF, we first design the mutual information transfer (MIT) module to mutually represent features from two modalities, roughly transferring complementary information into redundant one. Then, a redundant information acquisition supervised by source image (RIASSI) module is devised to further ensure the complementary-redundant information transfer after MIT. Meanwhile, we also propose a structure information preservation (SIP) module to guarantee that the edge structure information of the source images can be transferred to the fusion results.

Title: GLAD: Content-aware Dynamic Graphs For Log Anomaly Detection. (arXiv:2309.05953v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.05953
• Code URL: null
• Copy Paste: [[2309.05953]] GLAD: Content-aware Dynamic Graphs For Log Anomaly Detection(http://arxiv.org/abs/2309.05953)
• Summary:

  Logs play a crucial role in system monitoring and debugging by recording valuable system information, including events and states. Although various methods have been proposed to detect anomalies in log sequences, they often overlook the significance of considering relations among system components, such as services and users, which can be identified from log contents. Understanding these relations is vital for detecting anomalies and their underlying causes. To address this issue, we introduce GLAD, a Graph-based Log Anomaly Detection framework designed to detect relational anomalies in system logs. GLAD incorporates log semantics, relational patterns, and sequential patterns into a unified framework for anomaly detection. Specifically, GLAD first introduces a field extraction module that utilizes prompt-based few-shot learning to identify essential fields from log contents. Then GLAD constructs dynamic log graphs for sliding windows by interconnecting extracted fields and log events parsed from the log parser. These graphs represent events and fields as nodes and their relations as edges. Subsequently, GLAD utilizes a temporal-attentive graph edge anomaly detection model for identifying anomalous relations in these dynamic log graphs. This model employs a Graph Neural Network (GNN)-based encoder enhanced with transformers to capture content, structural and temporal features. We evaluate our proposed method on three datasets, and the results demonstrate the effectiveness of GLAD in detecting anomalies indicated by varying relational patterns.

membership infer

federate

Title: Federated Learning for Large-Scale Scene Modeling with Neural Radiance Fields. (arXiv:2309.06030v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06030
• Code URL: null
• Copy Paste: [[2309.06030]] Federated Learning for Large-Scale Scene Modeling with Neural Radiance Fields(http://arxiv.org/abs/2309.06030)
• Summary:

  We envision a system to continuously build and maintain a map based on earth-scale neural radiance fields (NeRF) using data collected from vehicles and drones in a lifelong learning manner. However, existing large-scale modeling by NeRF has problems in terms of scalability and maintainability when modeling earth-scale environments. Therefore, to address these problems, we propose a federated learning pipeline for large-scale modeling with NeRF. We tailor the model aggregation pipeline in federated learning for NeRF, thereby allowing local updates of NeRF. In the aggregation step, the accuracy of the clients' global pose is critical. Thus, we also propose global pose alignment to align the noisy global pose of clients before the aggregation step. In experiments, we show the effectiveness of the proposed pose alignment and the federated learning pipeline on the large-scale scene dataset, Mill19.

fair

interpretability

Title: Evaluating the Reliability of CNN Models on Classifying Traffic and Road Signs using LIME. (arXiv:2309.05747v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05747
• Code URL: null
• Copy Paste: [[2309.05747]] Evaluating the Reliability of CNN Models on Classifying Traffic and Road Signs using LIME(http://arxiv.org/abs/2309.05747)
• Summary:

  The objective of this investigation is to evaluate and contrast the effectiveness of four state-of-the-art pre-trained models, ResNet-34, VGG-19, DenseNet-121, and Inception V3, in classifying traffic and road signs with the utilization of the GTSRB public dataset. The study focuses on evaluating the accuracy of these models' predictions as well as their ability to employ appropriate features for image categorization. To gain insights into the strengths and limitations of the model's predictions, the study employs the local interpretable model-agnostic explanations (LIME) framework. The findings of this experiment indicate that LIME is a crucial tool for improving the interpretability and dependability of machine learning models for image identification, regardless of the models achieving an f1 score of 0.99 on classifying traffic and road signs. The conclusion of this study has important ramifications for how these models are used in practice, as it is crucial to ensure that model predictions are founded on the pertinent image features.

explainability

Title: Measuring vagueness and subjectivity in texts: from symbolic to neural VAGO. (arXiv:2309.06132v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06132
• Code URL: null
• Copy Paste: [[2309.06132]] Measuring vagueness and subjectivity in texts: from symbolic to neural VAGO(http://arxiv.org/abs/2309.06132)
• Summary:

  We present a hybrid approach to the automated measurement of vagueness and subjectivity in texts. We first introduce the expert system VAGO, we illustrate it on a small benchmark of fact vs. opinion sentences, and then test it on the larger French press corpus FreSaDa to confirm the higher prevalence of subjective markers in satirical vs. regular texts. We then build a neural clone of VAGO, based on a BERT-like architecture, trained on the symbolic VAGO scores obtained on FreSaDa. Using explainability tools (LIME), we show the interest of this neural version for the enrichment of the lexicons of the symbolic version, and for the production of versions in other languages.

watermark

Title: Catch You Everything Everywhere: Guarding Textual Inversion via Concept Watermarking. (arXiv:2309.05940v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.05940
• Code URL: null
• Copy Paste: [[2309.05940]] Catch You Everything Everywhere: Guarding Textual Inversion via Concept Watermarking(http://arxiv.org/abs/2309.05940)
• Summary:

  AIGC (AI-Generated Content) has achieved tremendous success in many applications such as text-to-image tasks, where the model can generate high-quality images with diverse prompts, namely, different descriptions in natural languages. More surprisingly, the emerging personalization techniques even succeed in describing unseen concepts with only a few personal images as references, and there have been some commercial platforms for sharing the valuable personalized concept. However, such an advanced technique also introduces a severe threat, where malicious users can misuse the target concept to generate highly-realistic illegal images. Therefore, it becomes necessary for the platform to trace malicious users and hold them accountable.

  In this paper, we focus on guarding the most popular lightweight personalization model, ie, Textual Inversion (TI). To achieve it, we propose the novel concept watermarking, where watermark information is embedded into the target concept and then extracted from generated images based on the watermarked concept. Specifically, we jointly train a watermark encoder and a watermark decoder with the sampler in the loop.

  It shows great resilience to different diffusion sampling processes possibly chosen by malicious users, meanwhile preserving utility for normal use. In practice, the concept owner can upload his concept with different watermarks (ie, serial numbers) to the platform, and the platform allocates different users with different serial numbers for subsequent tracing and forensics.

diffusion

Title: PhotoVerse: Tuning-Free Image Customization with Text-to-Image Diffusion Models. (arXiv:2309.05793v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05793
• Code URL: null
• Copy Paste: [[2309.05793]] PhotoVerse: Tuning-Free Image Customization with Text-to-Image Diffusion Models(http://arxiv.org/abs/2309.05793)
• Summary:

  Personalized text-to-image generation has emerged as a powerful and sought-after tool, empowering users to create customized images based on their specific concepts and prompts. However, existing approaches to personalization encounter multiple challenges, including long tuning times, large storage requirements, the necessity for multiple input images per identity, and limitations in preserving identity and editability. To address these obstacles, we present PhotoVerse, an innovative methodology that incorporates a dual-branch conditioning mechanism in both text and image domains, providing effective control over the image generation process. Furthermore, we introduce facial identity loss as a novel component to enhance the preservation of identity during training. Remarkably, our proposed PhotoVerse eliminates the need for test time tuning and relies solely on a single facial photo of the target identity, significantly reducing the resource cost associated with image generation. After a single training phase, our approach enables generating high-quality images within only a few seconds. Moreover, our method can produce diverse images that encompass various scenes and styles. The extensive evaluation demonstrates the superior performance of our approach, which achieves the dual objectives of preserving identity and facilitating editability. Project page: https://photoverse2d.github.io/

Title: Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts. (arXiv:2309.06135v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06135
• Code URL: null
• Copy Paste: [[2309.06135]] Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts(http://arxiv.org/abs/2309.06135)
• Summary:

  Text-to-image diffusion models, e.g. Stable Diffusion (SD), lately have shown remarkable ability in high-quality content generation, and become one of the representatives for the recent wave of transformative AI. Nevertheless, such advance comes with an intensifying concern about the misuse of this generative technology, especially for producing copyrighted or NSFW (i.e. not safe for work) images. Although efforts have been made to filter inappropriate images/prompts or remove undesirable concepts/styles via model fine-tuning, the reliability of these safety mechanisms against diversified problematic prompts remains largely unexplored. In this work, we propose Prompting4Debugging (P4D) as a debugging and red-teaming tool that automatically finds problematic prompts for diffusion models to test the reliability of a deployed safety mechanism. We demonstrate the efficacy of our P4D tool in uncovering new vulnerabilities of SD models with safety mechanisms. Particularly, our result shows that around half of prompts in existing safe prompting benchmarks which were originally considered "safe" can actually be manipulated to bypass many deployed safety mechanisms, including concept removal, negative prompt, and safety guidance. Our findings suggest that, without comprehensive testing, the evaluations on limited safe prompting benchmarks can lead to a false sense of safety for text-to-image models.

Title: Elucidating the solution space of extended reverse-time SDE for diffusion models. (arXiv:2309.06169v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06169
• Code URL: null
• Copy Paste: [[2309.06169]] Elucidating the solution space of extended reverse-time SDE for diffusion models(http://arxiv.org/abs/2309.06169)
• Summary:

  Diffusion models (DMs) demonstrate potent image generation capabilities in various generative modeling tasks. Nevertheless, their primary limitation lies in slow sampling speed, requiring hundreds or thousands of sequential function evaluations through large neural networks to generate high-quality images. Sampling from DMs can be seen as solving corresponding stochastic differential equations (SDEs) or ordinary differential equations (ODEs). In this work, we formulate the sampling process as an extended reverse-time SDE (ER SDE), unifying prior explorations into ODEs and SDEs. Leveraging the semi-linear structure of ER SDE solutions, we offer exact solutions and arbitrarily high-order approximate solutions for VP SDE and VE SDE, respectively. Based on the solution space of the ER SDE, we yield mathematical insights elucidating the superior performance of ODE solvers over SDE solvers in terms of fast sampling. Additionally, we unveil that VP SDE solvers stand on par with their VE SDE counterparts. Finally, we devise fast and training-free samplers, ER-SDE Solvers, elevating the efficiency of stochastic samplers to unprecedented levels. Experimental results demonstrate achieving 3.45 FID in 20 function evaluations and 2.24 FID in 50 function evaluations on the ImageNet 64$\times$64 dataset.

Title: Fg-T2M: Fine-Grained Text-Driven Human Motion Generation via Diffusion Model. (arXiv:2309.06284v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06284
• Code URL: null
• Copy Paste: [[2309.06284]] Fg-T2M: Fine-Grained Text-Driven Human Motion Generation via Diffusion Model(http://arxiv.org/abs/2309.06284)
• Summary:

  Text-driven human motion generation in computer vision is both significant and challenging. However, current methods are limited to producing either deterministic or imprecise motion sequences, failing to effectively control the temporal and spatial relationships required to conform to a given text description. In this work, we propose a fine-grained method for generating high-quality, conditional human motion sequences supporting precise text description. Our approach consists of two key components: 1) a linguistics-structure assisted module that constructs accurate and complete language feature to fully utilize text information; and 2) a context-aware progressive reasoning module that learns neighborhood and overall semantic linguistics features from shallow and deep graph neural networks to achieve a multi-step inference. Experiments show that our approach outperforms text-driven motion generation methods on HumanML3D and KIT test sets and generates better visually confirmed motion to the text conditions.

Title: InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation. (arXiv:2309.06380v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06380
• Code URL: https://github.com/gnobitab/instaflow
• Copy Paste: [[2309.06380]] InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation(http://arxiv.org/abs/2309.06380)
• Summary:

  Diffusion models have revolutionized text-to-image generation with its exceptional quality and creativity. However, its multi-step sampling process is known to be slow, often requiring tens of inference steps to obtain satisfactory results. Previous attempts to improve its sampling speed and reduce computational costs through distillation have been unsuccessful in achieving a functional one-step model. In this paper, we explore a recent method called Rectified Flow, which, thus far, has only been applied to small datasets. The core of Rectified Flow lies in its \emph{reflow} procedure, which straightens the trajectories of probability flows, refines the coupling between noises and images, and facilitates the distillation process with student models. We propose a novel text-conditioned pipeline to turn Stable Diffusion (SD) into an ultra-fast one-step model, in which we find reflow plays a critical role in improving the assignment between noise and images. Leveraging our new pipeline, we create, to the best of our knowledge, the first one-step diffusion-based text-to-image generator with SD-level image quality, achieving an FID (Frechet Inception Distance) of $23.3$ on MS COCO 2017-5k, surpassing the previous state-of-the-art technique, progressive distillation, by a significant margin ($37.2$ $\rightarrow$ $23.3$ in FID). By utilizing an expanded network with 1.7B parameters, we further improve the FID to $22.4$. We call our one-step models \emph{InstaFlow}. On MS COCO 2014-30k, InstaFlow yields an FID of $13.1$ in just $0.09$ second, the best in $\leq 0.1$ second regime, outperforming the recent StyleGAN-T ($13.9$ in $0.1$ second). Notably, the training of InstaFlow only costs 199 A100 GPU days. Project page:~\url{https://github.com/gnobitab/InstaFlow}.

noise learning

data-free

transformer

Title: Divergences in Color Perception between Deep Neural Networks and Humans. (arXiv:2309.05809v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05809
• Code URL: null
• Copy Paste: [[2309.05809]] Divergences in Color Perception between Deep Neural Networks and Humans(http://arxiv.org/abs/2309.05809)
• Summary:

  Deep neural networks (DNNs) are increasingly proposed as models of human vision, bolstered by their impressive performance on image classification and object recognition tasks. Yet, the extent to which DNNs capture fundamental aspects of human vision such as color perception remains unclear. Here, we develop novel experiments for evaluating the perceptual coherence of color embeddings in DNNs, and we assess how well these algorithms predict human color similarity judgments collected via an online survey. We find that state-of-the-art DNN architectures $-$ including convolutional neural networks and vision transformers $-$ provide color similarity judgments that strikingly diverge from human color judgments of (i) images with controlled color properties, (ii) images generated from online searches, and (iii) real-world images from the canonical CIFAR-10 dataset. We compare DNN performance against an interpretable and cognitively plausible model of color perception based on wavelet decomposition, inspired by foundational theories in computational neuroscience. While one deep learning model $-$ a convolutional DNN trained on a style transfer task $-$ captures some aspects of human color perception, our wavelet algorithm provides more coherent color embeddings that better predict human color judgments compared to all DNNs we examine. These results hold when altering the high-level visual task used to train similar DNN architectures (e.g., image classification versus image segmentation), as well as when examining the color embeddings of different layers in a given DNN architecture. These findings break new ground in the effort to analyze the perceptual representations of machine learning algorithms and to improve their ability to serve as cognitively plausible models of human vision. Implications for machine learning, human perception, and embodied cognition are discussed.

Title: Mobile Vision Transformer-based Visual Object Tracking. (arXiv:2309.05829v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05829
• Code URL: https://github.com/goutamyg/mvt
• Copy Paste: [[2309.05829]] Mobile Vision Transformer-based Visual Object Tracking(http://arxiv.org/abs/2309.05829)
• Summary:

  The introduction of robust backbones, such as Vision Transformers, has improved the performance of object tracking algorithms in recent years. However, these state-of-the-art trackers are computationally expensive since they have a large number of model parameters and rely on specialized hardware (e.g., GPU) for faster inference. On the other hand, recent lightweight trackers are fast but are less accurate, especially on large-scale datasets. We propose a lightweight, accurate, and fast tracking algorithm using Mobile Vision Transformers (MobileViT) as the backbone for the first time. We also present a novel approach of fusing the template and search region representations in the MobileViT backbone, thereby generating superior feature encoding for target localization. The experimental results show that our MobileViT-based Tracker, MVT, surpasses the performance of recent lightweight trackers on the large-scale datasets GOT10k and TrackingNet, and with a high inference speed. In addition, our method outperforms the popular DiMP-50 tracker despite having 4.7 times fewer model parameters and running at 2.8 times its speed on a GPU. The tracker code and models are available at https://github.com/goutamyg/MVT

Title: Knowledge-Guided Short-Context Action Anticipation in Human-Centric Videos. (arXiv:2309.05943v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05943
• Code URL: null
• Copy Paste: [[2309.05943]] Knowledge-Guided Short-Context Action Anticipation in Human-Centric Videos(http://arxiv.org/abs/2309.05943)
• Summary:

  This work focuses on anticipating long-term human actions, particularly using short video segments, which can speed up editing workflows through improved suggestions while fostering creativity by suggesting narratives. To this end, we imbue a transformer network with a symbolic knowledge graph for action anticipation in video segments by boosting certain aspects of the transformer's attention mechanism at run-time. Demonstrated on two benchmark datasets, Breakfast and 50Salads, our approach outperforms current state-of-the-art methods for long-term action anticipation using short video context by up to 9%.

Title: How does representation impact in-context learning: A exploration on a synthetic task. (arXiv:2309.06054v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06054
• Code URL: null
• Copy Paste: [[2309.06054]] How does representation impact in-context learning: A exploration on a synthetic task(http://arxiv.org/abs/2309.06054)
• Summary:

  In-context learning, i.e., learning from in-context samples, is an impressive ability of Transformer. However, the mechanism driving the in-context learning is not yet fully understood. In this study, we aim to investigate from an underexplored perspective of representation learning. The representation is more complex for in-context learning senario, where the representation can be impacted by both model weights and in-context samples. We refer the above two conceptually aspects of representation as in-weight component and in-context component, respectively. To study how the two components affect in-context learning capabilities, we construct a novel synthetic task, making it possible to device two probes, in-weights probe and in-context probe, to evaluate the two components, respectively. We demonstrate that the goodness of in-context component is highly related to the in-context learning performance, which indicates the entanglement between in-context learning and representation learning. Furthermore, we find that a good in-weights component can actually benefit the learning of the in-context component, indicating that in-weights learning should be the foundation of in-context learning. To further understand the the in-context learning mechanism and importance of the in-weights component, we proof by construction that a simple Transformer, which uses pattern matching and copy-past mechanism to perform in-context learning, can match the in-context learning performance with more complex, best tuned Transformer under the perfect in-weights component assumption. In short, those discoveries from representation learning perspective shed light on new approaches to improve the in-context capacity.

Title: A 3M-Hybrid Model for the Restoration of Unique Giant Murals: A Case Study on the Murals of Yongle Palace. (arXiv:2309.06194v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06194
• Code URL: null
• Copy Paste: [[2309.06194]] A 3M-Hybrid Model for the Restoration of Unique Giant Murals: A Case Study on the Murals of Yongle Palace(http://arxiv.org/abs/2309.06194)
• Summary:

  The Yongle Palace murals, as valuable cultural heritage, have suffered varying degrees of damage, making their restoration of significant importance. However, the giant size and unique data of Yongle Palace murals present challenges for existing deep-learning based restoration methods: 1) The distinctive style introduces domain bias in traditional transfer learning-based restoration methods, while the scarcity of mural data further limits the applicability of these methods. 2) Additionally, the giant size of these murals results in a wider range of defect types and sizes, necessitating models with greater adaptability. Consequently, there is a lack of focus on deep learning-based restoration methods for the unique giant murals of Yongle Palace. Here, a 3M-Hybrid model is proposed to address these challenges. Firstly, based on the characteristic that the mural data frequency is prominent in the distribution of low and high frequency features, high and low frequency features are separately abstracted for complementary learning. Furthermore, we integrate a pre-trained Vision Transformer model (VIT) into the CNN module, allowing us to leverage the benefits of a large model while mitigating domain bias. Secondly, we mitigate seam and structural distortion issues resulting from the restoration of large defects by employing a multi-scale and multi-perspective strategy, including data segmentation and fusion. Experimental results demonstrate the efficacy of our proposed model. In regular-sized mural restoration, it improves SSIM and PSNR by 14.61% and 4.73%, respectively, compared to the best model among four representative CNN models. Additionally, it achieves favorable results in the final restoration of giant murals.

Title: SGFeat: Salient Geometric Feature for Point Cloud Registration. (arXiv:2309.06207v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06207
• Code URL: null
• Copy Paste: [[2309.06207]] SGFeat: Salient Geometric Feature for Point Cloud Registration(http://arxiv.org/abs/2309.06207)
• Summary:

  Point Cloud Registration (PCR) is a critical and challenging task in computer vision. One of the primary difficulties in PCR is identifying salient and meaningful points that exhibit consistent semantic and geometric properties across different scans. Previous methods have encountered challenges with ambiguous matching due to the similarity among patch blocks throughout the entire point cloud and the lack of consideration for efficient global geometric consistency. To address these issues, we propose a new framework that includes several novel techniques. Firstly, we introduce a semantic-aware geometric encoder that combines object-level and patch-level semantic information. This encoder significantly improves registration recall by reducing ambiguity in patch-level superpoint matching. Additionally, we incorporate a prior knowledge approach that utilizes an intrinsic shape signature to identify salient points. This enables us to extract the most salient super points and meaningful dense points in the scene. Secondly, we introduce an innovative transformer that encodes High-Order (HO) geometric features. These features are crucial for identifying salient points within initial overlap regions while considering global high-order geometric consistency. To optimize this high-order transformer further, we introduce an anchor node selection strategy. By encoding inter-frame triangle or polyhedron consistency features based on these anchor nodes, we can effectively learn high-order geometric features of salient super points. These high-order features are then propagated to dense points and utilized by a Sinkhorn matching module to identify key correspondences for successful registration. In our experiments conducted on well-known datasets such as 3DMatch/3DLoMatch and KITTI, our approach has shown promising results, highlighting the effectiveness of our novel method.

Title: IBAFormer: Intra-batch Attention Transformer for Domain Generalized Semantic Segmentation. (arXiv:2309.06282v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06282
• Code URL: null
• Copy Paste: [[2309.06282]] IBAFormer: Intra-batch Attention Transformer for Domain Generalized Semantic Segmentation(http://arxiv.org/abs/2309.06282)
• Summary:

  Domain generalized semantic segmentation (DGSS) is a critical yet challenging task, where the model is trained only on source data without access to any target data. Despite the proposal of numerous DGSS strategies, the generalization capability remains limited in CNN architectures. Though some Transformer-based segmentation models show promising performance, they primarily focus on capturing intra-sample attentive relationships, disregarding inter-sample correlations which can potentially benefit DGSS. To this end, we enhance the attention modules in Transformer networks for improving DGSS by incorporating information from other independent samples in the same batch, enriching contextual information, and diversifying the training data for each attention block. Specifically, we propose two alternative intra-batch attention mechanisms, namely mean-based intra-batch attention (MIBA) and element-wise intra-batch attention (EIBA), to capture correlations between different samples, enhancing feature representation and generalization capabilities. Building upon intra-batch attention, we introduce IBAFormer, which integrates self-attention modules with the proposed intra-batch attention for DGSS. Extensive experiments demonstrate that IBAFormer achieves SOTA performance in DGSS, and ablation studies further confirm the effectiveness of each introduced component.

Title: Uncovering mesa-optimization algorithms in Transformers. (arXiv:2309.05858v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.05858
• Code URL: null
• Copy Paste: [[2309.05858]] Uncovering mesa-optimization algorithms in Transformers(http://arxiv.org/abs/2309.05858)
• Summary:

  Transformers have become the dominant model in deep learning, but the reason for their superior performance is poorly understood. Here, we hypothesize that the strong performance of Transformers stems from an architectural bias towards mesa-optimization, a learned process running within the forward pass of a model consisting of the following two steps: (i) the construction of an internal learning objective, and (ii) its corresponding solution found through optimization. To test this hypothesis, we reverse-engineer a series of autoregressive Transformers trained on simple sequence modeling tasks, uncovering underlying gradient-based mesa-optimization algorithms driving the generation of predictions. Moreover, we show that the learned forward-pass optimization algorithm can be immediately repurposed to solve supervised few-shot tasks, suggesting that mesa-optimization might underlie the in-context learning capabilities of large language models. Finally, we propose a novel self-attention layer, the mesa-layer, that explicitly and efficiently solves optimization problems specified in context. We find that this layer can lead to improved performance in synthetic and preliminary language modeling experiments, adding weight to our hypothesis that mesa-optimization is an important operation hidden within the weights of trained Transformers.

Title: ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning. (arXiv:2309.05915v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.05915
• Code URL: null
• Copy Paste: [[2309.05915]] ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning(http://arxiv.org/abs/2309.05915)
• Summary:

  Decision Transformer (DT), which employs expressive sequence modeling techniques to perform action generation, has emerged as a promising approach to offline policy optimization. However, DT generates actions conditioned on a desired future return, which is known to bear some weaknesses such as the susceptibility to environmental stochasticity. To overcome DT's weaknesses, we propose to empower DT with dynamic programming. Our method comprises three steps. First, we employ in-sample value iteration to obtain approximated value functions, which involves dynamic programming over the MDP structure. Second, we evaluate action quality in context with estimated advantages. We introduce two types of advantage estimators, IAE and GAE, which are suitable for different tasks. Third, we train an Advantage-Conditioned Transformer (ACT) to generate actions conditioned on the estimated advantages. Finally, during testing, ACT generates actions conditioned on a desired advantage. Our evaluation results validate that, by leveraging the power of dynamic programming, ACT demonstrates effective trajectory stitching and robust action generation in spite of the environmental stochasticity, outperforming baseline methods across various benchmarks. Additionally, we conduct an in-depth analysis of ACT's various design choices through ablation studies.

Title: Neural Network Layer Matrix Decomposition reveals Latent Manifold Encoding and Memory Capacity. (arXiv:2309.05968v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.05968
• Code URL: null
• Copy Paste: [[2309.05968]] Neural Network Layer Matrix Decomposition reveals Latent Manifold Encoding and Memory Capacity(http://arxiv.org/abs/2309.05968)
• Summary:

  We prove the converse of the universal approximation theorem, i.e. a neural network (NN) encoding theorem which shows that for every stably converged NN of continuous activation functions, its weight matrix actually encodes a continuous function that approximates its training dataset to within a finite margin of error over a bounded domain. We further show that using the Eckart-Young theorem for truncated singular value decomposition of the weight matrix for every NN layer, we can illuminate the nature of the latent space manifold of the training dataset encoded and represented by every NN layer, and the geometric nature of the mathematical operations performed by each NN layer. Our results have implications for understanding how NNs break the curse of dimensionality by harnessing memory capacity for expressivity, and that the two are complementary. This Layer Matrix Decomposition (LMD) further suggests a close relationship between eigen-decomposition of NN layers and the latest advances in conceptualizations of Hopfield networks and Transformer NN models.

Title: Long-term drought prediction using deep neural networks based on geospatial weather data. (arXiv:2309.06212v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06212
• Code URL: null
• Copy Paste: [[2309.06212]] Long-term drought prediction using deep neural networks based on geospatial weather data(http://arxiv.org/abs/2309.06212)
• Summary:

  The accurate prediction of drought probability in specific regions is crucial for informed decision-making in agricultural practices. It is important to make predictions one year in advance, particularly for long-term decisions. However, forecasting this probability presents challenges due to the complex interplay of various factors within the region of interest and neighboring areas. In this study, we propose an end-to-end solution to address this issue based on various spatiotemporal neural networks. The models considered focus on predicting the drought intensity based on the Palmer Drought Severity Index (PDSI) for subregions of interest, leveraging intrinsic factors and insights from climate models to enhance drought predictions.

  Comparative evaluations demonstrate the superior accuracy of Convolutional LSTM (ConvLSTM) and transformer models compared to baseline gradient boosting and logistic regression solutions. The two former models achieved impressive ROC AUC scores from 0.90 to 0.70 for forecast horizons from one to six months, outperforming baseline models. The transformer showed superiority for shorter horizons, while ConvLSTM did so for longer horizons. Thus, we recommend selecting the models accordingly for long-term drought forecasting.

  To ensure the broad applicability of the considered models, we conduct extensive validation across regions worldwide, considering different environmental conditions. We also run several ablation and sensitivity studies to challenge our findings and provide additional information on how to solve the problem.

generative

Title: Characterizing Latent Perspectives of Media Houses Towards Public Figures. (arXiv:2309.06112v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06112
• Code URL: null
• Copy Paste: [[2309.06112]] Characterizing Latent Perspectives of Media Houses Towards Public Figures(http://arxiv.org/abs/2309.06112)
• Summary:

  Media houses reporting on public figures, often come with their own biases stemming from their respective worldviews. A characterization of these underlying patterns helps us in better understanding and interpreting news stories. For this, we need diverse or subjective summarizations, which may not be amenable for classifying into predefined class labels. This work proposes a zero-shot approach for non-extractive or generative characterizations of person entities from a corpus using GPT-2. We use well-articulated articles from several well-known news media houses as a corpus to build a sound argument for this approach. First, we fine-tune a GPT-2 pre-trained language model with a corpus where specific person entities are characterized. Second, we further fine-tune this with demonstrations of person entity characterizations, created from a corpus of programmatically constructed characterizations. This twice fine-tuned model is primed with manual prompts consisting of entity names that were not previously encountered in the second fine-tuning, to generate a simple sentence about the entity. The results were encouraging, when compared against actual characterizations from the corpus.

Title: ChemSpaceAL: An Efficient Active Learning Methodology Applied to Protein-Specific Molecular Generation. (arXiv:2309.05853v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.05853
• Code URL: null
• Copy Paste: [[2309.05853]] ChemSpaceAL: An Efficient Active Learning Methodology Applied to Protein-Specific Molecular Generation(http://arxiv.org/abs/2309.05853)
• Summary:

  The incredible capabilities of generative artificial intelligence models have inevitably led to their application in the domain of drug discovery. It is therefore of tremendous interest to develop methodologies that enhance the abilities and applicability of these powerful tools. In this work, we present a novel and efficient semi-supervised active learning methodology that allows for the fine-tuning of a generative model with respect to an objective function by strategically operating within a constructed representation of the sample space. In the context of targeted molecular generation, we demonstrate the ability to fine-tune a GPT-based molecular generator with respect to an attractive interaction-based scoring function by strategically operating within a chemical space proxy, thereby maximizing attractive interactions between the generated molecules and a protein target. Importantly, our approach does not require the individual evaluation of all data points that are used for fine-tuning, enabling the incorporation of computationally expensive metrics. We are hopeful that the inherent generality of this methodology ensures that it will remain applicable as this exciting field evolves. To facilitate implementation and reproducibility, we have made all of our software available through the open-source ChemSpaceAL Python package.

large language model

Title: Studying the impacts of pre-training using ChatGPT-generated text on downstream tasks. (arXiv:2309.05668v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.05668
• Code URL: null
• Copy Paste: [[2309.05668]] Studying the impacts of pre-training using ChatGPT-generated text on downstream tasks(http://arxiv.org/abs/2309.05668)
• Summary:

  In recent times, significant advancements have been witnessed in the field of language models, particularly with the emergence of Large Language Models (LLMs) that are trained on vast amounts of data extracted from internet archives. These LLMs, such as ChatGPT, have become widely accessible, allowing users to generate text for various purposes including articles, essays, jokes, and poetry. Given that LLMs are trained on a diverse range of text sources, encompassing platforms like Reddit and Twitter, it is foreseeable that future training datasets will also incorporate text generated by previous iterations of the models themselves. In light of this development, our research aims to investigate the influence of artificial text in the pre-training phase of language models. Specifically, we conducted a comparative analysis between a language model, RoBERTa, pre-trained using CNN/DailyMail news articles, and ChatGPT, which employed the same articles for its training and evaluated their performance on three downstream tasks as well as their potential gender bias, using sentiment analysis as a metric. Through a series of experiments, we demonstrate that the utilization of artificial text during pre-training does not have a significant impact on either the performance of the models in downstream tasks or their gender bias. In conclusion, our findings suggest that the inclusion of text generated by LLMs in their own pre-training process does not yield substantial effects on the subsequent performance of the models in downstream tasks or their potential gender bias.

Title: Large Language Model for Science: A Study on P vs. NP. (arXiv:2309.05689v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.05689
• Code URL: https://github.com/microsoft/LMOps/tree/main/LLM4Science
• Copy Paste: [[2309.05689]] Large Language Model for Science: A Study on P vs(http://arxiv.org/abs/2309.05689)
• Summary:

  In this work, we use large language models (LLMs) to augment and accelerate research on the P versus NP problem, one of the most important open problems in theoretical computer science and mathematics. Specifically, we propose Socratic reasoning, a general framework that promotes in-depth thinking with LLMs for complex problem-solving. Socratic reasoning encourages LLMs to recursively discover, solve, and integrate problems while facilitating self-evaluation and refinement. Our pilot study on the P vs. NP problem shows that GPT-4 successfully produces a proof schema and engages in rigorous reasoning throughout 97 dialogue turns, concluding "P $\neq$ NP", which is in alignment with (Xu and Zhou, 2023). The investigation uncovers novel insights within the extensive solution space of LLMs, shedding light on LLM for Science.

Title: PACE: Prompting and Augmentation for Calibrated Confidence Estimation with GPT-4 in Cloud Incident Root Cause Analysis. (arXiv:2309.05833v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.05833
• Code URL: null
• Copy Paste: [[2309.05833]] PACE: Prompting and Augmentation for Calibrated Confidence Estimation with GPT-4 in Cloud Incident Root Cause Analysis(http://arxiv.org/abs/2309.05833)
• Summary:

  In recent years, the transition to cloud-based platforms in the IT sector has emphasized the significance of cloud incident root cause analysis to ensure service reliability and maintain customer trust. Central to this process is the efficient determination of root causes, a task made challenging due to the complex nature of contemporary cloud infrastructures. Despite the proliferation of AI-driven tools for root cause identification, their applicability remains limited by the inconsistent quality of their outputs. This paper introduces a method for enhancing confidence estimation in root cause analysis tools by prompting retrieval-augmented large language models (LLMs). This approach operates in two phases. Initially, the model evaluates its confidence based on historical incident data, considering its assessment of the evidence strength. Subsequently, the model reviews the root cause generated by the predictor. An optimization step then combines these evaluations to determine the final confidence assignment. Experimental results illustrate that our method enables the model to articulate its confidence effectively, providing a more calibrated score. We address research questions evaluating the ability of our method to produce calibrated confidence scores using LLMs, the impact of domain-specific retrieved examples on confidence estimates, and its potential generalizability across various root cause analysis models. Through this, we aim to bridge the confidence estimation gap, aiding on-call engineers in decision-making and bolstering the efficiency of cloud incident management.

Title: Stochastic LLMs do not Understand Language: Towards Symbolic, Explainable and Ontologically Based LLMs. (arXiv:2309.05918v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.05918
• Code URL: null
• Copy Paste: [[2309.05918]] Stochastic LLMs do not Understand Language: Towards Symbolic, Explainable and Ontologically Based LLMs(http://arxiv.org/abs/2309.05918)
• Summary:

  In our opinion the exuberance surrounding the relative success of data-driven large language models (LLMs) is slightly misguided and for several reasons (i) LLMs cannot be relied upon for factual information since for LLMs all ingested text (factual or non-factual) was created equal; (ii) due to their subsymbolic na-ture, whatever 'knowledge' these models acquire about language will always be buried in billions of microfeatures (weights), none of which is meaningful on its own; and (iii) LLMs will often fail to make the correct inferences in several linguistic contexts (e.g., nominal compounds, copredication, quantifier scope ambi-guities, intensional contexts. Since we believe the relative success of data-driven large language models (LLMs) is not a reflection on the symbolic vs. subsymbol-ic debate but a reflection on applying the successful strategy of a bottom-up reverse engineering of language at scale, we suggest in this paper applying the effective bottom-up strategy in a symbolic setting resulting in symbolic, explainable, and ontologically grounded language models.

Title: Balanced and Explainable Social Media Analysis for Public Health with Large Language Models. (arXiv:2309.05951v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.05951
• Code URL: https://github.com/yanjiangjerry/alex
• Copy Paste: [[2309.05951]] Balanced and Explainable Social Media Analysis for Public Health with Large Language Models(http://arxiv.org/abs/2309.05951)
• Summary:

  As social media becomes increasingly popular, more and more public health activities emerge, which is worth noting for pandemic monitoring and government decision-making. Current techniques for public health analysis involve popular models such as BERT and large language models (LLMs). Although recent progress in LLMs has shown a strong ability to comprehend knowledge by being fine-tuned on specific domain datasets, the costs of training an in-domain LLM for every specific public health task are especially expensive. Furthermore, such kinds of in-domain datasets from social media are generally highly imbalanced, which will hinder the efficiency of LLMs tuning. To tackle these challenges, the data imbalance issue can be overcome by sophisticated data augmentation methods for social media datasets. In addition, the ability of the LLMs can be effectively utilised by prompting the model properly. In light of the above discussion, in this paper, a novel ALEX framework is proposed for social media analysis on public health. Specifically, an augmentation pipeline is developed to resolve the data imbalance issue. Furthermore, an LLMs explanation mechanism is proposed by prompting an LLM with the predicted results from BERT models. Extensive experiments conducted on three tasks at the Social Media Mining for Health 2023 (SMM4H) competition with the first ranking in two tasks demonstrate the superior performance of the proposed ALEX method. Our code has been released in https://github.com/YanJiangJerry/ALEX.

Title: The Moral Machine Experiment on Large Language Models. (arXiv:2309.05958v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.05958
• Code URL: https://github.com/kztakemoto/mmllm
• Copy Paste: [[2309.05958]] The Moral Machine Experiment on Large Language Models(http://arxiv.org/abs/2309.05958)
• Summary:

  As large language models (LLMs) become more deeply integrated into various sectors, understanding how they make moral judgments has become crucial, particularly in the realm of autonomous driving. This study utilized the Moral Machine framework to investigate the ethical decision-making tendencies of prominent LLMs, including GPT-3.5, GPT-4, PaLM 2, and Llama 2, comparing their responses to human preferences. While LLMs' and humans' preferences such as prioritizing humans over pets and favoring saving more lives are broadly aligned, PaLM 2 and Llama 2, especially, evidence distinct deviations. Additionally, despite the qualitative similarities between the LLM and human preferences, there are significant quantitative disparities, suggesting that LLMs might lean toward more uncompromising decisions, compared to the milder inclinations of humans. These insights elucidate the ethical frameworks of LLMs and their potential implications for autonomous driving.

Title: BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models. (arXiv:2309.06085v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06085
• Code URL: null
• Copy Paste: [[2309.06085]] BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models(http://arxiv.org/abs/2309.06085)
• Summary:

  The rapid development of Large Language Models (LLMs) and the emergence of novel abilities with scale have necessitated the construction of holistic, diverse and challenging benchmarks such as HELM and BIG-bench. However, at the moment, most of these benchmarks focus only on performance in English and evaluations that include Southeast Asian (SEA) languages are few in number. We therefore propose BHASA, a holistic linguistic and cultural evaluation suite for LLMs in SEA languages. It comprises three components: (1) a NLP benchmark covering eight tasks across Natural Language Understanding (NLU), Generation (NLG) and Reasoning (NLR) tasks, (2) LINDSEA, a linguistic diagnostic toolkit that spans the gamut of linguistic phenomena including syntax, semantics and pragmatics, and (3) a cultural diagnostics dataset that probes for both cultural representation and sensitivity. For this preliminary effort, we implement the NLP benchmark only for Indonesian, Vietnamese, Thai and Tamil, and we only include Indonesian and Tamil for LINDSEA and the cultural diagnostics dataset. As GPT-4 is purportedly one of the best-performing multilingual LLMs at the moment, we use it as a yardstick to gauge the capabilities of LLMs in the context of SEA languages. Our initial experiments on GPT-4 with BHASA find it lacking in various aspects of linguistic capabilities, cultural representation and sensitivity in the targeted SEA languages. BHASA is a work in progress and will continue to be improved and expanded in the future.

Title: The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models. (arXiv:2309.06236v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06236
• Code URL: null
• Copy Paste: [[2309.06236]] The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models(http://arxiv.org/abs/2309.06236)
• Summary:

  Large Language Models (LLMs) have demonstrated remarkable generalization across diverse tasks, leading individuals to increasingly use them as personal assistants and universal computing engines. Nevertheless, a notable obstacle emerges when feeding numerical/temporal data into these models, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. Here, we discuss recent works that employ LLMs for human-centric tasks such as in mobile health sensing and present a case study showing that popular LLMs tokenize temporal data incorrectly. To address that, we highlight potential solutions such as prompt tuning with lightweight embedding layers as well as multimodal adapters, that can help bridge this "modality gap". While the capability of language models to generalize to other modalities with minimal or no finetuning is exciting, this paper underscores the fact that their outputs cannot be meaningful if they stumble over input nuances.

Title: Learning to Predict Concept Ordering for Common Sense Generation. (arXiv:2309.06363v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06363
• Code URL: https://github.com/tianhuizhang/concept_ordering
• Copy Paste: [[2309.06363]] Learning to Predict Concept Ordering for Common Sense Generation(http://arxiv.org/abs/2309.06363)
• Summary:

  Prior work has shown that the ordering in which concepts are shown to a commonsense generator plays an important role, affecting the quality of the generated sentence. However, it remains a challenge to determine the optimal ordering of a given set of concepts such that a natural sentence covering all the concepts could be generated from a pretrained generator. To understand the relationship between the ordering of the input concepts and the quality of the generated sentences, we conduct a systematic study considering multiple language models (LMs) and concept ordering strategies. We find that BART-large model consistently outperforms all other LMs considered in this study when fine-tuned using the ordering of concepts as they appear in CommonGen training data as measured using multiple evaluation metrics. Moreover, the larger GPT3-based large language models (LLMs) variants do not necessarily outperform much smaller LMs on this task, even when fine-tuned on task-specific training data. Interestingly, human annotators significantly reorder input concept sets when manually writing sentences covering those concepts, and this ordering provides the best sentence generations independently of the LM used for the generation, outperforming a probabilistic concept ordering baseline

Title: Framework-Based Qualitative Analysis of Free Responses of Large Language Models: Algorithmic Fidelity. (arXiv:2309.06364v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06364
• Code URL: null
• Copy Paste: [[2309.06364]] Framework-Based Qualitative Analysis of Free Responses of Large Language Models: Algorithmic Fidelity(http://arxiv.org/abs/2309.06364)
• Summary:

  Today, using Large-scale generative Language Models (LLMs) it is possible to simulate free responses to interview questions like those traditionally analyzed using qualitative research methods. Qualitative methodology encompasses a broad family of techniques involving manual analysis of open-ended interviews or conversations conducted freely in natural language. Here we consider whether artificial "silicon participants" generated by LLMs may be productively studied using qualitative methods aiming to produce insights that could generalize to real human populations. The key concept in our analysis is algorithmic fidelity, a term introduced by Argyle et al. (2023) capturing the degree to which LLM-generated outputs mirror human sub-populations' beliefs and attitudes. By definition, high algorithmic fidelity suggests latent beliefs elicited from LLMs may generalize to real humans, whereas low algorithmic fidelity renders such research invalid. Here we used an LLM to generate interviews with silicon participants matching specific demographic characteristics one-for-one with a set of human participants. Using framework-based qualitative analysis, we showed the key themes obtained from both human and silicon participants were strikingly similar. However, when we analyzed the structure and tone of the interviews we found even more striking differences. We also found evidence of the hyper-accuracy distortion described by Aher et al. (2023). We conclude that the LLM we tested (GPT-3.5) does not have sufficient algorithmic fidelity to expect research on it to generalize to human populations. However, the rapid pace of LLM research makes it plausible this could change in the future. Thus we stress the need to establish epistemic norms now around how to assess validity of LLM-based qualitative research, especially concerning the need to ensure representation of heterogeneous lived experiences.

Title: Towards Reliable and Fluent Large Language Models: Incorporating Feedback Learning Loops in QA Systems. (arXiv:2309.06384v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06384
• Code URL: null
• Copy Paste: [[2309.06384]] Towards Reliable and Fluent Large Language Models: Incorporating Feedback Learning Loops in QA Systems(http://arxiv.org/abs/2309.06384)
• Summary:

  Large language models (LLMs) have emerged as versatile tools in various daily applications. However, they are fraught with issues that undermine their utility and trustworthiness. These include the incorporation of erroneous references (citation), the generation of hallucinated information (correctness), and the inclusion of superfluous or omission of crucial details (fluency). To ameliorate these concerns, this study makes several key contributions. First, we build a dataset to train a critic model capable of evaluating the citation, correctness, and fluency of responses generated by LLMs in QA systems. Second, we propose an automated feedback mechanism that leverages the critic model to offer real-time feedback on heterogeneous aspects of generated text. Third, we introduce a feedback learning loop that uses this critic model to iteratively improve the performance of the LLM responsible for response generation. Experimental results demonstrate the efficacy of our approach, showing substantial improvements in citation and fluency metrics for ChatGPT, including a 4% precision increase in citation and an approximately 8% enhancement in the MAUVE metric for fluency, while maintaining high levels of correctness.

Title: Radiology-Llama2: Best-in-Class Large Language Model for Radiology. (arXiv:2309.06419v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.06419
• Code URL: null
• Copy Paste: [[2309.06419]] Radiology-Llama2: Best-in-Class Large Language Model for Radiology(http://arxiv.org/abs/2309.06419)
• Summary:

  This paper introduces Radiology-Llama2, a large language model specialized for radiology through a process known as instruction tuning. Radiology-Llama2 is based on the Llama2 architecture and further trained on a large dataset of radiology reports to generate coherent and clinically useful impressions from radiological findings. Quantitative evaluations using ROUGE metrics on the MIMIC-CXR and OpenI datasets demonstrate that Radiology-Llama2 achieves state-of-the-art performance compared to other generative language models, with a Rouge-1 score of 0.4834 on MIMIC-CXR and 0.4185 on OpenI. Additional assessments by radiology experts highlight the model's strengths in understandability, coherence, relevance, conciseness, and clinical utility. The work illustrates the potential of localized language models designed and tuned for specialized domains like radiology. When properly evaluated and deployed, such models can transform fields like radiology by automating rote tasks and enhancing human expertise.

Title: A compendium of data sources for data science, machine learning, and artificial intelligence. (arXiv:2309.05682v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.05682
• Code URL: null
• Copy Paste: [[2309.05682]] A compendium of data sources for data science, machine learning, and artificial intelligence(http://arxiv.org/abs/2309.05682)
• Summary:

  Recent advances in data science, machine learning, and artificial intelligence, such as the emergence of large language models, are leading to an increasing demand for data that can be processed by such models. While data sources are application-specific, and it is impossible to produce an exhaustive list of such data sources, it seems that a comprehensive, rather than complete, list would still benefit data scientists and machine learning experts of all levels of seniority. The goal of this publication is to provide just such an (inevitably incomplete) list -- or compendium -- of data sources across multiple areas of applications, including finance and economics, legal (laws and regulations), life sciences (medicine and drug discovery), news sentiment and social media, retail and ecommerce, satellite imagery, and shipping and logistics, and sports.

Title: Efficient Memory Management for Large Language Model Serving with PagedAttention. (arXiv:2309.06180v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.06180
• Code URL: https://github.com/vllm-project/vllm
• Copy Paste: [[2309.06180]] Efficient Memory Management for Large Language Model Serving with PagedAttention(http://arxiv.org/abs/2309.06180)
• Summary:

  High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4$\times$ with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm

segmentation

Title: Self-Correlation and Cross-Correlation Learning for Few-Shot Remote Sensing Image Semantic Segmentation. (arXiv:2309.05840v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05840
• Code URL: https://github.com/linhanwang/sccnet
• Copy Paste: [[2309.05840]] Self-Correlation and Cross-Correlation Learning for Few-Shot Remote Sensing Image Semantic Segmentation(http://arxiv.org/abs/2309.05840)
• Summary:

  Remote sensing image semantic segmentation is an important problem for remote sensing image interpretation. Although remarkable progress has been achieved, existing deep neural network methods suffer from the reliance on massive training data. Few-shot remote sensing semantic segmentation aims at learning to segment target objects from a query image using only a few annotated support images of the target class. Most existing few-shot learning methods stem primarily from their sole focus on extracting information from support images, thereby failing to effectively address the large variance in appearance and scales of geographic objects. To tackle these challenges, we propose a Self-Correlation and Cross-Correlation Learning Network for the few-shot remote sensing image semantic segmentation. Our model enhances the generalization by considering both self-correlation and cross-correlation between support and query images to make segmentation predictions. To further explore the self-correlation with the query image, we propose to adopt a classical spectral method to produce a class-agnostic segmentation mask based on the basic visual information of the image. Extensive experiments on two remote sensing image datasets demonstrate the effectiveness and superiority of our model in few-shot remote sensing image semantic segmentation. Code and models will be accessed at https://github.com/linhanwang/SCCNe.

Title: Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning. (arXiv:2309.05904v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05904
• Code URL: null
• Copy Paste: [[2309.05904]] Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning(http://arxiv.org/abs/2309.05904)
• Summary:

  Recently, multi-modal vision-language foundation models have gained significant attention in the medical field. While these models offer great opportunities, they still face a number of challenges, such as the requirement for fine-grained knowledge understanding in computer-aided diagnosis and capability of utilizing very limited or no task-specific labeled data in real-world clinical applications. In this study, we present MaCo, a novel multi-modal medical foundation model that explores masked contrastive learning to achieve granular alignment and zero-shot learning for a variety of medical imaging tasks. MaCo incorporates a correlation weighting mechanism to adjust the correlation between masked image patches and their corresponding reports, thereby enhancing the representation learning capabilities. We evaluate MaCo on six well-known open-source X-ray datasets, and the experimental results show it outperforms seven state-of-the-art approaches for classification, segmentation, and zero-shot phase grounding, demonstrating its great potential to promote a wide range of medical image analysis tasks.

Title: Medical Image Segmentation with Belief Function Theory and Deep Learning. (arXiv:2309.05914v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05914
• Code URL: null
• Copy Paste: [[2309.05914]] Medical Image Segmentation with Belief Function Theory and Deep Learning(http://arxiv.org/abs/2309.05914)
• Summary:

  Deep learning has shown promising contributions in medical image segmentation with powerful learning and feature representation abilities. However, it has limitations for reasoning with and combining imperfect (imprecise, uncertain, and partial) information. In this thesis, we study medical image segmentation approaches with belief function theory and deep learning, specifically focusing on information modeling and fusion based on uncertain evidence.

  First, we review existing belief function theory-based medical image segmentation methods and discuss their advantages and challenges. Second, we present a semi-supervised medical image segmentation framework to decrease the uncertainty caused by the lack of annotations with evidential segmentation and evidence fusion. Third, we compare two evidential classifiers, evidential neural network and radial basis function network, and show the effectiveness of belief function theory in uncertainty quantification; we use the two evidential classifiers with deep neural networks to construct deep evidential models for lymphoma segmentation. Fourth, we present a multimodal medical image fusion framework taking into account the reliability of each MR image source when performing different segmentation tasks using mass functions and contextual discounting.

Title: Beyond Generation: Harnessing Text to Image Models for Object Detection and Segmentation. (arXiv:2309.05956v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05956
• Code URL: null
• Copy Paste: [[2309.05956]] Beyond Generation: Harnessing Text to Image Models for Object Detection and Segmentation(http://arxiv.org/abs/2309.05956)
• Summary:

  We propose a new paradigm to automatically generate training data with accurate labels at scale using the text-to-image synthesis frameworks (e.g., DALL-E, Stable Diffusion, etc.). The proposed approach1 decouples training data generation into foreground object generation, and contextually coherent background generation. To generate foreground objects, we employ a straightforward textual template, incorporating the object class name as input prompts. This is fed into a text-to-image synthesis framework, producing various foreground images set against isolated backgrounds. A foreground-background segmentation algorithm is then used to generate foreground object masks. To generate context images, we begin by creating language descriptions of the context. This is achieved by applying an image captioning method to a small set of images representing the desired context. These textual descriptions are then transformed into a diverse array of context images via a text-to-image synthesis framework. Subsequently, we composite these with the foreground object masks produced in the initial step, utilizing a cut-and-paste method, to formulate the training data. We demonstrate the advantages of our approach on five object detection and segmentation datasets, including Pascal VOC and COCO. We found that detectors trained solely on synthetic data produced by our method achieve performance comparable to those trained on real data (Fig. 1). Moreover, a combination of real and synthetic data yields even much better results. Further analysis indicates that the synthetic data distribution complements the real data distribution effectively. Additionally, we emphasize the compositional nature of our data generation approach in out-of-distribution and zero-shot data generation scenarios. We open-source our code at https://github.com/gyhandy/Text2Image-for-Detection

Title: FLDNet: A Foreground-Aware Network for Polyp Segmentation Leveraging Long-Distance Dependencies. (arXiv:2309.05987v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05987
• Code URL: null
• Copy Paste: [[2309.05987]] FLDNet: A Foreground-Aware Network for Polyp Segmentation Leveraging Long-Distance Dependencies(http://arxiv.org/abs/2309.05987)
• Summary:

  Given the close association between colorectal cancer and polyps, the diagnosis and identification of colorectal polyps play a critical role in the detection and surgical intervention of colorectal cancer. In this context, the automatic detection and segmentation of polyps from various colonoscopy images has emerged as a significant problem that has attracted broad attention. Current polyp segmentation techniques face several challenges: firstly, polyps vary in size, texture, color, and pattern; secondly, the boundaries between polyps and mucosa are usually blurred, existing studies have focused on learning the local features of polyps while ignoring the long-range dependencies of the features, and also ignoring the local context and global contextual information of the combined features. To address these challenges, we propose FLDNet (Foreground-Long-Distance Network), a Transformer-based neural network that captures long-distance dependencies for accurate polyp segmentation. Specifically, the proposed model consists of three main modules: a pyramid-based Transformer encoder, a local context module, and a foreground-Aware module. Multilevel features with long-distance dependency information are first captured by the pyramid-based transformer encoder. On the high-level features, the local context module obtains the local characteristics related to the polyps by constructing different local context information. The coarse map obtained by decoding the reconstructed highest-level features guides the feature fusion process in the foreground-Aware module of the high-level features to achieve foreground enhancement of the polyps. Our proposed method, FLDNet, was evaluated using seven metrics on common datasets and demonstrated superiority over state-of-the-art methods on widely-used evaluation measures.

Title: ATTA: Anomaly-aware Test-Time Adaptation for Out-of-Distribution Detection in Segmentation. (arXiv:2309.05994v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.05994
• Code URL: https://github.com/gaozhitong/atta
• Copy Paste: [[2309.05994]] ATTA: Anomaly-aware Test-Time Adaptation for Out-of-Distribution Detection in Segmentation(http://arxiv.org/abs/2309.05994)
• Summary:

  Recent advancements in dense out-of-distribution (OOD) detection have primarily focused on scenarios where the training and testing datasets share a similar domain, with the assumption that no domain shift exists between them. However, in real-world situations, domain shift often exits and significantly affects the accuracy of existing out-of-distribution (OOD) detection models. In this work, we propose a dual-level OOD detection framework to handle domain shift and semantic shift jointly. The first level distinguishes whether domain shift exists in the image by leveraging global low-level features, while the second level identifies pixels with semantic shift by utilizing dense high-level feature maps. In this way, we can selectively adapt the model to unseen domains as well as enhance model's capacity in detecting novel classes. We validate the efficacy of our proposed method on several OOD segmentation benchmarks, including those with significant domain shifts and those without, observing consistent performance improvements across various baseline models.

Title: Real-Time Semantic Segmentation: A Brief Survey & Comparative Study in Remote Sensing. (arXiv:2309.06047v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06047
• Code URL: null
• Copy Paste: [[2309.06047]] Real-Time Semantic Segmentation: A Brief Survey & Comparative Study in Remote Sensing(http://arxiv.org/abs/2309.06047)
• Summary:

  Real-time semantic segmentation of remote sensing imagery is a challenging task that requires a trade-off between effectiveness and efficiency. It has many applications including tracking forest fires, detecting changes in land use and land cover, crop health monitoring, and so on. With the success of efficient deep learning methods (i.e., efficient deep neural networks) for real-time semantic segmentation in computer vision, researchers have adopted these efficient deep neural networks in remote sensing image analysis. This paper begins with a summary of the fundamental compression methods for designing efficient deep neural networks and provides a brief but comprehensive survey, outlining the recent developments in real-time semantic segmentation of remote sensing imagery. We examine several seminal efficient deep learning methods, placing them in a taxonomy based on the network architecture design approach. Furthermore, we evaluate the quality and efficiency of some existing efficient deep neural networks on a publicly available remote sensing semantic segmentation benchmark dataset, the OpenEarthMap. The experimental results of an extensive comparative study demonstrate that most of the existing efficient deep neural networks have good segmentation quality, but they suffer low inference speed (i.e., high latency rate), which may limit their capability of deployment in real-time applications of remote sensing image segmentation. We provide some insights into the current trend and future research directions for real-time semantic segmentation of remote sensing imagery.

Title: Active Label Refinement for Semantic Segmentation of Satellite Images. (arXiv:2309.06159v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06159
• Code URL: null
• Copy Paste: [[2309.06159]] Active Label Refinement for Semantic Segmentation of Satellite Images(http://arxiv.org/abs/2309.06159)
• Summary:

  Remote sensing through semantic segmentation of satellite images contributes to the understanding and utilisation of the earth's surface. For this purpose, semantic segmentation networks are typically trained on large sets of labelled satellite images. However, obtaining expert labels for these images is costly. Therefore, we propose to rely on a low-cost approach, e.g. crowdsourcing or pretrained networks, to label the images in the first step. Since these initial labels are partially erroneous, we use active learning strategies to cost-efficiently refine the labels in the second step. We evaluate the active learning strategies using satellite images of Bengaluru in India, labelled with land cover and land use labels. Our experimental results suggest that an active label refinement to improve the semantic segmentation network's performance is beneficial.

Title: Computer Vision Pipeline for Automated Antarctic Krill Analysis. (arXiv:2309.06188v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06188
• Code URL: null
• Copy Paste: [[2309.06188]] Computer Vision Pipeline for Automated Antarctic Krill Analysis(http://arxiv.org/abs/2309.06188)
• Summary:

  British Antarctic Survey (BAS) researchers launch annual expeditions to the Antarctic in order to estimate Antarctic Krill biomass and assess the change from previous years. These comparisons provide insight into the effects of the current environment on this key component of the marine food chain. In this work we have developed tools for automating the data collection and analysis process, using web-based image annotation tools and deep learning image classification and regression models. We achieve highly accurate krill instance segmentation results with an average 77.28% AP score, as well as separate maturity stage and length estimation of krill specimens with 62.99% accuracy and a 1.96 mm length error respectively.

Title: 360$^\circ$ from a Single Camera: A Few-Shot Approach for LiDAR Segmentation. (arXiv:2309.06197v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06197
• Code URL: null
• Copy Paste: [[2309.06197]] 360$^\circ$ from a Single Camera: A Few-Shot Approach for LiDAR Segmentation(http://arxiv.org/abs/2309.06197)
• Summary:

  Deep learning applications on LiDAR data suffer from a strong domain gap when applied to different sensors or tasks. In order for these methods to obtain similar accuracy on different data in comparison to values reported on public benchmarks, a large scale annotated dataset is necessary. However, in practical applications labeled data is costly and time consuming to obtain. Such factors have triggered various research in label-efficient methods, but a large gap remains to their fully-supervised counterparts. Thus, we propose ImageTo360, an effective and streamlined few-shot approach to label-efficient LiDAR segmentation. Our method utilizes an image teacher network to generate semantic predictions for LiDAR data within a single camera view. The teacher is used to pretrain the LiDAR segmentation student network, prior to optional fine-tuning on 360$^\circ$ data. Our method is implemented in a modular manner on the point level and as such is generalizable to different architectures. We improve over the current state-of-the-art results for label-efficient methods and even surpass some traditional fully-supervised segmentation networks.

Title: OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentation. (arXiv:2309.06276v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06276
• Code URL: null
• Copy Paste: [[2309.06276]] OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentation(http://arxiv.org/abs/2309.06276)
• Summary:

  Temporal action segmentation is typically achieved by discovering the dramatic variances in global visual descriptors. In this paper, we explore the merits of local features by proposing the unsupervised framework of Object-centric Temporal Action Segmentation (OTAS). Broadly speaking, OTAS consists of self-supervised global and local feature extraction modules as well as a boundary selection module that fuses the features and detects salient boundaries for action segmentation. As a second contribution, we discuss the pros and cons of existing frame-level and boundary-level evaluation metrics. Through extensive experiments, we find OTAS is superior to the previous state-of-the-art method by $41\%$ on average in terms of our recommended F1 score. Surprisingly, OTAS even outperforms the ground-truth human annotations in the user study. Moreover, OTAS is efficient enough to allow real-time inference.

Title: Exploring Flat Minima for Domain Generalization with Large Learning Rates. (arXiv:2309.06337v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06337
• Code URL: null
• Copy Paste: [[2309.06337]] Exploring Flat Minima for Domain Generalization with Large Learning Rates(http://arxiv.org/abs/2309.06337)
• Summary:

  Domain Generalization (DG) aims to generalize to arbitrary unseen domains. A promising approach to improve model generalization in DG is the identification of flat minima. One typical method for this task is SWAD, which involves averaging weights along the training trajectory. However, the success of weight averaging depends on the diversity of weights, which is limited when training with a small learning rate. Instead, we observe that leveraging a large learning rate can simultaneously promote weight diversity and facilitate the identification of flat regions in the loss landscape. However, employing a large learning rate suffers from the convergence problem, which cannot be resolved by simply averaging the training weights. To address this issue, we introduce a training strategy called Lookahead which involves the weight interpolation, instead of average, between fast and slow weights. The fast weight explores the weight space with a large learning rate, which is not converged while the slow weight interpolates with it to ensure the convergence. Besides, weight interpolation also helps identify flat minima by implicitly optimizing the local entropy loss that measures flatness. To further prevent overfitting during training, we propose two variants to regularize the training weight with weighted averaged weight or with accumulated history weight. Taking advantage of this new perspective, our methods achieve state-of-the-art performance on both classification and semantic segmentation domain generalization benchmarks. The code is available at https://github.com/koncle/DG-with-Large-LR.

Title: Padding-free Convolution based on Preservation of Differential Characteristics of Kernels. (arXiv:2309.06370v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06370
• Code URL: null
• Copy Paste: [[2309.06370]] Padding-free Convolution based on Preservation of Differential Characteristics of Kernels(http://arxiv.org/abs/2309.06370)
• Summary:

  Convolution is a fundamental operation in image processing and machine learning. Aimed primarily at maintaining image size, padding is a key ingredient of convolution, which, however, can introduce undesirable boundary effects. We present a non-padding-based method for size-keeping convolution based on the preservation of differential characteristics of kernels. The main idea is to make convolution over an incomplete sliding window "collapse" to a linear differential operator evaluated locally at its central pixel, which no longer requires information from the neighbouring missing pixels. While the underlying theory is rigorous, our final formula turns out to be simple: the convolution over an incomplete window is achieved by convolving its nearest complete window with a transformed kernel. This formula is computationally lightweight, involving neither interpolation or extrapolation nor restrictions on image and kernel sizes. Our method favours data with smooth boundaries, such as high-resolution images and fields from physics. Our experiments include: i) filtering analytical and non-analytical fields from computational physics and, ii) training convolutional neural networks (CNNs) for the tasks of image classification, semantic segmentation and super-resolution reconstruction. In all these experiments, our method has exhibited visible superiority over the compared ones.

Title: Attention De-sparsification Matters: Inducing Diversity in Digital Pathology Representation Learning. (arXiv:2309.06439v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.06439
• Code URL: null
• Copy Paste: [[2309.06439]] Attention De-sparsification Matters: Inducing Diversity in Digital Pathology Representation Learning(http://arxiv.org/abs/2309.06439)
• Summary:

  We propose DiRL, a Diversity-inducing Representation Learning technique for histopathology imaging. Self-supervised learning techniques, such as contrastive and non-contrastive approaches, have been shown to learn rich and effective representations of digitized tissue samples with limited pathologist supervision. Our analysis of vanilla SSL-pretrained models' attention distribution reveals an insightful observation: sparsity in attention, i.e, models tends to localize most of their attention to some prominent patterns in the image. Although attention sparsity can be beneficial in natural images due to these prominent patterns being the object of interest itself, this can be sub-optimal in digital pathology; this is because, unlike natural images, digital pathology scans are not object-centric, but rather a complex phenotype of various spatially intermixed biological components. Inadequate diversification of attention in these complex images could result in crucial information loss. To address this, we leverage cell segmentation to densely extract multiple histopathology-specific representations, and then propose a prior-guided dense pretext task for SSL, designed to match the multiple corresponding representations between the views. Through this, the model learns to attend to various components more closely and evenly, thus inducing adequate diversification in attention for capturing context rich representations. Through quantitative and qualitative analysis on multiple tasks across cancer types, we demonstrate the efficacy of our method and observe that the attention is more globally distributed.