secure

Title: Secure Computation over Encrypted Databases. (arXiv:2308.02878v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02878
• Code URL: null
• Copy Paste: [[2308.02878]] Secure Computation over Encrypted Databases(http://arxiv.org/abs/2308.02878)
• Summary:

  Sensitive applications running on the cloud often require data to be stored in an encrypted domain. To run data mining algorithms on such data, partially homomorphic encryption schemes (allowing certain operations in the ciphertext domain) have been devised. One such line of work yields schemes for secure \textit{k-nearest neighbors} computation that is designed to provide both \textit{Data Privacy} and \textit{Query Privacy}. Enhancements in this area further ensure that the data owner approves each query issued by a query user before the cloud server processes it. In this work, we describe an attack that invalidates the \textit{key confidentiality} claim, which further invalidates the \textit{Data Privacy} claim for these schemes. We show that a query user can specially tailor a query to extract information about the secret key used to encrypt the data points. Furthermore, the recovered secret information can be used to derive all the plaintext data points breaking \textit{data privacy}. We then suggest enhanced encryption schemes that make such attacks on \textit{data privacy} impossible while incurring meager additional costs in performance.

security

Title: Incentivizing Gigaton-Scale Carbon Dioxide Removal via a Climate-Positive Blockchain. (arXiv:2308.02653v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02653
• Code URL: null
• Copy Paste: [[2308.02653]] Incentivizing Gigaton-Scale Carbon Dioxide Removal via a Climate-Positive Blockchain(http://arxiv.org/abs/2308.02653)
• Summary:

  A new crypto token is proposed as an incentive mechanism to remove CO2 from the atmosphere permanently at gigaton scale. The token facilitates CO2 removal (CDR) by providing financial incentives to those that are removing CO2 and an opportunity to provide additional financial resources for CDR by the public. The new token will be native to a blockchain that uses a Proof-of-Useful-Work (PoUW) consensus mechanism. The useful work will be conducted by direct air carbon capture and storage (DACCS) facilities that will compete with each other based on the amount of CO2 captured and permanently stored. In terms of energy consumption, we require that the entire process, comprising DACCS technology and all blockchain operations, be climate positive while accounting for life cycle analysis of equipment used. We describe the underlying reward mechanism coupled with a verification mechanism for CDR. In addition, we consider security features to limit attacks and fraudulent activity. Finally, we outline a roadmap of features that are necessary to fully implement and deploy such a system, but are beyond the current scope of this article.

Title: SoftFlow: Automated HW-SW Confidentiality Verification for Embedded Processors. (arXiv:2308.02694v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02694
• Code URL: null
• Copy Paste: [[2308.02694]] SoftFlow: Automated HW-SW Confidentiality Verification for Embedded Processors(http://arxiv.org/abs/2308.02694)
• Summary:

  Despite its ever-increasing impact, security is not considered as a design objective in commercial electronic design automation (EDA) tools. This results in vulnerabilities being overlooked during the software-hardware design process. Specifically, vulnerabilities that allow leakage of sensitive data might stay unnoticed by standard testing, as the leakage itself might not result in evident functional changes. Therefore, EDA tools are needed to elaborate the confidentiality of sensitive data during the design process. However, state-of-the-art implementations either solely consider the hardware or restrict the expressiveness of the security properties that must be proven. Consequently, more proficient tools are required to assist in the software and hardware design. To address this issue, we propose SoftFlow, an EDA tool that allows determining whether a given software exploits existing leakage paths in hardware. Based on our analysis, the leakage paths can be retained if proven not to be exploited by software. This is desirable if the removal significantly impacts the design's performance or functionality, or if the path cannot be removed as the chip is already manufactured. We demonstrate the feasibility of SoftFlow by identifying vulnerabilities in OpenSSL cryptographic C programs, and redesigning them to avoid leakage of cryptographic keys in a RISC-V architecture.

Title: Understanding the RSA algorithm. (arXiv:2308.02785v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02785
• Code URL: null
• Copy Paste: [[2308.02785]] Understanding the RSA algorithm(http://arxiv.org/abs/2308.02785)
• Summary:

  With the emerging importance of cybersecurity, it will be beneficial for a wide community to understand some of the fundamental security mechanisms. The RSA algorithm is one of the essential algorithms used in public-key cryptosystems. Understanding the RSA algorithm requires knowledge regarding number theory, modular arithmetic, etc., which is often beyond the knowledge pool of many beginners in cybersecurity. In this work, we provide an intuitive and onion-peeling style introduction to the RSA algorithm, in which we assume that readers will only have a basic background in mathematics and cybersecurity. Started from three essential goals of public-key cryptosystems, we explained step-by-step how the RSA algorithm achieved these goals. We also used a toy example to further help readers to understand the algorithm from a practical perspective.

Title: DiSPEL: Distributed Security Policy Enforcement for Bus-based SoC. (arXiv:2308.02792v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02792
• Code URL: null
• Copy Paste: [[2308.02792]] DiSPEL: Distributed Security Policy Enforcement for Bus-based SoC(http://arxiv.org/abs/2308.02792)
• Summary:

  The current zero trust model adopted in System-on-Chip (SoC) design is vulnerable to various malicious entities, and modern SoC designs must incorporate various security policies to protect sensitive assets from unauthorized access. These policies involve complex interactions between multiple IP blocks, which poses challenges for SoC designers and security experts when implementing these policies and for system validators when ensuring compliance. Difficulties arise when upgrading policies, reusing IPs for systems targeting different security requirements, and the subsequent increase in design time and time-to-market. This paper proposes a generic and flexible framework, called DiSPEL, for enforcing security policies defined by the user represented in a formal way for any bus-based SoC design. It employs a distributed deployment strategy while ensuring trusted bus operations despite the presence of untrusted IPs. It relies on incorporating a dedicated, centralized module capable of implementing diverse security policies involving bus-level interactions while generating the necessary logic and appending in the bus-level wrapper for IP-level policies. The proposed architecture is generic and independent of specific security policy types supporting both synthesizable and non-synthesizable solutions. The experimental results demonstrate its effectiveness and correctness in enforcing the security requirements and viability due to low overhead in terms of area, delay, and power consumption tested on open-source standard SoC benchmarks.

Title: Meta-Analysis and Systematic Review for Anomaly Network Intrusion Detection Systems: Detection Methods, Dataset, Validation Methodology, and Challenges. (arXiv:2308.02805v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02805
• Code URL: null
• Copy Paste: [[2308.02805]] Meta-Analysis and Systematic Review for Anomaly Network Intrusion Detection Systems: Detection Methods, Dataset, Validation Methodology, and Challenges(http://arxiv.org/abs/2308.02805)
• Summary:

  Intrusion detection systems (IDSs) built on artificial intelligence (AI) are presented as latent mechanisms for actively detecting fresh attacks over a complex network. Although review papers are used the systematic review or simple methods to analyse and criticize the anomaly NIDS works, the current review uses a traditional way as a quantitative description to find current gaps by synthesizing and summarizing the data comparison without considering algorithms performance. This paper presents a systematic and meta-analysis study of AI for network intrusion detection systems (NIDS) focusing on deep learning (DL) and machine learning (ML) approaches in network security. Deep learning algorithms are explained in their structure, and data intrusion network is justified based on an infrastructure of networks and attack types. By conducting a meta-analysis and debating the validation of the DL and ML approach by effectiveness, used dataset, detected attacks, classification task, and time complexity, we offer a thorough benchmarking assessment of the current NIDS-based publications-based systematic approach. The proposed method is considered reviewing works for the anomaly-based network intrusion detection system (anomaly-NIDS) models. Furthermore, the effectiveness of proposed algorithms and selected datasets are discussed for the recent direction and improvements of ML and DL to the NIDS. The future trends for improving an anomaly-IDS for continuing detection in the evolution of cyberattacks are highlighted in several research studies.

privacy

Title: PPIMCE: An In-memory Computing Fabric for Privacy Preserving Computing. (arXiv:2308.02648v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02648
• Code URL: null
• Copy Paste: [[2308.02648]] PPIMCE: An In-memory Computing Fabric for Privacy Preserving Computing(http://arxiv.org/abs/2308.02648)
• Summary:

  Privacy has rapidly become a major concern/design consideration. Homomorphic Encryption (HE) and Garbled Circuits (GC) are privacy-preserving techniques that support computations on encrypted data. HE and GC can complement each other, as HE is more efficient for linear operations, while GC is more effective for non-linear operations. Together, they enable complex computing tasks, such as machine learning, to be performed exactly on ciphertexts. However, HE and GC introduce two major bottlenecks: an elevated computational overhead and high data transfer costs. This paper presents PPIMCE, an in-memory computing (IMC) fabric designed to mitigate both computational overhead and data transfer issues. Through the use of multiple IMC cores for high parallelism, and by leveraging in-SRAM IMC for data management, PPIMCE offers a compact, energy-efficient solution for accelerating HE and GC. PPIMCE achieves a 107X speedup against a CPU implementation of GC. Additionally, PPIMCE achieves a 1,500X and 800X speedup compared to CPU and GPU implementations of CKKS-based HE multiplications. For privacy-preserving machine learning inference, PPIMCE attains a 1,000X speedup compared to CPU and a 12X speedup against CraterLake, the state-of-art privacy preserving computation accelerator.

Title: Resilient and Privacy-Preserving Threshold Vehicular Public Key Infrastructure (VPKI). (arXiv:2308.02711v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02711
• Code URL: null
• Copy Paste: [[2308.02711]] Resilient and Privacy-Preserving Threshold Vehicular Public Key Infrastructure (VPKI)(http://arxiv.org/abs/2308.02711)
• Summary:

  Vehicular Public Key Infrastructure (VPKI) plays a vital role in ensuring secure and privacy-preserving communication in vehicular ad hoc networks (VANETs). However, current VPKI architectures face significant challenges in terms of scalability, resilience, and privacy preservation. This paper proposes a novel threshold-based VPKI architecture to overcome these limitations. Leveraging a Schnorr threshold signature scheme based on elliptic curve cryptography, the proposed architecture eliminates the reliance on individual certificate authorities (CAs) and distributes trust among multiple CAs in a threshold certificate signing approach. This enhances resilience and mitigates the single point-of-failure vulnerability. The architecture also addresses sybil-based misbehaviors through a time-restrictive pseudonym design that eliminates multiple simultaneous use of pseudonyms. Furthermore, the scheme reduces the size and latency of Certificate Revocation List (CRL) distribution by clustering multiple CAs in a threshold setting and adopting a region-specific CRL. The paper presents detailed analysis of the security, privacy and performance benefits of the proposed architecture. Results from the performance evaluation shows the improved resiliency, reduced handover rates, and better scalability potential of the proposed threshold-based VPKI architecture compared to existing techniques. The proposed threshold-based VPKI holds great promise in ensuring secure and privacy-preserving communication in VANETs, paving the way for safer and more efficient vehicular networks.

protect

defense

attack

Title: Can Self-Supervised Representation Learning Methods Withstand Distribution Shifts and Corruptions?. (arXiv:2308.02525v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02525
• Code URL: null
• Copy Paste: [[2308.02525]] Can Self-Supervised Representation Learning Methods Withstand Distribution Shifts and Corruptions?(http://arxiv.org/abs/2308.02525)
• Summary:

  Self-supervised learning in computer vision aims to leverage the inherent structure and relationships within data to learn meaningful representations without explicit human annotation, enabling a holistic understanding of visual scenes. Robustness in vision machine learning ensures reliable and consistent performance, enhancing generalization, adaptability, and resistance to noise, variations, and adversarial attacks. Self-supervised paradigms, namely contrastive learning, knowledge distillation, mutual information maximization, and clustering, have been considered to have shown advances in invariant learning representations. This work investigates the robustness of learned representations of self-supervised learning approaches focusing on distribution shifts and image corruptions in computer vision. Detailed experiments have been conducted to study the robustness of self-supervised learning methods on distribution shifts and image corruptions. The empirical analysis demonstrates a clear relationship between the performance of learned representations within self-supervised paradigms and the severity of distribution shifts and corruptions. Notably, higher levels of shifts and corruptions are found to significantly diminish the robustness of the learned representations. These findings highlight the critical impact of distribution shifts and image corruptions on the performance and resilience of self-supervised learning methods, emphasizing the need for effective strategies to mitigate their adverse effects. The study strongly advocates for future research in the field of self-supervised representation learning to prioritize the key aspects of safety and robustness in order to ensure practical applicability. The source code and results are available on GitHub.

robust

Title: Improving Generalization of Adversarial Training via Robust Critical Fine-Tuning. (arXiv:2308.02533v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02533
• Code URL: null
• Copy Paste: [[2308.02533]] Improving Generalization of Adversarial Training via Robust Critical Fine-Tuning(http://arxiv.org/abs/2308.02533)
• Summary:

  Deep neural networks are susceptible to adversarial examples, posing a significant security risk in critical applications. Adversarial Training (AT) is a well-established technique to enhance adversarial robustness, but it often comes at the cost of decreased generalization ability. This paper proposes Robustness Critical Fine-Tuning (RiFT), a novel approach to enhance generalization without compromising adversarial robustness. The core idea of RiFT is to exploit the redundant capacity for robustness by fine-tuning the adversarially trained model on its non-robust-critical module. To do so, we introduce module robust criticality (MRC), a measure that evaluates the significance of a given module to model robustness under worst-case weight perturbations. Using this measure, we identify the module with the lowest MRC value as the non-robust-critical module and fine-tune its weights to obtain fine-tuned weights. Subsequently, we linearly interpolate between the adversarially trained weights and fine-tuned weights to derive the optimal fine-tuned model weights. We demonstrate the efficacy of RiFT on ResNet18, ResNet34, and WideResNet34-10 models trained on CIFAR10, CIFAR100, and Tiny-ImageNet datasets. Our experiments show that \method can significantly improve both generalization and out-of-distribution robustness by around 1.5% while maintaining or even slightly enhancing adversarial robustness. Code is available at https://github.com/microsoft/robustlearn.

Title: Learning to Generate Training Datasets for Robust Semantic Segmentation. (arXiv:2308.02535v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02535
• Code URL: null
• Copy Paste: [[2308.02535]] Learning to Generate Training Datasets for Robust Semantic Segmentation(http://arxiv.org/abs/2308.02535)
• Summary:

  Semantic segmentation techniques have shown significant progress in recent years, but their robustness to real-world perturbations and data samples not seen during training remains a challenge, particularly in safety-critical applications. In this paper, we propose a novel approach to improve the robustness of semantic segmentation techniques by leveraging the synergy between label-to-image generators and image-to-label segmentation models. Specifically, we design and train Robusta, a novel robust conditional generative adversarial network to generate realistic and plausible perturbed or outlier images that can be used to train reliable segmentation models. We conduct in-depth studies of the proposed generative model, assess the performance and robustness of the downstream segmentation network, and demonstrate that our approach can significantly enhance the robustness of semantic segmentation techniques in the face of real-world perturbations, distribution shifts, and out-of-distribution samples. Our results suggest that this approach could be valuable in safety-critical applications, where the reliability of semantic segmentation techniques is of utmost importance and comes with a limited computational budget in inference. We will release our code shortly.

Title: Food Classification using Joint Representation of Visual and Textual Data. (arXiv:2308.02562v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02562
• Code URL: null
• Copy Paste: [[2308.02562]] Food Classification using Joint Representation of Visual and Textual Data(http://arxiv.org/abs/2308.02562)
• Summary:

  Food classification is an important task in health care. In this work, we propose a multimodal classification framework that uses the modified version of EfficientNet with the Mish activation function for image classification, and the traditional BERT transformer-based network is used for text classification. The proposed network and the other state-of-the-art methods are evaluated on a large open-source dataset, UPMC Food-101. The experimental results show that the proposed network outperforms the other methods, a significant difference of 11.57% and 6.34% in accuracy is observed for image and text classification, respectively, when compared with the second-best performing method. We also compared the performance in terms of accuracy, precision, and recall for text classification using both machine learning and deep learning-based models. The comparative analysis from the prediction results of both images and text demonstrated the efficiency and robustness of the proposed approach.

Title: Discrimination of Radiologists Utilizing Eye-Tracking Technology and Machine Learning: A Case Study. (arXiv:2308.02748v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02748
• Code URL: null
• Copy Paste: [[2308.02748]] Discrimination of Radiologists Utilizing Eye-Tracking Technology and Machine Learning: A Case Study(http://arxiv.org/abs/2308.02748)
• Summary:

  Perception-related errors comprise most diagnostic mistakes in radiology. To mitigate this problem, radiologists employ personalized and high-dimensional visual search strategies, otherwise known as search patterns. Qualitative descriptions of these search patterns, which involve the physician verbalizing or annotating the order he/she analyzes the image, can be unreliable due to discrepancies in what is reported versus the actual visual patterns. This discrepancy can interfere with quality improvement interventions and negatively impact patient care. This study presents a novel discretized feature encoding based on spatiotemporal binning of fixation data for efficient geometric alignment and temporal ordering of eye movement when reading chest X-rays. The encoded features of the eye-fixation data are employed by machine learning classifiers to discriminate between faculty and trainee radiologists. We include a clinical trial case study utilizing the Area Under the Curve (AUC), Accuracy, F1, Sensitivity, and Specificity metrics for class separability to evaluate the discriminability between the two subjects in regard to their level of experience. We then compare the classification performance to state-of-the-art methodologies. A repeatability experiment using a separate dataset, experimental protocol, and eye tracker was also performed using eight subjects to evaluate the robustness of the proposed approach. The numerical results from both experiments demonstrate that classifiers employing the proposed feature encoding methods outperform the current state-of-the-art in differentiating between radiologists in terms of experience level. This signifies the potential impact of the proposed method for identifying radiologists' level of expertise and those who would benefit from additional training.

Title: DeDrift: Robust Similarity Search under Content Drift. (arXiv:2308.02752v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02752
• Code URL: null
• Copy Paste: [[2308.02752]] DeDrift: Robust Similarity Search under Content Drift(http://arxiv.org/abs/2308.02752)
• Summary:

  The statistical distribution of content uploaded and searched on media sharing sites changes over time due to seasonal, sociological and technical factors. We investigate the impact of this "content drift" for large-scale similarity search tools, based on nearest neighbor search in embedding space. Unless a costly index reconstruction is performed frequently, content drift degrades the search accuracy and efficiency. The degradation is especially severe since, in general, both the query and database distributions change.

  We introduce and analyze real-world image and video datasets for which temporal information is available over a long time period. Based on the learnings, we devise DeDrift, a method that updates embedding quantizers to continuously adapt large-scale indexing structures on-the-fly. DeDrift almost eliminates the accuracy degradation due to the query and database content drift while being up to 100x faster than a full index reconstruction.

Title: One-stage Low-resolution Text Recognition with High-resolution Knowledge Transfer. (arXiv:2308.02770v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02770
• Code URL: https://github.com/csguoh/kd-ltr
• Copy Paste: [[2308.02770]] One-stage Low-resolution Text Recognition with High-resolution Knowledge Transfer(http://arxiv.org/abs/2308.02770)
• Summary:

  Recognizing characters from low-resolution (LR) text images poses a significant challenge due to the information deficiency as well as the noise and blur in low-quality images. Current solutions for low-resolution text recognition (LTR) typically rely on a two-stage pipeline that involves super-resolution as the first stage followed by the second-stage recognition. Although this pipeline is straightforward and intuitive, it has to use an additional super-resolution network, which causes inefficiencies during training and testing. Moreover, the recognition accuracy of the second stage heavily depends on the reconstruction quality of the first stage, causing ineffectiveness. In this work, we attempt to address these challenges from a novel perspective: adapting the recognizer to low-resolution inputs by transferring the knowledge from the high-resolution. Guided by this idea, we propose an efficient and effective knowledge distillation framework to achieve multi-level knowledge transfer. Specifically, the visual focus loss is proposed to extract the character position knowledge with resolution gap reduction and character region focus, the semantic contrastive loss is employed to exploit the contextual semantic knowledge with contrastive learning, and the soft logits loss facilitates both local word-level and global sequence-level learning from the soft teacher label. Extensive experiments show that the proposed one-stage pipeline significantly outperforms super-resolution based two-stage frameworks in terms of effectiveness and efficiency, accompanied by favorable robustness. Code is available at https://github.com/csguoh/KD-LTR.

Title: Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratings. (arXiv:2308.02575v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02575
• Code URL: null
• Copy Paste: [[2308.02575]] Is GPT-4 a reliable rater? Evaluating Consistency in GPT-4 Text Ratings(http://arxiv.org/abs/2308.02575)
• Summary:

  This study investigates the consistency of feedback ratings generated by OpenAI's GPT-4, a state-of-the-art artificial intelligence language model, across multiple iterations, time spans and stylistic variations. The model rated responses to tasks within the Higher Education (HE) subject domain of macroeconomics in terms of their content and style. Statistical analysis was conducted in order to learn more about the interrater reliability, consistency of the ratings across iterations and the correlation between ratings in terms of content and style. The results revealed a high interrater reliability with ICC scores ranging between 0.94 and 0.99 for different timespans, suggesting that GPT-4 is capable of generating consistent ratings across repetitions with a clear prompt. Style and content ratings show a high correlation of 0.87. When applying a non-adequate style the average content ratings remained constant, while style ratings decreased, which indicates that the large language model (LLM) effectively distinguishes between these two criteria during evaluation. The prompt used in this study is furthermore presented and explained. Further research is necessary to assess the robustness and reliability of AI models in various use cases.

Title: How Good Are SOTA Fake News Detectors. (arXiv:2308.02727v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02727
• Code URL: https://github.com/miceland2/fake_news_detection
• Copy Paste: [[2308.02727]] How Good Are SOTA Fake News Detectors(http://arxiv.org/abs/2308.02727)
• Summary:

  Automatic fake news detection with machine learning can prevent the dissemination of false statements before they gain many views. Several datasets labeling statements as legitimate or false have been created since the 2016 United States presidential election for the prospect of training machine learning models. We evaluate the robustness of both traditional and deep state-of-the-art models to gauge how well they may perform in the real world. We find that traditional models tend to generalize better to data outside the distribution it was trained on compared to more recently-developed large language models, though the best model to use may depend on the specific task at hand.

Title: SABRE: Robust Bayesian Peer-to-Peer Federated Learning. (arXiv:2308.02747v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02747
• Code URL: null
• Copy Paste: [[2308.02747]] SABRE: Robust Bayesian Peer-to-Peer Federated Learning(http://arxiv.org/abs/2308.02747)
• Summary:

  We introduce SABRE, a novel framework for robust variational Bayesian peer-to-peer federated learning. We analyze the robustness of the known variational Bayesian peer-to-peer federated learning framework (BayP2PFL) against poisoning attacks and subsequently show that BayP2PFL is not robust against those attacks. The new SABRE aggregation methodology is then devised to overcome the limitations of the existing frameworks. SABRE works well in non-IID settings, does not require the majority of the benign nodes over the compromised ones, and even outperforms the baseline algorithm in benign settings. We theoretically prove the robustness of our algorithm against data / model poisoning attacks in a decentralized linear regression setting. Proof-of-Concept evaluations on benchmark data from image classification demonstrate the superiority of SABRE over the existing frameworks under various poisoning attacks.

Title: Approximating Positive Homogeneous Functions with Scale Invariant Neural Networks. (arXiv:2308.02836v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02836
• Code URL: null
• Copy Paste: [[2308.02836]] Approximating Positive Homogeneous Functions with Scale Invariant Neural Networks(http://arxiv.org/abs/2308.02836)
• Summary:

  We investigate to what extent it is possible to solve linear inverse problems with $ReLu$ networks. Due to the scaling invariance arising from the linearity, an optimal reconstruction function $f$ for such a problem is positive homogeneous, i.e., satisfies $f(\lambda x) = \lambda f(x)$ for all non-negative $\lambda$. In a $ReLu$ network, this condition translates to considering networks without bias terms. We first consider recovery of sparse vectors from few linear measurements. We prove that $ReLu$- networks with only one hidden layer cannot even recover $1$-sparse vectors, not even approximately, and regardless of the width of the network. However, with two hidden layers, approximate recovery with arbitrary precision and arbitrary sparsity level $s$ is possible in a stable way. We then extend our results to a wider class of recovery problems including low-rank matrix recovery and phase retrieval. Furthermore, we also consider the approximation of general positive homogeneous functions with neural networks. Extending previous work, we establish new results explaining under which conditions such functions can be approximated with neural networks. Our results also shed some light on the seeming contradiction between previous works showing that neural networks for inverse problems typically have very large Lipschitz constants, but still perform very well also for adversarial noise. Namely, the error bounds in our expressivity results include a combination of a small constant term and a term that is linear in the noise level, indicating that robustness issues may occur only for very small noise levels.

biometric

steal

extraction

Title: SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning. (arXiv:2308.02565v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02565
• Code URL: https://github.com/vermouthdky/simteg
• Copy Paste: [[2308.02565]] SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning(http://arxiv.org/abs/2308.02565)
• Summary:

  Textual graphs (TGs) are graphs whose nodes correspond to text (sentences or documents), which are widely prevalent. The representation learning of TGs involves two stages: (i) unsupervised feature extraction and (ii) supervised graph representation learning. In recent years, extensive efforts have been devoted to the latter stage, where Graph Neural Networks (GNNs) have dominated. However, the former stage for most existing graph benchmarks still relies on traditional feature engineering techniques. More recently, with the rapid development of language models (LMs), researchers have focused on leveraging LMs to facilitate the learning of TGs, either by jointly training them in a computationally intensive framework (merging the two stages), or designing complex self-supervised training tasks for feature extraction (enhancing the first stage). In this work, we present SimTeG, a frustratingly Simple approach for Textual Graph learning that does not innovate in frameworks, models, and tasks. Instead, we first perform supervised parameter-efficient fine-tuning (PEFT) on a pre-trained LM on the downstream task, such as node classification. We then generate node embeddings using the last hidden states of finetuned LM. These derived features can be further utilized by any GNN for training on the same task. We evaluate our approach on two fundamental graph representation learning tasks: node classification and link prediction. Through extensive experiments, we show that our approach significantly improves the performance of various GNNs on multiple graph benchmarks.

Title: BioBERT Based SNP-traits Associations Extraction from Biomedical Literature. (arXiv:2308.02569v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02569
• Code URL: null
• Copy Paste: [[2308.02569]] BioBERT Based SNP-traits Associations Extraction from Biomedical Literature(http://arxiv.org/abs/2308.02569)
• Summary:

  Scientific literature contains a considerable amount of information that provides an excellent opportunity for developing text mining methods to extract biomedical relationships. An important type of information is the relationship between singular nucleotide polymorphisms (SNP) and traits. In this paper, we present a BioBERT-GRU method to identify SNP- traits associations. Based on the evaluation of our method on the SNPPhenA dataset, it is concluded that this new method performs better than previous machine learning and deep learning based methods. BioBERT-GRU achieved the result a precision of 0.883, recall of 0.882 and F1-score of 0.881.

membership infer

federate

Title: Vehicles Control: Collision Avoidance using Federated Deep Reinforcement Learning. (arXiv:2308.02614v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02614
• Code URL: null
• Copy Paste: [[2308.02614]] Vehicles Control: Collision Avoidance using Federated Deep Reinforcement Learning(http://arxiv.org/abs/2308.02614)
• Summary:

  In the face of growing urban populations and the escalating number of vehicles on the roads, managing transportation efficiently and ensuring safety have become critical challenges. To tackle these issues, the development of intelligent control systems for vehicles is paramount. This paper presents a comprehensive study on vehicle control for collision avoidance, leveraging the power of Federated Deep Reinforcement Learning (FDRL) techniques. Our main goal is to minimize travel delays and enhance the average speed of vehicles while prioritizing safety and preserving data privacy. To accomplish this, we conducted a comparative analysis between the local model, Deep Deterministic Policy Gradient (DDPG), and the global model, Federated Deep Deterministic Policy Gradient (FDDPG), to determine their effectiveness in optimizing vehicle control for collision avoidance. The results obtained indicate that the FDDPG algorithm outperforms DDPG in terms of effectively controlling vehicles and preventing collisions. Significantly, the FDDPG-based algorithm demonstrates substantial reductions in travel delays and notable improvements in average speed compared to the DDPG algorithm.

Title: Private Federated Learning with Dynamic Power Control via Non-Coherent Over-the-Air Computation. (arXiv:2308.02881v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02881
• Code URL: null
• Copy Paste: [[2308.02881]] Private Federated Learning with Dynamic Power Control via Non-Coherent Over-the-Air Computation(http://arxiv.org/abs/2308.02881)
• Summary:

  To further preserve model weight privacy and improve model performance in Federated Learning (FL), FL via Over-the-Air Computation (AirComp) scheme based on dynamic power control is proposed. The edge devices (EDs) transmit the signs of local stochastic gradients by activating two adjacent orthogonal frequency division multi-plexing (OFDM) subcarriers, and majority votes (MVs) at the edge server (ES) are obtained by exploiting the energy accumulation on the subcarriers. Then, we propose a dynamic power control algorithm to further offset the biased aggregation of the MV aggregation values. We show that the whole scheme can mitigate the impact of the time synchronization error, channel fading and noise. The theoretical convergence proof of the scheme is re-derived.

fair

Title: ALE: A Simulation-Based Active Learning Evaluation Framework for the Parameter-Driven Comparison of Query Strategies for NLP. (arXiv:2308.02537v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02537
• Code URL: https://github.com/philipp-kohl/active-learning-evaluation-framework
• Copy Paste: [[2308.02537]] ALE: A Simulation-Based Active Learning Evaluation Framework for the Parameter-Driven Comparison of Query Strategies for NLP(http://arxiv.org/abs/2308.02537)
• Summary:

  Supervised machine learning and deep learning require a large amount of labeled data, which data scientists obtain in a manual, and time-consuming annotation process. To mitigate this challenge, Active Learning (AL) proposes promising data points to annotators they annotate next instead of a subsequent or random sample. This method is supposed to save annotation effort while maintaining model performance. However, practitioners face many AL strategies for different tasks and need an empirical basis to choose between them. Surveys categorize AL strategies into taxonomies without performance indications. Presentations of novel AL strategies compare the performance to a small subset of strategies. Our contribution addresses the empirical basis by introducing a reproducible active learning evaluation (ALE) framework for the comparative evaluation of AL strategies in NLP. The framework allows the implementation of AL strategies with low effort and a fair data-driven comparison through defining and tracking experiment parameters (e.g., initial dataset size, number of data points per query step, and the budget). ALE helps practitioners to make more informed decisions, and researchers can focus on developing new, effective AL strategies and deriving best practices for specific use cases. With best practices, practitioners can lower their annotation costs. We present a case study to illustrate how to use the framework.

Title: EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Education. (arXiv:2308.02773v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02773
• Code URL: https://github.com/icalk-nlp/educhat
• Copy Paste: [[2308.02773]] EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Education(http://arxiv.org/abs/2308.02773)
• Summary:

  EduChat (https://www.educhat.top/) is a large-scale language model (LLM)-based chatbot system in the education domain. Its goal is to support personalized, fair, and compassionate intelligent education, serving teachers, students, and parents. Guided by theories from psychology and education, it further strengthens educational functions such as open question answering, essay assessment, Socratic teaching, and emotional support based on the existing basic LLMs. Particularly, we learn domain-specific knowledge by pre-training on the educational corpus and stimulate various skills with tool use by fine-tuning on designed system prompts and instructions. Currently, EduChat is available online as an open-source project, with its code, data, and model parameters available on platforms (e.g., GitHub https://github.com/icalk-nlp/EduChat, Hugging Face https://huggingface.co/ecnu-icalk ). We also prepare a demonstration of its capabilities online (https://vimeo.com/851004454). This initiative aims to promote research and applications of LLMs for intelligent education.

interpretability

Title: Knowledge-Driven Multi-Agent Reinforcement Learning for Computation Offloading in Cybertwin-Enabled Internet of Vehicles. (arXiv:2308.02603v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02603
• Code URL: null
• Copy Paste: [[2308.02603]] Knowledge-Driven Multi-Agent Reinforcement Learning for Computation Offloading in Cybertwin-Enabled Internet of Vehicles(http://arxiv.org/abs/2308.02603)
• Summary:

  By offloading computation-intensive tasks of vehicles to roadside units (RSUs), mobile edge computing (MEC) in the Internet of Vehicles (IoV) can relieve the onboard computation burden. However, existing model-based task offloading methods suffer from heavy computational complexity with the increase of vehicles and data-driven methods lack interpretability. To address these challenges, in this paper, we propose a knowledge-driven multi-agent reinforcement learning (KMARL) approach to reduce the latency of task offloading in cybertwin-enabled IoV. Specifically, in the considered scenario, the cybertwin serves as a communication agent for each vehicle to exchange information and make offloading decisions in the virtual space. To reduce the latency of task offloading, a KMARL approach is proposed to select the optimal offloading option for each vehicle, where graph neural networks are employed by leveraging domain knowledge concerning graph-structure communication topology and permutation invariance into neural networks. Numerical results show that our proposed KMARL yields higher rewards and demonstrates improved scalability compared with other methods, benefitting from the integration of domain knowledge.

explainability

Title: Exploring the Role of Explainability in AI-Assisted Embryo Selection. (arXiv:2308.02534v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02534
• Code URL: null
• Copy Paste: [[2308.02534]] Exploring the Role of Explainability in AI-Assisted Embryo Selection(http://arxiv.org/abs/2308.02534)
• Summary:

  In Vitro Fertilization is among the most widespread treatments for infertility. One of its main challenges is the evaluation and selection of embryo for implantation, a process with large inter- and intra-clinician variability. Deep learning based methods are gaining attention, but their opaque nature compromises their acceptance in the clinical context, where transparency in the decision making is key. In this paper we analyze the current work in the explainability of AI-assisted embryo analysis models, identifying the limitations. We also discuss how these models could be integrated in the clinical context as decision support systems, considering the needs of clinicians and patients. Finally, we propose guidelines for the sake of increasing interpretability and trustworthiness, pushing this technology forward towards established clinical practice.

watermark

diffusion

Title: Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusion. (arXiv:2308.02552v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02552
• Code URL: null
• Copy Paste: [[2308.02552]] Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusion(http://arxiv.org/abs/2308.02552)
• Summary:

  Owing to the unrestricted nature of the content in the training data, large text-to-image diffusion models, such as Stable Diffusion (SD), are capable of generating images with potentially copyrighted or dangerous content based on corresponding textual concepts information. This includes specific intellectual property (IP), human faces, and various artistic styles. However, Negative Prompt, a widely used method for content removal, frequently fails to conceal this content due to inherent limitations in its inference logic. In this work, we propose a novel strategy named \textbf{Degeneration-Tuning (DT)} to shield contents of unwanted concepts from SD weights. By utilizing Scrambled Grid to reconstruct the correlation between undesired concepts and their corresponding image domain, we guide SD to generate meaningless content when such textual concepts are provided as input. As this adaptation occurs at the level of the model's weights, the SD, after DT, can be grafted onto other conditional diffusion frameworks like ControlNet to shield unwanted concepts. In addition to qualitatively showcasing the effectiveness of our DT method in protecting various types of concepts, a quantitative comparison of the SD before and after DT indicates that the DT method does not significantly impact the generative quality of other contents. The FID and IS scores of the model on COCO-30K exhibit only minor changes after DT, shifting from 12.61 and 39.20 to 13.04 and 38.25, respectively, which clearly outperforms the previous methods.

Title: ConceptLab: Creative Generation using Diffusion Prior Constraints. (arXiv:2308.02669v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02669
• Code URL: null
• Copy Paste: [[2308.02669]] ConceptLab: Creative Generation using Diffusion Prior Constraints(http://arxiv.org/abs/2308.02669)
• Summary:

  Recent text-to-image generative models have enabled us to transform our words into vibrant, captivating imagery. The surge of personalization techniques that has followed has also allowed us to imagine unique concepts in new scenes. However, an intriguing question remains: How can we generate a new, imaginary concept that has never been seen before? In this paper, we present the task of creative text-to-image generation, where we seek to generate new members of a broad category (e.g., generating a pet that differs from all existing pets). We leverage the under-studied Diffusion Prior models and show that the creative generation problem can be formulated as an optimization process over the output space of the diffusion prior, resulting in a set of "prior constraints". To keep our generated concept from converging into existing members, we incorporate a question-answering model that adaptively adds new constraints to the optimization problem, encouraging the model to discover increasingly more unique creations. Finally, we show that our prior constraints can also serve as a strong mixing mechanism allowing us to create hybrids between generated concepts, introducing even more flexibility into the creative process.

Title: Sketch and Text Guided Diffusion Model for Colored Point Cloud Generation. (arXiv:2308.02874v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02874
• Code URL: null
• Copy Paste: [[2308.02874]] Sketch and Text Guided Diffusion Model for Colored Point Cloud Generation(http://arxiv.org/abs/2308.02874)
• Summary:

  Diffusion probabilistic models have achieved remarkable success in text guided image generation. However, generating 3D shapes is still challenging due to the lack of sufficient data containing 3D models along with their descriptions. Moreover, text based descriptions of 3D shapes are inherently ambiguous and lack details. In this paper, we propose a sketch and text guided probabilistic diffusion model for colored point cloud generation that conditions the denoising process jointly with a hand drawn sketch of the object and its textual description. We incrementally diffuse the point coordinates and color values in a joint diffusion process to reach a Gaussian distribution. Colored point cloud generation thus amounts to learning the reverse diffusion process, conditioned by the sketch and text, to iteratively recover the desired shape and color. Specifically, to learn effective sketch-text embedding, our model adaptively aggregates the joint embedding of text prompt and the sketch based on a capsule attention network. Our model uses staged diffusion to generate the shape and then assign colors to different parts conditioned on the appearance prompt while preserving precise shapes from the first stage. This gives our model the flexibility to extend to multiple tasks, such as appearance re-editing and part segmentation. Experimental results demonstrate that our model outperforms recent state-of-the-art in point cloud generation.

noise learning

data-free

transformer

Title: Survey on Computer Vision Techniques for Internet-of-Things Devices. (arXiv:2308.02553v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02553
• Code URL: null
• Copy Paste: [[2308.02553]] Survey on Computer Vision Techniques for Internet-of-Things Devices(http://arxiv.org/abs/2308.02553)
• Summary:

  Deep neural networks (DNNs) are state-of-the-art techniques for solving most computer vision problems. DNNs require billions of parameters and operations to achieve state-of-the-art results. This requirement makes DNNs extremely compute, memory, and energy-hungry, and consequently difficult to deploy on small battery-powered Internet-of-Things (IoT) devices with limited computing resources. Deployment of DNNs on Internet-of-Things devices, such as traffic cameras, can improve public safety by enabling applications such as automatic accident detection and emergency response.Through this paper, we survey the recent advances in low-power and energy-efficient DNN implementations that improve the deployability of DNNs without significantly sacrificing accuracy. In general, these techniques either reduce the memory requirements, the number of arithmetic operations, or both. The techniques can be divided into three major categories: neural network compression, network architecture search and design, and compiler and graph optimizations. In this paper, we survey both low-power techniques for both convolutional and transformer DNNs, and summarize the advantages, disadvantages, and open research problems.

Title: Attention-free Spikformer: Mixing Spike Sequences with Simple Linear Transforms. (arXiv:2308.02557v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02557
• Code URL: null
• Copy Paste: [[2308.02557]] Attention-free Spikformer: Mixing Spike Sequences with Simple Linear Transforms(http://arxiv.org/abs/2308.02557)
• Summary:

  By integrating the self-attention capability and the biological properties of Spiking Neural Networks (SNNs), Spikformer applies the flourishing Transformer architecture to SNN design. It introduces a Spiking Self-Attention (SSA) module to mix sparse visual features using spike-form Query, Key, and Value, resulting in State-Of-The-Art (SOTA) performance on numerous datasets compared to previous SNN-like frameworks. In this paper, we demonstrate that the Spikformer architecture can be accelerated by replacing the SSA with an unparameterized Linear Transform (LT) such as Fourier and Wavelet transforms. These transforms are utilized to mix spike sequences, reducing the quadratic time complexity to log-linear time complexity. They alternate between the frequency and time domains to extract sparse visual features, showcasing powerful performance and efficiency. We conduct extensive experiments on image classification using both neuromorphic and static datasets. The results indicate that compared to the SOTA Spikformer with SSA, Spikformer with LT achieves higher Top-1 accuracy on neuromorphic datasets and comparable Top-1 accuracy on static datasets. Moreover, Spikformer with LT achieves approximately $29$-$51\%$ improvement in training speed, $61$-$70\%$ improvement in inference speed, and reduces memory usage by $4$-$26\%$ due to not requiring learnable parameters.

Title: Lightweight Endoscopic Depth Estimation with CNN-Transformer Encoder. (arXiv:2308.02716v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02716
• Code URL: null
• Copy Paste: [[2308.02716]] Lightweight Endoscopic Depth Estimation with CNN-Transformer Encoder(http://arxiv.org/abs/2308.02716)
• Summary:

  In this study, we tackle the key challenges concerning accuracy and robustness in depth estimation for endoscopic imaging, with a particular emphasis on real-time inference and the impact of reflections. We propose an innovative lightweight solution that integrates Convolutional Neural Networks (CNN) and Transformers to predict multi-scale depth maps. Our approach includes optimizing the network architecture, incorporating multi-scale dilated convolution, and a multi-channel attention mechanism. We also introduce a statistical confidence boundary mask to minimize the impact of reflective areas. Moreover, we propose a novel complexity evaluation metric that considers network parameter size, floating-point operations, and inference frames per second. Our research aims to enhance the efficiency and safety of laparoscopic surgery significantly. We comprehensively evaluate our proposed method and compare it with existing solutions. The results demonstrate that our method ensures depth estimation accuracy while being lightweight.

Title: Dual Degradation-Inspired Deep Unfolding Network for Low-Light Image Enhancement. (arXiv:2308.02776v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02776
• Code URL: null
• Copy Paste: [[2308.02776]] Dual Degradation-Inspired Deep Unfolding Network for Low-Light Image Enhancement(http://arxiv.org/abs/2308.02776)
• Summary:

  Although low-light image enhancement has achieved great stride based on deep enhancement models, most of them mainly stress on enhancement performance via an elaborated black-box network and rarely explore the physical significance of enhancement models. Towards this issue, we propose a Dual degrAdation-inSpired deep Unfolding network, termed DASUNet, for low-light image enhancement. Specifically, we construct a dual degradation model (DDM) to explicitly simulate the deterioration mechanism of low-light images. It learns two distinct image priors via considering degradation specificity between luminance and chrominance spaces. To make the proposed scheme tractable, we design an alternating optimization solution to solve the proposed DDM. Further, the designed solution is unfolded into a specified deep network, imitating the iteration updating rules, to form DASUNet. Local and long-range information are obtained by prior modeling module (PMM), inheriting the advantages of convolution and Transformer, to enhance the representation capability of dual degradation priors. Additionally, a space aggregation module (SAM) is presented to boost the interaction of two degradation models. Extensive experiments on multiple popular low-light image datasets validate the effectiveness of DASUNet compared to canonical state-of-the-art low-light image enhancement methods. Our source code and pretrained model will be publicly available.

Title: Unfolding Once is Enough: A Deployment-Friendly Transformer Unit for Super-Resolution. (arXiv:2308.02794v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02794
• Code URL: https://github.com/yongliuy/ditn
• Copy Paste: [[2308.02794]] Unfolding Once is Enough: A Deployment-Friendly Transformer Unit for Super-Resolution(http://arxiv.org/abs/2308.02794)
• Summary:

  Recent years have witnessed a few attempts of vision transformers for single image super-resolution (SISR). Since the high resolution of intermediate features in SISR models increases memory and computational requirements, efficient SISR transformers are more favored. Based on some popular transformer backbone, many methods have explored reasonable schemes to reduce the computational complexity of the self-attention module while achieving impressive performance. However, these methods only focus on the performance on the training platform (e.g., Pytorch/Tensorflow) without further optimization for the deployment platform (e.g., TensorRT). Therefore, they inevitably contain some redundant operators, posing challenges for subsequent deployment in real-world applications. In this paper, we propose a deployment-friendly transformer unit, namely UFONE (i.e., UnFolding ONce is Enough), to alleviate these problems. In each UFONE, we introduce an Inner-patch Transformer Layer (ITL) to efficiently reconstruct the local structural information from patches and a Spatial-Aware Layer (SAL) to exploit the long-range dependencies between patches. Based on UFONE, we propose a Deployment-friendly Inner-patch Transformer Network (DITN) for the SISR task, which can achieve favorable performance with low latency and memory usage on both training and deployment platforms. Furthermore, to further boost the deployment efficiency of the proposed DITN on TensorRT, we also provide an efficient substitution for layer normalization and propose a fusion optimization strategy for specific operators. Extensive experiments show that our models can achieve competitive results in terms of qualitative and quantitative performance with high deployment efficiency. Code is available at \url{https://github.com/yongliuy/DITN}.

generative

Title: Learning Implicit Entity-object Relations by Bidirectional Generative Alignment for Multimodal NER. (arXiv:2308.02570v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02570
• Code URL: null
• Copy Paste: [[2308.02570]] Learning Implicit Entity-object Relations by Bidirectional Generative Alignment for Multimodal NER(http://arxiv.org/abs/2308.02570)
• Summary:

  The challenge posed by multimodal named entity recognition (MNER) is mainly two-fold: (1) bridging the semantic gap between text and image and (2) matching the entity with its associated object in image. Existing methods fail to capture the implicit entity-object relations, due to the lack of corresponding annotation. In this paper, we propose a bidirectional generative alignment method named BGA-MNER to tackle these issues. Our BGA-MNER consists of \texttt{image2text} and \texttt{text2image} generation with respect to entity-salient content in two modalities. It jointly optimizes the bidirectional reconstruction objectives, leading to aligning the implicit entity-object relations under such direct and powerful constraints. Furthermore, image-text pairs usually contain unmatched components which are noisy for generation. A stage-refined context sampler is proposed to extract the matched cross-modal content for generation. Extensive experiments on two benchmarks demonstrate that our method achieves state-of-the-art performance without image input during inference.

Title: Generation of Realistic Synthetic Raw Radar Data for Automated Driving Applications using Generative Adversarial Networks. (arXiv:2308.02632v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02632
• Code URL: null
• Copy Paste: [[2308.02632]] Generation of Realistic Synthetic Raw Radar Data for Automated Driving Applications using Generative Adversarial Networks(http://arxiv.org/abs/2308.02632)
• Summary:

  The main approaches for simulating FMCW radar are based on ray tracing, which is usually computationally intensive and do not account for background noise. This work proposes a faster method for FMCW radar simulation capable of generating synthetic raw radar data using generative adversarial networks (GAN). The code and pre-trained weights are open-source and available on GitHub. This method generates 16 simultaneous chirps, which allows the generated data to be used for the further development of algorithms for processing radar data (filtering and clustering). This can increase the potential for data augmentation, e.g., by generating data in non-existent or safety-critical scenarios that are not reproducible in real life. In this work, the GAN was trained with radar measurements of a motorcycle and used to generate synthetic raw radar data of a motorcycle traveling in a straight line. For generating this data, the distance of the motorcycle and Gaussian noise are used as input to the neural network. The synthetic generated radar chirps were evaluated using the Frechet Inception Distance (FID). Then, the Range-Azimuth (RA) map is calculated twice: (1\textsuperscript{st}) based on synthetic data using this GAN and (2\textsuperscript{nd}) based on real data. Based on these RA maps, an algorithm with adaptive threshold and edge detection is used for object detection. The results have shown that the data is realistic in terms of coherent radar reflections of the motorcycle and background noise based on the comparison of chirps, the RA maps and the object detection results. Thus, the proposed method in this work has shown to minimize the simulation-to-reality gap for the generation of radar data.

Title: CoSMo: A constructor specification language for Abstract Wikipedia's content selection process. (arXiv:2308.02539v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02539
• Code URL: null
• Copy Paste: [[2308.02539]] CoSMo: A constructor specification language for Abstract Wikipedia's content selection process(http://arxiv.org/abs/2308.02539)
• Summary:

  Representing snippets of information abstractly is a task that needs to be performed for various purposes, such as database view specification and the first stage in the natural language generation pipeline for generative AI from structured input, i.e., the content selection stage to determine what needs to be verbalised. For the Abstract Wikipedia project, requirements analysis revealed that such an abstract representation requires multilingual modelling, content selection covering declarative content and functions, and both classes and instances. There is no modelling language that meets either of the three features, let alone a combination. Following a rigorous language design process inclusive of broad stakeholder consultation, we created CoSMo, a novel {\sc Co}ntent {\sc S}election {\sc Mo}deling language that meets these and other requirements so that it may be useful both in Abstract Wikipedia as well as other contexts. We describe the design process, rationale and choices, the specification, and preliminary evaluation of the language.

Title: Evaluating ChatGPT and GPT-4 for Visual Programming. (arXiv:2308.02522v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02522
• Code URL: null
• Copy Paste: [[2308.02522]] Evaluating ChatGPT and GPT-4 for Visual Programming(http://arxiv.org/abs/2308.02522)
• Summary:

  Generative AI and large language models have the potential to drastically improve the landscape of computing education by automatically generating personalized feedback and content. Recent works have studied the capabilities of these models for different programming education scenarios; however, these works considered only text-based programming, in particular, Python programming. Consequently, they leave open the question of how well these models would perform in visual programming domains popularly used for K-8 programming education. The main research question we study is: Do state-of-the-art generative models show advanced capabilities in visual programming on par with their capabilities in text-based Python programming? In our work, we evaluate two models, ChatGPT (based on GPT-3.5) and GPT-4, in visual programming domains for various scenarios and assess performance using expert-based annotations. In particular, we base our evaluation using reference tasks from the domains of Hour of Code: Maze Challenge by Code-dot-org and Karel. Our results show that these models perform poorly and struggle to combine spatial, logical, and programming skills crucial for visual programming. These results also provide exciting directions for future work on developing techniques to improve the performance of generative models in visual programming.

Title: A Review of Change of Variable Formulas for Generative Modeling. (arXiv:2308.02652v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02652
• Code URL: null
• Copy Paste: [[2308.02652]] A Review of Change of Variable Formulas for Generative Modeling(http://arxiv.org/abs/2308.02652)
• Summary:

  Change-of-variables (CoV) formulas allow to reduce complicated probability densities to simpler ones by a learned transformation with tractable Jacobian determinant. They are thus powerful tools for maximum-likelihood learning, Bayesian inference, outlier detection, model selection, etc. CoV formulas have been derived for a large variety of model types, but this information is scattered over many separate works. We present a systematic treatment from the unifying perspective of encoder/decoder architectures, which collects 28 CoV formulas in a single place, reveals interesting relationships between seemingly diverse methods, emphasizes important distinctions that are not always clear in the literature, and identifies surprising gaps for future research.

Title: A generative model for surrogates of spatial-temporal wildfire nowcasting. (arXiv:2308.02810v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02810
• Code URL: null
• Copy Paste: [[2308.02810]] A generative model for surrogates of spatial-temporal wildfire nowcasting(http://arxiv.org/abs/2308.02810)
• Summary:

  Recent increase in wildfires worldwide has led to the need for real-time fire nowcasting. Physics-driven models, such as cellular automata and computational fluid dynamics can provide high-fidelity fire spread simulations but they are computationally expensive and time-consuming. Much effort has been put into developing machine learning models for fire prediction. However, these models are often region-specific and require a substantial quantity of simulation data for training purpose. This results in a significant amount of computational effort for different ecoregions. In this work, a generative model is proposed using a three-dimensional Vector-Quantized Variational Autoencoders to generate spatial-temporal sequences of unseen wildfire burned areas in a given ecoregion. The model is tested in the ecoregion of a recent massive wildfire event in California, known as the Chimney fire. Numerical results show that the model succeed in generating coherent and structured fire scenarios, taking into account the impact from geophysical variables, such as vegetation and slope. Generated data are also used to train a surrogate model for predicting wildfire dissemination, which has been tested on both simulation data and the real Chimney fire event.

large language model

Title: Towards More Human-like AI Communication: A Review of Emergent Communication Research. (arXiv:2308.02541v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02541
• Code URL: null
• Copy Paste: [[2308.02541]] Towards More Human-like AI Communication: A Review of Emergent Communication Research(http://arxiv.org/abs/2308.02541)
• Summary:

  In the recent shift towards human-centric AI, the need for machines to accurately use natural language has become increasingly important. While a common approach to achieve this is to train large language models, this method presents a form of learning misalignment where the model may not capture the underlying structure and reasoning humans employ in using natural language, potentially leading to unexpected or unreliable behavior. Emergent communication (Emecom) is a field of research that has seen a growing number of publications in recent years, aiming to develop artificial agents capable of using natural language in a way that goes beyond simple discriminative tasks and can effectively communicate and learn new concepts. In this review, we present Emecom under two aspects. Firstly, we delineate all the common proprieties we find across the literature and how they relate to human interactions. Secondly, we identify two subcategories and highlight their characteristics and open challenges. We encourage researchers to work together by demonstrating that different methods can be viewed as diverse solutions to a common problem and emphasize the importance of including diverse perspectives and expertise in the field. We believe a deeper understanding of human communication is crucial to developing machines that can accurately use natural language in human-machine interactions.

Title: Adapt and Decompose: Efficient Generalization of Text-to-SQL via Domain Adapted Least-To-Most Prompting. (arXiv:2308.02582v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02582
• Code URL: null
• Copy Paste: [[2308.02582]] Adapt and Decompose: Efficient Generalization of Text-to-SQL via Domain Adapted Least-To-Most Prompting(http://arxiv.org/abs/2308.02582)
• Summary:

  Cross-domain and cross-compositional generalization of Text-to-SQL semantic parsing is a challenging task. Existing Large Language Model (LLM) based solutions rely on inference-time retrieval of few-shot exemplars from the training set to synthesize a run-time prompt for each Natural Language (NL) test query. In contrast, we devise an algorithm which performs offline sampling of a minimal set-of few-shots from the training data, with complete coverage of SQL clauses, operators and functions, and maximal domain coverage within the allowed token length. This allows for synthesis of a fixed Generic Prompt (GP), with a diverse set-of exemplars common across NL test queries, avoiding expensive test time exemplar retrieval. We further auto-adapt the GP to the target database domain (DA-GP), to better handle cross-domain generalization; followed by a decomposed Least-To-Most-Prompting (LTMP-DA-GP) to handle cross-compositional generalization. The synthesis of LTMP-DA-GP is an offline task, to be performed one-time per new database with minimal human intervention. Our approach demonstrates superior performance on the KaggleDBQA dataset, designed to evaluate generalizability for the Text-to-SQL task. We further showcase consistent performance improvement of LTMP-DA-GP over GP, across LLMs and databases of KaggleDBQA, highlighting the efficacy and model agnostic benefits of our prompt based adapt and decompose approach.

segmentation

Title: Hierarchical Semi-Supervised Learning Framework for Surgical Gesture Segmentation and Recognition Based on Multi-Modality Data. (arXiv:2308.02529v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02529
• Code URL: null
• Copy Paste: [[2308.02529]] Hierarchical Semi-Supervised Learning Framework for Surgical Gesture Segmentation and Recognition Based on Multi-Modality Data(http://arxiv.org/abs/2308.02529)
• Summary:

  Segmenting and recognizing surgical operation trajectories into distinct, meaningful gestures is a critical preliminary step in surgical workflow analysis for robot-assisted surgery. This step is necessary for facilitating learning from demonstrations for autonomous robotic surgery, evaluating surgical skills, and so on. In this work, we develop a hierarchical semi-supervised learning framework for surgical gesture segmentation using multi-modality data (i.e. kinematics and vision data). More specifically, surgical tasks are initially segmented based on distance characteristics-based profiles and variance characteristics-based profiles constructed using kinematics data. Subsequently, a Transformer-based network with a pre-trained `ResNet-18' backbone is used to extract visual features from the surgical operation videos. By combining the potential segmentation points obtained from both modalities, we can determine the final segmentation points. Furthermore, gesture recognition can be implemented based on supervised learning. The proposed approach has been evaluated using data from the publicly available JIGSAWS database, including Suturing, Needle Passing, and Knot Tying tasks. The results reveal an average F1 score of 0.623 for segmentation and an accuracy of 0.856 for recognition.

Title: Guided Distillation for Semi-Supervised Instance Segmentation. (arXiv:2308.02668v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02668
• Code URL: null
• Copy Paste: [[2308.02668]] Guided Distillation for Semi-Supervised Instance Segmentation(http://arxiv.org/abs/2308.02668)
• Summary:

  Although instance segmentation methods have improved considerably, the dominant paradigm is to rely on fully-annotated training images, which are tedious to obtain. To alleviate this reliance, and boost results, semi-supervised approaches leverage unlabeled data as an additional training signal that limits overfitting to the labeled samples. In this context, we present novel design choices to significantly improve teacher-student distillation models. In particular, we (i) improve the distillation approach by introducing a novel "guided burn-in" stage, and (ii) evaluate different instance segmentation architectures, as well as backbone networks and pre-training strategies. Contrary to previous work which uses only supervised data for the burn-in period of the student model, we also use guidance of the teacher model to exploit unlabeled data in the burn-in period. Our improved distillation approach leads to substantial improvements over previous state-of-the-art results. For example, on the Cityscapes dataset we improve mask-AP from 23.7 to 33.9 when using labels for 10\% of images, and on the COCO dataset we improve mask-AP from 18.3 to 34.1 when using labels for only 1\% of the training data.

Title: Few-shot Class-Incremental Semantic Segmentation via Pseudo-Labeling and Knowledge Distillation. (arXiv:2308.02790v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02790
• Code URL: https://github.com/chasonjiang/fscilss
• Copy Paste: [[2308.02790]] Few-shot Class-Incremental Semantic Segmentation via Pseudo-Labeling and Knowledge Distillation(http://arxiv.org/abs/2308.02790)
• Summary:

  We address the problem of learning new classes for semantic segmentation models from few examples, which is challenging because of the following two reasons. Firstly, it is difficult to learn from limited novel data to capture the underlying class distribution. Secondly, it is challenging to retain knowledge for existing classes and to avoid catastrophic forgetting. For learning from limited data, we propose a pseudo-labeling strategy to augment the few-shot training annotations in order to learn novel classes more effectively. Given only one or a few images labeled with the novel classes and a much larger set of unlabeled images, we transfer the knowledge from labeled images to unlabeled images with a coarse-to-fine pseudo-labeling approach in two steps. Specifically, we first match each labeled image to its nearest neighbors in the unlabeled image set at the scene level, in order to obtain images with a similar scene layout. This is followed by obtaining pseudo-labels within this neighborhood by applying classifiers learned on the few-shot annotations. In addition, we use knowledge distillation on both labeled and unlabeled data to retain knowledge on existing classes. We integrate the above steps into a single convolutional neural network with a unified learning objective. Extensive experiments on the Cityscapes and KITTI datasets validate the efficacy of the proposed approach in the self-driving domain. Code is available from https://github.com/ChasonJiang/FSCILSS.

Title: NP-SemiSeg: When Neural Processes meet Semi-Supervised Semantic Segmentation. (arXiv:2308.02866v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02866
• Code URL: https://github.com/jianf-wang/np-semiseg
• Copy Paste: [[2308.02866]] NP-SemiSeg: When Neural Processes meet Semi-Supervised Semantic Segmentation(http://arxiv.org/abs/2308.02866)
• Summary:

  Semi-supervised semantic segmentation involves assigning pixel-wise labels to unlabeled images at training time. This is useful in a wide range of real-world applications where collecting pixel-wise labels is not feasible in time or cost. Current approaches to semi-supervised semantic segmentation work by predicting pseudo-labels for each pixel from a class-wise probability distribution output by a model. If the predicted probability distribution is incorrect, however, this leads to poor segmentation results, which can have knock-on consequences in safety critical systems, like medical images or self-driving cars. It is, therefore, important to understand what a model does not know, which is mainly achieved by uncertainty quantification. Recently, neural processes (NPs) have been explored in semi-supervised image classification, and they have been a computationally efficient and effective method for uncertainty quantification. In this work, we move one step forward by adapting NPs to semi-supervised semantic segmentation, resulting in a new model called NP-SemiSeg. We experimentally evaluated NP-SemiSeg on the public benchmarks PASCAL VOC 2012 and Cityscapes, with different training settings, and the results verify its effectiveness.

Title: Semi-supervised Learning for Segmentation of Bleeding Regions in Video Capsule Endoscopy. (arXiv:2308.02869v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02869
• Code URL: null
• Copy Paste: [[2308.02869]] Semi-supervised Learning for Segmentation of Bleeding Regions in Video Capsule Endoscopy(http://arxiv.org/abs/2308.02869)
• Summary:

  In the realm of modern diagnostic technology, video capsule endoscopy (VCE) is a standout for its high efficacy and non-invasive nature in diagnosing various gastrointestinal (GI) conditions, including obscure bleeding. Importantly, for the successful diagnosis and treatment of these conditions, accurate recognition of bleeding regions in VCE images is crucial. While deep learning-based methods have emerged as powerful tools for the automated analysis of VCE images, they often demand large training datasets with comprehensive annotations. Acquiring these labeled datasets tends to be time-consuming, costly, and requires significant domain expertise. To mitigate this issue, we have embraced a semi-supervised learning (SSL) approach for the bleeding regions segmentation within VCE. By adopting the `Mean Teacher' method, we construct a student U-Net equipped with an scSE attention block, alongside a teacher model of the same architecture. These models' parameters are alternately updated throughout the training process. We use the Kvasir-Capsule dataset for our experiments, which encompasses various GI bleeding conditions. Notably, we develop the segmentation annotations for this dataset ourselves. The findings from our experiments endorse the efficacy of the SSL-based segmentation strategy, demonstrating its capacity to reduce reliance on large volumes of annotations for model training, without compromising on the accuracy of identification.

Title: Cross-modal & Cross-domain Learning for Unsupervised LiDAR Semantic Segmentation. (arXiv:2308.02883v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02883
• Code URL: null
• Copy Paste: [[2308.02883]] Cross-modal & Cross-domain Learning for Unsupervised LiDAR Semantic Segmentation(http://arxiv.org/abs/2308.02883)
• Summary:

  In recent years, cross-modal domain adaptation has been studied on the paired 2D image and 3D LiDAR data to ease the labeling costs for 3D LiDAR semantic segmentation (3DLSS) in the target domain. However, in such a setting the paired 2D and 3D data in the source domain are still collected with additional effort. Since the 2D-3D projections can enable the 3D model to learn semantic information from the 2D counterpart, we ask whether we could further remove the need of source 3D data and only rely on the source 2D images. To answer it, this paper studies a new 3DLSS setting where a 2D dataset (source) with semantic annotations and a paired but unannotated 2D image and 3D LiDAR data (target) are available. To achieve 3DLSS in this scenario, we propose Cross-Modal and Cross-Domain Learning (CoMoDaL). Specifically, our CoMoDaL aims at modeling 1) inter-modal cross-domain distillation between the unpaired source 2D image and target 3D LiDAR data, and 2) the intra-domain cross-modal guidance between the target 2D image and 3D LiDAR data pair. In CoMoDaL, we propose to apply several constraints, such as point-to-pixel and prototype-to-pixel alignments, to associate the semantics in different modalities and domains by constructing mixed samples in two modalities. The experimental results on several datasets show that in the proposed setting, the developed CoMoDaL can achieve segmentation without the supervision of labeled LiDAR data. Ablations are also conducted to provide more analysis. Code will be available publicly.