secure

Title: Roll Up Your Sleeves: Working with a Collaborative and Engaging Task-Oriented Dialogue System. (arXiv:2307.16081v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.16081
• Code URL: null
• Copy Paste: [[2307.16081] Roll Up Your Sleeves: Working with a Collaborative and Engaging Task-Oriented Dialogue System](http://arxiv.org/abs/2307.16081) #secure
• Summary:

  We introduce TacoBot, a user-centered task-oriented digital assistant designed to guide users through complex real-world tasks with multiple steps. Covering a wide range of cooking and how-to tasks, we aim to deliver a collaborative and engaging dialogue experience. Equipped with language understanding, dialogue management, and response generation components supported by a robust search engine, TacoBot ensures efficient task assistance. To enhance the dialogue experience, we explore a series of data augmentation strategies using LLMs to train advanced neural models continuously. TacoBot builds upon our successful participation in the inaugural Alexa Prize TaskBot Challenge, where our team secured third place among ten competing teams. We offer TacoBot as an open-source framework that serves as a practical example for deploying task-oriented dialogue systems.

Title: Analyzing Cryptocurrency trends using Tweet Sentiment Data and User Meta-Data. (arXiv:2307.15956v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2307.15956
• Code URL: null
• Copy Paste: [[2307.15956] Analyzing Cryptocurrency trends using Tweet Sentiment Data and User Meta-Data](http://arxiv.org/abs/2307.15956) #secure
• Summary:

  Cryptocurrency is a form of digital currency using cryptographic techniques in a decentralized system for secure peer-to-peer transactions. It is gaining much popularity over traditional methods of payments because it facilitates a very fast, easy and secure way of transactions. However, it is very volatile and is influenced by a range of factors, with social media being a major one. Thus, with over four billion active users of social media, we need to understand its influence on the crypto market and how it can lead to fluctuations in the values of these cryptocurrencies. In our work, we analyze the influence of activities on Twitter, in particular the sentiments of the tweets posted regarding cryptocurrencies and how it influences their prices. In addition, we also collect metadata related to tweets and users. We use all these features to also predict the price of cryptocurrency for which we use some regression-based models and an LSTM-based model.

Title: Blockchain-based Decentralized Identity Management for Healthcare Systems. (arXiv:2307.16239v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2307.16239
• Code URL: null
• Copy Paste: [[2307.16239] Blockchain-based Decentralized Identity Management for Healthcare Systems](http://arxiv.org/abs/2307.16239) #secure
• Summary:

  Blockchain-based decentralized identity management provides a promising solution to improve the security and privacy of healthcare systems and make them scalable. Traditional Identity Management Systems are centralized, which makes them single-point-of-failure, vulnerable to attacks and data breaches, and non-scalable. In contrast, decentralized identity management based on the blockchain can ensure secure and transparent access to patient data while preserving privacy. This approach enables patients to control their personal health data while granting permission for medical personnel to access specific information as needed. We propose a decentralized identity management system for healthcare systems named BDIMHS based on a permissioned blockchain with Hyperledger Indy and Hyperledger Aries. We develop further descriptions of required functionalities and provide high-level procedures for network initialization, enrollment, registration, issuance, verification and revocation functionalities. The proposed solution improves data security, privacy, immutability, interoperability, and patient autonomy by using selective disclosure, zero-knowledge proofs, Decentralized Identifiers, and Verifiable Credentials. Furthermore, we discuss the potential challenges associated with implementing this technology in healthcare and evaluate the performance and security of the proposed solution.

security

Title: JFinder: A Novel Architecture for Java Vulnerability Identification Based Quad Self-Attention and Pre-training Mechanism. (arXiv:2307.15915v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2307.15915
• Code URL: null
• Copy Paste: [[2307.15915] JFinder: A Novel Architecture for Java Vulnerability Identification Based Quad Self-Attention and Pre-training Mechanism](http://arxiv.org/abs/2307.15915) #security
• Summary:

  Software vulnerabilities pose significant risks to computer systems, impacting our daily lives, productivity, and even our health. Identifying and addressing security vulnerabilities in a timely manner is crucial to prevent hacking and data breaches. Unfortunately, current vulnerability identification methods, including classical and deep learning-based approaches, exhibit critical drawbacks that prevent them from meeting the demands of the contemporary software industry. To tackle these issues, we present JFinder, a novel architecture for Java vulnerability identification that leverages quad self-attention and pre-training mechanisms to combine structural information and semantic representations. Experimental results demonstrate that JFinder outperforms all baseline methods, achieving an accuracy of 0.97 on the CWE dataset and an F1 score of 0.84 on the PROMISE dataset. Furthermore, a case study reveals that JFinder can accurately identify four cases of vulnerabilities after patching.

Title: An Effective LSTM-DDPM Scheme for Energy Theft Detection and Forecasting in Smart Grid. (arXiv:2307.16149v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.16149
• Code URL: null
• Copy Paste: [[2307.16149] An Effective LSTM-DDPM Scheme for Energy Theft Detection and Forecasting in Smart Grid](http://arxiv.org/abs/2307.16149) #security
• Summary:

  Energy theft detection (ETD) and energy consumption forecasting (ECF) are two interconnected challenges in smart grid systems. Addressing these issues collectively is crucial for ensuring system security. This paper addresses the interconnected challenges of ETD and ECF in smart grid systems. The proposed solution combines long short-term memory (LSTM) and a denoising diffusion probabilistic model (DDPM) to generate input reconstruction and forecasting. By leveraging the reconstruction and forecasting errors, the system identifies instances of energy theft, with the methods based on reconstruction error and forecasting error complementing each other in detecting different types of attacks. Through extensive experiments on real-world and synthetic datasets, the proposed scheme outperforms baseline methods in ETD and ECF problems. The ensemble method significantly enhances ETD performance, accurately detecting energy theft attacks that baseline methods fail to detect. The research offers a comprehensive and effective solution for addressing ETD and ECF challenges, demonstrating promising results and improved security in smart grid systems.

Title: "False negative -- that one is going to kill you": Understanding Industry Perspectives of Static Analysis based Security Testing. (arXiv:2307.16325v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2307.16325
• Code URL: null
• Copy Paste: [[2307.16325] "False negative -- that one is going to kill you": Understanding Industry Perspectives of Static Analysis based Security Testing](http://arxiv.org/abs/2307.16325) #security
• Summary:

  The demand for automated security analysis techniques, such as static analysis based security testing (SAST) tools continues to increase. To develop SASTs that are effectively leveraged by developers for finding vulnerabilities, researchers and tool designers must understand how developers perceive, select, and use SASTs, what they expect from the tools, whether they know of the limitations of the tools, and how they address those limitations. This paper describes a qualitative study that explores the assumptions, expectations, beliefs, and challenges experienced by developers who use SASTs. We perform in-depth, semi-structured interviews with 20 practitioners who possess a diverse range of software development expertise, as well as a variety of unique security, product, and organizational backgrounds. We identify $17$ key findings that shed light on developer perceptions and desires related to SASTs, and also expose gaps in the status quo -- challenging long-held beliefs in SAST design priorities. Finally, we provide concrete future directions for researchers and practitioners rooted in an analysis of our findings.

Title: Anomaly Detection in Industrial Machinery using IoT Devices and Machine Learning: a Systematic Mapping. (arXiv:2307.15807v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.15807
• Code URL: null
• Copy Paste: [[2307.15807] Anomaly Detection in Industrial Machinery using IoT Devices and Machine Learning: a Systematic Mapping](http://arxiv.org/abs/2307.15807) #security
• Summary:

  Anomaly detection is critical in the smart industry for preventing equipment failure, reducing downtime, and improving safety. Internet of Things (IoT) has enabled the collection of large volumes of data from industrial machinery, providing a rich source of information for Anomaly Detection. However, the volume and complexity of data generated by the Internet of Things ecosystems make it difficult for humans to detect anomalies manually. Machine learning (ML) algorithms can automate anomaly detection in industrial machinery by analyzing generated data. Besides, each technique has specific strengths and weaknesses based on the data nature and its corresponding systems. However, the current systematic mapping studies on Anomaly Detection primarily focus on addressing network and cybersecurity-related problems, with limited attention given to the industrial sector. Additionally, these studies do not cover the challenges involved in using ML for Anomaly Detection in industrial machinery within the context of the IoT ecosystems. This paper presents a systematic mapping study on Anomaly Detection for industrial machinery using IoT devices and ML algorithms to address this gap. The study comprehensively evaluates 84 relevant studies spanning from 2016 to 2023, providing an extensive review of Anomaly Detection research. Our findings identify the most commonly used algorithms, preprocessing techniques, and sensor types. Additionally, this review identifies application areas and points to future challenges and research opportunities.

privacy

Title: Catching Elusive Depression via Facial Micro-Expression Recognition. (arXiv:2307.15862v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.15862
• Code URL: null
• Copy Paste: [[2307.15862] Catching Elusive Depression via Facial Micro-Expression Recognition](http://arxiv.org/abs/2307.15862) #privacy
• Summary:

  Depression is a common mental health disorder that can cause consequential symptoms with continuously depressed mood that leads to emotional distress. One category of depression is Concealed Depression, where patients intentionally or unintentionally hide their genuine emotions through exterior optimism, thereby complicating and delaying diagnosis and treatment and leading to unexpected suicides. In this paper, we propose to diagnose concealed depression by using facial micro-expressions (FMEs) to detect and recognize underlying true emotions. However, the extremely low intensity and subtle nature of FMEs make their recognition a tough task. We propose a facial landmark-based Region-of-Interest (ROI) approach to address the challenge, and describe a low-cost and privacy-preserving solution that enables self-diagnosis using portable mobile devices in a personal setting (e.g., at home). We present results and findings that validate our method, and discuss other technical challenges and future directions in applying such techniques to real clinical settings.

Title: Mean Estimation with User-level Privacy under Data Heterogeneity. (arXiv:2307.15835v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2307.15835
• Code URL: null
• Copy Paste: [[2307.15835] Mean Estimation with User-level Privacy under Data Heterogeneity](http://arxiv.org/abs/2307.15835) #privacy
• Summary:

  A key challenge in many modern data analysis tasks is that user data are heterogeneous. Different users may possess vastly different numbers of data points. More importantly, it cannot be assumed that all users sample from the same underlying distribution. This is true, for example in language data, where different speech styles result in data heterogeneity. In this work we propose a simple model of heterogeneous user data that allows user data to differ in both distribution and quantity of data, and provide a method for estimating the population-level mean while preserving user-level differential privacy. We demonstrate asymptotic optimality of our estimator and also prove general lower bounds on the error achievable in the setting we introduce.

Title: zkDL: Efficient Zero-Knowledge Proofs of Deep Learning Training. (arXiv:2307.16273v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.16273
• Code URL: null
• Copy Paste: [[2307.16273] zkDL: Efficient Zero-Knowledge Proofs of Deep Learning Training](http://arxiv.org/abs/2307.16273) #privacy
• Summary:

  The recent advancements in deep learning have brought about significant changes in various aspects of people's lives. Meanwhile, these rapid developments have raised concerns about the legitimacy of the training process of deep networks. However, to protect the intellectual properties of untrusted AI developers, directly examining the training process by accessing the model parameters and training data by verifiers is often prohibited.

In response to this challenge, we present zkDL, an efficient zero-knowledge proof of deep learning training. At the core of zkDL is zkReLU, a specialized zero-knowledge proof protocol with optimized proving time and proof size for the ReLU activation function, a major obstacle in verifiable training due to its non-arithmetic nature. To integrate zkReLU into the proof system for the entire training process, we devise a novel construction of an arithmetic circuit from neural networks. By leveraging the abundant parallel computation resources, this construction reduces proving time and proof sizes by a factor of the network depth. As a result, zkDL enables the generation of complete and sound proofs, taking less than a minute with a size of less than 20 kB per training step, for a 16-layer neural network with 200M parameters, while ensuring the privacy of data and model parameters.

Title: Holistic Survey of Privacy and Fairness in Machine Learning. (arXiv:2307.15838v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.15838
• Code URL: null
• Copy Paste: [[2307.15838] Holistic Survey of Privacy and Fairness in Machine Learning](http://arxiv.org/abs/2307.15838) #privacy
• Summary:

  Privacy and fairness are two crucial pillars of responsible Artificial Intelligence (AI) and trustworthy Machine Learning (ML). Each objective has been independently studied in the literature with the aim of reducing utility loss in achieving them. Despite the significant interest attracted from both academia and industry, there remains an immediate demand for more in-depth research to unravel how these two objectives can be simultaneously integrated into ML models. As opposed to well-accepted trade-offs, i.e., privacy-utility and fairness-utility, the interrelation between privacy and fairness is not well-understood. While some works suggest a trade-off between the two objective functions, there are others that demonstrate the alignment of these functions in certain scenarios. To fill this research gap, we provide a thorough review of privacy and fairness in ML, including supervised, unsupervised, semi-supervised, and reinforcement learning. After examining and consolidating the literature on both objectives, we present a holistic survey on the impact of privacy on fairness, the impact of fairness on privacy, existing architectures, their interaction in application domains, and algorithms that aim to achieve both objectives while minimizing the utility sacrificed. Finally, we identify research challenges in achieving privacy and fairness concurrently in ML, particularly focusing on large language models.

protect

defense

Title: Theoretically Principled Trade-off for Stateful Defenses against Query-Based Black-Box Attacks. (arXiv:2307.16331v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.16331
• Code URL: null
• Copy Paste: [[2307.16331] Theoretically Principled Trade-off for Stateful Defenses against Query-Based Black-Box Attacks](http://arxiv.org/abs/2307.16331) #defense
• Summary:

  Adversarial examples threaten the integrity of machine learning systems with alarming success rates even under constrained black-box conditions. Stateful defenses have emerged as an effective countermeasure, detecting potential attacks by maintaining a buffer of recent queries and detecting new queries that are too similar. However, these defenses fundamentally pose a trade-off between attack detection and false positive rates, and this trade-off is typically optimized by hand-picking feature extractors and similarity thresholds that empirically work well. There is little current understanding as to the formal limits of this trade-off and the exact properties of the feature extractors/underlying problem domain that influence it. This work aims to address this gap by offering a theoretical characterization of the trade-off between detection and false positive rates for stateful defenses. We provide upper bounds for detection rates of a general class of feature extractors and analyze the impact of this trade-off on the convergence of black-box attacks. We then support our theoretical findings with empirical evaluations across multiple datasets and stateful defenses.

attack

Title: Auditing Frameworks Need Resource Isolation: A Systematic Study on the Super Producer Threat to System Auditing and Its Mitigation. (arXiv:2307.15895v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2307.15895
• Code URL: null
• Copy Paste: [[2307.15895] Auditing Frameworks Need Resource Isolation: A Systematic Study on the Super Producer Threat to System Auditing and Its Mitigation](http://arxiv.org/abs/2307.15895) #attack
• Summary:

  System auditing is a crucial technique for detecting APT attacks. However, attackers may try to compromise the system auditing frameworks to conceal their malicious activities. In this paper, we present a comprehensive and systematic study of the super producer threat in auditing frameworks, which enables attackers to either corrupt the auditing framework or paralyze the entire system. We analyze that the main cause of the super producer threat is the lack of data isolation in the centralized architecture of existing solutions. To address this threat, we propose a novel auditing framework, NODROP, which isolates provenance data generated by different processes with a threadlet-based architecture design. Our evaluation demonstrates that NODROP can ensure the integrity of the auditing frameworks while achieving an average 6.58% higher application overhead compared to vanilla Linux and 6.30% lower application overhead compared to a state-of-the-art commercial auditing framework, Sysdig across eight different hardware configurations.

Title: Exposing Hidden Attackers in Industrial Control Systems using Micro-distortions. (arXiv:2307.15926v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2307.15926
• Code URL: null
• Copy Paste: [[2307.15926] Exposing Hidden Attackers in Industrial Control Systems using Micro-distortions](http://arxiv.org/abs/2307.15926) #attack
• Summary:

  For industrial control systems (ICS), many existing defense solutions focus on detecting attacks only when they make the system behave anomalously. Instead, in this work, we study how to detect attackers who are still in their hiding phase. Specifically, we consider an off-path false-data-injection attacker who makes the original sensor's readings unavailable and then impersonates that sensor by sending out legitimate-looking fake readings, so that she can stay hidden in the system for a prolonged period of time (e.g., to gain more information or to launch the actual devastating attack on a specific time). To expose such hidden attackers, our approach relies on continuous injection of ``micro distortion'' to the original sensor's readings, either through digital or physical means. We keep the distortions strictly within a small magnitude (e.g., $0.5\%$ of the possible operating value range) to ensure that it does not affect the normal functioning of the ICS. Micro-distortions are generated based on secret key(s) shared only between the targeted sensor and the defender. For digitally-inserted micro-distortions, we propose and discuss the pros and cons of a two-layer least-significant-bit-based detection algorithm. Alternatively, when the micro-distortions are added physically, a main design challenge is to ensure the introduced micro-distortions do not get overwhelmed by the fluctuation of actual readings and can still provide accurate detection capability. Towards that, we propose a simple yet effective Filtered-$\Delta$-Mean-Difference algorithm that can expose the hidden attackers in a highly accurate and fast manner. We demonstrate the effectiveness and versatility of our defense by using real-world sensor reading traces from different industrial control (including smart grid) systems.

Title: Exploiting Parallel Memory Write Requests for Covert Channel Attacks in Integrated CPU-GPU Systems. (arXiv:2307.16123v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2307.16123
• Code URL: null
• Copy Paste: [[2307.16123] Exploiting Parallel Memory Write Requests for Covert Channel Attacks in Integrated CPU-GPU Systems](http://arxiv.org/abs/2307.16123) #attack
• Summary:

  In heterogeneous SoCs, accelerators like integrated GPUs (iGPUs) are integrated on the same chip as CPUs, sharing the memory subsystem. In such systems, the massive memory requests from throughput-oriented accelerators significantly interfere with CPU memory requests. In addition to the large performance impact, this interference provides an attacker with a strong leakage vector for covert attacks across the processors, which is hard to achieve across the cores in a multi-core CPU. In this paper, we demonstrate that parallel memory write requests of the iGPU and more specifically, the management policy of the write buffer in the memory controller (MC) can lead to significantly stalling CPU memory read requests in heterogeneous SoCs. We characterize the slowdown on the shared read and write buffers in the memory controller and exploit it to build a cross-processor covert channel in Intel-based integrated CPU-GPU systems. We develop two attack variants that achieve a bandwidth of 1.65 kbps and 4.41 kbps and error rates of 0.49% and 4.32% respectively.

Title: On Neural Network approximation of ideal adversarial attack and convergence of adversarial training. (arXiv:2307.16099v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.16099
• Code URL: null
• Copy Paste: [[2307.16099] On Neural Network approximation of ideal adversarial attack and convergence of adversarial training](http://arxiv.org/abs/2307.16099) #attack
• Summary:

  Adversarial attacks are usually expressed in terms of a gradient-based operation on the input data and model, this results in heavy computations every time an attack is generated. In this work, we solidify the idea of representing adversarial attacks as a trainable function, without further gradient computation. We first motivate that the theoretical best attacks, under proper conditions, can be represented as smooth piece-wise functions (piece-wise H\"older functions). Then we obtain an approximation result of such functions by a neural network. Subsequently, we emulate the ideal attack process by a neural network and reduce the adversarial training to a mathematical game between an attack network and a training model (a defense network). We also obtain convergence rates of adversarial loss in terms of the sample size $n$ for adversarial training in such a setting.

robust

Title: Fuzzy Logic Visual Network (FLVN): A neuro-symbolic approach for visual features matching. (arXiv:2307.16019v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16019
• Code URL: https://gitlab.com/grains2/flvn
• Copy Paste: [[2307.16019] Fuzzy Logic Visual Network (FLVN): A neuro-symbolic approach for visual features matching](http://arxiv.org/abs/2307.16019) #robust
• Summary:

  Neuro-symbolic integration aims at harnessing the power of symbolic knowledge representation combined with the learning capabilities of deep neural networks. In particular, Logic Tensor Networks (LTNs) allow to incorporate background knowledge in the form of logical axioms by grounding a first order logic language as differentiable operations between real tensors. Yet, few studies have investigated the potential benefits of this approach to improve zero-shot learning (ZSL) classification. In this study, we present the Fuzzy Logic Visual Network (FLVN) that formulates the task of learning a visual-semantic embedding space within a neuro-symbolic LTN framework. FLVN incorporates prior knowledge in the form of class hierarchies (classes and macro-classes) along with robust high-level inductive biases. The latter allow, for instance, to handle exceptions in class-level attributes, and to enforce similarity between images of the same class, preventing premature overfitting to seen classes and improving overall performance. FLVN reaches state of the art performance on the Generalized ZSL (GZSL) benchmarks AWA2 and CUB, improving by 1.3% and 3%, respectively. Overall, it achieves competitive performance to recent ZSL methods with less computational overhead. FLVN is available at https://gitlab.com/grains2/flvn.

Title: Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving. (arXiv:2307.16121v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16121
• Code URL: null
• Copy Paste: [[2307.16121] Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving](http://arxiv.org/abs/2307.16121) #robust
• Summary:

  Multi-modal fusion has shown initial promising results for object detection of autonomous driving perception. However, many existing fusion schemes do not consider the quality of each fusion input and may suffer from adverse conditions on one or more sensors. While predictive uncertainty has been applied to characterize single-modal object detection performance at run time, incorporating uncertainties into the multi-modal fusion still lacks effective solutions due primarily to the uncertainty's cross-modal incomparability and distinct sensitivities to various adverse conditions. To fill this gap, this paper proposes Uncertainty-Encoded Mixture-of-Experts (UMoE) that explicitly incorporates single-modal uncertainties into LiDAR-camera fusion. UMoE uses individual expert network to process each sensor's detection result together with encoded uncertainty. Then, the expert networks' outputs are analyzed by a gating network to determine the fusion weights. The proposed UMoE module can be integrated into any proposal fusion pipeline. Evaluation shows that UMoE achieves a maximum of 10.67%, 3.17%, and 5.40% performance gain compared with the state-of-the-art proposal-level multi-modal object detectors under extreme weather, adversarial, and blinding attack scenarios.

Title: Open-Set Domain Adaptation with Visual-Language Foundation Models. (arXiv:2307.16204v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16204
• Code URL: null
• Copy Paste: [[2307.16204] Open-Set Domain Adaptation with Visual-Language Foundation Models](http://arxiv.org/abs/2307.16204) #robust
• Summary:

  Unsupervised domain adaptation (UDA) has proven to be very effective in transferring knowledge obtained from a source domain with labeled data to a target domain with unlabeled data. Owing to the lack of labeled data in the target domain and the possible presence of unknown classes, open-set domain adaptation (ODA) has emerged as a potential solution to identify these classes during the training phase. Although existing ODA approaches aim to solve the distribution shifts between the source and target domains, most methods fine-tuned ImageNet pre-trained models on the source domain with the adaptation on the target domain. Recent visual-language foundation models (VLFM), such as Contrastive Language-Image Pre-Training (CLIP), are robust to many distribution shifts and, therefore, should substantially improve the performance of ODA. In this work, we explore generic ways to adopt CLIP, a popular VLFM, for ODA. We investigate the performance of zero-shot prediction using CLIP, and then propose an entropy optimization strategy to assist the ODA models with the outputs of CLIP. The proposed approach achieves state-of-the-art results on various benchmarks, demonstrating its effectiveness in addressing the ODA problem.

Title: Seeking the Yield Barrier: High-Dimensional SRAM Evaluation Through Optimal Manifold. (arXiv:2307.15773v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.15773
• Code URL: null
• Copy Paste: [[2307.15773] Seeking the Yield Barrier: High-Dimensional SRAM Evaluation Through Optimal Manifold](http://arxiv.org/abs/2307.15773) #robust
• Summary:

  Being able to efficiently obtain an accurate estimate of the failure probability of SRAM components has become a central issue as model circuits shrink their scale to submicrometer with advanced technology nodes. In this work, we revisit the classic norm minimization method. We then generalize it with infinite components and derive the novel optimal manifold concept, which bridges the surrogate-based and importance sampling (IS) yield estimation methods. We then derive a sub-optimal manifold, optimal hypersphere, which leads to an efficient sampling method being aware of the failure boundary called onion sampling. Finally, we use a neural coupling flow (which learns from samples like a surrogate model) as the IS proposal distribution. These combinations give rise to a novel yield estimation method, named Optimal Manifold Important Sampling (OPTIMIS), which keeps the advantages of the surrogate and IS methods to deliver state-of-the-art performance with robustness and consistency, with up to 3.5x in efficiency and 3x in accuracy over the best of SOTA methods in High-dimensional SRAM evaluation.

Title: Improving Realistic Worst-Case Performance of NVCiM DNN Accelerators through Training with Right-Censored Gaussian Noise. (arXiv:2307.15853v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.15853
• Code URL: null
• Copy Paste: [[2307.15853] Improving Realistic Worst-Case Performance of NVCiM DNN Accelerators through Training with Right-Censored Gaussian Noise](http://arxiv.org/abs/2307.15853) #robust
• Summary:

  Compute-in-Memory (CiM), built upon non-volatile memory (NVM) devices, is promising for accelerating deep neural networks (DNNs) owing to its in-situ data processing capability and superior energy efficiency. Unfortunately, the well-trained model parameters, after being mapped to NVM devices, can often exhibit large deviations from their intended values due to device variations, resulting in notable performance degradation in these CiM-based DNN accelerators. There exists a long list of solutions to address this issue. However, they mainly focus on improving the mean performance of CiM DNN accelerators. How to guarantee the worst-case performance under the impact of device variations, which is crucial for many safety-critical applications such as self-driving cars, has been far less explored. In this work, we propose to use the k-th percentile performance (KPP) to capture the realistic worst-case performance of DNN models executing on CiM accelerators. Through a formal analysis of the properties of KPP and the noise injection-based DNN training, we demonstrate that injecting a novel right-censored Gaussian noise, as opposed to the conventional Gaussian noise, significantly improves the KPP of DNNs. We further propose an automated method to determine the optimal hyperparameters for injecting this right-censored Gaussian noise during the training process. Our method achieves up to a 26% improvement in KPP compared to the state-of-the-art methods employed to enhance DNN robustness under the impact of device variations.

Title: Multi-output Headed Ensembles for Product Item Classification. (arXiv:2307.15858v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.15858
• Code URL: null
• Copy Paste: [[2307.15858] Multi-output Headed Ensembles for Product Item Classification](http://arxiv.org/abs/2307.15858) #robust
• Summary:

  In this paper, we revisit the problem of product item classification for large-scale e-commerce catalogs. The taxonomy of e-commerce catalogs consists of thousands of genres to which are assigned items that are uploaded by merchants on a continuous basis. The genre assignments by merchants are often wrong but treated as ground truth labels in automatically generated training sets, thus creating a feedback loop that leads to poorer model quality over time. This problem of taxonomy classification becomes highly pronounced due to the unavailability of sizable curated training sets.

Under such a scenario it is common to combine multiple classifiers to combat poor generalization performance from a single classifier. We propose an extensible deep learning based classification model framework that benefits from the simplicity and robustness of averaging ensembles and fusion based classifiers. We are also able to use metadata features and low-level feature engineering to boost classification performance. We show these improvements against robust industry standard baseline models that employ hyperparameter optimization.

Additionally, due to continuous insertion, deletion and updates to real-world high-volume e-commerce catalogs, assessing model performance for deployment using A/B testing and/or manual annotation becomes a bottleneck. To this end, we also propose a novel way to evaluate model performance using user sessions that provides better insights in addition to traditional measures of precision and recall.

Title: Multi-view Sparse Laplacian Eigenmaps for nonlinear Spectral Feature Selection. (arXiv:2307.15905v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.15905
• Code URL: null
• Copy Paste: [[2307.15905] Multi-view Sparse Laplacian Eigenmaps for nonlinear Spectral Feature Selection](http://arxiv.org/abs/2307.15905) #robust
• Summary:

  The complexity of high-dimensional datasets presents significant challenges for machine learning models, including overfitting, computational complexity, and difficulties in interpreting results. To address these challenges, it is essential to identify an informative subset of features that captures the essential structure of the data. In this study, the authors propose Multi-view Sparse Laplacian Eigenmaps (MSLE) for feature selection, which effectively combines multiple views of the data, enforces sparsity constraints, and employs a scalable optimization algorithm to identify a subset of features that capture the fundamental data structure. MSLE is a graph-based approach that leverages multiple views of the data to construct a more robust and informative representation of high-dimensional data. The method applies sparse eigendecomposition to reduce the dimensionality of the data, yielding a reduced feature set. The optimization problem is solved using an iterative algorithm alternating between updating the sparse coefficients and the Laplacian graph matrix. The sparse coefficients are updated using a soft-thresholding operator, while the graph Laplacian matrix is updated using the normalized graph Laplacian. To evaluate the performance of the MSLE technique, the authors conducted experiments on the UCI-HAR dataset, which comprises 561 features, and reduced the feature space by 10 to 90%. Our results demonstrate that even after reducing the feature space by 90%, the Support Vector Machine (SVM) maintains an error rate of 2.72%. Moreover, the authors observe that the SVM exhibits an accuracy of 96.69% with an 80% reduction in the overall feature space.

Title: An Automata-Theoretic Approach to Synthesizing Binarized Neural Networks. (arXiv:2307.15907v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.15907
• Code URL: null
• Copy Paste: [[2307.15907] An Automata-Theoretic Approach to Synthesizing Binarized Neural Networks](http://arxiv.org/abs/2307.15907) #robust
• Summary:

  Deep neural networks, (DNNs, a.k.a. NNs), have been widely used in various tasks and have been proven to be successful. However, the accompanied expensive computing and storage costs make the deployments in resource-constrained devices a significant concern. To solve this issue, quantization has emerged as an effective way to reduce the costs of DNNs with little accuracy degradation by quantizing floating-point numbers to low-width fixed-point representations. Quantized neural networks (QNNs) have been developed, with binarized neural networks (BNNs) restricted to binary values as a special case. Another concern about neural networks is their vulnerability and lack of interpretability. Despite the active research on trustworthy of DNNs, few approaches have been proposed to QNNs. To this end, this paper presents an automata-theoretic approach to synthesizing BNNs that meet designated properties. More specifically, we define a temporal logic, called BLTL, as the specification language. We show that each BLTL formula can be transformed into an automaton on finite words. To deal with the state-explosion problem, we provide a tableau-based approach in real implementation. For the synthesis procedure, we utilize SMT solvers to detect the existence of a model (i.e., a BNN) in the construction process. Notably, synthesis provides a way to determine the hyper-parameters of the network before training.Moreover, we experimentally evaluate our approach and demonstrate its effectiveness in improving the individual fairness and local robustness of BNNs while maintaining accuracy to a great extent.

Title: Dynamic deep-reinforcement-learning algorithm in Partially Observed Markov Decision Processes. (arXiv:2307.15931v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.15931
• Code URL: null
• Copy Paste: [[2307.15931] Dynamic deep-reinforcement-learning algorithm in Partially Observed Markov Decision Processes](http://arxiv.org/abs/2307.15931) #robust
• Summary:

  Reinforcement learning has been greatly improved in recent studies and an increased interest in real-world implementation has emerged in recent years. In many cases, due to the non-static disturbances, it becomes challenging for the agent to keep the performance. The disturbance results in the environment called Partially Observable Markov Decision Process. In common practice, Partially Observable Markov Decision Process is handled by introducing an additional estimator, or Recurrent Neural Network is utilized in the context of reinforcement learning. Both of the cases require to process sequential information on the trajectory. However, there are only a few studies investigating the effect of information to consider and the network structure to handle them. This study shows the benefit of action sequence inclusion in order to solve Partially Observable Markov Decision Process. Several structures and approaches are proposed to extend one of the latest deep reinforcement learning algorithms with LSTM networks. The developed algorithms showed enhanced robustness of controller performance against different types of external disturbances that are added to observation.

Title: A Noisy-Label-Learning Formulation for Immune Repertoire Classification and Disease-Associated Immune Receptor Sequence Identification. (arXiv:2307.15934v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.15934
• Code URL: https://github.com/tencentailabhealthcare/nll-irc
• Copy Paste: [[2307.15934] A Noisy-Label-Learning Formulation for Immune Repertoire Classification and Disease-Associated Immune Receptor Sequence Identification](http://arxiv.org/abs/2307.15934) #robust
• Summary:

  Immune repertoire classification, a typical multiple instance learning (MIL) problem, is a frontier research topic in computational biology that makes transformative contributions to new vaccines and immune therapies. However, the traditional instance-space MIL, directly assigning bag-level labels to instances, suffers from the massive amount of noisy labels and extremely low witness rate. In this work, we propose a noisy-label-learning formulation to solve the immune repertoire classification task. To remedy the inaccurate supervision of repertoire-level labels for a sequence-level classifier, we design a robust training strategy: The initial labels are smoothed to be asymmetric and are progressively corrected using the model's predictions throughout the training process. Furthermore, two models with the same architecture but different parameter initialization are co-trained simultaneously to remedy the known "confirmation bias" problem in the self-training-like schema. As a result, we obtain accurate sequence-level classification and, subsequently, repertoire-level classification. Experiments on the Cytomegalovirus (CMV) and Cancer datasets demonstrate our method's effectiveness and superior performance on sequence-level and repertoire-level tasks.

Title: An Efficient Approach to Mitigate Numerical Instability in Backpropagation for 16-bit Neural Network Training. (arXiv:2307.16189v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.16189
• Code URL: null
• Copy Paste: [[2307.16189] An Efficient Approach to Mitigate Numerical Instability in Backpropagation for 16-bit Neural Network Training](http://arxiv.org/abs/2307.16189) #robust
• Summary:

  In this research, we delve into the intricacies of the numerical instability observed in 16-bit computations of machine learning models, particularly when employing popular optimization algorithms such as RMSProp and Adam. This instability is commonly experienced during the training phase of deep neural networks, leading to disrupted learning processes and hindering the effective deployment of such models. We identify the single hyperparameter, epsilon, as the main culprit behind this numerical instability. An in-depth exploration of the role of epsilon in these optimizers within 16-bit computations reveals that a minor adjustment of its value can restore the functionality of RMSProp and Adam, consequently enabling the effective utilization of 16-bit neural networks. We propose a novel method to mitigate the identified numerical instability issues. This method capitalizes on the updates from the Adam optimizer and significantly improves the robustness of the learning process in 16-bit computations. This study contributes to better understanding of optimization in low-precision computations and provides an effective solution to a longstanding issue in training deep neural networks, opening new avenues for more efficient and stable model training.

Title: Robust Multi-Agent Reinforcement Learning with State Uncertainty. (arXiv:2307.16212v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.16212
• Code URL: https://github.com/sihongho/robust_marl_with_state_uncertainty
• Copy Paste: [[2307.16212] Robust Multi-Agent Reinforcement Learning with State Uncertainty](http://arxiv.org/abs/2307.16212) #robust
• Summary:

  In real-world multi-agent reinforcement learning (MARL) applications, agents may not have perfect state information (e.g., due to inaccurate measurement or malicious attacks), which challenges the robustness of agents' policies. Though robustness is getting important in MARL deployment, little prior work has studied state uncertainties in MARL, neither in problem formulation nor algorithm design. Motivated by this robustness issue and the lack of corresponding studies, we study the problem of MARL with state uncertainty in this work. We provide the first attempt to the theoretical and empirical analysis of this challenging problem. We first model the problem as a Markov Game with state perturbation adversaries (MG-SPA) by introducing a set of state perturbation adversaries into a Markov Game. We then introduce robust equilibrium (RE) as the solution concept of an MG-SPA. We conduct a fundamental analysis regarding MG-SPA such as giving conditions under which such a robust equilibrium exists. Then we propose a robust multi-agent Q-learning (RMAQ) algorithm to find such an equilibrium, with convergence guarantees. To handle high-dimensional state-action space, we design a robust multi-agent actor-critic (RMAAC) algorithm based on an analytical expression of the policy gradient derived in the paper. Our experiments show that the proposed RMAQ algorithm converges to the optimal value function; our RMAAC algorithm outperforms several MARL and robust MARL methods in multiple multi-agent environments when state uncertainty is present. The source code is public on \url{https://github.com/sihongho/robust_marl_with_state_uncertainty}.

biometric

Title: GaitASMS: Gait Recognition by Adaptive Structured Spatial Representation and Multi-Scale Temporal Aggregation. (arXiv:2307.15981v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.15981
• Code URL: null
• Copy Paste: [[2307.15981] GaitASMS: Gait Recognition by Adaptive Structured Spatial Representation and Multi-Scale Temporal Aggregation](http://arxiv.org/abs/2307.15981) #biometric
• Summary:

  Gait recognition is one of the most promising video-based biometric technologies. The edge of silhouettes and motion are the most informative feature and previous studies have explored them separately and achieved notable results. However, due to occlusions and variations in viewing angles, their gait recognition performance is often affected by the predefined spatial segmentation strategy. Moreover, traditional temporal pooling usually neglects distinctive temporal information in gait. To address the aforementioned issues, we propose a novel gait recognition framework, denoted as GaitASMS, which can effectively extract the adaptive structured spatial representations and naturally aggregate the multi-scale temporal information. The Adaptive Structured Representation Extraction Module (ASRE) separates the edge of silhouettes by using the adaptive edge mask and maximizes the representation in semantic latent space. Moreover, the Multi-Scale Temporal Aggregation Module (MSTA) achieves effective modeling of long-short-range temporal information by temporally aggregated structure. Furthermore, we propose a new data augmentation, denoted random mask, to enrich the sample space of long-term occlusion and enhance the generalization of the model. Extensive experiments conducted on two datasets demonstrate the competitive advantage of proposed method, especially in complex scenes, i.e. BG and CL. On the CASIA-B dataset, GaitASMS achieves the average accuracy of 93.5\% and outperforms the baseline on rank-1 accuracies by 3.4\% and 6.3\%, respectively, in BG and CL. The ablation experiments demonstrate the effectiveness of ASRE and MSTA.

steal

extraction

Title: Deep Convolutional Neural Networks with Zero-Padding: Feature Extraction and Learning. (arXiv:2307.16203v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.16203
• Code URL: null
• Copy Paste: [[2307.16203] Deep Convolutional Neural Networks with Zero-Padding: Feature Extraction and Learning](http://arxiv.org/abs/2307.16203) #extraction
• Summary:

  This paper studies the performance of deep convolutional neural networks (DCNNs) with zero-padding in feature extraction and learning. After verifying the roles of zero-padding in enabling translation-equivalence, and pooling in its translation-invariance driven nature, we show that with similar number of free parameters, any deep fully connected networks (DFCNs) can be represented by DCNNs with zero-padding. This demonstrates that DCNNs with zero-padding is essentially better than DFCNs in feature extraction. Consequently, we derive universal consistency of DCNNs with zero-padding and show its translation-invariance in the learning process. All our theoretical results are verified by numerical experiments including both toy simulations and real-data running.

Title: ATESA-B{\AE}RT: A Heterogeneous Ensemble Learning Model for Aspect-Based Sentiment Analysis. (arXiv:2307.15920v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.15920
• Code URL: null
• Copy Paste: [[2307.15920] ATESA-B{\AE}RT: A Heterogeneous Ensemble Learning Model for Aspect-Based Sentiment Analysis](http://arxiv.org/abs/2307.15920) #extraction
• Summary:

  The increasing volume of online reviews has made possible the development of sentiment analysis models for determining the opinion of customers regarding different products and services. Until now, sentiment analysis has proven to be an effective tool for determining the overall polarity of reviews. To improve the granularity at the aspect level for a better understanding of the service or product, the task of aspect-based sentiment analysis aims to first identify aspects and then determine the user's opinion about them. The complexity of this task lies in the fact that the same review can present multiple aspects, each with its own polarity. Current solutions have poor performance on such data. We address this problem by proposing ATESA-B{\AE}RT, a heterogeneous ensemble learning model for Aspect-Based Sentiment Analysis. Firstly, we divide our problem into two sub-tasks, i.e., Aspect Term Extraction and Aspect Term Sentiment Analysis. Secondly, we use the \textit{argmax} multi-class classification on six transformers-based learners for each sub-task. Initial experiments on two datasets prove that ATESA-B{\AE}RT outperforms current state-of-the-art solutions while solving the many aspects problem.

Title: Automatic Extraction of the Romanian Academic Word List: Data and Methods. (arXiv:2307.16045v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.16045
• Code URL: https://github.com/bucuram/ro-awl
• Copy Paste: [[2307.16045] Automatic Extraction of the Romanian Academic Word List: Data and Methods](http://arxiv.org/abs/2307.16045) #extraction
• Summary:

  This paper presents the methodology and data used for the automatic extraction of the Romanian Academic Word List (Ro-AWL). Academic Word Lists are useful in both L2 and L1 teaching contexts. For the Romanian language, no such resource exists so far. Ro-AWL has been generated by combining methods from corpus and computational linguistics with L2 academic writing approaches. We use two types of data: (a) existing data, such as the Romanian Frequency List based on the ROMBAC corpus, and (b) self-compiled data, such as the expert academic writing corpus EXPRES. For constructing the academic word list, we follow the methodology for building the Academic Vocabulary List for the English language. The distribution of Ro-AWL features (general distribution, POS distribution) into four disciplinary datasets is in line with previous research. Ro-AWL is freely available and can be used for teaching, research and NLP applications.

Title: EnrichEvent: Enriching Social Data with Contextual Information for Emerging Event Extraction. (arXiv:2307.16082v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.16082
• Code URL: null
• Copy Paste: [[2307.16082] EnrichEvent: Enriching Social Data with Contextual Information for Emerging Event Extraction](http://arxiv.org/abs/2307.16082) #extraction
• Summary:

  Social platforms have emerged as a crucial platform for disseminating and discussing information about real-life events, which offers an excellent opportunity for early detection of newsworthy events. However, most existing approaches for event detection solely exploit keyword burstiness or network structures to detect hot events. Thus, they often fail to identify emerging social events before reaching a trending state regarding the challenging nature of events and social data. Social data, e.g., tweets, is characterized by misspellings, incompleteness, ambiguity, and irregular language, as well as variation in aspects of opinions. Moreover, learning the evolving characteristics of the events utilizing limited contextual knowledge is almost infeasible for machine learning models. To address these problems, in this paper, we propose a framework that exploits the lexical, semantic, and contextual representations of streaming social data. In particular, we leverage contextual knowledge to detect semantically related tweets in their earliest emergence and enhance the quality of produced clusters. We next produce a cluster chains for each event to show the evolving variation of the event through time. We conducted extensive experiments to evaluate our framework, validating the effectiveness of the proposed framework in detecting and distinguishing social events.

Title: A Knowledge-enhanced Two-stage Generative Framework for Medical Dialogue Information Extraction. (arXiv:2307.16200v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.16200
• Code URL: https://github.com/flyingcat-fa/ktgf
• Copy Paste: [[2307.16200] A Knowledge-enhanced Two-stage Generative Framework for Medical Dialogue Information Extraction](http://arxiv.org/abs/2307.16200) #extraction
• Summary:

  This paper focuses on term-status pair extraction from medical dialogues (MD-TSPE), which is essential in diagnosis dialogue systems and the automatic scribe of electronic medical records (EMRs). In the past few years, works on MD-TSPE have attracted increasing research attention, especially after the remarkable progress made by generative methods. However, these generative methods output a whole sequence consisting of term-status pairs in one stage and ignore integrating prior knowledge, which demands a deeper understanding to model the relationship between terms and infer the status of each term. This paper presents a knowledge-enhanced two-stage generative framework (KTGF) to address the above challenges. Using task-specific prompts, we employ a single model to complete the MD-TSPE through two phases in a unified generative form: we generate all terms the first and then generate the status of each generated term. In this way, the relationship between terms can be learned more effectively from the sequence containing only terms in the first phase, and our designed knowledge-enhanced prompt in the second phase can leverage the category and status candidates of the generated term for status generation. Furthermore, our proposed special status ``not mentioned" makes more terms available and enriches the training data in the second phase, which is critical in the low-resource setting. The experiments on the Chunyu and CMDD datasets show that the proposed method achieves superior results compared to the state-of-the-art models in the full training and low-resource settings.

Title: Text Analysis Using Deep Neural Networks in Digital Humanities and Information Science. (arXiv:2307.16217v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.16217
• Code URL: null
• Copy Paste: [[2307.16217] Text Analysis Using Deep Neural Networks in Digital Humanities and Information Science](http://arxiv.org/abs/2307.16217) #extraction
• Summary:

  Combining computational technologies and humanities is an ongoing effort aimed at making resources such as texts, images, audio, video, and other artifacts digitally available, searchable, and analyzable. In recent years, deep neural networks (DNN) dominate the field of automatic text analysis and natural language processing (NLP), in some cases presenting a super-human performance. DNNs are the state-of-the-art machine learning algorithms solving many NLP tasks that are relevant for Digital Humanities (DH) research, such as spell checking, language detection, entity extraction, author detection, question answering, and other tasks. These supervised algorithms learn patterns from a large number of "right" and "wrong" examples and apply them to new examples. However, using DNNs for analyzing the text resources in DH research presents two main challenges: (un)availability of training data and a need for domain adaptation. This paper explores these challenges by analyzing multiple use-cases of DH studies in recent literature and their possible solutions and lays out a practical decision model for DH experts for when and how to choose the appropriate deep learning approaches for their research. Moreover, in this paper, we aim to raise awareness of the benefits of utilizing deep learning models in the DH community.

membership infer

federate

Title: You Can Backdoor Personalized Federated Learning. (arXiv:2307.15971v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2307.15971
• Code URL: https://github.com/bapfl/code
• Copy Paste: [[2307.15971] You Can Backdoor Personalized Federated Learning](http://arxiv.org/abs/2307.15971) #federate
• Summary:

  Backdoor attacks pose a significant threat to the security of federated learning systems. However, existing research primarily focuses on backdoor attacks and defenses within the generic FL scenario, where all clients collaborate to train a single global model. \citet{qin2023revisiting} conduct the first study of backdoor attacks in the personalized federated learning (pFL) scenario, where each client constructs a personalized model based on its local data. Notably, the study demonstrates that pFL methods with partial model-sharing can significantly boost robustness against backdoor attacks. In this paper, we whistleblow that pFL methods with partial model-sharing are still vulnerable to backdoor attacks in the absence of any defense. We propose three backdoor attack methods: BapFL, BapFL+, and Gen-BapFL, and we empirically demonstrate that they can effectively attack the pFL methods. Specifically, the key principle of BapFL lies in maintaining clean local parameters while implanting the backdoor into the global parameters. BapFL+ generalizes the attack success to benign clients by introducing Gaussian noise to the local parameters. Furthermore, we assume the collaboration of malicious clients and propose Gen-BapFL, which leverages meta-learning techniques to further enhances attack generalization. We evaluate our proposed attack methods against two classic pFL methods with partial model-sharing, FedPer and LG-FedAvg. Extensive experiments on four FL benchmark datasets demonstrate the effectiveness of our proposed attack methods. Additionally, we assess the defense efficacy of various defense strategies against our proposed attacks and find that Gradient Norm-Clipping is particularly effective. It is crucial to note that pFL method is not always secure in the presence of backdoor attacks, and we hope to inspire further research on attack and defense in pFL scenarios.

Title: Shuffled Differentially Private Federated Learning for Time Series Data Analytics. (arXiv:2307.16196v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.16196
• Code URL: null
• Copy Paste: [[2307.16196] Shuffled Differentially Private Federated Learning for Time Series Data Analytics](http://arxiv.org/abs/2307.16196) #federate
• Summary:

  Trustworthy federated learning aims to achieve optimal performance while ensuring clients' privacy. Existing privacy-preserving federated learning approaches are mostly tailored for image data, lacking applications for time series data, which have many important applications, like machine health monitoring, human activity recognition, etc. Furthermore, protective noising on a time series data analytics model can significantly interfere with temporal-dependent learning, leading to a greater decline in accuracy. To address these issues, we develop a privacy-preserving federated learning algorithm for time series data. Specifically, we employ local differential privacy to extend the privacy protection trust boundary to the clients. We also incorporate shuffle techniques to achieve a privacy amplification, mitigating the accuracy decline caused by leveraging local differential privacy. Extensive experiments were conducted on five time series datasets. The evaluation results reveal that our algorithm experienced minimal accuracy loss compared to non-private federated learning in both small and large client scenarios. Under the same level of privacy protection, our algorithm demonstrated improved accuracy compared to the centralized differentially private federated learning in both scenarios.

Title: Efficient Semi-Supervised Federated Learning for Heterogeneous Participants. (arXiv:2307.15870v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.15870
• Code URL: null
• Copy Paste: [[2307.15870] Efficient Semi-Supervised Federated Learning for Heterogeneous Participants](http://arxiv.org/abs/2307.15870) #federate
• Summary:

  Federated Learning (FL) has emerged to allow multiple clients to collaboratively train machine learning models on their private data. However, training and deploying large models for broader applications is challenging in resource-constrained environments. Fortunately, Split Federated Learning (SFL) offers an excellent solution by alleviating the computation and communication burden on the clients SFL often assumes labeled data for local training on clients, however, it is not the case in practice.Prior works have adopted semi-supervised techniques for leveraging unlabeled data in FL, but data non-IIDness poses another challenge to ensure training efficiency. Herein, we propose Pseudo-Clustering Semi-SFL, a novel system for training models in scenarios where labeled data reside on the server. By introducing Clustering Regularization, model performance under data non-IIDness can be improved. Besides, our theoretical and experimental investigations into model convergence reveal that the inconsistent training processes on labeled and unlabeled data impact the effectiveness of clustering regularization. Upon this, we develop a control algorithm for global updating frequency adaptation, which dynamically adjusts the number of supervised training iterations to mitigate the training inconsistency. Extensive experiments on benchmark models and datasets show that our system provides a 3.3x speed-up in training time and reduces the communication cost by about 80.1% while reaching the target accuracy, and achieves up to 6.9% improvement in accuracy under non-IID scenarios compared to the state-of-the-art.

Title: UPFL: Unsupervised Personalized Federated Learning towards New Clients. (arXiv:2307.15994v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.15994
• Code URL: https://github.com/anonymous-federated-learning/code
• Copy Paste: [[2307.15994] UPFL: Unsupervised Personalized Federated Learning towards New Clients](http://arxiv.org/abs/2307.15994) #federate
• Summary:

  Personalized federated learning has gained significant attention as a promising approach to address the challenge of data heterogeneity. In this paper, we address a relatively unexplored problem in federated learning. When a federated model has been trained and deployed, and an unlabeled new client joins, providing a personalized model for the new client becomes a highly challenging task. To address this challenge, we extend the adaptive risk minimization technique into the unsupervised personalized federated learning setting and propose our method, FedTTA. We further improve FedTTA with two simple yet effective optimization strategies: enhancing the training of the adaptation model with proxy regularization and early-stopping the adaptation through entropy. Moreover, we propose a knowledge distillation loss specifically designed for FedTTA to address the device heterogeneity. Extensive experiments on five datasets against eleven baselines demonstrate the effectiveness of our proposed FedTTA and its variants. The code is available at: https://github.com/anonymous-federated-learning/code.

fair

interpretability

explainability

watermark

Title: Towards Codable Text Watermarking for Large Language Models. (arXiv:2307.15992v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.15992
• Code URL: https://github.com/lancopku/codable-watermarking-for-llm
• Copy Paste: [[2307.15992] Towards Codable Text Watermarking for Large Language Models](http://arxiv.org/abs/2307.15992) #watermark
• Summary:

  As large language models (LLMs) generate texts with increasing fluency and realism, there is a growing need to identify the source of texts to prevent the abuse of LLMs. Text watermarking techniques have proven reliable in distinguishing whether a text is generated by LLMs by injecting hidden patterns into the generated texts. However, we argue that existing watermarking methods for LLMs are encoding-inefficient (only contain one bit of information - whether it is generated from an LLM or not) and cannot flexibly meet the diverse information encoding needs (such as encoding model version, generation time, user id, etc.) in different LLMs application scenarios. In this work, we conduct the first systematic study on the topic of Codable Text Watermarking for LLMs (CTWL) that allows text watermarks to carry more customizable information. First of all, we study the taxonomy of LLM watermarking technology and give a mathematical formulation for CTWL. Additionally, we provide a comprehensive evaluation system for CTWL: (1) watermarking success rate, (2) robustness against various corruptions, (3) coding rate of payload information, (4) encoding and decoding efficiency, (5) impacts on the quality of the generated text. To meet the requirements of these non-Pareto-improving metrics, we devise a CTWL method named Balance-Marking, based on the motivation of ensuring that available and unavailable vocabularies for encoding information have approximately equivalent probabilities. Compared to the random vocabulary partitioning extended from the existing work, a probability-balanced vocabulary partition can significantly improve the quality of the generated text. Extensive experimental results have shown that our method outperforms a direct baseline under comprehensive evaluation.

Title: A Private Watermark for Large Language Models. (arXiv:2307.16230v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.16230
• Code URL: null
• Copy Paste: [[2307.16230] A Private Watermark for Large Language Models](http://arxiv.org/abs/2307.16230) #watermark
• Summary:

  Recently, text watermarking algorithms for large language models (LLMs) have been mitigating the potential harms of text generated by the LLMs, including fake news and copyright issues. However, the watermark detection of current text algorithms requires the key from the generation process, making them susceptible to breaches and counterfeiting. In this work, we propose the first private watermarking algorithm, which extends the current text watermarking algorithms by using two different neural networks respectively for watermark generation and detection, rather than using the same key at both stages. Meanwhile, part of the parameters of the watermark generation and detection networks are shared, which makes the detection network achieve a high accuracy very efficiently. Experiments show that our algorithm ensures high detection accuracy with minimal impact on generation and detection speed, due to the small parameter size of both networks. Additionally, our subsequent analysis demonstrates the difficulty of reverting the watermark generation rules from the detection network.

diffusion

Title: RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects. (arXiv:2307.15988v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.15988
• Code URL: null
• Copy Paste: [[2307.15988] RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects](http://arxiv.org/abs/2307.15988) #diffusion
• Summary:

  We present RGB-D-Fusion, a multi-modal conditional denoising diffusion probabilistic model to generate high resolution depth maps from low-resolution monocular RGB images of humanoid subjects. RGB-D-Fusion first generates a low-resolution depth map using an image conditioned denoising diffusion probabilistic model and then upsamples the depth map using a second denoising diffusion probabilistic model conditioned on a low-resolution RGB-D image. We further introduce a novel augmentation technique, depth noise augmentation, to increase the robustness of our super-resolution model.

Title: Ultrasound Image Reconstruction with Denoising Diffusion Restoration Models. (arXiv:2307.15990v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.15990
• Code URL: https://github.com/yuxin-zhang-jasmine/drus-v1
• Copy Paste: [[2307.15990] Ultrasound Image Reconstruction with Denoising Diffusion Restoration Models](http://arxiv.org/abs/2307.15990) #diffusion
• Summary:

  Ultrasound image reconstruction can be approximately cast as a linear inverse problem that has traditionally been solved with penalized optimization using the $l_1$ or $l_2$ norm, or wavelet-based terms. However, such regularization functions often struggle to balance the sparsity and the smoothness. A promising alternative is using learned priors to make the prior knowledge closer to reality. In this paper, we rely on learned priors under the framework of Denoising Diffusion Restoration Models (DDRM), initially conceived for restoration tasks with natural images. We propose and test two adaptions of DDRM to ultrasound inverse problem models, DRUS and WDRUS. Our experiments on synthetic and PICMUS data show that from a single plane wave our method can achieve image quality comparable to or better than DAS and state-of-the-art methods. The code is available at: https://github.com/Yuxin-Zhang-Jasmine/DRUS-v1.

Title: HD-Fusion: Detailed Text-to-3D Generation Leveraging Multiple Noise Estimation. (arXiv:2307.16183v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16183
• Code URL: null
• Copy Paste: [[2307.16183] HD-Fusion: Detailed Text-to-3D Generation Leveraging Multiple Noise Estimation](http://arxiv.org/abs/2307.16183) #diffusion
• Summary:

  In this paper, we study Text-to-3D content generation leveraging 2D diffusion priors to enhance the quality and detail of the generated 3D models. Recent progress (Magic3D) in text-to-3D has shown that employing high-resolution (e.g., 512 x 512) renderings can lead to the production of high-quality 3D models using latent diffusion priors. To enable rendering at even higher resolutions, which has the potential to further augment the quality and detail of the models, we propose a novel approach that combines multiple noise estimation processes with a pretrained 2D diffusion prior. Distinct from the Bar-Tal et al.s' study which binds multiple denoised results to generate images from texts, our approach integrates the computation of scoring distillation losses such as SDS loss and VSD loss which are essential techniques for the 3D content generation with 2D diffusion priors. We experimentally evaluated the proposed approach. The results show that the proposed approach can generate high-quality details compared to the baselines.

Title: ADR-GNN: Advection-Diffusion-Reaction Graph Neural Networks. (arXiv:2307.16092v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.16092
• Code URL: null
• Copy Paste: [[2307.16092] ADR-GNN: Advection-Diffusion-Reaction Graph Neural Networks](http://arxiv.org/abs/2307.16092) #diffusion
• Summary:

  Graph neural networks (GNNs) have shown remarkable success in learning representations for graph-structured data. However, GNNs still face challenges in modeling complex phenomena that involve advection. In this paper, we propose a novel GNN architecture based on Advection-Diffusion-Reaction systems, called ADR-GNN. Advection models the directed transportation of information, diffusion captures the local smoothing of information, and reaction represents the non-linear transformation of information in channels. We provide an analysis of the qualitative behavior of ADR-GNN, that shows the benefit of combining advection, diffusion, and reaction. To demonstrate its efficacy, we evaluate ADR-GNN on real-world node classification and spatio-temporal datasets, and show that it improves or offers competitive performance compared to state-of-the-art networks.

noise learning

data-free

transformer

Title: Transformer on Shuttlecock Flying Direction Prediction for Hit-frame Detection. (arXiv:2307.16000v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16000
• Code URL: https://github.com/arthur900530/Transformer-on-Shuttlecock-Flying-Direction-Prediction-for-Hit-frame-Detection
• Copy Paste: [[2307.16000] Transformer on Shuttlecock Flying Direction Prediction for Hit-frame Detection](http://arxiv.org/abs/2307.16000) #transformer
• Summary:

  Sports professionals constantly under pressure to perform at the highest level can benefit from sports analysis, which allows coaches and players to reduce manual efforts and systematically evaluate their performance using automated tools. This research aims to advance sports analysis in badminton, systematically detecting hit-frames automatically from match videos using modern deep learning techniques. The data included in hit-frames can subsequently be utilized to synthesize players' strokes and on-court movement, as well as for other downstream applications such as analyzing training tasks and competition strategy. The proposed approach in this study comprises several automated procedures like rally-wise video trimming, player and court keypoints detection, shuttlecock flying direction prediction, and hit-frame detection. In the study, we achieved 99% accuracy on shot angle recognition for video trimming, over 92% accuracy for applying player keypoints sequences on shuttlecock flying direction prediction, and reported the evaluation results of rally-wise video trimming and hit-frame detection.

Title: Enhancing Object Detection in Ancient Documents with Synthetic Data Generation and Transformer-Based Models. (arXiv:2307.16005v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16005
• Code URL: null
• Copy Paste: [[2307.16005] Enhancing Object Detection in Ancient Documents with Synthetic Data Generation and Transformer-Based Models](http://arxiv.org/abs/2307.16005) #transformer
• Summary:

  The study of ancient documents provides a glimpse into our past. However, the low image quality and intricate details commonly found in these documents present significant challenges for accurate object detection. The objective of this research is to enhance object detection in ancient documents by reducing false positives and improving precision. To achieve this, we propose a method that involves the creation of synthetic datasets through computational mediation, along with the integration of visual feature extraction into the object detection process. Our approach includes associating objects with their component parts and introducing a visual feature map to enable the model to discern between different symbols and document elements. Through our experiments, we demonstrate that improved object detection has a profound impact on the field of Paleography, enabling in-depth analysis and fostering a greater understanding of these valuable historical artifacts.

Title: HandMIM: Pose-Aware Self-Supervised Learning for 3D Hand Mesh Estimation. (arXiv:2307.16061v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16061
• Code URL: null
• Copy Paste: [[2307.16061] HandMIM: Pose-Aware Self-Supervised Learning for 3D Hand Mesh Estimation](http://arxiv.org/abs/2307.16061) #transformer
• Summary:

  With an enormous number of hand images generated over time, unleashing pose knowledge from unlabeled images for supervised hand mesh estimation is an emerging yet challenging topic. To alleviate this issue, semi-supervised and self-supervised approaches have been proposed, but they are limited by the reliance on detection models or conventional ResNet backbones. In this paper, inspired by the rapid progress of Masked Image Modeling (MIM) in visual classification tasks, we propose a novel self-supervised pre-training strategy for regressing 3D hand mesh parameters. Our approach involves a unified and multi-granularity strategy that includes a pseudo keypoint alignment module in the teacher-student framework for learning pose-aware semantic class tokens. For patch tokens with detailed locality, we adopt a self-distillation manner between teacher and student network based on MIM pre-training. To better fit low-level regression tasks, we incorporate pixel reconstruction tasks for multi-level representation learning. Additionally, we design a strong pose estimation baseline using a simple vanilla vision Transformer (ViT) as the backbone and attach a PyMAF head after tokens for regression. Extensive experiments demonstrate that our proposed approach, named HandMIM, achieves strong performance on various hand mesh estimation tasks. Notably, HandMIM outperforms specially optimized architectures, achieving 6.29mm and 8.00mm PAVPE (Vertex-Point-Error) on challenging FreiHAND and HO3Dv2 test sets, respectively, establishing new state-of-the-art records on 3D hand mesh estimation.

Title: Video Frame Interpolation with Flow Transformer. (arXiv:2307.16144v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16144
• Code URL: null
• Copy Paste: [[2307.16144] Video Frame Interpolation with Flow Transformer](http://arxiv.org/abs/2307.16144) #transformer
• Summary:

  Video frame interpolation has been actively studied with the development of convolutional neural networks. However, due to the intrinsic limitations of kernel weight sharing in convolution, the interpolated frame generated by it may lose details. In contrast, the attention mechanism in Transformer can better distinguish the contribution of each pixel, and it can also capture long-range pixel dependencies, which provides great potential for video interpolation. Nevertheless, the original Transformer is commonly used for 2D images; how to develop a Transformer-based framework with consideration of temporal self-attention for video frame interpolation remains an open issue. In this paper, we propose Video Frame Interpolation Flow Transformer to incorporate motion dynamics from optical flows into the self-attention mechanism. Specifically, we design a Flow Transformer Block that calculates the temporal self-attention in a matched local area with the guidance of flow, making our framework suitable for interpolating frames with large motion while maintaining reasonably low complexity. In addition, we construct a multi-scale architecture to account for multi-scale motion, further improving the overall performance. Extensive experiments on three benchmarks demonstrate that the proposed method can generate interpolated frames with better visual quality than state-of-the-art methods.

Title: StylePrompter: All Styles Need Is Attention. (arXiv:2307.16151v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16151
• Code URL: https://github.com/i2-multimedia-lab/styleprompter
• Copy Paste: [[2307.16151] StylePrompter: All Styles Need Is Attention](http://arxiv.org/abs/2307.16151) #transformer
• Summary:

  GAN inversion aims at inverting given images into corresponding latent codes for Generative Adversarial Networks (GANs), especially StyleGAN where exists a disentangled latent space that allows attribute-based image manipulation at latent level. As most inversion methods build upon Convolutional Neural Networks (CNNs), we transfer a hierarchical vision Transformer backbone innovatively to predict $\mathcal{W^+}$ latent codes at token level. We further apply a Style-driven Multi-scale Adaptive Refinement Transformer (SMART) in $\mathcal{F}$ space to refine the intermediate style features of the generator. By treating style features as queries to retrieve lost identity information from the encoder's feature maps, SMART can not only produce high-quality inverted images but also surprisingly adapt to editing tasks. We then prove that StylePrompter lies in a more disentangled $\mathcal{W^+}$ and show the controllability of SMART. Finally, quantitative and qualitative experiments demonstrate that StylePrompter can achieve desirable performance in balancing reconstruction quality and editability, and is "smart" enough to fit into most edits, outperforming other $\mathcal{F}$-involved inversion methods.

Title: Around the GLOBE: Numerical Aggregation Question-Answering on Heterogeneous Genealogical Knowledge Graphs with Deep Neural Networks. (arXiv:2307.16208v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.16208
• Code URL: null
• Copy Paste: [[2307.16208] Around the GLOBE: Numerical Aggregation Question-Answering on Heterogeneous Genealogical Knowledge Graphs with Deep Neural Networks](http://arxiv.org/abs/2307.16208) #transformer
• Summary:

  One of the key AI tools for textual corpora exploration is natural language question-answering (QA). Unlike keyword-based search engines, QA algorithms receive and process natural language questions and produce precise answers to these questions, rather than long lists of documents that need to be manually scanned by the users. State-of-the-art QA algorithms based on DNNs were successfully employed in various domains. However, QA in the genealogical domain is still underexplored, while researchers in this field (and other fields in humanities and social sciences) can highly benefit from the ability to ask questions in natural language, receive concrete answers and gain insights hidden within large corpora. While some research has been recently conducted for factual QA in the genealogical domain, to the best of our knowledge, there is no previous research on the more challenging task of numerical aggregation QA (i.e., answering questions combining aggregation functions, e.g., count, average, max). Numerical aggregation QA is critical for distant reading and analysis for researchers (and the general public) interested in investigating cultural heritage domains. Therefore, in this study, we present a new end-to-end methodology for numerical aggregation QA for genealogical trees that includes: 1) an automatic method for training dataset generation; 2) a transformer-based table selection method, and 3) an optimized transformer-based numerical aggregation QA model. The findings indicate that the proposed architecture, GLOBE, outperforms the state-of-the-art models and pipelines by achieving 87% accuracy for this task compared to only 21% by current state-of-the-art models. This study may have practical implications for genealogical information centers and museums, making genealogical data research easy and scalable for experts as well as the general public.

generative

Title: What can Discriminator do? Towards Box-free Ownership Verification of Generative Adversarial Network. (arXiv:2307.15860v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.15860
• Code URL: null
• Copy Paste: [[2307.15860] What can Discriminator do? Towards Box-free Ownership Verification of Generative Adversarial Network](http://arxiv.org/abs/2307.15860) #generative
• Summary:

  In recent decades, Generative Adversarial Network (GAN) and its variants have achieved unprecedented success in image synthesis. However, well-trained GANs are under the threat of illegal steal or leakage. The prior studies on remote ownership verification assume a black-box setting where the defender can query the suspicious model with specific inputs, which we identify is not enough for generation tasks. To this end, in this paper, we propose a novel IP protection scheme for GANs where ownership verification can be done by checking outputs only, without choosing the inputs (i.e., box-free setting). Specifically, we make use of the unexploited potential of the discriminator to learn a hypersphere that captures the unique distribution learned by the paired generator. Extensive evaluations on two popular GAN tasks and more than 10 GAN architectures demonstrate our proposed scheme to effectively verify the ownership. Our proposed scheme shown to be immune to popular input-based removal attacks and robust against other existing attacks. The source code and models are available at https://github.com/AbstractTeen/gan_ownership_verification

Title: Fingerprints of Generative Models in the Frequency Domain. (arXiv:2307.15977v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.15977
• Code URL: null
• Copy Paste: [[2307.15977] Fingerprints of Generative Models in the Frequency Domain](http://arxiv.org/abs/2307.15977) #generative
• Summary:

  It is verified in existing works that CNN-based generative models leave unique fingerprints on generated images. There is a lack of analysis about how they are formed in generative models. Interpreting network components in the frequency domain, we derive sources for frequency distribution and grid-like pattern discrepancies exhibited on the spectrum. These insights are leveraged to develop low-cost synthetic models, which generate images emulating the frequency patterns observed in real generative models. The resulting fingerprint extractor pre-trained on synthetic data shows superior transferability in verifying, identifying, and analyzing the relationship of real CNN-based generative models such as GAN, VAE, Flow, and diffusion.

Title: SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension. (arXiv:2307.16125v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.16125
• Code URL: https://github.com/ailab-cvc/seed-bench
• Copy Paste: [[2307.16125] SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](http://arxiv.org/abs/2307.16125) #generative
• Summary:

  Based on powerful Large Language Models (LLMs), recent generative Multimodal Large Language Models (MLLMs) have gained prominence as a pivotal research area, exhibiting remarkable capability for both comprehension and generation. In this work, we address the evaluation of generative comprehension in MLLMs as a preliminary step towards a comprehensive assessment of generative models, by introducing a benchmark named SEED-Bench. SEED-Bench consists of 19K multiple choice questions with accurate human annotations (x 6 larger than existing benchmarks), which spans 12 evaluation dimensions including the comprehension of both the image and video modality. We develop an advanced pipeline for generating multiple-choice questions that target specific evaluation dimensions, integrating both automatic filtering and manual verification processes. Multiple-choice questions with groundtruth options derived from human annotation enables an objective and efficient assessment of model performance, eliminating the need for human or GPT intervention during evaluation. We further evaluate the performance of 18 models across all 12 dimensions, covering both the spatial and temporal understanding. By revealing the limitations of existing MLLMs through evaluation results, we aim for SEED-Bench to provide insights for motivating future research. We will launch and consistently maintain a leaderboard to provide a platform for the community to assess and investigate model capability.

Title: Stylized Projected GAN: A Novel Architecture for Fast and Realistic Image Generation. (arXiv:2307.16275v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16275
• Code URL: null
• Copy Paste: [[2307.16275] Stylized Projected GAN: A Novel Architecture for Fast and Realistic Image Generation](http://arxiv.org/abs/2307.16275) #generative
• Summary:

  Generative Adversarial Networks are used for generating the data using a generator and a discriminator, GANs usually produce high-quality images, but training GANs in an adversarial setting is a difficult task. GANs require high computation power and hyper-parameter regularization for converging. Projected GANs tackle the training difficulty of GANs by using transfer learning to project the generated and real samples into a pre-trained feature space. Projected GANs improve the training time and convergence but produce artifacts in the generated images which reduce the quality of the generated samples, we propose an optimized architecture called Stylized Projected GANs which integrates the mapping network of the Style GANs with Skip Layer Excitation of Fast GAN. The integrated modules are incorporated within the generator architecture of the Fast GAN to mitigate the problem of artifacts in the generated images.

Title: Improving Primary Healthcare Workflow Using Extreme Summarization of Scientific Literature Based on Generative AI. (arXiv:2307.15715v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.15715
• Code URL: null
• Copy Paste: [[2307.15715] Improving Primary Healthcare Workflow Using Extreme Summarization of Scientific Literature Based on Generative AI](http://arxiv.org/abs/2307.15715) #generative
• Summary:

  Primary care professionals struggle to keep up to date with the latest scientific literature critical in guiding evidence-based practice related to their daily work. To help solve the above-mentioned problem, we employed generative artificial intelligence techniques based on large-scale language models to summarize abstracts of scientific papers. Our objective is to investigate the potential of generative artificial intelligence in diminishing the cognitive load experienced by practitioners, thus exploring its ability to alleviate mental effort and burden. The study participants were provided with two use cases related to preventive care and behavior change, simulating a search for new scientific literature. The study included 113 university students from Slovenia and the United States randomized into three distinct study groups. The first group was assigned to the full abstracts. The second group was assigned to the short abstracts generated by AI. The third group had the option to select a full abstract in addition to the AI-generated short summary. Each use case study included ten retrieved abstracts. Our research demonstrates that the use of generative AI for literature review is efficient and effective. The time needed to answer questions related to the content of abstracts was significantly lower in groups two and three compared to the first group using full abstracts. The results, however, also show significantly lower accuracy in extracted knowledge in cases where full abstract was not available. Such a disruptive technology could significantly reduce the time required for healthcare professionals to keep up with the most recent scientific literature; nevertheless, further developments are needed to help them comprehend the knowledge accurately.

Title: SAFE: Saliency-Aware Counterfactual Explanations for DNN-based Automated Driving Systems. (arXiv:2307.15786v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.15786
• Code URL: null
• Copy Paste: [[2307.15786] SAFE: Saliency-Aware Counterfactual Explanations for DNN-based Automated Driving Systems](http://arxiv.org/abs/2307.15786) #generative
• Summary:

  A CF explainer identifies the minimum modifications in the input that would alter the model's output to its complement. In other words, a CF explainer computes the minimum modifications required to cross the model's decision boundary. Current deep generative CF models often work with user-selected features rather than focusing on the discriminative features of the black-box model. Consequently, such CF examples may not necessarily lie near the decision boundary, thereby contradicting the definition of CFs. To address this issue, we propose in this paper a novel approach that leverages saliency maps to generate more informative CF explanations. Source codes are available at: https://github.com/Amir-Samadi//Saliency_Aware_CF.

Title: Adaptive learning of density ratios in RKHS. (arXiv:2307.16164v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2307.16164
• Code URL: null
• Copy Paste: [[2307.16164] Adaptive learning of density ratios in RKHS](http://arxiv.org/abs/2307.16164) #generative
• Summary:

  Estimating the ratio of two probability densities from finitely many observations of the densities is a central problem in machine learning and statistics with applications in two-sample testing, divergence estimation, generative modeling, covariate shift adaptation, conditional density estimation, and novelty detection. In this work, we analyze a large class of density ratio estimation methods that minimize a regularized Bregman divergence between the true density ratio and a model in a reproducing kernel Hilbert space (RKHS). We derive new finite-sample error bounds, and we propose a Lepskii type parameter choice principle that minimizes the bounds without knowledge of the regularity of the density ratio. In the special case of quadratic loss, our method adaptively achieves a minimax optimal error rate. A numerical illustration is provided.

large language model

Title: Unified Model for Image, Video, Audio and Language Tasks. (arXiv:2307.16184v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16184
• Code URL: https://github.com/mshukor/unival
• Copy Paste: [[2307.16184] Unified Model for Image, Video, Audio and Language Tasks](http://arxiv.org/abs/2307.16184) #large language model
• Summary:

  Large Language Models (LLMs) have made the ambitious quest for generalist agents significantly far from being a fantasy. A key hurdle for building such general models is the diversity and heterogeneity of tasks and modalities. A promising solution is unification, allowing the support of a myriad of tasks and modalities within one unified framework. While few large models (e.g., Flamingo (Alayrac et al., 2022), trained on massive datasets, can support more than two modalities, current small to mid-scale unified models are still limited to 2 modalities, usually image-text or video-text. The question that we ask is: is it possible to build efficiently a unified model that can support all modalities? To answer this, we propose UnIVAL, a step further towards this ambitious goal. Without relying on fancy datasets sizes or models with billions of parameters, the ~ 0.25B parameter UnIVAL model goes beyond two modalities and unifies text, images, video, and audio into a single model. Our model is efficiently pretrained on many tasks, based on task balancing and multimodal curriculum learning. UnIVAL shows competitive performance to existing state-of-the-art approaches, across image and video-text tasks. The feature representations learned from image and video-text modalities, allows the model to achieve competitive performance when finetuned on audio-text tasks, despite not being pretrained on audio. Thanks to the unified model, we propose a novel study on multimodal model merging via weight interpolation of models trained on different multimodal tasks, showing their benefits in particular for out-of-distribution generalization. Finally, we motivate unification by showing the synergy between tasks. The model weights and code are released here: https://github.com/mshukor/UnIVAL.

Title: Utilizing Large Language Models for Natural Interface to Pharmacology Databases. (arXiv:2307.15717v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.15717
• Code URL: null
• Copy Paste: [[2307.15717] Utilizing Large Language Models for Natural Interface to Pharmacology Databases](http://arxiv.org/abs/2307.15717) #large language model
• Summary:

  The drug development process necessitates that pharmacologists undertake various tasks, such as reviewing literature, formulating hypotheses, designing experiments, and interpreting results. Each stage requires accessing and querying vast amounts of information. In this abstract, we introduce a Large Language Model (LLM)-based Natural Language Interface designed to interact with structured information stored in databases. Our experiments demonstrate the feasibility and effectiveness of the proposed framework. This framework can generalize to query a wide range of pharmaceutical data and knowledge bases.

Title: LLM-Rec: Personalized Recommendation via Prompting Large Language Models. (arXiv:2307.15780v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.15780
• Code URL: null
• Copy Paste: [[2307.15780] LLM-Rec: Personalized Recommendation via Prompting Large Language Models](http://arxiv.org/abs/2307.15780) #large language model
• Summary:

  We investigate various prompting strategies for enhancing personalized content recommendation performance with large language models (LLMs) through input augmentation. Our proposed approach, termed LLM-Rec, encompasses four distinct prompting strategies: (1) basic prompting, (2) recommendation-driven prompting, (3) engagement-guided prompting, and (4) recommendation-driven + engagement-guided prompting. Our empirical experiments show that combining the original content description with the augmented input text generated by LLM using these prompting strategies leads to improved recommendation performance. This finding highlights the importance of incorporating diverse prompts and input augmentation techniques to enhance the recommendation capabilities with large language models for personalized content recommendation.

Title: Dialogue Shaping: Empowering Agents through NPC Interaction. (arXiv:2307.15833v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.15833
• Code URL: null
• Copy Paste: [[2307.15833] Dialogue Shaping: Empowering Agents through NPC Interaction](http://arxiv.org/abs/2307.15833) #large language model
• Summary:

  One major challenge in reinforcement learning (RL) is the large amount of steps for the RL agent needs to converge in the training process and learn the optimal policy, especially in text-based game environments where the action space is extensive. However, non-player characters (NPCs) sometimes hold some key information about the game, which can potentially help to train RL agents faster. Thus, this paper explores how to interact and converse with NPC agents to get the key information using large language models (LLMs), as well as incorporate this information to speed up RL agent's training using knowledge graphs (KGs) and Story Shaping.

Title: RoCar: A Relationship Network-based Evaluation Method to Large Language Models. (arXiv:2307.15997v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.15997
• Code URL: null
• Copy Paste: [[2307.15997] RoCar: A Relationship Network-based Evaluation Method to Large Language Models](http://arxiv.org/abs/2307.15997) #large language model
• Summary:

  Large language models (LLMs) have received increasing attention. However, due to the complexity of its capabilities, how to rationally evaluate the capabilities of LLMs is still a task to be solved. We propose the RoCar method, which utilizes the defined basic schemas to randomly construct a task graph and generates natural language evaluation tasks based on the task graph to evaluate the reasoning and memory abilities of LLMs respectively. Due to the very large randomness of the task construction process, it is possible to ensure that none of the LLMs to be tested has directly learned the evaluation tasks, guaranteeing the fairness of the evaluation method.

Title: Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback. (arXiv:2307.16039v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.16039
• Code URL: null
• Copy Paste: [[2307.16039] Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](http://arxiv.org/abs/2307.16039) #large language model
• Summary:

  A key technology for the development of large language models (LLMs) involves instruction tuning that helps align the models' responses with human expectations to realize impressive learning abilities. Two major approaches for instruction tuning characterize supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), which are currently applied to produce the best commercial LLMs (e.g., ChatGPT). To improve the accessibility of LLMs for research and development efforts, various instruction-tuned open-source LLMs have also been introduced recently, e.g., Alpaca, Vicuna, to name a few. However, existing open-source LLMs have only been instruction-tuned for English and a few popular languages, thus hindering their impacts and accessibility to many other languages in the world. Among a few very recent work to explore instruction tuning for LLMs in multiple languages, SFT has been used as the only approach to instruction-tune LLMs for multiple languages. This has left a significant gap for fine-tuned LLMs based on RLHF in diverse languages and raised important questions on how RLHF can boost the performance of multilingual instruction tuning. To overcome this issue, we present Okapi, the first system with instruction-tuned LLMs based on RLHF for multiple languages. Okapi introduces instruction and response-ranked data in 26 diverse languages to facilitate the experiments and development of future multilingual LLM research. We also present benchmark datasets to enable the evaluation of generative LLMs in multiple languages. Our experiments demonstrate the advantages of RLHF for multilingual instruction over SFT for different base models and datasets. Our framework and resources are released at \url{https://github.com/nlp-uoregon/Okapi}.

Title: User-Controlled Knowledge Fusion in Large Language Models: Balancing Creativity and Hallucination. (arXiv:2307.16139v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.16139
• Code URL: null
• Copy Paste: [[2307.16139] User-Controlled Knowledge Fusion in Large Language Models: Balancing Creativity and Hallucination](http://arxiv.org/abs/2307.16139) #large language model
• Summary:

  In modern dialogue systems, the use of Large Language Models (LLMs) has grown exponentially due to their capacity to generate diverse, relevant, and creative responses. Despite their strengths, striking a balance between the LLMs' creativity and their faithfulness to external knowledge remains a key challenge. This paper presents an innovative user-controllable mechanism that modulates the balance between an LLM's imaginative capabilities and its adherence to factual information. Our approach incorporates a numerical tag during the fine-tuning phase of the LLM's training, representing the degree of faithfulness to the reference knowledge in the generated responses. This degree is computed through an automated process that measures lexical overlap using ROUGE scores, semantic similarity using Sentence-BERT embeddings, and an LLM's self-evaluation score. During model inference, users can manipulate this numerical tag, thus controlling the degree of the LLM's reliance on external knowledge. We conduct extensive experiments across various scenarios, demonstrating the adaptability of our method and its efficacy in ensuring the quality and accuracy of the LLM's responses. The results highlight the potential of our approach to enhance the versatility of LLMs while maintaining a balance between creativity and hallucination.

Title: Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models. (arXiv:2307.16180v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.16180
• Code URL: https://github.com/harderthenharder/transformers_tasks
• Copy Paste: [[2307.16180] Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models](http://arxiv.org/abs/2307.16180) #large language model
• Summary:

  The field of large language models (LLMs) has made significant progress, and their knowledge storage capacity is approaching that of human beings. Furthermore, advanced techniques, such as prompt learning and reinforcement learning, are being employed to address ethical concerns and hallucination problems associated with LLMs, bringing them closer to aligning with human values. This situation naturally raises the question of whether LLMs with human-like abilities possess a human-like personality? In this paper, we aim to investigate the feasibility of using the Myers-Briggs Type Indicator (MBTI), a widespread human personality assessment tool, as an evaluation metric for LLMs. Specifically, extensive experiments will be conducted to explore: 1) the personality types of different LLMs, 2) the possibility of changing the personality types by prompt engineering, and 3) How does the training dataset affect the model's personality. Although the MBTI is not a rigorous assessment, it can still reflect the similarity between LLMs and human personality. In practice, the MBTI has the potential to serve as a rough indicator. Our codes are available at https://github.com/HarderThenHarder/transformers_tasks/tree/main/LLM/llms_mbti.

Title: Distractor generation for multiple-choice questions with predictive prompting and large language models. (arXiv:2307.16338v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.16338
• Code URL: https://github.com/semerekiros/distractgpt
• Copy Paste: [[2307.16338] Distractor generation for multiple-choice questions with predictive prompting and large language models](http://arxiv.org/abs/2307.16338) #large language model
• Summary:

  Large Language Models (LLMs) such as ChatGPT have demonstrated remarkable performance across various tasks and have garnered significant attention from both researchers and practitioners. However, in an educational context, we still observe a performance gap in generating distractors -- i.e., plausible yet incorrect answers -- with LLMs for multiple-choice questions (MCQs). In this study, we propose a strategy for guiding LLMs such as ChatGPT, in generating relevant distractors by prompting them with question items automatically retrieved from a question bank as well-chosen in-context examples. We evaluate our LLM-based solutions using a quantitative assessment on an existing test set, as well as through quality annotations by human experts, i.e., teachers. We found that on average 53% of the generated distractors presented to the teachers were rated as high-quality, i.e., suitable for immediate use as is, outperforming the state-of-the-art model. We also show the gains of our approach 1 in generating high-quality distractors by comparing it with a zero-shot ChatGPT and a few-shot ChatGPT prompted with static examples.

segmentation

Title: CMDA: Cross-Modality Domain Adaptation for Nighttime Semantic Segmentation. (arXiv:2307.15942v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.15942
• Code URL: https://github.com/xiarho/cmda
• Copy Paste: [[2307.15942] CMDA: Cross-Modality Domain Adaptation for Nighttime Semantic Segmentation](http://arxiv.org/abs/2307.15942) #segmentation
• Summary:

  Most nighttime semantic segmentation studies are based on domain adaptation approaches and image input. However, limited by the low dynamic range of conventional cameras, images fail to capture structural details and boundary information in low-light conditions. Event cameras, as a new form of vision sensors, are complementary to conventional cameras with their high dynamic range. To this end, we propose a novel unsupervised Cross-Modality Domain Adaptation (CMDA) framework to leverage multi-modality (Images and Events) information for nighttime semantic segmentation, with only labels on daytime images. In CMDA, we design the Image Motion-Extractor to extract motion information and the Image Content-Extractor to extract content information from images, in order to bridge the gap between different modalities (Images to Events) and domains (Day to Night). Besides, we introduce the first image-event nighttime semantic segmentation dataset. Extensive experiments on both the public image dataset and the proposed image-event dataset demonstrate the effectiveness of our proposed approach. We open-source our code, models, and dataset at https://github.com/XiaRho/CMDA.

Title: XMem++: Production-level Video Segmentation From Few Annotated Frames. (arXiv:2307.15958v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.15958
• Code URL: null
• Copy Paste: [[2307.15958] XMem++: Production-level Video Segmentation From Few Annotated Frames](http://arxiv.org/abs/2307.15958) #segmentation
• Summary:

  Despite advancements in user-guided video segmentation, extracting complex objects consistently for highly complex scenes is still a labor-intensive task, especially for production. It is not uncommon that a majority of frames need to be annotated. We introduce a novel semi-supervised video object segmentation (SSVOS) model, XMem++, that improves existing memory-based models, with a permanent memory module. Most existing methods focus on single frame annotations, while our approach can effectively handle multiple user-selected frames with varying appearances of the same object or region. Our method can extract highly consistent results while keeping the required number of frame annotations low. We further introduce an iterative and attention-based frame suggestion mechanism, which computes the next best frame for annotation. Our method is real-time and does not require retraining after each user input. We also introduce a new dataset, PUMaVOS, which covers new challenging use cases not found in previous benchmarks. We demonstrate SOTA performance on challenging (partial and multi-class) segmentation scenarios as well as long videos, while ensuring significantly fewer frame annotations than any existing method.

Title: PD-SEG: Population Disaggregation Using Deep Segmentation Networks For Improved Built Settlement Mask. (arXiv:2307.16084v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16084
• Code URL: null
• Copy Paste: [[2307.16084] PD-SEG: Population Disaggregation Using Deep Segmentation Networks For Improved Built Settlement Mask](http://arxiv.org/abs/2307.16084) #segmentation
• Summary:

  Any policy-level decision-making procedure and academic research involving the optimum use of resources for development and planning initiatives depends on accurate population density statistics. The current cutting-edge datasets offered by WorldPop and Meta do not succeed in achieving this aim for developing nations like Pakistan; the inputs to their algorithms provide flawed estimates that fail to capture the spatial and land-use dynamics. In order to precisely estimate population counts at a resolution of 30 meters by 30 meters, we use an accurate built settlement mask obtained using deep segmentation networks and satellite imagery. The Points of Interest (POI) data is also used to exclude non-residential areas.

Title: ScribbleVC: Scribble-supervised Medical Image Segmentation with Vision-Class Embedding. (arXiv:2307.16226v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16226
• Code URL: https://github.com/huanglizi/scribblevc
• Copy Paste: [[2307.16226] ScribbleVC: Scribble-supervised Medical Image Segmentation with Vision-Class Embedding](http://arxiv.org/abs/2307.16226) #segmentation
• Summary:

  Medical image segmentation plays a critical role in clinical decision-making, treatment planning, and disease monitoring. However, accurate segmentation of medical images is challenging due to several factors, such as the lack of high-quality annotation, imaging noise, and anatomical differences across patients. In addition, there is still a considerable gap in performance between the existing label-efficient methods and fully-supervised methods. To address the above challenges, we propose ScribbleVC, a novel framework for scribble-supervised medical image segmentation that leverages vision and class embeddings via the multimodal information enhancement mechanism. In addition, ScribbleVC uniformly utilizes the CNN features and Transformer features to achieve better visual feature extraction. The proposed method combines a scribble-based approach with a segmentation network and a class-embedding module to produce accurate segmentation masks. We evaluate ScribbleVC on three benchmark datasets and compare it with state-of-the-art methods. The experimental results demonstrate that our method outperforms existing approaches in terms of accuracy, robustness, and efficiency. The datasets and code are released on GitHub.

Title: 3D Medical Image Segmentation with Sparse Annotation via Cross-Teaching between 3D and 2D Networks. (arXiv:2307.16256v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2307.16256
• Code URL: null
• Copy Paste: [[2307.16256] 3D Medical Image Segmentation with Sparse Annotation via Cross-Teaching between 3D and 2D Networks](http://arxiv.org/abs/2307.16256) #segmentation
• Summary:

  Medical image segmentation typically necessitates a large and precisely annotated dataset. However, obtaining pixel-wise annotation is a labor-intensive task that requires significant effort from domain experts, making it challenging to obtain in practical clinical scenarios. In such situations, reducing the amount of annotation required is a more practical approach. One feasible direction is sparse annotation, which involves annotating only a few slices, and has several advantages over traditional weak annotation methods such as bounding boxes and scribbles, as it preserves exact boundaries. However, learning from sparse annotation is challenging due to the scarcity of supervision signals. To address this issue, we propose a framework that can robustly learn from sparse annotation using the cross-teaching of both 3D and 2D networks. Considering the characteristic of these networks, we develop two pseudo label selection strategies, which are hard-soft confidence threshold and consistent label fusion. Our experimental results on the MMWHS dataset demonstrate that our method outperforms the state-of-the-art (SOTA) semi-supervised segmentation methods. Moreover, our approach achieves results that are comparable to the fully-supervised upper bound result.

Title: Improving TTS for Shanghainese: Addressing Tone Sandhi via Word Segmentation. (arXiv:2307.16199v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2307.16199
• Code URL: https://github.com/edward-martyr/shanghainese-tts
• Copy Paste: [[2307.16199] Improving TTS for Shanghainese: Addressing Tone Sandhi via Word Segmentation](http://arxiv.org/abs/2307.16199) #segmentation
• Summary:

  Tone is a crucial component of the prosody of Shanghainese, a Wu Chinese variety spoken primarily in urban Shanghai. Tone sandhi, which applies to all multi-syllabic words in Shanghainese, then, is key to natural-sounding speech. Unfortunately, recent work on Shanghainese TTS (text-to-speech) such as Apple's VoiceOver has shown poor performance with tone sandhi, especially LD (left-dominant sandhi). Here I show that word segmentation during text preprocessing can improve the quality of tone sandhi production in TTS models. Syllables within the same word are annotated with a special symbol, which serves as a proxy for prosodic information of the domain of LD. Contrary to the common practice of using prosodic annotation mainly for static pauses, this paper demonstrates that prosodic annotation can also be applied to dynamic tonal phenomena. I anticipate this project to be a starting point for bringing formal linguistic accounts of Shanghainese into computational projects. Too long have we been using the Mandarin models to approximate Shanghainese, but it is a different language with its own linguistic features, and its digitisation and revitalisation should be treated as such.