secure

security

Title: Vulnerability Clustering and other Machine Learning Applications of Semantic Vulnerability Embeddings. (arXiv:2310.05935v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.05935
• Code URL: null
• Copy Paste: [[2310.05935]] Vulnerability Clustering and other Machine Learning Applications of Semantic Vulnerability Embeddings(http://arxiv.org/abs/2310.05935)
• Summary:

  Cyber-security vulnerabilities are usually published in form of short natural language descriptions (e.g., in form of MITRE's CVE list) that over time are further manually enriched with labels such as those defined by the Common Vulnerability Scoring System (CVSS). In the Vulnerability AI (Analytics and Intelligence) project, we investigated different types of semantic vulnerability embeddings based on natural language processing (NLP) techniques to obtain a concise representation of the vulnerability space. We also evaluated their use as a foundation for machine learning applications that can support cyber-security researchers and analysts in risk assessment and other related activities. The particular applications we explored and briefly summarize in this report are clustering, classification, and visualization, as well as a new logic-based approach to evaluate theories about the vulnerability space.

Title: LLM for SoC Security: A Paradigm Shift. (arXiv:2310.06046v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06046
• Code URL: null
• Copy Paste: [[2310.06046]] LLM for SoC Security: A Paradigm Shift(http://arxiv.org/abs/2310.06046)
• Summary:

  As the ubiquity and complexity of system-on-chip (SoC) designs increase across electronic devices, the task of incorporating security into an SoC design flow poses significant challenges. Existing security solutions are inadequate to provide effective verification of modern SoC designs due to their limitations in scalability, comprehensiveness, and adaptability. On the other hand, Large Language Models (LLMs) are celebrated for their remarkable success in natural language understanding, advanced reasoning, and program synthesis tasks. Recognizing an opportunity, our research delves into leveraging the emergent capabilities of Generative Pre-trained Transformers (GPTs) to address the existing gaps in SoC security, aiming for a more efficient, scalable, and adaptable methodology. By integrating LLMs into the SoC security verification paradigm, we open a new frontier of possibilities and challenges to ensure the security of increasingly complex SoCs. This paper offers an in-depth analysis of existing works, showcases practical case studies, demonstrates comprehensive experiments, and provides useful promoting guidelines. We also present the achievements, prospects, and challenges of employing LLM in different SoC security verification tasks.

Title: Dynamic S-BOX using Chaotic Map for VPN Data Security. (arXiv:2310.05940v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.05940
• Code URL: null
• Copy Paste: [[2310.05940]] Dynamic S-BOX using Chaotic Map for VPN Data Security(http://arxiv.org/abs/2310.05940)
• Summary:

  A dynamic SBox using a chaotic map is a cryptography technique that changes the SBox during encryption based on iterations of a chaotic map, adding an extra layer of confusion and security to symmetric encryption algorithms like AES. The chaotic map introduces unpredictability, non-linearity, and key dependency, enhancing the overall security of the encryption process. The existing work on dynamic SBox using chaotic maps lacks standardized guidelines and extensive security analysis, leaving potential vulnerabilities and performance concerns unaddressed. Key management and the sensitivity of chaotic maps to initial conditions are challenges that need careful consideration. The main objective of using a dynamic SBox with a chaotic map in cryptography systems is to enhance the security and robustness of symmetric encryption algorithms. The method of dynamic SBox using a chaotic map involves initializing the SBox, selecting a chaotic map, iterating the map to generate chaotic values, and updating the SBox based on these values during the encryption process to enhance security and resist cryptanalytic attacks. This article proposes a novel chaotic map that can be utilized to create a fresh, lively SBox. The performance assessment of the suggested S resilience Box against various attacks involves metrics such as nonlinearity (NL), strict avalanche criterion (SAC), bit independence criterion (BIC), linear approximation probability (LP), and differential approximation probability (DP). These metrics help gauge the Box ability to handle and respond to different attack scenarios. Assess the cryptography strength of the proposed S-Box for usage in practical security applications, it is compared to other recently developed SBoxes. The comparative research shows that the suggested SBox has the potential to be an important advancement in the field of data security.

Title: SCAR: Power Side-Channel Analysis at RTL-Level. (arXiv:2310.06257v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06257
• Code URL: null
• Copy Paste: [[2310.06257]] SCAR: Power Side-Channel Analysis at RTL-Level(http://arxiv.org/abs/2310.06257)
• Summary:

  Power side-channel attacks exploit the dynamic power consumption of cryptographic operations to leak sensitive information of encryption hardware. Therefore, it is necessary to conduct power side-channel analysis for assessing the susceptibility of cryptographic systems and mitigating potential risks. Existing power side-channel analysis primarily focuses on post-silicon implementations, which are inflexible in addressing design flaws, leading to costly and time-consuming post-fabrication design re-spins. Hence, pre-silicon power side-channel analysis is required for early detection of vulnerabilities to improve design robustness. In this paper, we introduce SCAR, a novel pre-silicon power side-channel analysis framework based on Graph Neural Networks (GNN). SCAR converts register-transfer level (RTL) designs of encryption hardware into control-data flow graphs and use that to detect the design modules susceptible to side-channel leakage. Furthermore, we incorporate a deep learning-based explainer in SCAR to generate quantifiable and human-accessible explanation of our detection and localization decisions. We have also developed a fortification component as a part of SCAR that uses large-language models (LLM) to automatically generate and insert additional design code at the localized zone to shore up the side-channel leakage. When evaluated on popular encryption algorithms like AES, RSA, and PRESENT, and postquantum cryptography algorithms like Saber and CRYSTALS-Kyber, SCAR, achieves up to 94.49% localization accuracy, 100% precision, and 90.48% recall. Additionally, through explainability analysis, SCAR reduces features for GNN model training by 57% while maintaining comparable accuracy. We believe that SCAR will transform the security-critical hardware design cycle, resulting in faster design closure at a reduced design cost.

Title: NetShaper: A Differentially Private Network Side-Channel Mitigation System. (arXiv:2310.06293v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06293
• Code URL: null
• Copy Paste: [[2310.06293]] NetShaper: A Differentially Private Network Side-Channel Mitigation System(http://arxiv.org/abs/2310.06293)
• Summary:

  The widespread adoption of encryption in network protocols has significantly improved the overall security of many Internet applications. However, these protocols cannot prevent network side-channel leaks -- leaks of sensitive information through the sizes and timing of network packets. We present NetShaper, a system that mitigates such leaks based on the principle of traffic shaping. NetShaper's traffic shaping provides differential privacy guarantees while adapting to the prevailing workload and congestion condition, and allows configuring a tradeoff between privacy guarantees, bandwidth and latency overheads. Furthermore, NetShaper provides a modular and portable tunnel endpoint design that can support diverse applications. We present a middlebox-based implementation of NetShaper and demonstrate its applicability in a video streaming and a web service application.

Title: Toward a Reference Architecture for Software Supply Chain Metadata Management. (arXiv:2310.06300v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06300
• Code URL: null
• Copy Paste: [[2310.06300]] Toward a Reference Architecture for Software Supply Chain Metadata Management(http://arxiv.org/abs/2310.06300)
• Summary:

  An Software Supply Chain (SSC) attack combines an upstream attack, where malicious codes are injected into a software artefact via a compromised life cycle activity, and a downstream attack on the consumers who use the compromised artefact. Organisations need thorough and trustworthy visibility over the entire SSC of their software inventory to detect risks early and rapidly identify compromised assets in the event of an SSC attack. One way to achieve such visibility is through SSC metadata, machine-readable and authenticated documents describing an artefact's lifecycle, such as how it was constructed and the utilised ``ingredients''. Adopting SSC metadata requires organisations to procure or develop a Software Supply Chain Metadata Management system (SCM2), a suite of software tools for performing life cycle activities of SSC metadata documents such as creation, signing, distribution, and consumption. Selecting or developing an SCM2 is challenging due to the lack of a comprehensive domain model and architectural blueprint to aid practitioners in navigating the vast design space of SSC metadata terminologies, frameworks, and solutions. This paper addresses the above-mentioned challenge with a Systematisation of Knowledge about SSC metadata and SCM2, presented as a Reference Architecture (RA). The RA comprises a domain model and an architectural blueprint for SCM2 systems, constructed from the concepts and building blocks scattered across existing SSC security frameworks and standards. Our evaluation shows that the RA framework is effective for analysing existing SCM2 solutions and guiding the engineering of new SCM2.

Title: Better Safe than Sorry: Recovering after Adversarial Majority. (arXiv:2310.06338v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06338
• Code URL: null
• Copy Paste: [[2310.06338]] Better Safe than Sorry: Recovering after Adversarial Majority(http://arxiv.org/abs/2310.06338)
• Summary:

  The security of blockchain protocols is a combination of two properties: safety and liveness. It is well known that no blockchain protocol can provide both to sleepy (intermittently online) clients under adversarial majority. However, safety is more critical in that a single safety violation can cause users to lose money. At the same time, liveness must not be lost forever. We show that, in a synchronous network, it is possible to maintain safety for all clients even during adversarial majority, and recover liveness after honest majority is restored. Our solution takes the form of a recovery gadget that can be applied to any protocol with certificates (such as HotStuff, Streamlet, Tendermint, and their variants).

privacy

Title: Technocracy, pseudoscience and performative compliance: the risks of privacy risk assessments. Lessons from NIST's Privacy Risk Assessment Methodology. (arXiv:2310.05936v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.05936
• Code URL: null
• Copy Paste: [[2310.05936]] Technocracy, pseudoscience and performative compliance: the risks of privacy risk assessments(http://arxiv.org/abs/2310.05936)
• Summary:

  Privacy risk assessments have been touted as an objective, principled way to encourage organizations to implement privacy-by-design. They are central to a new regulatory model of collaborative governance, as embodied by the GDPR. However, existing guidelines and methods remain vague, and there is little empirical evidence on privacy harms. In this paper we conduct a close analysis of US NIST's Privacy Risk Assessment Methodology, highlighting multiple sites of discretion that create countless opportunities for adversarial organizations to engage in performative compliance. Our analysis shows that the premises on which the success of privacy risk assessments depends do not hold, particularly in regard to organizations' incentives and regulators auditing capabilities. We highlight the limitations and pitfalls of what is essentially a utilitarian and technocratic approach, leading us to discuss alternatives and a realignment of our policy and research objectives.

Title: Efficient Network Representation for GNN-based Intrusion Detection. (arXiv:2310.05956v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.05956
• Code URL: null
• Copy Paste: [[2310.05956]] Efficient Network Representation for GNN-based Intrusion Detection(http://arxiv.org/abs/2310.05956)
• Summary:

  The last decades have seen a growth in the number of cyber-attacks with severe economic and privacy damages, which reveals the need for network intrusion detection approaches to assist in preventing cyber-attacks and reducing their risks. In this work, we propose a novel network representation as a graph of flows that aims to provide relevant topological information for the intrusion detection task, such as malicious behavior patterns, the relation between phases of multi-step attacks, and the relation between spoofed and pre-spoofed attackers activities. In addition, we present a Graph Neural Network (GNN) based framework responsible for exploiting the proposed graph structure to classify communication flows by assigning them a maliciousness score. The framework comprises three main steps that aim to embed nodes features and learn relevant attack patterns from the network representation. Finally, we highlight a potential data leakage issue with classical evaluation procedures and suggest a solution to ensure a reliable validation of intrusion detection systems performance. We implement the proposed framework and prove that exploiting the flow-based graph structure outperforms the classical machine learning-based and the previous GNN-based solutions.

Title: Differentially Private Multi-Site Treatment Effect Estimation. (arXiv:2310.06237v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06237
• Code URL: null
• Copy Paste: [[2310.06237]] Differentially Private Multi-Site Treatment Effect Estimation(http://arxiv.org/abs/2310.06237)
• Summary:

  Patient privacy is a major barrier to healthcare AI. For confidentiality reasons, most patient data remains in silo in separate hospitals, preventing the design of data-driven healthcare AI systems that need large volumes of patient data to make effective decisions. A solution to this is collective learning across multiple sites through federated learning with differential privacy. However, literature in this space typically focuses on differentially private statistical estimation and machine learning, which is different from the causal inference-related problems that arise in healthcare. In this work, we take a fresh look at federated learning with a focus on causal inference; specifically, we look at estimating the average treatment effect (ATE), an important task in causal inference for healthcare applications, and provide a federated analytics approach to enable ATE estimation across multiple sites along with differential privacy (DP) guarantees at each site. The main challenge comes from site heterogeneity -- different sites have different sample sizes and privacy budgets. We address this through a class of per-site estimation algorithms that reports the ATE estimate and its variance as a quality measure, and an aggregation algorithm on the server side that minimizes the overall variance of the final ATE estimate. Our experiments on real and synthetic data show that our method reliably aggregates private statistics across sites and provides better privacy-utility tradeoff under site heterogeneity than baselines.

Title: Using Participants' Utility Functions to Compare Versions of Differential Privacy. (arXiv:2310.06258v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06258
• Code URL: null
• Copy Paste: [[2310.06258]] Using Participants' Utility Functions to Compare Versions of Differential Privacy(http://arxiv.org/abs/2310.06258)
• Summary:

  We use decision theory to compare variants of differential privacy from the perspective of prospective study participants. We posit the existence of a preference ordering on the set of potential consequences that study participants can incur, which enables the analysis of individual utility functions. Drawing upon the theory of measurement, we argue that changes in expected utilities should be measured via the classic Euclidean metric. We then consider the question of which privacy guarantees would be more appealing for individuals under different decision settings. Through our analysis, we found that the nature of the potential participant's utility function, along with the specific values of $\epsilon$ and $\delta$, can greatly alter which privacy guarantees are preferable.

Title: Partition-based differentially private synthetic data generation. (arXiv:2310.06371v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06371
• Code URL: null
• Copy Paste: [[2310.06371]] Partition-based differentially private synthetic data generation(http://arxiv.org/abs/2310.06371)
• Summary:

  Private synthetic data sharing is preferred as it keeps the distribution and nuances of original data compared to summary statistics. The state-of-the-art methods adopt a select-measure-generate paradigm, but measuring large domain marginals still results in much error and allocating privacy budget iteratively is still difficult. To address these issues, our method employs a partition-based approach that effectively reduces errors and improves the quality of synthetic data, even with a limited privacy budget. Results from our experiments demonstrate the superiority of our method over existing approaches. The synthetic data produced using our approach exhibits improved quality and utility, making it a preferable choice for private synthetic data sharing.

protect

Title: Top of the Heap: Efficient Memory Error Protection for Many Heap Objects. (arXiv:2310.06397v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06397
• Code URL: null
• Copy Paste: [[2310.06397]] Top of the Heap: Efficient Memory Error Protection for Many Heap Objects(http://arxiv.org/abs/2310.06397)
• Summary:

  Exploits against heap memory errors continue to be a major concern. Although many defenses have been proposed, heap data are not protected from attacks that exploit memory errors systematically. Research defenses focus on complete coverage of heap objects, often giving up on comprehensive memory safety protection and/or incurring high costs in performance overhead and memory usage. In this paper, we propose a solution for heap memory safety enforcement that aims to provide comprehensive protection from memory errors efficiently by protecting those heap objects whose accesses are provably safe from memory errors. Specifically, we present the Uriah system that statically validates spatial and type memory safety for heap objects, isolating compliant objects on a safe heap that enforces temporal type safety to prevent attacks on memory reuse. Using Uriah, 71.9% of heap allocation sites can be shown to produce objects (73% of allocations are found safe) that satisfy spatial and type safety, which are then isolated using Uriah's heap allocator from memory accesses via unsafe heap objects. Uriah only incurs 2.9% overhead and only uses 9.3% more memory on SPEC CPU2006 (C/C++) benchmarks, showing that many heap objects can be protected from all classes of memory errors efficiently.

Title: DASICS: Enhancing Memory Protection with Dynamic Compartmentalization. (arXiv:2310.06435v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06435
• Code URL: null
• Copy Paste: [[2310.06435]] DASICS: Enhancing Memory Protection with Dynamic Compartmentalization(http://arxiv.org/abs/2310.06435)
• Summary:

  In the existing software development ecosystem, security issues introduced by third-party code cannot be overlooked. Among these security concerns, memory access vulnerabilities stand out prominently, leading to risks such as the theft or tampering of sensitive data. To address this issue, software-based defense mechanisms have been established at the programming language, compiler, and operating system levels. However, as a trade-off, these mechanisms significantly reduce software execution efficiency. Hardware-software co-design approaches have sought to either construct entirely isolated trusted execution environments or attempt to partition security domains within the same address space. While such approaches enhance efficiency compared to pure software methods, they also encounter challenges related to granularity of protection, performance overhead, and portability. In response to these challenges, we present the DASICS (Dynamic in-Address-Space Isolation by Code Segments) secure processor design, which offers dynamic and flexible security protection across multiple privilege levels, addressing data flow protection, control flow protection, and secure system calls. We have implemented hardware FPGA prototypes and software QEMU simulator prototypes based on DASICS, along with necessary modifications to system software for adaptability. We illustrate the protective mechanisms and effectiveness of DASICS with two practical examples and provide potential real-world use cases where DASICS could be applied.

defense

Title: Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations. (arXiv:2310.06387v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06387
• Code URL: null
• Copy Paste: [[2310.06387]] Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations(http://arxiv.org/abs/2310.06387)
• Summary:

  Large Language Models (LLMs) have shown remarkable success in various tasks, but concerns about their safety and the potential for generating malicious content have emerged. In this paper, we explore the power of In-Context Learning (ICL) in manipulating the alignment ability of LLMs. We find that by providing just few in-context demonstrations without fine-tuning, LLMs can be manipulated to increase or decrease the probability of jailbreaking, i.e. answering malicious prompts. Based on these observations, we propose In-Context Attack (ICA) and In-Context Defense (ICD) methods for jailbreaking and guarding aligned language model purposes. ICA crafts malicious contexts to guide models in generating harmful outputs, while ICD enhances model robustness by demonstrations of rejecting to answer harmful prompts. Our experiments show the effectiveness of ICA and ICD in increasing or reducing the success rate of adversarial jailbreaking attacks. Overall, we shed light on the potential of ICL to influence LLM behavior and provide a new perspective for enhancing the safety and alignment of LLMs.

attack

Title: Robust and Efficient Interference Neural Networks for Defending Against Adversarial Attacks in ImageNet. (arXiv:2310.05947v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.05947
• Code URL: null
• Copy Paste: [[2310.05947]] Robust and Efficient Interference Neural Networks for Defending Against Adversarial Attacks in ImageNet(http://arxiv.org/abs/2310.05947)
• Summary:

  The existence of adversarial images has seriously affected the task of image recognition and practical application of deep learning, it is also a key scientific problem that deep learning urgently needs to solve. By far the most effective approach is to train the neural network with a large number of adversarial examples. However, this adversarial training method requires a huge amount of computing resources when applied to ImageNet, and has not yet achieved satisfactory results for high-intensity adversarial attacks. In this paper, we construct an interference neural network by applying additional background images and corresponding labels, and use pre-trained ResNet-152 to efficiently complete the training. Compared with the state-of-the-art results under the PGD attack, it has a better defense effect with much smaller computing resources. This work provides new ideas for academic research and practical applications of effective defense against adversarial attacks.

Title: Fingerprint Attack: Client De-Anonymization in Federated Learning. (arXiv:2310.05960v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.05960
• Code URL: null
• Copy Paste: [[2310.05960]] Fingerprint Attack: Client De-Anonymization in Federated Learning(http://arxiv.org/abs/2310.05960)
• Summary:

  Federated Learning allows collaborative training without data sharing in settings where participants do not trust the central server and one another. Privacy can be further improved by ensuring that communication between the participants and the server is anonymized through a shuffle; decoupling the participant identity from their data. This paper seeks to examine whether such a defense is adequate to guarantee anonymity, by proposing a novel fingerprinting attack over gradients sent by the participants to the server. We show that clustering of gradients can easily break the anonymization in an empirical study of learning federated language models on two language corpora. We then show that training with differential privacy can provide a practical defense against our fingerprint attack.

Title: Learning Cyber Defence Tactics from Scratch with Multi-Agent Reinforcement Learning. (arXiv:2310.05939v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.05939
• Code URL: null
• Copy Paste: [[2310.05939]] Learning Cyber Defence Tactics from Scratch with Multi-Agent Reinforcement Learning(http://arxiv.org/abs/2310.05939)
• Summary:

  Recent advancements in deep learning techniques have opened new possibilities for designing solutions for autonomous cyber defence. Teams of intelligent agents in computer network defence roles may reveal promising avenues to safeguard cyber and kinetic assets. In a simulated game environment, agents are evaluated on their ability to jointly mitigate attacker activity in host-based defence scenarios. Defender systems are evaluated against heuristic attackers with the goals of compromising network confidentiality, integrity, and availability. Value-based Independent Learning and Centralized Training Decentralized Execution (CTDE) cooperative Multi-Agent Reinforcement Learning (MARL) methods are compared revealing that both approaches outperform a simple multi-agent heuristic defender. This work demonstrates the ability of cooperative MARL to learn effective cyber defence tactics against varied threats.

Title: Mitigating Denial of Service Attacks in Fog-Based Wireless Sensor Networks Using Machine Learning Techniques. (arXiv:2310.05952v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.05952
• Code URL: null
• Copy Paste: [[2310.05952]] Mitigating Denial of Service Attacks in Fog-Based Wireless Sensor Networks Using Machine Learning Techniques(http://arxiv.org/abs/2310.05952)
• Summary:

  Wireless sensor networks are considered to be among the most significant and innovative technologies in the 21st century due to their wide range of industrial applications. Sensor nodes in these networks are susceptible to a variety of assaults due to their special qualities and method of deployment. In WSNs, denial of service attacks are common attacks in sensor networks. It is difficult to design a detection and prevention system that would effectively reduce the impact of these attacks on WSNs. In order to identify assaults on WSNs, this study suggests using two machine learning models: decision trees and XGBoost. The WSNs dataset was the subject of extensive tests to identify denial of service attacks. The experimental findings demonstrate that the XGBoost model, when applied to the entire dataset, has a higher true positive rate (98.3%) than the Decision tree approach (97.3%) and a lower false positive rate (1.7%) than the Decision tree technique (2.7%). Like this, with selected dataset assaults, the XGBoost approach has a higher true positive rate (99.01%) than the Decision tree technique (97.50%) and a lower false positive rate (0.99%) than the Decision tree technique (2.50%).

Title: Exploring adversarial attacks in federated learning for medical imaging. (arXiv:2310.06227v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06227
• Code URL: null
• Copy Paste: [[2310.06227]] Exploring adversarial attacks in federated learning for medical imaging(http://arxiv.org/abs/2310.06227)
• Summary:

  Federated learning offers a privacy-preserving framework for medical image analysis but exposes the system to adversarial attacks. This paper aims to evaluate the vulnerabilities of federated learning networks in medical image analysis against such attacks. Employing domain-specific MRI tumor and pathology imaging datasets, we assess the effectiveness of known threat scenarios in a federated learning environment. Our tests reveal that domain-specific configurations can increase the attacker's success rate significantly. The findings emphasize the urgent need for effective defense mechanisms and suggest a critical re-evaluation of current security protocols in federated medical image analysis systems.

robust

Title: Deep Learning based Tomato Disease Detection and Remedy Suggestions using Mobile Application. (arXiv:2310.05929v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.05929
• Code URL: null
• Copy Paste: [[2310.05929]] Deep Learning based Tomato Disease Detection and Remedy Suggestions using Mobile Application(http://arxiv.org/abs/2310.05929)
• Summary:

  We have developed a comprehensive computer system to assist farmers who practice traditional farming methods and have limited access to agricultural experts for addressing crop diseases. Our system utilizes artificial intelligence (AI) to identify and provide remedies for vegetable diseases. To ensure ease of use, we have created a mobile application that offers a user-friendly interface, allowing farmers to inquire about vegetable diseases and receive suitable solutions in their local language. The developed system can be utilized by any farmer with a basic understanding of a smartphone. Specifically, we have designed an AI-enabled mobile application for identifying and suggesting remedies for vegetable diseases, focusing on tomato diseases to benefit the local farming community in Nepal. Our system employs state-of-the-art object detection methodology, namely You Only Look Once (YOLO), to detect tomato diseases. The detected information is then relayed to the mobile application, which provides remedy suggestions guided by domain experts. In order to train our system effectively, we curated a dataset consisting of ten classes of tomato diseases. We utilized various data augmentation methods to address overfitting and trained a YOLOv5 object detector. The proposed method achieved a mean average precision of 0.76 and offers an efficient mobile interface for interacting with the AI system. While our system is currently in the development phase, we are actively working towards enhancing its robustness and real-time usability by accumulating more training samples.

Title: Automating global landslide detection with heterogeneous ensemble deep-learning classification. (arXiv:2310.05959v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.05959
• Code URL: null
• Copy Paste: [[2310.05959]] Automating global landslide detection with heterogeneous ensemble deep-learning classification(http://arxiv.org/abs/2310.05959)
• Summary:

  With changing climatic conditions, we are already seeing an increase in extreme weather events and their secondary consequences, including landslides. Landslides threaten infrastructure, including roads, railways, buildings, and human life. Hazard-based spatial planning and early warning systems are cost-effective strategies to reduce the risk to society from landslides. However, these both rely on data from previous landslide events, which is often scarce. Many deep learning (DL) models have recently been applied for landside mapping using medium- to high-resolution satellite images as input. However, they often suffer from sensitivity problems, overfitting, and low mapping accuracy. This study addresses some of these limitations by using a diverse global landslide dataset, using different segmentation models, such as Unet, Linknet, PSP-Net, PAN, and DeepLab and based on their performances, building an ensemble model. The ensemble model achieved the highest F1-score (0.69) when combining both Sentinel-1 and Sentinel-2 bands, with the highest average improvement of 6.87 % when the ensemble size was 20. On the other hand, Sentinel-2 bands only performed very well, with an F1 score of 0.61 when the ensemble size is 20 with an improvement of 14.59 % when the ensemble size is 20. This result shows considerable potential in building a robust and reliable monitoring system based on changes in vegetation index dNDVI only.

Title: CrowdRec: 3D Crowd Reconstruction from Single Color Images. (arXiv:2310.06332v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06332
• Code URL: https://github.com/boycehbz/crowdrec
• Copy Paste: [[2310.06332]] CrowdRec: 3D Crowd Reconstruction from Single Color Images(http://arxiv.org/abs/2310.06332)
• Summary:

  This is a technical report for the GigaCrowd challenge. Reconstructing 3D crowds from monocular images is a challenging problem due to mutual occlusions, server depth ambiguity, and complex spatial distribution. Since no large-scale 3D crowd dataset can be used to train a robust model, the current multi-person mesh recovery methods can hardly achieve satisfactory performance in crowded scenes. In this paper, we exploit the crowd features and propose a crowd-constrained optimization to improve the common single-person method on crowd images. To avoid scale variations, we first detect human bounding-boxes and 2D poses from the original images with off-the-shelf detectors. Then, we train a single-person mesh recovery network using existing in-the-wild image datasets. To promote a more reasonable spatial distribution, we further propose a crowd constraint to refine the single-person network parameters. With the optimization, we can obtain accurate body poses and shapes with reasonable absolute positions from a large-scale crowd image using a single-person backbone. The code will be publicly available at~\url{https://github.com/boycehbz/CrowdRec}.

Title: Leveraging Diffusion-Based Image Variations for Robust Training on Poisoned Data. (arXiv:2310.06372v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06372
• Code URL: https://github.com/lukasstruppek/robust_training_on_poisoned_samples
• Copy Paste: [[2310.06372]] Leveraging Diffusion-Based Image Variations for Robust Training on Poisoned Data(http://arxiv.org/abs/2310.06372)
• Summary:

  Backdoor attacks pose a serious security threat for training neural networks as they surreptitiously introduce hidden functionalities into a model. Such backdoors remain silent during inference on clean inputs, evading detection due to inconspicuous behavior. However, once a specific trigger pattern appears in the input data, the backdoor activates, causing the model to execute its concealed function. Detecting such poisoned samples within vast datasets is virtually impossible through manual inspection. To address this challenge, we propose a novel approach that enables model training on potentially poisoned datasets by utilizing the power of recent diffusion models. Specifically, we create synthetic variations of all training samples, leveraging the inherent resilience of diffusion models to potential trigger patterns in the data. By combining this generative approach with knowledge distillation, we produce student models that maintain their general performance on the task while exhibiting robust resistance to backdoor triggers.

Title: Let Models Speak Ciphers: Multiagent Debate through Embeddings. (arXiv:2310.06272v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06272
• Code URL: null
• Copy Paste: [[2310.06272]] Let Models Speak Ciphers: Multiagent Debate through Embeddings(http://arxiv.org/abs/2310.06272)
• Summary:

  Discussion and debate among Large Language Models (LLMs) have gained considerable attention due to their potential to enhance the reasoning ability of LLMs. Although natural language is an obvious choice for communication due to LLM's language understanding capability, the token sampling step needed when generating natural language poses a potential risk of information loss, as it uses only one token to represent the model's belief across the entire vocabulary. In this paper, we introduce a communication regime named CIPHER (Communicative Inter-Model Protocol Through Embedding Representation) to address this issue. Specifically, we remove the token sampling step from LLMs and let them communicate their beliefs across the vocabulary through the expectation of the raw transformer output embeddings. Remarkably, by deviating from natural language, CIPHER offers an advantage of encoding a broader spectrum of information without any modification to the model weights. While the state-of-the-art LLM debate methods using natural language outperforms traditional inference by a margin of 1.5-8%, our experiment results show that CIPHER debate further extends this lead by 1-3.5% across five reasoning tasks and multiple open-source LLMs of varying sizes. This showcases the superiority and robustness of embeddings as an alternative "language" for communication among LLMs.

Title: A Semantic Invariant Robust Watermark for Large Language Models. (arXiv:2310.06356v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2310.06356
• Code URL: null
• Copy Paste: [[2310.06356]] A Semantic Invariant Robust Watermark for Large Language Models(http://arxiv.org/abs/2310.06356)
• Summary:

  Watermark algorithms for large language models (LLMs) have achieved extremely high accuracy in detecting text generated by LLMs. Such algorithms typically involve adding extra watermark logits to the LLM's logits at each generation step. However, prior algorithms face a trade-off between attack robustness and security robustness. This is because the watermark logits for a token are determined by a certain number of preceding tokens; a small number leads to low security robustness, while a large number results in insufficient attack robustness. In this work, we propose a semantic invariant watermarking method for LLMs that provides both attack robustness and security robustness. The watermark logits in our work are determined by the semantics of all preceding tokens. Specifically, we utilize another embedding LLM to generate semantic embeddings for all preceding tokens, and then these semantic embeddings are transformed into the watermark logits through our trained watermark model. Subsequent analyses and experiments demonstrated the attack robustness of our method in semantically invariant settings: synonym substitution and text paraphrasing settings. Finally, we also show that our watermark possesses adequate security robustness. Our code and data are available at https://github.com/THU-BPM/Robust_Watermark.

Title: Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approach. (arXiv:2310.06112v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06112
• Code URL: https://github.com/fshp971/adv-ntk
• Copy Paste: [[2310.06112]] Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approach(http://arxiv.org/abs/2310.06112)
• Summary:

  Adversarial training (AT) is a canonical method for enhancing the robustness of deep neural networks (DNNs). However, recent studies empirically demonstrated that it suffers from robust overfitting, i.e., a long time AT can be detrimental to the robustness of DNNs. This paper presents a theoretical explanation of robust overfitting for DNNs. Specifically, we non-trivially extend the neural tangent kernel (NTK) theory to AT and prove that an adversarially trained wide DNN can be well approximated by a linearized DNN. Moreover, for squared loss, closed-form AT dynamics for the linearized DNN can be derived, which reveals a new AT degeneration phenomenon: a long-term AT will result in a wide DNN degenerates to that obtained without AT and thus cause robust overfitting. Based on our theoretical results, we further design a method namely Adv-NTK, the first AT algorithm for infinite-width DNNs. Experiments on real-world datasets show that Adv-NTK can help infinite-width DNNs enhance comparable robustness to that of their finite-width counterparts, which in turn justifies our theoretical findings. The code is available at https://github.com/fshp971/adv-ntk.

Title: Provably Accelerating Ill-Conditioned Low-rank Estimation via Scaled Gradient Descent, Even with Overparameterization. (arXiv:2310.06159v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06159
• Code URL: null
• Copy Paste: [[2310.06159]] Provably Accelerating Ill-Conditioned Low-rank Estimation via Scaled Gradient Descent, Even with Overparameterization(http://arxiv.org/abs/2310.06159)
• Summary:

  Many problems encountered in science and engineering can be formulated as estimating a low-rank object (e.g., matrices and tensors) from incomplete, and possibly corrupted, linear measurements. Through the lens of matrix and tensor factorization, one of the most popular approaches is to employ simple iterative algorithms such as gradient descent (GD) to recover the low-rank factors directly, which allow for small memory and computation footprints. However, the convergence rate of GD depends linearly, and sometimes even quadratically, on the condition number of the low-rank object, and therefore, GD slows down painstakingly when the problem is ill-conditioned. This chapter introduces a new algorithmic approach, dubbed scaled gradient descent (ScaledGD), that provably converges linearly at a constant rate independent of the condition number of the low-rank object, while maintaining the low per-iteration cost of gradient descent for a variety of tasks including sensing, robust principal component analysis and completion. In addition, ScaledGD continues to admit fast global convergence to the minimax-optimal solution, again almost independent of the condition number, from a small random initialization when the rank is over-specified in the presence of Gaussian noise. In total, ScaledGD highlights the power of appropriate preconditioning in accelerating nonconvex statistical estimation, where the iteration-varying preconditioners promote desirable invariance properties of the trajectory with respect to the symmetry in low-rank factorization without hurting generalization.

Title: Mitigating Simplicity Bias in Deep Learning for Improved OOD Generalization and Robustness. (arXiv:2310.06161v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06161
• Code URL: https://github.com/estija/cmid
• Copy Paste: [[2310.06161]] Mitigating Simplicity Bias in Deep Learning for Improved OOD Generalization and Robustness(http://arxiv.org/abs/2310.06161)
• Summary:

  Neural networks (NNs) are known to exhibit simplicity bias where they tend to prefer learning 'simple' features over more 'complex' ones, even when the latter may be more informative. Simplicity bias can lead to the model making biased predictions which have poor out-of-distribution (OOD) generalization. To address this, we propose a framework that encourages the model to use a more diverse set of features to make predictions. We first train a simple model, and then regularize the conditional mutual information with respect to it to obtain the final model. We demonstrate the effectiveness of this framework in various problem settings and real-world applications, showing that it effectively addresses simplicity bias and leads to more features being used, enhances OOD generalization, and improves subgroup robustness and fairness. We complement these results with theoretical analyses of the effect of the regularization and its OOD generalization properties.

Title: PAC-Bayesian Spectrally-Normalized Bounds for Adversarially Robust Generalization. (arXiv:2310.06182v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06182
• Code URL: null
• Copy Paste: [[2310.06182]] PAC-Bayesian Spectrally-Normalized Bounds for Adversarially Robust Generalization(http://arxiv.org/abs/2310.06182)
• Summary:

  Deep neural networks (DNNs) are vulnerable to adversarial attacks. It is found empirically that adversarially robust generalization is crucial in establishing defense algorithms against adversarial attacks. Therefore, it is interesting to study the theoretical guarantee of robust generalization. This paper focuses on norm-based complexity, based on a PAC-Bayes approach (Neyshabur et al., 2017). The main challenge lies in extending the key ingredient, which is a weight perturbation bound in standard settings, to the robust settings. Existing attempts heavily rely on additional strong assumptions, leading to loose bounds. In this paper, we address this issue and provide a spectrally-normalized robust generalization bound for DNNs. Compared to existing bounds, our bound offers two significant advantages: Firstly, it does not depend on additional assumptions. Secondly, it is considerably tighter, aligning with the bounds of standard generalization. Therefore, our result provides a different perspective on understanding robust generalization: The mismatch terms between standard and robust generalization bounds shown in previous studies do not contribute to the poor robust generalization. Instead, these disparities solely due to mathematical issues. Finally, we extend the main result to adversarial robustness against general non-$\ell_p$ attacks and other neural network architectures.

Title: Transfer learning-based physics-informed convolutional neural network for simulating flow in porous media with time-varying controls. (arXiv:2310.06319v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06319
• Code URL: null
• Copy Paste: [[2310.06319]] Transfer learning-based physics-informed convolutional neural network for simulating flow in porous media with time-varying controls(http://arxiv.org/abs/2310.06319)
• Summary:

  A physics-informed convolutional neural network is proposed to simulate two phase flow in porous media with time-varying well controls. While most of PICNNs in existing literatures worked on parameter-to-state mapping, our proposed network parameterizes the solution with time-varying controls to establish a control-to-state regression. Firstly, finite volume scheme is adopted to discretize flow equations and formulate loss function that respects mass conservation laws. Neumann boundary conditions are seamlessly incorporated into the semi-discretized equations so no additional loss term is needed. The network architecture comprises two parallel U-Net structures, with network inputs being well controls and outputs being the system states. To capture the time-dependent relationship between inputs and outputs, the network is well designed to mimic discretized state space equations. We train the network progressively for every timestep, enabling it to simultaneously predict oil pressure and water saturation at each timestep. After training the network for one timestep, we leverage transfer learning techniques to expedite the training process for subsequent timestep. The proposed model is used to simulate oil-water porous flow scenarios with varying reservoir gridblocks and aspects including computation efficiency and accuracy are compared against corresponding numerical approaches. The results underscore the potential of PICNN in effectively simulating systems with numerous grid blocks, as computation time does not scale with model dimensionality. We assess the temporal error using 10 different testing controls with variation in magnitude and another 10 with higher alternation frequency with proposed control-to-state architecture. Our observations suggest the need for a more robust and reliable model when dealing with controls that exhibit significant variations in magnitude or frequency.

Title: Exploit the antenna response consistency to define the alignment criteria for CSI data. (arXiv:2310.06328v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06328
• Code URL: null
• Copy Paste: [[2310.06328]] Exploit the antenna response consistency to define the alignment criteria for CSI data(http://arxiv.org/abs/2310.06328)
• Summary:

  Self-supervised learning (SSL) for WiFi-based human activity recognition (HAR) holds great promise due to its ability to address the challenge of insufficient labeled data. However, directly transplanting SSL algorithms, especially contrastive learning, originally designed for other domains to CSI data, often fails to achieve the expected performance. We attribute this issue to the inappropriate alignment criteria, which disrupt the semantic distance consistency between the feature space and the input space. To address this challenge, we introduce \textbf{A}netenna \textbf{R}esponse \textbf{C}onsistency (ARC) as a solution to define proper alignment criteria. ARC is designed to retain semantic information from the input space while introducing robustness to real-world noise. We analyze ARC from the perspective of CSI data structure, demonstrating that its optimal solution leads to a direct mapping from input CSI data to action vectors in the feature map. Furthermore, we provide extensive experimental evidence to validate the effectiveness of ARC in improving the performance of self-supervised learning for WiFi-based HAR.

Title: CAST: Cluster-Aware Self-Training for Tabular Data. (arXiv:2310.06380v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06380
• Code URL: null
• Copy Paste: [[2310.06380]] CAST: Cluster-Aware Self-Training for Tabular Data(http://arxiv.org/abs/2310.06380)
• Summary:

  Self-training has gained attraction because of its simplicity and versatility, yet it is vulnerable to noisy pseudo-labels. Several studies have proposed successful approaches to tackle this issue, but they have diminished the advantages of self-training because they require specific modifications in self-training algorithms or model architectures. Furthermore, most of them are incompatible with gradient boosting decision trees, which dominate the tabular domain. To address this, we revisit the cluster assumption, which states that data samples that are close to each other tend to belong to the same class. Inspired by the assumption, we propose Cluster-Aware Self-Training (CAST) for tabular data. CAST is a simple and universally adaptable approach for enhancing existing self-training algorithms without significant modifications. Concretely, our method regularizes the confidence of the classifier, which represents the value of the pseudo-label, forcing the pseudo-labels in low-density regions to have lower confidence by leveraging prior knowledge for each class within the training data. Extensive empirical evaluations on up to 20 real-world datasets confirm not only the superior performance of CAST but also its robustness in various setups in self-training contexts.

Title: Adversarial Robustness in Graph Neural Networks: A Hamiltonian Approach. (arXiv:2310.06396v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06396
• Code URL: https://github.com/zknus/neurips-2023-hang-robustness
• Copy Paste: [[2310.06396]] Adversarial Robustness in Graph Neural Networks: A Hamiltonian Approach(http://arxiv.org/abs/2310.06396)
• Summary:

  Graph neural networks (GNNs) are vulnerable to adversarial perturbations, including those that affect both node features and graph topology. This paper investigates GNNs derived from diverse neural flows, concentrating on their connection to various stability notions such as BIBO stability, Lyapunov stability, structural stability, and conservative stability. We argue that Lyapunov stability, despite its common use, does not necessarily ensure adversarial robustness. Inspired by physics principles, we advocate for the use of conservative Hamiltonian neural flows to construct GNNs that are robust to adversarial attacks. The adversarial robustness of different neural flow GNNs is empirically compared on several benchmark datasets under a variety of adversarial attacks. Extensive numerical experiments demonstrate that GNNs leveraging conservative Hamiltonian flows with Lyapunov stability substantially improve robustness against adversarial perturbations. The implementation code of experiments is available at https://github.com/zknus/NeurIPS-2023-HANG-Robustness.

biometric

steal

extraction

Title: Fire Detection From Image and Video Using YOLOv5. (arXiv:2310.06351v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06351
• Code URL: null
• Copy Paste: [[2310.06351]] Fire Detection From Image and Video Using YOLOv5(http://arxiv.org/abs/2310.06351)
• Summary:

  For the detection of fire-like targets in indoor, outdoor and forest fire images, as well as fire detection under different natural lights, an improved YOLOv5 fire detection deep learning algorithm is proposed. The YOLOv5 detection model expands the feature extraction network from three dimensions, which enhances feature propagation of fire small targets identification, improves network performance, and reduces model parameters. Furthermore, through the promotion of the feature pyramid, the top-performing prediction box is obtained. Fire-YOLOv5 attains excellent results compared to state-of-the-art object detection networks, notably in the detection of small targets of fire and smoke with mAP 90.5% and f1 score 88%. Overall, the Fire-YOLOv5 detection model can effectively deal with the inspection of small fire targets, as well as fire-like and smoke-like objects with F1 score 0.88. When the input image size is 416 x 416 resolution, the average detection time is 0.12 s per frame, which can provide real-time forest fire detection. Moreover, the algorithm proposed in this paper can also be applied to small target detection under other complicated situations. The proposed system shows an improved approach in all fire detection metrics such as precision, recall, and mean average precision.

Title: Skeleton Ground Truth Extraction: Methodology, Annotation Tool and Benchmarks. (arXiv:2310.06437v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06437
• Code URL: null
• Copy Paste: [[2310.06437]] Skeleton Ground Truth Extraction: Methodology, Annotation Tool and Benchmarks(http://arxiv.org/abs/2310.06437)
• Summary:

  Skeleton Ground Truth (GT) is critical to the success of supervised skeleton extraction methods, especially with the popularity of deep learning techniques. Furthermore, we see skeleton GTs used not only for training skeleton detectors with Convolutional Neural Networks (CNN) but also for evaluating skeleton-related pruning and matching algorithms. However, most existing shape and image datasets suffer from the lack of skeleton GT and inconsistency of GT standards. As a result, it is difficult to evaluate and reproduce CNN-based skeleton detectors and algorithms on a fair basis. In this paper, we present a heuristic strategy for object skeleton GT extraction in binary shapes and natural images. Our strategy is built on an extended theory of diagnosticity hypothesis, which enables encoding human-in-the-loop GT extraction based on clues from the target's context, simplicity, and completeness. Using this strategy, we developed a tool, SkeView, to generate skeleton GT of 17 existing shape and image datasets. The GTs are then structurally evaluated with representative methods to build viable baselines for fair comparisons. Experiments demonstrate that GTs generated by our strategy yield promising quality with respect to standard consistency, and also provide a balance between simplicity and completeness.

Title: Enhancing Document-level Event Argument Extraction with Contextual Clues and Role Relevance. (arXiv:2310.05991v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.05991
• Code URL: https://github.com/LWL-cpu/SCPRG-master
• Copy Paste: [[2310.05991]] Enhancing Document-level Event Argument Extraction with Contextual Clues and Role Relevance(http://arxiv.org/abs/2310.05991)
• Summary:

  Document-level event argument extraction poses new challenges of long input and cross-sentence inference compared to its sentence-level counterpart. However, most prior works focus on capturing the relations between candidate arguments and the event trigger in each event, ignoring two crucial points: a) non-argument contextual clue information; b) the relevance among argument roles. In this paper, we propose a SCPRG (Span-trigger-based Contextual Pooling and latent Role Guidance) model, which contains two novel and effective modules for the above problem. The Span-Trigger-based Contextual Pooling(STCP) adaptively selects and aggregates the information of non-argument clue words based on the context attention weights of specific argument-trigger pairs from pre-trained model. The Role-based Latent Information Guidance (RLIG) module constructs latent role representations, makes them interact through role-interactive encoding to capture semantic relevance, and merges them into candidate arguments. Both STCP and RLIG introduce no more than 1% new parameters compared with the base model and can be easily applied to other event extraction models, which are compact and transplantable. Experiments on two public datasets show that our SCPRG outperforms previous state-of-the-art methods, with 1.13 F1 and 2.64 F1 improvements on RAMS and WikiEvents respectively. Further analyses illustrate the interpretability of our model.

Title: CAW-coref: Conjunction-Aware Word-level Coreference Resolution. (arXiv:2310.06165v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06165
• Code URL: https://github.com/kareldo/wl-coref
• Copy Paste: [[2310.06165]] CAW-coref: Conjunction-Aware Word-level Coreference Resolution(http://arxiv.org/abs/2310.06165)
• Summary:

  State-of-the-art coreference resolutions systems depend on multiple LLM calls per document and are thus prohibitively expensive for many use cases (e.g., information extraction with large corpora). The leading word-level coreference system (WL-coref) attains 96.6% of these SOTA systems' performance while being much more efficient. In this work, we identify a routine yet important failure case of WL-coref: dealing with conjoined mentions such as 'Tom and Mary'. We offer a simple yet effective solution that improves the performance on the OntoNotes test set by 0.9% F1, shrinking the gap between efficient word-level coreference resolution and expensive SOTA approaches by 34.6%. Our Conjunction-Aware Word-level coreference model (CAW-coref) and code is available at https://github.com/KarelDO/wl-coref.

Title: Model Tuning or Prompt Tuning? A Study of Large Language Models for Clinical Concept and Relation Extraction. (arXiv:2310.06239v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06239
• Code URL: null
• Copy Paste: [[2310.06239]] Model Tuning or Prompt Tuning? A Study of Large Language Models for Clinical Concept and Relation Extraction(http://arxiv.org/abs/2310.06239)
• Summary:

  Objective To develop soft prompt-based learning algorithms for large language models (LLMs), examine the shape of prompts, prompt-tuning using frozen/unfrozen LLMs, transfer learning, and few-shot learning abilities. Methods We developed a soft prompt-based LLM model and compared 4 training strategies including (1) fine-tuning without prompts; (2) hard-prompt with unfrozen LLMs; (3) soft-prompt with unfrozen LLMs; and (4) soft-prompt with frozen LLMs. We evaluated 7 pretrained LLMs using the 4 training strategies for clinical concept and relation extraction on two benchmark datasets. We evaluated the transfer learning ability of the prompt-based learning algorithms in a cross-institution setting. We also assessed the few-shot learning ability. Results and Conclusion When LLMs are unfrozen, GatorTron-3.9B with soft prompting achieves the best strict F1-scores of 0.9118 and 0.8604 for concept extraction, outperforming the traditional fine-tuning and hard prompt-based models by 0.6~3.1% and 1.2~2.9%, respectively; GatorTron-345M with soft prompting achieves the best F1-scores of 0.8332 and 0.7488 for end-to-end relation extraction, outperforming the other two models by 0.2~2% and 0.6~11.7%, respectively. When LLMs are frozen, small (i.e., 345 million parameters) LLMs have a big gap to be competitive with unfrozen models; scaling LLMs up to billions of parameters makes frozen LLMs competitive with unfrozen LLMs. For cross-institute evaluation, soft prompting with a frozen GatorTron-8.9B model achieved the best performance. This study demonstrates that (1) machines can learn soft prompts better than humans, (2) frozen LLMs have better few-shot learning ability and transfer learning ability to facilitate muti-institution applications, and (3) frozen LLMs require large models.

membership infer

federate

Title: Text-driven Prompt Generation for Vision-Language Models in Federated Learning. (arXiv:2310.06123v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06123
• Code URL: null
• Copy Paste: [[2310.06123]] Text-driven Prompt Generation for Vision-Language Models in Federated Learning(http://arxiv.org/abs/2310.06123)
• Summary:

  Prompt learning for vision-language models, e.g., CoOp, has shown great success in adapting CLIP to different downstream tasks, making it a promising solution for federated learning due to computational reasons. Existing prompt learning techniques replace hand-crafted text prompts with learned vectors that offer improvements on seen classes, but struggle to generalize to unseen classes. Our work addresses this challenge by proposing Federated Text-driven Prompt Generation (FedTPG), which learns a unified prompt generation network across multiple remote clients in a scalable manner. The prompt generation network is conditioned on task-related text input, thus is context-aware, making it suitable to generalize for both seen and unseen classes. Our comprehensive empirical evaluations on nine diverse image classification datasets show that our method is superior to existing federated prompt learning methods, that achieve overall better generalization on both seen and unseen classes and is also generalizable to unseen datasets.

Title: Federated Multi-Level Optimization over Decentralized Networks. (arXiv:2310.06217v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06217
• Code URL: null
• Copy Paste: [[2310.06217]] Federated Multi-Level Optimization over Decentralized Networks(http://arxiv.org/abs/2310.06217)
• Summary:

  Multi-level optimization has gained increasing attention in recent years, as it provides a powerful framework for solving complex optimization problems that arise in many fields, such as meta-learning, multi-player games, reinforcement learning, and nested composition optimization. In this paper, we study the problem of distributed multi-level optimization over a network, where agents can only communicate with their immediate neighbors. This setting is motivated by the need for distributed optimization in large-scale systems, where centralized optimization may not be practical or feasible. To address this problem, we propose a novel gossip-based distributed multi-level optimization algorithm that enables networked agents to solve optimization problems at different levels in a single timescale and share information through network propagation. Our algorithm achieves optimal sample complexity, scaling linearly with the network size, and demonstrates state-of-the-art performance on various applications, including hyper-parameter tuning, decentralized reinforcement learning, and risk-averse optimization.

Title: Federated Learning with Reduced Information Leakage and Computation. (arXiv:2310.06341v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06341
• Code URL: null
• Copy Paste: [[2310.06341]] Federated Learning with Reduced Information Leakage and Computation(http://arxiv.org/abs/2310.06341)
• Summary:

  Federated learning (FL) is a distributed learning paradigm that allows multiple decentralized clients to collaboratively learn a common model without sharing local data. Although local data is not exposed directly, privacy concerns nonetheless exist as clients' sensitive information can be inferred from intermediate computations. Moreover, such information leakage accumulates substantially over time as the same data is repeatedly used during the iterative learning process. As a result, it can be particularly difficult to balance the privacy-accuracy trade-off when designing privacy-preserving FL algorithms. In this paper, we introduce Upcycled-FL, a novel federated learning framework with first-order approximation applied at every even iteration. Under this framework, half of the FL updates incur no information leakage and require much less computation. We first conduct the theoretical analysis on the convergence (rate) of Upcycled-FL, and then apply perturbation mechanisms to preserve privacy. Experiments on real-world data show that Upcycled-FL consistently outperforms existing methods over heterogeneous data, and significantly improves privacy-accuracy trade-off while reducing 48% of the training time on average.

Title: Asynchronous Federated Learning with Incentive Mechanism Based on Contract Theory. (arXiv:2310.06448v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06448
• Code URL: null
• Copy Paste: [[2310.06448]] Asynchronous Federated Learning with Incentive Mechanism Based on Contract Theory(http://arxiv.org/abs/2310.06448)
• Summary:

  To address the challenges posed by the heterogeneity inherent in federated learning (FL) and to attract high-quality clients, various incentive mechanisms have been employed. However, existing incentive mechanisms are typically utilized in conventional synchronous aggregation, resulting in significant straggler issues. In this study, we propose a novel asynchronous FL framework that integrates an incentive mechanism based on contract theory. Within the incentive mechanism, we strive to maximize the utility of the task publisher by adaptively adjusting clients' local model training epochs, taking into account factors such as time delay and test accuracy. In the asynchronous scheme, considering client quality, we devise aggregation weights and an access control algorithm to facilitate asynchronous aggregation. Through experiments conducted on the MNIST dataset, the simulation results demonstrate that the test accuracy achieved by our framework is 3.12% and 5.84% higher than that achieved by FedAvg and FedProx without any attacks, respectively. The framework exhibits a 1.35% accuracy improvement over the ideal Local SGD under attacks. Furthermore, aiming for the same target accuracy, our framework demands notably less computation time than both FedAvg and FedProx.

fair

Title: CFDBench: A Comprehensive Benchmark for Machine Learning Methods in Fluid Dynamics. (arXiv:2310.05963v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.05963
• Code URL: null
• Copy Paste: [[2310.05963]] CFDBench: A Comprehensive Benchmark for Machine Learning Methods in Fluid Dynamics(http://arxiv.org/abs/2310.05963)
• Summary:

  In recent years, applying deep learning to solve physics problems has attracted much attention. Data-driven deep learning methods produce operators that can learn solutions to the whole system of partial differential equations. However, the existing methods are only evaluated on simple flow equations (e.g., Burger's equation), and only consider the generalization ability on different initial conditions. In this paper, we construct CFDBench, a benchmark with four classic problems in computational fluid dynamics (CFD): lid-driven cavity flow, laminar boundary layer flow in circular tubes, dam flows through the steps, and periodic Karman vortex street. Each flow problem includes data with different boundary conditions, fluid physical properties, and domain geometry. Compared to existing datasets, the advantages of CFDBench are (1) comprehensive. It contains common physical parameters such as velocity, pressure, and cavity fraction. (2) realistic. It is very suitable for deep learning solutions of fluid mechanics equations. (3) challenging. It has a certain learning difficulty, prompting to find models with strong learning ability. (4) standardized. CFDBench facilitates a comprehensive and fair comparison of different deep learning methods for CFD. We make appropriate modifications to popular deep neural networks to apply them to CFDBench and enable the accommodation of more changing inputs. The evaluation on CFDBench reveals some new shortcomings of existing works and we propose possible directions for solving such problems.

Title: Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis. (arXiv:2310.06119v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06119
• Code URL: null
• Copy Paste: [[2310.06119]] Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis(http://arxiv.org/abs/2310.06119)
• Summary:

  Multivariate Time Series (MTS) widely exists in real-word complex systems, such as traffic and energy systems, making their forecasting crucial for understanding and influencing these systems. Recently, deep learning-based approaches have gained much popularity for effectively modeling temporal and spatial dependencies in MTS, specifically in Long-term Time Series Forecasting (LTSF) and Spatial-Temporal Forecasting (STF). However, the fair benchmarking issue and the choice of technical approaches have been hotly debated in related work. Such controversies significantly hinder our understanding of progress in this field. Thus, this paper aims to address these controversies to present insights into advancements achieved. To resolve benchmarking issues, we introduce BasicTS, a benchmark designed for fair comparisons in MTS forecasting. BasicTS establishes a unified training pipeline and reasonable evaluation settings, enabling an unbiased evaluation of over 30 popular MTS forecasting models on more than 18 datasets. Furthermore, we highlight the heterogeneity among MTS datasets and classify them based on temporal and spatial characteristics. We further prove that neglecting heterogeneity is the primary reason for generating controversies in technical approaches. Moreover, based on the proposed BasicTS and rich heterogeneous MTS datasets, we conduct an exhaustive and reproducible performance and efficiency comparison of popular models, providing insights for researchers in selecting and designing MTS forecasting models.

Title: Fair Classifiers that Abstain without Harm. (arXiv:2310.06205v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06205
• Code URL: null
• Copy Paste: [[2310.06205]] Fair Classifiers that Abstain without Harm(http://arxiv.org/abs/2310.06205)
• Summary:

  In critical applications, it is vital for classifiers to defer decision-making to humans. We propose a post-hoc method that makes existing classifiers selectively abstain from predicting certain samples. Our abstaining classifier is incentivized to maintain the original accuracy for each sub-population (i.e. no harm) while achieving a set of group fairness definitions to a user specified degree. To this end, we design an Integer Programming (IP) procedure that assigns abstention decisions for each training sample to satisfy a set of constraints. To generalize the abstaining decisions to test samples, we then train a surrogate model to learn the abstaining decisions based on the IP solutions in an end-to-end manner. We analyze the feasibility of the IP procedure to determine the possible abstention rate for different levels of unfairness tolerance and accuracy constraint for achieving no harm. To the best of our knowledge, this work is the first to identify the theoretical relationships between the constraint parameters and the required abstention rate. Our theoretical results are important since a high abstention rate is often infeasible in practice due to a lack of human resources. Our framework outperforms existing methods in terms of fairness disparity without sacrificing accuracy at similar abstention rates.

interpretability

Title: CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding. (arXiv:2310.06214v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06214
• Code URL: null
• Copy Paste: [[2310.06214]] CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding(http://arxiv.org/abs/2310.06214)
• Summary:

  3D visual grounding is the ability to localize objects in 3D scenes conditioned by utterances. Most existing methods devote the referring head to localize the referred object directly, causing failure in complex scenarios. In addition, it does not illustrate how and why the network reaches the final decision. In this paper, we address this question Can we design an interpretable 3D visual grounding framework that has the potential to mimic the human perception system?. To this end, we formulate the 3D visual grounding problem as a sequence-to-sequence task by first predicting a chain of anchors and then the final target. Interpretability not only improves the overall performance but also helps us identify failure cases. Following the chain of thoughts approach enables us to decompose the referring task into interpretable intermediate steps, boosting the performance and making our framework extremely data-efficient. Moreover, our proposed framework can be easily integrated into any existing architecture. We validate our approach through comprehensive experiments on the Nr3D, Sr3D, and Scanrefer benchmarks and show consistent performance gains compared to existing methods without requiring manually annotated data. Furthermore, our proposed framework, dubbed CoT3DRef, is significantly data-efficient, whereas on the Sr3D dataset, when trained only on 10% of the data, we match the SOTA performance that trained on the entire data.

Title: A novel Network Science Algorithm for Improving Triage of Patients. (arXiv:2310.05996v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.05996
• Code URL: null
• Copy Paste: [[2310.05996]] A novel Network Science Algorithm for Improving Triage of Patients(http://arxiv.org/abs/2310.05996)
• Summary:

  Patient triage plays a crucial role in healthcare, ensuring timely and appropriate care based on the urgency of patient conditions. Traditional triage methods heavily rely on human judgment, which can be subjective and prone to errors. Recently, a growing interest has been in leveraging artificial intelligence (AI) to develop algorithms for triaging patients. This paper presents the development of a novel algorithm for triaging patients. It is based on the analysis of patient data to produce decisions regarding their prioritization. The algorithm was trained on a comprehensive data set containing relevant patient information, such as vital signs, symptoms, and medical history. The algorithm was designed to accurately classify patients into triage categories through rigorous preprocessing and feature engineering. Experimental results demonstrate that our algorithm achieved high accuracy and performance, outperforming traditional triage methods. By incorporating computer science into the triage process, healthcare professionals can benefit from improved efficiency, accuracy, and consistency, prioritizing patients effectively and optimizing resource allocation. Although further research is needed to address challenges such as biases in training data and model interpretability, the development of AI-based algorithms for triaging patients shows great promise in enhancing healthcare delivery and patient outcomes.

explainability

watermark

diffusion

Title: DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion. (arXiv:2310.05934v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.05934
• Code URL: null
• Copy Paste: [[2310.05934]] DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion(http://arxiv.org/abs/2310.05934)
• Summary:

  Speech-driven 3D facial animation has gained significant attention for its ability to create realistic and expressive facial animations in 3D space based on speech. Learning-based methods have shown promising progress in achieving accurate facial motion synchronized with speech. However, one-to-many nature of speech-to-3D facial synthesis has not been fully explored: while the lip accurately synchronizes with the speech content, other facial attributes beyond speech-related motions are variable with respect to the speech. To account for the potential variance in the facial attributes within a single speech, we propose DF-3DFace, a diffusion-driven speech-to-3D face mesh synthesis. DF-3DFace captures the complex one-to-many relationships between speech and 3D face based on diffusion. It concurrently achieves aligned lip motion by exploiting audio-mesh synchronization and masked conditioning. Furthermore, the proposed method jointly models identity and pose in addition to facial motions so that it can generate 3D face animation without requiring a reference identity mesh and produce natural head poses. We contribute a new large-scale 3D facial mesh dataset, 3D-HDTF to enable the synthesis of variations in identities, poses, and facial motions of 3D face mesh. Extensive experiments demonstrate that our method successfully generates highly variable facial shapes and motions from speech and simultaneously achieves more realistic facial animation than the state-of-the-art methods.

Title: Layout Sequence Prediction From Noisy Mobile Modality. (arXiv:2310.06138v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06138
• Code URL: null
• Copy Paste: [[2310.06138]] Layout Sequence Prediction From Noisy Mobile Modality(http://arxiv.org/abs/2310.06138)
• Summary:

  Trajectory prediction plays a vital role in understanding pedestrian movement for applications such as autonomous driving and robotics. Current trajectory prediction models depend on long, complete, and accurately observed sequences from visual modalities. Nevertheless, real-world situations often involve obstructed cameras, missed objects, or objects out of sight due to environmental factors, leading to incomplete or noisy trajectories. To overcome these limitations, we propose LTrajDiff, a novel approach that treats objects obstructed or out of sight as equally important as those with fully visible trajectories. LTrajDiff utilizes sensor data from mobile phones to surmount out-of-sight constraints, albeit introducing new challenges such as modality fusion, noisy data, and the absence of spatial layout and object size information. We employ a denoising diffusion model to predict precise layout sequences from noisy mobile data using a coarse-to-fine diffusion strategy, incorporating the RMS, Siamese Masked Encoding Module, and MFM. Our model predicts layout sequences by implicitly inferring object size and projection status from a single reference timestamp or significantly obstructed sequences. Achieving SOTA results in randomly obstructed experiments and extremely short input experiments, our model illustrates the effectiveness of leveraging noisy mobile data. In summary, our approach offers a promising solution to the challenges faced by layout sequence and trajectory prediction models in real-world settings, paving the way for utilizing sensor data from mobile phones to accurately predict pedestrian bounding box trajectories. To the best of our knowledge, this is the first work that addresses severely obstructed and extremely short layout sequences by combining vision with noisy mobile modality, making it the pioneering work in the field of layout sequence trajectory prediction.

Title: Improving Compositional Text-to-image Generation with Large Vision-Language Models. (arXiv:2310.06311v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06311
• Code URL: null
• Copy Paste: [[2310.06311]] Improving Compositional Text-to-image Generation with Large Vision-Language Models(http://arxiv.org/abs/2310.06311)
• Summary:

  Recent advancements in text-to-image models, particularly diffusion models, have shown significant promise. However, compositional text-to-image models frequently encounter difficulties in generating high-quality images that accurately align with input texts describing multiple objects, variable attributes, and intricate spatial relationships. To address this limitation, we employ large vision-language models (LVLMs) for multi-dimensional assessment of the alignment between generated images and their corresponding input texts. Utilizing this assessment, we fine-tune the diffusion model to enhance its alignment capabilities. During the inference phase, an initial image is produced using the fine-tuned diffusion model. The LVLM is then employed to pinpoint areas of misalignment in the initial image, which are subsequently corrected using the image editing algorithm until no further misalignments are detected by the LVLM. The resultant image is consequently more closely aligned with the input text. Our experimental results validate that the proposed methodology significantly improves text-image alignment in compositional image generation, particularly with respect to object number, attribute binding, spatial relationships, and aesthetic quality.

Title: Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models. (arXiv:2310.06313v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06313
• Code URL: https://github.com/muzishen/pcdms
• Copy Paste: [[2310.06313]] Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models(http://arxiv.org/abs/2310.06313)
• Summary:

  Recent work has showcased the significant potential of diffusion models in pose-guided person image synthesis. However, owing to the inconsistency in pose between the source and target images, synthesizing an image with a distinct pose, relying exclusively on the source image and target pose information, remains a formidable challenge. This paper presents Progressive Conditional Diffusion Models (PCDMs) that incrementally bridge the gap between person images under the target and source poses through three stages. Specifically, in the first stage, we design a simple prior conditional diffusion model that predicts the global features of the target image by mining the global alignment relationship between pose coordinates and image appearance. Then, the second stage establishes a dense correspondence between the source and target images using the global features from the previous stage, and an inpainting conditional diffusion model is proposed to further align and enhance the contextual features, generating a coarse-grained person image. In the third stage, we propose a refining conditional diffusion model to utilize the coarsely generated image from the previous stage as a condition, achieving texture restoration and enhancing fine-detail consistency. The three-stage PCDMs work progressively to generate the final high-quality and high-fidelity synthesized image. Both qualitative and quantitative results demonstrate the consistency and photorealism of our proposed PCDMs under challenging scenarios.The code and model will be available at https://github.com/muzishen/PCDMs.

Title: JointNet: Extending Text-to-Image Diffusion for Dense Distribution Modeling. (arXiv:2310.06347v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06347
• Code URL: null
• Copy Paste: [[2310.06347]] JointNet: Extending Text-to-Image Diffusion for Dense Distribution Modeling(http://arxiv.org/abs/2310.06347)
• Summary:

  We introduce JointNet, a novel neural network architecture for modeling the joint distribution of images and an additional dense modality (e.g., depth maps). JointNet is extended from a pre-trained text-to-image diffusion model, where a copy of the original network is created for the new dense modality branch and is densely connected with the RGB branch. The RGB branch is locked during network fine-tuning, which enables efficient learning of the new modality distribution while maintaining the strong generalization ability of the large-scale pre-trained diffusion model. We demonstrate the effectiveness of JointNet by using RGBD diffusion as an example and through extensive experiments, showcasing its applicability in a variety of applications, including joint RGBD generation, dense depth prediction, depth-conditioned image generation, and coherent tile-based 3D panorama generation.

Title: Learning Stackable and Skippable LEGO Bricks for Efficient, Reconfigurable, and Variable-Resolution Diffusion Modeling. (arXiv:2310.06389v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06389
• Code URL: null
• Copy Paste: [[2310.06389]] Learning Stackable and Skippable LEGO Bricks for Efficient, Reconfigurable, and Variable-Resolution Diffusion Modeling(http://arxiv.org/abs/2310.06389)
• Summary:

  Diffusion models excel at generating photo-realistic images but come with significant computational costs in both training and sampling. While various techniques address these computational challenges, a less-explored issue is designing an efficient and adaptable network backbone for iterative refinement. Current options like U-Net and Vision Transformer often rely on resource-intensive deep networks and lack the flexibility needed for generating images at variable resolutions or with a smaller network than used in training. This study introduces LEGO bricks, which seamlessly integrate Local-feature Enrichment and Global-content Orchestration. These bricks can be stacked to create a test-time reconfigurable diffusion backbone, allowing selective skipping of bricks to reduce sampling costs and generate higher-resolution images than the training data. LEGO bricks enrich local regions with an MLP and transform them using a Transformer block while maintaining a consistent full-resolution image across all bricks. Experimental results demonstrate that LEGO bricks enhance training efficiency, expedite convergence, and facilitate variable-resolution image generation while maintaining strong generative performance. Moreover, LEGO significantly reduces sampling time compared to other methods, establishing it as a valuable enhancement for diffusion models.

Title: AnoDODE: Anomaly Detection with Diffusion ODE. (arXiv:2310.06420v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06420
• Code URL: null
• Copy Paste: [[2310.06420]] AnoDODE: Anomaly Detection with Diffusion ODE(http://arxiv.org/abs/2310.06420)
• Summary:

  Anomaly detection is the process of identifying atypical data samples that significantly deviate from the majority of the dataset. In the realm of clinical screening and diagnosis, detecting abnormalities in medical images holds great importance. Typically, clinical practice provides access to a vast collection of normal images, while abnormal images are relatively scarce. We hypothesize that abnormal images and their associated features tend to manifest in low-density regions of the data distribution. Following this assumption, we turn to diffusion ODEs for unsupervised anomaly detection, given their tractability and superior performance in density estimation tasks. More precisely, we propose a new anomaly detection method based on diffusion ODEs by estimating the density of features extracted from multi-scale medical images. Our anomaly scoring mechanism depends on computing the negative log-likelihood of features extracted from medical images at different scales, quantified in bits per dimension. Furthermore, we propose a reconstruction-based anomaly localization suitable for our method. Our proposed method not only identifie anomalies but also provides interpretability at both the image and pixel levels. Through experiments on the BraTS2021 medical dataset, our proposed method outperforms existing methods. These results confirm the effectiveness and robustness of our method.

Title: Latent Diffusion Model for DNA Sequence Generation. (arXiv:2310.06150v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06150
• Code URL: null
• Copy Paste: [[2310.06150]] Latent Diffusion Model for DNA Sequence Generation(http://arxiv.org/abs/2310.06150)
• Summary:

  The harnessing of machine learning, especially deep generative models, has opened up promising avenues in the field of synthetic DNA sequence generation. Whilst Generative Adversarial Networks (GANs) have gained traction for this application, they often face issues such as limited sample diversity and mode collapse. On the other hand, Diffusion Models are a promising new class of generative models that are not burdened with these problems, enabling them to reach the state-of-the-art in domains such as image generation. In light of this, we propose a novel latent diffusion model, DiscDiff, tailored for discrete DNA sequence generation. By simply embedding discrete DNA sequences into a continuous latent space using an autoencoder, we are able to leverage the powerful generative abilities of continuous diffusion models for the generation of discrete data. Additionally, we introduce Fr\'echet Reconstruction Distance (FReD) as a new metric to measure the sample quality of DNA sequence generations. Our DiscDiff model demonstrates an ability to generate synthetic DNA sequences that align closely with real DNA in terms of Motif Distribution, Latent Embedding Distribution (FReD), and Chromatin Profiles. Additionally, we contribute a comprehensive cross-species dataset of 150K unique promoter-gene sequences from 15 species, enriching resources for future generative modelling in genomics. We will make our code public upon publication.

Title: Memory-Consistent Neural Networks for Imitation Learning. (arXiv:2310.06171v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06171
• Code URL: null
• Copy Paste: [[2310.06171]] Memory-Consistent Neural Networks for Imitation Learning(http://arxiv.org/abs/2310.06171)
• Summary:

  Imitation learning considerably simplifies policy synthesis compared to alternative approaches by exploiting access to expert demonstrations. For such imitation policies, errors away from the training samples are particularly critical. Even rare slip-ups in the policy action outputs can compound quickly over time, since they lead to unfamiliar future states where the policy is still more likely to err, eventually causing task failures. We revisit simple supervised ``behavior cloning'' for conveniently training the policy from nothing more than pre-recorded demonstrations, but carefully design the model class to counter the compounding error phenomenon. Our ``memory-consistent neural network'' (MCNN) outputs are hard-constrained to stay within clearly specified permissible regions anchored to prototypical ``memory'' training samples. We provide a guaranteed upper bound for the sub-optimality gap induced by MCNN policies. Using MCNNs on 9 imitation learning tasks, with MLP, Transformer, and Diffusion backbones, spanning dexterous robotic manipulation and driving, proprioceptive inputs and visual inputs, and varying sizes and types of demonstration data, we find large and consistent gains in performance, validating that MCNNs are better-suited than vanilla deep neural networks for imitation learning applications. Website: https://sites.google.com/view/mcnn-imitation

Title: DockGame: Cooperative Games for Multimeric Rigid Protein Docking. (arXiv:2310.06177v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06177
• Code URL: https://github.com/vsomnath/dockgame
• Copy Paste: [[2310.06177]] DockGame: Cooperative Games for Multimeric Rigid Protein Docking(http://arxiv.org/abs/2310.06177)
• Summary:

  Protein interactions and assembly formation are fundamental to most biological processes. Predicting the assembly structure from constituent proteins -- referred to as the protein docking task -- is thus a crucial step in protein design applications. Most traditional and deep learning methods for docking have focused mainly on binary docking, following either a search-based, regression-based, or generative modeling paradigm. In this paper, we focus on the less-studied multimeric (i.e., two or more proteins) docking problem. We introduce DockGame, a novel game-theoretic framework for docking -- we view protein docking as a cooperative game between proteins, where the final assembly structure(s) constitute stable equilibria w.r.t. the underlying game potential. Since we do not have access to the true potential, we consider two approaches - i) learning a surrogate game potential guided by physics-based energy functions and computing equilibria by simultaneous gradient updates, and ii) sampling from the Gibbs distribution of the true potential by learning a diffusion generative model over the action spaces (rotations and translations) of all proteins. Empirically, on the Docking Benchmark 5.5 (DB5.5) dataset, DockGame has much faster runtimes than traditional docking methods, can generate multiple plausible assembly structures, and achieves comparable performance to existing binary docking baselines, despite solving the harder task of coordinating multiple protein chains.

Title: Boosting Continuous Control with Consistency Policy. (arXiv:2310.06343v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06343
• Code URL: null
• Copy Paste: [[2310.06343]] Boosting Continuous Control with Consistency Policy(http://arxiv.org/abs/2310.06343)
• Summary:

  Due to its training stability and strong expression, the diffusion model has attracted considerable attention in offline reinforcement learning. However, several challenges have also come with it: 1) The demand for a large number of diffusion steps makes the diffusion-model-based methods time inefficient and limits their applications in real-time control; 2) How to achieve policy improvement with accurate guidance for diffusion model-based policy is still an open problem. Inspired by the consistency model, we propose a novel time-efficiency method named Consistency Policy with Q-Learning (CPQL), which derives action from noise by a single step. By establishing a mapping from the reverse diffusion trajectories to the desired policy, we simultaneously address the issues of time efficiency and inaccurate guidance when updating diffusion model-based policy with the learned Q-function. We demonstrate that CPQL can achieve policy improvement with accurate guidance for offline reinforcement learning, and can be seamlessly extended for online RL tasks. Experimental results indicate that CPQL achieves new state-of-the-art performance on 11 offline and 21 online tasks, significantly improving inference speed by nearly 45 times compared to Diffusion-QL. We will release our code later.

Title: Advective Diffusion Transformers for Topological Generalization in Graph Learning. (arXiv:2310.06417v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06417
• Code URL: null
• Copy Paste: [[2310.06417]] Advective Diffusion Transformers for Topological Generalization in Graph Learning(http://arxiv.org/abs/2310.06417)
• Summary:

  Graph diffusion equations are intimately related to graph neural networks (GNNs) and have recently attracted attention as a principled framework for analyzing GNN dynamics, formalizing their expressive power, and justifying architectural choices. One key open questions in graph learning is the generalization capabilities of GNNs. A major limitation of current approaches hinges on the assumption that the graph topologies in the training and test sets come from the same distribution. In this paper, we make steps towards understanding the generalization of GNNs by exploring how graph diffusion equations extrapolate and generalize in the presence of varying graph topologies. We first show deficiencies in the generalization capability of existing models built upon local diffusion on graphs, stemming from the exponential sensitivity to topology variation. Our subsequent analysis reveals the promise of non-local diffusion, which advocates for feature propagation over fully-connected latent graphs, under the assumption of a specific data-generating condition. In addition to these findings, we propose a novel graph encoder backbone, Advective Diffusion Transformer (ADiT), inspired by advective graph diffusion equations that have a closed-form solution backed up with theoretical guarantees of desired generalization under topological distribution shifts. The new model, functioning as a versatile graph Transformer, demonstrates superior performance across a wide range of graph learning tasks.

noise learning

data-free

transformer

Title: DyST: Towards Dynamic Neural Scene Representations on Real-World Videos. (arXiv:2310.06020v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06020
• Code URL: null
• Copy Paste: [[2310.06020]] DyST: Towards Dynamic Neural Scene Representations on Real-World Videos(http://arxiv.org/abs/2310.06020)
• Summary:

  Visual understanding of the world goes beyond the semantics and flat structure of individual images. In this work, we aim to capture both the 3D structure and dynamics of real-world scenes from monocular real-world videos. Our Dynamic Scene Transformer (DyST) model leverages recent work in neural scene representation to learn a latent decomposition of monocular real-world videos into scene content, per-view scene dynamics, and camera pose. This separation is achieved through a novel co-training scheme on monocular videos and our new synthetic dataset DySO. DyST learns tangible latent representations for dynamic scenes that enable view generation with separate control over the camera and the content of the scene.

Title: Factorized Tensor Networks for Multi-Task and Multi-Domain Learning. (arXiv:2310.06124v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06124
• Code URL: null
• Copy Paste: [[2310.06124]] Factorized Tensor Networks for Multi-Task and Multi-Domain Learning(http://arxiv.org/abs/2310.06124)
• Summary:

  Multi-task and multi-domain learning methods seek to learn multiple tasks/domains, jointly or one after another, using a single unified network. The key challenge and opportunity is to exploit shared information across tasks and domains to improve the efficiency of the unified network. The efficiency can be in terms of accuracy, storage cost, computation, or sample complexity. In this paper, we propose a factorized tensor network (FTN) that can achieve accuracy comparable to independent single-task/domain networks with a small number of additional parameters. FTN uses a frozen backbone network from a source model and incrementally adds task/domain-specific low-rank tensor factors to the shared frozen network. This approach can adapt to a large number of target domains and tasks without catastrophic forgetting. Furthermore, FTN requires a significantly smaller number of task-specific parameters compared to existing methods. We performed experiments on widely used multi-domain and multi-task datasets. We show the experiments on convolutional-based architecture with different backbones and on transformer-based architecture. We observed that FTN achieves similar accuracy as single-task/domain methods while using only a fraction of additional parameters per task.

Title: DiPS: Discriminative Pseudo-Label Sampling with Self-Supervised Transformers for Weakly Supervised Object Localization. (arXiv:2310.06196v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06196
• Code URL: https://github.com/shakeebmurtaza/dips
• Copy Paste: [[2310.06196]] DiPS: Discriminative Pseudo-Label Sampling with Self-Supervised Transformers for Weakly Supervised Object Localization(http://arxiv.org/abs/2310.06196)
• Summary:

  Self-supervised vision transformers (SSTs) have shown great potential to yield rich localization maps that highlight different objects in an image. However, these maps remain class-agnostic since the model is unsupervised. They often tend to decompose the image into multiple maps containing different objects while being unable to distinguish the object of interest from background noise objects. In this paper, Discriminative Pseudo-label Sampling (DiPS) is introduced to leverage these class-agnostic maps for weakly-supervised object localization (WSOL), where only image-class labels are available. Given multiple attention maps, DiPS relies on a pre-trained classifier to identify the most discriminative regions of each attention map. This ensures that the selected ROIs cover the correct image object while discarding the background ones, and, as such, provides a rich pool of diverse and discriminative proposals to cover different parts of the object. Subsequently, these proposals are used as pseudo-labels to train our new transformer-based WSOL model designed to perform classification and localization tasks. Unlike standard WSOL methods, DiPS optimizes performance in both tasks by using a transformer encoder and a dedicated output head for each task, each trained using dedicated loss functions. To avoid overfitting a single proposal and promote better object coverage, a single proposal is randomly selected among the top ones for a training image at each training step. Experimental results on the challenging CUB, ILSVRC, OpenImages, and TelDrone datasets indicate that our architecture, in combination with our transformer-based proposals, can yield better localization performance than state-of-the-art methods.

Title: Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing. (arXiv:2310.06234v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06234
• Code URL: https://github.com/davidyanande/arc
• Copy Paste: [[2310.06234]] Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing(http://arxiv.org/abs/2310.06234)
• Summary:

  The advent of high-capacity pre-trained models has revolutionized problem-solving in computer vision, shifting the focus from training task-specific models to adapting pre-trained models. Consequently, effectively adapting large pre-trained models to downstream tasks in an efficient manner has become a prominent research area. Existing solutions primarily concentrate on designing lightweight adapters and their interaction with pre-trained models, with the goal of minimizing the number of parameters requiring updates. In this study, we propose a novel Adapter Re-Composing (ARC) strategy that addresses efficient pre-trained model adaptation from a fresh perspective. Our approach considers the reusability of adaptation parameters and introduces a parameter-sharing scheme. Specifically, we leverage symmetric down-/up-projections to construct bottleneck operations, which are shared across layers. By learning low-dimensional re-scaling coefficients, we can effectively re-compose layer-adaptive adapters. This parameter-sharing strategy in adapter design allows us to significantly reduce the number of new parameters while maintaining satisfactory performance, thereby offering a promising approach to compress the adaptation cost. We conduct experiments on 24 downstream image classification tasks using various Vision Transformer variants to evaluate our method. The results demonstrate that our approach achieves compelling transfer learning performance with a reduced parameter count. Our code is available at \href{https://github.com/DavidYanAnDe/ARC}{https://github.com/DavidYanAnDe/ARC}.

Title: Exploring Embeddings for Measuring Text Relatedness: Unveiling Sentiments and Relationships in Online Comments. (arXiv:2310.05964v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.05964
• Code URL: null
• Copy Paste: [[2310.05964]] Exploring Embeddings for Measuring Text Relatedness: Unveiling Sentiments and Relationships in Online Comments(http://arxiv.org/abs/2310.05964)
• Summary:

  After a pandemic that caused internet usage to grow by 70%, there has been an increased number of people all across the world using social media. Applications like Twitter, Meta Threads, YouTube, and Reddit have become increasingly pervasive, leaving almost no digital space where public opinion is not expressed. This paper investigates sentiment and semantic relationships among comments across various social media platforms, as well as discusses the importance of shared opinions across these different media platforms, using word embeddings to analyze components in sentences and documents. It allows researchers, politicians, and business representatives to trace a path of shared sentiment among users across the world. This research paper presents multiple approaches that measure the relatedness of text extracted from user comments on these popular online platforms. By leveraging embeddings, which capture semantic relationships between words and help analyze sentiments across the web, we can uncover connections regarding public opinion as a whole. The study utilizes pre-existing datasets from YouTube, Reddit, Twitter, and more. We made use of popular natural language processing models like Bidirectional Encoder Representations from Transformers (BERT) to analyze sentiments and explore relationships between comment embeddings. Additionally, we aim to utilize clustering and Kl-divergence to find semantic relationships within these comment embeddings across various social media platforms. Our analysis will enable a deeper understanding of the interconnectedness of online comments and will investigate the notion of the internet functioning as a large interconnected brain.

Title: Multi-Modal Knowledge Graph Transformer Framework for Multi-Modal Entity Alignment. (arXiv:2310.06365v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06365
• Code URL: https://github.com/xiaoqian19940510/moalign
• Copy Paste: [[2310.06365]] Multi-Modal Knowledge Graph Transformer Framework for Multi-Modal Entity Alignment(http://arxiv.org/abs/2310.06365)
• Summary:

  Multi-Modal Entity Alignment (MMEA) is a critical task that aims to identify equivalent entity pairs across multi-modal knowledge graphs (MMKGs). However, this task faces challenges due to the presence of different types of information, including neighboring entities, multi-modal attributes, and entity types. Directly incorporating the above information (e.g., concatenation or attention) can lead to an unaligned information space. To address these challenges, we propose a novel MMEA transformer, called MoAlign, that hierarchically introduces neighbor features, multi-modal attributes, and entity types to enhance the alignment task. Taking advantage of the transformer's ability to better integrate multiple information, we design a hierarchical modifiable self-attention block in a transformer encoder to preserve the unique semantics of different information. Furthermore, we design two entity-type prefix injection methods to integrate entity-type information using type prefixes, which help to restrict the global information of entities not present in the MMKGs. Our extensive experiments on benchmark datasets demonstrate that our approach outperforms strong competitors and achieves excellent entity alignment performance.

Title: Transformers and Large Language Models for Chemistry and Drug Discovery. (arXiv:2310.06083v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06083
• Code URL: null
• Copy Paste: [[2310.06083]] Transformers and Large Language Models for Chemistry and Drug Discovery(http://arxiv.org/abs/2310.06083)
• Summary:

  Language modeling has seen impressive progress over the last years, mainly prompted by the invention of the Transformer architecture, sparking a revolution in many fields of machine learning, with breakthroughs in chemistry and biology. In this chapter, we explore how analogies between chemical and natural language have inspired the use of Transformers to tackle important bottlenecks in the drug discovery process, such as retrosynthetic planning and chemical space exploration. The revolution started with models able to perform particular tasks with a single type of data, like linearised molecular graphs, which then evolved to include other types of data, like spectra from analytical instruments, synthesis actions, and human language. A new trend leverages recent developments in large language models, giving rise to a wave of models capable of solving generic tasks in chemistry, all facilitated by the flexibility of natural language. As we continue to explore and harness these capabilities, we can look forward to a future where machine learning plays an even more integral role in accelerating scientific discovery.

Title: Predicting Three Types of Freezing of Gait Events Using Deep Learning Models. (arXiv:2310.06322v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06322
• Code URL: null
• Copy Paste: [[2310.06322]] Predicting Three Types of Freezing of Gait Events Using Deep Learning Models(http://arxiv.org/abs/2310.06322)
• Summary:

  Freezing of gait is a Parkinson's Disease symptom that episodically inflicts a patient with the inability to step or turn while walking. While medical experts have discovered various triggers and alleviating actions for freezing of gait, the underlying causes and prediction models are still being explored today. Current freezing of gait prediction models that utilize machine learning achieve high sensitivity and specificity in freezing of gait predictions based on time-series data; however, these models lack specifications on the type of freezing of gait events. We develop various deep learning models using the transformer encoder architecture plus Bidirectional LSTM layers and different feature sets to predict the three different types of freezing of gait events. The best performing model achieves a score of 0.427 on testing data, which would rank top 5 in Kaggle's Freezing of Gait prediction competition, hosted by THE MICHAEL J. FOX FOUNDATION. However, we also recognize overfitting in training data that could be potentially improved through pseudo labelling on additional data and model architecture simplification.

generative

Title: Quantile-based Maximum Likelihood Training for Outlier Detection. (arXiv:2310.06085v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06085
• Code URL: null
• Copy Paste: [[2310.06085]] Quantile-based Maximum Likelihood Training for Outlier Detection(http://arxiv.org/abs/2310.06085)
• Summary:

  Discriminative learning effectively predicts true object class for image classification. However, it often results in false positives for outliers, posing critical concerns in applications like autonomous driving and video surveillance systems. Previous attempts to address this challenge involved training image classifiers through contrastive learning using actual outlier data or synthesizing outliers for self-supervised learning. Furthermore, unsupervised generative modeling of inliers in pixel space has shown limited success for outlier detection. In this work, we introduce a quantile-based maximum likelihood objective for learning the inlier distribution to improve the outlier separation during inference. Our approach fits a normalizing flow to pre-trained discriminative features and detects the outliers according to the evaluated log-likelihood. The experimental evaluation demonstrates the effectiveness of our method as it surpasses the performance of the state-of-the-art unsupervised methods for outlier detection. The results are also competitive compared with a recent self-supervised approach for outlier detection. Our work allows to reduce dependency on well-sampled negative training data, which is especially important for domains like medical diagnostics or remote sensing.

Title: Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understanding. (arXiv:2310.06103v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06103
• Code URL: https://github.com/digitalphonetics/multilingual-seq2seq-slu
• Copy Paste: [[2310.06103]] Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understanding(http://arxiv.org/abs/2310.06103)
• Summary:

  A number of methods have been proposed for End-to-End Spoken Language Understanding (E2E-SLU) using pretrained models, however their evaluation often lacks multilingual setup and tasks that require prediction of lexical fillers, such as slot filling. In this work, we propose a unified method that integrates multilingual pretrained speech and text models and performs E2E-SLU on six datasets in four languages in a generative manner, including the prediction of lexical fillers. We investigate how the proposed method can be improved by pretraining on widely available speech recognition data using several training objectives. Pretraining on 7000 hours of multilingual data allows us to outperform the state-of-the-art ultimately on two SLU datasets and partly on two more SLU datasets. Finally, we examine the cross-lingual capabilities of the proposed model and improve on the best known result on the PortMEDIA-Language dataset by almost half, achieving a Concept/Value Error Rate of 23.65%.

Title: Hexa: Self-Improving for Knowledge-Grounded Dialogue System. (arXiv:2310.06404v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06404
• Code URL: null
• Copy Paste: [[2310.06404]] Hexa: Self-Improving for Knowledge-Grounded Dialogue System(http://arxiv.org/abs/2310.06404)
• Summary:

  A common practice in knowledge-grounded dialogue generation is to explicitly utilize intermediate steps (e.g., web-search, memory retrieval) with modular approaches. However, data for such steps are often inaccessible compared to those of dialogue responses as they are unobservable in an ordinary dialogue. To fill in the absence of these data, we develop a self-improving method to improve the generative performances of intermediate steps without the ground truth data. In particular, we propose a novel bootstrapping scheme with a guided prompt and a modified loss function to enhance the diversity of appropriate self-generated responses. Through experiments on various benchmark datasets, we empirically demonstrate that our method successfully leverages a self-improving mechanism in generating intermediate and final responses and improves the performances on the task of knowledge-grounded dialogue generation.

Title: Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition. (arXiv:2310.06434v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06434
• Code URL: https://github.com/srijith-rkr/whispering-llama
• Copy Paste: [[2310.06434]] Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition(http://arxiv.org/abs/2310.06434)
• Summary:

  We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR). Our methodology leverages both acoustic information and external linguistic representations to generate accurate speech transcription contexts. This marks a step towards a fresh paradigm in generative error correction within the realm of n-best hypotheses. Unlike the existing ranking-based rescoring methods, our approach adeptly uses distinct initialization techniques and parameter-efficient algorithms to boost ASR performance derived from pre-trained speech and text models. Through evaluation across diverse ASR datasets, we evaluate the stability and reproducibility of our fusion technique, demonstrating its improved word error rate relative (WERR) performance in comparison to n-best hypotheses by relatively 37.66%. To encourage future research, we have made our code and pre-trained models open source at https://github.com/Srijith-rkr/Whispering-LLaMA.

Title: Generative ensemble deep learning severe weather prediction from a deterministic convection-allowing model. (arXiv:2310.06045v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06045
• Code URL: https://github.com/yingkaisha/severe_weather_cgan
• Copy Paste: [[2310.06045]] Generative ensemble deep learning severe weather prediction from a deterministic convection-allowing model(http://arxiv.org/abs/2310.06045)
• Summary:

  An ensemble post-processing method is developed for the probabilistic prediction of severe weather (tornadoes, hail, and wind gusts) over the conterminous United States (CONUS). The method combines conditional generative adversarial networks (CGANs), a type of deep generative model, with a convolutional neural network (CNN) to post-process convection-allowing model (CAM) forecasts. The CGANs are designed to create synthetic ensemble members from deterministic CAM forecasts, and their outputs are processed by the CNN to estimate the probability of severe weather. The method is tested using High-Resolution Rapid Refresh (HRRR) 1--24 hr forecasts as inputs and Storm Prediction Center (SPC) severe weather reports as targets. The method produced skillful predictions with up to 20% Brier Skill Score (BSS) increases compared to other neural-network-based reference methods using a testing dataset of HRRR forecasts in 2021. For the evaluation of uncertainty quantification, the method is overconfident but produces meaningful ensemble spreads that can distinguish good and bad forecasts. The quality of CGAN outputs is also evaluated. Results show that the CGAN outputs behave similarly to a numerical ensemble; they preserved the inter-variable correlations and the contribution of influential predictors as in the original HRRR forecasts. This work provides a novel approach to post-process CAM output using neural networks that can be applied to severe weather prediction.

Title: When is Agnostic Reinforcement Learning Statistically Tractable?. (arXiv:2310.06113v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06113
• Code URL: null
• Copy Paste: [[2310.06113]] When is Agnostic Reinforcement Learning Statistically Tractable?(http://arxiv.org/abs/2310.06113)
• Summary:

  We study the problem of agnostic PAC reinforcement learning (RL): given a policy class $\Pi$, how many rounds of interaction with an unknown MDP (with a potentially large state and action space) are required to learn an $\epsilon$-suboptimal policy with respect to $\Pi$? Towards that end, we introduce a new complexity measure, called the \emph{spanning capacity}, that depends solely on the set $\Pi$ and is independent of the MDP dynamics. With a generative model, we show that for any policy class $\Pi$, bounded spanning capacity characterizes PAC learnability. However, for online RL, the situation is more subtle. We show there exists a policy class $\Pi$ with a bounded spanning capacity that requires a superpolynomial number of samples to learn. This reveals a surprising separation for agnostic learnability between generative access and online access models (as well as between deterministic/stochastic MDPs under online access). On the positive side, we identify an additional \emph{sunflower} structure, which in conjunction with bounded spanning capacity enables statistically efficient online RL via a new algorithm called POPLER, which takes inspiration from classical importance sampling methods as well as techniques for reachable-state identification and policy evaluation in reward-free exploration.

large language model

Title: MuseChat: A Conversational Music Recommendation System for Videos. (arXiv:2310.06282v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06282
• Code URL: null
• Copy Paste: [[2310.06282]] MuseChat: A Conversational Music Recommendation System for Videos(http://arxiv.org/abs/2310.06282)
• Summary:

  We introduce MuseChat, an innovative dialog-based music recommendation system. This unique platform not only offers interactive user engagement but also suggests music tailored for input videos, so that users can refine and personalize their music selections. In contrast, previous systems predominantly emphasized content compatibility, often overlooking the nuances of users' individual preferences. For example, all the datasets only provide basic music-video pairings or such pairings with textual music descriptions. To address this gap, our research offers three contributions. First, we devise a conversation-synthesis method that simulates a two-turn interaction between a user and a recommendation system, which leverages pre-trained music tags and artist information. In this interaction, users submit a video to the system, which then suggests a suitable music piece with a rationale. Afterwards, users communicate their musical preferences, and the system presents a refined music recommendation with reasoning. Second, we introduce a multi-modal recommendation engine that matches music either by aligning it with visual cues from the video or by harmonizing visual information, feedback from previously recommended music, and the user's textual input. Third, we bridge music representations and textual data with a Large Language Model(Vicuna-7B). This alignment equips MuseChat to deliver music recommendations and their underlying reasoning in a manner resembling human communication. Our evaluations show that MuseChat surpasses existing state-of-the-art models in music retrieval tasks and pioneers the integration of the recommendation process within a natural language framework.

Title: BYOC: Personalized Few-Shot Classification with Co-Authored Class Descriptions. (arXiv:2310.06111v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06111
• Code URL: null
• Copy Paste: [[2310.06111]] BYOC: Personalized Few-Shot Classification with Co-Authored Class Descriptions(http://arxiv.org/abs/2310.06111)
• Summary:

  Text classification is a well-studied and versatile building block for many NLP applications. Yet, existing approaches require either large annotated corpora to train a model with or, when using large language models as a base, require carefully crafting the prompt as well as using a long context that can fit many examples. As a result, it is not possible for end-users to build classifiers for themselves. To address this issue, we propose a novel approach to few-shot text classification using an LLM. Rather than few-shot examples, the LLM is prompted with descriptions of the salient features of each class. These descriptions are coauthored by the user and the LLM interactively: while the user annotates each few-shot example, the LLM asks relevant questions that the user answers. Examples, questions, and answers are summarized to form the classification prompt. Our experiments show that our approach yields high accuracy classifiers, within 82% of the performance of models trained with significantly larger datasets while using only 1% of their training sets. Additionally, in a study with 30 participants, we show that end-users are able to build classifiers to suit their specific needs. The personalized classifiers show an average accuracy of 90%, which is 15% higher than the state-of-the-art approach.

Title: Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models. (arXiv:2310.06117v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06117
• Code URL: null
• Copy Paste: [[2310.06117]] Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models(http://arxiv.org/abs/2310.06117)
• Summary:

  We present Step-Back Prompting, a simple prompting technique that enables LLMs to do abstractions to derive high-level concepts and first principles from instances containing specific details. Using the concepts and principles to guide the reasoning steps, LLMs significantly improve their abilities in following a correct reasoning path towards the solution. We conduct experiments of Step-Back Prompting with PaLM-2L models and observe substantial performance gains on a wide range of challenging reasoning-intensive tasks including STEM, Knowledge QA, and Multi-Hop Reasoning. For instance, Step-Back Prompting improves PaLM-2L performance on MMLU Physics and Chemistry by 7% and 11%, TimeQA by 27%, and MuSiQue by 7%.

Title: Compressing Context to Enhance Inference Efficiency of Large Language Models. (arXiv:2310.06201v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06201
• Code URL: https://github.com/liyucheng09/selective_context
• Copy Paste: [[2310.06201]] Compressing Context to Enhance Inference Efficiency of Large Language Models(http://arxiv.org/abs/2310.06201)
• Summary:

  Large language models (LLMs) achieved remarkable performance across various tasks. However, they face challenges in managing long documents and extended conversations, due to significantly increased computational requirements, both in memory and inference time, and potential context truncation when the input exceeds the LLM's fixed context length. This paper proposes a method called Selective Context that enhances the inference efficiency of LLMs by identifying and pruning redundancy in the input context to make the input more compact. We test our approach using common data sources requiring long context processing: arXiv papers, news articles, and long conversations, on tasks of summarisation, question answering, and response generation. Experimental results show that Selective Context significantly reduces memory cost and decreases generation latency while maintaining comparable performance compared to that achieved when full context is used. Specifically, we achieve a 50\% reduction in context cost, resulting in a 36\% reduction in inference memory usage and a 32\% reduction in inference time, while observing only a minor drop of .023 in BERTscore and .038 in faithfulness on four downstream applications, indicating that our method strikes a good balance between efficiency and performance.

Title: GPT-who: An Information Density-based Machine-Generated Text Detector. (arXiv:2310.06202v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06202
• Code URL: null
• Copy Paste: [[2310.06202]] GPT-who: An Information Density-based Machine-Generated Text Detector(http://arxiv.org/abs/2310.06202)
• Summary:

  The Uniform Information Density principle posits that humans prefer to spread information evenly during language production. In this work, we examine if the UID principle can help capture differences between Large Language Models (LLMs) and human-generated text. We propose GPT-who, the first psycholinguistically-aware multi-class domain-agnostic statistical-based detector. This detector employs UID-based features to model the unique statistical signature of each LLM and human author for accurate authorship attribution. We evaluate our method using 4 large-scale benchmark datasets and find that GPT-who outperforms state-of-the-art detectors (both statistical- & non-statistical-based) such as GLTR, GPTZero, OpenAI detector, and ZeroGPT by over $20$% across domains. In addition to superior performance, it is computationally inexpensive and utilizes an interpretable representation of text articles. We present the largest analysis of the UID-based representations of human and machine-generated texts (over 400k articles) to demonstrate how authors distribute information differently, and in ways that enable their detection using an off-the-shelf LM without any fine-tuning. We find that GPT-who can distinguish texts generated by very sophisticated LLMs, even when the overlying text is indiscernible.

Title: GeoLLM: Extracting Geospatial Knowledge from Large Language Models. (arXiv:2310.06213v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06213
• Code URL: null
• Copy Paste: [[2310.06213]] GeoLLM: Extracting Geospatial Knowledge from Large Language Models(http://arxiv.org/abs/2310.06213)
• Summary:

  The application of machine learning (ML) in a range of geospatial tasks is increasingly common but often relies on globally available covariates such as satellite imagery that can either be expensive or lack predictive power. Here we explore the question of whether the vast amounts of knowledge found in Internet language corpora, now compressed within large language models (LLMs), can be leveraged for geospatial prediction tasks. We first demonstrate that LLMs embed remarkable spatial information about locations, but naively querying LLMs using geographic coordinates alone is ineffective in predicting key indicators like population density. We then present GeoLLM, a novel method that can effectively extract geospatial knowledge from LLMs with auxiliary map data from OpenStreetMap. We demonstrate the utility of our approach across multiple tasks of central interest to the international community, including the measurement of population density and economic livelihoods. Across these tasks, our method demonstrates a 70% improvement in performance (measured using Pearson's $r^2$) relative to baselines that use nearest neighbors or use information directly from the prompt, and performance equal to or exceeding satellite-based benchmarks in the literature. With GeoLLM, we observe that GPT-3.5 outperforms Llama 2 and RoBERTa by 19% and 51% respectively, suggesting that the performance of our method scales well with the size of the model and its pretraining dataset. Our experiments reveal that LLMs are remarkably sample-efficient, rich in geospatial information, and robust across the globe. Crucially, GeoLLM shows promise in mitigating the limitations of existing geospatial covariates and complementing them well.

Title: Get the gist? Using large language models for few-shot decontextualization. (arXiv:2310.06254v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06254
• Code URL: null
• Copy Paste: [[2310.06254]] Get the gist? Using large language models for few-shot decontextualization(http://arxiv.org/abs/2310.06254)
• Summary:

  In many NLP applications that involve interpreting sentences within a rich context -- for instance, information retrieval systems or dialogue systems -- it is desirable to be able to preserve the sentence in a form that can be readily understood without context, for later reuse -- a process known as ``decontextualization''. While previous work demonstrated that generative Seq2Seq models could effectively perform decontextualization after being fine-tuned on a specific dataset, this approach requires expensive human annotations and may not transfer to other domains. We propose a few-shot method of decontextualization using a large language model, and present preliminary results showing that this method achieves viable performance on multiple domains using only a small set of examples.

Title: Towards Mitigating Hallucination in Large Language Models via Self-Reflection. (arXiv:2310.06271v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06271
• Code URL: null
• Copy Paste: [[2310.06271]] Towards Mitigating Hallucination in Large Language Models via Self-Reflection(http://arxiv.org/abs/2310.06271)
• Summary:

  Large language models (LLMs) have shown promise for generative and knowledge-intensive tasks including question-answering (QA) tasks. However, the practical deployment still faces challenges, notably the issue of "hallucination", where models generate plausible-sounding but unfaithful or nonsensical information. This issue becomes particularly critical in the medical domain due to the uncommon professional concepts and potential social risks involved. This paper analyses the phenomenon of hallucination in medical generative QA systems using widely adopted LLMs and datasets. Our investigation centers on the identification and comprehension of common problematic answers, with a specific emphasis on hallucination. To tackle this challenge, we present an interactive self-reflection methodology that incorporates knowledge acquisition and answer generation. Through this feedback process, our approach steadily enhances the factuality, consistency, and entailment of the generated answers. Consequently, we harness the interactivity and multitasking ability of LLMs and produce progressively more precise and accurate answers. Experimental results on both automatic and human evaluation demonstrate the superiority of our approach in hallucination reduction compared to baselines.

Title: Selective Demonstrations for Cross-domain Text-to-SQL. (arXiv:2310.06302v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06302
• Code URL: null
• Copy Paste: [[2310.06302]] Selective Demonstrations for Cross-domain Text-to-SQL(http://arxiv.org/abs/2310.06302)
• Summary:

  Large language models (LLMs) with in-context learning have demonstrated impressive generalization capabilities in the cross-domain text-to-SQL task, without the use of in-domain annotations. However, incorporating in-domain demonstration examples has been found to greatly enhance LLMs' performance. In this paper, we delve into the key factors within in-domain examples that contribute to the improvement and explore whether we can harness these benefits without relying on in-domain annotations. Based on our findings, we propose a demonstration selection framework ODIS which utilizes both out-of-domain examples and synthetically generated in-domain examples to construct demonstrations. By retrieving demonstrations from hybrid sources, ODIS leverages the advantages of both, showcasing its effectiveness compared to baseline methods that rely on a single data source. Furthermore, ODIS outperforms state-of-the-art approaches on two cross-domain text-to-SQL datasets, with improvements of 1.1 and 11.8 points in execution accuracy, respectively.

Title: Large Language Models for Propaganda Detection. (arXiv:2310.06422v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06422
• Code URL: https://github.com/submissionemnlp/llm_propaganda_detection
• Copy Paste: [[2310.06422]] Large Language Models for Propaganda Detection(http://arxiv.org/abs/2310.06422)
• Summary:

  The prevalence of propaganda in our digital society poses a challenge to societal harmony and the dissemination of truth. Detecting propaganda through NLP in text is challenging due to subtle manipulation techniques and contextual dependencies. To address this issue, we investigate the effectiveness of modern Large Language Models (LLMs) such as GPT-3 and GPT-4 for propaganda detection. We conduct experiments using the SemEval-2020 task 11 dataset, which features news articles labeled with 14 propaganda techniques as a multi-label classification problem. Five variations of GPT-3 and GPT-4 are employed, incorporating various prompt engineering and fine-tuning strategies across the different models. We evaluate the models' performance by assessing metrics such as $F1$ score, $Precision$, and $Recall$, comparing the results with the current state-of-the-art approach using RoBERTa. Our findings demonstrate that GPT-4 achieves comparable results to the current state-of-the-art. Further, this study analyzes the potential and challenges of LLMs in complex tasks like propaganda detection.

Title: Constructive Large Language Models Alignment with Diverse Feedback. (arXiv:2310.06450v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2310.06450
• Code URL: null
• Copy Paste: [[2310.06450]] Constructive Large Language Models Alignment with Diverse Feedback(http://arxiv.org/abs/2310.06450)
• Summary:

  In recent research on large language models (LLMs), there has been a growing emphasis on aligning these models with human values to reduce the impact of harmful content. However, current alignment methods often rely solely on singular forms of human feedback, such as preferences, annotated labels, or natural language critiques, overlooking the potential advantages of combining these feedback types. This limitation leads to suboptimal performance, even when ample training data is available. In this paper, we introduce Constructive and Diverse Feedback (CDF) as a novel method to enhance LLM alignment, inspired by constructivist learning theory. Our approach involves collecting three distinct types of feedback tailored to problems of varying difficulty levels within the training dataset. Specifically, we exploit critique feedback for easy problems, refinement feedback for medium problems, and preference feedback for hard problems. By training our model with this diversified feedback, we achieve enhanced alignment performance while using less training data. To assess the effectiveness of CDF, we evaluate it against previous methods in three downstream tasks: question answering, dialog generation, and text summarization. Experimental results demonstrate that CDF achieves superior performance even with a smaller training dataset.

Title: Rethinking Memory and Communication Cost for Efficient Large Language Model Training. (arXiv:2310.06003v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06003
• Code URL: null
• Copy Paste: [[2310.06003]] Rethinking Memory and Communication Cost for Efficient Large Language Model Training(http://arxiv.org/abs/2310.06003)
• Summary:

  As model sizes and training datasets continue to increase, large-scale model training frameworks reduce memory consumption by various sharding techniques. However, the huge communication overhead reduces the training efficiency, especially in public cloud environments with varying network bandwidths. In this paper, we rethink the impact of memory consumption and communication overhead on the training speed of large language model, and propose a memory-communication balanced \underline{Pa}rtial \underline{R}edundancy \underline{O}ptimizer (PaRO). PaRO reduces the amount and frequency of inter-group communication by grouping GPU clusters and introducing minor intra-group memory redundancy, thereby improving the training efficiency of the model. Additionally, we propose a Hierarchical Overlapping Ring (HO-Ring) communication topology to enhance communication efficiency between nodes or across switches in large model training. Our experiments demonstrate that the HO-Ring algorithm improves communication efficiency by 32.6\% compared to the traditional Ring algorithm. Compared to the baseline ZeRO, PaRO significantly improves training throughput by 1.2x-2.6x and achieves a near-linear scalability. Therefore, the PaRO strategy provides more fine-grained options for the trade-off between memory consumption and communication overhead in different training scenarios.

Title: Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond. (arXiv:2310.06147v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2310.06147
• Code URL: null
• Copy Paste: [[2310.06147]] Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond(http://arxiv.org/abs/2310.06147)
• Summary:

  Recent advancements in Large Language Models (LLMs) have garnered wide attention and led to successful products such as ChatGPT and GPT-4. Their proficiency in adhering to instructions and delivering harmless, helpful, and honest (3H) responses can largely be attributed to the technique of Reinforcement Learning from Human Feedback (RLHF). In this paper, we aim to link the research in conventional RL to RL techniques used in LLM research. Demystify this technique by discussing why, when, and how RL excels. Furthermore, we explore potential future avenues that could either benefit from or contribute to RLHF research.

  Highlighted Takeaways:

  1. RLHF is Online Inverse RL with Offline Demonstration Data.

  2. RLHF $>$ SFT because Imitation Learning (and Inverse RL) $>$ Behavior Cloning (BC) by alleviating the problem of compounding error.

  3. The RM step in RLHF generates a proxy of the expensive human feedback, such an insight can be generalized to other LLM tasks such as prompting evaluation and optimization where feedback is also expensive.

  4. The policy learning in RLHF is more challenging than conventional problems studied in IRL due to their high action dimensionality and feedback sparsity.

  5. The main superiority of PPO over off-policy value-based methods is its stability gained from (almost) on-policy data and conservative policy updates.

segmentation

Title: CoBEVFusion: Cooperative Perception with LiDAR-Camera Bird's-Eye View Fusion. (arXiv:2310.06008v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06008
• Code URL: null
• Copy Paste: [[2310.06008]] CoBEVFusion: Cooperative Perception with LiDAR-Camera Bird's-Eye View Fusion(http://arxiv.org/abs/2310.06008)
• Summary:

  Autonomous Vehicles (AVs) use multiple sensors to gather information about their surroundings. By sharing sensor data between Connected Autonomous Vehicles (CAVs), the safety and reliability of these vehicles can be improved through a concept known as cooperative perception. However, recent approaches in cooperative perception only share single sensor information such as cameras or LiDAR. In this research, we explore the fusion of multiple sensor data sources and present a framework, called CoBEVFusion, that fuses LiDAR and camera data to create a Bird's-Eye View (BEV) representation. The CAVs process the multi-modal data locally and utilize a Dual Window-based Cross-Attention (DWCA) module to fuse the LiDAR and camera features into a unified BEV representation. The fused BEV feature maps are shared among the CAVs, and a 3D Convolutional Neural Network is applied to aggregate the features from the CAVs. Our CoBEVFusion framework was evaluated on the cooperative perception dataset OPV2V for two perception tasks: BEV semantic segmentation and 3D object detection. The results show that our DWCA LiDAR-camera fusion model outperforms perception models with single-modal data and state-of-the-art BEV fusion models. Our overall cooperative perception architecture, CoBEVFusion, also achieves comparable performance with other cooperative perception models.

Title: CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation. (arXiv:2310.06368v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2310.06368
• Code URL: https://github.com/zkzhang98/coinseg
• Copy Paste: [[2310.06368]] CoinSeg: Contrast Inter- and Intra- Class Representations for Incremental Segmentation(http://arxiv.org/abs/2310.06368)
• Summary:

  Class incremental semantic segmentation aims to strike a balance between the model's stability and plasticity by maintaining old knowledge while adapting to new concepts. However, most state-of-the-art methods use the freeze strategy for stability, which compromises the model's plasticity.In contrast, releasing parameter training for plasticity could lead to the best performance for all categories, but this requires discriminative feature representation.Therefore, we prioritize the model's plasticity and propose the Contrast inter- and intra-class representations for Incremental Segmentation (CoinSeg), which pursues discriminative representations for flexible parameter tuning. Inspired by the Gaussian mixture model that samples from a mixture of Gaussian distributions, CoinSeg emphasizes intra-class diversity with multiple contrastive representation centroids. Specifically, we use mask proposals to identify regions with strong objectness that are likely to be diverse instances/centroids of a category. These mask proposals are then used for contrastive representations to reinforce intra-class diversity. Meanwhile, to avoid bias from intra-class diversity, we also apply category-level pseudo-labels to enhance category-level consistency and inter-category diversity. Additionally, CoinSeg ensures the model's stability and alleviates forgetting through a specific flexible tuning strategy. We validate CoinSeg on Pascal VOC 2012 and ADE20K datasets with multiple incremental scenarios and achieve superior results compared to previous state-of-the-art methods, especially in more challenging and realistic long-term scenarios. Code is available at https://github.com/zkzhang98/CoinSeg.