2024-01-02

secure

Title: Realizing Open and Decentralized Marketplace for Exchanging Data of Expected IoT Behaviors. (arXiv:2401.00141v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00141
• Code URL: null
• Copy Paste: [[2401.00141]] Realizing Open and Decentralized Marketplace for Exchanging Data of Expected IoT Behaviors(http://arxiv.org/abs/2401.00141)
• Summary:

  With rising concerns about the security of IoT devices, network operators need better ways to handle potential risks. Luckily, IoT devices show consistent patterns in how they communicate. But despite previous efforts, it remains unclear how knowledge of these patterns can be made available. As data marketplaces become popular in different domains, this paper1 proposes creating a special marketplace focused on IoT cybersecurity. The goal is to openly share knowledge about IoT devices' behavior, using structured data formats like Manufacturer Usage Description (MUD) files. To make this work, we employ technologies like blockchain and smart contracts to build a practical and secure foundation for sharing and accessing important information about how IoT devices should behave on the network. Our contributions are two-fold. (1) We identify the essential features of an effective marketplace for sharing data related to the expected behaviors of IoT devices. We develop a smart contract on the Ethereum blockchain with five concrete functions; and, (2) We implement a prototype of our marketplace in a private chain environment-our codes are publicly released. We demonstrate how effectively our marketplace functions through experiments involving MUD files from consumer IoT devices. Our marketplace enables suppliers and consumers to share MUD data on the Ethereum blockchain for under a hundred dollars, promoting accessibility and participation.

Title: HexE -- Securing Audio Contents in Voice Chat using Puzzle and Timestamp. (arXiv:2401.00765v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00765
• Code URL: null
• Copy Paste: [[2401.00765]] HexE -- Securing Audio Contents in Voice Chat using Puzzle and Timestamp(http://arxiv.org/abs/2401.00765)
• Summary:

  Cryptography is the study of securing information. It is the physical process that scrambles the information by rearrangement and substitution of content, so that it becomes difficult for anyone to understand. In today's world, security has become an inevitable part of our day-to-day life, right from normal browsing to performing critical payment transactions. Hackers work endlessly to break the security present in the apps/websites on which we perform day-to-day operations and salvage valuable information. Because of this, many illegal activities have taken place which affect the user. One such illegal activity is tapping the voice communication between two users. If left unencrypted, the communication between the users is compromised, thereby causing issues. One way to prevent this act is to encrypt the audio in that the contents cannot have tampered with unless the receiver has the valid key to decrypt it. The proposed solution termed "HexE" aims to create a puzzle-based algorithm which would encrypt and decrypt the audio files without manipulating the file header, thus securing the contents. The algorithm works on an NxN SuDoKu-based puzzle which is accepted both by the sender and receiver. Using the timestamp of the event (UNIX based), a grid from the puzzle is chosen which in turn will act as the key for both encryption and decryption. If the timestamp is slightly adjusted, the process will end up in failure during decryption, thus ensuring confidentiality. Another approach to secure the audio files is to implement IPFS (Inter Planetary File System) alongside the puzzle algorithm in which the encrypted audio is stored on it and the receiver can fetch the audio provided if the valid IPFS Hash of the file is present. In this way, the audio file is secured.

security

Title: SSL-OTA: Unveiling Backdoor Threats in Self-Supervised Learning for Object Detection. (arXiv:2401.00137v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00137
• Code URL: null
• Copy Paste: [[2401.00137]] SSL-OTA: Unveiling Backdoor Threats in Self-Supervised Learning for Object Detection(http://arxiv.org/abs/2401.00137)
• Summary:

  The extensive adoption of Self-supervised learning (SSL) has led to an increased security threat from backdoor attacks. While existing research has mainly focused on backdoor attacks in image classification, there has been limited exploration into their implications for object detection. In this work, we propose the first backdoor attack designed for object detection tasks in SSL scenarios, termed Object Transform Attack (SSL-OTA). SSL-OTA employs a trigger capable of altering predictions of the target object to the desired category, encompassing two attacks: Data Poisoning Attack (NA) and Dual-Source Blending Attack (DSBA). NA conducts data poisoning during downstream fine-tuning of the object detector, while DSBA additionally injects backdoors into the pre-trained encoder. We establish appropriate metrics and conduct extensive experiments on benchmark datasets, demonstrating the effectiveness and utility of our proposed attack. Notably, both NA and DSBA achieve high attack success rates (ASR) at extremely low poisoning rates (0.5%). The results underscore the importance of considering backdoor threats in SSL-based object detection and contribute a novel perspective to the field.

Title: Is It Possible to Backdoor Face Forgery Detection with Natural Triggers?. (arXiv:2401.00414v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00414
• Code URL: null
• Copy Paste: [[2401.00414]] Is It Possible to Backdoor Face Forgery Detection with Natural Triggers?(http://arxiv.org/abs/2401.00414)
• Summary:

  Deep neural networks have significantly improved the performance of face forgery detection models in discriminating Artificial Intelligent Generated Content (AIGC). However, their security is significantly threatened by the injection of triggers during model training (i.e., backdoor attacks). Although existing backdoor defenses and manual data selection can mitigate those using human-eye-sensitive triggers, such as patches or adversarial noises, the more challenging natural backdoor triggers remain insufficiently researched. To further investigate natural triggers, we propose a novel analysis-by-synthesis backdoor attack against face forgery detection models, which embeds natural triggers in the latent space. We thoroughly study such backdoor vulnerability from two perspectives: (1) Model Discrimination (Optimization-Based Trigger): we adopt a substitute detection model and find the trigger by minimizing the cross-entropy loss; (2) Data Distribution (Custom Trigger): we manipulate the uncommon facial attributes in the long-tailed distribution to generate poisoned samples without the supervision from detection models. Furthermore, to completely evaluate the detection models towards the latest AIGC, we utilize both state-of-the-art StyleGAN and Stable Diffusion for trigger generation. Finally, these backdoor triggers introduce specific semantic features to the generated poisoned samples (e.g., skin textures and smile), which are more natural and robust. Extensive experiments show that our method is superior from three levels: (1) Attack Success Rate: ours achieves a high attack success rate (over 99%) and incurs a small model accuracy drop (below 0.2%) with a low poisoning rate (less than 3%); (2) Backdoor Defense: ours shows better robust performance when faced with existing backdoor defense methods; (3) Human Inspection: ours is less human-eye-sensitive from a comprehensive user study.

Title: Advancing TTP Analysis: Harnessing the Power of Encoder-Only and Decoder-Only Language Models with Retrieval Augmented Generation. (arXiv:2401.00280v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00280
• Code URL: null
• Copy Paste: [[2401.00280]] Advancing TTP Analysis: Harnessing the Power of Encoder-Only and Decoder-Only Language Models with Retrieval Augmented Generation(http://arxiv.org/abs/2401.00280)
• Summary:

  Tactics, Techniques, and Procedures (TTPs) outline the methods attackers use to exploit vulnerabilities. The interpretation of TTPs in the MITRE ATT&CK framework can be challenging for cybersecurity practitioners due to presumed expertise, complex dependencies, and inherent ambiguity. Meanwhile, advancements with Large Language Models (LLMs) have led to recent surge in studies exploring its uses in cybersecurity operations. This leads us to question how well encoder-only (e.g., RoBERTa) and decoder-only (e.g., GPT-3.5) LLMs can comprehend and summarize TTPs to inform analysts of the intended purposes (i.e., tactics) of a cyberattack procedure. The state-of-the-art LLMs have shown to be prone to hallucination by providing inaccurate information, which is problematic in critical domains like cybersecurity. Therefore, we propose the use of Retrieval Augmented Generation (RAG) techniques to extract relevant contexts for each cyberattack procedure for decoder-only LLMs (without fine-tuning). We further contrast such approach against supervised fine-tuning (SFT) of encoder-only LLMs. Our results reveal that both the direct-use of decoder-only LLMs (i.e., its pre-trained knowledge) and the SFT of encoder-only LLMs offer inaccurate interpretation of cyberattack procedures. Significant improvements are shown when RAG is used for decoder-only LLMs, particularly when directly relevant context is found. This study further sheds insights on the limitations and capabilities of using RAG for LLMs in interpreting TTPs.

Title: Autonomous Threat Hunting: A Future Paradigm for AI-Driven Threat Intelligence. (arXiv:2401.00286v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00286
• Code URL: null
• Copy Paste: [[2401.00286]] Autonomous Threat Hunting: A Future Paradigm for AI-Driven Threat Intelligence(http://arxiv.org/abs/2401.00286)
• Summary:

  The evolution of cybersecurity has spurred the emergence of autonomous threat hunting as a pivotal paradigm in the realm of AI-driven threat intelligence. This review navigates through the intricate landscape of autonomous threat hunting, exploring its significance and pivotal role in fortifying cyber defense mechanisms. Delving into the amalgamation of artificial intelligence (AI) and traditional threat intelligence methodologies, this paper delineates the necessity and evolution of autonomous approaches in combating contemporary cyber threats. Through a comprehensive exploration of foundational AI-driven threat intelligence, the review accentuates the transformative influence of AI and machine learning on conventional threat intelligence practices. It elucidates the conceptual framework underpinning autonomous threat hunting, spotlighting its components, and the seamless integration of AI algorithms within threat hunting processes.. Insightful discussions on challenges encompassing scalability, interpretability, and ethical considerations in AI-driven models enrich the discourse. Moreover, through illuminating case studies and evaluations, this paper showcases real-world implementations, underscoring success stories and lessons learned by organizations adopting AI-driven threat intelligence. In conclusion, this review consolidates key insights, emphasizing the substantial implications of autonomous threat hunting for the future of cybersecurity. It underscores the significance of continual research and collaborative efforts in harnessing the potential of AI-driven approaches to fortify cyber defenses against evolving threats.

Title: Blockchain and Deep Learning-Based IDS for Securing SDN-Enabled Industrial IoT Environments. (arXiv:2401.00468v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00468
• Code URL: null
• Copy Paste: [[2401.00468]] Blockchain and Deep Learning-Based IDS for Securing SDN-Enabled Industrial IoT Environments(http://arxiv.org/abs/2401.00468)
• Summary:

  The industrial Internet of Things (IIoT) involves the integration of Internet of Things (IoT) technologies into industrial settings. However, given the high sensitivity of the industry to the security of industrial control system networks and IIoT, the use of software-defined networking (SDN) technology can provide improved security and automation of communication processes. Despite this, the architecture of SDN can give rise to various security threats. Therefore, it is of paramount importance to consider the impact of these threats on SDN-based IIoT environments. Unlike previous research, which focused on security in IIoT and SDN architectures separately, we propose an integrated method including two components that work together seamlessly for better detecting and preventing security threats associated with SDN-based IIoT architectures. The two components consist in a convolutional neural network-based Intrusion Detection System (IDS) implemented as an SDN application and a Blockchain-based system (BS) to empower application layer and network layer security, respectively. A significant advantage of the proposed method lies in jointly minimizing the impact of attacks such as command injection and rule injection on SDN-based IIoT architecture layers. The proposed IDS exhibits superior classification accuracy in both binary and multiclass categories.

Title: TBDD: A New Trust-based, DRL-driven Framework for Blockchain Sharding in IoT. (arXiv:2401.00632v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00632
• Code URL: null
• Copy Paste: [[2401.00632]] TBDD: A New Trust-based, DRL-driven Framework for Blockchain Sharding in IoT(http://arxiv.org/abs/2401.00632)
• Summary:

  Integrating sharded blockchain with IoT presents a solution for trust issues and optimized data flow. Sharding boosts blockchain scalability by dividing its nodes into parallel shards, yet it's vulnerable to the $1\%$ attacks where dishonest nodes target a shard to corrupt the entire blockchain. Balancing security with scalability is pivotal for such systems. Deep Reinforcement Learning (DRL) adeptly handles dynamic, complex systems and multi-dimensional optimization. This paper introduces a Trust-based and DRL-driven (\textsc{TbDd}) framework, crafted to counter shard collusion risks and dynamically adjust node allocation, enhancing throughput while maintaining network security. With a comprehensive trust evaluation mechanism, \textsc{TbDd} discerns node types and performs targeted resharding against potential threats. The model maximizes tolerance for dishonest nodes, optimizes node movement frequency, ensures even node distribution in shards, and balances sharding risks. Rigorous evaluations prove \textsc{TbDd}'s superiority over conventional random-, community-, and trust-based sharding methods in shard risk equilibrium and reducing cross-shard transactions.

privacy

Title: CamPro: Camera-based Anti-Facial Recognition. (arXiv:2401.00151v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00151
• Code URL: null
• Copy Paste: [[2401.00151]] CamPro: Camera-based Anti-Facial Recognition(http://arxiv.org/abs/2401.00151)
• Summary:

  The proliferation of images captured from millions of cameras and the advancement of facial recognition (FR) technology have made the abuse of FR a severe privacy threat. Existing works typically rely on obfuscation, synthesis, or adversarial examples to modify faces in images to achieve anti-facial recognition (AFR). However, the unmodified images captured by camera modules that contain sensitive personally identifiable information (PII) could still be leaked. In this paper, we propose a novel approach, CamPro, to capture inborn AFR images. CamPro enables well-packed commodity camera modules to produce images that contain little PII and yet still contain enough information to support other non-sensitive vision applications, such as person detection. Specifically, CamPro tunes the configuration setup inside the camera image signal processor (ISP), i.e., color correction matrix and gamma correction, to achieve AFR, and designs an image enhancer to keep the image quality for possible human viewers. We implemented and validated CamPro on a proof-of-concept camera, and our experiments demonstrate its effectiveness on ten state-of-the-art black-box FR models. The results show that CamPro images can significantly reduce face identification accuracy to 0.3\% while having little impact on the targeted non-sensitive vision application. Furthermore, we find that CamPro is resilient to adaptive attackers who have re-trained their FR models using images generated by CamPro, even with full knowledge of privacy-preserving ISP parameters.

Title: SecFormer: Towards Fast and Accurate Privacy-Preserving Inference for Large Language Models. (arXiv:2401.00793v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00793
• Code URL: null
• Copy Paste: [[2401.00793]] SecFormer: Towards Fast and Accurate Privacy-Preserving Inference for Large Language Models(http://arxiv.org/abs/2401.00793)
• Summary:

  With the growing use of large language models hosted on cloud platforms to offer inference services, privacy concerns are escalating, especially concerning sensitive data like investment plans and bank account details. Secure Multi-Party Computing (SMPC) emerges as a promising solution to protect the privacy of inference data and model parameters. However, the application of SMPC in Privacy-Preserving Inference (PPI) for large language models, particularly those based on the Transformer architecture, often leads to considerable slowdowns or declines in performance. This is largely due to the multitude of nonlinear operations in the Transformer architecture, which are not well-suited to SMPC and are difficult to circumvent or optimize effectively. To address this concern, we introduce an advanced optimization framework called SecFormer, designed to strike an optimal balance between performance and efficiency in PPI for Transformer models. By implementing knowledge distillation techniques, we successfully eliminate the high-cost exponential and maximum operations in PPI without sacrificing model performance. Additionally, we have developed a suite of efficient SMPC protocols that utilize segmented polynomials and Goldschmidt's method to handle other complex nonlinear functions within PPI, such as GeLU, LayerNorm, and Softmax. Our extensive experiments reveal that SecFormer outperforms MPCFormer in performance, showing improvements of $5.6\%$ and $24.2\%$ for BERT$_{\text{BASE}}$ and BERT$_{\text{LARGE}}$, respectively. In terms of efficiency, SecFormer is 3.4 and 3.2 times faster than Puma, demonstrating its effectiveness and speed.

Title: Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learners. (arXiv:2401.00583v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00583
• Code URL: null
• Copy Paste: [[2401.00583]] Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learners(http://arxiv.org/abs/2401.00583)
• Summary:

  In the arena of privacy-preserving machine learning, differentially private stochastic gradient descent (DP-SGD) has outstripped the objective perturbation mechanism in popularity and interest. Though unrivaled in versatility, DP-SGD requires a non-trivial privacy overhead (for privately tuning the model's hyperparameters) and a computational complexity which might be extravagant for simple models such as linear and logistic regression. This paper revamps the objective perturbation mechanism with tighter privacy analyses and new computational tools that boost it to perform competitively with DP-SGD on unconstrained convex generalized linear problems.

Title: Privacy-Preserving Data in IoT-based Cloud Systems: A Comprehensive Survey with AI Integration. (arXiv:2401.00794v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00794
• Code URL: null
• Copy Paste: [[2401.00794]] Privacy-Preserving Data in IoT-based Cloud Systems: A Comprehensive Survey with AI Integration(http://arxiv.org/abs/2401.00794)
• Summary:

  As the integration of Internet of Things devices with cloud computing proliferates, the paramount importance of privacy preservation comes to the forefront. This survey paper meticulously explores the landscape of privacy issues in the dynamic intersection of IoT and cloud systems. The comprehensive literature review synthesizes existing research, illuminating key challenges and discerning emerging trends in privacy preserving techniques. The categorization of diverse approaches unveils a nuanced understanding of encryption techniques, anonymization strategies, access control mechanisms, and the burgeoning integration of artificial intelligence. Notable trends include the infusion of machine learning for dynamic anonymization, homomorphic encryption for secure computation, and AI-driven access control systems. The culmination of this survey contributes a holistic view, laying the groundwork for understanding the multifaceted strategies employed in securing sensitive data within IoT-based cloud environments. The insights garnered from this survey provide a valuable resource for researchers, practitioners, and policymakers navigating the complex terrain of privacy preservation in the evolving landscape of IoT and cloud computing

Title: Synthetic Data Applications in Finance. (arXiv:2401.00081v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00081
• Code URL: null
• Copy Paste: [[2401.00081]] Synthetic Data Applications in Finance(http://arxiv.org/abs/2401.00081)
• Summary:

  Synthetic data has made tremendous strides in various commercial settings including finance, healthcare, and virtual reality. We present a broad overview of prototypical applications of synthetic data in the financial sector and in particular provide richer details for a few select ones. These cover a wide variety of data modalities including tabular, time-series, event-series, and unstructured arising from both markets and retail financial applications. Since finance is a highly regulated industry, synthetic data is a potential approach for dealing with issues related to privacy, fairness, and explainability. Various metrics are utilized in evaluating the quality and effectiveness of our approaches in these applications. We conclude with open directions in synthetic data in the context of the financial domain.

Title: A review on different techniques used to combat the non-IID and heterogeneous nature of data in FL. (arXiv:2401.00809v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00809
• Code URL: null
• Copy Paste: [[2401.00809]] A review on different techniques used to combat the non-IID and heterogeneous nature of data in FL(http://arxiv.org/abs/2401.00809)
• Summary:

  Federated Learning (FL) is a machine-learning approach enabling collaborative model training across multiple decentralized edge devices that hold local data samples, all without exchanging these samples. This collaborative process occurs under the supervision of a central server orchestrating the training or via a peer-to-peer network. The significance of FL is particularly pronounced in industries such as healthcare and finance, where data privacy holds paramount importance. However, training a model under the Federated learning setting brings forth several challenges, with one of the most prominent being the heterogeneity of data distribution among the edge devices. The data is typically non-independently and non-identically distributed (non-IID), thereby presenting challenges to model convergence. This report delves into the issues arising from non-IID and heterogeneous data and explores current algorithms designed to address these challenges.

protect

Title: Quantifying Policy Administration Cost in an Active Learning Framework. (arXiv:2401.00086v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00086
• Code URL: null
• Copy Paste: [[2401.00086]] Quantifying Policy Administration Cost in an Active Learning Framework(http://arxiv.org/abs/2401.00086)
• Summary:

  This paper proposes a computational model for policy administration. As an organization evolves, new users and resources are gradually placed under the mediation of the access control model. Each time such new entities are added, the policy administrator must deliberate on how the access control policy shall be revised to reflect the new reality. A well-designed access control model must anticipate such changes so that the administration cost does not become prohibitive when the organization scales up. Unfortunately, past Access Control research does not offer a formal way to quantify the cost of policy administration. In this work, we propose to model ongoing policy administration in an active learning framework. Administration cost can be quantified in terms of query complexity. We demonstrate the utility of this approach by applying it to the evolution of protection domains. We also modelled different policy administration strategies in our framework. This allowed us to formally demonstrate that domain-based policies have a cost advantage over access control matrices because of the use of heuristic reasoning when the policy evolves. To the best of our knowledge, this is the first work to employ an active learning framework to study the cost of policy deliberation and demonstrate the cost advantage of heuristic policy administration.

Title: Multi-spatial Multi-temporal Air Quality Forecasting with Integrated Monitoring and Reanalysis Data. (arXiv:2401.00521v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00521
• Code URL: null
• Copy Paste: [[2401.00521]] Multi-spatial Multi-temporal Air Quality Forecasting with Integrated Monitoring and Reanalysis Data(http://arxiv.org/abs/2401.00521)
• Summary:

  Accurate air quality forecasting is crucial for public health, environmental monitoring and protection, and urban planning. However, existing methods fail to effectively utilize multi-scale information, both spatially and temporally. Spatially, there is a lack of integration between individual monitoring stations and city-wide scales. Temporally, the periodic nature of air quality variations is often overlooked or inadequately considered. To address these limitations, we present a novel Multi-spatial Multi-temporal air quality forecasting method based on Graph Convolutional Networks and Gated Recurrent Units (M2G2), bridging the gap in air quality forecasting across spatial and temporal scales. The proposed framework consists of two modules: Multi-scale Spatial GCN (MS-GCN) for spatial information fusion and Multi-scale Temporal GRU(MT-GRU) for temporal information integration. In the spatial dimension, the MS-GCN module employs a bidirectional learnable structure and a residual structure, enabling comprehensive information exchange between individual monitoring stations and the city-scale graph. Regarding the temporal dimension, the MT-GRU module adaptively combines information from different temporal scales through parallel hidden states. Leveraging meteorological indicators and four air quality indicators, we present comprehensive comparative analyses and ablation experiments, showcasing the higher accuracy of M2G2 in comparison to nine currently available advanced approaches across all aspects. The improvements of M2G2 over the second-best method on RMSE of the 24h/48h/72h are as follows: PM2.5: (7.72%, 6.67%, 10.45%); PM10: (6.43%, 5.68%, 7.73%); NO2: (5.07%, 7.76%, 16.60%); O3: (6.46%, 6.86%, 9.79%). Furthermore, we demonstrate the effectiveness of each module of M2G2 by ablation study.

defense

Title: TPatch: A Triggered Physical Adversarial Patch. (arXiv:2401.00148v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00148
• Code URL: null
• Copy Paste: [[2401.00148]] TPatch: A Triggered Physical Adversarial Patch(http://arxiv.org/abs/2401.00148)
• Summary:

  Autonomous vehicles increasingly utilize the vision-based perception module to acquire information about driving environments and detect obstacles. Correct detection and classification are important to ensure safe driving decisions. Existing works have demonstrated the feasibility of fooling the perception models such as object detectors and image classifiers with printed adversarial patches. However, most of them are indiscriminately offensive to every passing autonomous vehicle. In this paper, we propose TPatch, a physical adversarial patch triggered by acoustic signals. Unlike other adversarial patches, TPatch remains benign under normal circumstances but can be triggered to launch a hiding, creating or altering attack by a designed distortion introduced by signal injection attacks towards cameras. To avoid the suspicion of human drivers and make the attack practical and robust in the real world, we propose a content-based camouflage method and an attack robustness enhancement method to strengthen it. Evaluations with three object detectors, YOLO V3/V5 and Faster R-CNN, and eight image classifiers demonstrate the effectiveness of TPatch in both the simulation and the real world. We also discuss possible defenses at the sensor, algorithm, and system levels.

Title: The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness. (arXiv:2401.00287v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00287
• Code URL: null
• Copy Paste: [[2401.00287]] The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensiveness(http://arxiv.org/abs/2401.00287)
• Summary:

  As Large Language Models (LLMs) play an increasingly pivotal role in natural language processing applications, their safety concerns become critical areas of NLP research. This paper presents Safety and Over-Defensiveness Evaluation (SODE) benchmark: a collection of diverse safe and unsafe prompts with carefully designed evaluation methods that facilitate systematic evaluation, comparison, and analysis over 'safety' and 'over-defensiveness.' With SODE, we study a variety of LLM defense strategies over multiple state-of-the-art LLMs, which reveals several interesting and important findings, such as (a) the widely popular 'self-checking' techniques indeed improve the safety against unsafe inputs, but this comes at the cost of extreme over-defensiveness on the safe inputs, (b) providing a safety instruction along with in-context exemplars (of both safe and unsafe inputs) consistently improves safety and also mitigates undue over-defensiveness of the models, (c) providing contextual knowledge easily breaks the safety guardrails and makes the models more vulnerable to generating unsafe responses. Overall, our work reveals numerous such critical findings that we believe will pave the way and facilitate further research in improving the safety of LLMs.

attack

Title: Argumentation in Waltz's "Emerging Structure of International Politics''. (arXiv:2401.00366v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00366
• Code URL: null
• Copy Paste: [[2401.00366]] Argumentation in Waltz's "Emerging Structure of International Politics''(http://arxiv.org/abs/2401.00366)
• Summary:

  We present an annotation scheme for argumentative and domain-specific aspects of scholarly articles on the theory of International Relations. At argumentation level we identify Claims and Support/Attack relations. At domain level we model discourse content in terms of Theory and Data-related statements. We annotate Waltz's 1993 text on structural realism and show that our scheme can be reliably applied by domain experts enables insights on two research questions on justifications of claims.

Title: A clean-label graph backdoor attack method in node classification task. (arXiv:2401.00163v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00163
• Code URL: null
• Copy Paste: [[2401.00163]] A clean-label graph backdoor attack method in node classification task(http://arxiv.org/abs/2401.00163)
• Summary:

  Backdoor attacks in the traditional graph neural networks (GNNs) field are easily detectable due to the dilemma of confusing labels. To explore the backdoor vulnerability of GNNs and create a more stealthy backdoor attack method, a clean-label graph backdoor attack method(CGBA) in the node classification task is proposed in this paper. Differently from existing backdoor attack methods, CGBA requires neither modification of node labels nor graph structure. Specifically, to solve the problem of inconsistency between the contents and labels of the samples, CGBA selects poisoning samples in a specific target class and uses the label of sample as the target label (i.e., clean-label) after injecting triggers into the target samples. To guarantee the similarity of neighboring nodes, the raw features of the nodes are elaborately picked as triggers to further improve the concealment of the triggers. Extensive experiments results show the effectiveness of our method. When the poisoning rate is 0.04, CGBA can achieve an average attack success rate of 87.8%, 98.9%, 89.1%, and 98.5%, respectively.

Title: RASP for LSASS: Preventing Mimikatz-Related Attacks. (arXiv:2401.00316v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00316
• Code URL: null
• Copy Paste: [[2401.00316]] RASP for LSASS: Preventing Mimikatz-Related Attacks(http://arxiv.org/abs/2401.00316)
• Summary:

  The Windows authentication infrastructure relies on the Local Security Authority (LSA) system, with its integral component being lsass.exe. Regrettably, this framework is not impervious, presenting vulnerabilities that attract threat actors with malicious intent. By exploiting documented vulnerabilities sourced from the CVE database or leveraging sophisticated tools such as mimikatz, adversaries can successfully compromise user password-address information.

  In this comprehensive analysis, we delve into proactive measures aimed at fortifying the local authentication subsystem against potential threats. Moreover, we present empirical evidence derived from practical assessments of various defensive methodologies, including those articulated previously. This examination not only underscores the importance of proactive security measures but also assesses the practical efficacy of these strategies in real-world contexts.

robust

Title: A comprehensive framework for occluded human pose estimation. (arXiv:2401.00155v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00155
• Code URL: null
• Copy Paste: [[2401.00155]] A comprehensive framework for occluded human pose estimation(http://arxiv.org/abs/2401.00155)
• Summary:

  Occlusion presents a significant challenge in human pose estimation. The challenges posed by occlusion can be attributed to the following factors: 1) Data: The collection and annotation of occluded human pose samples are relatively challenging. 2) Feature: Occlusion can cause feature confusion due to the high similarity between the target person and interfering individuals. 3) Inference: Robust inference becomes challenging due to the loss of complete body structural information. The existing methods designed for occluded human pose estimation usually focus on addressing only one of these factors. In this paper, we propose a comprehensive framework DAG (Data, Attention, Graph) to address the performance degradation caused by occlusion. Specifically, we introduce the mask joints with instance paste data augmentation technique to simulate occlusion scenarios. Additionally, an Adaptive Discriminative Attention Module (ADAM) is proposed to effectively enhance the features of target individuals. Furthermore, we present the Feature-Guided Multi-Hop GCN (FGMP-GCN) to fully explore the prior knowledge of body structure and improve pose estimation results. Through extensive experiments conducted on three benchmark datasets for occluded human pose estimation, we demonstrate that the proposed method outperforms existing methods. Code and data will be publicly available.

Title: BusReF: Infrared-Visible images registration and fusion focus on reconstructible area using one set of features. (arXiv:2401.00285v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00285
• Code URL: null
• Copy Paste: [[2401.00285]] BusReF: Infrared-Visible images registration and fusion focus on reconstructible area using one set of features(http://arxiv.org/abs/2401.00285)
• Summary:

  In a scenario where multi-modal cameras are operating together, the problem of working with non-aligned images cannot be avoided. Yet, existing image fusion algorithms rely heavily on strictly registered input image pairs to produce more precise fusion results, as a way to improve the performance of downstream high-level vision tasks. In order to relax this assumption, one can attempt to register images first. However, the existing methods for registering multiple modalities have limitations, such as complex structures and reliance on significant semantic information. This paper aims to address the problem of image registration and fusion in a single framework, called BusRef. We focus on Infrared-Visible image registration and fusion task (IVRF). In this framework, the input unaligned image pairs will pass through three stages: Coarse registration, Fine registration and Fusion. It will be shown that the unified approach enables more robust IVRF. We also propose a novel training and evaluation strategy, involving the use of masks to reduce the influence of non-reconstructible regions on the loss functions, which greatly improves the accuracy and robustness of the fusion task. Last but not least, a gradient-aware fusion network is designed to preserve the complementary information. The advanced performance of this algorithm is demonstrated by

Title: SHARE: Single-view Human Adversarial REconstruction. (arXiv:2401.00343v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00343
• Code URL: null
• Copy Paste: [[2401.00343]] SHARE: Single-view Human Adversarial REconstruction(http://arxiv.org/abs/2401.00343)
• Summary:

  The accuracy of 3D Human Pose and Shape reconstruction (HPS) from an image is progressively improving. Yet, no known method is robust across all image distortion. To address issues due to variations of camera poses, we introduce SHARE, a novel fine-tuning method that utilizes adversarial data augmentation to enhance the robustness of existing HPS techniques. We perform a comprehensive analysis on the impact of camera poses on HPS reconstruction outcomes. We first generated large-scale image datasets captured systematically from diverse camera perspectives. We then established a mapping between camera poses and reconstruction errors as a continuous function that characterizes the relationship between camera poses and HPS quality. Leveraging this representation, we introduce RoME (Regions of Maximal Error), a novel sampling technique for our adversarial fine-tuning method.

  The SHARE framework is generalizable across various single-view HPS methods and we demonstrate its performance on HMR, SPIN, PARE, CLIFF and ExPose. Our results illustrate a reduction in mean joint errors across single-view HPS techniques, for images captured from multiple camera positions without compromising their baseline performance. In many challenging cases, our method surpasses the performance of existing models, highlighting its practical significance for diverse real-world applications.

Title: Generalizing Single-View 3D Shape Retrieval to Occlusions and Unseen Objects. (arXiv:2401.00405v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00405
• Code URL: null
• Copy Paste: [[2401.00405]] Generalizing Single-View 3D Shape Retrieval to Occlusions and Unseen Objects(http://arxiv.org/abs/2401.00405)
• Summary:

  Single-view 3D shape retrieval is a challenging task that is increasingly important with the growth of available 3D data. Prior work that has studied this task has not focused on evaluating how realistic occlusions impact performance, and how shape retrieval methods generalize to scenarios where either the target 3D shape database contains unseen shapes, or the input image contains unseen objects. In this paper, we systematically evaluate single-view 3D shape retrieval along three different axes: the presence of object occlusions and truncations, generalization to unseen 3D shape data, and generalization to unseen objects in the input images. We standardize two existing datasets of real images and propose a dataset generation pipeline to produce a synthetic dataset of scenes with multiple objects exhibiting realistic occlusions. Our experiments show that training on occlusion-free data as was commonly done in prior work leads to significant performance degradation for inputs with occlusion. We find that that by first pretraining on our synthetic dataset with occlusions and then finetuning on real data, we can significantly outperform models from prior work and demonstrate robustness to both unseen 3D shapes and unseen objects.

Title: From Text to Pixels: A Context-Aware Semantic Synergy Solution for Infrared and Visible Image Fusion. (arXiv:2401.00421v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00421
• Code URL: null
• Copy Paste: [[2401.00421]] From Text to Pixels: A Context-Aware Semantic Synergy Solution for Infrared and Visible Image Fusion(http://arxiv.org/abs/2401.00421)
• Summary:

  With the rapid progression of deep learning technologies, multi-modality image fusion has become increasingly prevalent in object detection tasks. Despite its popularity, the inherent disparities in how different sources depict scene content make fusion a challenging problem. Current fusion methodologies identify shared characteristics between the two modalities and integrate them within this shared domain using either iterative optimization or deep learning architectures, which often neglect the intricate semantic relationships between modalities, resulting in a superficial understanding of inter-modal connections and, consequently, suboptimal fusion outcomes. To address this, we introduce a text-guided multi-modality image fusion method that leverages the high-level semantics from textual descriptions to integrate semantics from infrared and visible images. This method capitalizes on the complementary characteristics of diverse modalities, bolstering both the accuracy and robustness of object detection. The codebook is utilized to enhance a streamlined and concise depiction of the fused intra- and inter-domain dynamics, fine-tuned for optimal performance in detection tasks. We present a bilevel optimization strategy that establishes a nexus between the joint problem of fusion and detection, optimizing both processes concurrently. Furthermore, we introduce the first dataset of paired infrared and visible images accompanied by text prompts, paving the way for future research. Extensive experiments on several datasets demonstrate that our method not only produces visually superior fusion results but also achieves a higher detection mAP over existing methods, achieving state-of-the-art results.

Title: Bidirectional Trained Tree-Structured Decoder for Handwritten Mathematical Expression Recognition. (arXiv:2401.00435v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00435
• Code URL: null
• Copy Paste: [[2401.00435]] Bidirectional Trained Tree-Structured Decoder for Handwritten Mathematical Expression Recognition(http://arxiv.org/abs/2401.00435)
• Summary:

  The Handwritten Mathematical Expression Recognition (HMER) task is a critical branch in the field of OCR. Recent studies have demonstrated that incorporating bidirectional context information significantly improves the performance of HMER models. However, existing methods fail to effectively utilize bidirectional context information during the inference stage. Furthermore, current bidirectional training methods are primarily designed for string decoders and cannot adequately generalize to tree decoders, which offer superior generalization capabilities and structural analysis capacity. In order to overcome these limitations, we propose the Mirror-Flipped Symbol Layout Tree (MF-SLT) and Bidirectional Asynchronous Training (BAT) structure. Our method extends the bidirectional training strategy to the tree decoder, allowing for more effective training by leveraging bidirectional information. Additionally, we analyze the impact of the visual and linguistic perception of the HMER model separately and introduce the Shared Language Modeling (SLM) mechanism. Through the SLM, we enhance the model's robustness and generalization when dealing with visual ambiguity, particularly in scenarios with abundant training data. Our approach has been validated through extensive experiments, demonstrating its ability to achieve new state-of-the-art results on the CROHME 2014, 2016, and 2019 datasets, as well as the HME100K dataset. The code used in our experiments will be publicly available.

Title: Geometry Depth Consistency in RGBD Relative Pose Estimation. (arXiv:2401.00639v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00639
• Code URL: null
• Copy Paste: [[2401.00639]] Geometry Depth Consistency in RGBD Relative Pose Estimation(http://arxiv.org/abs/2401.00639)
• Summary:

  Relative pose estimation for RGBD cameras is crucial in a number of applications. Previous approaches either rely on the RGB aspect of the images to estimate pose thus not fully making use of depth in the estimation process or estimate pose from the 3D cloud of points that each image produces, thus not making full use of RGB information. This paper shows that if one pair of correspondences is hypothesized from the RGB-based ranked-ordered correspondence list, then the space of remaining correspondences is restricted to corresponding pairs of curves nested around the hypothesized correspondence, implicitly capturing depth consistency. This simple Geometric Depth Constraint (GDC) significantly reduces potential matches. In effect this becomes a filter on possible correspondences that helps reduce the number of outliers and thus expedites RANSAC significantly. As such, the same budget of time allows for more RANSAC iterations and therefore additional robustness and a significant speedup. In addition, the paper proposed a Nested RANSAC approach that also speeds up the process, as shown through experiments on TUM, ICL-NUIM, and RGBD Scenes v2 datasets.

Title: From Covert Hiding to Visual Editing: Robust Generative Video Steganography. (arXiv:2401.00652v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00652
• Code URL: null
• Copy Paste: [[2401.00652]] From Covert Hiding to Visual Editing: Robust Generative Video Steganography(http://arxiv.org/abs/2401.00652)
• Summary:

  Traditional video steganography methods are based on modifying the covert space for embedding, whereas we propose an innovative approach that embeds secret message within semantic feature for steganography during the video editing process. Although existing traditional video steganography methods display a certain level of security and embedding capacity, they lack adequate robustness against common distortions in online social networks (OSNs). In this paper, we introduce an end-to-end robust generative video steganography network (RoGVS), which achieves visual editing by modifying semantic feature of videos to embed secret message. We employ face-swapping scenario to showcase the visual editing effects. We first design a secret message embedding module to adaptively hide secret message into the semantic feature of videos. Extensive experiments display that the proposed RoGVS method applied to facial video datasets demonstrate its superiority over existing video and image steganography techniques in terms of both robustness and capacity.

Title: PROMPT-IML: Image Manipulation Localization with Pre-trained Foundation Models Through Prompt Tuning. (arXiv:2401.00653v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00653
• Code URL: null
• Copy Paste: [[2401.00653]] PROMPT-IML: Image Manipulation Localization with Pre-trained Foundation Models Through Prompt Tuning(http://arxiv.org/abs/2401.00653)
• Summary:

  Deceptive images can be shared in seconds with social networking services, posing substantial risks. Tampering traces, such as boundary artifacts and high-frequency information, have been significantly emphasized by massive networks in the Image Manipulation Localization (IML) field. However, they are prone to image post-processing operations, which limit the generalization and robustness of existing methods. We present a novel Prompt-IML framework. We observe that humans tend to discern the authenticity of an image based on both semantic and high-frequency information, inspired by which, the proposed framework leverages rich semantic knowledge from pre-trained visual foundation models to assist IML. We are the first to design a framework that utilizes visual foundation models specially for the IML task. Moreover, we design a Feature Alignment and Fusion module to align and fuse features of semantic features with high-frequency features, which aims at locating tampered regions from multiple perspectives. Experimental results demonstrate that our model can achieve better performance on eight typical fake image datasets and outstanding robustness.

Title: Depth Map Denoising Network and Lightweight Fusion Network for Enhanced 3D Face Recognition. (arXiv:2401.00719v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00719
• Code URL: null
• Copy Paste: [[2401.00719]] Depth Map Denoising Network and Lightweight Fusion Network for Enhanced 3D Face Recognition(http://arxiv.org/abs/2401.00719)
• Summary:

  With the increasing availability of consumer depth sensors, 3D face recognition (FR) has attracted more and more attention. However, the data acquired by these sensors are often coarse and noisy, making them impractical to use directly. In this paper, we introduce an innovative Depth map denoising network (DMDNet) based on the Denoising Implicit Image Function (DIIF) to reduce noise and enhance the quality of facial depth images for low-quality 3D FR. After generating clean depth faces using DMDNet, we further design a powerful recognition network called Lightweight Depth and Normal Fusion network (LDNFNet), which incorporates a multi-branch fusion block to learn unique and complementary features between different modalities such as depth and normal images. Comprehensive experiments conducted on four distinct low-quality databases demonstrate the effectiveness and robustness of our proposed methods. Furthermore, when combining DMDNet and LDNFNet, we achieve state-of-the-art results on the Lock3DFace database.

Title: Deblurring 3D Gaussian Splatting. (arXiv:2401.00834v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00834
• Code URL: null
• Copy Paste: [[2401.00834]] Deblurring 3D Gaussian Splatting(http://arxiv.org/abs/2401.00834)
• Summary:

  Recent studies in Radiance Fields have paved the robust way for novel view synthesis with their photorealistic rendering quality. Nevertheless, they usually employ neural networks and volumetric rendering, which are costly to train and impede their broad use in various real-time applications due to the lengthy rendering time. Lately 3D Gaussians splatting-based approach has been proposed to model the 3D scene, and it achieves remarkable visual quality while rendering the images in real-time. However, it suffers from severe degradation in the rendering quality if the training images are blurry. Blurriness commonly occurs due to the lens defocusing, object motion, and camera shake, and it inevitably intervenes in clean image acquisition. Several previous studies have attempted to render clean and sharp images from blurry input images using neural fields. The majority of those works, however, are designed only for volumetric rendering-based neural radiance fields and are not straightforwardly applicable to rasterization-based 3D Gaussian splatting methods. Thus, we propose a novel real-time deblurring framework, deblurring 3D Gaussian Splatting, using a small Multi-Layer Perceptron (MLP) that manipulates the covariance of each 3D Gaussian to model the scene blurriness. While deblurring 3D Gaussian Splatting can still enjoy real-time rendering, it can reconstruct fine and sharp details from blurry images. A variety of experiments have been conducted on the benchmark, and the results have revealed the effectiveness of our approach for deblurring. Qualitative results are available at https://benhenryl.github.io/Deblurring-3D-Gaussian-Splatting/

Title: Mitigating the Impact of False Negatives in Dense Retrieval with Contrastive Confidence Regularization. (arXiv:2401.00165v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00165
• Code URL: null
• Copy Paste: [[2401.00165]] Mitigating the Impact of False Negatives in Dense Retrieval with Contrastive Confidence Regularization(http://arxiv.org/abs/2401.00165)
• Summary:

  In open-domain Question Answering (QA), dense retrieval is crucial for finding relevant passages for answer generation. Typically, contrastive learning is used to train a retrieval model that maps passages and queries to the same semantic space. The objective is to make similar ones closer and dissimilar ones further apart. However, training such a system is challenging due to the false negative issue, where relevant passages may be missed during data annotation. Hard negative sampling, which is commonly used to improve contrastive learning, can introduce more noise in training. This is because hard negatives are those closer to a given query, and thus more likely to be false negatives. To address this issue, we propose a novel contrastive confidence regularizer for Noise Contrastive Estimation (NCE) loss, a commonly used loss for dense retrieval. Our analysis shows that the regularizer helps dense retrieval models be more robust against false negatives with a theoretical guarantee. Additionally, we propose a model-agnostic method to filter out noisy negative passages in the dataset, improving any downstream dense retrieval models. Through experiments on three datasets, we demonstrate that our method achieves better retrieval performance in comparison to existing state-of-the-art dense retrieval systems.

Title: BatchEval: Towards Human-like Text Evaluation. (arXiv:2401.00437v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00437
• Code URL: null
• Copy Paste: [[2401.00437]] BatchEval: Towards Human-like Text Evaluation(http://arxiv.org/abs/2401.00437)
• Summary:

  Significant progress has been made in automatic text evaluation with the introduction of large language models (LLMs) as evaluators. However, current sample-wise evaluation paradigm suffers from the following issues: (1) Sensitive to prompt design; (2) Poor resistance to noise; (3) Inferior ensemble performance with static reference. Inspired by the fact that humans treat both criterion definition and inter sample comparison as references for evaluation, we propose BatchEval, a paradigm that conducts batch-wise evaluation iteratively to alleviate the above problems. We explore variants under this paradigm and confirm the optimal settings are two stage procedure with heterogeneous batch composition strategy and decimal scoring format. Comprehensive experiments across 3 LLMs on 4 text evaluation tasks demonstrate that BatchEval outperforms state-of-the-art methods by 10.5% on Pearson correlations with only 64% API cost on average. Further analyses have been conducted to verify the robustness, generalization, and working mechanism of BatchEval.

Title: State of What Art? A Call for Multi-Prompt LLM Evaluation. (arXiv:2401.00595v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00595
• Code URL: null
• Copy Paste: [[2401.00595]] State of What Art? A Call for Multi-Prompt LLM Evaluation(http://arxiv.org/abs/2401.00595)
• Summary:

  Recent advances in large language models (LLMs) have led to the development of various evaluation benchmarks. These benchmarks typically rely on a single instruction template for evaluating all LLMs on a specific task. In this paper, we comprehensively analyze the brittleness of results obtained via single-prompt evaluations across 6.5M instances, involving 20 different LLMs and 39 tasks from 3 benchmarks. To improve robustness of the analysis, we propose to evaluate LLMs with a set of diverse prompts instead. We discuss tailored evaluation metrics for specific use cases (e.g., LLM developers vs. developers interested in a specific downstream task), ensuring a more reliable and meaningful assessment of LLM capabilities. We then implement these criteria and conduct evaluations of multiple models, providing insights into the true strengths and limitations of current LLMs.

Title: Adversarially Trained Actor Critic for offline CMDPs. (arXiv:2401.00629v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00629
• Code URL: null
• Copy Paste: [[2401.00629]] Adversarially Trained Actor Critic for offline CMDPs(http://arxiv.org/abs/2401.00629)
• Summary:

  We propose a Safe Adversarial Trained Actor Critic (SATAC) algorithm for offline reinforcement learning (RL) with general function approximation in the presence of limited data coverage. SATAC operates as a two-player Stackelberg game featuring a refined objective function. The actor (leader player) optimizes the policy against two adversarially trained value critics (follower players), who focus on scenarios where the actor's performance is inferior to the behavior policy. Our framework provides both theoretical guarantees and a robust deep-RL implementation. Theoretically, we demonstrate that when the actor employs a no-regret optimization oracle, SATAC achieves two guarantees: (i) For the first time in the offline RL setting, we establish that SATAC can produce a policy that outperforms the behavior policy while maintaining the same level of safety, which is critical to designing an algorithm for offline RL. (ii) We demonstrate that the algorithm guarantees policy improvement across a broad range of hyperparameters, indicating its practical robustness. Additionally, we offer a practical version of SATAC and compare it with existing state-of-the-art offline safe-RL algorithms in continuous control environments. SATAC outperforms all baselines across a range of tasks, thus validating the theoretical performance.

Title: Unsupervised Outlier Detection using Random Subspace and Subsampling Ensembles of Dirichlet Process Mixtures. (arXiv:2401.00773v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00773
• Code URL: null
• Copy Paste: [[2401.00773]] Unsupervised Outlier Detection using Random Subspace and Subsampling Ensembles of Dirichlet Process Mixtures(http://arxiv.org/abs/2401.00773)
• Summary:

  Probabilistic mixture models are acknowledged as a valuable tool for unsupervised outlier detection owing to their interpretability and intuitive grounding in statistical principles. Within this framework, Dirichlet process mixture models emerge as a compelling alternative to conventional finite mixture models for both clustering and outlier detection tasks. However, despite their evident advantages, the widespread adoption of Dirichlet process mixture models in unsupervised outlier detection has been hampered by challenges related to computational inefficiency and sensitivity to outliers during the construction of detectors. To tackle these challenges, we propose a novel outlier detection method based on ensembles of Dirichlet process Gaussian mixtures. The proposed method is a fully unsupervised algorithm that capitalizes on random subspace and subsampling ensembles, not only ensuring efficient computation but also enhancing the robustness of the resulting outlier detector. Moreover, the proposed method leverages variational inference for Dirichlet process mixtures to ensure efficient and fast computation. Empirical studies with benchmark datasets demonstrate that our method outperforms existing approaches for unsupervised outlier detection.

Title: Inferring Heterogeneous Treatment Effects of Crashes on Highway Traffic: A Doubly Robust Causal Machine Learning Approach. (arXiv:2401.00781v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00781
• Code URL: null
• Copy Paste: [[2401.00781]] Inferring Heterogeneous Treatment Effects of Crashes on Highway Traffic: A Doubly Robust Causal Machine Learning Approach(http://arxiv.org/abs/2401.00781)
• Summary:

  Highway traffic crashes exert a considerable impact on both transportation systems and the economy. In this context, accurate and dependable emergency responses are crucial for effective traffic management. However, the influence of crashes on traffic status varies across diverse factors and may be biased due to selection bias. Therefore, there arises a necessity to accurately estimate the heterogeneous causal effects of crashes, thereby providing essential insights to facilitate individual-level emergency decision-making. This paper proposes a novel causal machine learning framework to estimate the causal effect of different types of crashes on highway speed. The Neyman-Rubin Causal Model (RCM) is employed to formulate this problem from a causal perspective. The Conditional Shapley Value Index (CSVI) is proposed based on causal graph theory to filter adverse variables, and the Structural Causal Model (SCM) is then adopted to define the statistical estimand for causal effects. The treatment effects are estimated by Doubly Robust Learning (DRL) methods, which combine doubly robust causal inference with classification and regression machine learning models. Experimental results from 4815 crashes on Highway Interstate 5 in Washington State reveal the heterogeneous treatment effects of crashes at varying distances and durations. The rear-end crashes cause more severe congestion and longer durations than other types of crashes, and the sideswipe crashes have the longest delayed impact. Additionally, the findings show that rear-end crashes affect traffic greater at night, while crash to objects has the most significant influence during peak hours. Statistical hypothesis tests, error metrics based on matched "counterfactual outcomes", and sensitive analyses are employed for assessment, and the results validate the accuracy and effectiveness of our method.

biometric

Title: SynCDR : Training Cross Domain Retrieval Models with Synthetic Data. (arXiv:2401.00420v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00420
• Code URL: null
• Copy Paste: [[2401.00420]] SynCDR : Training Cross Domain Retrieval Models with Synthetic Data(http://arxiv.org/abs/2401.00420)
• Summary:

  In cross-domain retrieval, a model is required to identify images from the same semantic category across two visual domains. For instance, given a sketch of an object, a model needs to retrieve a real image of it from an online store's catalog. A standard approach for such a problem is learning a feature space of images where Euclidean distances reflect similarity. Even without human annotations, which may be expensive to acquire, prior methods function reasonably well using unlabeled images for training. Our problem constraint takes this further to scenarios where the two domains do not necessarily share any common categories in training data. This can occur when the two domains in question come from different versions of some biometric sensor recording identities of different people. We posit a simple solution, which is to generate synthetic data to fill in these missing category examples across domains. This, we do via category preserving translation of images from one visual domain to another. We compare approaches specifically trained for this translation for a pair of domains, as well as those that can use large-scale pre-trained text-to-image diffusion models via prompts, and find that the latter can generate better replacement synthetic data, leading to more accurate cross-domain retrieval models. Code for our work is available at https://github.com/samarth4149/SynCDR .

steal

extraction

Title: Rethinking RAFT for Efficient Optical Flow. (arXiv:2401.00833v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00833
• Code URL: https://github.com/n3slami/Ef-RAFT
• Copy Paste: [[2401.00833]] Rethinking RAFT for Efficient Optical Flow(http://arxiv.org/abs/2401.00833)
• Summary:

  Despite significant progress in deep learning-based optical flow methods, accurately estimating large displacements and repetitive patterns remains a challenge. The limitations of local features and similarity search patterns used in these algorithms contribute to this issue. Additionally, some existing methods suffer from slow runtime and excessive graphic memory consumption. To address these problems, this paper proposes a novel approach based on the RAFT framework. The proposed Attention-based Feature Localization (AFL) approach incorporates the attention mechanism to handle global feature extraction and address repetitive patterns. It introduces an operator for matching pixels with corresponding counterparts in the second frame and assigning accurate flow values. Furthermore, an Amorphous Lookup Operator (ALO) is proposed to enhance convergence speed and improve RAFTs ability to handle large displacements by reducing data redundancy in its search operator and expanding the search space for similarity extraction. The proposed method, Efficient RAFT (Ef-RAFT),achieves significant improvements of 10% on the Sintel dataset and 5% on the KITTI dataset over RAFT. Remarkably, these enhancements are attained with a modest 33% reduction in speed and a mere 13% increase in memory usage. The code is available at: https://github.com/n3slami/Ef-RAFT

Title: L3Cube-MahaSocialNER: A Social Media based Marathi NER Dataset and BERT models. (arXiv:2401.00170v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00170
• Code URL: null
• Copy Paste: [[2401.00170]] L3Cube-MahaSocialNER: A Social Media based Marathi NER Dataset and BERT models(http://arxiv.org/abs/2401.00170)
• Summary:

  This work introduces the L3Cube-MahaSocialNER dataset, the first and largest social media dataset specifically designed for Named Entity Recognition (NER) in the Marathi language. The dataset comprises 18,000 manually labeled sentences covering eight entity classes, addressing challenges posed by social media data, including non-standard language and informal idioms. Deep learning models, including CNN, LSTM, BiLSTM, and Transformer models, are evaluated on the individual dataset with IOB and non-IOB notations. The results demonstrate the effectiveness of these models in accurately recognizing named entities in Marathi informal text. The L3Cube-MahaSocialNER dataset offers user-centric information extraction and supports real-time applications, providing a valuable resource for public opinion analysis, news, and marketing on social media platforms. We also show that the zero-shot results of the regular NER model are poor on the social NER test set thus highlighting the need for more social NER datasets. The datasets and models are publicly available at https://github.com/l3cube-pune/MarathiNLP

Title: Exploring the Effectiveness of Instruction Tuning in Biomedical Language Processing. (arXiv:2401.00579v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00579
• Code URL: null
• Copy Paste: [[2401.00579]] Exploring the Effectiveness of Instruction Tuning in Biomedical Language Processing(http://arxiv.org/abs/2401.00579)
• Summary:

  Large Language Models (LLMs), particularly those similar to ChatGPT, have significantly influenced the field of Natural Language Processing (NLP). While these models excel in general language tasks, their performance in domain-specific downstream tasks such as biomedical and clinical Named Entity Recognition (NER), Relation Extraction (RE), and Medical Natural Language Inference (NLI) is still evolving. In this context, our study investigates the potential of instruction tuning for biomedical language processing, applying this technique to two general LLMs of substantial scale. We present a comprehensive, instruction-based model trained on a dataset that consists of approximately $200,000$ instruction-focused samples. This dataset represents a carefully curated compilation of existing data, meticulously adapted and reformatted to align with the specific requirements of our instruction-based tasks. This initiative represents an important step in utilising such models to achieve results on par with specialised encoder-only models like BioBERT and BioClinicalBERT for various classical biomedical NLP tasks. Our work includes an analysis of the dataset's composition and its impact on model performance, providing insights into the intricacies of instruction tuning. By sharing our codes, models, and the distinctively assembled instruction-based dataset, we seek to encourage ongoing research and development in this area.

Title: PerSHOP -- A Persian dataset for shopping dialogue systems modeling. (arXiv:2401.00811v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00811
• Code URL: null
• Copy Paste: [[2401.00811]] PerSHOP -- A Persian dataset for shopping dialogue systems modeling(http://arxiv.org/abs/2401.00811)
• Summary:

  Nowadays, dialogue systems are used in many fields of industry and research. There are successful instances of these systems, such as Apple Siri, Google Assistant, and IBM Watson. Task-oriented dialogue system is a category of these, that are used in specific tasks. They can perform tasks such as booking plane tickets or making restaurant reservations. Shopping is one of the most popular areas on these systems. The bot replaces the human salesperson and interacts with the customers by speaking. To train the models behind the scenes of these systems, annotated data is needed. In this paper, we developed a dataset of dialogues in the Persian language through crowd-sourcing. We annotated these dialogues to train a model. This dataset contains nearly 22k utterances in 15 different domains and 1061 dialogues. This is the largest Persian dataset in this field, which is provided freely so that future researchers can use it. Also, we proposed some baseline models for natural language understanding (NLU) tasks. These models perform two tasks for NLU: intent classification and entity extraction. The F-1 score metric obtained for intent classification is around 91% and for entity extraction is around 93%, which can be a baseline for future research.

Title: MPRE: Multi-perspective Patient Representation Extractor for Disease Prediction. (arXiv:2401.00756v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00756
• Code URL: null
• Copy Paste: [[2401.00756]] MPRE: Multi-perspective Patient Representation Extractor for Disease Prediction(http://arxiv.org/abs/2401.00756)
• Summary:

  Patient representation learning based on electronic health records (EHR) is a critical task for disease prediction. This task aims to effectively extract useful information on dynamic features. Although various existing works have achieved remarkable progress, the model performance can be further improved by fully extracting the trends, variations, and the correlation between the trends and variations in dynamic features. In addition, sparse visit records limit the performance of deep learning models. To address these issues, we propose the Multi-perspective Patient Representation Extractor (MPRE) for disease prediction. Specifically, we propose Frequency Transformation Module (FTM) to extract the trend and variation information of dynamic features in the time-frequency domain, which can enhance the feature representation. In the 2D Multi-Extraction Network (2D MEN), we form the 2D temporal tensor based on trend and variation. Then, the correlations between trend and variation are captured by the proposed dilated operation. Moreover, we propose the First-Order Difference Attention Mechanism (FODAM) to calculate the contributions of differences in adjacent variations to the disease diagnosis adaptively. To evaluate the performance of MPRE and baseline methods, we conduct extensive experiments on two real-world public datasets. The experiment results show that MPRE outperforms state-of-the-art baseline methods in terms of AUROC and AUPRC.

membership infer

federate

Title: Horizontal Federated Computer Vision. (arXiv:2401.00390v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00390
• Code URL: null
• Copy Paste: [[2401.00390]] Horizontal Federated Computer Vision(http://arxiv.org/abs/2401.00390)
• Summary:

  In the modern world, the amount of visual data recorded has been rapidly increasing. In many cases, data is stored in geographically distinct locations and thus requires a large amount of time and space to consolidate. Sometimes, there are also regulations for privacy protection which prevent data consolidation. In this work, we present federated implementations for object detection and recognition using a federated Faster R-CNN (FRCNN) and image segmentation using a federated Fully Convolutional Network (FCN). Our FRCNN was trained on 5000 examples of the COCO2017 dataset while our FCN was trained on the entire train set of the CamVid dataset. The proposed federated models address the challenges posed by the increasing volume and decentralized nature of visual data, offering efficient solutions in compliance with privacy regulations.

Title: Client-wise Modality Selection for Balanced Multi-modal Federated Learning. (arXiv:2401.00403v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00403
• Code URL: null
• Copy Paste: [[2401.00403]] Client-wise Modality Selection for Balanced Multi-modal Federated Learning(http://arxiv.org/abs/2401.00403)
• Summary:

  Selecting proper clients to participate in the iterative federated learning (FL) rounds is critical to effectively harness a broad range of distributed datasets. Existing client selection methods simply consider the variability among FL clients with uni-modal data, however, have yet to consider clients with multi-modalities. We reveal that traditional client selection scheme in MFL may suffer from a severe modality-level bias, which impedes the collaborative exploitation of multi-modal data, leading to insufficient local data exploration and global aggregation. To tackle this challenge, we propose a Client-wise Modality Selection scheme for MFL (CMSFed) that can comprehensively utilize information from each modality via avoiding such client selection bias caused by modality imbalance. Specifically, in each MFL round, the local data from different modalities are selectively employed to participate in local training and aggregation to mitigate potential modality imbalance of the global model. To approximate the fully aggregated model update in a balanced way, we introduce a novel local training loss function to enhance the weak modality and align the divergent feature spaces caused by inconsistent modality adoption strategies for different clients simultaneously. Then, a modality-level gradient decoupling method is designed to derive respective submodular functions to maintain the gradient diversity during the selection progress and balance MFL according to local modality imbalance in each iteration. Our extensive experiments showcase the superiority of CMSFed over baselines and its effectiveness in multi-modal data exploitation.

Title: Federated Class-Incremental Learning with New-Class Augmented Self-Distillation. (arXiv:2401.00622v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00622
• Code URL: null
• Copy Paste: [[2401.00622]] Federated Class-Incremental Learning with New-Class Augmented Self-Distillation(http://arxiv.org/abs/2401.00622)
• Summary:

  Federated Learning (FL) enables collaborative model training among participants while guaranteeing the privacy of raw data. Mainstream FL methodologies overlook the dynamic nature of real-world data, particularly its tendency to grow in volume and diversify in classes over time. This oversight results in FL methods suffering from catastrophic forgetting, where models inadvertently discard previously learned information upon assimilating new data. In response to this challenge, we propose a novel Federated Class-Incremental Learning (FCIL) method, named FCIL with New-Class Augmented Self-Distillation (FedNASD). FedNASD combines new class scores, which are inferred from current models, with historical models' predictions. Based on the combined past and present knowledge, it incorporates self-distillation over models on clients, aiming to achieve effective knowledge transfer from historical models to current models. Theoretical analysis demonstrates that FedNASD is equivalent to modeling old class scores as conditional probabilities in the absence of new classes. Additionally, it reconciles the predictions of new classes with current models to refine the conditional probabilities of historical scores where new classes do not exist. Empirical experiments demonstrate the superiority of FedNASD over four baseline algorithms in reducing the average forgetting rate and boosting global accuracy.

Title: Communication-Efficient Federated Learning for LEO Constellations Integrated with HAPs Using Hybrid NOMA-OFDM. (arXiv:2401.00685v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00685
• Code URL: null
• Copy Paste: [[2401.00685]] Communication-Efficient Federated Learning for LEO Constellations Integrated with HAPs Using Hybrid NOMA-OFDM(http://arxiv.org/abs/2401.00685)
• Summary:

  Space AI has become increasingly important and sometimes even necessary for government, businesses, and society. An active research topic under this mission is integrating federated learning (FL) with satellite communications (SatCom) so that numerous low Earth orbit (LEO) satellites can collaboratively train a machine learning model. However, the special communication environment of SatCom leads to a very slow FL training process up to days and weeks. This paper proposes NomaFedHAP, a novel FL-SatCom approach tailored to LEO satellites, that (1) utilizes high-altitude platforms (HAPs) as distributed parameter servers (PS) to enhance satellite visibility, and (2) introduces non-orthogonal multiple access (NOMA) into LEO to enable fast and bandwidth-efficient model transmissions. In addition, NomaFedHAP includes (3) a new communication topology that exploits HAPs to bridge satellites among different orbits to mitigate the Doppler shift, and (4) a new FL model aggregation scheme that optimally balances models between different orbits and shells. Moreover, we (5) derive a closed-form expression of the outage probability for satellites in near and far shells, as well as for the entire system. Our extensive simulations have validated the mathematical analysis and demonstrated the superior performance of NomaFedHAP in achieving fast and efficient FL model convergence with high accuracy as compared to the state-of-the-art.

fair

Title: Automatic Essay Scoring in a Brazilian Scenario. (arXiv:2401.00095v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00095
• Code URL: null
• Copy Paste: [[2401.00095]] Automatic Essay Scoring in a Brazilian Scenario(http://arxiv.org/abs/2401.00095)
• Summary:

  This paper presents a novel Automatic Essay Scoring (AES) algorithm tailored for the Portuguese-language essays of Brazil's Exame Nacional do Ensino M\'edio (ENEM), addressing the challenges in traditional human grading systems. Our approach leverages advanced deep learning techniques to align closely with human grading criteria, targeting efficiency and scalability in evaluating large volumes of student essays. This research not only responds to the logistical and financial constraints of manual grading in Brazilian educational assessments but also promises to enhance fairness and consistency in scoring, marking a significant step forward in the application of AES in large-scale academic settings.

Title: Fairness-Enhancing Vehicle Rebalancing in the Ride-hailing System. (arXiv:2401.00093v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00093
• Code URL: null
• Copy Paste: [[2401.00093]] Fairness-Enhancing Vehicle Rebalancing in the Ride-hailing System(http://arxiv.org/abs/2401.00093)
• Summary:

  The rapid growth of the ride-hailing industry has revolutionized urban transportation worldwide. Despite its benefits, equity concerns arise as underserved communities face limited accessibility to affordable ride-hailing services. A key issue in this context is the vehicle rebalancing problem, where idle vehicles are moved to areas with anticipated demand. Without equitable approaches in demand forecasting and rebalancing strategies, these practices can further deepen existing inequities. In the realm of ride-hailing, three main facets of fairness are recognized: algorithmic fairness, fairness to drivers, and fairness to riders. This paper focuses on enhancing both algorithmic and rider fairness through a novel vehicle rebalancing method. We introduce an approach that combines a Socio-Aware Spatial-Temporal Graph Convolutional Network (SA-STGCN) for refined demand prediction and a fairness-integrated Matching-Integrated Vehicle Rebalancing (MIVR) model for subsequent vehicle rebalancing. Our methodology is designed to reduce prediction discrepancies and ensure equitable service provision across diverse regions. The effectiveness of our system is evaluated using simulations based on real-world ride-hailing data. The results suggest that our proposed method enhances both accuracy and fairness in forecasting ride-hailing demand, ultimately resulting in more equitable vehicle rebalancing in subsequent operations. Specifically, the algorithm developed in this study effectively reduces the standard deviation and average customer wait times by 6.48% and 0.49%, respectively. This achievement signifies a beneficial outcome for ride-hailing platforms, striking a balance between operational efficiency and fairness.

interpretability

Title: Graph-Convolutional Autoencoder Ensembles for the Humanities, Illustrated with a Study of the American Slave Trade. (arXiv:2401.00824v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00824
• Code URL: null
• Copy Paste: [[2401.00824]] Graph-Convolutional Autoencoder Ensembles for the Humanities, Illustrated with a Study of the American Slave Trade(http://arxiv.org/abs/2401.00824)
• Summary:

  We introduce a graph-aware autoencoder ensemble framework, with associated formalisms and tooling, designed to facilitate deep learning for scholarship in the humanities. By composing sub-architectures to produce a model isomorphic to a humanistic domain we maintain interpretability while providing function signatures for each sub-architectural choice, allowing both traditional and computational researchers to collaborate without disrupting established practices. We illustrate a practical application of our approach to a historical study of the American post-Atlantic slave trade, and make several specific technical contributions: a novel hybrid graph-convolutional autoencoder mechanism, batching policies for common graph topologies, and masking techniques for particular use-cases. The effectiveness of the framework for broadening participation of diverse domains is demonstrated by a growing suite of two dozen studies, both collaborations with humanists and established tasks from machine learning literature, spanning a variety of fields and data modalities. We make performance comparisons of several different architectural choices and conclude with an ambitious list of imminent next steps for this research.

Title: Enabling Smart Retrofitting and Performance Anomaly Detection for a Sensorized Vessel: A Maritime Industry Experience. (arXiv:2401.00112v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00112
• Code URL: null
• Copy Paste: [[2401.00112]] Enabling Smart Retrofitting and Performance Anomaly Detection for a Sensorized Vessel: A Maritime Industry Experience(http://arxiv.org/abs/2401.00112)
• Summary:

  The integration of sensorized vessels, enabling real-time data collection and machine learning-driven data analysis marks a pivotal advancement in the maritime industry. This transformative technology not only can enhance safety, efficiency, and sustainability but also usher in a new era of cost-effective and smart maritime transportation in our increasingly interconnected world. This study presents a deep learning-driven anomaly detection system augmented with interpretable machine learning models for identifying performance anomalies in an industrial sensorized vessel, called TUCANA. We Leverage a human-in-the-loop unsupervised process that involves utilizing standard and Long Short-Term Memory (LSTM) autoencoders augmented with interpretable surrogate models, i.e., random forest and decision tree, to add transparency and interpretability to the results provided by the deep learning models. The interpretable models also enable automated rule generation for translating the inference into human-readable rules. Additionally, the process also includes providing a projection of the results using t-distributed stochastic neighbor embedding (t-SNE), which helps with a better understanding of the structure and relationships within the data and assessment of the identified anomalies. We empirically evaluate the system using real data acquired from the vessel TUCANA and the results involve achieving over 80% precision and 90% recall with the LSTM model used in the process. The interpretable models also provide logical rules aligned with expert thinking, and the t-SNE-based projection enhances interpretability. Our system demonstrates that the proposed approach can be used effectively in real-world scenarios, offering transparency and precision in performance anomaly detection.

Title: Interpreting the Curse of Dimensionality from Distance Concentration and Manifold Effect. (arXiv:2401.00422v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00422
• Code URL: null
• Copy Paste: [[2401.00422]] Interpreting the Curse of Dimensionality from Distance Concentration and Manifold Effect(http://arxiv.org/abs/2401.00422)
• Summary:

  The characteristics and interpretability of data become more abstract and complex as the dimensionality increases. Common patterns and relationships that hold in in low-dimensional space may fail to hold in higher-dimensional space. This phenomenon leads to a decreasing performance for the regression, classification or clustering models or algorithms, which is known as curse of dimensionality. Curse of dimensionality can be attributed to many causes. In this paper, we first summarize five challenges associated with manipulating high-dimensional data, and explains the potential causes for the failure of regression, classification or clustering tasks. Subsequently, we delve into two major causes of the curse of dimensionality, distance concentration and manifold effect, by performing theoretical and empirical analyses. The results demonstrate that nearest neighbor search (NNS) using three typical distance measurements, Minkowski distance, Chebyshev distance, and cosine distance, becomes meaningless as the dimensionality increases. Meanwhile, the data incorporates more redundant features, and the variance contribution of principal component analysis (PCA) is skewed towards a few dimensions. By interpreting the causes of the curse of dimensionality, we can better understand the limitations of current models and algorithms, and drive to improve the performance of data analysis and machine learning tasks in high-dimensional space.

Title: Financial Time-Series Forecasting: Towards Synergizing Performance And Interpretability Within a Hybrid Machine Learning Approach. (arXiv:2401.00534v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00534
• Code URL: null
• Copy Paste: [[2401.00534]] Financial Time-Series Forecasting: Towards Synergizing Performance And Interpretability Within a Hybrid Machine Learning Approach(http://arxiv.org/abs/2401.00534)
• Summary:

  In the realm of cryptocurrency, the prediction of Bitcoin prices has garnered substantial attention due to its potential impact on financial markets and investment strategies. This paper propose a comparative study on hybrid machine learning algorithms and leverage on enhancing model interpretability. Specifically, linear regression(OLS, LASSO), long-short term memory(LSTM), decision tree regressors are introduced. Through the grounded experiments, we observe linear regressor achieves the best performance among candidate models. For the interpretability, we carry out a systematic overview on the preprocessing techniques of time-series statistics, including decomposition, auto-correlational function, exponential triple forecasting, which aim to excavate latent relations and complex patterns appeared in the financial time-series forecasting. We believe this work may derive more attention and inspire more researches in the realm of time-series analysis and its realistic applications.

explainability

Title: Explainability-Driven Leaf Disease Classification using Adversarial Training and Knowledge Distillation. (arXiv:2401.00334v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00334
• Code URL: null
• Copy Paste: [[2401.00334]] Explainability-Driven Leaf Disease Classification using Adversarial Training and Knowledge Distillation(http://arxiv.org/abs/2401.00334)
• Summary:

  This work focuses on plant leaf disease classification and explores three crucial aspects: adversarial training, model explainability, and model compression. The models' robustness against adversarial attacks is enhanced through adversarial training, ensuring accurate classification even in the presence of threats. Leveraging explainability techniques, we gain insights into the model's decision-making process, improving trust and transparency. Additionally, we explore model compression techniques to optimize computational efficiency while maintaining classification performance. Through our experiments, we determine that on a benchmark dataset, the robustness can be the price of the classification accuracy with performance reductions of 3%-20% for regular tests and gains of 50%-70% for adversarial attack tests. We also demonstrate that a student model can be 15-25 times more computationally efficient for a slight performance reduction, distilling the knowledge of more complex models.

watermark

diffusion

Title: 6D-Diff: A Keypoint Diffusion Framework for 6D Object Pose Estimation. (arXiv:2401.00029v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00029
• Code URL: null
• Copy Paste: [[2401.00029]] 6D-Diff: A Keypoint Diffusion Framework for 6D Object Pose Estimation(http://arxiv.org/abs/2401.00029)
• Summary:

  Estimating the 6D object pose from a single RGB image often involves noise and indeterminacy due to challenges such as occlusions and cluttered backgrounds. Meanwhile, diffusion models have shown appealing performance in generating high-quality images from random noise with high indeterminacy through step-by-step denoising. Inspired by their denoising capability, we propose a novel diffusion-based framework (6D-Diff) to handle the noise and indeterminacy in object pose estimation for better performance. In our framework, to establish accurate 2D-3D correspondence, we formulate 2D keypoints detection as a reverse diffusion (denoising) process. To facilitate such a denoising process, we design a Mixture-of-Cauchy-based forward diffusion process and condition the reverse process on the object features. Extensive experiments on the LM-O and YCB-V datasets demonstrate the effectiveness of our framework.

Title: Generating Enhanced Negatives for Training Language-Based Object Detectors. (arXiv:2401.00094v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00094
• Code URL: null
• Copy Paste: [[2401.00094]] Generating Enhanced Negatives for Training Language-Based Object Detectors(http://arxiv.org/abs/2401.00094)
• Summary:

  The recent progress in language-based open-vocabulary object detection can be largely attributed to finding better ways of leveraging large-scale data with free-form text annotations. Training such models with a discriminative objective function has proven successful, but requires good positive and negative samples. However, the free-form nature and the open vocabulary of object descriptions make the space of negatives extremely large. Prior works randomly sample negatives or use rule-based techniques to build them. In contrast, we propose to leverage the vast knowledge built into modern generative models to automatically build negatives that are more relevant to the original data. Specifically, we use large-language-models to generate negative text descriptions, and text-to-image diffusion models to also generate corresponding negative images. Our experimental analysis confirms the relevance of the generated negative data, and its use in language-based detectors improves performance on two complex benchmarks.

Title: Diffusion Model with Perceptual Loss. (arXiv:2401.00110v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00110
• Code URL: null
• Copy Paste: [[2401.00110]] Diffusion Model with Perceptual Loss(http://arxiv.org/abs/2401.00110)
• Summary:

  Diffusion models trained with mean squared error loss tend to generate unrealistic samples. Current state-of-the-art models rely on classifier-free guidance to improve sample quality, yet its surprising effectiveness is not fully understood. In this paper, We show that the effectiveness of classifier-free guidance partly originates from it being a form of implicit perceptual guidance. As a result, we can directly incorporate perceptual loss in diffusion training to improve sample quality. Since the score matching objective used in diffusion training strongly resembles the denoising autoencoder objective used in unsupervised training of perceptual networks, the diffusion model itself is a perceptual network and can be used to generate meaningful perceptual loss. We propose a novel self-perceptual objective that results in diffusion models capable of generating more realistic samples. For conditional generation, our method only improves sample quality without entanglement with the conditional input and therefore does not sacrifice sample diversity. Our method can also improve sample quality for unconditional generation, which was not possible with classifier-free guidance before.

Title: Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with Generative Diffusion Models. (arXiv:2401.00208v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00208
• Code URL: null
• Copy Paste: [[2401.00208]] Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with Generative Diffusion Models(http://arxiv.org/abs/2401.00208)
• Summary:

  Current Neural Radiance Fields (NeRF) can generate photorealistic novel views. For editing 3D scenes represented by NeRF, with the advent of generative models, this paper proposes Inpaint4DNeRF to capitalize on state-of-the-art stable diffusion models (e.g., ControlNet) for direct generation of the underlying completed background content, regardless of static or dynamic. The key advantages of this generative approach for NeRF inpainting are twofold. First, after rough mask propagation, to complete or fill in previously occluded content, we can individually generate a small subset of completed images with plausible content, called seed images, from which simple 3D geometry proxies can be derived. Second and the remaining problem is thus 3D multiview consistency among all completed images, now guided by the seed images and their 3D proxies. Without other bells and whistles, our generative Inpaint4DNeRF baseline framework is general which can be readily extended to 4D dynamic NeRFs, where temporal consistency can be naturally handled in a similar way as our multiview consistency.

Title: Probing the Limits and Capabilities of Diffusion Models for the Anatomic Editing of Digital Twins. (arXiv:2401.00247v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00247
• Code URL: null
• Copy Paste: [[2401.00247]] Probing the Limits and Capabilities of Diffusion Models for the Anatomic Editing of Digital Twins(http://arxiv.org/abs/2401.00247)
• Summary:

  Numerical simulations can model the physical processes that govern cardiovascular device deployment. When such simulations incorporate digital twins; computational models of patient-specific anatomy, they can expedite and de-risk the device design process. Nonetheless, the exclusive use of patient-specific data constrains the anatomic variability which can be precisely or fully explored. In this study, we investigate the capacity of Latent Diffusion Models (LDMs) to edit digital twins to create anatomic variants, which we term digital siblings. Digital twins and their corresponding siblings can serve as the basis for comparative simulations, enabling the study of how subtle anatomic variations impact the simulated deployment of cardiovascular devices, as well as the augmentation of virtual cohorts for device assessment. However, while diffusion models have been characterized in their ability to edit natural images, their capacity to anatomically edit digital twins has yet to be studied. Using a case example centered on 3D digital twins of cardiac anatomy, we implement various methods for generating digital siblings and characterize them through morphological and topological analyses. We specifically edit digital twins to introduce anatomic variation at different spatial scales and within localized regions, demonstrating the existence of bias towards common anatomic features. We further show that such anatomic bias can be leveraged for virtual cohort augmentation through selective editing, partially alleviating issues related to dataset imbalance and lack of diversity. Our experimental framework thus delineates the limits and capabilities of using latent diffusion models in synthesizing anatomic variation for in silico trials.

Title: Diff-PCR: Diffusion-Based Correspondence Searching in Doubly Stochastic Matrix Space for Point Cloud Registration. (arXiv:2401.00436v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00436
• Code URL: null
• Copy Paste: [[2401.00436]] Diff-PCR: Diffusion-Based Correspondence Searching in Doubly Stochastic Matrix Space for Point Cloud Registration(http://arxiv.org/abs/2401.00436)
• Summary:

  Efficiently finding optimal correspondences between point clouds is crucial for solving both rigid and non-rigid point cloud registration problems. Existing methods often rely on geometric or semantic feature embedding to establish correspondences and estimate transformations or flow fields. Recently, state-of-the-art methods have employed RAFT-like iterative updates to refine the solution. However, these methods have certain limitations. Firstly, their iterative refinement design lacks transparency, and their iterative updates follow a fixed path during the refinement process, which can lead to suboptimal results. Secondly, these methods overlook the importance of refining or optimizing correspondences (or matching matrices) as a precursor to solving transformations or flow fields. They typically compute candidate correspondences based on distances in the point feature space. However, they only project the candidate matching matrix into some matrix space once with Sinkhorn or dual softmax operations to obtain final correspondences. This one-shot projected matching matrix may be far from the globally optimal one, and these approaches do not consider the distribution of the target matching matrix. In this paper, we propose a novel approach that exploits the Denoising Diffusion Model to predict a searching gradient for the optimal matching matrix within the Doubly Stochastic Matrix Space. During the reverse denoising process, our method iteratively searches for better solutions along this denoising gradient, which points towards the maximum likelihood direction of the target matching matrix. Our method offers flexibility by allowing the search to start from any initial matching matrix provided by the online backbone or white noise. Experimental evaluations on the 3DMatch/3DLoMatch and 4DMatch/4DLoMatch datasets demonstrate the effectiveness of our newly designed framework.

Title: A Generalist FaceX via Learning Unified Facial Representation. (arXiv:2401.00551v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00551
• Code URL: null
• Copy Paste: [[2401.00551]] A Generalist FaceX via Learning Unified Facial Representation(http://arxiv.org/abs/2401.00551)
• Summary:

  This work presents FaceX framework, a novel facial generalist model capable of handling diverse facial tasks simultaneously. To achieve this goal, we initially formulate a unified facial representation for a broad spectrum of facial editing tasks, which macroscopically decomposes a face into fundamental identity, intra-personal variation, and environmental factors. Based on this, we introduce Facial Omni-Representation Decomposing (FORD) for seamless manipulation of various facial components, microscopically decomposing the core aspects of most facial editing tasks. Furthermore, by leveraging the prior of a pretrained StableDiffusion (SD) to enhance generation quality and accelerate training, we design Facial Omni-Representation Steering (FORS) to first assemble unified facial representations and then effectively steer the SD-aware generation process by the efficient Facial Representation Controller (FRC). %Without any additional features, Our versatile FaceX achieves competitive performance compared to elaborate task-specific models on popular facial editing tasks. Full codes and models will be available at https://github.com/diffusion-facex/FaceX.

Title: GD^2-NeRF: Generative Detail Compensation via GAN and Diffusion for One-shot Generalizable Neural Radiance Fields. (arXiv:2401.00616v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00616
• Code URL: null
• Copy Paste: [[2401.00616]] GD^2-NeRF: Generative Detail Compensation via GAN and Diffusion for One-shot Generalizable Neural Radiance Fields(http://arxiv.org/abs/2401.00616)
• Summary:

  In this paper, we focus on the One-shot Novel View Synthesis (O-NVS) task which targets synthesizing photo-realistic novel views given only one reference image per scene. Previous One-shot Generalizable Neural Radiance Fields (OG-NeRF) methods solve this task in an inference-time finetuning-free manner, yet suffer the blurry issue due to the encoder-only architecture that highly relies on the limited reference image. On the other hand, recent diffusion-based image-to-3d methods show vivid plausible results via distilling pre-trained 2D diffusion models into a 3D representation, yet require tedious per-scene optimization. Targeting these issues, we propose the GD^2-NeRF, a Generative Detail compensation framework via GAN and Diffusion that is both inference-time finetuning-free and with vivid plausible details. In detail, following a coarse-to-fine strategy, GD^2-NeRF is mainly composed of a One-stage Parallel Pipeline (OPP) and a 3D-consistent Detail Enhancer (Diff3DE). At the coarse stage, OPP first efficiently inserts the GAN model into the existing OG-NeRF pipeline for primarily relieving the blurry issue with in-distribution priors captured from the training dataset, achieving a good balance between sharpness (LPIPS, FID) and fidelity (PSNR, SSIM). Then, at the fine stage, Diff3DE further leverages the pre-trained image diffusion models to complement rich out-distribution details while maintaining decent 3D consistency. Extensive experiments on both the synthetic and real-world datasets show that GD$^2$-NeRF noticeably improves the details while without per-scene finetuning.

Title: Diffusion Models, Image Super-Resolution And Everything: A Survey. (arXiv:2401.00736v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00736
• Code URL: null
• Copy Paste: [[2401.00736]] Diffusion Models, Image Super-Resolution And Everything: A Survey(http://arxiv.org/abs/2401.00736)
• Summary:

  Diffusion Models (DMs) represent a significant advancement in image Super-Resolution (SR), aligning technical image quality more closely with human preferences and expanding SR applications. DMs address critical limitations of previous methods, enhancing overall realism and details in SR images. However, DMs suffer from color-shifting issues, and their high computational costs call for efficient sampling alternatives, underscoring the challenge of balancing computational efficiency and image quality. This survey gives an overview of DMs applied to image SR and offers a detailed analysis that underscores the unique characteristics and methodologies within this domain, distinct from broader existing reviews in the field. It presents a unified view of DM fundamentals and explores research directions, including alternative input domains, conditioning strategies, guidance, corruption spaces, and zero-shot methods. This survey provides insights into the evolution of image SR with DMs, addressing current trends, challenges, and future directions in this rapidly evolving field.

Title: DiffMorph: Text-less Image Morphing with Diffusion Models. (arXiv:2401.00739v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00739
• Code URL: null
• Copy Paste: [[2401.00739]] DiffMorph: Text-less Image Morphing with Diffusion Models(http://arxiv.org/abs/2401.00739)
• Summary:

  Text-conditioned image generation models are a prevalent use of AI image synthesis, yet intuitively controlling output guided by an artist remains challenging. Current methods require multiple images and textual prompts for each object to specify them as concepts to generate a single customized image.

  On the other hand, our work, \verb|DiffMorph|, introduces a novel approach that synthesizes images that mix concepts without the use of textual prompts. Our work integrates a sketch-to-image module to incorporate user sketches as input. \verb|DiffMorph| takes an initial image with conditioning artist-drawn sketches to generate a morphed image.

  We employ a pre-trained text-to-image diffusion model and fine-tune it to reconstruct each image faithfully. We seamlessly merge images and concepts from sketches into a cohesive composition. The image generation capability of our work is demonstrated through our results and a comparison of these with prompt-based image generation.

noise learning

data-free

transformer

Title: Image Super-resolution Reconstruction Network based on Enhanced Swin Transformer via Alternating Aggregation of Local-Global Features. (arXiv:2401.00241v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00241
• Code URL: null
• Copy Paste: [[2401.00241]] Image Super-resolution Reconstruction Network based on Enhanced Swin Transformer via Alternating Aggregation of Local-Global Features(http://arxiv.org/abs/2401.00241)
• Summary:

  The Swin Transformer image super-resolution reconstruction network only relies on the long-range relationship of window attention and shifted window attention to explore features. This mechanism has two limitations. On the one hand, it only focuses on global features while ignoring local features. On the other hand, it is only concerned with spatial feature interactions while ignoring channel features and channel interactions, thus limiting its non-linear mapping ability. To address the above limitations, this paper proposes enhanced Swin Transformer modules via alternating aggregation of local-global features. In the local feature aggregation stage, this paper introduces shift convolution to realize the interaction between local spatial information and channel information. This paper proposes a block sparse global perception module in the global feature aggregation stage. This module organizes the spatial information first, then sends the recombination information into a spatial gating unit to implement the further interaction of spatial and channel information. Then, a multi-scale self-attention module and a low-parameter residual channel attention module are introduced to realize information aggregation at different scales. Finally, the proposed network is validated on five publicly available datasets. The experimental results show that the proposed network outperforms the other state-of-the-art super-resolution networks.

Title: Masked Image Modeling via Dynamic Token Morphing. (arXiv:2401.00254v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00254
• Code URL: null
• Copy Paste: [[2401.00254]] Masked Image Modeling via Dynamic Token Morphing(http://arxiv.org/abs/2401.00254)
• Summary:

  Masked Image Modeling (MIM) arises as a promising option for Vision Transformers among various self-supervised learning (SSL) methods. The essence of MIM lies in token-wise masked patch predictions, with targets patchified from images; or generated by pre-trained tokenizers or models. We argue targets from the pre-trained models usually exhibit spatial inconsistency, which makes it excessively challenging for the model to follow to learn more discriminative representations. To mitigate the issue, we introduce a novel self-supervision signal based on Dynamic Token Morphing (DTM), which dynamically aggregates contextually related tokens. DTM can be generally applied to various SSL frameworks, yet we propose a simple MIM that employs DTM to effectively improve the performance barely introducing extra training costs. Our experiments on ImageNet-1K and ADE20K evidently demonstrate the superiority of our methods. Furthermore, the comparative evaluation of iNaturalist and Fine-grained Visual Classification datasets further validates the transferability of our method on various downstream tasks. Our code will be released publicly.

Title: COMMA: Co-Articulated Multi-Modal Learning. (arXiv:2401.00268v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00268
• Code URL: null
• Copy Paste: [[2401.00268]] COMMA: Co-Articulated Multi-Modal Learning(http://arxiv.org/abs/2401.00268)
• Summary:

  Pretrained large-scale vision-language models such as CLIP have demonstrated excellent generalizability over a series of downstream tasks. However, they are sensitive to the variation of input text prompts and need a selection of prompt templates to achieve satisfactory performance. Recently, various methods have been proposed to dynamically learn the prompts as the textual inputs to avoid the requirements of laboring hand-crafted prompt engineering in the fine-tuning process. We notice that these methods are suboptimal in two aspects. First, the prompts of the vision and language branches in these methods are usually separated or uni-directionally correlated. Thus, the prompts of both branches are not fully correlated and may not provide enough guidance to align the representations of both branches. Second, it's observed that most previous methods usually achieve better performance on seen classes but cause performance degeneration on unseen classes compared to CLIP. This is because the essential generic knowledge learned in the pretraining stage is partly forgotten in the fine-tuning process. In this paper, we propose Co-Articulated Multi-Modal Learning (COMMA) to handle the above limitations. Especially, our method considers prompts from both branches to generate the prompts to enhance the representation alignment of both branches. Besides, to alleviate forgetting about the essential knowledge, we minimize the feature discrepancy between the learned prompts and the embeddings of hand-crafted prompts in the pre-trained CLIP in the late transformer layers. We evaluate our method across three representative tasks of generalization to novel classes, new target datasets and unseen domain shifts. Experimental results demonstrate the superiority of our method by exhibiting a favorable performance boost upon all tasks with high efficiency.

Title: HybridGait: A Benchmark for Spatial-Temporal Cloth-Changing Gait Recognition with Hybrid Explorations. (arXiv:2401.00271v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00271
• Code URL: null
• Copy Paste: [[2401.00271]] HybridGait: A Benchmark for Spatial-Temporal Cloth-Changing Gait Recognition with Hybrid Explorations(http://arxiv.org/abs/2401.00271)
• Summary:

  Existing gait recognition benchmarks mostly include minor clothing variations in the laboratory environments, but lack persistent changes in appearance over time and space. In this paper, we propose the first in-the-wild benchmark CCGait for cloth-changing gait recognition, which incorporates diverse clothing changes, indoor and outdoor scenes, and multi-modal statistics over 92 days. To further address the coupling effect of clothing and viewpoint variations, we propose a hybrid approach HybridGait that exploits both temporal dynamics and the projected 2D information of 3D human meshes. Specifically, we introduce a Canonical Alignment Spatial-Temporal Transformer (CA-STT) module to encode human joint position-aware features, and fully exploit 3D dense priors via a Silhouette-guided Deformation with 3D-2D Appearance Projection (SilD) strategy. Our contributions are twofold: we provide a challenging benchmark CCGait that captures realistic appearance changes across an expanded and space, and we propose a hybrid framework HybridGait that outperforms prior works on CCGait and Gait3D benchmarks. Our project page is available at https://github.com/HCVLab/HybridGait.

Title: EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Masked Audio Gesture Modeling. (arXiv:2401.00374v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00374
• Code URL: null
• Copy Paste: [[2401.00374]] EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Masked Audio Gesture Modeling(http://arxiv.org/abs/2401.00374)
• Summary:

  We propose EMAGE, a framework to generate full-body human gestures from audio and masked gestures, encompassing facial, local body, hands, and global movements. To achieve this, we first introduce BEATX (BEAT-SMPLX-FLAME), a new mesh-level holistic co-speech dataset. BEATX combines MoShed SMPLX body with FLAME head parameters and further refines the modeling of head, neck, and finger movements, offering a community-standardized, high-quality 3D motion captured dataset. EMAGE leverages masked body gesture priors during training to boost inference performance. It involves a Masked Audio Gesture Transformer, facilitating joint training on audio-to-gesture generation and masked gesture reconstruction to effectively encode audio and body gesture hints. Encoded body hints from masked gestures are then separately employed to generate facial and body movements. Moreover, EMAGE adaptively merges speech features from the audio's rhythm and content and utilizes four compositional VQ-VAEs to enhance the results' fidelity and diversity. Experiments demonstrate that EMAGE generates holistic gestures with state-of-the-art performance and is flexible in accepting predefined spatial-temporal gesture inputs, generating complete, audio-synchronized results. Our code and dataset are available at https://pantomatrix.github.io/EMAGE/

Title: A Two-stream Hybrid CNN-Transformer Network for Skeleton-based Human Interaction Recognition. (arXiv:2401.00409v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00409
• Code URL: null
• Copy Paste: [[2401.00409]] A Two-stream Hybrid CNN-Transformer Network for Skeleton-based Human Interaction Recognition(http://arxiv.org/abs/2401.00409)
• Summary:

  Human Interaction Recognition is the process of identifying interactive actions between multiple participants in a specific situation. The aim is to recognise the action interactions between multiple entities and their meaning. Many single Convolutional Neural Network has issues, such as the inability to capture global instance interaction features or difficulty in training, leading to ambiguity in action semantics. In addition, the computational complexity of the Transformer cannot be ignored, and its ability to capture local information and motion features in the image is poor. In this work, we propose a Two-stream Hybrid CNN-Transformer Network (THCT-Net), which exploits the local specificity of CNN and models global dependencies through the Transformer. CNN and Transformer simultaneously model the entity, time and space relationships between interactive entities respectively. Specifically, Transformer-based stream integrates 3D convolutions with multi-head self-attention to learn inter-token correlations; We propose a new multi-branch CNN framework for CNN-based streams that automatically learns joint spatio-temporal features from skeleton sequences. The convolutional layer independently learns the local features of each joint neighborhood and aggregates the features of all joints. And the raw skeleton coordinates as well as their temporal difference are integrated with a dual-branch paradigm to fuse the motion features of the skeleton. Besides, a residual structure is added to speed up training convergence. Finally, the recognition results of the two branches are fused using parallel splicing. Experimental results on diverse and challenging datasets, demonstrate that the proposed method can better comprehend and infer the meaning and context of various actions, outperforming state-of-the-art methods.

Title: SVFAP: Self-supervised Video Facial Affect Perceiver. (arXiv:2401.00416v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00416
• Code URL: null
• Copy Paste: [[2401.00416]] SVFAP: Self-supervised Video Facial Affect Perceiver(http://arxiv.org/abs/2401.00416)
• Summary:

  Video-based facial affect analysis has recently attracted increasing attention owing to its critical role in human-computer interaction. Previous studies mainly focus on developing various deep learning architectures and training them in a fully supervised manner. Although significant progress has been achieved by these supervised methods, the longstanding lack of large-scale high-quality labeled data severely hinders their further improvements. Motivated by the recent success of self-supervised learning in computer vision, this paper introduces a self-supervised approach, termed Self-supervised Video Facial Affect Perceiver (SVFAP), to address the dilemma faced by supervised methods. Specifically, SVFAP leverages masked facial video autoencoding to perform self-supervised pre-training on massive unlabeled facial videos. Considering that large spatiotemporal redundancy exists in facial videos, we propose a novel temporal pyramid and spatial bottleneck Transformer as the encoder of SVFAP, which not only enjoys low computational cost but also achieves excellent performance. To verify the effectiveness of our method, we conduct experiments on nine datasets spanning three downstream tasks, including dynamic facial expression recognition, dimensional emotion recognition, and personality recognition. Comprehensive results demonstrate that SVFAP can learn powerful affect-related representations via large-scale self-supervised pre-training and it significantly outperforms previous state-of-the-art methods on all datasets. Codes will be available at https://github.com/sunlicai/SVFAP.

Title: Analyzing Local Representations of Self-supervised Vision Transformers. (arXiv:2401.00463v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00463
• Code URL: null
• Copy Paste: [[2401.00463]] Analyzing Local Representations of Self-supervised Vision Transformers(http://arxiv.org/abs/2401.00463)
• Summary:

  In this paper, we present a comparative analysis of various self-supervised Vision Transformers (ViTs), focusing on their local representative power. Inspired by large language models, we examine the abilities of ViTs to perform various computer vision tasks with little to no fine-tuning. We design an evaluation framework to analyze the quality of local, i.e. patch-level, representations in the context of few-shot semantic segmentation, instance identification, object retrieval, and tracking. We discover that contrastive learning based methods like DINO produce more universal patch representations that can be immediately applied for downstream tasks with no parameter tuning, compared to masked image modeling. The embeddings learned using the latter approach, e.g. in masked autoencoders, have high variance features that harm distance-based algorithms, such as k-NN, and do not contain useful information for most downstream tasks. Furthermore, we demonstrate that removing these high-variance features enhances k-NN by providing an analysis of the benchmarks for this work and for Scale-MAE, a recent extension of masked autoencoders. Finally, we find an object instance retrieval setting where DINOv2, a model pretrained on two orders of magnitude more data, performs worse than its less compute-intensive counterpart DINO.

Title: BRAU-Net++: U-Shaped Hybrid CNN-Transformer Network for Medical Image Segmentation. (arXiv:2401.00722v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00722
• Code URL: null
• Copy Paste: [[2401.00722]] BRAU-Net++: U-Shaped Hybrid CNN-Transformer Network for Medical Image Segmentation(http://arxiv.org/abs/2401.00722)
• Summary:

  Accurate medical image segmentation is essential for clinical quantification, disease diagnosis, treatment planning and many other applications. Both convolution-based and transformer-based u-shaped architectures have made significant success in various medical image segmentation tasks. The former can efficiently learn local information of images while requiring much more image-specific inductive biases inherent to convolution operation. The latter can effectively capture long-range dependency at different feature scales using self-attention, whereas it typically encounters the challenges of quadratic compute and memory requirements with sequence length increasing. To address this problem, through integrating the merits of these two paradigms in a well-designed u-shaped architecture, we propose a hybrid yet effective CNN-Transformer network, named BRAU-Net++, for an accurate medical image segmentation task. Specifically, BRAU-Net++ uses bi-level routing attention as the core building block to design our u-shaped encoder-decoder structure, in which both encoder and decoder are hierarchically constructed, so as to learn global semantic information while reducing computational complexity. Furthermore, this network restructures skip connection by incorporating channel-spatial attention which adopts convolution operations, aiming to minimize local spatial information loss and amplify global dimension-interaction of multi-scale features. Extensive experiments on three public benchmark datasets demonstrate that our proposed approach surpasses other state-of-the-art methods including its baseline: BRAU-Net under almost all evaluation metrics. We achieve the average Dice-Similarity Coefficient (DSC) of 82.47, 90.10, and 92.94 on Synapse multi-organ segmentation, ISIC-2018 Challenge, and CVC-ClinicDB, as well as the mIoU of 84.01 and 88.17 on ISIC-2018 Challenge and CVC-ClinicDB, respectively.

Title: Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera. (arXiv:2401.00847v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00847
• Code URL: null
• Copy Paste: [[2401.00847]] Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera(http://arxiv.org/abs/2401.00847)
• Summary:

  We present a lightweight and affordable motion capture method based on two smartwatches and a head-mounted camera. In contrast to the existing approaches that use six or more expert-level IMU devices, our approach is much more cost-effective and convenient. Our method can make wearable motion capture accessible to everyone everywhere, enabling 3D full-body motion capture in diverse environments. As a key idea to overcome the extreme sparsity and ambiguities of sensor inputs, we integrate 6D head poses obtained from the head-mounted cameras for motion estimation. To enable capture in expansive indoor and outdoor scenes, we propose an algorithm to track and update floor level changes to define head poses, coupled with a multi-stage Transformer-based regression module. We also introduce novel strategies leveraging visual cues of egocentric images to further enhance the motion capture quality while reducing ambiguities. We demonstrate the performance of our method on various challenging scenarios, including complex outdoor environments and everyday motions including object interactions and social interactions among multiple individuals.

Title: Temporal Validity Change Prediction. (arXiv:2401.00779v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00779
• Code URL: null
• Copy Paste: [[2401.00779]] Temporal Validity Change Prediction(http://arxiv.org/abs/2401.00779)
• Summary:

  Temporal validity is an important property of text that is useful for many downstream applications, such as recommender systems, conversational AI, or story understanding. Existing benchmarking tasks often require models to identify the temporal validity duration of a single statement. However, in many cases, additional contextual information, such as sentences in a story or posts on a social media profile, can be collected from the available text stream. This contextual information may greatly alter the duration for which a statement is expected to be valid. We propose Temporal Validity Change Prediction, a natural language processing task benchmarking the capability of machine learning models to detect contextual statements that induce such change. We create a dataset consisting of temporal target statements sourced from Twitter and crowdsource sample context statements. We then benchmark a set of transformer-based language models on our dataset. Finally, we experiment with temporal validity duration prediction as an auxiliary task to improve the performance of the state-of-the-art model.

Title: Transformer Multivariate Forecasting: Less is More?. (arXiv:2401.00230v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00230
• Code URL: null
• Copy Paste: [[2401.00230]] Transformer Multivariate Forecasting: Less is More?(http://arxiv.org/abs/2401.00230)
• Summary:

  In the domain of multivariate forecasting, transformer models stand out as powerful apparatus, displaying exceptional capabilities in handling messy datasets from real-world contexts. However, the inherent complexity of these datasets, characterized by numerous variables and lengthy temporal sequences, poses challenges, including increased noise and extended model runtime. This paper focuses on reducing redundant information to elevate forecasting accuracy while optimizing runtime efficiency. We propose a novel transformer forecasting framework enhanced by Principal Component Analysis (PCA) to tackle this challenge. The framework is evaluated by five state-of-the-art (SOTA) models and four diverse real-world datasets. Our experimental results demonstrate the framework's ability to minimize prediction errors across all models and datasets while significantly reducing runtime. From the model perspective, one of the PCA-enhanced models: PCA+Crossformer, reduces mean square errors (MSE) by 33.3% and decreases runtime by 49.2% on average. From the dataset perspective, the framework delivers 14.3% MSE and 76.6% runtime reduction on Electricity datasets, as well as 4.8% MSE and 86.9% runtime reduction on Traffic datasets. This study aims to advance various SOTA models and enhance transformer-based time series forecasting for intricate data.

Title: GraphGPT: Graph Learning with Generative Pre-trained Transformers. (arXiv:2401.00529v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00529
• Code URL: null
• Copy Paste: [[2401.00529]] GraphGPT: Graph Learning with Generative Pre-trained Transformers(http://arxiv.org/abs/2401.00529)
• Summary:

  We introduce \textit{GraphGPT}, a novel model for Graph learning by self-supervised Generative Pre-training Transformers. Our model transforms each graph or sampled subgraph into a sequence of tokens representing the node, edge and attributes reversibly using the Eulerian path first. Then we feed the tokens into a standard transformer decoder and pre-train it with the next-token-prediction (NTP) task. Lastly, we fine-tune the GraphGPT model with the supervised tasks. This intuitive, yet effective model achieves superior or close results to the state-of-the-art methods for the graph-, edge- and node-level tasks on the large scale molecular dataset PCQM4Mv2, the protein-protein association dataset ogbl-ppa and the ogbn-proteins dataset from the Open Graph Benchmark (OGB). Furthermore, the generative pre-training enables us to train GraphGPT up to 400M+ parameters with consistently increasing performance, which is beyond the capability of GNNs and previous graph transformers. The source code and pre-trained checkpoints will be released soon\footnote{\url{https://github.com/alibaba/graph-gpt}} to pave the way for the graph foundation model research, and also to assist the scientific discovery in pharmaceutical, chemistry, material and bio-informatics domains, etc.

generative

Title: Discrete Distribution Networks. (arXiv:2401.00036v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00036
• Code URL: null
• Copy Paste: [[2401.00036]] Discrete Distribution Networks(http://arxiv.org/abs/2401.00036)
• Summary:

  We introduce a novel generative model, the Discrete Distribution Networks (DDN), that approximates data distribution using hierarchical discrete distributions. We posit that since the features within a network inherently contain distributional information, liberating the network from a single output to concurrently generate multiple samples proves to be highly effective. Therefore, DDN fits the target distribution, including continuous ones, by generating multiple discrete sample points. To capture finer details of the target data, DDN selects the output that is closest to the Ground Truth (GT) from the coarse results generated in the first layer. This selected output is then fed back into the network as a condition for the second layer, thereby generating new outputs more similar to the GT. As the number of DDN layers increases, the representational space of the outputs expands exponentially, and the generated samples become increasingly similar to the GT. This hierarchical output pattern of discrete distributions endows DDN with two intriguing properties: highly compressed representation and more general zero-shot conditional generation. We demonstrate the efficacy of DDN and these intriguing properties through experiments on CIFAR-10 and FFHQ.

Title: UGPNet: Universal Generative Prior for Image Restoration. (arXiv:2401.00370v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00370
• Code URL: null
• Copy Paste: [[2401.00370]] UGPNet: Universal Generative Prior for Image Restoration(http://arxiv.org/abs/2401.00370)
• Summary:

  Recent image restoration methods can be broadly categorized into two classes: (1) regression methods that recover the rough structure of the original image without synthesizing high-frequency details and (2) generative methods that synthesize perceptually-realistic high-frequency details even though the resulting image deviates from the original structure of the input. While both directions have been extensively studied in isolation, merging their benefits with a single framework has been rarely studied. In this paper, we propose UGPNet, a universal image restoration framework that can effectively achieve the benefits of both approaches by simply adopting a pair of an existing regression model and a generative model. UGPNet first restores the image structure of a degraded input using a regression model and synthesizes a perceptually-realistic image with a generative model on top of the regressed output. UGPNet then combines the regressed output and the synthesized output, resulting in a final result that faithfully reconstructs the structure of the original image in addition to perceptually-realistic textures. Our extensive experiments on deblurring, denoising, and super-resolution demonstrate that UGPNet can successfully exploit both regression and generative methods for high-fidelity image restoration.

Title: Generative Model-Driven Synthetic Training Image Generation: An Approach to Cognition in Rail Defect Detection. (arXiv:2401.00393v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00393
• Code URL: null
• Copy Paste: [[2401.00393]] Generative Model-Driven Synthetic Training Image Generation: An Approach to Cognition in Rail Defect Detection(http://arxiv.org/abs/2401.00393)
• Summary:

  Recent advancements in cognitive computing, with the integration of deep learning techniques, have facilitated the development of intelligent cognitive systems (ICS). This is particularly beneficial in the context of rail defect detection, where the ICS would emulate human-like analysis of image data for defect patterns. Despite the success of Convolutional Neural Networks (CNN) in visual defect classification, the scarcity of large datasets for rail defect detection remains a challenge due to infrequent accident events that would result in defective parts and images. Contemporary researchers have addressed this data scarcity challenge by exploring rule-based and generative data augmentation models. Among these, Variational Autoencoder (VAE) models can generate realistic data without extensive baseline datasets for noise modeling. This study proposes a VAE-based synthetic image generation technique for rail defects, incorporating weight decay regularization and image reconstruction loss to prevent overfitting. The proposed method is applied to create a synthetic dataset for the Canadian Pacific Railway (CPR) with just 50 real samples across five classes. Remarkably, 500 synthetic samples are generated with a minimal reconstruction loss of 0.021. A Visual Transformer (ViT) model underwent fine-tuning using this synthetic CPR dataset, achieving high accuracy rates (98%-99%) in classifying the five defect classes. This research offers a promising solution to the data scarcity challenge in rail defect detection, showcasing the potential for robust ICS development in this domain.

Title: TSGAN: An Optical-to-SAR Dual Conditional GAN for Optical based SAR Temporal Shifting. (arXiv:2401.00440v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00440
• Code URL: null
• Copy Paste: [[2401.00440]] TSGAN: An Optical-to-SAR Dual Conditional GAN for Optical based SAR Temporal Shifting(http://arxiv.org/abs/2401.00440)
• Summary:

  In contrast to the well-investigated field of SAR-to-Optical translation, this study explores the lesser-investigated domain of Optical-to-SAR translation, a challenging field due to the ill-posed nature of this translation. The complexity arises as a single optical data can have multiple SAR representations based on the SAR viewing geometry. We propose a novel approach, termed SAR Temporal Shifting, which inputs an optical data from the desired timestamp along with a SAR data from a different temporal point but with a consistent viewing geometry as the expected SAR data, both complemented with a change map of optical data during the intervening period. This model modifies the SAR data based on the changes observed in optical data to generate the SAR data for the desired timestamp. Our model, a dual conditional Generative Adversarial Network (GAN), named Temporal Shifting GAN (TSGAN), incorporates a siamese encoder in both the Generator and the Discriminator. To prevent the model from overfitting on the input SAR data, we employed a change weighted loss function. Our approach surpasses traditional translation methods by eliminating the GAN's fiction phenomenon, particularly in unchanged regions, resulting in higher SSIM and PSNR in these areas. Additionally, modifications to the Pix2Pix architecture and the inclusion of attention mechanisms have enhanced the model's performance on all regions of the data. This research paves the way for leveraging legacy optical datasets, the most abundant and longstanding source of Earth datary data, extending their use to SAR domains and temporal analyses. To foster further research, we provide the code, datasets used in our study, and a framework for generating paired SAR-Optical datasets for new regions of interest. These resources are available on github.com/moienr/TemporalGAN

Title: An attempt to generate new bridge types from latent space of generative adversarial network. (arXiv:2401.00700v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00700
• Code URL: https://github.com/QQ583304953/Bridge-GAN
• Copy Paste: [[2401.00700]] An attempt to generate new bridge types from latent space of generative adversarial network(http://arxiv.org/abs/2401.00700)
• Summary:

  Try to generate new bridge types using generative artificial intelligence technology. Symmetric structured image dataset of three-span beam bridge, arch bridge, cable-stayed bridge and suspension bridge are used . Based on Python programming language, TensorFlow and Keras deep learning platform framework , as well as Wasserstein loss function and Lipschitz constraints, generative adversarial network is constructed and trained. From the obtained low dimensional bridge-type latent space sampling, new bridge types with asymmetric structures can be generated. Generative adversarial network can create new bridge types by organically combining different structural components on the basis of human original bridge types. It has a certain degree of human original ability. Generative artificial intelligence technology can open up imagination space and inspire humanity.

Title: Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Models. (arXiv:2401.00284v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00284
• Code URL: null
• Copy Paste: [[2401.00284]] Evaluation is all you need(http://arxiv.org/abs/2401.00284)
• Summary:

  This paper explores the use of open generative Large Language Models (LLMs) for annotation tasks in the social sciences. The study highlights the challenges associated with proprietary models, such as limited reproducibility and privacy concerns, and advocates for the adoption of open (source) models that can be operated on independent devices. Two examples of annotation tasks, sentiment analysis in tweets and identification of leisure activities in childhood aspirational essays are provided. The study evaluates the performance of different prompting strategies and models (neural-chat-7b-v3-2, Starling-LM-7B-alpha, openchat_3.5, zephyr-7b-alpha and zephyr-7b-beta). The results indicate the need for careful validation and tailored prompt engineering. The study highlights the advantages of open models for data privacy and reproducibility.

Title: Deep Generative Symbolic Regression. (arXiv:2401.00282v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00282
• Code URL: null
• Copy Paste: [[2401.00282]] Deep Generative Symbolic Regression(http://arxiv.org/abs/2401.00282)
• Summary:

  Symbolic regression (SR) aims to discover concise closed-form mathematical equations from data, a task fundamental to scientific discovery. However, the problem is highly challenging because closed-form equations lie in a complex combinatorial search space. Existing methods, ranging from heuristic search to reinforcement learning, fail to scale with the number of input variables. We make the observation that closed-form equations often have structural characteristics and invariances (e.g., the commutative law) that could be further exploited to build more effective symbolic regression solutions. Motivated by this observation, our key contribution is to leverage pre-trained deep generative models to capture the intrinsic regularities of equations, thereby providing a solid foundation for subsequent optimization steps. We show that our novel formalism unifies several prominent approaches of symbolic regression and offers a new perspective to justify and improve on the previous ad hoc designs, such as the usage of cross-entropy loss during pre-training. Specifically, we propose an instantiation of our framework, Deep Generative Symbolic Regression (DGSR). In our experiments, we show that DGSR achieves a higher recovery rate of true equations in the setting of a larger number of input variables, and it is more computationally efficient at inference time than state-of-the-art RL symbolic regression solutions.

Title: Viz: A QLoRA-based Copyright Marketplace for Legally Compliant Generative AI. (arXiv:2401.00503v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00503
• Code URL: null
• Copy Paste: [[2401.00503]] Viz: A QLoRA-based Copyright Marketplace for Legally Compliant Generative AI(http://arxiv.org/abs/2401.00503)
• Summary:

  This paper aims to introduce and analyze the Viz system in a comprehensive way, a novel system architecture that integrates Quantized Low-Rank Adapters (QLoRA) to fine-tune large language models (LLM) within a legally compliant and resource efficient marketplace. Viz represents a significant contribution to the field of artificial intelligence, particularly in addressing the challenges of computational efficiency, legal compliance, and economic sustainability in the utilization and monetization of LLMs. The paper delineates the scholarly discourse and developments that have informed the creation of Viz, focusing primarily on the advancements in LLM models, copyright issues in AI training (NYT case, 2023), and the evolution of model fine-tuning techniques, particularly low-rank adapters and quantized low-rank adapters, to create a sustainable and economically compliant framework for LLM utilization. The economic model it proposes benefits content creators, AI developers, and end-users, delineating a harmonious integration of technology, economy, and law, offering a comprehensive solution to the complex challenges of today's AI landscape.

large language model

Title: COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training. (arXiv:2401.00849v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00849
• Code URL: null
• Copy Paste: [[2401.00849]] COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training(http://arxiv.org/abs/2401.00849)
• Summary:

  In the evolution of Vision-Language Pre-training, shifting from short-text comprehension to encompassing extended textual contexts is pivotal. Recent autoregressive vision-language models like \cite{flamingo, palme}, leveraging the long-context capability of Large Language Models, have excelled in few-shot text generation tasks but face challenges in alignment tasks. Addressing this gap, we introduce the contrastive loss into text generation models, presenting the COntrastive-Streamlined MultimOdal framework (\ModelName), strategically partitioning the language model into dedicated unimodal text processing and adept multimodal data handling components. \ModelName, our unified framework, merges unimodal and multimodal elements, enhancing model performance for tasks involving textual and visual data while notably reducing learnable parameters. However, these models demand extensive long-text datasets, yet the availability of high-quality long-text video datasets remains limited. To bridge this gap, this work introduces \VideoDatasetName, an inaugural interleaved video-text dataset featuring comprehensive captions, marking a significant step forward. Demonstrating its impact, we illustrate how \VideoDatasetName{} enhances model performance in image-text tasks. With 34% learnable parameters and utilizing 72\% of the available data, our model demonstrates significant superiority over OpenFlamingo~\cite{openflamingo}. For instance, in the 4-shot flickr captioning task, performance notably improves from 57.2% to 65.\%. The contributions of \ModelName{} and \VideoDatasetName{} are underscored by notable performance gains across 14 diverse downstream datasets encompassing both image-text and video-text tasks.

Title: The Problem of Alignment. (arXiv:2401.00210v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00210
• Code URL: null
• Copy Paste: [[2401.00210]] The Problem of Alignment(http://arxiv.org/abs/2401.00210)
• Summary:

  Large Language Models produce sequences learned as statistical patterns from large corpora. In order not to reproduce corpus biases, after initial training models must be aligned with human values, preferencing certain continuations over others. Alignment, which can be viewed as the superimposition of normative structure onto a statistical model, reveals a conflicted and complex interrelationship between language and technology. This relationship shapes theories of language, linguistic practice and subjectivity, which are especially relevant to the current sophistication in artificially produced text. We examine this practice of structuration as a two-way interaction between users and models by analysing how ChatGPT4 redacts perceived `anomalous' language in fragments of Joyce's Ulysses and the new linguistic practice of prompt engineering. We then situate this alignment problem historically, revisiting earlier postwar linguistic debates which counterposed two views of meaning: as discrete structures, and as continuous probability distributions. We discuss the largely occluded work of the Moscow Linguistic School, which sought to reconcile this opposition. Our attention to the Moscow School and later related arguments by Searle and Kristeva casts the problem of alignment in a new light: as one involving attention to the social structuration of linguistic practice, including structuration of anomalies that, like the Joycean text, exist in defiance of expressive conventions. These debates around the communicative orientation toward language can help explain some of the contemporary behaviours and interdependencies that take place between users and LLMs.

Title: Boosting Large Language Model for Speech Synthesis: An Empirical Study. (arXiv:2401.00246v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00246
• Code URL: null
• Copy Paste: [[2401.00246]] Boosting Large Language Model for Speech Synthesis: An Empirical Study(http://arxiv.org/abs/2401.00246)
• Summary:

  Large language models (LLMs) have made significant advancements in natural language processing and are concurrently extending the language ability to other modalities, such as speech and vision. Nevertheless, most of the previous work focuses on prompting LLMs with perception abilities like auditory comprehension, and the effective approach for augmenting LLMs with speech synthesis capabilities remains ambiguous. In this paper, we conduct a comprehensive empirical exploration of boosting LLMs with the ability to generate speech, by combining pre-trained LLM LLaMA/OPT and text-to-speech synthesis model VALL-E. We compare three integration methods between LLMs and speech synthesis models, including directly fine-tuned LLMs, superposed layers of LLMs and VALL-E, and coupled LLMs and VALL-E using LLMs as a powerful text encoder. Experimental results show that, using LoRA method to fine-tune LLMs directly to boost the speech synthesis capability does not work well, and superposed LLMs and VALL-E can improve the quality of generated speech both in speaker similarity and word error rate (WER). Among these three methods, coupled methods leveraging LLMs as the text encoder can achieve the best performance, making it outperform original speech synthesis models with a consistently better speaker similarity and a significant (10.9%) WER reduction.

Title: Red Teaming for Large Language Models At Scale: Tackling Hallucinations on Mathematics Tasks. (arXiv:2401.00290v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00290
• Code URL: null
• Copy Paste: [[2401.00290]] Red Teaming for Large Language Models At Scale: Tackling Hallucinations on Mathematics Tasks(http://arxiv.org/abs/2401.00290)
• Summary:

  We consider the problem of red teaming LLMs on elementary calculations and algebraic tasks to evaluate how various prompting techniques affect the quality of outputs. We present a framework to procedurally generate numerical questions and puzzles, and compare the results with and without the application of several red teaming techniques. Our findings suggest that even though structured reasoning and providing worked-out examples slow down the deterioration of the quality of answers, the gpt-3.5-turbo and gpt-4 models are not well suited for elementary calculations and reasoning tasks, also when being red teamed.

Title: Improving Text Embeddings with Large Language Models. (arXiv:2401.00368v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00368
• Code URL: null
• Copy Paste: [[2401.00368]] Improving Text Embeddings with Large Language Models(http://arxiv.org/abs/2401.00368)
• Summary:

  In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. Unlike existing methods that often depend on multi-stage intermediate pre-training with billions of weakly-supervised text pairs, followed by fine-tuning with a few labeled datasets, our method does not require building complex training pipelines or relying on manually collected datasets that are often constrained by task diversity and language coverage. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across nearly 100 languages. We then fine-tune open-source decoder-only LLMs on the synthetic data using standard contrastive loss. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data. Furthermore, when fine-tuned with a mixture of synthetic and labeled data, our model sets new state-of-the-art results on the BEIR and MTEB benchmarks.

Title: RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. (arXiv:2401.00396v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00396
• Code URL: null
• Copy Paste: [[2401.00396]] RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models(http://arxiv.org/abs/2401.00396)
• Summary:

  Retrieval-augmented generation (RAG) has become a main technique for alleviating hallucinations in large language models (LLMs). Despite the integration of RAG, LLMs may still present unsupported or contradictory claims to the retrieved contents. In order to develop effective hallucination prevention strategies under RAG, it is important to create benchmark datasets that can measure the extent of hallucination. This paper presents RAGTruth, a corpus tailored for analyzing word-level hallucinations in various domains and tasks within the standard RAG frameworks for LLM applications. RAGTruth comprises nearly 18,000 naturally generated responses from diverse LLMs using RAG. These responses have undergone meticulous manual annotations at both the individual cases and word levels, incorporating evaluations of hallucination intensity. We not only benchmark hallucination frequencies across different LLMs, but also critically assess the effectiveness of several existing hallucination detection methodologies. Furthermore, we show that using a high-quality dataset such as RAGTruth, it is possible to finetune a relatively small LLM and achieve a competitive level of performance in hallucination detection when compared to the existing prompt-based approaches using state-of-the-art large language models such as GPT-4.

Title: keqing: knowledge-based question answering is a nature chain-of-thought mentor of LLM. (arXiv:2401.00426v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00426
• Code URL: null
• Copy Paste: [[2401.00426]] keqing: knowledge-based question answering is a nature chain-of-thought mentor of LLM(http://arxiv.org/abs/2401.00426)
• Summary:

  Large language models (LLMs) have exhibited remarkable performance on various natural language processing (NLP) tasks, especially for question answering. However, in the face of problems beyond the scope of knowledge, these LLMs tend to talk nonsense with a straight face, where the potential solution could be incorporating an Information Retrieval (IR) module and generating response based on these retrieved knowledge. In this paper, we present a novel framework to assist LLMs, such as ChatGPT, to retrieve question-related structured information on the knowledge graph, and demonstrate that Knowledge-based question answering (Keqing) could be a nature Chain-of-Thought (CoT) mentor to guide the LLM to sequentially find the answer entities of a complex question through interpretable logical chains. Specifically, the workflow of Keqing will execute decomposing a complex question according to predefined templates, retrieving candidate entities on knowledge graph, reasoning answers of sub-questions, and finally generating response with reasoning paths, which greatly improves the reliability of LLM's response. The experimental results on KBQA datasets show that Keqing can achieve competitive performance and illustrate the logic of answering each question.

Title: GeoGalactica: A Scientific Large Language Model in Geoscience. (arXiv:2401.00434v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00434
• Code URL: null
• Copy Paste: [[2401.00434]] GeoGalactica: A Scientific Large Language Model in Geoscience(http://arxiv.org/abs/2401.00434)
• Summary:

  Large language models (LLMs) have achieved huge success for their general knowledge and ability to solve a wide spectrum of tasks in natural language processing (NLP). Due to their impressive abilities, LLMs have shed light on potential inter-discipline applications to foster scientific discoveries of a specific domain by using artificial intelligence (AI for science, AI4S). In the meantime, utilizing NLP techniques in geoscience research and practice is wide and convoluted, contributing from knowledge extraction and document classification to question answering and knowledge discovery. In this work, we take the initial step to leverage LLM for science, through a rather straightforward approach. We try to specialize an LLM into geoscience, by further pre-training the model with a vast amount of texts in geoscience, as well as supervised fine-tuning (SFT) the resulting model with our custom collected instruction tuning dataset. These efforts result in a model GeoGalactica consisting of 30 billion parameters. To our best knowledge, it is the largest language model for the geoscience domain. More specifically, GeoGalactica is from further pre-training of Galactica. We train GeoGalactica over a geoscience-related text corpus containing 65 billion tokens curated from extensive data sources in the big science project Deep-time Digital Earth (DDE), preserving as the largest geoscience-specific text corpus. Then we fine-tune the model with 1 million pairs of instruction-tuning data consisting of questions that demand professional geoscience knowledge to answer. In this technical report, we will illustrate in detail all aspects of GeoGalactica, including data collection, data cleaning, base model selection, pre-training, SFT, and evaluation. We open-source our data curation tools and the checkpoints of GeoGalactica during the first 3/4 of pre-training.

Title: Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws. (arXiv:2401.00448v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00448
• Code URL: null
• Copy Paste: [[2401.00448]] Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws(http://arxiv.org/abs/2401.00448)
• Summary:

  Large language model (LLM) scaling laws are empirical formulas that estimate changes in model quality as a result of increasing parameter count and training data. However, these formulas, including the popular DeepMind Chinchilla scaling laws, neglect to include the cost of inference. We modify the Chinchilla scaling laws to calculate the optimal LLM parameter count and pre-training data size to train and deploy a model of a given quality and inference demand. We conduct our analysis both in terms of a compute budget and real-world costs and find that LLM researchers expecting reasonably large inference demand (~1B requests) should train models smaller and longer than Chinchilla-optimal.

Title: HSC-GPT: A Large Language Model for Human Settlements Construction. (arXiv:2401.00504v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00504
• Code URL: null
• Copy Paste: [[2401.00504]] HSC-GPT: A Large Language Model for Human Settlements Construction(http://arxiv.org/abs/2401.00504)
• Summary:

  The field of human settlement construction encompasses a range of spatial designs and management tasks, including urban planning and landscape architecture design. These tasks involve a plethora of instructions and descriptions presented in natural language, which are essential for understanding design requirements and producing effective design solutions. Recent research has sought to integrate natural language processing (NLP) and generative artificial intelligence (AI) into human settlement construction tasks. Due to the efficient processing and analysis capabilities of AI with data, significant successes have been achieved in design within this domain. However, this task still faces several fundamental challenges. The semantic information involved includes complex spatial details, diverse data source formats, high sensitivity to regional culture, and demanding requirements for innovation and rigor in work scenarios. These factors lead to limitations when applying general generative AI in this field, further exacerbated by a lack of high-quality data for model training. To address these challenges, this paper first proposes HSC-GPT, a large-scale language model framework specifically designed for tasks in human settlement construction, considering the unique characteristics of this domain.

Title: An Analysis of Embedding Layers and Similarity Scores using Siamese Neural Networks. (arXiv:2401.00582v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00582
• Code URL: null
• Copy Paste: [[2401.00582]] An Analysis of Embedding Layers and Similarity Scores using Siamese Neural Networks(http://arxiv.org/abs/2401.00582)
• Summary:

  Large Lanugage Models (LLMs) are gaining increasing popularity in a variety of use cases, from language understanding and writing to assistance in application development. One of the most important aspects for optimal funcionality of LLMs is embedding layers. Word embeddings are distributed representations of words in a continuous vector space. In the context of LLMs, words or tokens from the input text are transformed into high-dimensional vectors using unique algorithms specific to the model. Our research examines the embedding algorithms from leading companies in the industry, such as OpenAI, Google's PaLM, and BERT. Using medical data, we have analyzed similarity scores of each embedding layer, observing differences in performance among each algorithm. To enhance each model and provide an additional encoding layer, we also implemented Siamese Neural Networks. After observing changes in performance with the addition of the model, we measured the carbon footage per epoch of training. The carbon footprint associated with large language models (LLMs) is a significant concern, and should be taken into consideration when selecting algorithms for a variety of use cases. Overall, our research compared the accuracy different, leading embedding algorithms and their carbon footage, allowing for a holistic review of each embedding algorithm.

Title: Predicting Anti-microbial Resistance using Large Language Models. (arXiv:2401.00642v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00642
• Code URL: null
• Copy Paste: [[2401.00642]] Predicting Anti-microbial Resistance using Large Language Models(http://arxiv.org/abs/2401.00642)
• Summary:

  During times of increasing antibiotic resistance and the spread of infectious diseases like COVID-19, it is important to classify genes related to antibiotic resistance. As natural language processing has advanced with transformer-based language models, many language models that learn characteristics of nucleotide sequences have also emerged. These models show good performance in classifying various features of nucleotide sequences. When classifying nucleotide sequences, not only the sequence itself, but also various background knowledge is utilized. In this study, we use not only a nucleotide sequence-based language model but also a text language model based on PubMed articles to reflect more biological background knowledge in the model. We propose a method to fine-tune the nucleotide sequence language model and the text language model based on various databases of antibiotic resistance genes. We also propose an LLM-based augmentation technique to supplement the data and an ensemble method to effectively combine the two models. We also propose a benchmark for evaluating the model. Our method achieved better performance than the nucleotide sequence language model in the drug resistance class prediction.

Title: Digger: Detecting Copyright Content Mis-usage in Large Language Model Training. (arXiv:2401.00676v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00676
• Code URL: null
• Copy Paste: [[2401.00676]] Digger: Detecting Copyright Content Mis-usage in Large Language Model Training(http://arxiv.org/abs/2401.00676)
• Summary:

  Pre-training, which utilizes extensive and varied datasets, is a critical factor in the success of Large Language Models (LLMs) across numerous applications. However, the detailed makeup of these datasets is often not disclosed, leading to concerns about data security and potential misuse. This is particularly relevant when copyrighted material, still under legal protection, is used inappropriately, either intentionally or unintentionally, infringing on the rights of the authors.

  In this paper, we introduce a detailed framework designed to detect and assess the presence of content from potentially copyrighted books within the training datasets of LLMs. This framework also provides a confidence estimation for the likelihood of each content sample's inclusion. To validate our approach, we conduct a series of simulated experiments, the results of which affirm the framework's effectiveness in identifying and addressing instances of content misuse in LLM training processes. Furthermore, we investigate the presence of recognizable quotes from famous literary works within these datasets. The outcomes of our study have significant implications for ensuring the ethical use of copyrighted materials in the development of LLMs, highlighting the need for more transparent and responsible data management practices in this field.

Title: Large language model for Bible sentiment analysis: Sermon on the Mount. (arXiv:2401.00689v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00689
• Code URL: null
• Copy Paste: [[2401.00689]] Large language model for Bible sentiment analysis: Sermon on the Mount(http://arxiv.org/abs/2401.00689)
• Summary:

  The revolution of natural language processing via large language models has motivated its use in multidisciplinary areas that include social sciences and humanities and more specifically, comparative religion. Sentiment analysis provides a mechanism to study the emotions expressed in text. Recently, sentiment analysis has been used to study and compare translations of the Bhagavad Gita, which is a fundamental and sacred Hindu text. In this study, we use sentiment analysis for studying selected chapters of the Bible. These chapters are known as the Sermon on the Mount. We utilize a pre-trained language model for sentiment analysis by reviewing five translations of the Sermon on the Mount, which include the King James version, the New International Version, the New Revised Standard Version, the Lamsa Version, and the Basic English Version. We provide a chapter-by-chapter and verse-by-verse comparison using sentiment and semantic analysis and review the major sentiments expressed. Our results highlight the varying sentiments across the chapters and verses. We found that the vocabulary of the respective translations is significantly different. We detected different levels of humour, optimism, and empathy in the respective chapters that were used by Jesus to deliver his message.

Title: Benchmarking Large Language Models on Controllable Generation under Diversified Instructions. (arXiv:2401.00690v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00690
• Code URL: null
• Copy Paste: [[2401.00690]] Benchmarking Large Language Models on Controllable Generation under Diversified Instructions(http://arxiv.org/abs/2401.00690)
• Summary:

  While large language models (LLMs) have exhibited impressive instruction-following capabilities, it is still unclear whether and to what extent they can respond to explicit constraints that might be entailed in various instructions. As a significant aspect of LLM alignment, it is thus important to formulate such a specialized set of instructions as well as investigate the resulting behavior of LLMs. To address this vacancy, we propose a new benchmark CoDI-Eval to systematically and comprehensively evaluate LLMs' responses to instructions with various constraints. We construct a large collection of constraints-attributed instructions as a test suite focused on both generalization and coverage. Specifically, we advocate an instruction diversification process to synthesize diverse forms of constraint expression and also deliberate the candidate task taxonomy with even finer-grained sub-categories. Finally, we automate the entire evaluation process to facilitate further developments. Different from existing studies on controllable text generation, CoDI-Eval extends the scope to the prevalent instruction-following paradigm for the first time. We provide extensive evaluations of representative LLMs (e.g., ChatGPT, Vicuna) on CoDI-Eval, revealing their limitations in following instructions with specific constraints and there is still a significant gap between open-source and commercial closed-source LLMs. We believe this benchmark will facilitate research into improving the controllability of LLMs' responses to instructions. Our data and code are available at https://github.com/Xt-cyh/CoDI-Eval.

Title: Large Language Models aren't all that you need. (arXiv:2401.00698v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00698
• Code URL: null
• Copy Paste: [[2401.00698]] Large Language Models aren't all that you need(http://arxiv.org/abs/2401.00698)
• Summary:

  This paper describes the architecture and systems built towards solving the SemEval 2023 Task 2: MultiCoNER II (Multilingual Complex Named Entity Recognition) [1]. We evaluate two approaches (a) a traditional Conditional Random Fields model and (b) a Large Language Model (LLM) fine-tuned with a customized head and compare the two approaches. The novel ideas explored are: 1) Decaying auxiliary loss (with residual) - where we train the model on an auxiliary task of Coarse-Grained NER and include this task as a part of the loss function 2) Triplet token blending - where we explore ways of blending the embeddings of neighboring tokens in the final NER layer prior to prediction 3) Task-optimal heads - where we explore a variety of custom heads and learning rates for the final layer of the LLM. We also explore multiple LLMs including GPT-3 and experiment with a variety of dropout and other hyperparameter settings before arriving at our final model which achieves micro & macro f1 of 0.85/0.84 (on dev) and 0.67/0.61 on the test data . We show that while pre-trained LLMs, by themselves, bring about a large improvement in scores as compared to traditional models, we also demonstrate that tangible improvements to the Macro-F1 score can be made by augmenting the LLM with additional feature/loss/model engineering techniques described above.

Title: ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios. (arXiv:2401.00741v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00741
• Code URL: null
• Copy Paste: [[2401.00741]] ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios(http://arxiv.org/abs/2401.00741)
• Summary:

  Existing evaluations of tool learning primarily focus on validating the alignment of selected tools for large language models (LLMs) with expected outcomes. However, these approaches rely on a limited set of scenarios where answers can be pre-determined, diverging from genuine needs. Furthermore, a sole emphasis on outcomes disregards the intricate capabilities essential for LLMs to effectively utilize tools. To tackle this issue, we propose ToolEyes, a fine-grained system tailored for the evaluation of the LLMs' tool learning capabilities in authentic scenarios. The system meticulously examines seven real-world scenarios, analyzing five dimensions crucial to LLMs in tool learning: format alignment, intent comprehension, behavior planning, tool selection, and answer organization. Additionally, ToolEyes incorporates a tool library boasting approximately 600 tools, serving as an intermediary between LLMs and the physical world. Evaluations involving ten LLMs across three categories reveal a preference for specific scenarios and limited cognitive abilities in tool learning. Intriguingly, expanding the model size even exacerbates the hindrance to tool learning. These findings offer instructive insights aimed at advancing the field of tool learning. The data is available att https://github.com/Junjie-Ye/ToolEyes.git.

Title: Astraios: Parameter-Efficient Instruction Tuning Code Large Language Models. (arXiv:2401.00788v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00788
• Code URL: null
• Copy Paste: [[2401.00788]] Astraios: Parameter-Efficient Instruction Tuning Code Large Language Models(http://arxiv.org/abs/2401.00788)
• Summary:

  The high cost of full-parameter fine-tuning (FFT) of Large Language Models (LLMs) has led to a series of parameter-efficient fine-tuning (PEFT) methods. However, it remains unclear which methods provide the best cost-performance trade-off at different model scales. We introduce Astraios, a suite of 28 instruction-tuned OctoCoder models using 7 tuning methods and 4 model sizes up to 16 billion parameters. Through investigations across 5 tasks and 8 different datasets encompassing both code comprehension and code generation tasks, we find that FFT generally leads to the best downstream performance across all scales, and PEFT methods differ significantly in their efficacy based on the model scale. LoRA usually offers the most favorable trade-off between cost and performance. Further investigation into the effects of these methods on both model robustness and code security reveals that larger models tend to demonstrate reduced robustness and less security. At last, we explore the relationships among updated parameters, cross-entropy loss, and task performance. We find that the tuning effectiveness observed in small models generalizes well to larger models, and the validation loss in instruction tuning can be a reliable indicator of overall downstream performance.

Title: If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents. (arXiv:2401.00812v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00812
• Code URL: null
• Copy Paste: [[2401.00812]] If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents(http://arxiv.org/abs/2401.00812)
• Summary:

  The prominent large language models (LLMs) of today differ from past language models not only in size, but also in the fact that they are trained on a combination of natural language and formal language (code). As a medium between humans and computers, code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity. In this survey, we present an overview of the various benefits of integrating code into LLMs' training data. Specifically, beyond enhancing LLMs in code generation, we observe that these unique properties of code help (i) unlock the reasoning ability of LLMs, enabling their applications to a range of more complex natural language tasks; (ii) steer LLMs to produce structured and precise intermediate steps, which can then be connected to external execution ends through function calls; and (iii) take advantage of code compilation and execution environment, which also provides diverse feedback for model improvement. In addition, we trace how these profound capabilities of LLMs, brought by code, have led to their emergence as intelligent agents (IAs) in situations where the ability to understand instructions, decompose goals, plan and execute actions, and refine from feedback are crucial to their success on downstream tasks. Finally, we present several key challenges and future directions of empowering LLMs with code.

Title: A Computational Framework for Behavioral Assessment of LLM Therapists. (arXiv:2401.00820v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.00820
• Code URL: null
• Copy Paste: [[2401.00820]] A Computational Framework for Behavioral Assessment of LLM Therapists(http://arxiv.org/abs/2401.00820)
• Summary:

  The emergence of ChatGPT and other large language models (LLMs) has greatly increased interest in utilizing LLMs as therapists to support individuals struggling with mental health challenges. However, due to the lack of systematic studies, our understanding of how LLM therapists behave, i.e., ways in which they respond to clients, is significantly limited. Understanding their behavior across a wide range of clients and situations is crucial to accurately assess their capabilities and limitations in the high-risk setting of mental health, where undesirable behaviors can lead to severe consequences. In this paper, we propose BOLT, a novel computational framework to study the conversational behavior of LLMs when employed as therapists. We develop an in-context learning method to quantitatively measure the behavior of LLMs based on 13 different psychotherapy techniques including reflections, questions, solutions, normalizing, and psychoeducation. Subsequently, we compare the behavior of LLM therapists against that of high- and low-quality human therapy, and study how their behavior can be modulated to better reflect behaviors observed in high-quality therapy. Our analysis of GPT and Llama-variants reveals that these LLMs often resemble behaviors more commonly exhibited in low-quality therapy rather than high-quality therapy, such as offering a higher degree of problem-solving advice when clients share emotions, which is against typical recommendations. At the same time, unlike low-quality therapy, LLMs reflect significantly more upon clients' needs and strengths. Our analysis framework suggests that despite the ability of LLMs to generate anecdotal examples that appear similar to human therapists, LLM therapists are currently not fully consistent with high-quality care, and thus require additional research to ensure quality care.

Title: KernelGPT: Enhanced Kernel Fuzzing via Large Language Models. (arXiv:2401.00563v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2401.00563
• Code URL: null
• Copy Paste: [[2401.00563]] KernelGPT: Enhanced Kernel Fuzzing via Large Language Models(http://arxiv.org/abs/2401.00563)
• Summary:

  Bugs in operating system kernels can affect billions of devices and users all over the world. As a result, a large body of research has been focused on kernel fuzzing, i.e., automatically generating syscall (system call) sequences to detect potential kernel bugs or vulnerabilities. Syzkaller, one of the most widely studied kernel fuzzers, aims to generate valid syscall sequences based on predefined specifications written in syzlang, a domain-specific language for defining syscalls, their arguments, and the relationships between them. While there has been existing work trying to automate Syzkaller specification generation, this still remains largely manual work and a large number of important syscalls are still uncovered. In this paper, we propose KernelGPT, the first approach to automatically inferring Syzkaller specifications via Large Language Models (LLMs) for enhanced kernel fuzzing. Our basic insight is that LLMs have seen massive kernel code, documentation, and use cases during pre-training, and thus can automatically distill the necessary information for making valid syscalls. More specifically, KernelGPT leverages an iterative approach to automatically infer all the necessary specification components, and further leverages the validation feedback to repair/refine the initial specifications. Our preliminary results demonstrate that KernelGPT can help Syzkaller achieve higher coverage and find multiple previously unknown bugs. Moreover, we also received a request from the Syzkaller team to upstream specifications inferred by KernelGPT.

Title: Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles. (arXiv:2401.00243v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00243
• Code URL: null
• Copy Paste: [[2401.00243]] Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles(http://arxiv.org/abs/2401.00243)
• Summary:

  Reinforcement learning from human feedback (RLHF) emerges as a promising paradigm for aligning large language models (LLMs). However, a notable challenge in RLHF is overoptimization, where beyond a certain threshold, the pursuit of higher rewards leads to a decline in human preferences. In this paper, we observe the weakness of KL regularization which is commonly employed in existing RLHF methods to address overoptimization. To mitigate this limitation, we scrutinize the RLHF objective in the offline dataset and propose uncertainty-penalized RLHF (UP-RLHF), which incorporates uncertainty regularization during RL-finetuning. To enhance the uncertainty quantification abilities for reward models, we first propose a diverse low-rank adaptation (LoRA) ensemble by maximizing the nuclear norm of LoRA matrix concatenations. Then we optimize policy models utilizing penalized rewards, determined by both rewards and uncertainties provided by the diverse reward LoRA ensembles. Our experimental results, based on two real human preference datasets, showcase the effectiveness of diverse reward LoRA ensembles in quantifying reward uncertainty. Additionally, uncertainty regularization in UP-RLHF proves to be pivotal in mitigating overoptimization, thereby contributing to the overall performance.

Title: Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models. (arXiv:2401.00625v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.00625
• Code URL: null
• Copy Paste: [[2401.00625]] Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models(http://arxiv.org/abs/2401.00625)
• Summary:

  The burgeoning field of Large Language Models (LLMs), exemplified by sophisticated models like OpenAI's ChatGPT, represents a significant advancement in artificial intelligence. These models, however, bring forth substantial challenges in the high consumption of computational, memory, energy, and financial resources, especially in environments with limited resource capabilities. This survey aims to systematically address these challenges by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs. We categorize methods based on their optimization focus: computational, memory, energy, financial, and network resources and their applicability across various stages of an LLM's lifecycle, including architecture design, pretraining, finetuning, and system design. Additionally, the survey introduces a nuanced categorization of resource efficiency techniques by their specific resource types, which uncovers the intricate relationships and mappings between various resources and corresponding optimization techniques. A standardized set of evaluation metrics and datasets is also presented to facilitate consistent and fair comparisons across different models and techniques. By offering a comprehensive overview of the current sota and identifying open research avenues, this survey serves as a foundational reference for researchers and practitioners, aiding them in developing more sustainable and efficient LLMs in a rapidly evolving landscape.

segmentation

Title: Promoting Segment Anything Model towards Highly Accurate Dichotomous Image Segmentation. (arXiv:2401.00248v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00248
• Code URL: null
• Copy Paste: [[2401.00248]] Promoting Segment Anything Model towards Highly Accurate Dichotomous Image Segmentation(http://arxiv.org/abs/2401.00248)
• Summary:

  Segmenting any object represents a crucial step towards achieving artificial general intelligence, and the "Segment Anything Model" (SAM) has significantly advanced the development of foundational models in computer vision. We have high expectations regarding whether SAM can enhance highly accurate dichotomous image segmentation. In fact, the evidence presented in this article demonstrates that by inputting SAM with simple prompt boxes and utilizing the results output by SAM as input for IS5Net, we can greatly improve the effectiveness of highly accurate dichotomous image segmentation.

Title: SFGANS Self-supervised Future Generator for human ActioN Segmentation. (arXiv:2401.00438v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00438
• Code URL: null
• Copy Paste: [[2401.00438]] SFGANS Self-supervised Future Generator for human ActioN Segmentation(http://arxiv.org/abs/2401.00438)
• Summary:

  The ability to locate and classify action segments in long untrimmed video is of particular interest to many applications such as autonomous cars, robotics and healthcare applications. Today, the most popular pipeline for action segmentation is composed of encoding the frames into feature vectors, which are then processed by a temporal model for segmentation. In this paper we present a self-supervised method that comes in the middle of the standard pipeline and generated refined representations of the original feature vectors. Experiments show that this method improves the performance of existing models on different sub-tasks of action segmentation, even without additional hyper parameter tuning.

Title: Double-well Net for Image Segmentation. (arXiv:2401.00456v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00456
• Code URL: null
• Copy Paste: [[2401.00456]] Double-well Net for Image Segmentation(http://arxiv.org/abs/2401.00456)
• Summary:

  In this study, our goal is to integrate classical mathematical models with deep neural networks by introducing two novel deep neural network models for image segmentation known as Double-well Nets. Drawing inspiration from the Potts model, our models leverage neural networks to represent a region force functional. We extend the well-know MBO (Merriman-Bence-Osher) scheme to solve the Potts model. The widely recognized Potts model is approximated using a double-well potential and then solved by an operator-splitting method, which turns out to be an extension of the well-known MBO scheme. Subsequently, we replace the region force functional in the Potts model with a UNet-type network, which is data-driven, and also introduce control variables to enhance effectiveness. The resulting algorithm is a neural network activated by a function that minimizes the double-well potential. What sets our proposed Double-well Nets apart from many existing deep learning methods for image segmentation is their strong mathematical foundation. They are derived from the network approximation theory and employ the MBO scheme to approximately solve the Potts model. By incorporating mathematical principles, Double-well Nets bridge the MBO scheme and neural networks, and offer an alternative perspective for designing networks with mathematical backgrounds. Through comprehensive experiments, we demonstrate the performance of Double-well Nets, showcasing their superior accuracy and robustness compared to state-of-the-art neural networks. Overall, our work represents a valuable contribution to the field of image segmentation by combining the strengths of classical variational models and deep neural networks. The Double-well Nets introduce an innovative approach that leverages mathematical foundations to enhance segmentation performance.

Title: RainSD: Rain Style Diversification Module for Image Synthesis Enhancement using Feature-Level Style Distribution. (arXiv:2401.00460v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00460
• Code URL: null
• Copy Paste: [[2401.00460]] RainSD: Rain Style Diversification Module for Image Synthesis Enhancement using Feature-Level Style Distribution(http://arxiv.org/abs/2401.00460)
• Summary:

  Autonomous driving technology nowadays targets to level 4 or beyond, but the researchers are faced with some limitations for developing reliable driving algorithms in diverse challenges. To promote the autonomous vehicles to spread widely, it is important to address safety issues on this technology. Among various safety concerns, the sensor blockage problem by severe weather conditions can be one of the most frequent threats for multi-task learning based perception algorithms during autonomous driving. To handle this problem, the importance of the generation of proper datasets is becoming more significant. In this paper, a synthetic road dataset with sensor blockage generated from real road dataset BDD100K is suggested in the format of BDD100K annotation. Rain streaks for each frame were made by an experimentally established equation and translated utilizing the image-to-image translation network based on style transfer. Using this dataset, the degradation of the diverse multi-task networks for autonomous driving, such as lane detection, driving area segmentation, and traffic object detection, has been thoroughly evaluated and analyzed. The tendency of the performance degradation of deep neural network-based perception systems for autonomous vehicle has been analyzed in depth. Finally, we discuss the limitation and the future directions of the deep neural network-based perception algorithms and autonomous driving dataset generation based on image-to-image translation.

Title: SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge. (arXiv:2401.00496v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00496
• Code URL: null
• Copy Paste: [[2401.00496]] SAR-RARP50: Segmentation of surgical instrumentation and Action Recognition on Robot-Assisted Radical Prostatectomy Challenge(http://arxiv.org/abs/2401.00496)
• Summary:

  Surgical tool segmentation and action recognition are fundamental building blocks in many computer-assisted intervention applications, ranging from surgical skills assessment to decision support systems. Nowadays, learning-based action recognition and segmentation approaches outperform classical methods, relying, however, on large, annotated datasets. Furthermore, action recognition and tool segmentation algorithms are often trained and make predictions in isolation from each other, without exploiting potential cross-task relationships. With the EndoVis 2022 SAR-RARP50 challenge, we release the first multimodal, publicly available, in-vivo, dataset for surgical action recognition and semantic instrumentation segmentation, containing 50 suturing video segments of Robotic Assisted Radical Prostatectomy (RARP). The aim of the challenge is twofold. First, to enable researchers to leverage the scale of the provided dataset and develop robust and highly accurate single-task action recognition and tool segmentation approaches in the surgical domain. Second, to further explore the potential of multitask-based learning approaches and determine their comparative advantage against their single-task counterparts. A total of 12 teams participated in the challenge, contributing 7 action recognition methods, 9 instrument segmentation techniques, and 4 multitask approaches that integrated both action recognition and instrument segmentation.

Title: 1st Place Solution for 5th LSVOS Challenge: Referring Video Object Segmentation. (arXiv:2401.00663v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.00663
• Code URL: null
• Copy Paste: [[2401.00663]] 1st Place Solution for 5th LSVOS Challenge: Referring Video Object Segmentation(http://arxiv.org/abs/2401.00663)
• Summary:

  The recent transformer-based models have dominated the Referring Video Object Segmentation (RVOS) task due to the superior performance. Most prior works adopt unified DETR framework to generate segmentation masks in query-to-instance manner. In this work, we integrate strengths of that leading RVOS models to build up an effective paradigm. We first obtain binary mask sequences from the RVOS models. To improve the consistency and quality of masks, we propose Two-Stage Multi-Model Fusion strategy. Each stage rationally ensembles RVOS models based on framework design as well as training strategy, and leverages different video object segmentation (VOS) models to enhance mask coherence by object propagation mechanism. Our method achieves 75.7% J&F on Ref-Youtube-VOS validation set and 70% J&F on test set, which ranks 1st place on 5th Large-scale Video Object Segmentation Challenge (ICCV 2023) track 3. Code is available at https://github.com/RobertLuo1/iccv2023_RVOS_Challenge.