secure

Title: Prompt Highlighter: Interactive Control for Multi-Modal LLMs. (arXiv:2312.04302v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04302
• Code URL: https://github.com/dvlab-research/prompt-highlighter
• Copy Paste: [[2312.04302]] Prompt Highlighter: Interactive Control for Multi-Modal LLMs(http://arxiv.org/abs/2312.04302)
• Summary:

  This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs) inference: explicit controllable text generation. Multi-modal LLMs empower multi-modality understanding with the capability of semantic generation yet bring less explainability and heavier reliance on prompt contents due to their autoregressive generative nature. While manipulating prompt formats could improve outputs, designing specific and precise prompts per task can be challenging and ineffective. To tackle this issue, we introduce a novel inference method, Prompt Highlighter, which enables users to highlight specific prompt spans to interactively control the focus during generation. Motivated by the classifier-free diffusion guidance, we form regular and unconditional context pairs based on highlighted tokens, demonstrating that the autoregressive generation in models can be guided in a classifier-free way. Notably, we find that, during inference, guiding the models with highlighted tokens through the attention weights leads to more desired outputs. Our approach is compatible with current LLMs and VLMs, achieving impressive customized generation results without training. Experiments confirm its effectiveness in focusing on input contexts and generating reliable content. Without tuning on LLaVA-v1.5, our method secured 69.5 in the MMBench test and 1552.5 in MME-perception. The code is available at: https://github.com/dvlab-research/Prompt-Highlighter/

Title: Secure Ranging with IEEE 802.15.4z HRP UWB. (arXiv:2312.03964v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.03964
• Code URL: null
• Copy Paste: [[2312.03964]] Secure Ranging with IEEE 802(http://arxiv.org/abs/2312.03964)
• Summary:

  Secure ranging refers to the capability of upper-bounding the actual physical distance between two devices with reliability. This is essential in a variety of applications, including to unlock physical systems. In this work, we will look at secure ranging in the context of ultra-wideband impulse radio (UWB-IR) as specified in IEEE 802.15.4z (a.k.a. 4z). In particular, an encrypted waveform, i.e. the scrambled timestamp sequence (STS), is defined in the high rate pulse repetition frequency (HRP) mode of operation in 4z for secure ranging. This work demonstrates the security analysis of 4z HRP when implemented with an adequate receiver design and shows the STS waveform can enable secure ranging. We first review the STS receivers adopted in previous studies and analyze their security vulnerabilities. Then we present a reference STS receiver and prove that secure ranging can be achieved by employing the STS waveform in 4z HRP. The performance bounds of the reference secure STS receiver are also characterized. Numerical experiments corroborate the analyses and demonstrate the security of the reference STS receiver.

Title: Contract Wallet Using Emails. (arXiv:2312.04173v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.04173
• Code URL: null
• Copy Paste: [[2312.04173]] Contract Wallet Using Emails(http://arxiv.org/abs/2312.04173)
• Summary:

  We proposed a new construction for contract wallets, smart contract applications that allow users to control their crypto assets. Users can manipulate their crypto assets by simply sending emails with no need to manage keys. These emails are verified using zero-knowledge proof (ZKP) along with their attached digital signatures that the sender domain server (SDS) generates according to DomainKeys Identified Mail. Unless the SDS forges the emails, the crypto assets remain secure in the proposed system. Moreover, the existing SDSs can be used as is by outsourcing additional work to a third party that is not necessarily trusted. The system supports various functions to manipulate crypto assets. We produced a tool for variable-regex mapping (VRM) that enables developers to build a new function without ZKP skills. For example, using the tool, we built a demo application where users can exchange crypto assets via Uniswap only with emails. The published version of this paper is available at https://doi.org/10.1109/ICBC56567.2023.10174932.

security

Title: Dynamic Data-Driven Digital Twins for Blockchain Systems. (arXiv:2312.04226v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.04226
• Code URL: null
• Copy Paste: [[2312.04226]] Dynamic Data-Driven Digital Twins for Blockchain Systems(http://arxiv.org/abs/2312.04226)
• Summary:

  In recent years, we have seen an increase in the adoption of blockchain-based systems in non-financial applications, looking to benefit from what the technology has to offer. Although many fields have managed to include blockchain in their core functionalities, the adoption of blockchain, in general, is constrained by the so-called trilemma trade-off between decentralization, scalability, and security. In our previous work, we have shown that using a digital twin for dynamically managing blockchain systems during runtime can be effective in managing the trilemma trade-off. Our Digital Twin leverages DDDAS feedback loop, which is responsible for getting the data from the system to the digital twin, conducting optimisation, and updating the physical system. This paper examines how leveraging DDDAS feedback loop can support the optimisation component of the trilemma benefiting from Reinforcement Learning agents and a simulation component to augment the quality of the learned model while reducing the computational overhead required for decision-making.

privacy

Title: Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection. (arXiv:2312.04095v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04095
• Code URL: https://github.com/hnanhtuan/projected_gradient_unlearning
• Copy Paste: [[2312.04095]] Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection(http://arxiv.org/abs/2312.04095)
• Summary:

  Recent data-privacy laws have sparked interest in machine unlearning, which involves removing the effect of specific training samples from a learnt model as if they were never present in the original training dataset. The challenge of machine unlearning is to discard information about the ``forget'' data in the learnt model without altering the knowledge about the remaining dataset and to do so more efficiently than the naive retraining approach. To achieve this, we adopt a projected-gradient based learning method, named as Projected-Gradient Unlearning (PGU), in which the model takes steps in the orthogonal direction to the gradient subspaces deemed unimportant for the retaining dataset, so as to its knowledge is preserved. By utilizing Stochastic Gradient Descent (SGD) to update the model weights, our method can efficiently scale to any model and dataset size. We provide empirically evidence to demonstrate that our unlearning method can produce models that behave similar to models retrained from scratch across various metrics even when the training dataset is no longer accessible. Our code is available at https://github.com/hnanhtuan/projected_gradient_unlearning.

Title: Identity-Obscured Neural Radiance Fields: Privacy-Preserving 3D Facial Reconstruction. (arXiv:2312.04106v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04106
• Code URL: null
• Copy Paste: [[2312.04106]] Identity-Obscured Neural Radiance Fields: Privacy-Preserving 3D Facial Reconstruction(http://arxiv.org/abs/2312.04106)
• Summary:

  Neural radiance fields (NeRF) typically require a complete set of images taken from multiple camera perspectives to accurately reconstruct geometric details. However, this approach raise significant privacy concerns in the context of facial reconstruction. The critical need for privacy protection often leads invidividuals to be reluctant in sharing their facial images, due to fears of potential misuse or security risks. Addressing these concerns, we propose a method that leverages privacy-preserving images for reconstructing 3D head geometry within the NeRF framework. Our method stands apart from traditional facial reconstruction techniques as it does not depend on RGB information from images containing sensitive facial data. Instead, it effectively generates plausible facial geometry using a series of identity-obscured inputs, thereby protecting facial privacy.

Title: DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer. (arXiv:2312.03724v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03724
• Code URL: null
• Copy Paste: [[2312.03724]] DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer(http://arxiv.org/abs/2312.03724)
• Summary:

  Large Language Models (LLMs) have emerged as dominant tools for various tasks, particularly when tailored for a specific target by prompt tuning. Nevertheless, concerns surrounding data privacy present obstacles due to the tuned prompts' dependency on sensitive private information. A practical solution is to host a local LLM and optimize a soft prompt privately using data. Yet, hosting a local model becomes problematic when model ownership is protected. Alternative methods, like sending data to the model's provider for training, intensify these privacy issues facing an untrusted provider. In this paper, we present a novel solution called Differentially-Private Offsite Prompt Tuning (DP-OPT) to address this challenge. Our approach involves tuning a discrete prompt on the client side and then applying it to the desired cloud models. We demonstrate that prompts suggested by LLMs themselves can be transferred without compromising performance significantly. To ensure that the prompts do not leak private information, we introduce the first private prompt generation mechanism, by a differentially-private (DP) ensemble of in-context learning with private demonstrations. With DP-OPT, generating privacy-preserving prompts by Vicuna-7b can yield competitive performance compared to non-private in-context learning on GPT3.5 or local private prompt tuning. Codes are available at https://github.com/VITA-Group/DP-OPT .

Title: Clinical Risk Prediction Using Language Models: Benefits And Considerations. (arXiv:2312.03742v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03742
• Code URL: null
• Copy Paste: [[2312.03742]] Clinical Risk Prediction Using Language Models: Benefits And Considerations(http://arxiv.org/abs/2312.03742)
• Summary:

  The utilization of Electronic Health Records (EHRs) for clinical risk prediction is on the rise. However, strict privacy regulations limit access to comprehensive health records, making it challenging to apply standard machine learning algorithms in practical real-world scenarios. Previous research has addressed this data limitation by incorporating medical ontologies and employing transfer learning methods. In this study, we investigate the potential of leveraging language models (LMs) as a means to incorporate supplementary domain knowledge for improving the performance of various EHR-based risk prediction tasks. Unlike applying LMs to unstructured EHR data such as clinical notes, this study focuses on using textual descriptions within structured EHR to make predictions exclusively based on that information. We extensively compare against previous approaches across various data types and sizes. We find that employing LMs to represent structured EHRs, such as diagnostic histories, leads to improved or at least comparable performance in diverse risk prediction tasks. Furthermore, LM-based approaches offer numerous advantages, including few-shot learning, the capability to handle previously unseen medical concepts, and adaptability to various medical vocabularies. Nevertheless, we underscore, through various experiments, the importance of being cautious when employing such models, as concerns regarding the reliability of LMs persist.

Title: Making Translators Privacy-aware on the User's Side. (arXiv:2312.04068v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.04068
• Code URL: null
• Copy Paste: [[2312.04068]] Making Translators Privacy-aware on the User's Side(http://arxiv.org/abs/2312.04068)
• Summary:

  We propose PRISM to enable users of machine translation systems to preserve the privacy of data on their own initiative. There is a growing demand to apply machine translation systems to data that require privacy protection. While several machine translation engines claim to prioritize privacy, the extent and specifics of such protection are largely ambiguous. First, there is often a lack of clarity on how and to what degree the data is protected. Even if service providers believe they have sufficient safeguards in place, sophisticated adversaries might still extract sensitive information. Second, vulnerabilities may exist outside of these protective measures, such as within communication channels, potentially leading to data leakage. As a result, users are hesitant to utilize machine translation engines for data demanding high levels of privacy protection, thereby missing out on their benefits. PRISM resolves this problem. Instead of relying on the translation service to keep data safe, PRISM provides the means to protect data on the user's side. This approach ensures that even machine translation engines with inadequate privacy measures can be used securely. For platforms already equipped with privacy safeguards, PRISM acts as an additional protection layer, reinforcing their security furthermore. PRISM adds these privacy features without significantly compromising translation accuracy. Our experiments demonstrate the effectiveness of PRISM using real-world translators, T5 and ChatGPT (GPT-3.5-turbo), and the datasets with two languages. PRISM effectively balances privacy protection with translation accuracy.

Title: PCDP-SGD: Improving the Convergence of Differentially Private SGD via Projection in Advance. (arXiv:2312.03792v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.03792
• Code URL: null
• Copy Paste: [[2312.03792]] PCDP-SGD: Improving the Convergence of Differentially Private SGD via Projection in Advance(http://arxiv.org/abs/2312.03792)
• Summary:

  The paradigm of Differentially Private SGD~(DP-SGD) can provide a theoretical guarantee for training data in both centralized and federated settings. However, the utility degradation caused by DP-SGD limits its wide application in high-stakes tasks, such as medical image diagnosis. In addition to the necessary perturbation, the convergence issue is attributed to the information loss on the gradient clipping. In this work, we propose a general framework PCDP-SGD, which aims to compress redundant gradient norms and preserve more crucial top gradient components via projection operation before gradient clipping. Additionally, we extend PCDP-SGD as a fundamental component in differential privacy federated learning~(DPFL) for mitigating the data heterogeneous challenge and achieving efficient communication. We prove that pre-projection enhances the convergence of DP-SGD by reducing the dependence of clipping error and bias to a fraction of the top gradient eigenspace, and in theory, limits cross-client variance to improve the convergence under heterogeneous federation. Experimental results demonstrate that PCDP-SGD achieves higher accuracy compared with state-of-the-art DP-SGD variants in computer vision tasks. Moreover, PCDP-SGD outperforms current federated learning frameworks when DP is guaranteed on local training sets.

Title: On the Impact of Multi-dimensional Local Differential Privacy on Fairness. (arXiv:2312.04404v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04404
• Code URL: null
• Copy Paste: [[2312.04404]] On the Impact of Multi-dimensional Local Differential Privacy on Fairness(http://arxiv.org/abs/2312.04404)
• Summary:

  Automated decision systems are increasingly used to make consequential decisions in people's lives. Due to the sensitivity of the manipulated data as well as the resulting decisions, several ethical concerns need to be addressed for the appropriate use of such technologies, in particular, fairness and privacy. Unlike previous work, which focused on centralized differential privacy (DP) or local DP (LDP) for a single sensitive attribute, in this paper, we examine the impact of LDP in the presence of several sensitive attributes (i.e., multi-dimensional data) on fairness. Detailed empirical analysis on synthetic and benchmark datasets revealed very relevant observations. In particular, (1) multi-dimensional LDP is an efficient approach to reduce disparity, (2) the multi-dimensional approach of LDP (independent vs. combined) matters only at low privacy guarantees, and (3) the outcome Y distribution has an important effect on which group is more sensitive to the obfuscation. Last, we summarize our findings in the form of recommendations to guide practitioners in adopting effective privacy-preserving practices while maintaining fairness and utility in ML applications.

protect

Title: UID as a Guiding Metric for Automated Authorship Obfuscation. (arXiv:2312.03709v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03709
• Code URL: null
• Copy Paste: [[2312.03709]] UID as a Guiding Metric for Automated Authorship Obfuscation(http://arxiv.org/abs/2312.03709)
• Summary:

  Protecting the anonymity of authors has become a difficult task given the rise of automated authorship attributors. These attributors are capable of attributing the author of a text amongst a pool of authors with great accuracy. In order to counter the rise of these automated attributors, there has also been a rise of automated obfuscators. These obfuscators are capable of taking some text, perturbing the text in some manner, and, if successful, deceive an automated attributor in misattributing the wrong author. We devised three novel authorship obfuscation methods that utilized a Psycho-linguistic theory known as Uniform Information Density (UID) theory. This theory states that humans evenly distribute information amongst speech or text so as to maximize efficiency. Utilizing this theory in our three obfuscation methods, we attempted to see how successfully we could deceive two separate attributors. Obfuscating 50 human and 50 GPT-3 generated articles from the TuringBench dataset, we observed how well each method did on deceiving the attributors. While the quality of the obfuscation in terms of semantic preservation and sensical changes was high, we were not able to find any evidence to indicate UID was a viable guiding metric for obfuscation. However, due to restrictions in time we were unable to test a large enough sample of article or tune the parameters for our attributors to comment conclusively on UID in obfuscation.

Title: De-identification of clinical free text using natural language processing: A systematic review of current approaches. (arXiv:2312.03736v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03736
• Code URL: null
• Copy Paste: [[2312.03736]] De-identification of clinical free text using natural language processing: A systematic review of current approaches(http://arxiv.org/abs/2312.03736)
• Summary:

  Background: Electronic health records (EHRs) are a valuable resource for data-driven medical research. However, the presence of protected health information (PHI) makes EHRs unsuitable to be shared for research purposes. De-identification, i.e. the process of removing PHI is a critical step in making EHR data accessible. Natural language processing has repeatedly demonstrated its feasibility in automating the de-identification process. Objectives: Our study aims to provide systematic evidence on how the de-identification of clinical free text has evolved in the last thirteen years, and to report on the performances and limitations of the current state-of-the-art systems. In addition, we aim to identify challenges and potential research opportunities in this field. Methods: A systematic search in PubMed, Web of Science and the DBLP was conducted for studies published between January 2010 and February 2023. Titles and abstracts were examined to identify the relevant studies. Selected studies were then analysed in-depth, and information was collected on de-identification methodologies, data sources, and measured performance. Results: A total of 2125 publications were identified for the title and abstract screening. 69 studies were found to be relevant. Machine learning (37 studies) and hybrid (26 studies) approaches are predominant, while six studies relied only on rules. Majority of the approaches were trained and evaluated on public corpora. The 2014 i2b2/UTHealth corpus is the most frequently used (36 studies), followed by the 2006 i2b2 (18 studies) and 2016 CEGS N-GRID (10 studies) corpora.

defense

Title: Defense against ML-based Power Side-channel Attacks on DNN Accelerators with Adversarial Attacks. (arXiv:2312.04035v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.04035
• Code URL: null
• Copy Paste: [[2312.04035]] Defense against ML-based Power Side-channel Attacks on DNN Accelerators with Adversarial Attacks(http://arxiv.org/abs/2312.04035)
• Summary:

  Artificial Intelligence (AI) hardware accelerators have been widely adopted to enhance the efficiency of deep learning applications. However, they also raise security concerns regarding their vulnerability to power side-channel attacks (SCA). In these attacks, the adversary exploits unintended communication channels to infer sensitive information processed by the accelerator, posing significant privacy and copyright risks to the models. Advanced machine learning algorithms are further employed to facilitate the side-channel analysis and exacerbate the privacy issue of AI accelerators. Traditional defense strategies naively inject execution noise to the runtime of AI models, which inevitably introduce large overheads.

  In this paper, we present AIAShield, a novel defense methodology to safeguard FPGA-based AI accelerators and mitigate model extraction threats via power-based SCAs. The key insight of AIAShield is to leverage the prominent adversarial attack technique from the machine learning community to craft delicate noise, which can significantly obfuscate the adversary's side-channel observation while incurring minimal overhead to the execution of the protected model. At the hardware level, we design a new module based on ring oscillators to achieve fine-grained noise generation. At the algorithm level, we repurpose Neural Architecture Search to worsen the adversary's extraction results. Extensive experiments on the Nvidia Deep Learning Accelerator (NVDLA) demonstrate that AIAShield outperforms existing solutions with excellent transferability.

attack

Title: On the Robustness of Large Multimodal Models Against Image Adversarial Attacks. (arXiv:2312.03777v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03777
• Code URL: null
• Copy Paste: [[2312.03777]] On the Robustness of Large Multimodal Models Against Image Adversarial Attacks(http://arxiv.org/abs/2312.03777)
• Summary:

  Recent advances in instruction tuning have led to the development of State-of-the-Art Large Multimodal Models (LMMs). Given the novelty of these models, the impact of visual adversarial attacks on LMMs has not been thoroughly examined. We conduct a comprehensive study of the robustness of various LMMs against different adversarial attacks, evaluated across tasks including image classification, image captioning, and Visual Question Answer (VQA). We find that in general LMMs are not robust to visual adversarial inputs. However, our findings suggest that context provided to the model via prompts, such as questions in a QA pair helps to mitigate the effects of visual adversarial inputs. Notably, the LMMs evaluated demonstrated remarkable resilience to such attacks on the ScienceQA task with only an 8.10% drop in performance compared to their visual counterparts which dropped 99.73%. We also propose a new approach to real-world image classification which we term query decomposition. By incorporating existence queries into our input prompt we observe diminished attack effectiveness and improvements in image classification accuracy. This research highlights a previously under-explored facet of LMM robustness and sets the stage for future work aimed at strengthening the resilience of multimodal systems in adversarial environments.

Title: Forensic Iris Image Synthesis. (arXiv:2312.04125v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04125
• Code URL: null
• Copy Paste: [[2312.04125]] Forensic Iris Image Synthesis(http://arxiv.org/abs/2312.04125)
• Summary:

  Post-mortem iris recognition is an emerging application of iris-based human identification in a forensic setup, able to correctly identify deceased subjects even three weeks post-mortem. This technique thus is considered as an important component of future forensic toolkits. The current advancements in this field are seriously slowed down by exceptionally difficult data collection, which can happen in mortuary conditions, at crime scenes, or in ``body farm'' facilities. This paper makes a novel contribution to facilitate progress in post-mortem iris recognition by offering a conditional StyleGAN-based iris synthesis model, trained on the largest-available dataset of post-mortem iris samples acquired from more than 350 subjects, generating -- through appropriate exploration of StyleGAN latent space -- multiple within-class (same identity) and between-class (different new identities) post-mortem iris images, compliant with ISO/IEC 29794-6, and with decomposition deformations controlled by the requested PMI (post mortem interval). Besides an obvious application to enhance the existing, very sparse, post-mortem iris datasets to advance -- among others -- iris presentation attack endeavors, we anticipate it may be useful to generate samples that would expose professional forensic human examiners to never-seen-before deformations for various PMIs, increasing their training effectiveness. The source codes and model weights are made available with the paper.

Title: OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization. (arXiv:2312.04403v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04403
• Code URL: null
• Copy Paste: [[2312.04403]] OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimization(http://arxiv.org/abs/2312.04403)
• Summary:

  Vision-language pre-training (VLP) models demonstrate impressive abilities in processing both images and text. However, they are vulnerable to multi-modal adversarial examples (AEs). Investigating the generation of high-transferability adversarial examples is crucial for uncovering VLP models' vulnerabilities in practical scenarios. Recent works have indicated that leveraging data augmentation and image-text modal interactions can enhance the transferability of adversarial examples for VLP models significantly. However, they do not consider the optimal alignment problem between dataaugmented image-text pairs. This oversight leads to adversarial examples that are overly tailored to the source model, thus limiting improvements in transferability. In our research, we first explore the interplay between image sets produced through data augmentation and their corresponding text sets. We find that augmented image samples can align optimally with certain texts while exhibiting less relevance to others. Motivated by this, we propose an Optimal Transport-based Adversarial Attack, dubbed OT-Attack. The proposed method formulates the features of image and text sets as two distinct distributions and employs optimal transport theory to determine the most efficient mapping between them. This optimal mapping informs our generation of adversarial examples to effectively counteract the overfitting issues. Extensive experiments across various network architectures and datasets in image-text matching tasks reveal that our OT-Attack outperforms existing state-of-the-art methods in terms of adversarial transferability.

Title: Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak. (arXiv:2312.04127v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.04127
• Code URL: null
• Copy Paste: [[2312.04127]] Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak(http://arxiv.org/abs/2312.04127)
• Summary:

  Extensive work has been devoted to improving the safety mechanism of Large Language Models (LLMs). However, in specific scenarios, LLMs still generate harmful responses when faced with malicious instructions, a phenomenon referred to as "Jailbreak Attack". In our research, we introduce a novel jailbreak attack method (\textbf{RADIAL}), which consists of two steps: 1) Inherent Response Tendency Analysis: we analyze the inherent affirmation and rejection tendency of LLMs to react to real-world instructions. 2) Real-World Instructions-Driven Jailbreak: based on our analysis, we strategically choose several real-world instructions and embed malicious instructions into them to amplify the LLM's potential to generate harmful responses. On three open-source human-aligned LLMs, our method achieves excellent jailbreak attack performance for both Chinese and English malicious instructions. Besides, we guided detailed ablation experiments and verified the effectiveness of our core idea "Inherent Response Tendency Analysis". Our exploration also exposes the vulnerability of LLMs to being induced into generating more detailed harmful responses in subsequent rounds of dialogue.

Title: Dr. Jekyll and Mr. Hyde: Two Faces of LLMs. (arXiv:2312.03853v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.03853
• Code URL: null
• Copy Paste: [[2312.03853]] Dr(http://arxiv.org/abs/2312.03853)
• Summary:

  This year, we witnessed a rise in the use of Large Language Models, especially when combined with applications like chatbot assistants. Safety mechanisms and specialized training procedures are put in place to prevent improper responses from these assistants. In this work, we bypass these measures for ChatGPT and Bard (and, to some extent, Bing chat) by making them impersonate complex personas with opposite characteristics as those of the truthful assistants they are supposed to be. We start by creating elaborate biographies of these personas, which we then use in a new session with the same chatbots. Our conversation followed a role-play style to get the response the assistant was not allowed to provide. By making use of personas, we show that the response that is prohibited is actually provided, making it possible to obtain unauthorized, illegal, or harmful information. This work shows that by using adversarial personas, one can overcome safety mechanisms set out by ChatGPT and Bard. It also introduces several ways of activating such adversarial personas, altogether showing that both chatbots are vulnerable to this kind of attack.

Title: Node-aware Bi-smoothing: Certified Robustness against Graph Injection Attacks. (arXiv:2312.03979v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.03979
• Code URL: null
• Copy Paste: [[2312.03979]] Node-aware Bi-smoothing: Certified Robustness against Graph Injection Attacks(http://arxiv.org/abs/2312.03979)
• Summary:

  Deep Graph Learning (DGL) has emerged as a crucial technique across various domains. However, recent studies have exposed vulnerabilities in DGL models, such as susceptibility to evasion and poisoning attacks. While empirical and provable robustness techniques have been developed to defend against graph modification attacks (GMAs), the problem of certified robustness against graph injection attacks (GIAs) remains largely unexplored. To bridge this gap, we introduce the node-aware bi-smoothing framework, which is the first certifiably robust approach for general node classification tasks against GIAs. Notably, the proposed node-aware bi-smoothing scheme is model-agnostic and is applicable for both evasion and poisoning attacks. Through rigorous theoretical analysis, we establish the certifiable conditions of our smoothing scheme. We also explore the practical implications of our node-aware bi-smoothing schemes in two contexts: as an empirical defense approach against real-world GIAs and in the context of recommendation systems. Furthermore, we extend two state-of-the-art certified robustness frameworks to address node injection attacks and compare our approach against them. Extensive evaluations demonstrate the effectiveness of our proposed certificates.

Title: MediHunt: A Network Forensics Framework for Medical IoT Devices. (arXiv:2312.04096v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.04096
• Code URL: null
• Copy Paste: [[2312.04096]] MediHunt: A Network Forensics Framework for Medical IoT Devices(http://arxiv.org/abs/2312.04096)
• Summary:

  The Medical Internet of Things (MIoT) has enabled small, ubiquitous medical devices to communicate with each other to facilitate interconnected healthcare delivery. These devices interact using communication protocols like MQTT, Bluetooth, and Wi-Fi. However, as MIoT devices proliferate, these networked devices are vulnerable to cyber-attacks. This paper focuses on the vulnerabilities present in the Message Queuing Telemetry and Transport (MQTT) protocol. The MQTT protocol is prone to cyber-attacks that can harm the system's functionality. The memory-constrained MIoT devices enforce a limitation on storing all data logs that are required for comprehensive network forensics. This paper solves the data log availability challenge by detecting the attack in real-time and storing the corresponding logs for further analysis with the proposed network forensics framework: MediHunt. Machine learning (ML) techniques are the most real safeguard against cyber-attacks. However, these models require a specific dataset that covers diverse attacks on the MQTT-based IoT system for training. The currently available datasets do not encompass a variety of applications and TCP layer attacks. To address this issue, we leveraged the usage of a flow-based dataset containing flow data for TCP/IP layer and application layer attacks. Six different ML models are trained with the generated dataset to evaluate the effectiveness of the MediHunt framework in detecting real-time attacks. F1 scores and detection accuracy exceeded 0.99 for the proposed MediHunt framework with our custom dataset.

Title: A framework for securing email entrances and mitigating phishing impersonation attacks. (arXiv:2312.04100v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.04100
• Code URL: null
• Copy Paste: [[2312.04100]] A framework for securing email entrances and mitigating phishing impersonation attacks(http://arxiv.org/abs/2312.04100)
• Summary:

  Emails are used every day for communication, and many countries and organisations mostly use email for official communications. It is highly valued and recognised for confidential conversations and transactions in day-to-day business. The Often use of this channel and the quality of information it carries attracted cyber attackers to it. There are many existing techniques to mitigate attacks on email, however, the systems are more focused on email content and behaviour and not securing entrances to email boxes, composition, and settings. This work intends to protect users' email composition and settings to prevent attackers from using an account when it gets hacked or hijacked and stop them from setting forwarding on the victim's email account to a different account which automatically stops the user from receiving emails. A secure code is applied to the composition send button to curtail insider impersonation attack. Also, to secure open applications on public and private devices.

Title: TI-DNS: A Trusted and Incentive DNS Resolution Architecture based on Blockchain. (arXiv:2312.04114v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.04114
• Code URL: null
• Copy Paste: [[2312.04114]] TI-DNS: A Trusted and Incentive DNS Resolution Architecture based on Blockchain(http://arxiv.org/abs/2312.04114)
• Summary:

  Domain Name System (DNS) is a critical component of the Internet infrastructure, responsible for translating domain names into IP addresses. However, DNS is vulnerable to some malicious attacks, including DNS cache poisoning, which redirects users to malicious websites displaying offensive or illegal content. Existing countermeasures often suffer from at least one of the following weakness: weak attack resistance, high overhead, or complex implementation. To address these challenges, this paper presents TI-DNS, a blockchain-based DNS resolution architecture designed to detect and correct the forged DNS records caused by the cache poisoning attacks in the DNS resolution process. TI-DNS leverages a multi-resolver Query Vote mechanism to ensure the credibility of verified records on the blockchain ledger and a stake-based incentive mechanism to promote well-behaved participation. Importantly, TI-DNS is easy to be adopted as it only requires modifications to the resolver side of current DNS infrastructure. Finally, we develop a prototype and evaluate it against alternative solutions. The result demonstrates that TI-DNS effectively and efficiently solves DNS cache poisoning.

Title: Improved Efficient Two-Stage Denoising Diffusion Power System Measurement Recovery Against False Data Injection Attacks and Data Losses. (arXiv:2312.04346v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04346
• Code URL: null
• Copy Paste: [[2312.04346]] Improved Efficient Two-Stage Denoising Diffusion Power System Measurement Recovery Against False Data Injection Attacks and Data Losses(http://arxiv.org/abs/2312.04346)
• Summary:

  Measurement uncertainties, represented by cyber-attacks and data losses, seriously degrade the quality of power system measurements. Fortunately, the powerful generation ability of the denoising diffusion models can enable more precise measurement generation for power system data recovery. However, the controllable data generation and efficient computing methods of denoising diffusion models for deterministic trajectory still need further investigation. To this end, this paper proposes an improved two-stage denoising diffusion model (TSDM) to identify and reconstruct the measurements with various measurement uncertainties. The first stage of the model comprises a classifier-guided conditional anomaly detection component, while the second stage involves diffusion-based measurement imputation component. Moreover, the proposed TSDM adopts precise means and optimal variances to accelerate the diffusion generation process with subsequence sampling. Extensive numerical case studies demonstrate that the proposed TSDM can accurately recover power system measurements despite strong randomness under renewable energy integration and highly nonlinear dynamics under complex cyber-physical contingencies. Additionally, the proposed TSDM has stronger robustness compared to existing reconstruction networks and exhibits lower computational complexity than general denoising diffusion models.

Title: FreqFed: A Frequency Analysis-Based Approach for Mitigating Poisoning Attacks in Federated Learning. (arXiv:2312.04432v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.04432
• Code URL: null
• Copy Paste: [[2312.04432]] FreqFed: A Frequency Analysis-Based Approach for Mitigating Poisoning Attacks in Federated Learning(http://arxiv.org/abs/2312.04432)
• Summary:

  Federated learning (FL) is a collaborative learning paradigm allowing multiple clients to jointly train a model without sharing their training data. However, FL is susceptible to poisoning attacks, in which the adversary injects manipulated model updates into the federated model aggregation process to corrupt or destroy predictions (untargeted poisoning) or implant hidden functionalities (targeted poisoning or backdoors). Existing defenses against poisoning attacks in FL have several limitations, such as relying on specific assumptions about attack types and strategies or data distributions or not sufficiently robust against advanced injection techniques and strategies and simultaneously maintaining the utility of the aggregated model. To address the deficiencies of existing defenses, we take a generic and completely different approach to detect poisoning (targeted and untargeted) attacks. We present FreqFed, a novel aggregation mechanism that transforms the model updates (i.e., weights) into the frequency domain, where we can identify the core frequency components that inherit sufficient information about weights. This allows us to effectively filter out malicious updates during local training on the clients, regardless of attack types, strategies, and clients' data distributions. We extensively evaluate the efficiency and effectiveness of FreqFed in different application domains, including image classification, word prediction, IoT intrusion detection, and speech recognition. We demonstrate that FreqFed can mitigate poisoning attacks effectively with a negligible impact on the utility of the aggregated model.

robust

Title: Low-power, Continuous Remote Behavioral Localization with Event Cameras. (arXiv:2312.03799v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03799
• Code URL: null
• Copy Paste: [[2312.03799]] Low-power, Continuous Remote Behavioral Localization with Event Cameras(http://arxiv.org/abs/2312.03799)
• Summary:

  Researchers in natural science need reliable methods for quantifying animal behavior. Recently, numerous computer vision methods emerged to automate the process. However, observing wild species at remote locations remains a challenging task due to difficult lighting conditions and constraints on power supply and data storage. Event cameras offer unique advantages for battery-dependent remote monitoring due to their low power consumption and high dynamic range capabilities. We use this novel sensor to quantify a behavior in Chinstrap penguins called ecstatic display. We formulate the problem as a temporal action detection task, determining the start and end times of the behavior. For this purpose, we recorded a colony of breeding penguins in Antarctica during several weeks and labeled event data on 16 nests. The developed method consists of a generator of candidate time intervals (proposals) and a classifier of the actions within them. The experiments show that the event cameras' natural response to motion is effective for continuous behavior monitoring and detection, reaching a mean average precision (mAP) of 58% (which increases to 63% in good weather conditions). The results also demonstrate the robustness against various lighting conditions contained in the challenging dataset. The low-power capabilities of the event camera allows to record three times longer than with a conventional camera. This work pioneers the use of event cameras for remote wildlife observation, opening new interdisciplinary opportunities. https://tub-rip.github.io/eventpenguins/

Title: LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures. (arXiv:2312.04000v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04000
• Code URL: null
• Copy Paste: [[2312.04000]] LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures(http://arxiv.org/abs/2312.04000)
• Summary:

  Joint embedding (JE) architectures have emerged as a promising avenue for acquiring transferable data representations. A key obstacle to using JE methods, however, is the inherent challenge of evaluating learned representations without access to a downstream task, and an annotated dataset. Without efficient and reliable evaluation, it is difficult to iterate on architectural and training choices for JE methods. In this paper, we introduce LiDAR (Linear Discriminant Analysis Rank), a metric designed to measure the quality of representations within JE architectures. Our metric addresses several shortcomings of recent approaches based on feature covariance rank by discriminating between informative and uninformative features. In essence, LiDAR quantifies the rank of the Linear Discriminant Analysis (LDA) matrix associated with the surrogate SSL task -- a measure that intuitively captures the information content as it pertains to solving the SSL task. We empirically demonstrate that LiDAR significantly surpasses naive rank based approaches in its predictive power of optimal hyperparameters. Our proposed criterion presents a more robust and intuitive means of assessing the quality of representations within JE architectures, which we hope facilitates broader adoption of these powerful techniques in various domains.

Title: Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching. (arXiv:2312.04060v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04060
• Code URL: https://github.com/junshengzhou/vp2p-match
• Copy Paste: [[2312.04060]] Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching(http://arxiv.org/abs/2312.04060)
• Summary:

  Cross-modality registration between 2D images from cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic. Previous methods estimate 2D-3D correspondences by matching point and pixel patterns learned by neural networks, and use Perspective-n-Points (PnP) to estimate rigid transformation during post-processing. However, these methods struggle to map points and pixels to a shared latent space robustly since points and pixels have very different characteristics with patterns learned in different manners (MLP and CNN), and they also fail to construct supervision directly on the transformation since the PnP is non-differentiable, which leads to unstable registration results. To address these problems, we propose to learn a structured cross-modality latent space to represent pixel features and 3D features via a differentiable probabilistic PnP solver. Specifically, we design a triplet network to learn VoxelPoint-to-Pixel matching, where we represent 3D elements using both voxels and points to learn the cross-modality latent space with pixels. We design both the voxel and pixel branch based on CNNs to operate convolutions on voxels/pixels represented in grids, and integrate an additional point branch to regain the information lost during voxelization. We train our framework end-to-end by imposing supervisions directly on the predicted pose distribution with a probabilistic PnP solver. To explore distinctive patterns of cross-modality features, we design a novel loss with adaptive-weighted optimization for cross-modality feature description. The experimental results on KITTI and nuScenes datasets show significant improvements over the state-of-the-art methods. The code and models are available at https://github.com/junshengzhou/VP2P-Match.

Title: Towards 4D Human Video Stylization. (arXiv:2312.04143v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04143
• Code URL: https://github.com/tiantianwang/4d_video_stylization
• Copy Paste: [[2312.04143]] Towards 4D Human Video Stylization(http://arxiv.org/abs/2312.04143)
• Summary:

  We present a first step towards 4D (3D and time) human video stylization, which addresses style transfer, novel view synthesis and human animation within a unified framework. While numerous video stylization methods have been developed, they are often restricted to rendering images in specific viewpoints of the input video, lacking the capability to generalize to novel views and novel poses in dynamic scenes. To overcome these limitations, we leverage Neural Radiance Fields (NeRFs) to represent videos, conducting stylization in the rendered feature space. Our innovative approach involves the simultaneous representation of both the human subject and the surrounding scene using two NeRFs. This dual representation facilitates the animation of human subjects across various poses and novel viewpoints. Specifically, we introduce a novel geometry-guided tri-plane representation, significantly enhancing feature representation robustness compared to direct tri-plane optimization. Following the video reconstruction, stylization is performed within the NeRFs' rendered feature space. Extensive experiments demonstrate that the proposed method strikes a superior balance between stylized textures and temporal coherence, surpassing existing approaches. Furthermore, our framework uniquely extends its capabilities to accommodate novel poses and viewpoints, making it a versatile tool for creative human video stylization.

Title: EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer. (arXiv:2312.04152v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04152
• Code URL: https://github.com/vut-hfut/eulermormer
• Copy Paste: [[2312.04152]] EulerMormer: Robust Eulerian Motion Magnification via Dynamic Filtering within Transformer(http://arxiv.org/abs/2312.04152)
• Summary:

  Video Motion Magnification (VMM) aims to break the resolution limit of human visual perception capability and reveal the imperceptible minor motion that contains valuable information in the macroscopic domain. However, challenges arise in this task due to photon noise inevitably introduced by photographic devices and spatial inconsistency in amplification, leading to flickering artifacts in static fields and motion blur and distortion in dynamic fields in the video. Existing methods focus on explicit motion modeling without emphasizing prioritized denoising during the motion magnification process. This paper proposes a novel dynamic filtering strategy to achieve static-dynamic field adaptive denoising. Specifically, based on Eulerian theory, we separate texture and shape to extract motion representation through inter-frame shape differences, expecting to leverage these subdivided features to solve this task finely. Then, we introduce a novel dynamic filter that eliminates noise cues and preserves critical features in the motion magnification and amplification generation phases. Overall, our unified framework, EulerMormer, is a pioneering effort to first equip with Transformer in learning-based VMM. The core of the dynamic filter lies in a global dynamic sparse cross-covariance attention mechanism that explicitly removes noise while preserving vital information, coupled with a multi-scale dual-path gating mechanism that selectively regulates the dependence on different frequency features to reduce spatial attenuation and complement motion boundaries. We demonstrate extensive experiments that EulerMormer achieves more robust video motion magnification from the Eulerian perspective, significantly outperforming state-of-the-art methods. The source code is available at https://github.com/VUT-HFUT/EulerMormer.

Title: Joint-Individual Fusion Structure with Fusion Attention Module for Multi-Modal Skin Cancer Classification. (arXiv:2312.04189v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04189
• Code URL: null
• Copy Paste: [[2312.04189]] Joint-Individual Fusion Structure with Fusion Attention Module for Multi-Modal Skin Cancer Classification(http://arxiv.org/abs/2312.04189)
• Summary:

  Most convolutional neural network (CNN) based methods for skin cancer classification obtain their results using only dermatological images. Although good classification results have been shown, more accurate results can be achieved by considering the patient's metadata, which is valuable clinical information for dermatologists. Current methods only use the simple joint fusion structure (FS) and fusion modules (FMs) for the multi-modal classification methods, there still is room to increase the accuracy by exploring more advanced FS and FM. Therefore, in this paper, we design a new fusion method that combines dermatological images (dermoscopy images or clinical images) and patient metadata for skin cancer classification from the perspectives of FS and FM. First, we propose a joint-individual fusion (JIF) structure that learns the shared features of multi-modality data and preserves specific features simultaneously. Second, we introduce a fusion attention (FA) module that enhances the most relevant image and metadata features based on both the self and mutual attention mechanism to support the decision-making pipeline. We compare the proposed JIF-MMFA method with other state-of-the-art fusion methods on three different public datasets. The results show that our JIF-MMFA method improves the classification results for all tested CNN backbones and performs better than the other fusion methods on the three public datasets, demonstrating our method's effectiveness and robustness

Title: Adventures of Trustworthy Vision-Language Models: A Survey. (arXiv:2312.04231v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04231
• Code URL: null
• Copy Paste: [[2312.04231]] Adventures of Trustworthy Vision-Language Models: A Survey(http://arxiv.org/abs/2312.04231)
• Summary:

  Recently, transformers have become incredibly popular in computer vision and vision-language tasks. This notable rise in their usage can be primarily attributed to the capabilities offered by attention mechanisms and the outstanding ability of transformers to adapt and apply themselves to a variety of tasks and domains. Their versatility and state-of-the-art performance have established them as indispensable tools for a wide array of applications. However, in the constantly changing landscape of machine learning, the assurance of the trustworthiness of transformers holds utmost importance. This paper conducts a thorough examination of vision-language transformers, employing three fundamental principles of responsible AI: Bias, Robustness, and Interpretability. The primary objective of this paper is to delve into the intricacies and complexities associated with the practical use of transformers, with the overarching goal of advancing our comprehension of how to enhance their reliability and accountability.

Title: A Multi-scale Information Integration Framework for Infrared and Visible Image Fusion. (arXiv:2312.04328v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04328
• Code URL: https://github.com/ssyangguang/mda
• Copy Paste: [[2312.04328]] A Multi-scale Information Integration Framework for Infrared and Visible Image Fusion(http://arxiv.org/abs/2312.04328)
• Summary:

  Infrared and visible image fusion aims at generating a fused image containing the intensity and detail information of source images, and the key issue is effectively measuring and integrating the complementary information of multi-modality images from the same scene. Existing methods mostly adopt a simple weight in the loss function to decide the information retention of each modality rather than adaptively measuring complementary information for different image pairs. In this study, we propose a multi-scale dual attention (MDA) framework for infrared and visible image fusion, which is designed to measure and integrate complementary information in both structure and loss function at the image and patch level. In our method, the residual downsample block decomposes source images into three scales first. Then, dual attention fusion block integrates complementary information and generates a spatial and channel attention map at each scale for feature fusion. Finally, the output image is reconstructed by the residual reconstruction block. Loss function consists of image-level, feature-level and patch-level three parts, of which the calculation of the image-level and patch-level two parts are based on the weights generated by the complementary information measurement. Indeed, to constrain the pixel intensity distribution between the output and infrared image, a style loss is added. Our fusion results perform robust and informative across different scenarios. Qualitative and quantitative results on two datasets illustrate that our method is able to preserve both thermal radiation and detailed information from two modalities and achieve comparable results compared with the other state-of-the-art methods. Ablation experiments show the effectiveness of our information integration architecture and adaptively measure complementary information retention in the loss function.

Title: Multi-View Unsupervised Image Generation with Cross Attention Guidance. (arXiv:2312.04337v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04337
• Code URL: null
• Copy Paste: [[2312.04337]] Multi-View Unsupervised Image Generation with Cross Attention Guidance(http://arxiv.org/abs/2312.04337)
• Summary:

  The growing interest in novel view synthesis, driven by Neural Radiance Field (NeRF) models, is hindered by scalability issues due to their reliance on precisely annotated multi-view images. Recent models address this by fine-tuning large text2image diffusion models on synthetic multi-view data. Despite robust zero-shot generalization, they may need post-processing and can face quality issues due to the synthetic-real domain gap. This paper introduces a novel pipeline for unsupervised training of a pose-conditioned diffusion model on single-category datasets. With the help of pretrained self-supervised Vision Transformers (DINOv2), we identify object poses by clustering the dataset through comparing visibility and locations of specific object parts. The pose-conditioned diffusion model, trained on pose labels, and equipped with cross-frame attention at inference time ensures cross-view consistency, that is further aided by our novel hard-attention guidance. Our model, MIRAGE, surpasses prior work in novel view synthesis on real images. Furthermore, MIRAGE is robust to diverse textures and geometries, as demonstrated with our experiments on synthetic images generated with pretrained Stable Diffusion.

Title: DemoCaricature: Democratising Caricature Generation with a Rough Sketch. (arXiv:2312.04364v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04364
• Code URL: null
• Copy Paste: [[2312.04364]] DemoCaricature: Democratising Caricature Generation with a Rough Sketch(http://arxiv.org/abs/2312.04364)
• Summary:

  In this paper, we democratise caricature generation, empowering individuals to effortlessly craft personalised caricatures with just a photo and a conceptual sketch. Our objective is to strike a delicate balance between abstraction and identity, while preserving the creativity and subjectivity inherent in a sketch. To achieve this, we present Explicit Rank-1 Model Editing alongside single-image personalisation, selectively applying nuanced edits to cross-attention layers for a seamless merge of identity and style. Additionally, we propose Random Mask Reconstruction to enhance robustness, directing the model to focus on distinctive identity and style features. Crucially, our aim is not to replace artists but to eliminate accessibility barriers, allowing enthusiasts to engage in the artistry.

Title: Exploring the Robustness of Model-Graded Evaluations and Automated Interpretability. (arXiv:2312.03721v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03721
• Code URL: null
• Copy Paste: [[2312.03721]] Exploring the Robustness of Model-Graded Evaluations and Automated Interpretability(http://arxiv.org/abs/2312.03721)
• Summary:

  There has been increasing interest in evaluations of language models for a variety of risks and characteristics. Evaluations relying on natural language understanding for grading can often be performed at scale by using other language models. We test the robustness of these model-graded evaluations to injections on different datasets including a new Deception Eval. These injections resemble direct communication between the testee and the evaluator to change their grading. We extrapolate that future, more intelligent models might manipulate or cooperate with their evaluation model. We find significant susceptibility to these injections in state-of-the-art commercial models on all examined evaluations. Furthermore, similar injections can be used on automated interpretability frameworks to produce misleading model-written explanations. The results inspire future work and should caution against unqualified trust in evaluations and automated interpretability.

Title: MultiGPrompt for Multi-Task Pre-Training and Prompting on Graphs. (arXiv:2312.03731v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03731
• Code URL: null
• Copy Paste: [[2312.03731]] MultiGPrompt for Multi-Task Pre-Training and Prompting on Graphs(http://arxiv.org/abs/2312.03731)
• Summary:

  Graphs can inherently model interconnected objects on the Web, thereby facilitating a series of Web applications, such as web analyzing and content recommendation. Recently, Graph Neural Networks (GNNs) have emerged as a mainstream technique for graph representation learning. However, their efficacy within an end-to-end supervised framework is significantly tied to the availabilityof task-specific labels. To mitigate labeling costs and enhance robustness in few-shot settings, pre-training on self-supervised tasks has emerged as a promising method, while prompting has been proposed to further narrow the objective gap between pretext and downstream tasks. Although there has been some initial exploration of prompt-based learning on graphs, they primarily leverage a single pretext task, resulting in a limited subset of general knowledge that could be learned from the pre-training data. Hence, in this paper, we propose MultiGPrompt, a novel multi-task pre-training and prompting framework to exploit multiple pretext tasks for more comprehensive pre-trained knowledge. First, in pre-training, we design a set of pretext tokens to synergize multiple pretext tasks. Second, we propose a dual-prompt mechanism consisting of composed and open prompts to leverage task-specific and global pre-training knowledge, to guide downstream tasks in few-shot settings. Finally, we conduct extensive experiments on six public datasets to evaluate and analyze MultiGPrompt.

Title: Syntactic Fusion: Enhancing Aspect-Level Sentiment Analysis Through Multi-Tree Graph Integration. (arXiv:2312.03738v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03738
• Code URL: null
• Copy Paste: [[2312.03738]] Syntactic Fusion: Enhancing Aspect-Level Sentiment Analysis Through Multi-Tree Graph Integration(http://arxiv.org/abs/2312.03738)
• Summary:

  Recent progress in aspect-level sentiment classification has been propelled by the incorporation of graph neural networks (GNNs) leveraging syntactic structures, particularly dependency trees. Nevertheless, the performance of these models is often hampered by the innate inaccuracies of parsing algorithms. To mitigate this challenge, we introduce SynthFusion, an innovative graph ensemble method that amalgamates predictions from multiple parsers. This strategy blends diverse dependency relations prior to the application of GNNs, enhancing robustness against parsing errors while avoiding extra computational burdens. SynthFusion circumvents the pitfalls of overparameterization and diminishes the risk of overfitting, prevalent in models with stacked GNN layers, by optimizing graph connectivity. Our empirical evaluations on the SemEval14 and Twitter14 datasets affirm that SynthFusion not only outshines models reliant on single dependency trees but also eclipses alternative ensemble techniques, achieving this without an escalation in model complexity.

Title: Easy Data Augmentation in Sentiment Analysis of Cyberbullying. (arXiv:2312.03743v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03743
• Code URL: null
• Copy Paste: [[2312.03743]] Easy Data Augmentation in Sentiment Analysis of Cyberbullying(http://arxiv.org/abs/2312.03743)
• Summary:

  Instagram, a social media platform, has in the vicinity of 2 billion active users in 2023. The platform allows users to post photos and videos with one another. However, cyberbullying remains a significant problem for about 50% of young Indonesians. To address this issue, sentiment analysis for comment filtering uses a Support Vector Machine (SVM) and Easy Data Augmentation (EDA). EDA will augment the dataset, enabling robust prediction and analysis of cyberbullying by introducing more variation. Based on the tests, SVM combination with EDA results in a 2.52% increase in the k-Fold Cross Validation score. Our proposed approach shows an improved accuracy of 92.5%, 2.5% higher than that of the existing state-of-the-art method. To maintain the reproducibility and replicability of this research, the source code can be accessed at uns.id/eda_svm.

Title: English to Arabic machine translation of mathematical documents. (arXiv:2312.03753v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03753
• Code URL: null
• Copy Paste: [[2312.03753]] English to Arabic machine translation of mathematical documents(http://arxiv.org/abs/2312.03753)
• Summary:

  This paper is about the development of a machine translation system tailored specifically for LATEX mathematical documents. The system focuses on translating English LATEX mathematical documents into Arabic LATEX, catering to the growing demand for multilingual accessibility in scientific and mathematical literature. With the vast proliferation of LATEX mathematical documents the need for an efficient and accurate translation system has become increasingly essential. This paper addresses the necessity for a robust translation tool that enables seamless communication and comprehension of complex mathematical content across language barriers. The proposed system leverages a Transformer model as the core of the translation system, ensuring enhanced accuracy and fluency in the translated Arabic LATEX documents. Furthermore, the integration of RyDArab, an Arabic mathematical TEX extension, along with a rule-based translator for Arabic mathematical expressions, contributes to the precise rendering of complex mathematical symbols and equations in the translated output. The paper discusses the architecture, methodology, of the developed system, highlighting its efficacy in bridging the language gap in the domain of mathematical documentation

Title: RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training. (arXiv:2312.04032v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.04032
• Code URL: https://github.com/bbuing9/roast
• Copy Paste: [[2312.04032]] RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training(http://arxiv.org/abs/2312.04032)
• Summary:

  Fine-tuning pre-trained language models (LMs) has become the de facto standard in many NLP tasks. Nevertheless, fine-tuned LMs are still prone to robustness issues, such as adversarial robustness and model calibration. Several perspectives of robustness for LMs have been studied independently, but lacking a unified consideration in multiple perspectives. In this paper, we propose Robustifying LMs via Adversarial perturbation with Selective Training (RoAST), a simple yet effective fine-tuning technique to enhance the multi-perspective robustness of LMs in a unified way. RoAST effectively incorporates two important sources for the model robustness, robustness on the perturbed inputs and generalizable knowledge in pre-trained LMs. To be specific, RoAST introduces adversarial perturbation during fine-tuning while the model parameters are selectively updated upon their relative importance to minimize unnecessary deviation. Under a unified evaluation of fine-tuned LMs by incorporating four representative perspectives of model robustness, we demonstrate the effectiveness of RoAST compared to state-of-the-art fine-tuning methods on six different types of LMs, which indicates its usefulness in practice.

Title: Similarity-based Knowledge Transfer for Cross-Domain Reinforcement Learning. (arXiv:2312.03764v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.03764
• Code URL: null
• Copy Paste: [[2312.03764]] Similarity-based Knowledge Transfer for Cross-Domain Reinforcement Learning(http://arxiv.org/abs/2312.03764)
• Summary:

  Transferring knowledge in cross-domain reinforcement learning is a challenging setting in which learning is accelerated by reusing knowledge from a task with different observation and/or action space. However, it is often necessary to carefully select the source of knowledge for the receiving end to benefit from the transfer process. In this article, we study how to measure the similarity between cross-domain reinforcement learning tasks to select a source of knowledge that will improve the performance of the learning agent. We developed a semi-supervised alignment loss to match different spaces with a set of encoder-decoders, and use them to measure similarity and transfer policies across tasks. In comparison to prior works, our method does not require data to be aligned, paired or collected by expert policies. Experimental results, on a set of varied Mujoco control tasks, show the robustness of our method in effectively selecting and transferring knowledge, without the supervision of a tailored set of source tasks.

Title: Multi-Scale and Multi-Modal Contrastive Learning Network for Biomedical Time Series. (arXiv:2312.03796v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.03796
• Code URL: null
• Copy Paste: [[2312.03796]] Multi-Scale and Multi-Modal Contrastive Learning Network for Biomedical Time Series(http://arxiv.org/abs/2312.03796)
• Summary:

  Multi-modal biomedical time series (MBTS) data offers a holistic view of the physiological state, holding significant importance in various bio-medical applications. Owing to inherent noise and distribution gaps across different modalities, MBTS can be complex to model. Various deep learning models have been developed to learn representations of MBTS but still fall short in robustness due to the ignorance of modal-to-modal variations. This paper presents a multi-scale and multi-modal biomedical time series representation learning (MBSL) network with contrastive learning to migrate these variations. Firstly, MBTS is grouped based on inter-modal distances, then each group with minimum intra-modal variations can be effectively modeled by individual encoders. Besides, to enhance the multi-scale feature extraction (encoder), various patch lengths and mask ratios are designed to generate tokens with semantic information at different scales and diverse contextual perspectives respectively. Finally, cross-modal contrastive learning is proposed to maximize consistency among inter-modal groups, maintaining useful information and eliminating noises. Experiments against four bio-medical applications show that MBSL outperforms state-of-the-art models by 33.9% mean average errors (MAE) in respiration rate, by 13.8% MAE in exercise heart rate, by 1.41% accuracy in human activity recognition, and by 1.14% F1-score in obstructive sleep apnea-hypopnea syndrome.

Title: Learning Genomic Sequence Representations using Graph Neural Networks over De Bruijn Graphs. (arXiv:2312.03865v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.03865
• Code URL: https://github.com/ratschlab/genomic-gnn
• Copy Paste: [[2312.03865]] Learning Genomic Sequence Representations using Graph Neural Networks over De Bruijn Graphs(http://arxiv.org/abs/2312.03865)
• Summary:

  The rapid expansion of genomic sequence data calls for new methods to achieve robust sequence representations. Existing techniques often neglect intricate structural details, emphasizing mainly contextual information. To address this, we developed k-mer embeddings that merge contextual and structural string information by enhancing De Bruijn graphs with structural similarity connections. Subsequently, we crafted a self-supervised method based on Contrastive Learning that employs a heterogeneous Graph Convolutional Network encoder and constructs positive pairs based on node similarities. Our embeddings consistently outperform prior techniques for Edit Distance Approximation and Closest String Retrieval tasks.

Title: MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator. (arXiv:2312.03991v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.03991
• Code URL: null
• Copy Paste: [[2312.03991]] MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator(http://arxiv.org/abs/2312.03991)
• Summary:

  Offline reinforcement learning (RL) faces a significant challenge of distribution shift. Model-free offline RL penalizes the Q value for out-of-distribution (OOD) data or constrains the policy closed to the behavior policy to tackle this problem, but this inhibits the exploration of the OOD region. Model-based offline RL, which uses the trained environment model to generate more OOD data and performs conservative policy optimization within that model, has become an effective method for this problem. However, the current model-based algorithms rarely consider agent robustness when incorporating conservatism into policy. Therefore, the new model-based offline algorithm with a conservative Bellman operator (MICRO) is proposed. This method trades off performance and robustness via introducing the robust Bellman operator into the algorithm. Compared with previous model-based algorithms with robust adversarial models, MICRO can significantly reduce the computation cost by only choosing the minimal Q value in the state uncertainty set. Extensive experiments demonstrate that MICRO outperforms prior RL algorithms in offline RL benchmark and is considerably robust to adversarial perturbations.

Title: A Robust and Efficient Boundary Point Detection Method by Measuring Local Direction Dispersion. (arXiv:2312.04065v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04065
• Code URL: null
• Copy Paste: [[2312.04065]] A Robust and Efficient Boundary Point Detection Method by Measuring Local Direction Dispersion(http://arxiv.org/abs/2312.04065)
• Summary:

  Boundary points pose a significant challenge for machine learning tasks, including classification, clustering, and dimensionality reduction. Due to the similarity of features, boundary areas can result in mixed-up classes or clusters, leading to a crowding problem in dimensionality reduction. To address this challenge, numerous boundary point detection methods have been developed, but they are insufficiently to accurately and efficiently identify the boundary points in non-convex structures and high-dimensional manifolds. In this work, we propose a robust and efficient method for detecting boundary points using Local Direction Dispersion (LoDD). LoDD considers that internal points are surrounded by neighboring points in all directions, while neighboring points of a boundary point tend to be distributed only in a certain directional range. LoDD adopts a density-independent K-Nearest Neighbors (KNN) method to determine neighboring points, and defines a statistic-based metric using the eigenvalues of the covariance matrix of KNN coordinates to measure the centrality of a query point. We demonstrated the validity of LoDD on five synthetic datasets (2-D and 3-D) and ten real-world benchmarks, and tested its clustering performance by equipping with two typical clustering methods, K-means and Ncut. Our results show that LoDD achieves promising and robust detection accuracy in a time-efficient manner.

Title: MeanCut: A Greedy-Optimized Graph Clustering via Path-based Similarity and Degree Descent Criterion. (arXiv:2312.04067v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04067
• Code URL: https://github.com/zpguigroupwhu/meancut-clustering
• Copy Paste: [[2312.04067]] MeanCut: A Greedy-Optimized Graph Clustering via Path-based Similarity and Degree Descent Criterion(http://arxiv.org/abs/2312.04067)
• Summary:

  As the most typical graph clustering method, spectral clustering is popular and attractive due to the remarkable performance, easy implementation, and strong adaptability. Classical spectral clustering measures the edge weights of graph using pairwise Euclidean-based metric, and solves the optimal graph partition by relaxing the constraints of indicator matrix and performing Laplacian decomposition. However, Euclidean-based similarity might cause skew graph cuts when handling non-spherical data distributions, and the relaxation strategy introduces information loss. Meanwhile, spectral clustering requires specifying the number of clusters, which is hard to determine without enough prior knowledge. In this work, we leverage the path-based similarity to enhance intra-cluster associations, and propose MeanCut as the objective function and greedily optimize it in degree descending order for a nondestructive graph partition. This algorithm enables the identification of arbitrary shaped clusters and is robust to noise. To reduce the computational complexity of similarity calculation, we transform optimal path search into generating the maximum spanning tree (MST), and develop a fast MST (FastMST) algorithm to further improve its time-efficiency. Moreover, we define a density gradient factor (DGF) for separating the weakly connected clusters. The validity of our algorithm is demonstrated by testifying on real-world benchmarks and application of face recognition. The source code of MeanCut is available at https://github.com/ZPGuiGroupWhu/MeanCut-Clustering.

Title: On the adaptation of in-context learners for system identification. (arXiv:2312.04083v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04083
• Code URL: null
• Copy Paste: [[2312.04083]] On the adaptation of in-context learners for system identification(http://arxiv.org/abs/2312.04083)
• Summary:

  In-context system identification aims at constructing meta-models to describe classes of systems, differently from traditional approaches that model single systems. This paradigm facilitates the leveraging of knowledge acquired from observing the behaviour of different, yet related dynamics. This paper discusses the role of meta-model adaptation. Through numerical examples, we demonstrate how meta-model adaptation can enhance predictive performance in three realistic scenarios: tailoring the meta-model to describe a specific system rather than a class; extending the meta-model to capture the behaviour of systems beyond the initial training class; and recalibrating the model for new prediction tasks. Results highlight the effectiveness of meta-model adaptation to achieve a more robust and versatile meta-learning framework for system identification.

Title: Finding Interpretable Class-Specific Patterns through Efficient Neural Search. (arXiv:2312.04311v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04311
• Code URL: null
• Copy Paste: [[2312.04311]] Finding Interpretable Class-Specific Patterns through Efficient Neural Search(http://arxiv.org/abs/2312.04311)
• Summary:

  Discovering patterns in data that best describe the differences between classes allows to hypothesize and reason about class-specific mechanisms. In molecular biology, for example, this bears promise of advancing the understanding of cellular processes differing between tissues or diseases, which could lead to novel treatments. To be useful in practice, methods that tackle the problem of finding such differential patterns have to be readily interpretable by domain experts, and scalable to the extremely high-dimensional data.

  In this work, we propose a novel, inherently interpretable binary neural network architecture DIFFNAPS that extracts differential patterns from data. DiffNaps is scalable to hundreds of thousands of features and robust to noise, thus overcoming the limitations of current state-of-the-art methods in large-scale applications such as in biology. We show on synthetic and real world data, including three biological applications, that, unlike its competitors, DiffNaps consistently yields accurate, succinct, and interpretable class descriptions

biometric

steal

extraction

Title: Single Image Reflection Removal with Reflection Intensity Prior Knowledge. (arXiv:2312.03798v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03798
• Code URL: null
• Copy Paste: [[2312.03798]] Single Image Reflection Removal with Reflection Intensity Prior Knowledge(http://arxiv.org/abs/2312.03798)
• Summary:

  Single Image Reflection Removal (SIRR) in real-world images is a challenging task due to diverse image degradations occurring on the glass surface during light transmission and reflection. Many existing methods rely on specific prior assumptions to resolve the problem. In this paper, we propose a general reflection intensity prior that captures the intensity of the reflection phenomenon and demonstrate its effectiveness. To learn the reflection intensity prior, we introduce the Reflection Prior Extraction Network (RPEN). By segmenting images into regional patches, RPEN learns non-uniform reflection prior in an image. We propose Prior-based Reflection Removal Network (PRRN) using a simple transformer U-Net architecture that adapts reflection prior fed from RPEN. Experimental results on real-world benchmarks demonstrate the effectiveness of our approach achieving state-of-the-art accuracy in SIRR.

Title: Leveraging AI-derived Data for Carbon Accounting: Information Extraction from Alternative Sources. (arXiv:2312.03722v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03722
• Code URL: null
• Copy Paste: [[2312.03722]] Leveraging AI-derived Data for Carbon Accounting: Information Extraction from Alternative Sources(http://arxiv.org/abs/2312.03722)
• Summary:

  Carbon accounting is a fundamental building block in our global path to emissions reduction and decarbonization, yet many challenges exist in achieving reliable and trusted carbon accounting measures. We motivate that carbon accounting not only needs to be more data-driven, but also more methodologically sound. We discuss the need for alternative, more diverse data sources that can play a significant role on our path to trusted carbon accounting procedures and elaborate on not only why, but how Artificial Intelligence (AI) in general and Natural Language Processing (NLP) in particular can unlock reasonable access to a treasure trove of alternative data sets in light of the recent advances in the field that better enable the utilization of unstructured data in this process. We present a case study of the recent developments on real-world data via an NLP-powered analysis using OpenAI's GPT API on financial and shipping data. We conclude the paper with a discussion on how these methods and approaches can be integrated into a broader framework for AI-enabled integrative carbon accounting.

Title: Near-real-time Earthquake-induced Fatality Estimation using Crowdsourced Data and Large-Language Models. (arXiv:2312.03755v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03755
• Code URL: null
• Copy Paste: [[2312.03755]] Near-real-time Earthquake-induced Fatality Estimation using Crowdsourced Data and Large-Language Models(http://arxiv.org/abs/2312.03755)
• Summary:

  When a damaging earthquake occurs, immediate information about casualties is critical for time-sensitive decision-making by emergency response and aid agencies in the first hours and days. Systems such as Prompt Assessment of Global Earthquakes for Response (PAGER) by the U.S. Geological Survey (USGS) were developed to provide a forecast within about 30 minutes of any significant earthquake globally. Traditional systems for estimating human loss in disasters often depend on manually collected early casualty reports from global media, a process that's labor-intensive and slow with notable time delays. Recently, some systems have employed keyword matching and topic modeling to extract relevant information from social media. However, these methods struggle with the complex semantics in multilingual texts and the challenge of interpreting ever-changing, often conflicting reports of death and injury numbers from various unverified sources on social media platforms. In this work, we introduce an end-to-end framework to significantly improve the timeliness and accuracy of global earthquake-induced human loss forecasting using multi-lingual, crowdsourced social media. Our framework integrates (1) a hierarchical casualty extraction model built upon large language models, prompt design, and few-shot learning to retrieve quantitative human loss claims from social media, (2) a physical constraint-aware, dynamic-truth discovery model that discovers the truthful human loss from massive noisy and potentially conflicting human loss claims, and (3) a Bayesian updating loss projection model that dynamically updates the final loss estimation using discovered truths. We test the framework in real-time on a series of global earthquake events in 2021 and 2022 and show that our framework streamlines casualty data retrieval, achieving speed and accuracy comparable to manual methods by USGS.

membership infer

federate

Title: A Novel Federated Learning-based Intrusion Detection System for Flying Ad Hoc Networks. (arXiv:2312.04135v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.04135
• Code URL: null
• Copy Paste: [[2312.04135]] A Novel Federated Learning-based Intrusion Detection System for Flying Ad Hoc Networks(http://arxiv.org/abs/2312.04135)
• Summary:

  Unmanned aerial vehicles (UAVs) in flying ad-hoc networks (FANETs) face security challenges due to the dynamic and distributed nature of these networks. This paper presents the Federated Learning-based Intrusion Detection System (FL-IDS), an innovative approach designed to improve FANET security. FL-IDS leverages federated learning to address privacy concerns of centralized intrusion detection systems. FL-IDS operates in a decentralized manner, enabling UAVs to collaboratively train a global intrusion detection model without sharing raw data. Local models are assigned to each UAV, using client-specific data, and only updated model weights are shared with a central server. This preserves privacy while utilizing collective intelligence for effective intrusion detection. Experimental results show FL-IDS's competitive performance with Central IDS (C-IDS) while mitigating privacy concerns. The Bias Towards Specific Clients (BTSC) method further enhances FL-IDS performance, surpassing C-IDS even at lower attacker ratios. A comparative analysis with traditional intrusion detection methods, including Local IDS (L-IDS), provides insights into FL-IDS's strengths. This study significantly contributes to FANET security by introducing a privacy-aware, decentralized intrusion detection approach tailored to the unique challenges of UAV networks.

Title: A Masked Pruning Approach for Dimensionality Reduction in Communication-Efficient Federated Learning Systems. (arXiv:2312.03889v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.03889
• Code URL: null
• Copy Paste: [[2312.03889]] A Masked Pruning Approach for Dimensionality Reduction in Communication-Efficient Federated Learning Systems(http://arxiv.org/abs/2312.03889)
• Summary:

  Federated Learning (FL) represents a growing machine learning (ML) paradigm designed for training models across numerous nodes that retain local datasets, all without directly exchanging the underlying private data with the parameter server (PS). Its increasing popularity is attributed to notable advantages in terms of training deep neural network (DNN) models under privacy aspects and efficient utilization of communication resources. Unfortunately, DNNs suffer from high computational and communication costs, as well as memory consumption in intricate tasks. These factors restrict the applicability of FL algorithms in communication-constrained systems with limited hardware resources.

  In this paper, we develop a novel algorithm that overcomes these limitations by synergistically combining a pruning-based method with the FL process, resulting in low-dimensional representations of the model with minimal communication cost, dubbed Masked Pruning over FL (MPFL). The algorithm operates by initially distributing weights to the nodes through the PS. Subsequently, each node locally trains its model and computes pruning masks. These low-dimensional masks are then transmitted back to the PS, which generates a consensus pruning mask, broadcasted back to the nodes. This iterative process enhances the robustness and stability of the masked pruning model. The generated mask is used to train the FL model, achieving significant bandwidth savings. We present an extensive experimental study demonstrating the superior performance of MPFL compared to existing methods. Additionally, we have developed an open-source software package for the benefit of researchers and developers in related fields.

Title: Improving Communication Efficiency of Federated Distillation via Accumulating Local Updates. (arXiv:2312.04166v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04166
• Code URL: null
• Copy Paste: [[2312.04166]] Improving Communication Efficiency of Federated Distillation via Accumulating Local Updates(http://arxiv.org/abs/2312.04166)
• Summary:

  As an emerging federated learning paradigm, federated distillation enables communication-efficient model training by transmitting only small-scale knowledge during the learning process. To further improve the communication efficiency of federated distillation, we propose a novel technique, ALU, which accumulates multiple rounds of local updates before transferring the knowledge to the central server. ALU drastically decreases the frequency of communication in federated distillation, thereby significantly reducing the communication overhead during the training process. Empirical experiments demonstrate the substantial effect of ALU in improving the communication efficiency of federated distillation.

fair

Title: Multi-Group Fairness Evaluation via Conditional Value-at-Risk Testing. (arXiv:2312.03867v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.03867
• Code URL: null
• Copy Paste: [[2312.03867]] Multi-Group Fairness Evaluation via Conditional Value-at-Risk Testing(http://arxiv.org/abs/2312.03867)
• Summary:

  Machine learning (ML) models used in prediction and classification tasks may display performance disparities across population groups determined by sensitive attributes (e.g., race, sex, age). We consider the problem of evaluating the performance of a fixed ML model across population groups defined by multiple sensitive attributes (e.g., race and sex and age). Here, the sample complexity for estimating the worst-case performance gap across groups (e.g., the largest difference in error rates) increases exponentially with the number of group-denoting sensitive attributes. To address this issue, we propose an approach to test for performance disparities based on Conditional Value-at-Risk (CVaR). By allowing a small probabilistic slack on the groups over which a model has approximately equal performance, we show that the sample complexity required for discovering performance violations is reduced exponentially to be at most upper bounded by the square root of the number of groups. As a byproduct of our analysis, when the groups are weighted by a specific prior distribution, we show that R\'enyi entropy of order $2/3$ of the prior distribution captures the sample complexity of the proposed CVaR test algorithm. Finally, we also show that there exists a non-i.i.d. data collection strategy that results in a sample complexity independent of the number of groups.

Title: On The Fairness Impacts of Hardware Selection in Machine Learning. (arXiv:2312.03886v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.03886
• Code URL: null
• Copy Paste: [[2312.03886]] On The Fairness Impacts of Hardware Selection in Machine Learning(http://arxiv.org/abs/2312.03886)
• Summary:

  In the machine learning ecosystem, hardware selection is often regarded as a mere utility, overshadowed by the spotlight on algorithms and data. This oversight is particularly problematic in contexts like ML-as-a-service platforms, where users often lack control over the hardware used for model deployment. How does the choice of hardware impact generalization properties? This paper investigates the influence of hardware on the delicate balance between model performance and fairness. We demonstrate that hardware choices can exacerbate existing disparities, attributing these discrepancies to variations in gradient flows and loss surfaces across different demographic groups. Through both theoretical and empirical analysis, the paper not only identifies the underlying factors but also proposes an effective strategy for mitigating hardware-induced performance imbalances.

interpretability

explainability

Title: FakeWatch ElectionShield: A Benchmarking Framework to Detect Fake News for Credible US Elections. (arXiv:2312.03730v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03730
• Code URL: null
• Copy Paste: [[2312.03730]] FakeWatch ElectionShield: A Benchmarking Framework to Detect Fake News for Credible US Elections(http://arxiv.org/abs/2312.03730)
• Summary:

  In today's technologically driven world, the spread of fake news, particularly during crucial events such as elections, presents an increasing challenge to the integrity of information. To address this challenge, we introduce FakeWatch ElectionShield, an innovative framework carefully designed to detect fake news. We have created a novel dataset of North American election-related news articles through a blend of advanced language models (LMs) and thorough human verification, for precision and relevance. We propose a model hub of LMs for identifying fake news. Our goal is to provide the research community with adaptable and accurate classification models in recognizing the dynamic nature of misinformation. Extensive evaluation of fake news classifiers on our dataset and a benchmark dataset shows our that while state-of-the-art LMs slightly outperform the traditional ML models, classical models are still competitive with their balance of accuracy, explainability, and computational efficiency. This research sets the foundation for future studies to address misinformation related to elections.

Title: Causality and Explainability for Trustworthy Integrated Pest Management. (arXiv:2312.04343v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04343
• Code URL: null
• Copy Paste: [[2312.04343]] Causality and Explainability for Trustworthy Integrated Pest Management(http://arxiv.org/abs/2312.04343)
• Summary:

  Pesticides serve as a common tool in agricultural pest control but significantly contribute to the climate crisis. To combat this, Integrated Pest Management (IPM) stands as a climate-smart alternative. Despite its potential, IPM faces low adoption rates due to farmers' skepticism about its effectiveness. To address this challenge, we introduce an advanced data analysis framework tailored to enhance IPM adoption. Our framework provides i) robust pest population predictions across diverse environments with invariant and causal learning, ii) interpretable pest presence predictions using transparent models, iii) actionable advice through counterfactual explanations for in-season IPM interventions, iv) field-specific treatment effect estimations, and v) assessments of the effectiveness of our advice using causal inference. By incorporating these features, our framework aims to alleviate skepticism and encourage wider adoption of IPM practices among farmers.

watermark

diffusion

Title: Gaussian3Diff: 3D Gaussian Diffusion for 3D Full Head Synthesis and Editing. (arXiv:2312.03763v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03763
• Code URL: null
• Copy Paste: [[2312.03763]] Gaussian3Diff: 3D Gaussian Diffusion for 3D Full Head Synthesis and Editing(http://arxiv.org/abs/2312.03763)
• Summary:

  We present a novel framework for generating photorealistic 3D human head and subsequently manipulating and reposing them with remarkable flexibility. The proposed approach leverages an implicit function representation of 3D human heads, employing 3D Gaussians anchored on a parametric face model. To enhance representational capabilities and encode spatial information, we embed a lightweight tri-plane payload within each Gaussian rather than directly storing color and opacity. Additionally, we parameterize the Gaussians in a 2D UV space via a 3DMM, enabling effective utilization of the diffusion model for 3D head avatar generation. Our method facilitates the creation of diverse and realistic 3D human heads with fine-grained editing over facial features and expressions. Extensive experiments demonstrate the effectiveness of our method.

Title: DreamInpainter: Text-Guided Subject-Driven Image Inpainting with Diffusion Models. (arXiv:2312.03771v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03771
• Code URL: null
• Copy Paste: [[2312.03771]] DreamInpainter: Text-Guided Subject-Driven Image Inpainting with Diffusion Models(http://arxiv.org/abs/2312.03771)
• Summary:

  This study introduces Text-Guided Subject-Driven Image Inpainting, a novel task that combines text and exemplar images for image inpainting. While both text and exemplar images have been used independently in previous efforts, their combined utilization remains unexplored. Simultaneously accommodating both conditions poses a significant challenge due to the inherent balance required between editability and subject fidelity. To tackle this challenge, we propose a two-step approach DreamInpainter. First, we compute dense subject features to ensure accurate subject replication. Then, we employ a discriminative token selection module to eliminate redundant subject details, preserving the subject's identity while allowing changes according to other conditions such as mask shape and text prompts. Additionally, we introduce a decoupling regularization technique to enhance text control in the presence of exemplar images. Our extensive experiments demonstrate the superior performance of our method in terms of visual quality, identity preservation, and text control, showcasing its effectiveness in the context of text-guided subject-driven image inpainting.

Title: DiffusionAtlas: High-Fidelity Consistent Diffusion Video Editing. (arXiv:2312.03772v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03772
• Code URL: null
• Copy Paste: [[2312.03772]] DiffusionAtlas: High-Fidelity Consistent Diffusion Video Editing(http://arxiv.org/abs/2312.03772)
• Summary:

  We present a diffusion-based video editing framework, namely DiffusionAtlas, which can achieve both frame consistency and high fidelity in editing video object appearance. Despite the success in image editing, diffusion models still encounter significant hindrances when it comes to video editing due to the challenge of maintaining spatiotemporal consistency in the object's appearance across frames. On the other hand, atlas-based techniques allow propagating edits on the layered representations consistently back to frames. However, they often struggle to create editing effects that adhere correctly to the user-provided textual or visual conditions due to the limitation of editing the texture atlas on a fixed UV mapping field. Our method leverages a visual-textual diffusion model to edit objects directly on the diffusion atlases, ensuring coherent object identity across frames. We design a loss term with atlas-based constraints and build a pretrained text-driven diffusion model as pixel-wise guidance for refining shape distortions and correcting texture deviations. Qualitative and quantitative experiments show that our method outperforms state-of-the-art methods in achieving consistent high-fidelity video-object editing.

Title: FAAC: Facial Animation Generation with Anchor Frame and Conditional Control for Superior Fidelity and Editability. (arXiv:2312.03775v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03775
• Code URL: null
• Copy Paste: [[2312.03775]] FAAC: Facial Animation Generation with Anchor Frame and Conditional Control for Superior Fidelity and Editability(http://arxiv.org/abs/2312.03775)
• Summary:

  Over recent years, diffusion models have facilitated significant advancements in video generation. Yet, the creation of face-related videos still confronts issues such as low facial fidelity, lack of frame consistency, limited editability and uncontrollable human poses. To address these challenges, we introduce a facial animation generation method that enhances both face identity fidelity and editing capabilities while ensuring frame consistency. This approach incorporates the concept of an anchor frame to counteract the degradation of generative ability in original text-to-image models when incorporating a motion module. We propose two strategies towards this objective: training-free and training-based anchor frame methods. Our method's efficacy has been validated on multiple representative DreamBooth and LoRA models, delivering substantial improvements over the original outcomes in terms of facial fidelity, text-to-image editability, and video motion. Moreover, we introduce conditional control using a 3D parametric face model to capture accurate facial movements and expressions. This solution augments the creative possibilities for facial animation generation through the integration of multiple control signals. For additional samples, please visit https://anonymous.4open.science/r/FAAC.

Title: AnimateZero: Video Diffusion Models are Zero-Shot Image Animators. (arXiv:2312.03793v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03793
• Code URL: https://github.com/vvictoryuki/animatezero
• Copy Paste: [[2312.03793]] AnimateZero: Video Diffusion Models are Zero-Shot Image Animators(http://arxiv.org/abs/2312.03793)
• Summary:

  Large-scale text-to-video (T2V) diffusion models have great progress in recent years in terms of visual quality, motion and temporal consistency. However, the generation process is still a black box, where all attributes (e.g., appearance, motion) are learned and generated jointly without precise control ability other than rough text descriptions. Inspired by image animation which decouples the video as one specific appearance with the corresponding motion, we propose AnimateZero to unveil the pre-trained text-to-video diffusion model, i.e., AnimateDiff, and provide more precise appearance and motion control abilities for it. For appearance control, we borrow intermediate latents and their features from the text-to-image (T2I) generation for ensuring the generated first frame is equal to the given generated image. For temporal control, we replace the global temporal attention of the original T2V model with our proposed positional-corrected window attention to ensure other frames align with the first frame well. Empowered by the proposed methods, AnimateZero can successfully control the generating progress without further training. As a zero-shot image animator for given images, AnimateZero also enables multiple new applications, including interactive video generation and real image animation. The detailed experiments demonstrate the effectiveness of the proposed method in both T2V and related applications.

Title: AnimatableDreamer: Text-Guided Non-rigid 3D Model Generation and Reconstruction with Canonical Score Distillation. (arXiv:2312.03795v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03795
• Code URL: null
• Copy Paste: [[2312.03795]] AnimatableDreamer: Text-Guided Non-rigid 3D Model Generation and Reconstruction with Canonical Score Distillation(http://arxiv.org/abs/2312.03795)
• Summary:

  Text-to-3D model adaptations have advanced static 3D model quality, but sequential 3D model generation, particularly for animatable objects with large motions, is still scarce. Our work proposes AnimatableDreamer, a text-to-4D generation framework capable of generating diverse categories of non-rigid objects while adhering to the object motions extracted from a monocular video. At its core, AnimatableDreamer is equipped with our novel optimization design dubbed Canonical Score Distillation (CSD), which simplifies the generation dimension from 4D to 3D by denoising over different frames in the time-varying camera spaces while conducting the distillation process in a unique canonical space shared per video. Concretely, CSD ensures that score gradients back-propagate to the canonical space through differentiable warping, hence guaranteeing the time-consistent generation and maintaining morphological plausibility across different poses. By lifting the 3D generator to 4D with warping functions, AnimatableDreamer offers a novel perspective on non-rigid 3D model generation and reconstruction. Besides, with inductive knowledge from a multi-view consistent diffusion model, CSD regularizes reconstruction from novel views, thus cyclically enhancing the generation process. Extensive experiments demonstrate the capability of our method in generating high-flexibility text-guided 3D models from the monocular video, while also showing improved reconstruction performance over typical non-rigid reconstruction methods. Project page https://AnimatableDreamer.github.io.

Title: AVID: Any-Length Video Inpainting with Diffusion Model. (arXiv:2312.03816v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03816
• Code URL: null
• Copy Paste: [[2312.03816]] AVID: Any-Length Video Inpainting with Diffusion Model(http://arxiv.org/abs/2312.03816)
• Summary:

  Recent advances in diffusion models have successfully enabled text-guided image inpainting. While it seems straightforward to extend such editing capability into video domain, there has been fewer works regarding text-guided video inpainting. Given a video, a masked region at its initial frame, and an editing prompt, it requires a model to do infilling at each frame following the editing guidance while keeping the out-of-mask region intact. There are three main challenges in text-guided video inpainting: ($i$) temporal consistency of the edited video, ($ii$) supporting different inpainting types at different structural fidelity level, and ($iii$) dealing with variable video length. To address these challenges, we introduce Any-Length Video Inpainting with Diffusion Model, dubbed as AVID. At its core, our model is equipped with effective motion modules and adjustable structure guidance, for fixed-length video inpainting. Building on top of that, we propose a novel Temporal MultiDiffusion sampling pipeline with an middle-frame attention guidance mechanism, facilitating the generation of videos with any desired duration. Our comprehensive experiments show our model can robustly deal with various inpainting types at different video duration range, with high quality. More visualization results is made publicly available at https://zhang-zx.github.io/AVID/ .

Title: Diffusion Illusions: Hiding Images in Plain Sight. (arXiv:2312.03817v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03817
• Code URL: null
• Copy Paste: [[2312.03817]] Diffusion Illusions: Hiding Images in Plain Sight(http://arxiv.org/abs/2312.03817)
• Summary:

  We explore the problem of computationally generating special `prime' images that produce optical illusions when physically arranged and viewed in a certain way. First, we propose a formal definition for this problem. Next, we introduce Diffusion Illusions, the first comprehensive pipeline designed to automatically generate a wide range of these illusions. Specifically, we both adapt the existing `score distillation loss' and propose a new `dream target loss' to optimize a group of differentially parametrized prime images, using a frozen text-to-image diffusion model. We study three types of illusions, each where the prime images are arranged in different ways and optimized using the aforementioned losses such that images derived from them align with user-chosen text prompts or images. We conduct comprehensive experiments on these illusions and verify the effectiveness of our proposed method qualitatively and quantitatively. Additionally, we showcase the successful physical fabrication of our illusions -- as they are all designed to work in the real world. Our code and examples are publicly available at our interactive project website: https://diffusionillusions.com

Title: LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning. (arXiv:2312.03849v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03849
• Code URL: null
• Copy Paste: [[2312.03849]] LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning(http://arxiv.org/abs/2312.03849)
• Summary:

  Generating instructional images of human daily actions from an egocentric viewpoint serves a key step towards efficient skill transfer. In this paper, we introduce a novel problem -- egocentric action frame generation. The goal is to synthesize the action frame conditioning on the user prompt question and an input egocentric image that captures user's environment. Notably, existing egocentric datasets lack the detailed annotations that describe the execution of actions. Additionally, the diffusion-based image manipulation models fail to control the state change of an action within the corresponding egocentric image pixel space. To this end, we finetune a visual large language model (VLLM) via visual instruction tuning for curating the enriched action descriptions to address our proposed problem. Moreover, we propose to Learn EGOcentric (LEGO) action frame generation using image and text embeddings from VLLM as additional conditioning. We validate our proposed model on two egocentric datasets -- Ego4D and Epic-Kitchens. Our experiments show prominent improvement over prior image manipulation models in both quantitative and qualitative evaluation. We also conduct detailed ablation studies and analysis to provide insights on our method.

Title: Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion. (arXiv:2312.03869v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03869
• Code URL: null
• Copy Paste: [[2312.03869]] Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion(http://arxiv.org/abs/2312.03869)
• Summary:

  This paper presents a novel approach to inpainting 3D regions of a scene, given masked multi-view images, by distilling a 2D diffusion model into a learned 3D scene representation (e.g. a NeRF). Unlike 3D generative methods that explicitly condition the diffusion model on camera pose or multi-view information, our diffusion model is conditioned only on a single masked 2D image. Nevertheless, we show that this 2D diffusion model can still serve as a generative prior in a 3D multi-view reconstruction problem where we optimize a NeRF using a combination of score distillation sampling and NeRF reconstruction losses. Predicted depth is used as additional supervision to encourage accurate geometry. We compare our approach to 3D inpainting methods that focus on object removal. Because our method can generate content to fill any 3D masked region, we additionally demonstrate 3D object completion, 3D object replacement, and 3D scene completion.

Title: Controllable Human-Object Interaction Synthesis. (arXiv:2312.03913v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03913
• Code URL: null
• Copy Paste: [[2312.03913]] Controllable Human-Object Interaction Synthesis(http://arxiv.org/abs/2312.03913)
• Summary:

  Synthesizing semantic-aware, long-horizon, human-object interaction is critical to simulate realistic human behaviors. In this work, we address the challenging problem of generating synchronized object motion and human motion guided by language descriptions in 3D scenes. We propose Controllable Human-Object Interaction Synthesis (CHOIS), an approach that generates object motion and human motion simultaneously using a conditional diffusion model given a language description, initial object and human states, and sparse object waypoints. While language descriptions inform style and intent, waypoints ground the motion in the scene and can be effectively extracted using high-level planning methods. Naively applying a diffusion model fails to predict object motion aligned with the input waypoints and cannot ensure the realism of interactions that require precise hand-object contact and appropriate contact grounded by the floor. To overcome these problems, we introduce an object geometry loss as additional supervision to improve the matching between generated object motion and input object waypoints. In addition, we design guidance terms to enforce contact constraints during the sampling process of the trained diffusion model.

Title: Adapting HouseDiffusion for conditional Floor Plan generation on Modified Swiss Dwellings dataset. (arXiv:2312.03938v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03938
• Code URL: null
• Copy Paste: [[2312.03938]] Adapting HouseDiffusion for conditional Floor Plan generation on Modified Swiss Dwellings dataset(http://arxiv.org/abs/2312.03938)
• Summary:

  Automated floor plan generation has recently gained momentum with several methods that have been proposed. The CVAAD Floor Plan Auto-Completion workshop challenge introduced MSD, a new dataset that includes existing structural walls of the building as an additional input constraint. This technical report presents an approach for extending a recent work, HouseDiffusion (arXiv:2211.13287 [cs.CV]), to the MSD dataset. The adaption involves modifying the model's transformer layers to condition on a set of wall lines. The report introduces a pre-processing pipeline to extract wall lines from the binary mask of the building structure provided as input. Additionally, it was found that a data processing procedure that simplifies all room polygons to rectangles leads to better performance. This indicates that future work should explore better representations of variable-length polygons in diffusion models. The code will be made available at a later date.

Title: Style Transfer to Calvin and Hobbes comics using Stable Diffusion. (arXiv:2312.03993v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03993
• Code URL: null
• Copy Paste: [[2312.03993]] Style Transfer to Calvin and Hobbes comics using Stable Diffusion(http://arxiv.org/abs/2312.03993)
• Summary:

  This project report summarizes our journey to perform stable diffusion fine-tuning on a dataset containing Calvin and Hobbes comics. The purpose is to convert any given input image into the comic style of Calvin and Hobbes, essentially performing style transfer. We train stable-diffusion-v1.5 using Low Rank Adaptation (LoRA) to efficiently speed up the fine-tuning process. The diffusion itself is handled by a Variational Autoencoder (VAE), which is a U-net. Our results were visually appealing for the amount of training time and the quality of input data that went into training.

Title: Stable diffusion for Data Augmentation in COCO and Weed Datasets. (arXiv:2312.03996v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03996
• Code URL: null
• Copy Paste: [[2312.03996]] Stable diffusion for Data Augmentation in COCO and Weed Datasets(http://arxiv.org/abs/2312.03996)
• Summary:

  Generative models have increasingly impacted relative tasks ranging from image revision and object detection in computer vision to interior design and idea illustration in more general fields. Stable diffusion is an outstanding model series that paves the way for producing high-resolution images with thorough details from text prompts or reference images. It will be an interesting topic about how to leverage the capability of stable diffusion to elevate the image variations of certain categories (e.g., vehicles, humans, and daily objects); particularly, it has the potential to gain improvements for small datasets with image-sparse categories. This study utilized seven categories in the popular COCO dataset and three widespread weed species in Michigan to evaluate the efficiency of a recent version of stable diffusion. In detail, Stable diffusion was used to generate synthetic images belonging to these classes; then, YOLOv8 models were trained based on these synthetic images, whose performance was compared to the models trained on original images. In addition, several techniques (e.g., Image-to-image translation, Dreambooth, ControlNet) of Stable diffusion were leveraged for image generation with different focuses. In spite of the overall results being disappointing, promising results have been achieved in some classes, illustrating the potential of stable diffusion models to improve the performance of detection models, which represent more helpful information being conveyed into the models by the generated images. This seminal study may expedite the adaption of stable diffusion models to classification and detection tasks in different fields.

Title: KOALA: Self-Attention Matters in Knowledge Distillation of Latent Diffusion Models for Memory-Efficient and Fast Image Synthesis. (arXiv:2312.04005v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04005
• Code URL: null
• Copy Paste: [[2312.04005]] KOALA: Self-Attention Matters in Knowledge Distillation of Latent Diffusion Models for Memory-Efficient and Fast Image Synthesis(http://arxiv.org/abs/2312.04005)
• Summary:

  Stable diffusion is the mainstay of the text-to-image (T2I) synthesis in the community due to its generation performance and open-source nature. Recently, Stable Diffusion XL (SDXL), the successor of stable diffusion, has received a lot of attention due to its significant performance improvements with a higher resolution of 1024x1024 and a larger model. However, its increased computation cost and model size require higher-end hardware(e.g., bigger VRAM GPU) for end-users, incurring higher costs of operation. To address this problem, in this work, we propose an efficient latent diffusion model for text-to-image synthesis obtained by distilling the knowledge of SDXL. To this end, we first perform an in-depth analysis of the denoising U-Net in SDXL, which is the main bottleneck of the model, and then design a more efficient U-Net based on the analysis. Secondly, we explore how to effectively distill the generation capability of SDXL into an efficient U-Net and eventually identify four essential factors, the core of which is that self-attention is the most important part. With our efficient U-Net and self-attention-based knowledge distillation strategy, we build our efficient T2I models, called KOALA-1B & -700M, while reducing the model size up to 54% and 69% of the original SDXL model. In particular, the KOALA-700M is more than twice as fast as SDXL while still retaining a decent generation quality. We hope that due to its balanced speed-performance tradeoff, our KOALA models can serve as a cost-effective alternative to SDXL in resource-constrained environments.

Title: DiffusionPhase: Motion Diffusion in Frequency Domain. (arXiv:2312.04036v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04036
• Code URL: null
• Copy Paste: [[2312.04036]] DiffusionPhase: Motion Diffusion in Frequency Domain(http://arxiv.org/abs/2312.04036)
• Summary:

  In this study, we introduce a learning-based method for generating high-quality human motion sequences from text descriptions (e.g., ``A person walks forward"). Existing techniques struggle with motion diversity and smooth transitions in generating arbitrary-length motion sequences, due to limited text-to-motion datasets and the pose representations used that often lack expressiveness or compactness. To address these issues, we propose the first method for text-conditioned human motion generation in the frequency domain of motions. We develop a network encoder that converts the motion space into a compact yet expressive parameterized phase space with high-frequency details encoded, capturing the local periodicity of motions in time and space with high accuracy. We also introduce a conditional diffusion model for predicting periodic motion parameters based on text descriptions and a start pose, efficiently achieving smooth transitions between motion sequences associated with different text descriptions. Experiments demonstrate that our approach outperforms current methods in generating a broader variety of high-quality motions, and synthesizing long sequences with natural transitions.

Title: MTVG : Multi-text Video Generation with Text-to-Video Models. (arXiv:2312.04086v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04086
• Code URL: null
• Copy Paste: [[2312.04086]] MTVG : Multi-text Video Generation with Text-to-Video Models(http://arxiv.org/abs/2312.04086)
• Summary:

  Recently, video generation has attracted massive attention and yielded noticeable outcomes. Concerning the characteristics of video, multi-text conditioning incorporating sequential events is necessary for next-step video generation. In this work, we propose a novel multi-text video generation~(MTVG) by directly utilizing a pre-trained diffusion-based text-to-video~(T2V) generation model without additional fine-tuning. To generate consecutive video segments, visual consistency generated by distinct prompts is necessary with diverse variations, such as motion and content-related transitions. Our proposed MTVG includes Dynamic Noise and Last Frame Aware Inversion which reinitialize the noise latent to preserve visual coherence between videos of different prompts and prevent repetitive motion or contents. Furthermore, we present Structure Guiding Sampling to maintain the global appearance across the frames in a single video clip, where we leverage iterative latent updates across the preceding frame. Additionally, our Prompt Generator allows for arbitrary format of text conditions consisting of diverse events. As a result, our extensive experiments, including diverse transitions of descriptions, demonstrate that our proposed methods show superior generated outputs in terms of semantically coherent and temporally seamless video.Video examples are available in our project page: https://kuai-lab.github.io/mtvg-page.

Title: Diffusing Colors: Image Colorization with Text Guided Diffusion. (arXiv:2312.04145v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04145
• Code URL: null
• Copy Paste: [[2312.04145]] Diffusing Colors: Image Colorization with Text Guided Diffusion(http://arxiv.org/abs/2312.04145)
• Summary:

  The colorization of grayscale images is a complex and subjective task with significant challenges. Despite recent progress in employing large-scale datasets with deep neural networks, difficulties with controllability and visual quality persist. To tackle these issues, we present a novel image colorization framework that utilizes image diffusion techniques with granular text prompts. This integration not only produces colorization outputs that are semantically appropriate but also greatly improves the level of control users have over the colorization process. Our method provides a balance between automation and control, outperforming existing techniques in terms of visual quality and semantic coherence. We leverage a pretrained generative Diffusion Model, and show that we can finetune it for the colorization task without losing its generative power or attention to text prompts. Moreover, we present a novel CLIP-based ranking model that evaluates color vividness, enabling automatic selection of the most suitable level of vividness based on the specific scene semantics. Our approach holds potential particularly for color enhancement and historical image colorization.

Title: Detecting and Restoring Non-Standard Hands in Stable Diffusion Generated Images. (arXiv:2312.04236v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04236
• Code URL: null
• Copy Paste: [[2312.04236]] Detecting and Restoring Non-Standard Hands in Stable Diffusion Generated Images(http://arxiv.org/abs/2312.04236)
• Summary:

  We introduce a pipeline to address anatomical inaccuracies in Stable Diffusion generated hand images. The initial step involves constructing a specialized dataset, focusing on hand anomalies, to train our models effectively. A finetuned detection model is pivotal for precise identification of these anomalies, ensuring targeted correction. Body pose estimation aids in understanding hand orientation and positioning, crucial for accurate anomaly correction. The integration of ControlNet and InstructPix2Pix facilitates sophisticated inpainting and pixel-level transformation, respectively. This dual approach allows for high-fidelity image adjustments. This comprehensive approach ensures the generation of images with anatomically accurate hands, closely resembling real-world appearances. Our experimental results demonstrate the pipeline's efficacy in enhancing hand image realism in Stable Diffusion outputs. We provide an online demo at https://fixhand.yiqun.io

Title: iDesigner: A High-Resolution and Complex-Prompt Following Text-to-Image Diffusion Model for Interior Design. (arXiv:2312.04326v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04326
• Code URL: null
• Copy Paste: [[2312.04326]] iDesigner: A High-Resolution and Complex-Prompt Following Text-to-Image Diffusion Model for Interior Design(http://arxiv.org/abs/2312.04326)
• Summary:

  With the open-sourcing of text-to-image models (T2I) such as stable diffusion (SD) and stable diffusion XL (SD-XL), there is an influx of models fine-tuned in specific domains based on the open-source SD model, such as in anime, character portraits, etc. However, there are few specialized models in certain domains, such as interior design, which is attributed to the complex textual descriptions and detailed visual elements inherent in design, alongside the necessity for adaptable resolution. Therefore, text-to-image models for interior design are required to have outstanding prompt-following capabilities, as well as iterative collaboration with design professionals to achieve the desired outcome. In this paper, we collect and optimize text-image data in the design field and continue training in both English and Chinese on the basis of the open-source CLIP model. We also proposed a fine-tuning strategy with curriculum learning and reinforcement learning from CLIP feedback to enhance the prompt-following capabilities of our approach so as to improve the quality of image generation. The experimental results on the collected dataset demonstrate the effectiveness of the proposed approach, which achieves impressive results and outperforms strong baselines.

Title: Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models. (arXiv:2312.04410v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04410
• Code URL: https://github.com/shi-labs/smooth-diffusion
• Copy Paste: [[2312.04410]] Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models(http://arxiv.org/abs/2312.04410)
• Summary:

  Recently, diffusion models have made remarkable progress in text-to-image (T2I) generation, synthesizing images with high fidelity and diverse contents. Despite this advancement, latent space smoothness within diffusion models remains largely unexplored. Smooth latent spaces ensure that a perturbation on an input latent corresponds to a steady change in the output image. This property proves beneficial in downstream tasks, including image interpolation, inversion, and editing. In this work, we expose the non-smoothness of diffusion latent spaces by observing noticeable visual fluctuations resulting from minor latent variations. To tackle this issue, we propose Smooth Diffusion, a new category of diffusion models that can be simultaneously high-performing and smooth. Specifically, we introduce Step-wise Variation Regularization to enforce the proportion between the variations of an arbitrary input latent and that of the output image is a constant at any diffusion training step. In addition, we devise an interpolation standard deviation (ISTD) metric to effectively assess the latent space smoothness of a diffusion model. Extensive quantitative and qualitative experiments demonstrate that Smooth Diffusion stands out as a more desirable solution not only in T2I generation but also across various downstream tasks. Smooth Diffusion is implemented as a plug-and-play Smooth-LoRA to work with various community models. Code is available at https://github.com/SHI-Labs/Smooth-Diffusion.

Title: Cascade-Zero123: One Image to Highly Consistent 3D with Self-Prompted Nearby Views. (arXiv:2312.04424v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04424
• Code URL: null
• Copy Paste: [[2312.04424]] Cascade-Zero123: One Image to Highly Consistent 3D with Self-Prompted Nearby Views(http://arxiv.org/abs/2312.04424)
• Summary:

  Synthesizing multi-view 3D from one single image is a significant and challenging task. For this goal, Zero-1-to-3 methods aim to extend a 2D latent diffusion model to the 3D scope. These approaches generate the target-view image with a single-view source image and the camera pose as condition information. However, the one-to-one manner adopted in Zero-1-to-3 incurs challenges for building geometric and visual consistency across views, especially for complex objects. We propose a cascade generation framework constructed with two Zero-1-to-3 models, named Cascade-Zero123, to tackle this issue, which progressively extracts 3D information from the source image. Specifically, a self-prompting mechanism is designed to generate several nearby views at first. These views are then fed into the second-stage model along with the source image as generation conditions. With self-prompted multiple views as the supplementary information, our Cascade-Zero123 generates more highly consistent novel-view images than Zero-1-to-3. The promotion is significant for various complex and challenging scenes, involving insects, humans, transparent objects, and stacked multiple objects etc. The project page is at https://cascadezero123.github.io/.

Title: Approximate Caching for Efficiently Serving Diffusion Models. (arXiv:2312.04429v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04429
• Code URL: null
• Copy Paste: [[2312.04429]] Approximate Caching for Efficiently Serving Diffusion Models(http://arxiv.org/abs/2312.04429)
• Summary:

  Text-to-image generation using diffusion models has seen explosive popularity owing to their ability in producing high quality images adhering to text prompts. However, production-grade diffusion model serving is a resource intensive task that not only require high-end GPUs which are expensive but also incurs considerable latency. In this paper, we introduce a technique called approximate-caching that can reduce such iterative denoising steps for an image generation based on a prompt by reusing intermediate noise states created during a prior image generation for similar prompts. Based on this idea, we present an end to end text-to-image system, Nirvana, that uses the approximate-caching with a novel cache management-policy Least Computationally Beneficial and Frequently Used (LCBFU) to provide % GPU compute savings, 19.8% end-to-end latency reduction and 19% dollar savings, on average, on two real production workloads. We further present an extensive characterization of real production text-to-image prompts from the perspective of caching, popularity and reuse of intermediate states in a large production environment.

Title: DreamVideo: Composing Your Dream Videos with Customized Subject and Motion. (arXiv:2312.04433v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04433
• Code URL: null
• Copy Paste: [[2312.04433]] DreamVideo: Composing Your Dream Videos with Customized Subject and Motion(http://arxiv.org/abs/2312.04433)
• Summary:

  Customized generation using diffusion models has made impressive progress in image generation, but remains unsatisfactory in the challenging video generation task, as it requires the controllability of both subjects and motions. To that end, we present DreamVideo, a novel approach to generating personalized videos from a few static images of the desired subject and a few videos of target motion. DreamVideo decouples this task into two stages, subject learning and motion learning, by leveraging a pre-trained video diffusion model. The subject learning aims to accurately capture the fine appearance of the subject from provided images, which is achieved by combining textual inversion and fine-tuning of our carefully designed identity adapter. In motion learning, we architect a motion adapter and fine-tune it on the given videos to effectively model the target motion pattern. Combining these two lightweight and efficient adapters allows for flexible customization of any subject with any motion. Extensive experimental results demonstrate the superior performance of our DreamVideo over the state-of-the-art methods for customized video generation. Our project page is at https://dreamvideo-t2v.github.io.

noise learning

data-free

transformer

Title: Lite-Mind: Towards Efficient and Versatile Brain Representation Network. (arXiv:2312.03781v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03781
• Code URL: null
• Copy Paste: [[2312.03781]] Lite-Mind: Towards Efficient and Versatile Brain Representation Network(http://arxiv.org/abs/2312.03781)
• Summary:

  Research in decoding visual information from the brain, particularly through the non-invasive fMRI method, is rapidly progressing. The challenge arises from the limited data availability and the low signal-to-noise ratio of fMRI signals, leading to a low-precision task of fMRI-to-image retrieval. State-of-the-art MindEye remarkably improves fMRI-to-image retrieval performance by leveraging a deep MLP with a high parameter count orders of magnitude, i.e., a 996M MLP Backbone per subject, to align fMRI embeddings to the final hidden layer of CLIP's vision transformer. However, significant individual variations exist among subjects, even within identical experimental setups, mandating the training of subject-specific models. The substantial parameters pose significant challenges in deploying fMRI decoding on practical devices, especially with the necessitating of specific models for each subject. To this end, we propose Lite-Mind, a lightweight, efficient, and versatile brain representation network based on discrete Fourier transform, that efficiently aligns fMRI voxels to fine-grained information of CLIP. Our experiments demonstrate that Lite-Mind achieves an impressive 94.3% fMRI-to-image retrieval accuracy on the NSD dataset for Subject 1, with 98.7% fewer parameters than MindEye. Lite-Mind is also proven to be able to be migrated to smaller brain datasets and establishes a new state-of-the-art for zero-shot classification on the GOD dataset. The code is available at https://github.com/gongzix/Lite-Mind.

Title: Memory-Efficient Optical Flow via Radius-Distribution Orthogonal Cost Volume. (arXiv:2312.03790v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03790
• Code URL: null
• Copy Paste: [[2312.03790]] Memory-Efficient Optical Flow via Radius-Distribution Orthogonal Cost Volume(http://arxiv.org/abs/2312.03790)
• Summary:

  The full 4D cost volume in Recurrent All-Pairs Field Transforms (RAFT) or global matching by Transformer achieves impressive performance for optical flow estimation. However, their memory consumption increases quadratically with input resolution, rendering them impractical for high-resolution images. In this paper, we present MeFlow, a novel memory-efficient method for high-resolution optical flow estimation. The key of MeFlow is a recurrent local orthogonal cost volume representation, which decomposes the 2D search space dynamically into two 1D orthogonal spaces, enabling our method to scale effectively to very high-resolution inputs. To preserve essential information in the orthogonal space, we utilize self attention to propagate feature information from the 2D space to the orthogonal space. We further propose a radius-distribution multi-scale lookup strategy to model the correspondences of large displacements at a negligible cost. We verify the efficiency and effectiveness of our method on the challenging Sintel and KITTI benchmarks, and real-world 4K ($2160\!\times\!3840$) images. Our method achieves competitive performance on both Sintel and KITTI benchmarks, while maintaining the highest memory efficiency on high-resolution inputs.

Title: A Layer-Wise Tokens-to-Token Transformer Network for Improved Historical Document Image Enhancement. (arXiv:2312.03946v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03946
• Code URL: null
• Copy Paste: [[2312.03946]] A Layer-Wise Tokens-to-Token Transformer Network for Improved Historical Document Image Enhancement(http://arxiv.org/abs/2312.03946)
• Summary:

  Document image enhancement is a fundamental and important stage for attaining the best performance in any document analysis assignment because there are many degradation situations that could harm document images, making it more difficult to recognize and analyze them. In this paper, we propose \textbf{T2T-BinFormer} which is a novel document binarization encoder-decoder architecture based on a Tokens-to-token vision transformer. Each image is divided into a set of tokens with a defined length using the ViT model, which is then applied several times to model the global relationship between the tokens. However, the conventional tokenization of input data does not adequately reflect the crucial local structure between adjacent pixels of the input image, which results in low efficiency. Instead of using a simple ViT and hard splitting of images for the document image enhancement task, we employed a progressive tokenization technique to capture this local information from an image to achieve more effective results. Experiments on various DIBCO and H-DIBCO benchmarks demonstrate that the proposed model outperforms the existing CNN and ViT-based state-of-the-art methods. In this research, the primary area of examination is the application of the proposed architecture to the task of document binarization. The source code will be made available at https://github.com/RisabBiswas/T2T-BinFormer.

Title: Intelligent Anomaly Detection for Lane Rendering Using Transformer with Self-Supervised Pre-Training and Customized Fine-Tuning. (arXiv:2312.04398v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04398
• Code URL: null
• Copy Paste: [[2312.04398]] Intelligent Anomaly Detection for Lane Rendering Using Transformer with Self-Supervised Pre-Training and Customized Fine-Tuning(http://arxiv.org/abs/2312.04398)
• Summary:

  The burgeoning navigation services using digital maps provide great convenience to drivers. Nevertheless, the presence of anomalies in lane rendering map images occasionally introduces potential hazards, as such anomalies can be misleading to human drivers and consequently contribute to unsafe driving conditions. In response to this concern and to accurately and effectively detect the anomalies, this paper transforms lane rendering image anomaly detection into a classification problem and proposes a four-phase pipeline consisting of data pre-processing, self-supervised pre-training with the masked image modeling (MiM) method, customized fine-tuning using cross-entropy based loss with label smoothing, and post-processing to tackle it leveraging state-of-the-art deep learning techniques, especially those involving Transformer models. Various experiments verify the effectiveness of the proposed pipeline. Results indicate that the proposed pipeline exhibits superior performance in lane rendering image anomaly detection, and notably, the self-supervised pre-training with MiM can greatly enhance the detection accuracy while significantly reducing the total training time. For instance, employing the Swin Transformer with Uniform Masking as self-supervised pretraining (Swin-Trans-UM) yielded a heightened accuracy at 94.77% and an improved Area Under The Curve (AUC) score of 0.9743 compared with the pure Swin Transformer without pre-training (Swin-Trans) with an accuracy of 94.01% and an AUC of 0.9498. The fine-tuning epochs were dramatically reduced to 41 from the original 280. In conclusion, the proposed pipeline, with its incorporation of self-supervised pre-training using MiM and other advanced deep learning techniques, emerges as a robust solution for enhancing the accuracy and efficiency of lane rendering image anomaly detection in digital navigation systems.

Title: A Generic NLI approach for Classification of Sentiment Associated with Therapies. (arXiv:2312.03737v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03737
• Code URL: null
• Copy Paste: [[2312.03737]] A Generic NLI approach for Classification of Sentiment Associated with Therapies(http://arxiv.org/abs/2312.03737)
• Summary:

  This paper describes our system for addressing SMM4H 2023 Shared Task 2 on "Classification of sentiment associated with therapies (aspect-oriented)". In our work, we adopt an approach based on Natural language inference (NLI) to formulate this task as a sentence pair classification problem, and train transformer models to predict sentiment associated with a therapy on a given text. Our best model achieved 75.22\% F1-score which was 11\% (4\%) more than the mean (median) score of all teams' submissions.

Title: Classifying patient voice in social media data using neural networks: A comparison of AI models on different data sources and therapeutic domains. (arXiv:2312.03747v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03747
• Code URL: null
• Copy Paste: [[2312.03747]] Classifying patient voice in social media data using neural networks: A comparison of AI models on different data sources and therapeutic domains(http://arxiv.org/abs/2312.03747)
• Summary:

  It is essential that healthcare professionals and members of the healthcare community can access and easily understand patient experiences in the real world, so that care standards can be improved and driven towards personalised drug treatment. Social media platforms and message boards are deemed suitable sources of patient experience information, as patients have been observed to discuss and exchange knowledge, look for and provide support online. This paper tests the hypothesis that not all online patient experience information can be treated and collected in the same way, as a result of the inherent differences in the way individuals talk about their journeys, in different therapeutic domains and or data sources.

  We used linguistic analysis to understand and identify similarities between datasets, across patient language, between data sources (Reddit, SocialGist) and therapeutic domains (cardiovascular, oncology, immunology, neurology). We detected common vocabulary used by patients in the same therapeutic domain across data sources, except for immunology patients, who use unique vocabulary between the two data sources, and compared to all other datasets. We combined linguistically similar datasets to train classifiers (CNN, transformer) to accurately identify patient experience posts from social media, a task we refer to as patient voice classification. The cardiovascular and neurology transformer classifiers perform the best in their respective comparisons for the Reddit data source, achieving F1-scores of 0.865 and 1.0 respectively. The overall best performing classifier is the transformer classifier trained on all data collected for this experiment, achieving F1-scores ranging between 0.863 and 0.995 across all therapeutic domain and data source specific test datasets.

Title: Comparative Analysis of Multilingual Text Classification & Identification through Deep Learning and Embedding Visualization. (arXiv:2312.03789v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03789
• Code URL: null
• Copy Paste: [[2312.03789]] Comparative Analysis of Multilingual Text Classification & Identification through Deep Learning and Embedding Visualization(http://arxiv.org/abs/2312.03789)
• Summary:

  This research conducts a comparative study on multilingual text classification methods, utilizing deep learning and embedding visualization. The study employs LangDetect, LangId, FastText, and Sentence Transformer on a dataset encompassing 17 languages. It explores dimensionality's impact on clustering, revealing FastText's clearer clustering in 2D visualization due to its extensive multilingual corpus training. Notably, the FastText multi-layer perceptron model achieved remarkable accuracy, precision, recall, and F1 score, outperforming the Sentence Transformer model. The study underscores the effectiveness of these techniques in multilingual text classification, emphasizing the importance of large multilingual corpora for training embeddings. It lays the groundwork for future research and assists practitioners in developing language detection and classification systems. Additionally, it includes the comparison of multi-layer perceptron, LSTM, and Convolution models for classification.

Title: A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints. (arXiv:2312.03905v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.03905
• Code URL: null
• Copy Paste: [[2312.03905]] A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints(http://arxiv.org/abs/2312.03905)
• Summary:

  Neuro-symbolic AI bridges the gap between purely symbolic and neural approaches to learning. This often requires maximizing the likelihood of a symbolic constraint w.r.t the neural network's output distribution. Such output distributions are typically assumed to be fully-factorized. This limits the applicability of neuro-symbolic learning to the more expressive autoregressive distributions, e.g., transformers. Under such distributions, computing the likelihood of even simple constraints is #P-hard. Instead of attempting to enforce the constraint on the entire output distribution, we propose to do so on a random, local approximation thereof. More precisely, we optimize the likelihood of the constraint under a pseudolikelihood-based approximation centered around a model sample. Our approximation is factorized, allowing the reuse of solutions to sub-problems, a main tenet for efficiently computing neuro-symbolic losses. Moreover, it is a local, high-fidelity approximation of the likelihood, exhibiting low entropy and KL-divergence around the model sample. We evaluate our approach on Sudoku and shortest-path prediction cast as autoregressive generation, and observe that we greatly improve upon the base model's ability to predict logically-consistent outputs. We also evaluate on the task of detoxifying large language models. Using a simple constraint disallowing a list of toxic words, we are able to steer the model's outputs away from toxic generations, achieving SoTA detoxification compared to previous approaches.

Title: Collaboration or Corporate Capture? Quantifying NLP's Reliance on Industry Artifacts and Contributions. (arXiv:2312.03912v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03912
• Code URL: null
• Copy Paste: [[2312.03912]] Collaboration or Corporate Capture? Quantifying NLP's Reliance on Industry Artifacts and Contributions(http://arxiv.org/abs/2312.03912)
• Summary:

  The advent of transformers, higher computational budgets, and big data has engendered remarkable progress in Natural Language Processing (NLP). Impressive performance of industry pre-trained models has garnered public attention in recent years and made news headlines. That these are industry models is noteworthy. Rarely, if ever, are academic institutes producing exciting new NLP models. Using these models is critical for competing on NLP benchmarks and correspondingly to stay relevant in NLP research. We surveyed 100 papers published at EMNLP 2022 to determine whether this phenomenon constitutes a reliance on industry for NLP publications.

  We find that there is indeed a substantial reliance. Citations of industry artifacts and contributions across categories is at least three times greater than industry publication rates per year. Quantifying this reliance does not settle how we ought to interpret the results. We discuss two possible perspectives in our discussion: 1) Is collaboration with industry still collaboration in the absence of an alternative? Or 2) has free NLP inquiry been captured by the motivations and research direction of private corporations?

Title: Multimodal Misinformation Detection in a South African Social Media Environment. (arXiv:2312.04052v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.04052
• Code URL: null
• Copy Paste: [[2312.04052]] Multimodal Misinformation Detection in a South African Social Media Environment(http://arxiv.org/abs/2312.04052)
• Summary:

  With the constant spread of misinformation on social media networks, a need has arisen to continuously assess the veracity of digital content. This need has inspired numerous research efforts on the development of misinformation detection (MD) models. However, many models do not use all information available to them and existing research contains a lack of relevant datasets to train the models, specifically within the South African social media environment. The aim of this paper is to investigate the transferability of knowledge of a MD model between different contextual environments. This research contributes a multimodal MD model capable of functioning in the South African social media environment, as well as introduces a South African misinformation dataset. The model makes use of multiple sources of information for misinformation detection, namely: textual and visual elements. It uses bidirectional encoder representations from transformers (BERT) as the textual encoder and a residual network (ResNet) as the visual encoder. The model is trained and evaluated on the Fakeddit dataset and a South African misinformation dataset. Results show that using South African samples in the training of the model increases model performance, in a South African contextual environment, and that a multimodal model retains significantly more knowledge than both the textual and visual unimodal models. Our study suggests that the performance of a misinformation detection model is influenced by the cultural nuances of its operating environment and multimodal models assist in the transferability of knowledge between different contextual environments. Therefore, local data should be incorporated into the training process of a misinformation detection model in order to optimize model performance.

Title: nerblackbox: A High-level Library for Named Entity Recognition in Python. (arXiv:2312.04306v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.04306
• Code URL: https://github.com/flxst/nerblackbox
• Copy Paste: [[2312.04306]] nerblackbox: A High-level Library for Named Entity Recognition in Python(http://arxiv.org/abs/2312.04306)
• Summary:

  We present nerblackbox, a python library to facilitate the use of state-of-the-art transformer-based models for named entity recognition. It provides simple-to-use yet powerful methods to access data and models from a wide range of sources, for fully automated model training and evaluation as well as versatile model inference. While many technical challenges are solved and hidden from the user by default, nerblackbox also offers fine-grained control and a rich set of customizable features. It is thus targeted both at application-oriented developers as well as machine learning experts and researchers.

Title: Generalization to New Sequential Decision Making Tasks with In-Context Learning. (arXiv:2312.03801v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.03801
• Code URL: null
• Copy Paste: [[2312.03801]] Generalization to New Sequential Decision Making Tasks with In-Context Learning(http://arxiv.org/abs/2312.03801)
• Summary:

  Training autonomous agents that can learn new tasks from only a handful of demonstrations is a long-standing problem in machine learning. Recently, transformers have been shown to learn new language or vision tasks without any weight updates from only a few examples, also referred to as in-context learning. However, the sequential decision making setting poses additional challenges having a lower tolerance for errors since the environment's stochasticity or the agent's actions can lead to unseen, and sometimes unrecoverable, states. In this paper, we use an illustrative example to show that naively applying transformers to sequential decision making problems does not enable in-context learning of new tasks. We then demonstrate how training on sequences of trajectories with certain distributional properties leads to in-context learning of new sequential decision making tasks. We investigate different design choices and find that larger model and dataset sizes, as well as more task diversity, environment stochasticity, and trajectory burstiness, all result in better in-context learning of new out-of-distribution tasks. By training on large diverse offline datasets, our model is able to learn new MiniHack and Procgen tasks without any weight updates from just a handful of demonstrations.

Title: A Transformer Model for Symbolic Regression towards Scientific Discovery. (arXiv:2312.04070v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04070
• Code URL: https://github.com/omron-sinicx/transformer4sr
• Copy Paste: [[2312.04070]] A Transformer Model for Symbolic Regression towards Scientific Discovery(http://arxiv.org/abs/2312.04070)
• Summary:

  Symbolic Regression (SR) searches for mathematical expressions which best describe numerical datasets. This allows to circumvent interpretation issues inherent to artificial neural networks, but SR algorithms are often computationally expensive. This work proposes a new Transformer model aiming at Symbolic Regression particularly focused on its application for Scientific Discovery. We propose three encoder architectures with increasing flexibility but at the cost of column-permutation equivariance violation. Training results indicate that the most flexible architecture is required to prevent from overfitting. Once trained, we apply our best model to the SRSD datasets (Symbolic Regression for Scientific Discovery datasets) which yields state-of-the-art results using the normalized tree-based edit distance, at no extra computational cost.

Title: Graph Convolutions Enrich the Self-Attention in Transformers!. (arXiv:2312.04234v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04234
• Code URL: null
• Copy Paste: [[2312.04234]] Graph Convolutions Enrich the Self-Attention in Transformers!(http://arxiv.org/abs/2312.04234)
• Summary:

  Transformers, renowned for their self-attention mechanism, have achieved state-of-the-art performance across various tasks in natural language processing, computer vision, time-series modeling, etc. However, one of the challenges with deep Transformer models is the oversmoothing problem, where representations across layers converge to indistinguishable values, leading to significant performance degradation. We interpret the original self-attention as a simple graph filter and redesign it from a graph signal processing (GSP) perspective. We propose graph-filter-based self-attention (GFSA) to learn a general yet effective one, whose complexity, however, is slightly larger than that of the original self-attention mechanism. We demonstrate that GFSA improves the performance of Transformers in various fields, including computer vision, natural language processing, graph pattern classification, speech recognition, and code classification.

generative

Title: XCube ($\mathcal{X}^3$): Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies. (arXiv:2312.03806v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03806
• Code URL: null
• Copy Paste: [[2312.03806]] XCube ($\mathcal{X}^3$): Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies(http://arxiv.org/abs/2312.03806)
• Summary:

  We present $\mathcal{X}^3$ (pronounced XCube), a novel generative model for high-resolution sparse 3D voxel grids with arbitrary attributes. Our model can generate millions of voxels with a finest effective resolution of up to $1024^3$ in a feed-forward fashion without time-consuming test-time optimization. To achieve this, we employ a hierarchical voxel latent diffusion model which generates progressively higher resolution grids in a coarse-to-fine manner using a custom framework built on the highly efficient VDB data structure. Apart from generating high-resolution objects, we demonstrate the effectiveness of XCube on large outdoor scenes at scales of 100m$\times$100m with a voxel size as small as 10cm. We observe clear qualitative and quantitative improvements over past approaches. In addition to unconditional generation, we show that our model can be used to solve a variety of tasks such as user-guided editing, scene completion from a single scan, and text-to-3D. More results and details can be found at https://research.nvidia.com/labs/toronto-ai/xcube/.

Title: Comparing Generative Chatbots Based on Process Requirements. (arXiv:2312.03741v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03741
• Code URL: null
• Copy Paste: [[2312.03741]] Comparing Generative Chatbots Based on Process Requirements(http://arxiv.org/abs/2312.03741)
• Summary:

  Business processes are commonly represented by modelling languages, such as Event-driven Process Chain (EPC), Yet Another Workflow Language (YAWL), and the most popular standard notation for modelling business processes, the Business Process Model and Notation (BPMN). Most recently, chatbots, programs that allow users to interact with a machine using natural language, have been increasingly used for business process execution support. A recent category of chatbots worth mentioning is generative-based chatbots, powered by Large Language Models (LLMs) such as OpenAI's Generative Pre-Trained Transformer (GPT) model and Google's Pathways Language Model (PaLM), which are trained on billions of parameters and support conversational intelligence. However, it is not clear whether generative-based chatbots are able to understand and meet the requirements of constructs such as those provided by BPMN for process execution support. This paper presents a case study to compare the performance of prominent generative models, GPT and PaLM, in the context of process execution support. The research sheds light into the challenging problem of using conversational approaches supported by generative chatbots as a means to understand process-aware modelling notations and support users to execute their tasks.

Title: Beyond Surface: Probing LLaMA Across Scales and Layers. (arXiv:2312.04333v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.04333
• Code URL: https://github.com/nuochenpku/llama_analysis
• Copy Paste: [[2312.04333]] Beyond Surface: Probing LLaMA Across Scales and Layers(http://arxiv.org/abs/2312.04333)
• Summary:

  This paper presents an in-depth analysis of Large Language Models (LLMs), focusing on LLaMA, a prominent open-source foundational model in natural language processing. Instead of assessing LLaMA through its generative output, we design multiple-choice tasks to probe its intrinsic understanding in high-order tasks such as reasoning and computation. We examine the model horizontally, comparing different sizes, and vertically, assessing different layers. We unveil several key and uncommon findings based on the designed probing tasks: (1) Horizontally, enlarging model sizes almost could not automatically impart additional knowledge or computational prowess. Instead, it can enhance reasoning abilities, especially in math problem solving, and helps reduce hallucinations, but only beyond certain size thresholds; (2) In vertical analysis, the lower layers of LLaMA lack substantial arithmetic and factual knowledge, showcasing logical thinking, multilingual and recognitive abilities, with top layers housing most computational power and real-world knowledge.

Title: Improving Gradient-guided Nested Sampling for Posterior Inference. (arXiv:2312.03911v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.03911
• Code URL: https://github.com/pablo-lemos/ggns
• Copy Paste: [[2312.03911]] Improving Gradient-guided Nested Sampling for Posterior Inference(http://arxiv.org/abs/2312.03911)
• Summary:

  We present a performant, general-purpose gradient-guided nested sampling algorithm, ${\tt GGNS}$, combining the state of the art in differentiable programming, Hamiltonian slice sampling, clustering, mode separation, dynamic nested sampling, and parallelization. This unique combination allows ${\tt GGNS}$ to scale well with dimensionality and perform competitively on a variety of synthetic and real-world problems. We also show the potential of combining nested sampling with generative flow networks to obtain large amounts of high-quality samples from the posterior distribution. This combination leads to faster mode discovery and more accurate estimates of the partition function.

Title: Mixture of Dynamical Variational Autoencoders for Multi-Source Trajectory Modeling and Separation. (arXiv:2312.04167v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04167
• Code URL: null
• Copy Paste: [[2312.04167]] Mixture of Dynamical Variational Autoencoders for Multi-Source Trajectory Modeling and Separation(http://arxiv.org/abs/2312.04167)
• Summary:

  In this paper, we propose a latent-variable generative model called mixture of dynamical variational autoencoders (MixDVAE) to model the dynamics of a system composed of multiple moving sources. A DVAE model is pre-trained on a single-source dataset to capture the source dynamics. Then, multiple instances of the pre-trained DVAE model are integrated into a multi-source mixture model with a discrete observation-to-source assignment latent variable. The posterior distributions of both the discrete observation-to-source assignment variable and the continuous DVAE variables representing the sources content/position are estimated using a variational expectation-maximization algorithm, leading to multi-source trajectories estimation. We illustrate the versatility of the proposed MixDVAE model on two tasks: a computer vision task, namely multi-object tracking, and an audio processing task, namely single-channel audio source separation. Experimental results show that the proposed method works well on these two tasks, and outperforms several baseline methods.

Title: Learning to sample in Cartesian MRI. (arXiv:2312.04327v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.04327
• Code URL: null
• Copy Paste: [[2312.04327]] Learning to sample in Cartesian MRI(http://arxiv.org/abs/2312.04327)
• Summary:

  Despite its exceptional soft tissue contrast, Magnetic Resonance Imaging (MRI) faces the challenge of long scanning times compared to other modalities like X-ray radiography. Shortening scanning times is crucial in clinical settings, as it increases patient comfort, decreases examination costs and improves throughput. Recent advances in compressed sensing (CS) and deep learning allow accelerated MRI acquisition by reconstructing high-quality images from undersampled data. While reconstruction algorithms have received most of the focus, designing acquisition trajectories to optimize reconstruction quality remains an open question. This thesis explores two approaches to address this gap in the context of Cartesian MRI. First, we propose two algorithms, lazy LBCS and stochastic LBCS, that significantly improve upon G\"ozc\"u et al.'s greedy learning-based CS (LBCS) approach. These algorithms scale to large, clinically relevant scenarios like multi-coil 3D MR and dynamic MRI, previously inaccessible to LBCS. Additionally, we demonstrate that generative adversarial networks (GANs) can serve as a natural criterion for adaptive sampling by leveraging variance in the measurement domain to guide acquisition. Second, we delve into the underlying structures or assumptions that enable mask design algorithms to perform well in practice. Our experiments reveal that state-of-the-art deep reinforcement learning (RL) approaches, while capable of adaptation and long-horizon planning, offer only marginal improvements over stochastic LBCS, which is neither adaptive nor does long-term planning. Altogether, our findings suggest that stochastic LBCS and similar methods represent promising alternatives to deep RL. They shine in particular by their scalability and computational efficiency and could be key in the deployment of optimized acquisition trajectories in Cartesian MRI.

large language model

Title: Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignment. (arXiv:2312.03766v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03766
• Code URL: null
• Copy Paste: [[2312.03766]] Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignment(http://arxiv.org/abs/2312.03766)
• Summary:

  While existing image-text alignment models reach high quality binary assessments, they fall short of pinpointing the exact source of misalignment. In this paper, we present a method to provide detailed textual and visual explanation of detected misalignments between text-image pairs. We leverage large language models and visual grounding models to automatically construct a training set that holds plausible misaligned captions for a given image and corresponding textual explanations and visual indicators. We also publish a new human curated test set comprising ground-truth textual and visual misalignment annotations. Empirical results show that fine-tuning vision language models on our training set enables them to articulate misalignments and visually indicate them within images, outperforming strong baselines both on the binary alignment classification and the explanation generation tasks. Our method code and human curated test set are available at: https://mismatch-quest.github.io/

Title: Alpha-CLIP: A CLIP Model Focusing on Wherever You Want. (arXiv:2312.03818v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03818
• Code URL: https://github.com/sunzey/alphaclip
• Copy Paste: [[2312.03818]] Alpha-CLIP: A CLIP Model Focusing on Wherever You Want(http://arxiv.org/abs/2312.03818)
• Summary:

  Contrastive Language-Image Pre-training (CLIP) plays an essential role in extracting valuable content information from images across diverse tasks. It aligns textual and visual modalities to comprehend the entire image, including all the details, even those irrelevant to specific tasks. However, for a finer understanding and controlled editing of images, it becomes crucial to focus on specific regions of interest, which can be indicated as points, masks, or boxes by humans or perception models. To fulfill the requirements, we introduce Alpha-CLIP, an enhanced version of CLIP with an auxiliary alpha channel to suggest attentive regions and fine-tuned with constructed millions of RGBA region-text pairs. Alpha-CLIP not only preserves the visual recognition ability of CLIP but also enables precise control over the emphasis of image contents. It demonstrates effectiveness in various tasks, including but not limited to open-world recognition, multimodal large language models, and conditional 2D / 3D generation. It has a strong potential to serve as a versatile tool for image-related tasks.

Title: Improving Medical Report Generation with Adapter Tuning and Knowledge Enhancement in Vision-Language Foundation Models. (arXiv:2312.03970v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03970
• Code URL: null
• Copy Paste: [[2312.03970]] Improving Medical Report Generation with Adapter Tuning and Knowledge Enhancement in Vision-Language Foundation Models(http://arxiv.org/abs/2312.03970)
• Summary:

  Medical report generation demands automatic creation of coherent and precise descriptions for medical images. However, the scarcity of labelled medical image-report pairs poses formidable challenges in developing large-scale neural networks capable of harnessing the potential of artificial intelligence, exemplified by large language models. This study builds upon the state-of-the-art vision-language pre-training and fine-tuning approach, BLIP-2, to customize general large-scale foundation models. Integrating adapter tuning and a medical knowledge enhancement loss, our model significantly improves accuracy and coherence. Validation on the dataset of ImageCLEFmedical 2023 demonstrates our model's prowess, achieving the best-averaged results against several state-of-the-art methods. Significant improvements in ROUGE and CIDEr underscore our method's efficacy, highlighting promising outcomes for the rapid medical-domain adaptation of the vision-language foundation models in addressing challenges posed by data scarcity.

Title: Large Language Models are Good Prompt Learners for Low-Shot Image Classification. (arXiv:2312.04076v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04076
• Code URL: null
• Copy Paste: [[2312.04076]] Large Language Models are Good Prompt Learners for Low-Shot Image Classification(http://arxiv.org/abs/2312.04076)
• Summary:

  Low-shot image classification, where training images are limited or inaccessible, has benefited from recent progress on pre-trained vision-language (VL) models with strong generalizability, e.g. CLIP. Prompt learning methods built with VL models generate text features from the class names that only have confined class-specific information. Large Language Models (LLMs), with their vast encyclopedic knowledge, emerge as the complement. Thus, in this paper, we discuss the integration of LLMs to enhance pre-trained VL models, specifically on low-shot classification. However, the domain gap between language and vision blocks the direct application of LLMs. Thus, we propose LLaMP, Large Language Models as Prompt learners, that produces adaptive prompts for the CLIP text encoder, establishing it as the connecting bridge. Experiments show that, compared with other state-of-the-art prompt learning methods, LLaMP yields better performance on both zero-shot generalization and few-shot image classification, over a spectrum of 11 datasets.

Title: Text as Image: Learning Transferable Adapter for Multi-Label Classification. (arXiv:2312.04160v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04160
• Code URL: null
• Copy Paste: [[2312.04160]] Text as Image: Learning Transferable Adapter for Multi-Label Classification(http://arxiv.org/abs/2312.04160)
• Summary:

  Pre-trained vision-language models have notably accelerated progress of open-world concept recognition. Their impressive zero-shot ability has recently been transferred to multi-label image classification via prompt tuning, enabling to discover novel labels in an open-vocabulary manner. However, this paradigm suffers from non-trivial training costs, and becomes computationally prohibitive for a large number of candidate labels. To address this issue, we note that vision-language pre-training aligns images and texts in a unified embedding space, making it potential for an adapter network to identify labels in visual modality while be trained in text modality. To enhance such cross-modal transfer ability, a simple yet effective method termed random perturbation is proposed, which enables the adapter to search for potential visual embeddings by perturbing text embeddings with noise during training, resulting in better performance in visual modality. Furthermore, we introduce an effective approach to employ large language models for multi-label instruction-following text generation. In this way, a fully automated pipeline for visual label recognition is developed without relying on any manual data. Extensive experiments on public benchmarks show the superiority of our method in various multi-label classification tasks.

Title: An Evaluation of State-of-the-Art Large Language Models for Sarcasm Detection. (arXiv:2312.03706v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03706
• Code URL: null
• Copy Paste: [[2312.03706]] An Evaluation of State-of-the-Art Large Language Models for Sarcasm Detection(http://arxiv.org/abs/2312.03706)
• Summary:

  Sarcasm, as defined by Merriam-Webster, is the use of words by someone who means the opposite of what he is trying to say. In the field of sentimental analysis of Natural Language Processing, the ability to correctly identify sarcasm is necessary for understanding people's true opinions. Because the use of sarcasm is often context-based, previous research has used language representation models, such as Support Vector Machine (SVM) and Long Short-Term Memory (LSTM), to identify sarcasm with contextual-based information. Recent innovations in NLP have provided more possibilities for detecting sarcasm. In BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Jacob Devlin et al. (2018) introduced a new language representation model and demonstrated higher precision in interpreting contextualized language. As proposed by Hazarika et al. (2018), CASCADE is a context-driven model that produces good results for detecting sarcasm. This study analyzes a Reddit corpus using these two state-of-the-art models and evaluates their performance against baseline models to find the ideal approach to sarcasm detection.

Title: Large Language Models in Law: A Survey. (arXiv:2312.03718v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03718
• Code URL: null
• Copy Paste: [[2312.03718]] Large Language Models in Law: A Survey(http://arxiv.org/abs/2312.03718)
• Summary:

  The advent of artificial intelligence (AI) has significantly impacted the traditional judicial industry. Moreover, recently, with the development of AI-generated content (AIGC), AI and law have found applications in various domains, including image recognition, automatic text generation, and interactive chat. With the rapid emergence and growing popularity of large models, it is evident that AI will drive transformation in the traditional judicial industry. However, the application of legal large language models (LLMs) is still in its nascent stage. Several challenges need to be addressed. In this paper, we aim to provide a comprehensive survey of legal LLMs. We not only conduct an extensive survey of LLMs, but also expose their applications in the judicial system. We first provide an overview of AI technologies in the legal field and showcase the recent research in LLMs. Then, we discuss the practical implementation presented by legal LLMs, such as providing legal advice to users and assisting judges during trials. In addition, we explore the limitations of legal LLMs, including data, algorithms, and judicial practice. Finally, we summarize practical recommendations and propose future development directions to address these challenges.

Title: Negotiating with LLMS: Prompt Hacks, Skill Gaps, and Reasoning Deficits. (arXiv:2312.03720v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03720
• Code URL: null
• Copy Paste: [[2312.03720]] Negotiating with LLMS: Prompt Hacks, Skill Gaps, and Reasoning Deficits(http://arxiv.org/abs/2312.03720)
• Summary:

  Large language models LLMs like ChatGPT have reached the 100 Mio user barrier in record time and might increasingly enter all areas of our life leading to a diverse set of interactions between those Artificial Intelligence models and humans. While many studies have discussed governance and regulations deductively from first-order principles, few studies provide an inductive, data-driven lens based on observing dialogues between humans and LLMs especially when it comes to non-collaborative, competitive situations that have the potential to pose a serious threat to people. In this work, we conduct a user study engaging over 40 individuals across all age groups in price negotiations with an LLM. We explore how people interact with an LLM, investigating differences in negotiation outcomes and strategies. Furthermore, we highlight shortcomings of LLMs with respect to their reasoning capabilities and, in turn, susceptiveness to prompt hacking, which intends to manipulate the LLM to make agreements that are against its instructions or beyond any rationality. We also show that the negotiated prices humans manage to achieve span a broad range, which points to a literacy gap in effectively interacting with LLMs.

Title: Real Customization or Just Marketing: Are Customized Versions of Chat GPT Useful?. (arXiv:2312.03728v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03728
• Code URL: null
• Copy Paste: [[2312.03728]] Real Customization or Just Marketing: Are Customized Versions of Chat GPT Useful?(http://arxiv.org/abs/2312.03728)
• Summary:

  Large Language Models (LLMs), as the case of OpenAI ChatGPT-4 Turbo, are revolutionizing several industries, including higher education. In this context, LLMs can be personalized through a fine-tuning process to meet the student demands on every particular subject, like statistics. Recently, OpenAI has launched the possibility to fine-tune their model with a natural language web interface, enabling the possibility to create customized GPT version deliberately conditioned to meet the demands of a specific task. The objective of this research is to assess the potential of the customized GPTs that have recently been launched by OpenAI. After developing a Business Statistics Virtual Professor (BSVP), tailored for students at the Universidad Pontificia Comillas, its behavior was evaluated and compared with that of ChatGPT-4 Turbo. The results lead to several conclusions. Firstly, a substantial modification in the style of communication was observed. Following the instructions it was trained with, BSVP provided responses in a more relatable and friendly tone, even incorporating a few minor jokes. Secondly, and this is a matter of relevance, when explicitly asked for something like, "I would like to practice a programming exercise similar to those in R practice 4," BSVP was capable of providing a far superior response: having access to contextual documentation, it could fulfill the request, something beyond ChatGPT-4 Turbo's capabilities. On the downside, the response times were generally higher. Lastly, regarding overall performance, quality, depth, and alignment with the specific content of the course, no statistically significant differences were observed in the responses between BSVP and ChatGPT-4 Turbo. It appears that customized assistants trained with prompts present advantages as virtual aids for students, yet they do not constitute a substantial improvement over ChatGPT-4 Turbo.

Title: A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. (arXiv:2312.03732v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03732
• Code URL: null
• Copy Paste: [[2312.03732]] A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA(http://arxiv.org/abs/2312.03732)
• Summary:

  As large language models (LLMs) have become increasingly compute and memory intensive, parameter-efficient fine-tuning (PEFT) methods are now a common strategy to fine-tune LLMs. A popular PEFT method is Low-Rank Adapters (LoRA), which adds trainable low-rank "adapters" to selected layers. Each adapter consists of a low-rank matrix product, multiplicatively scaled by a rank-dependent factor. This scaling factor, which divides adapters by a factor of the rank, results in slowed learning and stunted performance for LoRA with higher-rank adapters. Consequently, the use of LoRA in practice has generally been limited to very low ranks. In this work, we study the impact of the scaling factor on the learning process and prove that LoRA adapters should be divided by a factor of the square root of the rank. Modifying LoRA with the appropriate scaling factor, which we call the rank-stabilized LoRA (rsLoRA) method, easily provides for a fine-tuning compute/performance trade-off, where larger ranks can be used to trade off increased computational resources during training for better fine-tuning performance, with no change in inference computing cost.

Title: Methods to Estimate Large Language Model Confidence. (arXiv:2312.03733v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03733
• Code URL: null
• Copy Paste: [[2312.03733]] Methods to Estimate Large Language Model Confidence(http://arxiv.org/abs/2312.03733)
• Summary:

  Large Language Models have difficulty communicating uncertainty, which is a significant obstacle to applying LLMs to complex medical tasks. This study evaluates methods to measure LLM confidence when suggesting a diagnosis for challenging clinical vignettes. GPT4 was asked a series of challenging case questions using Chain of Thought and Self Consistency prompting. Multiple methods were investigated to assess model confidence and evaluated on their ability to predict the models observed accuracy. The methods evaluated were Intrinsic Confidence, SC Agreement Frequency and CoT Response Length. SC Agreement Frequency correlated with observed accuracy, yielding a higher Area under the Receiver Operating Characteristic Curve compared to Intrinsic Confidence and CoT Length analysis. SC agreement is the most useful proxy for model confidence, especially for medical diagnosis. Model Intrinsic Confidence and CoT Response Length exhibit a weaker ability to differentiate between correct and incorrect answers, preventing them from being reliable and interpretable markers for model confidence. We conclude GPT4 has a limited ability to assess its own diagnostic accuracy. SC Agreement Frequency is the most useful method to measure GPT4 confidence.

Title: Prompting in Autoregressive Large Language Models. (arXiv:2312.03740v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03740
• Code URL: null
• Copy Paste: [[2312.03740]] Prompting in Autoregressive Large Language Models(http://arxiv.org/abs/2312.03740)
• Summary:

  Autoregressive Large Language Models have transformed the landscape of Natural Language Processing. Pre-train and prompt paradigm has replaced the conventional approach of pre-training and fine-tuning for many downstream NLP tasks. This shift has been possible largely due to LLMs and innovative prompting techniques. LLMs have shown great promise for a variety of downstream tasks owing to their vast parameters and huge datasets that they are pre-trained on. However, in order to fully realize their potential, their outputs must be guided towards the desired outcomes. Prompting, in which a specific input or instruction is provided to guide the LLMs toward the intended output, has become a tool for achieving this goal. In this paper, we discuss the various prompting techniques that have been applied to fully harness the power of LLMs. We present a taxonomy of existing literature on prompting techniques and provide a concise survey based on this taxonomy. Further, we identify some open problems in the realm of prompting in autoregressive LLMs which could serve as a direction for future research.

Title: Evaluating Large Language Model Creativity from a Literary Perspective. (arXiv:2312.03746v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03746
• Code URL: null
• Copy Paste: [[2312.03746]] Evaluating Large Language Model Creativity from a Literary Perspective(http://arxiv.org/abs/2312.03746)
• Summary:

  This paper assesses the potential for large language models (LLMs) to serve as assistive tools in the creative writing process, by means of a single, in-depth case study. In the course of the study, we develop interactive and multi-voice prompting strategies that interleave background descriptions (scene setting, plot elements), instructions that guide composition, samples of text in the target style, and critical discussion of the given samples. We qualitatively evaluate the results from a literary critical perspective, as well as from the standpoint of computational creativity (a sub-field of artificial intelligence). Our findings lend support to the view that the sophistication of the results that can be achieved with an LLM mirrors the sophistication of the prompting.

Title: Applying Large Language Models and Chain-of-Thought for Automatic Scoring. (arXiv:2312.03748v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03748
• Code URL: null
• Copy Paste: [[2312.03748]] Applying Large Language Models and Chain-of-Thought for Automatic Scoring(http://arxiv.org/abs/2312.03748)
• Summary:

  This study investigates the application of large language models (LLMs), specifically GPT-3.5 and GPT-4, with Chain-of-Though (CoT)in the automatic scoring of student-written responses to science assessments. We focused on overcoming the challenges of accessibility, technical complexity, and lack of explainability that have previously limited the use of automatic assessment tools among researchers and educators. We used a testing dataset comprising six assessment tasks (three binomial and three trinomial) with 1,650 student responses. We employed six prompt engineering strategies, combining zero-shot or few-shot learning with CoT, either alone or alongside item stem and scoring rubrics. Results indicated that few-shot (acc = .67) outperformed zero-shot learning (acc = .60), with 12.6\% increase. CoT, when used without item stem and scoring rubrics, did not significantly affect scoring accuracy (acc = .60). However, CoT prompting paired with contextual item stems and rubrics proved to be a significant contributor to scoring accuracy (13.44\% increase for zero-shot; 3.7\% increase for few-shot). Using a novel approach PPEAS, we found a more balanced accuracy across different proficiency categories, highlighting the importance of domain-specific reasoning in enhancing the effectiveness of LLMs in scoring tasks. Additionally, we also found that GPT-4 demonstrated superior performance over GPT-3.5 in various scoring tasks, showing 8.64\% difference. The study revealed that the single-call strategy with GPT-4, particularly using greedy sampling, outperformed other approaches, including ensemble voting strategies. This study demonstrates the potential of LLMs in facilitating automatic scoring, emphasizing that CoT enhances accuracy, particularly when used with item stem and scoring rubrics.

Title: Conceptual Engineering Using Large Language Models. (arXiv:2312.03749v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03749
• Code URL: null
• Copy Paste: [[2312.03749]] Conceptual Engineering Using Large Language Models(http://arxiv.org/abs/2312.03749)
• Summary:

  We describe a method, based on Jennifer Nado's definition of classification procedures as targets of conceptual engineering, that implements such procedures using a large language model. We then apply this method using data from the Wikidata knowledge graph to evaluate concept definitions from two paradigmatic conceptual engineering projects: the International Astronomical Union's redefinition of PLANET and Haslanger's ameliorative analysis of WOMAN. We discuss implications of this work for the theory and practice of conceptual engineering. The code and data can be found on GitHub.

Title: How should the advent of large language models affect the practice of science?. (arXiv:2312.03759v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03759
• Code URL: null
• Copy Paste: [[2312.03759]] How should the advent of large language models affect the practice of science?(http://arxiv.org/abs/2312.03759)
• Summary:

  Large language models (LLMs) are being increasingly incorporated into scientific workflows. However, we have yet to fully grasp the implications of this integration. How should the advent of large language models affect the practice of science? For this opinion piece, we have invited four diverse groups of scientists to reflect on this query, sharing their perspectives and engaging in debate. Schulz et al. make the argument that working with LLMs is not fundamentally different from working with human collaborators, while Bender et al. argue that LLMs are often misused and over-hyped, and that their limitations warrant a focus on more specialized, easily interpretable tools. Marelli et al. emphasize the importance of transparent attribution and responsible use of LLMs. Finally, Botvinick and Gershman advocate that humans should retain responsibility for determining the scientific roadmap. To facilitate the discussion, the four perspectives are complemented with a response from each group. By putting these different perspectives in conversation, we aim to bring attention to important considerations within the academic community regarding the adoption of LLMs and their impact on both current and future scientific practices.

Title: GPT vs Human for Scientific Reviews: A Dual Source Review on Applications of ChatGPT in Science. (arXiv:2312.03769v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03769
• Code URL: null
• Copy Paste: [[2312.03769]] GPT vs Human for Scientific Reviews: A Dual Source Review on Applications of ChatGPT in Science(http://arxiv.org/abs/2312.03769)
• Summary:

  The new polymath Large Language Models (LLMs) can speed-up greatly scientific reviews, possibly using more unbiased quantitative metrics, facilitating cross-disciplinary connections, and identifying emerging trends and research gaps by analyzing large volumes of data. However, at the present time, they lack the required deep understanding of complex methodologies, they have difficulty in evaluating innovative claims, and they are unable to assess ethical issues and conflicts of interest. Herein, we consider 13 GPT-related papers across different scientific domains, reviewed by a human reviewer and SciSpace, a large language model, with the reviews evaluated by three distinct types of evaluators, namely GPT-3.5, a crowd panel, and GPT-4. We found that 50% of SciSpace's responses to objective questions align with those of a human reviewer, with GPT-4 (informed evaluator) often rating the human reviewer higher in accuracy, and SciSpace higher in structure, clarity, and completeness. In subjective questions, the uninformed evaluators (GPT-3.5 and crowd panel) showed varying preferences between SciSpace and human responses, with the crowd panel showing a preference for the human responses. However, GPT-4 rated them equally in accuracy and structure but favored SciSpace for completeness.

Title: SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM. (arXiv:2312.03788v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.03788
• Code URL: null
• Copy Paste: [[2312.03788]] SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM(http://arxiv.org/abs/2312.03788)
• Summary:

  Large language models (LLMs) have shown remarkable capabilities in various tasks. However their huge model size and the consequent demand for computational and memory resources also pose challenges to model deployment. Currently, 4-bit post-training quantization (PTQ) has achieved some success in LLMs, reducing the memory footprint by approximately 75% compared to FP16 models, albeit with some accuracy loss. In this paper, we propose SmoothQuant+, an accurate and efficient 4-bit weight-only PTQ that requires no additional training, which enables lossless in accuracy for LLMs for the first time. Based on the fact that the loss of weight quantization is amplified by the activation outliers, SmoothQuant+ smoothes the activation outliers by channel before quantization, while adjusting the corresponding weights for mathematical equivalence, and then performs group-wise 4-bit weight quantization for linear layers. We have integrated SmoothQuant+ into the vLLM framework, an advanced high-throughput inference engine specially developed for LLMs, and equipped it with an efficient W4A16 CUDA kernels, so that vLLM can seamlessly support SmoothQuant+ 4-bit weight quantization. Our results show that, with SmoothQuant+, the Code Llama-34B model can be quantized and deployed on a A100 40GB GPU, achieving lossless accuracy and a throughput increase of 1.9 to 4.0 times compared to the FP16 model deployed on two A100 40GB GPUs. Moreover, the latency per token is only 68% of the FP16 model deployed on two A100 40GB GPUs. This is the state-of-the-art 4-bit weight quantization for LLMs as we know.

Title: Improving Activation Steering in Language Models with Mean-Centring. (arXiv:2312.03813v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03813
• Code URL: null
• Copy Paste: [[2312.03813]] Improving Activation Steering in Language Models with Mean-Centring(http://arxiv.org/abs/2312.03813)
• Summary:

  Recent work in activation steering has demonstrated the potential to better control the outputs of Large Language Models (LLMs), but it involves finding steering vectors. This is difficult because engineers do not typically know how features are represented in these models. We seek to address this issue by applying the idea of mean-centring to steering vectors. We find that taking the average of activations associated with a target dataset, and then subtracting the mean of all training activations, results in effective steering vectors. We test this method on a variety of models on natural language tasks by steering away from generating toxic text, and steering the completion of a story towards a target genre. We also apply mean-centring to extract function vectors, more effectively triggering the execution of a range of natural language tasks by a significant margin (compared to previous baselines). This suggests that mean-centring can be used to easily improve the effectiveness of activation steering in a wide range of contexts.

Title: Efficient Large Language Models: A Survey. (arXiv:2312.03863v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.03863
• Code URL: https://github.com/aiot-mlsys-lab/efficientllms
• Copy Paste: [[2312.03863]] Efficient Large Language Models: A Survey(http://arxiv.org/abs/2312.03863)
• Summary:

  Large Language Models (LLMs) have demonstrated remarkable capabilities in important tasks such as natural language understanding, language generation, and complex reasoning and have the potential to make a substantial impact on our society. Such capabilities, however, come with the considerable resources they demand, highlighting the strong need to develop effective techniques for addressing their efficiency challenges. In this survey, we provide a systematic and comprehensive review of efficient LLMs research. We organize the literature in a taxonomy consisting of three main categories, covering distinct yet interconnected efficient LLMs topics from model-centric, data-centric, and framework-centric perspective, respectively. We have also created a GitHub repository where we compile the papers featured in this survey at https://github.com/AIoT-MLSys-Lab/EfficientLLMs, https://github.com/AIoT-MLSys-Lab/Efficient-LLMs-Survey, and will actively maintain this repository and incorporate new research as it emerges. We hope our survey can serve as a valuable resource to help researchers and practitioners gain a systematic understanding of the research developments in efficient LLMs and inspire them to contribute to this important and exciting field.

Title: Comparing Large Language Model AI and Human-Generated Coaching Messages for Behavioral Weight Loss. (arXiv:2312.04059v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.04059
• Code URL: null
• Copy Paste: [[2312.04059]] Comparing Large Language Model AI and Human-Generated Coaching Messages for Behavioral Weight Loss(http://arxiv.org/abs/2312.04059)
• Summary:

  Automated coaching messages for weight control can save time and costs, but their repetitive, generic nature may limit their effectiveness compared to human coaching. Large language model (LLM) based artificial intelligence (AI) chatbots, like ChatGPT, could offer more personalized and novel messages to address repetition with their data-processing abilities. While LLM AI demonstrates promise to encourage healthier lifestyles, studies have yet to examine the feasibility and acceptability of LLM-based BWL coaching. 87 adults in a weight-loss trial rated ten coaching messages' helpfulness (five human-written, five ChatGPT-generated) using a 5-point Likert scale, providing additional open-ended feedback to justify their ratings. Participants also identified which messages they believed were AI-generated. The evaluation occurred in two phases: messages in Phase 1 were perceived as impersonal and negative, prompting revisions for Phase 2 messages. In Phase 1, AI-generated messages were rated less helpful than human-written ones, with 66 percent receiving a helpfulness rating of 3 or higher. However, in Phase 2, the AI messages matched the human-written ones regarding helpfulness, with 82% scoring three or above. Additionally, 50% were misidentified as human-written, suggesting AI's sophistication in mimicking human-generated content. A thematic analysis of open-ended feedback revealed that participants appreciated AI's empathy and personalized suggestions but found them more formulaic, less authentic, and too data-focused. This study reveals the preliminary feasibility and acceptability of LLM AIs, like ChatGPT, in crafting potentially effective weight control coaching messages. Our findings also underscore areas for future enhancement.

Title: CLadder: A Benchmark to Assess Causal Reasoning Capabilities of Language Models. (arXiv:2312.04350v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.04350
• Code URL: https://github.com/causalnlp/cladder
• Copy Paste: [[2312.04350]] CLadder: A Benchmark to Assess Causal Reasoning Capabilities of Language Models(http://arxiv.org/abs/2312.04350)
• Summary:

  The ability to perform causal reasoning is widely considered a core feature of intelligence. In this work, we investigate whether large language models (LLMs) can coherently reason about causality. Much of the existing work in natural language processing (NLP) focuses on evaluating commonsense causal reasoning in LLMs, thus failing to assess whether a model can perform causal inference in accordance with a set of well-defined formal rules. To address this, we propose a new NLP task, causal inference in natural language, inspired by the "causal inference engine" postulated by Judea Pearl et al. We compose a large dataset, CLadder, with 10K samples: based on a collection of causal graphs and queries (associational, interventional, and counterfactual), we obtain symbolic questions and ground-truth answers, through an oracle causal inference engine. These are then translated into natural language. We evaluate multiple LLMs on our dataset, and we introduce and evaluate a bespoke chain-of-thought prompting strategy, CausalCoT. We show that our task is highly challenging for LLMs, and we conduct an in-depth analysis to gain deeper insight into the causal reasoning abilities of LLMs. Our data is open-sourced at https://huggingface.co/datasets/causalNLP/cladder, and our code can be found at https://github.com/causalNLP/cladder.

Title: LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs. (arXiv:2312.04372v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.04372
• Code URL: null
• Copy Paste: [[2312.04372]] LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs(http://arxiv.org/abs/2312.04372)
• Summary:

  We present LaMPilot, a novel framework for planning in the field of autonomous driving, rethinking the task as a code-generation process that leverages established behavioral primitives. This approach aims to address the challenge of interpreting and executing spontaneous user instructions such as "overtake the car ahead," which have typically posed difficulties for existing frameworks. We introduce the LaMPilot benchmark specifically designed to quantitatively evaluate the efficacy of Large Language Models (LLMs) in translating human directives into actionable driving policies. We then evaluate a wide range of state-of-the-art code generation language models on tasks from the LaMPilot Benchmark. The results of the experiments showed that GPT-4, with human feedback, achieved an impressive task completion rate of 92.7% and a minimal collision rate of 0.9%. To encourage further investigation in this area, our code and dataset will be made available.

Title: OpenAsp: A Benchmark for Multi-document Open Aspect-based Summarization. (arXiv:2312.04440v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.04440
• Code URL: https://github.com/liatschiff/openasp
• Copy Paste: [[2312.04440]] OpenAsp: A Benchmark for Multi-document Open Aspect-based Summarization(http://arxiv.org/abs/2312.04440)
• Summary:

  The performance of automatic summarization models has improved dramatically in recent years. Yet, there is still a gap in meeting specific information needs of users in real-world scenarios, particularly when a targeted summary is sought, such as in the useful aspect-based summarization setting targeted in this paper. Previous datasets and studies for this setting have predominantly concentrated on a limited set of pre-defined aspects, focused solely on single document inputs, or relied on synthetic data. To advance research on more realistic scenarios, we introduce OpenAsp, a benchmark for multi-document \textit{open} aspect-based summarization. This benchmark is created using a novel and cost-effective annotation protocol, by which an open aspect dataset is derived from existing generic multi-document summarization datasets. We analyze the properties of OpenAsp showcasing its high-quality content. Further, we show that the realistic open-aspect setting realized in OpenAsp poses a challenge for current state-of-the-art summarization models, as well as for large language models.

Title: FoMo Rewards: Can we cast foundation models as reward functions?. (arXiv:2312.03881v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.03881
• Code URL: null
• Copy Paste: [[2312.03881]] FoMo Rewards: Can we cast foundation models as reward functions?(http://arxiv.org/abs/2312.03881)
• Summary:

  We explore the viability of casting foundation models as generic reward functions for reinforcement learning. To this end, we propose a simple pipeline that interfaces an off-the-shelf vision model with a large language model. Specifically, given a trajectory of observations, we infer the likelihood of an instruction describing the task that the user wants an agent to perform. We show that this generic likelihood function exhibits the characteristics ideally expected from a reward function: it associates high values with the desired behaviour and lower values for several similar, but incorrect policies. Overall, our work opens the possibility of designing open-ended agents for interactive tasks via foundation models.

segmentation

Title: Novel class discovery meets foundation models for 3D semantic segmentation. (arXiv:2312.03782v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.03782
• Code URL: null
• Copy Paste: [[2312.03782]] Novel class discovery meets foundation models for 3D semantic segmentation(http://arxiv.org/abs/2312.03782)
• Summary:

  The task of Novel Class Discovery (NCD) in semantic segmentation entails training a model able to accurately segment unlabelled (novel) classes, relying on the available supervision from annotated (base) classes. Although extensively investigated in 2D image data, the extension of the NCD task to the domain of 3D point clouds represents a pioneering effort, characterized by assumptions and challenges that are not present in the 2D case. This paper represents an advancement in the analysis of point cloud data in four directions. Firstly, it introduces the novel task of NCD for point cloud semantic segmentation. Secondly, it demonstrates that directly transposing the only existing NCD method for 2D image semantic segmentation to 3D data yields suboptimal results. Thirdly, a new NCD approach based on online clustering, uncertainty estimation, and semantic distillation is presented. Lastly, a novel evaluation protocol is proposed to rigorously assess the performance of NCD in point cloud semantic segmentation. Through comprehensive evaluations on the SemanticKITTI, SemanticPOSS, and S3DIS datasets, the paper demonstrates substantial superiority of the proposed method over the considered baselines.

Title: PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation. (arXiv:2312.04016v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04016
• Code URL: null
• Copy Paste: [[2312.04016]] PartDistill: 3D Shape Part Segmentation by Vision-Language Model Distillation(http://arxiv.org/abs/2312.04016)
• Summary:

  This paper proposes a cross-modal distillation framework, PartDistill, which transfers 2D knowledge from vision-language models (VLMs) to facilitate 3D shape part segmentation. PartDistill addresses three major challenges in this task: the lack of 3D segmentation in invisible or undetected regions in the 2D projections, inaccurate and inconsistent 2D predictions by VLMs, and the lack of knowledge accumulation across different 3D shapes. PartDistill consists of a teacher network that uses a VLM to make 2D predictions and a student network that learns from the 2D predictions while extracting geometrical features from multiple 3D shapes to carry out 3D part segmentation. A bi-directional distillation, including forward and backward distillations, is carried out within the framework, where the former forward distills the 2D predictions to the student network, and the latter improves the quality of the 2D predictions, which subsequently enhances the final 3D part segmentation. Moreover, PartDistill can exploit generative models that facilitate effortless 3D shape creation for generating knowledge sources to be distilled. Through extensive experiments, PartDistill boosts the existing methods with substantial margins on widely used ShapeNetPart and PartE datasets, by more than 15% and 12% higher mIoU scores, respectively.

Title: Residual Graph Convolutional Network for Bird's-Eye-View Semantic Segmentation. (arXiv:2312.04044v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04044
• Code URL: null
• Copy Paste: [[2312.04044]] Residual Graph Convolutional Network for Bird's-Eye-View Semantic Segmentation(http://arxiv.org/abs/2312.04044)
• Summary:

  Retrieving spatial information and understanding the semantic information of the surroundings are important for Bird's-Eye-View (BEV) semantic segmentation. In the application of autonomous driving, autonomous vehicles need to be aware of their surroundings to drive safely. However, current BEV semantic segmentation techniques, deep Convolutional Neural Networks (CNNs) and transformers, have difficulties in obtaining the global semantic relationships of the surroundings at the early layers of the network. In this paper, we propose to incorporate a novel Residual Graph Convolutional (RGC) module in deep CNNs to acquire both the global information and the region-level semantic relationship in the multi-view image domain. Specifically, the RGC module employs a non-overlapping graph space projection to efficiently project the complete BEV information into graph space. It then builds interconnected spatial and channel graphs to extract spatial information between each node and channel information within each node (i.e., extract contextual relationships of the global features). Furthermore, it uses a downsample residual process to enhance the coordinate feature reuse to maintain the global information. The segmentation data augmentation and alignment module helps to simultaneously augment and align BEV features and ground truth to geometrically preserve their alignment to achieve better segmentation results. Our experimental results on the nuScenes benchmark dataset demonstrate that the RGC network outperforms four state-of-the-art networks and its four variants in terms of IoU and mIoU. The proposed RGC network achieves a higher mIoU of 3.1% than the best state-of-the-art network, BEVFusion. Code and models will be released.

Title: An unsupervised approach towards promptable defect segmentation in laser-based additive manufacturing by Segment Anything. (arXiv:2312.04063v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04063
• Code URL: null
• Copy Paste: [[2312.04063]] An unsupervised approach towards promptable defect segmentation in laser-based additive manufacturing by Segment Anything(http://arxiv.org/abs/2312.04063)
• Summary:

  Foundation models are currently driving a paradigm shift in computer vision tasks for various fields including biology, astronomy, and robotics among others, leveraging user-generated prompts to enhance their performance. In the manufacturing domain, accurate image-based defect segmentation is imperative to ensure product quality and facilitate real-time process control. However, such tasks are often characterized by multiple challenges including the absence of labels and the requirement for low latency inference among others. To address these issues, we construct a framework for image segmentation using a state-of-the-art Vision Transformer (ViT) based Foundation model (Segment Anything Model) with a novel multi-point prompt generation scheme using unsupervised clustering. We apply our framework to perform real-time porosity segmentation in a case study of laser base powder bed fusion (L-PBF) and obtain high Dice Similarity Coefficients (DSC) without the necessity for any supervised fine-tuning in the model. Using such lightweight foundation model inference in conjunction with unsupervised prompt generation, we envision the construction of a real-time anomaly detection pipeline that has the potential to revolutionize the current laser-based additive manufacturing processes, thereby facilitating the shift towards Industry 4.0 and promoting defect-free production along with operational efficiency.

Title: Open-Vocabulary Segmentation with Semantic-Assisted Calibration. (arXiv:2312.04089v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04089
• Code URL: null
• Copy Paste: [[2312.04089]] Open-Vocabulary Segmentation with Semantic-Assisted Calibration(http://arxiv.org/abs/2312.04089)
• Summary:

  This paper studies open-vocabulary segmentation (OVS) through calibrating in-vocabulary and domain-biased embedding space with generalized contextual prior of CLIP. As the core of open-vocabulary understanding, alignment of visual content with the semantics of unbounded text has become the bottleneck of this field. To address this challenge, recent works propose to utilize CLIP as an additional classifier and aggregate model predictions with CLIP classification results. Despite their remarkable progress, performance of OVS methods in relevant scenarios is still unsatisfactory compared with supervised counterparts. We attribute this to the in-vocabulary embedding and domain-biased CLIP prediction. To this end, we present a Semantic-assisted CAlibration Network (SCAN). In SCAN, we incorporate generalized semantic prior of CLIP into proposal embedding to avoid collapsing on known categories. Besides, a contextual shift strategy is applied to mitigate the lack of global context and unnatural background noise. With above designs, SCAN achieves state-of-the-art performance on all popular open-vocabulary segmentation benchmarks. Furthermore, we also focus on the problem of existing evaluation system that ignores semantic duplication across categories, and propose a new metric called Semantic-Guided IoU (SG-IoU).

Title: Instance Tracking in 3D Scenes from Egocentric Videos. (arXiv:2312.04117v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04117
• Code URL: null
• Copy Paste: [[2312.04117]] Instance Tracking in 3D Scenes from Egocentric Videos(http://arxiv.org/abs/2312.04117)
• Summary:

  Egocentric sensors such as AR/VR devices capture human-object interactions and offer the potential to provide task-assistance by recalling 3D locations of objects of interest in the surrounding environment. This capability requires instance tracking in real-world 3D scenes from egocentric videos (IT3DEgo). We explore this problem by first introducing a new benchmark dataset, consisting of RGB and depth videos, per-frame camera pose, and instance-level annotations in both 2D camera and 3D world coordinates. We present an evaluation protocol which evaluates tracking performance in 3D coordinates with two settings for enrolling instances to track: (1) single-view online enrollment where an instance is specified on-the-fly based on the human wearer's interactions. and (2) multi-view pre-enrollment where images of an instance to be tracked are stored in memory ahead of time. To address IT3DEgo, we first re-purpose methods from relevant areas, e.g., single object tracking (SOT) -- running SOT methods to track instances in 2D frames and lifting them to 3D using camera pose and depth. We also present a simple method that leverages pretrained segmentation and detection models to generate proposals from RGB frames and match proposals with enrolled instance images. Perhaps surprisingly, our extensive experiments show that our method (with no finetuning) significantly outperforms SOT-based approaches. We conclude by arguing that the problem of egocentric instance tracking is made easier by leveraging camera pose and using a 3D allocentric (world) coordinate representation.

Title: Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation. (arXiv:2312.04168v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04168
• Code URL: https://github.com/osvai/af-dcd
• Copy Paste: [[2312.04168]] Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation(http://arxiv.org/abs/2312.04168)
• Summary:

  In recent years, knowledge distillation methods based on contrastive learning have achieved promising results on image classification and object detection tasks. However, in this line of research, we note that less attention is paid to semantic segmentation. Existing methods heavily rely on data augmentation and memory buffer, which entail high computational resource demands when applying them to handle semantic segmentation that requires to preserve high-resolution feature maps for making dense pixel-wise predictions. In order to address this problem, we present Augmentation-free Dense Contrastive Knowledge Distillation (Af-DCD), a new contrastive distillation learning paradigm to train compact and accurate deep neural networks for semantic segmentation applications. Af-DCD leverages a masked feature mimicking strategy, and formulates a novel contrastive learning loss via taking advantage of tactful feature partitions across both channel and spatial dimensions, allowing to effectively transfer dense and structured local knowledge learnt by the teacher model to a target student model while maintaining training efficiency. Extensive experiments on five mainstream benchmarks with various teacher-student network pairs demonstrate the effectiveness of our approach. For instance, the DeepLabV3-Res18|DeepLabV3-MBV2 model trained by Af-DCD reaches 77.03%|76.38% mIOU on Cityscapes dataset when choosing DeepLabV3-Res101 as the teacher, setting new performance records. Besides that, Af-DCD achieves an absolute mIOU improvement of 3.26%|3.04%|2.75%|2.30%|1.42% compared with individually trained counterpart on Cityscapes|Pascal VOC|Camvid|ADE20K|COCO-Stuff-164K. Code is available at https://github.com/OSVAI/Af-DCD

Title: SAMBA: A Trainable Segmentation Web-App with Smart Labelling. (arXiv:2312.04197v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04197
• Code URL: null
• Copy Paste: [[2312.04197]] SAMBA: A Trainable Segmentation Web-App with Smart Labelling(http://arxiv.org/abs/2312.04197)
• Summary:

  Segmentation is the assigning of a semantic class to every pixel in an image and is a prerequisite for various statistical analysis tasks in materials science, like phase quantification, physics simulations or morphological characterization. The wide range of length scales, imaging techniques and materials studied in materials science means any segmentation algorithm must generalise to unseen data and support abstract, user-defined semantic classes. Trainable segmentation is a popular interactive segmentation paradigm where a classifier is trained to map from image features to user drawn labels. SAMBA is a trainable segmentation tool that uses Meta's Segment Anything Model (SAM) for fast, high-quality label suggestions and a random forest classifier for robust, generalizable segmentations. It is accessible in the browser (https://www.sambasegment.com/) without the need to download any external dependencies. The segmentation backend is run in the cloud, so does not require the user to have powerful hardware.

Title: Fine-tune vision foundation model for crack segmentation in civil infrastructures. (arXiv:2312.04233v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04233
• Code URL: null
• Copy Paste: [[2312.04233]] Fine-tune vision foundation model for crack segmentation in civil infrastructures(http://arxiv.org/abs/2312.04233)
• Summary:

  Large-scale foundation models have become the mainstream method in the field of deep learning, while in civil engineering, the scale of AI models is strictly limited. In this work, vision foundation model is introduced for crack segmentation. Two Parameter-efficient fine-tuning methods, adapter and low-rank adaptation, are adopted to fine-tune the foundation model in the field of semantic segmentation: Segment Anything Model (SAM). The fine-tuned model CrackSAM is much larger than all the existing crack segmentation models, but shows excellent performance. To test the zero-shot performance of the proposed method, two unique datasets related to road and exterior wall cracks are collected, annotated and open-sourced, in total 810 images. Comparative experiments are conducted with twelve mature semantic segmentation models. On datasets with artificial noise and previously unseen datasets, the performance of CrackSAM far exceeds that of all state-of-the-art models. CrackSAM exhibits remarkable superiority, particularly in challenging conditions such as dim lighting, shadows, road markings, construction joints, and other interference factors. Such cross-scenario results demonstrate the outstanding zero-shot capability of foundation models, and provide new ideas for the development of vision models in civil engineering.

Title: Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation. (arXiv:2312.04265v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04265
• Code URL: null
• Copy Paste: [[2312.04265]] Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation(http://arxiv.org/abs/2312.04265)
• Summary:

  In this paper, we first assess and harness various Vision Foundation Models (VFMs) in the context of Domain Generalized Semantic Segmentation (DGSS). Driven by the motivation that Leveraging Stronger pre-trained models and Fewer trainable parameters for Superior generalizability, we introduce a robust fine-tuning approach, namely Rein, to parameter-efficiently harness VFMs for DGSS. Built upon a set of trainable tokens, each linked to distinct instances, Rein precisely refines and forwards the feature maps from each layer to the next layer within the backbone. This process produces diverse refinements for different categories within a single image. With fewer trainable parameters, Rein efficiently fine-tunes VFMs for DGSS tasks, surprisingly surpassing full parameter fine-tuning. Extensive experiments across various settings demonstrate that Rein significantly outperforms state-of-the-art methods. Remarkably, with just an extra 1% of trainable parameters within the frozen backbone, Rein achieves a mIoU of 68.1% on the Cityscapes, without accessing any real urban-scene datasets.

Title: Activity Grammars for Temporal Action Segmentation. (arXiv:2312.04266v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.04266
• Code URL: https://github.com/gongda0e/kari
• Copy Paste: [[2312.04266]] Activity Grammars for Temporal Action Segmentation(http://arxiv.org/abs/2312.04266)
• Summary:

  Sequence prediction on temporal data requires the ability to understand compositional structures of multi-level semantics beyond individual and contextual properties. The task of temporal action segmentation, which aims at translating an untrimmed activity video into a sequence of action segments, remains challenging for this reason. This paper addresses the problem by introducing an effective activity grammar to guide neural predictions for temporal action segmentation. We propose a novel grammar induction algorithm that extracts a powerful context-free grammar from action sequence data. We also develop an efficient generalized parser that transforms frame-level probability distributions into a reliable sequence of actions according to the induced grammar with recursive rules. Our approach can be combined with any neural network for temporal action segmentation to enhance the sequence prediction and discover its compositional structure. Experimental results demonstrate that our method significantly improves temporal action segmentation in terms of both performance and interpretability on two standard benchmarks, Breakfast and 50 Salads.