2024-08-12

Title: Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspective

• Authors: Yiqun Zhang, Xiaocui Yang, Xingle Xu, Zeran Gao, Yijie Huang, Shiyi Mu, Shi Feng, Daling Wang, Yifei Zhang, Kaisong Song, Ge Yu
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.04638
• Pdf URL: https://arxiv.org/pdf/2408.04638
• Copy Paste: [[2408.04638]] Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspective(https://arxiv.org/abs/2408.04638)
• Keywords: in-context
• Abstract: Affective Computing (AC), integrating computer science, psychology, and cognitive science knowledge, aims to enable machines to recognize, interpret, and simulate human this http URL create more value, AC can be applied to diverse scenarios, including social media, finance, healthcare, education, etc. Affective Computing (AC) includes two mainstream tasks, i.e., Affective Understanding (AU) and Affective Generation (AG). Fine-tuning Pre-trained Language Models (PLMs) for AU tasks has succeeded considerably. However, these models lack generalization ability, requiring specialized models for specific tasks. Additionally, traditional PLMs face challenges in AG, particularly in generating diverse and emotionally rich responses. The emergence of Large Language Models (LLMs), such as the ChatGPT series and LLaMA models, brings new opportunities and challenges, catalyzing a paradigm shift in AC. LLMs possess capabilities of in-context learning, common sense reasoning, and advanced sequence generation, which present unprecedented opportunities for AU. To provide a comprehensive overview of AC in the LLMs era from an NLP perspective, we summarize the development of LLMs research in this field, aiming to offer new insights. Specifically, we first summarize the traditional tasks related to AC and introduce the preliminary study based on LLMs. Subsequently, we outline the relevant techniques of popular LLMs to improve AC tasks, including Instruction Tuning and Prompt Engineering. For Instruction Tuning, we discuss full parameter fine-tuning and parameter-efficient methods such as LoRA, P-Tuning, and Prompt Tuning. In Prompt Engineering, we examine Zero-shot, Few-shot, Chain of Thought (CoT), and Agent-based methods for AU and AG. To clearly understand the performance of LLMs on different Affective Computing tasks, we further summarize the existing benchmarks and evaluation methods.

Title: GPT-3 Powered Information Extraction for Building Robust Knowledge Bases

• Authors: Ritabrata Roy Choudhury, Soumik Dey
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04641
• Pdf URL: https://arxiv.org/pdf/2408.04641
• Copy Paste: [[2408.04641]] GPT-3 Powered Information Extraction for Building Robust Knowledge Bases(https://arxiv.org/abs/2408.04641)
• Keywords: in-context
• Abstract: This work uses the state-of-the-art language model GPT-3 to offer a novel method of information extraction for knowledge base development. The suggested method attempts to solve the difficulties associated with obtaining relevant entities and relationships from unstructured text in order to extract structured information. We conduct experiments on a huge corpus of text from diverse fields to assess the performance of our suggested technique. The evaluation measures, which are frequently employed in information extraction tasks, include precision, recall, and F1-score. The findings demonstrate that GPT-3 can be used to efficiently and accurately extract pertinent and correct information from text, hence increasing the precision and productivity of knowledge base creation. We also assess how well our suggested approach performs in comparison to the most advanced information extraction techniques already in use. The findings show that by utilizing only a small number of instances in in-context learning, our suggested strategy yields competitive outcomes with notable savings in terms of data annotation and engineering expense. Additionally, we use our proposed method to retrieve Biomedical information, demonstrating its practicality in a real-world setting. All things considered, our suggested method offers a viable way to overcome the difficulties involved in obtaining structured data from unstructured text in order to create knowledge bases. It can greatly increase the precision and effectiveness of information extraction, which is necessary for many applications including chatbots, recommendation engines, and question-answering systems.

Title: Distinguishing Chatbot from Human

• Authors: Gauri Anil Godghase, Rishit Agrawal, Tanush Obili, Mark Stamp
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04647
• Pdf URL: https://arxiv.org/pdf/2408.04647
• Copy Paste: [[2408.04647]] Distinguishing Chatbot from Human(https://arxiv.org/abs/2408.04647)
• Keywords: generative
• Abstract: There have been many recent advances in the fields of generative Artificial Intelligence (AI) and Large Language Models (LLM), with the Generative Pre-trained Transformer (GPT) model being a leading "chatbot." LLM-based chatbots have become so powerful that it may seem difficult to differentiate between human-written and machine-generated text. To analyze this problem, we have developed a new dataset consisting of more than 750,000 human-written paragraphs, with a corresponding chatbot-generated paragraph for each. Based on this dataset, we apply Machine Learning (ML) techniques to determine the origin of text (human or chatbot). Specifically, we consider two methodologies for tackling this issue: feature analysis and embeddings. Our feature analysis approach involves extracting a collection of features from the text for classification. We also explore the use of contextual embeddings and transformer-based architectures to train classification models. Our proposed solutions offer high classification accuracy and serve as useful tools for textual analysis, resulting in a better understanding of chatbot-generated text in this era of advanced AI technology.

Title: LLM-based MOFs Synthesis Condition Extraction using Few-Shot Demonstrations

• Authors: Lei Shi, Zhimeng Liu, Yi Yang, Weize Wu, Yuyang Zhang, Hongbo Zhang, Jing Lin, Siyu Wu, Zihan Chen, Ruiming Li, Nan Wang, Zipeng Liu, Huobin Tan, Hongyi Gao, Yue Zhang, Ge Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04665
• Pdf URL: https://arxiv.org/pdf/2408.04665
• Copy Paste: [[2408.04665]] LLM-based MOFs Synthesis Condition Extraction using Few-Shot Demonstrations(https://arxiv.org/abs/2408.04665)
• Keywords: in-context
• Abstract: The extraction of Metal-Organic Frameworks (MOFs) synthesis conditions from literature text has been challenging but crucial for the logical design of new MOFs with desirable functionality. The recent advent of large language models (LLMs) provides disruptively new solution to this long-standing problem and latest researches have reported over 90% F1 in extracting correct conditions from MOFs literature. We argue in this paper that most existing synthesis extraction practices with LLMs stay with the primitive zero-shot learning, which could lead to downgraded extraction and application performance due to the lack of specialized knowledge. This work pioneers and optimizes the few-shot in-context learning paradigm for LLM extraction of material synthesis conditions. First, we propose a human-AI joint data curation process to secure high-quality ground-truth demonstrations for few-shot learning. Second, we apply a BM25 algorithm based on the retrieval-augmented generation (RAG) technique to adaptively select few-shot demonstrations for each MOF's extraction. Over a dataset randomly sampled from 84,898 well-defined MOFs, the proposed few-shot method achieves much higher average F1 performance (0.93 vs. 0.81, +14.8%) than the native zero-shot LLM using the same GPT-4 model, under fully automatic evaluation that are more objective than the previous human evaluation. The proposed method is further validated through real-world material experiments: compared with the baseline zero-shot LLM, the proposed few-shot approach increases the MOFs structural inference performance (R^2) by 29.4% in average.

Title: Towards Linguistic Neural Representation Learning and Sentence Retrieval from Electroencephalogram Recordings

• Authors: Jinzhao Zhou, Yiqun Duan, Ziyi Zhao, Yu-Cheng Chang, Yu-Kai Wang, Thomas Do, Chin-Teng Lin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04679
• Pdf URL: https://arxiv.org/pdf/2408.04679
• Copy Paste: [[2408.04679]] Towards Linguistic Neural Representation Learning and Sentence Retrieval from Electroencephalogram Recordings(https://arxiv.org/abs/2408.04679)
• Keywords: generative
• Abstract: Decoding linguistic information from non-invasive brain signals using EEG has gained increasing research attention due to its vast applicational potential. Recently, a number of works have adopted a generative-based framework to decode electroencephalogram (EEG) signals into sentences by utilizing the power generative capacity of pretrained large language models (LLMs). However, this approach has several drawbacks that hinder the further development of linguistic applications for brain-computer interfaces (BCIs). Specifically, the ability of the EEG encoder to learn semantic information from EEG data remains questionable, and the LLM decoder's tendency to generate sentences based on its training memory can be hard to avoid. These issues necessitate a novel approach for converting EEG signals into sentences. In this paper, we propose a novel two-step pipeline that addresses these limitations and enhances the validity of linguistic EEG decoding research. We first confirm that word-level semantic information can be learned from EEG data recorded during natural reading by training a Conformer encoder via a masked contrastive objective for word-level classification. To achieve sentence decoding results, we employ a training-free retrieval method to retrieve sentences based on the predictions from the EEG encoder. Extensive experiments and ablation studies were conducted in this paper for a comprehensive evaluation of the proposed approach. Visualization of the top prediction candidates reveals that our model effectively groups EEG segments into semantic categories with similar meanings, thereby validating its ability to learn patterns from unspoken EEG recordings. Despite the exploratory nature of this work, these results suggest that our method holds promise for providing more reliable solutions for converting EEG signals into text.

Title: Conversational AI Powered by Large Language Models Amplifies False Memories in Witness Interviews

• Authors: Samantha Chan, Pat Pataranutaporn, Aditya Suri, Wazeer Zulfikar, Pattie Maes, Elizabeth F. Loftus
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.04681
• Pdf URL: https://arxiv.org/pdf/2408.04681
• Copy Paste: [[2408.04681]] Conversational AI Powered by Large Language Models Amplifies False Memories in Witness Interviews(https://arxiv.org/abs/2408.04681)
• Keywords: generative
• Abstract: This study examines the impact of AI on human false memories -- recollections of events that did not occur or deviate from actual occurrences. It explores false memory induction through suggestive questioning in Human-AI interactions, simulating crime witness interviews. Four conditions were tested: control, survey-based, pre-scripted chatbot, and generative chatbot using a large language model (LLM). Participants (N=200) watched a crime video, then interacted with their assigned AI interviewer or survey, answering questions including five misleading ones. False memories were assessed immediately and after one week. Results show the generative chatbot condition significantly increased false memory formation, inducing over 3 times more immediate false memories than the control and 1.7 times more than the survey method. 36.4% of users' responses to the generative chatbot were misled through the interaction. After one week, the number of false memories induced by generative chatbots remained constant. However, confidence in these false memories remained higher than the control after one week. Moderating factors were explored: users who were less familiar with chatbots but more familiar with AI technology, and more interested in crime investigations, were more susceptible to false memories. These findings highlight the potential risks of using advanced AI in sensitive contexts, like police interviews, emphasizing the need for ethical considerations.

Title: Zero-Shot Uncertainty Quantification using Diffusion Probabilistic Models

• Authors: Dule Shu, Amir Barati Farimani
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.04718
• Pdf URL: https://arxiv.org/pdf/2408.04718
• Copy Paste: [[2408.04718]] Zero-Shot Uncertainty Quantification using Diffusion Probabilistic Models(https://arxiv.org/abs/2408.04718)
• Keywords: diffusion, generative
• Abstract: The success of diffusion probabilistic models in generative tasks, such as text-to-image generation, has motivated the exploration of their application to regression problems commonly encountered in scientific computing and various other domains. In this context, the use of diffusion regression models for ensemble prediction is becoming a practice with increasing popularity. Under such background, we conducted a study to quantitatively evaluate the effectiveness of ensemble methods on solving different regression problems using diffusion models. We consider the ensemble prediction of a diffusion model as a means for zero-shot uncertainty quantification, since the diffusion models in our study are not trained with a loss function containing any uncertainty estimation. Through extensive experiments on 1D and 2D data, we demonstrate that ensemble methods consistently improve model prediction accuracy across various regression tasks. Notably, we observed a larger accuracy gain in auto-regressive prediction compared with point-wise prediction, and that enhancements take place in both the mean-square error and the physics-informed loss. Additionally, we reveal a statistical correlation between ensemble prediction error and ensemble variance, offering insights into balancing computational complexity with prediction accuracy and monitoring prediction confidence in practical applications where the ground truth is unknown. Our study provides a comprehensive view of the utility of diffusion ensembles, serving as a useful reference for practitioners employing diffusion models in regression problem-solving.

Title: BRAT: Bonus oRthogonAl Token for Architecture Agnostic Textual Inversion

• Authors: James Baker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04785
• Pdf URL: https://arxiv.org/pdf/2408.04785
• Copy Paste: [[2408.04785]] BRAT: Bonus oRthogonAl Token for Architecture Agnostic Textual Inversion(https://arxiv.org/abs/2408.04785)
• Keywords: diffusion
• Abstract: Textual Inversion remains a popular method for personalizing diffusion models, in order to teach models new subjects and styles. We note that textual inversion has been underexplored using alternatives to the UNet, and experiment with textual inversion with a vision transformer. We also seek to optimize textual inversion using a strategy that does not require explicit use of the UNet and its idiosyncratic layers, so we add bonus tokens and enforce orthogonality. We find the use of the bonus token improves adherence to the source images and the use of the vision transformer improves adherence to the prompt. Code is available at this https URL.

Title: Performance Metric for Multiple Anomaly Score Distributions with Discrete Severity Levels

• Authors: Wonjun Yi, Yong-Hwa Park, Wonho Jung
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04817
• Pdf URL: https://arxiv.org/pdf/2408.04817
• Copy Paste: [[2408.04817]] Performance Metric for Multiple Anomaly Score Distributions with Discrete Severity Levels(https://arxiv.org/abs/2408.04817)
• Keywords: anomaly
• Abstract: The rise of smart factories has heightened the demand for automated maintenance, and normal-data-based anomaly detection has proved particularly effective in environments where anomaly data are scarce. This method, which does not require anomaly data during training, has prompted researchers to focus not only on detecting anomalies but also on classifying severity levels by using anomaly scores. However, the existing performance metrics, such as the area under the receiver operating characteristic curve (AUROC), do not effectively reflect the performance of models in classifying severity levels based on anomaly scores. To address this limitation, we propose the weighted sum of the area under the receiver operating characteristic curve (WS-AUROC), which combines AUROC with a penalty for severity level differences. We conducted various experiments using different penalty assignment methods: uniform penalty regardless of severity level differences, penalty based on severity level index differences, and penalty based on actual physical quantities that cause anomalies. The latter method was the most sensitive. Additionally, we propose an anomaly detector that achieves clear separation of distributions and outperforms the ablation models on the WS-AUROC and AUROC metrics.

Title: Adversarially Robust Industrial Anomaly Detection Through Diffusion Model

• Authors: Yuanpu Cao, Lu Lin, Jinghui Chen
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04839
• Pdf URL: https://arxiv.org/pdf/2408.04839
• Copy Paste: [[2408.04839]] Adversarially Robust Industrial Anomaly Detection Through Diffusion Model(https://arxiv.org/abs/2408.04839)
• Keywords: diffusion, anomaly
• Abstract: Deep learning-based industrial anomaly detection models have achieved remarkably high accuracy on commonly used benchmark datasets. However, the robustness of those models may not be satisfactory due to the existence of adversarial examples, which pose significant threats to the practical deployment of deep anomaly detectors. Recently, it has been shown that diffusion models can be used to purify the adversarial noises and thus build a robust classifier against adversarial attacks. Unfortunately, we found that naively applying this strategy in anomaly detection (i.e., placing a purifier before an anomaly detector) will suffer from a high anomaly miss rate since the purifying process can easily remove both the anomaly signal and the adversarial perturbations, causing the later anomaly detector failed to detect anomalies. To tackle this issue, we explore the possibility of performing anomaly detection and adversarial purification simultaneously. We propose a simple yet effective adversarially robust anomaly detection method, \textit{AdvRAD}, that allows the diffusion model to act both as an anomaly detector and adversarial purifier. We also extend our proposed method for certified robustness to $l_2$ norm bounded perturbations. Through extensive experiments, we show that our proposed method exhibits outstanding (certified) adversarial robustness while also maintaining equally strong anomaly detection performance on par with the state-of-the-art methods on industrial anomaly detection benchmark datasets.

Title: SCOI: Syntax-augmented Coverage-based In-context Example Selection for Machine Translation

• Authors: Chenming Tang, Zhixiang Wang, Yunfang Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04872
• Pdf URL: https://arxiv.org/pdf/2408.04872
• Copy Paste: [[2408.04872]] SCOI: Syntax-augmented Coverage-based In-context Example Selection for Machine Translation(https://arxiv.org/abs/2408.04872)
• Keywords: in-context
• Abstract: In-context learning (ICL) greatly improves the performance of large language models (LLMs) on various down-stream tasks, where the improvement highly depends on the quality of demonstrations. In this work, we introduce syntactic knowledge to select better in-context examples for machine translation (MT). We propose a new strategy, namely Syntax-augmented COverage-based In-context example selection (SCOI), leveraging the deep syntactic structure beyond conventional word matching. Specifically, we measure the set-level syntactic coverage by computing the coverage of polynomial terms with the help of a simplified tree-to-polynomial algorithm, and lexical coverage using word overlap. Furthermore, we devise an alternate selection approach to combine both coverage measures, taking advantage of syntactic and lexical information. We conduct experiments with two multi-lingual LLMs on six translation directions. Empirical results show that our proposed SCOI obtains the highest average COMET score among all learning-free methods, indicating that combining syntactic and lexical coverage successfully helps to select better in-context examples for MT.

Title: On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Survey

• Authors: Jingcai Guo, Zhijie Rao, Zhi Chen, Song Guo, Jingren Zhou, Dacheng Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04879
• Pdf URL: https://arxiv.org/pdf/2408.04879
• Copy Paste: [[2408.04879]] On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Survey(https://arxiv.org/abs/2408.04879)
• Keywords: foundation model
• Abstract: Zero-shot image recognition (ZSIR) aims at empowering models to recognize and reason in unseen domains via learning generalized knowledge from limited data in the seen domain. The gist for ZSIR is to execute element-wise representation and reasoning from the input visual space to the target semantic space, which is a bottom-up modeling paradigm inspired by the process by which humans observe the world, i.e., capturing new concepts by learning and combining the basic components or shared characteristics. In recent years, element-wise learning techniques have seen significant progress in ZSIR as well as widespread application. However, to the best of our knowledge, there remains a lack of a systematic overview of this topic. To enrich the literature and provide a sound basis for its future development, this paper presents a broad review of recent advances in element-wise ZSIR. Concretely, we first attempt to integrate the three basic ZSIR tasks of object recognition, compositional recognition, and foundation model-based open-world recognition into a unified element-wise perspective and provide a detailed taxonomy and analysis of the main research approaches. Then, we collect and summarize some key information and benchmarks, such as detailed technical implementations and common datasets. Finally, we sketch out the wide range of its related applications, discuss vital challenges, and suggest potential future directions.

Title: ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentation

• Authors: Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, Wayne Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04883
• Pdf URL: https://arxiv.org/pdf/2408.04883
• Copy Paste: [[2408.04883]] ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentation(https://arxiv.org/abs/2408.04883)
• Keywords: foundation model
• Abstract: Open-vocabulary semantic segmentation requires models to effectively integrate visual representations with open-vocabulary semantic labels. While Contrastive Language-Image Pre-training (CLIP) models shine in recognizing visual concepts from text, they often struggle with segment coherence due to their limited localization ability. In contrast, Vision Foundation Models (VFMs) excel at acquiring spatially consistent local visual representations, yet they fall short in semantic understanding. This paper introduces ProxyCLIP, an innovative framework designed to harmonize the strengths of both CLIP and VFMs, facilitating enhanced open-vocabulary semantic segmentation. ProxyCLIP leverages the spatial feature correspondence from VFMs as a form of proxy attention to augment CLIP, thereby inheriting the VFMs' robust local consistency and maintaining CLIP's exceptional zero-shot transfer capacity. We propose an adaptive normalization and masking strategy to get the proxy attention from VFMs, allowing for adaptation across different VFMs. Remarkably, as a training-free approach, ProxyCLIP significantly improves the average mean Intersection over Union (mIoU) across eight benchmarks from 40.3 to 44.4, showcasing its exceptional efficacy in bridging the gap between spatial precision and semantic richness for the open-vocabulary segmentation task.

Title: Towards a Generative Approach for Emotion Detection and Reasoning

• Authors: Ankita Bhaumik, Tomek Strzalkowski
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.04906
• Pdf URL: https://arxiv.org/pdf/2408.04906
• Copy Paste: [[2408.04906]] Towards a Generative Approach for Emotion Detection and Reasoning(https://arxiv.org/abs/2408.04906)
• Keywords: generative
• Abstract: Large language models (LLMs) have demonstrated impressive performance in mathematical and commonsense reasoning tasks using chain-of-thought (CoT) prompting techniques. But can they perform emotional reasoning by concatenating `Let's think step-by-step' to the input prompt? In this paper we investigate this question along with introducing a novel approach to zero-shot emotion detection and emotional reasoning using LLMs. Existing state of the art zero-shot approaches rely on textual entailment models to choose the most appropriate emotion label for an input text. We argue that this strongly restricts the model to a fixed set of labels which may not be suitable or sufficient for many applications where emotion analysis is required. Instead, we propose framing the problem of emotion analysis as a generative question-answering (QA) task. Our approach uses a two step methodology of generating relevant context or background knowledge to answer the emotion detection question step-by-step. Our paper is the first work on using a generative approach to jointly address the tasks of emotion detection and emotional reasoning for texts. We evaluate our approach on two popular emotion detection datasets and also release the fine-grained emotion labels and explanations for further training and fine-tuning of emotional reasoning systems.

Title: DAFT-GAN: Dual Affine Transformation Generative Adversarial Network for Text-Guided Image Inpainting

• Authors: Jihoon Lee, Yunhong Min, Hwidong Kim, Sangtae Ahn
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04962
• Pdf URL: https://arxiv.org/pdf/2408.04962
• Copy Paste: [[2408.04962]] DAFT-GAN: Dual Affine Transformation Generative Adversarial Network for Text-Guided Image Inpainting(https://arxiv.org/abs/2408.04962)
• Keywords: generative
• Abstract: In recent years, there has been a significant focus on research related to text-guided image inpainting. However, the task remains challenging due to several constraints, such as ensuring alignment between the image and the text, and maintaining consistency in distribution between corrupted and uncorrupted regions. In this paper, thus, we propose a dual affine transformation generative adversarial network (DAFT-GAN) to maintain the semantic consistency for text-guided inpainting. DAFT-GAN integrates two affine transformation networks to combine text and image features gradually for each decoding block. Moreover, we minimize information leakage of uncorrupted features for fine-grained image generation by encoding corrupted and uncorrupted regions of the masked image separately. Our proposed model outperforms the existing GAN-based models in both qualitative and quantitative assessments with three benchmark datasets (MS-COCO, CUB, and Oxford) for text-guided image inpainting.

Title: XNN: Paradigm Shift in Mitigating Identity Leakage within Cloud-Enabled Deep Learning

• Authors: Kaixin Liu, Huixin Xiong, Bingyu Duan, Zexuan Cheng, Xinyu Zhou, Wanqian Zhang, Xiangyu Zhang
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.04974
• Pdf URL: https://arxiv.org/pdf/2408.04974
• Copy Paste: [[2408.04974]] XNN: Paradigm Shift in Mitigating Identity Leakage within Cloud-Enabled Deep Learning(https://arxiv.org/abs/2408.04974)
• Keywords: generative
• Abstract: In the domain of cloud-based deep learning, the imperative for external computational resources coexists with acute privacy concerns, particularly identity leakage. To address this challenge, we introduce XNN and XNN-d, pioneering methodologies that infuse neural network features with randomized perturbations, striking a harmonious balance between utility and privacy. XNN, designed for the training phase, ingeniously blends random permutation with matrix multiplication techniques to obfuscate feature maps, effectively shielding private data from potential breaches without compromising training integrity. Concurrently, XNN-d, devised for the inference phase, employs adversarial training to integrate generative adversarial noise. This technique effectively counters black-box access attacks aimed at identity extraction, while a distilled face recognition network adeptly processes the perturbed features, ensuring accurate identification. Our evaluation demonstrates XNN's effectiveness, significantly outperforming existing methods in reducing identity leakage while maintaining a high model accuracy.

Title: \textit{re}CSE: Portable Reshaping Features for Sentence Embedding in Self-supervised Contrastive Learning

• Authors: Fufangchen Zhao, Gao Jian, Danfeng Yan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.04975
• Pdf URL: https://arxiv.org/pdf/2408.04975
• Copy Paste: [[2408.04975]] \textit{re}CSE: Portable Reshaping Features for Sentence Embedding in Self-supervised Contrastive Learning(https://arxiv.org/abs/2408.04975)
• Keywords: self-supervised
• Abstract: We propose \textit{re}CSE, a self supervised contrastive learning sentence representation framework based on feature reshaping. This framework is different from the current advanced models that use discrete data augmentation methods, but instead reshapes the input features of the original sentence, aggregates the global information of each token in the sentence, and alleviates the common problems of representation polarity and GPU memory consumption linear increase in current advanced models. In addition, our \textit{re}CSE has achieved competitive performance in semantic similarity tasks. And the experiment proves that our proposed feature reshaping method has strong universality, which can be transplanted to other self supervised contrastive learning frameworks and enhance their representation ability, even achieving state-of-the-art performance. Our code is available at this https URL.

Title: DreamCouple: Exploring High Quality Text-to-3D Generation Via Rectified Flow

• Authors: Hangyu Li, Xiangxiang Chu, Dingyuan Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.05008
• Pdf URL: https://arxiv.org/pdf/2408.05008
• Copy Paste: [[2408.05008]] DreamCouple: Exploring High Quality Text-to-3D Generation Via Rectified Flow(https://arxiv.org/abs/2408.05008)
• Keywords: diffusion
• Abstract: The Score Distillation Sampling (SDS), which exploits pretrained text-to-image model diffusion models as priors to 3D model training, has achieved significant success. Currently, the flow-based diffusion model has become a new trend for generations. Yet, adapting SDS to flow-based diffusion models in 3D generation remains unexplored. Our work is aimed to bridge this gap. In this paper, we adapt SDS to rectified flow and re-examine the over-smoothing issue under this novel framework. The issue can be explained that the model learns an average of multiple ODE trajectories. Then we propose DreamCouple, which instead of randomly sampling noise, uses a rectified flow model to find the coupled noise. Its Unique Couple Matching (UCM) loss guides the model to learn different trajectories and thus solves the over-smoothing issue. We apply our method to both NeRF and 3D Gaussian splatting and achieve state-of-the-art performances. We also identify some other interesting open questions such as initialization issues for NeRF and faster training convergence. Our code will be released soon.

Title: Generalizing Few Data to Unseen Domains Flexibly Based on Label Smoothing Integrated with Distributionally Robust Optimization

• Authors: Yangdi Wang, Zhi-Hai Zhang, Su Xiu Xu, Wenming Guo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.05082
• Pdf URL: https://arxiv.org/pdf/2408.05082
• Copy Paste: [[2408.05082]] Generalizing Few Data to Unseen Domains Flexibly Based on Label Smoothing Integrated with Distributionally Robust Optimization(https://arxiv.org/abs/2408.05082)
• Keywords: anomaly
• Abstract: Overfitting commonly occurs when applying deep neural networks (DNNs) on small-scale datasets, where DNNs do not generalize well from existing data to unseen data. The main reason resulting in overfitting is that small-scale datasets cannot reflect the situations of the real world. Label smoothing (LS) is an effective regularization method to prevent overfitting, avoiding it by mixing one-hot labels with uniform label vectors. However, LS only focuses on labels while ignoring the distribution of existing data. In this paper, we introduce the distributionally robust optimization (DRO) to LS, achieving shift the existing data distribution flexibly to unseen domains when training DNNs. Specifically, we prove that the regularization of LS can be extended to a regularization term for the DNNs parameters when integrating DRO. The regularization term can be utilized to shift existing data to unseen domains and generate new data. Furthermore, we propose an approximate gradient-iteration label smoothing algorithm (GI-LS) to achieve the findings and train DNNs. We prove that the shift for the existing data does not influence the convergence of GI-LS. Since GI-LS incorporates a series of hyperparameters, we further consider using Bayesian optimization (BO) to find the relatively optimal combinations of these hyperparameters. Taking small-scale anomaly classification tasks as a case, we evaluate GI-LS, and the results clearly demonstrate its superior performance.

Title: PreciseControl: Enhancing Text-To-Image Diffusion Models with Fine-Grained Attribute Control

• Authors: Rishubh Parihar, Sachidanand VS, Sabariswaran Mani, Tejan Karmali, R. Venkatesh Babu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.05083
• Pdf URL: https://arxiv.org/pdf/2408.05083
• Copy Paste: [[2408.05083]] PreciseControl: Enhancing Text-To-Image Diffusion Models with Fine-Grained Attribute Control(https://arxiv.org/abs/2408.05083)
• Keywords: diffusion
• Abstract: Recently, we have seen a surge of personalization methods for text-to-image (T2I) diffusion models to learn a concept using a few images. Existing approaches, when used for face personalization, suffer to achieve convincing inversion with identity preservation and rely on semantic text-based editing of the generated face. However, a more fine-grained control is desired for facial attribute editing, which is challenging to achieve solely with text prompts. In contrast, StyleGAN models learn a rich face prior and enable smooth control towards fine-grained attribute editing by latent manipulation. This work uses the disentangled $\mathcal{W+}$ space of StyleGANs to condition the T2I model. This approach allows us to precisely manipulate facial attributes, such as smoothly introducing a smile, while preserving the existing coarse text-based control inherent in T2I models. To enable conditioning of the T2I model on the $\mathcal{W+}$ space, we train a latent mapper to translate latent codes from $\mathcal{W+}$ to the token embedding space of the T2I model. The proposed approach excels in the precise inversion of face images with attribute preservation and facilitates continuous control for fine-grained attribute editing. Furthermore, our approach can be readily extended to generate compositions involving multiple individuals. We perform extensive experiments to validate our method for face personalization and fine-grained attribute editing.

Title: Bootstrap Latents of Nodes and Neighbors for Graph Self-Supervised Learning

• Authors: Yunhui Liu, Huaisong Zhang, Tieke He, Tao Zheng, Jianhua Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.05087
• Pdf URL: https://arxiv.org/pdf/2408.05087
• Copy Paste: [[2408.05087]] Bootstrap Latents of Nodes and Neighbors for Graph Self-Supervised Learning(https://arxiv.org/abs/2408.05087)
• Keywords: self-supervised
• Abstract: Contrastive learning is a significant paradigm in graph self-supervised learning. However, it requires negative samples to prevent model collapse and learn discriminative representations. These negative samples inevitably lead to heavy computation, memory overhead and class collision, compromising the representation learning. Recent studies present that methods obviating negative samples can attain competitive performance and scalability enhancements, exemplified by bootstrapped graph latents (BGRL). However, BGRL neglects the inherent graph homophily, which provides valuable insights into underlying positive pairs. Our motivation arises from the observation that subtly introducing a few ground-truth positive pairs significantly improves BGRL. Although we can't obtain ground-truth positive pairs without labels under the self-supervised setting, edges in the graph can reflect noisy positive pairs, i.e., neighboring nodes often share the same label. Therefore, we propose to expand the positive pair set with node-neighbor pairs. Subsequently, we introduce a cross-attention module to predict the supportiveness score of a neighbor with respect to the anchor node. This score quantifies the positive support from each neighboring node, and is encoded into the training objective. Consequently, our method mitigates class collision from negative and noisy positive samples, concurrently enhancing intra-class compactness. Extensive experiments are conducted on five benchmark datasets and three downstream task node classification, node clustering, and node similarity search. The results demonstrate that our method generates node representations with enhanced intra-class compactness and achieves state-of-the-art performance.

Title: Semantic Successive Refinement: A Generative AI-aided Semantic Communication Framework

• Authors: Kexin Zhang, Lixin Li, Wensheng Lin, Yuna Yan, Rui Li, Wenchi Cheng, Zhu Han
• Subjects: cs.LG, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.05112
• Pdf URL: https://arxiv.org/pdf/2408.05112
• Copy Paste: [[2408.05112]] Semantic Successive Refinement: A Generative AI-aided Semantic Communication Framework(https://arxiv.org/abs/2408.05112)
• Keywords: diffusion, generative
• Abstract: Semantic Communication (SC) is an emerging technology aiming to surpass the Shannon limit. Traditional SC strategies often minimize signal distortion between the original and reconstructed data, neglecting perceptual quality, especially in low Signal-to-Noise Ratio (SNR) environments. To address this issue, we introduce a novel Generative AI Semantic Communication (GSC) system for single-user scenarios. This system leverages deep generative models to establish a new paradigm in SC. Specifically, At the transmitter end, it employs a joint source-channel coding mechanism based on the Swin Transformer for efficient semantic feature extraction and compression. At the receiver end, an advanced Diffusion Model (DM) reconstructs high-quality images from degraded signals, enhancing perceptual details. Additionally, we present a Multi-User Generative Semantic Communication (MU-GSC) system utilizing an asynchronous processing model. This model effectively manages multiple user requests and optimally utilizes system resources for parallel processing. Simulation results on public datasets demonstrate that our generative AI semantic communication systems achieve superior transmission efficiency and enhanced communication content quality across various channel conditions. Compared to CNN-based DeepJSCC, our methods improve the Peak Signal-to-Noise Ratio (PSNR) by 17.75% in Additive White Gaussian Noise (AWGN) channels and by 20.86% in Rayleigh channels.

Title: ECG-FM: An Open Electrocardiogram Foundation Model

• Authors: Kaden McKeen, Laura Oliva, Sameer Masood, Augustin Toma, Barry Rubin, Bo Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.05178
• Pdf URL: https://arxiv.org/pdf/2408.05178
• Copy Paste: [[2408.05178]] ECG-FM: An Open Electrocardiogram Foundation Model(https://arxiv.org/abs/2408.05178)
• Keywords: foundation model
• Abstract: The electrocardiogram (ECG) is a ubiquitous diagnostic test. Conventional task-specific ECG analysis models require large numbers of expensive ECG annotations or associated labels to train. Transfer learning techniques have been shown to improve generalization and reduce reliance on labeled data. We present ECG-FM, an open foundation model for ECG analysis, and conduct a comprehensive study performed on a dataset of 1.66 million ECGs sourced from both publicly available and private institutional sources. ECG-FM adopts a transformer-based architecture and is pretrained on 2.5 million samples using ECG-specific augmentations and contrastive learning, as well as a continuous signal masking objective. Our transparent evaluation includes a diverse range of downstream tasks, where we predict ECG interpretation labels, reduced left ventricular ejection fraction, and abnormal cardiac troponin. Affirming ECG-FM's effectiveness as a foundation model, we demonstrate how its command of contextual information results in strong performance, rich pretrained embeddings, and reliable interpretability. Due to a lack of open-weight practices, we highlight how ECG analysis is lagging behind other medical machine learning subfields in terms of foundation model adoption. Our code is available at this https URL.

Title: Cross-Domain Learning for Video Anomaly Detection with Limited Supervision

• Authors: Yashika Jain, Ali Dabouei, Min Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.05191
• Pdf URL: https://arxiv.org/pdf/2408.05191
• Copy Paste: [[2408.05191]] Cross-Domain Learning for Video Anomaly Detection with Limited Supervision(https://arxiv.org/abs/2408.05191)
• Keywords: anomaly
• Abstract: Video Anomaly Detection (VAD) automates the identification of unusual events, such as security threats in surveillance videos. In real-world applications, VAD models must effectively operate in cross-domain settings, identifying rare anomalies and scenarios not well-represented in the training data. However, existing cross-domain VAD methods focus on unsupervised learning, resulting in performance that falls short of real-world expectations. Since acquiring weak supervision, i.e., video-level labels, for the source domain is cost-effective, we conjecture that combining it with external unlabeled data has notable potential to enhance cross-domain performance. To this end, we introduce a novel weakly-supervised framework for Cross-Domain Learning (CDL) in VAD that incorporates external data during training by estimating its prediction bias and adaptively minimizing that using the predicted uncertainty. We demonstrate the effectiveness of the proposed CDL framework through comprehensive experiments conducted in various configurations on two large-scale VAD datasets: UCF-Crime and XD-Violence. Our method significantly surpasses the state-of-the-art works in cross-domain evaluations, achieving an average absolute improvement of 19.6% on UCF-Crime and 12.87% on XD-Violence.

Title: Cell Morphology-Guided Small Molecule Generation with GFlowNets

• Authors: Stephen Zhewen Lu, Ziqing Lu, Ehsan Hajiramezanali, Tommaso Biancalani, Yoshua Bengio, Gabriele Scalia, Michał Koziarski
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2408.05196
• Pdf URL: https://arxiv.org/pdf/2408.05196
• Copy Paste: [[2408.05196]] Cell Morphology-Guided Small Molecule Generation with GFlowNets(https://arxiv.org/abs/2408.05196)
• Keywords: generative
• Abstract: High-content phenotypic screening, including high-content imaging (HCI), has gained popularity in the last few years for its ability to characterize novel therapeutics without prior knowledge of the protein target. When combined with deep learning techniques to predict and represent molecular-phenotype interactions, these advancements hold the potential to significantly accelerate and enhance drug discovery applications. This work focuses on the novel task of HCI-guided molecular design. Generative models for molecule design could be guided by HCI data, for example with a supervised model that links molecules to phenotypes of interest as a reward function. However, limited labeled data, combined with the high-dimensional readouts, can make training these methods challenging and impractical. We consider an alternative approach in which we leverage an unsupervised multimodal joint embedding to define a latent similarity as a reward for GFlowNets. The proposed model learns to generate new molecules that could produce phenotypic effects similar to those of the given image target, without relying on pre-annotated phenotypic labels. We demonstrate that the proposed method generates molecules with high morphological and structural similarity to the target, increasing the likelihood of similar biological activity, as confirmed by an independent oracle model.

Title: Multi-Garment Customized Model Generation

• Authors: Yichen Liu, Penghui Du, Yi Liu Quanwei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.05206
• Pdf URL: https://arxiv.org/pdf/2408.05206
• Copy Paste: [[2408.05206]] Multi-Garment Customized Model Generation(https://arxiv.org/abs/2408.05206)
• Keywords: diffusion
• Abstract: This paper introduces Multi-Garment Customized Model Generation, a unified framework based on Latent Diffusion Models (LDMs) aimed at addressing the unexplored task of synthesizing images with free combinations of multiple pieces of clothing. The method focuses on generating customized models wearing various targeted outfits according to different text prompts. The primary challenge lies in maintaining the natural appearance of the dressed model while preserving the complex textures of each piece of clothing, ensuring that the information from different garments does not interfere with each other. To tackle these challenges, we first developed a garment encoder, which is a trainable UNet copy with shared weights, capable of extracting detailed features of garments in parallel. Secondly, our framework supports the conditional generation of multiple garments through decoupled multi-garment feature fusion, allowing multiple clothing features to be injected into the backbone network, significantly alleviating conflicts between garment information. Additionally, the proposed garment encoder is a plug-and-play module that can be combined with other extension modules such as IP-Adapter and ControlNet, enhancing the diversity and controllability of the generated models. Extensive experiments demonstrate the superiority of our approach over existing alternatives, opening up new avenues for the task of generating images with multiple-piece clothing combinations