2024-02-20

Title: Text2Data: Low-Resource Data Generation with Textual Control

• Authors: Shiyu Wang, Yihao Feng, Tian Lan, Ning Yu, Yu Bai, Ran Xu, Huan Wang, Caiming Xiong, Silvio Savarese
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.10941
• Pdf URL: https://arxiv.org/pdf/2402.10941
• Copy Paste: [[2402.10941]] Text2Data: Low-Resource Data Generation with Textual Control(https://arxiv.org/abs/2402.10941)
• Keywords: diffusion, generative
• Abstract: Natural language serves as a common and straightforward control signal for humans to interact seamlessly with machines. Recognizing the importance of this interface, the machine learning community is investing considerable effort in generating data that is semantically coherent with textual instructions. While strides have been made in text-to-data generation spanning image editing, audio synthesis, video creation, and beyond, low-resource areas characterized by expensive annotations or complex data structures, such as molecules, motion dynamics, and time series, often lack textual labels. This deficiency impedes supervised learning, thereby constraining the application of advanced generative models for text-to-data tasks. In response to these challenges in the low-resource scenario, we propose Text2Data, a novel approach that utilizes unlabeled data to understand the underlying data distribution through an unsupervised diffusion model. Subsequently, it undergoes controllable finetuning via a novel constraint optimization-based learning objective that ensures controllability and effectively counteracts catastrophic forgetting. Comprehensive experiments demonstrate that Text2Data is able to achieve enhanced performance regarding controllability across various modalities, including molecules, motions and time series, when compared to existing baselines.

Title: Zero-shot Explainable Mental Health Analysis on Social Media by incorporating Mental Scales

• Authors: Wenyu Li, Yinuo Zhu, Xin Lin, Ming Li, Ziyue Jiang, Ziqian Zeng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10948
• Pdf URL: https://arxiv.org/pdf/2402.10948
• Copy Paste: [[2402.10948]] Zero-shot Explainable Mental Health Analysis on Social Media by incorporating Mental Scales(https://arxiv.org/abs/2402.10948)
• Keywords: generative
• Abstract: Traditional discriminative approaches in mental health analysis are known for their strong capacity but lack interpretability and demand large-scale annotated data. On the other hand, generative approaches, such as those based on large language models (LLMs),have the potential to get rid of heavy annotations and provide explanations. However, their capabilities still fall short compared to discriminative approaches, and their explanations may be unreliable due to the fact that the generation of explanation is a black-box process. Inspired by the psychological assessment practice of using scales to evaluate mental states, our method incorporates two procedures via LLMs. First, the patient completes mental health questionnaires, and second, the psychologist interprets the collected information from the mental health questions and makes informed decisions. Experimental results show that our method outperforms other zero-shot methods. Our method can generate more rigorous explanation based on the outputs of mental questionnaires.

Title: Generative AI and Process Systems Engineering: The Next Frontier

• Authors: Benjamin Decardi-Nelson, Abdulelah S. Alshehri, Akshay Ajagekar, Fengqi You
• Subjects: cs.LG, eess.SY, math.OC
• Abstract URL: https://arxiv.org/abs/2402.10977
• Pdf URL: https://arxiv.org/pdf/2402.10977
• Copy Paste: [[2402.10977]] Generative AI and Process Systems Engineering: The Next Frontier(https://arxiv.org/abs/2402.10977)
• Keywords: foundation model, generative
• Abstract: This article explores how emerging generative artificial intelligence (GenAI) models, such as large language models (LLMs), can enhance solution methodologies within process systems engineering (PSE). These cutting-edge GenAI models, particularly foundation models (FMs), which are pre-trained on extensive, general-purpose datasets, offer versatile adaptability for a broad range of tasks, including responding to queries, image generation, and complex decision-making. Given the close relationship between advancements in PSE and developments in computing and systems technologies, exploring the synergy between GenAI and PSE is essential. We begin our discussion with a compact overview of both classic and emerging GenAI models, including FMs, and then dive into their applications within key PSE domains: synthesis and design, optimization and integration, and process monitoring and control. In each domain, we explore how GenAI models could potentially advance PSE methodologies, providing insights and prospects for each area. Furthermore, the article identifies and discusses potential challenges in fully leveraging GenAI within PSE, including multiscale modeling, data requirements, evaluation metrics and benchmarks, and trust and safety, thereby deepening the discourse on effective GenAI integration into systems analysis, design, optimization, operations, monitoring, and control. This paper provides a guide for future research focused on the applications of emerging GenAI in PSE.

Title: "Understanding AI": Semantic Grounding in Large Language Models

• Authors: Holger Lyre
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.10992
• Pdf URL: https://arxiv.org/pdf/2402.10992
• Copy Paste: [[2402.10992]] "Understanding AI": Semantic Grounding in Large Language Models(https://arxiv.org/abs/2402.10992)
• Keywords: self-supervised, generative
• Abstract: Do LLMs understand the meaning of the texts they generate? Do they possess a semantic grounding? And how could we understand whether and what they understand? I start the paper with the observation that we have recently witnessed a generative turn in AI, since generative models, including LLMs, are key for self-supervised learning. To assess the question of semantic grounding, I distinguish and discuss five methodological ways. The most promising way is to apply core assumptions of theories of meaning in philosophy of mind and language to LLMs. Grounding proves to be a gradual affair with a three-dimensional distinction between functional, social and causal grounding. LLMs show basic evidence in all three dimensions. A strong argument is that LLMs develop world models. Hence, LLMs are neither stochastic parrots nor semantic zombies, but already understand the language they generate, at least in an elementary sense.

Title: The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains

• Authors: Benjamin L. Edelman, Ezra Edelman, Surbhi Goel, Eran Malach, Nikolaos Tsilivis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.11004
• Pdf URL: https://arxiv.org/pdf/2402.11004
• Copy Paste: [[2402.11004]] The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains(https://arxiv.org/abs/2402.11004)
• Keywords: in-context
• Abstract: Large language models have the ability to generate text that mimics patterns in their inputs. We introduce a simple Markov Chain sequence modeling task in order to study how this in-context learning (ICL) capability emerges. In our setting, each example is sampled from a Markov chain drawn from a prior distribution over Markov chains. Transformers trained on this task form \emph{statistical induction heads} which compute accurate next-token probabilities given the bigram statistics of the context. During the course of training, models pass through multiple phases: after an initial stage in which predictions are uniform, they learn to sub-optimally predict using in-context single-token statistics (unigrams); then, there is a rapid phase transition to the correct in-context bigram solution. We conduct an empirical and theoretical investigation of this multi-phase process, showing how successful learning results from the interaction between the transformer's layers, and uncovering evidence that the presence of the simpler unigram solution may delay formation of the final bigram solution. We examine how learning is affected by varying the prior distribution over Markov chains, and consider the generalization of our in-context learning of Markov chains (ICL-MC) task to $n$-grams for $n > 2$.

Title: Exploring Value Biases: How LLMs Deviate Towards the Ideal

• Authors: Sarath Sivaprasad, Pramod Kaushik, Sahar Abdelnabi, Mario Fritz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.11005
• Pdf URL: https://arxiv.org/pdf/2402.11005
• Copy Paste: [[2402.11005]] Exploring Value Biases: How LLMs Deviate Towards the Ideal(https://arxiv.org/abs/2402.11005)
• Keywords: in-context
• Abstract: Large-Language-Models (LLMs) are deployed in a wide range of applications, and their response has an increasing social impact. Understanding the non-deliberate(ive) mechanism of LLMs in giving responses is essential in explaining their performance and discerning their biases in real-world applications. This is analogous to human studies, where such inadvertent responses are referred to as sampling. We study this sampling of LLMs in light of value bias and show that the sampling of LLMs tends to favour high-value options. Value bias corresponds to this shift of response from the most likely towards an ideal value represented in the LLM. In fact, this effect can be reproduced even with new entities learnt via in-context prompting. We show that this bias manifests in unexpected places and has implications on relevant application scenarios, like choosing exemplars. The results show that value bias is strong in LLMs across different categories, similar to the results found in human studies.

Title: AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators

• Authors: Jingwei Ni, Minjing Shi, Dominik Stammbach, Mrinmaya Sachan, Elliott Ash, Markus Leippold
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.11073
• Pdf URL: https://arxiv.org/pdf/2402.11073
• Copy Paste: [[2402.11073]] AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators(https://arxiv.org/abs/2402.11073)
• Keywords: generative
• Abstract: With the rise of generative AI, automated fact-checking methods to combat misinformation are becoming more and more important. However, factual claim detection, the first step in a fact-checking pipeline, suffers from two key issues that limit its scalability and generalizability: (1) inconsistency in definitions of the task and what a claim is, and (2) the high cost of manual annotation. To address (1), we review the definitions in related work and propose a unifying definition of factual claims that focuses on verifiability. To address (2), we introduce AFaCTA (Automatic Factual Claim deTection Annotator), a novel framework that assists in the annotation of factual claims with the help of large language models (LLMs). AFaCTA calibrates its annotation confidence with consistency along three predefined reasoning paths. Extensive evaluation and experiments in the domain of political speech reveal that AFaCTA can efficiently assist experts in annotating factual claims and training high-quality classifiers, and can work with or without expert supervision. Our analyses also result in PoliClaim, a comprehensive claim detection dataset spanning diverse political topics.

Title: The Male CEO and the Female Assistant: Probing Gender Biases in Text-To-Image Models Through Paired Stereotype Test

• Authors: Yixin Wan, Kai-Wei Chang
• Subjects: cs.CV, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.11089
• Pdf URL: https://arxiv.org/pdf/2402.11089
• Copy Paste: [[2402.11089]] The Male CEO and the Female Assistant: Probing Gender Biases in Text-To-Image Models Through Paired Stereotype Test(https://arxiv.org/abs/2402.11089)
• Keywords: generative
• Abstract: Recent large-scale Text-To-Image (T2I) models such as DALLE-3 demonstrate great potential in new applications, but also face unprecedented fairness challenges. Prior studies revealed gender biases in single-person image generation, but T2I model applications might require portraying two or more people simultaneously. Potential biases in this setting remain unexplored, leading to fairness-related risks in usage. To study these underlying facets of gender biases in T2I models, we propose a novel Paired Stereotype Test (PST) bias evaluation framework. PST prompts the model to generate two individuals in the same image. They are described with two social identities that are stereotypically associated with the opposite gender. Biases can then be measured by the level of conformation to gender stereotypes in generated images. Using PST, we evaluate DALLE-3 from 2 perspectives: biases in gendered occupation and biases in organizational power. Despite seemingly fair or even anti-stereotype single-person generations, PST still unveils gendered occupational and power associations. Moreover, compared to single-person settings, DALLE-3 generates noticeably more masculine figures under PST for individuals with male-stereotypical identities. PST is therefore effective in revealing underlying gender biases in DALLE-3 that single-person settings cannot capture. Our findings reveal the complicated patterns of gender biases in modern T2I models, further highlighting the critical fairness challenges in multimodal generative systems.

Title: Navigating the Dual Facets: A Comprehensive Evaluation of Sequential Memory Editing in Large Language Models

• Authors: Zihao Lin, Mohammad Beigi, Hongxuan Li, Yufan Zhou, Yuxiang Zhang, Qifan Wang, Wenpeng Yin, Lifu Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.11122
• Pdf URL: https://arxiv.org/pdf/2402.11122
• Copy Paste: [[2402.11122]] Navigating the Dual Facets: A Comprehensive Evaluation of Sequential Memory Editing in Large Language Models(https://arxiv.org/abs/2402.11122)
• Keywords: in-context
• Abstract: Memory Editing (ME) has emerged as an efficient method to modify erroneous facts or inject new facts into Large Language Models (LLMs). Two mainstream ME methods exist: parameter-modifying ME and parameter-preserving ME (integrating extra modules while preserving original parameters). Regrettably, previous studies on ME evaluation have two critical limitations: (i) evaluating LLMs with single edit only, neglecting the need for continuous editing, and (ii) evaluations focusing solely on basic factual triples, overlooking broader LLM capabilities like logical reasoning and reading understanding. This study addresses these limitations with contributions threefold: (i) We explore how ME affects a wide range of fundamental capabilities of LLMs under sequential editing. Experimental results reveal an intriguing phenomenon: Most parameter-modifying ME consistently degrade performance across all tasks after a few sequential edits. In contrast, parameter-preserving ME effectively maintains LLMs' fundamental capabilities but struggles to accurately recall edited knowledge presented in a different format. (ii) We extend our evaluation to different editing settings, such as layers to edit, model size, instruction tuning, etc. Experimental findings indicate several strategies that can potentially mitigate the adverse effects of ME. (iii) We further explain why parameter-modifying ME damages LLMs from three dimensions: parameter changes after editing, language modeling capability, and the in-context learning capability. Our in-depth study advocates more careful use of ME in real-world scenarios.

Title: TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks

• Authors: Benjamin Feuer, Robin Tibor Schirrmeister, Valeriia Cherepanova, Chinmay Hegde, Frank Hutter, Micah Goldblum, Niv Cohen, Colin White
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.11137
• Pdf URL: https://arxiv.org/pdf/2402.11137
• Copy Paste: [[2402.11137]] TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks(https://arxiv.org/abs/2402.11137)
• Keywords: in-context
• Abstract: While tabular classification has traditionally relied on from-scratch training, a recent breakthrough called prior-data fitted networks (PFNs) challenges this approach. Similar to large language models, PFNs make use of pretraining and in-context learning to achieve strong performance on new tasks in a single forward pass. However, current PFNs have limitations that prohibit their widespread adoption. Notably, TabPFN achieves very strong performance on small tabular datasets but is not designed to make predictions for datasets of size larger than 1000. In this work, we overcome these limitations and substantially improve the performance of PFNs by developing context optimization techniques for PFNs. Specifically, we propose TuneTables, a novel prompt-tuning strategy that compresses large datasets into a smaller learned context. TuneTables scales TabPFN to be competitive with state-of-the-art tabular classification methods on larger datasets, while having a substantially lower inference time than TabPFN. Furthermore, we show that TuneTables can be used as an interpretability tool and can even be used to mitigate biases by optimizing a fairness objective.

Title: GenDec: A robust generative Question-decomposition method for Multi-hop reasoning

• Authors: Jian Wu, Linyi Yang, Yuliang Ji, Wenhao Huang, Börje F. Karlsson, Manabu Okumura
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.11166
• Pdf URL: https://arxiv.org/pdf/2402.11166
• Copy Paste: [[2402.11166]] GenDec: A robust generative Question-decomposition method for Multi-hop reasoning(https://arxiv.org/abs/2402.11166)
• Keywords: generative
• Abstract: Multi-hop QA (MHQA) involves step-by-step reasoning to answer complex questions and find multiple relevant supporting facts. However, Existing large language models'(LLMs) reasoning ability in multi-hop question answering remains exploration, which is inadequate in answering multi-hop questions. Moreover, it is unclear whether LLMs follow a desired reasoning chain to reach the right final answer. In this paper, we propose a \textbf{gen}erative question \textbf{dec}omposition method (GenDec) from the perspective of explainable QA by generating independent and complete sub-questions based on incorporating additional extracted evidence for enhancing LLMs' reasoning ability in RAG. To demonstrate the impact, generalization, and robustness of Gendec, we conduct two experiments, the first is combining GenDec with small QA systems on paragraph retrieval and QA tasks. We secondly examine the reasoning capabilities of various state-of-the-art LLMs including GPT-4 and GPT-3.5 combined with GenDec. We experiment on the HotpotQA, 2WikihopMultiHopQA, MuSiQue, and PokeMQA datasets.

Title: Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs

• Authors: Minh-Vuong Nguyen, Linhao Luo, Fatemeh Shiri, Dinh Phung, Yuan-Fang Li, Thuy-Trang Vu, Gholamreza Haffari
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.11199
• Pdf URL: https://arxiv.org/pdf/2402.11199
• Copy Paste: [[2402.11199]] Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs(https://arxiv.org/abs/2402.11199)
• Keywords: generative
• Abstract: Large language models (LLMs) demonstrate strong reasoning abilities when prompted to generate chain-of-thought (CoT) explanations alongside answers. However, previous research on evaluating LLMs has solely focused on answer accuracy, neglecting the correctness of the generated CoT. In this paper, we delve deeper into the CoT reasoning capabilities of LLMs in multi-hop question answering by utilizing knowledge graphs (KGs). We propose a novel discriminative and generative CoT evaluation paradigm to assess LLMs' knowledge of reasoning and the accuracy of the generated CoT. Through experiments conducted on 5 different families of LLMs across 2 multi-hop question-answering datasets, we find that LLMs possess sufficient knowledge to perform reasoning. However, there exists a significant disparity between answer accuracy and faithfulness of the CoT reasoning generated by LLMs, indicating that they often arrive at correct answers through incorrect reasoning.

Title: Enhancing Security in Blockchain Networks: Anomalies, Frauds, and Advanced Detection Techniques

• Authors: Joerg Osterrieder, Stephen Chan, Jeffrey Chu, Yuanyuan Zhang, Branka Hadji Misheva, Codruta Mare
• Subjects: cs.CR, q-fin.GN
• Abstract URL: https://arxiv.org/abs/2402.11231
• Pdf URL: https://arxiv.org/pdf/2402.11231
• Copy Paste: [[2402.11231]] Enhancing Security in Blockchain Networks: Anomalies, Frauds, and Advanced Detection Techniques(https://arxiv.org/abs/2402.11231)
• Keywords: anomaly
• Abstract: Blockchain technology, a foundational distributed ledger system, enables secure and transparent multi-party transactions. Despite its advantages, blockchain networks are susceptible to anomalies and frauds, posing significant risks to their integrity and security. This paper offers a detailed examination of blockchain's key definitions and properties, alongside a thorough analysis of the various anomalies and frauds that undermine these networks. It describes an array of detection and prevention strategies, encompassing statistical and machine learning methods, game-theoretic solutions, digital forensics, reputation-based systems, and comprehensive risk assessment techniques. Through case studies, we explore practical applications of anomaly and fraud detection in blockchain networks, extracting valuable insights and implications for both current practice and future research. Moreover, we spotlight emerging trends and challenges within the field, proposing directions for future investigation and technological development. Aimed at both practitioners and researchers, this paper seeks to provide a technical, in-depth overview of anomaly and fraud detection within blockchain networks, marking a significant step forward in the search for enhanced network security and reliability.

Title: ZeroG: Investigating Cross-dataset Zero-shot Transferability in Graphs

• Authors: Yuhan Li, Peisong Wang, Zhixun Li, Jeffrey Xu Yu, Jia Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.11235
• Pdf URL: https://arxiv.org/pdf/2402.11235
• Copy Paste: [[2402.11235]] ZeroG: Investigating Cross-dataset Zero-shot Transferability in Graphs(https://arxiv.org/abs/2402.11235)
• Keywords: foundation model, generative
• Abstract: With the development of foundation models such as large language models, zero-shot transfer learning has become increasingly significant. This is highlighted by the generative capabilities of NLP models like GPT-4, and the retrieval-based approaches of CV models like CLIP, both of which effectively bridge the gap between seen and unseen data. In the realm of graph learning, the continuous emergence of new graphs and the challenges of human labeling also amplify the necessity for zero-shot transfer learning, driving the exploration of approaches that can generalize across diverse graph data without necessitating dataset-specific and label-specific fine-tuning. In this study, we extend such paradigms to zero-shot transferability in graphs by introducing ZeroG, a new framework tailored to enable cross-dataset generalization. Addressing the inherent challenges such as feature misalignment, mismatched label spaces, and negative transfer, we leverage a language model to encode both node attributes and class semantics, ensuring consistent feature dimensions across datasets. We also propose a prompt-based subgraph sampling module that enriches the semantic information and structure information of extracted subgraphs using prompting nodes and neighborhood aggregation, respectively. We further adopt a lightweight fine-tuning strategy that reduces the risk of overfitting and maintains the zero-shot learning efficacy of the language model. The results underscore the effectiveness of our model in achieving significant cross-dataset zero-shot transferability, opening pathways for the development of graph foundation models. Especially, ZeroG, as a zero-shot method, can even achieve results comparable to those of semi-supervised learning on Pubmed.

Title: DiffPoint: Single and Multi-view Point Cloud Reconstruction with ViT Based Diffusion Model

• Authors: Yu Feng, Xing Shi, Mengli Cheng, Yun Xiong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.11241
• Pdf URL: https://arxiv.org/pdf/2402.11241
• Copy Paste: [[2402.11241]] DiffPoint: Single and Multi-view Point Cloud Reconstruction with ViT Based Diffusion Model(https://arxiv.org/abs/2402.11241)
• Keywords: diffusion
• Abstract: As the task of 2D-to-3D reconstruction has gained significant attention in various real-world scenarios, it becomes crucial to be able to generate high-quality point clouds. Despite the recent success of deep learning models in generating point clouds, there are still challenges in producing high-fidelity results due to the disparities between images and point clouds. While vision transformers (ViT) and diffusion models have shown promise in various vision tasks, their benefits for reconstructing point clouds from images have not been demonstrated yet. In this paper, we first propose a neat and powerful architecture called DiffPoint that combines ViT and diffusion models for the task of point cloud reconstruction. At each diffusion step, we divide the noisy point clouds into irregular patches. Then, using a standard ViT backbone that treats all inputs as tokens (including time information, image embeddings, and noisy patches), we train our model to predict target points based on input images. We evaluate DiffPoint on both single-view and multi-view reconstruction tasks and achieve state-of-the-art results. Additionally, we introduce a unified and flexible feature fusion module for aggregating image features from single or multiple input images. Furthermore, our work demonstrates the feasibility of applying unified architectures across languages and images to improve 3D reconstruction tasks.

Title: C-ICL: Contrastive In-context Learning for Information Extraction

• Authors: Ying Mo, Jian Yang, Jiahao Liu, Shun Zhang, Jingang Wang, Zhoujun Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.11254
• Pdf URL: https://arxiv.org/pdf/2402.11254
• Copy Paste: [[2402.11254]] C-ICL: Contrastive In-context Learning for Information Extraction(https://arxiv.org/abs/2402.11254)
• Keywords: in-context
• Abstract: Recently, there has been increasing interest in exploring the capabilities of advanced large language models (LLMs) in the field of information extraction (IE), specifically focusing on tasks related to named entity recognition (NER) and relation extraction (RE). Although researchers are exploring the use of few-shot information extraction through in-context learning with LLMs, they tend to focus only on using correct or positive examples for demonstration, neglecting the potential value of incorporating incorrect or negative examples into the learning process. In this paper, we present c-ICL, a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. This approach enhances the ability of LLMs to extract entities and relations by utilizing prompts that incorporate not only the positive samples but also the reasoning behind them. This method allows for the identification and correction of potential interface errors. Specifically, our proposed method taps into the inherent contextual information and valuable information in hard negative samples and the nearest positive neighbors to the test and then applies the in-context learning demonstrations based on LLMs. Our experiments on various datasets indicate that c-ICL outperforms previous few-shot in-context learning methods, delivering substantial enhancements in performance across a broad spectrum of related tasks. These improvements are noteworthy, showcasing the versatility of our approach in miscellaneous scenarios.

Title: Human-AI Interactions in the Communication Era: Autophagy Makes Large Models Achieving Local Optima

• Authors: Shu Yang, Lijie Hu, Lu Yu, Muhammad Asif Ali, Di Wang
• Subjects: cs.CL, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2402.11271
• Pdf URL: https://arxiv.org/pdf/2402.11271
• Copy Paste: [[2402.11271]] Human-AI Interactions in the Communication Era: Autophagy Makes Large Models Achieving Local Optima(https://arxiv.org/abs/2402.11271)
• Keywords: generative
• Abstract: The increasing significance of large language and multimodal models in societal information processing has ignited debates on social safety and ethics. However, few studies have approached the analysis of these limitations from the comprehensive perspective of human and artificial intelligence system interactions. This study investigates biases and preferences when humans and large models are used as key links in communication. To achieve this, we design a multimodal dataset and three different experiments to evaluate generative models in their roles as producers and disseminators of information. Our main findings highlight that synthesized information is more likely to be incorporated into model training datasets and messaging than human-generated information. Additionally, large models, when acting as transmitters of information, tend to modify and lose specific content selectively. Conceptually, we present two realistic models of autophagic ("self-consumption") loops to account for the suppression of human-generated information in the exchange of information between humans and AI systems. We generalize the declining diversity of social information and the bottleneck in model performance caused by the above trends to the local optima of large models.

Title: On Good Practices for Task-Specific Distillation of Large Pretrained Models

• Authors: Juliette Marrie, Michael Arbel, Julien Mairal, Diane Larlus
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.11305
• Pdf URL: https://arxiv.org/pdf/2402.11305
• Copy Paste: [[2402.11305]] On Good Practices for Task-Specific Distillation of Large Pretrained Models(https://arxiv.org/abs/2402.11305)
• Keywords: diffusion
• Abstract: Large pretrained visual models exhibit remarkable generalization across diverse recognition tasks. Yet, real-world applications often demand compact models tailored to specific problems. Variants of knowledge distillation have been devised for such a purpose, enabling task-specific compact models (the students) to learn from a generic large pretrained one (the teacher). In this paper, we show that the excellent robustness and versatility of recent pretrained models challenge common practices established in the literature, calling for a new set of optimal guidelines for task-specific distillation. To address the lack of samples in downstream tasks, we also show that a variant of Mixup based on stable diffusion complements standard data augmentation. This strategy eliminates the need for engineered text prompts and improves distillation of generic models into streamlined specialized networks.

Title: ChatEarthNet: A Global-Scale, High-Quality Image-Text Dataset for Remote Sensing

• Authors: Zhenghang Yuan, Zhitong Xiong, Lichao Mou, Xiao Xiang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.11325
• Pdf URL: https://arxiv.org/pdf/2402.11325
• Copy Paste: [[2402.11325]] ChatEarthNet: A Global-Scale, High-Quality Image-Text Dataset for Remote Sensing(https://arxiv.org/abs/2402.11325)
• Keywords: foundation model
• Abstract: An in-depth comprehension of global land cover is essential in Earth observation, forming the foundation for a multitude of applications. Although remote sensing technology has advanced rapidly, leading to a proliferation of satellite imagery, the inherent complexity of these images often makes them difficult for non-expert users to understand. Natural language, as a carrier of human knowledge, can be a bridge between common users and complicated satellite imagery. In this context, we introduce a global-scale, high-quality image-text dataset for remote sensing, providing natural language descriptions for Sentinel-2 data to facilitate the understanding of satellite imagery for common users. Specifically, we utilize Sentinel-2 data for its global coverage as the foundational image source, employing semantic segmentation labels from the European Space Agency's (ESA) WorldCover project to enrich the descriptions of land covers. By conducting in-depth semantic analysis, we formulate detailed prompts to elicit rich descriptions from ChatGPT. To enhance the dataset's quality, we introduce the manual verification process. This step involves manual inspection and correction to refine the dataset, thus significantly improving its accuracy and quality. Finally, we offer the community ChatEarthNet, a large-scale image-text dataset characterized by global coverage, high quality, wide-ranging diversity, and detailed descriptions. ChatEarthNet consists of 163,488 image-text pairs with captions generated by ChatGPT-3.5 and an additional 10,000 image-text pairs with captions generated by ChatGPT-4V(ision). This dataset has significant potential for training vision-language foundation models and evaluating large vision-language models for remote sensing. The dataset will be made publicly available.

Title: PhaseEvo: Towards Unified In-Context Prompt Optimization for Large Language Models

• Authors: Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley Malin, Sricharan Kumar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.11347
• Pdf URL: https://arxiv.org/pdf/2402.11347
• Copy Paste: [[2402.11347]] PhaseEvo: Towards Unified In-Context Prompt Optimization for Large Language Models(https://arxiv.org/abs/2402.11347)
• Keywords: generative, in-context
• Abstract: Crafting an ideal prompt for Large Language Models (LLMs) is a challenging task that demands significant resources and expert human input. Existing work treats the optimization of prompt instruction and in-context learning examples as distinct problems, leading to sub-optimal prompt performance. This research addresses this limitation by establishing a unified in-context prompt optimization framework, which aims to achieve joint optimization of the prompt instruction and examples. However, formulating such optimization in the discrete and high-dimensional natural language space introduces challenges in terms of convergence and computational efficiency. To overcome these issues, we present PhaseEvo, an efficient automatic prompt optimization framework that combines the generative capability of LLMs with the global search proficiency of evolution algorithms. Our framework features a multi-phase design incorporating innovative LLM-based mutation operators to enhance search efficiency and accelerate convergence. We conduct an extensive evaluation of our approach across 35 benchmark tasks. The results demonstrate that PhaseEvo significantly outperforms the state-of-the-art baseline methods by a large margin whilst maintaining good efficiency.

Title: Data Distribution Distilled Generative Model for Generalized Zero-Shot Recognition

• Authors: Yijie Wang, Mingjian Hong, Luwen Huangfu, Sheng Huang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.11424
• Pdf URL: https://arxiv.org/pdf/2402.11424
• Copy Paste: [[2402.11424]] Data Distribution Distilled Generative Model for Generalized Zero-Shot Recognition(https://arxiv.org/abs/2402.11424)
• Keywords: generative
• Abstract: In the realm of Zero-Shot Learning (ZSL), we address biases in Generalized Zero-Shot Learning (GZSL) models, which favor seen data. To counter this, we introduce an end-to-end generative GZSL framework called D$^3$GZSL. This framework respects seen and synthesized unseen data as in-distribution and out-of-distribution data, respectively, for a more balanced model. D$^3$GZSL comprises two core modules: in-distribution dual space distillation (ID$^2$SD) and out-of-distribution batch distillation (O$^2$DBD). ID$^2$SD aligns teacher-student outcomes in embedding and label spaces, enhancing learning coherence. O$^2$DBD introduces low-dimensional out-of-distribution representations per batch sample, capturing shared structures between seen and unseen categories. Our approach demonstrates its effectiveness across established GZSL benchmarks, seamlessly integrating into mainstream generative frameworks. Extensive experiments consistently showcase that D$^3$GZSL elevates the performance of existing generative GZSL methods, underscoring its potential to refine zero-shot learning practices.The code is available at: https://github.com/PJBQ/D3GZSL.git

Title: In-Context Example Ordering Guided by Label Distributions

• Authors: Zhichao Xu, Daniel Cohen, Bei Wang, Vivek Srikumar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.11447
• Pdf URL: https://arxiv.org/pdf/2402.11447
• Copy Paste: [[2402.11447]] In-Context Example Ordering Guided by Label Distributions(https://arxiv.org/abs/2402.11447)
• Keywords: in-context
• Abstract: By allowing models to predict without task-specific training, in-context learning (ICL) with pretrained LLMs has enormous potential in NLP. However, a number of problems persist in ICL. In particular, its performance is sensitive to the choice and order of in-context examples. Given the same set of in-context examples with different orderings, model performance may vary between near random to near state-of-the-art. In this work, we formulate in-context example ordering as an optimization problem. We examine three problem settings that differ in the assumptions they make about what is known about the task. Inspired by the idea of learning from label proportions, we propose two principles for in-context example ordering guided by model's probability predictions. We apply our proposed principles to thirteen text classification datasets and nine different autoregressive LLMs with 700M to 13B parameters. We demonstrate that our approach outperforms the baselines by improving the classification accuracy, reducing model miscalibration, and also by selecting better in-context examples.

Title: AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition

• Authors: Zhaorun Chen, Zhuokai Zhao, Zhihong Zhu, Ruiqi Zhang, Xiang Li, Bhiksha Raj, Huaxiu Yao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.11452
• Pdf URL: https://arxiv.org/pdf/2402.11452
• Copy Paste: [[2402.11452]] AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition(https://arxiv.org/abs/2402.11452)
• Keywords: self-supervised
• Abstract: Recent advancements in large language models (LLMs) have shown promise in multi-step reasoning tasks, yet their reliance on extensive manual labeling to provide procedural feedback remains a significant impediment. To address this challenge, in this paper, we propose a novel self-supervised framework AutoPRM that efficiently enhances the fine-tuning of LLMs for intricate reasoning challenges. Specifically, AutoPRM first decomposes complex problems into more manageable subquestions with a controllable granularity switch, then sequentially apply reinforcement learning to iteratively improve the subquestion solver. Additionally, we propose context-guided-decoding to avoid reward tampering and guide the subquestion solver towards the solution of the holistic problem. Extensive experiments show that AutoPRM significantly improves performance on mathematical and commonsense reasoning tasks over SOTA. More encouragingly, AutoPRM can be easily integrated with other orthogonal reasoning pipelines.

Title: LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks

• Authors: Hanqing Wang, Bowen Ping, Shuo Wang, Xu Han, Yun Chen, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.11455
• Pdf URL: https://arxiv.org/pdf/2402.11455
• Copy Paste: [[2402.11455]] LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks(https://arxiv.org/abs/2402.11455)
• Keywords: generative
• Abstract: LoRA employs lightweight modules to customize large language models (LLMs) for each downstream task or domain, where different learned additional modules represent diverse skills. Combining existing LoRAs to address new tasks can enhance the reusability of learned LoRAs, particularly beneficial for tasks with limited annotated data. Most prior works on LoRA combination primarily rely on task-level weights for each involved LoRA, making different examples and tokens share the same LoRA weights. However, in generative tasks, different tokens may necessitate diverse skills to manage. Taking the Chinese math task as an example, understanding the problem description may depend more on the Chinese LoRA, while the calculation part may rely more on the math LoRA. To this end, we propose LoRA-Flow, which utilizes dynamic weights to adjust the impact of different LoRAs. The weights at each step are determined by a fusion gate with extremely few parameters, which can be learned with only 200 training examples. Experiments across six generative tasks demonstrate that our method consistently outperforms baselines with task-level fusion weights. This underscores the necessity of introducing dynamic fusion weights for LoRA combination.

Title: Visual Concept-driven Image Generation with Text-to-Image Diffusion Model

• Authors: Tanzila Rahman, Shweta Mahajan, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Leonid Sigal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.11487
• Pdf URL: https://arxiv.org/pdf/2402.11487
• Copy Paste: [[2402.11487]] Visual Concept-driven Image Generation with Text-to-Image Diffusion Model(https://arxiv.org/abs/2402.11487)
• Keywords: diffusion
• Abstract: Text-to-image (TTI) diffusion models have demonstrated impressive results in generating high-resolution images of complex and imaginative scenes. Recent approaches have further extended these methods with personalization techniques that allow them to integrate user-illustrated concepts (e.g., the user him/herself) using a few sample image illustrations. However, the ability to generate images with multiple interacting concepts, such as human subjects, as well as concepts that may be entangled in one, or across multiple, image illustrations remains illusive. In this work, we propose a concept-driven TTI personalization framework that addresses these core challenges. We build on existing works that learn custom tokens for user-illustrated concepts, allowing those to interact with existing text tokens in the TTI model. However, importantly, to disentangle and better learn the concepts in question, we jointly learn (latent) segmentation masks that disentangle these concepts in user-provided image illustrations. We do so by introducing an Expectation Maximization (EM)-like optimization procedure where we alternate between learning the custom tokens and estimating masks encompassing corresponding concepts in user-supplied images. We obtain these masks based on cross-attention, from within the U-Net parameterized latent diffusion model and subsequent Dense CRF optimization. We illustrate that such joint alternating refinement leads to the learning of better tokens for concepts and, as a bi-product, latent masks. We illustrate the benefits of the proposed approach qualitatively and quantitatively (through user studies) with a number of examples and use cases that can combine up to three entangled concepts.

Title: Graph Out-of-Distribution Generalization via Causal Intervention

• Authors: Qitian Wu, Fan Nie, Chenxiao Yang, Tianyi Bao, Junchi Yan
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2402.11494
• Pdf URL: https://arxiv.org/pdf/2402.11494
• Copy Paste: [[2402.11494]] Graph Out-of-Distribution Generalization via Causal Intervention(https://arxiv.org/abs/2402.11494)
• Keywords: generative
• Abstract: Out-of-distribution (OOD) generalization has gained increasing attentions for learning on graphs, as graph neural networks (GNNs) often exhibit performance degradation with distribution shifts. The challenge is that distribution shifts on graphs involve intricate interconnections between nodes, and the environment labels are often absent in data. In this paper, we adopt a bottom-up data-generative perspective and reveal a key observation through causal analysis: the crux of GNNs' failure in OOD generalization lies in the latent confounding bias from the environment. The latter misguides the model to leverage environment-sensitive correlations between ego-graph features and target nodes' labels, resulting in undesirable generalization on new unseen nodes. Built upon this analysis, we introduce a conceptually simple yet principled approach for training robust GNNs under node-level distribution shifts, without prior knowledge of environment labels. Our method resorts to a new learning objective derived from causal inference that coordinates an environment estimator and a mixture-of-expert GNN predictor. The new approach can counteract the confounding bias in training data and facilitate learning generalizable predictive relations. Extensive experiment demonstrates that our model can effectively enhance generalization with various types of distribution shifts and yield up to 27.4\% accuracy improvement over state-of-the-arts on graph OOD generalization benchmarks. Source codes are available at https://github.com/fannie1208/CaNet.

Title: URLBERT:A Contrastive and Adversarial Pre-trained Model for URL Classification

• Authors: Yujie Li, Yanbin Wang, Haitao Xu, Zhenhao Guo, Zheng Cao, Lun Zhang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.11495
• Pdf URL: https://arxiv.org/pdf/2402.11495
• Copy Paste: [[2402.11495]] URLBERT:A Contrastive and Adversarial Pre-trained Model for URL Classification(https://arxiv.org/abs/2402.11495)
• Keywords: self-supervised
• Abstract: URLs play a crucial role in understanding and categorizing web content, particularly in tasks related to security control and online recommendations. While pre-trained models are currently dominating various fields, the domain of URL analysis still lacks specialized pre-trained models. To address this gap, this paper introduces URLBERT, the first pre-trained representation learning model applied to a variety of URL classification or detection tasks. We first train a URL tokenizer on a corpus of billions of URLs to address URL data tokenization. Additionally, we propose two novel pre-training tasks: (1) self-supervised contrastive learning tasks, which strengthen the model's understanding of URL structure and the capture of category differences by distinguishing different variants of the same URL; (2) virtual adversarial training, aimed at improving the model's robustness in extracting semantic features from URLs. Finally, our proposed methods are evaluated on tasks including phishing URL detection, web page classification, and ad filtering, achieving state-of-the-art performance. Importantly, we also explore multi-task learning with URLBERT, and experimental results demonstrate that multi-task learning model based on URLBERT exhibit equivalent effectiveness compared to independently fine-tuned models, showing the simplicity of URLBERT in handling complex task requirements. The code for our work is available at https://github.com/Davidup1/URLBERT.

Title: Thyroid ultrasound diagnosis improvement via multi-view self-supervised learning and two-stage pre-training

• Authors: Jian Wang, Xin Yang, Xiaohong Jia, Wufeng Xue, Rusi Chen, Yanlin Chen, Xiliang Zhu, Lian Liu, Yan Cao, Jianqiao Zhou, Dong Ni, Ning Gu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.11497
• Pdf URL: https://arxiv.org/pdf/2402.11497
• Copy Paste: [[2402.11497]] Thyroid ultrasound diagnosis improvement via multi-view self-supervised learning and two-stage pre-training(https://arxiv.org/abs/2402.11497)
• Keywords: self-supervised
• Abstract: Thyroid nodule classification and segmentation in ultrasound images are crucial for computer-aided diagnosis; however, they face limitations owing to insufficient labeled data. In this study, we proposed a multi-view contrastive self-supervised method to improve thyroid nodule classification and segmentation performance with limited manual labels. Our method aligns the transverse and longitudinal views of the same nodule, thereby enabling the model to focus more on the nodule area. We designed an adaptive loss function that eliminates the limitations of the paired data. Additionally, we adopted a two-stage pre-training to exploit the pre-training on ImageNet and thyroid ultrasound images. Extensive experiments were conducted on a large-scale dataset collected from multiple centers. The results showed that the proposed method significantly improves nodule classification and segmentation performance with limited manual labels and outperforms state-of-the-art self-supervised methods. The two-stage pre-training also significantly exceeded ImageNet pre-training.

Title: GenAD: Generative End-to-End Autonomous Driving

• Authors: Wenzhao Zheng, Ruiqi Song, Xianda Guo, Long Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.11502
• Pdf URL: https://arxiv.org/pdf/2402.11502
• Copy Paste: [[2402.11502]] GenAD: Generative End-to-End Autonomous Driving(https://arxiv.org/abs/2402.11502)
• Keywords: generative
• Abstract: Directly producing planning results from raw sensors has been a long-desired solution for autonomous driving and has attracted increasing attention recently. Most existing end-to-end autonomous driving methods factorize this problem into perception, motion prediction, and planning. However, we argue that the conventional progressive pipeline still cannot comprehensively model the entire traffic evolution process, e.g., the future interaction between the ego car and other traffic participants and the structural trajectory prior. In this paper, we explore a new paradigm for end-to-end autonomous driving, where the key is to predict how the ego car and the surroundings evolve given past scenes. We propose GenAD, a generative framework that casts autonomous driving into a generative modeling problem. We propose an instance-centric scene tokenizer that first transforms the surrounding scenes into map-aware instance tokens. We then employ a variational autoencoder to learn the future trajectory distribution in a structural latent space for trajectory prior modeling. We further adopt a temporal model to capture the agent and ego movements in the latent space to generate more effective future trajectories. GenAD finally simultaneously performs motion prediction and planning by sampling distributions in the learned structural latent space conditioned on the instance tokens and using the learned temporal model to generate futures. Extensive experiments on the widely used nuScenes benchmark show that the proposed GenAD achieves state-of-the-art performance on vision-centric end-to-end autonomous driving with high efficiency.

Title: MAL: Motion-Aware Loss with Temporal and Distillation Hints for Self-Supervised Depth Estimation

• Authors: Yup-Jiang Dong, Fang-Lue Zhang, Song-Hai Zhang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.11507
• Pdf URL: https://arxiv.org/pdf/2402.11507
• Copy Paste: [[2402.11507]] MAL: Motion-Aware Loss with Temporal and Distillation Hints for Self-Supervised Depth Estimation(https://arxiv.org/abs/2402.11507)
• Keywords: self-supervised
• Abstract: Depth perception is crucial for a wide range of robotic applications. Multi-frame self-supervised depth estimation methods have gained research interest due to their ability to leverage large-scale, unlabeled real-world data. However, the self-supervised methods often rely on the assumption of a static scene and their performance tends to degrade in dynamic environments. To address this issue, we present Motion-Aware Loss, which leverages the temporal relation among consecutive input frames and a novel distillation scheme between the teacher and student networks in the multi-frame self-supervised depth estimation methods. Specifically, we associate the spatial locations of moving objects with the temporal order of input frames to eliminate errors induced by object motion. Meanwhile, we enhance the original distillation scheme in multi-frame methods to better exploit the knowledge from a teacher network. MAL is a novel, plug-and-play module designed for seamless integration into multi-frame self-supervised monocular depth estimation methods. Adding MAL into previous state-of-the-art methods leads to a reduction in depth estimation errors by up to 4.2% and 10.8% on KITTI and CityScapes benchmarks, respectively.

Title: Temporal Disentangled Contrastive Diffusion Model for Spatiotemporal Imputation

• Authors: Yakun Chen, Kaize Shi, Zhangkai Wu, Juan Chen, Xianzhi Wang, Julian McAuley, Guandong Xu, Shui Yu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.11558
• Pdf URL: https://arxiv.org/pdf/2402.11558
• Copy Paste: [[2402.11558]] Temporal Disentangled Contrastive Diffusion Model for Spatiotemporal Imputation(https://arxiv.org/abs/2402.11558)
• Keywords: diffusion, generative
• Abstract: Spatiotemporal data analysis is pivotal across various domains, including transportation, meteorology, and healthcare. However, the data collected in real-world scenarios often suffers incompleteness due to sensor malfunctions and network transmission errors. Spatiotemporal imputation endeavours to predict missing values by exploiting the inherent spatial and temporal dependencies present in the observed data. Traditional approaches, which rely on classical statistical and machine learning techniques, are often inadequate, particularly when the data fails to meet strict distributional assumptions. In contrast, recent deep learning-based methods, leveraging graph and recurrent neural networks, have demonstrated enhanced efficacy. Nonetheless, these approaches are prone to error accumulation. Generative models have been increasingly adopted to circumvent the reliance on potentially inaccurate historical imputed values for future predictions. These models grapple with the challenge of producing unstable results, a particular issue in diffusion-based models. We aim to address these challenges by designing conditional features to guide the generative process and expedite training. Specifically, we introduce C$^2$TSD, a novel approach incorporating trend and seasonal information as conditional features and employing contrastive learning to improve model generalizability. The extensive experiments on three real-world datasets demonstrate the superior performance of C$^2$TSD over various state-of-the-art baselines.

Title: Visual In-Context Learning for Large Vision-Language Models

• Authors: Yucheng Zhou, Xiang Li, Qianning Wang, Jianbing Shen
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.11574
• Pdf URL: https://arxiv.org/pdf/2402.11574
• Copy Paste: [[2402.11574]] Visual In-Context Learning for Large Vision-Language Models(https://arxiv.org/abs/2402.11574)
• Keywords: in-context
• Abstract: In Large Visual Language Models (LVLMs), the efficacy of In-Context Learning (ICL) remains limited by challenges in cross-modal interactions and representation disparities. To overcome these challenges, we introduce a novel Visual In-Context Learning (VICL) method comprising Visual Demonstration Retrieval, Intent-Oriented Image Summarization, and Intent-Oriented Demonstration Composition. Our approach retrieves images via ''Retrieval & Rerank'' paradigm, summarises images with task intent and task-specific visual parsing, and composes language-based demonstrations that reduce token count and alleviate cross-modal interaction problem. Experimental evaluations on five visual reasoning datasets demonstrate the effectiveness of our method. Moreover, our extensive experiments leverage information flow analysis to elucidate the effectiveness of our method, and investigate the impact of length and position of demonstrations for LVLM. The use of in-context unlearning further shows promise in resetting specific model knowledge without retraining.

Title: SDiT: Spiking Diffusion Model with Transformer

• Authors: Shu Yang, Hanzhi Ma, Chengting Yu, Aili Wang, Er-Ping Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.11588
• Pdf URL: https://arxiv.org/pdf/2402.11588
• Copy Paste: [[2402.11588]] SDiT: Spiking Diffusion Model with Transformer(https://arxiv.org/abs/2402.11588)
• Keywords: diffusion, generative
• Abstract: Spiking neural networks (SNNs) have low power consumption and bio-interpretable characteristics, and are considered to have tremendous potential for energy-efficient computing. However, the exploration of SNNs on image generation tasks remains very limited, and a unified and effective structure for SNN-based generative models has yet to be proposed. In this paper, we explore a novel diffusion model architecture within spiking neural networks. We utilize transformer to replace the commonly used U-net structure in mainstream diffusion models. It can generate higher quality images with relatively lower computational cost and shorter sampling time. It aims to provide an empirical baseline for research of generative models based on SNNs. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 datasets demonstrate that our work is highly competitive compared to existing SNN generative models.

Title: Simplifying Hyperparameter Tuning in Online Machine Learning -- The spotRiverGUI

• Authors: Thomas Bartz-Beielstein
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.11594
• Pdf URL: https://arxiv.org/pdf/2402.11594
• Copy Paste: [[2402.11594]] Simplifying Hyperparameter Tuning in Online Machine Learning -- The spotRiverGUI(https://arxiv.org/abs/2402.11594)
• Keywords: anomaly
• Abstract: Batch Machine Learning (BML) reaches its limits when dealing with very large amounts of streaming data. This is especially true for available memory, handling drift in data streams, and processing new, unknown data. Online Machine Learning (OML) is an alternative to BML that overcomes the limitations of BML. OML is able to process data in a sequential manner, which is especially useful for data streams. The `river` package is a Python OML-library, which provides a variety of online learning algorithms for classification, regression, clustering, anomaly detection, and more. The `spotRiver` package provides a framework for hyperparameter tuning of OML models. The `spotRiverGUI` is a graphical user interface for the `spotRiver` package. The `spotRiverGUI` releases the user from the burden of manually searching for the optimal hyperparameter setting. After the data is provided, users can compare different OML algorithms from the powerful `river` package in a convenient way and tune the selected algorithms very efficiently.

Title: In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness

• Authors: Liam Collins, Advait Parulekar, Aryan Mokhtari, Sujay Sanghavi, Sanjay Shakkottai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.11639
• Pdf URL: https://arxiv.org/pdf/2402.11639
• Copy Paste: [[2402.11639]] In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness(https://arxiv.org/abs/2402.11639)
• Keywords: in-context
• Abstract: A striking property of transformers is their ability to perform in-context learning (ICL), a machine learning framework in which the learner is presented with a novel context during inference implicitly through some data, and tasked with making a prediction in that context. As such that learner must adapt to the context without additional training. We explore the role of softmax attention in an ICL setting where each context encodes a regression task. We show that an attention unit learns a window that it uses to implement a nearest-neighbors predictor adapted to the landscape of the pretraining tasks. Specifically, we show that this window widens with decreasing Lipschitzness and increasing label noise in the pretraining tasks. We also show that on low-rank, linear problems, the attention unit learns to project onto the appropriate subspace before inference. Further, we show that this adaptivity relies crucially on the softmax activation and thus cannot be replicated by the linear activation often studied in prior theoretical analyses.

Title: 3D Point Cloud Compression with Recurrent Neural Network and Image Compression Methods

• Authors: Till Beemelmanns, Yuchen Tao, Bastian Lampe, Lennart Reiher, Raphael van Kempen, Timo Woopen, Lutz Eckstein
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.11680
• Pdf URL: https://arxiv.org/pdf/2402.11680
• Copy Paste: [[2402.11680]] 3D Point Cloud Compression with Recurrent Neural Network and Image Compression Methods(https://arxiv.org/abs/2402.11680)
• Keywords: self-supervised
• Abstract: Storing and transmitting LiDAR point cloud data is essential for many AV applications, such as training data collection, remote control, cloud services or SLAM. However, due to the sparsity and unordered structure of the data, it is difficult to compress point cloud data to a low volume. Transforming the raw point cloud data into a dense 2D matrix structure is a promising way for applying compression algorithms. We propose a new lossless and calibrated 3D-to-2D transformation which allows compression algorithms to efficiently exploit spatial correlations within the 2D representation. To compress the structured representation, we use common image compression methods and also a self-supervised deep compression approach using a recurrent neural network. We also rearrange the LiDAR's intensity measurements to a dense 2D representation and propose a new metric to evaluate the compression performance of the intensity. Compared to approaches that are based on generic octree point cloud compression or based on raw point cloud data compression, our approach achieves the best quantitative and visual performance. Source code and dataset are available at https://github.com/ika-rwth-aachen/Point-Cloud-Compression.

Title: GNNavi: Navigating the Information Flow in Large Language Models by Graph Neural Network

• Authors: Shuzhou Yuan, Ercong Nie, Michael Färber, Helmut Schmid, Hinrich Schütze
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.11709
• Pdf URL: https://arxiv.org/pdf/2402.11709
• Copy Paste: [[2402.11709]] GNNavi: Navigating the Information Flow in Large Language Models by Graph Neural Network(https://arxiv.org/abs/2402.11709)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) exhibit strong In-Context Learning (ICL) capabilities when prompts with demonstrations are applied to them. However, fine-tuning still remains crucial to further enhance their adaptability. Prompt-based fine-tuning proves to be an effective fine-tuning method in low-data scenarios, but high demands on computing resources limit its practicality. We address this issue by introducing a prompt-based parameter-efficient fine-tuning (PEFT) approach. GNNavi leverages insights into ICL's information flow dynamics, which indicates that label words act in prompts as anchors for information propagation. GNNavi employs a Graph Neural Network (GNN) layer to precisely guide the aggregation and distribution of information flow during the processing of prompts by hardwiring the desired information flow into the GNN. Our experiments on text classification tasks with GPT-2 and Llama2 shows GNNavi surpasses standard prompt-based fine-tuning methods in few-shot settings by updating just 0.2% to 0.5% of parameters. We compare GNNavi with prevalent PEFT approaches, such as prefix tuning, LoRA and Adapter in terms of performance and efficiency. Our analysis reveals that GNNavi enhances information flow and ensures a clear aggregation process.

Title: In-Context Learning Demonstration Selection via Influence Analysis

• Authors: Vinay M.S., Minh-Hao Van, Xintao Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.11750
• Pdf URL: https://arxiv.org/pdf/2402.11750
• Copy Paste: [[2402.11750]] In-Context Learning Demonstration Selection via Influence Analysis(https://arxiv.org/abs/2402.11750)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) have demonstrated their In-Context Learning (ICL) capabilities which provides an opportunity to perform few shot learning without any gradient update. Despite its multiple benefits, ICL generalization performance is sensitive to the selected demonstrations. Selecting effective demonstrations for ICL is still an open research challenge. To address this challenge, we propose a demonstration selection method called InfICL which analyzes influences of training samples through influence functions. Identifying highly influential training samples can potentially aid in uplifting the ICL generalization performance. To limit the running cost of InfICL, we only employ the LLM to generate sample embeddings, and don't perform any costly fine tuning. We perform empirical study on multiple real-world datasets and show merits of our InfICL against state-of-the-art baselines.

Title: MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs

• Authors: Yavuz Faruk Bakman, Duygu Nur Yaldiz, Baturalp Buyukates, Chenyang Tao, Dimitrios Dimitriadis, Salman Avestimehr
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.11756
• Pdf URL: https://arxiv.org/pdf/2402.11756
• Copy Paste: [[2402.11756]] MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs(https://arxiv.org/abs/2402.11756)
• Keywords: generative
• Abstract: Generative Large Language Models (LLMs) are widely utilized for their excellence in various tasks. However, their tendency to produce inaccurate or misleading outputs poses a potential risk, particularly in high-stakes environments. Therefore, estimating the correctness of generative LLM outputs is an important task for enhanced reliability. Uncertainty Estimation (UE) in generative LLMs is an evolving domain, where SOTA probability-based methods commonly employ length-normalized scoring. In this work, we propose Meaning-Aware Response Scoring (MARS) as an alternative to length-normalized scoring for UE methods. MARS is a novel scoring function that considers the semantic contribution of each token in the generated sequence in the context of the question. We demonstrate that integrating MARS into UE methods results in a universal and significant improvement in UE performance. We conduct experiments using three distinct closed-book question-answering datasets across five popular pre-trained LLMs. Lastly, we validate the efficacy of MARS on a Medical QA dataset. Code can be found https://anonymous.4open.science/r/LLM_Uncertainity-309B.

Title: Towards Theoretical Understandings of Self-Consuming Generative Models

• Authors: Shi Fu, Sen Zhang, Yingjie Wang, Xinmei Tian, Dacheng Tao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.11778
• Pdf URL: https://arxiv.org/pdf/2402.11778
• Copy Paste: [[2402.11778]] Towards Theoretical Understandings of Self-Consuming Generative Models(https://arxiv.org/abs/2402.11778)
• Keywords: diffusion, generative
• Abstract: This paper tackles the emerging challenge of training generative models within a self-consuming loop, wherein successive generations of models are recursively trained on mixtures of real and synthetic data from previous generations. We construct a theoretical framework to rigorously evaluate how this training regimen impacts the data distributions learned by future models. Specifically, we derive bounds on the total variation (TV) distance between the synthetic data distributions produced by future models and the original real data distribution under various mixed training scenarios. Our analysis demonstrates that this distance can be effectively controlled under the condition that mixed training dataset sizes or proportions of real data are large enough. Interestingly, we further unveil a phase transition induced by expanding synthetic data amounts, proving theoretically that while the TV distance exhibits an initial ascent, it declines beyond a threshold point. Finally, we specialize our general results to diffusion models, delivering nuanced insights such as the efficacy of optimal early stopping within the self-consuming loop.

Title: SDGE: Stereo Guided Depth Estimation for 360° Camera Sets

• Authors: Jialei Xu, Xianming Liu, Junjun Jiang, Xiangyang Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.11791
• Pdf URL: https://arxiv.org/pdf/2402.11791
• Copy Paste: [[2402.11791]] SDGE: Stereo Guided Depth Estimation for 360° Camera Sets(https://arxiv.org/abs/2402.11791)
• Keywords: self-supervised
• Abstract: Depth estimation is a critical technology in autonomous driving, and multi-camera systems are often used to achieve a 360{\deg} perception. These 360{\deg} camera sets often have limited or low-quality overlap regions, making multi-view stereo methods infeasible for the entire image. Alternatively, monocular methods may not produce consistent cross-view predictions. To address these issues, we propose the Stereo Guided Depth Estimation (SGDE) method, which enhances depth estimation of the full image by explicitly utilizing multi-view stereo results on the overlap. We suggest building virtual pinhole cameras to resolve the distortion problem of fisheye cameras and unify the processing for the two types of 360{\deg} cameras. For handling the varying noise on camera poses caused by unstable movement, the approach employs a self-calibration method to obtain highly accurate relative poses of the adjacent cameras with minor overlap. These enable the use of robust stereo methods to obtain high-quality depth prior in the overlap region. This prior serves not only as an additional input but also as pseudo-labels that enhance the accuracy of depth estimation methods and improve cross-view prediction consistency. The effectiveness of SGDE is evaluated on one fisheye camera dataset, Synthetic Urban, and two pinhole camera datasets, DDAD and nuScenes. Our experiments demonstrate that SGDE is effective for both supervised and self-supervised depth estimation, and highlight the potential of our method for advancing downstream autonomous driving technologies, such as 3D object detection and occupancy prediction.

Title: Generative Kaleidoscopic Networks

• Authors: Harsh Shrivastava
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.11793
• Pdf URL: https://arxiv.org/pdf/2402.11793
• Copy Paste: [[2402.11793]] Generative Kaleidoscopic Networks(https://arxiv.org/abs/2402.11793)
• Keywords: generative
• Abstract: We discovered that the Deep ReLU networks (or Multilayer Perceptron architecture) demonstrate an 'over-generalization' phenomenon. That is, the output values for the inputs that were not seen during training are mapped close to the output range that were observed during the learning process. In other words, the MLP learns a many-to-one mapping and this effect is more prominent as we increase the number of layers or depth of the MLP. We utilize this property of Deep ReLU networks to design a dataset kaleidoscope, termed as 'Generative Kaleidoscopic Networks'. Briefly, if we learn a MLP to map from input $x\in\mathbb{R}^D$ to itself $f_\mathcal{N}(x)\rightarrow x$, the 'Kaleidoscopic sampling' procedure starts with a random input noise $z\in\mathbb{R}^D$ and recursively applies $f_\mathcal{N}(\cdots f_\mathcal{N}(z)\cdots )$. After a burn-in period duration, we start observing samples from the input distribution and we found that deeper the MLP, higher is the quality of samples recovered. Scope: We observed this phenomenon to various degrees for the other deep learning architectures like CNNs, Transformers & U-Nets and we are currently investigating them further.

Title: Avoiding Feature Suppression in Contrastive Learning: Learning What Has Not Been Learned Before

• Authors: Jihai Zhang, Xiang Lan, Xiaoye Qu, Yu Cheng, Mengling Feng, Bryan Hooi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.11816
• Pdf URL: https://arxiv.org/pdf/2402.11816
• Copy Paste: [[2402.11816]] Avoiding Feature Suppression in Contrastive Learning: Learning What Has Not Been Learned Before(https://arxiv.org/abs/2402.11816)
• Keywords: self-supervised
• Abstract: Self-Supervised contrastive learning has emerged as a powerful method for obtaining high-quality representations from unlabeled data. However, feature suppression has recently been identified in standard contrastive learning ($e.g.$, SimCLR, CLIP): in a single end-to-end training stage, the contrastive model captures only parts of the shared information across contrasting views, while ignore the other potentially useful information. With feature suppression, contrastive models often fail to learn sufficient representations capable for various downstream tasks. To mitigate the feature suppression problem and ensure the contrastive model to learn comprehensive representations, we develop a novel Multistage Contrastive Learning (MCL) framework. Unlike standard contrastive learning that often result in feature suppression, MCL progressively learn new features that have not been explored in the previous stage, while maintaining the well-learned features. Extensive experiments conducted on various publicly available benchmarks validate the effectiveness of our proposed framework. In addition, we demonstrate that the proposed MCL can be adapted to a variety of popular contrastive learning backbones and boost their performance by learning features that could not be gained from standard contrastive learning procedures.

Title: Where It Really Matters: Few-Shot Environmental Conservation Media Monitoring for Low-Resource Languages

• Authors: Sameer Jain, Sedrick Scott Keh, Shova Chettri, Karun Dewan, Pablo Izquierdo, Johanna Prussman, Pooja Shreshtha, Cesar Suarez, Zheyuan Ryan Shi, Lei Li, Fei Fang
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.11818
• Pdf URL: https://arxiv.org/pdf/2402.11818
• Copy Paste: [[2402.11818]] Where It Really Matters: Few-Shot Environmental Conservation Media Monitoring for Low-Resource Languages(https://arxiv.org/abs/2402.11818)
• Keywords: in-context
• Abstract: Environmental conservation organizations routinely monitor news content on conservation in protected areas to maintain situational awareness of developments that can have an environmental impact. Existing automated media monitoring systems require large amounts of data labeled by domain experts, which is only feasible at scale for high-resource languages like English. However, such tools are most needed in the global south where news of interest is mainly in local low-resource languages, and far fewer experts are available to annotate datasets sustainably. In this paper, we propose NewsSerow, a method to automatically recognize environmental conservation content in low-resource languages. NewsSerow is a pipeline of summarization, in-context few-shot classification, and self-reflection using large language models (LLMs). Using at most 10 demonstration example news articles in Nepali, NewsSerow significantly outperforms other few-shot methods and achieves comparable performance with models fully fine-tuned using thousands of examples. The World Wide Fund for Nature (WWF) has deployed NewsSerow for media monitoring in Nepal, significantly reducing their operational burden, and ensuring that AI tools for conservation actually reach the communities that need them the most. NewsSerow has also been deployed for countries with other languages like Colombia.

Title: UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Prediction

• Authors: Yuan Yuan, Jingtao Ding, Jie Feng, Depeng Jin, Yong Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.11838
• Pdf URL: https://arxiv.org/pdf/2402.11838
• Copy Paste: [[2402.11838]] UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Prediction(https://arxiv.org/abs/2402.11838)
• Keywords: foundation model, generative
• Abstract: Urban spatio-temporal prediction is crucial for informed decision-making, such as transportation management, resource optimization, and urban planning. Although pretrained foundation models for natural languages have experienced remarkable breakthroughs, wherein one general-purpose model can tackle multiple tasks across various domains, urban spatio-temporal modeling lags behind. Existing approaches for urban prediction are usually tailored for specific spatio-temporal scenarios, requiring task-specific model designs and extensive in-domain training data. In this work, we propose a universal model, UniST, for urban spatio-temporal prediction. Drawing inspiration from large language models, UniST achieves success through: (i) flexibility towards diverse spatio-temporal data characteristics, (ii) effective generative pre-training with elaborated masking strategies to capture complex spatio-temporal relationships, (iii) spatio-temporal knowledge-guided prompts that align and leverage intrinsic and shared knowledge across scenarios. These designs together unlock the potential of a one-for-all model for spatio-temporal prediction with powerful generalization capability. Extensive experiments on 15 cities and 6 domains demonstrate the universality of UniST in advancing state-of-the-art prediction performance, especially in few-shot and zero-shot scenarios.

Title: WildFake: A Large-scale Challenging Dataset for AI-Generated Images Detection

• Authors: Yan Hong, Jianfu Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.11843
• Pdf URL: https://arxiv.org/pdf/2402.11843
• Copy Paste: [[2402.11843]] WildFake: A Large-scale Challenging Dataset for AI-Generated Images Detection(https://arxiv.org/abs/2402.11843)
• Keywords: diffusion, generative
• Abstract: The extraordinary ability of generative models enabled the generation of images with such high quality that human beings cannot distinguish Artificial Intelligence (AI) generated images from real-life photographs. The development of generation techniques opened up new opportunities but concurrently introduced potential risks to privacy, authenticity, and security. Therefore, the task of detecting AI-generated imagery is of paramount importance to prevent illegal activities. To assess the generalizability and robustness of AI-generated image detection, we present a large-scale dataset, referred to as WildFake, comprising state-of-the-art generators, diverse object categories, and real-world applications. WildFake dataset has the following advantages: 1) Rich Content with Wild collection: WildFake collects fake images from the open-source community, enriching its diversity with a broad range of image classes and image styles. 2) Hierarchical structure: WildFake contains fake images synthesized by different types of generators from GANs, diffusion models, to other generative models. These key strengths enhance the generalization and robustness of detectors trained on WildFake, thereby demonstrating WildFake's considerable relevance and effectiveness for AI-generated detectors in real-world scenarios. Moreover, our extensive evaluation experiments are tailored to yield profound insights into the capabilities of different levels of generative models, a distinctive advantage afforded by WildFake's unique hierarchical structure.

Title: Modularized Networks for Few-shot Hateful Meme Detection

• Authors: Rui Cao, Roy Ka-Wei Lee, Jing Jiang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.11845
• Pdf URL: https://arxiv.org/pdf/2402.11845
• Copy Paste: [[2402.11845]] Modularized Networks for Few-shot Hateful Meme Detection(https://arxiv.org/abs/2402.11845)
• Keywords: in-context
• Abstract: In this paper, we address the challenge of detecting hateful memes in the low-resource setting where only a few labeled examples are available. Our approach leverages the compositionality of Low-rank adaptation (LoRA), a widely used parameter-efficient tuning technique. We commence by fine-tuning large language models (LLMs) with LoRA on selected tasks pertinent to hateful meme detection, thereby generating a suite of LoRA modules. These modules are capable of essential reasoning skills for hateful meme detection. We then use the few available annotated samples to train a module composer, which assigns weights to the LoRA modules based on their relevance. The model's learnable parameters are directly proportional to the number of LoRA modules. This modularized network, underpinned by LLMs and augmented with LoRA modules, exhibits enhanced generalization in the context of hateful meme detection. Our evaluation spans three datasets designed for hateful meme detection in a few-shot learning context. The proposed method demonstrates superior performance to traditional in-context learning, which is also more computationally intensive during inference.We then use the few available annotated samples to train a module composer, which assigns weights to the LoRA modules based on their relevance. The model's learnable parameters are directly proportional to the number of LoRA modules. This modularized network, underpinned by LLMs and augmented with LoRA modules, exhibits enhanced generalization in the context of hateful meme detection. Our evaluation spans three datasets designed for hateful meme detection in a few-shot learning context. The proposed method demonstrates superior performance to traditional in-context learning, which is also more computationally intensive during inference.

Title: UnlearnCanvas: A Stylized Image Dataset to Benchmark Machine Unlearning for Diffusion Models

• Authors: Yihua Zhang, Yimeng Zhang, Yuguang Yao, Jinghan Jia, Jiancheng Liu, Xiaoming Liu, Sijia Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.11846
• Pdf URL: https://arxiv.org/pdf/2402.11846
• Copy Paste: [[2402.11846]] UnlearnCanvas: A Stylized Image Dataset to Benchmark Machine Unlearning for Diffusion Models(https://arxiv.org/abs/2402.11846)
• Keywords: diffusion, generative
• Abstract: The rapid advancement of diffusion models (DMs) has not only transformed various real-world industries but has also introduced negative societal concerns, including the generation of harmful content, copyright disputes, and the rise of stereotypes and biases. To mitigate these issues, machine unlearning (MU) has emerged as a potential solution, demonstrating its ability to remove undesired generative capabilities of DMs in various applications. However, by examining existing MU evaluation methods, we uncover several key challenges that can result in incomplete, inaccurate, or biased evaluations for MU in DMs. To address them, we enhance the evaluation metrics for MU, including the introduction of an often-overlooked retainability measurement for DMs post-unlearning. Additionally, we introduce UnlearnCanvas, a comprehensive high-resolution stylized image dataset that facilitates us to evaluate the unlearning of artistic painting styles in conjunction with associated image objects. We show that this dataset plays a pivotal role in establishing a standardized and automated evaluation framework for MU techniques on DMs, featuring 7 quantitative metrics to address various aspects of unlearning effectiveness. Through extensive experiments, we benchmark 5 state-of-the-art MU methods, revealing novel insights into their pros and cons, and the underlying unlearning mechanisms. Furthermore, we demonstrate the potential of UnlearnCanvas to benchmark other generative modeling tasks, such as style transfer. The UnlearnCanvas dataset, benchmark, and the codes to reproduce all the results in this work can be found at https://github.com/OPTML-Group/UnlearnCanvas.

Title: ComFusion: Personalized Subject Generation in Multiple Specific Scenes From Single Image

• Authors: Yan Hong, Jianfu Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.11849
• Pdf URL: https://arxiv.org/pdf/2402.11849
• Copy Paste: [[2402.11849]] ComFusion: Personalized Subject Generation in Multiple Specific Scenes From Single Image(https://arxiv.org/abs/2402.11849)
• Keywords: diffusion
• Abstract: Recent advancements in personalizing text-to-image (T2I) diffusion models have shown the capability to generate images based on personalized visual concepts using a limited number of user-provided examples. However, these models often struggle with maintaining high visual fidelity, particularly in manipulating scenes as defined by textual inputs. Addressing this, we introduce ComFusion, a novel approach that leverages pretrained models generating composition of a few user-provided subject images and predefined-text scenes, effectively fusing visual-subject instances with textual-specific scenes, resulting in the generation of high-fidelity instances within diverse scenes. ComFusion integrates a class-scene prior preservation regularization, which leverages composites the subject class and scene-specific knowledge from pretrained models to enhance generation fidelity. Additionally, ComFusion uses coarse generated images, ensuring they align effectively with both the instance image and scene texts. Consequently, ComFusion maintains a delicate balance between capturing the essence of the subject and maintaining scene fidelity.Extensive evaluations of ComFusion against various baselines in T2I personalization have demonstrated its qualitative and quantitative superiority.

Title: Generative Semi-supervised Graph Anomaly Detection

• Authors: Hezhe Qiao, Qingsong Wen, Xiaoli Li, Ee-Peng Lim, Guansong Pang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.11887
• Pdf URL: https://arxiv.org/pdf/2402.11887
• Copy Paste: [[2402.11887]] Generative Semi-supervised Graph Anomaly Detection(https://arxiv.org/abs/2402.11887)
• Keywords: generative, anomaly
• Abstract: This work considers a practical semi-supervised graph anomaly detection (GAD) scenario, where part of the nodes in a graph are known to be normal, contrasting to the unsupervised setting in most GAD studies with a fully unlabeled graph. As expected, we find that having access to these normal nodes helps enhance the detection performance of existing unsupervised GAD methods when they are adapted to the semi-supervised setting. However, their utilization of these normal nodes is limited. In this paper, we propose a novel Generative GAD approach (GGAD) for the semi-supervised scenario to better exploit the normal nodes. The key idea is to generate outlier nodes that assimilate anomaly nodes in both local structure and node representations for providing effective negative node samples in training a discriminative one-class classifier. There have been many generative anomaly detection approaches, but they are designed for non-graph data, and as a result, they fail to take account of the graph structure information. Our approach tackles this problem by generating graph structure-aware outlier nodes that have asymmetric affinity separability from normal nodes while being enforced to achieve egocentric closeness to normal nodes in the node representation space. Comprehensive experiments on four real-world datasets are performed to establish a benchmark for semi-supervised GAD and show that GGAD substantially outperforms state-of-the-art unsupervised and semi-supervised GAD methods with varying numbers of training normal nodes. Code will be made available at https://github.com/mala-lab/GGAD.

Title: One2Avatar: Generative Implicit Head Avatar For Few-shot User Adaptation

• Authors: Zhixuan Yu, Ziqian Bai, Abhimitra Meka, Feitong Tan, Qiangeng Xu, Rohit Pandey, Sean Fanello, Hyun Soo Park, Yinda Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.11909
• Pdf URL: https://arxiv.org/pdf/2402.11909
• Copy Paste: [[2402.11909]] One2Avatar: Generative Implicit Head Avatar For Few-shot User Adaptation(https://arxiv.org/abs/2402.11909)
• Keywords: generative
• Abstract: Traditional methods for constructing high-quality, personalized head avatars from monocular videos demand extensive face captures and training time, posing a significant challenge for scalability. This paper introduces a novel approach to create high quality head avatar utilizing only a single or a few images per user. We learn a generative model for 3D animatable photo-realistic head avatar from a multi-view dataset of expressions from 2407 subjects, and leverage it as a prior for creating personalized avatar from few-shot images. Different from previous 3D-aware face generative models, our prior is built with a 3DMM-anchored neural radiance field backbone, which we show to be more effective for avatar creation through auto-decoding based on few-shot inputs. We also handle unstable 3DMM fitting by jointly optimizing the 3DMM fitting and camera calibration that leads to better few-shot adaptation. Our method demonstrates compelling results and outperforms existing state-of-the-art methods for few-shot avatar adaptation, paving the way for more efficient and personalized avatar creation.

Title: PhySU-Net: Long Temporal Context Transformer for rPPG with Self-Supervised Pre-training

• Authors: Marko Savic, Guoying Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.11913
• Pdf URL: https://arxiv.org/pdf/2402.11913
• Copy Paste: [[2402.11913]] PhySU-Net: Long Temporal Context Transformer for rPPG with Self-Supervised Pre-training(https://arxiv.org/abs/2402.11913)
• Keywords: self-supervised
• Abstract: Remote photoplethysmography (rPPG) is a promising technology that consists of contactless measuring of cardiac activity from facial videos. Most recent approaches utilize convolutional networks with limited temporal modeling capability or ignore long temporal context. Supervised rPPG methods are also severely limited by scarce data availability. In this work, we propose PhySU-Net, the first long spatial-temporal map rPPG transformer network and a self-supervised pre-training strategy that exploits unlabeled data to improve our model. Our strategy leverages traditional methods and image masking to provide pseudo-labels for self-supervised pre-training. Our model is tested on two public datasets (OBF and VIPL-HR) and shows superior performance in supervised training. Furthermore, we demonstrate that our self-supervised pre-training strategy further improves our model's performance by leveraging representations learned from unlabeled data.

Title: A Generative Pre-Training Framework for Spatio-Temporal Graph Transfer Learning

• Authors: Yuan Yuan, Chenyang Shao, Jingtao Ding, Depeng Jin, Yong Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.11922
• Pdf URL: https://arxiv.org/pdf/2402.11922
• Copy Paste: [[2402.11922]] A Generative Pre-Training Framework for Spatio-Temporal Graph Transfer Learning(https://arxiv.org/abs/2402.11922)
• Keywords: diffusion, generative
• Abstract: Spatio-temporal graph (STG) learning is foundational for smart city applications, yet it is often hindered by data scarcity in many cities and regions. To bridge this gap, we propose a novel generative pre-training framework, GPDiff, for STG transfer learning. Unlike conventional approaches that heavily rely on common feature extraction or intricate transfer learning designs, our solution takes a novel approach by performing generative pre-training on a collection of model parameters optimized with data from source cities. We recast STG transfer learning as pre-training a generative hypernetwork, which generates tailored model parameters guided by prompts, allowing for adaptability to diverse data distributions and city-specific characteristics. GPDiff employs a diffusion model with a transformer-based denoising network, which is model-agnostic to integrate with powerful STG models. By addressing challenges arising from data gaps and the complexity of generalizing knowledge across cities, our framework consistently outperforms state-of-the-art baselines on multiple real-world datasets for tasks such as traffic speed prediction and crowd flow prediction. The implementation of our approach is available: https://github.com/PLUTO-SCY/GPDiff.

Title: DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation

• Authors: Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, Xin Tong
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2402.11929
• Pdf URL: https://arxiv.org/pdf/2402.11929
• Copy Paste: [[2402.11929]] DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation(https://arxiv.org/abs/2402.11929)
• Keywords: diffusion
• Abstract: This paper presents a novel method for exerting fine-grained lighting control during text-driven diffusion-based image generation. While existing diffusion models already have the ability to generate images under any lighting condition, without additional guidance these models tend to correlate image content and lighting. Moreover, text prompts lack the necessary expressional power to describe detailed lighting setups. To provide the content creator with fine-grained control over the lighting during image generation, we augment the text-prompt with detailed lighting information in the form of radiance hints, i.e., visualizations of the scene geometry with a homogeneous canonical material under the target lighting. However, the scene geometry needed to produce the radiance hints is unknown. Our key observation is that we only need to guide the diffusion process, hence exact radiance hints are not necessary; we only need to point the diffusion model in the right direction. Based on this observation, we introduce a three stage method for controlling the lighting during image generation. In the first stage, we leverage a standard pretrained diffusion model to generate a provisional image under uncontrolled lighting. Next, in the second stage, we resynthesize and refine the foreground object in the generated image by passing the target lighting to a refined diffusion model, named DiLightNet, using radiance hints computed on a coarse shape of the foreground object inferred from the provisional image. To retain the texture details, we multiply the radiance hints with a neural encoding of the provisional synthesized image before passing it to DiLightNet. Finally, in the third stage, we resynthesize the background to be consistent with the lighting on the foreground object. We demonstrate and validate our lighting controlled diffusion model on a variety of text prompts and lighting conditions.

Title: SLADE: Detecting Dynamic Anomalies in Edge Streams without Labels via Self-Supervised Learning

• Authors: Jongha Lee, Sunwoo Kim, Kijung Shin
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2402.11933
• Pdf URL: https://arxiv.org/pdf/2402.11933
• Copy Paste: [[2402.11933]] SLADE: Detecting Dynamic Anomalies in Edge Streams without Labels via Self-Supervised Learning(https://arxiv.org/abs/2402.11933)
• Keywords: self-supervised, anomaly
• Abstract: To detect anomalies in real-world graphs, such as social, email, and financial networks, various approaches have been developed. While they typically assume static input graphs, most real-world graphs grow over time, naturally represented as edge streams. In this context, we aim to achieve three goals: (a) instantly detecting anomalies as they occur, (b) adapting to dynamically changing states, and (c) handling the scarcity of dynamic anomaly labels. In this paper, we propose SLADE (Self-supervised Learning for Anomaly Detection in Edge Streams) for rapid detection of dynamic anomalies in edge streams, without relying on labels. SLADE detects the shifts of nodes into abnormal states by observing deviations in their interaction patterns over time. To this end, it trains a deep neural network to perform two self-supervised tasks: (a) minimizing drift in node representations and (b) generating long-term interaction patterns from short-term ones. Failure in these tasks for a node signals its deviation from the norm. Notably, the neural network and tasks are carefully designed so that all required operations can be performed in constant time (w.r.t. the graph size) in response to each new edge in the input stream. In dynamic anomaly detection across four real-world datasets, SLADE outperforms nine competing methods, even those leveraging label supervision.

Title: Privacy-Preserving Low-Rank Adaptation for Latent Diffusion Models

• Authors: Zihao Luo, Xilie Xu, Feng Liu, Yun Sing Koh, Di Wang, Jingfeng Zhang
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.11989
• Pdf URL: https://arxiv.org/pdf/2402.11989
• Copy Paste: [[2402.11989]] Privacy-Preserving Low-Rank Adaptation for Latent Diffusion Models(https://arxiv.org/abs/2402.11989)
• Keywords: diffusion
• Abstract: Low-rank adaptation (LoRA) is an efficient strategy for adapting latent diffusion models (LDMs) on a training dataset to generate specific objects by minimizing the adaptation loss. However, adapted LDMs via LoRA are vulnerable to membership inference (MI) attacks that can judge whether a particular data point belongs to private training datasets, thus facing severe risks of privacy leakage. To defend against MI attacks, we make the first effort to propose a straightforward solution: privacy-preserving LoRA (PrivateLoRA). PrivateLoRA is formulated as a min-max optimization problem where a proxy attack model is trained by maximizing its MI gain while the LDM is adapted by minimizing the sum of the adaptation loss and the proxy attack model's MI gain. However, we empirically disclose that PrivateLoRA has the issue of unstable optimization due to the large fluctuation of the gradient scale which impedes adaptation. To mitigate this issue, we propose Stable PrivateLoRA that adapts the LDM by minimizing the ratio of the adaptation loss to the MI gain, which implicitly rescales the gradient and thus stabilizes the optimization. Our comprehensive empirical results corroborate that adapted LDMs via Stable PrivateLoRA can effectively defend against MI attacks while generating high-quality images. Our code is available at https://github.com/WilliamLUO0/StablePrivateLoRA.

Title: ISCUTE: Instance Segmentation of Cables Using Text Embedding

• Authors: Shir Kozlovsky, Omkar Joglekar, Dotan Di Castro
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.11996
• Pdf URL: https://arxiv.org/pdf/2402.11996
• Copy Paste: [[2402.11996]] ISCUTE: Instance Segmentation of Cables Using Text Embedding(https://arxiv.org/abs/2402.11996)
• Keywords: foundation model
• Abstract: In the field of robotics and automation, conventional object recognition and instance segmentation methods face a formidable challenge when it comes to perceiving Deformable Linear Objects (DLOs) like wires, cables, and flexible tubes. This challenge arises primarily from the lack of distinct attributes such as shape, color, and texture, which calls for tailored solutions to achieve precise identification. In this work, we propose a foundation model-based DLO instance segmentation technique that is text-promptable and user-friendly. Specifically, our approach combines the text-conditioned semantic segmentation capabilities of CLIPSeg model with the zero-shot generalization capabilities of Segment Anything Model (SAM). We show that our method exceeds SOTA performance on DLO instance segmentation, achieving a mIoU of $91.21\%$. We also introduce a rich and diverse DLO-specific dataset for instance segmentation.

Title: Direct Consistency Optimization for Compositional Text-to-Image Personalization

• Authors: Kyungmin Lee, Sangkyung Kwak, Kihyuk Sohn, Jinwoo Shin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12004
• Pdf URL: https://arxiv.org/pdf/2402.12004
• Copy Paste: [[2402.12004]] Direct Consistency Optimization for Compositional Text-to-Image Personalization(https://arxiv.org/abs/2402.12004)
• Keywords: diffusion
• Abstract: Text-to-image (T2I) diffusion models, when fine-tuned on a few personal images, are able to generate visuals with a high degree of consistency. However, they still lack in synthesizing images of different scenarios or styles that are possible in the original pretrained models. To address this, we propose to fine-tune the T2I model by maximizing consistency to reference images, while penalizing the deviation from the pretrained model. We devise a novel training objective for T2I diffusion models that minimally fine-tunes the pretrained model to achieve consistency. Our method, dubbed \emph{Direct Consistency Optimization}, is as simple as regular diffusion loss, while significantly enhancing the compositionality of personalized T2I models. Also, our approach induces a new sampling method that controls the tradeoff between image fidelity and prompt fidelity. Lastly, we emphasize the necessity of using a comprehensive caption for reference images to further enhance the image-text alignment. We show the efficacy of the proposed method on the T2I personalization for subject, style, or both. In particular, our method results in a superior Pareto frontier to the baselines. Generated examples and codes are in our project page( https://dco-t2i.github.io/).

Title: A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change

• Authors: Francesco Periti, Nina Tahmasebi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12011
• Pdf URL: https://arxiv.org/pdf/2402.12011
• Copy Paste: [[2402.12011]] A Systematic Comparison of Contextualized Word Embeddings for Lexical Semantic Change(https://arxiv.org/abs/2402.12011)
• Keywords: in-context
• Abstract: Contextualized embeddings are the preferred tool for modeling Lexical Semantic Change (LSC). Current evaluations typically focus on a specific task known as Graded Change Detection (GCD). However, performance comparison across work are often misleading due to their reliance on diverse settings. In this paper, we evaluate state-of-the-art models and approaches for GCD under equal conditions. We further break the LSC problem into Word-in-Context (WiC) and Word Sense Induction (WSI) tasks, and compare models across these different levels. Our evaluation is performed across different languages on eight available benchmarks for LSC, and shows that (i) APD outperforms other approaches for GCD; (ii) XL-LEXEME outperforms other contextualized models for WiC, WSI, and GCD, while being comparable to GPT-4; (iii) there is a clear need for improving the modeling of word meanings, as well as focus on how, when, and why these meanings change, rather than solely focusing on the extent of semantic change.

Title: Distilling Large Language Models for Text-Attributed Graph Learning

• Authors: Bo Pan, Zheng Zhang, Yifei Zhang, Yuntong Hu, Liang Zhao
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12022
• Pdf URL: https://arxiv.org/pdf/2402.12022
• Copy Paste: [[2402.12022]] Distilling Large Language Models for Text-Attributed Graph Learning(https://arxiv.org/abs/2402.12022)
• Keywords: generative
• Abstract: Text-Attributed Graphs (TAGs) are graphs of connected textual documents. Graph models can efficiently learn TAGs, but their training heavily relies on human-annotated labels, which are scarce or even unavailable in many applications. Large language models (LLMs) have recently demonstrated remarkable capabilities in few-shot and zero-shot TAG learning, but they suffer from scalability, cost, and privacy issues. Therefore, in this work, we focus on synergizing LLMs and graph models with their complementary strengths by distilling the power of LLMs to a local graph model on TAG learning. To address the inherent gaps between LLMs (generative models for texts) and graph models (discriminative models for graphs), we propose first to let LLMs teach an interpreter with rich textual rationale and then let a student model mimic the interpreter's reasoning without LLMs' textual rationale. Extensive experiments validate the efficacy of our proposed framework.

Title: Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?

• Authors: Marco Gaido, Sara Papi, Matteo Negri, Luisa Bentivogli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12025
• Pdf URL: https://arxiv.org/pdf/2402.12025
• Copy Paste: [[2402.12025]] Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?(https://arxiv.org/abs/2402.12025)
• Keywords: foundation model
• Abstract: The field of natural language processing (NLP) has recently witnessed a transformative shift with the emergence of foundation models, particularly Large Language Models (LLMs) that have revolutionized text-based NLP. This paradigm has extended to other modalities, including speech, where researchers are actively exploring the combination of Speech Foundation Models (SFMs) and LLMs into single, unified models capable of addressing multimodal tasks. Among such tasks, this paper focuses on speech-to-text translation (ST). By examining the published papers on the topic, we propose a unified view of the architectural solutions and training strategies presented so far, highlighting similarities and differences among them. Based on this examination, we not only organize the lessons learned but also show how diverse settings and evaluation approaches hinder the identification of the best-performing solution for each architectural building block and training choice. Lastly, we outline recommendations for future works on the topic aimed at better understanding the strengths and weaknesses of the SFM+LLM solutions for ST.

Title: Self-AMPLIFY: Improving Small Language Models with Self Post Hoc Explanations

• Authors: Milan Bhan, Jean-Noel Vittaut, Nicolas Chesneau, Marie-Jeanne Lesot
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12038
• Pdf URL: https://arxiv.org/pdf/2402.12038
• Copy Paste: [[2402.12038]] Self-AMPLIFY: Improving Small Language Models with Self Post Hoc Explanations(https://arxiv.org/abs/2402.12038)
• Keywords: in-context
• Abstract: Incorporating natural language rationales in the prompt and In-Context Learning (ICL) has led to a significant improvement of Large Language Models (LLMs) performance. However, rationales currently require human-annotation or the use of auxiliary proxy models to target promising samples or generate high-quality rationales. In this work, we propose Self-AMPLIFY to generate automatically rationales from post hoc explanation methods applied to Small Language Models (SLMs) to improve their own performance. Self-AMPLIFY is a 3-step method that targets samples, generates rationales and builds a final prompt to leverage ICL. Self-AMPLIFY performance is evaluated on two SLMs and two datasets requiring reasoning abilities: these experiments show that Self-AMPLIFY achieves good results against competitors. Self-AMPLIFY is the first method to apply post hoc explanation methods to SLM to generate rationales to improve their own performance in a fully automated manner.

Title: A Lightweight Parallel Framework for Blind Image Quality Assessment

• Authors: Qunyue Huang, Bin Fang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12043
• Pdf URL: https://arxiv.org/pdf/2402.12043
• Copy Paste: [[2402.12043]] A Lightweight Parallel Framework for Blind Image Quality Assessment(https://arxiv.org/abs/2402.12043)
• Keywords: self-supervised
• Abstract: Existing blind image quality assessment (BIQA) methods focus on designing complicated networks based on convolutional neural networks (CNNs) or transformer. In addition, some BIQA methods enhance the performance of the model in a two-stage training manner. Despite the significant advancements, these methods remarkably raise the parameter count of the model, thus requiring more training time and computational resources. To tackle the above issues, we propose a lightweight parallel framework (LPF) for BIQA. First, we extract the visual features using a pre-trained feature extraction network. Furthermore, we construct a simple yet effective feature embedding network (FEN) to transform the visual features, aiming to generate the latent representations that contain salient distortion information. To improve the robustness of the latent representations, we present two novel self-supervised subtasks, including a sample-level category prediction task and a batch-level quality comparison task. The sample-level category prediction task is presented to help the model with coarse-grained distortion perception. The batch-level quality comparison task is formulated to enhance the training data and thus improve the robustness of the latent representations. Finally, the latent representations are fed into a distortion-aware quality regression network (DaQRN), which simulates the human vision system (HVS) and thus generates accurate quality scores. Experimental results on multiple benchmark datasets demonstrate that the proposed method achieves superior performance over state-of-the-art approaches. Moreover, extensive analyses prove that the proposed method has lower computational complexity and faster convergence speed.

Title: Do Large Language Models Understand Logic or Just Mimick Context?

• Authors: Junbing Yan, Chengyu Wang, Jun Huang, Wei Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12091
• Pdf URL: https://arxiv.org/pdf/2402.12091
• Copy Paste: [[2402.12091]] Do Large Language Models Understand Logic or Just Mimick Context?(https://arxiv.org/abs/2402.12091)
• Keywords: in-context
• Abstract: Over the past few years, the abilities of large language models (LLMs) have received extensive attention, which have performed exceptionally well in complicated scenarios such as logical reasoning and symbolic inference. A significant factor contributing to this progress is the benefit of in-context learning and few-shot prompting. However, the reasons behind the success of such models using contextual reasoning have not been fully explored. Do LLMs have understand logical rules to draw inferences, or do they ``guess'' the answers by learning a type of probabilistic mapping through context? This paper investigates the reasoning capabilities of LLMs on two logical reasoning datasets by using counterfactual methods to replace context text and modify logical concepts. Based on our analysis, it is found that LLMs do not truly understand logical rules; rather, in-context learning has simply enhanced the likelihood of these models arriving at the correct answers. If one alters certain words in the context text or changes the concepts of logical terms, the outputs of LLMs can be significantly disrupted, leading to counter-intuitive responses. This work provides critical insights into the limitations of LLMs, underscoring the need for more robust mechanisms to ensure reliable logical reasoning in LLMs.

Title: Human Video Translation via Query Warping

• Authors: Haiming Zhu, Yangyang Xu, Shengfeng He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12099
• Pdf URL: https://arxiv.org/pdf/2402.12099
• Copy Paste: [[2402.12099]] Human Video Translation via Query Warping(https://arxiv.org/abs/2402.12099)
• Keywords: diffusion
• Abstract: In this paper, we present QueryWarp, a novel framework for temporally coherent human motion video translation. Existing diffusion-based video editing approaches that rely solely on key and value tokens to ensure temporal consistency, which scarifies the preservation of local and structural regions. In contrast, we aim to consider complementary query priors by constructing the temporal correlations among query tokens from different frames. Initially, we extract appearance flows from source poses to capture continuous human foreground motion. Subsequently, during the denoising process of the diffusion model, we employ appearance flows to warp the previous frame's query token, aligning it with the current frame's query. This query warping imposes explicit constraints on the outputs of self-attention layers, effectively guaranteeing temporally coherent translation. We perform experiments on various human motion video translation tasks, and the results demonstrate that our QueryWarp framework surpasses state-of-the-art methods both qualitatively and quantitatively.

Title: Groot: Adversarial Testing for Generative Text-to-Image Models with Tree-based Semantic Transformation

• Authors: Yi Liu, Guowei Yang, Gelei Deng, Feiyue Chen, Yuqi Chen, Ling Shi, Tianwei Zhang, Yang Liu
• Subjects: cs.CL, cs.AI, cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.12100
• Pdf URL: https://arxiv.org/pdf/2402.12100
• Copy Paste: [[2402.12100]] Groot: Adversarial Testing for Generative Text-to-Image Models with Tree-based Semantic Transformation(https://arxiv.org/abs/2402.12100)
• Keywords: generative
• Abstract: With the prevalence of text-to-image generative models, their safety becomes a critical concern. adversarial testing techniques have been developed to probe whether such models can be prompted to produce Not-Safe-For-Work (NSFW) content. However, existing solutions face several challenges, including low success rate and inefficiency. We introduce Groot, the first automated framework leveraging tree-based semantic transformation for adversarial testing of text-to-image models. Groot employs semantic decomposition and sensitive element drowning strategies in conjunction with LLMs to systematically refine adversarial prompts. Our comprehensive evaluation confirms the efficacy of Groot, which not only exceeds the performance of current state-of-the-art approaches but also achieves a remarkable success rate (93.66%) on leading text-to-image models such as DALL-E 3 and Midjourney.

Title: Unsupervised LLM Adaptation for Question Answering

• Authors: Kuniaki Saito, Kihyuk Sohn, Chen-Yu Lee, Yoshitaka Ushiku
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12170
• Pdf URL: https://arxiv.org/pdf/2402.12170
• Copy Paste: [[2402.12170]] Unsupervised LLM Adaptation for Question Answering(https://arxiv.org/abs/2402.12170)
• Keywords: self-supervised
• Abstract: Large language models (LLM) learn diverse knowledge present in the large-scale training dataset via self-supervised training. Followed by instruction-tuning, LLM acquires the ability to return correct information for diverse questions. However, adapting these pre-trained LLMs to new target domains, such as different organizations or periods, for the question-answering (QA) task incurs a substantial annotation cost. To tackle this challenge, we propose a novel task, unsupervised LLM adaptation for question answering. In this task, we leverage a pre-trained LLM, a publicly available QA dataset (source data), and unlabeled documents from the target domain. Our goal is to learn LLM that can answer questions about the target domain. We introduce one synthetic and two real datasets to evaluate models fine-tuned on the source and target data, and reveal intriguing insights; (i) fine-tuned models exhibit the ability to provide correct answers for questions about the target domain even though they do not see any questions about the information described in the unlabeled documents, but (ii) they have difficulties in accessing information located in the middle or at the end of documents, and (iii) this challenge can be partially mitigated by replacing input tokens with random ones during adaptation.

Title: ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning

• Authors: Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, Yu Qiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12185
• Pdf URL: https://arxiv.org/pdf/2402.12185
• Copy Paste: [[2402.12185]] ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning(https://arxiv.org/abs/2402.12185)
• Keywords: foundation model
• Abstract: Recently, many versatile Multi-modal Large Language Models (MLLMs) have emerged continuously. However, their capacity to query information depicted in visual charts and engage in reasoning based on the queried contents remains under-explored. In this paper, to comprehensively and rigorously benchmark the ability of the off-the-shelf MLLMs in the chart domain, we construct ChartX, a multi-modal evaluation set covering 18 chart types, 7 chart tasks, 22 disciplinary topics, and high-quality chart data. Besides, we develop ChartVLM to offer a new perspective on handling multi-modal tasks that strongly depend on interpretable patterns, such as reasoning tasks in the field of charts or geometric images. We evaluate the chart-related ability of mainstream MLLMs and our ChartVLM on the proposed ChartX evaluation set. Extensive experiments demonstrate that ChartVLM surpasses both versatile and chart-related large models, achieving results comparable to GPT-4V. We believe that our study can pave the way for further exploration in creating a more comprehensive chart evaluation set and developing more interpretable multi-modal models. Both ChartX and ChartVLM are available at: https://github.com/UniModal4Reasoning/ChartVLM

Title: Adversarial Feature Alignment: Balancing Robustness and Accuracy in Deep Learning via Adversarial Training

• Authors: Leo Hyun Park, Jaeuk Kim, Myung Gyo Oh, Jaewoo Park, Taekyoung Kwon
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12187
• Pdf URL: https://arxiv.org/pdf/2402.12187
• Copy Paste: [[2402.12187]] Adversarial Feature Alignment: Balancing Robustness and Accuracy in Deep Learning via Adversarial Training(https://arxiv.org/abs/2402.12187)
• Keywords: diffusion
• Abstract: Deep learning models continue to advance in accuracy, yet they remain vulnerable to adversarial attacks, which often lead to the misclassification of adversarial examples. Adversarial training is used to mitigate this problem by increasing robustness against these attacks. However, this approach typically reduces a model's standard accuracy on clean, non-adversarial samples. The necessity for deep learning models to balance both robustness and accuracy for security is obvious, but achieving this balance remains challenging, and the underlying reasons are yet to be clarified. This paper proposes a novel adversarial training method called Adversarial Feature Alignment (AFA), to address these problems. Our research unveils an intriguing insight: misalignment within the feature space often leads to misclassification, regardless of whether the samples are benign or adversarial. AFA mitigates this risk by employing a novel optimization algorithm based on contrastive learning to alleviate potential feature misalignment. Through our evaluations, we demonstrate the superior performance of AFA. The baseline AFA delivers higher robust accuracy than previous adversarial contrastive learning methods while minimizing the drop in clean accuracy to 1.86% and 8.91% on CIFAR10 and CIFAR100, respectively, in comparison to cross-entropy. We also show that joint optimization of AFA and TRADES, accompanied by data augmentation using a recent diffusion model, achieves state-of-the-art accuracy and robustness.

Title: Polarization of Autonomous Generative AI Agents Under Echo Chambers

• Authors: Masaya Ohagi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12212
• Pdf URL: https://arxiv.org/pdf/2402.12212
• Copy Paste: [[2402.12212]] Polarization of Autonomous Generative AI Agents Under Echo Chambers(https://arxiv.org/abs/2402.12212)
• Keywords: generative
• Abstract: Online social networks often create echo chambers where people only hear opinions reinforcing their beliefs. An echo chamber often generates polarization, leading to conflicts caused by people with radical opinions, such as the January 6, 2021, attack on the US Capitol. The echo chamber has been viewed as a human-specific problem, but this implicit assumption is becoming less reasonable as large language models, such as ChatGPT, acquire social abilities. In response to this situation, we investigated the potential for polarization to occur among a group of autonomous AI agents based on generative language models in an echo chamber environment. We had AI agents discuss specific topics and analyzed how the group's opinions changed as the discussion progressed. As a result, we found that the group of agents based on ChatGPT tended to become polarized in echo chamber environments. The analysis of opinion transitions shows that this result is caused by ChatGPT's high prompt understanding ability to update its opinion by considering its own and surrounding agents' opinions. We conducted additional experiments to investigate under what specific conditions AI agents tended to polarize. As a result, we identified factors that strongly influence polarization, such as the agent's persona. These factors should be monitored to prevent the polarization of AI agents.

Title: AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

• Authors: Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yugang Jiang, Xipeng Qiu
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12226
• Pdf URL: https://arxiv.org/pdf/2402.12226
• Copy Paste: [[2402.12226]] AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling(https://arxiv.org/abs/2402.12226)
• Keywords: generative
• Abstract: We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images, and music. AnyGPT can be trained stably without any alterations to the current large language model (LLM) architecture or training paradigms. Instead, it relies exclusively on data-level preprocessing, facilitating the seamless integration of new modalities into LLMs, akin to the incorporation of new languages. We build a multimodal text-centric dataset for multimodal alignment pre-training. Utilizing generative models, we synthesize the first large-scale any-to-any multimodal instruction dataset. It consists of 108k samples of multi-turn conversations that intricately interweave various modalities, thus equipping the model to handle arbitrary combinations of multimodal inputs and outputs. Experimental results demonstrate that AnyGPT is capable of facilitating any-to-any multimodal conversation while achieving performance comparable to specialized models across all modalities, proving that discrete representations can effectively and conveniently unify multiple modalities within a language model. Demos are shown in https://junzhan2000.github.io/AnyGPT.github.io/

Title: Diffusion Tempering Improves Parameter Estimation with Probabilistic Integrators for Ordinary Differential Equations

• Authors: Jonas Beck, Nathanael Bosch, Michael Deistler, Kyra L. Kadhim, Jakob H. Macke, Philipp Hennig, Philipp Berens
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12231
• Pdf URL: https://arxiv.org/pdf/2402.12231
• Copy Paste: [[2402.12231]] Diffusion Tempering Improves Parameter Estimation with Probabilistic Integrators for Ordinary Differential Equations(https://arxiv.org/abs/2402.12231)
• Keywords: diffusion
• Abstract: Ordinary differential equations (ODEs) are widely used to describe dynamical systems in science, but identifying parameters that explain experimental measurements is challenging. In particular, although ODEs are differentiable and would allow for gradient-based parameter optimization, the nonlinear dynamics of ODEs often lead to many local minima and extreme sensitivity to initial conditions. We therefore propose diffusion tempering, a novel regularization technique for probabilistic numerical methods which improves convergence of gradient-based parameter optimization in ODEs. By iteratively reducing a noise parameter of the probabilistic integrator, the proposed method converges more reliably to the true parameters. We demonstrate that our method is effective for dynamical systems of different complexity and show that it obtains reliable parameter estimates for a Hodgkin-Huxley model with a practically relevant number of parameters.

Title: Task-Oriented Dialogue with In-Context Learning

• Authors: Tom Bocklisch, Thomas Werkmeister, Daksh Varshneya, Alan Nichol
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.12234
• Pdf URL: https://arxiv.org/pdf/2402.12234
• Copy Paste: [[2402.12234]] Task-Oriented Dialogue with In-Context Learning(https://arxiv.org/abs/2402.12234)
• Keywords: in-context
• Abstract: We describe a system for building task-oriented dialogue systems combining the in-context learning abilities of large language models (LLMs) with the deterministic execution of business logic. LLMs are used to translate between the surface form of the conversation and a domain-specific language (DSL) which is used to progress the business logic. We compare our approach to the intent-based NLU approach predominantly used in industry today. Our experiments show that developing chatbots with our system requires significantly less effort than established approaches, that these chatbots can successfully navigate complex dialogues which are extremely challenging for NLU-based systems, and that our system has desirable properties for scaling task-oriented dialogue systems to a large number of tasks. We make our implementation available for use and further study.

Title: Mixed Gaussian Flow for Diverse Trajectory Prediction

• Authors: Jiahe Chen, Jinkun Cao, Dahua Lin, Kris Kitani, Jiangmiao Pang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12238
• Pdf URL: https://arxiv.org/pdf/2402.12238
• Copy Paste: [[2402.12238]] Mixed Gaussian Flow for Diverse Trajectory Prediction(https://arxiv.org/abs/2402.12238)
• Keywords: generative
• Abstract: Existing trajectory prediction studies intensively leverage generative models. Normalizing flow is one of the genres with the advantage of being invertible to derive the probability density of predicted trajectories. However, mapping from a standard Gaussian by a flow-based model hurts the capacity to capture complicated patterns of trajectories, ignoring the under-represented motion intentions in the training data. To solve the problem, we propose a flow-based model to transform a mixed Gaussian prior into the future trajectory manifold. The model shows a better capacity for generating diverse trajectory patterns. Also, by associating each sub-Gaussian with a certain subspace of trajectories, we can generate future trajectories with controllable motion intentions. In such a fashion, the flow-based model is not encouraged to simply seek the most likelihood of the intended manifold anymore but a family of controlled manifolds with explicit interpretability. Our proposed method is demonstrated to show state-of-the-art performance in the quantitative evaluation of sampling well-aligned trajectories in top-M generated candidates. We also demonstrate that it can generate diverse, controllable, and out-of-distribution trajectories. Code is available at https://github.com/mulplue/MGF.

Title: Synthetic location trajectory generation using categorical diffusion models

• Authors: Simon Dirmeier, Ye Hong, Fernando Perez-Cruz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.12242
• Pdf URL: https://arxiv.org/pdf/2402.12242
• Copy Paste: [[2402.12242]] Synthetic location trajectory generation using categorical diffusion models(https://arxiv.org/abs/2402.12242)
• Keywords: diffusion, generative
• Abstract: Diffusion probabilistic models (DPMs) have rapidly evolved to be one of the predominant generative models for the simulation of synthetic data, for instance, for computer vision, audio, natural language processing, or biomolecule generation. Here, we propose using DPMs for the generation of synthetic individual location trajectories (ILTs) which are sequences of variables representing physical locations visited by individuals. ILTs are of major importance in mobility research to understand the mobility behavior of populations and to ultimately inform political decision-making. We represent ILTs as multi-dimensional categorical random variables and propose to model their joint distribution using a continuous DPM by first applying the diffusion process in a continuous unconstrained space and then mapping the continuous variables into a discrete space. We demonstrate that our model can synthesize realistic ILPs by comparing conditionally and unconditionally generated sequences to real-world ILPs from a GNSS tracking data set which suggests the potential use of our model for synthetic data generation, for example, for benchmarking models used in mobility research.

Title: Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships

• Authors: Sebastian Koch, Narunas Vaskevicius, Mirco Colosi, Pedro Hermosilla, Timo Ropinski
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12259
• Pdf URL: https://arxiv.org/pdf/2402.12259
• Copy Paste: [[2402.12259]] Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships(https://arxiv.org/abs/2402.12259)
• Keywords: foundation model
• Abstract: Current approaches for 3D scene graph prediction rely on labeled datasets to train models for a fixed set of known object classes and relationship categories. We present Open3DSG, an alternative approach to learn 3D scene graph prediction in an open world without requiring labeled scene graph data. We co-embed the features from a 3D scene graph prediction backbone with the feature space of powerful open world 2D vision language foundation models. This enables us to predict 3D scene graphs from 3D point clouds in a zero-shot manner by querying object classes from an open vocabulary and predicting the inter-object relationships from a grounded LLM with scene graph features and queried object classes as context. Open3DSG is the first 3D point cloud method to predict not only explicit open-vocabulary object classes, but also open-set relationships that are not limited to a predefined label set, making it possible to express rare as well as specific objects and relationships in the predicted 3D scene graph. Our experiments show that Open3DSG is effective at predicting arbitrary object classes as well as their complex inter-object relationships describing spatial, supportive, semantic and comparative relationships.

Title: Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks

• Authors: Nadezhda Chirkova, Vassilina Nikoulina
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.12279
• Pdf URL: https://arxiv.org/pdf/2402.12279
• Copy Paste: [[2402.12279]] Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks(https://arxiv.org/abs/2402.12279)
• Keywords: generative
• Abstract: Zero-shot cross-lingual generation implies finetuning of the multilingual pretrained language model on a generation task in one language and then using it to make predictions for this task in other languages. Previous works notice a frequent problem of generation in a wrong language and propose approaches to address it, usually using mT5 as a backbone model. In this work we compare various approaches proposed from the literature in unified settings, also including alternative backbone models, namely mBART and NLLB-200. We first underline the importance of tuning learning rate used for finetuning, which helps to substantially alleviate the problem of generation in the wrong language. Then, we show that with careful learning rate tuning, the simple full finetuning of the model acts as a very strong baseline and alternative approaches bring only marginal improvements. Finally, we find that mBART performs similarly to mT5 of the same size, and NLLB-200 can be competitive in some cases. Our final models reach the performance of the approach based on data translation which is usually considered as an upper baseline for zero-shot cross-lingual generation.

Title: Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models

• Authors: Christian Schlarmann, Naman Deep Singh, Francesco Croce, Matthias Hein
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.12336
• Pdf URL: https://arxiv.org/pdf/2402.12336
• Copy Paste: [[2402.12336]] Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models(https://arxiv.org/abs/2402.12336)
• Keywords: foundation model
• Abstract: Multi-modal foundation models like OpenFlamingo, LLaVA, and GPT-4 are increasingly used for various real-world tasks. Prior work has shown that these models are highly vulnerable to adversarial attacks on the vision modality. These attacks can be leveraged to spread fake information or defraud users, and thus pose a significant risk, which makes the robustness of large multi-modal foundation models a pressing problem. The CLIP model, or one of its variants, is used as a frozen vision encoder in many vision-language models (VLMs), e.g. LLaVA and OpenFlamingo. We propose an unsupervised adversarial fine-tuning scheme to obtain a robust CLIP vision encoder, which yields robustness on all vision down-stream tasks (VLMs, zero-shot classification) that rely on CLIP. In particular, we show that stealth-attacks on users of VLMs by a malicious third party providing manipulated images are no longer possible once one replaces the original CLIP model with our robust one. No retraining or fine-tuning of the VLM is required. The code and robust models are available at https://github.com/chs20/RobustVLM

Title: FiT: Flexible Vision Transformer for Diffusion Model

• Authors: Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, Lei Bai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.12376
• Pdf URL: https://arxiv.org/pdf/2402.12376
• Copy Paste: [[2402.12376]] FiT: Flexible Vision Transformer for Diffusion Model(https://arxiv.org/abs/2402.12376)
• Keywords: diffusion
• Abstract: Nature is infinitely resolution-free. In the context of this reality, existing diffusion models, such as Diffusion Transformers, often face challenges when processing image resolutions outside of their trained domain. To overcome this limitation, we present the Flexible Vision Transformer (FiT), a transformer architecture specifically designed for generating images with unrestricted resolutions and aspect ratios. Unlike traditional methods that perceive images as static-resolution grids, FiT conceptualizes images as sequences of dynamically-sized tokens. This perspective enables a flexible training strategy that effortlessly adapts to diverse aspect ratios during both training and inference phases, thus promoting resolution generalization and eliminating biases induced by image cropping. Enhanced by a meticulously adjusted network structure and the integration of training-free extrapolation techniques, FiT exhibits remarkable flexibility in resolution extrapolation generation. Comprehensive experiments demonstrate the exceptional performance of FiT across a broad range of resolutions, showcasing its effectiveness both within and beyond its training resolution distribution. Repository available at https://github.com/whlzy/FiT.