2024-01-17

Title: Human-Instruction-Free LLM Self-Alignment with Limited Samples

• Authors: Hongyi Guo, Yuanshun Yao, Wei Shen, Jiaheng Wei, Xiaoying Zhang, Zhaoran Wang, Yang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06785
• Pdf URL: https://arxiv.org/pdf/2401.06785
• Copy Paste: [[2401.06785]] Human-Instruction-Free LLM Self-Alignment with Limited Samples(https://arxiv.org/abs/2401.06785)
• Keywords: in-context
• Abstract: Aligning large language models (LLMs) with human values is a vital task for LLM practitioners. Current alignment techniques have several limitations: (1) requiring a large amount of annotated data; (2) demanding heavy human involvement; (3) lacking a systematic mechanism to continuously improve. In this work, we study aligning LLMs to a new domain with limited samples (e.g. < 100). We propose an algorithm that can self-align LLMs iteratively without active human involvement. Unlike existing works, our algorithm relies on neither human-crafted instructions nor labeled rewards, significantly reducing human involvement. In addition, our algorithm can self-improve the alignment continuously. The key idea is to first retrieve high-quality samples related to the target domain and use them as In-context Learning examples to generate more samples. Then we use the self-generated samples to finetune the LLM iteratively. We show that our method can unlock the LLMs' self-generalization ability to perform alignment with near-zero human supervision. We test our algorithm on three benchmarks in safety, truthfulness, and instruction-following, and show good performance in alignment, domain adaptability, and scalability.

Title: AI and Generative AI for Research Discovery and Summarization

• Authors: Mark Glickman, Yi Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06795
• Pdf URL: https://arxiv.org/pdf/2401.06795
• Copy Paste: [[2401.06795]] AI and Generative AI for Research Discovery and Summarization(https://arxiv.org/abs/2401.06795)
• Keywords: generative
• Abstract: AI and generative AI tools, including chatbots like ChatGPT that rely on large language models (LLMs), have burst onto the scene this year, creating incredible opportunities to increase work productivity and improve our lives. Statisticians and data scientists have begun experiencing the benefits from the availability of these tools in numerous ways, such as the generation of programming code from text prompts to analyze data or fit statistical models. One area that these tools can make a substantial impact is in research discovery and summarization. Standalone tools and plugins to chatbots are being developed that allow researchers to more quickly find relevant literature than pre-2023 search tools. Furthermore, generative AI tools have improved to the point where they can summarize and extract the key points from research articles in succinct language. Finally, chatbots based on highly parameterized LLMs can be used to simulate abductive reasoning, which provides researchers the ability to make connections among related technical topics, which can also be used for research discovery. We review the developments in AI and generative AI for research discovery and summarization, and propose directions where these types of tools are likely to head in the future that may be of interest to statistician and data scientists.

Title: Generative AI Meets Semantic Communication: Evolution and Revolution of Communication Tasks

• Authors: Eleonora Grassucci, Jihong Park, Sergio Barbarossa, Seong-Lyun Kim, Jinho Choi, Danilo Comminiello
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06803
• Pdf URL: https://arxiv.org/pdf/2401.06803
• Copy Paste: [[2401.06803]] Generative AI Meets Semantic Communication: Evolution and Revolution of Communication Tasks(https://arxiv.org/abs/2401.06803)
• Keywords: generative
• Abstract: While deep generative models are showing exciting abilities in computer vision and natural language processing, their adoption in communication frameworks is still far underestimated. These methods are demonstrated to evolve solutions to classic communication problems such as denoising, restoration, or compression. Nevertheless, generative models can unveil their real potential in semantic communication frameworks, in which the receiver is not asked to recover the sequence of bits used to encode the transmitted (semantic) message, but only to regenerate content that is semantically consistent with the transmitted message. Disclosing generative models capabilities in semantic communication paves the way for a paradigm shift with respect to conventional communication systems, which has great potential to reduce the amount of data traffic and offers a revolutionary versatility to novel tasks and applications that were not even conceivable a few years ago. In this paper, we present a unified perspective of deep generative models in semantic communication and we unveil their revolutionary role in future communication frameworks, enabling emerging applications and tasks. Finally, we analyze the challenges and opportunities to face to develop generative models specifically tailored for communication systems.

Title: ChatGPT, Let us Chat Sign Language: Experiments, Architectural Elements, Challenges and Research Directions

• Authors: Nada Shahin, Leila Ismail
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06804
• Pdf URL: https://arxiv.org/pdf/2401.06804
• Copy Paste: [[2401.06804]] ChatGPT, Let us Chat Sign Language: Experiments, Architectural Elements, Challenges and Research Directions(https://arxiv.org/abs/2401.06804)
• Keywords: generative
• Abstract: ChatGPT is a language model based on Generative AI. Existing research work on ChatGPT focused on its use in various domains. However, its potential for Sign Language Translation (SLT) is yet to be explored. This paper addresses this void. Therefore, we present GPT's evolution aiming a retrospective analysis of the improvements to its architecture for SLT. We explore ChatGPT's capabilities in translating different sign languages in paving the way to better accessibility for deaf and hard-of-hearing community. Our experimental results indicate that ChatGPT can accurately translate from English to American (ASL), Australian (AUSLAN), and British (BSL) sign languages and from Arabic Sign Language (ArSL) to English with only one prompt iteration. However, the model failed to translate from Arabic to ArSL and ASL, AUSLAN, and BSL to Arabic. Consequently, we present challenges and derive insights for future research directions.

Title: When ChatGPT is gone: Creativity reverts and homogeneity persists

• Authors: Qinghan Liu, Yiyong Zhou, Jihao Huang, Guiquan Li
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.06816
• Pdf URL: https://arxiv.org/pdf/2401.06816
• Copy Paste: [[2401.06816]] When ChatGPT is gone: Creativity reverts and homogeneity persists(https://arxiv.org/abs/2401.06816)
• Keywords: generative
• Abstract: ChatGPT has been evidenced to enhance human performance in creative tasks. Yet, it is still unclear if this boosting effect sustains with and without ChatGPT. In a pre-registered seven-day lab experiment and a follow-up survey after 30 days of experiment completion, we examined the impacts of ChatGPT presence and absence on sustained creativity using a text dataset of 3302 creative ideas and 427 creative solutions from 61 college students. Participants in the treatment group used ChatGPT in creative tasks, while those in the control group completed the tasks by themselves. The findings show that although the boosting effect of ChatGPT was consistently observed over a five-day creative journey, human creative performance reverted to baseline when ChatGPT was down on the 7th and the 30th day. More critically, the use of ChatGPT in creative tasks resulted in increasingly homogenized contents, and this homogenization effect persisted even when ChatGPT was absence. These findings pose a challenge to the prevailing argument that ChatGPT can enhance human creativity. In fact, generative AI like ChatGPT lends to human with a temporary rise in creative performance but boxes human creative capability in the long run, highlighting the imperative for cautious generative AI integration in creative endeavors.

Title: A Survey on the Applications of Frontier AI, Foundation Models, and Large Language Models to Intelligent Transportation Systems

• Authors: Mohamed R. Shoaib, Heba M. Emara, Jun Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06831
• Pdf URL: https://arxiv.org/pdf/2401.06831
• Copy Paste: [[2401.06831]] A Survey on the Applications of Frontier AI, Foundation Models, and Large Language Models to Intelligent Transportation Systems(https://arxiv.org/abs/2401.06831)
• Keywords: foundation model
• Abstract: This survey paper explores the transformative influence of frontier AI, foundation models, and Large Language Models (LLMs) in the realm of Intelligent Transportation Systems (ITS), emphasizing their integral role in advancing transportation intelligence, optimizing traffic management, and contributing to the realization of smart cities. Frontier AI refers to the forefront of AI technology, encompassing the latest advancements, innovations, and experimental techniques in the field, especially AI foundation models and LLMs. Foundation models, like GPT-4, are large, general-purpose AI models that provide a base for a wide range of applications. They are characterized by their versatility and scalability. LLMs are obtained from finetuning foundation models with a specific focus on processing and generating natural language. They excel in tasks like language understanding, text generation, translation, and summarization. By leveraging vast textual data, including traffic reports and social media interactions, LLMs extract critical insights, fostering the evolution of ITS. The survey navigates the dynamic synergy between LLMs and ITS, delving into applications in traffic management, integration into autonomous vehicles, and their role in shaping smart cities. It provides insights into ongoing research, innovations, and emerging trends, aiming to inspire collaboration at the intersection of language, intelligence, and mobility for safer, more efficient, and sustainable transportation systems. The paper further surveys interactions between LLMs and various aspects of ITS, exploring roles in traffic management, facilitating autonomous vehicles, and contributing to smart city development, while addressing challenges brought by frontier AI and foundation models. This paper offers valuable inspiration for future research and innovation in the transformative domain of intelligent transportation.

Title: MiTTenS: A Dataset for Evaluating Misgendering in Translation

• Authors: Kevin Robinson, Sneha Kudugunta, Romina Stella, Sunipa Dev, Jasmijn Bastings
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.06935
• Pdf URL: https://arxiv.org/pdf/2401.06935
• Copy Paste: [[2401.06935]] MiTTenS: A Dataset for Evaluating Misgendering in Translation(https://arxiv.org/abs/2401.06935)
• Keywords: foundation model
• Abstract: Misgendering is the act of referring to someone in a way that does not reflect their gender identity. Translation systems, including foundation models capable of translation, can produce errors that result in misgendering harms. To measure the extent of such potential harms when translating into and out of English, we introduce a dataset, MiTTenS, covering 26 languages from a variety of language families and scripts, including several traditionally underpresented in digital resources. The dataset is constructed with handcrafted passages that target known failure patterns, longer synthetically generated passages, and natural passages sourced from multiple domains. We demonstrate the usefulness of the dataset by evaluating both dedicated neural machine translation systems and foundation models, and show that all systems exhibit errors resulting in misgendering harms, even in high resource languages.

Title: Accelerated Sampling of Rare Events using a Neural Network Bias Potential

• Authors: Xinru Hua, Rasool Ahmad, Jose Blanchet, Wei Cai
• Subjects: cs.LG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2401.06936
• Pdf URL: https://arxiv.org/pdf/2401.06936
• Copy Paste: [[2401.06936]] Accelerated Sampling of Rare Events using a Neural Network Bias Potential(https://arxiv.org/abs/2401.06936)
• Keywords: diffusion
• Abstract: In the field of computational physics and material science, the efficient sampling of rare events occurring at atomic scale is crucial. It aids in understanding mechanisms behind a wide range of important phenomena, including protein folding, conformal changes, chemical reactions and materials diffusion and deformation. Traditional simulation methods, such as Molecular Dynamics and Monte Carlo, often prove inefficient in capturing the timescale of these rare events by brute force. In this paper, we introduce a practical approach by combining the idea of importance sampling with deep neural networks (DNNs) that enhance the sampling of these rare events. In particular, we approximate the variance-free bias potential function with DNNs which is trained to maximize the probability of rare event transition under the importance potential function. This method is easily scalable to high-dimensional problems and provides robust statistical guarantees on the accuracy of the estimated probability of rare event transition. Furthermore, our algorithm can actively generate and learn from any successful samples, which is a novel improvement over existing methods. Using a 2D system as a test bed, we provide comparisons between results obtained from different training strategies, traditional Monte Carlo sampling and numerically solved optimal bias potential function under different temperatures. Our numerical results demonstrate the efficacy of the DNN-based importance sampling of rare events.

Title: Transformer for Object Re-Identification: A Survey

• Authors: Mang Ye, Shuoyi Chen, Chenyue Li, Wei-Shi Zheng, David Crandall, Bo Du
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06960
• Pdf URL: https://arxiv.org/pdf/2401.06960
• Copy Paste: [[2401.06960]] Transformer for Object Re-Identification: A Survey(https://arxiv.org/abs/2401.06960)
• Keywords: foundation model
• Abstract: Object Re-Identification (Re-ID) aims to identify and retrieve specific objects from varying viewpoints. For a prolonged period, this field has been predominantly driven by deep convolutional neural networks. In recent years, the Transformer has witnessed remarkable advancements in computer vision, prompting an increasing body of research to delve into the application of Transformer in Re-ID. This paper provides a comprehensive review and in-depth analysis of the Transformer-based Re-ID. In categorizing existing works into Image/Video-Based Re-ID, Re-ID with limited data/annotations, Cross-Modal Re-ID, and Special Re-ID Scenarios, we thoroughly elucidate the advantages demonstrated by the Transformer in addressing a multitude of challenges across these domains. Considering the trending unsupervised Re-ID, we propose a new Transformer baseline, UntransReID, achieving state-of-the-art performance on both single-/cross modal tasks. Besides, this survey also covers a wide range of Re-ID research objects, including progress in animal Re-ID. Given the diversity of species in animal Re-ID, we devise a standardized experimental benchmark and conduct extensive experiments to explore the applicability of Transformer for this task to facilitate future research. Finally, we discuss some important yet under-investigated open issues in the big foundation model era, we believe it will serve as a new handbook for researchers in this field.

Title: Domain Adaptation for Large-Vocabulary Object Detectors

• Authors: Kai Jiang, Jiaxing Huang, Weiying Xie, Yunsong Li, Ling Shao, Shijian Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06969
• Pdf URL: https://arxiv.org/pdf/2401.06969
• Copy Paste: [[2401.06969]] Domain Adaptation for Large-Vocabulary Object Detectors(https://arxiv.org/abs/2401.06969)
• Keywords: foundation model
• Abstract: Large-vocabulary object detectors (LVDs) aim to detect objects of many categories, which learn super objectness features and can locate objects accurately while applied to various downstream data. However, LVDs often struggle in recognizing the located objects due to domain discrepancy in data distribution and object vocabulary. At the other end, recent vision-language foundation models such as CLIP demonstrate superior open-vocabulary recognition capability. This paper presents KGD, a Knowledge Graph Distillation technique that exploits the implicit knowledge graphs (KG) in CLIP for effectively adapting LVDs to various downstream domains. KGD consists of two consecutive stages: 1) KG extraction that employs CLIP to encode downstream domain data as nodes and their feature distances as edges, constructing KG that inherits the rich semantic relations in CLIP explicitly; and 2) KG encapsulation that transfers the extracted KG into LVDs to enable accurate cross-domain object classification. In addition, KGD can extract both visual and textual KG independently, providing complementary vision and language knowledge for object localization and object classification in detection tasks over various downstream domains. Experiments over multiple widely adopted detection benchmarks show that KGD outperforms the state-of-the-art consistently by large margins.

Title: Edge-Enabled Anomaly Detection and Information Completion for Social Network Knowledge Graphs

• Authors: Fan Lu, Quan Qi, Huaibin Qin
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.07022
• Pdf URL: https://arxiv.org/pdf/2401.07022
• Copy Paste: [[2401.07022]] Edge-Enabled Anomaly Detection and Information Completion for Social Network Knowledge Graphs(https://arxiv.org/abs/2401.07022)
• Keywords: anomaly
• Abstract: In the rapidly advancing information era, various human behaviors are being precisely recorded in the form of data, including identity information, criminal records, and communication data. Law enforcement agencies can effectively maintain social security and precisely combat criminal activities by analyzing the aforementioned data. In comparison to traditional data analysis methods, deep learning models, relying on the robust computational power in cloud centers, exhibit higher accuracy in extracting data features and inferring data. However, within the architecture of cloud centers, the transmission of data from end devices introduces significant latency, hindering real-time inference of data. Furthermore, low-latency edge computing architectures face limitations in direct deployment due to relatively weak computing and storage capacities of nodes. To address these challenges, a lightweight distributed knowledge graph completion architecture is proposed. Firstly, we introduce a lightweight distributed knowledge graph completion architecture that utilizes knowledge graph embedding for data analysis. Subsequently, to filter out substandard data, a personnel data quality assessment method named PDQA is proposed. Lastly, we present a model pruning algorithm that significantly reduces the model size while maximizing performance, enabling lightweight deployment. In experiments, we compare the effects of 11 advanced models on completing the knowledge graph of public security personnel information. The results indicate that the RotatE model outperforms other models significantly in knowledge graph completion, with the pruned model size reduced by 70\%, and hits@10 reaching 86.97\%.}

Title: xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning

• Authors: Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xiannian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, Zhoujun Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.07037
• Pdf URL: https://arxiv.org/pdf/2401.07037
• Copy Paste: [[2401.07037]] xCoT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning(https://arxiv.org/abs/2401.07037)
• Keywords: in-context
• Abstract: Chain-of-thought (CoT) has emerged as a powerful technique to elicit reasoning in large language models and improve a variety of downstream tasks. CoT mainly demonstrates excellent performance in English, but its usage in low-resource languages is constrained due to poor language generalization. To bridge the gap among different languages, we propose a cross-lingual instruction fine-tuning framework (xCOT) to transfer knowledge from high-resource languages to low-resource languages. Specifically, the multilingual instruction training data (xCOT-INSTRUCT) is created to encourage the semantic alignment of multiple languages. We introduce cross-lingual in-context few-shot learning (xICL)) to accelerate multilingual agreement in instruction tuning, where some fragments of source languages in examples are randomly substituted by their counterpart translations of target languages. During multilingual instruction tuning, we adopt the randomly online CoT strategy to enhance the multilingual reasoning ability of the large language model by first translating the query to another language and then answering in English. To further facilitate the language transfer, we leverage the high-resource CoT to supervise the training of low-resource languages with cross-lingual distillation. Experimental results on previous benchmarks demonstrate the superior performance of xCoT in reducing the gap among different languages, highlighting its potential to reduce the cross-lingual gap.

Title: Exploring Adversarial Attacks against Latent Diffusion Model from the Perspective of Adversarial Transferability

• Authors: Junxi Chen, Junhao Dong, Xiaohua Xie
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.07087
• Pdf URL: https://arxiv.org/pdf/2401.07087
• Copy Paste: [[2401.07087]] Exploring Adversarial Attacks against Latent Diffusion Model from the Perspective of Adversarial Transferability(https://arxiv.org/abs/2401.07087)
• Keywords: diffusion
• Abstract: Recently, many studies utilized adversarial examples (AEs) to raise the cost of malicious image editing and copyright violation powered by latent diffusion models (LDMs). Despite their successes, a few have studied the surrogate model they used to generate AEs. In this paper, from the perspective of adversarial transferability, we investigate how the surrogate model's property influences the performance of AEs for LDMs. Specifically, we view the time-step sampling in the Monte-Carlo-based (MC-based) adversarial attack as selecting surrogate models. We find that the smoothness of surrogate models at different time steps differs, and we substantially improve the performance of the MC-based AEs by selecting smoother surrogate models. In the light of the theoretical framework on adversarial transferability in image classification, we also conduct a theoretical analysis to explain why smooth surrogate models can also boost AEs for LDMs.

Title: Enhanced Few-Shot Class-Incremental Learning via Ensemble Models

• Authors: Mingli Zhu, Zihao Zhu, Sihong Chen, Chen Chen, Baoyuan Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.07208
• Pdf URL: https://arxiv.org/pdf/2401.07208
• Copy Paste: [[2401.07208]] Enhanced Few-Shot Class-Incremental Learning via Ensemble Models(https://arxiv.org/abs/2401.07208)
• Keywords: self-supervised
• Abstract: Few-shot class-incremental learning (FSCIL) aims to continually fit new classes with limited training data, while maintaining the performance of previously learned classes. The main challenges are overfitting the rare new training samples and forgetting old classes. While catastrophic forgetting has been extensively studied, the overfitting problem has attracted less attention in FSCIL. To tackle overfitting challenge, we design a new ensemble model framework cooperated with data augmentation to boost generalization. In this way, the enhanced model works as a library storing abundant features to guarantee fast adaptation to downstream tasks. Specifically, the multi-input multi-output ensemble structure is applied with a spatial-aware data augmentation strategy, aiming at diversifying the feature extractor and alleviating overfitting in incremental sessions. Moreover, self-supervised learning is also integrated to further improve the model generalization. Comprehensive experimental results show that the proposed method can indeed mitigate the overfitting problem in FSCIL, and outperform the state-of-the-art methods.

Title: Self-supervised Event-based Monocular Depth Estimation using Cross-modal Consistency

• Authors: Junyu Zhu, Lina Liu, Bofeng Jiang, Feng Wen, Hongbo Zhang, Wanlong Li, Yong Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.07218
• Pdf URL: https://arxiv.org/pdf/2401.07218
• Copy Paste: [[2401.07218]] Self-supervised Event-based Monocular Depth Estimation using Cross-modal Consistency(https://arxiv.org/abs/2401.07218)
• Keywords: self-supervised
• Abstract: An event camera is a novel vision sensor that can capture per-pixel brightness changes and output a stream of asynchronous ``events''. It has advantages over conventional cameras in those scenes with high-speed motions and challenging lighting conditions because of the high temporal resolution, high dynamic range, low bandwidth, low power consumption, and no motion blur. Therefore, several supervised monocular depth estimation from events is proposed to address scenes difficult for conventional cameras. However, depth annotation is costly and time-consuming. In this paper, to lower the annotation cost, we propose a self-supervised event-based monocular depth estimation framework named EMoDepth. EMoDepth constrains the training process using the cross-modal consistency from intensity frames that are aligned with events in the pixel coordinate. Moreover, in inference, only events are used for monocular depth prediction. Additionally, we design a multi-scale skip-connection architecture to effectively fuse features for depth estimation while maintaining high inference speed. Experiments on MVSEC and DSEC datasets demonstrate that our contributions are effective and that the accuracy can outperform existing supervised event-based and unsupervised frame-based methods.

Title: Distilling Event Sequence Knowledge From Large Language Models

• Authors: Somin Wadhwa, Oktie Hassanzadeh, Debarun Bhattacharjya, Ken Barker, Jian Ni
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.07237
• Pdf URL: https://arxiv.org/pdf/2401.07237
• Copy Paste: [[2401.07237]] Distilling Event Sequence Knowledge From Large Language Models(https://arxiv.org/abs/2401.07237)
• Keywords: generative
• Abstract: Event sequence models have been found to be highly effective in the analysis and prediction of events. Building such models requires availability of abundant high-quality event sequence data. In certain applications, however, clean structured event sequences are not available, and automated sequence extraction results in data that is too noisy and incomplete. In this work, we explore the use of Large Language Models (LLMs) to generate event sequences that can effectively be used for probabilistic event model construction. This can be viewed as a mechanism of distilling event sequence knowledge from LLMs. Our approach relies on a Knowledge Graph (KG) of event concepts with partial causal relations to guide the generative language model for causal event sequence generation. We show that our approach can generate high-quality event sequences, filling a knowledge gap in the input KG. Furthermore, we explore how the generated sequences can be leveraged to discover useful and more complex structured knowledge from pattern mining and probabilistic event models. We release our sequence generation code and evaluation framework, as well as corpus of event sequence data.

Title: MIMIC: Mask Image Pre-training with Mix Contrastive Fine-tuning for Facial Expression Recognition

• Authors: Fan Zhang, Xiaobao Guo, Xiaojiang Peng, Alex Kot
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.07245
• Pdf URL: https://arxiv.org/pdf/2401.07245
• Copy Paste: [[2401.07245]] MIMIC: Mask Image Pre-training with Mix Contrastive Fine-tuning for Facial Expression Recognition(https://arxiv.org/abs/2401.07245)
• Keywords: self-supervised
• Abstract: Cutting-edge research in facial expression recognition (FER) currently favors the utilization of convolutional neural networks (CNNs) backbone which is supervisedly pre-trained on face recognition datasets for feature extraction. However, due to the vast scale of face recognition datasets and the high cost associated with collecting facial labels, this pre-training paradigm incurs significant expenses. Towards this end, we propose to pre-train vision Transformers (ViTs) through a self-supervised approach on a mid-scale general image dataset. In addition, when compared with the domain disparity existing between face datasets and FER datasets, the divergence between general datasets and FER datasets is more pronounced. Therefore, we propose a contrastive fine-tuning approach to effectively mitigate this domain disparity. Specifically, we introduce a novel FER training paradigm named Mask Image pre-training with MIx Contrastive fine-tuning (MIMIC). In the initial phase, we pre-train the ViT via masked image reconstruction on general images. Subsequently, in the fine-tuning stage, we introduce a mix-supervised contrastive learning process, which enhances the model with a more extensive range of positive samples by the mixing strategy. Through extensive experiments conducted on three benchmark datasets, we demonstrate that our MIMIC outperforms the previous training paradigm, showing its capability to learn better representations. Remarkably, the results indicate that the vanilla ViT can achieve impressive performance without the need for intricate, auxiliary-designed modules. Moreover, when scaling up the model size, MIMIC exhibits no performance saturation and is superior to the current state-of-the-art methods.

Title: Small Language Model Can Self-correct

• Authors: Haixia Han, Jiaqing Liang, Jie Shi, Qianyu He, Yanghua Xiao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.07301
• Pdf URL: https://arxiv.org/pdf/2401.07301
• Copy Paste: [[2401.07301]] Small Language Model Can Self-correct(https://arxiv.org/abs/2401.07301)
• Keywords: generative
• Abstract: Generative Language Models (LMs) such as ChatGPT have exhibited remarkable performance across various downstream tasks. Nevertheless, one of their most prominent drawbacks is generating inaccurate or false information with a confident tone. Previous studies have devised sophisticated pipelines and prompts to induce large LMs to exhibit the capability for self-correction. However, large LMs are explicitly prompted to verify and modify its answers separately rather than completing all steps spontaneously like humans. Moreover, these complex prompts are extremely challenging for small LMs to follow. In this paper, we introduce the \underline{I}ntrinsic \underline{S}elf-\underline{C}orrection (ISC) in generative language models, aiming to correct the initial output of LMs in a self-triggered manner, even for those small LMs with 6 billion parameters. Specifically, we devise a pipeline for constructing self-correction data and propose Partial Answer Masking (PAM), aiming to endow the model with the capability for intrinsic self-correction through fine-tuning. We conduct experiments using LMs with parameters sizes ranging from 6 billion to 13 billion in two tasks, including commonsense reasoning and factual knowledge reasoning. Our experiments demonstrate that the outputs generated using ISC outperform those generated without self-correction. We believe that the output quality of even small LMs can be further improved by empowering them with the ability to intrinsic self-correct.

Title: PersonalityChat: Conversation Distillation for Personalized Dialog Modeling with Facts and Traits

• Authors: Ehsan Lotfi, Maxime De Bruyn, Jeska Buhmann, Walter Daelemans
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.07363
• Pdf URL: https://arxiv.org/pdf/2401.07363
• Copy Paste: [[2401.07363]] PersonalityChat: Conversation Distillation for Personalized Dialog Modeling with Facts and Traits(https://arxiv.org/abs/2401.07363)
• Keywords: generative
• Abstract: The new wave of Large Language Models (LLM) has offered an efficient tool to curate sizeable conversational datasets. So far studies have mainly focused on task-oriented or generic open-domain dialogs, and have not fully explored the ability of LLMs in following complicated prompts. In this work, we focus on personalization, and employ LLMs to curate a dataset which is difficult and costly to crowd-source: PersonalityChat is a synthetic conversational dataset based upon the popular PersonaChat dataset, but conditioned on both personas and (Big-5) personality traits. Evaluating models fine-tuned on this dataset, we show that the personality trait labels can be used for trait-based personalization of generative dialogue models. We also perform a head-to-head comparison between PersonalityChat and PersonaChat, and show that training on the distilled dataset results in more fluent and coherent dialog agents in the small-model regime.

Title: PDE Generalization of In-Context Operator Networks: A Study on 1D Scalar Nonlinear Conservation Laws

• Authors: Liu Yang, Stanley J. Osher
• Subjects: cs.LG, cs.AI, math.NA
• Abstract URL: https://arxiv.org/abs/2401.07364
• Pdf URL: https://arxiv.org/pdf/2401.07364
• Copy Paste: [[2401.07364]] PDE Generalization of In-Context Operator Networks: A Study on 1D Scalar Nonlinear Conservation Laws(https://arxiv.org/abs/2401.07364)
• Keywords: foundation model, in-context
• Abstract: Can we build a single large model for a wide range of PDE-related scientific learning tasks? Can this model generalize to new PDEs, even of new forms, without any fine-tuning? In-context operator learning and the corresponding model In-Context Operator Networks (ICON) [1] represent an initial exploration of these questions. The capability of ICON regarding the first question has been demonstrated in [1]. In this paper, we explore the second question by investigating the generalization capabilities of ICON for conservation laws, a family of PDEs with temporal evolution. We show the positive answer to the second question, i.e., ICON can generalize well to some PDEs with new forms without any fine-tuning. We also show how to broaden the range of problems that ICON can address, by transforming functions and equations to ICON's capability scope. We believe that the progress in this paper is a significant step towards the goal of training a foundation model for PDE-related tasks under the in-context operator learning framework.

Title: Generation of Synthetic Images for Pedestrian Detection Using a Sequence of GANs

• Authors: Viktor Seib, Malte Roosen, Ida Germann, Stefan Wirtz, Dietrich Paulus
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.07370
• Pdf URL: https://arxiv.org/pdf/2401.07370
• Copy Paste: [[2401.07370]] Generation of Synthetic Images for Pedestrian Detection Using a Sequence of GANs(https://arxiv.org/abs/2401.07370)
• Keywords: generative
• Abstract: Creating annotated datasets demands a substantial amount of manual effort. In this proof-of-concept work, we address this issue by proposing a novel image generation pipeline. The pipeline consists of three distinct generative adversarial networks (previously published), combined in a novel way to augment a dataset for pedestrian detection. Despite the fact that the generated images are not always visually pleasant to the human eye, our detection benchmark reveals that the results substantially surpass the baseline. The presented proof-of-concept work was done in 2020 and is now published as a technical report after a three years retention period.

Title: Cross Domain Early Crop Mapping using CropGAN and CNN Classifier

• Authors: Yiqun Wang, Hui Huang, Radu State
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.07398
• Pdf URL: https://arxiv.org/pdf/2401.07398
• Copy Paste: [[2401.07398]] Cross Domain Early Crop Mapping using CropGAN and CNN Classifier(https://arxiv.org/abs/2401.07398)
• Keywords: generative
• Abstract: Driven by abundant satellite imagery, machine learning-based approaches have recently been promoted to generate high-resolution crop cultivation maps to support many agricultural applications. One of the major challenges faced by these approaches is the limited availability of ground truth labels. In the absence of ground truth, existing work usually adopts the "direct transfer strategy" that trains a classifier using historical labels collected from other regions and then applies the trained model to the target region. Unfortunately, the spectral features of crops exhibit inter-region and inter-annual variability due to changes in soil composition, climate conditions, and crop progress, the resultant models perform poorly on new and unseen regions or years. This paper presents the Crop Generative Adversarial Network (CropGAN) to address the above cross-domain issue. Our approach does not need labels from the target domain. Instead, it learns a mapping function to transform the spectral features of the target domain to the source domain (with labels) while preserving their local structure. The classifier trained by the source domain data can be directly applied to the transformed data to produce high-accuracy early crop maps of the target domain. Comprehensive experiments across various regions and years demonstrate the benefits and effectiveness of the proposed approach. Compared with the widely adopted direct transfer strategy, the F1 score after applying the proposed CropGAN is improved by 13.13% - 50.98%

Title: Hierarchical Fashion Design with Multi-stage Diffusion Models

• Authors: Zhifeng Xie, Hao li, Huiming Ding, Mengtian Li, Ying Cao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.07450
• Pdf URL: https://arxiv.org/pdf/2401.07450
• Copy Paste: [[2401.07450]] Hierarchical Fashion Design with Multi-stage Diffusion Models(https://arxiv.org/abs/2401.07450)
• Keywords: diffusion
• Abstract: Cross-modal fashion synthesis and editing offer intelligent support to fashion designers by enabling the automatic generation and local modification of design drafts.While current diffusion models demonstrate commendable stability and controllability in image synthesis,they still face significant challenges in generating fashion design from abstract design elements and fine-grained editing.Abstract sensory expressions, \eg office, business, and party, form the high-level design concepts, while measurable aspects like sleeve length, collar type, and pant length are considered the low-level attributes of clothing.Controlling and editing fashion images using lengthy text descriptions poses a difficulty.In this paper, we propose HieraFashDiff,a novel fashion design method using the shared multi-stage diffusion model encompassing high-level design concepts and low-level clothing attributes in a hierarchical structure.Specifically, we categorized the input text into different levels and fed them in different time step to the diffusion model according to the criteria of professional clothing designers.HieraFashDiff allows designers to add low-level attributes after high-level prompts for interactive editing incrementally.In addition, we design a differentiable loss function in the sampling process with a mask to keep non-edit areas.Comprehensive experiments performed on our newly conducted Hierarchical fashion dataset,demonstrate that our proposed method outperforms other state-of-the-art competitors.

Title: PolMERLIN: Self-Supervised Polarimetric Complex SAR Image Despeckling with Masked Networks

• Authors: Shunya Kato, Masaki Saito, Katsuhiko Ishiguro, Sol Cummings
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.07503
• Pdf URL: https://arxiv.org/pdf/2401.07503
• Copy Paste: [[2401.07503]] PolMERLIN: Self-Supervised Polarimetric Complex SAR Image Despeckling with Masked Networks(https://arxiv.org/abs/2401.07503)
• Keywords: self-supervised
• Abstract: Despeckling is a crucial noise reduction task in improving the quality of synthetic aperture radar (SAR) images. Directly obtaining noise-free SAR images is a challenging task that has hindered the development of accurate despeckling algorithms. The advent of deep learning has facilitated the study of denoising models that learn from only noisy SAR images. However, existing methods deal solely with single-polarization images and cannot handle the multi-polarization images captured by modern satellites. In this work, we present an extension of the existing model for generating single-polarization SAR images to handle multi-polarization SAR images. Specifically, we propose a novel self-supervised despeckling approach called channel masking, which exploits the relationship between polarizations. Additionally, we utilize a spatial masking method that addresses pixel-to-pixel correlations to further enhance the performance of our approach. By effectively incorporating multiple polarization information, our method surpasses current state-of-the-art methods in quantitative evaluation in both synthetic and real-world scenarios.

Title: InstantID: Zero-shot Identity-Preserving Generation in Seconds

• Authors: Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, Anthony Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.07519
• Pdf URL: https://arxiv.org/pdf/2401.07519
• Copy Paste: [[2401.07519]] InstantID: Zero-shot Identity-Preserving Generation in Seconds(https://arxiv.org/abs/2401.07519)
• Keywords: diffusion
• Abstract: There has been significant progress in personalized image synthesis with methods such as Textual Inversion, DreamBooth, and LoRA. Yet, their real-world applicability is hindered by high storage demands, lengthy fine-tuning processes, and the need for multiple reference images. Conversely, existing ID embedding-based methods, while requiring only a single forward inference, face challenges: they either necessitate extensive fine-tuning across numerous model parameters, lack compatibility with community pre-trained models, or fail to maintain high face fidelity. Addressing these limitations, we introduce InstantID, a powerful diffusion model-based solution. Our plug-and-play module adeptly handles image personalization in various styles using just a single facial image, while ensuring high fidelity. To achieve this, we design a novel IdentityNet by imposing strong semantic and weak spatial conditions, integrating facial and landmark images with textual prompts to steer the image generation. InstantID demonstrates exceptional performance and efficiency, proving highly beneficial in real-world applications where identity preservation is paramount. Moreover, our work seamlessly integrates with popular pre-trained text-to-image diffusion models like SD1.5 and SDXL, serving as an adaptable plugin. Our codes and pre-trained checkpoints will be available at https://github.com/InstantID/InstantID.

Title: One for All: Toward Unified Foundation Models for Earth Vision

• Authors: Zhitong Xiong, Yi Wang, Fahong Zhang, Xiao Xiang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.07527
• Pdf URL: https://arxiv.org/pdf/2401.07527
• Copy Paste: [[2401.07527]] One for All: Toward Unified Foundation Models for Earth Vision(https://arxiv.org/abs/2401.07527)
• Keywords: foundation model
• Abstract: Foundation models characterized by extensive parameters and trained on large-scale datasets have demonstrated remarkable efficacy across various downstream tasks for remote sensing data. Current remote sensing foundation models typically specialize in a single modality or a specific spatial resolution range, limiting their versatility for downstream datasets. While there have been attempts to develop multi-modal remote sensing foundation models, they typically employ separate vision encoders for each modality or spatial resolution, necessitating a switch in backbones contingent upon the input data. To address this issue, we introduce a simple yet effective method, termed OFA-Net (One-For-All Network): employing a single, shared Transformer backbone for multiple data modalities with different spatial resolutions. Using the masked image modeling mechanism, we pre-train a single Transformer backbone on a curated multi-modal dataset with this simple design. Then the backbone model can be used in different downstream tasks, thus forging a path towards a unified foundation backbone model in Earth vision. The proposed method is evaluated on 12 distinct downstream tasks and demonstrates promising performance.

Title: Exploiting GPT-4 Vision for Zero-shot Point Cloud Understanding

• Authors: Qi Sun, Xiao Cui, Wengang Zhou, Houqiang Li
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.07572
• Pdf URL: https://arxiv.org/pdf/2401.07572
• Copy Paste: [[2401.07572]] Exploiting GPT-4 Vision for Zero-shot Point Cloud Understanding(https://arxiv.org/abs/2401.07572)
• Keywords: generative
• Abstract: In this study, we tackle the challenge of classifying the object category in point clouds, which previous works like PointCLIP struggle to address due to the inherent limitations of the CLIP architecture. Our approach leverages GPT-4 Vision (GPT-4V) to overcome these challenges by employing its advanced generative abilities, enabling a more adaptive and robust classification process. We adapt the application of GPT-4V to process complex 3D data, enabling it to achieve zero-shot recognition capabilities without altering the underlying model architecture. Our methodology also includes a systematic strategy for point cloud image visualization, mitigating domain gap and enhancing GPT-4V's efficiency. Experimental validation demonstrates our approach's superiority in diverse scenarios, setting a new benchmark in zero-shot point cloud classification.

Title: Collaboratively Self-supervised Video Representation Learning for Action Recognition

• Authors: Jie Zhang, Zhifan Wan, Lanqing Hu, Stephen Lin, Shuzhe Wu, Shiguang Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.07584
• Pdf URL: https://arxiv.org/pdf/2401.07584
• Copy Paste: [[2401.07584]] Collaboratively Self-supervised Video Representation Learning for Action Recognition(https://arxiv.org/abs/2401.07584)
• Keywords: self-supervised, generative
• Abstract: Considering the close connection between action recognition and human pose estimation, we design a Collaboratively Self-supervised Video Representation (CSVR) learning framework specific to action recognition by jointly considering generative pose prediction and discriminative context matching as pretext tasks. Specifically, our CSVR consists of three branches: a generative pose prediction branch, a discriminative context matching branch, and a video generating branch. Among them, the first one encodes dynamic motion feature by utilizing Conditional-GAN to predict the human poses of future frames, and the second branch extracts static context features by pulling the representations of clips and compressed key frames from the same video together while pushing apart the pairs from different videos. The third branch is designed to recover the current video frames and predict the future ones, for the purpose of collaboratively improving dynamic motion features and static context features. Extensive experiments demonstrate that our method achieves state-of-the-art performance on the UCF101 and HMDB51 datasets.

Title: Multimodal Crowd Counting with Pix2Pix GANs

• Authors: Muhammad Asif Khan, Hamid Menouar, Ridha Hamila
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.07591
• Pdf URL: https://arxiv.org/pdf/2401.07591
• Copy Paste: [[2401.07591]] Multimodal Crowd Counting with Pix2Pix GANs(https://arxiv.org/abs/2401.07591)
• Keywords: generative
• Abstract: Most state-of-the-art crowd counting methods use color (RGB) images to learn the density map of the crowd. However, these methods often struggle to achieve higher accuracy in densely crowded scenes with poor illumination. Recently, some studies have reported improvement in the accuracy of crowd counting models using a combination of RGB and thermal images. Although multimodal data can lead to better predictions, multimodal data might not be always available beforehand. In this paper, we propose the use of generative adversarial networks (GANs) to automatically generate thermal infrared (TIR) images from color (RGB) images and use both to train crowd counting models to achieve higher accuracy. We use a Pix2Pix GAN network first to translate RGB images to TIR images. Our experiments on several state-of-the-art crowd counting models and benchmark crowd datasets report significant improvement in accuracy.

Title: Foundation Models for Biomedical Image Segmentation: A Survey

• Authors: Ho Hin Lee, Yu Gu, Theodore Zhao, Yanbo Xu, Jianwei Yang, Naoto Usuyama, Cliff Wong, Mu Wei, Bennett A. Landman, Yuankai Huo, Alberto Santamaria-Pang, Hoifung Poon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.07654
• Pdf URL: https://arxiv.org/pdf/2401.07654
• Copy Paste: [[2401.07654]] Foundation Models for Biomedical Image Segmentation: A Survey(https://arxiv.org/abs/2401.07654)
• Keywords: foundation model
• Abstract: Recent advancements in biomedical image analysis have been significantly driven by the Segment Anything Model (SAM). This transformative technology, originally developed for general-purpose computer vision, has found rapid application in medical image processing. Within the last year, marked by over 100 publications, SAM has demonstrated its prowess in zero-shot learning adaptations for medical imaging. The fundamental premise of SAM lies in its capability to segment or identify objects in images without prior knowledge of the object type or imaging modality. This approach aligns well with tasks achievable by the human visual system, though its application in non-biological vision contexts remains more theoretically challenging. A notable feature of SAM is its ability to adjust segmentation according to a specified resolution scale or area of interest, akin to semantic priming. This adaptability has spurred a wave of creativity and innovation in applying SAM to medical imaging. Our review focuses on the period from April 1, 2023, to September 30, 2023, a critical first six months post-initial publication. We examine the adaptations and integrations of SAM necessary to address longstanding clinical challenges, particularly in the context of 33 open datasets covered in our analysis. While SAM approaches or achieves state-of-the-art performance in numerous applications, it falls short in certain areas, such as segmentation of the carotid artery, adrenal glands, optic nerve, and mandible bone. Our survey delves into the innovative techniques where SAM's foundational approach excels and explores the core concepts in translating and applying these models effectively in diverse medical imaging scenarios.

Title: Prompting open-source and commercial language models for grammatical error correction of English learner text

• Authors: Christopher Davis, Andrew Caines, Øistein Andersen, Shiva Taslimipoor, Helen Yannakoudakis, Zheng Yuan, Christopher Bryant, Marek Rei, Paula Buttery
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.07702
• Pdf URL: https://arxiv.org/pdf/2401.07702
• Copy Paste: [[2401.07702]] Prompting open-source and commercial language models for grammatical error correction of English learner text(https://arxiv.org/abs/2401.07702)
• Keywords: generative
• Abstract: Thanks to recent advances in generative AI, we are able to prompt large language models (LLMs) to produce texts which are fluent and grammatical. In addition, it has been shown that we can elicit attempts at grammatical error correction (GEC) from LLMs when prompted with ungrammatical input sentences. We evaluate how well LLMs can perform at GEC by measuring their performance on established benchmark datasets. We go beyond previous studies, which only examined GPT* models on a selection of English GEC datasets, by evaluating seven open-source and three commercial LLMs on four established GEC benchmarks. We investigate model performance and report results against individual error types. Our results indicate that LLMs do not always outperform supervised English GEC models except in specific contexts -- namely commercial LLMs on benchmarks annotated with fluency corrections as opposed to minimal edits. We find that several open-source models outperform commercial ones on minimal edit benchmarks, and that in some settings zero-shot prompting is just as competitive as few-shot prompting.

Title: Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks

• Authors: Siyu Zou, Jiji Tang, Yiyi Zhou, Jing He, Chaoyi Zhao, Rongsheng Zhang, Zhipeng Hu, Xiaoshuai Sun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.07709
• Pdf URL: https://arxiv.org/pdf/2401.07709
• Copy Paste: [[2401.07709]] Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks(https://arxiv.org/abs/2401.07709)
• Keywords: diffusion
• Abstract: Diffusion-based Image Editing (DIE) is an emerging research hot-spot, which often applies a semantic mask to control the target area for diffusion-based editing. However, most existing solutions obtain these masks via manual operations or off-line processing, greatly reducing their efficiency. In this paper, we propose a novel and efficient image editing method for Text-to-Image (T2I) diffusion models, termed Instant Diffusion Editing(InstDiffEdit). In particular, InstDiffEdit aims to employ the cross-modal attention ability of existing diffusion models to achieve instant mask guidance during the diffusion steps. To reduce the noise of attention maps and realize the full automatics, we equip InstDiffEdit with a training-free refinement scheme to adaptively aggregate the attention distributions for the automatic yet accurate mask generation. Meanwhile, to supplement the existing evaluations of DIE, we propose a new benchmark called Editing-Mask to examine the mask accuracy and local editing ability of existing methods. To validate InstDiffEdit, we also conduct extensive experiments on ImageNet and Imagen, and compare it with a bunch of the SOTA methods. The experimental results show that InstDiffEdit not only outperforms the SOTA methods in both image quality and editing results, but also has a much faster inference speed, i.e., +5 to +6 times. Our code available at https://anonymous.4open.science/r/InstDiffEdit-C306/

Title: Graph Transformer GANs with Graph Masked Modeling for Architectural Layout Generation

• Authors: Hao Tang, Ling Shao, Nicu Sebe, Luc Van Gool
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.07721
• Pdf URL: https://arxiv.org/pdf/2401.07721
• Copy Paste: [[2401.07721]] Graph Transformer GANs with Graph Masked Modeling for Architectural Layout Generation(https://arxiv.org/abs/2401.07721)
• Keywords: generative
• Abstract: We present a novel graph Transformer generative adversarial network (GTGAN) to learn effective graph node relations in an end-to-end fashion for challenging graph-constrained architectural layout generation tasks. The proposed graph-Transformer-based generator includes a novel graph Transformer encoder that combines graph convolutions and self-attentions in a Transformer to model both local and global interactions across connected and non-connected graph nodes. Specifically, the proposed connected node attention (CNA) and non-connected node attention (NNA) aim to capture the global relations across connected nodes and non-connected nodes in the input graph, respectively. The proposed graph modeling block (GMB) aims to exploit local vertex interactions based on a house layout topology. Moreover, we propose a new node classification-based discriminator to preserve the high-level semantic and discriminative node features for different house components. To maintain the relative spatial relationships between ground truth and predicted graphs, we also propose a novel graph-based cycle-consistency loss. Finally, we propose a novel self-guided pre-training method for graph representation learning. This approach involves simultaneous masking of nodes and edges at an elevated mask ratio (i.e., 40%) and their subsequent reconstruction using an asymmetric graph-centric autoencoder architecture. This method markedly improves the model's learning proficiency and expediency. Experiments on three challenging graph-constrained architectural layout generation tasks (i.e., house layout generation, house roof generation, and building layout generation) with three public datasets demonstrate the effectiveness of the proposed method in terms of objective quantitative scores and subjective visual realism. New state-of-the-art results are established by large margins on these three tasks.

Title: HexaGen3D: StableDiffusion is just one step away from Fast and Diverse Text-to-3D Generation

• Authors: Antoine Mercier, Ramin Nakhli, Mahesh Reddy, Rajeev Yasarla, Hong Cai, Fatih Porikli, Guillaume Berger
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.07727
• Pdf URL: https://arxiv.org/pdf/2401.07727
• Copy Paste: [[2401.07727]] HexaGen3D: StableDiffusion is just one step away from Fast and Diverse Text-to-3D Generation(https://arxiv.org/abs/2401.07727)
• Keywords: diffusion, generative
• Abstract: Despite the latest remarkable advances in generative modeling, efficient generation of high-quality 3D assets from textual prompts remains a difficult task. A key challenge lies in data scarcity: the most extensive 3D datasets encompass merely millions of assets, while their 2D counterparts contain billions of text-image pairs. To address this, we propose a novel approach which harnesses the power of large, pretrained 2D diffusion models. More specifically, our approach, HexaGen3D, fine-tunes a pretrained text-to-image model to jointly predict 6 orthographic projections and the corresponding latent triplane. We then decode these latents to generate a textured mesh. HexaGen3D does not require per-sample optimization, and can infer high-quality and diverse objects from textual prompts in 7 seconds, offering significantly better quality-to-latency trade-offs when comparing to existing approaches. Furthermore, HexaGen3D demonstrates strong generalization to new objects or compositions.

Title: MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation

• Authors: Mi Yan, Jiazhao Zhang, Yan Zhu, He Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.07745
• Pdf URL: https://arxiv.org/pdf/2401.07745
• Copy Paste: [[2401.07745]] MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation(https://arxiv.org/abs/2401.07745)
• Keywords: foundation model
• Abstract: Open-vocabulary 3D instance segmentation has emerged as a frontier topic due to its capability to segment 3D instances beyond a predefined set of categories. However, compared to significant progress in the 2D domain, methods for 3D open-vocabulary instance segmentation are hindered by the limited scale of high-quality annotated 3D data. To harness the capabilities of 2D models, recent efforts have focused on merging 2D masks based on metrics such as geometric and semantic similarity to form 3D instances. In contrast to these local metrics, we propose a novel metric called view consensus to better exploit multi-view observation. The key insight is that two 2D masks should be considered as belonging to the same instance if a considerable number of other 2D masks from other views contain both these two masks. Based on this metric, we build a global mask graph and iteratively cluster masks, prioritizing mask pairs with solid view consensus. The corresponding 3D points cluster of these 2D mask clusters can be regarded as 3D instances, along with the fused open-vocabulary features from clustered 2D masks. Through this multi-view verification and fusion mechanism, our method effectively leverages the prior instance knowledge from massive 2D masks predicted by visual foundation models, eliminating the need for training on 3D data. Experiments on publicly available datasets, including ScanNet200 and MatterPort3D, demonstrate that our method achieves state-of-the-art performance in both open-vocabulary instance segmentation and class-agnostic mask generation. Our project page is at https://pku-epic.github.io/MaskClustering.

Title: Towards A Better Metric for Text-to-Video Generation

• Authors: Jay Zhangjie Wu, Guian Fang, Haoning Wu, Xintao Wang, Yixiao Ge, Xiaodong Cun, David Junhao Zhang, Jia-Wei Liu, Yuchao Gu, Rui Zhao, Weisi Lin, Wynne Hsu, Ying Shan, Mike Zheng Shou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.07781
• Pdf URL: https://arxiv.org/pdf/2401.07781
• Copy Paste: [[2401.07781]] Towards A Better Metric for Text-to-Video Generation(https://arxiv.org/abs/2401.07781)
• Keywords: generative
• Abstract: Generative models have demonstrated remarkable capability in synthesizing high-quality text, images, and videos. For video generation, contemporary text-to-video models exhibit impressive capabilities, crafting visually stunning videos. Nonetheless, evaluating such videos poses significant challenges. Current research predominantly employs automated metrics such as FVD, IS, and CLIP Score. However, these metrics provide an incomplete analysis, particularly in the temporal assessment of video content, thus rendering them unreliable indicators of true video quality. Furthermore, while user studies have the potential to reflect human perception accurately, they are hampered by their time-intensive and laborious nature, with outcomes that are often tainted by subjective bias. In this paper, we investigate the limitations inherent in existing metrics and introduce a novel evaluation pipeline, the Text-to-Video Score (T2VScore). This metric integrates two pivotal criteria: (1) Text-Video Alignment, which scrutinizes the fidelity of the video in representing the given text description, and (2) Video Quality, which evaluates the video's overall production caliber with a mixture of experts. Moreover, to evaluate the proposed metrics and facilitate future improvements on them, we present the TVGE dataset, collecting human judgements of 2,543 text-to-video generated videos on the two criteria. Experiments on the TVGE dataset demonstrate the superiority of the proposed T2VScore on offering a better metric for text-to-video generation.

Title: Fusing Echocardiography Images and Medical Records for Continuous Patient Stratification

• Authors: Nathan Painchaud, Pierre-Yves Courand, Pierre-Marc Jodoin, Nicolas Duchateau, Olivier Bernard
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.07796
• Pdf URL: https://arxiv.org/pdf/2401.07796
• Copy Paste: [[2401.07796]] Fusing Echocardiography Images and Medical Records for Continuous Patient Stratification(https://arxiv.org/abs/2401.07796)
• Keywords: foundation model
• Abstract: Deep learning now enables automatic and robust extraction of cardiac function descriptors from echocardiographic sequences, such as ejection fraction or strain. These descriptors provide fine-grained information that physicians consider, in conjunction with more global variables from the clinical record, to assess patients' condition. Drawing on novel transformer models applied to tabular data (e.g., variables from electronic health records), we propose a method that considers all descriptors extracted from medical records and echocardiograms to learn the representation of a difficult-to-characterize cardiovascular pathology, namely hypertension. Our method first projects each variable into its own representation space using modality-specific approaches. These standardized representations of multimodal data are then fed to a transformer encoder, which learns to merge them into a comprehensive representation of the patient through a pretext task of predicting a clinical rating. This pretext task is formulated as an ordinal classification to enforce a pathological continuum in the representation space. We observe the major trends along this continuum for a cohort of 239 hypertensive patients to describe, with unprecedented gradation, the effect of hypertension on a number of cardiac function descriptors. Our analysis shows that i) pretrained weights from a foundation model allow to reach good performance (83% accuracy) even with limited data (less than 200 training samples), ii) trends across the population are reproducible between trainings, and iii) for descriptors whose interactions with hypertension are well documented, patterns are consistent with prior physiological knowledge.

Title: Discovery of Generalizable TBI Phenotypes Using Multivariate Time-Series Clustering

• Authors: Hamid Ghaderi, Brandon Foreman, Chandan K. Reddy, Vignesh Subbian
• Subjects: cs.LG, q-bio.QM, stat.AP
• Abstract URL: https://arxiv.org/abs/2401.08002
• Pdf URL: https://arxiv.org/pdf/2401.08002
• Copy Paste: [[2401.08002]] Discovery of Generalizable TBI Phenotypes Using Multivariate Time-Series Clustering(https://arxiv.org/abs/2401.08002)
• Keywords: self-supervised
• Abstract: Traumatic Brain Injury (TBI) presents a broad spectrum of clinical presentations and outcomes due to its inherent heterogeneity, leading to diverse recovery trajectories and varied therapeutic responses. While many studies have delved into TBI phenotyping for distinct patient populations, identifying TBI phenotypes that consistently generalize across various settings and populations remains a critical research gap. Our research addresses this by employing multivariate time-series clustering to unveil TBI's dynamic intricates. Utilizing a self-supervised learning-based approach to clustering multivariate time-Series data with missing values (SLAC-Time), we analyzed both the research-centric TRACK-TBI and the real-world MIMIC-IV datasets. Remarkably, the optimal hyperparameters of SLAC-Time and the ideal number of clusters remained consistent across these datasets, underscoring SLAC-Time's stability across heterogeneous datasets. Our analysis revealed three generalizable TBI phenotypes ({\alpha}, \b{eta}, and {\gamma}), each exhibiting distinct non-temporal features during emergency department visits, and temporal feature profiles throughout ICU stays. Specifically, phenotype {\alpha} represents mild TBI with a remarkably consistent clinical presentation. In contrast, phenotype \b{eta} signifies severe TBI with diverse clinical manifestations, and phenotype {\gamma} represents a moderate TBI profile in terms of severity and clinical diversity. Age is a significant determinant of TBI outcomes, with older cohorts recording higher mortality rates. Importantly, while certain features varied by age, the core characteristics of TBI manifestations tied to each phenotype remain consistent across diverse populations.

Title: Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities

• Authors: Xu Yan, Haiming Zhang, Yingjie Cai, Jingming Guo, Weichao Qiu, Bin Gao, Kaiqiang Zhou, Yue Zhao, Huan Jin, Jiantao Gao, Zhen Li, Lihui Jiang, Wei Zhang, Hongbo Zhang, Dengxin Dai, Bingbing Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08045
• Pdf URL: https://arxiv.org/pdf/2401.08045
• Copy Paste: [[2401.08045]] Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities(https://arxiv.org/abs/2401.08045)
• Keywords: diffusion, foundation model
• Abstract: The rise of large foundation models, trained on extensive datasets, is revolutionizing the field of AI. Models such as SAM, DALL-E2, and GPT-4 showcase their adaptability by extracting intricate patterns and performing effectively across diverse tasks, thereby serving as potent building blocks for a wide range of AI applications. Autonomous driving, a vibrant front in AI applications, remains challenged by the lack of dedicated vision foundation models (VFMs). The scarcity of comprehensive training data, the need for multi-sensor integration, and the diverse task-specific architectures pose significant obstacles to the development of VFMs in this field. This paper delves into the critical challenge of forging VFMs tailored specifically for autonomous driving, while also outlining future directions. Through a systematic analysis of over 250 papers, we dissect essential techniques for VFM development, including data preparation, pre-training strategies, and downstream task adaptation. Moreover, we explore key advancements such as NeRF, diffusion models, 3D Gaussian Splatting, and world models, presenting a comprehensive roadmap for future research. To empower researchers, we have built and maintained https://github.com/zhanghm1995/Forge_VFM4AD, an open-access repository constantly updated with the latest advancements in forging VFMs for autonomous driving.

Title: Enhancing Robustness of LLM-Synthetic Text Detectors for Academic Writing: A Comprehensive Analysis

• Authors: Zhicheng Dou, Yuchen Guo, Ching-Chun Chang, Huy H. Nguyen, Isao Echizen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.08046
• Pdf URL: https://arxiv.org/pdf/2401.08046
• Copy Paste: [[2401.08046]] Enhancing Robustness of LLM-Synthetic Text Detectors for Academic Writing: A Comprehensive Analysis(https://arxiv.org/abs/2401.08046)
• Keywords: generative
• Abstract: The emergence of large language models (LLMs), such as Generative Pre-trained Transformer 4 (GPT-4) used by ChatGPT, has profoundly impacted the academic and broader community. While these models offer numerous advantages in terms of revolutionizing work and study methods, they have also garnered significant attention due to their potential negative consequences. One example is generating academic reports or papers with little to no human contribution. Consequently, researchers have focused on developing detectors to address the misuse of LLMs. However, most existing methods prioritize achieving higher accuracy on restricted datasets, neglecting the crucial aspect of generalizability. This limitation hinders their practical application in real-life scenarios where reliability is paramount. In this paper, we present a comprehensive analysis of the impact of prompts on the text generated by LLMs and highlight the potential lack of robustness in one of the current state-of-the-art GPT detectors. To mitigate these issues concerning the misuse of LLMs in academic writing, we propose a reference-based Siamese detector named Synthetic-Siamese which takes a pair of texts, one as the inquiry and the other as the reference. Our method effectively addresses the lack of robustness of previous detectors (OpenAI detector and DetectGPT) and significantly improves the baseline performances in realistic academic writing scenarios by approximately 67% to 95%.

Title: EmoTalker: Emotionally Editable Talking Face Generation via Diffusion Model

• Authors: Bingyuan Zhang, Xulong Zhang, Ning Cheng, Jun Yu, Jing Xiao, Jianzong Wang
• Subjects: cs.CV, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2401.08049
• Pdf URL: https://arxiv.org/pdf/2401.08049
• Copy Paste: [[2401.08049]] EmoTalker: Emotionally Editable Talking Face Generation via Diffusion Model(https://arxiv.org/abs/2401.08049)
• Keywords: diffusion
• Abstract: In recent years, the field of talking faces generation has attracted considerable attention, with certain methods adept at generating virtual faces that convincingly imitate human expressions. However, existing methods face challenges related to limited generalization, particularly when dealing with challenging identities. Furthermore, methods for editing expressions are often confined to a singular emotion, failing to adapt to intricate emotions. To overcome these challenges, this paper proposes EmoTalker, an emotionally editable portraits animation approach based on the diffusion model. EmoTalker modifies the denoising process to ensure preservation of the original portrait's identity during inference. To enhance emotion comprehension from text input, Emotion Intensity Block is introduced to analyze fine-grained emotions and strengths derived from prompts. Additionally, a crafted dataset is harnessed to enhance emotion comprehension within prompts. Experiments show the effectiveness of EmoTalker in generating high-quality, emotionally customizable facial expressions.

Title: SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generation

• Authors: Zhixuan Liu, Peter Schaldenbrand, Beverley-Claire Okogwu, Wenxuan Peng, Youngsik Yun, Andrew Hundt, Jihie Kim, Jean Oh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08053
• Pdf URL: https://arxiv.org/pdf/2401.08053
• Copy Paste: [[2401.08053]] SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generation(https://arxiv.org/abs/2401.08053)
• Keywords: diffusion, generative
• Abstract: Accurate representation in media is known to improve the well-being of the people who consume it. Generative image models trained on large web-crawled datasets such as LAION are known to produce images with harmful stereotypes and misrepresentations of cultures. We improve inclusive representation in generated images by (1) engaging with communities to collect a culturally representative dataset that we call the Cross-Cultural Understanding Benchmark (CCUB) and (2) proposing a novel Self-Contrastive Fine-Tuning (SCoFT) method that leverages the model's known biases to self-improve. SCoFT is designed to prevent overfitting on small datasets, encode only high-level information from the data, and shift the generated distribution away from misrepresentations encoded in a pretrained model. Our user study conducted on 51 participants from 5 different countries based on their self-selected national cultural affiliation shows that fine-tuning on CCUB consistently generates images with higher cultural relevance and fewer stereotypes when compared to the Stable Diffusion baseline, which is further improved with our SCoFT technique.

Title: Adversarial Masking Contrastive Learning for vein recognition

• Authors: Huafeng Qin, Yiquan Wu, Mounim A. El-Yacoubi, Jun Wang, Guangxiang Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08079
• Pdf URL: https://arxiv.org/pdf/2401.08079
• Copy Paste: [[2401.08079]] Adversarial Masking Contrastive Learning for vein recognition(https://arxiv.org/abs/2401.08079)
• Keywords: generative
• Abstract: Vein recognition has received increasing attention due to its high security and privacy. Recently, deep neural networks such as Convolutional neural networks (CNN) and Transformers have been introduced for vein recognition and achieved state-of-the-art performance. Despite the recent advances, however, existing solutions for finger-vein feature extraction are still not optimal due to scarce training image samples. To overcome this problem, in this paper, we propose an adversarial masking contrastive learning (AMCL) approach, that generates challenging samples to train a more robust contrastive learning model for the downstream palm-vein recognition task, by alternatively optimizing the encoder in the contrastive learning model and a set of latent variables. First, a huge number of masks are generated to train a robust generative adversarial network (GAN). The trained generator transforms a latent variable from the latent variable space into a mask space. Then, we combine the trained generator with a contrastive learning model to obtain our AMCL, where the generator produces challenging masking images to increase the contrastive loss and the contrastive learning model is trained based on the harder images to learn a more robust feature representation. After training, the trained encoder in the contrastive learning model is combined with a classification layer to build a classifier, which is further fine-tuned on labeled training data for vein recognition. The experimental results on three databases demonstrate that our approach outperforms existing contrastive learning approaches in terms of improving identification accuracy of vein classifiers and achieves state-of-the-art recognition results.

Title: UV-SAM: Adapting Segment Anything Model for Urban Village Identification

• Authors: Xin Zhang, Yu Liu, Yuming Lin, Qingming Liao, Yong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08083
• Pdf URL: https://arxiv.org/pdf/2401.08083
• Copy Paste: [[2401.08083]] UV-SAM: Adapting Segment Anything Model for Urban Village Identification(https://arxiv.org/abs/2401.08083)
• Keywords: foundation model
• Abstract: Urban villages, defined as informal residential areas in or around urban centers, are characterized by inadequate infrastructures and poor living conditions, closely related to the Sustainable Development Goals (SDGs) on poverty, adequate housing, and sustainable cities. Traditionally, governments heavily depend on field survey methods to monitor the urban villages, which however are time-consuming, labor-intensive, and possibly delayed. Thanks to widely available and timely updated satellite images, recent studies develop computer vision techniques to detect urban villages efficiently. However, existing studies either focus on simple urban village image classification or fail to provide accurate boundary information. To accurately identify urban village boundaries from satellite images, we harness the power of the vision foundation model and adapt the Segment Anything Model (SAM) to urban village segmentation, named UV-SAM. Specifically, UV-SAM first leverages a small-sized semantic segmentation model to produce mixed prompts for urban villages, including mask, bounding box, and image representations, which are then fed into SAM for fine-grained boundary identification. Extensive experimental results on two datasets in China demonstrate that UV-SAM outperforms existing baselines, and identification results over multiple years show that both the number and area of urban villages are decreasing over time, providing deeper insights into the development trends of urban villages and sheds light on the vision foundation models for sustainable cities. The dataset and codes of this study are available at https://github.com/tsinghua-fib-lab/UV-SAM.

Title: A Survey of Resource-efficient LLM and Multimodal Foundation Models

• Authors: Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, Qiyang Zhang, Zhenyan Lu, Li Zhang, Shangguang Wang, Yuanchun Li, Yunxin Liu, Xin Jin, Xuanzhe Liu
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.08092
• Pdf URL: https://arxiv.org/pdf/2401.08092
• Copy Paste: [[2401.08092]] A Survey of Resource-efficient LLM and Multimodal Foundation Models(https://arxiv.org/abs/2401.08092)
• Keywords: diffusion, foundation model
• Abstract: Large foundation models, including large language models (LLMs), vision transformers (ViTs), diffusion, and LLM-based multimodal models, are revolutionizing the entire machine learning lifecycle, from training to deployment. However, the substantial advancements in versatility and performance these models offer come at a significant cost in terms of hardware resources. To support the growth of these large models in a scalable and environmentally sustainable way, there has been a considerable focus on developing resource-efficient strategies. This survey delves into the critical importance of such research, examining both algorithmic and systemic aspects. It offers a comprehensive analysis and valuable insights gleaned from existing literature, encompassing a broad array of topics from cutting-edge model architectures and training/serving algorithms to practical system designs and implementations. The goal of this survey is to provide an overarching understanding of how current approaches are tackling the resource challenges posed by large foundation models and to potentially inspire future breakthroughs in this field.

Title: Inpainting Normal Maps for Lightstage data

• Authors: Hancheng Zuo, Bernard Tiddeman
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.08099
• Pdf URL: https://arxiv.org/pdf/2401.08099
• Copy Paste: [[2401.08099]] Inpainting Normal Maps for Lightstage data(https://arxiv.org/abs/2401.08099)
• Keywords: generative
• Abstract: This study introduces a novel method for inpainting normal maps using a generative adversarial network (GAN). Normal maps, often derived from a lightstage, are crucial in performance capture but can have obscured areas due to movement (e.g., by arms, hair, or props). Inpainting fills these missing areas with plausible data. Our approach extends previous general image inpainting techniques, employing a bow tie-like generator network and a discriminator network, with alternating training phases. The generator aims to synthesize images aligning with the ground truth and deceive the discriminator, which differentiates between real and processed images. Periodically, the discriminator undergoes retraining to enhance its ability to identify processed images. Importantly, our method adapts to the unique characteristics of normal map data, necessitating modifications to the loss function. We utilize a cosine loss instead of mean squared error loss for generator training. Limited training data availability, even with synthetic datasets, demands significant augmentation, considering the specific nature of the input data. This includes appropriate image flipping and in-plane rotations to accurately alter normal vectors. Throughout training, we monitored key metrics such as average loss, Structural Similarity Index Measure (SSIM), and Peak Signal-to-Noise Ratio (PSNR) for the generator, along with average loss and accuracy for the discriminator. Our findings suggest that the proposed model effectively generates high-quality, realistic inpainted normal maps, suitable for performance capture applications. These results establish a foundation for future research, potentially involving more advanced networks and comparisons with inpainting of source images used to create the normal maps.

Title: SpecSTG: A Fast Spectral Diffusion Framework for Probabilistic Spatio-Temporal Traffic Forecasting

• Authors: Lequan Lin, Dai Shi, Andi Han, Junbin Gao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08119
• Pdf URL: https://arxiv.org/pdf/2401.08119
• Copy Paste: [[2401.08119]] SpecSTG: A Fast Spectral Diffusion Framework for Probabilistic Spatio-Temporal Traffic Forecasting(https://arxiv.org/abs/2401.08119)
• Keywords: diffusion
• Abstract: Traffic forecasting, a crucial application of spatio-temporal graph (STG) learning, has traditionally relied on deterministic models for accurate point estimations. Yet, these models fall short of identifying latent risks of unexpected volatility in future observations. To address this gap, probabilistic methods, especially variants of diffusion models, have emerged as uncertainty-aware solutions. However, existing diffusion methods typically focus on generating separate future time series for individual sensors in the traffic network, resulting in insufficient involvement of spatial network characteristics in the probabilistic learning process. To better leverage spatial dependencies and systematic patterns inherent in traffic data, we propose SpecSTG, a novel spectral diffusion framework. Our method generates the Fourier representation of future time series, transforming the learning process into the spectral domain enriched with spatial information. Additionally, our approach incorporates a fast spectral graph convolution designed for Fourier input, alleviating the computational burden associated with existing models. Numerical experiments show that SpecSTG achieves outstanding performance with traffic flow and traffic speed datasets compared to state-of-the-art baselines. The source code for SpecSTG is available at https://anonymous.4open.science/r/SpecSTG.

Title: IoTWarden: A Deep Reinforcement Learning Based Real-time Defense System to Mitigate Trigger-action IoT Attacks

• Authors: Md Morshed Alam (1), Israt Jahan (2), Weichao Wang (1) ((1) Department of Software and Information Systems, University of North Carolina at Charlotte, Charlotte, USA, (2) Department of Computer Science, University of Memphis, Memphis, USA)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2401.08141
• Pdf URL: https://arxiv.org/pdf/2401.08141
• Copy Paste: [[2401.08141]] IoTWarden: A Deep Reinforcement Learning Based Real-time Defense System to Mitigate Trigger-action IoT Attacks(https://arxiv.org/abs/2401.08141)
• Keywords: anomaly
• Abstract: In trigger-action IoT platforms, IoT devices report event conditions to IoT hubs notifying their cyber states and let the hubs invoke actions in other IoT devices based on functional dependencies defined as rules in a rule engine. These functional dependencies create a chain of interactions that help automate network tasks. Adversaries exploit this chain to report fake event conditions to IoT hubs and perform remote injection attacks upon a smart environment to indirectly control targeted IoT devices. Existing defense efforts usually depend on static analysis over IoT apps to develop rule-based anomaly detection mechanisms. We also see ML-based defense mechanisms in the literature that harness physical event fingerprints to determine anomalies in an IoT network. However, these methods often demonstrate long response time and lack of adaptability when facing complicated attacks. In this paper, we propose to build a deep reinforcement learning based real-time defense system for injection attacks. We define the reward functions for defenders and implement a deep Q-network based approach to identify the optimal defense policy. Our experiments show that the proposed mechanism can effectively and accurately identify and defend against injection attacks with reasonable computation overhead.

Title: Completely Occluded and Dense Object Instance Segmentation Using Box Prompt-Based Segmentation Foundation Models

• Authors: Zhen Zhou, Junfeng Fan, Yunkai Ma, Sihan Zhao, Fengshui Jing, Min Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08174
• Pdf URL: https://arxiv.org/pdf/2401.08174
• Copy Paste: [[2401.08174]] Completely Occluded and Dense Object Instance Segmentation Using Box Prompt-Based Segmentation Foundation Models(https://arxiv.org/abs/2401.08174)
• Keywords: foundation model
• Abstract: Completely occluded and dense object instance segmentation (IS) is an important and challenging task. Although current amodal IS methods can predict invisible regions of occluded objects, they are difficult to directly predict completely occluded objects. For dense object IS, existing box-based methods are overly dependent on the performance of bounding box detection. In this paper, we propose CFNet, a coarse-to-fine IS framework for completely occluded and dense objects, which is based on box prompt-based segmentation foundation models (BSMs). Specifically, CFNet first detects oriented bounding boxes (OBBs) to distinguish instances and provide coarse localization information. Then, it predicts OBB prompt-related masks for fine segmentation. To predict completely occluded object instances, CFNet performs IS on occluders and utilizes prior geometric properties, which overcomes the difficulty of directly predicting completely occluded object instances. Furthermore, based on BSMs, CFNet reduces the dependence on bounding box detection performance, improving dense object IS performance. Moreover, we propose a novel OBB prompt encoder for BSMs. To make CFNet more lightweight, we perform knowledge distillation on it and introduce a Gaussian smoothing method for teacher targets. Experimental results demonstrate that CFNet achieves the best performance on both industrial and publicly available datasets.

Title: Key-point Guided Deformable Image Manipulation Using Diffusion Model

• Authors: Seok-Hwan Oh, Guil Jung, Myeong-Gee Kim, Sang-Yun Kim, Young-Min Kim, Hyeon-Jik Lee, Hyuk-Sool Kwon, Hyeon-Min Bae
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08178
• Pdf URL: https://arxiv.org/pdf/2401.08178
• Copy Paste: [[2401.08178]] Key-point Guided Deformable Image Manipulation Using Diffusion Model(https://arxiv.org/abs/2401.08178)
• Keywords: diffusion, generative
• Abstract: In this paper, we introduce a Key-point-guided Diffusion probabilistic Model (KDM) that gains precise control over images by manipulating the object's key-point. We propose a two-stage generative model incorporating an optical flow map as an intermediate output. By doing so, a dense pixel-wise understanding of the semantic relation between the image and sparse key point is configured, leading to more realistic image generation. Additionally, the integration of optical flow helps regulate the inter-frame variance of sequential images, demonstrating an authentic sequential image generation. The KDM is evaluated with diverse key-point conditioned image synthesis tasks, including facial image generation, human pose synthesis, and echocardiography video prediction, demonstrating the KDM is proving consistency enhanced and photo-realistic images compared with state-of-the-art models.

Title: Multi-scale 2D Temporal Map Diffusion Models for Natural Language Video Localization

• Authors: Chongzhi Zhang, Mingyuan Zhang, Zhiyang Teng, Jiayi Li, Xizhou Zhu, Lewei Lu, Ziwei Liu, Aixin Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08232
• Pdf URL: https://arxiv.org/pdf/2401.08232
• Copy Paste: [[2401.08232]] Multi-scale 2D Temporal Map Diffusion Models for Natural Language Video Localization(https://arxiv.org/abs/2401.08232)
• Keywords: diffusion
• Abstract: Natural Language Video Localization (NLVL), grounding phrases from natural language descriptions to corresponding video segments, is a complex yet critical task in video understanding. Despite ongoing advancements, many existing solutions lack the capability to globally capture temporal dynamics of the video data. In this study, we present a novel approach to NLVL that aims to address this issue. Our method involves the direct generation of a global 2D temporal map via a conditional denoising diffusion process, based on the input video and language query. The main challenges are the inherent sparsity and discontinuity of a 2D temporal map in devising the diffusion decoder. To address these challenges, we introduce a multi-scale technique and develop an innovative diffusion decoder. Our approach effectively encapsulates the interaction between the query and video data across various time scales. Experiments on the Charades and DiDeMo datasets underscore the potency of our design.

Title: A Generative Adversarial Attack for Multilingual Text Classifiers

• Authors: Tom Roth, Inigo Jauregi Unanue, Alsharif Abuadbba, Massimo Piccardi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.08255
• Pdf URL: https://arxiv.org/pdf/2401.08255
• Copy Paste: [[2401.08255]] A Generative Adversarial Attack for Multilingual Text Classifiers(https://arxiv.org/abs/2401.08255)
• Keywords: generative
• Abstract: Current adversarial attack algorithms, where an adversary changes a text to fool a victim model, have been repeatedly shown to be effective against text classifiers. These attacks, however, generally assume that the victim model is monolingual and cannot be used to target multilingual victim models, a significant limitation given the increased use of these models. For this reason, in this work we propose an approach to fine-tune a multilingual paraphrase model with an adversarial objective so that it becomes able to generate effective adversarial examples against multilingual classifiers. The training objective incorporates a set of pre-trained models to ensure text quality and language consistency of the generated text. In addition, all the models are suitably connected to the generator by vocabulary-mapping matrices, allowing for full end-to-end differentiability of the overall training pipeline. The experimental validation over two multilingual datasets and five languages has shown the effectiveness of the proposed approach compared to existing baselines, particularly in terms of query efficiency. We also provide a detailed analysis of the generated attacks and discuss limitations and opportunities for future research.

Title: Modeling Spoof Noise by De-spoofing Diffusion and its Application in Face Anti-spoofing

• Authors: Bin Zhang, Xiangyu Zhu, Xiaoyu Zhang, Zhen Lei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08275
• Pdf URL: https://arxiv.org/pdf/2401.08275
• Copy Paste: [[2401.08275]] Modeling Spoof Noise by De-spoofing Diffusion and its Application in Face Anti-spoofing(https://arxiv.org/abs/2401.08275)
• Keywords: diffusion
• Abstract: Face anti-spoofing is crucial for ensuring the security and reliability of face recognition systems. Several existing face anti-spoofing methods utilize GAN-like networks to detect presentation attacks by estimating the noise pattern of a spoof image and recovering the corresponding genuine image. But GAN's limited face appearance space results in the denoised faces cannot cover the full data distribution of genuine faces, thereby undermining the generalization performance of such methods. In this work, we present a pioneering attempt to employ diffusion models to denoise a spoof image and restore the genuine image. The difference between these two images is considered as the spoof noise, which can serve as a discriminative cue for face anti-spoofing. We evaluate our proposed method on several intra-testing and inter-testing protocols, where the experimental results showcase the effectiveness of our method in achieving competitive performance in terms of both accuracy and generalization.

Title: Generative Denoise Distillation: Simple Stochastic Noises Induce Efficient Knowledge Transfer for Dense Prediction

• Authors: Zhaoge Liu, Xiaohao Xu, Yunkang Cao, Weiming Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08332
• Pdf URL: https://arxiv.org/pdf/2401.08332
• Copy Paste: [[2401.08332]] Generative Denoise Distillation: Simple Stochastic Noises Induce Efficient Knowledge Transfer for Dense Prediction(https://arxiv.org/abs/2401.08332)
• Keywords: generative
• Abstract: Knowledge distillation is the process of transferring knowledge from a more powerful large model (teacher) to a simpler counterpart (student). Numerous current approaches involve the student imitating the knowledge of the teacher directly. However, redundancy still exists in the learned representations through these prevalent methods, which tend to learn each spatial location's features indiscriminately. To derive a more compact representation (concept feature) from the teacher, inspired by human cognition, we suggest an innovative method, termed Generative Denoise Distillation (GDD), where stochastic noises are added to the concept feature of the student to embed them into the generated instance feature from a shallow network. Then, the generated instance feature is aligned with the knowledge of the instance from the teacher. We extensively experiment with object detection, instance segmentation, and semantic segmentation to demonstrate the versatility and effectiveness of our method. Notably, GDD achieves new state-of-the-art performance in the tasks mentioned above. We have achieved substantial improvements in semantic segmentation by enhancing PspNet and DeepLabV3, both of which are based on ResNet-18, resulting in mIoU scores of 74.67 and 77.69, respectively, surpassing their previous scores of 69.85 and 73.20 on the Cityscapes dataset of 20 categories. The source code of GDD is available at https://github.com/ZhgLiu/GDD.

Title: Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference

• Authors: Jinghan Yao, Quentin Anthony, Aamir Shafi, Hari Subramoni, Dhabaleswar K. (DK)Panda
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.08383
• Pdf URL: https://arxiv.org/pdf/2401.08383
• Copy Paste: [[2401.08383]] Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference(https://arxiv.org/abs/2401.08383)
• Keywords: generative
• Abstract: In large language models like the Generative Pre-trained Transformer, the Mixture of Experts paradigm has emerged as a powerful technique for enhancing model expressiveness and accuracy. However, deploying GPT MoE models for parallel inference on distributed systems presents significant challenges, primarily due to the extensive Alltoall communication required for expert routing and aggregation. This communication bottleneck exacerbates the already complex computational landscape, hindering the efficient utilization of high-performance computing resources. In this paper, we propose a lightweight optimization technique called ExFlow, to largely accelerate the inference of these MoE models. We take a new perspective on alleviating the communication overhead by exploiting the inter-layer expert affinity. Unlike previous methods, our solution can be directly applied to pre-trained MoE models without any fine-tuning or accuracy degradation. By proposing a context-coherent expert parallelism on distributed systems, our design only uses one Alltoall communication to deliver the same functionality while previous methods all require two Alltoalls. By carefully examining the conditional probability in tokens' routing across multiple layers, we proved that pre-trained GPT MoE models implicitly exhibit a strong inter-layer expert affinity. We then design an efficient integer programming model to capture such features and show that by properly placing the experts on corresponding GPUs, we can reduce up to 67% cross-GPU routing latency. Our solution beats the cutting-edge MoE implementations with experts from 8 to 64, with up to 2.2x improvement in inference throughput. We further provide a detailed study of how the model implicitly acquires this expert affinity at the very early training stage and how this affinity evolves and stabilizes during training.

Title: Hidden Flaws Behind Expert-Level Accuracy of GPT-4 Vision in Medicine

• Authors: Qiao Jin, Fangyuan Chen, Yiliang Zhou, Ziyang Xu, Justin M. Cheung, Robert Chen, Ronald M. Summers, Justin F. Rousseau, Peiyun Ni, Marc J Landsman, Sally L. Baxter, Subhi J. Al'Aref, Yijia Li, Michael F. Chiang, Yifan Peng, Zhiyong Lu
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.08396
• Pdf URL: https://arxiv.org/pdf/2401.08396
• Copy Paste: [[2401.08396]] Hidden Flaws Behind Expert-Level Accuracy of GPT-4 Vision in Medicine(https://arxiv.org/abs/2401.08396)
• Keywords: generative
• Abstract: Recent studies indicate that Generative Pre-trained Transformer 4 with Vision (GPT-4V) outperforms human physicians in medical challenge tasks. However, these evaluations primarily focused on the accuracy of multi-choice questions alone. Our study extends the current scope by conducting a comprehensive analysis of GPT-4V's rationales of image comprehension, recall of medical knowledge, and step-by-step multimodal reasoning when solving New England Journal of Medicine (NEJM) Image Challenges - an imaging quiz designed to test the knowledge and diagnostic capabilities of medical professionals. Evaluation results confirmed that GPT-4V outperforms human physicians regarding multi-choice accuracy (88.0% vs. 77.0%, p=0.034). GPT-4V also performs well in cases where physicians incorrectly answer, with over 80% accuracy. However, we discovered that GPT-4V frequently presents flawed rationales in cases where it makes the correct final choices (27.3%), most prominent in image comprehension (21.6%). Regardless of GPT-4V's high accuracy in multi-choice questions, our findings emphasize the necessity for further in-depth evaluations of its rationales before integrating such models into clinical workflows.

Title: Machine Translation with Large Language Models: Prompt Engineering for Persian, English, and Russian Directions

• Authors: Nooshin Pourkamali, Shler Ebrahim Sharifi
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08429
• Pdf URL: https://arxiv.org/pdf/2401.08429
• Copy Paste: [[2401.08429]] Machine Translation with Large Language Models: Prompt Engineering for Persian, English, and Russian Directions(https://arxiv.org/abs/2401.08429)
• Keywords: generative, in-context
• Abstract: Generative large language models (LLMs) have demonstrated exceptional proficiency in various natural language processing (NLP) tasks, including machine translation, question answering, text summarization, and natural language understanding. To further enhance the performance of LLMs in machine translation, we conducted an investigation into two popular prompting methods and their combination, focusing on cross-language combinations of Persian, English, and Russian. We employed n-shot feeding and tailored prompting frameworks. Our findings indicate that multilingual LLMs like PaLM exhibit human-like machine translation outputs, enabling superior fine-tuning of desired translation nuances in accordance with style guidelines and linguistic considerations. These models also excel in processing and applying prompts. However, the choice of language model, machine translation task, and the specific source and target languages necessitate certain considerations when adopting prompting frameworks and utilizing n-shot in-context learning. Furthermore, we identified errors and limitations inherent in popular LLMs as machine translation tools and categorized them based on various linguistic metrics. This typology of errors provides valuable insights for utilizing LLMs effectively and offers methods for designing prompts for in-context learning. Our report aims to contribute to the advancement of machine translation with LLMs by improving both the accuracy and reliability of evaluation metrics.

Title: Instilling Multi-round Thinking to Text-guided Image Generation

• Authors: Lidong Zeng, Zhedong Zheng, Yinwei Wei, Tat-seng Chua
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08472
• Pdf URL: https://arxiv.org/pdf/2401.08472
• Copy Paste: [[2401.08472]] Instilling Multi-round Thinking to Text-guided Image Generation(https://arxiv.org/abs/2401.08472)
• Keywords: self-supervised
• Abstract: In this paper, we study the text-guided image generation task. Our focus lies in the modification of a reference image, given user text feedback, to imbue it with specific desired properties. Despite recent strides in this field, a persistent challenge remains that single-round optimization often overlooks crucial details, particularly in the realm of fine-grained changes like shoes or sleeves. This misalignment accumulation significantly hampers multi-round customization during interaction. In an attempt to address this challenge, we introduce a new self-supervised regularization into the existing framework, i.e., multi-round regularization. It builds upon the observation that the modification order does not affect the final result. As the name suggests, the multi-round regularization encourages the model to maintain consistency across different modification orders. Specifically, our proposed approach addresses the issue where an initial failure to capture fine-grained details leads to substantial discrepancies after multiple rounds, as opposed to traditional one-round learning. Both qualitative and quantitative experiments show the proposed method achieves high-fidelity generation quality over the text-guided generation task, especially the local modification. Furthermore, we extend the evaluation to semantic alignment with text by applying our method to text-guided retrieval datasets, such as FahisonIQ, where it demonstrates competitive performance.

Title: Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models

• Authors: Tassilo Klein, Moin Nabi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08491
• Pdf URL: https://arxiv.org/pdf/2401.08491
• Copy Paste: [[2401.08491]] Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models(https://arxiv.org/abs/2401.08491)
• Keywords: self-supervised
• Abstract: The generation of undesirable and factually incorrect content of large language models poses a significant challenge and remains largely an unsolved issue. This paper studies the integration of a contrastive learning objective for fine-tuning LLMs for implicit knowledge editing and controlled text generation. Optimizing the training objective entails aligning text perplexities in a contrastive fashion. To facilitate training the model in a self-supervised fashion, we leverage an off-the-shelf LLM for training data generation. We showcase applicability in the domain of detoxification. Herein, the proposed approach leads to a significant decrease in the generation of toxic content while preserving general utility for downstream tasks such as commonsense reasoning and reading comprehension. The proposed approach is conceptually simple but empirically powerful.

Title: Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis

• Authors: Zhenhui Ye, Tianyun Zhong, Yi Ren, Jiaqi Yang, Weichuang Li, Jiawei Huang, Ziyue Jiang, Jinzheng He, Rongjie Huang, Jinglin Liu, Chen Zhang, Xiang Yin, Zejun Ma, Zhou Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08503
• Pdf URL: https://arxiv.org/pdf/2401.08503
• Copy Paste: [[2401.08503]] Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis(https://arxiv.org/abs/2401.08503)
• Keywords: generative
• Abstract: One-shot 3D talking portrait generation aims to reconstruct a 3D avatar from an unseen image, and then animate it with a reference video or audio to generate a talking portrait video. The existing methods fail to simultaneously achieve the goals of accurate 3D avatar reconstruction and stable talking face animation. Besides, while the existing works mainly focus on synthesizing the head part, it is also vital to generate natural torso and background segments to obtain a realistic talking portrait video. To address these limitations, we present Real3D-Potrait, a framework that (1) improves the one-shot 3D reconstruction power with a large image-to-plane model that distills 3D prior knowledge from a 3D face generative model; (2) facilitates accurate motion-conditioned animation with an efficient motion adapter; (3) synthesizes realistic video with natural torso movement and switchable background using a head-torso-background super-resolution model; and (4) supports one-shot audio-driven talking face generation with a generalizable audio-to-motion model. Extensive experiments show that Real3D-Portrait generalizes well to unseen identities and generates more realistic talking portrait videos compared to previous methods.

Title: The Gaps between Pre-train and Downstream Settings in Bias Evaluation and Debiasing

• Authors: Masahiro Kaneko, Danushka Bollegala, Timothy Baldwin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.08511
• Pdf URL: https://arxiv.org/pdf/2401.08511
• Copy Paste: [[2401.08511]] The Gaps between Pre-train and Downstream Settings in Bias Evaluation and Debiasing(https://arxiv.org/abs/2401.08511)
• Keywords: in-context
• Abstract: The output tendencies of Pre-trained Language Models (PLM) vary markedly before and after Fine-Tuning (FT) due to the updates to the model parameters. These divergences in output tendencies result in a gap in the social biases of PLMs. For example, there exits a low correlation between intrinsic bias scores of a PLM and its extrinsic bias scores under FT-based debiasing methods. Additionally, applying FT-based debiasing methods to a PLM leads to a decline in performance in downstream tasks. On the other hand, PLMs trained on large datasets can learn without parameter updates via In-Context Learning (ICL) using prompts. ICL induces smaller changes to PLMs compared to FT-based debiasing methods. Therefore, we hypothesize that the gap observed in pre-trained and FT models does not hold true for debiasing methods that use ICL. In this study, we demonstrate that ICL-based debiasing methods show a higher correlation between intrinsic and extrinsic bias scores compared to FT-based methods. Moreover, the performance degradation due to debiasing is also lower in the ICL case compared to that in the FT case.

Title: Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation

• Authors: Mathis Petrovich, Or Litany, Umar Iqbal, Michael J. Black, Gül Varol, Xue Bin Peng, Davis Rempe
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08559
• Pdf URL: https://arxiv.org/pdf/2401.08559
• Copy Paste: [[2401.08559]] Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation(https://arxiv.org/abs/2401.08559)
• Keywords: diffusion, generative
• Abstract: Recent advances in generative modeling have led to promising progress on synthesizing 3D human motion from text, with methods that can generate character animations from short prompts and specified durations. However, using a single text prompt as input lacks the fine-grained control needed by animators, such as composing multiple actions and defining precise durations for parts of the motion. To address this, we introduce the new problem of timeline control for text-driven motion synthesis, which provides an intuitive, yet fine-grained, input interface for users. Instead of a single prompt, users can specify a multi-track timeline of multiple prompts organized in temporal intervals that may overlap. This enables specifying the exact timings of each action and composing multiple actions in sequence or at overlapping intervals. To generate composite animations from a multi-track timeline, we propose a new test-time denoising method. This method can be integrated with any pre-trained motion diffusion model to synthesize realistic motions that accurately reflect the timeline. At every step of denoising, our method processes each timeline interval (text prompt) individually, subsequently aggregating the predictions with consideration for the specific body parts engaged in each action. Experimental comparisons and ablations validate that our method produces realistic motions that respect the semantics and timing of given text prompts. Our code and models are publicly available at https://mathis.petrovich.fr/stmc.

Title: ADVENT: Attack/Anomaly Detection in VANETs

• Authors: Hamideh Baharlouei, Adetokunbo Makanju, Nur Zincir-Heywood
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.08564
• Pdf URL: https://arxiv.org/pdf/2401.08564
• Copy Paste: [[2401.08564]] ADVENT: Attack/Anomaly Detection in VANETs(https://arxiv.org/abs/2401.08564)
• Keywords: anomaly
• Abstract: In the domain of Vehicular Ad hoc Networks (VANETs), where the imperative of having a real-world malicious detector capable of detecting attacks in real-time and unveiling their perpetrators is crucial, our study introduces a system with this goal. This system is designed for real-time detection of malicious behavior, addressing the critical need to first identify the onset of attacks and subsequently the responsible actors. Prior work in this area have never addressed both requirements, which we believe are necessary for real world deployment, simultaneously. By seamlessly integrating statistical and machine learning techniques, the proposed system prioritizes simplicity and efficiency. It excels in swiftly detecting attack onsets with a remarkable F1-score of 99.66%, subsequently identifying malicious vehicles with an average F1-score of approximately 97.85%. Incorporating federated learning in both stages enhances privacy and improves the efficiency of malicious node detection, effectively reducing the false negative rate.

Title: RoHM: Robust Human Motion Reconstruction via Diffusion

• Authors: Siwei Zhang, Bharat Lal Bhatnagar, Yuanlu Xu, Alexander Winkler, Petr Kadlecek, Siyu Tang, Federica Bogo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.08570
• Pdf URL: https://arxiv.org/pdf/2401.08570
• Copy Paste: [[2401.08570]] RoHM: Robust Human Motion Reconstruction via Diffusion(https://arxiv.org/abs/2401.08570)
• Keywords: diffusion
• Abstract: We propose RoHM, an approach for robust 3D human motion reconstruction from monocular RGB(-D) videos in the presence of noise and occlusions. Most previous approaches either train neural networks to directly regress motion in 3D or learn data-driven motion priors and combine them with optimization at test time. The former do not recover globally coherent motion and fail under occlusions; the latter are time-consuming, prone to local minima, and require manual tuning. To overcome these shortcomings, we exploit the iterative, denoising nature of diffusion models. RoHM is a novel diffusion-based motion model that, conditioned on noisy and occluded input data, reconstructs complete, plausible motions in consistent global coordinates. Given the complexity of the problem -- requiring one to address different tasks (denoising and infilling) in different solution spaces (local and global motion) -- we decompose it into two sub-tasks and learn two models, one for global trajectory and one for local motion. To capture the correlations between the two, we then introduce a novel conditioning module, combining it with an iterative inference scheme. We apply RoHM to a variety of tasks -- from motion reconstruction and denoising to spatial and temporal infilling. Extensive experiments on three popular datasets show that our method outperforms state-of-the-art approaches qualitatively and quantitatively, while being faster at test time. The code will be available at https://sanweiliti.github.io/ROHM/ROHM.html.

Title: Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability

• Authors: Afra Feyza Akyürek, Ekin Akyürek, Leshem Choshen, Derry Wijaya, Jacob Andreas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.08574
• Pdf URL: https://arxiv.org/pdf/2401.08574
• Copy Paste: [[2401.08574]] Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability(https://arxiv.org/abs/2401.08574)
• Keywords: self-supervised
• Abstract: While language models (LMs) can sometimes generate factually correct text and estimate truth values of individual claims, these generally do not reflect a globally coherent, manipulable model of the world. As a consequence, current LMs also generate incorrect or nonsensical content, and are difficult to edit and bring up to date. We present a method called Deductive Closure Training (DCT) that uses LMs themselves to identify implications of (and contradictions within) the text that they generate, yielding an efficient self-supervised procedure for improving LM factuality. Given a collection of seed documents, DCT prompts LMs to generate additional text implied by these documents, reason globally about the correctness of this generated text, and finally fine-tune on text inferred to be correct. Given seed documents from a trusted source, DCT provides a tool for supervised model updating; if seed documents are sampled from the LM itself, DCT enables fully unsupervised fine-tuning for improved coherence and accuracy. Across the CREAK, MQUaKE, and Reversal Curse datasets, supervised DCT improves LM fact verification and text generation accuracy by 3-26%; on CREAK fully unsupervised DCT improves verification accuracy by 12%. These results show that LMs' reasoning capabilities during inference can be leveraged during training to improve their reliability.