2024-04-09

Title: Similar Data Points Identification with LLM: A Human-in-the-loop Strategy Using Summarization and Hidden State Insights

• Authors: Xianlong Zeng, Fanghao Song, Ang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04281
• Pdf URL: https://arxiv.org/pdf/2404.04281
• Copy Paste: [[2404.04281]] Similar Data Points Identification with LLM: A Human-in-the-loop Strategy Using Summarization and Hidden State Insights(https://arxiv.org/abs/2404.04281)
• Keywords: generative
• Abstract: This study introduces a simple yet effective method for identifying similar data points across non-free text domains, such as tabular and image data, using Large Language Models (LLMs). Our two-step approach involves data point summarization and hidden state extraction. Initially, data is condensed via summarization using an LLM, reducing complexity and highlighting essential information in sentences. Subsequently, the summarization sentences are fed through another LLM to extract hidden states, serving as compact, feature-rich representations. This approach leverages the advanced comprehension and generative capabilities of LLMs, offering a scalable and efficient strategy for similarity identification across diverse datasets. We demonstrate the effectiveness of our method in identifying similar data points on multiple datasets. Additionally, our approach enables non-technical domain experts, such as fraud investigators or marketing operators, to quickly identify similar data points tailored to specific scenarios, demonstrating its utility in practical applications. In general, our results open new avenues for leveraging LLMs in data analysis across various domains.

Title: A Real-time Anomaly Detection Using Convolutional Autoencoder with Dynamic Threshold

• Authors: Sarit Maitra, Sukanya Kundu, Aishwarya Shankar
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04311
• Pdf URL: https://arxiv.org/pdf/2404.04311
• Copy Paste: [[2404.04311]] A Real-time Anomaly Detection Using Convolutional Autoencoder with Dynamic Threshold(https://arxiv.org/abs/2404.04311)
• Keywords: anomaly
• Abstract: The majority of modern consumer-level energy is generated by real-time smart metering systems. These frequently contain anomalies, which prevent reliable estimates of the series' evolution. This work introduces a hybrid modeling approach combining statistics and a Convolutional Autoencoder with a dynamic threshold. The threshold is determined based on Mahalanobis distance and moving averages. It has been tested using real-life energy consumption data collected from smart metering systems. The solution includes a real-time, meter-level anomaly detection system that connects to an advanced monitoring system. This makes a substantial contribution by detecting unusual data movements and delivering an early warning. Early detection and subsequent troubleshooting can financially benefit organizations and consumers and prevent disasters from occurring.

Title: Koala: Key frame-conditioned long video-LLM

• Authors: Reuben Tan, Ximeng Sun, Ping Hu, Jui-hsien Wang, Hanieh Deilamsalehy, Bryan A. Plummer, Bryan Russell, Kate Saenko
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04346
• Pdf URL: https://arxiv.org/pdf/2404.04346
• Copy Paste: [[2404.04346]] Koala: Key frame-conditioned long video-LLM(https://arxiv.org/abs/2404.04346)
• Keywords: self-supervised
• Abstract: Long video question answering is a challenging task that involves recognizing short-term activities and reasoning about their fine-grained relationships. State-of-the-art video Large Language Models (vLLMs) hold promise as a viable solution due to their demonstrated emergent capabilities on new tasks. However, despite being trained on millions of short seconds-long videos, vLLMs are unable to understand minutes-long videos and accurately answer questions about them. To address this limitation, we propose a lightweight and self-supervised approach, Key frame-conditioned long video-LLM (Koala), that introduces learnable spatiotemporal queries to adapt pretrained vLLMs for generalizing to longer videos. Our approach introduces two new tokenizers that condition on visual tokens computed from sparse video key frames for understanding short and long video moments. We train our proposed approach on HowTo100M and demonstrate its effectiveness on zero-shot long video understanding benchmarks, where it outperforms state-of-the-art large models by 3 - 6% in absolute accuracy across all tasks. Surprisingly, we also empirically show that our approach not only helps a pretrained vLLM to understand long videos but also improves its accuracy on short-term action recognition.

Title: Assisting humans in complex comparisons: automated information comparison at scale

• Authors: Truman Yuen, Graham A. Watt, Yuri Lawryshyn
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04351
• Pdf URL: https://arxiv.org/pdf/2404.04351
• Copy Paste: [[2404.04351]] Assisting humans in complex comparisons: automated information comparison at scale(https://arxiv.org/abs/2404.04351)
• Keywords: generative
• Abstract: Generative Large Language Models enable efficient analytics across knowledge domains, rivalling human experts in information comparisons. However, the applications of LLMs for information comparisons face scalability challenges due to the difficulties in maintaining information across large contexts and overcoming model token limitations. To address these challenges, we developed the novel Abstractive Summarization \& Criteria-driven Comparison Endpoint (ASC$^2$End) system to automate information comparison at scale. Our system employs Semantic Text Similarity comparisons for generating evidence-supported analyses. We utilize proven data-handling strategies such as abstractive summarization and retrieval augmented generation to overcome token limitations and retain relevant information during model inference. Prompts were designed using zero-shot strategies to contextualize information for improved model reasoning. We evaluated abstractive summarization using ROUGE scoring and assessed the generated comparison quality using survey responses. Models evaluated on the ASC$^2$End system show desirable results providing insights on the expected performance of the system. ASC$^2$End is a novel system and tool that enables accurate, automated information comparison at scale across knowledge domains, overcoming limitations in context length and retrieval.

Title: Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedback

• Authors: Mo Kordzanganeh, Danial Keshvary, Nariman Arian
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04356
• Pdf URL: https://arxiv.org/pdf/2404.04356
• Copy Paste: [[2404.04356]] Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedback(https://arxiv.org/abs/2404.04356)
• Keywords: diffusion
• Abstract: Latent diffusion models are the state-of-the-art for synthetic image generation. To align these models with human preferences, training the models using reinforcement learning on human feedback is crucial. Black et. al 2024 introduced denoising diffusion policy optimisation (DDPO), which accounts for the iterative denoising nature of the generation by modelling it as a Markov chain with a final reward. As the reward is a single value that determines the model's performance on the entire image, the model has to navigate a very sparse reward landscape and so requires a large sample count. In this work, we extend the DDPO by presenting the Pixel-wise Policy Optimisation (PXPO) algorithm, which can take feedback for each pixel, providing a more nuanced reward to the model.

Title: Deciphering Political Entity Sentiment in News with Large Language Models: Zero-Shot and Few-Shot Strategies

• Authors: Alapan Kuila, Sudeshna Sarkar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04361
• Pdf URL: https://arxiv.org/pdf/2404.04361
• Copy Paste: [[2404.04361]] Deciphering Political Entity Sentiment in News with Large Language Models: Zero-Shot and Few-Shot Strategies(https://arxiv.org/abs/2404.04361)
• Keywords: in-context
• Abstract: Sentiment analysis plays a pivotal role in understanding public opinion, particularly in the political domain where the portrayal of entities in news articles influences public perception. In this paper, we investigate the effectiveness of Large Language Models (LLMs) in predicting entity-specific sentiment from political news articles. Leveraging zero-shot and few-shot strategies, we explore the capability of LLMs to discern sentiment towards political entities in news content. Employing a chain-of-thought (COT) approach augmented with rationale in few-shot in-context learning, we assess whether this method enhances sentiment prediction accuracy. Our evaluation on sentiment-labeled datasets demonstrates that LLMs, outperform fine-tuned BERT models in capturing entity-specific sentiment. We find that learning in-context significantly improves model performance, while the self-consistency mechanism enhances consistency in sentiment prediction. Despite the promising results, we observe inconsistencies in the effectiveness of the COT prompting method. Overall, our findings underscore the potential of LLMs in entity-centric sentiment analysis within the political news domain and highlight the importance of suitable prompting strategies and model architectures.

Title: ClickDiffusion: Harnessing LLMs for Interactive Precise Image Editing

• Authors: Alec Helbling, Seongmin Lee, Polo Chau
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04376
• Pdf URL: https://arxiv.org/pdf/2404.04376
• Copy Paste: [[2404.04376]] ClickDiffusion: Harnessing LLMs for Interactive Precise Image Editing(https://arxiv.org/abs/2404.04376)
• Keywords: diffusion
• Abstract: Recently, researchers have proposed powerful systems for generating and manipulating images using natural language instructions. However, it is difficult to precisely specify many common classes of image transformations with text alone. For example, a user may wish to change the location and breed of a particular dog in an image with several similar dogs. This task is quite difficult with natural language alone, and would require a user to write a laboriously complex prompt that both disambiguates the target dog and describes the destination. We propose ClickDiffusion, a system for precise image manipulation and generation that combines natural language instructions with visual feedback provided by the user through a direct manipulation interface. We demonstrate that by serializing both an image and a multi-modal instruction into a textual representation it is possible to leverage LLMs to perform precise transformations of the layout and appearance of an image. Code available at https://github.com/poloclub/ClickDiffusion.

Title: Increased LLM Vulnerabilities from Fine-tuning and Quantization

• Authors: Divyanshu Kumar, Anurakt Kumar, Sahil Agarwal, Prashanth Harshangi
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04392
• Pdf URL: https://arxiv.org/pdf/2404.04392
• Copy Paste: [[2404.04392]] Increased LLM Vulnerabilities from Fine-tuning and Quantization(https://arxiv.org/abs/2404.04392)
• Keywords: foundation model
• Abstract: Large Language Models (LLMs) have become very popular and have found use cases in many domains, such as chatbots, auto-task completion agents, and much more. However, LLMs are vulnerable to different types of attacks, such as jailbreaking, prompt injection attacks, and privacy leakage attacks. Foundational LLMs undergo adversarial and alignment training to learn not to generate malicious and toxic content. For specialized use cases, these foundational LLMs are subjected to fine-tuning or quantization for better performance and efficiency. We examine the impact of downstream tasks such as fine-tuning and quantization on LLM vulnerability. We test foundation models like Mistral, Llama, MosaicML, and their fine-tuned versions. Our research shows that fine-tuning and quantization reduces jailbreak resistance significantly, leading to increased LLM vulnerabilities. Finally, we demonstrate the utility of external guardrails in reducing LLM vulnerabilities.

Title: PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos

• Authors: Yufei Zhang, Jeffrey O. Kephart, Zijun Cui, Qiang Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04430
• Pdf URL: https://arxiv.org/pdf/2404.04430
• Copy Paste: [[2404.04430]] PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos(https://arxiv.org/abs/2404.04430)
• Keywords: self-supervised
• Abstract: While current methods have shown promising progress on estimating 3D human motion from monocular videos, their motion estimates are often physically unrealistic because they mainly consider kinematics. In this paper, we introduce Physics-aware Pretrained Transformer (PhysPT), which improves kinematics-based motion estimates and infers motion forces. PhysPT exploits a Transformer encoder-decoder backbone to effectively learn human dynamics in a self-supervised manner. Moreover, it incorporates physics principles governing human motion. Specifically, we build a physics-based body representation and contact force model. We leverage them to impose novel physics-inspired training losses (i.e., force loss, contact loss, and Euler-Lagrange loss), enabling PhysPT to capture physical properties of the human body and the forces it experiences. Experiments demonstrate that, once trained, PhysPT can be directly applied to kinematics-based estimates to significantly enhance their physical plausibility and generate favourable motion forces. Furthermore, we show that these physically meaningful quantities translate into improved accuracy of an important downstream task: human action recognition.

Title: Aligning Diffusion Models by Optimizing Human Utility

• Authors: Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, Kazuki Kozuka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04465
• Pdf URL: https://arxiv.org/pdf/2404.04465
• Copy Paste: [[2404.04465]] Aligning Diffusion Models by Optimizing Human Utility(https://arxiv.org/abs/2404.04465)
• Keywords: diffusion
• Abstract: We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Since this objective applies to each generation independently, Diffusion-KTO does not require collecting costly pairwise preference data nor training a complex reward model. Instead, our objective requires simple per-image binary feedback signals, e.g. likes or dislikes, which are abundantly available. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit superior performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary signals and broadens the applicability of aligning text-to-image diffusion models with human preferences.

Title: Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Models

• Authors: Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, Junshi Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04478
• Pdf URL: https://arxiv.org/pdf/2404.04478
• Copy Paste: [[2404.04478]] Diffusion-RWKV: Scaling RWKV-Like Architectures for Diffusion Models(https://arxiv.org/abs/2404.04478)
• Keywords: diffusion
• Abstract: Transformers have catalyzed advancements in computer vision and natural language processing (NLP) fields. However, substantial computational complexity poses limitations for their application in long-context tasks, such as high-resolution image generation. This paper introduces a series of architectures adapted from the RWKV model used in the NLP, with requisite modifications tailored for diffusion model applied to image generation tasks, referred to as Diffusion-RWKV. Similar to the diffusion with Transformers, our model is designed to efficiently handle patchnified inputs in a sequence with extra conditions, while also scaling up effectively, accommodating both large-scale parameters and extensive datasets. Its distinctive advantage manifests in its reduced spatial aggregation complexity, rendering it exceptionally adept at processing high-resolution images, thereby eliminating the necessity for windowing or group cached operations. Experimental results on both condition and unconditional image generation tasks demonstrate that Diffison-RWKV achieves performance on par with or surpasses existing CNN or Transformer-based diffusion models in FID and IS metrics while significantly reducing total computation FLOP usage.

Title: Latent-based Diffusion Model for Long-tailed Recognition

• Authors: Pengxiao Han, Changkun Ye, Jieming Zhou, Jing Zhang, Jie Hong, Xuesong Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04517
• Pdf URL: https://arxiv.org/pdf/2404.04517
• Copy Paste: [[2404.04517]] Latent-based Diffusion Model for Long-tailed Recognition(https://arxiv.org/abs/2404.04517)
• Keywords: diffusion
• Abstract: Long-tailed imbalance distribution is a common issue in practical computer vision applications. Previous works proposed methods to address this problem, which can be categorized into several classes: re-sampling, re-weighting, transfer learning, and feature augmentation. In recent years, diffusion models have shown an impressive generation ability in many sub-problems of deep computer vision. However, its powerful generation has not been explored in long-tailed problems. We propose a new approach, the Latent-based Diffusion Model for Long-tailed Recognition (LDMLR), as a feature augmentation method to tackle the issue. First, we encode the imbalanced dataset into features using the baseline model. Then, we train a Denoising Diffusion Implicit Model (DDIM) using these encoded features to generate pseudo-features. Finally, we train the classifier using the encoded and pseudo-features from the previous two steps. The model's accuracy shows an improvement on the CIFAR-LT and ImageNet-LT datasets by using the proposed method.

Title: MedIAnomaly: A comparative study of anomaly detection in medical images

• Authors: Yu Cai, Weiwen Zhang, Hao Chen, Kwang-Ting Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04518
• Pdf URL: https://arxiv.org/pdf/2404.04518
• Copy Paste: [[2404.04518]] MedIAnomaly: A comparative study of anomaly detection in medical images(https://arxiv.org/abs/2404.04518)
• Keywords: self-supervised, anomaly
• Abstract: Anomaly detection (AD) aims at detecting abnormal samples that deviate from the expected normal patterns. Generally, it can be trained on merely normal data without the requirement for abnormal samples, and thereby plays an important role in the recognition of rare diseases and health screening in the medical domain. Despite numerous related studies, we observe a lack of a fair and comprehensive evaluation, which causes some ambiguous conclusions and hinders the development of this field. This paper focuses on building a benchmark with unified implementation and comparison to address this problem. In particular, seven medical datasets with five image modalities, including chest X-rays, brain MRIs, retinal fundus images, dermatoscopic images, and histopathology whole slide images are organized for extensive evaluation. Twenty-seven typical AD methods, including reconstruction and self-supervised learning-based methods, are involved in comparison of image-level anomaly classification and pixel-level anomaly segmentation. Furthermore, we for the first time formally explore the effect of key components in existing methods, clearly revealing unresolved challenges and potential future directions. The datasets and code are available at \url{https://github.com/caiyu6666/MedIAnomaly}.

Title: DATENeRF: Depth-Aware Text-based Editing of NeRFs

• Authors: Sara Rojas, Julien Philip, Kai Zhang, Sai Bi, Fujun Luan, Bernard Ghanem, Kalyan Sunkavall
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04526
• Pdf URL: https://arxiv.org/pdf/2404.04526
• Copy Paste: [[2404.04526]] DATENeRF: Depth-Aware Text-based Editing of NeRFs(https://arxiv.org/abs/2404.04526)
• Keywords: diffusion
• Abstract: Recent advancements in diffusion models have shown remarkable proficiency in editing 2D images based on text prompts. However, extending these techniques to edit scenes in Neural Radiance Fields (NeRF) is complex, as editing individual 2D frames can result in inconsistencies across multiple views. Our crucial insight is that a NeRF scene's geometry can serve as a bridge to integrate these 2D edits. Utilizing this geometry, we employ a depth-conditioned ControlNet to enhance the coherence of each 2D image modification. Moreover, we introduce an inpainting approach that leverages the depth information of NeRF scenes to distribute 2D edits across different images, ensuring robustness against errors and resampling challenges. Our results reveal that this methodology achieves more consistent, lifelike, and detailed edits than existing leading methods for text-driven NeRF scene editing.

Title: Frequency Decomposition-Driven Unsupervised Domain Adaptation for Remote Sensing Image Semantic Segmentation

• Authors: Xianping Ma, Xiaokang Zhang, Xingchen Ding, Man-On Pun, Siwei Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04531
• Pdf URL: https://arxiv.org/pdf/2404.04531
• Copy Paste: [[2404.04531]] Frequency Decomposition-Driven Unsupervised Domain Adaptation for Remote Sensing Image Semantic Segmentation(https://arxiv.org/abs/2404.04531)
• Keywords: generative
• Abstract: Cross-domain semantic segmentation of remote sensing (RS) imagery based on unsupervised domain adaptation (UDA) techniques has significantly advanced deep-learning applications in the geosciences. Recently, with its ingenious and versatile architecture, the Transformer model has been successfully applied in RS-UDA tasks. However, existing UDA methods mainly focus on domain alignment in the high-level feature space. It is still challenging to retain cross-domain local spatial details and global contextual semantics simultaneously, which is crucial for the RS image semantic segmentation task. To address these problems, we propose novel high/low-frequency decomposition (HLFD) techniques to guide representation alignment in cross-domain semantic segmentation. Specifically, HLFD attempts to decompose the feature maps into high- and low-frequency components before performing the domain alignment in the corresponding subspaces. Secondly, to further facilitate the alignment of decomposed features, we propose a fully global-local generative adversarial network, namely GLGAN, to learn domain-invariant detailed and semantic features across domains by leveraging global-local transformer blocks (GLTBs). By integrating HLFD techniques and the GLGAN, a novel UDA framework called FD-GLGAN is developed to improve the cross-domain transferability and generalization capability of semantic segmentation models. Extensive experiments on two fine-resolution benchmark datasets, namely ISPRS Potsdam and ISPRS Vaihingen, highlight the effectiveness and superiority of the proposed approach as compared to the state-of-the-art UDA methods. The source code for this work will be accessible at https://github.com/sstary/SSRS.

Title: BeyondScene: Higher-Resolution Human-Centric Scene Generation With Pretrained Diffusion

• Authors: Gwanghyun Kim, Hayeon Kim, Hoigi Seo, Dong Un Kang, Se Young Chun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04544
• Pdf URL: https://arxiv.org/pdf/2404.04544
• Copy Paste: [[2404.04544]] BeyondScene: Higher-Resolution Human-Centric Scene Generation With Pretrained Diffusion(https://arxiv.org/abs/2404.04544)
• Keywords: diffusion
• Abstract: Generating higher-resolution human-centric scenes with details and controls remains a challenge for existing text-to-image diffusion models. This challenge stems from limited training image size, text encoder capacity (limited tokens), and the inherent difficulty of generating complex scenes involving multiple humans. While current methods attempted to address training size limit only, they often yielded human-centric scenes with severe artifacts. We propose BeyondScene, a novel framework that overcomes prior limitations, generating exquisite higher-resolution (over 8K) human-centric scenes with exceptional text-image correspondence and naturalness using existing pretrained diffusion models. BeyondScene employs a staged and hierarchical approach to initially generate a detailed base image focusing on crucial elements in instance creation for multiple humans and detailed descriptions beyond token limit of diffusion model, and then to seamlessly convert the base image to a higher-resolution output, exceeding training image size and incorporating details aware of text and instances via our novel instance-aware hierarchical enlargement process that consists of our proposed high-frequency injected forward diffusion and adaptive joint diffusion. BeyondScene surpasses existing methods in terms of correspondence with detailed text descriptions and naturalness, paving the way for advanced applications in higher-resolution human-centric scene creation beyond the capacity of pretrained diffusion models without costly retraining. Project page: https://janeyeon.github.io/beyond-scene.

Title: Diffusion Time-step Curriculum for One Image to 3D Generation

• Authors: Xuanyu Yi, Zike Wu, Qingshan Xu, Pan Zhou, Joo-Hwee Lim, Hanwang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04562
• Pdf URL: https://arxiv.org/pdf/2404.04562
• Copy Paste: [[2404.04562]] Diffusion Time-step Curriculum for One Image to 3D Generation(https://arxiv.org/abs/2404.04562)
• Keywords: diffusion
• Abstract: Score distillation sampling~(SDS) has been widely adopted to overcome the absence of unseen views in reconstructing 3D objects from a \textbf{single} image. It leverages pre-trained 2D diffusion models as teacher to guide the reconstruction of student 3D models. Despite their remarkable success, SDS-based methods often encounter geometric artifacts and texture saturation. We find out the crux is the overlooked indiscriminate treatment of diffusion time-steps during optimization: it unreasonably treats the student-teacher knowledge distillation to be equal at all time-steps and thus entangles coarse-grained and fine-grained modeling. Therefore, we propose the Diffusion Time-step Curriculum one-image-to-3D pipeline (DTC123), which involves both the teacher and student models collaborating with the time-step curriculum in a coarse-to-fine manner. Extensive experiments on NeRF4, RealFusion15, GSO and Level50 benchmark demonstrate that DTC123 can produce multi-view consistent, high-quality, and diverse 3D assets. Codes and more generation demos will be released in https://github.com/yxymessi/DTC123.

Title: To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO

• Authors: Zi-Hao Qiu, Siqi Guo, Mao Xu, Tuo Zhao, Lijun Zhang, Tianbao Yang
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2404.04575
• Pdf URL: https://arxiv.org/pdf/2404.04575
• Copy Paste: [[2404.04575]] To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO(https://arxiv.org/abs/2404.04575)
• Keywords: foundation model
• Abstract: The temperature parameter plays a profound role during training and/or inference with large foundation models (LFMs) such as large language models (LLMs) and CLIP models. Particularly, it adjusts the logits in the softmax function in LLMs, which is crucial for next token generation, and it scales the similarities in the contrastive loss for training CLIP models. A significant question remains: Is it viable to learn a neural network to predict a personalized temperature of any input data for enhancing LFMs"? In this paper, we present a principled framework for learning a small yet generalizable temperature prediction network (TempNet) to improve LFMs. Our solution is composed of a novel learning framework with a robust loss underpinned by constrained distributionally robust optimization (DRO), and a properly designed TempNet with theoretical inspiration. TempNet can be trained together with a large foundation model from scratch or learned separately given a pretrained foundation model. It is not only useful for predicting personalized temperature to promote the training of LFMs but also generalizable and transferable to new tasks. Our experiments on LLMs and CLIP models demonstrate that TempNet greatly improves the performance of existing solutions or models, e.g. Table 1. The code to reproduce the experimental results in this paper can be found at https://github.com/zhqiu/TempNet.

Title: SDFR: Synthetic Data for Face Recognition Competition

• Authors: Hatef Otroshi Shahreza, Christophe Ecabert, Anjith George, Alexander Unnervik, Sébastien Marcel, Nicolò Di Domenico, Guido Borghi, Davide Maltoni, Fadi Boutros, Julia Vogel, Naser Damer, Ángela Sánchez-Pérez, EnriqueMas-Candela, Jorge Calvo-Zaragoza, Bernardo Biesseck, Pedro Vidal, Roger Granada, David Menotti, Ivan DeAndres-Tame, Simone Maurizio La Cava, Sara Concas, Pietro Melzi, Ruben Tolosana, Ruben Vera-Rodriguez, Gianpaolo Perelli, Giulia Orrù, Gian Luca Marcialis, Julian Fierrez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04580
• Pdf URL: https://arxiv.org/pdf/2404.04580
• Copy Paste: [[2404.04580]] SDFR: Synthetic Data for Face Recognition Competition(https://arxiv.org/abs/2404.04580)
• Keywords: generative
• Abstract: Large-scale face recognition datasets are collected by crawling the Internet and without individuals' consent, raising legal, ethical, and privacy concerns. With the recent advances in generative models, recently several works proposed generating synthetic face recognition datasets to mitigate concerns in web-crawled face recognition datasets. This paper presents the summary of the Synthetic Data for Face Recognition (SDFR) Competition held in conjunction with the 18th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2024) and established to investigate the use of synthetic data for training face recognition models. The SDFR competition was split into two tasks, allowing participants to train face recognition systems using new synthetic datasets and/or existing ones. In the first task, the face recognition backbone was fixed and the dataset size was limited, while the second task provided almost complete freedom on the model backbone, the dataset, and the training pipeline. The submitted models were trained on existing and also new synthetic datasets and used clever methods to improve training with synthetic data. The submissions were evaluated and ranked on a diverse set of seven benchmarking datasets. The paper gives an overview of the submitted face recognition models and reports achieved performance compared to baseline models trained on real and synthetic datasets. Furthermore, the evaluation of submissions is extended to bias assessment across different demography groups. Lastly, an outlook on the current state of the research in training face recognition models using synthetic data is presented, and existing problems as well as potential future directions are also discussed.

Title: D$^3$: Scaling Up Deepfake Detection by Learning from Discrepancy

• Authors: Yongqi Yang, Zhihao Qian, Ye Zhu, Yu Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04584
• Pdf URL: https://arxiv.org/pdf/2404.04584
• Copy Paste: [[2404.04584]] D$^3$: Scaling Up Deepfake Detection by Learning from Discrepancy(https://arxiv.org/abs/2404.04584)
• Keywords: generative
• Abstract: The boom of Generative AI brings opportunities entangled with risks and concerns. In this work, we seek a step toward a universal deepfake detection system with better generalization and robustness, to accommodate the responsible deployment of diverse image generative models. We do so by first scaling up the existing detection task setup from the one-generator to multiple-generators in training, during which we disclose two challenges presented in prior methodological designs. Specifically, we reveal that the current methods tailored for training on one specific generator either struggle to learn comprehensive artifacts from multiple generators or tend to sacrifice their ability to identify fake images from seen generators (i.e., In-Domain performance) to exchange the generalization for unseen generators (i.e., Out-Of-Domain performance). To tackle the above challenges, we propose our Discrepancy Deepfake Detector (D$^3$) framework, whose core idea is to learn the universal artifacts from multiple generators by introducing a parallel network branch that takes a distorted image as extra discrepancy signal to supplement its original counterpart. Extensive scaled-up experiments on the merged UFD and GenImage datasets with six detection models demonstrate the effectiveness of our framework, achieving a 5.3% accuracy improvement in the OOD testing compared to the current SOTA methods while maintaining the ID performance.

Title: DifFUSER: Diffusion Model for Robust Multi-Sensor Fusion in 3D Object Detection and BEV Segmentation

• Authors: Duy-Tho Le, Hengcan Shi, Jianfei Cai, Hamid Rezatofighi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04629
• Pdf URL: https://arxiv.org/pdf/2404.04629
• Copy Paste: [[2404.04629]] DifFUSER: Diffusion Model for Robust Multi-Sensor Fusion in 3D Object Detection and BEV Segmentation(https://arxiv.org/abs/2404.04629)
• Keywords: diffusion, generative
• Abstract: Diffusion models have recently gained prominence as powerful deep generative models, demonstrating unmatched performance across various domains. However, their potential in multi-sensor fusion remains largely unexplored. In this work, we introduce DifFUSER, a novel approach that leverages diffusion models for multi-modal fusion in 3D object detection and BEV map segmentation. Benefiting from the inherent denoising property of diffusion, DifFUSER is able to refine or even synthesize sensor features in case of sensor malfunction, thereby improving the quality of the fused output. In terms of architecture, our DifFUSER blocks are chained together in a hierarchical BiFPN fashion, termed cMini-BiFPN, offering an alternative architecture for latent diffusion. We further introduce a Gated Self-conditioned Modulated (GSM) latent diffusion module together with a Progressive Sensor Dropout Training (PSDT) paradigm, designed to add stronger conditioning to the diffusion process and robustness to sensor failures. Our extensive evaluations on the Nuscenes dataset reveal that DifFUSER not only achieves state-of-the-art performance with a 69.1% mIOU in BEV map segmentation tasks but also competes effectively with leading transformer-based fusion techniques in 3D object detection.

Title: InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization

• Authors: Xiefan Guo, Jinlin Liu, Miaomiao Cui, Jiankai Li, Hongyu Yang, Di Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04650
• Pdf URL: https://arxiv.org/pdf/2404.04650
• Copy Paste: [[2404.04650]] InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization(https://arxiv.org/abs/2404.04650)
• Keywords: diffusion
• Abstract: Recent strides in the development of diffusion models, exemplified by advancements such as Stable Diffusion, have underscored their remarkable prowess in generating visually compelling images. However, the imperative of achieving a seamless alignment between the generated image and the provided prompt persists as a formidable challenge. This paper traces the root of these difficulties to invalid initial noise, and proposes a solution in the form of Initial Noise Optimization (InitNO), a paradigm that refines this noise. Considering text prompts, not all random noises are effective in synthesizing semantically-faithful images. We design the cross-attention response score and the self-attention conflict score to evaluate the initial noise, bifurcating the initial latent space into valid and invalid sectors. A strategically crafted noise optimization pipeline is developed to guide the initial noise towards valid regions. Our method, validated through rigorous experimentation, shows a commendable proficiency in generating images in strict accordance with text prompts. Our code is available at https://github.com/xiefan-guo/initno.

Title: Salient Sparse Visual Odometry With Pose-Only Supervision

• Authors: Siyu Chen, Kangcheng Liu, Chen Wang, Shenghai Yuan, Jianfei Yang, Lihua Xie
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.04677
• Pdf URL: https://arxiv.org/pdf/2404.04677
• Copy Paste: [[2404.04677]] Salient Sparse Visual Odometry With Pose-Only Supervision(https://arxiv.org/abs/2404.04677)
• Keywords: self-supervised
• Abstract: Visual Odometry (VO) is vital for the navigation of autonomous systems, providing accurate position and orientation estimates at reasonable costs. While traditional VO methods excel in some conditions, they struggle with challenges like variable lighting and motion blur. Deep learning-based VO, though more adaptable, can face generalization problems in new environments. Addressing these drawbacks, this paper presents a novel hybrid visual odometry (VO) framework that leverages pose-only supervision, offering a balanced solution between robustness and the need for extensive labeling. We propose two cost-effective and innovative designs: a self-supervised homographic pre-training for enhancing optical flow learning from pose-only labels and a random patch-based salient point detection strategy for more accurate optical flow patch extraction. These designs eliminate the need for dense optical flow labels for training and significantly improve the generalization capability of the system in diverse and challenging environments. Our pose-only supervised method achieves competitive performance on standard datasets and greater robustness and generalization ability in extreme and unseen scenarios, even compared to dense optical flow-supervised state-of-the-art methods.

Title: GenEARL: A Training-Free Generative Framework for Multimodal Event Argument Role Labeling

• Authors: Hritik Bansal, Po-Nien Kung, P. Jeffrey Brantingham, Kai-Wei Chang, Nanyun Peng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04763
• Pdf URL: https://arxiv.org/pdf/2404.04763
• Copy Paste: [[2404.04763]] GenEARL: A Training-Free Generative Framework for Multimodal Event Argument Role Labeling(https://arxiv.org/abs/2404.04763)
• Keywords: generative
• Abstract: Multimodal event argument role labeling (EARL), a task that assigns a role for each event participant (object) in an image is a complex challenge. It requires reasoning over the entire image, the depicted event, and the interactions between various objects participating in the event. Existing models heavily rely on high-quality event-annotated training data to understand the event semantics and structures, and they fail to generalize to new event types and domains. In this paper, we propose GenEARL, a training-free generative framework that harness the power of the modern generative models to understand event task descriptions given image contexts to perform the EARL task. Specifically, GenEARL comprises two stages of generative prompting with a frozen vision-language model (VLM) and a frozen large language model (LLM). First, a generative VLM learns the semantics of the event argument roles and generates event-centric object descriptions based on the image. Subsequently, a LLM is prompted with the generated object descriptions with a predefined template for EARL (i.e., assign an object with an event argument role). We show that GenEARL outperforms the contrastive pretraining (CLIP) baseline by 9.4% and 14.2% accuracy for zero-shot EARL on the M2E2 and SwiG datasets, respectively. In addition, we outperform CLIP-Event by 22% precision on M2E2 dataset. The framework also allows flexible adaptation and generalization to unseen domains.

Title: Rethinking Diffusion Model for Multi-Contrast MRI Super-Resolution

• Authors: Guangyuan Li, Chen Rao, Juncheng Mo, Zhanjie Zhang, Wei Xing, Lei Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04785
• Pdf URL: https://arxiv.org/pdf/2404.04785
• Copy Paste: [[2404.04785]] Rethinking Diffusion Model for Multi-Contrast MRI Super-Resolution(https://arxiv.org/abs/2404.04785)
• Keywords: diffusion
• Abstract: Recently, diffusion models (DM) have been applied in magnetic resonance imaging (MRI) super-resolution (SR) reconstruction, exhibiting impressive performance, especially with regard to detailed reconstruction. However, the current DM-based SR reconstruction methods still face the following issues: (1) They require a large number of iterations to reconstruct the final image, which is inefficient and consumes a significant amount of computational resources. (2) The results reconstructed by these methods are often misaligned with the real high-resolution images, leading to remarkable distortion in the reconstructed MR images. To address the aforementioned issues, we propose an efficient diffusion model for multi-contrast MRI SR, named as DiffMSR. Specifically, we apply DM in a highly compact low-dimensional latent space to generate prior knowledge with high-frequency detail information. The highly compact latent space ensures that DM requires only a few simple iterations to produce accurate prior knowledge. In addition, we design the Prior-Guide Large Window Transformer (PLWformer) as the decoder for DM, which can extend the receptive field while fully utilizing the prior knowledge generated by DM to ensure that the reconstructed MR image remains undistorted. Extensive experiments on public and clinical datasets demonstrate that our DiffMSR outperforms state-of-the-art methods.

Title: Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Driving

• Authors: Jinlong Li, Baolu Li, Zhengzhong Tu, Xinyu Liu, Qing Guo, Felix Juefei-Xu, Runsheng Xu, Hongkai Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04804
• Pdf URL: https://arxiv.org/pdf/2404.04804
• Copy Paste: [[2404.04804]] Light the Night: A Multi-Condition Diffusion Framework for Unpaired Low-Light Enhancement in Autonomous Driving(https://arxiv.org/abs/2404.04804)
• Keywords: diffusion
• Abstract: Vision-centric perception systems for autonomous driving have gained considerable attention recently due to their cost-effectiveness and scalability, especially compared to LiDAR-based systems. However, these systems often struggle in low-light conditions, potentially compromising their performance and safety. To address this, our paper introduces LightDiff, a domain-tailored framework designed to enhance the low-light image quality for autonomous driving applications. Specifically, we employ a multi-condition controlled diffusion model. LightDiff works without any human-collected paired data, leveraging a dynamic data degradation process instead. It incorporates a novel multi-condition adapter that adaptively controls the input weights from different modalities, including depth maps, RGB images, and text captions, to effectively illuminate dark scenes while maintaining context consistency. Furthermore, to align the enhanced images with the detection model's knowledge, LightDiff employs perception-specific scores as rewards to guide the diffusion training process through reinforcement learning. Extensive experiments on the nuScenes datasets demonstrate that LightDiff can significantly improve the performance of several state-of-the-art 3D detectors in night-time conditions while achieving high visual quality scores, highlighting its potential to safeguard autonomous driving.

Title: Mixup Domain Adaptations for Dynamic Remaining Useful Life Predictions

• Authors: Muhammad Tanzil Furqon, Mahardhika Pratama, Lin Liu, Habibullah, Kutluyil Dogancay
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.04824
• Pdf URL: https://arxiv.org/pdf/2404.04824
• Copy Paste: [[2404.04824]] Mixup Domain Adaptations for Dynamic Remaining Useful Life Predictions(https://arxiv.org/abs/2404.04824)
• Keywords: self-supervised
• Abstract: Remaining Useful Life (RUL) predictions play vital role for asset planning and maintenance leading to many benefits to industries such as reduced downtime, low maintenance costs, etc. Although various efforts have been devoted to study this topic, most existing works are restricted for i.i.d conditions assuming the same condition of the training phase and the deployment phase. This paper proposes a solution to this problem where a mix-up domain adaptation (MDAN) is put forward. MDAN encompasses a three-staged mechanism where the mix-up strategy is not only performed to regularize the source and target domains but also applied to establish an intermediate mix-up domain where the source and target domains are aligned. The self-supervised learning strategy is implemented to prevent the supervision collapse problem. Rigorous evaluations have been performed where MDAN is compared to recently published works for dynamic RUL predictions. MDAN outperforms its counterparts with substantial margins in 12 out of 12 cases. In addition, MDAN is evaluated with the bearing machine dataset where it beats prior art with significant gaps in 8 of 12 cases. Source codes of MDAN are made publicly available in \url{https://github.com/furqon3009/MDAN}.

Title: Strictly-ID-Preserved and Controllable Accessory Advertising Image Generation

• Authors: Youze Xue, Binghui Chen, Yifeng Geng, Xuansong Xie, Jiansheng Chen, Hongbing Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04828
• Pdf URL: https://arxiv.org/pdf/2404.04828
• Copy Paste: [[2404.04828]] Strictly-ID-Preserved and Controllable Accessory Advertising Image Generation(https://arxiv.org/abs/2404.04828)
• Keywords: generative
• Abstract: Customized generative text-to-image models have the ability to produce images that closely resemble a given subject. However, in the context of generating advertising images for e-commerce scenarios, it is crucial that the generated subject's identity aligns perfectly with the product being advertised. In order to address the need for strictly-ID preserved advertising image generation, we have developed a Control-Net based customized image generation pipeline and have taken earring model advertising as an example. Our approach facilitates a seamless interaction between the earrings and the model's face, while ensuring that the identity of the earrings remains intact. Furthermore, to achieve a diverse and controllable display, we have proposed a multi-branch cross-attention architecture, which allows for control over the scale, pose, and appearance of the model, going beyond the limitations of text prompts. Our method manages to achieve fine-grained control of the generated model's face, resulting in controllable and captivating advertising effects.

Title: ShoeModel: Learning to Wear on the User-specified Shoes via Diffusion Model

• Authors: Binghui Chen, Wenyu Li, Yifeng Geng, Xuansong Xie, Wangmeng Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04833
• Pdf URL: https://arxiv.org/pdf/2404.04833
• Copy Paste: [[2404.04833]] ShoeModel: Learning to Wear on the User-specified Shoes via Diffusion Model(https://arxiv.org/abs/2404.04833)
• Keywords: diffusion
• Abstract: With the development of the large-scale diffusion model, Artificial Intelligence Generated Content (AIGC) techniques are popular recently. However, how to truly make it serve our daily lives remains an open question. To this end, in this paper, we focus on employing AIGC techniques in one filed of E-commerce marketing, i.e., generating hyper-realistic advertising images for displaying user-specified shoes by human. Specifically, we propose a shoe-wearing system, called Shoe-Model, to generate plausible images of human legs interacting with the given shoes. It consists of three modules: (1) shoe wearable-area detection module (WD), (2) leg-pose synthesis module (LpS) and the final (3) shoe-wearing image generation module (SW). Them three are performed in ordered stages. Compared to baselines, our ShoeModel is shown to generalize better to different type of shoes and has ability of keeping the ID-consistency of the given shoes, as well as automatically producing reasonable interactions with human. Extensive experiments show the effectiveness of our proposed shoe-wearing system. Figure 1 shows the input and output examples of our ShoeModel.

Title: SLPL SHROOM at SemEval\-2024 Task 06: A comprehensive study on models ability to detect hallucination

• Authors: Pouya Fallah, Soroush Gooran, Mohammad Jafarinasab, Pouya Sadeghi, Reza Farnia, Amirreza Tarabkhah, Zainab Sadat Taghavi, Hossein Sameti
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04845
• Pdf URL: https://arxiv.org/pdf/2404.04845
• Copy Paste: [[2404.04845]] SLPL SHROOM at SemEval\-2024 Task 06: A comprehensive study on models ability to detect hallucination(https://arxiv.org/abs/2404.04845)
• Keywords: generative
• Abstract: Language models, particularly generative models, are susceptible to hallucinations, generating outputs that contradict factual knowledge or the source text. This study explores methods for detecting hallucinations in three SemEval-2024 Task 6 tasks: Machine Translation, Definition Modeling, and Paraphrase Generation. We evaluate two methods: semantic similarity between the generated text and factual references, and an ensemble of language models that judge each other's outputs. Our results show that semantic similarity achieves moderate accuracy and correlation scores in trial data, while the ensemble method offers insights into the complexities of hallucination detection but falls short of expectations. This work highlights the challenges of hallucination detection and underscores the need for further research in this critical area.

Title: Contextual Chart Generation for Cyber Deception

• Authors: David D. Nguyen, David Liebowitz, Surya Nepal, Salil S. Kanhere, Sharif Abuadbba
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04854
• Pdf URL: https://arxiv.org/pdf/2404.04854
• Copy Paste: [[2404.04854]] Contextual Chart Generation for Cyber Deception(https://arxiv.org/abs/2404.04854)
• Keywords: generative
• Abstract: Honeyfiles are security assets designed to attract and detect intruders on compromised systems. Honeyfiles are a type of honeypot that mimic real, sensitive documents, creating the illusion of the presence of valuable data. Interaction with a honeyfile reveals the presence of an intruder, and can provide insights into their goals and intentions. Their practical use, however, is limited by the time, cost and effort associated with manually creating realistic content. The introduction of large language models has made high-quality text generation accessible, but honeyfiles contain a variety of content including charts, tables and images. This content needs to be plausible and realistic, as well as semantically consistent both within honeyfiles and with the real documents they mimic, to successfully deceive an intruder. In this paper, we focus on an important component of the honeyfile content generation problem: document charts. Charts are ubiquitous in corporate documents and are commonly used to communicate quantitative and scientific data. Existing image generation models, such as DALL-E, are rather prone to generating charts with incomprehensible text and unconvincing data. We take a multi-modal approach to this problem by combining two purpose-built generative models: a multitask Transformer and a specialized multi-head autoencoder. The Transformer generates realistic captions and plot text, while the autoencoder generates the underlying tabular data for the plot. To advance the field of automated honeyplot generation, we also release a new document-chart dataset and propose a novel metric Keyword Semantic Matching (KSM). This metric measures the semantic consistency between keywords of a corpus and a smaller bag of words. Extensive experiments demonstrate excellent performance against multiple large language models, including ChatGPT and GPT4.

Title: ByteEdit: Boost, Comply and Accelerate Generative Image Editing

• Authors: Yuxi Ren, Jie Wu, Yanzuo Lu, Huafeng Kuang, Xin Xia, Xionghui Wang, Qianqian Wang, Yixing Zhu, Pan Xie, Shiyin Wang, Xuefeng Xiao, Yitong Wang, Min Zheng, Lean Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04860
• Pdf URL: https://arxiv.org/pdf/2404.04860
• Copy Paste: [[2404.04860]] ByteEdit: Boost, Comply and Accelerate Generative Image Editing(https://arxiv.org/abs/2404.04860)
• Keywords: diffusion, generative
• Abstract: Recent advancements in diffusion-based generative image editing have sparked a profound revolution, reshaping the landscape of image outpainting and inpainting tasks. Despite these strides, the field grapples with inherent challenges, including: i) inferior quality; ii) poor consistency; iii) insufficient instrcution adherence; iv) suboptimal generation efficiency. To address these obstacles, we present ByteEdit, an innovative feedback learning framework meticulously designed to Boost, Comply, and Accelerate Generative Image Editing tasks. ByteEdit seamlessly integrates image reward models dedicated to enhancing aesthetics and image-text alignment, while also introducing a dense, pixel-level reward model tailored to foster coherence in the output. Furthermore, we propose a pioneering adversarial and progressive feedback learning strategy to expedite the model's inference speed. Through extensive large-scale user evaluations, we demonstrate that ByteEdit surpasses leading generative image editing products, including Adobe, Canva, and MeiTu, in both generation quality and consistency. ByteEdit-Outpainting exhibits a remarkable enhancement of 388% and 135% in quality and consistency, respectively, when compared to the baseline model. Experiments also verfied that our acceleration models maintains excellent performance results in terms of quality and consistency.

Title: Graph Neural Networks for Binary Programming

• Authors: Moshe Eliasof, Eldad Haber
• Subjects: cs.LG, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2404.04874
• Pdf URL: https://arxiv.org/pdf/2404.04874
• Copy Paste: [[2404.04874]] Graph Neural Networks for Binary Programming(https://arxiv.org/abs/2404.04874)
• Keywords: self-supervised
• Abstract: This paper investigates a link between Graph Neural Networks (GNNs) and Binary Programming (BP) problems, laying the groundwork for GNNs to approximate solutions for these computationally challenging problems. By analyzing the sensitivity of BP problems, we are able to frame the solution of BP problems as a heterophilic node classification task. We then propose Binary-Programming GNN (BPGNN), an architecture that integrates graph representation learning techniques with BP-aware features to approximate BP solutions efficiently. Additionally, we introduce a self-supervised data generation mechanism, to enable efficient and tractable training data acquisition even for large-scale BP problems. Experimental evaluations of BPGNN across diverse BP problem sizes showcase its superior performance compared to exhaustive search and heuristic approaches. Finally, we discuss open challenges in the under-explored field of BP problems with GNNs.

Title: Mixture of Low-rank Experts for Transferable AI-Generated Image Detection

• Authors: Zihan Liu, Hanyi Wang, Yaoyu Kang, Shilin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04883
• Pdf URL: https://arxiv.org/pdf/2404.04883
• Copy Paste: [[2404.04883]] Mixture of Low-rank Experts for Transferable AI-Generated Image Detection(https://arxiv.org/abs/2404.04883)
• Keywords: diffusion, generative
• Abstract: Generative models have shown a giant leap in synthesizing photo-realistic images with minimal expertise, sparking concerns about the authenticity of online information. This study aims to develop a universal AI-generated image detector capable of identifying images from diverse sources. Existing methods struggle to generalize across unseen generative models when provided with limited sample sources. Inspired by the zero-shot transferability of pre-trained vision-language models, we seek to harness the nontrivial visual-world knowledge and descriptive proficiency of CLIP-ViT to generalize over unknown domains. This paper presents a novel parameter-efficient fine-tuning approach, mixture of low-rank experts, to fully exploit CLIP-ViT's potential while preserving knowledge and expanding capacity for transferable detection. We adapt only the MLP layers of deeper ViT blocks via an integration of shared and separate LoRAs within an MoE-based structure. Extensive experiments on public benchmarks show that our method achieves superiority over state-of-the-art approaches in cross-generator generalization and robustness to perturbations. Remarkably, our best-performing ViT-L/14 variant requires training only 0.08% of its parameters to surpass the leading baseline by +3.64% mAP and +12.72% avg.Acc across unseen diffusion and autoregressive models. This even outperforms the baseline with just 0.28% of the training data. Our code and pre-trained models will be available at https://github.com/zhliuworks/CLIPMoLE.

Title: TimeGPT in Load Forecasting: A Large Time Series Model Perspective

• Authors: Wenlong Liao, Fernando Porte-Agel, Jiannong Fang, Christian Rehtanz, Shouxiang Wang, Dechang Yang, Zhe Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04885
• Pdf URL: https://arxiv.org/pdf/2404.04885
• Copy Paste: [[2404.04885]] TimeGPT in Load Forecasting: A Large Time Series Model Perspective(https://arxiv.org/abs/2404.04885)
• Keywords: generative
• Abstract: Machine learning models have made significant progress in load forecasting, but their forecast accuracy is limited in cases where historical load data is scarce. Inspired by the outstanding performance of large language models (LLMs) in computer vision and natural language processing, this paper aims to discuss the potential of large time series models in load forecasting with scarce historical data. Specifically, the large time series model is constructed as a time series generative pre-trained transformer (TimeGPT), which is trained on massive and diverse time series datasets consisting of 100 billion data points (e.g., finance, transportation, banking, web traffic, weather, energy, healthcare, etc.). Then, the scarce historical load data is used to fine-tune the TimeGPT, which helps it to adapt to the data distribution and characteristics associated with load forecasting. Simulation results show that TimeGPT outperforms the benchmarks (e.g., popular machine learning models and statistical models) for load forecasting on several real datasets with scarce training samples, particularly for short look-ahead times. However, it cannot be guaranteed that TimeGPT is always superior to benchmarks for load forecasting with scarce data, since the performance of TimeGPT may be affected by the distribution differences between the load data and the training data. In practical applications, we can divide the historical data into a training set and a validation set, and then use the validation set loss to decide whether TimeGPT is the best choice for a specific dataset.

Title: PagPassGPT: Pattern Guided Password Guessing via Generative Pretrained Transformer

• Authors: Xingyu Su, Xiaojie Zhu, Yang Li, Yong Li, Chi Chen, Paulo Esteves-Veríssimo
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04886
• Pdf URL: https://arxiv.org/pdf/2404.04886
• Copy Paste: [[2404.04886]] PagPassGPT: Pattern Guided Password Guessing via Generative Pretrained Transformer(https://arxiv.org/abs/2404.04886)
• Keywords: generative
• Abstract: Amidst the surge in deep learning-based password guessing models, challenges of generating high-quality passwords and reducing duplicate passwords persist. To address these challenges, we present PagPassGPT, a password guessing model constructed on Generative Pretrained Transformer (GPT). It can perform pattern guided guessing by incorporating pattern structure information as background knowledge, resulting in a significant increase in the hit rate. Furthermore, we propose D&C-GEN to reduce the repeat rate of generated passwords, which adopts the concept of a divide-and-conquer approach. The primary task of guessing passwords is recursively divided into non-overlapping subtasks. Each subtask inherits the knowledge from the parent task and predicts succeeding tokens. In comparison to the state-of-the-art model, our proposed scheme exhibits the capability to correctly guess 12% more passwords while producing 25% fewer duplicates.

Title: A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals

• Authors: Jiangnan Tang, Jingya Wang, Kaiyang Ji, Lan Xu, Jingyi Yu, Ye Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04890
• Pdf URL: https://arxiv.org/pdf/2404.04890
• Copy Paste: [[2404.04890]] A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals(https://arxiv.org/abs/2404.04890)
• Keywords: diffusion
• Abstract: Estimating full-body human motion via sparse tracking signals from head-mounted displays and hand controllers in 3D scenes is crucial to applications in AR/VR. One of the biggest challenges to this task is the one-to-many mapping from sparse observations to dense full-body motions, which endowed inherent ambiguities. To help resolve this ambiguous problem, we introduce a new framework to combine rich contextual information provided by scenes to benefit full-body motion tracking from sparse observations. To estimate plausible human motions given sparse tracking signals and 3D scenes, we develop $\text{S}^2$Fusion, a unified framework fusing \underline{S}cene and sparse \underline{S}ignals with a conditional dif\underline{Fusion} model. $\text{S}^2$Fusion first extracts the spatial-temporal relations residing in the sparse signals via a periodic autoencoder, and then produces time-alignment feature embedding as additional inputs. Subsequently, by drawing initial noisy motion from a pre-trained prior, $\text{S}^2$Fusion utilizes conditional diffusion to fuse scene geometry and sparse tracking signals to generate full-body scene-aware motions. The sampling procedure of $\text{S}^2$Fusion is further guided by a specially designed scene-penetration loss and phase-matching loss, which effectively regularizes the motion of the lower body even in the absence of any tracking signals, making the generated motion much more plausible and coherent. Extensive experimental results have demonstrated that our $\text{S}^2$Fusion outperforms the state-of-the-art in terms of estimation quality and smoothness.

Title: Regularized Conditional Diffusion Model for Multi-Task Preference Alignment

• Authors: Xudong Yu, Chenjia Bai, Haoran He, Changhong Wang, Xuelong Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04920
• Pdf URL: https://arxiv.org/pdf/2404.04920
• Copy Paste: [[2404.04920]] Regularized Conditional Diffusion Model for Multi-Task Preference Alignment(https://arxiv.org/abs/2404.04920)
• Keywords: diffusion
• Abstract: Sequential decision-making is desired to align with human intents and exhibit versatility across various tasks. Previous methods formulate it as a conditional generation process, utilizing return-conditioned diffusion models to directly model trajectory distributions. Nevertheless, the return-conditioned paradigm relies on pre-defined reward functions, facing challenges when applied in multi-task settings characterized by varying reward functions (versatility) and showing limited controllability concerning human preferences (alignment). In this work, we adopt multi-task preferences as a unified condition for both single- and multi-task decision-making, and propose preference representations aligned with preference labels. The learned representations are used to guide the conditional generation process of diffusion models, and we introduce an auxiliary objective to maximize the mutual information between representations and corresponding generated trajectories, improving alignment between trajectories and preferences. Extensive experiments in D4RL and Meta-World demonstrate that our method presents favorable performance in single- and multi-task scenarios, and exhibits superior alignment with preferences.

Title: Anomaly Detection in Electrocardiograms: Advancing Clinical Diagnosis Through Self-Supervised Learning

• Authors: Aofan Jiang, Chaoqin Huang, Qing Cao, Yuchen Xu, Zi Zeng, Kang Chen, Ya Zhang, Yanfeng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04935
• Pdf URL: https://arxiv.org/pdf/2404.04935
• Copy Paste: [[2404.04935]] Anomaly Detection in Electrocardiograms: Advancing Clinical Diagnosis Through Self-Supervised Learning(https://arxiv.org/abs/2404.04935)
• Keywords: self-supervised, anomaly
• Abstract: The electrocardiogram (ECG) is an essential tool for diagnosing heart disease, with computer-aided systems improving diagnostic accuracy and reducing healthcare costs. Despite advancements, existing systems often miss rare cardiac anomalies that could be precursors to serious, life-threatening issues or alterations in the cardiac macro/microstructure. We address this gap by focusing on self-supervised anomaly detection (AD), training exclusively on normal ECGs to recognize deviations indicating anomalies. We introduce a novel self-supervised learning framework for ECG AD, utilizing a vast dataset of normal ECGs to autonomously detect and localize cardiac anomalies. It proposes a novel masking and restoration technique alongside a multi-scale cross-attention module, enhancing the model's ability to integrate global and local signal features. The framework emphasizes accurate localization of anomalies within ECG signals, ensuring the method's clinical relevance and reliability. To reduce the impact of individual variability, the approach further incorporates crucial patient-specific information from ECG reports, such as age and gender, thus enabling accurate identification of a broad spectrum of cardiac anomalies, including rare ones. Utilizing an extensive dataset of 478,803 ECG graphic reports from real-world clinical practice, our method has demonstrated exceptional effectiveness in AD across all tested conditions, regardless of their frequency of occurrence, significantly outperforming existing models. It achieved superior performance metrics, including an AUROC of 91.2%, an F1 score of 83.7%, a sensitivity rate of 84.2%, a specificity of 83.0%, and a precision of 75.6% with a fixed recall rate of 90%. It has also demonstrated robust localization capabilities, with an AUROC of 76.5% and a Dice coefficient of 65.3% for anomaly localization.

Title: AnimateZoo: Zero-shot Video Generation of Cross-Species Animation via Subject Alignment

• Authors: Yuanfeng Xu, Yuhao Chen, Zhongzhan Huang, Zijian He, Guangrun Wang, Philip Torr, Liang Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.04946
• Pdf URL: https://arxiv.org/pdf/2404.04946
• Copy Paste: [[2404.04946]] AnimateZoo: Zero-shot Video Generation of Cross-Species Animation via Subject Alignment(https://arxiv.org/abs/2404.04946)
• Keywords: diffusion
• Abstract: Recent video editing advancements rely on accurate pose sequences to animate subjects. However, these efforts are not suitable for cross-species animation due to pose misalignment between species (for example, the poses of a cat differs greatly from that of a pig due to differences in body structure). In this paper, we present AnimateZoo, a zero-shot diffusion-based video generator to address this challenging cross-species animation issue, aiming to accurately produce animal animations while preserving the background. The key technique used in our AnimateZoo is subject alignment, which includes two steps. First, we improve appearance feature extraction by integrating a Laplacian detail booster and a prompt-tuning identity extractor. These components are specifically designed to capture essential appearance information, including identity and fine details. Second, we align shape features and address conflicts from differing subjects by introducing a scale-information remover. This ensures accurate cross-species animation. Moreover, we introduce two high-quality animal video datasets featuring a wide variety of species. Trained on these extensive datasets, our model is capable of generating videos characterized by accurate movements, consistent appearance, and high-fidelity frames, without the need for the pre-inference fine-tuning that prior arts required. Extensive experiments showcase the outstanding performance of our method in cross-species action following tasks, demonstrating exceptional shape adaptation capability. The project page is available at https://justinxu0.github.io/AnimateZoo/.

Title: Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models

• Authors: Zijin Yang, Kai Zeng, Kejiang Chen, Han Fang, Weiming Zhang, Nenghai Yu
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.04956
• Pdf URL: https://arxiv.org/pdf/2404.04956
• Copy Paste: [[2404.04956]] Gaussian Shading: Provable Performance-Lossless Image Watermarking for Diffusion Models(https://arxiv.org/abs/2404.04956)
• Keywords: diffusion
• Abstract: Ethical concerns surrounding copyright protection and inappropriate content generation pose challenges for the practical implementation of diffusion models. One effective solution involves watermarking the generated images. However, existing methods often compromise the model performance or require additional training, which is undesirable for operators and users. To address this issue, we propose Gaussian Shading, a diffusion model watermarking technique that is both performance-lossless and training-free, while serving the dual purpose of copyright protection and tracing of offending content. Our watermark embedding is free of model parameter modifications and thus is plug-and-play. We map the watermark to latent representations following a standard Gaussian distribution, which is indistinguishable from latent representations obtained from the non-watermarked diffusion model. Therefore we can achieve watermark embedding with lossless performance, for which we also provide theoretical proof. Furthermore, since the watermark is intricately linked with image semantics, it exhibits resilience to lossy processing and erasure attempts. The watermark can be extracted by Denoising Diffusion Implicit Models (DDIM) inversion and inverse sampling. We evaluate Gaussian Shading on multiple versions of Stable Diffusion, and the results demonstrate that Gaussian Shading not only is performance-lossless but also outperforms existing methods in terms of robustness.

Title: Temporal Generalization Estimation in Evolving Graphs

• Authors: Bin Lu, Tingyan Ma, Xiaoying Gan, Xinbing Wang, Yunqiang Zhu, Chenghu Zhou, Shiyu Liang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04969
• Pdf URL: https://arxiv.org/pdf/2404.04969
• Copy Paste: [[2404.04969]] Temporal Generalization Estimation in Evolving Graphs(https://arxiv.org/abs/2404.04969)
• Keywords: self-supervised
• Abstract: Graph Neural Networks (GNNs) are widely deployed in vast fields, but they often struggle to maintain accurate representations as graphs evolve. We theoretically establish a lower bound, proving that under mild conditions, representation distortion inevitably occurs over time. To estimate the temporal distortion without human annotation after deployment, one naive approach is to pre-train a recurrent model (e.g., RNN) before deployment and use this model afterwards, but the estimation is far from satisfactory. In this paper, we analyze the representation distortion from an information theory perspective, and attribute it primarily to inaccurate feature extraction during evolution. Consequently, we introduce Smart, a straightforward and effective baseline enhanced by an adaptive feature extractor through self-supervised graph reconstruction. In synthetic random graphs, we further refine the former lower bound to show the inevitable distortion over time and empirically observe that Smart achieves good estimation performance. Moreover, we observe that Smart consistently shows outstanding generalization estimation on four real-world evolving graphs. The ablation studies underscore the necessity of graph reconstruction. For example, on OGB-arXiv dataset, the estimation metric MAPE deteriorates from 2.19% to 8.00% without reconstruction.

Title: Dynamic Distinction Learning: Adaptive Pseudo Anomalies for Video Anomaly Detection

• Authors: Demetris Lappas, Vasileios Argyriou, Dimitrios Makris
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04986
• Pdf URL: https://arxiv.org/pdf/2404.04986
• Copy Paste: [[2404.04986]] Dynamic Distinction Learning: Adaptive Pseudo Anomalies for Video Anomaly Detection(https://arxiv.org/abs/2404.04986)
• Keywords: anomaly
• Abstract: We introduce Dynamic Distinction Learning (DDL) for Video Anomaly Detection, a novel video anomaly detection methodology that combines pseudo-anomalies, dynamic anomaly weighting, and a distinction loss function to improve detection accuracy. By training on pseudo-anomalies, our approach adapts to the variability of normal and anomalous behaviors without fixed anomaly thresholds. Our model showcases superior performance on the Ped2, Avenue and ShanghaiTech datasets, where individual models are tailored for each scene. These achievements highlight DDL's effectiveness in advancing anomaly detection, offering a scalable and adaptable solution for video surveillance challenges.

Title: DinoBloom: A Foundation Model for Generalizable Cell Embeddings in Hematology

• Authors: Valentin Koch, Sophia J. Wagner, Salome Kazeminia, Ece Sancar, Matthias Hehr, Julia Schnabel, Tingying Peng, Carsten Marr
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05022
• Pdf URL: https://arxiv.org/pdf/2404.05022
• Copy Paste: [[2404.05022]] DinoBloom: A Foundation Model for Generalizable Cell Embeddings in Hematology(https://arxiv.org/abs/2404.05022)
• Keywords: foundation model
• Abstract: In hematology, computational models offer significant potential to improve diagnostic accuracy, streamline workflows, and reduce the tedious work of analyzing single cells in peripheral blood or bone marrow smears. However, clinical adoption of computational models has been hampered by the lack of generalization due to large batch effects, small dataset sizes, and poor performance in transfer learning from natural images. To address these challenges, we introduce DinoBloom, the first foundation model for single cell images in hematology, utilizing a tailored DINOv2 pipeline. Our model is built upon an extensive collection of 13 diverse, publicly available datasets of peripheral blood and bone marrow smears, the most substantial open-source cohort in hematology so far, comprising over 380,000 white blood cell images. To assess its generalization capability, we evaluate it on an external dataset with a challenging domain shift. We show that our model outperforms existing medical and non-medical vision models in (i) linear probing and k-nearest neighbor evaluations for cell-type classification on blood and bone marrow smears and (ii) weakly supervised multiple instance learning for acute myeloid leukemia subtyping by a large margin. A family of four DinoBloom models (small, base, large, and giant) can be adapted for a wide range of downstream applications, be a strong baseline for classification problems, and facilitate the assessment of batch effects in new datasets. All models are available at github.com/marrlab/DinoBloom.

Title: TimeCSL: Unsupervised Contrastive Learning of General Shapelets for Explorable Time Series Analysis

• Authors: Zhiyu Liang, Chen Liang, Zheng Liang, Hongzhi Wang, Bo Zheng
• Subjects: cs.LG, cs.DB
• Abstract URL: https://arxiv.org/abs/2404.05057
• Pdf URL: https://arxiv.org/pdf/2404.05057
• Copy Paste: [[2404.05057]] TimeCSL: Unsupervised Contrastive Learning of General Shapelets for Explorable Time Series Analysis(https://arxiv.org/abs/2404.05057)
• Keywords: self-supervised, anomaly
• Abstract: Unsupervised (a.k.a. Self-supervised) representation learning (URL) has emerged as a new paradigm for time series analysis, because it has the ability to learn generalizable time series representation beneficial for many downstream tasks without using labels that are usually difficult to obtain. Considering that existing approaches have limitations in the design of the representation encoder and the learning objective, we have proposed Contrastive Shapelet Learning (CSL), the first URL method that learns the general-purpose shapelet-based representation through unsupervised contrastive learning, and shown its superior performance in several analysis tasks, such as time series classification, clustering, and anomaly detection. In this paper, we develop TimeCSL, an end-to-end system that makes full use of the general and interpretable shapelets learned by CSL to achieve explorable time series analysis in a unified pipeline. We introduce the system components and demonstrate how users interact with TimeCSL to solve different analysis tasks in the unified pipeline, and gain insight into their time series by exploring the learned shapelets and representation.

Title: Automated Prediction of Breast Cancer Response to Neoadjuvant Chemotherapy from DWI Data

• Authors: Shir Nitzan, Maya Gilad, Moti Freiman
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05061
• Pdf URL: https://arxiv.org/pdf/2404.05061
• Copy Paste: [[2404.05061]] Automated Prediction of Breast Cancer Response to Neoadjuvant Chemotherapy from DWI Data(https://arxiv.org/abs/2404.05061)
• Keywords: diffusion
• Abstract: Effective surgical planning for breast cancer hinges on accurately predicting pathological complete response (pCR) to neoadjuvant chemotherapy (NAC). Diffusion-weighted MRI (DWI) and machine learning offer a non-invasive approach for early pCR assessment. However, most machine-learning models require manual tumor segmentation, a cumbersome and error-prone task. We propose a deep learning model employing "Size-Adaptive Lesion Weighting" for automatic DWI tumor segmentation to enhance pCR prediction accuracy. Despite histopathological changes during NAC complicating DWI image segmentation, our model demonstrates robust performance. Utilizing the BMMR2 challenge dataset, it matches human experts in pCR prediction pre-NAC with an area under the curve (AUC) of 0.76 vs. 0.796, and surpasses standard automated methods mid-NAC, with an AUC of 0.729 vs. 0.654 and 0.576. Our approach represents a significant advancement in automating breast cancer treatment planning, enabling more reliable pCR predictions without manual segmentation.

Title: HaVTR: Improving Video-Text Retrieval Through Augmentation Using Large Foundation Models

• Authors: Yimu Wang, Shuai Yuan, Xiangru Jian, Wei Pang, Mushi Wang, Ning Yu
• Subjects: cs.CV, cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05083
• Pdf URL: https://arxiv.org/pdf/2404.05083
• Copy Paste: [[2404.05083]] HaVTR: Improving Video-Text Retrieval Through Augmentation Using Large Foundation Models(https://arxiv.org/abs/2404.05083)
• Keywords: foundation model, generative
• Abstract: While recent progress in video-text retrieval has been driven by the exploration of powerful model architectures and training strategies, the representation learning ability of video-text retrieval models is still limited due to low-quality and scarce training data annotations. To address this issue, we present a novel video-text learning paradigm, HaVTR, which augments video and text data to learn more generalized features. Specifically, we first adopt a simple augmentation method, which generates self-similar data by randomly duplicating or dropping subwords and frames. In addition, inspired by the recent advancement in visual and language generative models, we propose a more powerful augmentation method through textual paraphrasing and video stylization using large language models (LLMs) and visual generative models (VGMs). Further, to bring richer information into video and text, we propose a hallucination-based augmentation method, where we use LLMs and VGMs to generate and add new relevant information to the original data. Benefiting from the enriched data, extensive experiments on several video-text retrieval benchmarks demonstrate the superiority of HaVTR over existing methods.

Title: How much reliable is ChatGPT's prediction on Information Extraction under Input Perturbations?

• Authors: Ishani Mondal, Abhilasha Sancheti
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05088
• Pdf URL: https://arxiv.org/pdf/2404.05088
• Copy Paste: [[2404.05088]] How much reliable is ChatGPT's prediction on Information Extraction under Input Perturbations?(https://arxiv.org/abs/2404.05088)
• Keywords: in-context
• Abstract: In this paper, we assess the robustness (reliability) of ChatGPT under input perturbations for one of the most fundamental tasks of Information Extraction (IE) i.e. Named Entity Recognition (NER). Despite the hype, the majority of the researchers have vouched for its language understanding and generation capabilities; a little attention has been paid to understand its robustness: How the input-perturbations affect 1) the predictions, 2) the confidence of predictions and 3) the quality of rationale behind its prediction. We perform a systematic analysis of ChatGPT's robustness (under both zero-shot and few-shot setup) on two NER datasets using both automatic and human evaluation. Based on automatic evaluation metrics, we find that 1) ChatGPT is more brittle on Drug or Disease replacements (rare entities) compared to the perturbations on widely known Person or Location entities, 2) the quality of explanations for the same entity considerably differ under different types of "Entity-Specific" and "Context-Specific" perturbations and the quality can be significantly improved using in-context learning, and 3) it is overconfident for majority of the incorrect predictions, and hence it could lead to misguidance of the end-users.

Title: Reconstructing Retinal Visual Images from 3T fMRI Data Enhanced by Unsupervised Learning

• Authors: Yujian Xiong, Wenhui Zhu, Zhong-Lin Lu, Yalin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05107
• Pdf URL: https://arxiv.org/pdf/2404.05107
• Copy Paste: [[2404.05107]] Reconstructing Retinal Visual Images from 3T fMRI Data Enhanced by Unsupervised Learning(https://arxiv.org/abs/2404.05107)
• Keywords: generative
• Abstract: The reconstruction of human visual inputs from brain activity, particularly through functional Magnetic Resonance Imaging (fMRI), holds promising avenues for unraveling the mechanisms of the human visual system. Despite the significant strides made by deep learning methods in improving the quality and interpretability of visual reconstruction, there remains a substantial demand for high-quality, long-duration, subject-specific 7-Tesla fMRI experiments. The challenge arises in integrating diverse smaller 3-Tesla datasets or accommodating new subjects with brief and low-quality fMRI scans. In response to these constraints, we propose a novel framework that generates enhanced 3T fMRI data through an unsupervised Generative Adversarial Network (GAN), leveraging unpaired training across two distinct fMRI datasets in 7T and 3T, respectively. This approach aims to overcome the limitations of the scarcity of high-quality 7-Tesla data and the challenges associated with brief and low-quality scans in 3-Tesla experiments. In this paper, we demonstrate the reconstruction capabilities of the enhanced 3T fMRI data, highlighting its proficiency in generating superior input visual images compared to data-intensive methods trained and tested on a single subject.

Title: Self-Supervised Multi-Object Tracking with Path Consistency

• Authors: Zijia Lu, Bing Shuai, Yanbei Chen, Zhenlin Xu, Davide Modolo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05136
• Pdf URL: https://arxiv.org/pdf/2404.05136
• Copy Paste: [[2404.05136]] Self-Supervised Multi-Object Tracking with Path Consistency(https://arxiv.org/abs/2404.05136)
• Keywords: self-supervised
• Abstract: In this paper, we propose a novel concept of path consistency to learn robust object matching without using manual object identity supervision. Our key idea is that, to track a object through frames, we can obtain multiple different association results from a model by varying the frames it can observe, i.e., skipping frames in observation. As the differences in observations do not alter the identities of objects, the obtained association results should be consistent. Based on this rationale, we generate multiple observation paths, each specifying a different set of frames to be skipped, and formulate the Path Consistency Loss that enforces the association results are consistent across different observation paths. We use the proposed loss to train our object matching model with only self-supervision. By extensive experiments on three tracking datasets (MOT17, PersonPath22, KITTI), we demonstrate that our method outperforms existing unsupervised methods with consistent margins on various evaluation metrics, and even achieves performance close to supervised methods.

Title: Linguistic Changes in Spontaneous Speech for Detecting Parkinsons Disease Using Large Language Models

• Authors: Jonathan Crawford
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.05160
• Pdf URL: https://arxiv.org/pdf/2404.05160
• Copy Paste: [[2404.05160]] Linguistic Changes in Spontaneous Speech for Detecting Parkinsons Disease Using Large Language Models(https://arxiv.org/abs/2404.05160)
• Keywords: generative
• Abstract: Parkinsons disease is the second most prevalent neurodegenerative disorder with over ten million active cases worldwide and one million new diagnoses per year. Detecting and subsequently diagnosing the disease is challenging because of symptom heterogeneity with respect to complexity, as well as the type and timing of phenotypic manifestations. Typically, language impairment can present in the prodromal phase and precede motor symptoms suggesting that a linguistic-based approach could serve as a diagnostic method for incipient Parkinsons disease. Additionally, improved linguistic models may enhance other approaches through ensemble techniques. The field of large language models is advancing rapidly, presenting the opportunity to explore the use of these new models for detecting Parkinsons disease and to improve on current linguistic approaches with high-dimensional representations of linguistics. We evaluate the application of state-of-the-art large language models to detect Parkinsons disease automatically from spontaneous speech with up to 73% accuracy.

Title: LGSDF: Continual Global Learning of Signed Distance Fields Aided by Local Updating

• Authors: Yufeng Yue, Yinan Deng, Jiahui Wang, Yi Yang
• Subjects: cs.CV, cs.GR, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.05187
• Pdf URL: https://arxiv.org/pdf/2404.05187
• Copy Paste: [[2404.05187]] LGSDF: Continual Global Learning of Signed Distance Fields Aided by Local Updating(https://arxiv.org/abs/2404.05187)
• Keywords: self-supervised
• Abstract: Implicit reconstruction of ESDF (Euclidean Signed Distance Field) involves training a neural network to regress the signed distance from any point to the nearest obstacle, which has the advantages of lightweight storage and continuous querying. However, existing algorithms usually rely on conflicting raw observations as training data, resulting in poor map performance. In this paper, we propose LGSDF, an ESDF continual Global learning algorithm aided by Local updating. At the front end, axis-aligned grids are dynamically updated by pre-processed sensor observations, where incremental fusion alleviates estimation error caused by limited viewing directions. At the back end, a randomly initialized implicit ESDF neural network performs continual self-supervised learning guided by these grids to generate smooth and continuous maps. The results on multiple scenes show that LGSDF can construct more accurate ESDF maps and meshes compared with SOTA (State Of The Art) explicit and implicit mapping algorithms. The source code of LGSDF is publicly available at https://github.com/BIT-DYN/LGSDF.

Title: A secure and private ensemble matcher using multi-vault obfuscated templates

• Authors: Babak Poorebrahim Gilkalaye, Shubhabrata Mukherjee, Reza Derakhshani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05205
• Pdf URL: https://arxiv.org/pdf/2404.05205
• Copy Paste: [[2404.05205]] A secure and private ensemble matcher using multi-vault obfuscated templates(https://arxiv.org/abs/2404.05205)
• Keywords: generative
• Abstract: Given the irrevocability of biometric samples and mounting privacy concerns, biometric template security and secure matching are among the essential features of any well-designed modern biometric system. In this paper, we propose an obfuscation method that hides the biometric template information with just enough chaff. The main idea is to reduce the number of chaff points to a practical level by creating n sub-templates from the original template and hiding each sub-template with m chaff points. During verification, s closest vectors to the biometric query are retrieved from each vault and then combined to generate hash values that are compared with the stored hash value. We demonstrate the effectiveness of synthetic facial images, generated by a Generative Adversarial Network (GAN), as ``random chaff points'' within a secure-vault authorization system. This approach safeguards user identities during training and deployment. We tested our protocol using the AT&T, GT, and LFW face datasets, with the ROC areas under the curve being 0.99, 0.99, and 0.90, respectively. These numbers were close to those of the unprotected templates, showing that our method does not adversely affect accuracy.

Title: SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos

• Authors: Changan Chen, Kumar Ashutosh, Rohit Girdhar, David Harwath, Kristen Grauman
• Subjects: cs.CV, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.05206
• Pdf URL: https://arxiv.org/pdf/2404.05206
• Copy Paste: [[2404.05206]] SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos(https://arxiv.org/abs/2404.05206)
• Keywords: self-supervised
• Abstract: We propose a novel self-supervised embedding to learn how actions sound from narrated in-the-wild egocentric videos. Whereas existing methods rely on curated data with known audio-visual correspondence, our multimodal contrastive-consensus coding (MC3) embedding reinforces the associations between audio, language, and vision when all modality pairs agree, while diminishing those associations when any one pair does not. We show our approach can successfully discover how the long tail of human actions sound from egocentric video, outperforming an array of recent multimodal embedding techniques on two datasets (Ego4D and EPIC-Sounds) and multiple cross-modal tasks.

Title: DiffCJK: Conditional Diffusion Model for High-Quality and Wide-coverage CJK Character Generation

• Authors: Yingtao Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05212
• Pdf URL: https://arxiv.org/pdf/2404.05212
• Copy Paste: [[2404.05212]] DiffCJK: Conditional Diffusion Model for High-Quality and Wide-coverage CJK Character Generation(https://arxiv.org/abs/2404.05212)
• Keywords: diffusion, generative
• Abstract: Chinese, Japanese, and Korean (CJK), with a vast number of native speakers, has profound influence on society and culture. The typesetting of CJK languages carries a wide range of requirements due to the complexity of their scripts and unique literary traditions. A critical aspect of this typesetting process is that CJK fonts need to provide a set of consistent-looking glyphs for approximately one hundred thousand characters. However, creating such a font is inherently labor-intensive and expensive, which significantly hampers the development of new CJK fonts for typesetting, historical, aesthetic, or artistic purposes. To bridge this gap, we are motivated by recent advancements in diffusion-based generative models and propose a novel diffusion method for generating glyphs in a targeted style from a \emph{single} conditioned, standard glyph form. Our experiments show that our method is capable of generating fonts of both printed and hand-written styles, the latter of which presents a greater challenge. Moreover, our approach shows remarkable zero-shot generalization capabilities for non-CJK but Chinese-inspired scripts. We also show our method facilitates smooth style interpolation and generates bitmap images suitable for vectorization, which is crucial in the font creation process. In summary, our proposed method opens the door to high-quality, generative model-assisted font creation for CJK characters, for both typesetting and artistic endeavors.

Title: Out-of-Distribution Data: An Acquaintance of Adversarial Examples -- A Survey

• Authors: Naveen Karunanayake, Ravin Gunawardena, Suranga Seneviratne, Sanjay Chawla
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05219
• Pdf URL: https://arxiv.org/pdf/2404.05219
• Copy Paste: [[2404.05219]] Out-of-Distribution Data: An Acquaintance of Adversarial Examples -- A Survey(https://arxiv.org/abs/2404.05219)
• Keywords: anomaly
• Abstract: Deep neural networks (DNNs) deployed in real-world applications can encounter out-of-distribution (OOD) data and adversarial examples. These represent distinct forms of distributional shifts that can significantly impact DNNs' reliability and robustness. Traditionally, research has addressed OOD detection and adversarial robustness as separate challenges. This survey focuses on the intersection of these two areas, examining how the research community has investigated them together. Consequently, we identify two key research directions: robust OOD detection and unified robustness. Robust OOD detection aims to differentiate between in-distribution (ID) data and OOD data, even when they are adversarially manipulated to deceive the OOD detector. Unified robustness seeks a single approach to make DNNs robust against both adversarial attacks and OOD inputs. Accordingly, first, we establish a taxonomy based on the concept of distributional shifts. This framework clarifies how robust OOD detection and unified robustness relate to other research areas addressing distributional shifts, such as OOD detection, open set recognition, and anomaly detection. Subsequently, we review existing work on robust OOD detection and unified robustness. Finally, we highlight the limitations of the existing work and propose promising research directions that explore adversarial and OOD inputs within a unified framework.

Title: PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detection

• Authors: Xiaofan Li, Zhizhong Zhang, Xin Tan, Chengwei Chen, Yanyun Qu, Yuan Xie, Lizhuang Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05231
• Pdf URL: https://arxiv.org/pdf/2404.05231
• Copy Paste: [[2404.05231]] PromptAD: Learning Prompts with only Normal Samples for Few-Shot Anomaly Detection(https://arxiv.org/abs/2404.05231)
• Keywords: anomaly
• Abstract: The vision-language model has brought great improvement to few-shot industrial anomaly detection, which usually needs to design of hundreds of prompts through prompt engineering. For automated scenarios, we first use conventional prompt learning with many-class paradigm as the baseline to automatically learn prompts but found that it can not work well in one-class anomaly detection. To address the above problem, this paper proposes a one-class prompt learning method for few-shot anomaly detection, termed PromptAD. First, we propose semantic concatenation which can transpose normal prompts into anomaly prompts by concatenating normal prompts with anomaly suffixes, thus constructing a large number of negative samples used to guide prompt learning in one-class setting. Furthermore, to mitigate the training challenge caused by the absence of anomaly images, we introduce the concept of explicit anomaly margin, which is used to explicitly control the margin between normal prompt features and anomaly prompt features through a hyper-parameter. For image-level/pixel-level anomaly detection, PromptAD achieves first place in 11/12 few-shot settings on MVTec and VisA.

Title: Product Description and QA Assisted Self-Supervised Opinion Summarization

• Authors: Tejpalsingh Siledar, Rupasai Rangaraju, Sankara Sri Raghava Ravindra Muddu, Suman Banerjee, Amey Patil, Sudhanshu Shekhar Singh, Muthusamy Chelliah, Nikesh Garera, Swaprava Nath, Pushpak Bhattacharyya
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05243
• Pdf URL: https://arxiv.org/pdf/2404.05243
• Copy Paste: [[2404.05243]] Product Description and QA Assisted Self-Supervised Opinion Summarization(https://arxiv.org/abs/2404.05243)
• Keywords: self-supervised
• Abstract: In e-commerce, opinion summarization is the process of summarizing the consensus opinions found in product reviews. However, the potential of additional sources such as product description and question-answers (QA) has been considered less often. Moreover, the absence of any supervised training data makes this task challenging. To address this, we propose a novel synthetic dataset creation (SDC) strategy that leverages information from reviews as well as additional sources for selecting one of the reviews as a pseudo-summary to enable supervised training. Our Multi-Encoder Decoder framework for Opinion Summarization (MEDOS) employs a separate encoder for each source, enabling effective selection of information while generating the summary. For evaluation, due to the unavailability of test sets with additional sources, we extend the Amazon, Oposum+, and Flipkart test sets and leverage ChatGPT to annotate summaries. Experiments across nine test sets demonstrate that the combination of our SDC approach and MEDOS model achieves on average a 14.5% improvement in ROUGE-1 F1 over the SOTA. Moreover, comparative analysis underlines the significance of incorporating additional sources for generating more informative summaries. Human evaluations further indicate that MEDOS scores relatively higher in coherence and fluency with 0.41 and 0.5 (-1 to 1) respectively, compared to existing models. To the best of our knowledge, we are the first to generate opinion summaries leveraging additional sources in a self-supervised setting.

Title: Text-to-Image Synthesis for Any Artistic Styles: Advancements in Personalized Artistic Image Generation via Subdivision and Dual Binding

• Authors: Junseo Park, Beomseok Ko, Hyeryung Jang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05256
• Pdf URL: https://arxiv.org/pdf/2404.05256
• Copy Paste: [[2404.05256]] Text-to-Image Synthesis for Any Artistic Styles: Advancements in Personalized Artistic Image Generation via Subdivision and Dual Binding(https://arxiv.org/abs/2404.05256)
• Keywords: diffusion
• Abstract: Recent advancements in text-to-image models, such as Stable Diffusion, have demonstrated their ability to synthesize visual images through natural language prompts. One approach of personalizing text-to-image models, exemplified by DreamBooth, fine-tunes the pre-trained model by binding unique text identifiers with a few images of a specific subject. Although existing fine-tuning methods have demonstrated competence in rendering images according to the styles of famous painters, it is still challenging to learn to produce images encapsulating distinct art styles due to abstract and broad visual perceptions of stylistic attributes such as lines, shapes, textures, and colors. In this paper, we introduce a new method, Single-StyleForge, for personalization. It fine-tunes pre-trained text-to-image diffusion models to generate diverse images in specified styles from text prompts. By using around 15-20 images of the target style, the approach establishes a foundational binding of a unique token identifier with a broad range of the target style. It also utilizes auxiliary images to strengthen this binding, resulting in offering specific guidance on representing elements such as persons in a target style-consistent manner. In addition, we present ways to improve the quality of style and text-image alignment through a method called Multi-StyleForge, which inherits the strategy used in StyleForge and learns tokens in multiple. Experimental evaluation conducted on six distinct artistic styles demonstrates substantial improvements in both the quality of generated images and the perceptual fidelity metrics, such as FID, KID, and CLIP scores.

Title: Mask-ControlNet: Higher-Quality Image Generation with An Additional Mask Prompt

• Authors: Zhiqi Huang, Huixin Xiong, Haoyu Wang, Longguang Wang, Zhiheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05331
• Pdf URL: https://arxiv.org/pdf/2404.05331
• Copy Paste: [[2404.05331]] Mask-ControlNet: Higher-Quality Image Generation with An Additional Mask Prompt(https://arxiv.org/abs/2404.05331)
• Keywords: diffusion
• Abstract: Text-to-image generation has witnessed great progress, especially with the recent advancements in diffusion models. Since texts cannot provide detailed conditions like object appearance, reference images are usually leveraged for the control of objects in the generated images. However, existing methods still suffer limited accuracy when the relationship between the foreground and background is complicated. To address this issue, we develop a framework termed Mask-ControlNet by introducing an additional mask prompt. Specifically, we first employ large vision models to obtain masks to segment the objects of interest in the reference image. Then, the object images are employed as additional prompts to facilitate the diffusion model to better understand the relationship between foreground and background regions during image generation. Experiments show that the mask prompts enhance the controllability of the diffusion model to maintain higher fidelity to the reference image while achieving better image quality. Comparison with previous text-to-image generation methods demonstrates our method's superior quantitative and qualitative performance on the benchmark datasets.

Title: Certified PEFTSmoothing: Parameter-Efficient Fine-Tuning with Randomized Smoothing

• Authors: Chengyan Fu, Wenjie Wang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.05350
• Pdf URL: https://arxiv.org/pdf/2404.05350
• Copy Paste: [[2404.05350]] Certified PEFTSmoothing: Parameter-Efficient Fine-Tuning with Randomized Smoothing(https://arxiv.org/abs/2404.05350)
• Keywords: diffusion
• Abstract: Randomized smoothing is the primary certified robustness method for accessing the robustness of deep learning models to adversarial perturbations in the l2-norm, by adding isotropic Gaussian noise to the input image and returning the majority votes over the base classifier. Theoretically, it provides a certified norm bound, ensuring predictions of adversarial examples are stable within this bound. A notable constraint limiting widespread adoption is the necessity to retrain base models entirely from scratch to attain a robust version. This is because the base model fails to learn the noise-augmented data distribution to give an accurate vote. One intuitive way to overcome this challenge is to involve a custom-trained denoiser to eliminate the noise. However, this approach is inefficient and sub-optimal. Inspired by recent large model training procedures, we explore an alternative way named PEFTSmoothing to adapt the base model to learn the Gaussian noise-augmented data with Parameter-Efficient Fine-Tuning (PEFT) methods in both white-box and black-box settings. Extensive results demonstrate the effectiveness and efficiency of PEFTSmoothing, which allow us to certify over 98% accuracy for ViT on CIFAR-10, 20% higher than SoTA denoised smoothing, and over 61% accuracy on ImageNet which is 30% higher than CNN-based denoiser and comparable to the Diffusion-based denoiser.

Title: Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance

• Authors: Dazhong Shen, Guanglu Song, Zeyue Xue, Fu-Yun Wang, Yu Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05384
• Pdf URL: https://arxiv.org/pdf/2404.05384
• Copy Paste: [[2404.05384]] Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance(https://arxiv.org/abs/2404.05384)
• Keywords: diffusion
• Abstract: Classifier-Free Guidance (CFG) has been widely used in text-to-image diffusion models, where the CFG scale is introduced to control the strength of text guidance on the whole image space. However, we argue that a global CFG scale results in spatial inconsistency on varying semantic strengths and suboptimal image quality. To address this problem, we present a novel approach, Semantic-aware Classifier-Free Guidance (S-CFG), to customize the guidance degrees for different semantic units in text-to-image diffusion models. Specifically, we first design a training-free semantic segmentation method to partition the latent image into relatively independent semantic regions at each denoising step. In particular, the cross-attention map in the denoising U-net backbone is renormalized for assigning each patch to the corresponding token, while the self-attention map is used to complete the semantic regions. Then, to balance the amplification of diverse semantic units, we adaptively adjust the CFG scales across different semantic regions to rescale the text guidance degrees into a uniform level. Finally, extensive experiments demonstrate the superiority of S-CFG over the original CFG strategy on various text-to-image diffusion models, without requiring any extra training cost. our codes are available at https://github.com/SmilesDZgk/S-CFG.

Title: Relation Extraction Using Large Language Models: A Case Study on Acupuncture Point Locations

• Authors: Yiming Li, Xueqing Peng, Jianfu Li, Xu Zuo, Suyuan Peng, Donghong Pei, Cui Tao, Hua Xu, Na Hong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05415
• Pdf URL: https://arxiv.org/pdf/2404.05415
• Copy Paste: [[2404.05415]] Relation Extraction Using Large Language Models: A Case Study on Acupuncture Point Locations(https://arxiv.org/abs/2404.05415)
• Keywords: generative
• Abstract: In acupuncture therapy, the accurate location of acupoints is essential for its effectiveness. The advanced language understanding capabilities of large language models (LLMs) like Generative Pre-trained Transformers (GPT) present a significant opportunity for extracting relations related to acupoint locations from textual knowledge sources. This study aims to compare the performance of GPT with traditional deep learning models (Long Short-Term Memory (LSTM) and Bidirectional Encoder Representations from Transformers for Biomedical Text Mining (BioBERT)) in extracting acupoint-related location relations and assess the impact of pretraining and fine-tuning on GPT's performance. We utilized the World Health Organization Standard Acupuncture Point Locations in the Western Pacific Region (WHO Standard) as our corpus, which consists of descriptions of 361 acupoints. Five types of relations ('direction_of,' 'distance_of,' 'part_of,' 'near_acupoint,' and 'located_near') (n= 3,174) between acupoints were annotated. Five models were compared: BioBERT, LSTM, pre-trained GPT-3.5, and fine-tuned GPT-3.5, as well as pre-trained GPT-4. Performance metrics included micro-average exact match precision, recall, and F1 scores. Our results demonstrate that fine-tuned GPT-3.5 consistently outperformed other models in F1 scores across all relation types. Overall, it achieved the highest micro-average F1 score of 0.92. This study underscores the effectiveness of LLMs like GPT in extracting relations related to acupoint locations, with implications for accurately modeling acupuncture knowledge and promoting standard implementation in acupuncture training and practice. The findings also contribute to advancing informatics applications in traditional and complementary medicine, showcasing the potential of LLMs in natural language processing.

Title: Test-Time Zero-Shot Temporal Action Localization

• Authors: Benedetta Liberatori, Alessandro Conti, Paolo Rota, Yiming Wang, Elisa Ricci
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05426
• Pdf URL: https://arxiv.org/pdf/2404.05426
• Copy Paste: [[2404.05426]] Test-Time Zero-Shot Temporal Action Localization(https://arxiv.org/abs/2404.05426)
• Keywords: self-supervised
• Abstract: Zero-Shot Temporal Action Localization (ZS-TAL) seeks to identify and locate actions in untrimmed videos unseen during training. Existing ZS-TAL methods involve fine-tuning a model on a large amount of annotated training data. While effective, training-based ZS-TAL approaches assume the availability of labeled data for supervised learning, which can be impractical in some applications. Furthermore, the training process naturally induces a domain bias into the learned model, which may adversely affect the model's generalization ability to arbitrary videos. These considerations prompt us to approach the ZS-TAL problem from a radically novel perspective, relaxing the requirement for training data. To this aim, we introduce a novel method that performs Test-Time adaptation for Temporal Action Localization (T3AL). In a nutshell, T3AL adapts a pre-trained Vision and Language Model (VLM). T3AL operates in three steps. First, a video-level pseudo-label of the action category is computed by aggregating information from the entire video. Then, action localization is performed adopting a novel procedure inspired by self-supervised learning. Finally, frame-level textual descriptions extracted with a state-of-the-art captioning model are employed for refining the action region proposals. We validate the effectiveness of T3AL by conducting experiments on the THUMOS14 and the ActivityNet-v1.3 datasets. Our results demonstrate that T3AL significantly outperforms zero-shot baselines based on state-of-the-art VLMs, confirming the benefit of a test-time adaptation approach.

Title: Taming Transformers for Realistic Lidar Point Cloud Generation

• Authors: Hamed Haghighi, Amir Samadi, Mehrdad Dianati, Valentina Donzella, Kurt Debattista
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.05505
• Pdf URL: https://arxiv.org/pdf/2404.05505
• Copy Paste: [[2404.05505]] Taming Transformers for Realistic Lidar Point Cloud Generation(https://arxiv.org/abs/2404.05505)
• Keywords: diffusion, generative
• Abstract: Diffusion Models (DMs) have achieved State-Of-The-Art (SOTA) results in the Lidar point cloud generation task, benefiting from their stable training and iterative refinement during sampling. However, DMs often fail to realistically model Lidar raydrop noise due to their inherent denoising process. To retain the strength of iterative sampling while enhancing the generation of raydrop noise, we introduce LidarGRIT, a generative model that uses auto-regressive transformers to iteratively sample the range images in the latent space rather than image space. Furthermore, LidarGRIT utilises VQ-VAE to separately decode range images and raydrop masks. Our results show that LidarGRIT achieves superior performance compared to SOTA models on KITTI-360 and KITTI odometry datasets. Code available at:https://github.com/hamedhaghighi/LidarGRIT.

Title: Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Models

• Authors: Saman Motamed, Wouter Van Gansbeke, Luc Van Gool
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05519
• Pdf URL: https://arxiv.org/pdf/2404.05519
• Copy Paste: [[2404.05519]] Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Models(https://arxiv.org/abs/2404.05519)
• Keywords: diffusion
• Abstract: With recent advances in image and video diffusion models for content creation, a plethora of techniques have been proposed for customizing their generated content. In particular, manipulating the cross-attention layers of Text-to-Image (T2I) diffusion models has shown great promise in controlling the shape and location of objects in the scene. Transferring image-editing techniques to the video domain, however, is extremely challenging as object motion and temporal consistency are difficult to capture accurately. In this work, we take a first look at the role of cross-attention in Text-to-Video (T2V) diffusion models for zero-shot video editing. While one-shot models have shown potential in controlling motion and camera movement, we demonstrate zero-shot control over object shape, position and movement in T2V models. We show that despite the limitations of current T2V models, cross-attention guidance can be a promising approach for editing videos.

Title: Dynamic Backtracking in GFlowNet: Enhancing Decision Steps with Reward-Dependent Adjustment Mechanisms

• Authors: Shuai Guo, Jielei Chu, Lei Zhu, Tianrui Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05576
• Pdf URL: https://arxiv.org/pdf/2404.05576
• Copy Paste: [[2404.05576]] Dynamic Backtracking in GFlowNet: Enhancing Decision Steps with Reward-Dependent Adjustment Mechanisms(https://arxiv.org/abs/2404.05576)
• Keywords: generative
• Abstract: Generative Flow Networks (GFlowNets) are probabilistic models predicated on Markov flows, employing specific amortization algorithms to learn stochastic policies that generate compositional substances including biomolecules, chemical materials, and more. Demonstrating formidable prowess in generating high-performance biochemical molecules, GFlowNets accelerate the discovery of scientific substances, effectively circumventing the time-consuming, labor-intensive, and costly shortcomings intrinsic to conventional material discovery. However, previous work often struggles to accumulate exploratory experience and is prone to becoming disoriented within expansive sampling spaces. Attempts to address this issue, such as LS-GFN, are limited to local greedy searches and lack broader global adjustments. This paper introduces a novel GFlowNet variant, the Dynamic Backtracking GFN (DB-GFN), which enhances the adaptability of decision-making steps through a reward-based dynamic backtracking mechanism. DB-GFN permits backtracking during the network construction process according to the current state's reward value, thus correcting disadvantageous decisions and exploring alternative pathways during the exploration process. Applied to generative tasks of biochemical molecules and genetic material sequences, DB-GFN surpasses existing GFlowNet models and traditional reinforcement learning methods in terms of sample quality, exploration sample quantity, and training convergence speed. Furthermore, the orthogonal nature of DB-GFN suggests its potential as a powerful tool for future improvements in GFN networks, with the promise of integrating with other strategies to achieve more efficient search performance.

Title: Towards More General Video-based Deepfake Detection through Facial Feature Guided Adaptation for Foundation Model

• Authors: Yue-Hua Han, Tai-Ming Huang, Shu-Tzu Lo, Po-Han Huang, Kai-Lung Hua, Jun-Cheng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05583
• Pdf URL: https://arxiv.org/pdf/2404.05583
• Copy Paste: [[2404.05583]] Towards More General Video-based Deepfake Detection through Facial Feature Guided Adaptation for Foundation Model(https://arxiv.org/abs/2404.05583)
• Keywords: foundation model, generative
• Abstract: With the rise of deep learning, generative models have enabled the creation of highly realistic synthetic images, presenting challenges due to their potential misuse. While research in Deepfake detection has grown rapidly in response, many detection methods struggle with unseen Deepfakes generated by new synthesis techniques. To address this generalisation challenge, we propose a novel Deepfake detection approach by adapting rich information encoded inside the Foundation Models with rich information encoded inside, specifically using the image encoder from CLIP which has demonstrated strong zero-shot capability for downstream tasks. Inspired by the recent advances of parameter efficient fine-tuning, we propose a novel side-network-based decoder to extract spatial and temporal cues from the given video clip, with the promotion of the Facial Component Guidance (FCG) to guidencourage the spatial feature to include features of key facial parts for more robust and general Deepfake detection. Through extensive cross-dataset evaluations, our approach exhibits superior effectiveness in identifying unseen Deepfake samples, achieving notable performance improvementsuccess even with limited training samples and manipulation types. Our model secures an average performance enhancement of 0.9% AUROC in cross-dataset assessments comparing with state-of-the-art methods, especiallytablishing a significant lead of achieving 4.4% improvement on the challenging DFDC dataset.

Title: Enhancing Software Related Information Extraction with Generative Language Models through Single-Choice Question Answering

• Authors: Wolfgang Otto, Sharmila Upadhyaya, Stefan Dietze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05587
• Pdf URL: https://arxiv.org/pdf/2404.05587
• Copy Paste: [[2404.05587]] Enhancing Software Related Information Extraction with Generative Language Models through Single-Choice Question Answering(https://arxiv.org/abs/2404.05587)
• Keywords: generative, in-context
• Abstract: This paper describes our participation in the Shared Task on Software Mentions Disambiguation (SOMD), with a focus on improving relation extraction in scholarly texts through Generative Language Models (GLMs) using single-choice question-answering. The methodology prioritises the use of in-context learning capabilities of GLMs to extract software-related entities and their descriptive attributes, such as distributive information. Our approach uses Retrieval-Augmented Generation (RAG) techniques and GLMs for Named Entity Recognition (NER) and Attributive NER to identify relationships between extracted software entities, providing a structured solution for analysing software citations in academic literature. The paper provides a detailed description of our approach, demonstrating how using GLMs in a single-choice QA paradigm can greatly enhance IE methodologies. Our participation in the SOMD shared task highlights the importance of precise software citation practices and showcases our system's ability to overcome the challenges of disambiguating and extracting relationships between software mentions. This sets the groundwork for future research and development in this field.

Title: UniFL: Improve Stable Diffusion via Unified Feedback Learning

• Authors: Jiacheng Zhang, Jie Wu, Yuxi Ren, Xin Xia, Huafeng Kuang, Pan Xie, Jiashi Li, Xuefeng Xiao, Weilin Huang, Min Zheng, Lean Fu, Guanbin Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05595
• Pdf URL: https://arxiv.org/pdf/2404.05595
• Copy Paste: [[2404.05595]] UniFL: Improve Stable Diffusion via Unified Feedback Learning(https://arxiv.org/abs/2404.05595)
• Keywords: diffusion
• Abstract: Diffusion models have revolutionized the field of image generation, leading to the proliferation of high-quality models and diverse downstream applications. However, despite these significant advancements, the current competitive solutions still suffer from several limitations, including inferior visual quality, a lack of aesthetic appeal, and inefficient inference, without a comprehensive solution in sight. To address these challenges, we present UniFL, a unified framework that leverages feedback learning to enhance diffusion models comprehensively. UniFL stands out as a universal, effective, and generalizable solution applicable to various diffusion models, such as SD1.5 and SDXL. Notably, UniFL incorporates three key components: perceptual feedback learning, which enhances visual quality; decoupled feedback learning, which improves aesthetic appeal; and adversarial feedback learning, which optimizes inference speed. In-depth experiments and extensive user studies validate the superior performance of our proposed method in enhancing both the quality of generated models and their acceleration. For instance, UniFL surpasses ImageReward by 17% user preference in terms of generation quality and outperforms LCM and SDXL Turbo by 57% and 20% in 4-step inference. Moreover, we have verified the efficacy of our approach in downstream tasks, including Lora, ControlNet, and AnimateDiff.

Title: A Training-Free Plug-and-Play Watermark Framework for Stable Diffusion

• Authors: Guokai Zhang, Lanjun Wang, Yuting Su, An-An Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05607
• Pdf URL: https://arxiv.org/pdf/2404.05607
• Copy Paste: [[2404.05607]] A Training-Free Plug-and-Play Watermark Framework for Stable Diffusion(https://arxiv.org/abs/2404.05607)
• Keywords: diffusion
• Abstract: Nowadays, the family of Stable Diffusion (SD) models has gained prominence for its high quality outputs and scalability. This has also raised security concerns on social media, as malicious users can create and disseminate harmful content. Existing approaches involve training components or entire SDs to embed a watermark in generated images for traceability and responsibility attribution. However, in the era of AI-generated content (AIGC), the rapid iteration of SDs renders retraining with watermark models costly. To address this, we propose a training-free plug-and-play watermark framework for SDs. Without modifying any components of SDs, we embed diverse watermarks in the latent space, adapting to the denoising process. Our experimental findings reveal that our method effectively harmonizes image quality and watermark invisibility. Furthermore, it performs robustly under various attacks. We also have validated that our method is generalized to multiple versions of SDs, even without retraining the watermark model.

Title: Learning a Category-level Object Pose Estimator without Pose Annotations

• Authors: Fengrui Tian, Yaoyao Liu, Adam Kortylewski, Yueqi Duan, Shaoyi Du, Alan Yuille, Angtian Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05626
• Pdf URL: https://arxiv.org/pdf/2404.05626
• Copy Paste: [[2404.05626]] Learning a Category-level Object Pose Estimator without Pose Annotations(https://arxiv.org/abs/2404.05626)
• Keywords: diffusion
• Abstract: 3D object pose estimation is a challenging task. Previous works always require thousands of object images with annotated poses for learning the 3D pose correspondence, which is laborious and time-consuming for labeling. In this paper, we propose to learn a category-level 3D object pose estimator without pose annotations. Instead of using manually annotated images, we leverage diffusion models (e.g., Zero-1-to-3) to generate a set of images under controlled pose differences and propose to learn our object pose estimator with those images. Directly using the original diffusion model leads to images with noisy poses and artifacts. To tackle this issue, firstly, we exploit an image encoder, which is learned from a specially designed contrastive pose learning, to filter the unreasonable details and extract image feature maps. Additionally, we propose a novel learning strategy that allows the model to learn object poses from those generated image sets without knowing the alignment of their canonical poses. Experimental results show that our method has the capability of category-level object pose estimation from a single shot setting (as pose definition), while significantly outperforming other state-of-the-art methods on the few-shot category-level object pose estimation benchmarks.

Title: Fighting crime with Transformers: Empirical analysis of address parsing methods in payment data

• Authors: Haitham Hammami, Louis Baligand, Bojan Petrovski
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05632
• Pdf URL: https://arxiv.org/pdf/2404.05632
• Copy Paste: [[2404.05632]] Fighting crime with Transformers: Empirical analysis of address parsing methods in payment data(https://arxiv.org/abs/2404.05632)
• Keywords: generative
• Abstract: In the financial industry, identifying the location of parties involved in payments is a major challenge in the context of various regulatory requirements. For this purpose address parsing entails extracting fields such as street, postal code, or country from free text message attributes. While payment processing platforms are updating their standards with more structured formats such as SWIFT with ISO 20022, address parsing remains essential for a considerable volume of messages. With the emergence of Transformers and Generative Large Language Models (LLM), we explore the performance of state-of-the-art solutions given the constraint of processing a vast amount of daily data. This paper also aims to show the need for training robust models capable of dealing with real-world noisy transactional data. Our results suggest that a well fine-tuned Transformer model using early-stopping significantly outperforms other approaches. Nevertheless, generative LLMs demonstrate strong zero-shot performance and warrant further investigations.

Title: BinaryDM: Towards Accurate Binarization of Diffusion Model

• Authors: Xingyu Zheng, Haotong Qin, Xudong Ma, Mingyuan Zhang, Haojie Hao, Jiakai Wang, Zixiang Zhao, Jinyang Guo, Xianglong Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05662
• Pdf URL: https://arxiv.org/pdf/2404.05662
• Copy Paste: [[2404.05662]] BinaryDM: Towards Accurate Binarization of Diffusion Model(https://arxiv.org/abs/2404.05662)
• Keywords: diffusion
• Abstract: With the advancement of diffusion models (DMs) and the substantially increased computational requirements, quantization emerges as a practical solution to obtain compact and efficient low-bit DMs. However, the highly discrete representation leads to severe accuracy degradation, hindering the quantization of diffusion models to ultra-low bit-widths. In this paper, we propose BinaryDM, a novel accurate quantization-aware training approach to push the weights of diffusion models towards the limit of 1-bit. Firstly, we present a Learnable Multi-basis Binarizer (LMB) to recover the representations generated by the binarized DM, which improves the information in details of representations crucial to the DM. Secondly, a Low-rank Representation Mimicking (LRM) is applied to enhance the binarization-aware optimization of the DM, alleviating the optimization direction ambiguity caused by fine-grained alignment. Moreover, a progressive initialization strategy is applied to training DMs to avoid convergence difficulties. Comprehensive experiments demonstrate that BinaryDM achieves significant accuracy and efficiency gains compared to SOTA quantization methods of DMs under ultra-low bit-widths. As the first binarization method for diffusion models, BinaryDM achieves impressive 16.0 times FLOPs and 27.1 times storage savings with 1-bit weight and 4-bit activation, showcasing its substantial advantages and potential for deploying DMs on resource-limited scenarios.

Title: YaART: Yet Another ART Rendering Technology

• Authors: Sergey Kastryulin, Artem Konev, Alexander Shishenya, Eugene Lyapustin, Artem Khurshudov, Alexander Tselousov, Nikita Vinokurov, Denis Kuznedelev, Alexander Markovich, Grigoriy Livshits, Alexey Kirillov, Anastasiia Tabisheva, Liubov Chubarova, Marina Kaminskaia, Alexander Ustyuzhanin, Artemii Shvetsov, Daniil Shlenskii, Valerii Startsev, Dmitrii Kornilov, Mikhail Romanov, Artem Babenko, Sergei Ovcharenko, Valentin Khrulkov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05666
• Pdf URL: https://arxiv.org/pdf/2404.05666
• Copy Paste: [[2404.05666]] YaART: Yet Another ART Rendering Technology(https://arxiv.org/abs/2404.05666)
• Keywords: diffusion, generative
• Abstract: In the rapidly progressing field of generative models, the development of efficient and high-fidelity text-to-image diffusion systems represents a significant frontier. This study introduces YaART, a novel production-grade text-to-image cascaded diffusion model aligned to human preferences using Reinforcement Learning from Human Feedback (RLHF). During the development of YaART, we especially focus on the choices of the model and training dataset sizes, the aspects that were not systematically investigated for text-to-image cascaded diffusion models before. In particular, we comprehensively analyze how these choices affect both the efficiency of the training process and the quality of the generated images, which are highly important in practice. Furthermore, we demonstrate that models trained on smaller datasets of higher-quality images can successfully compete with those trained on larger datasets, establishing a more efficient scenario of diffusion models training. From the quality perspective, YaART is consistently preferred by users over many existing state-of-the-art models.

Title: NAF-DPM: A Nonlinear Activation-Free Diffusion Probabilistic Model for Document Enhancement

• Authors: Giordano Cicchetti, Danilo Comminiello
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05669
• Pdf URL: https://arxiv.org/pdf/2404.05669
• Copy Paste: [[2404.05669]] NAF-DPM: A Nonlinear Activation-Free Diffusion Probabilistic Model for Document Enhancement(https://arxiv.org/abs/2404.05669)
• Keywords: diffusion, generative
• Abstract: Real-world documents may suffer various forms of degradation, often resulting in lower accuracy in optical character recognition (OCR) systems. Therefore, a crucial preprocessing step is essential to eliminate noise while preserving text and key features of documents. In this paper, we propose NAF-DPM, a novel generative framework based on a diffusion probabilistic model (DPM) designed to restore the original quality of degraded documents. While DPMs are recognized for their high-quality generated images, they are also known for their large inference time. To mitigate this problem we provide the DPM with an efficient nonlinear activation-free (NAF) network and we employ as a sampler a fast solver of ordinary differential equations, which can converge in a few iterations. To better preserve text characters, we introduce an additional differentiable module based on convolutional recurrent neural networks, simulating the behavior of an OCR system during training. Experiments conducted on various datasets showcase the superiority of our approach, achieving state-of-the-art performance in terms of pixel-level and perceptual similarity metrics. Furthermore, the results demonstrate a notable character error reduction made by OCR systems when transcribing real-world document images enhanced by our framework. Code and pre-trained models are available at https://github.com/ispamm/NAF-DPM.

Title: CoReS: Orchestrating the Dance of Reasoning and Segmentation

• Authors: Xiaoyi Bao, Siyang Sun, Shuailei Ma, Kecheng Zheng, Yuxin Guo, Guosheng Zhao, Yun Zheng, Xingang Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05673
• Pdf URL: https://arxiv.org/pdf/2404.05673
• Copy Paste: [[2404.05673]] CoReS: Orchestrating the Dance of Reasoning and Segmentation(https://arxiv.org/abs/2404.05673)
• Keywords: in-context
• Abstract: The reasoning segmentation task, which demands a nuanced comprehension of intricate queries to accurately pinpoint object regions, is attracting increasing attention. However, Multi-modal Large Language Models (MLLM) often find it difficult to accurately localize the objects described in complex reasoning contexts. We believe that the act of reasoning segmentation should mirror the cognitive stages of human visual search, where each step is a progressive refinement of thought toward the final object. Thus we introduce the Chains of Reasoning and Segmenting (CoReS) and find this top-down visual hierarchy indeed enhances the visual search process. Specifically, we propose a dual-chain structure that generates multi-modal, chain-like outputs to aid the segmentation process. Furthermore, to steer the MLLM's outputs into this intended hierarchy, we incorporate in-context inputs as guidance. Extensive experiments demonstrate the superior performance of our CoReS, which surpasses the state-of-the-art method by 7.1\% on the ReasonSeg dataset. The code will be released at https://github.com/baoxiaoyi/CoReS.

Title: MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation

• Authors: Kunpeng Song, Yizhe Zhu, Bingchen Liu, Qing Yan, Ahmed Elgammal, Xiao Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05674
• Pdf URL: https://arxiv.org/pdf/2404.05674
• Copy Paste: [[2404.05674]] MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation(https://arxiv.org/abs/2404.05674)
• Keywords: diffusion
• Abstract: In this paper, we present MoMA: an open-vocabulary, training-free personalized image model that boasts flexible zero-shot capabilities. As foundational text-to-image models rapidly evolve, the demand for robust image-to-image translation grows. Addressing this need, MoMA specializes in subject-driven personalized image generation. Utilizing an open-source, Multimodal Large Language Model (MLLM), we train MoMA to serve a dual role as both a feature extractor and a generator. This approach effectively synergizes reference image and text prompt information to produce valuable image features, facilitating an image diffusion model. To better leverage the generated features, we further introduce a novel self-attention shortcut method that efficiently transfers image features to an image diffusion model, improving the resemblance of the target object in generated images. Remarkably, as a tuning-free plug-and-play module, our model requires only a single reference image and outperforms existing methods in generating images with high detail fidelity, enhanced identity-preservation and prompt faithfulness. Our work is open-source, thereby providing universal access to these advancements.

Title: SphereHead: Stable 3D Full-head Synthesis with Spherical Tri-plane Representation

• Authors: Heyuan Li, Ce Chen, Tianhao Shi, Yuda Qiu, Sizhe An, Guanying Chen, Xiaoguang Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05680
• Pdf URL: https://arxiv.org/pdf/2404.05680
• Copy Paste: [[2404.05680]] SphereHead: Stable 3D Full-head Synthesis with Spherical Tri-plane Representation(https://arxiv.org/abs/2404.05680)
• Keywords: generative
• Abstract: While recent advances in 3D-aware Generative Adversarial Networks (GANs) have aided the development of near-frontal view human face synthesis, the challenge of comprehensively synthesizing a full 3D head viewable from all angles still persists. Although PanoHead proves the possibilities of using a large-scale dataset with images of both frontal and back views for full-head synthesis, it often causes artifacts for back views. Based on our in-depth analysis, we found the reasons are mainly twofold. First, from network architecture perspective, we found each plane in the utilized tri-plane/tri-grid representation space tends to confuse the features from both sides, causing "mirroring" artifacts (e.g., the glasses appear in the back). Second, from data supervision aspect, we found that existing discriminator training in 3D GANs mainly focuses on the quality of the rendered image itself, and does not care much about its plausibility with the perspective from which it was rendered. This makes it possible to generate "face" in non-frontal views, due to its easiness to fool the discriminator. In response, we propose SphereHead, a novel tri-plane representation in the spherical coordinate system that fits the human head's geometric characteristics and efficiently mitigates many of the generated artifacts. We further introduce a view-image consistency loss for the discriminator to emphasize the correspondence of the camera parameters and the images. The combination of these efforts results in visually superior outcomes with significantly fewer artifacts. Our code and dataset are publicly available at https://lhyfst.github.io/spherehead.

Title: Learning 3D-Aware GANs from Unposed Images with Template Feature Field

• Authors: Xinya Chen, Hanlei Guo, Yanrui Bin, Shangzhan Zhang, Yuanbo Yang, Yue Wang, Yujun Shen, Yiyi Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05705
• Pdf URL: https://arxiv.org/pdf/2404.05705
• Copy Paste: [[2404.05705]] Learning 3D-Aware GANs from Unposed Images with Template Feature Field(https://arxiv.org/abs/2404.05705)
• Keywords: generative
• Abstract: Collecting accurate camera poses of training images has been shown to well serve the learning of 3D-aware generative adversarial networks (GANs) yet can be quite expensive in practice. This work targets learning 3D-aware GANs from unposed images, for which we propose to perform on-the-fly pose estimation of training images with a learned template feature field (TeFF). Concretely, in addition to a generative radiance field as in previous approaches, we ask the generator to also learn a field from 2D semantic features while sharing the density from the radiance field. Such a framework allows us to acquire a canonical 3D feature template leveraging the dataset mean discovered by the generative model, and further efficiently estimate the pose parameters on real data. Experimental results on various challenging datasets demonstrate the superiority of our approach over state-of-the-art alternatives from both the qualitative and the quantitative perspectives.

Title: MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding

• Authors: Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, Ser-Nam Lim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05726
• Pdf URL: https://arxiv.org/pdf/2404.05726
• Copy Paste: [[2404.05726]] MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding(https://arxiv.org/abs/2404.05726)
• Keywords: foundation model
• Abstract: With the success of large language models (LLMs), integrating the vision model into LLMs to build vision-language foundation models has gained much more interest recently. However, existing LLM-based large multimodal models (e.g., Video-LLaMA, VideoChat) can only take in a limited number of frames for short video understanding. In this study, we mainly focus on designing an efficient and effective model for long-term video understanding. Instead of trying to process more frames simultaneously like most existing work, we propose to process videos in an online manner and store past video information in a memory bank. This allows our model to reference historical video content for long-term analysis without exceeding LLMs' context length constraints or GPU memory limits. Our memory bank can be seamlessly integrated into current multimodal LLMs in an off-the-shelf manner. We conduct extensive experiments on various video understanding tasks, such as long-video understanding, video question answering, and video captioning, and our model can achieve state-of-the-art performances across multiple datasets. Code available at https://boheumd.github.io/MA-LMM/.

Title: Finding Visual Task Vectors

• Authors: Alberto Hojel, Yutong Bai, Trevor Darrell, Amir Globerson, Amir Bar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.05729
• Pdf URL: https://arxiv.org/pdf/2404.05729
• Copy Paste: [[2404.05729]] Finding Visual Task Vectors(https://arxiv.org/abs/2404.05729)
• Keywords: in-context
• Abstract: Visual Prompting is a technique for teaching models to perform a visual task via in-context examples, without any additional training. In this work, we analyze the activations of MAE-VQGAN, a recent Visual Prompting model, and find task vectors, activations that encode task-specific information. Equipped with this insight, we demonstrate that it is possible to identify the task vectors and use them to guide the network towards performing different tasks without providing any input-output examples. To find task vectors, we compute the average intermediate activations per task and use the REINFORCE algorithm to search for the subset of task vectors. The resulting task vectors guide the model towards performing a task better than the original model without the need for input-output examples.