2024-09-11

Title: OPAL: Outlier-Preserved Microscaling Quantization A ccelerator for Generative Large Language Models

• Authors: Jahyun Koo, Dahoon Park, Sangwoo Jung, Jaeha Kung
• Subjects: cs.LG, cs.AR
• Abstract URL: https://arxiv.org/abs/2409.05902
• Pdf URL: https://arxiv.org/pdf/2409.05902
• Copy Paste: [[2409.05902]] OPAL: Outlier-Preserved Microscaling Quantization A ccelerator for Generative Large Language Models(https://arxiv.org/abs/2409.05902)
• Keywords: generative
• Abstract: To overcome the burden on the memory size and bandwidth due to ever-increasing size of large language models (LLMs), aggressive weight quantization has been recently studied, while lacking research on quantizing activations. In this paper, we present a hardware-software co-design method that results in an energy-efficient LLM accelerator, named OPAL, for generation tasks. First of all, a novel activation quantization method that leverages the microscaling data format while preserving several outliers per sub-tensor block (e.g., four out of 128 elements) is proposed. Second, on top of preserving outliers, mixed precision is utilized that sets 5-bit for inputs to sensitive layers in the decoder block of an LLM, while keeping inputs to less sensitive layers to 3-bit. Finally, we present the OPAL hardware architecture that consists of FP units for handling outliers and vectorized INT multipliers for dominant non-outlier related operations. In addition, OPAL uses log2-based approximation on softmax operations that only requires shift and subtraction to maximize power efficiency. As a result, we are able to improve the energy efficiency by 1.6~2.2x, and reduce the area by 2.4~3.1x with negligible accuracy loss, i.e., <1 perplexity increase.

Title: STLLM-DF: A Spatial-Temporal Large Language Model with Diffusion for Enhanced Multi-Mode Traffic System Forecasting

• Authors: Zhiqi Shao, Haoning Xi, Haohui Lu, Ze Wang, Michael G.H. Bell, Junbin Gao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05921
• Pdf URL: https://arxiv.org/pdf/2409.05921
• Copy Paste: [[2409.05921]] STLLM-DF: A Spatial-Temporal Large Language Model with Diffusion for Enhanced Multi-Mode Traffic System Forecasting(https://arxiv.org/abs/2409.05921)
• Keywords: diffusion
• Abstract: The rapid advancement of Intelligent Transportation Systems (ITS) presents challenges, particularly with missing data in multi-modal transportation and the complexity of handling diverse sequential tasks within a centralized framework. To address these issues, we propose the Spatial-Temporal Large Language Model Diffusion (STLLM-DF), an innovative model that leverages Denoising Diffusion Probabilistic Models (DDPMs) and Large Language Models (LLMs) to improve multi-task transportation prediction. The DDPM's robust denoising capabilities enable it to recover underlying data patterns from noisy inputs, making it particularly effective in complex transportation systems. Meanwhile, the non-pretrained LLM dynamically adapts to spatial-temporal relationships within multi-modal networks, allowing the system to efficiently manage diverse transportation tasks in both long-term and short-term predictions. Extensive experiments demonstrate that STLLM-DF consistently outperforms existing models, achieving an average reduction of 2.40\% in MAE, 4.50\% in RMSE, and 1.51\% in MAPE. This model significantly advances centralized ITS by enhancing predictive accuracy, robustness, and overall system performance across multiple tasks, thus paving the way for more effective spatio-temporal traffic forecasting through the integration of frozen transformer language models and diffusion techniques.

Title: Self-Supervised State Space Model for Real-Time Traffic Accident Prediction Using eKAN Networks

• Authors: Xin Tan, Meng Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05933
• Pdf URL: https://arxiv.org/pdf/2409.05933
• Copy Paste: [[2409.05933]] Self-Supervised State Space Model for Real-Time Traffic Accident Prediction Using eKAN Networks(https://arxiv.org/abs/2409.05933)
• Keywords: self-supervised
• Abstract: Accurate prediction of traffic accidents across different times and regions is vital for public safety. However, existing methods face two key challenges: 1) Generalization: Current models rely heavily on manually constructed multi-view structures, like POI distributions and road network densities, which are labor-intensive and difficult to scale across cities. 2) Real-Time Performance: While some methods improve accuracy with complex architectures, they often incur high computational costs, limiting their real-time applicability. To address these challenges, we propose SSL-eKamba, an efficient self-supervised framework for traffic accident prediction. To enhance generalization, we design two self-supervised auxiliary tasks that adaptively improve traffic pattern representation through spatiotemporal discrepancy awareness. For real-time performance, we introduce eKamba, an efficient model that redesigns the Kolmogorov-Arnold Network (KAN) architecture. This involves using learnable univariate functions for input activation and applying a selective mechanism (Selective SSM) to capture multi-variate correlations, thereby improving computational efficiency. Extensive experiments on two real-world datasets demonstrate that SSL-eKamba consistently outperforms state-of-the-art baselines. This framework may also offer new insights for other spatiotemporal tasks. Our source code is publicly available at this http URL.

Title: CoDiCast: Conditional Diffusion Model for Weather Prediction with Uncertainty Quantification

• Authors: Jimeng Shi, Bowen Jin, Jiawei Han, Giri Narasimhan
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2409.05975
• Pdf URL: https://arxiv.org/pdf/2409.05975
• Copy Paste: [[2409.05975]] CoDiCast: Conditional Diffusion Model for Weather Prediction with Uncertainty Quantification(https://arxiv.org/abs/2409.05975)
• Keywords: diffusion
• Abstract: Accurate weather forecasting is critical for science and society. Yet, existing methods have not managed to simultaneously have the properties of high accuracy, low uncertainty, and high computational efficiency. On one hand, to quantify the uncertainty in weather predictions, the strategy of ensemble forecast (i.e., generating a set of diverse predictions) is often employed. However, traditional ensemble numerical weather prediction (NWP) is computationally intensive. On the other hand, most existing machine learning-based weather prediction (MLWP) approaches are efficient and accurate. Nevertheless, they are deterministic and cannot capture the uncertainty of weather forecasting. In this work, we propose CoDiCast, a conditional diffusion model to generate accurate global weather prediction, while achieving uncertainty quantification with ensemble forecasts and modest computational cost. The key idea is to simulate a conditional version of the reverse denoising process in diffusion models, which starts from pure Gaussian noise to generate realistic weather scenarios for a future time point. Each denoising step is conditioned on observations from the recent past. Ensemble forecasts are achieved by repeatedly sampling from stochastic Gaussian noise to represent uncertainty quantification. CoDiCast is trained on a decade of ERA5 reanalysis data from the European Centre for Medium-Range Weather Forecasts (ECMWF). Experimental results demonstrate that our approach outperforms several existing data-driven methods in accuracy. Our conditional diffusion model, CoDiCast, can generate 3-day global weather forecasts, at 6-hour steps and $5.625^\circ$ latitude-longitude resolution, for over 5 variables, in about 12 minutes on a commodity A100 GPU machine with 80GB memory. The open-souced code is provided at \url{this https URL}.

Title: Enhanced Generative Data Augmentation for Semantic Segmentation via Stronger Guidance

• Authors: Quang-Huy Che, Duc-Tri Le, Vinh-Tiep Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06002
• Pdf URL: https://arxiv.org/pdf/2409.06002
• Copy Paste: [[2409.06002]] Enhanced Generative Data Augmentation for Semantic Segmentation via Stronger Guidance(https://arxiv.org/abs/2409.06002)
• Keywords: diffusion, generative
• Abstract: Data augmentation is a widely used technique for creating training data for tasks that require labeled data, such as semantic segmentation. This method benefits pixel-wise annotation tasks requiring much effort and intensive labor. Traditional data augmentation methods involve simple transformations like rotations and flips to create new images from existing ones. However, these new images may lack diversity along the main semantic axes in the data and not change high-level semantic properties. To address this issue, generative models have emerged as an effective solution for augmenting data by generating synthetic images. Controllable generative models offer a way to augment data for semantic segmentation tasks using a prompt and visual reference from the original image. However, using these models directly presents challenges, such as creating an effective prompt and visual reference to generate a synthetic image that accurately reflects the content and structure of the original. In this work, we introduce an effective data augmentation method for semantic segmentation using the Controllable Diffusion Model. Our proposed method includes efficient prompt generation using Class-Prompt Appending and Visual Prior Combination to enhance attention to labeled classes in real images. These techniques allow us to generate images that accurately depict segmented classes in the real image. In addition, we employ the class balancing algorithm to ensure efficiency when merging the synthetic and original images to generate balanced data for the training dataset. We evaluated our method on the PASCAL VOC datasets and found it highly effective for synthesizing images in semantic segmentation.

Title: Identifying the sources of ideological bias in GPT models through linguistic variation in output

• Authors: Christina Walker, Joan C. Timoneda
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06043
• Pdf URL: https://arxiv.org/pdf/2409.06043
• Copy Paste: [[2409.06043]] Identifying the sources of ideological bias in GPT models through linguistic variation in output(https://arxiv.org/abs/2409.06043)
• Keywords: generative
• Abstract: Extant work shows that generative AI models such as GPT-3.5 and 4 perpetuate social stereotypes and biases. One concerning but less explored source of bias is ideology. Do GPT models take ideological stances on politically sensitive topics? In this article, we provide an original approach to identifying ideological bias in generative models, showing that bias can stem from both the training data and the filtering algorithm. We leverage linguistic variation in countries with contrasting political attitudes to evaluate bias in average GPT responses to sensitive political topics in those languages. First, we find that GPT output is more conservative in languages that map well onto conservative societies (i.e., Polish), and more liberal in languages used uniquely in liberal societies (i.e., Swedish). This result provides strong evidence of training data bias in GPT models. Second, differences across languages observed in GPT-3.5 persist in GPT-4, even though GPT-4 is significantly more liberal due to OpenAI's filtering policy. Our main takeaway is that generative model training must focus on high-quality, curated datasets to reduce bias, even if it entails a compromise in training data size. Filtering responses after training only introduces new biases and does not remove the underlying training biases.

Title: Statistical Mechanics of Min-Max Problems

• Authors: Yuma Ichikawa, Koji Hukushima
• Subjects: cs.LG, cond-mat.dis-nn, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.06053
• Pdf URL: https://arxiv.org/pdf/2409.06053
• Copy Paste: [[2409.06053]] Statistical Mechanics of Min-Max Problems(https://arxiv.org/abs/2409.06053)
• Keywords: generative
• Abstract: Min-max optimization problems, also known as saddle point problems, have attracted significant attention due to their applications in various fields, such as fair beamforming, generative adversarial networks (GANs), and adversarial learning. However, understanding the properties of these min-max problems has remained a substantial challenge. This study introduces a statistical mechanical formalism for analyzing the equilibrium values of min-max problems in the high-dimensional limit, while appropriately addressing the order of operations for min and max. As a first step, we apply this formalism to bilinear min-max games and simple GANs, deriving the relationship between the amount of training data and generalization error and indicating the optimal ratio of fake to real data for effective learning. This formalism provides a groundwork for a deeper theoretical analysis of the equilibrium properties in various machine learning methods based on min-max problems and encourages the development of new algorithms and architectures.

Title: DiffusionPen: Towards Controlling the Style of Handwritten Text Generation

• Authors: Konstantina Nikolaidou, George Retsinas, Giorgos Sfikas, Marcus Liwicki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06065
• Pdf URL: https://arxiv.org/pdf/2409.06065
• Copy Paste: [[2409.06065]] DiffusionPen: Towards Controlling the Style of Handwritten Text Generation(https://arxiv.org/abs/2409.06065)
• Keywords: diffusion
• Abstract: Handwritten Text Generation (HTG) conditioned on text and style is a challenging task due to the variability of inter-user characteristics and the unlimited combinations of characters that form new words unseen during training. Diffusion Models have recently shown promising results in HTG but still remain under-explored. We present DiffusionPen (DiffPen), a 5-shot style handwritten text generation approach based on Latent Diffusion Models. By utilizing a hybrid style extractor that combines metric learning and classification, our approach manages to capture both textual and stylistic characteristics of seen and unseen words and styles, generating realistic handwritten samples. Moreover, we explore several variation strategies of the data with multi-style mixtures and noisy embeddings, enhancing the robustness and diversity of the generated data. Extensive experiments using IAM offline handwriting database show that our method outperforms existing methods qualitatively and quantitatively, and its additional generated data can improve the performance of Handwriting Text Recognition (HTR) systems. The code is available at: this https URL.

Title: SVS-GAN: Leveraging GANs for Semantic Video Synthesis

• Authors: Khaled M. Seyam, Julian Wiederer, Markus Braun, Bin Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06074
• Pdf URL: https://arxiv.org/pdf/2409.06074
• Copy Paste: [[2409.06074]] SVS-GAN: Leveraging GANs for Semantic Video Synthesis(https://arxiv.org/abs/2409.06074)
• Keywords: diffusion, generative
• Abstract: In recent years, there has been a growing interest in Semantic Image Synthesis (SIS) through the use of Generative Adversarial Networks (GANs) and diffusion models. This field has seen innovations such as the implementation of specialized loss functions tailored for this task, diverging from the more general approaches in Image-to-Image (I2I) translation. While the concept of Semantic Video Synthesis (SVS)$\unicode{x2013}$the generation of temporally coherent, realistic sequences of images from semantic maps$\unicode{x2013}$is newly formalized in this paper, some existing methods have already explored aspects of this field. Most of these approaches rely on generic loss functions designed for video-to-video translation or require additional data to achieve temporal coherence. In this paper, we introduce the SVS-GAN, a framework specifically designed for SVS, featuring a custom architecture and loss functions. Our approach includes a triple-pyramid generator that utilizes SPADE blocks. Additionally, we employ a U-Net-based network for the image discriminator, which performs semantic segmentation for the OASIS loss. Through this combination of tailored architecture and objective engineering, our framework aims to bridge the existing gap between SIS and SVS, outperforming current state-of-the-art models on datasets like Cityscapes and KITTI-360.

Title: SGC-VQGAN: Towards Complex Scene Representation via Semantic Guided Clustering Codebook

• Authors: Chenjing Ding, Chiyu Wang, Boshi Liu, Xi Guo, Weixuan Tang, Wei Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06105
• Pdf URL: https://arxiv.org/pdf/2409.06105
• Copy Paste: [[2409.06105]] SGC-VQGAN: Towards Complex Scene Representation via Semantic Guided Clustering Codebook(https://arxiv.org/abs/2409.06105)
• Keywords: self-supervised
• Abstract: Vector quantization (VQ) is a method for deterministically learning features through discrete codebook representations. Recent works have utilized visual tokenizers to discretize visual regions for self-supervised representation learning. However, a notable limitation of these tokenizers is lack of semantics, as they are derived solely from the pretext task of reconstructing raw image pixels in an auto-encoder paradigm. Additionally, issues like imbalanced codebook distribution and codebook collapse can adversely impact performance due to inefficient codebook utilization. To address these challenges, We introduce SGC-VQGAN through Semantic Online Clustering method to enhance token semantics through Consistent Semantic Learning. Utilizing inference results from segmentation model , our approach constructs a temporospatially consistent semantic codebook, addressing issues of codebook collapse and imbalanced token semantics. Our proposed Pyramid Feature Learning pipeline integrates multi-level features to capture both image details and semantics simultaneously. As a result, SGC-VQGAN achieves SOTA performance in both reconstruction quality and various downstream tasks. Its simplicity, requiring no additional parameter learning, enables its direct application in downstream tasks, presenting significant potential.

Title: Estimating the Completeness of Discrete Speech Units

• Authors: Sung-Lin Yeh, Hao Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06109
• Pdf URL: https://arxiv.org/pdf/2409.06109
• Copy Paste: [[2409.06109]] Estimating the Completeness of Discrete Speech Units(https://arxiv.org/abs/2409.06109)
• Keywords: self-supervised
• Abstract: Representing speech with discrete units has been widely used in speech codec and speech generation. However, there are several unverified claims about self-supervised discrete units, such as disentangling phonetic and speaker information with k-means, or assuming information loss after k-means. In this work, we take an information-theoretic perspective to answer how much information is present (information completeness) and how much information is accessible (information accessibility), before and after residual vector quantization. We show a lower bound for information completeness and estimate completeness on discretized HuBERT representations after residual vector quantization. We find that speaker information is sufficiently present in HuBERT discrete units, and that phonetic information is sufficiently present in the residual, showing that vector quantization does not achieve disentanglement. Our results offer a comprehensive assessment on the choice of discrete units, and suggest that a lot more information in the residual should be mined rather than discarded.

Title: DECOLLAGE: 3D Detailization by Controllable, Localized, and Learned Geometry Enhancement

• Authors: Qimin Chen, Zhiqin Chen, Vladimir G. Kim, Noam Aigerman, Hao Zhang, Siddhartha Chaudhuri
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06129
• Pdf URL: https://arxiv.org/pdf/2409.06129
• Copy Paste: [[2409.06129]] DECOLLAGE: 3D Detailization by Controllable, Localized, and Learned Geometry Enhancement(https://arxiv.org/abs/2409.06129)
• Keywords: generative
• Abstract: We present a 3D modeling method which enables end-users to refine or detailize 3D shapes using machine learning, expanding the capabilities of AI-assisted 3D content creation. Given a coarse voxel shape (e.g., one produced with a simple box extrusion tool or via generative modeling), a user can directly "paint" desired target styles representing compelling geometric details, from input exemplar shapes, over different regions of the coarse shape. These regions are then up-sampled into high-resolution geometries which adhere with the painted styles. To achieve such controllable and localized 3D detailization, we build on top of a Pyramid GAN by making it masking-aware. We devise novel structural losses and priors to ensure that our method preserves both desired coarse structures and fine-grained features even if the painted styles are borrowed from diverse sources, e.g., different semantic parts and even different shape categories. Through extensive experiments, we show that our ability to localize details enables novel interactive creative workflows and applications. Our experiments further demonstrate that in comparison to prior techniques built on global detailization, our method generates structure-preserving, high-resolution stylized geometries with more coherent shape details and style transitions.

Title: UniLearn: Enhancing Dynamic Facial Expression Recognition through Unified Pre-Training and Fine-Tuning on Images and Videos

• Authors: Yin Chen, Jia Li, Yu Zhang, Zhenzhen Hu, Shiguang Shan, Meng Wang, Richang Hong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06154
• Pdf URL: https://arxiv.org/pdf/2409.06154
• Copy Paste: [[2409.06154]] UniLearn: Enhancing Dynamic Facial Expression Recognition through Unified Pre-Training and Fine-Tuning on Images and Videos(https://arxiv.org/abs/2409.06154)
• Keywords: self-supervised
• Abstract: Dynamic facial expression recognition (DFER) is essential for understanding human emotions and behavior. However, conventional DFER methods, which primarily use dynamic facial data, often underutilize static expression images and their labels, limiting their performance and robustness. To overcome this, we introduce UniLearn, a novel unified learning paradigm that integrates static facial expression recognition (SFER) data to enhance DFER task. UniLearn employs a dual-modal self-supervised pre-training method, leveraging both facial expression images and videos to enhance a ViT model's spatiotemporal representation capability. Then, the pre-trained model is fine-tuned on both static and dynamic expression datasets using a joint fine-tuning strategy. To prevent negative transfer during joint fine-tuning, we introduce an innovative Mixture of Adapter Experts (MoAE) module that enables task-specific knowledge acquisition and effectively integrates information from both static and dynamic expression data. Extensive experiments demonstrate UniLearn's effectiveness in leveraging complementary information from static and dynamic facial data, leading to more accurate and robust DFER. UniLearn consistently achieves state-of-the-art performance on FERV39K, MAFW, and DFEW benchmarks, with weighted average recall (WAR) of 53.65\%, 58.44\%, and 76.68\%, respectively. The source code and model weights will be publicly available at \url{this https URL}.

Title: Larger Language Models Don't Care How You Think: Why Chain-of-Thought Prompting Fails in Subjective Tasks

• Authors: Georgios Chochlakis, Niyantha Maruthu Pandiyan, Kristina Lerman, Shrikanth Narayanan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06173
• Pdf URL: https://arxiv.org/pdf/2409.06173
• Copy Paste: [[2409.06173]] Larger Language Models Don't Care How You Think: Why Chain-of-Thought Prompting Fails in Subjective Tasks(https://arxiv.org/abs/2409.06173)
• Keywords: in-context
• Abstract: In-Context Learning (ICL) in Large Language Models (LLM) has emerged as the dominant technique for performing natural language tasks, as it does not require updating the model parameters with gradient-based methods. ICL promises to "adapt" the LLM to perform the present task at a competitive or state-of-the-art level at a fraction of the computational cost. ICL can be augmented by incorporating the reasoning process to arrive at the final label explicitly in the prompt, a technique called Chain-of-Thought (CoT) prompting. However, recent work has found that ICL relies mostly on the retrieval of task priors and less so on "learning" to perform tasks, especially for complex subjective domains like emotion and morality, where priors ossify posterior predictions. In this work, we examine whether "enabling" reasoning also creates the same behavior in LLMs, wherein the format of CoT retrieves reasoning priors that remain relatively unchanged despite the evidence in the prompt. We find that, surprisingly, CoT indeed suffers from the same posterior collapse as ICL for larger language models. Code is avalaible at this https URL.

Title: EDADepth: Enhanced Data Augmentation for Monocular Depth Estimation

• Authors: Nischal Khanal, Shivanand Venkanna Sheshappanavar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06183
• Pdf URL: https://arxiv.org/pdf/2409.06183
• Copy Paste: [[2409.06183]] EDADepth: Enhanced Data Augmentation for Monocular Depth Estimation(https://arxiv.org/abs/2409.06183)
• Keywords: diffusion
• Abstract: Due to their text-to-image synthesis feature, diffusion models have recently seen a rise in visual perception tasks, such as depth estimation. The lack of good-quality datasets makes the extraction of a fine-grain semantic context challenging for the diffusion models. The semantic context with fewer details further worsens the process of creating effective text embeddings that will be used as input for diffusion models. In this paper, we propose a novel EDADepth, an enhanced data augmentation method to estimate monocular depth without using additional training data. We use Swin2SR, a super-resolution model, to enhance the quality of input images. We employ the BEiT pre-trained semantic segmentation model for better extraction of text embeddings. We introduce BLIP-2 tokenizer to generate tokens from these text embeddings. The novelty of our approach is the introduction of Swin2SR, the BEiT model, and the BLIP-2 tokenizer in the diffusion-based pipeline for the monocular depth estimation. Our model achieves state-of-the-art results (SOTA) on the {\delta}3 metric on NYUv2 and KITTI datasets. It also achieves results comparable to those of the SOTA models in the RMSE and REL metrics. Finally, we also show improvements in the visualization of the estimated depth compared to the SOTA diffusion-based monocular depth estimation models. Code: this https URL.

Title: MyGo: Consistent and Controllable Multi-View Driving Video Generation with Camera Control

• Authors: Yining Yao, Xi Guo, Chenjing Ding, Wei Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06189
• Pdf URL: https://arxiv.org/pdf/2409.06189
• Copy Paste: [[2409.06189]] MyGo: Consistent and Controllable Multi-View Driving Video Generation with Camera Control(https://arxiv.org/abs/2409.06189)
• Keywords: diffusion, generative
• Abstract: High-quality driving video generation is crucial for providing training data for autonomous driving models. However, current generative models rarely focus on enhancing camera motion control under multi-view tasks, which is essential for driving video generation. Therefore, we propose MyGo, an end-to-end framework for video generation, introducing motion of onboard cameras as conditions to make progress in camera controllability and multi-view consistency. MyGo employs additional plug-in modules to inject camera parameters into the pre-trained video diffusion model, which retains the extensive knowledge of the pre-trained model as much as possible. Furthermore, we use epipolar constraints and neighbor view information during the generation process of each view to enhance spatial-temporal consistency. Experimental results show that MyGo has achieved state-of-the-art results in both general camera-controlled video generation and multi-view driving video generation tasks, which lays the foundation for more accurate environment simulation in autonomous driving. Project page: \href{this https URL}{this http URL\_project/MyGo/page.html}

Title: STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning

• Authors: Jaeseong Lee, seung-won hwang, Aurick Qiao, Daniel F Campos, Zhewei Yao, Yuxiong He
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06211
• Pdf URL: https://arxiv.org/pdf/2409.06211
• Copy Paste: [[2409.06211]] STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning(https://arxiv.org/abs/2409.06211)
• Keywords: generative
• Abstract: Mixture-of-experts (MoEs) have been adopted for reducing inference costs by sparsely activating experts in Large language models (LLMs). Despite this reduction, the massive number of experts in MoEs still makes them expensive to serve. In this paper, we study how to address this, by pruning MoEs. Among pruning methodologies, unstructured pruning has been known to achieve the highest performance for a given pruning ratio, compared to structured pruning, since the latter imposes constraints on the sparsification structure. This is intuitive, as the solution space of unstructured pruning subsumes that of structured pruning. However, our counterintuitive finding reveals that expert pruning, a form of structured pruning, can actually precede unstructured pruning to outperform unstructured-only pruning. As existing expert pruning, requiring $O(\frac{k^n}{\sqrt{n}})$ forward passes for $n$ experts, cannot scale for recent MoEs, we propose a scalable alternative with $O(1)$ complexity, yet outperforming the more expensive methods. The key idea is leveraging a latent structure between experts, based on behavior similarity, such that the greedy decision of whether to prune closely captures the joint pruning effect. Ours is highly effective -- for Snowflake Arctic, a 480B-sized MoE with 128 experts, our method needs only one H100 and two hours to achieve nearly no loss in performance with 40% sparsity, even in generative tasks such as GSM8K, where state-of-the-art unstructured pruning fails to. The code will be made publicly available.

Title: Towards Generalizable Scene Change Detection

• Authors: Jaewoo Kim, Uehwan Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06214
• Pdf URL: https://arxiv.org/pdf/2409.06214
• Copy Paste: [[2409.06214]] Towards Generalizable Scene Change Detection(https://arxiv.org/abs/2409.06214)
• Keywords: foundation model
• Abstract: Scene Change Detection (SCD) is vital for applications such as visual surveillance and mobile robotics. However, current SCD methods exhibit a bias to the temporal order of training datasets and limited performance on unseen domains; coventional SCD benchmarks are not able to evaluate generalization or temporal consistency. To tackle these limitations, we introduce a Generalizable Scene Change Detection Framework (GeSCF) in this work. The proposed GeSCF leverages localized semantics of a foundation model without any re-training or fine-tuning -- for generalization over unseen domains. Specifically, we design an adaptive thresholding of the similarity distribution derived from facets of the pre-trained foundation model to generate initial pseudo-change mask. We further utilize Segment Anything Model's (SAM) class-agnostic masks to refine pseudo-masks. Moreover, our proposed framework maintains commutative operations in all settings to ensure complete temporal consistency. Finally, we define new metrics, evaluation dataset, and evaluation protocol for Generalizable Scene Change Detection (GeSCD). Extensive experiments demonstrate that GeSCF excels across diverse and challenging environments -- establishing a new benchmark for SCD performance.

Title: Inference is All You Need: Self Example Retriever for Cross-domain Dialogue State Tracking with ChatGPT

• Authors: Jihyun Lee, Gary Geunbae Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06243
• Pdf URL: https://arxiv.org/pdf/2409.06243
• Copy Paste: [[2409.06243]] Inference is All You Need: Self Example Retriever for Cross-domain Dialogue State Tracking with ChatGPT(https://arxiv.org/abs/2409.06243)
• Keywords: in-context
• Abstract: Traditional dialogue state tracking approaches heavily rely on extensive training data and handcrafted features, limiting their scalability and adaptability to new domains. In this paper, we propose a novel method that leverages inference and in-context learning with ChatGPT for domain transfer in dialogue state tracking, without any parameter updates. By guiding ChatGPT's chain of thought, we enable it to retrieve relevant examples and generalize knowledge to accurately infer dialogue states, solely through inference. Experimental results on the MultiWOZ dataset demonstrate competitive performance and promising generalization across domains. Our parameter-free approach offers a scalable and adaptable solution, opening new research directions in domain transfer learning.

Title: Learning Augmentation Policies from A Model Zoo for Time Series Forecasting

• Authors: Haochen Yuan, Xuelin Li, Yunbo Wang, Xiaokang Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06282
• Pdf URL: https://arxiv.org/pdf/2409.06282
• Copy Paste: [[2409.06282]] Learning Augmentation Policies from A Model Zoo for Time Series Forecasting(https://arxiv.org/abs/2409.06282)
• Keywords: generative
• Abstract: Time series forecasting models typically rely on a fixed-size training set and treat all data uniformly, which may not effectively capture the specific patterns present in more challenging training samples. To address this issue, we introduce AutoTSAug, a learnable data augmentation method based on reinforcement learning. Our approach begins with an empirical analysis to determine which parts of the training data should be augmented. Specifically, we identify the so-called marginal samples by considering the prediction diversity across a set of pretrained forecasting models. Next, we propose using variational masked autoencoders as the augmentation model and applying the REINFORCE algorithm to transform the marginal samples into new data. The goal of this generative model is not only to mimic the distribution of real data but also to reduce the variance of prediction errors across the model zoo. By augmenting the marginal samples with a learnable policy, AutoTSAug substantially improves forecasting performance, advancing the prior art in this field with minimal additional computational cost.

Title: Context Enhancement with Reconstruction as Sequence for Unified Unsupervised Anomaly Detection

• Authors: Hui-Yue Yang, Hui Chen, Lihao Liu, Zijia Lin, Kai Chen, Liejun Wang, Jungong Han, Guiguang Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06285
• Pdf URL: https://arxiv.org/pdf/2409.06285
• Copy Paste: [[2409.06285]] Context Enhancement with Reconstruction as Sequence for Unified Unsupervised Anomaly Detection(https://arxiv.org/abs/2409.06285)
• Keywords: anomaly
• Abstract: Unsupervised anomaly detection (AD) aims to train robust detection models using only normal samples, while can generalize well to unseen anomalies. Recent research focuses on a unified unsupervised AD setting in which only one model is trained for all classes, i.e., n-class-one-model paradigm. Feature-reconstruction-based methods achieve state-of-the-art performance in this scenario. However, existing methods often suffer from a lack of sufficient contextual awareness, thereby compromising the quality of the reconstruction. To address this issue, we introduce a novel Reconstruction as Sequence (RAS) method, which enhances the contextual correspondence during feature reconstruction from a sequence modeling perspective. In particular, based on the transformer technique, we integrate a specialized RASFormer block into RAS. This block enables the capture of spatial relationships among different image regions and enhances sequential dependencies throughout the reconstruction process. By incorporating the RASFormer block, our RAS method achieves superior contextual awareness capabilities, leading to remarkable performance. Experimental results show that our RAS significantly outperforms competing methods, well demonstrating the effectiveness and superiority of our method. Our code is available at this https URL.

Title: Enhancing Long Video Understanding via Hierarchical Event-Based Memory

• Authors: Dingxin Cheng, Mingda Li, Jingyu Liu, Yongxin Guo, Bin Jiang, Qingbin Liu, Xi Chen, Bo Zhao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06299
• Pdf URL: https://arxiv.org/pdf/2409.06299
• Copy Paste: [[2409.06299]] Enhancing Long Video Understanding via Hierarchical Event-Based Memory(https://arxiv.org/abs/2409.06299)
• Keywords: foundation model
• Abstract: Recently, integrating visual foundation models into large language models (LLMs) to form video understanding systems has attracted widespread attention. Most of the existing models compress diverse semantic information within the whole video and feed it into LLMs for content comprehension. While this method excels in short video understanding, it may result in a blend of multiple event information in long videos due to coarse compression, which causes information redundancy. Consequently, the semantics of key events might be obscured within the vast information that hinders the model's understanding capabilities. To address this issue, we propose a Hierarchical Event-based Memory-enhanced LLM (HEM-LLM) for better understanding of long videos. Firstly, we design a novel adaptive sequence segmentation scheme to divide multiple events within long videos. In this way, we can perform individual memory modeling for each event to establish intra-event contextual connections, thereby reducing information redundancy. Secondly, while modeling current event, we compress and inject the information of the previous event to enhance the long-term inter-event dependencies in videos. Finally, we perform extensive experiments on various video understanding tasks and the results show that our model achieves state-of-the-art performances.

Title: High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study

• Authors: Shijie Chang, Lihe Zhang, Huchuan Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06305
• Pdf URL: https://arxiv.org/pdf/2409.06305
• Copy Paste: [[2409.06305]] High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study(https://arxiv.org/abs/2409.06305)
• Keywords: foundation model
• Abstract: Existing few-shot segmentation (FSS) methods mainly focus on designing novel support-query matching and self-matching mechanisms to exploit implicit knowledge in pre-trained backbones. However, the performance of these methods is often constrained by models pre-trained on classification tasks. The exploration of what types of pre-trained models can provide more beneficial implicit knowledge for FSS remains limited. In this paper, inspired by the representation consistency of foundational computer vision models, we develop a FSS framework based on foundation models. To be specific, we propose a simple approach to extract implicit knowledge from foundation models to construct coarse correspondence and introduce a lightweight decoder to refine coarse correspondence for fine-grained segmentation. We systematically summarize the performance of various foundation models on FSS and discover that the implicit knowledge within some of these models is more beneficial for FSS than models pre-trained on classification tasks. Extensive experiments on two widely used datasets demonstrate the effectiveness of our approach in leveraging the implicit knowledge of foundation models. Notably, the combination of DINOv2 and DFN exceeds previous state-of-the-art methods by 17.5% on COCO-20i. Code is available at this https URL.

Title: G3PT: Unleash the power of Autoregressive Modeling in 3D Generation via Cross-scale Querying Transformer

• Authors: Jinzhi Zhang, Feng Xiong, Mu Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06322
• Pdf URL: https://arxiv.org/pdf/2409.06322
• Copy Paste: [[2409.06322]] G3PT: Unleash the power of Autoregressive Modeling in 3D Generation via Cross-scale Querying Transformer(https://arxiv.org/abs/2409.06322)
• Keywords: generative
• Abstract: Autoregressive transformers have revolutionized generative models in language processing and shown substantial promise in image and video generation. However, these models face significant challenges when extended to 3D generation tasks due to their reliance on next-token prediction to learn token sequences, which is incompatible with the unordered nature of 3D data. Instead of imposing an artificial order on 3D data, in this paper, we introduce G3PT, a scalable coarse-to-fine 3D generative model utilizing a cross-scale querying transformer. The key is to map point-based 3D data into discrete tokens with different levels of detail, naturally establishing a sequential relationship between different levels suitable for autoregressive modeling. Additionally, the cross-scale querying transformer connects tokens globally across different levels of detail without requiring an ordered sequence. Benefiting from this approach, G3PT features a versatile 3D generation pipeline that effortlessly supports diverse conditional structures, enabling the generation of 3D shapes from various types of conditions. Extensive experiments demonstrate that G3PT achieves superior generation quality and generalization ability compared to previous 3D generation methods. Most importantly, for the first time in 3D generation, scaling up G3PT reveals distinct power-law scaling behaviors.

Title: Extracting Paragraphs from LLM Token Activations

• Authors: Nicholas Pochinkov, Angelo Benoit, Lovkush Agarwal, Zainab Ali Majid, Lucile Ter-Minassian
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06328
• Pdf URL: https://arxiv.org/pdf/2409.06328
• Copy Paste: [[2409.06328]] Extracting Paragraphs from LLM Token Activations(https://arxiv.org/abs/2409.06328)
• Keywords: generative
• Abstract: Generative large language models (LLMs) excel in natural language processing tasks, yet their inner workings remain underexplored beyond token-level predictions. This study investigates the degree to which these models decide the content of a paragraph at its onset, shedding light on their contextual understanding. By examining the information encoded in single-token activations, specifically the "\textbackslash n\textbackslash n" double newline token, we demonstrate that patching these activations can transfer significant information about the context of the following paragraph, providing further insights into the model's capacity to plan ahead.

Title: Improving Conditional Level Generation using Automated Validation in Match-3 Games

• Authors: Monica Villanueva Aylagas, Joakim Bergdahl, Jonas Gillberg, Alessandro Sestini, Theodor Tolstoy, Linus Gisslén
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06349
• Pdf URL: https://arxiv.org/pdf/2409.06349
• Copy Paste: [[2409.06349]] Improving Conditional Level Generation using Automated Validation in Match-3 Games(https://arxiv.org/abs/2409.06349)
• Keywords: generative
• Abstract: Generative models for level generation have shown great potential in game production. However, they often provide limited control over the generation, and the validity of the generated levels is unreliable. Despite this fact, only a few approaches that learn from existing data provide the users with ways of controlling the generation, simultaneously addressing the generation of unsolvable levels. %One of the main challenges it faces is that levels generated through automation may not be solvable thus requiring validation. are not always engaging, challenging, or even solvable. This paper proposes Avalon, a novel method to improve models that learn from existing level designs using difficulty statistics extracted from gameplay. In particular, we use a conditional variational autoencoder to generate layouts for match-3 levels, conditioning the model on pre-collected statistics such as game mechanics like difficulty and relevant visual features like size and symmetry. Our method is general enough that multiple approaches could potentially be used to generate these statistics. We quantitatively evaluate our approach by comparing it to an ablated model without difficulty conditioning. Additionally, we analyze both quantitatively and qualitatively whether the style of the dataset is preserved in the generated levels. Our approach generates more valid levels than the same method without difficulty conditioning.

Title: DiffQRCoder: Diffusion-based Aesthetic QR Code Generation with Scanning Robustness Guided Iterative Refinement

• Authors: Jia-Wei Liao, Winston Wang, Tzu-Sian Wang, Li-Xuan Peng, Ju-Hsuan Weng, Cheng-Fu Chou, Jun-Cheng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06355
• Pdf URL: https://arxiv.org/pdf/2409.06355
• Copy Paste: [[2409.06355]] DiffQRCoder: Diffusion-based Aesthetic QR Code Generation with Scanning Robustness Guided Iterative Refinement(https://arxiv.org/abs/2409.06355)
• Keywords: diffusion
• Abstract: With the success of Diffusion Models for image generation, the technologies also have revolutionized the aesthetic Quick Response (QR) code generation. Despite significant improvements in visual attractiveness for the beautified codes, their scannabilities are usually sacrificed and thus hinder their practical uses in real-world scenarios. To address this issue, we propose a novel Diffusion-based QR Code generator (DiffQRCoder) to effectively craft both scannable and visually pleasing QR codes. The proposed approach introduces Scanning-Robust Perceptual Guidance (SRPG), a new diffusion guidance for Diffusion Models to guarantee the generated aesthetic codes to obey the ground-truth QR codes while maintaining their attractiveness during the denoising process. Additionally, we present another post-processing technique, Scanning Robust Manifold Projected Gradient Descent (SR-MPGD), to further enhance their scanning robustness through iterative latent space optimization. With extensive experiments, the results demonstrate that our approach not only outperforms other compared methods in Scanning Success Rate (SSR) with better or comparable CLIP aesthetic score (CLIP-aes.) but also significantly improves the SSR of the ControlNet-only approach from 60% to 99%. The subjective evaluation indicates that our approach achieves promising visual attractiveness to users as well. Finally, even with different scanning angles and the most rigorous error tolerance settings, our approach robustly achieves over 95% SSR, demonstrating its capability for real-world applications.

Title: What happens to diffusion model likelihood when your model is conditional?

• Authors: Mattias Cross, Anton Ragni
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06364
• Pdf URL: https://arxiv.org/pdf/2409.06364
• Copy Paste: [[2409.06364]] What happens to diffusion model likelihood when your model is conditional?(https://arxiv.org/abs/2409.06364)
• Keywords: diffusion
• Abstract: Diffusion Models (DMs) iteratively denoise random samples to produce high-quality data. The iterative sampling process is derived from Stochastic Differential Equations (SDEs), allowing a speed-quality trade-off chosen at inference. Another advantage of sampling with differential equations is exact likelihood computation. These likelihoods have been used to rank unconditional DMs and for out-of-domain classification. Despite the many existing and possible uses of DM likelihoods, the distinct properties captured are unknown, especially in conditional contexts such as Text-To-Image (TTI) or Text-To-Speech synthesis (TTS). Surprisingly, we find that TTS DM likelihoods are agnostic to the text input. TTI likelihood is more expressive but cannot discern confounding prompts. Our results show that applying DMs to conditional tasks reveals inconsistencies and strengthens claims that the properties of DM likelihood are unknown. This impact sheds light on the previously unknown nature of DM likelihoods. Although conditional DMs maximise likelihood, the likelihood in question is not as sensitive to the conditioning input as one expects. This investigation provides a new point-of-view on diffusion likelihoods.

Title: Texture-AD: An Anomaly Detection Dataset and Benchmark for Real Algorithm Development

• Authors: Tianwu Lei, Bohan Wang, Silin Chen, Shurong Cao, Ningmu Zou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06367
• Pdf URL: https://arxiv.org/pdf/2409.06367
• Copy Paste: [[2409.06367]] Texture-AD: An Anomaly Detection Dataset and Benchmark for Real Algorithm Development(https://arxiv.org/abs/2409.06367)
• Keywords: anomaly
• Abstract: Anomaly detection is a crucial process in industrial manufacturing and has made significant advancements recently. However, there is a large variance between the data used in the development and the data collected by the production environment. Therefore, we present the Texture-AD benchmark based on representative texture-based anomaly detection to evaluate the effectiveness of unsupervised anomaly detection algorithms in real-world applications. This dataset includes images of 15 different cloth, 14 semiconductor wafers and 10 metal plates acquired under different optical schemes. In addition, it includes more than 10 different types of defects produced during real manufacturing processes, such as scratches, wrinkles, color variations and point defects, which are often more difficult to detect than existing datasets. All anomalous areas are provided with pixel-level annotations to facilitate comprehensive evaluation using anomaly detection models. Specifically, to adapt to diverse products in automated pipelines, we present a new evaluation method and results of baseline algorithms. The experimental results show that Texture-AD is a difficult challenge for state-of-the-art algorithms. To our knowledge, Texture-AD is the first dataset to be devoted to evaluating industrial defect detection algorithms in the real world. The dataset is available at https://XXX.

Title: Distilling Generative-Discriminative Representations for Very Low-Resolution Face Recognition

• Authors: Junzheng Zhang, Weijia Guo, Bochao Liu, Ruixin Shi, Yong Li, Shiming Ge
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.06371
• Pdf URL: https://arxiv.org/pdf/2409.06371
• Copy Paste: [[2409.06371]] Distilling Generative-Discriminative Representations for Very Low-Resolution Face Recognition(https://arxiv.org/abs/2409.06371)
• Keywords: diffusion, generative
• Abstract: Very low-resolution face recognition is challenging due to the serious loss of informative facial details in resolution degradation. In this paper, we propose a generative-discriminative representation distillation approach that combines generative representation with cross-resolution aligned knowledge distillation. This approach facilitates very low-resolution face recognition by jointly distilling generative and discriminative models via two distillation modules. Firstly, the generative representation distillation takes the encoder of a diffusion model pretrained for face super-resolution as the generative teacher to supervise the learning of the student backbone via feature regression, and then freezes the student backbone. After that, the discriminative representation distillation further considers a pretrained face recognizer as the discriminative teacher to supervise the learning of the student head via cross-resolution relational contrastive distillation. In this way, the general backbone representation can be transformed into discriminative head representation, leading to a robust and discriminative student model for very low-resolution face recognition. Our approach improves the recovery of the missing details in very low-resolution faces and achieves better knowledge transfer. Extensive experiments on face datasets demonstrate that our approach enhances the recognition accuracy of very low-resolution faces, showcasing its effectiveness and adaptability.

Title: Prompt2Fashion: An automatically generated fashion dataset

• Authors: Georgia Argyro, Angeliki Dimitriou, Maria Lymperaiou, Giorgos Filandrianos, Giorgos Stamou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06442
• Pdf URL: https://arxiv.org/pdf/2409.06442
• Copy Paste: [[2409.06442]] Prompt2Fashion: An automatically generated fashion dataset(https://arxiv.org/abs/2409.06442)
• Keywords: generative
• Abstract: Despite the rapid evolution and increasing efficacy of language and vision generative models, there remains a lack of comprehensive datasets that bridge the gap between personalized fashion needs and AI-driven design, limiting the potential for truly inclusive and customized fashion solutions. In this work, we leverage generative models to automatically construct a fashion image dataset tailored to various occasions, styles, and body types as instructed by users. We use different Large Language Models (LLMs) and prompting strategies to offer personalized outfits of high aesthetic quality, detail, and relevance to both expert and non-expert users' requirements, as demonstrated by qualitative analysis. Up until now the evaluation of the generated outfits has been conducted by non-expert human subjects. Despite the provided fine-grained insights on the quality and relevance of generation, we extend the discussion on the importance of expert knowledge for the evaluation of artistic AI-generated datasets such as this one. Our dataset is publicly available on GitHub at this https URL.

Title: Learning Generative Interactive Environments By Trained Agent Exploration

• Authors: Naser Kazemi, Nedko Savov, Danda Paudel, Luc Van Gool
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06445
• Pdf URL: https://arxiv.org/pdf/2409.06445
• Copy Paste: [[2409.06445]] Learning Generative Interactive Environments By Trained Agent Exploration(https://arxiv.org/abs/2409.06445)
• Keywords: generative
• Abstract: World models are increasingly pivotal in interpreting and simulating the rules and actions of complex environments. Genie, a recent model, excels at learning from visually diverse environments but relies on costly human-collected data. We observe that their alternative method of using random agents is too limited to explore the environment. We propose to improve the model by employing reinforcement learning based agents for data generation. This approach produces diverse datasets that enhance the model's ability to adapt and perform well across various scenarios and realistic actions within the environment. In this paper, we first release the model GenieRedux - an implementation based on Genie. Additionally, we introduce GenieRedux-G, a variant that uses the agent's readily available actions to factor out action prediction uncertainty during validation. Our evaluation, including a replication of the Coinrun case study, shows that GenieRedux-G achieves superior visual fidelity and controllability using the trained agent exploration. The proposed approach is reproducable, scalable and adaptable to new types of environments. Our codebase is available at this https URL .

Title: Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models

• Authors: Rohit Jena, Ali Taghibakhshi, Sahil Jain, Gerald Shen, Nima Tajbakhsh, Arash Vahdat
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06493
• Pdf URL: https://arxiv.org/pdf/2409.06493
• Copy Paste: [[2409.06493]] Elucidating Optimal Reward-Diversity Tradeoffs in Text-to-Image Diffusion Models(https://arxiv.org/abs/2409.06493)
• Keywords: diffusion
• Abstract: Text-to-image (T2I) diffusion models have become prominent tools for generating high-fidelity images from text prompts. However, when trained on unfiltered internet data, these models can produce unsafe, incorrect, or stylistically undesirable images that are not aligned with human preferences. To address this, recent approaches have incorporated human preference datasets to fine-tune T2I models or to optimize reward functions that capture these preferences. Although effective, these methods are vulnerable to reward hacking, where the model overfits to the reward function, leading to a loss of diversity in the generated images. In this paper, we prove the inevitability of reward hacking and study natural regularization techniques like KL divergence and LoRA scaling, and their limitations for diffusion models. We also introduce Annealed Importance Guidance (AIG), an inference-time regularization inspired by Annealed Importance Sampling, which retains the diversity of the base model while achieving Pareto-Optimal reward-diversity tradeoffs. Our experiments demonstrate the benefits of AIG for Stable Diffusion models, striking the optimal balance between reward optimization and image diversity. Furthermore, a user study confirms that AIG improves diversity and quality of generated images across different model architectures and reward functions.

Title: Aligning Machine and Human Visual Representations across Abstraction Levels

• Authors: Lukas Muttenthaler, Klaus Greff, Frieda Born, Bernhard Spitzer, Simon Kornblith, Michael C. Mozer, Klaus-Robert Müller, Thomas Unterthiner, Andrew K. Lampinen
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06509
• Pdf URL: https://arxiv.org/pdf/2409.06509
• Copy Paste: [[2409.06509]] Aligning Machine and Human Visual Representations across Abstraction Levels(https://arxiv.org/abs/2409.06509)
• Keywords: foundation model
• Abstract: Deep neural networks have achieved success across a wide range of applications, including as models of human behavior in vision tasks. However, neural network training and human learning differ in fundamental ways, and neural networks often fail to generalize as robustly as humans do, raising questions regarding the similarity of their underlying representations. What is missing for modern learning systems to exhibit more human-like behavior? We highlight a key misalignment between vision models and humans: whereas human conceptual knowledge is hierarchically organized from fine- to coarse-scale distinctions, model representations do not accurately capture all these levels of abstraction. To address this misalignment, we first train a teacher model to imitate human judgments, then transfer human-like structure from its representations into pretrained state-of-the-art vision foundation models. These human-aligned models more accurately approximate human behavior and uncertainty across a wide range of similarity tasks, including a new dataset of human judgments spanning multiple levels of semantic abstractions. They also perform better on a diverse set of machine learning tasks, increasing generalization and out-of-distribution robustness. Thus, infusing neural networks with additional human knowledge yields a best-of-both-worlds representation that is both more consistent with human cognition and more practically useful, thus paving the way toward more robust, interpretable, and human-like artificial intelligence systems.

Title: Quantifying and Enabling the Interpretability of CLIP-like Models

• Authors: Avinash Madasu, Yossi Gandelsman, Vasudev Lal, Phillip Howard
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06579
• Pdf URL: https://arxiv.org/pdf/2409.06579
• Copy Paste: [[2409.06579]] Quantifying and Enabling the Interpretability of CLIP-like Models(https://arxiv.org/abs/2409.06579)
• Keywords: in-context
• Abstract: CLIP is one of the most popular foundational models and is heavily used for many vision-language tasks. However, little is known about the inner workings of CLIP. To bridge this gap we propose a study to quantify the interpretability in CLIP like models. We conduct this study on six different CLIP models from OpenAI and OpenCLIP which vary by size, type of pre-training data and patch size. Our approach begins with using the TEXTSPAN algorithm and in-context learning to break down individual attention heads into specific properties. We then evaluate how easily these heads can be interpreted using new metrics which measure property consistency within heads and property disentanglement across heads. Our findings reveal that larger CLIP models are generally more interpretable than their smaller counterparts. To further assist users in understanding the inner workings of CLIP models, we introduce CLIP-InterpreT, a tool designed for interpretability analysis. CLIP-InterpreT offers five types of analyses: property-based nearest neighbor search, per-head topic segmentation, contrastive segmentation, per-head nearest neighbors of an image, and per-head nearest neighbors of text.

Title: Label-free Monitoring of Self-Supervised Learning Progress

• Authors: Isaac Xu, Scott Lowe, Thomas Trappenberg
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06612
• Pdf URL: https://arxiv.org/pdf/2409.06612
• Copy Paste: [[2409.06612]] Label-free Monitoring of Self-Supervised Learning Progress(https://arxiv.org/abs/2409.06612)
• Keywords: self-supervised
• Abstract: Self-supervised learning (SSL) is an effective method for exploiting unlabelled data to learn a high-level embedding space that can be used for various downstream tasks. However, existing methods to monitor the quality of the encoder -- either during training for one model or to compare several trained models -- still rely on access to annotated data. When SSL methodologies are applied to new data domains, a sufficiently large labelled dataset may not always be available. In this study, we propose several evaluation metrics which can be applied on the embeddings of unlabelled data and investigate their viability by comparing them to linear probe accuracy (a common metric which utilizes an annotated dataset). In particular, we apply $k$-means clustering and measure the clustering quality with the silhouette score and clustering agreement. We also measure the entropy of the embedding distribution. We find that while the clusters did correspond better to the ground truth annotations as training of the network progressed, label-free clustering metrics correlated with the linear probe accuracy only when training with SSL methods SimCLR and MoCo-v2, but not with SimSiam. Additionally, although entropy did not always have strong correlations with LP accuracy, this appears to be due to instability arising from early training, with the metric stabilizing and becoming more reliable at later stages of learning. Furthermore, while entropy generally decreases as learning progresses, this trend reverses for SimSiam. More research is required to establish the cause for this unexpected behaviour. Lastly, we find that while clustering based approaches are likely only viable for same-architecture comparisons, entropy may be architecture-independent.

Title: Hierarchical Multi-Label Classification with Missing Information for Benthic Habitat Imagery

• Authors: Isaac Xu, Benjamin Misiuk, Scott C. Lowe, Martin Gillis, Craig J. Brown, Thomas Trappenberg
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06618
• Pdf URL: https://arxiv.org/pdf/2409.06618
• Copy Paste: [[2409.06618]] Hierarchical Multi-Label Classification with Missing Information for Benthic Habitat Imagery(https://arxiv.org/abs/2409.06618)
• Keywords: self-supervised
• Abstract: In this work, we apply state-of-the-art self-supervised learning techniques on a large dataset of seafloor imagery, \textit{BenthicNet}, and study their performance for a complex hierarchical multi-label (HML) classification downstream task. In particular, we demonstrate the capacity to conduct HML training in scenarios where there exist multiple levels of missing annotation information, an important scenario for handling heterogeneous real-world data collected by multiple research groups with differing data collection protocols. We find that, when using smaller one-hot image label datasets typical of local or regional scale benthic science projects, models pre-trained with self-supervision on a larger collection of in-domain benthic data outperform models pre-trained on ImageNet. In the HML setting, we find the model can attain a deeper and more precise classification if it is pre-trained with self-supervision on in-domain data. We hope this work can establish a benchmark for future models in the field of automated underwater image annotation tasks and can guide work in other domains with hierarchical annotations of mixed resolution.

Title: SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation

• Authors: Teng Hu, Jiangning Zhang, Ran Yi, Hongrui Huang, Yabiao Wang, Lizhuang Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06633
• Pdf URL: https://arxiv.org/pdf/2409.06633
• Copy Paste: [[2409.06633]] SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation(https://arxiv.org/abs/2409.06633)
• Keywords: diffusion, generative
• Abstract: In recent years, the development of diffusion models has led to significant progress in image and video generation tasks, with pre-trained models like the Stable Diffusion series playing a crucial role. Inspired by model pruning which lightens large pre-trained models by removing unimportant parameters, we propose a novel model fine-tuning method to make full use of these ineffective parameters and enable the pre-trained model with new task-specified capabilities. In this work, we first investigate the importance of parameters in pre-trained diffusion models, and discover that the smallest 10% to 20% of parameters by absolute values do not contribute to the generation process. Based on this observation, we propose a method termed SaRA that re-utilizes these temporarily ineffective parameters, equating to optimizing a sparse weight matrix to learn the task-specific knowledge. To mitigate overfitting, we propose a nuclear-norm-based low-rank sparse training scheme for efficient fine-tuning. Furthermore, we design a new progressive parameter adjustment strategy to make full use of the re-trained/finetuned parameters. Finally, we propose a novel unstructural backpropagation strategy, which significantly reduces memory costs during fine-tuning. Our method enhances the generative capabilities of pre-trained models in downstream applications and outperforms traditional fine-tuning methods like LoRA in maintaining model's generalization ability. We validate our approach through fine-tuning experiments on SD models, demonstrating significant improvements. SaRA also offers a practical advantage that requires only a single line of code modification for efficient implementation and is seamlessly compatible with existing methods.

Title: EyeCLIP: A visual-language foundation model for multi-modal ophthalmic image analysis

• Authors: Danli Shi, Weiyi Zhang, Jiancheng Yang, Siyu Huang, Xiaolan Chen, Mayinuer Yusufu, Kai Jin, Shan Lin, Shunming Liu, Qing Zhang, Mingguang He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06644
• Pdf URL: https://arxiv.org/pdf/2409.06644
• Copy Paste: [[2409.06644]] EyeCLIP: A visual-language foundation model for multi-modal ophthalmic image analysis(https://arxiv.org/abs/2409.06644)
• Keywords: self-supervised, foundation model
• Abstract: Early detection of eye diseases like glaucoma, macular degeneration, and diabetic retinopathy is crucial for preventing vision loss. While artificial intelligence (AI) foundation models hold significant promise for addressing these challenges, existing ophthalmic foundation models primarily focus on a single modality, whereas diagnosing eye diseases requires multiple modalities. A critical yet often overlooked aspect is harnessing the multi-view information across various modalities for the same patient. Additionally, due to the long-tail nature of ophthalmic diseases, standard fully supervised or unsupervised learning approaches often struggle. Therefore, it is essential to integrate clinical text to capture a broader spectrum of diseases. We propose EyeCLIP, a visual-language foundation model developed using over 2.77 million multi-modal ophthalmology images with partial text data. To fully leverage the large multi-modal unlabeled and labeled data, we introduced a pretraining strategy that combines self-supervised reconstructions, multi-modal image contrastive learning, and image-text contrastive learning to learn a shared representation of multiple modalities. Through evaluation using 14 benchmark datasets, EyeCLIP can be transferred to a wide range of downstream tasks involving ocular and systemic diseases, achieving state-of-the-art performance in disease classification, visual question answering, and cross-modal retrieval. EyeCLIP represents a significant advancement over previous methods, especially showcasing few-shot, even zero-shot capabilities in real-world long-tail scenarios.

Title: Data Collection-free Masked Video Modeling

• Authors: Yuchi Ishikawa, Masayoshi Kondo, Yoshimitsu Aoki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06665
• Pdf URL: https://arxiv.org/pdf/2409.06665
• Copy Paste: [[2409.06665]] Data Collection-free Masked Video Modeling(https://arxiv.org/abs/2409.06665)
• Keywords: self-supervised
• Abstract: Pre-training video transformers generally requires a large amount of data, presenting significant challenges in terms of data collection costs and concerns related to privacy, licensing, and inherent biases. Synthesizing data is one of the promising ways to solve these issues, yet pre-training solely on synthetic data has its own challenges. In this paper, we introduce an effective self-supervised learning framework for videos that leverages readily available and less costly static images. Specifically, we define the Pseudo Motion Generator (PMG) module that recursively applies image transformations to generate pseudo-motion videos from images. These pseudo-motion videos are then leveraged in masked video modeling. Our approach is applicable to synthetic images as well, thus entirely freeing video pre-training from data collection costs and other concerns in real data. Through experiments in action recognition tasks, we demonstrate that this framework allows effective learning of spatio-temporal features through pseudo-motion videos, significantly improving over existing methods which also use static images and partially outperforming those using both real and synthetic videos. These results uncover fragments of what video transformers learn through masked video modeling.