2024-01-15

Title: DFU: scale-robust diffusion model for zero-shot super-resolution image generation

• Authors: Alex Havrilla, Kevin Rojas, Wenjing Liao, Molei Tao
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06144
• Pdf URL: https://arxiv.org/pdf/2401.06144
• Copy Paste: [[2401.06144]] DFU: scale-robust diffusion model for zero-shot super-resolution image generation(https://arxiv.org/abs/2401.06144)
• Keywords: diffusion, generative
• Abstract: Diffusion generative models have achieved remarkable success in generating images with a fixed resolution. However, existing models have limited ability to generalize to different resolutions when training data at those resolutions are not available. Leveraging techniques from operator learning, we present a novel deep-learning architecture, Dual-FNO UNet (DFU), which approximates the score operator by combining both spatial and spectral information at multiple resolutions. Comparisons of DFU to baselines demonstrate its scalability: 1) simultaneously training on multiple resolutions improves FID over training at any single fixed resolution; 2) DFU generalizes beyond its training resolutions, allowing for coherent, high-fidelity generation at higher-resolutions with the same model, i.e. zero-shot super-resolution image-generation; 3) we propose a fine-tuning strategy to further enhance the zero-shot super-resolution image-generation capability of our model, leading to a FID of 11.3 at 1.66 times the maximum training resolution on FFHQ, which no other method can come close to achieving.

Title: AAMDM: Accelerated Auto-regressive Motion Diffusion Model

• Authors: Tianyu Li, Calvin Qiao, Guanqiao Ren, KangKang Yin, Sehoon Ha
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.06146
• Pdf URL: https://arxiv.org/pdf/2401.06146
• Copy Paste: [[2401.06146]] AAMDM: Accelerated Auto-regressive Motion Diffusion Model(https://arxiv.org/abs/2401.06146)
• Keywords: diffusion
• Abstract: Interactive motion synthesis is essential in creating immersive experiences in entertainment applications, such as video games and virtual reality. However, generating animations that are both high-quality and contextually responsive remains a challenge. Traditional techniques in the game industry can produce high-fidelity animations but suffer from high computational costs and poor scalability. Trained neural network models alleviate the memory and speed issues, yet fall short on generating diverse motions. Diffusion models offer diverse motion synthesis with low memory usage, but require expensive reverse diffusion processes. This paper introduces the Accelerated Auto-regressive Motion Diffusion Model (AAMDM), a novel motion synthesis framework designed to achieve quality, diversity, and efficiency all together. AAMDM integrates Denoising Diffusion GANs as a fast Generation Module, and an Auto-regressive Diffusion Model as a Polishing Module. Furthermore, AAMDM operates in a lower-dimensional embedded space rather than the full-dimensional pose space, which reduces the training complexity as well as further improves the performance. We show that AAMDM outperforms existing methods in motion quality, diversity, and runtime efficiency, through comprehensive quantitative analyses and visual comparisons. We also demonstrate the effectiveness of each algorithmic component through ablation studies.

Title: Image Classifier Based Generative Method for Planar Antenna Design

• Authors: Yang Zhong, Weiping Dou, Andrew Cohen, Dia'a Bisharat, Yuandong Tian, Jiang Zhu, Qing Huo Liu
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.06149
• Pdf URL: https://arxiv.org/pdf/2401.06149
• Copy Paste: [[2401.06149]] Image Classifier Based Generative Method for Planar Antenna Design(https://arxiv.org/abs/2401.06149)
• Keywords: generative
• Abstract: To extend the antenna design on printed circuit boards (PCBs) for more engineers of interest, we propose a simple method that models PCB antennas with a few basic components. By taking two separate steps to decide their geometric dimensions and positions, antenna prototypes can be facilitated with no experience required. Random sampling statistics relate to the quality of dimensions are used in selecting among dimension candidates. A novel image-based classifier using a convolutional neural network (CNN) is introduced to further determine the positions of these fixed-dimension components. Two examples from wearable products have been chosen to examine the entire workflow. Their final designs are realistic and their performance metrics are not inferior to the ones designed by experienced engineers.

Title: TriNeRFLet: A Wavelet Based Multiscale Triplane NeRF Representation

• Authors: Rajaei Khatib, Raja Giryes
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06191
• Pdf URL: https://arxiv.org/pdf/2401.06191
• Copy Paste: [[2401.06191]] TriNeRFLet: A Wavelet Based Multiscale Triplane NeRF Representation(https://arxiv.org/abs/2401.06191)
• Keywords: diffusion
• Abstract: In recent years, the neural radiance field (NeRF) model has gained popularity due to its ability to recover complex 3D scenes. Following its success, many approaches proposed different NeRF representations in order to further improve both runtime and performance. One such example is Triplane, in which NeRF is represented using three 2D feature planes. This enables easily using existing 2D neural networks in this framework, e.g., to generate the three planes. Despite its advantage, the triplane representation lagged behind in its 3D recovery quality compared to NeRF solutions. In this work, we propose TriNeRFLet, a 2D wavelet-based multiscale triplane representation for NeRF, which closes the 3D recovery performance gap and is competitive with current state-of-the-art methods. Building upon the triplane framework, we also propose a novel super-resolution (SR) technique that combines a diffusion model with TriNeRFLet for improving NeRF resolution.

Title: Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications

• Authors: Yuwen Xiong, Zhiqi Li, Yuntao Chen, Feng Wang, Xizhou Zhu, Jiapeng Luo, Wenhai Wang, Tong Lu, Hongsheng Li, Yu Qiao, Lewei Lu, Jie Zhou, Jifeng Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06197
• Pdf URL: https://arxiv.org/pdf/2401.06197
• Copy Paste: [[2401.06197]] Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications(https://arxiv.org/abs/2401.06197)
• Keywords: diffusion, generative
• Abstract: We introduce Deformable Convolution v4 (DCNv4), a highly efficient and effective operator designed for a broad spectrum of vision applications. DCNv4 addresses the limitations of its predecessor, DCNv3, with two key enhancements: 1. removing softmax normalization in spatial aggregation to enhance its dynamic property and expressive power and 2. optimizing memory access to minimize redundant operations for speedup. These improvements result in a significantly faster convergence compared to DCNv3 and a substantial increase in processing speed, with DCNv4 achieving more than three times the forward speed. DCNv4 demonstrates exceptional performance across various tasks, including image classification, instance and semantic segmentation, and notably, image generation. When integrated into generative models like U-Net in the latent diffusion model, DCNv4 outperforms its baseline, underscoring its possibility to enhance generative models. In practical applications, replacing DCNv3 with DCNv4 in the InternImage model to create FlashInternImage results in up to 80% speed increase and further performance improvement without further modifications. The advancements in speed and efficiency of DCNv4, combined with its robust performance across diverse vision tasks, show its potential as a foundational building block for future vision models.

Title: Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

• Authors: Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, Saining Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06209
• Pdf URL: https://arxiv.org/pdf/2401.06209
• Copy Paste: [[2401.06209]] Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs(https://arxiv.org/abs/2401.06209)
• Keywords: self-supervised
• Abstract: Is vision good enough for language? Recent advancements in multimodal models primarily stem from the powerful reasoning abilities of large language models (LLMs). However, the visual component typically depends only on the instance-level contrastive language-image pre-training (CLIP). Our research reveals that the visual capabilities in recent multimodal LLMs (MLLMs) still exhibit systematic shortcomings. To understand the roots of these errors, we explore the gap between the visual embedding space of CLIP and vision-only self-supervised learning. We identify ''CLIP-blind pairs'' - images that CLIP perceives as similar despite their clear visual differences. With these pairs, we construct the Multimodal Visual Patterns (MMVP) benchmark. MMVP exposes areas where state-of-the-art systems, including GPT-4V, struggle with straightforward questions across nine basic visual patterns, often providing incorrect answers and hallucinated explanations. We further evaluate various CLIP-based vision-and-language models and found a notable correlation between visual patterns that challenge CLIP models and those problematic for multimodal LLMs. As an initial effort to address these issues, we propose a Mixture of Features (MoF) approach, demonstrating that integrating vision self-supervised learning features with MLLMs can significantly enhance their visual grounding capabilities. Together, our research suggests visual representation learning remains an open challenge, and accurate visual grounding is crucial for future successful multimodal systems.

Title: FedTabDiff: Federated Learning of Diffusion Probabilistic Models for Synthetic Mixed-Type Tabular Data Generation

• Authors: Timur Sattarov, Marco Schreyer, Damian Borth
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06263
• Pdf URL: https://arxiv.org/pdf/2401.06263
• Copy Paste: [[2401.06263]] FedTabDiff: Federated Learning of Diffusion Probabilistic Models for Synthetic Mixed-Type Tabular Data Generation(https://arxiv.org/abs/2401.06263)
• Keywords: diffusion, generative
• Abstract: Realistic synthetic tabular data generation encounters significant challenges in preserving privacy, especially when dealing with sensitive information in domains like finance and healthcare. In this paper, we introduce \textit{Federated Tabular Diffusion} (FedTabDiff) for generating high-fidelity mixed-type tabular data without centralized access to the original tabular datasets. Leveraging the strengths of \textit{Denoising Diffusion Probabilistic Models} (DDPMs), our approach addresses the inherent complexities in tabular data, such as mixed attribute types and implicit relationships. More critically, FedTabDiff realizes a decentralized learning scheme that permits multiple entities to collaboratively train a generative model while respecting data privacy and locality. We extend DDPMs into the federated setting for tabular data generation, which includes a synchronous update scheme and weighted averaging for effective model aggregation. Experimental evaluations on real-world financial and medical datasets attest to the framework's capability to produce synthetic data that maintains high fidelity, utility, privacy, and coverage.

Title: A Study on Self-Supervised Pretraining for Vision Problems in Gastrointestinal Endoscopy

• Authors: Edward Sanderson, Bogdan J. Matuszewski
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06278
• Pdf URL: https://arxiv.org/pdf/2401.06278
• Copy Paste: [[2401.06278]] A Study on Self-Supervised Pretraining for Vision Problems in Gastrointestinal Endoscopy(https://arxiv.org/abs/2401.06278)
• Keywords: self-supervised
• Abstract: Solutions to vision tasks in gastrointestinal endoscopy (GIE) conventionally use image encoders pretrained in a supervised manner with ImageNet-1k as backbones. However, the use of modern self-supervised pretraining algorithms and a recent dataset of 100k unlabelled GIE images (Hyperkvasir-unlabelled) may allow for improvements. In this work, we study the fine-tuned performance of models with ResNet50 and ViT-B backbones pretrained in self-supervised and supervised manners with ImageNet-1k and Hyperkvasir-unlabelled (self-supervised only) in a range of GIE vision tasks. In addition to identifying the most suitable pretraining pipeline and backbone architecture for each task, out of those considered, our results suggest: that self-supervised pretraining generally produces more suitable backbones for GIE vision tasks than supervised pretraining; that self-supervised pretraining with ImageNet-1k is typically more suitable than pretraining with Hyperkvasir-unlabelled, with the notable exception of monocular depth estimation in colonoscopy; and that ViT-Bs are more suitable in polyp segmentation and monocular depth estimation in colonoscopy, ResNet50s are more suitable in polyp detection, and both architectures perform similarly in anatomical landmark recognition and pathological finding characterisation. We hope this work draws attention to the complexity of pretraining for GIE vision tasks, informs this development of more suitable approaches than the convention, and inspires further research on this topic to help advance this development. Code available: \underline{github.com/ESandML/SSL4GIE}

Title: Demystifying Variational Diffusion Models

• Authors: Fabio De Sousa Ribeiro, Ben Glocker
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06281
• Pdf URL: https://arxiv.org/pdf/2401.06281
• Copy Paste: [[2401.06281]] Demystifying Variational Diffusion Models(https://arxiv.org/abs/2401.06281)
• Keywords: diffusion
• Abstract: Despite the growing popularity of diffusion models, gaining a deep understanding of the model class remains somewhat elusive for the uninitiated in non-equilibrium statistical physics. With that in mind, we present what we believe is a more straightforward introduction to diffusion models using directed graphical modelling and variational Bayesian principles, which imposes relatively fewer prerequisites on the average reader. Our exposition constitutes a comprehensive technical review spanning from foundational concepts like deep latent variable models to recent advances in continuous-time diffusion-based modelling, highlighting theoretical connections between model classes along the way. We provide additional mathematical insights that were omitted in the seminal works whenever possible to aid in understanding, while avoiding the introduction of new notation. We envision this article serving as a useful educational supplement for both researchers and practitioners in the area, and we welcome feedback and contributions from the community at https://github.com/biomedia-mira/demystifying-diffusion.

Title: Frequency-Time Diffusion with Neural Cellular Automata

• Authors: John Kalkhof, Arlene Kühn, Yannik Frisch, Anirban Mukhopadhyay
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06291
• Pdf URL: https://arxiv.org/pdf/2401.06291
• Copy Paste: [[2401.06291]] Frequency-Time Diffusion with Neural Cellular Automata(https://arxiv.org/abs/2401.06291)
• Keywords: diffusion, generative
• Abstract: Denoising Diffusion Models (DDMs) have become the leading generative technique for synthesizing high-quality images but are often constrained by their UNet-based architectures that impose certain limitations. In particular, the considerable size of often hundreds of millions of parameters makes them impractical when hardware resources are limited. However, even with powerful hardware, processing images in the gigapixel range is difficult. This is especially true in fields such as microscopy or satellite imaging, where such challenges arise from the limitation to a predefined generative size and the inefficient scaling to larger images. We present two variations of Neural Cellular Automata (NCA)-based DDM methods to address these challenges and jumpstart NCA-based DDMs: Diff-NCA and FourierDiff-NCA. Diff-NCA performs diffusion by using only local features of the underlying distribution, making it suitable for applications where local features are critical. To communicate global knowledge in image space, naive NCA setups require timesteps that increase with the image scale. We solve this bottleneck of current NCA architectures by introducing FourierDiff-NCA, which advances Diff-NCA by adding a Fourier-based diffusion process and combines the frequency-organized Fourier space with the image space. By initiating diffusion in the Fourier domain and finalizing it in the image space, FourierDiff-NCA accelerates global communication. We validate our techniques by using Diff-NCA (208k parameters) to generate high-resolution digital pathology scans at 576x576 resolution and FourierDiff-NCA (887k parameters) to synthesize CelebA images at 64x64, outperforming VNCA and five times bigger UNet-based DDMs. In addition, we demonstrate FourierDiff-NCA's capabilities in super-resolution, OOD image synthesis, and inpainting without additional training.

Title: Misconfidence-based Demonstration Selection for LLM In-Context Learning

• Authors: Shangqing Xu, Chao Zhang (Georgia Institute of Technology)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06301
• Pdf URL: https://arxiv.org/pdf/2401.06301
• Copy Paste: [[2401.06301]] Misconfidence-based Demonstration Selection for LLM In-Context Learning(https://arxiv.org/abs/2401.06301)
• Keywords: in-context
• Abstract: In-context learning with large language models (LLMs) excels at adapting to various tasks rapidly. However, its success hinges on carefully selecting demonstrations, which remains an obstacle in practice. Current approaches to this problem either rely on hard-to-acquire external supervision or require frequent interactions with LLMs, resulting in high costs. We propose a new method called In-Context Reflection (ICR) to overcome these challenges. ICR strategically selects demonstrations to reduce the discrepancy between the LLM's outputs and the actual input-output mappings. Specifically, ICR starts with a random set of initial demonstrations, then iteratively refines it. In each step, it analyzes a pool of candidate examples and identifies the ones most likely to challenge the LLM's current understanding, measured by a new metric called misconfidence. These most confusing examples are then selected to replace the less informative demonstrations in the current set. Our comprehensive evaluation across five diverse datasets encompassing 13 subtasks shows the efficacy of ICR. Compared to existing methods, ICR achieves an average performance boost of 4%, while demonstrating remarkable cross-task generalization capabilities.

Title: Learning from Semi-Factuals: A Debiased and Semantic-Aware Framework for Generalized Relation Discovery

• Authors: Jiaxin Wang, Lingling Zhang, Jun Liu, Tianlin Guo, Wenjun Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06327
• Pdf URL: https://arxiv.org/pdf/2401.06327
• Copy Paste: [[2401.06327]] Learning from Semi-Factuals: A Debiased and Semantic-Aware Framework for Generalized Relation Discovery(https://arxiv.org/abs/2401.06327)
• Keywords: self-supervised
• Abstract: We introduce a novel task, called Generalized Relation Discovery (GRD), for open-world relation extraction. GRD aims to identify unlabeled instances in existing pre-defined relations or discover novel relations by assigning instances to clusters as well as providing specific meanings for these clusters. The key challenges of GRD are how to mitigate the serious model biases caused by labeled pre-defined relations to learn effective relational representations and how to determine the specific semantics of novel relations during classifying or clustering unlabeled instances. We then propose a novel framework, SFGRD, for this task to solve the above issues by learning from semi-factuals in two stages. The first stage is semi-factual generation implemented by a tri-view debiased relation representation module, in which we take each original sentence as the main view and design two debiased views to generate semi-factual examples for this sentence. The second stage is semi-factual thinking executed by a dual-space tri-view collaborative relation learning module, where we design a cluster-semantic space and a class-index space to learn relational semantics and relation label indices, respectively. In addition, we devise alignment and selection strategies to integrate two spaces and establish a self-supervised learning loop for unlabeled data by doing semi-factual thinking across three views. Extensive experimental results show that SFGRD surpasses state-of-the-art models in terms of accuracy by 2.36\% $\sim$5.78\% and cosine similarity by 32.19\%$\sim$ 84.45\% for relation label index and relation semantic quality, respectively. To the best of our knowledge, we are the first to exploit the efficacy of semi-factuals in relation extraction.

Title: Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering

• Authors: Chang Yu, Junran Peng, Xiangyu Zhu, Zhaoxiang Zhang, Qi Tian, Zhen Lei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06345
• Pdf URL: https://arxiv.org/pdf/2401.06345
• Copy Paste: [[2401.06345]] Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering(https://arxiv.org/abs/2401.06345)
• Keywords: diffusion
• Abstract: The text-to-image synthesis by diffusion models has recently shown remarkable performance in generating high-quality images. Although performs well for simple texts, the models may get confused when faced with complex texts that contain multiple objects or spatial relationships. To get the desired images, a feasible way is to manually adjust the textual descriptions, i.e., narrating the texts or adding some words, which is labor-consuming. In this paper, we propose a framework to learn the proper textual descriptions for diffusion models through prompt learning. By utilizing the quality guidance and the semantic guidance derived from the pre-trained diffusion model, our method can effectively learn the prompts to improve the matches between the input text and the generated images. Extensive experiments and analyses have validated the effectiveness of the proposed method.

Title: SamLP: A Customized Segment Anything Model for License Plate Detection

• Authors: Haoxuan Ding, Junyu Gao, Yuan Yuan, Qi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06374
• Pdf URL: https://arxiv.org/pdf/2401.06374
• Copy Paste: [[2401.06374]] SamLP: A Customized Segment Anything Model for License Plate Detection(https://arxiv.org/abs/2401.06374)
• Keywords: foundation model
• Abstract: With the emergence of foundation model, this novel paradigm of deep learning has encouraged many powerful achievements in natural language processing and computer vision. There are many advantages of foundation model, such as excellent feature extraction power, mighty generalization ability, great few-shot and zero-shot learning capacity, etc. which are beneficial to vision tasks. As the unique identity of vehicle, different countries and regions have diverse license plate (LP) styles and appearances, and even different types of vehicles have different LPs. However, recent deep learning based license plate detectors are mainly trained on specific datasets, and these limited datasets constrain the effectiveness and robustness of LP detectors. To alleviate the negative impact of limited data, an attempt to exploit the advantages of foundation model is implement in this paper. We customize a vision foundation model, i.e. Segment Anything Model (SAM), for LP detection task and propose the first LP detector based on vision foundation model, named SamLP. Specifically, we design a Low-Rank Adaptation (LoRA) fine-tuning strategy to inject extra parameters into SAM and transfer SAM into LP detection task. And then, we further propose a promptable fine-tuning step to provide SamLP with prompatable segmentation capacity. The experiments show that our proposed SamLP achieves promising detection performance compared to other LP detectors. Meanwhile, the proposed SamLP has great few-shot and zero-shot learning ability, which shows the potential of transferring vision foundation model. The code is available at https://github.com/Dinghaoxuan/SamLP

Title: Adaptive Data Augmentation for Aspect Sentiment Quad Prediction

• Authors: Wenyuan Zhang, Xinghua Zhang, Shiyao Cui, Kun Huang, Xuebin Wang, Tingwen Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06394
• Pdf URL: https://arxiv.org/pdf/2401.06394
• Copy Paste: [[2401.06394]] Adaptive Data Augmentation for Aspect Sentiment Quad Prediction(https://arxiv.org/abs/2401.06394)
• Keywords: generative
• Abstract: Aspect sentiment quad prediction (ASQP) aims to predict the quad sentiment elements for a given sentence, which is a critical task in the field of aspect-based sentiment analysis. However, the data imbalance issue has not received sufficient attention in ASQP task. In this paper, we divide the issue into two-folds, quad-pattern imbalance and aspect-category imbalance, and propose an Adaptive Data Augmentation (ADA) framework to tackle the imbalance issue. Specifically, a data augmentation process with a condition function adaptively enhances the tail quad patterns and aspect categories, alleviating the data imbalance in ASQP. Following previous studies, we also further explore the generative framework for extracting complete quads by introducing the category prior knowledge and syntax-guided decoding target. Experimental results demonstrate that data augmentation for imbalance in ASQP task can improve the performance, and the proposed ADA method is superior to naive data oversampling.

Title: ModaVerse: Efficiently Transforming Modalities with LLMs

• Authors: Xinyu Wang, Bohan Zhuang, Qi Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06395
• Pdf URL: https://arxiv.org/pdf/2401.06395
• Copy Paste: [[2401.06395]] ModaVerse: Efficiently Transforming Modalities with LLMs(https://arxiv.org/abs/2401.06395)
• Keywords: generative
• Abstract: Humans possess the capability to comprehend diverse modalities and seamlessly transfer information between them. In this work, we introduce ModaVerse, a Multi-modal Large Language Model (MLLM) capable of comprehending and transforming content across various modalities including images, videos, and audio. Predominant MLLM frameworks have largely relied on the alignment of latent spaces of textual and non-textual features. This alignment process, which synchronizes a language model trained on textual data with encoders and decoders trained on multi-modal data, often necessitates extensive training of several projection layers in multiple stages. Inspired by LLM-as-agent methodologies, we propose a novel Input/Output (I/O) alignment mechanism that operates directly at the level of natural language. It aligns the LLM's output with the input of generative models, avoiding the complexities associated with latent feature alignments, and simplifying the multiple training stages of existing MLLMs into a single, efficient process. This conceptual advancement leads to significant reductions in both data and computational costs. By conducting experiments on several benchmarks, we demonstrate that our approach attains comparable performance with the state of the art while achieving considerable efficiencies in data usage and training duration.

Title: UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding

• Authors: Bowen Shi, Peisen Zhao, Zichen Wang, Yuhang Zhang, Yaoming Wang, Jin Li, Wenrui Dai, Junni Zou, Hongkai Xiong, Qi Tian, Xiaopeng Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06397
• Pdf URL: https://arxiv.org/pdf/2401.06397
• Copy Paste: [[2401.06397]] UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding(https://arxiv.org/abs/2401.06397)
• Keywords: foundation model
• Abstract: Vision-language foundation models, represented by Contrastive language-image pre-training (CLIP), have gained increasing attention for jointly understanding both vision and textual tasks. However, existing approaches primarily focus on training models to match global image representations with textual descriptions, thereby overlooking the critical alignment between local regions and corresponding text tokens. This paper extends CLIP with multi-granularity alignment. Notably, we deliberately construct a new dataset comprising pseudo annotations at various levels of granularities, encompassing image-level, region-level, and pixel-level captions/tags. Accordingly, we develop a unified multi-granularity learning framework, named UMG-CLIP, that simultaneously empowers the model with versatile perception abilities across different levels of detail. Equipped with parameter efficient tuning, UMG-CLIP surpasses current widely used CLIP models and achieves state-of-the-art performance on diverse image understanding benchmarks, including open-world recognition, retrieval, semantic segmentation, and panoptic segmentation tasks. We hope UMG-CLIP can serve as a valuable option for advancing vision-language foundation models.

Title: Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model

• Authors: Taehee Kim, Yeongjae Cho, Heejun Shin, Yohan Jo, Dongmyung Shin
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06400
• Pdf URL: https://arxiv.org/pdf/2401.06400
• Copy Paste: [[2401.06400]] Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model(https://arxiv.org/abs/2401.06400)
• Keywords: foundation model
• Abstract: Visual question answering (VQA) is a task where an image is given, and a series of questions are asked about the image. To build an efficient VQA algorithm, a large amount of QA data is required which is very expensive. Generating synthetic QA pairs based on templates is a practical way to obtain data. However, VQA models trained on those data do not perform well on complex, human-written questions. To address this issue, we propose a new method called {\it chain of QA for human-written questions} (CoQAH). CoQAH utilizes a sequence of QA interactions between a large language model and a VQA model trained on synthetic data to reason and derive logical answers for human-written questions. We tested the effectiveness of CoQAH on two types of human-written VQA datasets for 3D-rendered and chest X-ray images and found that it achieved state-of-the-art accuracy in both types of data. Notably, CoQAH outperformed general vision-language models, VQA models, and medical foundation models with no finetuning.

Title: Heterogeneous Low-Rank Approximation for Federated Fine-tuning of On-Device Foundation Models

• Authors: Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Gauri Joshi
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.06432
• Pdf URL: https://arxiv.org/pdf/2401.06432
• Copy Paste: [[2401.06432]] Heterogeneous Low-Rank Approximation for Federated Fine-tuning of On-Device Foundation Models(https://arxiv.org/abs/2401.06432)
• Keywords: foundation model
• Abstract: Large foundation models (FMs) adapt surprisingly well to specific domains or tasks with fine-tuning. Federated learning (FL) further enables private FM fine-tuning using the local data on devices. However, the standard FMs' large size poses challenges for resource-constrained and heterogeneous devices. To address this, we consider FMs with reduced parameter sizes, referred to as on-device FMs (ODFMs). While ODFMs allow on-device inference, computational constraints still hinder efficient federated fine-tuning. We propose a parameter-efficient federated fine-tuning method for ODFMs using heterogeneous low-rank approximations (LoRAs) that addresses system and data heterogeneity. We show that homogeneous LoRA ranks face a trade-off between overfitting and slow convergence, and propose HetLoRA, which employs heterogeneous ranks across clients and eliminates the shortcomings of homogeneous HetLoRA. By applying rank self-pruning locally and sparsity-weighted aggregation at the server, we combine the advantages of high and low-rank LoRAs, which achieves improved convergence speed and final performance compared to homogeneous LoRA. Furthermore, it offers enhanced computation efficiency compared to full fine-tuning, making it suitable for heterogeneous devices while preserving data privacy.

Title: RotationDrag: Point-based Image Editing with Rotated Diffusion Features

• Authors: Minxing Luo, Wentao Cheng, Jian Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06442
• Pdf URL: https://arxiv.org/pdf/2401.06442
• Copy Paste: [[2401.06442]] RotationDrag: Point-based Image Editing with Rotated Diffusion Features(https://arxiv.org/abs/2401.06442)
• Keywords: diffusion, generative
• Abstract: A precise and user-friendly manipulation of image content while preserving image fidelity has always been crucial to the field of image editing. Thanks to the power of generative models, recent point-based image editing methods allow users to interactively change the image content with high generalizability by clicking several control points. But the above mentioned editing process is usually based on the assumption that features stay constant in the motion supervision step from initial to target points. In this work, we conduct a comprehensive investigation in the feature space of diffusion models, and find that features change acutely under in-plane rotation. Based on this, we propose a novel approach named RotationDrag, which significantly improves point-based image editing performance when users intend to in-plane rotate the image content. Our method tracks handle points more precisely by utilizing the feature map of the rotated images, thus ensuring precise optimization and high image fidelity. Furthermore, we build a in-plane rotation focused benchmark called RotateBench, the first benchmark to evaluate the performance of point-based image editing method under in-plane rotation scenario on both real images and generated images. A thorough user study demonstrates the superior capability in accomplishing in-plane rotation that users intend to achieve, comparing the DragDiffusion baseline and other existing diffusion-based methods. See the project page https://github.com/Tony-Lowe/RotationDrag for code and experiment results.

Title: BOK-VQA: Bilingual Outside Knowledge-based Visual Question Answering via Graph Representation Pretraining

• Authors: Minjun Kim, Seungwoo Song, Youhan Lee, Haneol Jang, Kyungtae Lim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06443
• Pdf URL: https://arxiv.org/pdf/2401.06443
• Copy Paste: [[2401.06443]] BOK-VQA: Bilingual Outside Knowledge-based Visual Question Answering via Graph Representation Pretraining(https://arxiv.org/abs/2401.06443)
• Keywords: generative
• Abstract: The current research direction in generative models, such as the recently developed GPT4, aims to find relevant knowledge information for multimodal and multilingual inputs to provide answers. Under these research circumstances, the demand for multilingual evaluation of visual question answering (VQA) tasks, a representative task of multimodal systems, has increased. Accordingly, we propose a bilingual outside-knowledge VQA (BOK-VQA) dataset in this study that can be extended to multilingualism. The proposed data include 17K images, 17K question-answer pairs for both Korean and English and 280K instances of knowledge information related to question-answer content. We also present a framework that can effectively inject knowledge information into a VQA system by pretraining the knowledge information of BOK-VQA data in the form of graph embeddings. Finally, through in-depth analysis, we demonstrated the actual effect of the knowledge information contained in the constructed training data on VQA.

Title: Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learning

• Authors: Kaiyi Zhang, Ang Lv, Yuhan Chen, Hansen Ha, Tao Xu, Rui Yan
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06469
• Pdf URL: https://arxiv.org/pdf/2401.06469
• Copy Paste: [[2401.06469]] Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learning(https://arxiv.org/abs/2401.06469)
• Keywords: in-context
• Abstract: In this paper, by treating in-context learning (ICL) as a meta-optimization process, we explain why LLMs are sensitive to the order of ICL examples. This understanding leads us to the development of Batch-ICL, an effective, efficient, and order-agnostic inference algorithm for ICL. Differing from the standard N-shot learning approach, Batch-ICL employs $N$ separate 1-shot forward computations and aggregates the resulting meta-gradients. These aggregated meta-gradients are then applied to a zero-shot learning to generate the final prediction. This batch processing approach renders the LLM agnostic to the order of ICL examples. Through extensive experiments and analysis, we demonstrate that Batch-ICL consistently outperforms most permutations of example sequences. In some cases, it even exceeds the performance of the optimal order for standard ICL, all while reducing the computational resources required. Furthermore, we develop a novel variant of Batch-ICL featuring multiple "epochs" of meta-optimization. This variant implicitly explores permutations of ICL examples, further enhancing ICL performance.

Title: Self-supervised Learning of Dense Hierarchical Representations for Medical Image Segmentation

• Authors: Eytan Kats, Jochen G. Hirsch, Mattias P. Heinrich
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06473
• Pdf URL: https://arxiv.org/pdf/2401.06473
• Copy Paste: [[2401.06473]] Self-supervised Learning of Dense Hierarchical Representations for Medical Image Segmentation(https://arxiv.org/abs/2401.06473)
• Keywords: self-supervised
• Abstract: This paper demonstrates a self-supervised framework for learning voxel-wise coarse-to-fine representations tailored for dense downstream tasks. Our approach stems from the observation that existing methods for hierarchical representation learning tend to prioritize global features over local features due to inherent architectural bias. To address this challenge, we devise a training strategy that balances the contributions of features from multiple scales, ensuring that the learned representations capture both coarse and fine-grained details. Our strategy incorporates 3-fold improvements: (1) local data augmentations, (2) a hierarchically balanced architecture, and (3) a hybrid contrastive-restorative loss function. We evaluate our method on CT and MRI data and demonstrate that our new approach particularly beneficial for fine-tuning with limited annotated data and consistently outperforms the baseline counterpart in linear evaluation settings.

Title: Frequency Masking for Universal Deepfake Detection

• Authors: Chandler Timm Doloriel, Ngai-Man Cheung
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06506
• Pdf URL: https://arxiv.org/pdf/2401.06506
• Copy Paste: [[2401.06506]] Frequency Masking for Universal Deepfake Detection(https://arxiv.org/abs/2401.06506)
• Keywords: self-supervised, generative
• Abstract: We study universal deepfake detection. Our goal is to detect synthetic images from a range of generative AI approaches, particularly from emerging ones which are unseen during training of the deepfake detector. Universal deepfake detection requires outstanding generalization capability. Motivated by recently proposed masked image modeling which has demonstrated excellent generalization in self-supervised pre-training, we make the first attempt to explore masked image modeling for universal deepfake detection. We study spatial and frequency domain masking in training deepfake detectors. Based on empirical analysis, we propose a novel deepfake detector via frequency masking. Our focus on frequency domain is different from the majority, which primarily target spatial domain detection. Our comparative analyses reveal substantial performance gains over existing methods. Code and models are publicly available.

Title: Utilizing Layout Effects for Analog Logic Locking

• Authors: Muayad J. Aljafar, Florence Azais, Marie-Lise Flottes, Samuel Pagliarini
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2401.06508
• Pdf URL: https://arxiv.org/pdf/2401.06508
• Copy Paste: [[2401.06508]] Utilizing Layout Effects for Analog Logic Locking(https://arxiv.org/abs/2401.06508)
• Keywords: diffusion
• Abstract: While numerous obfuscation techniques are available for securing digital assets in the digital domain, there has been a notable lack of focus on protecting Intellectual Property (IP) in the analog domain. This is primarily due to the relatively smaller footprint of analog components within an Integrated Circuit (IC), with the majority of the surface dedicated to digital elements. However, despite their smaller nature, analog components are highly valuable IP and warrant effective protection. In this paper, we present a groundbreaking method for safeguarding analog IP by harnessing layout-based effects that are typically considered undesirable in IC design. Specifically, we exploit the impact of Length of Oxide Diffusion and Well Proximity Effect on transistors to fine-tune critical parameters such as transconductance (gm) and threshold voltage (Vth). These parameters remain concealed behind key inputs, akin to the logic locking approach employed in digital ICs. Our research explores the application of layout-based effects in two commercial CMOS technologies, namely a 28nm and a 65nm node. To demonstrate the efficacy of our proposed technique, we implement it for locking an Operational Transconductance Amplifier. Extensive simulations are performed, evaluating the obfuscation strength by applying a large number of key sets (over 50,000 and 300,000). The results exhibit a significant degradation in performance metrics, such as open-loop gain (up to 130dB), phase margin (up to 50 degrees), 3dB bandwidth (approximately 2.5MHz), and power consumption (around 1mW) when incorrect keys are employed. Our findings highlight the advantages of our approach as well as the associated overhead.

Title: Exploring Diverse Representations for Open Set Recognition

• Authors: Yu Wang, Junxian Mu, Pengfei Zhu, Qinghua Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06521
• Pdf URL: https://arxiv.org/pdf/2401.06521
• Copy Paste: [[2401.06521]] Exploring Diverse Representations for Open Set Recognition(https://arxiv.org/abs/2401.06521)
• Keywords: generative
• Abstract: Open set recognition (OSR) requires the model to classify samples that belong to closed sets while rejecting unknown samples during test. Currently, generative models often perform better than discriminative models in OSR, but recent studies show that generative models may be computationally infeasible or unstable on complex tasks. In this paper, we provide insights into OSR and find that learning supplementary representations can theoretically reduce the open space risk. Based on the analysis, we propose a new model, namely Multi-Expert Diverse Attention Fusion (MEDAF), that learns diverse representations in a discriminative way. MEDAF consists of multiple experts that are learned with an attention diversity regularization term to ensure the attention maps are mutually different. The logits learned by each expert are adaptively fused and used to identify the unknowns through the score function. We show that the differences in attention maps can lead to diverse representations so that the fused representations can well handle the open space. Extensive experiments are conducted on standard and OSR large-scale benchmarks. Results show that the proposed discriminative method can outperform existing generative models by up to 9.5% on AUROC and achieve new state-of-the-art performance with little computational cost. Our method can also seamlessly integrate existing classification models. Code is available at https://github.com/Vanixxz/MEDAF.

Title: 360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model

• Authors: Qian Wang, Weiqi Li, Chong Mou, Xinhua Cheng, Jian Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06578
• Pdf URL: https://arxiv.org/pdf/2401.06578
• Copy Paste: [[2401.06578]] 360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model(https://arxiv.org/abs/2401.06578)
• Keywords: diffusion
• Abstract: 360-degree panoramic videos recently attract more interest in both studies and applications, courtesy of the heightened immersive experiences they engender. Due to the expensive cost of capturing 360-degree panoramic videos, generating desirable panoramic videos by given prompts is urgently required. Recently, the emerging text-to-video (T2V) diffusion methods demonstrate notable effectiveness in standard video generation. However, due to the significant gap in content and motion patterns between panoramic and standard videos, these methods encounter challenges in yielding satisfactory 360-degree panoramic videos. In this paper, we propose a controllable panorama video generation pipeline named 360-Degree Video Diffusion model (360DVD) for generating panoramic videos based on the given prompts and motion conditions. Concretely, we introduce a lightweight module dubbed 360-Adapter and assisted 360 Enhancement Techniques to transform pre-trained T2V models for 360-degree video generation. We further propose a new panorama dataset named WEB360 consisting of 360-degree video-text pairs for training 360DVD, addressing the absence of captioned panoramic video datasets. Extensive experiments demonstrate the superiority and effectiveness of 360DVD for panorama video generation. The code and dataset will be released soon.

Title: Every Node is Different: Dynamically Fusing Self-Supervised Tasks for Attributed Graph Clustering

• Authors: Pengfei Zhu, Qian Wang, Yu Wang, Jialu Li, Qinghua Hu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06595
• Pdf URL: https://arxiv.org/pdf/2401.06595
• Copy Paste: [[2401.06595]] Every Node is Different: Dynamically Fusing Self-Supervised Tasks for Attributed Graph Clustering(https://arxiv.org/abs/2401.06595)
• Keywords: self-supervised
• Abstract: Attributed graph clustering is an unsupervised task that partitions nodes into different groups. Self-supervised learning (SSL) shows great potential in handling this task, and some recent studies simultaneously learn multiple SSL tasks to further boost performance. Currently, different SSL tasks are assigned the same set of weights for all graph nodes. However, we observe that some graph nodes whose neighbors are in different groups require significantly different emphases on SSL tasks. In this paper, we propose to dynamically learn the weights of SSL tasks for different nodes and fuse the embeddings learned from different SSL tasks to boost performance. We design an innovative graph clustering approach, namely Dynamically Fusing Self-Supervised Learning (DyFSS). Specifically, DyFSS fuses features extracted from diverse SSL tasks using distinct weights derived from a gating network. To effectively learn the gating network, we design a dual-level self-supervised strategy that incorporates pseudo labels and the graph structure. Extensive experiments on five datasets show that DyFSS outperforms the state-of-the-art multi-task SSL methods by up to 8.66% on the accuracy metric. The code of DyFSS is available at: https://github.com/q086/DyFSS.

Title: Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking

• Authors: Wei Cao, Chang Luo, Biao Zhang, Matthias Nießner, Jiapeng Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06614
• Pdf URL: https://arxiv.org/pdf/2401.06614
• Copy Paste: [[2401.06614]] Motion2VecSets: 4D Latent Vector Set Diffusion for Non-rigid Shape Reconstruction and Tracking(https://arxiv.org/abs/2401.06614)
• Keywords: diffusion
• Abstract: We introduce Motion2VecSets, a 4D diffusion model for dynamic surface reconstruction from point cloud sequences. While existing state-of-the-art methods have demonstrated success in reconstructing non-rigid objects using neural field representations, conventional feed-forward networks encounter challenges with ambiguous observations from noisy, partial, or sparse point clouds. To address these challenges, we introduce a diffusion model that explicitly learns the shape and motion distribution of non-rigid objects through an iterative denoising process of compressed latent representations. The diffusion-based prior enables more plausible and probabilistic reconstructions when handling ambiguous inputs. We parameterize 4D dynamics with latent vector sets instead of using a global latent. This novel 4D representation allows us to learn local surface shape and deformation patterns, leading to more accurate non-linear motion capture and significantly improving generalizability to unseen motions and identities. For more temporal-coherent object tracking, we synchronously denoise deformation latent sets and exchange information across multiple frames. To avoid the computational overhead, we design an interleaved space and time attention block to alternately aggregate deformation latents along spatial and temporal domains. Extensive comparisons against the state-of-the-art methods demonstrate the superiority of our Motion2VecSets in 4D reconstruction from various imperfect observations, notably achieving a 19% improvement in Intersection over Union (IoU) compared to CaDex for reconstructing unseen individuals from sparse point clouds on the DeformingThings4D-Animals dataset. More detailed information can be found at https://vveicao.github.io/projects/Motion2VecSets/.

Title: Adversarial Examples are Misaligned in Diffusion Model Manifolds

• Authors: Peter Lorenz, Ricard Durall, Jansi Keuper
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2401.06637
• Pdf URL: https://arxiv.org/pdf/2401.06637
• Copy Paste: [[2401.06637]] Adversarial Examples are Misaligned in Diffusion Model Manifolds(https://arxiv.org/abs/2401.06637)
• Keywords: diffusion, generative
• Abstract: In recent years, diffusion models (DMs) have drawn significant attention for their success in approximating data distributions, yielding state-of-the-art generative results. Nevertheless, the versatility of these models extends beyond their generative capabilities to encompass various vision applications, such as image inpainting, segmentation, adversarial robustness, among others. This study is dedicated to the investigation of adversarial attacks through the lens of diffusion models. However, our objective does not involve enhancing the adversarial robustness of image classifiers. Instead, our focus lies in utilizing the diffusion model to detect and analyze the anomalies introduced by these attacks on images. To that end, we systematically examine the alignment of the distributions of adversarial examples when subjected to the process of transformation using diffusion models. The efficacy of this approach is assessed across CIFAR-10 and ImageNet datasets, including varying image sizes in the latter. The results demonstrate a notable capacity to discriminate effectively between benign and attacked images, providing compelling evidence that adversarial instances do not align with the learned manifold of the DMs.

Title: Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently

• Authors: Kanishka Misra, Allyson Ettinger, Kyle Mahowald
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06640
• Pdf URL: https://arxiv.org/pdf/2401.06640
• Copy Paste: [[2401.06640]] Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently(https://arxiv.org/abs/2401.06640)
• Keywords: in-context
• Abstract: Recent zero-shot evaluations have highlighted important limitations in the abilities of language models (LMs) to perform meaning extraction. However, it is now well known that LMs can demonstrate radical improvements in the presence of experimental contexts such as in-context examples and instructions. How well does this translate to previously studied meaning-sensitive tasks? We present a case-study on the extent to which experimental contexts can improve LMs' robustness in performing property inheritance -- predicting semantic properties of novel concepts, a task that they have been previously shown to fail on. Upon carefully controlling the nature of the in-context examples and the instructions, our work reveals that they can indeed lead to non-trivial property inheritance behavior in LMs. However, this ability is inconsistent: with a minimal reformulation of the task, some LMs were found to pick up on shallow, non-semantic heuristics from their inputs, suggesting that the computational principles of semantic property inference are yet to be mastered by LMs.

Title: Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation

• Authors: Jan Cegin, Branislav Pecher, Jakub Simko, Ivan Srba, Maria Bielikova, Peter Brusilovsky
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06643
• Pdf URL: https://arxiv.org/pdf/2401.06643
• Copy Paste: [[2401.06643]] Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation(https://arxiv.org/abs/2401.06643)
• Keywords: generative
• Abstract: The latest generative large language models (LLMs) have found their application in data augmentation tasks, where small numbers of text samples are LLM-paraphrased and then used to fine-tune the model. However, more research is needed to assess how different prompts, seed data selection strategies, filtering methods, or model settings affect the quality of paraphrased data (and downstream models). In this study, we investigate three text diversity incentive methods well established in crowdsourcing: taboo words, hints by previous outlier solutions, and chaining on previous outlier solutions. Using these incentive methods as part of instructions to LLMs augmenting text datasets, we measure their effects on generated texts' lexical diversity and downstream model performance. We compare the effects over 5 different LLMs and 6 datasets. We show that diversity is most increased by taboo words, while downstream model performance is highest when previously created paraphrases are used as hints.

Title: Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarks

• Authors: Stefan Blücher, Johanna Vielhaben, Nils Strodthoff
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06654
• Pdf URL: https://arxiv.org/pdf/2401.06654
• Copy Paste: [[2401.06654]] Decoupling Pixel Flipping and Occlusion Strategy for Consistent XAI Benchmarks(https://arxiv.org/abs/2401.06654)
• Keywords: diffusion
• Abstract: Feature removal is a central building block for eXplainable AI (XAI), both for occlusion-based explanations (Shapley values) as well as their evaluation (pixel flipping, PF). However, occlusion strategies can vary significantly from simple mean replacement up to inpainting with state-of-the-art diffusion models. This ambiguity limits the usefulness of occlusion-based approaches. For example, PF benchmarks lead to contradicting rankings. This is amplified by competing PF measures: Features are either removed starting with most influential first (MIF) or least influential first (LIF). This study proposes two complementary perspectives to resolve this disagreement problem. Firstly, we address the common criticism of occlusion-based XAI, that artificial samples lead to unreliable model evaluations. We propose to measure the reliability by the R(eference)-Out-of-Model-Scope (OMS) score. The R-OMS score enables a systematic comparison of occlusion strategies and resolves the disagreement problem by grouping consistent PF rankings. Secondly, we show that the insightfulness of MIF and LIF is conversely dependent on the R-OMS score. To leverage this, we combine the MIF and LIF measures into the symmetric relevance gain (SRG) measure. This breaks the inherent connection to the underlying occlusion strategy and leads to consistent rankings. This resolves the disagreement problem, which we verify for a set of 40 different occlusion strategies.

Title: An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models

• Authors: Gantavya Bhatt, Yifang Chen, Arnav M. Das, Jifan Zhang, Sang T. Truong, Stephen Mussmann, Yinglun Zhu, Jeffrey Bilmes, Simon S. Du, Kevin Jamieson, Jordan T. Ash, Robert D. Nowak
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06692
• Pdf URL: https://arxiv.org/pdf/2401.06692
• Copy Paste: [[2401.06692]] An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models(https://arxiv.org/abs/2401.06692)
• Keywords: generative
• Abstract: Supervised finetuning (SFT) on instruction datasets has played a crucial role in achieving the remarkable zero-shot generalization capabilities observed in modern large language models (LLMs). However, the annotation efforts required to produce high quality responses for instructions are becoming prohibitively expensive, especially as the number of tasks spanned by instruction datasets continues to increase. Active learning is effective in identifying useful subsets of samples to annotate from an unlabeled pool, but its high computational cost remains a barrier to its widespread applicability in the context of LLMs. To mitigate the annotation cost of SFT and circumvent the computational bottlenecks of active learning, we propose using experimental design. Experimental design techniques select the most informative samples to label, and typically maximize some notion of uncertainty and/or diversity. In our work, we implement a framework that evaluates several existing and novel experimental design techniques and find that these methods consistently yield significant gains in label efficiency with little computational overhead. On generative tasks, our methods achieve the same generalization performance with only $50\%$ of annotation cost required by random sampling.

Title: Quantum Machine Learning in the Cognitive Domain: Alzheimer's Disease Study

• Authors: Emine Akpinar
• Subjects: cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2401.06697
• Pdf URL: https://arxiv.org/pdf/2401.06697
• Copy Paste: [[2401.06697]] Quantum Machine Learning in the Cognitive Domain: Alzheimer's Disease Study(https://arxiv.org/abs/2401.06697)
• Keywords: generative
• Abstract: Alzheimer's disease (AD) is the most prevalent neurodegenerative brain disorder, which results in significant cognitive impairments, especially in the elderly population. Cognitive impairments can manifest as a decline in various mental faculties, such as concentration, memory, and other higher-order cognitive abilities. These deficits can significantly impact an individual's capacity to comprehend information, acquire new knowledge, and communicate effectively. One of the affected activities due to cognitive impairments is handwriting. By analyzing different aspects of handwriting, including pressure, velocity, and spatial organization, researchers can detect subtle alterations that might indicate early-stage cognitive impairments, especially AD. Recently, several classical artificial intelligence (AI) approaches have been proposed for detecting AD in elderly individuals through handwriting analysis. However, advanced AI methods require more computational power as the size of the data increases. Additionally, diagnoses can be influenced by factors such as limited relevant classical vector space and correlations between features. Recent studies have shown that using quantum computing technologies in healthcare can not only address these problems but also accelerate complex data analysis and process large datasets more efficiently. In this study, we introduced a variational quantum classifier with fewer circuit elements to facilitate the early diagnosis of AD in elderly individuals based on handwriting data. We employed ZZFeatureMap for encoding features. To classify AD, a parameterized quantum circuit consisting of repeated Ry and Rz rotation gates, as well as CY and CZ two-qubit entangling gates, was designed and implemented. The proposed model achieved an accuracy of 0.75 in classifying AD.

Title: The Unreasonable Effectiveness of Easy Training Data for Hard Tasks

• Authors: Peter Hase, Mohit Bansal, Peter Clark, Sarah Wiegreffe
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06751
• Pdf URL: https://arxiv.org/pdf/2401.06751
• Copy Paste: [[2401.06751]] The Unreasonable Effectiveness of Easy Training Data for Hard Tasks(https://arxiv.org/abs/2401.06751)
• Keywords: in-context
• Abstract: How can we train models to perform well on hard test data when hard training data is by definition difficult to label correctly? This question has been termed the scalable oversight problem and has drawn increasing attention as language models have continually improved. In this paper, we present the surprising conclusion that current language models often generalize relatively well from easy to hard data, even performing as well as "oracle" models trained on hard data. We demonstrate this kind of easy-to-hard generalization using simple training methods like in-context learning, linear classifier heads, and QLoRA for seven different measures of datapoint hardness, including six empirically diverse human hardness measures (like grade level) and one model-based measure (loss-based). Furthermore, we show that even if one cares most about model performance on hard data, it can be better to collect and train on easy data rather than hard data, since hard data is generally noisier and costlier to collect. Our experiments use open models up to 70b in size and four publicly available question-answering datasets with questions ranging in difficulty from 3rd grade science questions to college level STEM questions and general-knowledge trivia. We conclude that easy-to-hard generalization in LMs is surprisingly strong for the tasks studied, suggesting the scalable oversight problem may be easier than previously thought. Our code is available at https://github.com/allenai/easy-to-hard-generalization

Title: Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements

• Authors: Anton Voronov, Lena Wolf, Max Ryabinin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06766
• Pdf URL: https://arxiv.org/pdf/2401.06766
• Copy Paste: [[2401.06766]] Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements(https://arxiv.org/abs/2401.06766)
• Keywords: in-context
• Abstract: Large language models demonstrate a remarkable capability for learning to solve new tasks from a few examples. The prompt template, or the way the input examples are formatted to obtain the prompt, is an important yet often overlooked aspect of in-context learning. In this work, we conduct a comprehensive study of the template format's influence on the in-context learning performance. We evaluate the impact of the prompt template across models (from 770M to 70B parameters) and 4 standard classification datasets. We show that a poor choice of the template can reduce the performance of the strongest models and inference methods to a random guess level. More importantly, the best templates do not transfer between different setups and even between models of the same family. Our findings show that the currently prevalent approach to evaluation, which ignores template selection, may give misleading results due to different templates in different works. As a first step towards mitigating this issue, we propose Template Ensembles that aggregate model predictions across several templates. This simple test-time augmentation boosts average performance while being robust to the choice of random set of templates.