2024-03-08

Title: Three Revisits to Node-Level Graph Anomaly Detection: Outliers, Message Passing and Hyperbolic Neural Networks

• Authors: Jing Gu, Dongmian Zou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04010
• Pdf URL: https://arxiv.org/pdf/2403.04010
• Copy Paste: [[2403.04010]] Three Revisits to Node-Level Graph Anomaly Detection: Outliers, Message Passing and Hyperbolic Neural Networks(https://arxiv.org/abs/2403.04010)
• Keywords: anomaly
• Abstract: Graph anomaly detection plays a vital role for identifying abnormal instances in complex networks. Despite advancements of methodology based on deep learning in recent years, existing benchmarking approaches exhibit limitations that hinder a comprehensive comparison. In this paper, we revisit datasets and approaches for unsupervised node-level graph anomaly detection tasks from three aspects. Firstly, we introduce outlier injection methods that create more diverse and graph-based anomalies in graph datasets. Secondly, we compare methods employing message passing against those without, uncovering the unexpected decline in performance associated with message passing. Thirdly, we explore the use of hyperbolic neural networks, specifying crucial architecture and loss design that contribute to enhanced performance. Through rigorous experiments and evaluations, our study sheds light on general strategies for improving node-level graph anomaly detection methods.

Title: Unsupervised Contrastive Learning for Robust RF Device Fingerprinting Under Time-Domain Shift

• Authors: Jun Chen, Weng-Keen Wong, Bechir Hamdaoui
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.04036
• Pdf URL: https://arxiv.org/pdf/2403.04036
• Copy Paste: [[2403.04036]] Unsupervised Contrastive Learning for Robust RF Device Fingerprinting Under Time-Domain Shift(https://arxiv.org/abs/2403.04036)
• Keywords: self-supervised
• Abstract: Radio Frequency (RF) device fingerprinting has been recognized as a potential technology for enabling automated wireless device identification and classification. However, it faces a key challenge due to the domain shift that could arise from variations in the channel conditions and environmental settings, potentially degrading the accuracy of RF-based device classification when testing and training data is collected in different domains. This paper introduces a novel solution that leverages contrastive learning to mitigate this domain shift problem. Contrastive learning, a state-of-the-art self-supervised learning approach from deep learning, learns a distance metric such that positive pairs are closer (i.e. more similar) in the learned metric space than negative pairs. When applied to RF fingerprinting, our model treats RF signals from the same transmission as positive pairs and those from different transmissions as negative pairs. Through experiments on wireless and wired RF datasets collected over several days, we demonstrate that our contrastive learning approach captures domain-invariant features, diminishing the effects of domain-specific variations. Our results show large and consistent improvements in accuracy (10.8\% to 27.8\%) over baseline models, thus underscoring the effectiveness of contrastive learning in improving device classification under domain shift.

Title: Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations

• Authors: Xiaolin Sun, Zizhan Zheng
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04050
• Pdf URL: https://arxiv.org/pdf/2403.04050
• Copy Paste: [[2403.04050]] Belief-Enriched Pessimistic Q-Learning against Adversarial State Perturbations(https://arxiv.org/abs/2403.04050)
• Keywords: diffusion
• Abstract: Reinforcement learning (RL) has achieved phenomenal success in various domains. However, its data-driven nature also introduces new vulnerabilities that can be exploited by malicious opponents. Recent work shows that a well-trained RL agent can be easily manipulated by strategically perturbing its state observations at the test stage. Existing solutions either introduce a regularization term to improve the smoothness of the trained policy against perturbations or alternatively train the agent's policy and the attacker's policy. However, the former does not provide sufficient protection against strong attacks, while the latter is computationally prohibitive for large environments. In this work, we propose a new robust RL algorithm for deriving a pessimistic policy to safeguard against an agent's uncertainty about true states. This approach is further enhanced with belief state inference and diffusion-based state purification to reduce uncertainty. Empirical results show that our approach obtains superb performance under strong attacks and has a comparable training overhead with regularization-based methods. Our code is available at https://github.com/SliencerX/Belief-enriched-robust-Q-learning.

Title: LoDisc: Learning Global-Local Discriminative Features for Self-Supervised Fine-Grained Visual Recognition

• Authors: Jialu Shi, Zhiqiang Wei, Jie Nie, Lei Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04066
• Pdf URL: https://arxiv.org/pdf/2403.04066
• Copy Paste: [[2403.04066]] LoDisc: Learning Global-Local Discriminative Features for Self-Supervised Fine-Grained Visual Recognition(https://arxiv.org/abs/2403.04066)
• Keywords: self-supervised
• Abstract: Self-supervised contrastive learning strategy has attracted remarkable attention due to its exceptional ability in representation learning. However, current contrastive learning tends to learn global coarse-grained representations of the image that benefit generic object recognition, whereas such coarse-grained features are insufficient for fine-grained visual recognition. In this paper, we present to incorporate the subtle local fine-grained feature learning into global self-supervised contrastive learning through a pure self-supervised global-local fine-grained contrastive learning framework. Specifically, a novel pretext task called Local Discrimination (LoDisc) is proposed to explicitly supervise self-supervised model's focus towards local pivotal regions which are captured by a simple-but-effective location-wise mask sampling strategy. We show that Local Discrimination pretext task can effectively enhance fine-grained clues in important local regions, and the global-local framework further refines the fine-grained feature representations of images. Extensive experimental results on different fine-grained object recognition tasks demonstrate that the proposed method can lead to a decent improvement in different evaluation settings. Meanwhile, the proposed method is also effective in general object recognition tasks.

Title: Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection

• Authors: Jianfeng He, Hang Su, Jason Cai, Igor Shalyminov, Hwanjun Song, Saab Mansour
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04073
• Pdf URL: https://arxiv.org/pdf/2403.04073
• Copy Paste: [[2403.04073]] Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection(https://arxiv.org/abs/2403.04073)
• Keywords: generative
• Abstract: Semi-supervised dialogue summarization (SSDS) leverages model-generated summaries to reduce reliance on human-labeled data and improve the performance of summarization models. While addressing label noise, previous works on semi-supervised learning primarily focus on natural language understanding tasks, assuming each sample has a unique label. However, these methods are not directly applicable to SSDS, as it is a generative task, and each dialogue can be summarized in different ways. In this work, we propose a novel scoring approach, SiCF, which encapsulates three primary dimensions of summarization model quality: Semantic invariance (indicative of model confidence), Coverage (factual recall), and Faithfulness (factual precision). Using the SiCF score, we select unlabeled dialogues with high-quality generated summaries to train summarization models. Comprehensive experiments on three public datasets demonstrate the effectiveness of SiCF scores in uncertainty estimation and semi-supervised learning for dialogue summarization tasks. Our code is available at \url{https://github.com/amazon-science/summarization-sicf-score}.

Title: Scalable and Robust Transformer Decoders for Interpretable Image Classification with Foundation Models

• Authors: Evelyn Mannix, Howard Bondell
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04125
• Pdf URL: https://arxiv.org/pdf/2403.04125
• Copy Paste: [[2403.04125]] Scalable and Robust Transformer Decoders for Interpretable Image Classification with Foundation Models(https://arxiv.org/abs/2403.04125)
• Keywords: self-supervised, foundation model
• Abstract: Interpretable computer vision models can produce transparent predictions, where the features of an image are compared with prototypes from a training dataset and the similarity between them forms a basis for classification. Nevertheless these methods are computationally expensive to train, introduce additional complexity and may require domain knowledge to adapt hyper-parameters to a new dataset. Inspired by developments in object detection, segmentation and large-scale self-supervised foundation vision models, we introduce Component Features (ComFe), a novel explainable-by-design image classification approach using a transformer-decoder head and hierarchical mixture-modelling. With only global image labels and no segmentation or part annotations, ComFe can identify consistent image components, such as the head, body, wings and tail of a bird, and the image background, and determine which of these features are informative in making a prediction. We demonstrate that ComFe obtains higher accuracy compared to previous interpretable models across a range of fine-grained vision benchmarks, without the need to individually tune hyper-parameters for each dataset. We also show that ComFe outperforms a non-interpretable linear head across a range of datasets, including ImageNet, and improves performance on generalisation and robustness benchmarks.

Title: Dual-path Frequency Discriminators for Few-shot Anomaly Detection

• Authors: Yuhu Bai, Jiangning Zhang, Yuhang Dong, Guanzhong Tian, Yunkang Cao, Yabiao Wang, Chengjie Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04151
• Pdf URL: https://arxiv.org/pdf/2403.04151
• Copy Paste: [[2403.04151]] Dual-path Frequency Discriminators for Few-shot Anomaly Detection(https://arxiv.org/abs/2403.04151)
• Keywords: anomaly
• Abstract: Few-shot anomaly detection (FSAD) is essential in industrial manufacturing. However, existing FSAD methods struggle to effectively leverage a limited number of normal samples, and they may fail to detect and locate inconspicuous anomalies in the spatial domain. We further discover that these subtle anomalies would be more noticeable in the frequency domain. In this paper, we propose a Dual-Path Frequency Discriminators (DFD) network from a frequency perspective to tackle these issues. Specifically, we generate anomalies at both image-level and feature-level. Differential frequency components are extracted by the multi-frequency information construction module and supplied into the fine-grained feature construction module to provide adapted features. We consider anomaly detection as a discriminative classification problem, wherefore the dual-path feature discrimination module is employed to detect and locate the image-level and feature-level anomalies in the feature space. The discriminators aim to learn a joint representation of anomalous features and normal features in the latent space. Extensive experiments conducted on MVTec AD and VisA benchmarks demonstrate that our DFD surpasses current state-of-the-art methods. Source code will be available.

Title: Stabilizing Policy Gradients for Stochastic Differential Equations via Consistency with Perturbation Process

• Authors: Xiangxin Zhou, Liang Wang, Yichi Zhou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04154
• Pdf URL: https://arxiv.org/pdf/2403.04154
• Copy Paste: [[2403.04154]] Stabilizing Policy Gradients for Stochastic Differential Equations via Consistency with Perturbation Process(https://arxiv.org/abs/2403.04154)
• Keywords: generative
• Abstract: Considering generating samples with high rewards, we focus on optimizing deep neural networks parameterized stochastic differential equations (SDEs), the advanced generative models with high expressiveness, with policy gradient, the leading algorithm in reinforcement learning. Nevertheless, when applying policy gradients to SDEs, since the policy gradient is estimated on a finite set of trajectories, it can be ill-defined, and the policy behavior in data-scarce regions may be uncontrolled. This challenge compromises the stability of policy gradients and negatively impacts sample complexity. To address these issues, we propose constraining the SDE to be consistent with its associated perturbation process. Since the perturbation process covers the entire space and is easy to sample, we can mitigate the aforementioned problems. Our framework offers a general approach allowing for a versatile selection of policy gradient methods to effectively and efficiently train SDEs. We evaluate our algorithm on the task of structure-based drug design and optimize the binding affinity of generated ligand molecules. Our method achieves the best Vina score -9.07 on the CrossDocked2020 dataset.

Title: ProMISe: Promptable Medical Image Segmentation using SAM

• Authors: Jinfeng Wang, Sifan Song, Xinkun Wang, Yiyi Wang, Yiyi Miao, Jionglong Su, S. Kevin Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04164
• Pdf URL: https://arxiv.org/pdf/2403.04164
• Copy Paste: [[2403.04164]] ProMISe: Promptable Medical Image Segmentation using SAM(https://arxiv.org/abs/2403.04164)
• Keywords: foundation model
• Abstract: With the proposal of the Segment Anything Model (SAM), fine-tuning SAM for medical image segmentation (MIS) has become popular. However, due to the large size of the SAM model and the significant domain gap between natural and medical images, fine-tuning-based strategies are costly with potential risk of instability, feature damage and catastrophic forgetting. Furthermore, some methods of transferring SAM to a domain-specific MIS through fine-tuning strategies disable the model's prompting capability, severely limiting its utilization scenarios. In this paper, we propose an Auto-Prompting Module (APM), which provides SAM-based foundation model with Euclidean adaptive prompts in the target domain. Our experiments demonstrate that such adaptive prompts significantly improve SAM's non-fine-tuned performance in MIS. In addition, we propose a novel non-invasive method called Incremental Pattern Shifting (IPS) to adapt SAM to specific medical domains. Experimental results show that the IPS enables SAM to achieve state-of-the-art or competitive performance in MIS without the need for fine-tuning. By coupling these two methods, we propose ProMISe, an end-to-end non-fine-tuned framework for Promptable Medical Image Segmentation. Our experiments demonstrate that both using our methods individually or in combination achieves satisfactory performance in low-cost pattern shifting, with all of SAM's parameters frozen.

Title: Generative AI for Synthetic Data Generation: Methods, Challenges and the Future

• Authors: Xu Guo, Yiqiang Chen
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04190
• Pdf URL: https://arxiv.org/pdf/2403.04190
• Copy Paste: [[2403.04190]] Generative AI for Synthetic Data Generation: Methods, Challenges and the Future(https://arxiv.org/abs/2403.04190)
• Keywords: generative
• Abstract: The recent surge in research focused on generating synthetic data from large language models (LLMs), especially for scenarios with limited data availability, marks a notable shift in Generative Artificial Intelligence (AI). Their ability to perform comparably to real-world data positions this approach as a compelling solution to low-resource challenges. This paper delves into advanced technologies that leverage these gigantic LLMs for the generation of task-specific training data. We outline methodologies, evaluation techniques, and practical applications, discuss the current limitations, and suggest potential pathways for future research.

Title: Large Language Models are In-Context Molecule Learners

• Authors: Jiatong Li, Wei Liu, Zhihao Ding, Wenqi Fan, Yuqiang Li, Qing Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04197
• Pdf URL: https://arxiv.org/pdf/2403.04197
• Copy Paste: [[2403.04197]] Large Language Models are In-Context Molecule Learners(https://arxiv.org/abs/2403.04197)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) have demonstrated exceptional performance in biochemical tasks, especially the molecule caption translation task, which aims to bridge the gap between molecules and natural language texts. However, previous methods in adapting LLMs to the molecule-caption translation task required extra domain-specific pre-training stages, suffered weak alignment between molecular and textual spaces, or imposed stringent demands on the scale of LLMs. To resolve the challenges, we propose In-Context Molecule Adaptation (ICMA), as a new paradigm allowing LLMs to learn the molecule-text alignment from context examples via In-Context Molecule Tuning. Specifically, ICMA incorporates the following three stages: Cross-modal Retrieval, Post-retrieval Re-ranking, and In-context Molecule Tuning. Initially, Cross-modal Retrieval utilizes BM25 Caption Retrieval and Molecule Graph Retrieval to retrieve informative context examples. Additionally, we also propose Post-retrieval Re-ranking with Sequence Reversal and Random Walk to further improve the quality of retrieval results. Finally, In-Context Molecule Tuning unlocks the in-context molecule learning capability of LLMs with retrieved examples and adapts the parameters of LLMs for the molecule-caption translation task. Experimental results demonstrate that ICMT can empower LLMs to achieve state-of-the-art or comparable performance without extra training corpora and intricate structures, showing that LLMs are inherently in-context molecule learners.

Title: DEEP-ICL: Definition-Enriched Experts for Language Model In-Context Learning

• Authors: Xingwei Qu, Yiming Liang, Yucheng Wang, Tianyu Zheng, Tommy Yue, Lei Ma, Stephen W. Huang, Jiajun Zhang, Wenhu Chen, Chenghua Lin, Jie Fu, Ge Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04233
• Pdf URL: https://arxiv.org/pdf/2403.04233
• Copy Paste: [[2403.04233]] DEEP-ICL: Definition-Enriched Experts for Language Model In-Context Learning(https://arxiv.org/abs/2403.04233)
• Keywords: in-context
• Abstract: It has long been assumed that the sheer number of parameters in large language models (LLMs) drives in-context learning (ICL) capabilities, enabling remarkable performance improvements by leveraging task-specific demonstrations. Challenging this hypothesis, we introduce DEEP-ICL, a novel task Definition Enriched ExPert Ensembling methodology for ICL. DEEP-ICL explicitly extracts task definitions from given demonstrations and generates responses through learning task-specific examples. We argue that improvement from ICL does not directly rely on model size, but essentially stems from understanding task definitions and task-guided learning. Inspired by this, DEEP-ICL combines two 3B models with distinct roles (one for concluding task definitions and the other for learning task demonstrations) and achieves comparable performance to LLaMA2-13B. Furthermore, our framework outperforms conventional ICL by overcoming pretraining sequence length limitations, by supporting unlimited demonstrations. We contend that DEEP-ICL presents a novel alternative for achieving efficient few-shot learning, extending beyond the conventional ICL.

Title: Controllable Generation with Text-to-Image Diffusion Models: A Survey

• Authors: Pu Cao, Feng Zhou, Qing Song, Lu Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04279
• Pdf URL: https://arxiv.org/pdf/2403.04279
• Copy Paste: [[2403.04279]] Controllable Generation with Text-to-Image Diffusion Models: A Survey(https://arxiv.org/abs/2403.04279)
• Keywords: diffusion, generative
• Abstract: In the rapidly advancing realm of visual generation, diffusion models have revolutionized the landscape, marking a significant shift in capabilities with their impressive text-guided generative functions. However, relying solely on text for conditioning these models does not fully cater to the varied and complex requirements of different applications and scenarios. Acknowledging this shortfall, a variety of studies aim to control pre-trained text-to-image (T2I) models to support novel conditions. In this survey, we undertake a thorough review of the literature on controllable generation with T2I diffusion models, covering both the theoretical foundations and practical advancements in this domain. Our review begins with a brief introduction to the basics of denoising diffusion probabilistic models (DDPMs) and widely used T2I diffusion models. We then reveal the controlling mechanisms of diffusion models, theoretically analyzing how novel conditions are introduced into the denoising process for conditional generation. Additionally, we offer a detailed overview of research in this area, organizing it into distinct categories from the condition perspective: generation with specific conditions, generation with multiple conditions, and universal controllable generation. For an exhaustive list of the controllable generation literature surveyed, please refer to our curated repository at \url{https://github.com/PRIV-Creation/Awesome-Controllable-T2I-Diffusion-Models}.

Title: Effectiveness Assessment of Recent Large Vision-Language Models

• Authors: Yao Jiang, Xinyu Yan, Ge-Peng Ji, Keren Fu, Meijun Sun, Huan Xiong, Deng-Ping Fan, Fahad Shahbaz Khan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04306
• Pdf URL: https://arxiv.org/pdf/2403.04306
• Copy Paste: [[2403.04306]] Effectiveness Assessment of Recent Large Vision-Language Models(https://arxiv.org/abs/2403.04306)
• Keywords: anomaly
• Abstract: The advent of large vision-language models (LVLMs) represents a noteworthy advancement towards the pursuit of artificial general intelligence. However, the extent of their efficacy across both specialized and general tasks warrants further investigation. This article endeavors to evaluate the competency of popular LVLMs in specialized and general tasks, respectively, aiming to offer a comprehensive comprehension of these innovative methodologies. To gauge their efficacy in specialized tasks, we tailor a comprehensive testbed comprising three distinct scenarios: natural, healthcare, and industrial, encompassing six challenging tasks. These tasks include salient, camouflaged, and transparent object detection, as well as polyp and skin lesion detection, alongside industrial anomaly detection. We examine the performance of three recent open-source LVLMs -- MiniGPT-v2, LLaVA-1.5, and Shikra -- in the realm of visual recognition and localization. Moreover, we conduct empirical investigations utilizing the aforementioned models alongside GPT-4V, assessing their multi-modal understanding capacities in general tasks such as object counting, absurd question answering, affordance reasoning, attribute recognition, and spatial relation reasoning. Our investigations reveal that these models demonstrate limited proficiency not only in specialized tasks but also in general tasks. We delve deeper into this inadequacy and suggest several potential factors, including limited cognition in specialized tasks, object hallucination, text-to-image interference, and decreased robustness in complex problems. We hope this study would provide valuable insights for the future development of LVLMs, augmenting their power in coping with both general and specialized applications.

Title: Discriminative Probing and Tuning for Text-to-Image Generation

• Authors: Leigang Qu, Wenjie Wang, Yongqi Li, Hanwang Zhang, Liqiang Nie, Tat-Seng Chua
• Subjects: cs.CV, cs.AI, cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.04321
• Pdf URL: https://arxiv.org/pdf/2403.04321
• Copy Paste: [[2403.04321]] Discriminative Probing and Tuning for Text-to-Image Generation(https://arxiv.org/abs/2403.04321)
• Keywords: generative
• Abstract: Despite advancements in text-to-image generation (T2I), prior methods often face text-image misalignment problems such as relation confusion in generated images. Existing solutions involve cross-attention manipulation for better compositional understanding or integrating large language models for improved layout planning. However, the inherent alignment capabilities of T2I models are still inadequate. By reviewing the link between generative and discriminative modeling, we posit that T2I models' discriminative abilities may reflect their text-image alignment proficiency during generation. In this light, we advocate bolstering the discriminative abilities of T2I models to achieve more precise text-to-image alignment for generation. We present a discriminative adapter built on T2I models to probe their discriminative abilities on two representative tasks and leverage discriminative fine-tuning to improve their text-image alignment. As a bonus of the discriminative adapter, a self-correction mechanism can leverage discriminative gradients to better align generated images to text prompts during inference. Comprehensive evaluations across three benchmark datasets, including both in-distribution and out-of-distribution scenarios, demonstrate our method's superior generation performance. Meanwhile, it achieves state-of-the-art discriminative performance on the two discriminative tasks compared to other generative models.

Title: Video-Driven Animation of Neural Head Avatars

• Authors: Wolfgang Paier, Paul Hinzer, Anna Hilsmann, Peter Eisert
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04380
• Pdf URL: https://arxiv.org/pdf/2403.04380
• Copy Paste: [[2403.04380]] Video-Driven Animation of Neural Head Avatars(https://arxiv.org/abs/2403.04380)
• Keywords: generative
• Abstract: We present a new approach for video-driven animation of high-quality neural 3D head models, addressing the challenge of person-independent animation from video input. Typically, high-quality generative models are learned for specific individuals from multi-view video footage, resulting in person-specific latent representations that drive the generation process. In order to achieve person-independent animation from video input, we introduce an LSTM-based animation network capable of translating person-independent expression features into personalized animation parameters of person-specific 3D head models. Our approach combines the advantages of personalized head models (high quality and realism) with the convenience of video-driven animation employing multi-person facial performance capture. We demonstrate the effectiveness of our approach on synthesized animations with high quality based on different source videos as well as an ablation study.

Title: Exploring the Influence of Dimensionality Reduction on Anomaly Detection Performance in Multivariate Time Series

• Authors: Mahsun Altin, Altan Cakir
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04429
• Pdf URL: https://arxiv.org/pdf/2403.04429
• Copy Paste: [[2403.04429]] Exploring the Influence of Dimensionality Reduction on Anomaly Detection Performance in Multivariate Time Series(https://arxiv.org/abs/2403.04429)
• Keywords: anomaly
• Abstract: This paper presents an extensive empirical study on the integration of dimensionality reduction techniques with advanced unsupervised time series anomaly detection models, focusing on the MUTANT and Anomaly-Transformer models. The study involves a comprehensive evaluation across three different datasets: MSL, SMAP, and SWaT. Each dataset poses unique challenges, allowing for a robust assessment of the models' capabilities in varied contexts. The dimensionality reduction techniques examined include PCA, UMAP, Random Projection, and t-SNE, each offering distinct advantages in simplifying high-dimensional data. Our findings reveal that dimensionality reduction not only aids in reducing computational complexity but also significantly enhances anomaly detection performance in certain scenarios. Moreover, a remarkable reduction in training times was observed, with reductions by approximately 300\% and 650\% when dimensionality was halved and minimized to the lowest dimensions, respectively. This efficiency gain underscores the dual benefit of dimensionality reduction in both performance enhancement and operational efficiency. The MUTANT model exhibits notable adaptability, especially with UMAP reduction, while the Anomaly-Transformer demonstrates versatility across various reduction techniques. These insights provide a deeper understanding of the synergistic effects of dimensionality reduction and anomaly detection, contributing valuable perspectives to the field of time series analysis. The study underscores the importance of selecting appropriate dimensionality reduction strategies based on specific model requirements and dataset characteristics, paving the way for more efficient, accurate, and scalable solutions in anomaly detection.

Title: On-demand Quantization for Green Federated Generative Diffusion in Mobile Edge Networks

• Authors: Bingkun Lai, Jiayi He, Jiawen Kang, Gaolei Li, Minrui Xu, Tao zhang, Shengli Xie
• Subjects: cs.LG, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2403.04430
• Pdf URL: https://arxiv.org/pdf/2403.04430
• Copy Paste: [[2403.04430]] On-demand Quantization for Green Federated Generative Diffusion in Mobile Edge Networks(https://arxiv.org/abs/2403.04430)
• Keywords: diffusion, generative
• Abstract: Generative Artificial Intelligence (GAI) shows remarkable productivity and creativity in Mobile Edge Networks, such as the metaverse and the Industrial Internet of Things. Federated learning is a promising technique for effectively training GAI models in mobile edge networks due to its data distribution. However, there is a notable issue with communication consumption when training large GAI models like generative diffusion models in mobile edge networks. Additionally, the substantial energy consumption associated with training diffusion-based models, along with the limited resources of edge devices and complexities of network environments, pose challenges for improving the training efficiency of GAI models. To address this challenge, we propose an on-demand quantized energy-efficient federated diffusion approach for mobile edge networks. Specifically, we first design a dynamic quantized federated diffusion training scheme considering various demands from the edge devices. Then, we study an energy efficiency problem based on specific quantization requirements. Numerical results show that our proposed method significantly reduces system energy consumption and transmitted model size compared to both baseline federated diffusion and fixed quantized federated diffusion methods while effectively maintaining reasonable quality and diversity of generated data.

Title: StableDrag: Stable Dragging for Point-based Image Editing

• Authors: Yutao Cui, Xiaotong Zhao, Guozhen Zhang, Shengming Cao, Kai Ma, Limin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04437
• Pdf URL: https://arxiv.org/pdf/2403.04437
• Copy Paste: [[2403.04437]] StableDrag: Stable Dragging for Point-based Image Editing(https://arxiv.org/abs/2403.04437)
• Keywords: diffusion, generative
• Abstract: Point-based image editing has attracted remarkable attention since the emergence of DragGAN. Recently, DragDiffusion further pushes forward the generative quality via adapting this dragging technique to diffusion models. Despite these great success, this dragging scheme exhibits two major drawbacks, namely inaccurate point tracking and incomplete motion supervision, which may result in unsatisfactory dragging outcomes. To tackle these issues, we build a stable and precise drag-based editing framework, coined as StableDrag, by designing a discirminative point tracking method and a confidence-based latent enhancement strategy for motion supervision. The former allows us to precisely locate the updated handle points, thereby boosting the stability of long-range manipulation, while the latter is responsible for guaranteeing the optimized latent as high-quality as possible across all the manipulation steps. Thanks to these unique designs, we instantiate two types of image editing models including StableDrag-GAN and StableDrag-Diff, which attains more stable dragging performance, through extensive qualitative experiments and quantitative assessment on DragBench.

Title: Disentangled Diffusion-Based 3D Human Pose Estimation with Hierarchical Spatial and Temporal Denoiser

• Authors: Qingyuan Cai, Xuecai Hu, Saihui Hou, Li Yao, Yongzhen Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04444
• Pdf URL: https://arxiv.org/pdf/2403.04444
• Copy Paste: [[2403.04444]] Disentangled Diffusion-Based 3D Human Pose Estimation with Hierarchical Spatial and Temporal Denoiser(https://arxiv.org/abs/2403.04444)
• Keywords: diffusion
• Abstract: Recently, diffusion-based methods for monocular 3D human pose estimation have achieved state-of-the-art (SOTA) performance by directly regressing the 3D joint coordinates from the 2D pose sequence. Although some methods decompose the task into bone length and bone direction prediction based on the human anatomical skeleton to explicitly incorporate more human body prior constraints, the performance of these methods is significantly lower than that of the SOTA diffusion-based methods. This can be attributed to the tree structure of the human skeleton. Direct application of the disentangled method could amplify the accumulation of hierarchical errors, propagating through each hierarchy. Meanwhile, the hierarchical information has not been fully explored by the previous methods. To address these problems, a Disentangled Diffusion-based 3D Human Pose Estimation method with Hierarchical Spatial and Temporal Denoiser is proposed, termed DDHPose. In our approach: (1) We disentangle the 3D pose and diffuse the bone length and bone direction during the forward process of the diffusion model to effectively model the human pose prior. A disentanglement loss is proposed to supervise diffusion model learning. (2) For the reverse process, we propose Hierarchical Spatial and Temporal Denoiser (HSTDenoiser) to improve the hierarchical modeling of each joint. Our HSTDenoiser comprises two components: the Hierarchical-Related Spatial Transformer (HRST) and the Hierarchical-Related Temporal Transformer (HRTT). HRST exploits joint spatial information and the influence of the parent joint on each joint for spatial modeling, while HRTT utilizes information from both the joint and its hierarchical adjacent joints to explore the hierarchical temporal correlations among joints.

Title: Membership Inference Attacks and Privacy in Topic Modeling

• Authors: Nico Manzonelli, Wanrong Zhang, Salil Vadhan
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04451
• Pdf URL: https://arxiv.org/pdf/2403.04451
• Copy Paste: [[2403.04451]] Membership Inference Attacks and Privacy in Topic Modeling(https://arxiv.org/abs/2403.04451)
• Keywords: generative
• Abstract: Recent research shows that large language models are susceptible to privacy attacks that infer aspects of the training data. However, it is unclear if simpler generative models, like topic models, share similar vulnerabilities. In this work, we propose an attack against topic models that can confidently identify members of the training data in Latent Dirichlet Allocation. Our results suggest that the privacy risks associated with generative modeling are not restricted to large neural models. Additionally, to mitigate these vulnerabilities, we explore differentially private (DP) topic modeling. We propose a framework for private topic modeling that incorporates DP vocabulary selection as a pre-processing step, and show that it improves privacy while having limited effects on practical utility.

Title: Do Large Language Model Understand Multi-Intent Spoken Language ?

• Authors: Shangjian Yin, Peijie Huang, Yuhong Xu, Haojing Huang, Jiatian Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04481
• Pdf URL: https://arxiv.org/pdf/2403.04481
• Copy Paste: [[2403.04481]] Do Large Language Model Understand Multi-Intent Spoken Language ?(https://arxiv.org/abs/2403.04481)
• Keywords: generative
• Abstract: This study marks a significant advancement by harnessing Large Language Models (LLMs) for multi-intent spoken language understanding (SLU), proposing a unique methodology that capitalizes on the generative power of LLMs within an SLU context. Our innovative technique reconfigures entity slots specifically for LLM application in multi-intent SLU environments and introduces the concept of Sub-Intent Instruction (SII), enhancing the dissection and interpretation of intricate, multi-intent communication within varied domains. The resultant datasets, dubbed LM-MixATIS and LM-MixSNIPS, are crafted from pre-existing benchmarks. Our research illustrates that LLMs can match and potentially excel beyond the capabilities of current state-of-the-art multi-intent SLU models. It further explores LLM efficacy across various intent configurations and dataset proportions. Moreover, we introduce two pioneering metrics, Entity Slot Accuracy (ESA) and Combined Semantic Accuracy (CSA), to provide an in-depth analysis of LLM proficiency in this complex field.

Title: What makes an image realistic?

• Authors: Lucas Theis
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.04493
• Pdf URL: https://arxiv.org/pdf/2403.04493
• Copy Paste: [[2403.04493]] What makes an image realistic?(https://arxiv.org/abs/2403.04493)
• Keywords: generative
• Abstract: The last decade has seen tremendous progress in our ability to generate realistic-looking data, be it images, text, audio, or video. Here, we discuss the closely related problem of quantifying realism, that is, designing functions that can reliably tell realistic data from unrealistic data. This problem turns out to be significantly harder to solve and remains poorly understood, despite its prevalence in machine learning and recent breakthroughs in generative AI. Drawing on insights from algorithmic information theory, we discuss why this problem is challenging, why a good generative model alone is insufficient to solve it, and what a good solution would look like. In particular, we introduce the notion of a universal critic, which unlike adversarial critics does not require adversarial training. While universal critics are not immediately practical, they can serve both as a North Star for guiding practical implementations and as a tool for analyzing existing attempts to capture realism.

Title: Where does In-context Translation Happen in Large Language Models

• Authors: Suzanna Sia, David Mueller, Kevin Duh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04510
• Pdf URL: https://arxiv.org/pdf/2403.04510
• Copy Paste: [[2403.04510]] Where does In-context Translation Happen in Large Language Models(https://arxiv.org/abs/2403.04510)
• Keywords: self-supervised, in-context
• Abstract: Self-supervised large language models have demonstrated the ability to perform Machine Translation (MT) via in-context learning, but little is known about where the model performs the task with respect to prompt instructions and demonstration examples. In this work, we attempt to characterize the region where large language models transition from in-context learners to translation models. Through a series of layer-wise context-masking experiments on \textsc{GPTNeo2.7B}, \textsc{Bloom3B}, \textsc{Llama7b} and \textsc{Llama7b-chat}, we demonstrate evidence of a "task recognition" point where the translation task is encoded into the input representations and attention to context is no longer necessary. We further observe correspondence between the low performance when masking out entire layers, and the task recognition layers. Taking advantage of this redundancy results in 45\% computational savings when prompting with 5 examples, and task recognition achieved at layer 14 / 32. Our layer-wise fine-tuning experiments indicate that the most effective layers for MT fine-tuning are the layers critical to task recognition.

Title: Enhancing Data Quality in Federated Fine-Tuning of Foundation Models

• Authors: Wanru Zhao, Yaxin Du, Nicholas Donald Lane, Siheng Chen, Yanfeng Wang
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.04529
• Pdf URL: https://arxiv.org/pdf/2403.04529
• Copy Paste: [[2403.04529]] Enhancing Data Quality in Federated Fine-Tuning of Foundation Models(https://arxiv.org/abs/2403.04529)
• Keywords: foundation model
• Abstract: In the current landscape of foundation model training, there is a significant reliance on public domain data, which is nearing exhaustion according to recent research. To further scale up, it is crucial to incorporate collaboration among multiple specialized and high-quality private domain data sources. However, the challenge of training models locally without sharing private data presents numerous obstacles in data quality control. To tackle this issue, we propose a data quality control pipeline for federated fine-tuning of foundation models. This pipeline computes scores reflecting the quality of training data and determines a global threshold for a unified standard, aiming for improved global performance. Our experiments show that the proposed quality control pipeline facilitates the effectiveness and reliability of the model training, leading to better performance.

Title: Reducing self-supervised learning complexity improves weakly-supervised classification performance in computational pathology

• Authors: Tim Lenz, Omar S. M. El Nahhas, Marta Ligero, Jakob Nikolas Kather
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04558
• Pdf URL: https://arxiv.org/pdf/2403.04558
• Copy Paste: [[2403.04558]] Reducing self-supervised learning complexity improves weakly-supervised classification performance in computational pathology(https://arxiv.org/abs/2403.04558)
• Keywords: self-supervised, foundation model
• Abstract: Deep Learning models have been successfully utilized to extract clinically actionable insights from routinely available histology data. Generally, these models require annotations performed by clinicians, which are scarce and costly to generate. The emergence of self-supervised learning (SSL) methods remove this barrier, allowing for large-scale analyses on non-annotated data. However, recent SSL approaches apply increasingly expansive model architectures and larger datasets, causing the rapid escalation of data volumes, hardware prerequisites, and overall expenses, limiting access to these resources to few institutions. Therefore, we investigated the complexity of contrastive SSL in computational pathology in relation to classification performance with the utilization of consumer-grade hardware. Specifically, we analyzed the effects of adaptations in data volume, architecture, and algorithms on downstream clas- sification tasks, emphasizing their impact on computational resources. We trained breast cancer foundation models on a large public patient cohort and validated them on various downstream classification tasks in a weakly supervised manner on two external public patient cohorts. Our experiments demonstrate that we can improve downstream classification performance whilst reducing SSL training duration by 90%. In summary, we propose a set of adaptations which enable the utilization of SSL in computational pathology in non-resource abundant environments.

Title: Pix2Gif: Motion-Guided Diffusion for GIF Generation

• Authors: Hitesh Kandala, Jianfeng Gao, Jianwei Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04634
• Pdf URL: https://arxiv.org/pdf/2403.04634
• Copy Paste: [[2403.04634]] Pix2Gif: Motion-Guided Diffusion for GIF Generation(https://arxiv.org/abs/2403.04634)
• Keywords: diffusion
• Abstract: We present Pix2Gif, a motion-guided diffusion model for image-to-GIF (video) generation. We tackle this problem differently by formulating the task as an image translation problem steered by text and motion magnitude prompts, as shown in teaser fig. To ensure that the model adheres to motion guidance, we propose a new motion-guided warping module to spatially transform the features of the source image conditioned on the two types of prompts. Furthermore, we introduce a perceptual loss to ensure the transformed feature map remains within the same space as the target image, ensuring content consistency and coherence. In preparation for the model training, we meticulously curated data by extracting coherent image frames from the TGIF video-caption dataset, which provides rich information about the temporal changes of subjects. After pretraining, we apply our model in a zero-shot manner to a number of video datasets. Extensive qualitative and quantitative experiments demonstrate the effectiveness of our model -- it not only captures the semantic prompt from text but also the spatial ones from motion guidance. We train all our models using a single node of 16xV100 GPUs. Code, dataset and models are made public at: https://hiteshk03.github.io/Pix2Gif/.

Title: Context-Based Multimodal Fusion

• Authors: Bilal Faye, Hanane Azzag, Mustapha Lebbah, Djamel Bouchaffra
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04650
• Pdf URL: https://arxiv.org/pdf/2403.04650
• Copy Paste: [[2403.04650]] Context-Based Multimodal Fusion(https://arxiv.org/abs/2403.04650)
• Keywords: self-supervised
• Abstract: The fusion models, which effectively combine information from different sources, are widely used in solving multimodal tasks. However, they have significant limitations related to aligning data distributions across different modalities. This challenge can lead to inconsistencies and difficulties in learning robust representations. Alignment models, while specifically addressing this issue, often require training "from scratch" with large datasets to achieve optimal results, which can be costly in terms of resources and time. To overcome these limitations, we propose an innovative model called Context-Based Multimodal Fusion (CBMF), which combines both modality fusion and data distribution alignment. In CBMF, each modality is represented by a specific context vector, fused with the embedding of each modality. This enables the use of large pre-trained models that can be frozen, reducing the computational and training data requirements. Additionally, the network learns to differentiate embeddings of different modalities through fusion with context and aligns data distributions using a contrastive approach for self-supervised learning. Thus, CBMF offers an effective and economical solution for solving complex multimodal tasks.

Title: Yi: Open Foundation Models by 01.AI

• Authors: 01.AI: Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, Zonghong Dai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04652
• Pdf URL: https://arxiv.org/pdf/2403.04652
• Copy Paste: [[2403.04652]] Yi: Open Foundation Models by 01.AI(https://arxiv.org/abs/2403.04652)
• Keywords: foundation model
• Abstract: We introduce the Yi model family, a series of language and multimodal models that demonstrate strong multi-dimensional capabilities. The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models. Our base models achieve strong performance on a wide range of benchmarks like MMLU, and our finetuned chat models deliver strong human preference rate on major evaluation platforms like AlpacaEval and Chatbot Arena. Building upon our scalable super-computing infrastructure and the classical transformer architecture, we attribute the performance of Yi models primarily to its data quality resulting from our data-engineering efforts. For pretraining, we construct 3.1 trillion tokens of English and Chinese corpora using a cascaded data deduplication and quality filtering pipeline. For finetuning, we polish a small scale (less than 10K) instruction dataset over multiple iterations such that every single instance has been verified directly by our machine learning engineers. For vision-language, we combine the chat language model with a vision transformer encoder and train the model to align visual representations to the semantic space of the language model. We further extend the context length to 200K through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. We show that extending the depth of the pretrained checkpoint through continual pretraining further improves performance. We believe that given our current results, continuing to scale up model parameters using thoroughly optimized data will lead to even stronger frontier models.

Title: Chain of Thought Explanation for Dialogue State Tracking

• Authors: Lin Xu, Ningxin Peng, Daquan Zhou, See-Kiong Ng, Jinlan Fu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.04656
• Pdf URL: https://arxiv.org/pdf/2403.04656
• Copy Paste: [[2403.04656]] Chain of Thought Explanation for Dialogue State Tracking(https://arxiv.org/abs/2403.04656)
• Keywords: generative
• Abstract: Dialogue state tracking (DST) aims to record user queries and goals during a conversational interaction achieved by maintaining a prede- fined set of slots and their corresponding values. Current approaches decide slot values opaquely, while humans usually adopt a more deliberate approach by collecting information from relevant dialogue turns and then reasoning the appropriate values. In this work, we focus on the steps needed to figure out slot values by proposing a model named Chain-of-Thought-Explanation (CoTE) for the DST task. CoTE, which is built on the generative DST framework, is designed to create detailed explanations step by step after determining the slot values. This process leads to more accurate and reliable slot values. More-over, to improve the reasoning ability of the CoTE, we further construct more fluent and high-quality explanations with automatic paraphrasing, leading the method CoTE-refined. Experimental results on three widely recognized DST benchmarks-MultiWOZ 2.2, WoZ 2.0, and M2M-demonstrate the remarkable effectiveness of the CoTE. Furthermore, through a meticulous fine-grained analysis, we observe significant benefits of our CoTE on samples characterized by longer dialogue turns, user responses, and reasoning steps.

Title: PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation

• Authors: Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, Zhenguo Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04692
• Pdf URL: https://arxiv.org/pdf/2403.04692
• Copy Paste: [[2403.04692]] PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation(https://arxiv.org/abs/2403.04692)
• Keywords: diffusion
• Abstract: In this paper, we introduce PixArt-\Sigma, a Diffusion Transformer model~(DiT) capable of directly generating images at 4K resolution. PixArt-\Sigma represents a significant advancement over its predecessor, PixArt-\alpha, offering images of markedly higher fidelity and improved alignment with text prompts. A key feature of PixArt-\Sigma is its training efficiency. Leveraging the foundational pre-training of PixArt-\alpha, it evolves from the `weaker' baseline to a `stronger' model via incorporating higher quality data, a process we term "weak-to-strong training". The advancements in PixArt-\Sigma are twofold: (1) High-Quality Training Data: PixArt-\Sigma incorporates superior-quality image data, paired with more precise and detailed image captions. (2) Efficient Token Compression: we propose a novel attention module within the DiT framework that compresses both keys and values, significantly improving efficiency and facilitating ultra-high-resolution image generation. Thanks to these improvements, PixArt-\Sigma achieves superior image quality and user prompt adherence capabilities with significantly smaller model size (0.6B parameters) than existing text-to-image diffusion models, such as SDXL (2.6B parameters) and SD Cascade (5.1B parameters). Moreover, PixArt-\Sigma's capability to generate 4K images supports the creation of high-resolution posters and wallpapers, efficiently bolstering the production of high-quality visual content in industries such as film and gaming.

Title: Delving into the Trajectory Long-tail Distribution for Muti-object Tracking

• Authors: Sijia Chen, En Yu, Jinyang Li, Wenbing Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04700
• Pdf URL: https://arxiv.org/pdf/2403.04700
• Copy Paste: [[2403.04700]] Delving into the Trajectory Long-tail Distribution for Muti-object Tracking(https://arxiv.org/abs/2403.04700)
• Keywords: diffusion
• Abstract: Multiple Object Tracking (MOT) is a critical area within computer vision, with a broad spectrum of practical implementations. Current research has primarily focused on the development of tracking algorithms and enhancement of post-processing techniques. Yet, there has been a lack of thorough examination concerning the nature of tracking data it self. In this study, we pioneer an exploration into the distribution patterns of tracking data and identify a pronounced long-tail distribution issue within existing MOT datasets. We note a significant imbalance in the distribution of trajectory lengths across different pedestrians, a phenomenon we refer to as "pedestrians trajectory long-tail distribution". Addressing this challenge, we introduce a bespoke strategy designed to mitigate the effects of this skewed distribution. Specifically, we propose two data augmentation strategies, including Stationary Camera View Data Augmentation (SVA) and Dynamic Camera View Data Augmentation (DVA) , designed for viewpoint states and the Group Softmax (GS) module for Re-ID. SVA is to backtrack and predict the pedestrian trajectory of tail classes, and DVA is to use diffusion model to change the background of the scene. GS divides the pedestrians into unrelated groups and performs softmax operation on each group individually. Our proposed strategies can be integrated into numerous existing tracking systems, and extensive experimentation validates the efficacy of our method in reducing the influence of long-tail distribution on multi-object tracking performance. The code is available at https://github.com/chen-si-jia/Trajectory-Long-tail-Distribution-for-MOT.

Title: ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changes

• Authors: Hashmat Shadab Malik, Muhammad Huzaifa, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.04701
• Pdf URL: https://arxiv.org/pdf/2403.04701
• Copy Paste: [[2403.04701]] ObjectCompose: Evaluating Resilience of Vision-Based Models on Object-to-Background Compositional Changes(https://arxiv.org/abs/2403.04701)
• Keywords: diffusion, generative
• Abstract: Given the large-scale multi-modal training of recent vision-based models and their generalization capabilities, understanding the extent of their robustness is critical for their real-world deployment. In this work, we evaluate the resilience of current vision-based models against diverse object-to-background context variations. The majority of robustness evaluation methods have introduced synthetic datasets to induce changes to object characteristics (viewpoints, scale, color) or utilized image transformation techniques (adversarial changes, common corruptions) on real images to simulate shifts in distributions. Recent works have explored leveraging large language models and diffusion models to generate changes in the background. However, these methods either lack in offering control over the changes to be made or distort the object semantics, making them unsuitable for the task. Our method, on the other hand, can induce diverse object-to-background changes while preserving the original semantics and appearance of the object. To achieve this goal, we harness the generative capabilities of text-to-image, image-to-text, and image-to-segment models to automatically generate a broad spectrum of object-to-background changes. We induce both natural and adversarial background changes by either modifying the textual prompts or optimizing the latents and textual embedding of text-to-image models. This allows us to quantify the role of background context in understanding the robustness and generalization of deep neural networks. We produce various versions of standard vision datasets (ImageNet, COCO), incorporating either diverse and realistic backgrounds into the images or introducing color, texture, and adversarial changes in the background. We conduct extensive experiment to analyze the robustness of vision-based models against object-to-background context variations across diverse tasks.

Title: Masked Capsule Autoencoders

• Authors: Miles Everett, Mingjun Zhong, Georgios Leontidis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.04724
• Pdf URL: https://arxiv.org/pdf/2403.04724
• Copy Paste: [[2403.04724]] Masked Capsule Autoencoders(https://arxiv.org/abs/2403.04724)
• Keywords: self-supervised
• Abstract: We propose Masked Capsule Autoencoders (MCAE), the first Capsule Network that utilises pretraining in a self-supervised manner. Capsule Networks have emerged as a powerful alternative to Convolutional Neural Networks (CNNs), and have shown favourable properties when compared to Vision Transformers (ViT), but have struggled to effectively learn when presented with more complex data, leading to Capsule Network models that do not scale to modern tasks. Our proposed MCAE model alleviates this issue by reformulating the Capsule Network to use masked image modelling as a pretraining stage before finetuning in a supervised manner. Across several experiments and ablations studies we demonstrate that similarly to CNNs and ViTs, Capsule Networks can also benefit from self-supervised pretraining, paving the way for further advancements in this neural network domain. For instance, pretraining on the Imagenette dataset, a dataset of 10 classes of Imagenet-sized images, we achieve not only state-of-the-art results for Capsule Networks but also a 9% improvement compared to purely supervised training. Thus we propose that Capsule Networks benefit from and should be trained within a masked image modelling framework, with a novel capsule decoder, to improve a Capsule Network's performance on realistic-sized images.

Title: LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error

• Authors: Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, Yu Su
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.04746
• Pdf URL: https://arxiv.org/pdf/2403.04746
• Copy Paste: [[2403.04746]] LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error(https://arxiv.org/abs/2403.04746)
• Keywords: in-context
• Abstract: Tools are essential for large language models (LLMs) to acquire up-to-date information and take consequential actions in external environments. Existing work on tool-augmented LLMs primarily focuses on the broad coverage of tools and the flexibility of adding new tools. However, a critical aspect that has surprisingly been understudied is simply how accurately an LLM uses tools for which it has been trained. We find that existing LLMs, including GPT-4 and open-source LLMs specifically fine-tuned for tool use, only reach a correctness rate in the range of 30% to 60%, far from reliable use in practice. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE), that orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory. Specifically, STE leverages an LLM's 'imagination' to simulate plausible scenarios for using a tool, after which the LLM interacts with the tool to learn from its execution feedback. Both short-term and long-term memory are employed to improve the depth and breadth of the exploration, respectively. Comprehensive experiments on ToolBench show that STE substantially improves tool learning for LLMs under both in-context learning and fine-tuning settings, bringing a boost of 46.7% to Mistral-Instruct-7B and enabling it to outperform GPT-4. We also show effective continual learning of tools via a simple experience replay strategy.