diffusion

Title: Patch-based Selection and Refinement for Early Object Detection. (arXiv:2311.02274v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.02274
• Code URL: null
• Copy Paste: [[2311.02274]] Patch-based Selection and Refinement for Early Object Detection(http://arxiv.org/abs/2311.02274)
• Summary:

  Early object detection (OD) is a crucial task for the safety of many dynamic systems. Current OD algorithms have limited success for small objects at a long distance. To improve the accuracy and efficiency of such a task, we propose a novel set of algorithms that divide the image into patches, select patches with objects at various scales, elaborate the details of a small object, and detect it as early as possible. Our approach is built upon a transformer-based network and integrates the diffusion model to improve the detection accuracy. As demonstrated on BDD100K, our algorithms enhance the mAP for small objects from 1.03 to 8.93, and reduce the data volume in computation by more than 77\%. The source code is available at \href{https://github.com/destiny301/dpr}{https://github.com/destiny301/dpr}

Title: Sparse Training of Discrete Diffusion Models for Graph Generation. (arXiv:2311.02142v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.02142
• Code URL: https://github.com/qym7/sparsediff
• Copy Paste: [[2311.02142]] Sparse Training of Discrete Diffusion Models for Graph Generation(http://arxiv.org/abs/2311.02142)
• Summary:

  Generative models for graphs often encounter scalability challenges due to the inherent need to predict interactions for every node pair. Despite the sparsity often exhibited by real-world graphs, the unpredictable sparsity patterns of their adjacency matrices, stemming from their unordered nature, leads to quadratic computational complexity. In this work, we introduce SparseDiff, a denoising diffusion model for graph generation that is able to exploit sparsity during its training phase. At the core of SparseDiff is a message-passing neural network tailored to predict only a subset of edges during each forward pass. When combined with a sparsity-preserving noise model, this model can efficiently work with edge lists representations of graphs, paving the way for scalability to much larger structures. During the sampling phase, SparseDiff iteratively populates the adjacency matrix from its prior state, ensuring prediction of the full graph while controlling memory utilization. Experimental results show that SparseDiff simultaneously matches state-of-the-art in generation performance on both small and large graphs, highlighting the versatility of our method.

self-supervised

Title: Self-Supervised Learning of Representations for Space Generates Multi-Modular Grid Cells. (arXiv:2311.02316v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.02316
• Code URL: null
• Copy Paste: [[2311.02316]] Self-Supervised Learning of Representations for Space Generates Multi-Modular Grid Cells(http://arxiv.org/abs/2311.02316)
• Summary:

  To solve the spatial problems of mapping, localization and navigation, the mammalian lineage has developed striking spatial representations. One important spatial representation is the Nobel-prize winning grid cells: neurons that represent self-location, a local and aperiodic quantity, with seemingly bizarre non-local and spatially periodic activity patterns of a few discrete periods. Why has the mammalian lineage learnt this peculiar grid representation? Mathematical analysis suggests that this multi-periodic representation has excellent properties as an algebraic code with high capacity and intrinsic error-correction, but to date, there is no satisfactory synthesis of core principles that lead to multi-modular grid cells in deep recurrent neural networks. In this work, we begin by identifying key insights from four families of approaches to answering the grid cell question: coding theory, dynamical systems, function optimization and supervised deep learning. We then leverage our insights to propose a new approach that combines the strengths of all four approaches. Our approach is a self-supervised learning (SSL) framework - including data, data augmentations, loss functions and a network architecture - motivated from a normative perspective, without access to supervised position information or engineering of particular readout representations as needed in previous approaches. We show that multiple grid cell modules can emerge in networks trained on our SSL framework and that the networks and emergent representations generalize well outside their training distribution. This work contains insights for neuroscientists interested in the origins of grid cells as well as machine learning researchers interested in novel SSL frameworks.

foundation model

Title: Robust Fine-Tuning of Vision-Language Models for Domain Generalization. (arXiv:2311.02236v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.02236
• Code URL: https://github.com/mit-ll/robust-vision-language-finetuning
• Copy Paste: [[2311.02236]] Robust Fine-Tuning of Vision-Language Models for Domain Generalization(http://arxiv.org/abs/2311.02236)
• Summary:

  Transfer learning enables the sharing of common knowledge among models for a variety of downstream tasks, but traditional methods suffer in limited training data settings and produce narrow models incapable of effectively generalizing under distribution shifts. Foundation models have recently demonstrated impressive zero-shot inference capabilities and robustness under distribution shifts. However, zero-shot evaluation for these models has been predominantly confined to benchmarks with simple distribution shifts, limiting our understanding of their effectiveness under the more realistic shifts found in practice. Moreover, common fine-tuning methods for these models have yet to be evaluated against vision models in few-shot scenarios where training data is limited. To address these gaps, we present a new recipe for few-shot fine-tuning of the popular vision-language foundation model CLIP and evaluate its performance on challenging benchmark datasets with realistic distribution shifts from the WILDS collection. Our experimentation demonstrates that, while zero-shot CLIP fails to match performance of trained vision models on more complex benchmarks, few-shot CLIP fine-tuning outperforms its vision-only counterparts in terms of in-distribution and out-of-distribution accuracy at all levels of training data availability. This provides a strong incentive for adoption of foundation models within few-shot learning applications operating with real-world data. Code is available at https://github.com/mit-ll/robust-vision-language-finetuning

generative

Title: Generative Artificial Intelligence in Healthcare: Ethical Considerations and Assessment Checklist. (arXiv:2311.02107v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.02107
• Code URL: null
• Copy Paste: [[2311.02107]] Generative Artificial Intelligence in Healthcare: Ethical Considerations and Assessment Checklist(http://arxiv.org/abs/2311.02107)
• Summary:

  The widespread use of ChatGPT and other emerging technology powered by generative artificial intelligence (AI) has drawn much attention to potential ethical issues, especially in high-stakes applications such as healthcare. However, less clear is how to resolve such issues beyond following guidelines and regulations that are still under discussion and development. On the other hand, other types of generative AI have been used to synthesize images and other types of data for research and practical purposes, which have resolved some ethical issues and exposed other ethical issues, but such technology is less often the focus of ongoing ethical discussions. Here we highlight gaps in current ethical discussions of generative AI via a systematic scoping review of relevant existing research in healthcare, and reduce the gaps by proposing an ethics checklist for comprehensive assessment and transparent documentation of ethical discussions in generative AI development. While the checklist can be readily integrated into the current peer review and publication system to enhance generative AI research, it may also be used in broader settings to disclose ethics-related considerations in generative AI-powered products (or real-life applications of such products) to help users establish reasonable trust in their capabilities.

Title: Resist Label Noise with PGM for Graph Neural Networks. (arXiv:2311.02116v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.02116
• Code URL: null
• Copy Paste: [[2311.02116]] Resist Label Noise with PGM for Graph Neural Networks(http://arxiv.org/abs/2311.02116)
• Summary:

  While robust graph neural networks (GNNs) have been widely studied for graph perturbation and attack, those for label noise have received significantly less attention. Most existing methods heavily rely on the label smoothness assumption to correct noisy labels, which adversely affects their performance on heterophilous graphs. Further, they generally perform poorly in high noise-rate scenarios. To address these problems, in this paper, we propose a novel probabilistic graphical model (PGM) based framework LNP. Given a noisy label set and a clean label set, our goal is to maximize the likelihood of labels in the clean set. We first present LNP-v1, which generates clean labels based on graphs only in the Bayesian network. To further leverage the information of clean labels in the noisy label set, we put forward LNP-v2, which incorporates the noisy label set into the Bayesian network to generate clean labels. The generative process can then be used to predict labels for unlabeled nodes. We conduct extensive experiments to show the robustness of LNP on varying noise types and rates, and also on graphs with different heterophilies. In particular, we show that LNP can lead to inspiring performance in high noise-rate situations.

Title: Joint Composite Latent Space Bayesian Optimization. (arXiv:2311.02213v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.02213
• Code URL: null
• Copy Paste: [[2311.02213]] Joint Composite Latent Space Bayesian Optimization(http://arxiv.org/abs/2311.02213)
• Summary:

  Bayesian Optimization (BO) is a technique for sample-efficient black-box optimization that employs probabilistic models to identify promising input locations for evaluation. When dealing with composite-structured functions, such as f=g o h, evaluating a specific location x yields observations of both the final outcome f(x) = g(h(x)) as well as the intermediate output(s) h(x). Previous research has shown that integrating information from these intermediate outputs can enhance BO performance substantially. However, existing methods struggle if the outputs h(x) are high-dimensional. Many relevant problems fall into this setting, including in the context of generative AI, molecular design, or robotics. To effectively tackle these challenges, we introduce Joint Composite Latent Space Bayesian Optimization (JoCo), a novel framework that jointly trains neural network encoders and probabilistic models to adaptively compress high-dimensional input and output spaces into manageable latent representations. This enables viable BO on these compressed representations, allowing JoCo to outperform other state-of-the-art methods in high-dimensional BO on a wide variety of simulated and real-world problems.

Title: Structured Neural Networks for Density Estimation and Causal Inference. (arXiv:2311.02221v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.02221
• Code URL: null
• Copy Paste: [[2311.02221]] Structured Neural Networks for Density Estimation and Causal Inference(http://arxiv.org/abs/2311.02221)
• Summary:

  Injecting structure into neural networks enables learning functions that satisfy invariances with respect to subsets of inputs. For instance, when learning generative models using neural networks, it is advantageous to encode the conditional independence structure of observed variables, often in the form of Bayesian networks. We propose the Structured Neural Network (StrNN), which injects structure through masking pathways in a neural network. The masks are designed via a novel relationship we explore between neural network architectures and binary matrix factorization, to ensure that the desired independencies are respected. We devise and study practical algorithms for this otherwise NP-hard design problem based on novel objectives that control the model architecture. We demonstrate the utility of StrNN in three applications: (1) binary and Gaussian density estimation with StrNN, (2) real-valued density estimation with Structured Autoregressive Flows (StrAFs) and Structured Continuous Normalizing Flows (StrCNF), and (3) interventional and counterfactual analysis with StrAFs for causal inference. Our work opens up new avenues for learning neural networks that enable data-efficient generative modeling and the use of normalizing flows for causal effect estimation.

anomaly

in-context

Title: COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning. (arXiv:2311.02248v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.02248
• Code URL: null
• Copy Paste: [[2311.02248]] COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning(http://arxiv.org/abs/2311.02248)
• Summary:

  We present a data and cost efficient way of incorporating the speech modality into a large language model (LLM). The resulting multi-modal LLM is a COntextual Speech Model with Instruction-following/in-context-learning Capabilities - COSMIC. Speech comprehension test question-answer (SQA) pairs are generated using GPT-3.5 based on the speech transcriptions as a part of the supervision for the instruction tuning. With fewer than 20M trainable parameters and as little as 450 hours of English speech data for SQA generation, COSMIC exhibits emergent instruction-following and in-context learning capabilities in speech-to-text tasks. The model is able to follow the given text instructions to generate text response even on the unseen EN$\to$X speech-to-text translation (S2TT) task with zero-shot setting. We evaluate the model's in-context learning via various tasks such as EN$\to$X S2TT and few-shot domain adaptation. And instruction-following capabilities are evaluated through a contextual biasing benchmark. Our results demonstrate the efficacy of the proposed low cost recipe for building a speech LLM and that with the new instruction-tuning data.

Title: Narrowing the Gap between Zero- and Few-shot Machine Translation by Matching Styles. (arXiv:2311.02310v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.02310
• Code URL: null
• Copy Paste: [[2311.02310]] Narrowing the Gap between Zero- and Few-shot Machine Translation by Matching Styles(http://arxiv.org/abs/2311.02310)
• Summary:

  Large language models trained primarily in a monolingual setting have demonstrated their ability to generalize to machine translation using zero- and few-shot examples with in-context learning. However, even though zero-shot translations are relatively good, there remains a discernible gap comparing their performance with the few-shot setting. In this paper, we investigate the factors contributing to this gap and find that this gap can largely be closed (for about 70%) by matching the writing styles of the target corpus. Additionally, we explore potential approaches to enhance zero-shot baselines without the need for parallel demonstration examples, providing valuable insights into how these methods contribute to improving translation metrics.