2024-10-04

Title: Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospects

• Authors: Awal Ahmed Fime, Saifuddin Mahmud, Arpita Das, Md. Sunzidul Islam, Hong-Hoon Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01816
• Pdf URL: https://arxiv.org/pdf/2410.01816
• Copy Paste: [[2410.01816]] Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospects(https://arxiv.org/abs/2410.01816)
• Keywords: diffusion, generative
• Abstract: Automatic scene generation is an essential area of research with applications in robotics, recreation, visual representation, training and simulation, education, and more. This survey provides a comprehensive review of the current state-of-the-arts in automatic scene generation, focusing on techniques that leverage machine learning, deep learning, embedded systems, and natural language processing (NLP). We categorize the models into four main types: Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), Transformers, and Diffusion Models. Each category is explored in detail, discussing various sub-models and their contributions to the field. We also review the most commonly used datasets, such as COCO-Stuff, Visual Genome, and MS-COCO, which are critical for training and evaluating these models. Methodologies for scene generation are examined, including image-to-3D conversion, text-to-3D generation, UI/layout design, graph-based methods, and interactive scene generation. Evaluation metrics such as Frechet Inception Distance (FID), Kullback-Leibler (KL) Divergence, Inception Score (IS), Intersection over Union (IoU), and Mean Average Precision (mAP) are discussed in the context of their use in assessing model performance. The survey identifies key challenges and limitations in the field, such as maintaining realism, handling complex scenes with multiple objects, and ensuring consistency in object relationships and spatial arrangements. By summarizing recent advances and pinpointing areas for improvement, this survey aims to provide a valuable resource for researchers and practitioners working on automatic scene generation.

Title: PixelBytes: Catching Unified Representation for Multimodal Generation

• Authors: Fabien Furfaro
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01820
• Pdf URL: https://arxiv.org/pdf/2410.01820
• Copy Paste: [[2410.01820]] PixelBytes: Catching Unified Representation for Multimodal Generation(https://arxiv.org/abs/2410.01820)
• Keywords: foundation model
• Abstract: This report introduces PixelBytes, a novel approach for unified multimodal representation learning. Inspired by existing sequence models such as Image Transformers, PixelCNN, and Mamba-Bytes, our method aims to capture diverse inputs in a cohesive representation, exploring the integration of different data types, particularly text, audio, and pixelated images (sprites). We conducted experiments on a specialized PixelBytes Pok{é}mon dataset. Initially, we investigated various model architectures, including Recurrent Neural Networks (RNNs), State Space Models (SSMs), and Attention-based models, focusing on bidirectional processing and our convolutional PxBy embedding technique. Subsequently, we evaluated models based on data reduction strategies and the effectiveness of autoregressive learning. We specifically examined Long Short-Term Memory (LSTM) networks in both predictive and autoregressive modes for our main experiments. Our findings suggest that autoregressive models outperform predictive models in this context. By adopting a flexible approach to multimodal modeling, PixelBytes contributes to the ongoing development of foundation models capable of understanding and generating multimodal data. The complete PixelBytes project, including code, models, and datasets, is available online.

Title: OCC-MLLM-Alpha:Empowering Multi-modal Large Language Model for the Understanding of Occluded Objects with Self-Supervised Test-Time Learning

• Authors: Shuxin Yang, Xinhan Di
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.01861
• Pdf URL: https://arxiv.org/pdf/2410.01861
• Copy Paste: [[2410.01861]] OCC-MLLM-Alpha:Empowering Multi-modal Large Language Model for the Understanding of Occluded Objects with Self-Supervised Test-Time Learning(https://arxiv.org/abs/2410.01861)
• Keywords: self-supervised
• Abstract: There is a gap in the understanding of occluded objects in existing large-scale visual language multi-modal models. Current state-of-the-art multi-modal models fail to provide satisfactory results in describing occluded objects through universal visual encoders and supervised learning strategies. Therefore, we introduce a multi-modal large language framework and corresponding self-supervised learning strategy with support of 3D generation. We start our experiments comparing with the state-of-the-art models in the evaluation of a large-scale dataset SOMVideo [18]. The initial results demonstrate the improvement of 16.92% in comparison with the state-of-the-art VLM models.

Title: Social Media Authentication and Combating Deepfakes using Semi-fragile Invisible Image Watermarking

• Authors: Aakash Varma Nadimpalli, Ajita Rattani
• Subjects: cs.CV, cs.AI, cs.CR, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.01906
• Pdf URL: https://arxiv.org/pdf/2410.01906
• Copy Paste: [[2410.01906]] Social Media Authentication and Combating Deepfakes using Semi-fragile Invisible Image Watermarking(https://arxiv.org/abs/2410.01906)
• Keywords: generative
• Abstract: With the significant advances in deep generative models for image and video synthesis, Deepfakes and manipulated media have raised severe societal concerns. Conventional machine learning classifiers for deepfake detection often fail to cope with evolving deepfake generation technology and are susceptible to adversarial attacks. Alternatively, invisible image watermarking is being researched as a proactive defense technique that allows media authentication by verifying an invisible secret message embedded in the image pixels. A handful of invisible image watermarking techniques introduced for media authentication have proven vulnerable to basic image processing operations and watermark removal attacks. In response, we have proposed a semi-fragile image watermarking technique that embeds an invisible secret message into real images for media authentication. Our proposed watermarking framework is designed to be fragile to facial manipulations or tampering while being robust to benign image-processing operations and watermark removal attacks. This is facilitated through a unique architecture of our proposed technique consisting of critic and adversarial networks that enforce high image quality and resiliency to watermark removal efforts, respectively, along with the backbone encoder-decoder and the discriminator networks. Thorough experimental investigations on SOTA facial Deepfake datasets demonstrate that our proposed model can embed a $64$-bit secret as an imperceptible image watermark that can be recovered with a high-bit recovery accuracy when benign image processing operations are applied while being non-recoverable when unseen Deepfake manipulations are applied. In addition, our proposed watermarking technique demonstrates high resilience to several white-box and black-box watermark removal attacks. Thus, obtaining state-of-the-art performance.

Title: A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generation

• Authors: Liang Chen, Sinan Tan, Zefan Cai, Weichu Xie, Haozhe Zhao, Yichi Zhang, Junyang Lin, Jinze Bai, Tianyu Liu, Baobao Chang
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01912
• Pdf URL: https://arxiv.org/pdf/2410.01912
• Copy Paste: [[2410.01912]] A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generation(https://arxiv.org/abs/2410.01912)
• Keywords: diffusion, self-supervised, generative
• Abstract: This work tackles the information loss bottleneck of vector-quantization (VQ) autoregressive image generation by introducing a novel model architecture called the 2-Dimensional Autoregression (DnD) Transformer. The DnD-Transformer predicts more codes for an image by introducing a new autoregression direction, \textit{model depth}, along with the sequence length direction. Compared to traditional 1D autoregression and previous work utilizing similar 2D image decomposition such as RQ-Transformer, the DnD-Transformer is an end-to-end model that can generate higher quality images with the same backbone model size and sequence length, opening a new optimization perspective for autoregressive image generation. Furthermore, our experiments reveal that the DnD-Transformer's potential extends beyond generating natural images. It can even generate images with rich text and graphical elements in a self-supervised manner, demonstrating an understanding of these combined modalities. This has not been previously demonstrated for popular vision generative models such as diffusion models, showing a spark of vision-language intelligence when trained solely on images. Code, datasets and models are open at this https URL.

Title: TAEGAN: Generating Synthetic Tabular Data For Data Augmentation

• Authors: Jiayu Li, Zilong Zhao, Kevin Yee, Uzair Javaid, Biplab Sikdar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01933
• Pdf URL: https://arxiv.org/pdf/2410.01933
• Copy Paste: [[2410.01933]] TAEGAN: Generating Synthetic Tabular Data For Data Augmentation(https://arxiv.org/abs/2410.01933)
• Keywords: self-supervised, generative
• Abstract: Synthetic tabular data generation has gained significant attention for its potential in data augmentation, software testing and privacy-preserving data sharing. However, most research has primarily focused on larger datasets and evaluating their quality in terms of metrics like column-wise statistical distributions and inter-feature correlations, while often overlooking its utility for data augmentation, particularly for datasets whose data is scarce. In this paper, we propose Tabular Auto-Encoder Generative Adversarial Network (TAEGAN), an improved GAN-based framework for generating high-quality tabular data. Although large language models (LLMs)-based methods represent the state-of-the-art in synthetic tabular data generation, they are often overkill for small datasets due to their extensive size and complexity. TAEGAN employs a masked auto-encoder as the generator, which for the first time introduces the power of self-supervised pre-training in tabular data generation so that essentially exposes the networks to more information. We extensively evaluate TAEGAN against five state-of-the-art synthetic tabular data generation algorithms. Results from 10 datasets show that TAEGAN outperforms existing deep-learning-based tabular data generation models on 9 out of 10 datasets on the machine learning efficacy and achieves superior data augmentation performance on 7 out of 8 smaller datasets.

Title: Discrete Copula Diffusion

• Authors: Anji Liu, Oliver Broadrick, Mathias Niepert, Guy Van den Broeck
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.01949
• Pdf URL: https://arxiv.org/pdf/2410.01949
• Copy Paste: [[2410.01949]] Discrete Copula Diffusion(https://arxiv.org/abs/2410.01949)
• Keywords: diffusion, generative
• Abstract: Discrete diffusion models have recently shown significant progress in modeling complex data, such as natural languages and DNA sequences. However, unlike diffusion models for continuous data, which can generate high-quality samples in just a few denoising steps, modern discrete diffusion models still require hundreds or even thousands of denoising steps to perform well. In this paper, we identify a fundamental limitation that prevents discrete diffusion models from achieving strong performance with fewer steps -- they fail to capture dependencies between output variables at each denoising step. To address this issue, we provide a formal explanation and introduce a general approach to supplement the missing dependency information by incorporating another deep generative model, termed the copula model. Our method does not require fine-tuning either the diffusion model or the copula model, yet it enables high-quality sample generation with significantly fewer denoising steps. When we apply this approach to autoregressive copula models, the combined model outperforms both models individually in unconditional and conditional text generation. Specifically, the hybrid model achieves better (un)conditional text generation using 8 to 32 times fewer denoising steps than the diffusion model alone. In addition to presenting an effective discrete diffusion generation algorithm, this paper emphasizes the importance of modeling inter-variable dependencies in discrete diffusion.

Title: Score-based pullback Riemannian geometry

• Authors: Willem Diepeveen, Georgios Batzolis, Zakhar Shumaylov, Carola-Bibiane Schönlieb
• Subjects: cs.LG, math.DG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.01950
• Pdf URL: https://arxiv.org/pdf/2410.01950
• Copy Paste: [[2410.01950]] Score-based pullback Riemannian geometry(https://arxiv.org/abs/2410.01950)
• Keywords: generative
• Abstract: Data-driven Riemannian geometry has emerged as a powerful tool for interpretable representation learning, offering improved efficiency in downstream tasks. Moving forward, it is crucial to balance cheap manifold mappings with efficient training algorithms. In this work, we integrate concepts from pullback Riemannian geometry and generative models to propose a framework for data-driven Riemannian geometry that is scalable in both geometry and learning: score-based pullback Riemannian geometry. Focusing on unimodal distributions as a first step, we propose a score-based Riemannian structure with closed-form geodesics that pass through the data probability density. With this structure, we construct a Riemannian autoencoder (RAE) with error bounds for discovering the correct data manifold dimension. This framework can naturally be used with anisotropic normalizing flows by adopting isometry regularization during training. Through numerical experiments on various datasets, we demonstrate that our framework not only produces high-quality geodesics through the data support, but also reliably estimates the intrinsic dimension of the data manifold and provides a global chart of the manifold, even in high-dimensional ambient spaces.

Title: Generate then Refine: Data Augmentation for Zero-shot Intent Detection

• Authors: I-Fan Lin, Faegheh Hasibi, Suzan Verberne
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.01953
• Pdf URL: https://arxiv.org/pdf/2410.01953
• Copy Paste: [[2410.01953]] Generate then Refine: Data Augmentation for Zero-shot Intent Detection(https://arxiv.org/abs/2410.01953)
• Keywords: generative
• Abstract: In this short paper we propose a data augmentation method for intent detection in zero-resource domains. Existing data augmentation methods rely on few labelled examples for each intent category, which can be expensive in settings with many possible intents. We use a two-stage approach: First, we generate utterances for intent labels using an open-source large language model in a zero-shot setting. Second, we develop a smaller sequence-to-sequence model (the Refiner), to improve the generated utterances. The Refiner is fine-tuned on seen domains and then applied to unseen domains. We evaluate our method by training an intent classifier on the generated data, and evaluating it on real (human) data. We find that the Refiner significantly improves the data utility and diversity over the zero-shot LLM baseline for unseen domains and over common baseline approaches. Our results indicate that a two-step approach of a generative LLM in zero-shot setting and a smaller sequence-to-sequence model can provide high-quality data for intent detection.

Title: UlcerGPT: A Multimodal Approach Leveraging Large Language and Vision Models for Diabetic Foot Ulcer Image Transcription

• Authors: Reza Basiri, Ali Abedi, Chau Nguyen, Milos R. Popovic, Shehroz S. Khan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.01989
• Pdf URL: https://arxiv.org/pdf/2410.01989
• Copy Paste: [[2410.01989]] UlcerGPT: A Multimodal Approach Leveraging Large Language and Vision Models for Diabetic Foot Ulcer Image Transcription(https://arxiv.org/abs/2410.01989)
• Keywords: generative
• Abstract: Diabetic foot ulcers (DFUs) are a leading cause of hospitalizations and lower limb amputations, placing a substantial burden on patients and healthcare systems. Early detection and accurate classification of DFUs are critical for preventing serious complications, yet many patients experience delays in receiving care due to limited access to specialized services. Telehealth has emerged as a promising solution, improving access to care and reducing the need for in-person visits. The integration of artificial intelligence and pattern recognition into telemedicine has further enhanced DFU management by enabling automatic detection, classification, and monitoring from images. Despite advancements in artificial intelligence-driven approaches for DFU image analysis, the application of large language models for DFU image transcription has not yet been explored. To address this gap, we introduce UlcerGPT, a novel multimodal approach leveraging large language and vision models for DFU image transcription. This framework combines advanced vision and language models, such as Large Language and Vision Assistant and Chat Generative Pre-trained Transformer, to transcribe DFU images by jointly detecting, classifying, and localizing regions of interest. Through detailed experiments on a public dataset, evaluated by expert clinicians, UlcerGPT demonstrates promising results in the accuracy and efficiency of DFU transcription, offering potential support for clinicians in delivering timely care via telemedicine.

Title: Using Style Ambiguity Loss to Improve Aesthetics of Diffusion Models

• Authors: James Baker
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02055
• Pdf URL: https://arxiv.org/pdf/2410.02055
• Copy Paste: [[2410.02055]] Using Style Ambiguity Loss to Improve Aesthetics of Diffusion Models(https://arxiv.org/abs/2410.02055)
• Keywords: diffusion
• Abstract: Teaching text-to-image models to be creative involves using style ambiguity loss. In this work, we explore using the style ambiguity training objective, used to approximate creativity, on a diffusion model. We then experiment with forms of style ambiguity loss that do not require training a classifier or a labeled dataset, and find that the models trained with style ambiguity loss can generate better images than the baseline diffusion models and GANs. Code is available at this https URL.

Title: Semi-Supervised Fine-Tuning of Vision Foundation Models with Content-Style Decomposition

• Authors: Mariia Drozdova, Vitaliy Kinakh, Yury Belousov, Erica Lastufka, Slava Voloshynovskiy
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02069
• Pdf URL: https://arxiv.org/pdf/2410.02069
• Copy Paste: [[2410.02069]] Semi-Supervised Fine-Tuning of Vision Foundation Models with Content-Style Decomposition(https://arxiv.org/abs/2410.02069)
• Keywords: foundation model
• Abstract: In this paper, we present a semi-supervised fine-tuning approach designed to improve the performance of foundation models on downstream tasks with limited labeled data. By leveraging content-style decomposition within an information-theoretic framework, our method enhances the latent representations of pre-trained vision foundation models, aligning them more effectively with specific task objectives and addressing the problem of distribution shift. We evaluate our approach on multiple datasets, including MNIST, its augmented variations (with yellow and white stripes), CIFAR-10, SVHN, and GalaxyMNIST. The experiments show improvements over purely supervised baselines, particularly in low-labeled data regimes, across both frozen and trainable backbones for the majority of the tested datasets.

Title: Learning from the Giants: A Practical Approach to Underwater Depth and Surface Normals Estimation

• Authors: Alzayat Saleh, Melanie Olsen, Bouchra Senadji, Mostafa Rahimi Azghadi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02072
• Pdf URL: https://arxiv.org/pdf/2410.02072
• Copy Paste: [[2410.02072]] Learning from the Giants: A Practical Approach to Underwater Depth and Surface Normals Estimation(https://arxiv.org/abs/2410.02072)
• Keywords: generative
• Abstract: Monocular Depth and Surface Normals Estimation (MDSNE) is crucial for tasks such as 3D reconstruction, autonomous navigation, and underwater exploration. Current methods rely either on discriminative models, which struggle with transparent or reflective surfaces, or generative models, which, while accurate, are computationally expensive. This paper presents a novel deep learning model for MDSNE, specifically tailored for underwater environments, using a hybrid architecture that integrates Convolutional Neural Networks (CNNs) with Transformers, leveraging the strengths of both approaches. Training effective MDSNE models is often hampered by noisy real-world datasets and the limited generalization of synthetic datasets. To address this, we generate pseudo-labeled real data using multiple pre-trained MDSNE models. To ensure the quality of this data, we propose the Depth Normal Evaluation and Selection Algorithm (DNESA), which evaluates and selects the most reliable pseudo-labeled samples using domain-specific metrics. A lightweight student model is then trained on this curated dataset. Our model reduces parameters by 90% and training costs by 80%, allowing real-time 3D perception on resource-constrained devices. Key contributions include: a novel and efficient MDSNE model, the DNESA algorithm, a domain-specific data pipeline, and a focus on real-time performance and scalability. Designed for real-world underwater applications, our model facilitates low-cost deployments in underwater robots and autonomous vehicles, bridging the gap between research and practical implementation.

Title: Depth Pro: Sharp Monocular Metric Depth in Less Than a Second

• Authors: Aleksei Bochkovskii, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R. Richter, Vladlen Koltun
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02073
• Pdf URL: https://arxiv.org/pdf/2410.02073
• Copy Paste: [[2410.02073]] Depth Pro: Sharp Monocular Metric Depth in Less Than a Second(https://arxiv.org/abs/2410.02073)
• Keywords: foundation model
• Abstract: We present a foundation model for zero-shot metric monocular depth estimation. Our model, Depth Pro, synthesizes high-resolution depth maps with unparalleled sharpness and high-frequency details. The predictions are metric, with absolute scale, without relying on the availability of metadata such as camera intrinsics. And the model is fast, producing a 2.25-megapixel depth map in 0.3 seconds on a standard GPU. These characteristics are enabled by a number of technical contributions, including an efficient multi-scale vision transformer for dense prediction, a training protocol that combines real and synthetic datasets to achieve high metric accuracy alongside fine boundary tracing, dedicated evaluation metrics for boundary accuracy in estimated depth maps, and state-of-the-art focal length estimation from a single image. Extensive experiments analyze specific design choices and demonstrate that Depth Pro outperforms prior work along multiple dimensions. We release code and weights at this https URL

Title: Deep Generative Modeling for Identification of Noisy, Non-Stationary Dynamical Systems

• Authors: Doris Voina, Steven Brunton, J. Nathan Kutz
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.02079
• Pdf URL: https://arxiv.org/pdf/2410.02079
• Copy Paste: [[2410.02079]] Deep Generative Modeling for Identification of Noisy, Non-Stationary Dynamical Systems(https://arxiv.org/abs/2410.02079)
• Keywords: generative
• Abstract: A significant challenge in many fields of science and engineering is making sense of time-dependent measurement data by recovering governing equations in the form of differential equations. We focus on finding parsimonious ordinary differential equation (ODE) models for nonlinear, noisy, and non-autonomous dynamical systems and propose a machine learning method for data-driven system identification. While many methods tackle noisy and limited data, non-stationarity - where differential equation parameters change over time - has received less attention. Our method, dynamic SINDy, combines variational inference with SINDy (sparse identification of nonlinear dynamics) to model time-varying coefficients of sparse ODEs. This framework allows for uncertainty quantification of ODE coefficients, expanding on previous methods for autonomous systems. These coefficients are then interpreted as latent variables and added to the system to obtain an autonomous dynamical model. We validate our approach using synthetic data, including nonlinear oscillators and the Lorenz system, and apply it to neuronal activity data from C. elegans. Dynamic SINDy uncovers a global nonlinear model, showing it can handle real, noisy, and chaotic datasets. We aim to apply our method to a variety of problems, specifically dynamic systems with complex time-dependent parameters.

Title: EMMA: Efficient Visual Alignment in Multi-Modal LLMs

• Authors: Sara Ghazanfari, Alexandre Araujo, Prashanth Krishnamurthy, Siddharth Garg, Farshad Khorrami
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02080
• Pdf URL: https://arxiv.org/pdf/2410.02080
• Copy Paste: [[2410.02080]] EMMA: Efficient Visual Alignment in Multi-Modal LLMs(https://arxiv.org/abs/2410.02080)
• Keywords: foundation model
• Abstract: Multi-modal Large Language Models (MLLMs) have recently exhibited impressive general-purpose capabilities by leveraging vision foundation models to encode the core concepts of images into representations. These are then combined with instructions and processed by the language model to generate high-quality responses. Despite significant progress in enhancing the language component, challenges persist in optimally fusing visual encodings within the language model for task-specific adaptability. Recent research has focused on improving this fusion through modality adaptation modules but at the cost of significantly increased model complexity and training data needs. In this paper, we propose EMMA (Efficient Multi-Modal Adaptation), a lightweight cross-modality module designed to efficiently fuse visual and textual encodings, generating instruction-aware visual representations for the language model. Our key contributions include: (1) an efficient early fusion mechanism that integrates vision and language representations with minimal added parameters (less than 0.2% increase in model size), (2) an in-depth interpretability analysis that sheds light on the internal mechanisms of the proposed method; (3) comprehensive experiments that demonstrate notable improvements on both specialized and general benchmarks for MLLMs. Empirical results show that EMMA boosts performance across multiple tasks by up to 9.3% while significantly improving robustness against hallucinations. Our code is available at this https URL

Title: FARM: Functional Group-Aware Representations for Small Molecules

• Authors: Thao Nguyen, Kuan-Hao Huang, Ge Liu, Martin D. Burke, Ying Diao, Heng Ji
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.02082
• Pdf URL: https://arxiv.org/pdf/2410.02082
• Copy Paste: [[2410.02082]] FARM: Functional Group-Aware Representations for Small Molecules(https://arxiv.org/abs/2410.02082)
• Keywords: foundation model
• Abstract: We introduce Functional Group-Aware Representations for Small Molecules (FARM), a novel foundation model designed to bridge the gap between SMILES, natural language, and molecular graphs. The key innovation of FARM lies in its functional group-aware tokenization, which incorporates functional group information directly into the representations. This strategic reduction in tokenization granularity in a way that is intentionally interfaced with key drivers of functional properties (i.e., functional groups) enhances the model's understanding of chemical language, expands the chemical lexicon, more effectively bridging SMILES and natural language, and ultimately advances the model's capacity to predict molecular properties. FARM also represents molecules from two perspectives: by using masked language modeling to capture atom-level features and by employing graph neural networks to encode the whole molecule topology. By leveraging contrastive learning, FARM aligns these two views of representations into a unified molecular embedding. We rigorously evaluate FARM on the MoleculeNet dataset, where it achieves state-of-the-art performance on 10 out of 12 tasks. These results highlight FARM's potential to improve molecular representation learning, with promising applications in drug discovery and pharmaceutical research.

Title: HyperBrain: Anomaly Detection for Temporal Hypergraph Brain Networks

• Authors: Sadaf Sadeghian, Xiaoxiao Li, Margo Seltzer
• Subjects: cs.LG, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2410.02087
• Pdf URL: https://arxiv.org/pdf/2410.02087
• Copy Paste: [[2410.02087]] HyperBrain: Anomaly Detection for Temporal Hypergraph Brain Networks(https://arxiv.org/abs/2410.02087)
• Keywords: anomaly
• Abstract: Identifying unusual brain activity is a crucial task in neuroscience research, as it aids in the early detection of brain disorders. It is common to represent brain networks as graphs, and researchers have developed various graph-based machine learning methods for analyzing them. However, the majority of existing graph learning tools for the brain face a combination of the following three key limitations. First, they focus only on pairwise correlations between regions of the brain, limiting their ability to capture synchronized activity among larger groups of regions. Second, they model the brain network as a static network, overlooking the temporal changes in the brain. Third, most are designed only for classifying brain networks as healthy or disordered, lacking the ability to identify abnormal brain activity patterns linked to biomarkers associated with disorders. To address these issues, we present HyperBrain, an unsupervised anomaly detection framework for temporal hypergraph brain networks. HyperBrain models fMRI time series data as temporal hypergraphs capturing dynamic higher-order interactions. It then uses a novel customized temporal walk (BrainWalk) and neural encodings to detect abnormal co-activations among brain regions. We evaluate the performance of HyperBrain in both synthetic and real-world settings for Autism Spectrum Disorder and Attention Deficit Hyperactivity Disorder(ADHD). HyperBrain outperforms all other baselines on detecting abnormal co-activations in brain networks. Furthermore, results obtained from HyperBrain are consistent with clinical research on these brain disorders. Our findings suggest that learning temporal and higher-order connections in the brain provides a promising approach to uncover intricate connectivity patterns in brain networks, offering improved diagnosis.

Title: EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing

• Authors: Haotian Sun, Bowen Zhang, Yanghao Li, Haoshuo Huang, Tao Lei, Ruoming Pang, Bo Dai, Nan Du
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02098
• Pdf URL: https://arxiv.org/pdf/2410.02098
• Copy Paste: [[2410.02098]] EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing(https://arxiv.org/abs/2410.02098)
• Keywords: diffusion
• Abstract: Diffusion transformers have been widely adopted for text-to-image synthesis. While scaling these models up to billions of parameters shows promise, the effectiveness of scaling beyond current sizes remains underexplored and challenging. By explicitly exploiting the computational heterogeneity of image generations, we develop a new family of Mixture-of-Experts (MoE) models (EC-DIT) for diffusion transformers with expert-choice routing. EC-DIT learns to adaptively optimize the compute allocated to understand the input texts and generate the respective image patches, enabling heterogeneous computation aligned with varying text-image complexities. This heterogeneity provides an efficient way of scaling EC-DIT up to 97 billion parameters and achieving significant improvements in training convergence, text-to-image alignment, and overall generation quality over dense models and conventional MoE models. Through extensive ablations, we show that EC-DIT demonstrates superior scalability and adaptive compute allocation by recognizing varying textual importance through end-to-end training. Notably, in text-to-image alignment evaluation, our largest models achieve a state-of-the-art GenEval score of 71.68% and still maintain competitive inference speed with intuitive interpretability.

Title: Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networks

• Authors: Siddharth Joshi, Jiayi Ni, Baharan Mirzasoleiman
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02116
• Pdf URL: https://arxiv.org/pdf/2410.02116
• Copy Paste: [[2410.02116]] Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networks(https://arxiv.org/abs/2410.02116)
• Keywords: self-supervised
• Abstract: Dataset distillation (DD) generates small synthetic datasets that can efficiently train deep networks with a limited amount of memory and compute. Despite the success of DD methods for supervised learning, DD for self-supervised pre-training of deep models has remained unaddressed. Pre-training on unlabeled data is crucial for efficiently generalizing to downstream tasks with limited labeled data. In this work, we propose the first effective DD method for SSL pre-training. First, we show, theoretically and empirically, that naive application of supervised DD methods to SSL fails, due to the high variance of the SSL gradient. Then, we address this issue by relying on insights from knowledge distillation (KD) literature. Specifically, we train a small student model to match the representations of a larger teacher model trained with SSL. Then, we generate a small synthetic dataset by matching the training trajectories of the student models. As the KD objective has considerably lower variance than SSL, our approach can generate synthetic datasets that can successfully pre-train high-quality encoders. Through extensive experiments, we show that our distilled sets lead to up to 13% higher accuracy than prior work, on a variety of downstream tasks, in the presence of limited labeled data.

Title: C-MELT: Contrastive Enhanced Masked Auto-Encoders for ECG-Language Pre-Training

• Authors: Manh Pham, Aaqib Saeed, Dong Ma
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02131
• Pdf URL: https://arxiv.org/pdf/2410.02131
• Copy Paste: [[2410.02131]] C-MELT: Contrastive Enhanced Masked Auto-Encoders for ECG-Language Pre-Training(https://arxiv.org/abs/2410.02131)
• Keywords: generative
• Abstract: Accurate interpretation of Electrocardiogram (ECG) signals is pivotal for diagnosing cardiovascular diseases. Integrating ECG signals with their accompanying textual reports holds immense potential to enhance clinical diagnostics through the combination of physiological data and qualitative insights. However, this integration faces significant challenges due to inherent modality disparities and the scarcity of labeled data for robust cross-modal learning. To address these obstacles, we propose C-MELT, a novel framework that pre-trains ECG and text data using a contrastive masked auto-encoder architecture. C-MELT uniquely combines the strengths of generative with enhanced discriminative capabilities to achieve robust cross-modal representations. This is accomplished through masked modality modeling, specialized loss functions, and an improved negative sampling strategy tailored for cross-modal alignment. Extensive experiments on five public datasets across diverse downstream tasks demonstrate that C-MELT significantly outperforms existing methods, achieving 15% and 2% increases in linear probing and zero-shot performance over state-of-the-art models, respectively. These results highlight the effectiveness of C-MELT, underscoring its potential to advance automated clinical diagnostics through multi-modal representations.

Title: TrajGPT: Irregular Time-Series Representation Learning for Health Trajectory Analysis

• Authors: Ziyang Song, Qingcheng Lu, He Zhu, David Buckeridge, Yue Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02133
• Pdf URL: https://arxiv.org/pdf/2410.02133
• Copy Paste: [[2410.02133]] TrajGPT: Irregular Time-Series Representation Learning for Health Trajectory Analysis(https://arxiv.org/abs/2410.02133)
• Keywords: generative
• Abstract: In many domains, such as healthcare, time-series data is often irregularly sampled with varying intervals between observations. This poses challenges for classical time-series models that require equally spaced data. To address this, we propose a novel time-series Transformer called Trajectory Generative Pre-trained Transformer (TrajGPT). TrajGPT employs a novel Selective Recurrent Attention (SRA) mechanism, which utilizes a data-dependent decay to adaptively filter out irrelevant past information based on contexts. By interpreting TrajGPT as discretized ordinary differential equations (ODEs), it effectively captures the underlying continuous dynamics and enables time-specific inference for forecasting arbitrary target timesteps. Experimental results demonstrate that TrajGPT excels in trajectory forecasting, drug usage prediction, and phenotype classification without requiring task-specific fine-tuning. By evolving the learned continuous dynamics, TrajGPT can interpolate and extrapolate disease risk trajectories from partially-observed time series. The visualization of predicted health trajectories shows that TrajGPT forecasts unseen diseases based on the history of clinically relevant phenotypes (i.e., contexts).

Title: Plug-and-Play Controllable Generation for Discrete Masked Models

• Authors: Wei Guo, Yuchen Zhu, Molei Tao, Yongxin Chen
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.02143
• Pdf URL: https://arxiv.org/pdf/2410.02143
• Copy Paste: [[2410.02143]] Plug-and-Play Controllable Generation for Discrete Masked Models(https://arxiv.org/abs/2410.02143)
• Keywords: generative
• Abstract: This article makes discrete masked models for the generative modeling of discrete data controllable. The goal is to generate samples of a discrete random variable that adheres to a posterior distribution, satisfies specific constraints, or optimizes a reward function. This methodological development enables broad applications across downstream tasks such as class-specific image generation and protein design. Existing approaches for controllable generation of masked models typically rely on task-specific fine-tuning or additional modifications, which can be inefficient and resource-intensive. To overcome these limitations, we propose a novel plug-and-play framework based on importance sampling that bypasses the need for training a conditional score. Our framework is agnostic to the choice of control criteria, requires no gradient information, and is well-suited for tasks such as posterior sampling, Bayesian inverse problems, and constrained generation. We demonstrate the effectiveness of our approach through extensive experiments, showcasing its versatility across multiple domains, including protein design.

Title: Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis

• Authors: Hongkang Li, Meng Wang, Songtao Lu, Xiaodong Cui, Pin-Yu Chen
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02167
• Pdf URL: https://arxiv.org/pdf/2410.02167
• Copy Paste: [[2410.02167]] Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis(https://arxiv.org/abs/2410.02167)
• Keywords: in-context
• Abstract: Chain-of-Thought (CoT) is an efficient prompting method that enables the reasoning ability of large language models by augmenting the query using multiple examples with multiple intermediate steps. Despite the empirical success, the theoretical understanding of how to train a Transformer to achieve the CoT ability remains less explored. This is primarily due to the technical challenges involved in analyzing the nonconvex optimization on nonlinear attention models. To the best of our knowledge, this work provides the first theoretical study of training Transformers with nonlinear attention to obtain the CoT generalization capability so that the resulting model can inference on unseen tasks when the input is augmented by examples of the new task. We first quantify the required training samples and iterations to train a Transformer model towards CoT ability. We then prove the success of its CoT generalization on unseen tasks with distribution-shifted testing data. Moreover, we theoretically characterize the conditions for an accurate reasoning output by CoT even when the provided reasoning examples contain noises and are not always accurate. In contrast, in-context learning (ICL), which can be viewed as one-step CoT without intermediate steps, may fail to provide an accurate output when CoT does. These theoretical findings are justified through experiments.

Title: Channel-aware Contrastive Conditional Diffusion for Multivariate Probabilistic Time Series Forecasting

• Authors: Siyang Li, Yize Chen, Hui Xiong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02168
• Pdf URL: https://arxiv.org/pdf/2410.02168
• Copy Paste: [[2410.02168]] Channel-aware Contrastive Conditional Diffusion for Multivariate Probabilistic Time Series Forecasting(https://arxiv.org/abs/2410.02168)
• Keywords: diffusion, generative
• Abstract: Forecasting faithful trajectories of multivariate time series from practical scopes is essential for reasonable decision-making. Recent methods majorly tailor generative conditional diffusion models to estimate the target temporal predictive distribution. However, it remains an obstacle to enhance the exploitation efficiency of given implicit temporal predictive information to bolster conditional diffusion learning. To this end, we propose a generic channel-aware Contrastive Conditional Diffusion model entitled CCDM to achieve desirable Multivariate probabilistic forecasting, obviating the need for curated temporal conditioning inductive biases. In detail, we first design a channel-centric conditional denoising network to manage intra-variate variations and cross-variate correlations, which can lead to scalability on diverse prediction horizons and channel numbers. Then, we devise an ad-hoc denoising-based temporal contrastive learning to explicitly amplify the predictive mutual information between past observations and future forecasts. It can coherently complement naive step-wise denoising diffusion training and improve the forecasting accuracy and generality on unknown test time series. Besides, we offer theoretic insights on the benefits of such auxiliary contrastive training refinement from both neural mutual information and temporal distribution generalization aspects. The proposed CCDM can exhibit superior forecasting capability compared to current state-of-the-art diffusion forecasters over a comprehensive benchmark, with best MSE and CRPS outcomes on $66.67\%$ and $83.33\%$ cases. Our code is publicly available at this https URL.

Title: Calibrate to Discriminate: Improve In-Context Learning with Label-Free Comparative Inference

• Authors: Wei Cheng, Tianlu Wang, Yanmin Ji, Fan Yang, Keren Tan, Yiyu Zheng
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02210
• Pdf URL: https://arxiv.org/pdf/2410.02210
• Copy Paste: [[2410.02210]] Calibrate to Discriminate: Improve In-Context Learning with Label-Free Comparative Inference(https://arxiv.org/abs/2410.02210)
• Keywords: in-context
• Abstract: While in-context learning with large language models (LLMs) has shown impressive performance, we have discovered a unique miscalibration behavior where both correct and incorrect predictions are assigned the same level of confidence. We refer to this phenomenon as indiscriminate miscalibration. We found that traditional calibration metrics, such as Expected Calibrated Errors (ECEs), are unable to capture this behavior effectively. To address this issue, we propose new metrics to measure the severity of indiscriminate miscalibration. Additionally, we develop a novel in-context comparative inference method to alleviate miscalibrations and improve classification performance. Through extensive experiments on five datasets, we demonstrate that our proposed method can achieve more accurate and calibrated predictions compared to regular zero-shot and few-shot prompting.

Title: Hard Negative Sample Mining for Whole Slide Image Classification

• Authors: Wentao Huang, Xiaoling Hu, Shahira Abousamra, Prateek Prasanna, Chao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02212
• Pdf URL: https://arxiv.org/pdf/2410.02212
• Copy Paste: [[2410.02212]] Hard Negative Sample Mining for Whole Slide Image Classification(https://arxiv.org/abs/2410.02212)
• Keywords: self-supervised
• Abstract: Weakly supervised whole slide image (WSI) classification is challenging due to the lack of patch-level labels and high computational costs. State-of-the-art methods use self-supervised patch-wise feature representations for multiple instance learning (MIL). Recently, methods have been proposed to fine-tune the feature representation on the downstream task using pseudo labeling, but mostly focusing on selecting high-quality positive patches. In this paper, we propose to mine hard negative samples during fine-tuning. This allows us to obtain better feature representations and reduce the training cost. Furthermore, we propose a novel patch-wise ranking loss in MIL to better exploit these hard negative samples. Experiments on two public datasets demonstrate the efficacy of these proposed ideas. Our codes are available at this https URL

Title: Mitigating Downstream Model Risks via Model Provenance

• Authors: Keyu Wang, Abdullah Norozi Iranzad, Scott Schaffter, Doina Precup, Jonathan Lebensold
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02230
• Pdf URL: https://arxiv.org/pdf/2410.02230
• Copy Paste: [[2410.02230]] Mitigating Downstream Model Risks via Model Provenance(https://arxiv.org/abs/2410.02230)
• Keywords: foundation model
• Abstract: Research and industry are rapidly advancing the innovation and adoption of foundation model-based systems, yet the tools for managing these models have not kept pace. Understanding the provenance and lineage of models is critical for researchers, industry, regulators, and public trust. While model cards and system cards were designed to provide transparency, they fall short in key areas: tracing model genealogy, enabling machine readability, offering reliable centralized management systems, and fostering consistent creation incentives. This challenge mirrors issues in software supply chain security, but AI/ML remains at an earlier stage of maturity. Addressing these gaps requires industry-standard tooling that can be adopted by foundation model publishers, open-source model innovators, and major distribution platforms. We propose a machine-readable model specification format to simplify the creation of model records, thereby reducing error-prone human effort, notably when a new model inherits most of its design from a foundation model. Our solution explicitly traces relationships between upstream and downstream models, enhancing transparency and traceability across the model lifecycle. To facilitate the adoption, we introduce the unified model record (UMR) repository , a semantically versioned system that automates the publication of model records to multiple formats (PDF, HTML, LaTeX) and provides a hosted web interface (this https URL). This proof of concept aims to set a new standard for managing foundation models, bridging the gap between innovation and responsible model management.

Title: SCA: Highly Efficient Semantic-Consistent Unrestricted Adversarial Attack

• Authors: Zihao Pan, Weibin Wu, Yuhang Cao, Zibin Zheng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02240
• Pdf URL: https://arxiv.org/pdf/2410.02240
• Copy Paste: [[2410.02240]] SCA: Highly Efficient Semantic-Consistent Unrestricted Adversarial Attack(https://arxiv.org/abs/2410.02240)
• Keywords: diffusion
• Abstract: Unrestricted adversarial attacks typically manipulate the semantic content of an image (e.g., color or texture) to create adversarial examples that are both effective and photorealistic. Recent works have utilized the diffusion inversion process to map images into a latent space, where high-level semantics are manipulated by introducing perturbations. However, they often results in substantial semantic distortions in the denoised output and suffers from low efficiency. In this study, we propose a novel framework called Semantic-Consistent Unrestricted Adversarial Attacks (SCA), which employs an inversion method to extract edit-friendly noise maps and utilizes Multimodal Large Language Model (MLLM) to provide semantic guidance throughout the process. Under the condition of rich semantic information provided by MLLM, we perform the DDPM denoising process of each step using a series of edit-friendly noise maps, and leverage DPM Solver++ to accelerate this process, enabling efficient sampling with semantic consistency. Compared to existing methods, our framework enables the efficient generation of adversarial examples that exhibit minimal discernible semantic changes. Consequently, we for the first time introduce Semantic-Consistent Adversarial Examples (SCAE). Extensive experiments and visualizations have demonstrated the high efficiency of SCA, particularly in being on average 12 times faster than the state-of-the-art attacks. Our code can be found at this https URL}{this https URL.

Title: PFGuard: A Generative Framework with Privacy and Fairness Safeguards

• Authors: Soyeon Kim, Yuji Roh, Geon Heo, Steven Euijong Whang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02246
• Pdf URL: https://arxiv.org/pdf/2410.02246
• Copy Paste: [[2410.02246]] PFGuard: A Generative Framework with Privacy and Fairness Safeguards(https://arxiv.org/abs/2410.02246)
• Keywords: generative
• Abstract: Generative models must ensure both privacy and fairness for Trustworthy AI. While these goals have been pursued separately, recent studies propose to combine existing privacy and fairness techniques to achieve both goals. However, naively combining these techniques can be insufficient due to privacy-fairness conflicts, where a sample in a minority group may be amplified for fairness, only to be suppressed for privacy. We demonstrate how these conflicts lead to adverse effects, such as privacy violations and unexpected fairness-utility tradeoffs. To mitigate these risks, we propose PFGuard, a generative framework with privacy and fairness safeguards, which simultaneously addresses privacy, fairness, and utility. By using an ensemble of multiple teacher models, PFGuard balances privacy-fairness conflicts between fair and private training stages and achieves high utility based on ensemble learning. Extensive experiments show that PFGuard successfully generates synthetic data on high-dimensional data while providing both fairness convergence and strict DP guarantees - the first of its kind to our knowledge.

Title: Correlation and Navigation in the Vocabulary Key Representation Space of Language Models

• Authors: Letian Peng, Chenyang An, Jingbo Shang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02284
• Pdf URL: https://arxiv.org/pdf/2410.02284
• Copy Paste: [[2410.02284]] Correlation and Navigation in the Vocabulary Key Representation Space of Language Models(https://arxiv.org/abs/2410.02284)
• Keywords: in-context
• Abstract: Language model (LM) decoding is based on the next-token prediction (NTP) probability distribution. For neural LMs (e.g., Transformer-based), NTP distribution is essentially a softmax-regularized dot product between an encoded input context (query) and fixed vocabulary representations (keys). In this paper, we study the effect of the key distribution on the NTP distribution, with a focus on whether the similarity between keys will trigger spurious correlations in NTP. Through knowledge-probing tasks, we show that in the NTP distribution, the few top-ranked tokens are typically accurate. However, the middle-ranked prediction is highly biased towards the tokens that are distributionally (not necessarily semantically) similar to these top ones. For instance, if "P" is predicted as the top-1 token, "A"-"Z" will all be ranked high in NTP, no matter whether they can lead to correct decoding results. This hurts the sampling diversity and makes the sampling of correct, long-tail results hopeless and noisy. We attempt to alleviate this issue via a novel in-context method that iteratively pushes the query representation away from explored regions. Specifically, we include the explored decoding results in the context and prompt the LM to generate something else, which encourages the LM to produce a query representation that has small dot products with explored keys. Experiments on knowledge-probing tasks show that our method leads to efficient navigation away from explored keys to correct new keys. We further extend our method to open-ended and chain-of-thought (for reasoning) generation. Experiment results show that ICN contributes to better generation diversity and improved self-consistency voting performance. Finally, we discuss potential training issues caused by the fixed key space together with the challenges and possible ways to address them in future research.

Title: Make Compound Sentences Simple to Analyze: Learning to Split Sentences for Aspect-based Sentiment Analysis

• Authors: Yongsik Seo, Sungwon Song, Ryang Heo, Jieyong Kim, Dongha Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.02297
• Pdf URL: https://arxiv.org/pdf/2410.02297
• Copy Paste: [[2410.02297]] Make Compound Sentences Simple to Analyze: Learning to Split Sentences for Aspect-based Sentiment Analysis(https://arxiv.org/abs/2410.02297)
• Keywords: generative
• Abstract: In the domain of Aspect-Based Sentiment Analysis (ABSA), generative methods have shown promising results and achieved substantial advancements. However, despite these advancements, the tasks of extracting sentiment quadruplets, which capture the nuanced sentiment expressions within a sentence, remain significant challenges. In particular, compound sentences can potentially contain multiple quadruplets, making the extraction task increasingly difficult as sentence complexity grows. To address this issue, we are focusing on simplifying sentence structures to facilitate the easier recognition of these elements and crafting a model that integrates seamlessly with various ABSA tasks. In this paper, we propose Aspect Term Oriented Sentence Splitter (ATOSS), which simplifies compound sentence into simpler and clearer forms, thereby clarifying their structure and intent. As a plug-and-play module, this approach retains the parameters of the ABSA model while making it easier to identify essential intent within input sentences. Extensive experimental results show that utilizing ATOSS outperforms existing methods in both ASQP and ACOS tasks, which are the primary tasks for extracting sentiment quadruplets.

Title: Decoupling Layout from Glyph in Online Chinese Handwriting Generation

• Authors: Ren-Min Si, Yan-Ming Zhang, Yi Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02309
• Pdf URL: https://arxiv.org/pdf/2410.02309
• Copy Paste: [[2410.02309]] Decoupling Layout from Glyph in Online Chinese Handwriting Generation(https://arxiv.org/abs/2410.02309)
• Keywords: diffusion, in-context
• Abstract: Text plays a crucial role in the transmission of human civilization, and teaching machines to generate online handwritten text in various styles presents an interesting and significant challenge. However, most prior work has concentrated on generating individual Chinese fonts, leaving {complete text line generation largely unexplored}. In this paper, we identify that text lines can naturally be divided into two components: layout and glyphs. Based on this division, we designed a text line layout generator coupled with a diffusion-based stylized font synthesizer to address this challenge hierarchically. More concretely, the layout generator performs in-context-like learning based on the text content and the provided style references to generate positions for each glyph autoregressively. Meanwhile, the font synthesizer which consists of a character embedding dictionary, a multi-scale calligraphy style encoder, and a 1D U-Net based diffusion denoiser will generate each font on its position while imitating the calligraphy style extracted from the given style references. Qualitative and quantitative experiments on the CASIA-OLHWDB demonstrate that our method is capable of generating structurally correct and indistinguishable imitation samples.

Title: Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis

• Authors: Zikun Zhang, Zixiang Chen, Quanquan Gu
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.02321
• Pdf URL: https://arxiv.org/pdf/2410.02321
• Copy Paste: [[2410.02321]] Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis(https://arxiv.org/abs/2410.02321)
• Keywords: diffusion
• Abstract: Diffusion models have achieved great success in generating high-dimensional samples across various applications. While the theoretical guarantees for continuous-state diffusion models have been extensively studied, the convergence analysis of the discrete-state counterparts remains under-explored. In this paper, we study the theoretical aspects of score-based discrete diffusion models under the Continuous Time Markov Chain (CTMC) framework. We introduce a discrete-time sampling algorithm in the general state space $[S]^d$ that utilizes score estimators at predefined time points. We derive convergence bounds for the Kullback-Leibler (KL) divergence and total variation (TV) distance between the generated sample distribution and the data distribution, considering both scenarios with and without early stopping under specific assumptions. Notably, our KL divergence bounds are nearly linear in dimension $d$, aligning with state-of-the-art results for diffusion models. Our convergence analysis employs a Girsanov-based method and establishes key properties of the discrete score function, which are essential for characterizing the discrete-time sampling process.

Title: Simplicity bias and optimization threshold in two-layer ReLU networks

• Authors: Etienne Boursier, Nicolas Flammarion
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.02348
• Pdf URL: https://arxiv.org/pdf/2410.02348
• Copy Paste: [[2410.02348]] Simplicity bias and optimization threshold in two-layer ReLU networks(https://arxiv.org/abs/2410.02348)
• Keywords: diffusion, in-context
• Abstract: Understanding generalization of overparametrized neural networks remains a fundamental challenge in machine learning. Most of the literature mostly studies generalization from an interpolation point of view, taking convergence of parameters towards a global minimum of the training loss for granted. While overparametrized architectures indeed interpolated the data for typical classification tasks, this interpolation paradigm does not seem valid anymore for more complex tasks such as in-context learning or diffusion. Instead for such tasks, it has been empirically observed that the trained models goes from global minima to spurious local minima of the training loss as the number of training samples becomes larger than some level we call optimization threshold. While the former yields a poor generalization to the true population loss, the latter was observed to actually correspond to the minimiser of this true loss. This paper explores theoretically this phenomenon in the context of two-layer ReLU networks. We demonstrate that, despite overparametrization, networks often converge toward simpler solutions rather than interpolating the training data, which can lead to a drastic improvement on the test loss with respect to interpolating solutions. Our analysis relies on the so called early alignment phase, during which neurons align towards specific directions. This directional alignment, which occurs in the early stage of training, leads to a simplicity bias, wherein the network approximates the ground truth model without converging to the global minimum of the training loss. Our results suggest that this bias, resulting in an optimization threshold from which interpolation is not reached anymore, is beneficial and enhances the generalization of trained models.

Title: From Concrete to Abstract: A Multimodal Generative Approach to Abstract Concept Learning

• Authors: Haodong Xie, Rahul Singh Maharjan, Federico Tavella, Angelo Cangelosi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02365
• Pdf URL: https://arxiv.org/pdf/2410.02365
• Copy Paste: [[2410.02365]] From Concrete to Abstract: A Multimodal Generative Approach to Abstract Concept Learning(https://arxiv.org/abs/2410.02365)
• Keywords: generative
• Abstract: Understanding and manipulating concrete and abstract concepts is fundamental to human intelligence. Yet, they remain challenging for artificial agents. This paper introduces a multimodal generative approach to high order abstract concept learning, which integrates visual and categorical linguistic information from concrete ones. Our model initially grounds subordinate level concrete concepts, combines them to form basic level concepts, and finally abstracts to superordinate level concepts via the grounding of basic-level concepts. We evaluate the model language learning ability through language-to-visual and visual-to-language tests with high order abstract concepts. Experimental results demonstrate the proficiency of the model in both language understanding and language naming tasks.

Title: Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation

• Authors: Muzhi Zhu, Yang Liu, Zekai Luo, Chenchen Jing, Hao Chen, Guangkai Xu, Xinlong Wang, Chunhua Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02369
• Pdf URL: https://arxiv.org/pdf/2410.02369
• Copy Paste: [[2410.02369]] Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation(https://arxiv.org/abs/2410.02369)
• Keywords: diffusion, generative, in-context
• Abstract: The Diffusion Model has not only garnered noteworthy achievements in the realm of image generation but has also demonstrated its potential as an effective pretraining method utilizing unlabeled data. Drawing from the extensive potential unveiled by the Diffusion Model in both semantic correspondence and open vocabulary segmentation, our work initiates an investigation into employing the Latent Diffusion Model for Few-shot Semantic Segmentation. Recently, inspired by the in-context learning ability of large language models, Few-shot Semantic Segmentation has evolved into In-context Segmentation tasks, morphing into a crucial element in assessing generalist segmentation models. In this context, we concentrate on Few-shot Semantic Segmentation, establishing a solid foundation for the future development of a Diffusion-based generalist model for segmentation. Our initial focus lies in understanding how to facilitate interaction between the query image and the support image, resulting in the proposal of a KV fusion method within the self-attention framework. Subsequently, we delve deeper into optimizing the infusion of information from the support mask and simultaneously re-evaluating how to provide reasonable supervision from the query mask. Based on our analysis, we establish a simple and effective framework named DiffewS, maximally retaining the original Latent Diffusion Model's generative framework and effectively utilizing the pre-training prior. Experimental results demonstrate that our method significantly outperforms the previous SOTA models in multiple settings.

Title: BiSSL: Bilevel Optimization for Self-Supervised Pre-Training and Fine-Tuning

• Authors: Gustav Wagner Zakarias, Lars Kai Hansen, Zheng-Hua Tan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02387
• Pdf URL: https://arxiv.org/pdf/2410.02387
• Copy Paste: [[2410.02387]] BiSSL: Bilevel Optimization for Self-Supervised Pre-Training and Fine-Tuning(https://arxiv.org/abs/2410.02387)
• Keywords: self-supervised
• Abstract: In this work, we present BiSSL, a first-of-its-kind training framework that introduces bilevel optimization to enhance the alignment between the pretext pre-training and downstream fine-tuning stages in self-supervised learning. BiSSL formulates the pretext and downstream task objectives as the lower- and upper-level objectives in a bilevel optimization problem and serves as an intermediate training stage within the self-supervised learning pipeline. By more explicitly modeling the interdependence of these training stages, BiSSL facilitates enhanced information sharing between them, ultimately leading to a backbone parameter initialization that is better suited for the downstream task. We propose a training algorithm that alternates between optimizing the two objectives defined in BiSSL. Using a ResNet-18 backbone pre-trained with SimCLR on the STL10 dataset, we demonstrate that our proposed framework consistently achieves improved or competitive classification accuracies across various downstream image classification datasets compared to the conventional self-supervised learning pipeline. Qualitative analyses of the backbone features further suggest that BiSSL enhances the alignment of downstream features in the backbone prior to fine-tuning.

Title: SynCo: Synthetic Hard Negatives in Contrastive Learning for Better Unsupervised Visual Representations

• Authors: Nikolaos Giakoumoglou, Tania Stathaki
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02401
• Pdf URL: https://arxiv.org/pdf/2410.02401
• Copy Paste: [[2410.02401]] SynCo: Synthetic Hard Negatives in Contrastive Learning for Better Unsupervised Visual Representations(https://arxiv.org/abs/2410.02401)
• Keywords: self-supervised
• Abstract: Contrastive learning has become a dominant approach in self-supervised visual representation learning, with hard negatives-samples that closely resemble the anchor-being key to enhancing the discriminative power of learned representations. However, efficiently leveraging hard negatives remains a challenge due to the difficulty in identifying and incorporating them without significantly increasing computational costs. To address this, we introduce SynCo (Synthetic Negatives in Contrastive learning), a novel contrastive learning approach that improves model performance by generating synthetic hard negatives. Built on the MoCo framework, SynCo introduces six novel strategies for creating diverse synthetic hard negatives that can be generated on-the-fly with minimal computational overhead. SynCo achieves faster training and better representation learning, achieving a top-1 accuracy of 68.1% in ImageNet linear evaluation after only 200 epochs on pretraining, surpassing MoCo's 67.5% with the same ResNet-50 encoder. Additionally, it transfers more effectively to detection tasks: on the PASCAL VOC, it outperforms both the supervised baseline and MoCo, achieving an AP of 82.5%; on the COCO dataset, it sets a new benchmark with 40.4% AP for bounding box detection and 35.4% AP for instance segmentation. Our synthetic hard negative generation procedure significantly enhances the quality of visual representations learned through self-supervised contrastive learning. Code is available at this https URL.

Title: Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models

• Authors: Seyedmorteza Sadat, Otmar Hilliges, Romann M. Weber
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02416
• Pdf URL: https://arxiv.org/pdf/2410.02416
• Copy Paste: [[2410.02416]] Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models(https://arxiv.org/abs/2410.02416)
• Keywords: diffusion
• Abstract: Classifier-free guidance (CFG) is crucial for improving both generation quality and alignment between the input condition and final output in diffusion models. While a high guidance scale is generally required to enhance these aspects, it also causes oversaturation and unrealistic artifacts. In this paper, we revisit the CFG update rule and introduce modifications to address this issue. We first decompose the update term in CFG into parallel and orthogonal components with respect to the conditional model prediction and observe that the parallel component primarily causes oversaturation, while the orthogonal component enhances image quality. Accordingly, we propose down-weighting the parallel component to achieve high-quality generations without oversaturation. Additionally, we draw a connection between CFG and gradient ascent and introduce a new rescaling and momentum method for the CFG update rule based on this insight. Our approach, termed adaptive projected guidance (APG), retains the quality-boosting advantages of CFG while enabling the use of higher guidance scales without oversaturation. APG is easy to implement and introduces practically no additional computational overhead to the sampling process. Through extensive experiments, we demonstrate that APG is compatible with various conditional diffusion models and samplers, leading to improved FID, recall, and saturation scores while maintaining precision comparable to CFG, making our method a superior plug-and-play alternative to standard classifier-free guidance.

Title: PnP-Flow: Plug-and-Play Image Restoration with Flow Matching

• Authors: Ségolène Martin, Anne Gagneux, Paul Hagemann, Gabriele Steidl
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02423
• Pdf URL: https://arxiv.org/pdf/2410.02423
• Copy Paste: [[2410.02423]] PnP-Flow: Plug-and-Play Image Restoration with Flow Matching(https://arxiv.org/abs/2410.02423)
• Keywords: generative
• Abstract: In this paper, we introduce Plug-and-Play (PnP) Flow Matching, an algorithm for solving imaging inverse problems. PnP methods leverage the strength of pre-trained denoisers, often deep neural networks, by integrating them in optimization schemes. While they achieve state-of-the-art performance on various inverse problems in imaging, PnP approaches face inherent limitations on more generative tasks like inpainting. On the other hand, generative models such as Flow Matching pushed the boundary in image sampling yet lack a clear method for efficient use in image restoration. We propose to combine the PnP framework with Flow Matching (FM) by defining a time-dependent denoiser using a pre-trained FM model. Our algorithm alternates between gradient descent steps on the data-fidelity term, reprojections onto the learned FM path, and denoising. Notably, our method is computationally efficient and memory-friendly, as it avoids backpropagation through ODEs and trace computations. We evaluate its performance on denoising, super-resolution, deblurring, and inpainting tasks, demonstrating superior results compared to existing PnP algorithms and Flow Matching based state-of-the-art methods.

Title: Learning the Latent Rules of a Game from Data: A Chess Story

• Authors: Ben Fauber
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02426
• Pdf URL: https://arxiv.org/pdf/2410.02426
• Copy Paste: [[2410.02426]] Learning the Latent Rules of a Game from Data: A Chess Story(https://arxiv.org/abs/2410.02426)
• Keywords: generative
• Abstract: We demonstrate that small pretrained foundational generative language models with millions of parameters can learn the latent rules of a process from data associated with the process. Inspired by Stefan Zweig's novella "Schachnovelle," also known as "The Royal Game" in English, we show that 28M and 125M parameter pretrained foundational small language models (SLMs) can be instruction fine-tuned with 1,000-to-1,000,000 examples to learn the rules of chess, propose legal moves, and accurately solve chess problems. We also explore the impact of successive language model fine-tuning epochs on improved outcomes and demonstrate reductions in model hallucinations by increasing the number of instruction fine-tuning examples.

Title: Personalized Federated Learning for Generative AI-Assisted Semantic Communications

• Authors: Yubo Peng, Feibo Jiang, Li Dong, Kezhi Wang, Kun Yang
• Subjects: cs.LG, cs.DC, cs.IT
• Abstract URL: https://arxiv.org/abs/2410.02450
• Pdf URL: https://arxiv.org/pdf/2410.02450
• Copy Paste: [[2410.02450]] Personalized Federated Learning for Generative AI-Assisted Semantic Communications(https://arxiv.org/abs/2410.02450)
• Keywords: generative
• Abstract: Semantic Communication (SC) focuses on transmitting only the semantic information rather than the raw data. This approach offers an efficient solution to the issue of spectrum resource utilization caused by the various intelligent applications on Mobile Users (MUs). Generative Artificial Intelligence (GAI) models have recently exhibited remarkable content generation and signal processing capabilities, presenting new opportunities for enhancing SC. Therefore, we propose a GAI-assisted SC (GSC) model deployed between MUs and the Base Station (BS). Then, to train the GSC model using the local data of MUs while ensuring privacy and accommodating heterogeneous requirements of MUs, we introduce Personalized Semantic Federated Learning (PSFL). This approach incorporates a novel Personalized Local Distillation (PLD) and Adaptive Global Pruning (AGP). In PLD, each MU selects a personalized GSC model as a mentor tailored to its local resources and a unified Convolutional Neural Networks (CNN)-based SC (CSC) model as a student. This mentor model is then distilled into the student model for global aggregation. In AGP, we perform network pruning on the aggregated global model according to real-time communication environments, reducing communication energy. Finally, numerical results demonstrate the feasibility and efficiency of the proposed PSFL scheme.

Title: Towards a Theoretical Understanding of Memorization in Diffusion Models

• Authors: Yunhao Chen, Xingjun Ma, Difan Zou, Yu-Gang Jiang
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02467
• Pdf URL: https://arxiv.org/pdf/2410.02467
• Copy Paste: [[2410.02467]] Towards a Theoretical Understanding of Memorization in Diffusion Models(https://arxiv.org/abs/2410.02467)
• Keywords: diffusion, generative
• Abstract: As diffusion probabilistic models (DPMs) are being employed as mainstream models for Generative Artificial Intelligence (GenAI), the study of their memorization of training data has attracted growing attention. Existing works in this direction aim to establish an understanding of whether or to what extent DPMs learn via memorization. Such an understanding is crucial for identifying potential risks of data leakage and copyright infringement in diffusion models and, more importantly, for trustworthy application of GenAI. Existing works revealed that conditional DPMs are more prone to training data memorization than unconditional DPMs, and the motivated data extraction methods are mostly for conditional DPMs. However, these understandings are primarily empirical, and extracting training data from unconditional models has been found to be extremely challenging. In this work, we provide a theoretical understanding of memorization in both conditional and unconditional DPMs under the assumption of model convergence. Our theoretical analysis indicates that extracting data from unconditional models can also be effective by constructing a proper surrogate condition. Based on this result, we propose a novel data extraction method named \textbf{Surrogate condItional Data Extraction (SIDE)} that leverages a time-dependent classifier trained on the generated data as a surrogate condition to extract training data from unconditional DPMs. Empirical results demonstrate that our SIDE can extract training data in challenging scenarios where previous methods fail, and it is, on average, over 50\% more effective across different scales of the CelebA dataset.

Title: Event-Customized Image Generation

• Authors: Zhen Wang, Yilei Jiang, Dong Zheng, Jun Xiao, Long Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02483
• Pdf URL: https://arxiv.org/pdf/2410.02483
• Copy Paste: [[2410.02483]] Event-Customized Image Generation(https://arxiv.org/abs/2410.02483)
• Keywords: diffusion
• Abstract: Customized Image Generation, generating customized images with user-specified concepts, has raised significant attention due to its creativity and novelty. With impressive progress achieved in subject customization, some pioneer works further explored the customization of action and interaction beyond entity (i.e., human, animal, and object) appearance. However, these approaches only focus on basic actions and interactions between two entities, and their effects are limited by insufficient ''exactly same'' reference images. To extend customized image generation to more complex scenes for general real-world applications, we propose a new task: event-customized image generation. Given a single reference image, we define the ''event'' as all specific actions, poses, relations, or interactions between different entities in the scene. This task aims at accurately capturing the complex event and generating customized images with various target entities. To solve this task, we proposed a novel training-free event customization method: FreeEvent. Specifically, FreeEvent introduces two extra paths alongside the general diffusion denoising process: 1) Entity switching path: it applies cross-attention guidance and regulation for target entity generation. 2) Event transferring path: it injects the spatial feature and self-attention maps from the reference image to the target image for event generation. To further facilitate this new task, we collected two evaluation benchmarks: SWiG-Event and Real-Event. Extensive experiments and ablations have demonstrated the effectiveness of FreeEvent.

Title: SAFLEX: Self-Adaptive Augmentation via Feature Label Extrapolation

• Authors: Mucong Ding, Bang An, Yuancheng Xu, Anirudh Satheesh, Furong Huang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02512
• Pdf URL: https://arxiv.org/pdf/2410.02512
• Copy Paste: [[2410.02512]] SAFLEX: Self-Adaptive Augmentation via Feature Label Extrapolation(https://arxiv.org/abs/2410.02512)
• Keywords: diffusion, generative
• Abstract: Data augmentation, a cornerstone technique in deep learning, is crucial in enhancing model performance, especially with scarce labeled data. While traditional techniques are effective, their reliance on hand-crafted methods limits their applicability across diverse data types and tasks. Although modern learnable augmentation methods offer increased adaptability, they are computationally expensive and challenging to incorporate within prevalent augmentation workflows. In this work, we present a novel, efficient method for data augmentation, effectively bridging the gap between existing augmentation strategies and emerging datasets and learning tasks. We introduce SAFLEX (Self-Adaptive Augmentation via Feature Label EXtrapolation), which learns the sample weights and soft labels of augmented samples provided by any given upstream augmentation pipeline, using a specifically designed efficient bilevel optimization algorithm. Remarkably, SAFLEX effectively reduces the noise and label errors of the upstream augmentation pipeline with a marginal computational cost. As a versatile module, SAFLEX excels across diverse datasets, including natural and medical images and tabular data, showcasing its prowess in few-shot learning and out-of-distribution generalization. SAFLEX seamlessly integrates with common augmentation strategies like RandAug, CutMix, and those from large pre-trained generative models like stable diffusion and is also compatible with frameworks such as CLIP's fine-tuning. Our findings highlight the potential to adapt existing augmentation pipelines for new data types and tasks, signaling a move towards more adaptable and resilient training frameworks.

Title: Learning from Offline Foundation Features with Tensor Augmentations

• Authors: Emir Konuk, Christos Matsoukas, Moein Sorkhei, Phitchapha Lertsiravaramet, Kevin Smith
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02527
• Pdf URL: https://arxiv.org/pdf/2410.02527
• Copy Paste: [[2410.02527]] Learning from Offline Foundation Features with Tensor Augmentations(https://arxiv.org/abs/2410.02527)
• Keywords: foundation model
• Abstract: We introduce Learning from Offline Foundation Features with Tensor Augmentations (LOFF-TA), an efficient training scheme designed to harness the capabilities of foundation models in limited resource settings where their direct development is not feasible. LOFF-TA involves training a compact classifier on cached feature embeddings from a frozen foundation model, resulting in up to $37\times$ faster training and up to $26\times$ reduced GPU memory usage. Because the embeddings of augmented images would be too numerous to store, yet the augmentation process is essential for training, we propose to apply tensor augmentations to the cached embeddings of the original non-augmented images. LOFF-TA makes it possible to leverage the power of foundation models, regardless of their size, in settings with limited computational capacity. Moreover, LOFF-TA can be used to apply foundation models to high-resolution images without increasing compute. In certain scenarios, we find that training with LOFF-TA yields better results than directly fine-tuning the foundation model.

Title: Pseudo-Stereo Inputs: A Solution to the Occlusion Challenge in Self-Supervised Stereo Matching

• Authors: Ruizhi Yang, Xingqiang Li, Jiajun Bai, Jinsong Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02534
• Pdf URL: https://arxiv.org/pdf/2410.02534
• Copy Paste: [[2410.02534]] Pseudo-Stereo Inputs: A Solution to the Occlusion Challenge in Self-Supervised Stereo Matching(https://arxiv.org/abs/2410.02534)
• Keywords: self-supervised
• Abstract: Self-supervised stereo matching holds great promise for application and research due to its independence from expensive labeled data. However, direct self-supervised stereo matching paradigms based on photometric loss functions have consistently struggled with performance issues due to the occlusion challenge. The crux of the occlusion challenge lies in the fact that the positions of occluded pixels consistently align with the epipolar search direction defined by the input stereo images, leading to persistent information loss and erroneous feedback at fixed locations during self-supervised training. In this work, we propose a simple yet highly effective pseudo-stereo inputs strategy to address the core occlusion challenge. This strategy decouples the input and feedback images, compelling the network to probabilistically sample information from both sides of the occluding objects. As a result, the persistent lack of information in the aforementioned fixed occlusion areas is mitigated. Building upon this, we further address feedback conflicts and overfitting issues arising from the strategy. By integrating these components, our method achieves stable and significant performance improvements compared to existing methods. Quantitative experiments are conducted to evaluate the performance. Qualitative experiments further demonstrate accurate disparity inference even at occluded regions. These results demonstrate a significant advancement over previous methods in the field of direct self-supervised stereo matching based on photometric loss. The proposed pseudo-stereo inputs strategy, due to its simplicity and effectiveness, has the potential to serve as a new paradigm for direct self-supervised stereo matching. Code is available at this https URL.

Title: Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions

• Authors: Angana Borah, Rada Mihalcea
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2410.02584
• Pdf URL: https://arxiv.org/pdf/2410.02584
• Copy Paste: [[2410.02584]] Towards Implicit Bias Detection and Mitigation in Multi-Agent LLM Interactions(https://arxiv.org/abs/2410.02584)
• Keywords: in-context
• Abstract: As Large Language Models (LLMs) continue to evolve, they are increasingly being employed in numerous studies to simulate societies and execute diverse social tasks. However, LLMs are susceptible to societal biases due to their exposure to human-generated data. Given that LLMs are being used to gain insights into various societal aspects, it is essential to mitigate these biases. To that end, our study investigates the presence of implicit gender biases in multi-agent LLM interactions and proposes two strategies to mitigate these biases. We begin by creating a dataset of scenarios where implicit gender biases might arise, and subsequently develop a metric to assess the presence of biases. Our empirical analysis reveals that LLMs generate outputs characterized by strong implicit bias associations (>= 50\% of the time). Furthermore, these biases tend to escalate following multi-agent interactions. To mitigate them, we propose two strategies: self-reflection with in-context examples (ICE); and supervised fine-tuning. Our research demonstrates that both methods effectively mitigate implicit biases, with the ensemble of fine-tuning and self-reflection proving to be the most successful.

Title: Beyond Squared Error: Exploring Loss Design for Enhanced Training of Generative Flow Networks

• Authors: Rui Hu, Yifan Zhang, Zhuoran Li, Longbo Huang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02596
• Pdf URL: https://arxiv.org/pdf/2410.02596
• Copy Paste: [[2410.02596]] Beyond Squared Error: Exploring Loss Design for Enhanced Training of Generative Flow Networks(https://arxiv.org/abs/2410.02596)
• Keywords: generative
• Abstract: Generative Flow Networks (GFlowNets) are a novel class of generative models designed to sample from unnormalized distributions and have found applications in various important tasks, attracting great research interest in their training algorithms. In general, GFlowNets are trained by fitting the forward flow to the backward flow on sampled training objects. Prior work focused on the choice of training objects, parameterizations, sampling and resampling strategies, and backward policies, aiming to enhance credit assignment, exploration, or exploitation of the training process. However, the choice of regression loss, which can highly influence the exploration and exploitation behavior of the under-training policy, has been overlooked. Due to the lack of theoretical understanding for choosing an appropriate regression loss, most existing algorithms train the flow network by minimizing the squared error of the forward and backward flows in log-space, i.e., using the quadratic regression loss. In this work, we rigorously prove that distinct regression losses correspond to specific divergence measures, enabling us to design and analyze regression losses according to the desired properties of the corresponding divergence measures. Specifically, we examine two key properties: zero-forcing and zero-avoiding, where the former promotes exploitation and higher rewards, and the latter encourages exploration and enhances diversity. Based on our theoretical framework, we propose three novel regression losses, namely, Shifted-Cosh, Linex(1/2), and Linex(1). We evaluate them across three benchmarks: hyper-grid, bit-sequence generation, and molecule generation. Our proposed losses are compatible with most existing training algorithms, and significantly improve the performances of the algorithms concerning convergence speed, sample diversity, and robustness.

Title: Diffusion & Adversarial Schr\"odinger Bridges via Iterative Proportional Markovian Fitting

• Authors: Sergei Kholkin, Grigoriy Ksenofontov, David Li, Nikita Kornilov, Nikita Gushchin, Evgeny Burnaev, Alexander Korotin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02601
• Pdf URL: https://arxiv.org/pdf/2410.02601
• Copy Paste: [[2410.02601]] Diffusion & Adversarial Schr\"odinger Bridges via Iterative Proportional Markovian Fitting(https://arxiv.org/abs/2410.02601)
• Keywords: diffusion
• Abstract: The Iterative Markovian Fitting (IMF) procedure based on iterative reciprocal and Markovian projections has recently been proposed as a powerful method for solving the Schrödinger Bridge problem. However, it has been observed that for the practical implementation of this procedure, it is crucial to alternate between fitting a forward and backward time diffusion at each iteration. Such implementation is thought to be a practical heuristic, which is required to stabilize training and obtain good results in applications such as unpaired domain translation. In our work, we show that this heuristic closely connects with the pioneer approaches for the Schrödinger Bridge based on the Iterative Proportional Fitting (IPF) procedure. Namely, we find that the practical implementation of IMF is, in fact, a combination of IMF and IPF procedures, and we call this combination the Iterative Proportional Markovian Fitting (IPMF) procedure. We show both theoretically and practically that this combined IPMF procedure can converge under more general settings, thus, showing that the IPMF procedure opens a door towards developing a unified framework for solving Schrödinger Bridge problems.

Title: Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers

• Authors: Shijie Chen, Bernal Jiménez Gutiérrez, Yu Su
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2410.02642
• Pdf URL: https://arxiv.org/pdf/2410.02642
• Copy Paste: [[2410.02642]] Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers(https://arxiv.org/abs/2410.02642)
• Keywords: generative, in-context
• Abstract: Information retrieval (IR) systems have played a vital role in modern digital life and have cemented their continued usefulness in this new era of generative AI via retrieval-augmented generation. With strong language processing capabilities and remarkable versatility, large language models (LLMs) have become popular choices for zero-shot re-ranking in IR systems. So far, LLM-based re-ranking methods rely on strong generative capabilities, which restricts their use to either specialized or powerful proprietary models. Given these restrictions, we ask: is autoregressive generation necessary and optimal for LLMs to perform re-ranking? We hypothesize that there are abundant signals relevant to re-ranking within LLMs that might not be used to their full potential via generation. To more directly leverage such signals, we propose in-context re-ranking (ICR), a novel method that leverages the change in attention pattern caused by the search query for accurate and efficient re-ranking. To mitigate the intrinsic biases in LLMs, we propose a calibration method using a content-free query. Due to the absence of generation, ICR only requires two ($O(1)$) forward passes to re-rank $N$ documents, making it substantially more efficient than generative re-ranking methods that require at least $O(N)$ forward passes. Our novel design also enables ICR to be applied to any LLM without specialized training while guaranteeing a well-formed ranking. Extensive experiments with two popular open-weight LLMs on standard single-hop and multi-hop information retrieval benchmarks show that ICR outperforms RankGPT while cutting the latency by more than 60% in practice. Through detailed analyses, we show that ICR's performance is specially strong on tasks that require more complex re-ranking signals. Our findings call for further exploration on novel ways of utilizing open-weight LLMs beyond text generation.

Title: Undesirable Memorization in Large Language Models: A Survey

• Authors: Ali Satvaty, Suzan Verberne, Fatih Turkmen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02650
• Pdf URL: https://arxiv.org/pdf/2410.02650
• Copy Paste: [[2410.02650]] Undesirable Memorization in Large Language Models: A Survey(https://arxiv.org/abs/2410.02650)
• Keywords: diffusion
• Abstract: While recent research increasingly showcases the remarkable capabilities of Large Language Models (LLMs), it's vital to confront their hidden pitfalls. Among these challenges, the issue of memorization stands out, posing significant ethical and legal risks. In this paper, we presents a Systematization of Knowledge (SoK) on the topic of memorization in LLMs. Memorization is the effect that a model tends to store and reproduce phrases or passages from the training data and has been shown to be the fundamental issue to various privacy and security attacks against LLMs. We begin by providing an overview of the literature on the memorization, exploring it across five key dimensions: intentionality, degree, retrievability, abstraction, and transparency. Next, we discuss the metrics and methods used to measure memorization, followed by an analysis of the factors that contribute to memorization phenomenon. We then examine how memorization manifests itself in specific model architectures and explore strategies for mitigating these effects. We conclude our overview by identifying potential research topics for the near future: to develop methods for balancing performance and privacy in LLMs, and the analysis of memorization in specific contexts, including conversational agents, retrieval-augmented generation, multilingual language models, and diffusion language models.

Title: Measuring and Improving Persuasiveness of Generative Models

• Authors: Somesh Singh, Yaman K Singla, Harini SI, Balaji Krishnamurthy
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02653
• Pdf URL: https://arxiv.org/pdf/2410.02653
• Copy Paste: [[2410.02653]] Measuring and Improving Persuasiveness of Generative Models(https://arxiv.org/abs/2410.02653)
• Keywords: generative
• Abstract: LLMs are increasingly being used in workflows involving generating content to be consumed by humans (e.g., marketing) and also in directly interacting with humans (e.g., through chatbots). The development of such systems that are capable of generating verifiably persuasive messages presents both opportunities and challenges for society. On the one hand, such systems could positively impact domains like advertising and social good, such as addressing drug addiction, and on the other, they could be misused for spreading misinformation and shaping political opinions. To channel LLMs' impact on society, we need to develop systems to measure and benchmark their persuasiveness. With this motivation, we introduce PersuasionBench and PersuasionArena, the first large-scale benchmark and arena containing a battery of tasks to measure the persuasion ability of generative models automatically. We investigate to what extent LLMs know and leverage linguistic patterns that can help them generate more persuasive language. Our findings indicate that the persuasiveness of LLMs correlates positively with model size, but smaller models can also be made to have a higher persuasiveness than much larger models. Notably, targeted training using synthetic and natural datasets significantly enhances smaller models' persuasive capabilities, challenging scale-dependent assumptions. Our findings carry key implications for both model developers and policymakers. For instance, while the EU AI Act and California's SB-1047 aim to regulate AI models based on the number of floating point operations, we demonstrate that simple metrics like this alone fail to capture the full scope of AI's societal impact. We invite the community to explore and contribute to PersuasionArena and PersuasionBench, available at this https URL, to advance our understanding of AI-driven persuasion and its societal implications.

Title: Scalable Simulation-free Entropic Unbalanced Optimal Transport

• Authors: Jaemoo Choi, Jaewoong Choi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02656
• Pdf URL: https://arxiv.org/pdf/2410.02656
• Copy Paste: [[2410.02656]] Scalable Simulation-free Entropic Unbalanced Optimal Transport(https://arxiv.org/abs/2410.02656)
• Keywords: generative
• Abstract: The Optimal Transport (OT) problem investigates a transport map that connects two distributions while minimizing a given cost function. Finding such a transport map has diverse applications in machine learning, such as generative modeling and image-to-image translation. In this paper, we introduce a scalable and simulation-free approach for solving the Entropic Unbalanced Optimal Transport (EUOT) problem. We derive the dynamical form of this EUOT problem, which is a generalization of the Schrödinger bridges (SB) problem. Based on this, we derive dual formulation and optimality conditions of the EUOT problem from the stochastic optimal control interpretation. By leveraging these properties, we propose a simulation-free algorithm to solve EUOT, called Simulation-free EUOT (SF-EUOT). While existing SB models require expensive simulation costs during training and evaluation, our model achieves simulation-free training and one-step generation by utilizing the reciprocal property. Our model demonstrates significantly improved scalability in generative modeling and image-to-image translation tasks compared to previous SB methods.

Title: GUD: Generation with Unified Diffusion

• Authors: Mathis Gerdes, Max Welling, Miranda C. N. Cheng
• Subjects: cs.LG, hep-th, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.02667
• Pdf URL: https://arxiv.org/pdf/2410.02667
• Copy Paste: [[2410.02667]] GUD: Generation with Unified Diffusion(https://arxiv.org/abs/2410.02667)
• Keywords: diffusion, generative
• Abstract: Diffusion generative models transform noise into data by inverting a process that progressively adds noise to data samples. Inspired by concepts from the renormalization group in physics, which analyzes systems across different scales, we revisit diffusion models by exploring three key design aspects: 1) the choice of representation in which the diffusion process operates (e.g. pixel-, PCA-, Fourier-, or wavelet-basis), 2) the prior distribution that data is transformed into during diffusion (e.g. Gaussian with covariance $\Sigma$), and 3) the scheduling of noise levels applied separately to different parts of the data, captured by a component-wise noise schedule. Incorporating the flexibility in these choices, we develop a unified framework for diffusion generative models with greatly enhanced design freedom. In particular, we introduce soft-conditioning models that smoothly interpolate between standard diffusion models and autoregressive models (in any basis), conceptually bridging these two approaches. Our framework opens up a wide design space which may lead to more efficient training and data generation, and paves the way to novel architectures integrating different generative approaches and generation tasks.

Title: ControlAR: Controllable Image Generation with Autoregressive Models

• Authors: Zongming Li, Tianheng Cheng, Shoufa Chen, Peize Sun, Haocheng Shen, Longjin Ran, Xiaoxin Chen, Wenyu Liu, Xinggang Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.02705
• Pdf URL: https://arxiv.org/pdf/2410.02705
• Copy Paste: [[2410.02705]] ControlAR: Controllable Image Generation with Autoregressive Models(https://arxiv.org/abs/2410.02705)
• Keywords: diffusion
• Abstract: Autoregressive (AR) models have reformulated image generation as next-token prediction, demonstrating remarkable potential and emerging as strong competitors to diffusion models. However, control-to-image generation, akin to ControlNet, remains largely unexplored within AR models. Although a natural approach, inspired by advancements in Large Language Models, is to tokenize control images into tokens and prefill them into the autoregressive model before decoding image tokens, it still falls short in generation quality compared to ControlNet and suffers from inefficiency. To this end, we introduce ControlAR, an efficient and effective framework for integrating spatial controls into autoregressive image generation models. Firstly, we explore control encoding for AR models and propose a lightweight control encoder to transform spatial inputs (e.g., canny edges or depth maps) into control tokens. Then ControlAR exploits the conditional decoding method to generate the next image token conditioned on the per-token fusion between control and image tokens, similar to positional encodings. Compared to prefilling tokens, using conditional decoding significantly strengthens the control capability of AR models but also maintains the model's efficiency. Furthermore, the proposed ControlAR surprisingly empowers AR models with arbitrary-resolution image generation via conditional decoding and specific controls. Extensive experiments can demonstrate the controllability of the proposed ControlAR for the autoregressive control-to-image generation across diverse inputs, including edges, depths, and segmentation masks. Furthermore, both quantitative and qualitative results indicate that ControlAR surpasses previous state-of-the-art controllable diffusion models, e.g., ControlNet++. Code, models, and demo will soon be available at this https URL.

Title: SteerDiff: Steering towards Safe Text-to-Image Diffusion Models

• Authors: Hongxiang Zhang, Yifeng He, Hao Chen
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2410.02710
• Pdf URL: https://arxiv.org/pdf/2410.02710
• Copy Paste: [[2410.02710]] SteerDiff: Steering towards Safe Text-to-Image Diffusion Models(https://arxiv.org/abs/2410.02710)
• Keywords: diffusion
• Abstract: Text-to-image (T2I) diffusion models have drawn attention for their ability to generate high-quality images with precise text alignment. However, these models can also be misused to produce inappropriate content. Existing safety measures, which typically rely on text classifiers or ControlNet-like approaches, are often insufficient. Traditional text classifiers rely on large-scale labeled datasets and can be easily bypassed by rephrasing. As diffusion models continue to scale, fine-tuning these safeguards becomes increasingly challenging and lacks flexibility. Recent red-teaming attack researches further underscore the need for a new paradigm to prevent the generation of inappropriate content. In this paper, we introduce SteerDiff, a lightweight adaptor module designed to act as an intermediary between user input and the diffusion model, ensuring that generated images adhere to ethical and safety standards with little to no impact on usability. SteerDiff identifies and manipulates inappropriate concepts within the text embedding space to guide the model away from harmful outputs. We conduct extensive experiments across various concept unlearning tasks to evaluate the effectiveness of our approach. Furthermore, we benchmark SteerDiff against multiple red-teaming strategies to assess its robustness. Finally, we explore the potential of SteerDiff for concept forgetting tasks, demonstrating its versatility in text-conditioned image generation.

Title: NETS: A Non-Equilibrium Transport Sampler

• Authors: Michael S. Albergo, Eric Vanden-Eijnden
• Subjects: cs.LG, cond-mat.stat-mech, hep-lat
• Abstract URL: https://arxiv.org/abs/2410.02711
• Pdf URL: https://arxiv.org/pdf/2410.02711
• Copy Paste: [[2410.02711]] NETS: A Non-Equilibrium Transport Sampler(https://arxiv.org/abs/2410.02711)
• Keywords: diffusion
• Abstract: We propose an algorithm, termed the Non-Equilibrium Transport Sampler (NETS), to sample from unnormalized probability distributions. NETS can be viewed as a variant of annealed importance sampling (AIS) based on Jarzynski's equality, in which the stochastic differential equation used to perform the non-equilibrium sampling is augmented with an additional learned drift term that lowers the impact of the unbiasing weights used in AIS. We show that this drift is the minimizer of a variety of objective functions, which can all be estimated in an unbiased fashion without backpropagating through solutions of the stochastic differential equations governing the sampling. We also prove that some these objectives control the Kullback-Leibler divergence of the estimated distribution from its target. NETS is shown to be unbiased and, in addition, has a tunable diffusion coefficient which can be adjusted post-training to maximize the effective sample size. We demonstrate the efficacy of the method on standard benchmarks, high-dimensional Gaussian mixture distributions, and a model from statistical lattice field theory, for which it surpasses the performances of related work and existing baselines.

Title: SynthFormer: Equivariant Pharmacophore-based Generation of Molecules for Ligand-Based Drug Design

• Authors: Zygimantas Jocys, Henriette M.G. Willems, Katayoun Farrahi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02718
• Pdf URL: https://arxiv.org/pdf/2410.02718
• Copy Paste: [[2410.02718]] SynthFormer: Equivariant Pharmacophore-based Generation of Molecules for Ligand-Based Drug Design(https://arxiv.org/abs/2410.02718)
• Keywords: generative
• Abstract: Drug discovery is a complex and resource-intensive process, with significant time and cost investments required to bring new medicines to patients. Recent advancements in generative machine learning (ML) methods offer promising avenues to accelerate early-stage drug discovery by efficiently exploring chemical space. This paper addresses the gap between in silico generative approaches and practical in vitro methodologies, highlighting the need for their integration to optimize molecule discovery. We introduce SynthFormer, a novel ML model that utilizes a 3D equivariant encoder for pharmacophores to generate fully synthesizable molecules, constructed as synthetic trees. Unlike previous methods, SynthFormer incorporates 3D information and provides synthetic paths, enhancing its ability to produce molecules with good docking scores across various proteins. Our contributions include a new methodology for efficient chemical space exploration using 3D information, a novel architecture called Synthformer for translating 3D pharmacophore representations into molecules, and a meaningful embedding space that organizes reagents for drug discovery optimization. Synthformer generates molecules that dock well and enables effective late-stage optimization restricted by synthesis paths.

Title: Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorization

• Authors: Ryan C. Barron, Ves Grantcharov, Selma Wanna, Maksim E. Eren, Manish Bhattarai, Nicholas Solovyev, George Tompkins, Charles Nicholas, Kim Ø. Rasmussen, Cynthia Matuszek, Boian S. Alexandrov
• Subjects: cs.CL, cs.AI, cs.IR, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.02721
• Pdf URL: https://arxiv.org/pdf/2410.02721
• Copy Paste: [[2410.02721]] Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorization(https://arxiv.org/abs/2410.02721)
• Keywords: anomaly
• Abstract: Large Language Models (LLMs) are pre-trained on large-scale corpora and excel in numerous general natural language processing (NLP) tasks, such as question answering (QA). Despite their advanced language capabilities, when it comes to domain-specific and knowledge-intensive tasks, LLMs suffer from hallucinations, knowledge cut-offs, and lack of knowledge attributions. Additionally, fine tuning LLMs' intrinsic knowledge to highly specific domains is an expensive and time consuming process. The retrieval-augmented generation (RAG) process has recently emerged as a method capable of optimization of LLM responses, by referencing them to a predetermined ontology. It was shown that using a Knowledge Graph (KG) ontology for RAG improves the QA accuracy, by taking into account relevant sub-graphs that preserve the information in a structured manner. In this paper, we introduce SMART-SLIC, a highly domain-specific LLM framework, that integrates RAG with KG and a vector store (VS) that store factual domain specific information. Importantly, to avoid hallucinations in the KG, we build these highly domain-specific KGs and VSs without the use of LLMs, but via NLP, data mining, and nonnegative tensor factorization with automatic model selection. Pairing our RAG with a domain-specific: (i) KG (containing structured information), and (ii) VS (containing unstructured information) enables the development of domain-specific chat-bots that attribute the source of information, mitigate hallucinations, lessen the need for fine-tuning, and excel in highly domain-specific question answering tasks. We pair SMART-SLIC with chain-of-thought prompting agents. The framework is designed to be generalizable to adapt to any specific or specialized domain. In this paper, we demonstrate the question answering capabilities of our framework on a corpus of scientific publications on malware analysis and anomaly detection.

Title: Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation

• Authors: Rohin Manvi, Anikait Singh, Stefano Ermon
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02725
• Pdf URL: https://arxiv.org/pdf/2410.02725
• Copy Paste: [[2410.02725]] Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation(https://arxiv.org/abs/2410.02725)
• Keywords: generative
• Abstract: Inference-time computation is a powerful paradigm to enhance the performance of large language models (LLMs), with Best-of-N sampling being a widely used technique. However, this method is computationally expensive, requiring both (1) an external reward model and (2) the generation of multiple samples. In this work, we introduce a new generative self-evaluation scheme designed to adaptively reduce the number of generated samples while maintaining or even improving performance. We use a generative reward model formulation, allowing the LLM to predict mid-generation the probability that restarting the generation will yield a better response. These predictions are obtained without an external reward model and can be used to decide whether or not to generate more samples, prune unpromising samples early on, or to pick the best sample. This capability is very inexpensive as it involves generating a single predefined token. Trained using a dataset constructed with real unfiltered LMSYS user prompts, Llama 3.1 8B's win rate against GPT-4 on AlpacaEval increases from 21% to 34% with 16 samples and math performance on GSM8K improves from 84% to 91%. By sampling only when the LLM determines that it is beneficial to do so and adaptively adjusting temperature annealing, we demonstrate that 74% of the improvement from using 16 samples can be achieved with only 1.2 samples on average. We further demonstrate that 50-75% of samples can be pruned early in generation with minimal degradation in performance. Overall, our methods enable more efficient and scalable compute utilization during inference for LLMs.

Title: Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models

• Authors: Zhengfeng Lai, Vasileios Saveris, Chen Chen, Hong-You Chen, Haotian Zhang, Bowen Zhang, Juan Lao Tebar, Wenze Hu, Zhe Gan, Peter Grasch, Meng Cao, Yinfei Yang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02740
• Pdf URL: https://arxiv.org/pdf/2410.02740
• Copy Paste: [[2410.02740]] Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models(https://arxiv.org/abs/2410.02740)
• Keywords: diffusion, foundation model
• Abstract: Recent advancements in multimodal models highlight the value of rewritten captions for improving performance, yet key challenges remain. For example, while synthetic captions often provide superior quality and image-text alignment, it is not clear whether they can fully replace AltTexts: the role of synthetic captions and their interaction with original web-crawled AltTexts in pre-training is still not well understood. Moreover, different multimodal foundation models may have unique preferences for specific caption formats, but efforts to identify the optimal captions for each model remain limited. In this work, we propose a novel, controllable, and scalable captioning pipeline designed to generate diverse caption formats tailored to various multimodal models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic Captions (DSC+) as case studies, we systematically explore their effects and interactions with AltTexts across models such as CLIP, multimodal LLMs, and diffusion models. Our findings reveal that a hybrid approach that keeps both synthetic captions and AltTexts can outperform the use of synthetic captions alone, improving both alignment and performance, with each model demonstrating preferences for particular caption formats. This comprehensive analysis provides valuable insights into optimizing captioning strategies, thereby advancing the pre-training of multimodal foundation models.

Title: Contrastive Localized Language-Image Pre-Training

• Authors: Hong-You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang, Marcin Eichner, Keen You, Meng Cao, Bowen Zhang, Yinfei Yang, Zhe Gan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02746
• Pdf URL: https://arxiv.org/pdf/2410.02746
• Copy Paste: [[2410.02746]] Contrastive Localized Language-Image Pre-Training(https://arxiv.org/abs/2410.02746)
• Keywords: foundation model
• Abstract: Contrastive Language-Image Pre-training (CLIP) has been a celebrated method for training vision encoders to generate image/text representations facilitating various applications. Recently, CLIP has been widely adopted as the vision backbone of multimodal large language models (MLLMs) to connect image inputs for language interactions. The success of CLIP as a vision-language foundation model relies on aligning web-crawled noisy text annotations at image levels. Nevertheless, such criteria may become insufficient for downstream tasks in need of fine-grained vision representations, especially when region-level understanding is demanding for MLLMs. In this paper, we improve the localization capability of CLIP with several advances. We propose a pre-training method called Contrastive Localized Language-Image Pre-training (CLOC) by complementing CLIP with region-text contrastive loss and modules. We formulate a new concept, promptable embeddings, of which the encoder produces image embeddings easy to transform into region representations given spatial hints. To support large-scale pre-training, we design a visually-enriched and spatially-localized captioning framework to effectively generate region-text pseudo-labels at scale. By scaling up to billions of annotated images, CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks.

Title: ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AI

• Authors: Ahmad Elawady, Gunjan Chhablani, Ram Ramrakhya, Karmesh Yadav, Dhruv Batra, Zsolt Kira, Andrew Szot
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.02751
• Pdf URL: https://arxiv.org/pdf/2410.02751
• Copy Paste: [[2410.02751]] ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AI(https://arxiv.org/abs/2410.02751)
• Keywords: in-context
• Abstract: Intelligent embodied agents need to quickly adapt to new scenarios by integrating long histories of experience into decision-making. For instance, a robot in an unfamiliar house initially wouldn't know the locations of objects needed for tasks and might perform inefficiently. However, as it gathers more experience, it should learn the layout of its environment and remember where objects are, allowing it to complete new tasks more efficiently. To enable such rapid adaptation to new tasks, we present ReLIC, a new approach for in-context reinforcement learning (RL) for embodied agents. With ReLIC, agents are capable of adapting to new environments using 64,000 steps of in-context experience with full attention while being trained through self-generated experience via RL. We achieve this by proposing a novel policy update scheme for on-policy RL called "partial updates'' as well as a Sink-KV mechanism that enables effective utilization of a long observation history for embodied agents. Our method outperforms a variety of meta-RL baselines in adapting to unseen houses in an embodied multi-object navigation task. In addition, we find that ReLIC is capable of few-shot imitation learning despite never being trained with expert demonstrations. We also provide a comprehensive analysis of ReLIC, highlighting that the combination of large-scale RL training, the proposed partial updates scheme, and the Sink-KV are essential for effective in-context learning. The code for ReLIC and all our experiments is at this https URL

Title: FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Models

• Authors: Zhipei Xu, Xuanyu Zhang, Runyi Li, Zecheng Tang, Qing Huang, Jian Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.02761
• Pdf URL: https://arxiv.org/pdf/2410.02761
• Copy Paste: [[2410.02761]] FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Models(https://arxiv.org/abs/2410.02761)
• Keywords: generative
• Abstract: The rapid development of generative AI is a double-edged sword, which not only facilitates content creation but also makes image manipulation easier and more difficult to detect. Although current image forgery detection and localization (IFDL) methods are generally effective, they tend to face two challenges: \textbf{1)} black-box nature with unknown detection principle, \textbf{2)} limited generalization across diverse tampering methods (e.g., Photoshop, DeepFake, AIGC-Editing). To address these issues, we propose the explainable IFDL task and design FakeShield, a multi-modal framework capable of evaluating image authenticity, generating tampered region masks, and providing a judgment basis based on pixel-level and image-level tampering clues. Additionally, we leverage GPT-4o to enhance existing IFDL datasets, creating the Multi-Modal Tamper Description dataSet (MMTD-Set) for training FakeShield's tampering analysis capabilities. Meanwhile, we incorporate a Domain Tag-guided Explainable Forgery Detection Module (DTE-FDM) and a Multi-modal Forgery Localization Module (MFLM) to address various types of tamper detection interpretation and achieve forgery localization guided by detailed textual descriptions. Extensive experiments demonstrate that FakeShield effectively detects and localizes various tampering techniques, offering an explainable and superior solution compared to previous IFDL methods.