2024-10-28

Title: VehicleSDF: A 3D generative model for constrained engineering design via surrogate modeling

• Authors: Hayata Morita, Kohei Shintani, Chenyang Yuan, Frank Permenter
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18986
• Pdf URL: https://arxiv.org/pdf/2410.18986
• Copy Paste: [[2410.18986]] VehicleSDF: A 3D generative model for constrained engineering design via surrogate modeling(https://arxiv.org/abs/2410.18986)
• Keywords: generative
• Abstract: A main challenge in mechanical design is to efficiently explore the design space while satisfying engineering constraints. This work explores the use of 3D generative models to explore the design space in the context of vehicle development, while estimating and enforcing engineering constraints. Specifically, we generate diverse 3D models of cars that meet a given set of geometric specifications, while also obtaining quick estimates of performance parameters such as aerodynamic drag. For this, we employ a data-driven approach (using the ShapeNet dataset) to train VehicleSDF, a DeepSDF based model that represents potential designs in a latent space witch can be decoded into a 3D model. We then train surrogate models to estimate engineering parameters from this latent space representation, enabling us to efficiently optimize latent vectors to match specifications. Our experiments show that we can generate diverse 3D models while matching the specified geometric parameters. Finally, we demonstrate that other performance parameters such as aerodynamic drag can be estimated in a differentiable pipeline.

Title: Generative Topology for Shape Synthesis

• Authors: Ernst Röell, Bastian Rieck
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.18987
• Pdf URL: https://arxiv.org/pdf/2410.18987
• Copy Paste: [[2410.18987]] Generative Topology for Shape Synthesis(https://arxiv.org/abs/2410.18987)
• Keywords: generative
• Abstract: The Euler Characteristic Transform (ECT) is a powerful invariant for assessing geometrical and topological characteristics of a large variety of objects, including graphs and embedded simplicial complexes. Although the ECT is invertible in theory, no explicit algorithm for general data sets exists. In this paper, we address this lack and demonstrate that it is possible to learn the inversion, permitting us to develop a novel framework for shape generation tasks on point clouds. Our model exhibits high quality in reconstruction and generation tasks, affords efficient latent-space interpolation, and is orders of magnitude faster than existing methods.

Title: Dual Space Training for GANs: A Pathway to Efficient and Creative Generative Models

• Authors: Beka Modrekiladze
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.19009
• Pdf URL: https://arxiv.org/pdf/2410.19009
• Copy Paste: [[2410.19009]] Dual Space Training for GANs: A Pathway to Efficient and Creative Generative Models(https://arxiv.org/abs/2410.19009)
• Keywords: generative
• Abstract: Generative Adversarial Networks (GANs) have demonstrated remarkable advancements in generative modeling; however, their training is often resource-intensive, requiring extensive computational time and hundreds of thousands of epochs. This paper proposes a novel optimization approach that transforms the training process by operating within a dual space of the initial data using invertible mappings, specifically autoencoders. By training GANs on the encoded representations in the dual space, which encapsulate the most salient features of the data, the generative process becomes significantly more efficient and potentially reveals underlying patterns beyond human recognition. This approach not only enhances training speed and resource usage but also explores the philosophical question of whether models can generate insights that transcend the human intelligence while being limited by the human-generated data.

Title: Large Language Models for Financial Aid in Financial Time-series Forecasting

• Authors: Md Khairul Islam, Ayush Karmacharya, Timothy Sue, Judy Fox
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.19025
• Pdf URL: https://arxiv.org/pdf/2410.19025
• Copy Paste: [[2410.19025]] Large Language Models for Financial Aid in Financial Time-series Forecasting(https://arxiv.org/abs/2410.19025)
• Keywords: foundation model
• Abstract: Considering the difficulty of financial time series forecasting in financial aid, much of the current research focuses on leveraging big data analytics in financial services. One modern approach is to utilize "predictive analysis", analogous to forecasting financial trends. However, many of these time series data in Financial Aid (FA) pose unique challenges due to limited historical datasets and high dimensional financial information, which hinder the development of effective predictive models that balance accuracy with efficient runtime and memory usage. Pre-trained foundation models are employed to address these challenging tasks. We use state-of-the-art time series models including pre-trained LLMs (GPT-2 as the backbone), transformers, and linear models to demonstrate their ability to outperform traditional approaches, even with minimal ("few-shot") or no fine-tuning ("zero-shot"). Our benchmark study, which includes financial aid with seven other time series tasks, shows the potential of using LLMs for scarce financial datasets.

Title: An Investigation on Machine Learning Predictive Accuracy Improvement and Uncertainty Reduction using VAE-based Data Augmentation

• Authors: Farah Alsafadi, Mahmoud Yaseen, Xu Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19063
• Pdf URL: https://arxiv.org/pdf/2410.19063
• Copy Paste: [[2410.19063]] An Investigation on Machine Learning Predictive Accuracy Improvement and Uncertainty Reduction using VAE-based Data Augmentation(https://arxiv.org/abs/2410.19063)
• Keywords: generative
• Abstract: The confluence of ultrafast computers with large memory, rapid progress in Machine Learning (ML) algorithms, and the availability of large datasets place multiple engineering fields at the threshold of dramatic progress. However, a unique challenge in nuclear engineering is data scarcity because experimentation on nuclear systems is usually more expensive and time-consuming than most other disciplines. One potential way to resolve the data scarcity issue is deep generative learning, which uses certain ML models to learn the underlying distribution of existing data and generate synthetic samples that resemble the real data. In this way, one can significantly expand the dataset to train more accurate predictive ML models. In this study, our objective is to evaluate the effectiveness of data augmentation using variational autoencoder (VAE)-based deep generative models. We investigated whether the data augmentation leads to improved accuracy in the predictions of a deep neural network (DNN) model trained using the augmented data. Additionally, the DNN prediction uncertainties are quantified using Bayesian Neural Networks (BNN) and conformal prediction (CP) to assess the impact on predictive uncertainty reduction. To test the proposed methodology, we used TRACE simulations of steady-state void fraction data based on the NUPEC Boiling Water Reactor Full-size Fine-mesh Bundle Test (BFBT) benchmark. We found that augmenting the training dataset using VAEs has improved the DNN model's predictive accuracy, improved the prediction confidence intervals, and reduced the prediction uncertainties.

Title: BIFR\"OST: 3D-Aware Image compositing with Language Instructions

• Authors: Lingxiao Li, Kaixiong Gong, Weihong Li, Xili Dai, Tao Chen, Xiaojun Yuan, Xiangyu Yue
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19079
• Pdf URL: https://arxiv.org/pdf/2410.19079
• Copy Paste: [[2410.19079]] BIFR\"OST: 3D-Aware Image compositing with Language Instructions(https://arxiv.org/abs/2410.19079)
• Keywords: diffusion, generative
• Abstract: This paper introduces Bifröst, a novel 3D-aware framework that is built upon diffusion models to perform instruction-based image composition. Previous methods concentrate on image compositing at the 2D level, which fall short in handling complex spatial relationships ($\textit{e.g.}$, occlusion). Bifröst addresses these issues by training MLLM as a 2.5D location predictor and integrating depth maps as an extra condition during the generation process to bridge the gap between 2D and 3D, which enhances spatial comprehension and supports sophisticated spatial interactions. Our method begins by fine-tuning MLLM with a custom counterfactual dataset to predict 2.5D object locations in complex backgrounds from language instructions. Then, the image-compositing model is uniquely designed to process multiple types of input features, enabling it to perform high-fidelity image compositions that consider occlusion, depth blur, and image harmonization. Extensive qualitative and quantitative evaluations demonstrate that Bifröst significantly outperforms existing methods, providing a robust solution for generating realistically composed images in scenarios demanding intricate spatial understanding. This work not only pushes the boundaries of generative image compositing but also reduces reliance on expensive annotated datasets by effectively utilizing existing resources in innovative ways.

Title: Watermarking Large Language Models and the Generated Content: Opportunities and Challenges

• Authors: Ruisi Zhang, Farinaz Koushanfar
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2410.19096
• Pdf URL: https://arxiv.org/pdf/2410.19096
• Copy Paste: [[2410.19096]] Watermarking Large Language Models and the Generated Content: Opportunities and Challenges(https://arxiv.org/abs/2410.19096)
• Keywords: generative
• Abstract: The widely adopted and powerful generative large language models (LLMs) have raised concerns about intellectual property rights violations and the spread of machine-generated misinformation. Watermarking serves as a promising approch to establish ownership, prevent unauthorized use, and trace the origins of LLM-generated content. This paper summarizes and shares the challenges and opportunities we found when watermarking LLMs. We begin by introducing techniques for watermarking LLMs themselves under different threat models and scenarios. Next, we investigate watermarking methods designed for the content generated by LLMs, assessing their effectiveness and resilience against various attacks. We also highlight the importance of watermarking domain-specific models and data, such as those used in code generation, chip design, and medical applications. Furthermore, we explore methods like hardware acceleration to improve the efficiency of the watermarking process. Finally, we discuss the limitations of current approaches and outline future research directions for the responsible use and protection of these generative AI tools.

Title: Context-Aware Trajectory Anomaly Detection

• Authors: Haoji Hu, Jina Kim, Jinwei Zhou, Sofia Kirsanova, JangHyeon Lee, Yao-Yi Chiang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19136
• Pdf URL: https://arxiv.org/pdf/2410.19136
• Copy Paste: [[2410.19136]] Context-Aware Trajectory Anomaly Detection(https://arxiv.org/abs/2410.19136)
• Keywords: generative, anomaly
• Abstract: Trajectory anomaly detection is crucial for effective decision-making in urban and human mobility management. Existing methods of trajectory anomaly detection generally focus on training a trajectory generative model and evaluating the likelihood of reconstructing a given trajectory. However, previous work often lacks important contextual information on the trajectory, such as the agent's information (e.g., agent ID) or geographic information (e.g., Points of Interest (POI)), which could provide additional information on accurately capturing anomalous behaviors. To fill this gap, we propose a context-aware anomaly detection approach that models contextual information related to trajectories. The proposed method is based on a trajectory reconstruction framework guided by contextual factors such as agent ID and contextual POI embedding. The injection of contextual information aims to improve the performance of anomaly detection. We conducted experiments in two cities and demonstrated that the proposed approach significantly outperformed existing methods by effectively modeling contextual information. Overall, this paper paves a new direction for advancing trajectory anomaly detection.

Title: Structured Diffusion Models with Mixture of Gaussians as Prior Distribution

• Authors: Nanshan Jia, Tingyu Zhu, Haoyu Liu, Zeyu Zheng
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.19149
• Pdf URL: https://arxiv.org/pdf/2410.19149
• Copy Paste: [[2410.19149]] Structured Diffusion Models with Mixture of Gaussians as Prior Distribution(https://arxiv.org/abs/2410.19149)
• Keywords: diffusion
• Abstract: We propose a class of structured diffusion models, in which the prior distribution is chosen as a mixture of Gaussians, rather than a standard Gaussian distribution. The specific mixed Gaussian distribution, as prior, can be chosen to incorporate certain structured information of the data. We develop a simple-to-implement training procedure that smoothly accommodates the use of mixed Gaussian as prior. Theory is provided to quantify the benefits of our proposed models, compared to the classical diffusion models. Numerical experiments with synthetic, image and operational data are conducted to show comparative advantages of our model. Our method is shown to be robust to mis-specifications and in particular suits situations where training resources are limited or faster training in real time is desired.

Title: Label Set Optimization via Activation Distribution Kurtosis for Zero-shot Classification with Generative Models

• Authors: Yue Li, Zhixue Zhao, Carolina Scarton
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.19195
• Pdf URL: https://arxiv.org/pdf/2410.19195
• Copy Paste: [[2410.19195]] Label Set Optimization via Activation Distribution Kurtosis for Zero-shot Classification with Generative Models(https://arxiv.org/abs/2410.19195)
• Keywords: generative, in-context
• Abstract: In-context learning (ICL) performance is known to be sensitive to the prompt design, yet the impact of class label options in zero-shot classification has been largely overlooked. This study presents the first comprehensive empirical study investigating how label option (e.g., lexical choice, order, and elaboration) influences zero-shot ICL classification performance. Our findings reveal that lexical choices for label names (e.g., agree this http URL in stance classification) play an important role, with effects also linked to label orders. An analysis of the model internal states further shows that optimal label names tend to activate fewer outlier neurons in the feed forward network. Based on this observation, we propose Label set Optimization via Activation Distribution kurtosiS (LOADS), a post-hoc approach requiring no gradient propagation. LOADS not only demonstrates effectiveness with only 100 unlabelled samples across different model types and sizes, but also shows cross-lingual transferability.

Title: No Free Lunch: Fundamental Limits of Learning Non-Hallucinating Generative Models

• Authors: Changlong Wu, Ananth Grama, Wojciech Szpankowski
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.19217
• Pdf URL: https://arxiv.org/pdf/2410.19217
• Copy Paste: [[2410.19217]] No Free Lunch: Fundamental Limits of Learning Non-Hallucinating Generative Models(https://arxiv.org/abs/2410.19217)
• Keywords: generative
• Abstract: Generative models have shown impressive capabilities in synthesizing high-quality outputs across various domains. However, a persistent challenge is the occurrence of "hallucinations", where the model produces outputs that are plausible but invalid. While empirical strategies have been explored to mitigate this issue, a rigorous theoretical understanding remains elusive. In this paper, we develop a theoretical framework to analyze the learnability of non-hallucinating generative models from a learning-theoretic perspective. Our results reveal that non-hallucinating learning is statistically impossible when relying solely on the training dataset, even for a hypothesis class of size two and when the entire training set is truthful. To overcome these limitations, we show that incorporating inductive biases aligned with the actual facts into the learning process is essential. We provide a systematic approach to achieve this by restricting the facts set to a concept class of finite VC-dimension and demonstrate its effectiveness under various learning paradigms. Although our findings are primarily conceptual, they represent a first step towards a principled approach to addressing hallucinations in learning generative models.

Title: Peptide-GPT: Generative Design of Peptides using Generative Pre-trained Transformers and Bio-informatic Supervision

• Authors: Aayush Shah, Chakradhar Guntuboina, Amir Barati Farimani
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2410.19222
• Pdf URL: https://arxiv.org/pdf/2410.19222
• Copy Paste: [[2410.19222]] Peptide-GPT: Generative Design of Peptides using Generative Pre-trained Transformers and Bio-informatic Supervision(https://arxiv.org/abs/2410.19222)
• Keywords: generative
• Abstract: In recent years, natural language processing (NLP) models have demonstrated remarkable capabilities in various domains beyond traditional text generation. In this work, we introduce PeptideGPT, a protein language model tailored to generate protein sequences with distinct properties: hemolytic activity, solubility, and non-fouling characteristics. To facilitate a rigorous evaluation of these generated sequences, we established a comprehensive evaluation pipeline consisting of ideas from bioinformatics to retain valid proteins with ordered structures. First, we rank the generated sequences based on their perplexity scores, then we filter out those lying outside the permissible convex hull of proteins. Finally, we predict the structure using ESMFold and select the proteins with pLDDT values greater than 70 to ensure ordered structure. The properties of generated sequences are evaluated using task-specific classifiers - PeptideBERT and HAPPENN. We achieved an accuracy of 76.26% in hemolytic, 72.46% in non-hemolytic, 78.84% in non-fouling, and 68.06% in solubility protein generation. Our experimental results demonstrate the effectiveness of PeptideGPT in de novo protein design and underscore the potential of leveraging NLP-based approaches for paving the way for future innovations and breakthroughs in synthetic biology and bioinformatics. Codes, models, and data used in this study are freely available at: this https URL.

Title: Flow Generator Matching

• Authors: Zemin Huang, Zhengyang Geng, Weijian Luo, Guo-jun Qi
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2410.19310
• Pdf URL: https://arxiv.org/pdf/2410.19310
• Copy Paste: [[2410.19310]] Flow Generator Matching(https://arxiv.org/abs/2410.19310)
• Keywords: diffusion, generative
• Abstract: In the realm of Artificial Intelligence Generated Content (AIGC), flow-matching models have emerged as a powerhouse, achieving success due to their robust theoretical underpinnings and solid ability for large-scale generative modeling. These models have demonstrated state-of-the-art performance, but their brilliance comes at a cost. The process of sampling from these models is notoriously demanding on computational resources, as it necessitates the use of multi-step numerical ordinary differential equations (ODEs). Against this backdrop, this paper presents a novel solution with theoretical guarantees in the form of Flow Generator Matching (FGM), an innovative approach designed to accelerate the sampling of flow-matching models into a one-step generation, while maintaining the original performance. On the CIFAR10 unconditional generation benchmark, our one-step FGM model achieves a new record Fréchet Inception Distance (FID) score of 3.08 among few-step flow-matching-based models, outperforming original 50-step flow-matching models. Furthermore, we use the FGM to distill the Stable Diffusion 3, a leading text-to-image flow-matching model based on the MM-DiT architecture. The resulting MM-DiT-FGM one-step text-to-image model demonstrates outstanding industry-level performance. When evaluated on the GenEval benchmark, MM-DiT-FGM has delivered remarkable generating qualities, rivaling other multi-step models in light of the efficiency of a single generation step.

Title: Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion

• Authors: Emiel Hoogeboom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, Tim Salimans
• Subjects: cs.CV, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.19324
• Pdf URL: https://arxiv.org/pdf/2410.19324
• Copy Paste: [[2410.19324]] Simpler Diffusion (SiD2): 1.5 FID on ImageNet512 with pixel-space diffusion(https://arxiv.org/abs/2410.19324)
• Keywords: diffusion
• Abstract: Latent diffusion models have become the popular choice for scaling up diffusion models for high resolution image synthesis. Compared to pixel-space models that are trained end-to-end, latent models are perceived to be more efficient and to produce higher image quality at high resolution. Here we challenge these notions, and show that pixel-space models can in fact be very competitive to latent approaches both in quality and efficiency, achieving 1.5 FID on ImageNet512 and new SOTA results on ImageNet128 and ImageNet256. We present a simple recipe for scaling end-to-end pixel-space diffusion models to high resolutions. 1: Use the sigmoid loss (Kingma & Gao, 2023) with our prescribed hyper-parameters. 2: Use our simplified memory-efficient architecture with fewer skip-connections. 3: Scale the model to favor processing the image at high resolution with fewer parameters, rather than using more parameters but at a lower resolution. When combining these three steps with recently proposed tricks like guidance intervals, we obtain a family of pixel-space diffusion models we call Simple Diffusion v2 (SiD2).

Title: FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality

• Authors: Zhengyao Lv, Chenyang Si, Junhao Song, Zhenyu Yang, Yu Qiao, Ziwei Liu, Kwan-Yee K. Wong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19355
• Pdf URL: https://arxiv.org/pdf/2410.19355
• Copy Paste: [[2410.19355]] FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality(https://arxiv.org/abs/2410.19355)
• Keywords: diffusion
• Abstract: In this paper, we present \textbf{\textit{FasterCache}}, a novel training-free strategy designed to accelerate the inference of video diffusion models with high-quality generation. By analyzing existing cache-based methods, we observe that \textit{directly reusing adjacent-step features degrades video quality due to the loss of subtle variations}. We further perform a pioneering investigation of the acceleration potential of classifier-free guidance (CFG) and reveal significant redundancy between conditional and unconditional features within the same timestep. Capitalizing on these observations, we introduce FasterCache to substantially accelerate diffusion-based video generation. Our key contributions include a dynamic feature reuse strategy that preserves both feature distinction and temporal continuity, and CFG-Cache which optimizes the reuse of conditional and unconditional outputs to further enhance inference speed without compromising video quality. We empirically evaluate FasterCache on recent video diffusion models. Experimental results show that FasterCache can significantly accelerate video generation (\eg 1.67$\times$ speedup on Vchitect-2.0) while keeping video quality comparable to the baseline, and consistently outperform existing methods in both inference speed and video quality.

Title: KAHANI: Culturally-Nuanced Visual Storytelling Pipeline for Non-Western Cultures

• Authors: Hamna, Deepthi Sudharsan, Agrima Seth, Ritvik Budhiraja, Deepika Khullar, Vyshak Jain, Kalika Bali, Aditya Vashistha, Sameer Segal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.19419
• Pdf URL: https://arxiv.org/pdf/2410.19419
• Copy Paste: [[2410.19419]] KAHANI: Culturally-Nuanced Visual Storytelling Pipeline for Non-Western Cultures(https://arxiv.org/abs/2410.19419)
• Keywords: diffusion
• Abstract: Large Language Models (LLMs) and Text-To-Image (T2I) models have demonstrated the ability to generate compelling text and visual stories. However, their outputs are predominantly aligned with the sensibilities of the Global North, often resulting in an outsider's gaze on other cultures. As a result, non-Western communities have to put extra effort into generating culturally specific stories. To address this challenge, we developed a visual storytelling pipeline called KAHANI that generates culturally grounded visual stories for non-Western cultures. Our pipeline leverages off-the-shelf models GPT-4 Turbo and Stable Diffusion XL (SDXL). By using Chain of Thought (CoT) and T2I prompting techniques, we capture the cultural context from user's prompt and generate vivid descriptions of the characters and scene compositions. To evaluate the effectiveness of KAHANI, we conducted a comparative user study with ChatGPT-4 (with DALL-E3) in which participants from different regions of India compared the cultural relevance of stories generated by the two tools. Results from the qualitative and quantitative analysis performed on the user study showed that KAHANI was able to capture and incorporate more Culturally Specific Items (CSIs) compared to ChatGPT-4. In terms of both its cultural competence and visual story generation quality, our pipeline outperformed ChatGPT-4 in 27 out of the 36 comparisons.

Title: Analyzing Generative Models by Manifold Entropic Metrics

• Authors: Daniel Galperin, Ullrich Köthe
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.19426
• Pdf URL: https://arxiv.org/pdf/2410.19426
• Copy Paste: [[2410.19426]] Analyzing Generative Models by Manifold Entropic Metrics(https://arxiv.org/abs/2410.19426)
• Keywords: generative
• Abstract: Good generative models should not only synthesize high quality data, but also utilize interpretable representations that aid human understanding of their behavior. However, it is difficult to measure objectively if and to what degree desirable properties of disentangled representations have been achieved. Inspired by the principle of independent mechanisms, we address this difficulty by introducing a novel set of tractable information-theoretic evaluation metrics. We demonstrate the usefulness of our metrics on illustrative toy examples and conduct an in-depth comparison of various normalizing flow architectures and $\beta$-VAEs on the EMNIST dataset. Our method allows to sort latent features by importance and assess the amount of residual correlations of the resulting concepts. The most interesting finding of our experiments is a ranking of model architectures and training procedures in terms of their inductive bias to converge to aligned and disentangled representations during training.

Title: Generative Diffusion Models for Sequential Recommendations

• Authors: Sharare Zolghadr, Ole Winther, Paul Jeha
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19429
• Pdf URL: https://arxiv.org/pdf/2410.19429
• Copy Paste: [[2410.19429]] Generative Diffusion Models for Sequential Recommendations(https://arxiv.org/abs/2410.19429)
• Keywords: diffusion, generative
• Abstract: Generative models such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) have shown promise in sequential recommendation tasks. However, they face challenges, including posterior collapse and limited representation capacity. The work by Li et al. (2023) introduces a novel approach that leverages diffusion models to address these challenges by representing item embeddings as distributions rather than fixed vectors. This approach allows for a more adaptive reflection of users' diverse interests and various item aspects. During the diffusion phase, the model converts the target item embedding into a Gaussian distribution by adding noise, facilitating the representation of sequential item distributions and the injection of uncertainty. An Approximator then processes this noisy item representation to reconstruct the target item. In the reverse phase, the model utilizes users' past interactions to reverse the noise and finalize the item prediction through a rounding operation. This research introduces enhancements to the DiffuRec architecture, particularly by adding offset noise in the diffusion process to improve robustness and incorporating a cross-attention mechanism in the Approximator to better capture relevant user-item interactions. These contributions led to the development of a new model, DiffuRecSys, which improves performance. Extensive experiments conducted on three public benchmark datasets demonstrate that these modifications enhance item representation, effectively capture diverse user preferences, and outperform existing baselines in sequential recommendation research.

Title: Transductive Learning for Near-Duplicate Image Detection in Scanned Photo Collections

• Authors: Francesc Net, Marc Folia, Pep Casals, Lluis Gomez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19437
• Pdf URL: https://arxiv.org/pdf/2410.19437
• Copy Paste: [[2410.19437]] Transductive Learning for Near-Duplicate Image Detection in Scanned Photo Collections(https://arxiv.org/abs/2410.19437)
• Keywords: self-supervised
• Abstract: This paper presents a comparative study of near-duplicate image detection techniques in a real-world use case scenario, where a document management company is commissioned to manually annotate a collection of scanned photographs. Detecting duplicate and near-duplicate photographs can reduce the time spent on manual annotation by archivists. This real use case differs from laboratory settings as the deployment dataset is available in advance, allowing the use of transductive learning. We propose a transductive learning approach that leverages state-of-the-art deep learning architectures such as convolutional neural networks (CNNs) and Vision Transformers (ViTs). Our approach involves pre-training a deep neural network on a large dataset and then fine-tuning the network on the unlabeled target collection with self-supervised learning. The results show that the proposed approach outperforms the baseline methods in the task of near-duplicate image detection in the UKBench and an in-house private dataset.

Title: Marked Temporal Bayesian Flow Point Processes

• Authors: Hui Chen, Xuhui Fan, Hengyu Liu, Longbing Cao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19512
• Pdf URL: https://arxiv.org/pdf/2410.19512
• Copy Paste: [[2410.19512]] Marked Temporal Bayesian Flow Point Processes(https://arxiv.org/abs/2410.19512)
• Keywords: generative
• Abstract: Marked event data captures events by recording their continuous-valued occurrence timestamps along with their corresponding discrete-valued types. They have appeared in various real-world scenarios such as social media, financial transactions, and healthcare records, and have been effectively modeled through Marked Temporal Point Process (MTPP) models. Recently, developing generative models for these MTPP models have seen rapid development due to their powerful generative capability and less restrictive functional forms. However, existing generative MTPP models are usually challenged in jointly modeling events' timestamps and types since: (1) mainstream methods design the generative mechanisms for timestamps only and do not include event types; (2) the complex interdependence between the timestamps and event types are overlooked. In this paper, we propose a novel generative MTPP model called BMTPP. Unlike existing generative MTPP models, BMTPP flexibly models marked temporal joint distributions using a parameter-based approach. Additionally, by adding joint noise to the marked temporal data space, BMTPP effectively captures and explicitly reveals the interdependence between timestamps and event types. Extensive experiments validate the superiority of our approach over other state-of-the-art models and its ability to effectively capture marked-temporal interdependence.

Title: Utilizing Image Transforms and Diffusion Models for Generative Modeling of Short and Long Time Series

• Authors: Ilan Naiman, Nimrod Berman, Itai Pemper, Idan Arbiv, Gal Fadlon, Omri Azencot
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19538
• Pdf URL: https://arxiv.org/pdf/2410.19538
• Copy Paste: [[2410.19538]] Utilizing Image Transforms and Diffusion Models for Generative Modeling of Short and Long Time Series(https://arxiv.org/abs/2410.19538)
• Keywords: diffusion, generative
• Abstract: Lately, there has been a surge in interest surrounding generative modeling of time series data. Most existing approaches are designed either to process short sequences or to handle long-range sequences. This dichotomy can be attributed to gradient issues with recurrent networks, computational costs associated with transformers, and limited expressiveness of state space models. Towards a unified generative model for varying-length time series, we propose in this work to transform sequences into images. By employing invertible transforms such as the delay embedding and the short-time Fourier transform, we unlock three main advantages: i) We can exploit advanced diffusion vision models; ii) We can remarkably process short- and long-range inputs within the same framework; and iii) We can harness recent and established tools proposed in the time series to image literature. We validate the effectiveness of our method through a comprehensive evaluation across multiple tasks, including unconditional generation, interpolation, and extrapolation. We show that our approach achieves consistently state-of-the-art results against strong baselines. In the unconditional generation tasks, we show remarkable mean improvements of 58.17% over previous diffusion models in the short discriminative score and 132.61% in the (ultra-)long classification scores. Code is at this https URL.

Title: Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning

• Authors: Shentong Mo, Shengbang Tong
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV, eess.SP
• Abstract URL: https://arxiv.org/abs/2410.19560
• Pdf URL: https://arxiv.org/pdf/2410.19560
• Copy Paste: [[2410.19560]] Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning(https://arxiv.org/abs/2410.19560)
• Keywords: self-supervised
• Abstract: In recent advancements in unsupervised visual representation learning, the Joint-Embedding Predictive Architecture (JEPA) has emerged as a significant method for extracting visual features from unlabeled imagery through an innovative masking strategy. Despite its success, two primary limitations have been identified: the inefficacy of Exponential Moving Average (EMA) from I-JEPA in preventing entire collapse and the inadequacy of I-JEPA prediction in accurately learning the mean of patch representations. Addressing these challenges, this study introduces a novel framework, namely C-JEPA (Contrastive-JEPA), which integrates the Image-based Joint-Embedding Predictive Architecture with the Variance-Invariance-Covariance Regularization (VICReg) strategy. This integration is designed to effectively learn the variance/covariance for preventing entire collapse and ensuring invariance in the mean of augmented views, thereby overcoming the identified limitations. Through empirical and theoretical evaluations, our work demonstrates that C-JEPA significantly enhances the stability and quality of visual representation learning. When pre-trained on the ImageNet-1K dataset, C-JEPA exhibits rapid and improved convergence in both linear probing and fine-tuning performance metrics.

Title: Neuromorphic IoT Architecture for Efficient Water Management: A Smart Village Case Study

• Authors: Mugdim Bublin, Heimo Hirner, Antoine-Martin Lanners, Radu Grosu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.19562
• Pdf URL: https://arxiv.org/pdf/2410.19562
• Copy Paste: [[2410.19562]] Neuromorphic IoT Architecture for Efficient Water Management: A Smart Village Case Study(https://arxiv.org/abs/2410.19562)
• Keywords: anomaly
• Abstract: The exponential growth of IoT networks necessitates a paradigm shift towards architectures that offer high flexibility and learning capabilities while maintaining low energy consumption, minimal communication overhead, and low latency. Traditional IoT systems, particularly when integrated with machine learning approaches, often suffer from high communication overhead and significant energy consumption. This work addresses these challenges by proposing a neuromorphic architecture inspired by biological systems. To illustrate the practical application of our proposed architecture, we present a case study focusing on water management in the Carinthian community of Neuhaus. Preliminary results regarding water consumption prediction and anomaly detection in this community are presented. We also introduce a novel neuromorphic IoT architecture that integrates biological principles into the design of IoT systems. This architecture is specifically tailored for edge computing scenarios, where low power and high efficiency are crucial. Our approach leverages the inherent advantages of neuromorphic computing, such as asynchronous processing and event-driven communication, to create an IoT framework that is both energy-efficient and responsive. This case study demonstrates how the neuromorphic IoT architecture can be deployed in a real-world scenario, highlighting its benefits in terms of energy savings, reduced communication overhead, and improved system responsiveness.

Title: Microplastic Identification Using AI-Driven Image Segmentation and GAN-Generated Ecological Context

• Authors: Alex Dils, David Raymond, Jack Spottiswood, Samay Kodige, Dylan Karmin, Rikhil Kokal, Win Cowger, Chris Sadée
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2410.19604
• Pdf URL: https://arxiv.org/pdf/2410.19604
• Copy Paste: [[2410.19604]] Microplastic Identification Using AI-Driven Image Segmentation and GAN-Generated Ecological Context(https://arxiv.org/abs/2410.19604)
• Keywords: generative
• Abstract: Current methods for microplastic identification in water samples are costly and require expert analysis. Here, we propose a deep learning segmentation model to automatically identify microplastics in microscopic images. We labeled images of microplastic from the Moore Institute for Plastic Pollution Research and employ a Generative Adversarial Network (GAN) to supplement and generate diverse training data. To verify the validity of the generated data, we conducted a reader study where an expert was able to discern the generated microplastic from real microplastic at a rate of 68 percent. Our segmentation model trained on the combined data achieved an F1-Score of 0.91 on a diverse dataset, compared to the model without generated data's 0.82. With our findings we aim to enhance the ability of both experts and citizens to detect microplastic across diverse ecological contexts, thereby improving the cost and accessibility of microplastic analysis.

Title: Frozen-DETR: Enhancing DETR with Image Understanding from Frozen Foundation Models

• Authors: Shenghao Fu, Junkai Yan, Qize Yang, Xihan Wei, Xiaohua Xie, Wei-Shi Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19635
• Pdf URL: https://arxiv.org/pdf/2410.19635
• Copy Paste: [[2410.19635]] Frozen-DETR: Enhancing DETR with Image Understanding from Frozen Foundation Models(https://arxiv.org/abs/2410.19635)
• Keywords: foundation model
• Abstract: Recent vision foundation models can extract universal representations and show impressive abilities in various tasks. However, their application on object detection is largely overlooked, especially without fine-tuning them. In this work, we show that frozen foundation models can be a versatile feature enhancer, even though they are not pre-trained for object detection. Specifically, we explore directly transferring the high-level image understanding of foundation models to detectors in the following two ways. First, the class token in foundation models provides an in-depth understanding of the complex scene, which facilitates decoding object queries in the detector's decoder by providing a compact context. Additionally, the patch tokens in foundation models can enrich the features in the detector's encoder by providing semantic details. Utilizing frozen foundation models as plug-and-play modules rather than the commonly used backbone can significantly enhance the detector's performance while preventing the problems caused by the architecture discrepancy between the detector's backbone and the foundation model. With such a novel paradigm, we boost the SOTA query-based detector DINO from 49.0% AP to 51.9% AP (+2.9% AP) and further to 53.8% AP (+4.8% AP) by integrating one or two foundation models respectively, on the COCO validation set after training for 12 epochs with R50 as the detector's backbone.

Title: A distributional simplicity bias in the learning dynamics of transformers

• Authors: Riccardo Rende, Federica Gerace, Alessandro Laio, Sebastian Goldt
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.19637
• Pdf URL: https://arxiv.org/pdf/2410.19637
• Copy Paste: [[2410.19637]] A distributional simplicity bias in the learning dynamics of transformers(https://arxiv.org/abs/2410.19637)
• Keywords: self-supervised
• Abstract: The remarkable capability of over-parameterised neural networks to generalise effectively has been explained by invoking a ``simplicity bias'': neural networks prevent overfitting by initially learning simple classifiers before progressing to more complex, non-linear functions. While simplicity biases have been described theoretically and experimentally in feed-forward networks for supervised learning, the extent to which they also explain the remarkable success of transformers trained with self-supervised techniques remains unclear. In our study, we demonstrate that transformers, trained on natural language data, also display a simplicity bias. Specifically, they sequentially learn many-body interactions among input tokens, reaching a saturation point in the prediction error for low-degree interactions while continuing to learn high-degree interactions. To conduct this analysis, we develop a procedure to generate \textit{clones} of a given natural language data set, which rigorously capture the interactions between tokens up to a specified order. This approach opens up the possibilities of studying how interactions of different orders in the data affect learning, in natural language processing and beyond.

Title: DiffGS: Functional Gaussian Splatting Diffusion

• Authors: Junsheng Zhou, Weiqi Zhang, Yu-Shen Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.19657
• Pdf URL: https://arxiv.org/pdf/2410.19657
• Copy Paste: [[2410.19657]] DiffGS: Functional Gaussian Splatting Diffusion(https://arxiv.org/abs/2410.19657)
• Keywords: diffusion, generative
• Abstract: 3D Gaussian Splatting (3DGS) has shown convincing performance in rendering speed and fidelity, yet the generation of Gaussian Splatting remains a challenge due to its discreteness and unstructured nature. In this work, we propose DiffGS, a general Gaussian generator based on latent diffusion models. DiffGS is a powerful and efficient 3D generative model which is capable of generating Gaussian primitives at arbitrary numbers for high-fidelity rendering with rasterization. The key insight is to represent Gaussian Splatting in a disentangled manner via three novel functions to model Gaussian probabilities, colors and transforms. Through the novel disentanglement of 3DGS, we represent the discrete and unstructured 3DGS with continuous Gaussian Splatting functions, where we then train a latent diffusion model with the target of generating these Gaussian Splatting functions both unconditionally and conditionally. Meanwhile, we introduce a discretization algorithm to extract Gaussians at arbitrary numbers from the generated functions via octree-guided sampling and optimization. We explore DiffGS for various tasks, including unconditional generation, conditional generation from text, image, and partial 3DGS, as well as Point-to-Gaussian generation. We believe that DiffGS provides a new direction for flexibly modeling and generating Gaussian Splatting.

Title: Adversarial Environment Design via Regret-Guided Diffusion Models

• Authors: Hojun Chung, Junseo Lee, Minsoo Kim, Dohyeong Kim, Songhwai Oh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.19715
• Pdf URL: https://arxiv.org/pdf/2410.19715
• Copy Paste: [[2410.19715]] Adversarial Environment Design via Regret-Guided Diffusion Models(https://arxiv.org/abs/2410.19715)
• Keywords: diffusion
• Abstract: Training agents that are robust to environmental changes remains a significant challenge in deep reinforcement learning (RL). Unsupervised environment design (UED) has recently emerged to address this issue by generating a set of training environments tailored to the agent's capabilities. While prior works demonstrate that UED has the potential to learn a robust policy, their performance is constrained by the capabilities of the environment generation. To this end, we propose a novel UED algorithm, adversarial environment design via regret-guided diffusion models (ADD). The proposed method guides the diffusion-based environment generator with the regret of the agent to produce environments that the agent finds challenging but conducive to further improvement. By exploiting the representation power of diffusion models, ADD can directly generate adversarial environments while maintaining the diversity of training environments, enabling the agent to effectively learn a robust policy. Our experimental results demonstrate that the proposed method successfully generates an instructive curriculum of environments, outperforming UED baselines in zero-shot generalization across novel, out-of-distribution environments. Project page: this https URL

Title: Enhanced Anomaly Detection in Industrial Control Systems aided by Machine Learning

• Authors: Vegard Berge, Chunlei Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2410.19717
• Pdf URL: https://arxiv.org/pdf/2410.19717
• Copy Paste: [[2410.19717]] Enhanced Anomaly Detection in Industrial Control Systems aided by Machine Learning(https://arxiv.org/abs/2410.19717)
• Keywords: anomaly
• Abstract: Traditional intrusion detection systems (IDSs) often rely on either network traffic or process data, but this single-source approach may miss complex attack patterns that span multiple layers within industrial control systems (ICSs) or persistent threats that target different layers of operational technology systems. This study investigates whether combining both network and process data can improve attack detection in ICSs environments. Leveraging the SWaT dataset, we evaluate various machine learning models on individual and combined data sources. Our findings suggest that integrating network traffic with operational process data can enhance detection capabilities, evidenced by improved recall rates for cyber attack classification. Serving as a proof-of-concept within a limited testing environment, this research explores the feasibility of advancing intrusion detection through a multi-source data approach in ICSs. Although the results are promising, they are preliminary and highlight the need for further studies across diverse datasets and refined methodologies.