2024-10-30

Title: TV-3DG: Mastering Text-to-3D Customized Generation with Visual Prompt

• Authors: Jiahui Yang, Donglin Di, Baorui Ma, Xun Yang, Yongjia Ma, Wenzhang Sun, Wei Chen, Jianxun Cui, Zhou Xue, Meng Wang, Yebin Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21299
• Pdf URL: https://arxiv.org/pdf/2410.21299
• Copy Paste: [[2410.21299]] TV-3DG: Mastering Text-to-3D Customized Generation with Visual Prompt(https://arxiv.org/abs/2410.21299)
• Keywords: generative
• Abstract: In recent years, advancements in generative models have significantly expanded the capabilities of text-to-3D generation. Many approaches rely on Score Distillation Sampling (SDS) technology. However, SDS struggles to accommodate multi-condition inputs, such as text and visual prompts, in customized generation tasks. To explore the core reasons, we decompose SDS into a difference term and a classifier-free guidance term. Our analysis identifies the core issue as arising from the difference term and the random noise addition during the optimization process, both contributing to deviations from the target mode during distillation. To address this, we propose a novel algorithm, Classifier Score Matching (CSM), which removes the difference term in SDS and uses a deterministic noise addition process to reduce noise during optimization, effectively overcoming the low-quality limitations of SDS in our customized generation framework. Based on CSM, we integrate visual prompt information with an attention fusion mechanism and sampling guidance techniques, forming the Visual Prompt CSM (VPCSM) algorithm. Furthermore, we introduce a Semantic-Geometry Calibration (SGC) module to enhance quality through improved textual information integration. We present our approach as TV-3DG, with extensive experiments demonstrating its capability to achieve stable, high-quality, customized 3D generation. Project page: \url{this https URL}

Title: Domain-Adaptive Pre-training of Self-Supervised Foundation Models for Medical Image Classification in Gastrointestinal Endoscopy

• Authors: Marcel Roth, Micha V. Nowak
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21302
• Pdf URL: https://arxiv.org/pdf/2410.21302
• Copy Paste: [[2410.21302]] Domain-Adaptive Pre-training of Self-Supervised Foundation Models for Medical Image Classification in Gastrointestinal Endoscopy(https://arxiv.org/abs/2410.21302)
• Keywords: self-supervised, foundation model
• Abstract: Video capsule endoscopy has transformed gastrointestinal endoscopy (GIE) diagnostics by offering a non-invasive method for capturing detailed images of the gastrointestinal tract, enabling early disease detection. However, its potential is limited by the sheer volume of images generated during the imaging procedure, which can take anywhere from 6-8 hours and often produce up to 1 million images, necessitating automated analysis. Additionally, the variability of these images, combined with the need for expert annotations and the scarcity of large, high-quality labeled datasets, constrains the effectiveness of current medical image analysis models. To address this, we introduce a novel large gastrointestinal endoscopy dataset, called EndoExtend24, created by merging and re-stratifying the train/test splits of ten existing public and private datasets, ensuring no overlap of patient data across splits. EndoExtend24 includes over 226,000 labeled images, as well as dynamic class mappings, which allow unified training across datasets with differing labeling granularity, supporting up to 123 distinct pathological findings. Further, we propose to leverage domain adaptive pre-training of foundation models in computer vision trained with self-supervision on generic image data, to adapt them to the task of GIE medical diagnosis. Specifically, the EVA-02 model, which is based on the vision transformer architecture and was trained on ImageNet-22k with masked image modeling (using EVA-CLIP as a MIM teacher), is pre-trained on the novel EndoExtend24 dataset to achieve domain adaptation, and finally trained on the Capsule Endoscopy 2024 Challenge dataset. Experimental results show promising results on the challenge validation set, with an AUC Macro score of 0.993 and a balanced accuracy of 89.3%.

Title: VideoSAM: A Large Vision Foundation Model for High-Speed Video Segmentation

• Authors: Chika Maduabuchi, Ericmoore Jossou, Matteo Bucci
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21304
• Pdf URL: https://arxiv.org/pdf/2410.21304
• Copy Paste: [[2410.21304]] VideoSAM: A Large Vision Foundation Model for High-Speed Video Segmentation(https://arxiv.org/abs/2410.21304)
• Keywords: foundation model
• Abstract: High-speed video (HSV) segmentation is essential for analyzing dynamic physical processes in scientific and industrial applications, such as boiling heat transfer. Existing models like U-Net struggle with generalization and accurately segmenting complex bubble formations. We present VideoSAM, a specialized adaptation of the Segment Anything Model (SAM), fine-tuned on a diverse HSV dataset for phase detection. Through diverse experiments, VideoSAM demonstrates superior performance across four fluid environments -- Water, FC-72, Nitrogen, and Argon -- significantly outperforming U-Net in complex segmentation tasks. In addition to introducing VideoSAM, we contribute an open-source HSV segmentation dataset designed for phase detection, enabling future research in this domain. Our findings underscore VideoSAM's potential to set new standards in robust and accurate HSV segmentation. The code and dataset used in this study are available online at this https URL .

Title: Decoding Diffusion: A Scalable Framework for Unsupervised Analysis of Latent Space Biases and Representations Using Natural Language Prompts

• Authors: E. Zhixuan Zeng, Yuhao Chen, Alexander Wong
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21314
• Pdf URL: https://arxiv.org/pdf/2410.21314
• Copy Paste: [[2410.21314]] Decoding Diffusion: A Scalable Framework for Unsupervised Analysis of Latent Space Biases and Representations Using Natural Language Prompts(https://arxiv.org/abs/2410.21314)
• Keywords: diffusion, generative
• Abstract: Recent advances in image generation have made diffusion models powerful tools for creating high-quality images. However, their iterative denoising process makes understanding and interpreting their semantic latent spaces more challenging than other generative models, such as GANs. Recent methods have attempted to address this issue by identifying semantically meaningful directions within the latent space. However, they often need manual interpretation or are limited in the number of vectors that can be trained, restricting their scope and utility. This paper proposes a novel framework for unsupervised exploration of diffusion latent spaces. We directly leverage natural language prompts and image captions to map latent directions. This method allows for the automatic understanding of hidden features and supports a broader range of analysis without the need to train specific vectors. Our method provides a more scalable and interpretable understanding of the semantic knowledge encoded within diffusion models, facilitating comprehensive analysis of latent biases and the nuanced representations these models learn. Experimental results show that our framework can uncover hidden patterns and associations in various domains, offering new insights into the interpretability of diffusion model latent spaces.

Title: Angel or Devil: Discriminating Hard Samples and Anomaly Contaminations for Unsupervised Time Series Anomaly Detection

• Authors: Ruyi Zhang, Hongzuo Xu, Songlei Jian, Yusong Tan, Haifang Zhou, Rulin Xu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21322
• Pdf URL: https://arxiv.org/pdf/2410.21322
• Copy Paste: [[2410.21322]] Angel or Devil: Discriminating Hard Samples and Anomaly Contaminations for Unsupervised Time Series Anomaly Detection(https://arxiv.org/abs/2410.21322)
• Keywords: anomaly
• Abstract: Training in unsupervised time series anomaly detection is constantly plagued by the discrimination between harmful `anomaly contaminations' and beneficial `hard normal samples'. These two samples exhibit analogous loss behavior that conventional loss-based methodologies struggle to differentiate. To tackle this problem, we propose a novel approach that supplements traditional loss behavior with `parameter behavior', enabling a more granular characterization of anomalous patterns. Parameter behavior is formalized by measuring the parametric response to minute perturbations in input samples. Leveraging the complementary nature of parameter and loss behaviors, we further propose a dual Parameter-Loss Data Augmentation method (termed PLDA), implemented within the reinforcement learning paradigm. During the training phase of anomaly detection, PLDA dynamically augments the training data through an iterative process that simultaneously mitigates anomaly contaminations while amplifying informative hard normal samples. PLDA demonstrates remarkable versatility, which can serve as an additional component that seamlessly integrated with existing anomaly detectors to enhance their detection performance. Extensive experiments on ten datasets show that PLDA significantly improves the performance of four distinct detectors by up to 8\%, outperforming three state-of-the-art data augmentation methods.

Title: Self-Supervised Learning and Opportunistic Inference for Continuous Monitoring of Freezing of Gait in Parkinson's Disease

• Authors: Shovito Barua Soumma, Kartik Mangipudi, Daniel Peterson, Shyamal Mehta, Hassan Ghasemzadeh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21326
• Pdf URL: https://arxiv.org/pdf/2410.21326
• Copy Paste: [[2410.21326]] Self-Supervised Learning and Opportunistic Inference for Continuous Monitoring of Freezing of Gait in Parkinson's Disease(https://arxiv.org/abs/2410.21326)
• Keywords: self-supervised
• Abstract: Parkinson's disease (PD) is a progressive neurological disorder that impacts the quality of life significantly, making in-home monitoring of motor symptoms such as Freezing of Gait (FoG) critical. However, existing symptom monitoring technologies are power-hungry, rely on extensive amounts of labeled data, and operate in controlled settings. These shortcomings limit real-world deployment of the technology. This work presents LIFT-PD, a computationally-efficient self-supervised learning framework for real-time FoG detection. Our method combines self-supervised pre-training on unlabeled data with a novel differential hopping windowing technique to learn from limited labeled instances. An opportunistic model activation module further minimizes power consumption by selectively activating the deep learning module only during active periods. Extensive experimental results show that LIFT-PD achieves a 7.25% increase in precision and 4.4% improvement in accuracy compared to supervised models while using as low as 40% of the labeled training data used for supervised learning. Additionally, the model activation module reduces inference time by up to 67% compared to continuous inference. LIFT-PD paves the way for practical, energy-efficient, and unobtrusive in-home monitoring of PD patients with minimal labeling requirements.

Title: E(3)-invaraint diffusion model for pocket-aware peptide generation

• Authors: Po-Yu Liang, Jun Bai
• Subjects: cs.LG, cs.AI, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2410.21335
• Pdf URL: https://arxiv.org/pdf/2410.21335
• Copy Paste: [[2410.21335]] E(3)-invaraint diffusion model for pocket-aware peptide generation(https://arxiv.org/abs/2410.21335)
• Keywords: diffusion
• Abstract: Biologists frequently desire protein inhibitors for a variety of reasons, including use as research tools for understanding biological processes and application to societal problems in agriculture, healthcare, etc. Immunotherapy, for instance, relies on immune checkpoint inhibitors to block checkpoint proteins, preventing their binding with partner proteins and boosting immune cell function against abnormal cells. Inhibitor discovery has long been a tedious process, which in recent years has been accelerated by computational approaches. Advances in artificial intelligence now provide an opportunity to make inhibitor discovery smarter than ever before. While extensive research has been conducted on computer-aided inhibitor discovery, it has mainly focused on either sequence-to-structure mapping, reverse mapping, or bio-activity prediction, making it unrealistic for biologists to utilize such tools. Instead, our work proposes a new method of computer-assisted inhibitor discovery: de novo pocket-aware peptide structure and sequence generation network. Our approach consists of two sequential diffusion models for end-to-end structure generation and sequence prediction. By leveraging angle and dihedral relationships between backbone atoms, we ensure an E(3)-invariant representation of peptide structures. Our results demonstrate that our method achieves comparable performance to state-of-the-art models, highlighting its potential in pocket-aware peptide design. This work offers a new approach for precise drug discovery using receptor-specific peptide generation.

Title: Energy-Based Diffusion Language Models for Text Generation

• Authors: Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie, Yilun Xu, Jure Leskovec, Stefano Ermon, Arash Vahdat
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21357
• Pdf URL: https://arxiv.org/pdf/2410.21357
• Copy Paste: [[2410.21357]] Energy-Based Diffusion Language Models for Text Generation(https://arxiv.org/abs/2410.21357)
• Keywords: diffusion, generative
• Abstract: Despite remarkable progress in autoregressive language models, alternative generative paradigms beyond left-to-right generation are still being actively explored. Discrete diffusion models, with the capacity for parallel generation, have recently emerged as a promising alternative. Unfortunately, these models still underperform the autoregressive counterparts, with the performance gap increasing when reducing the number of sampling steps. Our analysis reveals that this degradation is a consequence of an imperfect approximation used by diffusion models. In this work, we propose Energy-based Diffusion Language Model (EDLM), an energy-based model operating at the full sequence level for each diffusion step, introduced to improve the underlying approximation used by diffusion models. More specifically, we introduce an EBM in a residual form, and show that its parameters can be obtained by leveraging a pretrained autoregressive model or by finetuning a bidirectional transformer via noise contrastive estimation. We also propose an efficient generation algorithm via parallel important sampling. Comprehensive experiments on language modeling benchmarks show that our model can consistently outperform state-of-the-art diffusion models by a significant margin, and approaches autoregressive models' perplexity. We further show that, without any generation performance drop, our framework offers a 1.3$\times$ sampling speedup over existing diffusion models.

Title: A Survey on Automatic Credibility Assessment of Textual Credibility Signals in the Era of Large Language Models

• Authors: Ivan Srba, Olesya Razuvayevskaya, João A. Leite, Robert Moro, Ipek Baris Schlicht, Sara Tonelli, Francisco Moreno García, Santiago Barrio Lottmann, Denis Teyssou, Valentin Porcellini, Carolina Scarton, Kalina Bontcheva, Maria Bielikova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21360
• Pdf URL: https://arxiv.org/pdf/2410.21360
• Copy Paste: [[2410.21360]] A Survey on Automatic Credibility Assessment of Textual Credibility Signals in the Era of Large Language Models(https://arxiv.org/abs/2410.21360)
• Keywords: generative
• Abstract: In the current era of social media and generative AI, an ability to automatically assess the credibility of online social media content is of tremendous importance. Credibility assessment is fundamentally based on aggregating credibility signals, which refer to small units of information, such as content factuality, bias, or a presence of persuasion techniques, into an overall credibility score. Credibility signals provide a more granular, more easily explainable and widely utilizable information in contrast to currently predominant fake news detection, which utilizes various (mostly latent) features. A growing body of research on automatic credibility assessment and detection of credibility signals can be characterized as highly fragmented and lacking mutual interconnections. This issue is even more prominent due to a lack of an up-to-date overview of research works on automatic credibility assessment. In this survey, we provide such systematic and comprehensive literature review of 175 research papers while focusing on textual credibility signals and Natural Language Processing (NLP), which undergoes a significant advancement due to Large Language Models (LLMs). While positioning the NLP research into the context of other multidisciplinary research works, we tackle with approaches for credibility assessment as well as with 9 categories of credibility signals (we provide a thorough analysis for 3 of them, namely: 1) factuality, subjectivity and bias, 2) persuasion techniques and logical fallacies, and 3) claims and veracity). Following the description of the existing methods, datasets and tools, we identify future challenges and opportunities, while paying a specific attention to recent rapid development of generative AI.

Title: SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization

• Authors: Wanhua Li, Zibin Meng, Jiawei Zhou, Donglai Wei, Chuang Gan, Hanspeter Pfister
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21411
• Pdf URL: https://arxiv.org/pdf/2410.21411
• Copy Paste: [[2410.21411]] SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization(https://arxiv.org/abs/2410.21411)
• Keywords: foundation model, generative
• Abstract: Social relation reasoning aims to identify relation categories such as friends, spouses, and colleagues from images. While current methods adopt the paradigm of training a dedicated network end-to-end using labeled image data, they are limited in terms of generalizability and interpretability. To address these issues, we first present a simple yet well-crafted framework named {\name}, which combines the perception capability of Vision Foundation Models (VFMs) and the reasoning capability of Large Language Models (LLMs) within a modular framework, providing a strong baseline for social relation recognition. Specifically, we instruct VFMs to translate image content into a textual social story, and then utilize LLMs for text-based reasoning. {\name} introduces systematic design principles to adapt VFMs and LLMs separately and bridge their gaps. Without additional model training, it achieves competitive zero-shot results on two databases while offering interpretable answers, as LLMs can generate language-based explanations for the decisions. The manual prompt design process for LLMs at the reasoning phase is tedious and an automated prompt optimization method is desired. As we essentially convert a visual classification task into a generative task of LLMs, automatic prompt optimization encounters a unique long prompt optimization issue. To address this issue, we further propose the Greedy Segment Prompt Optimization (GSPO), which performs a greedy search by utilizing gradient information at the segment level. Experimental results show that GSPO significantly improves performance, and our method also generalizes to different image styles. The code is available at this https URL.

Title: AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion models

• Authors: Yaopei Zeng, Yuanpu Cao, Bochuan Cao, Yurui Chang, Jinghui Chen, Lu Lin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21471
• Pdf URL: https://arxiv.org/pdf/2410.21471
• Copy Paste: [[2410.21471]] AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion models(https://arxiv.org/abs/2410.21471)
• Keywords: diffusion
• Abstract: Recent advances in diffusion models have significantly enhanced the quality of image synthesis, yet they have also introduced serious safety concerns, particularly the generation of Not Safe for Work (NSFW) content. Previous research has demonstrated that adversarial prompts can be used to generate NSFW content. However, such adversarial text prompts are often easily detectable by text-based filters, limiting their efficacy. In this paper, we expose a previously overlooked vulnerability: adversarial image attacks targeting Image-to-Image (I2I) diffusion models. We propose AdvI2I, a novel framework that manipulates input images to induce diffusion models to generate NSFW content. By optimizing a generator to craft adversarial images, AdvI2I circumvents existing defense mechanisms, such as Safe Latent Diffusion (SLD), without altering the text prompts. Furthermore, we introduce AdvI2I-Adaptive, an enhanced version that adapts to potential countermeasures and minimizes the resemblance between adversarial images and NSFW concept embeddings, making the attack more resilient against defenses. Through extensive experiments, we demonstrate that both AdvI2I and AdvI2I-Adaptive can effectively bypass current safeguards, highlighting the urgent need for stronger security measures to address the misuse of I2I diffusion models.

Title: A Systematic Review of Machine Learning in Sports Betting: Techniques, Challenges, and Future Directions

• Authors: René Manassé Galekwa, Jean Marie Tshimula, Etienne Gael Tajeuna, Kyamakya Kyandoghere
• Subjects: cs.LG, cs.CE, cs.ET, cs.IR, cs.SI
• Abstract URL: https://arxiv.org/abs/2410.21484
• Pdf URL: https://arxiv.org/pdf/2410.21484
• Copy Paste: [[2410.21484]] A Systematic Review of Machine Learning in Sports Betting: Techniques, Challenges, and Future Directions(https://arxiv.org/abs/2410.21484)
• Keywords: anomaly
• Abstract: The sports betting industry has experienced rapid growth, driven largely by technological advancements and the proliferation of online platforms. Machine learning (ML) has played a pivotal role in the transformation of this sector by enabling more accurate predictions, dynamic odds-setting, and enhanced risk management for both bookmakers and bettors. This systematic review explores various ML techniques, including support vector machines, random forests, and neural networks, as applied in different sports such as soccer, basketball, tennis, and cricket. These models utilize historical data, in-game statistics, and real-time information to optimize betting strategies and identify value bets, ultimately improving profitability. For bookmakers, ML facilitates dynamic odds adjustment and effective risk management, while bettors leverage data-driven insights to exploit market inefficiencies. This review also underscores the role of ML in fraud detection, where anomaly detection models are used to identify suspicious betting patterns. Despite these advancements, challenges such as data quality, real-time decision-making, and the inherent unpredictability of sports outcomes remain. Ethical concerns related to transparency and fairness are also of significant importance. Future research should focus on developing adaptive models that integrate multimodal data and manage risk in a manner akin to financial portfolios. This review provides a comprehensive examination of the current applications of ML in sports betting, and highlights both the potential and the limitations of these technologies.

Title: RoBIn: A Transformer-Based Model For Risk Of Bias Inference With Machine Reading Comprehension

• Authors: Abel Corrêa Dias, Viviane Pereira Moreira, João Luiz Dihl Comba
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21495
• Pdf URL: https://arxiv.org/pdf/2410.21495
• Copy Paste: [[2410.21495]] RoBIn: A Transformer-Based Model For Risk Of Bias Inference With Machine Reading Comprehension(https://arxiv.org/abs/2410.21495)
• Keywords: generative
• Abstract: Objective: Scientific publications play a crucial role in uncovering insights, testing novel drugs, and shaping healthcare policies. Accessing the quality of publications requires evaluating their Risk of Bias (RoB), a process typically conducted by human reviewers. In this study, we introduce a new dataset for machine reading comprehension and RoB assessment and present RoBIn (Risk of Bias Inference), an innovative model crafted to automate such evaluation. The model employs a dual-task approach, extracting evidence from a given context and assessing the RoB based on the gathered evidence. Methods: We use data from the Cochrane Database of Systematic Reviews (CDSR) as ground truth to label open-access clinical trial publications from PubMed. This process enabled us to develop training and test datasets specifically for machine reading comprehension and RoB inference. Additionally, we created extractive (RoBInExt) and generative (RoBInGen) Transformer-based approaches to extract relevant evidence and classify the RoB effectively. Results: RoBIn is evaluated across various settings and benchmarked against state-of-the-art methods for RoB inference, including large language models in multiple scenarios. In most cases, the best-performing RoBIn variant surpasses traditional machine learning and LLM-based approaches, achieving an ROC AUC of 0.83. Conclusion: Based on the evidence extracted from clinical trial reports, RoBIn performs a binary classification to decide whether the trial is at a low RoB or a high/unclear RoB. We found that both RoBInGen and RoBInExt are robust and have the best results in many settings.

Title: Diffusion-nested Auto-Regressive Synthesis of Heterogeneous Tabular Data

• Authors: Hengrui Zhang, Liancheng Fang, Qitian Wu, Philip S. Yu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21523
• Pdf URL: https://arxiv.org/pdf/2410.21523
• Copy Paste: [[2410.21523]] Diffusion-nested Auto-Regressive Synthesis of Heterogeneous Tabular Data(https://arxiv.org/abs/2410.21523)
• Keywords: diffusion
• Abstract: Autoregressive models are predominant in natural language generation, while their application in tabular data remains underexplored. We posit that this can be attributed to two factors: 1) tabular data contains heterogeneous data type, while the autoregressive model is primarily designed to model discrete-valued data; 2) tabular data is column permutation-invariant, requiring a generation model to generate columns in arbitrary order. This paper proposes a Diffusion-nested Autoregressive model (TabDAR) to address these issues. To enable autoregressive methods for continuous columns, TabDAR employs a diffusion model to parameterize the conditional distribution of continuous features. To ensure arbitrary generation order, TabDAR resorts to masked transformers with bi-directional attention, which simulate various permutations of column order, hence enabling it to learn the conditional distribution of a target column given an arbitrary combination of other columns. These designs enable TabDAR to not only freely handle heterogeneous tabular data but also support convenient and flexible unconditional/conditional sampling. We conduct extensive experiments on ten datasets with distinct properties, and the proposed TabDAR outperforms previous state-of-the-art methods by 18% to 45% on eight metrics across three distinct aspects.

Title: Unveiling Context-Aware Criteria in Self-Assessing LLMs

• Authors: Taneesh Gupta, Shivam Shandilya, Xuchao Zhang, Supriyo Ghosh, Chetan Bansal, Huaxiu Yao, Saravan Rajmohan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21545
• Pdf URL: https://arxiv.org/pdf/2410.21545
• Copy Paste: [[2410.21545]] Unveiling Context-Aware Criteria in Self-Assessing LLMs(https://arxiv.org/abs/2410.21545)
• Keywords: generative
• Abstract: The use of large language models (LLMs) as evaluators has garnered significant attention due to their potential to rival human-level evaluations in long-form response assessments. However, current LLM evaluators rely heavily on static, human-defined criteria, limiting their ability to generalize across diverse generative tasks and incorporate context-specific knowledge. In this paper, we propose a novel Self-Assessing LLM framework that integrates Context-Aware Criteria (SALC) with dynamic knowledge tailored to each evaluation instance. This instance-level knowledge enhances the LLM evaluator's performance by providing relevant and context-aware insights that pinpoint the important criteria specific to the current instance. Additionally, the proposed framework adapts seamlessly to various tasks without relying on predefined human criteria, offering a more flexible evaluation approach. Empirical evaluations demonstrate that our approach significantly outperforms existing baseline evaluation frameworks, yielding improvements on average 4.8% across a wide variety of datasets. Furthermore, by leveraging knowledge distillation techniques, we fine-tuned smaller language models for criteria generation and evaluation, achieving comparable or superior performance to larger models with much lower cost. Our method also exhibits a improvement in LC Win-Rate in AlpacaEval2 leaderboard up to a 12% when employed for preference data generation in Direct Preference Optimization (DPO), underscoring its efficacy as a robust and scalable evaluation framework.

Title: Exploring the Design Space of Diffusion Bridge Models via Stochasticity Control

• Authors: Shaorong Zhang, Yuanbin Cheng, Xianghao Kong, Greg Ver Steeg
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21553
• Pdf URL: https://arxiv.org/pdf/2410.21553
• Copy Paste: [[2410.21553]] Exploring the Design Space of Diffusion Bridge Models via Stochasticity Control(https://arxiv.org/abs/2410.21553)
• Keywords: diffusion
• Abstract: Diffusion bridge models effectively facilitate image-to-image (I2I) translation by connecting two distributions. However, existing methods overlook the impact of noise in sampling SDEs, transition kernel, and the base distribution on sampling efficiency, image quality and diversity. To address this gap, we propose the Stochasticity-controlled Diffusion Bridge (SDB), a novel theoretical framework that extends the design space of diffusion bridges, and provides strategies to mitigate singularities during both training and sampling. By controlling stochasticity in the sampling SDEs, our sampler achieves speeds up to 5 times faster than the baseline, while also producing lower FID scores. After training, SDB sets new benchmarks in image quality and sampling efficiency via managing stochasticity within the transition kernel. Furthermore, introducing stochasticity into the base distribution significantly improves image diversity, as quantified by a newly introduced metric.

Title: Going Beyond H&E and Oncology: How Do Histopathology Foundation Models Perform for Multi-stain IHC and Immunology?

• Authors: Amaya Gallagher-Syed, Elena Pontarini, Myles J. Lewis, Michael R. Barnes, Gregory Slabaugh
• Subjects: cs.CV, cs.AI, q-bio.QM, q-bio.TO
• Abstract URL: https://arxiv.org/abs/2410.21560
• Pdf URL: https://arxiv.org/pdf/2410.21560
• Copy Paste: [[2410.21560]] Going Beyond H&E and Oncology: How Do Histopathology Foundation Models Perform for Multi-stain IHC and Immunology?(https://arxiv.org/abs/2410.21560)
• Keywords: foundation model
• Abstract: This study evaluates the generalisation capabilities of state-of-the-art histopathology foundation models on out-of-distribution multi-stain autoimmune Immunohistochemistry datasets. We compare 13 feature extractor models, including ImageNet-pretrained networks, and histopathology foundation models trained on both public and proprietary data, on Rheumatoid Arthritis subtyping and Sjogren's Disease detection tasks. Using a simple Attention-Based Multiple Instance Learning classifier, we assess the transferability of learned representations from cancer H&E images to autoimmune IHC images. Contrary to expectations, histopathology-pretrained models did not significantly outperform ImageNet-pretrained models. Furthermore, there was evidence of both autoimmune feature misinterpretation and biased feature importance. Our findings highlight the challenges in transferring knowledge from cancer to autoimmune histopathology and emphasise the need for careful evaluation of AI models across diverse histopathological tasks. The code to run this benchmark is available at this https URL.

Title: ImageNet-RIB Benchmark: Large Pre-Training Datasets Don't Guarantee Robustness after Fine-Tuning

• Authors: Jaedong Hwang, Brian Cheung, Zhang-Wei Hong, Akhilan Boopathy, Pulkit Agrawal, Ila Fiete
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21582
• Pdf URL: https://arxiv.org/pdf/2410.21582
• Copy Paste: [[2410.21582]] ImageNet-RIB Benchmark: Large Pre-Training Datasets Don't Guarantee Robustness after Fine-Tuning(https://arxiv.org/abs/2410.21582)
• Keywords: foundation model
• Abstract: Highly performant large-scale pre-trained models promise to also provide a valuable foundation for learning specialized tasks, by fine-tuning the model to the desired task. By starting from a good general-purpose model, the goal is to achieve both specialization in the target task and maintain robustness. To assess the robustness of models to out-of-distribution samples after fine-tuning on downstream datasets, we introduce a new robust fine-tuning benchmark, ImageNet-RIB (Robustness Inheritance Benchmark). The benchmark consists of a set of related but distinct specialized (downstream) tasks; pre-trained models are fine-tuned on one task in the set and their robustness is assessed on the rest, iterating across all tasks for fine-tuning and assessment. We find that the continual learning methods, EWC and LwF maintain robustness after fine-tuning though fine-tuning generally does reduce performance on generalization to related downstream tasks across models. Not surprisingly, models pre-trained on large and rich datasets exhibit higher initial robustness across datasets and suffer more pronounced degradation during fine-tuning. The distance between the pre-training and downstream datasets, measured by optimal transport, predicts this performance degradation on the pre-training dataset. However, counterintuitively, model robustness after fine-tuning on related downstream tasks is the worst when the pre-training dataset is the richest and the most diverse. This suggests that starting with the strongest foundation model is not necessarily the best approach for performance on specialist tasks. The benchmark thus offers key insights for developing more resilient fine-tuning strategies and building robust machine learning models. this https URL

Title: CaloChallenge 2022: A Community Challenge for Fast Calorimeter Simulation

• Authors: Claudius Krause, Michele Faucci Giannelli, Gregor Kasieczka, Benjamin Nachman, Dalila Salamani, David Shih, Anna Zaborowska, Oz Amram, Kerstin Borras, Matthew R. Buckley, Erik Buhmann, Thorsten Buss, Renato Paulo Da Costa Cardoso, Anthony L. Caterini, Nadezda Chernyavskaya, Federico A.G. Corchia, Jesse C. Cresswell, Sascha Diefenbacher, Etienne Dreyer, Vijay Ekambaram, Engin Eren, Florian Ernst, Luigi Favaro, Matteo Franchini, Frank Gaede, Eilam Gross, Shih-Chieh Hsu, Kristina Jaruskova, Benno Käch, Jayant Kalagnanam, Raghav Kansal, Taewoo Kim, Dmitrii Kobylianskii, Anatolii Korol, William Korcari, Dirk Krücker, Katja Krüger, Marco Letizia, Shu Li, Qibin Liu, Xiulong Liu, Gabriel Loaiza-Ganem, Thandikire Madula, Peter McKeown, Isabell-A. Melzer-Pellmann, Vinicius Mikuni, Nam Nguyen, Ayodele Ore, Sofia Palacios Schweitzer, Ian Pang, Kevin Pedro, Tilman Plehn, Witold Pokorski, Huilin Qu, Piyush Raikwar, John A. Raine, Humberto Reyes-Gonzalez, Lorenzo Rinaldi, Brendan Leigh Ross, Moritz A.W. Scham, Simon Schnake, Chase Shimmin, Eli Shlizerman, Nathalie Soybelman, Mudhakar Srivatsa, Kalliopi Tsolaki, Sofia Vallecorsa, Kyongmin Yeo, Rui Zhang
• Subjects: cs.LG, hep-ex, hep-ph, physics.ins-det
• Abstract URL: https://arxiv.org/abs/2410.21611
• Pdf URL: https://arxiv.org/pdf/2410.21611
• Copy Paste: [[2410.21611]] CaloChallenge 2022: A Community Challenge for Fast Calorimeter Simulation(https://arxiv.org/abs/2410.21611)
• Keywords: diffusion, generative
• Abstract: We present the results of the "Fast Calorimeter Simulation Challenge 2022" - the CaloChallenge. We study state-of-the-art generative models on four calorimeter shower datasets of increasing dimensionality, ranging from a few hundred voxels to a few tens of thousand voxels. The 31 individual submissions span a wide range of current popular generative architectures, including Variational AutoEncoders (VAEs), Generative Adversarial Networks (GANs), Normalizing Flows, Diffusion models, and models based on Conditional Flow Matching. We compare all submissions in terms of quality of generated calorimeter showers, as well as shower generation time and model size. To assess the quality we use a broad range of different metrics including differences in 1-dimensional histograms of observables, KPD/FPD scores, AUCs of binary classifiers, and the log-posterior of a multiclass classifier. The results of the CaloChallenge provide the most complete and comprehensive survey of cutting-edge approaches to calorimeter fast simulation to date. In addition, our work provides a uniquely detailed perspective on the important problem of how to evaluate generative models. As such, the results presented here should be applicable for other domains that use generative AI and require fast and faithful generation of samples in a large phase space.

Title: OFER: Occluded Face Expression Reconstruction

• Authors: Pratheba Selvaraju, Victoria Fernandez Abrevaya, Timo Bolkart, Rick Akkerman, Tianyu Ding, Faezeh Amjadi, Ilya Zharkov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21629
• Pdf URL: https://arxiv.org/pdf/2410.21629
• Copy Paste: [[2410.21629]] OFER: Occluded Face Expression Reconstruction(https://arxiv.org/abs/2410.21629)
• Keywords: diffusion
• Abstract: Reconstructing 3D face models from a single image is an inherently ill-posed problem, which becomes even more challenging in the presence of occlusions. In addition to fewer available observations, occlusions introduce an extra source of ambiguity, where multiple reconstructions can be equally valid. Despite the ubiquity of the problem, very few methods address its multi-hypothesis nature. In this paper we introduce OFER, a novel approach for single image 3D face reconstruction that can generate plausible, diverse, and expressive 3D faces, even under strong occlusions. Specifically, we train two diffusion models to generate the shape and expression coefficients of a face parametric model, conditioned on the input image. This approach captures the multi-modal nature of the problem, generating a distribution of solutions as output. Although this addresses the ambiguity problem, the challenge remains to pick the best matching shape to ensure consistency across diverse expressions. To achieve this, we propose a novel ranking mechanism that sorts the outputs of the shape diffusion network based on the predicted shape accuracy scores to select the best match. We evaluate our method using standard benchmarks and introduce CO-545, a new protocol and dataset designed to assess the accuracy of expressive faces under occlusion. Our results show improved performance over occlusion-based methods, with added ability to generate multiple expressions for a given image.

Title: Faster Local Solvers for Graph Diffusion Equations

• Authors: Jiahe Bai, Baojian Zhou, Deqing Yang, Yanghua Xiao
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2410.21634
• Pdf URL: https://arxiv.org/pdf/2410.21634
• Copy Paste: [[2410.21634]] Faster Local Solvers for Graph Diffusion Equations(https://arxiv.org/abs/2410.21634)
• Keywords: diffusion
• Abstract: Efficient computation of graph diffusion equations (GDEs), such as Personalized PageRank, Katz centrality, and the Heat kernel, is crucial for clustering, training neural networks, and many other graph-related problems. Standard iterative methods require accessing the whole graph per iteration, making them time-consuming for large-scale graphs. While existing local solvers approximate diffusion vectors through heuristic local updates, they often operate sequentially and are typically designed for specific diffusion types, limiting their applicability. Given that diffusion vectors are highly localizable, as measured by the participation ratio, this paper introduces a novel framework for approximately solving GDEs using a local diffusion process. This framework reveals the suboptimality of existing local solvers. Furthermore, our approach effectively localizes standard iterative solvers by designing simple and provably sublinear time algorithms. These new local solvers are highly parallelizable, making them well-suited for implementation on GPUs. We demonstrate the effectiveness of our framework in quickly obtaining approximate diffusion vectors, achieving up to a hundred-fold speed improvement, and its applicability to large-scale dynamic graphs. Our framework could also facilitate more efficient local message-passing mechanisms for GNNs.

Title: Are Paraphrases Generated by Large Language Models Invertible?

• Authors: Rafael Rivera Soto, Barry Chen, Nicholas Andrews
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21637
• Pdf URL: https://arxiv.org/pdf/2410.21637
• Copy Paste: [[2410.21637]] Are Paraphrases Generated by Large Language Models Invertible?(https://arxiv.org/abs/2410.21637)
• Keywords: in-context
• Abstract: Large language models can produce highly fluent paraphrases while retaining much of the original meaning. While this capability has a variety of helpful applications, it may also be abused by bad actors, for example to plagiarize content or to conceal their identity. This motivates us to consider the problem of paraphrase inversion: given a paraphrased document, attempt to recover the original text. To explore the feasibility of this task, we fine-tune paraphrase inversion models, both with and without additional author-specific context to help guide the inversion process. We explore two approaches to author-specific inversion: one using in-context examples of the target author's writing, and another using learned style representations that capture distinctive features of the author's style. We show that, when starting from paraphrased machine-generated text, we can recover significant portions of the document using a learned inversion model. When starting from human-written text, the variety of source writing styles poses a greater challenge for invertability. However, even when the original tokens can't be recovered, we find the inverted text is stylistically similar to the original, which significantly improves the performance of plagiarism detectors and authorship identification systems that rely on stylistic markers.

Title: Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis

• Authors: Deepak Sridhar, Abhishek Peri, Rohith Rachala, Nuno Vasconcelos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21638
• Pdf URL: https://arxiv.org/pdf/2410.21638
• Copy Paste: [[2410.21638]] Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis(https://arxiv.org/abs/2410.21638)
• Keywords: diffusion, generative
• Abstract: Recent advances in generative modeling with diffusion processes (DPs) enabled breakthroughs in image synthesis. Despite impressive image quality, these models have various prompt compliance problems, including low recall in generating multiple objects, difficulty in generating text in images, and meeting constraints like object locations and pose. For fine-grained editing and manipulation, they also require fine-grained semantic or instance maps that are tedious to produce manually. While prompt compliance can be enhanced by addition of loss functions at inference, this is time consuming and does not scale to complex scenes. To overcome these limitations, this work introduces a new family of \textit{Factor Graph Diffusion Models} (FG-DMs) that models the joint distribution of images and conditioning variables, such as semantic, sketch, depth or normal maps via a factor graph decomposition. This joint structure has several advantages, including support for efficient sampling based prompt compliance schemes, which produce images of high object recall, semi-automated fine-grained editing, text-based editing of conditions with noise inversion, explainability at intermediate levels, ability to produce labeled datasets for the training of downstream models such as segmentation or depth, training with missing data, and continual learning where new conditioning variables can be added with minimal or no modifications to the existing structure. We propose an implementation of FG-DMs by adapting a pre-trained Stable Diffusion (SD) model to implement all FG-DM factors, using only COCO dataset, and show that it is effective in generating images with 15\% higher recall than SD while retaining its generalization ability. We introduce an attention distillation loss that encourages consistency among the attention maps of all factors, improving the fidelity of the generated conditions and image.

Title: On filter design in deep convolutional neural network

• Authors: Gaurav Hirani, Waleed Abdulla
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21644
• Pdf URL: https://arxiv.org/pdf/2410.21644
• Copy Paste: [[2410.21644]] On filter design in deep convolutional neural network(https://arxiv.org/abs/2410.21644)
• Keywords: self-supervised
• Abstract: The deep convolutional neural network (DCNN) in computer vision has given promising results. It is widely applied in many areas, from medicine, agriculture, self-driving car, biometric system, and almost all computer vision-based applications. Filters or weights are the critical elements responsible for learning in DCNN. Backpropagation has been the primary learning algorithm for DCNN and provides promising results, but the size and numbers of the filters remain hyper-parameters. Various studies have been done in the last decade on semi-supervised, self-supervised, and unsupervised methods and their properties. The effects of filter initialization, size-shape selection, and the number of filters on learning and optimization have not been investigated in a separate publication to collate all the options. Such attributes are often treated as hyper-parameters and lack mathematical understanding. Computer vision algorithms have many limitations in real-life applications, and understanding the learning process is essential to have some significant improvement. To the best of our knowledge, no separate investigation has been published discussing the filters; this is our primary motivation. This study focuses on arguments for choosing specific physical parameters of filters, initialization, and learning technic over scattered methods. The promising unsupervised approaches have been evaluated. Additionally, the limitations, current challenges, and future scope have been discussed in this paper.

Title: Exploring Local Memorization in Diffusion Models via Bright Ending Attention

• Authors: Chen Chen, Daochang Liu, Mubarak Shah, Chang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21665
• Pdf URL: https://arxiv.org/pdf/2410.21665
• Copy Paste: [[2410.21665]] Exploring Local Memorization in Diffusion Models via Bright Ending Attention(https://arxiv.org/abs/2410.21665)
• Keywords: diffusion, anomaly
• Abstract: In this paper, we identify and leverage a novel `bright ending' (BE) anomaly in diffusion models prone to memorizing training images to address a new task: locating localized memorization regions within these models. BE refers to a distinct cross-attention pattern observed in text-to-image generations using diffusion models. Specifically, memorized image patches exhibit significantly greater attention to the end token during the final inference step compared to non-memorized patches. This attention map effectively highlights regions where the generated image replicates training data. Furthermore, driven by our observation that local memorization significantly underperforms in existing tasks of measuring, detecting, and mitigating memorization in diffusion models compared to global memorization, we propose a simple yet effective method to integrate BE and the results of the new localization task into these existing frameworks. This integration effectively improves their performances by narrowing the performance gap caused by local memorization. Our results not only demonstrate the successful execution of the new localization task but also establish new state-of-the-art performance across all existing tasks, underscoring the significance of the BE phenomenon.

Title: Investigating Memorization in Video Diffusion Models

• Authors: Chen Chen, Enhuai Liu, Daochang Liu, Mubarak Shah, Chang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21669
• Pdf URL: https://arxiv.org/pdf/2410.21669
• Copy Paste: [[2410.21669]] Investigating Memorization in Video Diffusion Models(https://arxiv.org/abs/2410.21669)
• Keywords: diffusion
• Abstract: Diffusion models, widely used for image and video generation, face a significant limitation: the risk of memorizing and reproducing training data during inference, potentially generating unauthorized copyrighted content. While prior research has focused on image diffusion models (IDMs), video diffusion models (VDMs) remain underexplored. To address this gap, we first formally define the two types of memorization in VDMs (content memorization and motion memorization) in a practical way that focuses on privacy preservation and applies to all generation types. We then introduce new metrics specifically designed to separately assess content and motion memorization in VDMs. Additionally, we curate a dataset of text prompts that are most prone to triggering memorization when used as conditioning in VDMs. By leveraging these prompts, we generate diverse videos from various open-source VDMs, successfully extracting numerous training videos from each tested model. Through the application of our proposed metrics, we systematically analyze memorization across various pretrained VDMs, including text-conditional and unconditional models, on a variety of datasets. Our comprehensive study reveals that memorization is widespread across all tested VDMs, indicating that VDMs can also memorize image training data in addition to video datasets. Finally, we propose efficient and effective detection strategies for both content and motion memorization, offering a foundational approach for improving privacy in VDMs.

Title: Pushing the Limits of All-Atom Geometric Graph Neural Networks: Pre-Training, Scaling and Zero-Shot Transfer

• Authors: Zihan Pengmei, Zhengyuan Shen, Zichen Wang, Marcus Collins, Huzefa Rangwala
• Subjects: cs.LG, physics.chem-ph
• Abstract URL: https://arxiv.org/abs/2410.21683
• Pdf URL: https://arxiv.org/pdf/2410.21683
• Copy Paste: [[2410.21683]] Pushing the Limits of All-Atom Geometric Graph Neural Networks: Pre-Training, Scaling and Zero-Shot Transfer(https://arxiv.org/abs/2410.21683)
• Keywords: self-supervised
• Abstract: Constructing transferable descriptors for conformation representation of molecular and biological systems finds numerous applications in drug discovery, learning-based molecular dynamics, and protein mechanism analysis. Geometric graph neural networks (Geom-GNNs) with all-atom information have transformed atomistic simulations by serving as a general learnable geometric descriptors for downstream tasks including prediction of interatomic potential and molecular properties. However, common practices involve supervising Geom-GNNs on specific downstream tasks, which suffer from the lack of high-quality data and inaccurate labels leading to poor generalization and performance degradation on out-of-distribution (OOD) scenarios. In this work, we explored the possibility of using pre-trained Geom-GNNs as transferable and highly effective geometric descriptors for improved generalization. To explore their representation power, we studied the scaling behaviors of Geom-GNNs under self-supervised pre-training, supervised and unsupervised learning setups. We find that the expressive power of different architectures can differ on the pre-training task. Interestingly, Geom-GNNs do not follow the power-law scaling on the pre-training task, and universally lack predictable scaling behavior on the supervised tasks with quantum chemical labels important for screening and design of novel molecules. More importantly, we demonstrate how all-atom graph embedding can be organically combined with other neural architectures to enhance the expressive power. Meanwhile, the low-dimensional projection of the latent space shows excellent agreement with conventional geometrical descriptors.

Title: On the Role of Depth and Looping for In-Context Learning with Task Diversity

• Authors: Khashayar Gatmiry, Nikunj Saunshi, Sashank J. Reddi, Stefanie Jegelka, Sanjiv Kumar
• Subjects: cs.LG, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.21698
• Pdf URL: https://arxiv.org/pdf/2410.21698
• Copy Paste: [[2410.21698]] On the Role of Depth and Looping for In-Context Learning with Task Diversity(https://arxiv.org/abs/2410.21698)
• Keywords: in-context
• Abstract: The intriguing in-context learning (ICL) abilities of deep Transformer models have lately garnered significant attention. By studying in-context linear regression on unimodal Gaussian data, recent empirical and theoretical works have argued that ICL emerges from Transformers' abilities to simulate learning algorithms like gradient descent. However, these works fail to capture the remarkable ability of Transformers to learn multiple tasks in context. To this end, we study in-context learning for linear regression with diverse tasks, characterized by data covariance matrices with condition numbers ranging from $[1, \kappa]$, and highlight the importance of depth in this setting. More specifically, (a) we show theoretical lower bounds of $\log(\kappa)$ (or $\sqrt{\kappa}$) linear attention layers in the unrestricted (or restricted) attention setting and, (b) we show that multilayer Transformers can indeed solve such tasks with a number of layers that matches the lower bounds. However, we show that this expressivity of multilayer Transformer comes at the price of robustness. In particular, multilayer Transformers are not robust to even distributional shifts as small as $O(e^{-L})$ in Wasserstein distance, where $L$ is the depth of the network. We then demonstrate that Looped Transformers -- a special class of multilayer Transformers with weight-sharing -- not only exhibit similar expressive power but are also provably robust under mild assumptions. Besides out-of-distribution generalization, we also show that Looped Transformers are the only models that exhibit a monotonic behavior of loss with respect to depth.

Title: AdaptGCD: Multi-Expert Adapter Tuning for Generalized Category Discovery

• Authors: Yuxun Qu, Yongqiang Tang, Chenyang Zhang, Wensheng Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21705
• Pdf URL: https://arxiv.org/pdf/2410.21705
• Copy Paste: [[2410.21705]] AdaptGCD: Multi-Expert Adapter Tuning for Generalized Category Discovery(https://arxiv.org/abs/2410.21705)
• Keywords: self-supervised
• Abstract: Different from the traditional semi-supervised learning paradigm that is constrained by the close-world assumption, Generalized Category Discovery (GCD) presumes that the unlabeled dataset contains new categories not appearing in the labeled set, and aims to not only classify old categories but also discover new categories in the unlabeled data. Existing studies on GCD typically devote to transferring the general knowledge from the self-supervised pretrained model to the target GCD task via some fine-tuning strategies, such as partial tuning and prompt learning. Nevertheless, these fine-tuning methods fail to make a sound balance between the generalization capacity of pretrained backbone and the adaptability to the GCD task. To fill this gap, in this paper, we propose a novel adapter-tuning-based method named AdaptGCD, which is the first work to introduce the adapter tuning into the GCD task and provides some key insights expected to enlighten future research. Furthermore, considering the discrepancy of supervision information between the old and new classes, a multi-expert adapter structure equipped with a route assignment constraint is elaborately devised, such that the data from old and new classes are separated into different expert groups. Extensive experiments are conducted on 7 widely-used datasets. The remarkable improvements in performance highlight the effectiveness of our proposals.

Title: Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation

• Authors: Ruihao Xia, Yu Liang, Peng-Tao Jiang, Hao Zhang, Bo Li, Yang Tang, Pan Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21708
• Pdf URL: https://arxiv.org/pdf/2410.21708
• Copy Paste: [[2410.21708]] Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation(https://arxiv.org/abs/2410.21708)
• Keywords: diffusion
• Abstract: Despite their success, unsupervised domain adaptation methods for semantic segmentation primarily focus on adaptation between image domains and do not utilize other abundant visual modalities like depth, infrared and event. This limitation hinders their performance and restricts their application in real-world multimodal scenarios. To address this issue, we propose Modality Adaptation with text-to-image Diffusion Models (MADM) for semantic segmentation task which utilizes text-to-image diffusion models pre-trained on extensive image-text pairs to enhance the model's cross-modality capabilities. Specifically, MADM comprises two key complementary components to tackle major challenges. First, due to the large modality gap, using one modal data to generate pseudo labels for another modality suffers from a significant drop in accuracy. To address this, MADM designs diffusion-based pseudo-label generation which adds latent noise to stabilize pseudo-labels and enhance label accuracy. Second, to overcome the limitations of latent low-resolution features in diffusion models, MADM introduces the label palette and latent regression which converts one-hot encoded labels into the RGB form by palette and regresses them in the latent space, thus ensuring the pre-trained decoder for up-sampling to obtain fine-grained features. Extensive experimental results demonstrate that MADM achieves state-of-the-art adaptation performance across various modality tasks, including images to depth, infrared, and event modalities. We open-source our code and models at this https URL.

Title: Sliced-Wasserstein-based Anomaly Detection and Open Dataset for Localized Critical Peak Rebates

• Authors: Julien Pallage, Bertrand Scherrer, Salma Naccache, Christophe Bélanger, Antoine Lesage-Landry
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21712
• Pdf URL: https://arxiv.org/pdf/2410.21712
• Copy Paste: [[2410.21712]] Sliced-Wasserstein-based Anomaly Detection and Open Dataset for Localized Critical Peak Rebates(https://arxiv.org/abs/2410.21712)
• Keywords: anomaly
• Abstract: In this work, we present a new unsupervised anomaly (outlier) detection (AD) method using the sliced-Wasserstein metric. This filtering technique is conceptually interesting for integration in MLOps pipelines deploying trustworthy machine learning models in critical sectors like energy. Additionally, we open the first dataset showcasing localized critical peak rebate demand response in a northern climate. We demonstrate the capabilities of our method on synthetic datasets as well as standard AD datasets and use it in the making of a first benchmark for our open-source localized critical peak rebate dataset.

Title: Generating Realistic Tabular Data with Large Language Models

• Authors: Dang Nguyen, Sunil Gupta, Kien Do, Thin Nguyen, Svetha Venkatesh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21717
• Pdf URL: https://arxiv.org/pdf/2410.21717
• Copy Paste: [[2410.21717]] Generating Realistic Tabular Data with Large Language Models(https://arxiv.org/abs/2410.21717)
• Keywords: generative
• Abstract: While most generative models show achievements in image data generation, few are developed for tabular data generation. Recently, due to success of large language models (LLM) in diverse tasks, they have also been used for tabular data generation. However, these methods do not capture the correct correlation between the features and the target variable, hindering their applications in downstream predictive tasks. To address this problem, we propose a LLM-based method with three important improvements to correctly capture the ground-truth feature-class correlation in the real data. First, we propose a novel permutation strategy for the input data in the fine-tuning phase. Second, we propose a feature-conditional sampling approach to generate synthetic samples. Finally, we generate the labels by constructing prompts based on the generated samples to query our fine-tuned LLM. Our extensive experiments show that our method significantly outperforms 10 SOTA baselines on 20 datasets in downstream tasks. It also produces highly realistic synthetic samples in terms of quality and diversity. More importantly, classifiers trained with our synthetic data can even compete with classifiers trained with the original data on half of the benchmark datasets, which is a significant achievement in tabular data generation.

Title: DiffSTR: Controlled Diffusion Models for Scene Text Removal

• Authors: Sanhita Pathak, Vinay Kaushik, Brejesh Lall
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21721
• Pdf URL: https://arxiv.org/pdf/2410.21721
• Copy Paste: [[2410.21721]] DiffSTR: Controlled Diffusion Models for Scene Text Removal(https://arxiv.org/abs/2410.21721)
• Keywords: diffusion
• Abstract: To prevent unauthorized use of text in images, Scene Text Removal (STR) has become a crucial task. It focuses on automatically removing text and replacing it with a natural, text-less background while preserving significant details such as texture, color, and contrast. Despite its importance in privacy protection, STR faces several challenges, including boundary artifacts, inconsistent texture and color, and preserving correct shadows. Most STR approaches estimate a text region mask to train a model, solving for image translation or inpainting to generate a text-free image. Thus, the quality of the generated image depends on the accuracy of the inpainting mask and the generator's capability. In this work, we leverage the superior capabilities of diffusion models in generating high-quality, consistent images to address the STR problem. We introduce a ControlNet diffusion model, treating STR as an inpainting task. To enhance the model's robustness, we develop a mask pretraining pipeline to condition our diffusion model. This involves training a masked autoencoder (MAE) using a combination of box masks and coarse stroke masks, and fine-tuning it using masks derived from our novel segmentation-based mask refinement framework. This framework iteratively refines an initial mask and segments it using the SLIC and Hierarchical Feature Selection (HFS) algorithms to produce an accurate final text mask. This improves mask prediction and utilizes rich textural information in natural scene images to provide accurate inpainting masks. Experiments on the SCUT-EnsText and SCUT-Syn datasets demonstrate that our method significantly outperforms existing state-of-the-art techniques.

Title: IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models

• Authors: Hang Guo, Yawei Li, Tao Dai, Shu-Tao Xia, Luca Benini
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21759
• Pdf URL: https://arxiv.org/pdf/2410.21759
• Copy Paste: [[2410.21759]] IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models(https://arxiv.org/abs/2410.21759)
• Keywords: diffusion
• Abstract: Fine-tuning large-scale text-to-image diffusion models for various downstream tasks has yielded impressive results. However, the heavy computational burdens of tuning large models prevent personal customization. Recent advances have attempted to employ parameter-efficient fine-tuning (PEFT) techniques to adapt the floating-point (FP) or quantized pre-trained weights. Nonetheless, the adaptation parameters in existing works are still restricted to FP arithmetic, hindering hardware-friendly acceleration. In this work, we propose IntLoRA, to further push the efficiency limits by using integer type (INT) low-rank parameters to adapt the quantized diffusion models. By working in the integer arithmetic, our IntLoRA offers three key advantages: (i) for fine-tuning, the pre-trained weights are quantized, reducing memory usage; (ii) for storage, both pre-trained and low-rank weights are in INT which consumes less disk space; (iii) for inference, IntLoRA weights can be naturally merged into quantized pre-trained weights through efficient integer multiplication or bit-shifting, eliminating additional post-training quantization. Extensive experiments demonstrate that IntLoRA can achieve performance on par with or even superior to the vanilla LoRA, accompanied by significant efficiency improvements. Code is available at \url{this https URL}.

Title: HairDiffusion: Vivid Multi-Colored Hair Editing via Latent Diffusion

• Authors: Yu Zeng, Yang Zhang, Jiachen Liu, Linlin Shen, Kaijun Deng, Weizhao He, Jinbao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21789
• Pdf URL: https://arxiv.org/pdf/2410.21789
• Copy Paste: [[2410.21789]] HairDiffusion: Vivid Multi-Colored Hair Editing via Latent Diffusion(https://arxiv.org/abs/2410.21789)
• Keywords: diffusion
• Abstract: Hair editing is a critical image synthesis task that aims to edit hair color and hairstyle using text descriptions or reference images, while preserving irrelevant attributes (e.g., identity, background, cloth). Many existing methods are based on StyleGAN to address this task. However, due to the limited spatial distribution of StyleGAN, it struggles with multiple hair color editing and facial preservation. Considering the advancements in diffusion models, we utilize Latent Diffusion Models (LDMs) for hairstyle editing. Our approach introduces Multi-stage Hairstyle Blend (MHB), effectively separating control of hair color and hairstyle in diffusion latent space. Additionally, we train a warping module to align the hair color with the target region. To further enhance multi-color hairstyle editing, we fine-tuned a CLIP model using a multi-color hairstyle dataset. Our method not only tackles the complexity of multi-color hairstyles but also addresses the challenge of preserving original colors during diffusion editing. Extensive experiments showcase the superiority of our method in editing multi-color hairstyles while preserving facial attributes given textual descriptions and reference images.

Title: SimSiam Naming Game: A Unified Approach for Representation Learning and Emergent Communication

• Authors: Nguyen Le Hoang, Tadahiro Taniguchi, Fang Tianwei, Akira Taniguchi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21803
• Pdf URL: https://arxiv.org/pdf/2410.21803
• Copy Paste: [[2410.21803]] SimSiam Naming Game: A Unified Approach for Representation Learning and Emergent Communication(https://arxiv.org/abs/2410.21803)
• Keywords: self-supervised, generative
• Abstract: Emergent communication, driven by generative models, enables agents to develop a shared language for describing their individual views of the same objects through interactions. Meanwhile, self-supervised learning (SSL), particularly SimSiam, uses discriminative representation learning to make representations of augmented views of the same data point closer in the representation space. Building on the prior work of VI-SimSiam, which incorporates a generative and Bayesian perspective into the SimSiam framework via variational inference (VI) interpretation, we propose SimSiam+VAE, a unified approach for both representation learning and emergent communication. SimSiam+VAE integrates a variational autoencoder (VAE) into the predictor of the SimSiam network to enhance representation learning and capture uncertainty. Experimental results show that SimSiam+VAE outperforms both SimSiam and VI-SimSiam. We further extend this model into a communication framework called the SimSiam Naming Game (SSNG), which applies the generative and Bayesian approach based on VI to develop internal representations and emergent language, while utilizing the discriminative process of SimSiam to facilitate mutual understanding between agents. In experiments with established models, despite the dynamic alternation of agent roles during interactions, SSNG demonstrates comparable performance to the referential game and slightly outperforms the Metropolis-Hastings naming game.

Title: Volumetric Conditioning Module to Control Pretrained Diffusion Models for 3D Medical Images

• Authors: Suhyun Ahn, Wonjung Park, Jihoon Cho, Seunghyuck Park, Jinah Park
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21826
• Pdf URL: https://arxiv.org/pdf/2410.21826
• Copy Paste: [[2410.21826]] Volumetric Conditioning Module to Control Pretrained Diffusion Models for 3D Medical Images(https://arxiv.org/abs/2410.21826)
• Keywords: diffusion
• Abstract: Spatial control methods using additional modules on pretrained diffusion models have gained attention for enabling conditional generation in natural images. These methods guide the generation process with new conditions while leveraging the capabilities of large models. They could be beneficial as training strategies in the context of 3D medical imaging, where training a diffusion model from scratch is challenging due to high computational costs and data scarcity. However, the potential application of spatial control methods with additional modules to 3D medical images has not yet been explored. In this paper, we present a tailored spatial control method for 3D medical images with a novel lightweight module, Volumetric Conditioning Module (VCM). Our VCM employs an asymmetric U-Net architecture to effectively encode complex information from various levels of 3D conditions, providing detailed guidance in image synthesis. To examine the applicability of spatial control methods and the effectiveness of VCM for 3D medical data, we conduct experiments under single- and multimodal conditions scenarios across a wide range of dataset sizes, from extremely small datasets with 10 samples to large datasets with 500 samples. The experimental results show that the VCM is effective for conditional generation and efficient in terms of requiring less training data and computational resources. We further investigate the potential applications for our spatial control method through axial super-resolution for medical images. Our code is available at \url{this https URL}

Title: Diffusion as Reasoning: Enhancing Object Goal Navigation with LLM-Biased Diffusion Model

• Authors: Yiming Ji, Yang Liu, Zhengpu Wang, Boyu Ma, Zongwu Xie, Hong Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21842
• Pdf URL: https://arxiv.org/pdf/2410.21842
• Copy Paste: [[2410.21842]] Diffusion as Reasoning: Enhancing Object Goal Navigation with LLM-Biased Diffusion Model(https://arxiv.org/abs/2410.21842)
• Keywords: diffusion
• Abstract: The Object Goal Navigation (ObjectNav) task requires the agent to navigate to a specified target in an unseen environment. Since the environment layout is unknown, the agent needs to perform semantic reasoning to infer the potential location of the target, based on its accumulated memory of the environment during the navigation process. Diffusion models have been shown to be able to learn the distribution relationships between features in RGB images, and thus generate new realistic this http URL this work, we propose a new approach to solving the ObjectNav task, by training a diffusion model to learn the statistical distribution patterns of objects in semantic maps, and using the map of the explored regions during navigation as the condition to generate the map of the unknown regions, thereby realizing the semantic reasoning of the target object, i.e., diffusion as reasoning (DAR). Meanwhile, we propose the global target bias and local LLM bias methods, where the former can constrain the diffusion model to generate the target object more effectively, and the latter utilizes the common sense knowledge extracted from the LLM to improve the generalization of the reasoning process. Based on the generated map in the unknown region, the agent sets the predicted location of the target as the goal and moves towards it. Experiments on Gibson and MP3D show the effectiveness of our method.

Title: Improving In-Context Learning with Small Language Model Ensembles

• Authors: M. Mehdi Mojarradi, Lingyi Yang, Robert McCraith, Adam Mahdi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21868
• Pdf URL: https://arxiv.org/pdf/2410.21868
• Copy Paste: [[2410.21868]] Improving In-Context Learning with Small Language Model Ensembles(https://arxiv.org/abs/2410.21868)
• Keywords: in-context
• Abstract: Large language models (LLMs) have shown impressive capabilities across various tasks, but their performance on domain-specific tasks remains limited. While methods like retrieval augmented generation and fine-tuning can help to address this, they require significant resources. In-context learning (ICL) is a cheap and efficient alternative but cannot match the accuracies of advanced methods. We present Ensemble SuperICL, a novel approach that enhances ICL by leveraging the expertise of multiple fine-tuned small language models (SLMs). Ensemble SuperICL achieves state of the art (SoTA) results on several natural language understanding benchmarks. Additionally, we test it on a medical-domain labelling task and showcase its practicality by using off-the-shelf SLMs fine-tuned on a general language task, achieving superior accuracy in large-scale data labelling compared to all baselines. Finally, we conduct an ablation study and sensitivity analyses to elucidate the underlying mechanism of Ensemble SuperICL. Our research contributes to the growing demand for efficient domain specialisation methods in LLMs, offering a cheap and effective method for practitioners.

Title: Cross-Entropy Is All You Need To Invert the Data Generating Process

• Authors: Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E. Vogt, Randall Balestriero, Wieland Brendel, David Klindt
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.21869
• Pdf URL: https://arxiv.org/pdf/2410.21869
• Copy Paste: [[2410.21869]] Cross-Entropy Is All You Need To Invert the Data Generating Process(https://arxiv.org/abs/2410.21869)
• Keywords: self-supervised
• Abstract: Supervised learning has become a cornerstone of modern machine learning, yet a comprehensive theory explaining its effectiveness remains elusive. Empirical phenomena, such as neural analogy-making and the linear representation hypothesis, suggest that supervised models can learn interpretable factors of variation in a linear fashion. Recent advances in self-supervised learning, particularly nonlinear Independent Component Analysis, have shown that these methods can recover latent structures by inverting the data generating process. We extend these identifiability results to parametric instance discrimination, then show how insights transfer to the ubiquitous setting of supervised learning with cross-entropy minimization. We prove that even in standard classification tasks, models learn representations of ground-truth factors of variation up to a linear transformation. We corroborate our theoretical contribution with a series of empirical studies. First, using simulated data matching our theoretical assumptions, we demonstrate successful disentanglement of latent factors. Second, we show that on DisLib, a widely-used disentanglement benchmark, simple classification tasks recover latent structures up to linear transformations. Finally, we reveal that models trained on ImageNet encode representations that permit linear decoding of proxy factors of variation. Together, our theoretical findings and experiments offer a compelling explanation for recent observations of linear representations, such as superposition in neural networks. This work takes a significant step toward a cohesive theory that accounts for the unreasonable effectiveness of supervised deep learning.

Title: LogSHIELD: A Graph-based Real-time Anomaly Detection Framework using Frequency Analysis

• Authors: Krishna Chandra Roy, Qian Chen
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.21936
• Pdf URL: https://arxiv.org/pdf/2410.21936
• Copy Paste: [[2410.21936]] LogSHIELD: A Graph-based Real-time Anomaly Detection Framework using Frequency Analysis(https://arxiv.org/abs/2410.21936)
• Keywords: anomaly
• Abstract: Anomaly-based cyber threat detection using deep learning is on a constant growth in popularity for novel cyber-attack detection and forensics. A robust, efficient, and real-time threat detector in a large-scale operational enterprise network requires high accuracy, high fidelity, and a high throughput model to detect malicious activities. Traditional anomaly-based detection models, however, suffer from high computational overhead and low detection accuracy, making them unsuitable for real-time threat detection. In this work, we propose LogSHIELD, a highly effective graph-based anomaly detection model in host data. We present a real-time threat detection approach using frequency-domain analysis of provenance graphs. To demonstrate the significance of graph-based frequency analysis we proposed two approaches. Approach-I uses a Graph Neural Network (GNN) LogGNN and approach-II performs frequency domain analysis on graph node samples for graph embedding. Both approaches use a statistical clustering algorithm for anomaly detection. The proposed models are evaluated using a large host log dataset consisting of 774M benign logs and 375K malware logs. LogSHIELD explores the provenance graph to extract contextual and causal relationships among logs, exposing abnormal activities. It can detect stealthy and sophisticated attacks with over 98% average AUC and F1 scores. It significantly improves throughput, achieves an average detection latency of 0.13 seconds, and outperforms state-of-the-art models in detection time.

Title: ReMix: Training Generalized Person Re-identification on a Mixture of Data

• Authors: Timur Mamedov, Anton Konushin, Vadim Konushin
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.21938
• Pdf URL: https://arxiv.org/pdf/2410.21938
• Copy Paste: [[2410.21938]] ReMix: Training Generalized Person Re-identification on a Mixture of Data(https://arxiv.org/abs/2410.21938)
• Keywords: self-supervised
• Abstract: Modern person re-identification (Re-ID) methods have a weak generalization ability and experience a major accuracy drop when capturing environments change. This is because existing multi-camera Re-ID datasets are limited in size and diversity, since such data is difficult to obtain. At the same time, enormous volumes of unlabeled single-camera records are available. Such data can be easily collected, and therefore, it is more diverse. Currently, single-camera data is used only for self-supervised pre-training of Re-ID methods. However, the diversity of single-camera data is suppressed by fine-tuning on limited multi-camera data after pre-training. In this paper, we propose ReMix, a generalized Re-ID method jointly trained on a mixture of limited labeled multi-camera and large unlabeled single-camera data. Effective training of our method is achieved through a novel data sampling strategy and new loss functions that are adapted for joint use with both types of data. Experiments show that ReMix has a high generalization ability and outperforms state-of-the-art methods in generalizable person Re-ID. To the best of our knowledge, this is the first work that explores joint training on a mixture of multi-camera and single-camera data in person Re-ID.

Title: SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types

• Authors: Yutao Mou, Shikun Zhang, Wei Ye
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.21965
• Pdf URL: https://arxiv.org/pdf/2410.21965
• Copy Paste: [[2410.21965]] SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types(https://arxiv.org/abs/2410.21965)
• Keywords: generative
• Abstract: Ensuring the safety of large language model (LLM) applications is essential for developing trustworthy artificial intelligence. Current LLM safety benchmarks have two limitations. First, they focus solely on either discriminative or generative evaluation paradigms while ignoring their interconnection. Second, they rely on standardized inputs, overlooking the effects of widespread prompting techniques, such as system prompts, few-shot demonstrations, and chain-of-thought prompting. To overcome these issues, we developed SG-Bench, a novel benchmark to assess the generalization of LLM safety across various tasks and prompt types. This benchmark integrates both generative and discriminative evaluation tasks and includes extended data to examine the impact of prompt engineering and jailbreak on LLM safety. Our assessment of 3 advanced proprietary LLMs and 10 open-source LLMs with the benchmark reveals that most LLMs perform worse on discriminative tasks than generative ones, and are highly susceptible to prompts, indicating poor generalization in safety alignment. We also explain these findings quantitatively and qualitatively to provide insights for future research.

Title: PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference

• Authors: Kendong Liu, Zhiyu Zhu, Chuanhao Li, Hui Liu, Huanqiang Zeng, Junhui Hou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21966
• Pdf URL: https://arxiv.org/pdf/2410.21966
• Copy Paste: [[2410.21966]] PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference(https://arxiv.org/abs/2410.21966)
• Keywords: diffusion, generative
• Abstract: In this paper, we make the first attempt to align diffusion models for image inpainting with human aesthetic standards via a reinforcement learning framework, significantly improving the quality and visual appeal of inpainted images. Specifically, instead of directly measuring the divergence with paired images, we train a reward model with the dataset we construct, consisting of nearly 51,000 images annotated with human preferences. Then, we adopt a reinforcement learning process to fine-tune the distribution of a pre-trained diffusion model for image inpainting in the direction of higher reward. Moreover, we theoretically deduce the upper bound on the error of the reward model, which illustrates the potential confidence of reward estimation throughout the reinforcement alignment process, thereby facilitating accurate regularization. Extensive experiments on inpainting comparison and downstream tasks, such as image extension and 3D reconstruction, demonstrate the effectiveness of our approach, showing significant improvements in the alignment of inpainted images with human preference compared with state-of-the-art methods. This research not only advances the field of image inpainting but also provides a framework for incorporating human preference into the iterative refinement of generative models based on modeling reward accuracy, with broad implications for the design of visually driven AI applications. Our code and dataset are publicly available at this https URL.

Title: A Survey on RGB, 3D, and Multimodal Approaches for Unsupervised Industrial Anomaly Detection

• Authors: Yuxuan Lin, Yang Chang, Xuan Tong, Jiawen Yu, Antonio Liotta, Guofan Huang, Wei Song, Deyu Zeng, Zongze Wu, Yan Wang, Wenqiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.21982
• Pdf URL: https://arxiv.org/pdf/2410.21982
• Copy Paste: [[2410.21982]] A Survey on RGB, 3D, and Multimodal Approaches for Unsupervised Industrial Anomaly Detection(https://arxiv.org/abs/2410.21982)
• Keywords: anomaly
• Abstract: In the advancement of industrial informatization, Unsupervised Industrial Anomaly Detection (UIAD) technology effectively overcomes the scarcity of abnormal samples and significantly enhances the automation and reliability of smart manufacturing. While RGB, 3D, and multimodal anomaly detection have demonstrated comprehensive and robust capabilities within the industrial informatization sector, existing reviews on industrial anomaly detection have not sufficiently classified and discussed methods in 3D and multimodal settings. We focus on 3D UIAD and multimodal UIAD, providing a comprehensive summary of unsupervised industrial anomaly detection in three modal settings. Firstly, we compare our surveys with recent works, introducing commonly used datasets, evaluation metrics, and the definitions of anomaly detection problems. Secondly, we summarize five research paradigms in RGB, 3D and multimodal UIAD and three emerging industrial manufacturing optimization directions in RGB UIAD, and review three multimodal feature fusion strategies in multimodal settings. Finally, we outline the primary challenges currently faced by UIAD in three modal settings, and offer insights into future development directions, aiming to provide researchers with a thorough reference and offer new perspectives for the advancement of industrial informatization. Corresponding resources are available at this https URL.

Title: FreeGaussian: Guidance-free Controllable 3D Gaussian Splats with Flow Derivatives

• Authors: Qizhi Chen, Delin Qu, Yiwen Tang, Haoming Song, Yiting Zhang, Dong Wang, Bin Zhao, Xuelong Li
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.22070
• Pdf URL: https://arxiv.org/pdf/2410.22070
• Copy Paste: [[2410.22070]] FreeGaussian: Guidance-free Controllable 3D Gaussian Splats with Flow Derivatives(https://arxiv.org/abs/2410.22070)
• Keywords: self-supervised
• Abstract: Reconstructing controllable Gaussian splats from monocular video is a challenging task due to its inherently insufficient constraints. Widely adopted approaches supervise complex interactions with additional masks and control signal annotations, limiting their real-world applications. In this paper, we propose an annotation guidance-free method, dubbed FreeGaussian, that mathematically derives dynamic Gaussian motion from optical flow and camera motion using novel dynamic Gaussian constraints. By establishing a connection between 2D flows and 3D Gaussian dynamic control, our method enables self-supervised optimization and continuity of dynamic Gaussian motions from flow priors. Furthermore, we introduce a 3D spherical vector controlling scheme, which represents the state with a 3D Gaussian trajectory, thereby eliminating the need for complex 1D control signal calculations and simplifying controllable Gaussian modeling. Quantitative and qualitative evaluations on extensive experiments demonstrate the state-of-the-art visual performance and control capability of our method. Project page: this https URL.

Title: TractShapeNet: Efficient Multi-Shape Learning with 3D Tractography Point Clouds

• Authors: Yui Lo, Yuqian Chen, Dongnan Liu, Jon Haitz Legarreta, Leo Zekelman, Fan Zhang, Jarrett Rushmore, Yogesh Rathi, Nikos Makris, Alexandra J. Golby, Weidong Cai, Lauren J. O'Donnell
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.22099
• Pdf URL: https://arxiv.org/pdf/2410.22099
• Copy Paste: [[2410.22099]] TractShapeNet: Efficient Multi-Shape Learning with 3D Tractography Point Clouds(https://arxiv.org/abs/2410.22099)
• Keywords: diffusion
• Abstract: Brain imaging studies have demonstrated that diffusion MRI tractography geometric shape descriptors can inform the study of the brain's white matter pathways and their relationship to brain function. In this work, we investigate the possibility of utilizing a deep learning model to compute shape measures of the brain's white matter connections. We introduce a novel framework, TractShapeNet, that leverages a point cloud representation of tractography to compute five shape measures: length, span, volume, total surface area, and irregularity. We assess the performance of the method on a large dataset including 1065 healthy young adults. Experiments for shape measure computation demonstrate that our proposed TractShapeNet outperforms other point cloud-based neural network models in both the Pearson correlation coefficient and normalized error metrics. We compare the inference runtime results with the conventional shape computation tool DSI-Studio. Our results demonstrate that a deep learning approach enables faster and more efficient shape measure computation. We also conduct experiments on two downstream language cognition prediction tasks, showing that shape measures from TractShapeNet perform similarly to those computed by DSI-Studio. Our code will be available at: this https URL.

Title: Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench

• Authors: Zheyuan Liu, Guangyao Dou, Mengzhao Jia, Zhaoxuan Tan, Qingkai Zeng, Yongle Yuan, Meng Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2410.22108
• Pdf URL: https://arxiv.org/pdf/2410.22108
• Copy Paste: [[2410.22108]] Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench(https://arxiv.org/abs/2410.22108)
• Keywords: generative
• Abstract: Generative models such as Large Language Models (LLM) and Multimodal Large Language models (MLLMs) trained on massive web corpora can memorize and disclose individuals' confidential and private data, raising legal and ethical concerns. While many previous works have addressed this issue in LLM via machine unlearning, it remains largely unexplored for MLLMs. To tackle this challenge, we introduce Multimodal Large Language Model Unlearning Benchmark (MLLMU-Bench), a novel benchmark aimed at advancing the understanding of multimodal machine unlearning. MLLMU-Bench consists of 500 fictitious profiles and 153 profiles for public celebrities, each profile feature over 14 customized question-answer pairs, evaluated from both multimodal (image+text) and unimodal (text) perspectives. The benchmark is divided into four sets to assess unlearning algorithms in terms of efficacy, generalizability, and model utility. Finally, we provide baseline results using existing generative model unlearning algorithms. Surprisingly, our experiments show that unimodal unlearning algorithms excel in generation and cloze tasks, while multimodal unlearning approaches perform better in classification tasks with multimodal inputs.

Title: AmpleGCG-Plus: A Strong Generative Model of Adversarial Suffixes to Jailbreak LLMs with Higher Success Rates in Fewer Attempts

• Authors: Vishal Kumar, Zeyi Liao, Jaylen Jones, Huan Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22143
• Pdf URL: https://arxiv.org/pdf/2410.22143
• Copy Paste: [[2410.22143]] AmpleGCG-Plus: A Strong Generative Model of Adversarial Suffixes to Jailbreak LLMs with Higher Success Rates in Fewer Attempts(https://arxiv.org/abs/2410.22143)
• Keywords: generative
• Abstract: Although large language models (LLMs) are typically aligned, they remain vulnerable to jailbreaking through either carefully crafted prompts in natural language or, interestingly, gibberish adversarial suffixes. However, gibberish tokens have received relatively less attention despite their success in attacking aligned LLMs. Recent work, AmpleGCG~\citep{liao2024amplegcg}, demonstrates that a generative model can quickly produce numerous customizable gibberish adversarial suffixes for any harmful query, exposing a range of alignment gaps in out-of-distribution (OOD) language spaces. To bring more attention to this area, we introduce AmpleGCG-Plus, an enhanced version that achieves better performance in fewer attempts. Through a series of exploratory experiments, we identify several training strategies to improve the learning of gibberish suffixes. Our results, verified under a strict evaluation setting, show that it outperforms AmpleGCG on both open-weight and closed-source models, achieving increases in attack success rate (ASR) of up to 17\% in the white-box setting against Llama-2-7B-chat, and more than tripling ASR in the black-box setting against GPT-4. Notably, AmpleGCG-Plus jailbreaks the newer GPT-4o series of models at similar rates to GPT-4, and, uncovers vulnerabilities against the recently proposed circuit breakers defense. We publicly release AmpleGCG-Plus along with our collected training datasets.

Title: Capacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models

• Authors: Raman Dutt, Pedro Sanchez, Ondrej Bohdal, Sotirios A. Tsaftaris, Timothy Hospedales
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22149
• Pdf URL: https://arxiv.org/pdf/2410.22149
• Copy Paste: [[2410.22149]] Capacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models(https://arxiv.org/abs/2410.22149)
• Keywords: diffusion
• Abstract: In this work, we present compelling evidence that controlling model capacity during fine-tuning can effectively mitigate memorization in diffusion models. Specifically, we demonstrate that adopting Parameter-Efficient Fine-Tuning (PEFT) within the pre-train fine-tune paradigm significantly reduces memorization compared to traditional full fine-tuning approaches. Our experiments utilize the MIMIC dataset, which comprises image-text pairs of chest X-rays and their corresponding reports. The results, evaluated through a range of memorization and generation quality metrics, indicate that PEFT not only diminishes memorization but also enhances downstream generation quality. Additionally, PEFT methods can be seamlessly combined with existing memorization mitigation techniques for further improvement. The code for our experiments is available at: this https URL

Title: Standardization Trends on Safety and Trustworthiness Technology for Advanced AI

• Authors: Jonghong Jeon
• Subjects: cs.LG, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2410.22151
• Pdf URL: https://arxiv.org/pdf/2410.22151
• Copy Paste: [[2410.22151]] Standardization Trends on Safety and Trustworthiness Technology for Advanced AI(https://arxiv.org/abs/2410.22151)
• Keywords: foundation model
• Abstract: Artificial Intelligence (AI) has rapidly evolved over the past decade and has advanced in areas such as language comprehension, image and video recognition, programming, and scientific reasoning. Recent AI technologies based on large language models and foundation models are approaching or surpassing artificial general intelligence. These systems demonstrate superior performance in complex problem solving, natural language processing, and multi-domain tasks, and can potentially transform fields such as science, industry, healthcare, and education. However, these advancements have raised concerns regarding the safety and trustworthiness of advanced AI, including risks related to uncontrollability, ethical conflicts, long-term socioeconomic impacts, and safety assurance. Efforts are being expended to develop internationally agreed-upon standards to ensure the safety and reliability of AI. This study analyzes international trends in safety and trustworthiness standardization for advanced AI, identifies key areas for standardization, proposes future directions and strategies, and draws policy implications. The goal is to support the safe and trustworthy development of advanced AI and enhance international competitiveness through effective standardization.

Title: Benchmarking LLM Guardrails in Handling Multilingual Toxicity

• Authors: Yahan Yang, Soham Dan, Dan Roth, Insup Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2410.22153
• Pdf URL: https://arxiv.org/pdf/2410.22153
• Copy Paste: [[2410.22153]] Benchmarking LLM Guardrails in Handling Multilingual Toxicity(https://arxiv.org/abs/2410.22153)
• Keywords: in-context
• Abstract: With the ubiquity of Large Language Models (LLMs), guardrails have become crucial to detect and defend against toxic content. However, with the increasing pervasiveness of LLMs in multilingual scenarios, their effectiveness in handling multilingual toxic inputs remains unclear. In this work, we introduce a comprehensive multilingual test suite, spanning seven datasets and over ten languages, to benchmark the performance of state-of-the-art guardrails. We also investigates the resilience of guardrails against recent jailbreaking techniques, and assess the impact of in-context safety policies and language resource availability on guardrails' performance. Our findings show that existing guardrails are still ineffective at handling multilingual toxicity and lack robustness against jailbreaking prompts. This work aims to identify the limitations of guardrails and to build a more reliable and trustworthy LLMs in multilingual scenarios.

Title: Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspective

• Authors: Shenghao Xie, Wenqiang Zu, Mingyang Zhao, Duo Su, Shilong Liu, Ruohua Shi, Guoqi Li, Shanghang Zhang, Lei Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2410.22217
• Pdf URL: https://arxiv.org/pdf/2410.22217
• Copy Paste: [[2410.22217]] Towards Unifying Understanding and Generation in the Era of Vision Foundation Models: A Survey from the Autoregression Perspective(https://arxiv.org/abs/2410.22217)
• Keywords: foundation model
• Abstract: Autoregression in large language models (LLMs) has shown impressive scalability by unifying all language tasks into the next token prediction paradigm. Recently, there is a growing interest in extending this success to vision foundation models. In this survey, we review the recent advances and discuss future directions for autoregressive vision foundation models. First, we present the trend for next generation of vision foundation models, i.e., unifying both understanding and generation in vision tasks. We then analyze the limitations of existing vision foundation models, and present a formal definition of autoregression with its advantages. Later, we categorize autoregressive vision foundation models from their vision tokenizers and autoregression backbones. Finally, we discuss several promising research challenges and directions. To the best of our knowledge, this is the first survey to comprehensively summarize autoregressive vision foundation models under the trend of unifying understanding and generation. A collection of related resources is available at this https URL.

Title: Hypergraph-based multi-scale spatio-temporal graph convolution network for Time-Series anomaly detection

• Authors: Hongyi Xu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.22256
• Pdf URL: https://arxiv.org/pdf/2410.22256
• Copy Paste: [[2410.22256]] Hypergraph-based multi-scale spatio-temporal graph convolution network for Time-Series anomaly detection(https://arxiv.org/abs/2410.22256)
• Keywords: anomaly
• Abstract: Multivariate time series anomaly detection technology plays an important role in many fields including aerospace, water treatment, cloud service providers, etc. Excellent anomaly detection models can greatly improve work efficiency and avoid major economic losses. However, with the development of technology, the increasing size and complexity of data, and the lack of labels for relevant abnormal data, it is becoming increasingly challenging to perform effective and accurate anomaly detection in high-dimensional and complex data sets. In this paper, we propose a hypergraph based spatiotemporal graph convolutional neural network model STGCN_Hyper, which explicitly captures high-order, multi-hop correlations between multiple variables through a hypergraph based dynamic graph structure learning module. On this basis, we further use the hypergraph based spatiotemporal graph convolutional network to utilize the learned hypergraph structure to effectively propagate and aggregate one-hop and multi-hop related node information in the convolutional network, thereby obtaining rich spatial information. Furthermore, through the multi-scale TCN dilated convolution module, the STGCN_hyper model can also capture the dependencies of features at different scales in the temporal dimension. An unsupervised anomaly detector based on PCA and GMM is also integrated into the STGCN_hyper model. Through the anomaly score of the detector, the model can detect the anomalies in an unsupervised way. Experimental results on multiple time series datasets show that our model can flexibly learn the multi-scale time series features in the data and the dependencies between features, and outperforms most existing baseline models in terms of precision, recall, F1-score on anomaly detection tasks. Our code is available on: this https URL

Title: Meta-Learning Adaptable Foundation Models

• Authors: Jacob L. Block, Sundararajan Srinivasan, Liam Collins, Aryan Mokhtari, Sanjay Shakkottai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2410.22264
• Pdf URL: https://arxiv.org/pdf/2410.22264
• Copy Paste: [[2410.22264]] Meta-Learning Adaptable Foundation Models(https://arxiv.org/abs/2410.22264)
• Keywords: foundation model
• Abstract: The power of foundation models (FMs) lies in their capacity to learn highly expressive representations that can be adapted to a broad spectrum of tasks. However, these pretrained models require multiple stages of fine-tuning to become effective for downstream applications. Conventionally, the model is first retrained on the aggregate of a diverse set of tasks of interest and then adapted to specific low-resource downstream tasks by utilizing a parameter-efficient fine-tuning (PEFT) scheme. While this two-phase procedure seems reasonable, the independence of the retraining and fine-tuning phases causes a major issue, as there is no guarantee the retrained model will achieve good performance post-fine-tuning. To explicitly address this issue, we introduce a meta-learning framework infused with PEFT in this intermediate retraining stage to learn a model that can be easily adapted to unseen tasks. For our theoretical results, we focus on linear models using low-rank adaptations. In this setting, we demonstrate the suboptimality of standard retraining for finding an adaptable set of parameters. Further, we prove that our method recovers the optimally adaptable parameters. We then apply these theoretical insights to retraining the RoBERTa model to predict the continuation of conversations between different personas within the ConvAI2 dataset. Empirically, we observe significant performance benefits using our proposed meta-learning scheme during retraining relative to the conventional approach.

Title: Fourier Head: Helping Large Language Models Learn Complex Probability Distributions

• Authors: Nate Gillman, Daksh Aggarwal, Michael Freeman, Saurabh Singh, Chen Sun
• Subjects: cs.LG, cs.AI, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2410.22269
• Pdf URL: https://arxiv.org/pdf/2410.22269
• Copy Paste: [[2410.22269]] Fourier Head: Helping Large Language Models Learn Complex Probability Distributions(https://arxiv.org/abs/2410.22269)
• Keywords: foundation model
• Abstract: As the quality of large language models has improved, there has been increased interest in using them to model non-linguistic tokens. For example, the Decision Transformer recasts agentic decision making as a sequence modeling problem, using a decoder-only LLM to model the distribution over the discrete action space for an Atari agent. However, when adapting LLMs to non-linguistic domains, it remains unclear if softmax over discrete bins captures the continuous structure of the tokens and the potentially complex distributions needed for high quality token generation. We introduce a neural network layer, constructed using Fourier series, which we can easily substitute for any linear layer if we want the outputs to have a more continuous structure. We perform extensive analysis on synthetic datasets, as well as on large-scale decision making and time series forecasting tasks. We also provide theoretical evidence that this layer can better learn signal from data while ignoring high-frequency noise. All of our results support the effectiveness of our proposed Fourier head in scenarios where the underlying data distribution has a natural continuous structure. For example, the Fourier head improves a Decision Transformer agent's returns by 46% on the Atari Seaquest game, and increases a state-of-the-art times series foundation model's forecasting performance by 3.5% across 20 benchmarks unseen during training.

Title: Embedding-based classifiers can detect prompt injection attacks

• Authors: Md. Ahsan Ayub, Subhabrata Majumdar
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2410.22284
• Pdf URL: https://arxiv.org/pdf/2410.22284
• Copy Paste: [[2410.22284]] Embedding-based classifiers can detect prompt injection attacks(https://arxiv.org/abs/2410.22284)
• Keywords: generative
• Abstract: Large Language Models (LLMs) are seeing significant adoption in every type of organization due to their exceptional generative capabilities. However, LLMs are found to be vulnerable to various adversarial attacks, particularly prompt injection attacks, which trick them into producing harmful or inappropriate content. Adversaries execute such attacks by crafting malicious prompts to deceive the LLMs. In this paper, we propose a novel approach based on embedding-based Machine Learning (ML) classifiers to protect LLM-based applications against this severe threat. We leverage three commonly used embedding models to generate embeddings of malicious and benign prompts and utilize ML classifiers to predict whether an input prompt is malicious. Out of several traditional ML methods, we achieve the best performance with classifiers built using Random Forest and XGBoost. Our classifiers outperform state-of-the-art prompt injection classifiers available in open-source implementations, which use encoder-only neural networks.