2024-11-07

Title: Undermining Image and Text Classification Algorithms Using Adversarial Attacks

• Authors: Langalibalele Lunga, Suhas Sreehari
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03348
• Pdf URL: https://arxiv.org/pdf/2411.03348
• Copy Paste: [[2411.03348]] Undermining Image and Text Classification Algorithms Using Adversarial Attacks(https://arxiv.org/abs/2411.03348)
• Keywords: generative
• Abstract: Machine learning models are prone to adversarial attacks, where inputs can be manipulated in order to cause misclassifications. While previous research has focused on techniques like Generative Adversarial Networks (GANs), there's limited exploration of GANs and Synthetic Minority Oversampling Technique (SMOTE) in text and image classification models to perform adversarial attacks. Our study addresses this gap by training various machine learning models and using GANs and SMOTE to generate additional data points aimed at attacking text classification models. Furthermore, we extend our investigation to face recognition models, training a Convolutional Neural Network(CNN) and subjecting it to adversarial attacks with fast gradient sign perturbations on key features identified by GradCAM, a technique used to highlight key image characteristics CNNs use in classification. Our experiments reveal a significant vulnerability in classification models. Specifically, we observe a 20 % decrease in accuracy for the top-performing text classification models post-attack, along with a 30 % decrease in facial recognition accuracy. This highlights the susceptibility of these models to manipulation of input data. Adversarial attacks not only compromise the security but also undermine the reliability of machine learning systems. By showcasing the impact of adversarial attacks on both text classification and face recognition models, our study underscores the urgent need for develop robust defenses against such vulnerabilities.

Title: Pedestrian Volume Prediction Using a Diffusion Convolutional Gated Recurrent Unit Model

• Authors: Yiwei Dong, Tingjin Chu, Lele Zhang, Hadi Ghaderi, Hanfang Yang
• Subjects: cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2411.03360
• Pdf URL: https://arxiv.org/pdf/2411.03360
• Copy Paste: [[2411.03360]] Pedestrian Volume Prediction Using a Diffusion Convolutional Gated Recurrent Unit Model(https://arxiv.org/abs/2411.03360)
• Keywords: diffusion
• Abstract: Effective models for analysing and predicting pedestrian flow are important to ensure the safety of both pedestrians and other road users. These tools also play a key role in optimising infrastructure design and geometry and supporting the economic utility of interconnected communities. The implementation of city-wide automatic pedestrian counting systems provides researchers with invaluable data, enabling the development and training of deep learning applications that offer better insights into traffic and crowd flows. Benefiting from real-world data provided by the City of Melbourne pedestrian counting system, this study presents a pedestrian flow prediction model, as an extension of Diffusion Convolutional Grated Recurrent Unit (DCGRU) with dynamic time warping, named DCGRU-DTW. This model captures the spatial dependencies of pedestrian flow through the diffusion process and the temporal dependency captured by Gated Recurrent Unit (GRU). Through extensive numerical experiments, we demonstrate that the proposed model outperforms the classic vector autoregressive model and the original DCGRU across multiple model accuracy metrics.

Title: DM4Steal: Diffusion Model For Link Stealing Attack On Graph Neural Networks

• Authors: Jinyin Chen, Haonan Ma, Haibin Zheng
• Subjects: cs.CR, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03364
• Pdf URL: https://arxiv.org/pdf/2411.03364
• Copy Paste: [[2411.03364]] DM4Steal: Diffusion Model For Link Stealing Attack On Graph Neural Networks(https://arxiv.org/abs/2411.03364)
• Keywords: diffusion
• Abstract: Graph has become increasingly integral to the advancement of recommendation systems, particularly with the fast development of graph neural network(GNN). By exploring the virtue of rich node features and link information, GNN is designed to provide personalized and accurate suggestions. Meanwhile, the privacy leakage of GNN in such contexts has also captured special attention. Prior work has revealed that a malicious user can utilize auxiliary knowledge to extract sensitive link data of the target graph, integral to recommendation systems, via the decision made by the target GNN model. This poses a significant risk to the integrity and confidentiality of data used in recommendation system. Though important, previous works on GNN's privacy leakage are still challenged in three aspects, i.e., limited stealing attack scenarios, sub-optimal attack performance, and adaptation against defense. To address these issues, we propose a diffusion model based link stealing attack, named DM4Steal. It differs previous work from three critical aspects. (i) Generality: aiming at six attack scenarios with limited auxiliary knowledge, we propose a novel training strategy for diffusion models so that DM4Steal is transferable to diverse attack scenarios. (ii) Effectiveness: benefiting from the retention of semantic structure in the diffusion model during the training process, DM4Steal is capable to learn the precise topology of the target graph through the GNN decision process. (iii) Adaptation: when GNN is defensive (e.g., DP, Dropout), DM4Steal relies on the stability that comes from sampling the score model multiple times to keep performance degradation to a minimum, thus DM4Steal implements successful adaptive attack on defensive GNN.

Title: Enhanced Real-Time Threat Detection in 5G Networks: A Self-Attention RNN Autoencoder Approach for Spectral Intrusion Analysis

• Authors: Mohammadreza Kouchaki, Minglong Zhang, Aly S. Abdalla, Guangchen Lan, Christopher G. Brinton, Vuk Marojevic
• Subjects: cs.CR, cs.AI, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.03365
• Pdf URL: https://arxiv.org/pdf/2411.03365
• Copy Paste: [[2411.03365]] Enhanced Real-Time Threat Detection in 5G Networks: A Self-Attention RNN Autoencoder Approach for Spectral Intrusion Analysis(https://arxiv.org/abs/2411.03365)
• Keywords: anomaly
• Abstract: In the rapidly evolving landscape of 5G technology, safeguarding Radio Frequency (RF) environments against sophisticated intrusions is paramount, especially in dynamic spectrum access and management. This paper presents an enhanced experimental model that integrates a self-attention mechanism with a Recurrent Neural Network (RNN)-based autoencoder for the detection of anomalous spectral activities in 5G networks at the waveform level. Our approach, grounded in time-series analysis, processes in-phase and quadrature (I/Q) samples to identify irregularities that could indicate potential jamming attacks. The model's architecture, augmented with a self-attention layer, extends the capabilities of RNN autoencoders, enabling a more nuanced understanding of temporal dependencies and contextual relationships within the RF spectrum. Utilizing a simulated 5G Radio Access Network (RAN) test-bed constructed with srsRAN 5G and Software Defined Radios (SDRs), we generated a comprehensive stream of data that reflects real-world RF spectrum conditions and attack scenarios. The model is trained to reconstruct standard signal behavior, establishing a normative baseline against which deviations, indicative of security threats, are identified. The proposed architecture is designed to balance between detection precision and computational efficiency, so the LSTM network, enriched with self-attention, continues to optimize for minimal execution latency and power consumption. Conducted on a real-world SDR-based testbed, our results demonstrate the model's improved performance and accuracy in threat detection. Keywords: self-attention, real-time intrusion detection, RNN autoencoder, Transformer architecture, LSTM, time series anomaly detection, 5G Security, spectrum access security.

Title: Personal Data Protection in AI-Native 6G Systems

• Authors: Keivan Navaie
• Subjects: cs.CR, cs.ET, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.03368
• Pdf URL: https://arxiv.org/pdf/2411.03368
• Copy Paste: [[2411.03368]] Personal Data Protection in AI-Native 6G Systems(https://arxiv.org/abs/2411.03368)
• Keywords: generative
• Abstract: As 6G evolves into an AI-native technology, the integration of artificial intelligence (AI) and Generative AI into cellular communication systems presents unparalleled opportunities for enhancing connectivity, network optimization, and personalized services. However, these advancements also introduce significant data protection challenges, as AI models increasingly depend on vast amounts of personal data for training and decision-making. In this context, ensuring compliance with stringent data protection regulations, such as the General Data Protection Regulation (GDPR), becomes critical for the design and operational integrity of 6G networks. These regulations shape key system architecture aspects, including transparency, accountability, fairness, bias mitigation, and data security. This paper identifies and examines the primary data protection risks associated with AI-driven 6G networks, focusing on the complex data flows and processing activities throughout the 6G lifecycle. By exploring these risks, we provide a comprehensive analysis of the potential privacy implications and propose effective mitigation strategies. Our findings stress the necessity of embedding privacy-by-design and privacy-by-default principles in the development of 6G standards to ensure both regulatory compliance and the protection of individual rights.

Title: Pathway-Guided Optimization of Deep Generative Molecular Design Models for Cancer Therapy

• Authors: Alif Bin Abdul Qayyum, Susan D. Mertins, Amanda K. Paulson, Nathan M. Urban, Byung-Jun Yoon
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2411.03460
• Pdf URL: https://arxiv.org/pdf/2411.03460
• Copy Paste: [[2411.03460]] Pathway-Guided Optimization of Deep Generative Molecular Design Models for Cancer Therapy(https://arxiv.org/abs/2411.03460)
• Keywords: generative
• Abstract: The data-driven drug design problem can be formulated as an optimization task of a potentially expensive black-box objective function over a huge high-dimensional and structured molecular space. The junction tree variational autoencoder (JTVAE) has been shown to be an efficient generative model that can be used for suggesting legitimate novel drug-like small molecules with improved properties. While the performance of the generative molecular design (GMD) scheme strongly depends on the initial training data, one can improve its sampling efficiency for suggesting better molecules with enhanced properties by optimizing the latent space. In this work, we propose how mechanistic models - such as pathway models described by differential equations - can be used for effective latent space optimization(LSO) of JTVAEs and other similar models for GMD. To demonstrate the potential of our proposed approach, we show how a pharmacodynamic model, assessing the therapeutic efficacy of a drug-like small molecule by predicting how it modulates a cancer pathway, can be incorporated for effective LSO of data-driven models for GMD.

Title: Self Supervised Networks for Learning Latent Space Representations of Human Body Scans and Motions

• Authors: Emmanuel Hartman, Nicolas Charon, Martin Bauer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03475
• Pdf URL: https://arxiv.org/pdf/2411.03475
• Copy Paste: [[2411.03475]] Self Supervised Networks for Learning Latent Space Representations of Human Body Scans and Motions(https://arxiv.org/abs/2411.03475)
• Keywords: self-supervised
• Abstract: This paper introduces self-supervised neural network models to tackle several fundamental problems in the field of 3D human body analysis and processing. First, we propose VariShaPE (Varifold Shape Parameter Estimator), a novel architecture for the retrieval of latent space representations of body shapes and poses. This network offers a fast and robust method to estimate the embedding of arbitrary unregistered meshes into the latent space. Second, we complement the estimation of latent codes with MoGeN (Motion Geometry Network) a framework that learns the geometry on the latent space itself. This is achieved by lifting the body pose parameter space into a higher dimensional Euclidean space in which body motion mini-sequences from a training set of 4D data can be approximated by simple linear interpolation. Using the SMPL latent space representation we illustrate how the combination of these network models, once trained, can be used to perform a variety of tasks with very limited computational cost. This includes operations such as motion interpolation, extrapolation and transfer as well as random shape and pose generation.

Title: Rainfall regression from C-band Synthetic Aperture Radar using Multi-Task Generative Adversarial Networks

• Authors: Aurélien Colin, Romain Husson
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03480
• Pdf URL: https://arxiv.org/pdf/2411.03480
• Copy Paste: [[2411.03480]] Rainfall regression from C-band Synthetic Aperture Radar using Multi-Task Generative Adversarial Networks(https://arxiv.org/abs/2411.03480)
• Keywords: generative
• Abstract: This paper introduces a data-driven approach to estimate precipitation rates from Synthetic Aperture Radar (SAR) at a spatial resolution of 200 meters per pixel. It addresses previous challenges related to the collocation of SAR and weather radar data, specifically the misalignment in collocations and the scarcity of rainfall examples under strong wind. To tackle these challenges, the paper proposes a multi-objective formulation, introducing patch-level components and an adversarial component. It exploits the full NEXRAD archive to look for potential co-locations with Sentinel-1 data. With additional enhancements to the training procedure and the incorporation of additional inputs, the resulting model demonstrates improved accuracy in rainfall estimates and the ability to extend its performance to scenarios up to 15 m/s.

Title: SynthSet: Generative Diffusion Model for Semantic Segmentation in Precision Agriculture

• Authors: Andrew Heschl, Mauricio Murillo, Keyhan Najafian, Farhad Maleki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03505
• Pdf URL: https://arxiv.org/pdf/2411.03505
• Copy Paste: [[2411.03505]] SynthSet: Generative Diffusion Model for Semantic Segmentation in Precision Agriculture(https://arxiv.org/abs/2411.03505)
• Keywords: diffusion, generative
• Abstract: This paper introduces a methodology for generating synthetic annotated data to address data scarcity in semantic segmentation tasks within the precision agriculture domain. Utilizing Denoising Diffusion Probabilistic Models (DDPMs) and Generative Adversarial Networks (GANs), we propose a dual diffusion model architecture for synthesizing realistic annotated agricultural data, without any human intervention. We employ super-resolution to enhance the phenotypic characteristics of the synthesized images and their coherence with the corresponding generated masks. We showcase the utility of the proposed method for wheat head segmentation. The high quality of synthesized data underscores the effectiveness of the proposed methodology in generating image-mask pairs. Furthermore, models trained on our generated data exhibit promising performance when tested on an external, diverse dataset of real wheat fields. The results show the efficacy of the proposed methodology for addressing data scarcity for semantic segmentation tasks. Moreover, the proposed approach can be readily adapted for various segmentation tasks in precision agriculture and beyond.

Title: Exploring the Benefits of Domain-Pretraining of Generative Large Language Models for Chemistry

• Authors: Anurag Acharya, Shivam Sharma, Robin Cosbey, Megha Subramanian, Scott Howland, Maria Glenski
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03542
• Pdf URL: https://arxiv.org/pdf/2411.03542
• Copy Paste: [[2411.03542]] Exploring the Benefits of Domain-Pretraining of Generative Large Language Models for Chemistry(https://arxiv.org/abs/2411.03542)
• Keywords: foundation model, generative
• Abstract: A proliferation of Large Language Models (the GPT series, BLOOM, LLaMA, and more) are driving forward novel development of multipurpose AI for a variety of tasks, particularly natural language processing (NLP) tasks. These models demonstrate strong performance on a range of tasks; however, there has been evidence of brittleness when applied to more niche or narrow domains where hallucinations or fluent but incorrect responses reduce performance. Given the complex nature of scientific domains, it is prudent to investigate the trade-offs of leveraging off-the-shelf versus more targeted foundation models for scientific domains. In this work, we examine the benefits of in-domain pre-training for a given scientific domain, chemistry, and compare these to open-source, off-the-shelf models with zero-shot and few-shot prompting. Our results show that not only do in-domain base models perform reasonably well on in-domain tasks in a zero-shot setting but that further adaptation using instruction fine-tuning yields impressive performance on chemistry-specific tasks such as named entity recognition and molecular formula generation.

Title: Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data

• Authors: Seunggeun Chi, Pin-Hao Huang, Enna Sachdeva, Hengbo Ma, Karthik Ramani, Kwonjoon Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03561
• Pdf URL: https://arxiv.org/pdf/2411.03561
• Copy Paste: [[2411.03561]] Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data(https://arxiv.org/abs/2411.03561)
• Keywords: diffusion
• Abstract: We study the problem of estimating the body movements of a camera wearer from egocentric videos. Current methods for ego-body pose estimation rely on temporally dense sensor data, such as IMU measurements from spatially sparse body parts like the head and hands. However, we propose that even temporally sparse observations, such as hand poses captured intermittently from egocentric videos during natural or periodic hand movements, can effectively constrain overall body motion. Naively applying diffusion models to generate full-body pose from head pose and sparse hand pose leads to suboptimal results. To overcome this, we develop a two-stage approach that decomposes the problem into temporal completion and spatial completion. First, our method employs masked autoencoders to impute hand trajectories by leveraging the spatiotemporal correlations between the head pose sequence and intermittent hand poses, providing uncertainty estimates. Subsequently, we employ conditional diffusion models to generate plausible full-body motions based on these temporally dense trajectories of the head and hands, guided by the uncertainty estimates from the imputation. The effectiveness of our method was rigorously tested and validated through comprehensive experiments conducted on various HMD setup with AMASS and Ego-Exo4D datasets.

Title: From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond

• Authors: Harsha Nori, Naoto Usuyama, Nicholas King, Scott Mayer McKinney, Xavier Fernandes, Sheng Zhang, Eric Horvitz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03590
• Pdf URL: https://arxiv.org/pdf/2411.03590
• Copy Paste: [[2411.03590]] From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond(https://arxiv.org/abs/2411.03590)
• Keywords: in-context
• Abstract: Run-time steering strategies like Medprompt are valuable for guiding large language models (LLMs) to top performance on challenging tasks. Medprompt demonstrates that a general LLM can be focused to deliver state-of-the-art performance on specialized domains like medicine by using a prompt to elicit a run-time strategy involving chain of thought reasoning and ensembling. OpenAI's o1-preview model represents a new paradigm, where a model is designed to do run-time reasoning before generating final responses. We seek to understand the behavior of o1-preview on a diverse set of medical challenge problem benchmarks. Following on the Medprompt study with GPT-4, we systematically evaluate the o1-preview model across various medical benchmarks. Notably, even without prompting techniques, o1-preview largely outperforms the GPT-4 series with Medprompt. We further systematically study the efficacy of classic prompt engineering strategies, as represented by Medprompt, within the new paradigm of reasoning models. We found that few-shot prompting hinders o1's performance, suggesting that in-context learning may no longer be an effective steering approach for reasoning-native models. While ensembling remains viable, it is resource-intensive and requires careful cost-performance optimization. Our cost and accuracy analysis across run-time strategies reveals a Pareto frontier, with GPT-4o representing a more affordable option and o1-preview achieving state-of-the-art performance at higher cost. Although o1-preview offers top performance, GPT-4o with steering strategies like Medprompt retains value in specific contexts. Moreover, we note that the o1-preview model has reached near-saturation on many existing medical benchmarks, underscoring the need for new, challenging benchmarks. We close with reflections on general directions for inference-time computation with LLMs.

Title: These Maps Are Made by Propagation: Adapting Deep Stereo Networks to Road Scenarios with Decisive Disparity Diffusion

• Authors: Chuang-Wei Liu, Yikang Zhang, Qijun Chen, Ioannis Pitas, Rui Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03717
• Pdf URL: https://arxiv.org/pdf/2411.03717
• Copy Paste: [[2411.03717]] These Maps Are Made by Propagation: Adapting Deep Stereo Networks to Road Scenarios with Decisive Disparity Diffusion(https://arxiv.org/abs/2411.03717)
• Keywords: diffusion
• Abstract: Stereo matching has emerged as a cost-effective solution for road surface 3D reconstruction, garnering significant attention towards improving both computational efficiency and accuracy. This article introduces decisive disparity diffusion (D3Stereo), marking the first exploration of dense deep feature matching that adapts pre-trained deep convolutional neural networks (DCNNs) to previously unseen road scenarios. A pyramid of cost volumes is initially created using various levels of learned representations. Subsequently, a novel recursive bilateral filtering algorithm is employed to aggregate these costs. A key innovation of D3Stereo lies in its alternating decisive disparity diffusion strategy, wherein intra-scale diffusion is employed to complete sparse disparity images, while inter-scale inheritance provides valuable prior information for higher resolutions. Extensive experiments conducted on our created UDTIRI-Stereo and Stereo-Road datasets underscore the effectiveness of D3Stereo strategy in adapting pre-trained DCNNs and its superior performance compared to all other explicit programming-based algorithms designed specifically for road surface 3D reconstruction. Additional experiments conducted on the Middlebury dataset with backbone DCNNs pre-trained on the ImageNet database further validate the versatility of D3Stereo strategy in tackling general stereo matching problems.

Title: NeurIPS 2023 Competition: Privacy Preserving Federated Learning Document VQA

• Authors: Marlon Tobaben, Mohamed Ali Souibgui, Rubèn Tito, Khanh Nguyen, Raouf Kerkouche, Kangsoo Jung, Joonas Jälkö, Lei Kang, Andrey Barsky, Vincent Poulain d'Andecy, Aurélie Joseph, Aashiq Muhamed, Kevin Kuo, Virginia Smith, Yusuke Yamasaki, Takumi Fukami, Kenta Niwa, Iifan Tyou, Hiro Ishii, Rio Yokota, Ragul N, Rintu Kutum, Josep Llados, Ernest Valveny, Antti Honkela, Mario Fritz, Dimosthenis Karatzas
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03730
• Pdf URL: https://arxiv.org/pdf/2411.03730
• Copy Paste: [[2411.03730]] NeurIPS 2023 Competition: Privacy Preserving Federated Learning Document VQA(https://arxiv.org/abs/2411.03730)
• Keywords: generative
• Abstract: The Privacy Preserving Federated Learning Document VQA (PFL-DocVQA) competition challenged the community to develop provably private and communication-efficient solutions in a federated setting for a real-life use case: invoice processing. The competition introduced a dataset of real invoice documents, along with associated questions and answers requiring information extraction and reasoning over the document images. Thereby, it brings together researchers and expertise from the document analysis, privacy, and federated learning communities. Participants fine-tuned a pre-trained, state-of-the-art Document Visual Question Answering model provided by the organizers for this new domain, mimicking a typical federated invoice processing setup. The base model is a multi-modal generative language model, and sensitive information could be exposed through either the visual or textual input modality. Participants proposed elegant solutions to reduce communication costs while maintaining a minimum utility threshold in track 1 and to protect all information from each document provider using differential privacy in track 2. The competition served as a new testbed for developing and testing private federated learning methods, simultaneously raising awareness about privacy within the document image analysis and recognition community. Ultimately, the competition analysis provides best practices and recommendations for successfully running privacy-focused federated learning challenges in the future.

Title: SA3DIP: Segment Any 3D Instance with Potential 3D Priors

• Authors: Xi Yang, Xu Gu, Xingyilang Yin, Xinbo Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03819
• Pdf URL: https://arxiv.org/pdf/2411.03819
• Copy Paste: [[2411.03819]] SA3DIP: Segment Any 3D Instance with Potential 3D Priors(https://arxiv.org/abs/2411.03819)
• Keywords: foundation model
• Abstract: The proliferation of 2D foundation models has sparked research into adapting them for open-world 3D instance segmentation. Recent methods introduce a paradigm that leverages superpoints as geometric primitives and incorporates 2D multi-view masks from Segment Anything model (SAM) as merging guidance, achieving outstanding zero-shot instance segmentation results. However, the limited use of 3D priors restricts the segmentation performance. Previous methods calculate the 3D superpoints solely based on estimated normal from spatial coordinates, resulting in under-segmentation for instances with similar geometry. Besides, the heavy reliance on SAM and hand-crafted algorithms in 2D space suffers from over-segmentation due to SAM's inherent part-level segmentation tendency. To address these issues, we propose SA3DIP, a novel method for Segmenting Any 3D Instances via exploiting potential 3D Priors. Specifically, on one hand, we generate complementary 3D primitives based on both geometric and textural priors, which reduces the initial errors that accumulate in subsequent procedures. On the other hand, we introduce supplemental constraints from the 3D space by using a 3D detector to guide a further merging process. Furthermore, we notice a considerable portion of low-quality ground truth annotations in ScanNetV2 benchmark, which affect the fair evaluations. Thus, we present ScanNetV2-INS with complete ground truth labels and supplement additional instances for 3D class-agnostic instance segmentation. Experimental evaluations on various 2D-3D datasets demonstrate the effectiveness and robustness of our approach. Our code and proposed ScanNetV2-INS dataset are available HERE.

Title: Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts

• Authors: Zhitong Gao, Bingnan Li, Mathieu Salzmann, Xuming He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03829
• Pdf URL: https://arxiv.org/pdf/2411.03829
• Copy Paste: [[2411.03829]] Generalize or Detect? Towards Robust Semantic Segmentation Under Multiple Distribution Shifts(https://arxiv.org/abs/2411.03829)
• Keywords: generative, anomaly
• Abstract: In open-world scenarios, where both novel classes and domains may exist, an ideal segmentation model should detect anomaly classes for safety and generalize to new domains. However, existing methods often struggle to distinguish between domain-level and semantic-level distribution shifts, leading to poor out-of-distribution (OOD) detection or domain generalization performance. In this work, we aim to equip the model to generalize effectively to covariate-shift regions while precisely identifying semantic-shift regions. To achieve this, we design a novel generative augmentation method to produce coherent images that incorporate both anomaly (or novel) objects and various covariate shifts at both image and object levels. Furthermore, we introduce a training strategy that recalibrates uncertainty specifically for semantic shifts and enhances the feature extractor to align features associated with domain shifts. We validate the effectiveness of our method across benchmarks featuring both semantic and domain shifts. Our method achieves state-of-the-art performance across all benchmarks for both OOD detection and domain generalization. Code is available at this https URL.

Title: ROBIN: Robust and Invisible Watermarks for Diffusion Models with Adversarial Optimization

• Authors: Huayang Huang, Yu Wu, Qian Wang
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.03862
• Pdf URL: https://arxiv.org/pdf/2411.03862
• Copy Paste: [[2411.03862]] ROBIN: Robust and Invisible Watermarks for Diffusion Models with Adversarial Optimization(https://arxiv.org/abs/2411.03862)
• Keywords: diffusion, generative
• Abstract: Watermarking generative content serves as a vital tool for authentication, ownership protection, and mitigation of potential misuse. Existing watermarking methods face the challenge of balancing robustness and concealment. They empirically inject a watermark that is both invisible and robust and passively achieve concealment by limiting the strength of the watermark, thus reducing the robustness. In this paper, we propose to explicitly introduce a watermark hiding process to actively achieve concealment, thus allowing the embedding of stronger watermarks. To be specific, we implant a robust watermark in an intermediate diffusion state and then guide the model to hide the watermark in the final generated image. We employ an adversarial optimization algorithm to produce the optimal hiding prompt guiding signal for each watermark. The prompt embedding is optimized to minimize artifacts in the generated image, while the watermark is optimized to achieve maximum strength. The watermark can be verified by reversing the generation process. Experiments on various diffusion models demonstrate the watermark remains verifiable even under significant image tampering and shows superior invisibility compared to other state-of-the-art robust watermarking methods.

Title: EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoning

• Authors: Kiran Purohit, Venktesh V, Raghuram Devalla, Krishna Mohan Yerragorla, Sourangshu Bhattacharya, Avishek Anand
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03877
• Pdf URL: https://arxiv.org/pdf/2411.03877
• Copy Paste: [[2411.03877]] EXPLORA: Efficient Exemplar Subset Selection for Complex Reasoning(https://arxiv.org/abs/2411.03877)
• Keywords: in-context
• Abstract: Answering reasoning-based complex questions over text and hybrid sources, including tables, is a challenging task. Recent advances in large language models (LLMs) have enabled in-context learning (ICL), allowing LLMs to acquire proficiency in a specific task using only a few demonstration samples (exemplars). A critical challenge in ICL is the selection of optimal exemplars, which can be either task-specific (static) or test-example-specific (dynamic). Static exemplars provide faster inference times and increased robustness across a distribution of test examples. In this paper, we propose an algorithm for static exemplar subset selection for complex reasoning tasks. We introduce EXPLORA, a novel exploration method designed to estimate the parameters of the scoring function, which evaluates exemplar subsets without incorporating confidence information. EXPLORA significantly reduces the number of LLM calls to ~11% of those required by state-of-the-art methods and achieves a substantial performance improvement of 12.24%. We open-source our code and data (this https URL).

Title: RAGulator: Lightweight Out-of-Context Detectors for Grounded Text Generation

• Authors: Ian Poey, Jiajun Liu, Qishuai Zhong, Adrien Chenailler
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03920
• Pdf URL: https://arxiv.org/pdf/2411.03920
• Copy Paste: [[2411.03920]] RAGulator: Lightweight Out-of-Context Detectors for Grounded Text Generation(https://arxiv.org/abs/2411.03920)
• Keywords: generative
• Abstract: Real-time detection of out-of-context LLM outputs is crucial for enterprises looking to safely adopt RAG applications. In this work, we train lightweight models to discriminate LLM-generated text that is semantically out-of-context from retrieved text documents. We preprocess a combination of summarisation and semantic textual similarity datasets to construct training data using minimal resources. We find that DeBERTa is not only the best-performing model under this pipeline, but it is also fast and does not require additional text preprocessing or feature engineering. While emerging work demonstrates that generative LLMs can also be fine-tuned and used in complex data pipelines to achieve state-of-the-art performance, we note that speed and resource limits are important considerations for on-premise deployment.

Title: Self-supervised Representation Learning for Cell Event Recognition through Time Arrow Prediction

• Authors: Cangxiong Chen, Vinay P. Namboodiri, Julia E. Sero
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03924
• Pdf URL: https://arxiv.org/pdf/2411.03924
• Copy Paste: [[2411.03924]] Self-supervised Representation Learning for Cell Event Recognition through Time Arrow Prediction(https://arxiv.org/abs/2411.03924)
• Keywords: self-supervised
• Abstract: The spatio-temporal nature of live-cell microscopy data poses challenges in the analysis of cell states which is fundamental in bioimaging. Deep-learning based segmentation or tracking methods rely on large amount of high quality annotations to work effectively. In this work, we explore an alternative solution: using feature maps obtained from self-supervised representation learning (SSRL) on time arrow prediction (TAP) for the downstream supervised task of cell event recognition. We demonstrate through extensive experiments and analysis that this approach can achieve better performance with limited annotation compared to models trained from end to end using fully supervised approach. Our analysis also provides insight into applications of the SSRL using TAP in live-cell microscopy.

Title: GUIDE-VAE: Advancing Data Generation with User Information and Pattern Dictionaries

• Authors: Kutay Bölat, Simon Tindemans
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.03936
• Pdf URL: https://arxiv.org/pdf/2411.03936
• Copy Paste: [[2411.03936]] GUIDE-VAE: Advancing Data Generation with User Information and Pattern Dictionaries(https://arxiv.org/abs/2411.03936)
• Keywords: generative
• Abstract: Generative modelling of multi-user datasets has become prominent in science and engineering. Generating a data point for a given user requires employing user information, and conventional generative models, including variational autoencoders (VAEs), often ignore that. This paper introduces GUIDE-VAE, a novel conditional generative model that leverages user embeddings to generate user-guided data. By allowing the model to benefit from shared patterns across users, GUIDE-VAE enhances performance in multi-user settings, even under significant data imbalance. In addition to integrating user information, GUIDE-VAE incorporates a pattern dictionary-based covariance composition (PDCC) to improve the realism of generated samples by capturing complex feature dependencies. While user embeddings drive performance gains, PDCC addresses common issues such as noise and over-smoothing typically seen in VAEs. The proposed GUIDE-VAE was evaluated on a multi-user smart meter dataset characterized by substantial data imbalance across users. Quantitative results show that GUIDE-VAE performs effectively in both synthetic data generation and missing record imputation tasks, while qualitative evaluations reveal that GUIDE-VAE produces more plausible and less noisy data. These results establish GUIDE-VAE as a promising tool for controlled, realistic data generation in multi-user datasets, with potential applications across various domains requiring user-informed modelling.

Title: Can Custom Models Learn In-Context? An Exploration of Hybrid Architecture Performance on In-Context Learning Tasks

• Authors: Ryan Campbell, Nelson Lojo, Kesava Viswanadha, Christoffer Grondal Tryggestad, Derrick Han Sun, Sriteja Vijapurapu, August Rolfsen, Anant Sahai
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03945
• Pdf URL: https://arxiv.org/pdf/2411.03945
• Copy Paste: [[2411.03945]] Can Custom Models Learn In-Context? An Exploration of Hybrid Architecture Performance on In-Context Learning Tasks(https://arxiv.org/abs/2411.03945)
• Keywords: in-context
• Abstract: In-Context Learning (ICL) is a phenomenon where task learning occurs through a prompt sequence without the necessity of parameter updates. ICL in Multi-Headed Attention (MHA) with absolute positional embedding has been the focus of more study than other sequence model varieties. We examine implications of architectural differences between GPT-2 and LLaMa as well as LlaMa and Mamba. We extend work done by Garg et al. (2022) and Park et al. (2024) to GPT-2/LLaMa hybrid and LLaMa/Mamba hybrid models - examining the interplay between sequence transformation blocks and regressive performance in-context. We note that certain architectural changes cause degraded training efficiency/ICL accuracy by converging to suboptimal predictors or converging slower. We also find certain hybrids showing optimistic performance improvements, informing potential future ICL-focused architecture modifications. Additionally, we propose the "ICL regression score", a scalar metric describing a model's whole performance on a specific task. Compute limitations impose restrictions on our architecture-space, training duration, number of training runs, function class complexity, and benchmark complexity. To foster reproducible and extensible research, we provide a typed, modular, and extensible Python package on which we run all experiments.

Title: Face Reconstruction from Face Embeddings using Adapter to a Face Foundation Model

• Authors: Hatef Otroshi Shahreza, Anjith George, Sébastien Marcel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03960
• Pdf URL: https://arxiv.org/pdf/2411.03960
• Copy Paste: [[2411.03960]] Face Reconstruction from Face Embeddings using Adapter to a Face Foundation Model(https://arxiv.org/abs/2411.03960)
• Keywords: foundation model
• Abstract: Face recognition systems extract embedding vectors from face images and use these embeddings to verify or identify individuals. Face reconstruction attack (also known as template inversion) refers to reconstructing face images from face embeddings and using the reconstructed face image to enter a face recognition system. In this paper, we propose to use a face foundation model to reconstruct face images from the embeddings of a blackbox face recognition model. The foundation model is trained with 42M images to generate face images from the facial embeddings of a fixed face recognition model. We propose to use an adapter to translate target embeddings into the embedding space of the foundation model. The generated images are evaluated on different face recognition models and different datasets, demonstrating the effectiveness of our method to translate embeddings of different face recognition models. We also evaluate the transferability of reconstructed face images when attacking different face recognition models. Our experimental results show that our reconstructed face images outperform previous reconstruction attacks against face recognition models.

Title: ReEdit: Multimodal Exemplar-Based Image Editing with Diffusion Models

• Authors: Ashutosh Srivastava, Tarun Ram Menta, Abhinav Java, Avadhoot Jadhav, Silky Singh, Surgan Jandial, Balaji Krishnamurthy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03982
• Pdf URL: https://arxiv.org/pdf/2411.03982
• Copy Paste: [[2411.03982]] ReEdit: Multimodal Exemplar-Based Image Editing with Diffusion Models(https://arxiv.org/abs/2411.03982)
• Keywords: diffusion
• Abstract: Modern Text-to-Image (T2I) Diffusion models have revolutionized image editing by enabling the generation of high-quality photorealistic images. While the de facto method for performing edits with T2I models is through text instructions, this approach non-trivial due to the complex many-to-many mapping between natural language and images. In this work, we address exemplar-based image editing -- the task of transferring an edit from an exemplar pair to a content image(s). We propose ReEdit, a modular and efficient end-to-end framework that captures edits in both text and image modalities while ensuring the fidelity of the edited image. We validate the effectiveness of ReEdit through extensive comparisons with state-of-the-art baselines and sensitivity analyses of key design choices. Our results demonstrate that ReEdit consistently outperforms contemporary approaches both qualitatively and quantitatively. Additionally, ReEdit boasts high practical applicability, as it does not require any task-specific optimization and is four times faster than the next best baseline.

Title: Towards Resource-Efficient Federated Learning in Industrial IoT for Multivariate Time Series Analysis

• Authors: Alexandros Gkillas, Aris Lalos
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2411.03996
• Pdf URL: https://arxiv.org/pdf/2411.03996
• Copy Paste: [[2411.03996]] Towards Resource-Efficient Federated Learning in Industrial IoT for Multivariate Time Series Analysis(https://arxiv.org/abs/2411.03996)
• Keywords: anomaly
• Abstract: Anomaly and missing data constitute a thorny problem in industrial applications. In recent years, deep learning enabled anomaly detection has emerged as a critical direction, however the improved detection accuracy is achieved with the utilization of large neural networks, increasing their storage and computational cost. Moreover, the data collected in edge devices contain user privacy, introducing challenges that can be successfully addressed by the privacy-preserving distributed paradigm, known as federated learning (FL). This framework allows edge devices to train and exchange models increasing also the communication cost. Thus, to deal with the increased communication, processing and storage challenges of the FL based deep anomaly detection NN pruning is expected to have significant benefits towards reducing the processing, storage and communication complexity. With this focus, a novel compression-based optimization problem is proposed at the server-side of a FL paradigm that fusses the received local models broadcast and performs pruning generating a more compressed model. Experiments in the context of anomaly detection and missing value imputation demonstrate that the proposed FL scenario along with the proposed compressed-based method are able to achieve high compression rates (more than $99.7\%$) with negligible performance losses (less than $1.18\%$ ) as compared to the centralized solutions.

Title: M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models

• Authors: Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, Arman Cohan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04075
• Pdf URL: https://arxiv.org/pdf/2411.04075
• Copy Paste: [[2411.04075]] M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models(https://arxiv.org/abs/2411.04075)
• Keywords: foundation model
• Abstract: Existing benchmarks for evaluating foundation models mainly focus on single-document, text-only tasks. However, they often fail to fully capture the complexity of research workflows, which typically involve interpreting non-textual data and gathering information across multiple documents. To address this gap, we introduce M3SciQA, a multi-modal, multi-document scientific question answering benchmark designed for a more comprehensive evaluation of foundation models. M3SciQA consists of 1,452 expert-annotated questions spanning 70 natural language processing paper clusters, where each cluster represents a primary paper along with all its cited documents, mirroring the workflow of comprehending a single paper by requiring multi-modal and multi-document data. With M3SciQA, we conduct a comprehensive evaluation of 18 foundation models. Our results indicate that current foundation models still significantly underperform compared to human experts in multi-modal information retrieval and in reasoning across multiple scientific documents. Additionally, we explore the implications of these findings for the future advancement of applying foundation models in multi-modal scientific literature analysis.

Title: Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?

• Authors: Daniel P. Jeong, Saurabh Garg, Zachary C. Lipton, Michael Oberst
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04118
• Pdf URL: https://arxiv.org/pdf/2411.04118
• Copy Paste: [[2411.04118]] Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?(https://arxiv.org/abs/2411.04118)
• Keywords: foundation model
• Abstract: Several recent works seek to develop foundation models specifically for medical applications, adapting general-purpose large language models (LLMs) and vision-language models (VLMs) via continued pretraining on publicly available biomedical corpora. These works typically claim that such domain-adaptive pretraining (DAPT) improves performance on downstream medical tasks, such as answering medical licensing exam questions. In this paper, we compare seven public "medical" LLMs and two VLMs against their corresponding base models, arriving at a different conclusion: all medical VLMs and nearly all medical LLMs fail to consistently improve over their base models in the zero-/few-shot prompting regime for medical question-answering (QA) tasks. For instance, across the tasks and model pairs we consider in the 3-shot setting, medical LLMs only outperform their base models in 12.1% of cases, reach a (statistical) tie in 49.8% of cases, and are significantly worse than their base models in the remaining 38.2% of cases. Our conclusions are based on (i) comparing each medical model head-to-head, directly against the corresponding base model; (ii) optimizing the prompts for each model separately; and (iii) accounting for statistical uncertainty in comparisons. While these basic practices are not consistently adopted in the literature, our ablations show that they substantially impact conclusions. Our findings suggest that state-of-the-art general-domain models may already exhibit strong medical knowledge and reasoning capabilities, and offer recommendations to strengthen the conclusions of future studies.

Title: Community Forensics: Using Thousands of Generators to Train Fake Image Detectors

• Authors: Jeongsoo Park, Andrew Owens
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04125
• Pdf URL: https://arxiv.org/pdf/2411.04125
• Copy Paste: [[2411.04125]] Community Forensics: Using Thousands of Generators to Train Fake Image Detectors(https://arxiv.org/abs/2411.04125)
• Keywords: diffusion, generative
• Abstract: One of the key challenges of detecting AI-generated images is spotting images that have been created by previously unseen generative models. We argue that the limited diversity of the training data is a major obstacle to addressing this problem, and we propose a new dataset that is significantly larger and more diverse than prior work. As part of creating this dataset, we systematically download thousands of text-to-image latent diffusion models and sample images from them. We also collect images from dozens of popular open source and commercial models. The resulting dataset contains 2.7M images that have been sampled from 4803 different models. These images collectively capture a wide range of scene content, generator architectures, and image processing settings. Using this dataset, we study the generalization abilities of fake image detectors. Our experiments suggest that detection performance improves as the number of models in the training set increases, even when these models have similar architectures. We also find that detection performance improves as the diversity of the models increases, and that our trained detectors generalize better than those trained on other datasets.