2024-11-08

Title: DiMSUM: Diffusion Mamba -- A Scalable and Unified Spatial-Frequency Method for Image Generation

• Authors: Hao Phung, Quan Dao, Trung Dao, Hoang Phan, Dimitris Metaxas, Anh Tran
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04168
• Pdf URL: https://arxiv.org/pdf/2411.04168
• Copy Paste: [[2411.04168]] DiMSUM: Diffusion Mamba -- A Scalable and Unified Spatial-Frequency Method for Image Generation(https://arxiv.org/abs/2411.04168)
• Keywords: diffusion, transformer
• Abstract: We introduce a novel state-space architecture for diffusion models, effectively harnessing spatial and frequency information to enhance the inductive bias towards local features in input images for image generation tasks. While state-space networks, including Mamba, a revolutionary advancement in recurrent neural networks, typically scan input sequences from left to right, they face difficulties in designing effective scanning strategies, especially in the processing of image data. Our method demonstrates that integrating wavelet transformation into Mamba enhances the local structure awareness of visual inputs and better captures long-range relations of frequencies by disentangling them into wavelet subbands, representing both low- and high-frequency components. These wavelet-based outputs are then processed and seamlessly fused with the original Mamba outputs through a cross-attention fusion layer, combining both spatial and frequency information to optimize the order awareness of state-space models which is essential for the details and overall quality of image generation. Besides, we introduce a globally-shared transformer to supercharge the performance of Mamba, harnessing its exceptional power to capture global relationships. Through extensive experiments on standard benchmarks, our method demonstrates superior results compared to DiT and DIFFUSSM, achieving faster training convergence and delivering high-quality outputs. The codes and pretrained models are released at this https URL.

Title: Scalable DP-SGD: Shuffling vs. Poisson Subsampling

• Authors: Lynn Chua, Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang
• Subjects: cs.LG, cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2411.04205
• Pdf URL: https://arxiv.org/pdf/2411.04205
• Copy Paste: [[2411.04205]] Scalable DP-SGD: Shuffling vs. Poisson Subsampling(https://arxiv.org/abs/2411.04205)
• Keywords: privacy
• Abstract: We provide new lower bounds on the privacy guarantee of the multi-epoch Adaptive Batch Linear Queries (ABLQ) mechanism with shuffled batch sampling, demonstrating substantial gaps when compared to Poisson subsampling; prior analysis was limited to a single epoch. Since the privacy analysis of Differentially Private Stochastic Gradient Descent (DP-SGD) is obtained by analyzing the ABLQ mechanism, this brings into serious question the common practice of implementing shuffling-based DP-SGD, but reporting privacy parameters as if Poisson subsampling was used. To understand the impact of this gap on the utility of trained machine learning models, we introduce a practical approach to implement Poisson subsampling at scale using massively parallel computation, and efficiently train models with the same. We compare the utility of models trained with Poisson-subsampling-based DP-SGD, and the optimistic estimates of utility when using shuffling, via our new lower bounds on the privacy guarantee of ABLQ with shuffling.

Title: Quantum Diffusion Models for Few-Shot Learning

• Authors: Ruhan Wang, Ye Wang, Jing Liu, Toshiaki Koike-Akino
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04217
• Pdf URL: https://arxiv.org/pdf/2411.04217
• Copy Paste: [[2411.04217]] Quantum Diffusion Models for Few-Shot Learning(https://arxiv.org/abs/2411.04217)
• Keywords: diffusion
• Abstract: Modern quantum machine learning (QML) methods involve the variational optimization of parameterized quantum circuits on training datasets, followed by predictions on testing datasets. Most state-of-the-art QML algorithms currently lack practical advantages due to their limited learning capabilities, especially in few-shot learning tasks. In this work, we propose three new frameworks employing quantum diffusion model (QDM) as a solution for the few-shot learning: label-guided generation inference (LGGI); label-guided denoising inference (LGDI); and label-guided noise addition inference (LGNAI). Experimental results demonstrate that our proposed algorithms significantly outperform existing methods.

Title: Equivariant Graph Network Approximations of High-Degree Polynomials for Force Field Prediction

• Authors: Zhao Xu, Haiyang Yu, Montgomery Bohde, Shuiwang Ji
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04219
• Pdf URL: https://arxiv.org/pdf/2411.04219
• Copy Paste: [[2411.04219]] Equivariant Graph Network Approximations of High-Degree Polynomials for Force Field Prediction(https://arxiv.org/abs/2411.04219)
• Keywords: robust
• Abstract: Recent advancements in equivariant deep models have shown promise in accurately predicting atomic potentials and force fields in molecular dynamics simulations. Using spherical harmonics (SH) and tensor products (TP), these equivariant networks gain enhanced physical understanding, like symmetries and many-body interactions. Beyond encoding physical insights, SH and TP are also crucial to represent equivariant polynomial functions. In this work, we analyze the equivariant polynomial functions for the equivariant architecture, and introduce a novel equivariant network, named PACE. The proposed PACE utilizes edge booster and the Atomic Cluster Expansion (ACE) technique to approximate a greater number of $SE(3) \times S_n$ equivariant polynomial functions with enhanced degrees. As experimented in commonly used benchmarks, PACE demonstrates state-of-the-art performance in predicting atomic energy and force fields, with robust generalization capability across various geometric distributions under molecular dynamics (MD) across different temperature conditions. Our code is publicly available as part of the AIRS library this https URL.

Title: Diversity Helps Jailbreak Large Language Models

• Authors: Weiliang Zhao, Daniel Ben-Levi, Junfeng Yang, Chengzhi Mao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04223
• Pdf URL: https://arxiv.org/pdf/2411.04223
• Copy Paste: [[2411.04223]] Diversity Helps Jailbreak Large Language Models(https://arxiv.org/abs/2411.04223)
• Keywords: security, attack, robust, large language model
• Abstract: We have uncovered a powerful jailbreak technique that leverages large language models' ability to diverge from prior context, enabling them to bypass safety constraints and generate harmful outputs. By simply instructing the LLM to deviate and obfuscate previous attacks, our method dramatically outperforms existing approaches, achieving up to a 62% higher success rate in compromising nine leading chatbots, including GPT-4, Gemini, and Llama, while using only 13% of the queries. This revelation exposes a critical flaw in current LLM safety training, suggesting that existing methods may merely mask vulnerabilities rather than eliminate them. Our findings sound an urgent alarm for the need to revolutionize testing methodologies to ensure robust and reliable LLM security.

Title: WiFlexFormer: Efficient WiFi-Based Person-Centric Sensing

• Authors: Julian Strohmayer, Matthias Wödlinger, Martin Kampel
• Subjects: cs.CV, cs.AI, cs.ET, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04224
• Pdf URL: https://arxiv.org/pdf/2411.04224
• Copy Paste: [[2411.04224]] WiFlexFormer: Efficient WiFi-Based Person-Centric Sensing(https://arxiv.org/abs/2411.04224)
• Keywords: transformer
• Abstract: We propose WiFlexFormer, a highly efficient Transformer-based architecture designed for WiFi Channel State Information (CSI)-based person-centric sensing. We benchmark WiFlexFormer against state-of-the-art vision and specialized architectures for processing radio frequency data and demonstrate that it achieves comparable Human Activity Recognition (HAR) performance while offering a significantly lower parameter count and faster inference times. With an inference time of just 10 ms on an Nvidia Jetson Orin Nano, WiFlexFormer is optimized for real-time inference. Additionally, its low parameter count contributes to improved cross-domain generalization, where it often outperforms larger models. Our comprehensive evaluation shows that WiFlexFormer is a potential solution for efficient, scalable WiFi-based sensing applications. The PyTorch implementation of WiFlexFormer is publicly available at: this https URL.

Title: Approximate Equivariance in Reinforcement Learning

• Authors: Jung Yeon Park, Sujay Bhatt, Sihan Zeng, Lawson L.S. Wong, Alec Koppel, Sumitra Ganesh, Robin Walters
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04225
• Pdf URL: https://arxiv.org/pdf/2411.04225
• Copy Paste: [[2411.04225]] Approximate Equivariance in Reinforcement Learning(https://arxiv.org/abs/2411.04225)
• Keywords: robust
• Abstract: Equivariant neural networks have shown great success in reinforcement learning, improving sample efficiency and generalization when there is symmetry in the task. However, in many problems, only approximate symmetry is present, which makes imposing exact symmetry inappropriate. Recently, approximately equivariant networks have been proposed for supervised classification and modeling physical systems. In this work, we develop approximately equivariant algorithms in reinforcement learning (RL). We define approximately equivariant MDPs and theoretically characterize the effect of approximate equivariance on the optimal Q function. We propose novel RL architectures using relaxed group convolutions and experiment on several continuous control domains and stock trading with real financial data. Our results demonstrate that approximate equivariance matches prior work when exact symmetries are present, and outperforms them when domains exhibit approximate symmetry. As an added byproduct of these techniques, we observe increased robustness to noise at test time.

Title: Differentially Private Finite Population Estimation via Survey Weight Regularization

• Authors: Jeremy Seeman, Yajuan Si, Jerome P Reiter
• Subjects: cs.CR, stat.ME
• Abstract URL: https://arxiv.org/abs/2411.04236
• Pdf URL: https://arxiv.org/pdf/2411.04236
• Copy Paste: [[2411.04236]] Differentially Private Finite Population Estimation via Survey Weight Regularization(https://arxiv.org/abs/2411.04236)
• Keywords: privacy
• Abstract: In general, it is challenging to release differentially private versions of survey-weighted statistics with low error for acceptable privacy loss. This is because weighted statistics from complex sample survey data can be more sensitive to individual survey response and weight values than unweighted statistics, resulting in differentially private mechanisms that can add substantial noise to the unbiased estimate of the finite population quantity. On the other hand, simply disregarding the survey weights adds noise to a biased estimator, which also can result in an inaccurate estimate. Thus, the problem of releasing an accurate survey-weighted estimate essentially involves a trade-off among bias, precision, and privacy. We leverage this trade-off to develop a differentially private method for estimating finite population quantities. The key step is to privately estimate a hyperparameter that determines how much to regularize or shrink survey weights as a function of privacy loss. We illustrate the differentially private finite population estimation using the Panel Study of Income Dynamics. We show that optimal strategies for releasing DP survey-weighted mean income estimates require orders-of-magnitude less noise than naively using the original survey weights without modification.

Title: Multimodal Structure-Aware Quantum Data Processing

• Authors: Hala Hawashin, Mehrnoosh Sadrzadeh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04242
• Pdf URL: https://arxiv.org/pdf/2411.04242
• Copy Paste: [[2411.04242]] Multimodal Structure-Aware Quantum Data Processing(https://arxiv.org/abs/2411.04242)
• Keywords: large language model
• Abstract: While large language models (LLMs) have advanced the field of natural language processing (NLP), their ``black box'' nature obscures their decision-making processes. To address this, researchers developed structured approaches using higher order tensors. These are able to model linguistic relations, but stall when training on classical computers due to their excessive size. Tensors are natural inhabitants of quantum systems and training on quantum computers provides a solution by translating text to variational quantum circuits. In this paper, we develop MultiQ-NLP: a framework for structure-aware data processing with multimodal text+image data. Here, ``structure'' refers to syntactic and grammatical relationships in language, as well as the hierarchical organization of visual elements in images. We enrich the translation with new types and type homomorphisms and develop novel architectures to represent structure. When tested on a main stream image classification task (SVO Probes), our best model showed a par performance with the state of the art classical models; moreover the best model was fully structured.

Title: PocoLoco: A Point Cloud Diffusion Model of Human Shape in Loose Clothing

• Authors: Siddharth Seth, Rishabh Dabral, Diogo Luvizon, Marc Habermann, Ming-Hsuan Yang, Christian Theobalt, Adam Kortylewski
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04249
• Pdf URL: https://arxiv.org/pdf/2411.04249
• Copy Paste: [[2411.04249]] PocoLoco: A Point Cloud Diffusion Model of Human Shape in Loose Clothing(https://arxiv.org/abs/2411.04249)
• Keywords: diffusion, generative
• Abstract: Modeling a human avatar that can plausibly deform to articulations is an active area of research. We present PocoLoco -- the first template-free, point-based, pose-conditioned generative model for 3D humans in loose clothing. We motivate our work by noting that most methods require a parametric model of the human body to ground pose-dependent deformations. Consequently, they are restricted to modeling clothing that is topologically similar to the naked body and do not extend well to loose clothing. The few methods that attempt to model loose clothing typically require either canonicalization or a UV-parameterization and need to address the challenging problem of explicitly estimating correspondences for the deforming clothes. In this work, we formulate avatar clothing deformation as a conditional point-cloud generation task within the denoising diffusion framework. Crucially, our framework operates directly on unordered point clouds, eliminating the need for a parametric model or a clothing template. This also enables a variety of practical applications, such as point-cloud completion and pose-based editing -- important features for virtual human animation. As current datasets for human avatars in loose clothing are far too small for training diffusion models, we release a dataset of two subjects performing various poses in loose clothing with a total of 75K point clouds. By contributing towards tackling the challenging task of effectively modeling loose clothing and expanding the available data for training these models, we aim to set the stage for further innovation in digital humans. The source code is available at this https URL .

Title: LSHBloom: Memory-efficient, Extreme-scale Document Deduplication

• Authors: Arham Khan, Robert Underwood, Carlo Siebenschuh, Yadu Babuji, Aswathy Ajith, Kyle Hippe, Ozan Gokdemir, Alexander Brace, Kyle Chard, Ian Foster
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04257
• Pdf URL: https://arxiv.org/pdf/2411.04257
• Copy Paste: [[2411.04257]] LSHBloom: Memory-efficient, Extreme-scale Document Deduplication(https://arxiv.org/abs/2411.04257)
• Keywords: large language model
• Abstract: Deduplication is a major focus for assembling and curating training datasets for large language models (LLM) -- detecting and eliminating additional instances of the same content -- in large collections of technical documents. Unrestrained, duplicates in the training dataset increase training costs and lead to undesirable properties such as memorization in trained models or cheating on evaluation. Contemporary approaches to document-level deduplication are often extremely expensive in both runtime and memory. We propose LSHBloom, an extension to MinhashLSH, which replaces the expensive LSHIndex with lightweight Bloom filters. LSHBloom demonstrates the same deduplication performance as MinhashLSH with only a marginal increase in false positives (as low as 1e-5 in our experiments); demonstrates competitive runtime (270\% faster than MinhashLSH on peS2o); and, crucially, uses just 0.6\% of the disk space required by MinhashLSH to deduplicate peS2o. We demonstrate that this space advantage scales with increased dataset size -- at the extreme scale of several billion documents, LSHBloom promises a 250\% speedup and a 54$\times$ space advantage over traditional MinHashLSH scaling deduplication of text datasets to many billions of documents.

Title: Generative Discrete Event Process Simulation for Hidden Markov Models to Predict Competitor Time-to-Market

• Authors: Nandakishore Santhi, Stephan Eidenbenz, Brian Key, George Tompkins
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04266
• Pdf URL: https://arxiv.org/pdf/2411.04266
• Copy Paste: [[2411.04266]] Generative Discrete Event Process Simulation for Hidden Markov Models to Predict Competitor Time-to-Market(https://arxiv.org/abs/2411.04266)
• Keywords: steal, generative
• Abstract: We study the challenge of predicting the time at which a competitor product, such as a novel high-capacity EV battery or a new car model, will be available to customers; as new information is obtained, this time-to-market estimate is revised. Our scenario is as follows: We assume that the product is under development at a Firm B, which is a competitor to Firm A; as they are in the same industry, Firm A has a relatively good understanding of the processes and steps required to produce the product. While Firm B tries to keep its activities hidden (think of stealth-mode for start-ups), Firm A is nevertheless able to gain periodic insights by observing what type of resources Firm B is using. We show how Firm A can build a model that predicts when Firm B will be ready to sell its product; the model leverages knowledge of the underlying processes and required resources to build a Parallel Discrete Simulation (PDES)-based process model that it then uses as a generative model to train a Hidden Markov Model (HMM). We study the question of how many resource observations Firm A requires in order to accurately assess the current state of development at Firm B. In order to gain general insights into the capabilities of this approach, we study the effect of different process graph densities, different densities of the resource-activity maps, etc., and also scaling properties as we increase the number resource counts. We find that in most cases, the HMM achieves a prediction accuracy of 70 to 80 percent after 20 (daily) observations of a production process that lasts 150 days on average and we characterize the effects of different problem instance densities on this prediction accuracy. Our results give insight into the level of market knowledge required for accurate and early time-to-market prediction.

Title: The Recurrent Sticky Hierarchical Dirichlet Process Hidden Markov Model

• Authors: Mikołaj Słupiński, Piotr Lipiński
• Subjects: cs.LG, cs.AI, math.DS, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.04278
• Pdf URL: https://arxiv.org/pdf/2411.04278
• Copy Paste: [[2411.04278]] The Recurrent Sticky Hierarchical Dirichlet Process Hidden Markov Model(https://arxiv.org/abs/2411.04278)
• Keywords: segmentation
• Abstract: The Hierarchical Dirichlet Process Hidden Markov Model (HDP-HMM) is a natural Bayesian nonparametric extension of the classical Hidden Markov Model for learning from (spatio-)temporal data. A sticky HDP-HMM has been proposed to strengthen the self-persistence probability in the HDP-HMM. Then, disentangled sticky HDP-HMM has been proposed to disentangle the strength of the self-persistence prior and transition prior. However, the sticky HDP-HMM assumes that the self-persistence probability is stationary, limiting its expressiveness. Here, we build on previous work on sticky HDP-HMM and disentangled sticky HDP-HMM, developing a more general model: the recurrent sticky HDP-HMM (RS-HDP-HMM). We develop a novel Gibbs sampling strategy for efficient inference in this model. We show that RS-HDP-HMM outperforms disentangled sticky HDP-HMM, sticky HDP-HMM, and HDP-HMM in both synthetic and real data segmentation.

Title: Bayesian Inference in Recurrent Explicit Duration Switching Linear Dynamical Systems

• Authors: Mikołaj Słupiński, Piotr Lipiński
• Subjects: cs.LG, cs.AI, math.DS, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.04280
• Pdf URL: https://arxiv.org/pdf/2411.04280
• Copy Paste: [[2411.04280]] Bayesian Inference in Recurrent Explicit Duration Switching Linear Dynamical Systems(https://arxiv.org/abs/2411.04280)
• Keywords: segmentation
• Abstract: In this paper, we propose a novel model called Recurrent Explicit Duration Switching Linear Dynamical Systems (REDSLDS) that incorporates recurrent explicit duration variables into the rSLDS model. We also propose an inference and learning scheme that involves the use of Pólya-gamma augmentation. We demonstrate the improved segmentation capabilities of our model on three benchmark datasets, including two quantitative datasets and one qualitative dataset.

Title: Generating Synthetic Electronic Health Record (EHR) Data: A Review with Benchmarking

• Authors: Xingran Chen, Zhenke Wu, Xu Shi, Hyunghoon Cho, Bhramar Mukherjee
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.04281
• Pdf URL: https://arxiv.org/pdf/2411.04281
• Copy Paste: [[2411.04281]] Generating Synthetic Electronic Health Record (EHR) Data: A Review with Benchmarking(https://arxiv.org/abs/2411.04281)
• Keywords: privacy, protect
• Abstract: We conduct a scoping review of existing approaches for synthetic EHR data generation, and benchmark major methods with proposed open-source software to offer recommendations for practitioners. We search three academic databases for our scoping review. Methods are benchmarked on open-source EHR datasets, MIMIC-III/IV. Seven existing methods covering major categories and two baseline methods are implemented and compared. Evaluation metrics concern data fidelity, downstream utility, privacy protection, and computational cost. 42 studies are identified and classified into five categories. Seven open-source methods covering all categories are selected, trained on MIMIC-III, and evaluated on MIMIC-III or MIMIC-IV for transportability considerations. Among them, GAN-based methods demonstrate competitive performance in fidelity and utility on MIMIC-III; rule-based methods excel in privacy protection. Similar findings are observed on MIMIC-IV, except that GAN-based methods further outperform the baseline methods in preserving fidelity. A Python package, ``SynthEHRella'', is provided to integrate various choices of approaches and evaluation metrics, enabling more streamlined exploration and evaluation of multiple methods. We found that method choice is governed by the relative importance of the evaluation metrics in downstream use cases. We provide a decision tree to guide the choice among the benchmarked methods. Based on the decision tree, GAN-based methods excel when distributional shifts exist between the training and testing populations. Otherwise, CorGAN and MedGAN are most suitable for association modeling and predictive modeling, respectively. Future research should prioritize enhancing fidelity of the synthetic data while controlling privacy exposure, and comprehensive benchmarking of longitudinal or conditional generation methods.

Title: Enhancing Security Control Production With Generative AI

• Authors: Chen Ling, Mina Ghashami, Vianne Gao, Ali Torkamani, Ruslan Vaulin, Nivedita Mangam, Bhavya Jain, Farhan Diwan, Malini SS, Mingrui Cheng, Shreya Tarur Kumar, Felix Candelario
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04284
• Pdf URL: https://arxiv.org/pdf/2411.04284
• Copy Paste: [[2411.04284]] Enhancing Security Control Production With Generative AI(https://arxiv.org/abs/2411.04284)
• Keywords: security, protect, robust, generative, large language model
• Abstract: Security controls are mechanisms or policies designed for cloud based services to reduce risk, protect information, and ensure compliance with security regulations. The development of security controls is traditionally a labor-intensive and time-consuming process. This paper explores the use of Generative AI to accelerate the generation of security controls. We specifically focus on generating Gherkin codes which are the domain-specific language used to define the behavior of security controls in a structured and understandable format. By leveraging large language models and in-context learning, we propose a structured framework that reduces the time required for developing security controls from 2-3 days to less than one minute. Our approach integrates detailed task descriptions, step-by-step instructions, and retrieval-augmented generation to enhance the accuracy and efficiency of the generated Gherkin code. Initial evaluations on AWS cloud services demonstrate promising results, indicating that GenAI can effectively streamline the security control development process, thus providing a robust and dynamic safeguard for cloud-based infrastructures.

Title: Robust Real-Time Mortality Prediction in the Intensive Care Unit using Temporal Difference Learning

• Authors: Thomas Frost, Kezhi Li, Steve Harris
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04285
• Pdf URL: https://arxiv.org/pdf/2411.04285
• Copy Paste: [[2411.04285]] Robust Real-Time Mortality Prediction in the Intensive Care Unit using Temporal Difference Learning(https://arxiv.org/abs/2411.04285)
• Keywords: robust
• Abstract: The task of predicting long-term patient outcomes using supervised machine learning is a challenging one, in part because of the high variance of each patient's trajectory, which can result in the model over-fitting to the training data. Temporal difference (TD) learning, a common reinforcement learning technique, may reduce variance by generalising learning to the pattern of state transitions rather than terminal outcomes. However, in healthcare this method requires several strong assumptions about patient states, and there appears to be limited literature evaluating the performance of TD learning against traditional supervised learning methods for long-term health outcome prediction tasks. In this study, we define a framework for applying TD learning to real-time irregularly sampled time series data using a Semi-Markov Reward Process. We evaluate the model framework in predicting intensive care mortality and show that TD learning under this framework can result in improved model robustness compared to standard supervised learning methods. and that this robustness is maintained even when validated on external datasets. This approach may offer a more reliable method when learning to predict patient outcomes using high-variance irregular time series data.

Title: Unfair Alignment: Examining Safety Alignment Across Vision Encoder Layers in Vision-Language Models

• Authors: Saketh Bachu, Erfan Shayegani, Trishna Chakraborty, Rohit Lal, Arindam Dutta, Chengyu Song, Yue Dong, Nael Abu-Ghazaleh, Amit K. Roy-Chowdhury
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04291
• Pdf URL: https://arxiv.org/pdf/2411.04291
• Copy Paste: [[2411.04291]] Unfair Alignment: Examining Safety Alignment Across Vision Encoder Layers in Vision-Language Models(https://arxiv.org/abs/2411.04291)
• Keywords: attack, robust, fair, large language model
• Abstract: Vision-language models (VLMs) have improved significantly in multi-modal tasks, but their more complex architecture makes their safety alignment more challenging than the alignment of large language models (LLMs). In this paper, we reveal an unfair distribution of safety across the layers of VLM's vision encoder, with earlier and middle layers being disproportionately vulnerable to malicious inputs compared to the more robust final layers. This 'cross-layer' vulnerability stems from the model's inability to generalize its safety training from the default architectural settings used during training to unseen or out-of-distribution scenarios, leaving certain layers exposed. We conduct a comprehensive analysis by projecting activations from various intermediate layers and demonstrate that these layers are more likely to generate harmful outputs when exposed to malicious inputs. Our experiments with LLaVA-1.5 and Llama 3.2 show discrepancies in attack success rates and toxicity scores across layers, indicating that current safety alignment strategies focused on a single default layer are insufficient.

Title: Fair Exploration and Exploitation

• Authors: Stephen Pasteris, Chris Hicks, Vasilios Mavroudis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04295
• Pdf URL: https://arxiv.org/pdf/2411.04295
• Copy Paste: [[2411.04295]] Fair Exploration and Exploitation(https://arxiv.org/abs/2411.04295)
• Keywords: protect, fair
• Abstract: In this paper we consider the contextual bandit problem with a finite (or infinite and clustered) context set. We consider the fully adversarial problem in which, apart from having bounded losses, there are no assumptions whatsoever on the generation of the contexts and losses. In our problem we assume that the context set is partitioned into a set of protected groups. At the start of each trial we are given a probability distribution over the context set and are required (on that trial) to be fair with respect to that distribution, in that if the context (for that trial) was drawn from the distribution then our choice of action would be unbiased towards any protected group. We develop an algorithm FexEx for this problem which has remarkable efficiency, having a space and per-trial time complexity at most linear in the dimensionality of the policy space. FexEx can handle non-stationarity, in that its regret can be bounded with respect to any sequence of policies satisfying the fairness constraints. For such a sequence the regret bound of FexEx is essentially the same as that of running Exp3.S for each context independently (an approach that does not satisfy the fairness constraints).

Title: A Capabilities Approach to Studying Bias and Harm in Language Technologies

• Authors: Hellina Hailu Nigatu, Zeerak Talat
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.04298
• Pdf URL: https://arxiv.org/pdf/2411.04298
• Copy Paste: [[2411.04298]] A Capabilities Approach to Studying Bias and Harm in Language Technologies(https://arxiv.org/abs/2411.04298)
• Keywords: fair
• Abstract: Mainstream Natural Language Processing (NLP) research has ignored the majority of the world's languages. In moving from excluding the majority of the world's languages to blindly adopting what we make for English, we first risk importing the same harms we have at best mitigated and at least measured for English. However, in evaluating and mitigating harms arising from adopting new technologies into such contexts, we often disregard (1) the actual community needs of Language Technologies, and (2) biases and fairness issues within the context of the communities. In this extended abstract, we consider fairness, bias, and inclusion in Language Technologies through the lens of the Capabilities Approach. The Capabilities Approach centers on what people are capable of achieving, given their intersectional social, political, and economic contexts instead of what resources are (theoretically) available to them. We detail the Capabilities Approach, its relationship to multilingual and multicultural evaluation, and how the framework affords meaningful collaboration with community members in defining and measuring the harms of Language Technologies.

Title: Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education

• Authors: Anand Syamkumar, Nora Tseng, Kaycie Barron, Shanglin Yang, Shamya Karumbaiah, Rheeya Uppal, Junjie Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04308
• Pdf URL: https://arxiv.org/pdf/2411.04308
• Copy Paste: [[2411.04308]] Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education(https://arxiv.org/abs/2411.04308)
• Keywords: large language model
• Abstract: Large language models (LLMs) offer promise in generating educational content, providing instructor feedback, and reducing teacher workload on assessments. While prior studies have focused on studying LLM-powered learning analytics, limited research has examined how effective LLMs are in a bilingual context. In this paper, we study the effectiveness of multilingual large language models (MLLMs) across monolingual (English-only, Spanish-only) and bilingual (Spanglish) student writing. We present a learning analytics use case that details LLM performance in assessing acceptable and unacceptable explanations of Science and Social Science concepts. Our findings reveal a significant bias in the grading performance of pre-trained models for bilingual writing compared to English-only and Spanish-only writing. Following this, we fine-tune open-source MLLMs including Llama 3.1 and Mistral NeMo using synthetic datasets generated in English, Spanish, and Spanglish. Our experiments indicate that the models perform significantly better for all three languages after fine-tuning with bilingual data. This study highlights the potential of enhancing MLLM effectiveness to support authentic language practices amongst bilingual learners. It also aims to illustrate the value of incorporating non-English languages into the design and implementation of language models in education.

Title: A Multilingual Sentiment Lexicon for Low-Resource Language Translation using Large Languages Models and Explainable AI

• Authors: Melusi Malinga, Isaac Lupanda, Mike Wa Nkongolo, Phil van Deventer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04316
• Pdf URL: https://arxiv.org/pdf/2411.04316
• Copy Paste: [[2411.04316]] A Multilingual Sentiment Lexicon for Low-Resource Language Translation using Large Languages Models and Explainable AI(https://arxiv.org/abs/2411.04316)
• Keywords: transformer, large language model
• Abstract: South Africa and the Democratic Republic of Congo (DRC) present a complex linguistic landscape with languages such as Zulu, Sepedi, Afrikaans, French, English, and Tshiluba (Ciluba), which creates unique challenges for AI-driven translation and sentiment analysis systems due to a lack of accurately labeled data. This study seeks to address these challenges by developing a multilingual lexicon designed for French and Tshiluba, now expanded to include translations in English, Afrikaans, Sepedi, and Zulu. The lexicon enhances cultural relevance in sentiment classification by integrating language-specific sentiment scores. A comprehensive testing corpus is created to support translation and sentiment analysis tasks, with machine learning models such as Random Forest, Support Vector Machine (SVM), Decision Trees, and Gaussian Naive Bayes (GNB) trained to predict sentiment across low resource languages (LRLs). Among them, the Random Forest model performed particularly well, capturing sentiment polarity and handling language-specific nuances effectively. Furthermore, Bidirectional Encoder Representations from Transformers (BERT), a Large Language Model (LLM), is applied to predict context-based sentiment with high accuracy, achieving 99% accuracy and 98% precision, outperforming other models. The BERT predictions were clarified using Explainable AI (XAI), improving transparency and fostering confidence in sentiment classification. Overall, findings demonstrate that the proposed lexicon and machine learning models significantly enhance translation and sentiment analysis for LRLs in South Africa and the DRC, laying a foundation for future AI models that support underrepresented languages, with applications across education, governance, and business in multilingual contexts.

Title: Efficient Symmetry-Aware Materials Generation via Hierarchical Generative Flow Networks

• Authors: Tri Minh Nguyen, Sherif Abdulkader Tawfik, Truyen Tran, Sunil Gupta, Santu Rana, Svetha Venkatesh
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2411.04323
• Pdf URL: https://arxiv.org/pdf/2411.04323
• Copy Paste: [[2411.04323]] Efficient Symmetry-Aware Materials Generation via Hierarchical Generative Flow Networks(https://arxiv.org/abs/2411.04323)
• Keywords: diffusion, generative
• Abstract: Discovering new solid-state materials requires rapidly exploring the vast space of crystal structures and locating stable regions. Generating stable materials with desired properties and compositions is extremely difficult as we search for very small isolated pockets in the exponentially many possibilities, considering elements from the periodic table and their 3D arrangements in crystal lattices. Materials discovery necessitates both optimized solution structures and diversity in the generated material structures. Existing methods struggle to explore large material spaces and generate diverse samples with desired properties and requirements. We propose the Symmetry-aware Hierarchical Architecture for Flow-based Traversal (SHAFT), a novel generative model employing a hierarchical exploration strategy to efficiently exploit the symmetry of the materials space to generate crystal structures given desired properties. In particular, our model decomposes the exponentially large materials space into a hierarchy of subspaces consisting of symmetric space groups, lattice parameters, and atoms. We demonstrate that SHAFT significantly outperforms state-of-the-art iterative generative methods, such as Generative Flow Networks (GFlowNets) and Crystal Diffusion Variational AutoEncoders (CDVAE), in crystal structure generation tasks, achieving higher validity, diversity, and stability of generated structures optimized for target properties and requirements.

Title: Gradient Boosting Trees and Large Language Models for Tabular Data Few-Shot Learning

• Authors: Carlos Huertas
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04324
• Pdf URL: https://arxiv.org/pdf/2411.04324
• Copy Paste: [[2411.04324]] Gradient Boosting Trees and Large Language Models for Tabular Data Few-Shot Learning(https://arxiv.org/abs/2411.04324)
• Keywords: large language model
• Abstract: Large Language Models (LLM) have brought numerous of new applications to Machine Learning (ML). In the context of tabular data (TD), recent studies show that TabLLM is a very powerful mechanism for few-shot-learning (FSL) applications, even if gradient boosting decisions trees (GBDT) have historically dominated the TD field. In this work we demonstrate that although LLMs are a viable alternative, the evidence suggests that baselines used to gauge performance can be improved. We replicated public benchmarks and our methodology improves LightGBM by 290%, this is mainly driven by forcing node splitting with few samples, a critical step in FSL with GBDT. Our results show an advantage to TabLLM for 8 or fewer shots, but as the number of samples increases GBDT provides competitive performance at a fraction of runtime. For other real-life applications with vast number of samples, we found FSL still useful to improve model diversity, and when combined with ExtraTrees it provides strong resilience to overfitting, our proposal was validated in a ML competition setting ranking first place.

Title: Balancing Transparency and Accuracy: A Comparative Analysis of Rule-Based and Deep Learning Models in Political Bias Classification

• Authors: Manuel Nunez Martinez, Sonja Schmer-Galunder, Zoey Liu, Sangpil Youm, Chathuri Jayaweera, Bonnie J. Dorr
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04328
• Pdf URL: https://arxiv.org/pdf/2411.04328
• Copy Paste: [[2411.04328]] Balancing Transparency and Accuracy: A Comparative Analysis of Rule-Based and Deep Learning Models in Political Bias Classification(https://arxiv.org/abs/2411.04328)
• Keywords: explainability
• Abstract: The unchecked spread of digital information, combined with increasing political polarization and the tendency of individuals to isolate themselves from opposing political viewpoints, has driven researchers to develop systems for automatically detecting political bias in media. This trend has been further fueled by discussions on social media. We explore methods for categorizing bias in US news articles, comparing rule-based and deep learning approaches. The study highlights the sensitivity of modern self-learning systems to unconstrained data ingestion, while reconsidering the strengths of traditional rule-based systems. Applying both models to left-leaning (CNN) and right-leaning (FOX) news articles, we assess their effectiveness on data beyond the original training and test this http URL analysis highlights each model's accuracy, offers a framework for exploring deep-learning explainability, and sheds light on political bias in US news media. We contrast the opaque architecture of a deep learning model with the transparency of a linguistically informed rule-based model, showing that the rule-based model performs consistently across different data conditions and offers greater transparency, whereas the deep learning model is dependent on the training set and struggles with unseen data.

Title: CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models

• Authors: Jierui Li, Hung Le, Yinbo Zhou, Caiming Xiong, Silvio Savarese, Doyen Sahoo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04329
• Pdf URL: https://arxiv.org/pdf/2411.04329
• Copy Paste: [[2411.04329]] CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models(https://arxiv.org/abs/2411.04329)
• Keywords: large language model
• Abstract: Pre-trained on massive amounts of code and text data, large language models (LLMs) have demonstrated remarkable achievements in performing code generation tasks. With additional execution-based feedback, these models can act as agents with capabilities to self-refine and improve generated code autonomously. However, on challenging coding tasks with extremely large search space, current agentic approaches still struggle with multi-stage planning, generating, and debugging. To address this problem, we propose CodeTree, a framework for LLM agents to efficiently explore the search space in different stages of the code generation process. Specifically, we adopted a unified tree structure to explicitly explore different coding strategies, generate corresponding coding solutions, and subsequently refine the solutions. In each stage, critical decision-making (ranking, termination, expanding) of the exploration process is guided by both the environmental execution-based feedback and LLM-agent-generated feedback. We comprehensively evaluated CodeTree on 7 code generation benchmarks and demonstrated the significant performance gains of CodeTree against strong baselines. Using GPT-4o as the base model, we consistently achieved top results of 95.1 on HumanEval, 98.7 on MBPP, and 43.0 on CodeContests. On the challenging SWEBench benchmark, our approach led to significant performance gains.

Title: HandCraft: Anatomically Correct Restoration of Malformed Hands in Diffusion Generated Images

• Authors: Zhenyue Qin, Yiqun Zhang, Yang Liu, Dylan Campbell
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04332
• Pdf URL: https://arxiv.org/pdf/2411.04332
• Copy Paste: [[2411.04332]] HandCraft: Anatomically Correct Restoration of Malformed Hands in Diffusion Generated Images(https://arxiv.org/abs/2411.04332)
• Keywords: diffusion, generative
• Abstract: Generative text-to-image models, such as Stable Diffusion, have demonstrated a remarkable ability to generate diverse, high-quality images. However, they are surprisingly inept when it comes to rendering human hands, which are often anatomically incorrect or reside in the "uncanny valley". In this paper, we propose a method HandCraft for restoring such malformed hands. This is achieved by automatically constructing masks and depth images for hands as conditioning signals using a parametric model, allowing a diffusion-based image editor to fix the hand's anatomy and adjust its pose while seamlessly integrating the changes into the original image, preserving pose, color, and style. Our plug-and-play hand restoration solution is compatible with existing pretrained diffusion models, and the restoration process facilitates adoption by eschewing any fine-tuning or training requirements for the diffusion models. We also contribute MalHand datasets that contain generated images with a wide variety of malformed hands in several styles for hand detector training and hand restoration benchmarking, and demonstrate through qualitative and quantitative evaluation that HandCraft not only restores anatomical correctness but also maintains the integrity of the overall image.

Title: GazeGen: Gaze-Driven User Interaction for Visual Content Generation

• Authors: He-Yen Hsieh, Ziyun Li, Sai Qian Zhang, Wei-Te Mark Ting, Kao-Den Chang, Barbara De Salvo, Chiao Liu, H. T. Kung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04335
• Pdf URL: https://arxiv.org/pdf/2411.04335
• Copy Paste: [[2411.04335]] GazeGen: Gaze-Driven User Interaction for Visual Content Generation(https://arxiv.org/abs/2411.04335)
• Keywords: generative
• Abstract: We present GazeGen, a user interaction system that generates visual content (images and videos) for locations indicated by the user's eye gaze. GazeGen allows intuitive manipulation of visual content by targeting regions of interest with gaze. Using advanced techniques in object detection and generative AI, GazeGen performs gaze-controlled image adding/deleting, repositioning, and surface material changes of image objects, and converts static images into videos. Central to GazeGen is the DFT Gaze (Distilled and Fine-Tuned Gaze) agent, an ultra-lightweight model with only 281K parameters, performing accurate real-time gaze predictions tailored to individual users' eyes on small edge devices. GazeGen is the first system to combine visual content generation with real-time gaze estimation, made possible exclusively by DFT Gaze. This real-time gaze estimation enables various visual content generation tasks, all controlled by the user's gaze. The input for DFT Gaze is the user's eye images, while the inputs for visual content generation are the user's view and the predicted gaze point from DFT Gaze. To achieve efficient gaze predictions, we derive the small model from a large model (10x larger) via novel knowledge distillation and personal adaptation techniques. We integrate knowledge distillation with a masked autoencoder, developing a compact yet powerful gaze estimation model. This model is further fine-tuned with Adapters, enabling highly accurate and personalized gaze predictions with minimal user input. DFT Gaze ensures low-latency and precise gaze tracking, supporting a wide range of gaze-driven tasks. We validate the performance of DFT Gaze on AEA and OpenEDS2020 benchmarks, demonstrating low angular gaze error and low latency on the edge device (Raspberry Pi 4). Furthermore, we describe applications of GazeGen, illustrating its versatility and effectiveness in various usage scenarios.

Title: Enhancing classroom teaching with LLMs and RAG

• Authors: Elizabeth A Mullins, Adrian Portillo, Kristalys Ruiz-Rohena, Aritran Piplai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04341
• Pdf URL: https://arxiv.org/pdf/2411.04341
• Copy Paste: [[2411.04341]] Enhancing classroom teaching with LLMs and RAG(https://arxiv.org/abs/2411.04341)
• Keywords: security, large language model
• Abstract: Large Language Models have become a valuable source of information for our daily inquiries. However, after training, its data source quickly becomes out-of-date, making RAG a useful tool for providing even more recent or pertinent data. In this work, we investigate how RAG pipelines, with the course materials serving as a data source, might help students in K-12 education. The initial research utilizes Reddit as a data source for up-to-date cybersecurity information. Chunk size is evaluated to determine the optimal amount of context needed to generate accurate answers. After running the experiment for different chunk sizes, answer correctness was evaluated using RAGAs with average answer correctness not exceeding 50 percent for any chunk size. This suggests that Reddit is not a good source to mine for data for questions about cybersecurity threats. The methodology was successful in evaluating the data source, which has implications for its use to evaluate educational resources for effectiveness.

Title: LidaRefer: Outdoor 3D Visual Grounding for Autonomous Driving with Transformers

• Authors: Yeong-Seung Baek, Heung-Seon Oh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04351
• Pdf URL: https://arxiv.org/pdf/2411.04351
• Copy Paste: [[2411.04351]] LidaRefer: Outdoor 3D Visual Grounding for Autonomous Driving with Transformers(https://arxiv.org/abs/2411.04351)
• Keywords: transformer
• Abstract: 3D visual grounding (VG) aims to locate relevant objects or regions within 3D scenes based on natural language descriptions. Although recent methods for indoor 3D VG have successfully transformer-based architectures to capture global contextual information and enable fine-grained cross-modal fusion, they are unsuitable for outdoor environments due to differences in the distribution of point clouds between indoor and outdoor settings. Specifically, first, extensive LiDAR point clouds demand unacceptable computational and memory resources within transformers due to the high-dimensional visual features. Second, dominant background points and empty spaces in sparse LiDAR point clouds complicate cross-modal fusion owing to their irrelevant visual information. To address these challenges, we propose LidaRefer, a transformer-based 3D VG framework designed for large-scale outdoor scenes. Moreover, during training, we introduce a simple and effective localization method, which supervises the decoder's queries to localize not only a target object but also ambiguous objects that might be confused as the target due to the exhibition of similar attributes in a scene or the incorrect understanding of a language description. This supervision enhances the model's ability to distinguish ambiguous objects from a target by learning the differences in their spatial relationships and attributes. LidaRefer achieves state-of-the-art performance on Talk2Car-3D, a 3D VG dataset for autonomous driving, with significant improvements under various evaluation settings.

Title: GaGSL: Global-augmented Graph Structure Learning via Graph Information Bottleneck

• Authors: Shuangjie Li, Jiangqing Song, Baoming Zhang, Gaoli Ruan, Junyuan Xie, Chongjun Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04356
• Pdf URL: https://arxiv.org/pdf/2411.04356
• Copy Paste: [[2411.04356]] GaGSL: Global-augmented Graph Structure Learning via Graph Information Bottleneck(https://arxiv.org/abs/2411.04356)
• Keywords: robust
• Abstract: Graph neural networks (GNNs) are prominent for their effectiveness in processing graph data for semi-supervised node classification tasks. Most works of GNNs assume that the observed structure accurately represents the underlying node relationships. However, the graph structure is inevitably noisy or incomplete in reality, which can degrade the quality of graph representations. Therefore, it is imperative to learn a clean graph structure that balances performance and robustness. In this paper, we propose a novel method named \textit{Global-augmented Graph Structure Learning} (GaGSL), guided by the Graph Information Bottleneck (GIB) principle. The key idea behind GaGSL is to learn a compact and informative graph structure for node classification tasks. Specifically, to mitigate the bias caused by relying solely on the original structure, we first obtain augmented features and augmented structure through global feature augmentation and global structure augmentation. We then input the augmented features and augmented structure into a structure estimator with different parameters for optimization and re-definition of the graph structure, respectively. The redefined structures are combined to form the final graph structure. Finally, we employ GIB based on mutual information to guide the optimization of the graph structure to obtain the minimum sufficient graph structure. Comprehensive evaluations across a range of datasets reveal the outstanding performance and robustness of GaGSL compared with the state-of-the-art methods.

Title: MegaPortrait: Revisiting Diffusion Control for High-fidelity Portrait Generation

• Authors: Han Yang, Sotiris Anagnostidis, Enis Simsar, Thomas Hofmann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04357
• Pdf URL: https://arxiv.org/pdf/2411.04357
• Copy Paste: [[2411.04357]] MegaPortrait: Revisiting Diffusion Control for High-fidelity Portrait Generation(https://arxiv.org/abs/2411.04357)
• Keywords: diffusion
• Abstract: We propose MegaPortrait. It's an innovative system for creating personalized portrait images in computer vision. It has three modules: Identity Net, Shading Net, and Harmonization Net. Identity Net generates learned identity using a customized model fine-tuned with source images. Shading Net re-renders portraits using extracted representations. Harmonization Net fuses pasted faces and the reference image's body for coherent results. Our approach with off-the-shelf Controlnets is better than state-of-the-art AI portrait products in identity preservation and image fidelity. MegaPortrait has a simple but effective design and we compare it with other methods and products to show its superiority.

Title: Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation

• Authors: Vaibhav Seth, Arinjay Pathak, Ayan Sengupta, Natraj Raman, Sriram Gopalakrishnan, Tanmoy Chakraborty
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04358
• Pdf URL: https://arxiv.org/pdf/2411.04358
• Copy Paste: [[2411.04358]] Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation(https://arxiv.org/abs/2411.04358)
• Keywords: robust, generative, large language model
• Abstract: Large Language Models (LLMs) are highly resource-intensive to fine-tune due to their enormous size. While low-rank adaptation is a prominent parameter-efficient fine-tuning approach, it suffers from sensitivity to hyperparameter choices, leading to instability in model performance on fine-tuning downstream tasks. This paper highlights the importance of effective parameterization in low-rank fine-tuning to reduce estimator variance and enhance the stability of final model outputs. We propose MonteCLoRA, an efficient fine-tuning technique, employing Monte Carlo estimation to learn an unbiased posterior estimation of low-rank parameters with low expected variance, which stabilizes fine-tuned LLMs with only O(1) additional parameters. MonteCLoRA shows significant improvements in accuracy and robustness, achieving up to 3.8% higher accuracy and 8.6% greater robustness than existing efficient fine-tuning methods on natural language understanding tasks with pre-trained RoBERTa-base. Furthermore, in generative tasks with pre-trained LLaMA-1-7B, MonteCLoRA demonstrates robust zero-shot performance with 50% lower variance than the contemporary efficient fine-tuning methods. The theoretical and empirical results presented in the paper underscore how parameterization and hyperpriors balance exploration-exploitation in the low-rank parametric space, therefore leading to more optimal and robust parameter estimation during efficient fine-tuning.

Title: Measuring short-form factuality in large language models

• Authors: Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, William Fedus
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04368
• Pdf URL: https://arxiv.org/pdf/2411.04368
• Copy Paste: [[2411.04368]] Measuring short-form factuality in large language models(https://arxiv.org/abs/2411.04368)
• Keywords: large language model
• Abstract: We present SimpleQA, a benchmark that evaluates the ability of language models to answer short, fact-seeking questions. We prioritized two properties in designing this eval. First, SimpleQA is challenging, as it is adversarially collected against GPT-4 responses. Second, responses are easy to grade, because questions are created such that there exists only a single, indisputable answer. Each answer in SimpleQA is graded as either correct, incorrect, or not attempted. A model with ideal behavior would get as many questions correct as possible while not attempting the questions for which it is not confident it knows the correct answer. SimpleQA is a simple, targeted evaluation for whether models "know what they know," and our hope is that this benchmark will remain relevant for the next few generations of frontier models. SimpleQA can be found at this https URL.

Title: ComFairGNN: Community Fair Graph Neural Network

• Authors: Yonas Sium, Qi Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04371
• Pdf URL: https://arxiv.org/pdf/2411.04371
• Copy Paste: [[2411.04371]] ComFairGNN: Community Fair Graph Neural Network(https://arxiv.org/abs/2411.04371)
• Keywords: fair
• Abstract: Graph Neural Networks (GNNs) have become the leading approach for addressing graph analytical problems in various real-world scenarios. However, GNNs may produce biased predictions against certain demographic subgroups due to node attributes and neighbors surrounding a node. Most current research on GNN fairness focuses predominantly on debiasing GNNs using oversimplified fairness evaluation metrics, which can give a misleading impression of fairness. Understanding the potential evaluation paradoxes due to the complicated nature of the graph structure is crucial for developing effective GNN debiasing mechanisms. In this paper, we examine the effectiveness of current GNN debiasing methods in terms of unfairness evaluation. Specifically, we introduce a community-level strategy to measure bias in GNNs and evaluate debiasing methods at this level. Further, We introduce ComFairGNN, a novel framework designed to mitigate community-level bias in GNNs. Our approach employs a learnable coreset-based debiasing function that addresses bias arising from diverse local neighborhood distributions during GNNs neighborhood aggregation. Comprehensive evaluations on three benchmark datasets demonstrate our model's effectiveness in both accuracy and fairness metrics.

Title: Benchmarking Large Language Models with Integer Sequence Generation Tasks

• Authors: Daniel O'Malley, Manish Bhattarai, Javier Santos
• Subjects: cs.LG, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2411.04372
• Pdf URL: https://arxiv.org/pdf/2411.04372
• Copy Paste: [[2411.04372]] Benchmarking Large Language Models with Integer Sequence Generation Tasks(https://arxiv.org/abs/2411.04372)
• Keywords: large language model
• Abstract: This paper presents a novel benchmark where the large language model (LLM) must write code that computes integer sequences from the Online Encyclopedia of Integer Sequences (OEIS), a widely-used resource for mathematical sequences. The benchmark is designed to evaluate both the correctness of the generated code and its computational efficiency. Our benchmark reveals that the o1 series of models outperform other frontier models from OpenAI, Anthropic, Meta, and Google in accuracy and cheating rates across both easy and hard integer sequences. In order to ensure models do not exploit memorized sequence values, we introduce an automated cheating detection mechanism that flags the use of lookup tables and validated this automation against human cheating evaluations. This benchmark provides a meaningful challenge for current LLMs, offering insights into their mathematical reasoning and code writing capabilities, which can guide future research directions and model development in mathematical reasoning and code synthesis.

Title: Game-Theoretic Defenses for Robust Conformal Prediction Against Adversarial Attacks in Medical Imaging

• Authors: Rui Luo, Jie Bao, Zhixin Zhou, Chuangyin Dang
• Subjects: cs.LG, cs.CR, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.04376
• Pdf URL: https://arxiv.org/pdf/2411.04376
• Copy Paste: [[2411.04376]] Game-Theoretic Defenses for Robust Conformal Prediction Against Adversarial Attacks in Medical Imaging(https://arxiv.org/abs/2411.04376)
• Keywords: defense, attack, robust
• Abstract: Adversarial attacks pose significant threats to the reliability and safety of deep learning models, especially in critical domains such as medical imaging. This paper introduces a novel framework that integrates conformal prediction with game-theoretic defensive strategies to enhance model robustness against both known and unknown adversarial perturbations. We address three primary research questions: constructing valid and efficient conformal prediction sets under known attacks (RQ1), ensuring coverage under unknown attacks through conservative thresholding (RQ2), and determining optimal defensive strategies within a zero-sum game framework (RQ3). Our methodology involves training specialized defensive models against specific attack types and employing maximum and minimum classifiers to aggregate defenses effectively. Extensive experiments conducted on the MedMNIST datasets, including PathMNIST, OrganAMNIST, and TissueMNIST, demonstrate that our approach maintains high coverage guarantees while minimizing prediction set sizes. The game-theoretic analysis reveals that the optimal defensive strategy often converges to a singular robust model, outperforming uniform and simple strategies across all evaluated datasets. This work advances the state-of-the-art in uncertainty quantification and adversarial robustness, providing a reliable mechanism for deploying deep learning models in adversarial environments.

Title: TrajGPT: Controlled Synthetic Trajectory Generation Using a Multitask Transformer-Based Spatiotemporal Model

• Authors: Shang-Ling Hsu, Emmanuel Tung, John Krumm, Cyrus Shahabi, Khurram Shafique
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04381
• Pdf URL: https://arxiv.org/pdf/2411.04381
• Copy Paste: [[2411.04381]] TrajGPT: Controlled Synthetic Trajectory Generation Using a Multitask Transformer-Based Spatiotemporal Model(https://arxiv.org/abs/2411.04381)
• Keywords: transformer, generative, large language model
• Abstract: Human mobility modeling from GPS-trajectories and synthetic trajectory generation are crucial for various applications, such as urban planning, disaster management and epidemiology. Both of these tasks often require filling gaps in a partially specified sequence of visits - a new problem that we call "controlled" synthetic trajectory generation. Existing methods for next-location prediction or synthetic trajectory generation cannot solve this problem as they lack the mechanisms needed to constrain the generated sequences of visits. Moreover, existing approaches (1) frequently treat space and time as independent factors, an assumption that fails to hold true in real-world scenarios, and (2) suffer from challenges in accuracy of temporal prediction as they fail to deal with mixed distributions and the inter-relationships of different modes with latent variables (e.g., day-of-the-week). These limitations become even more pronounced when the task involves filling gaps within sequences instead of solely predicting the next visit. We introduce TrajGPT, a transformer-based, multi-task, joint spatiotemporal generative model to address these issues. Taking inspiration from large language models, TrajGPT poses the problem of controlled trajectory generation as that of text infilling in natural language. TrajGPT integrates the spatial and temporal models in a transformer architecture through a Bayesian probability model that ensures that the gaps in a visit sequence are filled in a spatiotemporally consistent manner. Our experiments on public and private datasets demonstrate that TrajGPT not only excels in controlled synthetic visit generation but also outperforms competing models in next-location prediction tasks - Relatively, TrajGPT achieves a 26-fold improvement in temporal accuracy while retaining more than 98% of spatial accuracy on average.

Title: Unlearning in- vs. out-of-distribution data in LLMs under gradient-based method

• Authors: Teodora Baluta, Pascal Lamblin, Daniel Tarlow, Fabian Pedregosa, Gintare Karolina Dziugaite
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04388
• Pdf URL: https://arxiv.org/pdf/2411.04388
• Copy Paste: [[2411.04388]] Unlearning in- vs. out-of-distribution data in LLMs under gradient-based method(https://arxiv.org/abs/2411.04388)
• Keywords: generative, large language model
• Abstract: Machine unlearning aims to solve the problem of removing the influence of selected training examples from a learned model. Despite the increasing attention to this problem, it remains an open research question how to evaluate unlearning in large language models (LLMs), and what are the critical properties of the data to be unlearned that affect the quality and efficiency of unlearning. This work formalizes a metric to evaluate unlearning quality in generative models, and uses it to assess the trade-offs between unlearning quality and performance. We demonstrate that unlearning out-of-distribution examples requires more unlearning steps but overall presents a better trade-off overall. For in-distribution examples, however, we observe a rapid decay in performance as unlearning progresses. We further evaluate how example's memorization and difficulty affect unlearning under a classical gradient ascent-based approach.

Title: Variational Low-Rank Adaptation Using IVON

• Authors: Bai Cong, Nico Daheim, Yuesong Shen, Daniel Cremers, Rio Yokota, Mohammad Emtiyaz Khan, Thomas Möllenhoff
• Subjects: cs.LG, cs.AI, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.04421
• Pdf URL: https://arxiv.org/pdf/2411.04421
• Copy Paste: [[2411.04421]] Variational Low-Rank Adaptation Using IVON(https://arxiv.org/abs/2411.04421)
• Keywords: large language model
• Abstract: We show that variational learning can significantly improve the accuracy and calibration of Low-Rank Adaptation (LoRA) without a substantial increase in the cost. We replace AdamW by the Improved Variational Online Newton (IVON) algorithm to finetune large language models. For Llama-2 with 7 billion parameters, IVON improves the accuracy over AdamW by 2.8% and expected calibration error by 4.6%. The accuracy is also better than the other Bayesian alternatives, yet the cost is lower and the implementation is easier. Our work provides additional evidence for the effectiveness of IVON for large language models. The code is available at this https URL.

Title: Bayesian Calibration of Win Rate Estimation with LLM Evaluators

• Authors: Yicheng Gao, Gonghan Xu, Zhe Wang, Arman Cohan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04424
• Pdf URL: https://arxiv.org/pdf/2411.04424
• Copy Paste: [[2411.04424]] Bayesian Calibration of Win Rate Estimation with LLM Evaluators(https://arxiv.org/abs/2411.04424)
• Keywords: generative, large language model
• Abstract: Recent advances in large language models (LLMs) show the potential of using LLMs as evaluators for assessing the quality of text generations from LLMs. However, applying LLM evaluators naively to compare or judge between different systems can lead to unreliable results due to the intrinsic win rate estimation bias of LLM evaluators. In order to mitigate this problem, we propose two calibration methods, Bayesian Win Rate Sampling (BWRS) and Bayesian Dawid-Skene, both of which leverage Bayesian inference to more accurately infer the true win rate of generative language models. We empirically validate our methods on six datasets covering story generation, summarization, and instruction following tasks. We show that both our methods are effective in improving the accuracy of win rate estimation using LLMs as evaluators, offering a promising direction for reliable automatic text quality evaluation.

Title: DELIFT: Data Efficient Language model Instruction Fine Tuning

• Authors: Ishika Agarwal, Krishna Killamsetty, Lucian Popa, Marina Danilevksy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04425
• Pdf URL: https://arxiv.org/pdf/2411.04425
• Copy Paste: [[2411.04425]] DELIFT: Data Efficient Language model Instruction Fine Tuning(https://arxiv.org/abs/2411.04425)
• Keywords: large language model
• Abstract: Fine-tuning large language models (LLMs) is essential for enhancing their performance on specific tasks but is often resource-intensive due to redundant or uninformative data. To address this inefficiency, we introduce DELIFT (Data Efficient Language model Instruction Fine-Tuning), a novel algorithm that systematically optimizes data selection across the three key stages of fine-tuning: (1) instruction tuning, (2) task-specific fine-tuning (e.g., reasoning, question-answering), and (3) continual fine-tuning (e.g., incorporating new data versions). Unlike existing methods that focus on single-stage optimization or rely on computationally intensive gradient calculations, DELIFT operates efficiently across all stages. Central to our approach is a pairwise utility metric that quantifies how beneficial a data sample is for improving the model's responses to other samples, effectively measuring the informational value relative to the model's current capabilities. By leveraging different submodular functions applied to this metric, DELIFT selects diverse and optimal subsets that are useful across all stages of fine-tuning. Experiments across various tasks and model scales demonstrate that DELIFT can reduce the fine-tuning data size by up to 70% without compromising performance, offering significant computational savings and outperforming existing methods in both efficiency and efficacy.

Title: One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity

• Authors: Sonia K. Murthy, Tomer Ullman, Jennifer Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04427
• Pdf URL: https://arxiv.org/pdf/2411.04427
• Copy Paste: [[2411.04427]] One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity(https://arxiv.org/abs/2411.04427)
• Keywords: large language model
• Abstract: Researchers in social science and psychology have recently proposed using large language models (LLMs) as replacements for humans in behavioral research. In addition to arguments about whether LLMs accurately capture population-level patterns, this has raised questions about whether LLMs capture human-like conceptual diversity. Separately, it is debated whether post-training alignment (RLHF or RLAIF) affects models' internal diversity. Inspired by human studies, we use a new way of measuring the conceptual diversity of synthetically-generated LLM "populations" by relating the internal variability of simulated individuals to the population-level variability. We use this approach to evaluate non-aligned and aligned LLMs on two domains with rich human behavioral data. While no model reaches human-like diversity, aligned models generally display less diversity than their instruction fine-tuned counterparts. Our findings highlight potential trade-offs between increasing models' value alignment and decreasing the diversity of their conceptual representations.

Title: Towards Unifying Interpretability and Control: Evaluation via Intervention

• Authors: Usha Bhalla, Suraj Srinivas, Asma Ghandeharioun, Himabindu Lakkaraju
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04430
• Pdf URL: https://arxiv.org/pdf/2411.04430
• Copy Paste: [[2411.04430]] Towards Unifying Interpretability and Control: Evaluation via Intervention(https://arxiv.org/abs/2411.04430)
• Keywords: interpretability, large language model
• Abstract: With the growing complexity and capability of large language models, a need to understand model reasoning has emerged, often motivated by an underlying goal of controlling and aligning models. While numerous interpretability and steering methods have been proposed as solutions, they are typically designed either for understanding or for control, seldom addressing both, with the connection between interpretation and control more broadly remaining tenuous. Additionally, the lack of standardized applications, motivations, and evaluation metrics makes it difficult to assess these methods' practical utility and efficacy. To address this, we propose intervention as a fundamental goal of interpretability and introduce success criteria to evaluate how well methods are able to control model behavior through interventions. We unify and extend four popular interpretability methods--sparse autoencoders, logit lens, tuned lens, and probing--into an abstract encoder-decoder framework. This framework maps intermediate latent representations to human-interpretable feature spaces, enabling interventions on these interpretable features, which can then be mapped back to latent representations to control model outputs. We introduce two new evaluation metrics: intervention success rate and the coherence-intervention tradeoff, designed to measure the accuracy of explanations and their utility in controlling model behavior. Our findings reveal that (1) although current methods allow for intervention, they are inconsistent across models and features, (2) lens-based methods outperform others in achieving simple, concrete interventions, and (3) interventions often compromise model performance and coherence, underperforming simpler alternatives, such as prompting, for steering model behavior and highlighting a critical shortcoming of current interpretability approaches in real-world applications requiring control.

Title: Scaling Laws for Pre-training Agents and World Models

• Authors: Tim Pearce, Tabish Rashid, Dave Bignell, Raluca Georgescu, Sam Devlin, Katja Hofmann
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04434
• Pdf URL: https://arxiv.org/pdf/2411.04434
• Copy Paste: [[2411.04434]] Scaling Laws for Pre-training Agents and World Models(https://arxiv.org/abs/2411.04434)
• Keywords: generative
• Abstract: The performance of embodied agents has been shown to improve by increasing model parameters, dataset size, and compute. This has been demonstrated in domains from robotics to video games, when generative learning objectives on offline datasets (pre-training) are used to model an agent's behavior (imitation learning) or their environment (world modeling). This paper characterizes the role of scale in these tasks more precisely. Going beyond the simple intuition that `bigger is better', we show that the same types of power laws found in language modeling (e.g. between loss and optimal model size), also arise in world modeling and imitation learning. However, the coefficients of these laws are heavily influenced by the tokenizer, task \& architecture -- this has important implications on the optimal sizing of models and data.

Title: Gradient Localization Improves Lifelong Pretraining of Language Models

• Authors: Jared Fernandez, Yonatan Bisk, Emma Strubell
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04448
• Pdf URL: https://arxiv.org/pdf/2411.04448
• Copy Paste: [[2411.04448]] Gradient Localization Improves Lifelong Pretraining of Language Models(https://arxiv.org/abs/2411.04448)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) trained on web-scale text corpora have been shown to capture world knowledge in their parameters. However, the mechanism by which language models store different types of knowledge is poorly understood. In this work, we examine two types of knowledge relating to temporally sensitive entities and demonstrate that each type is localized to different sets of parameters within the LLMs. We hypothesize that the lack of consideration of the locality of knowledge in existing continual learning methods contributes to both: the failed uptake of new information, and catastrophic forgetting of previously learned information. We observe that sequences containing references to updated and newly mentioned entities exhibit larger gradient norms in a subset of layers. We demonstrate that targeting parameter updates to these relevant layers can improve the performance of continually pretraining on language containing temporal drift.

Title: Comparing Fairness of Generative Mobility Models

• Authors: Daniel Wang, Jack McFarland, Afra Mashhadi, Ekin Ugurel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04453
• Pdf URL: https://arxiv.org/pdf/2411.04453
• Copy Paste: [[2411.04453]] Comparing Fairness of Generative Mobility Models(https://arxiv.org/abs/2411.04453)
• Keywords: fair, generative
• Abstract: This work examines the fairness of generative mobility models, addressing the often overlooked dimension of equity in model performance across geographic regions. Predictive models built on crowd flow data are instrumental in understanding urban structures and movement patterns; however, they risk embedding biases, particularly in spatiotemporal contexts where model performance may reflect and reinforce existing inequities tied to geographic distribution. We propose a novel framework for assessing fairness by measuring the utility and equity of generated traces. Utility is assessed via the Common Part of Commuters (CPC), a similarity metric comparing generated and real mobility flows, while fairness is evaluated using demographic parity. By reformulating demographic parity to reflect the difference in CPC distribution between two groups, our analysis reveals disparities in how various models encode biases present in the underlying data. We utilized four models (Gravity, Radiation, Deep Gravity, and Non-linear Gravity) and our results indicate that traditional gravity and radiation models produce fairer outcomes, although Deep Gravity achieves higher CPC. This disparity underscores a trade-off between model accuracy and equity, with the feature-rich Deep Gravity model amplifying pre-existing biases in community representations. Our findings emphasize the importance of integrating fairness metrics in mobility modeling to avoid perpetuating inequities.

Title: Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity

• Authors: Robby Costales, Stefanos Nikolaidis
• Subjects: cs.LG, cs.AI, cs.RO, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.04466
• Pdf URL: https://arxiv.org/pdf/2411.04466
• Copy Paste: [[2411.04466]] Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity(https://arxiv.org/abs/2411.04466)
• Keywords: robust
• Abstract: The wider application of end-to-end learning methods to embodied decision-making domains remains bottlenecked by their reliance on a superabundance of training data representative of the target domain. Meta-reinforcement learning (meta-RL) approaches abandon the aim of zero-shot generalization--the goal of standard reinforcement learning (RL)--in favor of few-shot adaptation, and thus hold promise for bridging larger generalization gaps. While learning this meta-level adaptive behavior still requires substantial data, efficient environment simulators approaching real-world complexity are growing in prevalence. Even so, hand-designing sufficiently diverse and numerous simulated training tasks for these complex domains is prohibitively labor-intensive. Domain randomization (DR) and procedural generation (PG), offered as solutions to this problem, require simulators to possess carefully-defined parameters which directly translate to meaningful task diversity--a similarly prohibitive assumption. In this work, we present DIVA, an evolutionary approach for generating diverse training tasks in such complex, open-ended simulators. Like unsupervised environment design (UED) methods, DIVA can be applied to arbitrary parameterizations, but can additionally incorporate realistically-available domain knowledge--thus inheriting the flexibility and generality of UED, and the supervised structure embedded in well-designed simulators exploited by DR and PG. Our empirical results showcase DIVA's unique ability to overcome complex parameterizations and successfully train adaptive agent behavior, far outperforming competitive baselines from prior literature. These findings highlight the potential of such semi-supervised environment design (SSED) approaches, of which DIVA is the first humble constituent, to enable training in realistic simulated domains, and produce more robust and capable adaptive agents.

Title: LLM-R: A Framework for Domain-Adaptive Maintenance Scheme Generation Combining Hierarchical Agents and RAG

• Authors: Laifa Tao, Qixuan Huang, Xianjun Wu, Weiwei Zhang, Yunlong Wu, Bin Li, Chen Lu, Xingshuo Hai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04476
• Pdf URL: https://arxiv.org/pdf/2411.04476
• Copy Paste: [[2411.04476]] LLM-R: A Framework for Domain-Adaptive Maintenance Scheme Generation Combining Hierarchical Agents and RAG(https://arxiv.org/abs/2411.04476)
• Keywords: large language model
• Abstract: The increasing use of smart devices has emphasized the critical role of maintenance in production activities. Interactive Electronic Technical Manuals (IETMs) are vital tools that support the maintenance of smart equipment. However, traditional IETMs face challenges such as transitioning from Graphical User Interfaces (GUIs) to natural Language User Interfaces (LUIs) and managing complex logical relationships. Additionally, they must meet the current demands for higher intelligence. This paper proposes a Maintenance Scheme Generation Method based on Large Language Models (LLM-R). The proposed method includes several key innovations: We propose the Low Rank Adaptation-Knowledge Retention (LORA-KR) loss technology to proportionally adjust mixed maintenance data for fine-tuning the LLM. This method prevents knowledge conflicts caused by mixed data, improving the model's adaptability and reasoning ability in specific maintenance domains, Besides, Hierarchical Task-Based Agent and Instruction-level Retrieval-Augmented Generation (RAG) technologies are adopted to optimize the generation steps and mitigate the phenomenon of hallucination caused by the model's Inability to access contextual information. This enhancement improves the model's flexibility and accuracy in handling known or unknown maintenance objects and maintenance scheme scenarios. To validate the proposed method's effectiveness in maintenance tasks, a maintenance scheme dataset was constructed using objects from different fields. The experimental results show that the accuracy of the maintenance schemes generated by the proposed method reached 91.59%, indicating which improvement enhances the intelligence of maintenance schemes and introduces novel technical approaches for equipment maintenance.

Title: Series-to-Series Diffusion Bridge Model

• Authors: Hao Yang, Zhanbo Feng, Feng Zhou, Robert C Qiu, Zenan Ling
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04491
• Pdf URL: https://arxiv.org/pdf/2411.04491
• Copy Paste: [[2411.04491]] Series-to-Series Diffusion Bridge Model(https://arxiv.org/abs/2411.04491)
• Keywords: robust, diffusion
• Abstract: Diffusion models have risen to prominence in time series forecasting, showcasing their robust capability to model complex data distributions. However, their effectiveness in deterministic predictions is often constrained by instability arising from their inherent stochasticity. In this paper, we revisit time series diffusion models and present a comprehensive framework that encompasses most existing diffusion-based methods. Building on this theoretical foundation, we propose a novel diffusion-based time series forecasting model, the Series-to-Series Diffusion Bridge Model ($\mathrm{S^2DBM}$), which leverages the Brownian Bridge process to reduce randomness in reverse estimations and improves accuracy by incorporating informative priors and conditions derived from historical time series data. Experimental results demonstrate that $\mathrm{S^2DBM}$ delivers superior performance in point-to-point forecasting and competes effectively with other diffusion-based models in probabilistic forecasting.

Title: Synergy-Guided Regional Supervision of Pseudo Labels for Semi-Supervised Medical Image Segmentation

• Authors: Tao Wang, Xinlin Zhang, Yuanbin Chen, Yuanbo Zhou, Longxuan Zhao, Tao Tan, Tong Tong
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04493
• Pdf URL: https://arxiv.org/pdf/2411.04493
• Copy Paste: [[2411.04493]] Synergy-Guided Regional Supervision of Pseudo Labels for Semi-Supervised Medical Image Segmentation(https://arxiv.org/abs/2411.04493)
• Keywords: robust, segmentation
• Abstract: Semi-supervised learning has received considerable attention for its potential to leverage abundant unlabeled data to enhance model robustness. Pseudo labeling is a widely used strategy in semi supervised learning. However, existing methods often suffer from noise contamination, which can undermine model performance. To tackle this challenge, we introduce a novel Synergy-Guided Regional Supervision of Pseudo Labels (SGRS-Net) framework. Built upon the mean teacher network, we employ a Mix Augmentation module to enhance the unlabeled data. By evaluating the synergy before and after augmentation, we strategically partition the pseudo labels into distinct regions. Additionally, we introduce a Region Loss Evaluation module to assess the loss across each delineated area. Extensive experiments conducted on the LA dataset have demonstrated superior performance over state-of-the-art techniques, underscoring the efficiency and practicality of our framework.

Title: Thanos: Enhancing Conversational Agents with Skill-of-Mind-Infused Large Language Model

• Authors: Young-Jun Lee, Dokyong Lee, Junyoung Youn, Kyeongjin Oh, Ho-Jin Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04496
• Pdf URL: https://arxiv.org/pdf/2411.04496
• Copy Paste: [[2411.04496]] Thanos: Enhancing Conversational Agents with Skill-of-Mind-Infused Large Language Model(https://arxiv.org/abs/2411.04496)
• Keywords: large language model
• Abstract: To increase social bonding with interlocutors, humans naturally acquire the ability to respond appropriately in a given situation by considering which conversational skill is most suitable for the response - a process we call skill-of-mind. For large language model (LLM)-based conversational agents, planning appropriate conversational skills, as humans do, is challenging due to the complexity of social dialogue, especially in interactive scenarios. To address this, we propose a skill-of-mind-annotated conversation dataset, named Multifaceted Skill-of-Mind, which includes multi-turn and multifaceted conversational skills across various interactive scenarios (e.g., long-term, counseling, task-oriented), grounded in diverse social contexts (e.g., demographics, persona, rules of thumb). This dataset consists of roughly 100K conversations. Using this dataset, we introduce a new family of skill-of-mind-infused LLMs, named Thanos, with model sizes of 1B, 3B, and 8B parameters. With extensive experiments, these models successfully demonstrate the skill-of-mind process and exhibit strong generalizability in inferring multifaceted skills across a variety of domains. Moreover, we show that Thanos significantly enhances the quality of responses generated by LLM-based conversational agents and promotes prosocial behavior in human evaluations.

Title: Pose2Trajectory: Using Transformers on Body Pose to Predict Tennis Player's Trajectory

• Authors: Ali K. AlShami, Terrance Boult, Jugal Kalita
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04501
• Pdf URL: https://arxiv.org/pdf/2411.04501
• Copy Paste: [[2411.04501]] Pose2Trajectory: Using Transformers on Body Pose to Predict Tennis Player's Trajectory(https://arxiv.org/abs/2411.04501)
• Keywords: transformer
• Abstract: Tracking the trajectory of tennis players can help camera operators in production. Predicting future movement enables cameras to automatically track and predict a player's future trajectory without human intervention. Predicting future human movement in the context of complex physical tasks is also intellectually satisfying. Swift advancements in sports analytics and the wide availability of videos for tennis have inspired us to propose a novel method called Pose2Trajectory, which predicts a tennis player's future trajectory as a sequence derived from their body joints' data and ball position. Demonstrating impressive accuracy, our approach capitalizes on body joint information to provide a comprehensive understanding of the human body's geometry and motion, thereby enhancing the prediction of the player's trajectory. We use encoder-decoder Transformer architecture trained on the joints and trajectory information of the players with ball positions. The predicted sequence can provide information to help close-up cameras to keep tracking the tennis player, following centroid coordinates. We generate a high-quality dataset from multiple videos to assist tennis player movement prediction using object detection and human pose estimation methods. It contains bounding boxes and joint information for tennis players and ball positions in singles tennis games. Our method shows promising results in predicting the tennis player's movement trajectory with different sequence prediction lengths using the joints and trajectory information with the ball position.

Title: FedDP: Privacy-preserving method based on federated learning for histopathology image segmentation

• Authors: Liangrui Pan, Mao Huang, Lian Wang, Pinle Qin, Shaoliang Peng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04509
• Pdf URL: https://arxiv.org/pdf/2411.04509
• Copy Paste: [[2411.04509]] FedDP: Privacy-preserving method based on federated learning for histopathology image segmentation(https://arxiv.org/abs/2411.04509)
• Keywords: privacy, protect, attack, federate, transformer, segmentation
• Abstract: Hematoxylin and Eosin (H&E) staining of whole slide images (WSIs) is considered the gold standard for pathologists and medical practitioners for tumor diagnosis, surgical planning, and post-operative assessment. With the rapid advancement of deep learning technologies, the development of numerous models based on convolutional neural networks and transformer-based models has been applied to the precise segmentation of WSIs. However, due to privacy regulations and the need to protect patient confidentiality, centralized storage and processing of image data are impractical. Training a centralized model directly is challenging to implement in medical settings due to these privacy this http URL paper addresses the dispersed nature and privacy sensitivity of medical image data by employing a federated learning framework, allowing medical institutions to collaboratively learn while protecting patient privacy. Additionally, to address the issue of original data reconstruction through gradient inversion during the federated learning training process, differential privacy introduces noise into the model updates, preventing attackers from inferring the contributions of individual samples, thereby protecting the privacy of the training this http URL results show that the proposed method, FedDP, minimally impacts model accuracy while effectively safeguarding the privacy of cancer pathology image data, with only a slight decrease in Dice, Jaccard, and Acc indices by 0.55%, 0.63%, and 0.42%, respectively. This approach facilitates cross-institutional collaboration and knowledge sharing while protecting sensitive data privacy, providing a viable solution for further research and application in the medical field.

Title: Normalized Space Alignment: A Versatile Metric for Representation Analysis

• Authors: Danish Ebadulla, Aditya Gulati, Ambuj Singh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04512
• Pdf URL: https://arxiv.org/pdf/2411.04512
• Copy Paste: [[2411.04512]] Normalized Space Alignment: A Versatile Metric for Representation Analysis(https://arxiv.org/abs/2411.04512)
• Keywords: robust
• Abstract: We introduce a manifold analysis technique for neural network representations. Normalized Space Alignment (NSA) compares pairwise distances between two point clouds derived from the same source and having the same size, while potentially possessing differing dimensionalities. NSA can act as both an analytical tool and a differentiable loss function, providing a robust means of comparing and aligning representations across different layers and models. It satisfies the criteria necessary for both a similarity metric and a neural network loss function. We showcase NSA's versatility by illustrating its utility as a representation space analysis metric, a structure-preserving loss function, and a robustness analysis tool. NSA is not only computationally efficient but it can also approximate the global structural discrepancy during mini-batching, facilitating its use in a wide variety of neural network training paradigms.

Title: l0-Regularized Sparse Coding-based Interpretable Network for Multi-Modal Image Fusion

• Authors: Gargi Panda, Soumitra Kundu, Saumik Bhattacharya, Aurobinda Routray
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04519
• Pdf URL: https://arxiv.org/pdf/2411.04519
• Copy Paste: [[2411.04519]] l0-Regularized Sparse Coding-based Interpretable Network for Multi-Modal Image Fusion(https://arxiv.org/abs/2411.04519)
• Keywords: interpretability
• Abstract: Multi-modal image fusion (MMIF) enhances the information content of the fused image by combining the unique as well as common features obtained from different modality sensor images, improving visualization, object detection, and many more tasks. In this work, we introduce an interpretable network for the MMIF task, named FNet, based on an l0-regularized multi-modal convolutional sparse coding (MCSC) model. Specifically, for solving the l0-regularized CSC problem, we develop an algorithm unrolling-based l0-regularized sparse coding (LZSC) block. Given different modality source images, FNet first separates the unique and common features from them using the LZSC block and then these features are combined to generate the final fused image. Additionally, we propose an l0-regularized MCSC model for the inverse fusion process. Based on this model, we introduce an interpretable inverse fusion network named IFNet, which is utilized during FNet's training. Extensive experiments show that FNet achieves high-quality fusion results across five different MMIF tasks. Furthermore, we show that FNet enhances downstream object detection in visible-thermal image pairs. We have also visualized the intermediate results of FNet, which demonstrates the good interpretability of our network.

Title: Tomato, Tomahto, Tomate: Measuring the Role of Shared Semantics among Subwords in Multilingual Language Models

• Authors: Xinyu Zhang, Jing Lu, Vinh Q. Tran, Tal Schuster, Donald Metzler, Jimmy Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04530
• Pdf URL: https://arxiv.org/pdf/2411.04530
• Copy Paste: [[2411.04530]] Tomato, Tomahto, Tomate: Measuring the Role of Shared Semantics among Subwords in Multilingual Language Models(https://arxiv.org/abs/2411.04530)
• Keywords: robust
• Abstract: Human understanding of language is robust to different word choices as far as they represent similar semantic concepts. To what extent does our human intuition transfer to language models, which represent all subwords as distinct embeddings? In this work, we take an initial step on measuring the role of shared semantics among subwords in the encoder-only multilingual language models (mLMs). To this end, we form "semantic tokens" by merging the semantically similar subwords and their embeddings, and evaluate the updated mLMs on 5 heterogeneous multilingual downstream tasks. Results show that the general shared semantics could get the models a long way in making the predictions on mLMs with different tokenizers and model sizes. Inspections on the grouped subwords show that they exhibit a wide range of semantic similarities, including synonyms and translations across many languages and scripts. Lastly, we found the zero-shot results with semantic tokens are on par or even better than the original models on certain classification tasks, suggesting that the shared subword-level semantics may serve as the anchors for cross-lingual transferring.

Title: Neural Fingerprints for Adversarial Attack Detection

• Authors: Haim Fisher, Moni Shahar, Yehezkel S. Resheff
• Subjects: cs.CV, cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2411.04533
• Pdf URL: https://arxiv.org/pdf/2411.04533
• Copy Paste: [[2411.04533]] Neural Fingerprints for Adversarial Attack Detection(https://arxiv.org/abs/2411.04533)
• Keywords: security, attack, robust
• Abstract: Deep learning models for image classification have become standard tools in recent years. A well known vulnerability of these models is their susceptibility to adversarial examples. These are generated by slightly altering an image of a certain class in a way that is imperceptible to humans but causes the model to classify it wrongly as another class. Many algorithms have been proposed to address this problem, falling generally into one of two categories: (i) building robust classifiers (ii) directly detecting attacked images. Despite the good performance of these detectors, we argue that in a white-box setting, where the attacker knows the configuration and weights of the network and the detector, they can overcome the detector by running many examples on a local copy, and sending only those that were not detected to the actual model. This problem is common in security applications where even a very good model is not sufficient to ensure safety. In this paper we propose to overcome this inherent limitation of any static defence with randomization. To do so, one must generate a very large family of detectors with consistent performance, and select one or more of them randomly for each input. For the individual detectors, we suggest the method of neural fingerprints. In the training phase, for each class we repeatedly sample a tiny random subset of neurons from certain layers of the network, and if their average is sufficiently different between clean and attacked images of the focal class they are considered a fingerprint and added to the detector bank. During test time, we sample fingerprints from the bank associated with the label predicted by the model, and detect attacks using a likelihood ratio test. We evaluate our detectors on ImageNet with different attack methods and model architectures, and show near-perfect detection with low rates of false detection.

Title: Hypercube Policy Regularization Framework for Offline Reinforcement Learning

• Authors: Yi Shen, Hanyan Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04534
• Pdf URL: https://arxiv.org/pdf/2411.04534
• Copy Paste: [[2411.04534]] Hypercube Policy Regularization Framework for Offline Reinforcement Learning(https://arxiv.org/abs/2411.04534)
• Keywords: diffusion
• Abstract: Offline reinforcement learning has received extensive attention from scholars because it avoids the interaction between the agent and the environment by learning a policy through a static dataset. However, general reinforcement learning methods cannot get satisfactory results in offline reinforcement learning due to the out-of-distribution state actions that the dataset cannot cover during training. To solve this problem, the policy regularization method that tries to directly clone policies used in static datasets has received numerous studies due to its simplicity and effectiveness. However, policy constraint methods make the agent choose the corresponding actions in the static dataset. This type of constraint is usually over-conservative, which results in suboptimal policies, especially in low-quality static datasets. In this paper, a hypercube policy regularization framework is proposed, this method alleviates the constraints of policy constraint methods by allowing the agent to explore the actions corresponding to similar states in the static dataset, which increases the effectiveness of algorithms in low-quality datasets. It was also theoretically demonstrated that the hypercube policy regularization framework can effectively improve the performance of original algorithms. In addition, the hypercube policy regularization framework is combined with TD3-BC and Diffusion-QL for experiments on D4RL datasets which are called TD3-BC-C and Diffusion-QL-C. The experimental results of the score demonstrate that TD3-BC-C and Diffusion-QL-C perform better than state-of-the-art algorithms like IQL, CQL, TD3-BC and Diffusion-QL in most D4RL environments in approximate time.

Title: Meta-Reasoning Improves Tool Use in Large Language Models

• Authors: Lisa Alazraki, Marek Rei
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04535
• Pdf URL: https://arxiv.org/pdf/2411.04535
• Copy Paste: [[2411.04535]] Meta-Reasoning Improves Tool Use in Large Language Models(https://arxiv.org/abs/2411.04535)
• Keywords: large language model
• Abstract: External tools help large language models (LLMs) succeed at tasks where they would otherwise typically fail. In existing frameworks, LLMs learn tool use either by in-context demonstrations or via full model fine-tuning on annotated data. As these approaches do not easily scale, a recent trend is to abandon them in favor of lightweight, parameter-efficient tuning paradigms. These methods allow quickly alternating between the frozen LLM and its specialised fine-tuned version, by switching on or off a handful of additional custom parameters. Hence, we postulate that the generalization ability of the frozen model can be leveraged to improve tool selection. We present Tool selECTion via meta-reasONing (TECTON), a two-phase system that first reasons over a task using a custom fine-tuned LM head and outputs candidate tools. Then, with the custom head disabled, it meta-reasons (i.e., it reasons over the previous reasoning process) to make a final choice. We show that TECTON results in substantial gains - both in-distribution and out-of-distribution - on a range of math reasoning datasets.

Title: Peri-midFormer: Periodic Pyramid Transformer for Time Series Analysis

• Authors: Qiang Wu, Gechang Yao, Zhixi Feng, Shuyuan Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04554
• Pdf URL: https://arxiv.org/pdf/2411.04554
• Copy Paste: [[2411.04554]] Peri-midFormer: Periodic Pyramid Transformer for Time Series Analysis(https://arxiv.org/abs/2411.04554)
• Keywords: transformer
• Abstract: Time series analysis finds wide applications in fields such as weather forecasting, anomaly detection, and behavior recognition. Previous methods attempted to model temporal variations directly using 1D time series. However, this has been quite challenging due to the discrete nature of data points in time series and the complexity of periodic variation. In terms of periodicity, taking weather and traffic data as an example, there are multi-periodic variations such as yearly, monthly, weekly, and daily, etc. In order to break through the limitations of the previous methods, we decouple the implied complex periodic variations into inclusion and overlap relationships among different level periodic components based on the observation of the multi-periodicity therein and its inclusion relationships. This explicitly represents the naturally occurring pyramid-like properties in time series, where the top level is the original time series and lower levels consist of periodic components with gradually shorter periods, which we call the periodic pyramid. To further extract complex temporal variations, we introduce self-attention mechanism into the periodic pyramid, capturing complex periodic relationships by computing attention between periodic components based on their inclusion, overlap, and adjacency relationships. Our proposed Peri-midFormer demonstrates outstanding performance in five mainstream time series analysis tasks, including short- and long-term forecasting, imputation, classification, and anomaly detection.

Title: Pruning Literals for Highly Efficient Explainability at Word Level

• Authors: Rohan Kumar Yadav, Bimal Bhattarai, Abhik Jana, Lei Jiao, Seid Muhie Yimam
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04557
• Pdf URL: https://arxiv.org/pdf/2411.04557
• Copy Paste: [[2411.04557]] Pruning Literals for Highly Efficient Explainability at Word Level(https://arxiv.org/abs/2411.04557)
• Keywords: explainability
• Abstract: Designing an explainable model becomes crucial now for Natural Language Processing(NLP) since most of the state-of-the-art machine learning models provide a limited explanation for the prediction. In the spectrum of an explainable model, Tsetlin Machine(TM) is promising because of its capability of providing word-level explanation using proposition logic. However, concern rises over the elaborated combination of literals (propositional logic) in the clause that makes the model difficult for humans to comprehend, despite having a transparent learning process. In this paper, we design a post-hoc pruning of clauses that eliminate the randomly placed literals in the clause thereby making the model more efficiently interpretable than the vanilla TM. Experiments on the publicly available YELP-HAT Dataset demonstrate that the proposed pruned TM's attention map aligns more with the human attention map than the vanilla TM's attention map. In addition, the pairwise similarity measure also surpasses the attention map-based neural network models. In terms of accuracy, the proposed pruning method does not degrade the accuracy significantly but rather enhances the performance up to 4% to 9% in some test data.

Title: Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning

• Authors: Marvin Alles, Philip Becker-Ehmck, Patrick van der Smagt, Maximilian Karl
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04562
• Pdf URL: https://arxiv.org/pdf/2411.04562
• Copy Paste: [[2411.04562]] Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning(https://arxiv.org/abs/2411.04562)
• Keywords: generative
• Abstract: In offline reinforcement learning, a policy is learned using a static dataset in the absence of costly feedback from the environment. In contrast to the online setting, only using static datasets poses additional challenges, such as policies generating out-of-distribution samples. Model-based offline reinforcement learning methods try to overcome these by learning a model of the underlying dynamics of the environment and using it to guide policy search. It is beneficial but, with limited datasets, errors in the model and the issue of value overestimation among out-of-distribution states can worsen performance. Current model-based methods apply some notion of conservatism to the Bellman update, often implemented using uncertainty estimation derived from model ensembles. In this paper, we propose Constrained Latent Action Policies (C-LAP) which learns a generative model of the joint distribution of observations and actions. We cast policy learning as a constrained objective to always stay within the support of the latent action distribution, and use the generative capabilities of the model to impose an implicit constraint on the generated actions. Thereby eliminating the need to use additional uncertainty penalties on the Bellman update and significantly decreasing the number of gradient steps required to learn a policy. We empirically evaluate C-LAP on the D4RL and V-D4RL benchmark, and show that C-LAP is competitive to state-of-the-art methods, especially outperforming on datasets with visual observations.

Title: Higher-Order GNNs Meet Efficiency: Sparse Sobolev Graph Neural Networks

• Authors: Jhony H. Giraldo, Aref Einizade, Andjela Todorovic, Jhon A. Castro-Correa, Mohsen Badiey, Thierry Bouwmans, Fragkiskos D. Malliaros
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2411.04570
• Pdf URL: https://arxiv.org/pdf/2411.04570
• Copy Paste: [[2411.04570]] Higher-Order GNNs Meet Efficiency: Sparse Sobolev Graph Neural Networks(https://arxiv.org/abs/2411.04570)
• Keywords: robust
• Abstract: Graph Neural Networks (GNNs) have shown great promise in modeling relationships between nodes in a graph, but capturing higher-order relationships remains a challenge for large-scale networks. Previous studies have primarily attempted to utilize the information from higher-order neighbors in the graph, involving the incorporation of powers of the shift operator, such as the graph Laplacian or adjacency matrix. This approach comes with a trade-off in terms of increased computational and memory demands. Relying on graph spectral theory, we make a fundamental observation: the regular and the Hadamard power of the Laplacian matrix behave similarly in the spectrum. This observation has significant implications for capturing higher-order information in GNNs for various tasks such as node classification and semi-supervised learning. Consequently, we propose a novel graph convolutional operator based on the sparse Sobolev norm of graph signals. Our approach, known as Sparse Sobolev GNN (S2-GNN), employs Hadamard products between matrices to maintain the sparsity level in graph representations. S2-GNN utilizes a cascade of filters with increasing Hadamard powers to generate a diverse set of functions. We theoretically analyze the stability of S2-GNN to show the robustness of the model against possible graph perturbations. We also conduct a comprehensive evaluation of S2-GNN across various graph mining, semi-supervised node classification, and computer vision tasks. In particular use cases, our algorithm demonstrates competitive performance compared to state-of-the-art GNNs in terms of performance and running time.

Title: DomainGallery: Few-shot Domain-driven Image Generation by Attribute-centric Finetuning

• Authors: Yuxuan Duan, Yan Hong, Bo Zhang, Jun Lan, Huijia Zhu, Weiqiang Wang, Jianfu Zhang, Li Niu, Liqing Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04571
• Pdf URL: https://arxiv.org/pdf/2411.04571
• Copy Paste: [[2411.04571]] DomainGallery: Few-shot Domain-driven Image Generation by Attribute-centric Finetuning(https://arxiv.org/abs/2411.04571)
• Keywords: diffusion
• Abstract: The recent progress in text-to-image models pretrained on large-scale datasets has enabled us to generate various images as long as we provide a text prompt describing what we want. Nevertheless, the availability of these models is still limited when we expect to generate images that fall into a specific domain either hard to describe or just unseen to the models. In this work, we propose DomainGallery, a few-shot domain-driven image generation method which aims at finetuning pretrained Stable Diffusion on few-shot target datasets in an attribute-centric manner. Specifically, DomainGallery features prior attribute erasure, attribute disentanglement, regularization and enhancement. These techniques are tailored to few-shot domain-driven generation in order to solve key issues that previous works have failed to settle. Extensive experiments are given to validate the superior performance of DomainGallery on a variety of domain-driven generation scenarios. Codes are available at this https URL.

Title: Towards Robust Federated Analytics via Differentially Private Measurements of Statistical Heterogeneity

• Authors: Mary Scott, Graham Cormode, Carsten Maple
• Subjects: cs.LG, cs.DB
• Abstract URL: https://arxiv.org/abs/2411.04579
• Pdf URL: https://arxiv.org/pdf/2411.04579
• Copy Paste: [[2411.04579]] Towards Robust Federated Analytics via Differentially Private Measurements of Statistical Heterogeneity(https://arxiv.org/abs/2411.04579)
• Keywords: privacy, robust, federate
• Abstract: Statistical heterogeneity is a measure of how skewed the samples of a dataset are. It is a common problem in the study of differential privacy that the usage of a statistically heterogeneous dataset results in a significant loss of accuracy. In federated scenarios, statistical heterogeneity is more likely to happen, and so the above problem is even more pressing. We explore the three most promising ways to measure statistical heterogeneity and give formulae for their accuracy, while simultaneously incorporating differential privacy. We find the optimum privacy parameters via an analytic mechanism, which incorporates root finding methods. We validate the main theorems and related hypotheses experimentally, and test the robustness of the analytic mechanism to different heterogeneity levels. The analytic mechanism in a distributed setting delivers superior accuracy to all combinations involving the classic mechanism and/or the centralized setting. All measures of statistical heterogeneity do not lose significant accuracy when a heterogeneous sample is used.

Title: On the Inherent Robustness of One-Stage Object Detection against Out-of-Distribution Data

• Authors: Aitor Martinez-Seras, Javier Del Ser, Alain Andres, Pablo Garcia-Bringas
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04586
• Pdf URL: https://arxiv.org/pdf/2411.04586
• Copy Paste: [[2411.04586]] On the Inherent Robustness of One-Stage Object Detection against Out-of-Distribution Data(https://arxiv.org/abs/2411.04586)
• Keywords: robust
• Abstract: Robustness is a fundamental aspect for developing safe and trustworthy models, particularly when they are deployed in the open world. In this work we analyze the inherent capability of one-stage object detectors to robustly operate in the presence of out-of-distribution (OoD) data. Specifically, we propose a novel detection algorithm for detecting unknown objects in image data, which leverages the features extracted by the model from each sample. Differently from other recent approaches in the literature, our proposal does not require retraining the object detector, thereby allowing for the use of pretrained models. Our proposed OoD detector exploits the application of supervised dimensionality reduction techniques to mitigate the effects of the curse of dimensionality on the features extracted by the model. Furthermore, it utilizes high-resolution feature maps to identify potential unknown objects in an unsupervised fashion. Our experiments analyze the Pareto trade-off between the performance detecting known and unknown objects resulting from different algorithmic configurations and inference confidence thresholds. We also compare the performance of our proposed algorithm to that of logits-based post-hoc OoD methods, as well as possible fusion strategies. Finally, we discuss on the competitiveness of all tested methods against state-of-the-art OoD approaches for object detection models over the recently published Unknown Object Detection benchmark. The obtained results verify that the performance of avant-garde post-hoc OoD detectors can be further improved when combined with our proposed algorithm.

Title: Verification of Neural Networks against Convolutional Perturbations via Parameterised Kernels

• Authors: Benedikt Brückner, Alessio Lomuscio
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04594
• Pdf URL: https://arxiv.org/pdf/2411.04594
• Copy Paste: [[2411.04594]] Verification of Neural Networks against Convolutional Perturbations via Parameterised Kernels(https://arxiv.org/abs/2411.04594)
• Keywords: robust
• Abstract: We develop a method for the efficient verification of neural networks against convolutional perturbations such as blurring or sharpening. To define input perturbations we use well-known camera shake, box blur and sharpen kernels. We demonstrate that these kernels can be linearly parameterised in a way that allows for a variation of the perturbation strength while preserving desired kernel properties. To facilitate their use in neural network verification, we develop an efficient way of convolving a given input with these parameterised kernels. The result of this convolution can be used to encode the perturbation in a verification setting by prepending a linear layer to a given network. This leads to tight bounds and a high effectiveness in the resulting verification step. We add further precision by employing input splitting as a branch and bound strategy. We demonstrate that we are able to verify robustness on a number of standard benchmarks where the baseline is unable to provide any safety certificates. To the best of our knowledge, this is the first solution for verifying robustness against specific convolutional perturbations such as camera shake.

Title: Social EgoMesh Estimation

• Authors: Luca Scofano, Alessio Sampieri, Edoardo De Matteis, Indro Spinelli, Fabio Galasso
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04598
• Pdf URL: https://arxiv.org/pdf/2411.04598
• Copy Paste: [[2411.04598]] Social EgoMesh Estimation(https://arxiv.org/abs/2411.04598)
• Keywords: diffusion
• Abstract: Accurately estimating the 3D pose of the camera wearer in egocentric video sequences is crucial to modeling human behavior in virtual and augmented reality applications. The task presents unique challenges due to the limited visibility of the user's body caused by the front-facing camera mounted on their head. Recent research has explored the utilization of the scene and ego-motion, but it has overlooked humans' interactive nature. We propose a novel framework for Social Egocentric Estimation of body MEshes (SEE-ME). Our approach is the first to estimate the wearer's mesh using only a latent probabilistic diffusion model, which we condition on the scene and, for the first time, on the social wearer-interactee interactions. Our in-depth study sheds light on when social interaction matters most for ego-mesh estimation; it quantifies the impact of interpersonal distance and gaze direction. Overall, SEE-ME surpasses the current best technique, reducing the pose estimation error (MPJPE) by 53%. The code is available at this https URL.

Title: Cross- and Intra-image Prototypical Learning for Multi-label Disease Diagnosis and Interpretation

• Authors: Chong Wang, Fengbei Liu, Yuanhong Chen, Helen Frazer, Gustavo Carneiro
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04607
• Pdf URL: https://arxiv.org/pdf/2411.04607
• Copy Paste: [[2411.04607]] Cross- and Intra-image Prototypical Learning for Multi-label Disease Diagnosis and Interpretation(https://arxiv.org/abs/2411.04607)
• Keywords: robust, interpretability
• Abstract: Recent advances in prototypical learning have shown remarkable potential to provide useful decision interpretations associating activation maps and predictions with class-specific training prototypes. Such prototypical learning has been well-studied for various single-label diseases, but for quite relevant and more challenging multi-label diagnosis, where multiple diseases are often concurrent within an image, existing prototypical learning models struggle to obtain meaningful activation maps and effective class prototypes due to the entanglement of the multiple diseases. In this paper, we present a novel Cross- and Intra-image Prototypical Learning (CIPL) framework, for accurate multi-label disease diagnosis and interpretation from medical images. CIPL takes advantage of common cross-image semantics to disentangle the multiple diseases when learning the prototypes, allowing a comprehensive understanding of complicated pathological lesions. Furthermore, we propose a new two-level alignment-based regularisation strategy that effectively leverages consistent intra-image information to enhance interpretation robustness and predictive performance. Extensive experiments show that our CIPL attains the state-of-the-art (SOTA) classification accuracy in two public multi-label benchmarks of disease diagnosis: thoracic radiography and fundus images. Quantitative interpretability results show that CIPL also has superiority in weakly-supervised thoracic disease localisation over other leading saliency- and prototype-based explanation methods.

Title: Solar potential analysis over Indian cities using high-resolution satellite imagery and DEM

• Authors: Jai Singla
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04610
• Pdf URL: https://arxiv.org/pdf/2411.04610
• Copy Paste: [[2411.04610]] Solar potential analysis over Indian cities using high-resolution satellite imagery and DEM(https://arxiv.org/abs/2411.04610)
• Keywords: diffusion
• Abstract: Most of the research work in the solar potential analysis is performed utilizing aerial imagery, LiDAR data, and satellite imagery. However, in the existing studies using satellite data, parameters such as trees/ vegetation shadow, adjacent higher architectural structures, and eccentric roof structures in urban areas were not considered, and relatively coarser-resolution datasets were used for analysis. In this work, we have implemented a novel approach to estimate rooftop solar potential using inputs of high-resolution satellite imagery (0.5 cm), a digital elevation model (1m), along with ground station radiation data. Solar radiation analysis is performed using the diffusion proportion and transmissivity ratio derived from the ground station data hosted by IMD. It was observed that due to seasonal variations, environmental effects and technical reasons such as solar panel structure etc., there can be a significant loss of electricity generation up to 50%. Based on the results, it is also understood that using 1m DEM and 50cm satellite imagery, more authentic results are produced over the urban areas.

Title: Multi-temporal crack segmentation in concrete structure using deep learning approaches

• Authors: Said Harb, Pedro Achanccaray, Mehdi Maboudi, Markus Gerke
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.04620
• Pdf URL: https://arxiv.org/pdf/2411.04620
• Copy Paste: [[2411.04620]] Multi-temporal crack segmentation in concrete structure using deep learning approaches(https://arxiv.org/abs/2411.04620)
• Keywords: segmentation
• Abstract: Cracks are among the earliest indicators of deterioration in concrete structures. Early automatic detection of these cracks can significantly extend the lifespan of critical infrastructures, such as bridges, buildings, and tunnels, while simultaneously reducing maintenance costs and facilitating efficient structural health monitoring. This study investigates whether leveraging multi-temporal data for crack segmentation can enhance segmentation quality. Therefore, we compare a Swin UNETR trained on multi-temporal data with a U-Net trained on mono-temporal data to assess the effect of temporal information compared with conventional single-epoch approaches. To this end, a multi-temporal dataset comprising 1356 images, each with 32 sequential crack propagation images, was created. After training the models, experiments were conducted to analyze their generalization ability, temporal consistency, and segmentation quality. The multi-temporal approach consistently outperformed its mono-temporal counterpart, achieving an IoU of $82.72\%$ and a F1-score of $90.54\%$, representing a significant improvement over the mono-temporal model's IoU of $76.69\%$ and F1-score of $86.18\%$, despite requiring only half of the trainable parameters. The multi-temporal model also displayed a more consistent segmentation quality, with reduced noise and fewer errors. These results suggest that temporal information significantly enhances the performance of segmentation models, offering a promising solution for improved crack detection and the long-term monitoring of concrete structures, even with limited sequential data.

Title: Brain Tumour Removing and Missing Modality Generation using 3D WDM

• Authors: André Ferreira, Gijs Luijten, Behrus Puladi, Jens Kleesiek, Victor Alves, Jan Egger
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04630
• Pdf URL: https://arxiv.org/pdf/2411.04630
• Copy Paste: [[2411.04630]] Brain Tumour Removing and Missing Modality Generation using 3D WDM(https://arxiv.org/abs/2411.04630)
• Keywords: diffusion
• Abstract: This paper presents the second-placed solution for task 8 and the participation solution for task 7 of BraTS 2024. The adoption of automated brain analysis algorithms to support clinical practice is increasing. However, many of these algorithms struggle with the presence of brain lesions or the absence of certain MRI modalities. The alterations in the brain's morphology leads to high variability and thus poor performance of predictive models that were trained only on healthy brains. The lack of information that is usually provided by some of the missing MRI modalities also reduces the reliability of the prediction models trained with all modalities. In order to improve the performance of these models, we propose the use of conditional 3D wavelet diffusion models. The wavelet transform enabled full-resolution image training and prediction on a GPU with 48 GB VRAM, without patching or downsampling, preserving all information for prediction. For the inpainting task of BraTS 2024, the use of a large and variable number of healthy masks and the stability and efficiency of the 3D wavelet diffusion model resulted in 0.007, 22.61 and 0.842 in the validation set and 0.07 , 22.8 and 0.91 in the testing set (MSE, PSNR and SSIM respectively). The code for these tasks is available at this https URL.

Title: Improved Multi-Task Brain Tumour Segmentation with Synthetic Data Augmentation

• Authors: André Ferreira, Tiago Jesus, Behrus Puladi, Jens Kleesiek, Victor Alves, Jan Egger
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04632
• Pdf URL: https://arxiv.org/pdf/2411.04632
• Copy Paste: [[2411.04632]] Improved Multi-Task Brain Tumour Segmentation with Synthetic Data Augmentation(https://arxiv.org/abs/2411.04632)
• Keywords: robust, segmentation
• Abstract: This paper presents the winning solution of task 1 and the third-placed solution of task 3 of the BraTS challenge. The use of automated tools in clinical practice has increased due to the development of more and more sophisticated and reliable algorithms. However, achieving clinical standards and developing tools for real-life scenarios is a major challenge. To this end, BraTS has organised tasks to find the most advanced solutions for specific purposes. In this paper, we propose the use of synthetic data to train state-of-the-art frameworks in order to improve the segmentation of adult gliomas in a post-treatment scenario, and the segmentation of meningioma for radiotherapy planning. Our results suggest that the use of synthetic data leads to more robust algorithms, although the synthetic data generation pipeline is not directly suited to the meningioma task. The code for these tasks is available at this https URL.

Title: Cybercrime Prediction via Geographically Weighted Learning

• Authors: Muhammad Al-Zafar Khan, Jamal Al-Karaki, Emad Mahafzah
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04635
• Pdf URL: https://arxiv.org/pdf/2411.04635
• Copy Paste: [[2411.04635]] Cybercrime Prediction via Geographically Weighted Learning(https://arxiv.org/abs/2411.04635)
• Keywords: security
• Abstract: Inspired by the success of Geographically Weighted Regression and its accounting for spatial variations, we propose GeogGNN -- A graph neural network model that accounts for geographical latitude and longitudinal points. Using a synthetically generated dataset, we apply the algorithm for a 4-class classification problem in cybersecurity with seemingly realistic geographic coordinates centered in the Gulf Cooperation Council region. We demonstrate that it has higher accuracy than standard neural networks and convolutional neural networks that treat the coordinates as features. Encouraged by the speed-up in model accuracy by the GeogGNN model, we provide a general mathematical result that demonstrates that a geometrically weighted neural network will, in principle, always display higher accuracy in the classification of spatially dependent data by making use of spatial continuity and local averaging features.

Title: TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models

• Authors: Jonathan Fhima, Elad Ben Avraham, Oren Nuriel, Yair Kittenplon, Roy Ganz, Aviad Aberdam, Ron Litman
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04642
• Pdf URL: https://arxiv.org/pdf/2411.04642
• Copy Paste: [[2411.04642]] TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models(https://arxiv.org/abs/2411.04642)
• Keywords: transformer
• Abstract: Vision-Language (VL) models have garnered considerable research interest; however, they still face challenges in effectively handling text within images. To address this limitation, researchers have developed two approaches. The first method involves utilizing external Optical Character Recognition (OCR) tools to extract textual information from images, which is then prepended to other textual inputs. The second strategy focuses on employing extremely high-resolution images to improve text recognition capabilities. In this paper, we focus on enhancing the first strategy by introducing a novel method, named TAP-VL, which treats OCR information as a distinct modality and seamlessly integrates it into any VL model. TAP-VL employs a lightweight transformer-based OCR module to receive OCR with layout information, compressing it into a short fixed-length sequence for input into the LLM. Initially, we conduct model-agnostic pretraining of the OCR module on unlabeled documents, followed by its integration into any VL architecture through brief fine-tuning. Extensive experiments demonstrate consistent performance improvements when applying TAP-VL to top-performing VL models, across scene-text and document-based VL benchmarks.

Title: DanceFusion: A Spatio-Temporal Skeleton Diffusion Transformer for Audio-Driven Dance Motion Reconstruction

• Authors: Li Zhao, Zhengmin Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04646
• Pdf URL: https://arxiv.org/pdf/2411.04646
• Copy Paste: [[2411.04646]] DanceFusion: A Spatio-Temporal Skeleton Diffusion Transformer for Audio-Driven Dance Motion Reconstruction(https://arxiv.org/abs/2411.04646)
• Keywords: diffusion, transformer
• Abstract: This paper introduces DanceFusion, a novel framework for reconstructing and generating dance movements synchronized to music, utilizing a Spatio-Temporal Skeleton Diffusion Transformer. The framework adeptly handles incomplete and noisy skeletal data common in short-form dance videos on social media platforms like TikTok. DanceFusion incorporates a hierarchical Transformer-based Variational Autoencoder (VAE) integrated with a diffusion model, significantly enhancing motion realism and accuracy. Our approach introduces sophisticated masking techniques and a unique iterative diffusion process that refines the motion sequences, ensuring high fidelity in both motion generation and synchronization with accompanying audio cues. Comprehensive evaluations demonstrate that DanceFusion surpasses existing methods, providing state-of-the-art performance in generating dynamic, realistic, and stylistically diverse dance motions. Potential applications of this framework extend to content creation, virtual reality, and interactive entertainment, promising substantial advancements in automated dance generation. Visit our project page at this https URL.

Title: ICH-SCNet: Intracerebral Hemorrhage Segmentation and Prognosis Classification Network Using CLIP-guided SAM mechanism

• Authors: Xinlei Yu, Ahmed Elazab, Ruiquan Ge, Hui Jin, Xinchen Jiang, Gangyong Jia, Qing Wu, Qinglei Shi, Changmiao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04656
• Pdf URL: https://arxiv.org/pdf/2411.04656
• Copy Paste: [[2411.04656]] ICH-SCNet: Intracerebral Hemorrhage Segmentation and Prognosis Classification Network Using CLIP-guided SAM mechanism(https://arxiv.org/abs/2411.04656)
• Keywords: segmentation
• Abstract: Intracerebral hemorrhage (ICH) is the most fatal subtype of stroke and is characterized by a high incidence of disability. Accurate segmentation of the ICH region and prognosis prediction are critically important for developing and refining treatment plans for post-ICH patients. However, existing approaches address these two tasks independently and predominantly focus on imaging data alone, thereby neglecting the intrinsic correlation between the tasks and modalities. This paper introduces a multi-task network, ICH-SCNet, designed for both ICH segmentation and prognosis classification. Specifically, we integrate a SAM-CLIP cross-modal interaction mechanism that combines medical text and segmentation auxiliary information with neuroimaging data to enhance cross-modal feature recognition. Additionally, we develop an effective feature fusion module and a multi-task loss function to improve performance further. Extensive experiments on an ICH dataset reveal that our approach surpasses other state-of-the-art methods. It excels in the overall performance of classification tasks and outperforms competing models in all segmentation task metrics.

Title: EarCapAuth: Biometric Method for Earables Using Capacitive Sensing Eartips

• Authors: Richard Hanser, Tobias Röddiger, Till Riedel, Michael Beigl
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.04657
• Pdf URL: https://arxiv.org/pdf/2411.04657
• Copy Paste: [[2411.04657]] EarCapAuth: Biometric Method for Earables Using Capacitive Sensing Eartips(https://arxiv.org/abs/2411.04657)
• Keywords: security, biometric
• Abstract: Earphones can give access to sensitive information via voice assistants which demands security methods that prevent unauthorized use. Therefore, we developed EarCapAuth, an authentication mechanism using 48 capacitive electrodes embedded into the soft silicone eartips of two earables. For evaluation, we gathered capactive ear canal measurements from 20 participants in 20 wearing sessions (12 at rest, 8 while walking). A per user classifier trained for authentication achieves an EER of 7.62% and can be tuned to a FAR (False Acceptance Rate) of 1% at FRR (False Rejection Rate) of 16.14%. For identification, EarCapAuth achieves 89.95%. This outperforms some earable biometric principles from related work. Performance under motion slightly decreased to 9.76% EER for authentication and 86.40% accuracy for identification. Enrollment can be performed rapidly with multiple short earpiece insertions and a biometric decision is made every 0.33s. In the future, EarCapAuth could be integrated into high-resolution brain sensing electrode tips.

Title: Automated Image Color Mapping for a Historic Photographic Collection

• Authors: Taylor Arnold, Lauren Tilton
• Subjects: cs.CV, stat.AP
• Abstract URL: https://arxiv.org/abs/2411.04659
• Pdf URL: https://arxiv.org/pdf/2411.04659
• Copy Paste: [[2411.04659]] Automated Image Color Mapping for a Historic Photographic Collection(https://arxiv.org/abs/2411.04659)
• Keywords: protect
• Abstract: In the 1970s, the United States Environmental Protection Agency sponsored Documerica, a large-scale photography initiative to document environmental subjects nation-wide. While over 15,000 digitized public-domain photographs from the collection are available online, most of the images were scanned from damaged copies of the original prints. We present and evaluate a modified histogram matching technique based on the underlying chemistry of the prints for correcting the damaged images by using training data collected from a small set of undamaged prints. The entire set of color-adjusted Documerica images is made available in an open repository.

Title: Explainable Search and Discovery of Visual Cultural Heritage Collections with Multimodal Large Language Models

• Authors: Taylor Arnold, Lauren Tilton
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04663
• Pdf URL: https://arxiv.org/pdf/2411.04663
• Copy Paste: [[2411.04663]] Explainable Search and Discovery of Visual Cultural Heritage Collections with Multimodal Large Language Models(https://arxiv.org/abs/2411.04663)
• Keywords: privacy, large language model
• Abstract: Many cultural institutions have made large digitized visual collections available online, often under permissible re-use licences. Creating interfaces for exploring and searching these collections is difficult, particularly in the absence of granular metadata. In this paper, we introduce a method for using state-of-the-art multimodal large language models (LLMs) to enable an open-ended, explainable search and discovery interface for visual collections. We show how our approach can create novel clustering and recommendation systems that avoid common pitfalls of methods based directly on visual embeddings. Of particular interest is the ability to offer concrete textual explanations of each recommendation without the need to preselect the features of interest. Together, these features can create a digital interface that is more open-ended and flexible while also being better suited to addressing privacy and ethical concerns. Through a case study using a collection of documentary photographs, we provide several metrics showing the efficacy and possibilities of our approach.

Title: Differentially Private Continual Learning using Pre-Trained Models

• Authors: Marlon Tobaben, Marcus Klasson, Rui Li, Arno Solin, Antti Honkela
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.04680
• Pdf URL: https://arxiv.org/pdf/2411.04680
• Copy Paste: [[2411.04680]] Differentially Private Continual Learning using Pre-Trained Models(https://arxiv.org/abs/2411.04680)
• Keywords: privacy
• Abstract: This work explores the intersection of continual learning (CL) and differential privacy (DP). Crucially, continual learning models must retain knowledge across tasks, but this conflicts with the differential privacy requirement of restricting individual samples to be memorised in the model. We propose using pre-trained models to address the trade-offs between privacy and performance in a continual learning this http URL specifically, we present necessary assumptions to enable privacy-preservation and propose combining pre-trained models with parameter-free classifiers and parameter-efficient adapters that are learned under differential privacy. Our experiments demonstrate their effectiveness and provide insights into balancing the competing demands of continual learning and privacy.

Title: Personalized Federated Learning for Cross-view Geo-localization

• Authors: Christos Anagnostopoulos, Alexandros Gkillas, Nikos Piperigkos, Aris S. Lalos
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04692
• Pdf URL: https://arxiv.org/pdf/2411.04692
• Copy Paste: [[2411.04692]] Personalized Federated Learning for Cross-view Geo-localization(https://arxiv.org/abs/2411.04692)
• Keywords: privacy, robust, federate
• Abstract: In this paper we propose a methodology combining Federated Learning (FL) with Cross-view Image Geo-localization (CVGL) techniques. We address the challenges of data privacy and heterogeneity in autonomous vehicle environments by proposing a personalized Federated Learning scenario that allows selective sharing of model parameters. Our method implements a coarse-to-fine approach, where clients share only the coarse feature extractors while keeping fine-grained features specific to local environments. We evaluate our approach against traditional centralized and single-client training schemes using the KITTI dataset combined with satellite imagery. Results demonstrate that our federated CVGL method achieves performance close to centralized training while maintaining data privacy. The proposed partial model sharing strategy shows comparable or slightly better performance than classical FL, offering significant reduced communication overhead without sacrificing accuracy. Our work contributes to more robust and privacy-preserving localization systems for autonomous vehicles operating in diverse environments

Title: Reciprocal Point Learning Network with Large Electromagnetic Kernel for SAR Open-Set Recognition

• Authors: Xiayang Xiao, Zhuoxuan Li, Ruyi Zhang, Jiacheng Chen, Haipeng Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04693
• Pdf URL: https://arxiv.org/pdf/2411.04693
• Copy Paste: [[2411.04693]] Reciprocal Point Learning Network with Large Electromagnetic Kernel for SAR Open-Set Recognition(https://arxiv.org/abs/2411.04693)
• Keywords: robust
• Abstract: The limitations of existing Synthetic Aperture Radar (SAR) Automatic Target Recognition (ATR) methods lie in their confinement by the closed-environment assumption, hindering their effective and robust handling of unknown target categories in open environments. Open Set Recognition (OSR), a pivotal facet for algorithmic practicality, intends to categorize known classes while denoting unknown ones as "unknown." The chief challenge in OSR involves concurrently mitigating risks associated with generalizing features from a restricted set of known classes to numerous unknown samples and the open space exposure to potential unknown data. To enhance open-set SAR classification, a method called scattering kernel with reciprocal learning network is proposed. Initially, a feature learning framework is constructed based on reciprocal point learning (RPL), establishing a bounded space for potential unknown classes. This approach indirectly introduces unknown information into a learner confined to known classes, thereby acquiring more concise and discriminative representations. Subsequently, considering the variability in the imaging of targets at different angles and the discreteness of components in SAR images, a proposal is made to design convolutional kernels based on large-sized attribute scattering center models. This enhances the ability to extract intrinsic non-linear features and specific scattering characteristics in SAR images, thereby improving the discriminative features of the model and mitigating the impact of imaging variations on classification performance. Experiments on the MSTAR datasets substantiate the superior performance of the proposed approach called ASC-RPL over mainstream methods.

Title: The Multiple Dimensions of Spuriousness in Machine Learning

• Authors: Samuel J. Bell, Skyler Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04696
• Pdf URL: https://arxiv.org/pdf/2411.04696
• Copy Paste: [[2411.04696]] The Multiple Dimensions of Spuriousness in Machine Learning(https://arxiv.org/abs/2411.04696)
• Keywords: robust, fair
• Abstract: Learning correlations from data forms the foundation of today's machine learning (ML) and artificial intelligence (AI) research. While such an approach enables the automatic discovery of patterned relationships within big data corpora, it is susceptible to failure modes when unintended correlations are captured. This vulnerability has expanded interest in interrogating spuriousness, often critiqued as an impediment to model performance, fairness, and robustness. In this article, we trace deviations from the conventional definition of statistical spuriousness-which denotes a non-causal observation arising from either coincidence or confounding variables-to articulate how ML researchers make sense of spuriousness in practice. Drawing on a broad survey of ML literature, we conceptualize the "multiple dimensions of spuriousness," encompassing: relevance ("Models should only use correlations that are relevant to the task."), generalizability ("Models should only use correlations that generalize to unseen data"), human-likeness ("Models should only use correlations that a human would use to perform the same task"), and harmfulness ("Models should only use correlations that are not harmful"). These dimensions demonstrate that ML spuriousness goes beyond the causal/non-causal dichotomy and that the disparate interpretative paths researchers choose could meaningfully influence the trajectory of ML development. By underscoring how a fundamental problem in ML is contingently negotiated in research contexts, we contribute to ongoing debates about responsible practices in AI development.

Title: Dynamic Brightness Adaptation for Robust Multi-modal Image Fusion

• Authors: Yiming Sun, Bing Cao, Pengfei Zhu, Qinghua Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04697
• Pdf URL: https://arxiv.org/pdf/2411.04697
• Copy Paste: [[2411.04697]] Dynamic Brightness Adaptation for Robust Multi-modal Image Fusion(https://arxiv.org/abs/2411.04697)
• Keywords: robust
• Abstract: Infrared and visible image fusion aim to integrate modality strengths for visually enhanced, informative images. Visible imaging in real-world scenarios is susceptible to dynamic environmental brightness fluctuations, leading to texture degradation. Existing fusion methods lack robustness against such brightness perturbations, significantly compromising the visual fidelity of the fused imagery. To address this challenge, we propose the Brightness Adaptive multimodal dynamic fusion framework (BA-Fusion), which achieves robust image fusion despite dynamic brightness fluctuations. Specifically, we introduce a Brightness Adaptive Gate (BAG) module, which is designed to dynamically select features from brightness-related channels for normalization, while preserving brightness-independent structural information within the source images. Furthermore, we propose a brightness consistency loss function to optimize the BAG module. The entire framework is tuned via alternating training strategies. Extensive experiments validate that our method surpasses state-of-the-art methods in preserving multi-modal image information and visual fidelity, while exhibiting remarkable robustness across varying brightness levels. Our code is available: this https URL.

Title: ESC-MISR: Enhancing Spatial Correlations for Multi-Image Super-Resolution in Remote Sensing

• Authors: Zhihui Zhang, Jinhui Pang, Jianan Li, Xiaoshuai Hao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04706
• Pdf URL: https://arxiv.org/pdf/2411.04706
• Copy Paste: [[2411.04706]] ESC-MISR: Enhancing Spatial Correlations for Multi-Image Super-Resolution in Remote Sensing(https://arxiv.org/abs/2411.04706)
• Keywords: transformer
• Abstract: Multi-Image Super-Resolution (MISR) is a crucial yet challenging research task in the remote sensing community. In this paper, we address the challenging task of Multi-Image Super-Resolution in Remote Sensing (MISR-RS), aiming to generate a High-Resolution (HR) image from multiple Low-Resolution (LR) images obtained by satellites. Recently, the weak temporal correlations among LR images have attracted increasing attention in the MISR-RS task. However, existing MISR methods treat the LR images as sequences with strong temporal correlations, overlooking spatial correlations and imposing temporal dependencies. To address this problem, we propose a novel end-to-end framework named Enhancing Spatial Correlations in MISR (ESC-MISR), which fully exploits the spatial-temporal relations of multiple images for HR image reconstruction. Specifically, we first introduce a novel fusion module named Multi-Image Spatial Transformer (MIST), which emphasizes parts with clearer global spatial features and enhances the spatial correlations between LR images. Besides, we perform a random shuffle strategy for the sequential inputs of LR images to attenuate temporal dependencies and capture weak temporal correlations in the training stage. Compared with the state-of-the-art methods, our ESC-MISR achieves 0.70dB and 0.76dB cPSNR improvements on the two bands of the PROBA-V dataset respectively, demonstrating the superiority of our method.

Title: From CNN to ConvRNN: Adapting Visualization Techniques for Time-Series Anomaly Detection

• Authors: Fabien Poirier
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04707
• Pdf URL: https://arxiv.org/pdf/2411.04707
• Copy Paste: [[2411.04707]] From CNN to ConvRNN: Adapting Visualization Techniques for Time-Series Anomaly Detection(https://arxiv.org/abs/2411.04707)
• Keywords: explainability
• Abstract: Nowadays, neural networks are commonly used to solve various problems. Unfortunately, despite their effectiveness, they are often perceived as black boxes capable of providing answers without explaining their decisions, which raises numerous ethical and legal concerns. Fortunately, the field of explainability helps users understand these results. This aspect of machine learning allows users to grasp the decision-making process of a model and verify the relevance of its outcomes. In this article, we focus on the learning process carried out by a ``time distributed`` convRNN, which performs anomaly detection from video data.

Title: Exploring Hierarchical Molecular Graph Representation in Multimodal LLMs

• Authors: Chengxin Hu, Hao Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04708
• Pdf URL: https://arxiv.org/pdf/2411.04708
• Copy Paste: [[2411.04708]] Exploring Hierarchical Molecular Graph Representation in Multimodal LLMs(https://arxiv.org/abs/2411.04708)
• Keywords: large language model
• Abstract: Following the milestones in large language models (LLMs) and multimodal models, we have seen a surge in applying LLMs to biochemical tasks. Leveraging graph features and molecular text representations, LLMs can tackle various tasks, such as predicting chemical reaction outcomes and describing molecular properties. However, most current work overlooks the multi-level nature of graph features. The impact of different feature levels on LLMs and the importance of each level remain unexplored, and it is possible that different chemistry tasks require different feature levels. In this work, we first investigate the effect of feature granularity by fusing GNN-generated feature tokens, discovering that even reducing all tokens to a single token does not significantly impact performance. We then explore the effect of various feature levels on performance, finding that both the quality of LLM-generated molecules and performance on different tasks benefit from different feature levels. We conclude with two key insights: (1) current molecular Multimodal LLMs(MLLMs) lack a comprehensive understanding of graph features, and (2) static processing is not sufficient for hierarchical graph feature. Our code will be publicly available soon.

Title: TIP-I2V: A Million-Scale Real Text and Image Prompt Dataset for Image-to-Video Generation

• Authors: Wenhao Wang, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04709
• Pdf URL: https://arxiv.org/pdf/2411.04709
• Copy Paste: [[2411.04709]] TIP-I2V: A Million-Scale Real Text and Image Prompt Dataset for Image-to-Video Generation(https://arxiv.org/abs/2411.04709)
• Keywords: diffusion
• Abstract: Video generation models are revolutionizing content creation, with image-to-video models drawing increasing attention due to their enhanced controllability, visual consistency, and practical applications. However, despite their popularity, these models rely on user-provided text and image prompts, and there is currently no dedicated dataset for studying these prompts. In this paper, we introduce TIP-I2V, the first large-scale dataset of over 1.70 million unique user-provided Text and Image Prompts specifically for Image-to-Video generation. Additionally, we provide the corresponding generated videos from five state-of-the-art image-to-video models. We begin by outlining the time-consuming and costly process of curating this large-scale dataset. Next, we compare TIP-I2V to two popular prompt datasets, VidProM (text-to-video) and DiffusionDB (text-to-image), highlighting differences in both basic and semantic information. This dataset enables advancements in image-to-video research. For instance, to develop better models, researchers can use the prompts in TIP-I2V to analyze user preferences and evaluate the multi-dimensional performance of their trained models; and to enhance model safety, they may focus on addressing the misinformation issue caused by image-to-video models. The new research inspired by TIP-I2V and the differences with existing datasets emphasize the importance of a specialized image-to-video prompt dataset. The project is publicly available at this https URL.

Title: Differential Privacy Overview and Fundamental Techniques

• Authors: Ferdinando Fioretto, Pascal Van Hentenryck, Juba Ziani
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04710
• Pdf URL: https://arxiv.org/pdf/2411.04710
• Copy Paste: [[2411.04710]] Differential Privacy Overview and Fundamental Techniques(https://arxiv.org/abs/2411.04710)
• Keywords: privacy, protect, robust
• Abstract: This chapter is meant to be part of the book "Differential Privacy in Artificial Intelligence: From Theory to Practice" and provides an introduction to Differential Privacy. It starts by illustrating various attempts to protect data privacy, emphasizing where and why they failed, and providing the key desiderata of a robust privacy definition. It then defines the key actors, tasks, and scopes that make up the domain of privacy-preserving data analysis. Following that, it formalizes the definition of Differential Privacy and its inherent properties, including composition, post-processing immunity, and group privacy. The chapter also reviews the basic techniques and mechanisms commonly used to implement Differential Privacy in its pure and approximate forms.

Title: SEE-DPO: Self Entropy Enhanced Direct Preference Optimization

• Authors: Shivanshu Shekhar, Shreyas Singh, Tong Zhang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04712
• Pdf URL: https://arxiv.org/pdf/2411.04712
• Copy Paste: [[2411.04712]] SEE-DPO: Self Entropy Enhanced Direct Preference Optimization(https://arxiv.org/abs/2411.04712)
• Keywords: robust, diffusion, generative, large language model
• Abstract: Direct Preference Optimization (DPO) has been successfully used to align large language models (LLMs) according to human preferences, and more recently it has also been applied to improving the quality of text-to-image diffusion models. However, DPO-based methods such as SPO, Diffusion-DPO, and D3PO are highly susceptible to overfitting and reward hacking, especially when the generative model is optimized to fit out-of-distribution during prolonged training. To overcome these challenges and stabilize the training of diffusion models, we introduce a self-entropy regularization mechanism in reinforcement learning from human feedback. This enhancement improves DPO training by encouraging broader exploration and greater robustness. Our regularization technique effectively mitigates reward hacking, leading to improved stability and enhanced image quality across the latent space. Extensive experiments demonstrate that integrating human feedback with self-entropy regularization can significantly boost image diversity and specificity, achieving state-of-the-art results on key image generation metrics.

Title: Multi-Reward as Condition for Instruction-based Image Editing

• Authors: Xin Gu, Ming Li, Libo Zhang, Fan Chen, Longyin Wen, Tiejian Luo, Sijie Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04713
• Pdf URL: https://arxiv.org/pdf/2411.04713
• Copy Paste: [[2411.04713]] Multi-Reward as Condition for Instruction-based Image Editing(https://arxiv.org/abs/2411.04713)
• Keywords: diffusion, generative
• Abstract: High-quality training triplets (instruction, original image, edited image) are essential for instruction-based image editing. Predominant training datasets (e.g., InsPix2Pix) are created using text-to-image generative models (e.g., Stable Diffusion, DALL-E) which are not trained for image editing. Accordingly, these datasets suffer from inaccurate instruction following, poor detail preserving, and generation artifacts. In this paper, we propose to address the training data quality issue with multi-perspective reward data instead of refining the ground-truth image quality. 1) we first design a quantitative metric system based on best-in-class LVLM (Large Vision Language Model), i.e., GPT-4o in our case, to evaluate the generation quality from 3 perspectives, namely, instruction following, detail preserving, and generation quality. For each perspective, we collected quantitative score in $0\sim 5$ and text descriptive feedback on the specific failure points in ground-truth edited images, resulting in a high-quality editing reward dataset, i.e., RewardEdit20K. 2) We further proposed a novel training framework to seamlessly integrate the metric output, regarded as multi-reward, into editing models to learn from the imperfect training triplets. During training, the reward scores and text descriptions are encoded as embeddings and fed into both the latent space and the U-Net of the editing models as auxiliary conditions. During inference, we set these additional conditions to the highest score with no text description for failure points, to aim at the best generation outcome. Experiments indicate that our multi-reward conditioned model outperforms its no-reward counterpart on two popular editing pipelines, i.e., InsPix2Pix and SmartEdit. The code and dataset will be released.

Title: Revisiting Disparity from Dual-Pixel Images: Physics-Informed Lightweight Depth Estimation

• Authors: Teppei Kurita, Yuhi Kondo, Legong Sun, Takayuki Sasaki, Sho Nitta, Yasuhiro Hashimoto, Yoshinori Muramatsu, Yusuke Moriuchi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04714
• Pdf URL: https://arxiv.org/pdf/2411.04714
• Copy Paste: [[2411.04714]] Revisiting Disparity from Dual-Pixel Images: Physics-Informed Lightweight Depth Estimation(https://arxiv.org/abs/2411.04714)
• Keywords: robust
• Abstract: In this study, we propose a high-performance disparity (depth) estimation method using dual-pixel (DP) images with few parameters. Conventional end-to-end deep-learning methods have many parameters but do not fully exploit disparity constraints, which limits their performance. Therefore, we propose a lightweight disparity estimation method based on a completion-based network that explicitly constrains disparity and learns the physical and systemic disparity properties of DP. By modeling the DP-specific disparity error parametrically and using it for sampling during training, the network acquires the unique properties of DP and enhances robustness. This learning also allows us to use a common RGB-D dataset for training without a DP dataset, which is labor-intensive to acquire. Furthermore, we propose a non-learning-based refinement framework that efficiently handles inherent disparity expansion errors by appropriately refining the confidence map of the network output. As a result, the proposed method achieved state-of-the-art results while reducing the overall system size to 1/5 of that of the conventional method, even without using the DP dataset for training, thereby demonstrating its effectiveness. The code and dataset are available on our project site.

Title: NeuroFly: A framework for whole-brain single neuron reconstruction

• Authors: Rubin Zhao, Yang Liu, Shiqi Zhang, Zijian Yi, Yanyang Xiao, Fang Xu, Yi Yang, Pencheng Zhou
• Subjects: cs.CV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2411.04715
• Pdf URL: https://arxiv.org/pdf/2411.04715
• Copy Paste: [[2411.04715]] NeuroFly: A framework for whole-brain single neuron reconstruction(https://arxiv.org/abs/2411.04715)
• Keywords: robust, segmentation
• Abstract: Neurons, with their elongated, tree-like dendritic and axonal structures, enable efficient signal integration and long-range communication across brain regions. By reconstructing individual neurons' morphology, we can gain valuable insights into brain connectivity, revealing the structure basis of cognition, movement, and perception. Despite the accumulation of extensive 3D microscopic imaging data, progress has been considerably hindered by the absence of automated tools to streamline this process. Here we introduce NeuroFly, a validated framework for large-scale automatic single neuron reconstruction. This framework breaks down the process into three distinct stages: segmentation, connection, and proofreading. In the segmentation stage, we perform automatic segmentation followed by skeletonization to generate over-segmented neuronal fragments without branches. During the connection stage, we use a 3D image-based path following approach to extend each fragment and connect it with other fragments of the same neuron. Finally, human annotators are required only to proofread the few unresolved positions. The first two stages of our process are clearly defined computer vision problems, and we have trained robust baseline models to solve them. We validated NeuroFly's efficiency using in-house datasets that include a variety of challenging scenarios, such as dense arborizations, weak axons, images with contamination. We will release the datasets along with a suite of visualization and annotation tools for better reproducibility. Our goal is to foster collaboration among researchers to address the neuron reconstruction challenge, ultimately accelerating advancements in neuroscience research. The dataset and code are available at this https URL

Title: Subspace-Constrained Quadratic Matrix Factorization: Algorithm and Applications

• Authors: Zheng Zhai, Xiaohui Li
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04717
• Pdf URL: https://arxiv.org/pdf/2411.04717
• Copy Paste: [[2411.04717]] Subspace-Constrained Quadratic Matrix Factorization: Algorithm and Applications(https://arxiv.org/abs/2411.04717)
• Keywords: robust
• Abstract: Matrix Factorization has emerged as a widely adopted framework for modeling data exhibiting low-rank structures. To address challenges in manifold learning, this paper presents a subspace-constrained quadratic matrix factorization model. The model is designed to jointly learn key low-dimensional structures, including the tangent space, the normal subspace, and the quadratic form that links the tangent space to a low-dimensional representation. We solve the proposed factorization model using an alternating minimization method, involving an in-depth investigation of nonlinear regression and projection subproblems. Theoretical properties of the quadratic projection problem and convergence characteristics of the alternating strategy are also investigated. To validate our approach, we conduct numerical experiments on synthetic and real-world datasets. Results demonstrate that our model outperforms existing methods, highlighting its robustness and efficacy in capturing core low-dimensional structures.

Title: Controlling Human Shape and Pose in Text-to-Image Diffusion Models via Domain Adaptation

• Authors: Benito Buchheim, Max Reimann, Jürgen Döllner
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04724
• Pdf URL: https://arxiv.org/pdf/2411.04724
• Copy Paste: [[2411.04724]] Controlling Human Shape and Pose in Text-to-Image Diffusion Models via Domain Adaptation(https://arxiv.org/abs/2411.04724)
• Keywords: diffusion
• Abstract: We present a methodology for conditional control of human shape and pose in pretrained text-to-image diffusion models using a 3D human parametric model (SMPL). Fine-tuning these diffusion models to adhere to new conditions requires large datasets and high-quality annotations, which can be more cost-effectively acquired through synthetic data generation rather than real-world data. However, the domain gap and low scene diversity of synthetic data can compromise the pretrained model's visual fidelity. We propose a domain-adaptation technique that maintains image quality by isolating synthetically trained conditional information in the classifier-free guidance vector and composing it with another control network to adapt the generated images to the input domain. To achieve SMPL control, we fine-tune a ControlNet-based architecture on the synthetic SURREAL dataset of rendered humans and apply our domain adaptation at generation time. Experiments demonstrate that our model achieves greater shape and pose diversity than the 2d pose-based ControlNet, while maintaining the visual fidelity and improving stability, proving its usefulness for downstream tasks such as human animation.

Title: Taming Rectified Flow for Inversion and Editing

• Authors: Jiangshan Wang, Junfu Pu, Zhongang Qi, Jiayi Guo, Yue Ma, Nisha Huang, Yuxin Chen, Xiu Li, Ying Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04746
• Pdf URL: https://arxiv.org/pdf/2411.04746
• Copy Paste: [[2411.04746]] Taming Rectified Flow for Inversion and Editing(https://arxiv.org/abs/2411.04746)
• Keywords: robust, diffusion, transformer, generative
• Abstract: Rectified-flow-based diffusion transformers, such as FLUX and OpenSora, have demonstrated exceptional performance in the field of image and video generation. Despite their robust generative capabilities, these models often suffer from inaccurate inversion, which could further limit their effectiveness in downstream tasks such as image and video editing. To address this issue, we propose RF-Solver, a novel training-free sampler that enhances inversion precision by reducing errors in the process of solving rectified flow ODEs. Specifically, we derive the exact formulation of the rectified flow ODE and perform a high-order Taylor expansion to estimate its nonlinear components, significantly decreasing the approximation error at each timestep. Building upon RF-Solver, we further design RF-Edit, which comprises specialized sub-modules for image and video editing. By sharing self-attention layer features during the editing process, RF-Edit effectively preserves the structural information of the source image or video while achieving high-quality editing results. Our approach is compatible with any pre-trained rectified-flow-based models for image and video tasks, requiring no additional training or optimization. Extensive experiments on text-to-image generation, image & video inversion, and image & video editing demonstrate the robust performance and adaptability of our methods. Code is available at this https URL.

Title: Mining the Minoria: Unknown, Under-represented, and Under-performing Minority Groups

• Authors: Mohsen Dehghankar, Abolfazl Asudeh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04761
• Pdf URL: https://arxiv.org/pdf/2411.04761
• Copy Paste: [[2411.04761]] Mining the Minoria: Unknown, Under-represented, and Under-performing Minority Groups(https://arxiv.org/abs/2411.04761)
• Keywords: privacy
• Abstract: Due to a variety of reasons, such as privacy, data in the wild often misses the grouping information required for identifying minorities. On the other hand, it is known that machine learning models are only as good as the data they are trained on and, hence, may underperform for the under-represented minority groups. The missing grouping information presents a dilemma for responsible data scientists who find themselves in an unknown-unknown situation, where not only do they not have access to the grouping attributes but do not also know what groups to consider. This paper is an attempt to address this dilemma. Specifically, we propose a minority mining problem, where we find vectors in the attribute space that reveal potential groups that are under-represented and under-performing. Technically speaking, we propose a geometric transformation of data into a dual space and use notions such as the arrangement of hyperplanes to design an efficient algorithm for the problem in lower dimensions. Generalizing our solution to the higher dimensions is cursed by dimensionality. Therefore, we propose a solution based on smart exploration of the search space for such cases. We conduct comprehensive experiments using real-world and synthetic datasets alongside the theoretical analysis. Our experiment results demonstrate the effectiveness of our proposed solutions in mining the unknown, under-represented, and under-performing minorities.

Title: Attention Masks Help Adversarial Attacks to Bypass Safety Detectors

• Authors: Yunfan Shi
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04772
• Pdf URL: https://arxiv.org/pdf/2411.04772
• Copy Paste: [[2411.04772]] Attention Masks Help Adversarial Attacks to Bypass Safety Detectors(https://arxiv.org/abs/2411.04772)
• Keywords: protect, defense, attack, steal, explainability
• Abstract: Despite recent research advancements in adversarial attack methods, current approaches against XAI monitors are still discoverable and slower. In this paper, we present an adaptive framework for attention mask generation to enable stealthy, explainable and efficient PGD image classification adversarial attack under XAI monitors. Specifically, we utilize mutation XAI mixture and multitask self-supervised X-UNet for attention mask generation to guide PGD attack. Experiments on MNIST (MLP), CIFAR-10 (AlexNet) have shown that our system can outperform benchmark PGD, Sparsefool and SOTA SINIFGSM in balancing among stealth, efficiency and explainability which is crucial for effectively fooling SOTA defense protected classifiers.

Title: Learn to Solve Vehicle Routing Problems ASAP: A Neural Optimization Approach for Time-Constrained Vehicle Routing Problems with Finite Vehicle Fleet

• Authors: Elija Deineko, Carina Kehrt
• Subjects: cs.LG, cs.ET
• Abstract URL: https://arxiv.org/abs/2411.04777
• Pdf URL: https://arxiv.org/pdf/2411.04777
• Copy Paste: [[2411.04777]] Learn to Solve Vehicle Routing Problems ASAP: A Neural Optimization Approach for Time-Constrained Vehicle Routing Problems with Finite Vehicle Fleet(https://arxiv.org/abs/2411.04777)
• Keywords: robust, generative
• Abstract: Finding a feasible and prompt solution to the Vehicle Routing Problem (VRP) is a prerequisite for efficient freight transportation, seamless logistics, and sustainable mobility. Traditional optimization methods reach their limits when confronted with the real-world complexity of VRPs, which involve numerous constraints and objectives. Recently, the ability of generative Artificial Intelligence (AI) to solve combinatorial tasks, known as Neural Combinatorial Optimization (NCO), demonstrated promising results, offering new perspectives. In this study, we propose an NCO approach to solve a time-constrained capacitated VRP with a finite vehicle fleet size. The approach is based on an encoder-decoder architecture, formulated in line with the Policy Optimization with Multiple Optima (POMO) protocol and trained via a Proximal Policy Optimization (PPO) algorithm. We successfully trained the policy with multiple objectives (minimizing the total distance while maximizing vehicle utilization) and evaluated it on medium and large instances, benchmarking it against state-of-the-art heuristics. The method is able to find adequate and cost-efficient solutions, showing both flexibility and robust generalization. Finally, we provide a critical analysis of the solution generated by NCO and discuss the challenges and opportunities of this new branch of intelligent learning algorithms emerging in optimization science, focusing on freight transportation.

Title: AlignXIE: Improving Multilingual Information Extraction by Cross-Lingual Alignment

• Authors: Yuxin Zuo, Wenxuan Jiang, Wenxuan Liu, Zixuan Li, Long Bai, Hanbin Wang, Yutao Zeng, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04794
• Pdf URL: https://arxiv.org/pdf/2411.04794
• Copy Paste: [[2411.04794]] AlignXIE: Improving Multilingual Information Extraction by Cross-Lingual Alignment(https://arxiv.org/abs/2411.04794)
• Keywords: extraction
• Abstract: Empirical evidence suggests that LLMs exhibit spontaneous cross-lingual alignment. Our findings suggest that although LLMs also demonstrate promising cross-lingual alignment in Information Extraction, there remains significant imbalance across languages, revealing an underlying deficiency in the IE alignment. To address this issue, we propose AlignXIE, a powerful code-based LLM that significantly enhances cross-lingual IE alignment through two strategies. Firstly, AlignXIE formulates IE across different languages, especially non-English ones, as code generation tasks, standardizing the representation of various schemas using Python classes to ensure consistency of the same ontology in different languages and align the schema. Secondly, it incorporates an IE cross-lingual alignment phase through a translated instance prediction task proposed in this paper to align the extraction process, utilizing ParallelNER, an IE bilingual parallel dataset with 257,190 samples, generated by our proposed LLM-based automatic pipeline for IE parallel data construction, with manual annotation to ensure quality. Ultimately, we obtain AlignXIE through multilingual IE instruction tuning. Although without training in 9 unseen languages, AlignXIE surpasses ChatGPT by $30.17\%$ and SoTA by $20.03\%$, thereby demonstrating superior cross-lingual IE capabilities. Comprehensive evaluations on 63 IE benchmarks in Chinese and English under various settings, demonstrate that AlignXIE significantly enhances cross-lingual and multilingual IE through boosting the IE alignment.

Title: Defending Deep Regression Models against Backdoor Attacks

• Authors: Lingyu Du, Yupei Liu, Jinyuan Jia, Guohao Lan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04811
• Pdf URL: https://arxiv.org/pdf/2411.04811
• Copy Paste: [[2411.04811]] Defending Deep Regression Models against Backdoor Attacks(https://arxiv.org/abs/2411.04811)
• Keywords: defense, attack
• Abstract: Deep regression models are used in a wide variety of safety-critical applications, but are vulnerable to backdoor attacks. Although many defenses have been proposed for classification models, they are ineffective as they do not consider the uniqueness of regression models. First, the outputs of regression models are continuous values instead of discretized labels. Thus, the potential infected target of a backdoored regression model has infinite possibilities, which makes it impossible to be determined by existing defenses. Second, the backdoor behavior of backdoored deep regression models is triggered by the activation values of all the neurons in the feature space, which makes it difficult to be detected and mitigated using existing defenses. To resolve these problems, we propose DRMGuard, the first defense to identify if a deep regression model in the image domain is backdoored or not. DRMGuard formulates the optimization problem for reverse engineering based on the unique output-space and feature-space characteristics of backdoored deep regression models. We conduct extensive evaluations on two regression tasks and four datasets. The results show that DRMGuard can consistently defend against various backdoor attacks. We also generalize four state-of-the-art defenses designed for classifiers to regression models, and compare DRMGuard with them. The results show that DRMGuard significantly outperforms all those defenses.

Title: LuxBank: The First Universal Dependency Treebank for Luxembourgish

• Authors: Alistair Plum, Caroline Döhmer, Emilia Milano, Anne-Marie Lutgen, Christoph Purschke
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04813
• Pdf URL: https://arxiv.org/pdf/2411.04813
• Copy Paste: [[2411.04813]] LuxBank: The First Universal Dependency Treebank for Luxembourgish(https://arxiv.org/abs/2411.04813)
• Keywords: large language model
• Abstract: The Universal Dependencies (UD) project has significantly expanded linguistic coverage across 161 languages, yet Luxembourgish, a West Germanic language spoken by approximately 400,000 people, has remained absent until now. In this paper, we introduce LuxBank, the first UD Treebank for Luxembourgish, addressing the gap in syntactic annotation and analysis for this `low-research' language. We establish formal guidelines for Luxembourgish language annotation, providing the foundation for the first large-scale quantitative analysis of its syntax. LuxBank serves not only as a resource for linguists and language learners but also as a tool for developing spell checkers and grammar checkers, organising existing text archives and even training large language models. By incorporating Luxembourgish into the UD framework, we aim to enhance the understanding of syntactic variation within West Germanic languages and offer a model for documenting smaller, semi-standardised languages. This work positions Luxembourgish as a valuable resource in the broader linguistic and NLP communities, contributing to the study of languages with limited research and resources.

Title: End-to-end Inception-Unet based Generative Adversarial Networks for Snow and Rain Removals

• Authors: Ibrahim Kajo, Mohamed Kas, Yassine Ruichek
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.04821
• Pdf URL: https://arxiv.org/pdf/2411.04821
• Copy Paste: [[2411.04821]] End-to-end Inception-Unet based Generative Adversarial Networks for Snow and Rain Removals(https://arxiv.org/abs/2411.04821)
• Keywords: extraction, generative
• Abstract: The superior performance introduced by deep learning approaches in removing atmospheric particles such as snow and rain from a single image; favors their usage over classical ones. However, deep learning-based approaches still suffer from challenges related to the particle appearance characteristics such as size, type, and transparency. Furthermore, due to the unique characteristics of rain and snow particles, single network based deep learning approaches struggle in handling both degradation scenarios simultaneously. In this paper, a global framework that consists of two Generative Adversarial Networks (GANs) is proposed where each handles the removal of each particle individually. The architectures of both desnowing and deraining GANs introduce the integration of a feature extraction phase with the classical U-net generator network which in turn enhances the removal performance in the presence of severe variations in size and appearance. Furthermore, a realistic dataset that contains pairs of snowy images next to their groundtruth images estimated using a low-rank approximation approach; is presented. The experiments show that the proposed desnowing and deraining approaches achieve significant improvements in comparison to the state-of-the-art approaches when tested on both synthetic and realistic datasets.

Title: VTechAGP: An Academic-to-General-Audience Text Paraphrase Dataset and Benchmark Models

• Authors: Ming Cheng, Jiaying Gong, Chenhan Yuan, William A. Ingram, Edward Fox, Hoda Eldardiry
• Subjects: cs.CL, cs.DL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04825
• Pdf URL: https://arxiv.org/pdf/2411.04825
• Copy Paste: [[2411.04825]] VTechAGP: An Academic-to-General-Audience Text Paraphrase Dataset and Benchmark Models(https://arxiv.org/abs/2411.04825)
• Keywords: generative, large language model
• Abstract: Existing text simplification or paraphrase datasets mainly focus on sentence-level text generation in a general domain. These datasets are typically developed without using domain knowledge. In this paper, we release a novel dataset, VTechAGP, which is the first academic-to-general-audience text paraphrase dataset consisting of 4,938 document-level these and dissertation academic and general-audience abstract pairs from 8 colleges authored over 25 years. We also propose a novel dynamic soft prompt generative language model, DSPT5. For training, we leverage a contrastive-generative loss function to learn the keyword vectors in the dynamic prompt. For inference, we adopt a crowd-sampling decoding strategy at both semantic and structural levels to further select the best output candidate. We evaluate DSPT5 and various state-of-the-art large language models (LLMs) from multiple perspectives. Results demonstrate that the SOTA LLMs does not provide satisfactory outcomes, while the lightweight DSPT5 can achieve competitive results. To the best of our knowledge, we are the first to build a benchmark dataset and solutions for academic-to-general-audience text paraphrase dataset.

Title: Prompt-Guided Internal States for Hallucination Detection of Large Language Models

• Authors: Fujie Zhang, Peiqi Yu, Biao Yi, Baolei Zhang, Tong Li, Zheli Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04847
• Pdf URL: https://arxiv.org/pdf/2411.04847
• Copy Paste: [[2411.04847]] Prompt-Guided Internal States for Hallucination Detection of Large Language Models(https://arxiv.org/abs/2411.04847)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across a variety of tasks in different domains. However, they sometimes generate responses that are logically coherent but factually incorrect or misleading, which is known as LLM hallucinations. Data-driven supervised methods train hallucination detectors by leveraging the internal states of LLMs, but detectors trained on specific domains often struggle to generalize well to other domains. In this paper, we aim to enhance the cross-domain performance of supervised detectors with only in-domain data. We propose a novel framework, prompt-guided internal states for hallucination detection of LLMs, namely PRISM. By utilizing appropriate prompts to guide changes in the structure related to text truthfulness within the LLM's internal states, we make this structure more salient and consistent across texts from different domains. We integrated our framework with existing hallucination detection methods and conducted experiments on datasets from different domains. The experimental results indicate that our framework significantly enhances the cross-domain generalization of existing hallucination detection methods.

Title: ZAHA: Introducing the Level of Facade Generalization and the Large-Scale Point Cloud Facade Semantic Segmentation Benchmark Dataset

• Authors: Olaf Wysocki, Yue Tan, Thomas Froech, Yan Xia, Magdalena Wysocki, Ludwig Hoegner, Daniel Cremers, Christoph Holst
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04865
• Pdf URL: https://arxiv.org/pdf/2411.04865
• Copy Paste: [[2411.04865]] ZAHA: Introducing the Level of Facade Generalization and the Large-Scale Point Cloud Facade Semantic Segmentation Benchmark Dataset(https://arxiv.org/abs/2411.04865)
• Keywords: robust, segmentation
• Abstract: Facade semantic segmentation is a long-standing challenge in photogrammetry and computer vision. Although the last decades have witnessed the influx of facade segmentation methods, there is a lack of comprehensive facade classes and data covering the architectural variability. In ZAHA, we introduce Level of Facade Generalization (LoFG), novel hierarchical facade classes designed based on international urban modeling standards, ensuring compatibility with real-world challenging classes and uniform methods' comparison. Realizing the LoFG, we present to date the largest semantic 3D facade segmentation dataset, providing 601 million annotated points at five and 15 classes of LoFG2 and LoFG3, respectively. Moreover, we analyze the performance of baseline semantic segmentation methods on our introduced LoFG classes and data, complementing it with a discussion on the unresolved challenges for facade segmentation. We firmly believe that ZAHA shall facilitate further development of 3D facade semantic segmentation methods, enabling robust segmentation indispensable in creating urban digital twins.

Title: Boosting Latent Diffusion with Perceptual Objectives

• Authors: Tariq Berrada, Pietro Astolfi, Jakob Verbeek, Melissa Hall, Marton Havasi, Michal Drozdzal, Yohann Benchetrit, Adriana Romero-Soriano, Karteek Alahari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04873
• Pdf URL: https://arxiv.org/pdf/2411.04873
• Copy Paste: [[2411.04873]] Boosting Latent Diffusion with Perceptual Objectives(https://arxiv.org/abs/2411.04873)
• Keywords: diffusion, generative
• Abstract: Latent diffusion models (LDMs) power state-of-the-art high-resolution generative image models. LDMs learn the data distribution in the latent space of an autoencoder (AE) and produce images by mapping the generated latents into RGB image space using the AE decoder. While this approach allows for efficient model training and sampling, it induces a disconnect between the training of the diffusion model and the decoder, resulting in a loss of detail in the generated images. To remediate this disconnect, we propose to leverage the internal features of the decoder to define a latent perceptual loss (LPL). This loss encourages the models to create sharper and more realistic images. Our loss can be seamlessly integrated with common autoencoders used in latent diffusion models, and can be applied to different generative modeling paradigms such as DDPM with epsilon and velocity prediction, as well as flow matching. Extensive experiments with models trained on three datasets at 256 and 512 resolution show improved quantitative -- with boosts between 6% and 20% in FID -- and qualitative results when using our perceptual loss.

Title: In the Era of Prompt Learning with Vision-Language Models

• Authors: Ankit Jha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04892
• Pdf URL: https://arxiv.org/pdf/2411.04892
• Copy Paste: [[2411.04892]] In the Era of Prompt Learning with Vision-Language Models(https://arxiv.org/abs/2411.04892)
• Keywords: segmentation
• Abstract: Large-scale foundation models like CLIP have shown strong zero-shot generalization but struggle with domain shifts, limiting their adaptability. In our work, we introduce \textsc{StyLIP}, a novel domain-agnostic prompt learning strategy for Domain Generalization (DG). StyLIP disentangles visual style and content in CLIP`s vision encoder by using style projectors to learn domain-specific prompt tokens and combining them with content features. Trained contrastively, this approach enables seamless adaptation across domains, outperforming state-of-the-art methods on multiple DG benchmarks. Additionally, we propose AD-CLIP for unsupervised domain adaptation (DA), leveraging CLIP`s frozen vision backbone to learn domain-invariant prompts through image style and content features. By aligning domains in embedding space with entropy minimization, AD-CLIP effectively handles domain shifts, even when only target domain samples are available. Lastly, we outline future work on class discovery using prompt learning for semantic segmentation in remote sensing, focusing on identifying novel or rare classes in unstructured environments. This paves the way for more adaptive and generalizable models in complex, real-world scenarios.

Title: Sampling-guided Heterogeneous Graph Neural Network with Temporal Smoothing for Scalable Longitudinal Data Imputation

• Authors: Zhaoyang Zhang, Ziqi Chen, Qiao Liu, Jinhan Xie, Hongtu Zhu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04899
• Pdf URL: https://arxiv.org/pdf/2411.04899
• Copy Paste: [[2411.04899]] Sampling-guided Heterogeneous Graph Neural Network with Temporal Smoothing for Scalable Longitudinal Data Imputation(https://arxiv.org/abs/2411.04899)
• Keywords: robust
• Abstract: In this paper, we propose a novel framework, the Sampling-guided Heterogeneous Graph Neural Network (SHT-GNN), to effectively tackle the challenge of missing data imputation in longitudinal studies. Unlike traditional methods, which often require extensive preprocessing to handle irregular or inconsistent missing data, our approach accommodates arbitrary missing data patterns while maintaining computational efficiency. SHT-GNN models both observations and covariates as distinct node types, connecting observation nodes at successive time points through subject-specific longitudinal subnetworks, while covariate-observation interactions are represented by attributed edges within bipartite graphs. By leveraging subject-wise mini-batch sampling and a multi-layer temporal smoothing mechanism, SHT-GNN efficiently scales to large datasets, while effectively learning node representations and imputing missing data. Extensive experiments on both synthetic and real-world datasets, including the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, demonstrate that SHT-GNN significantly outperforms existing imputation methods, even with high missing data rates. The empirical results highlight SHT-GNN's robust imputation capabilities and superior performance, particularly in the context of complex, large-scale longitudinal data.

Title: OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

• Authors: Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J.H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, Wei Chu
• Subjects: cs.CL, cs.PL
• Abstract URL: https://arxiv.org/abs/2411.04905
• Pdf URL: https://arxiv.org/pdf/2411.04905
• Copy Paste: [[2411.04905]] OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models(https://arxiv.org/abs/2411.04905)
• Keywords: large language model
• Abstract: Large language models (LLMs) for code have become indispensable in various domains, including code generation, reasoning tasks and agent this http URL open-access code LLMs are increasingly approaching the performance levels of proprietary models, high-quality code LLMs suitable for rigorous scientific investigation, particularly those with reproducible data processing pipelines and transparent training protocols, remain limited. The scarcity is due to various challenges, including resource constraints, ethical considerations, and the competitive advantages of keeping models advanced. To address the gap, we introduce OpenCoder, a top-tier code LLM that not only achieves performance comparable to leading models but also serves as an ``open cookbook'' for the research community. Unlike most prior efforts, we release not only model weights and inference code, but also the reproducible training data, complete data processing pipeline, rigorous experimental ablation results, and detailed training protocols for open scientific research. Through this comprehensive release, we identify the key ingredients for building a top-tier code LLM: (1) code optimized heuristic rules for data cleaning and methods for data deduplication, (2) recall of text corpus related to code and (3) high-quality synthetic data in both annealing and supervised fine-tuning stages. By offering this level of openness, we aim to broaden access to all aspects of a top-tier code LLM, with OpenCoder serving as both a powerful model and an open foundation to accelerate research, and enable reproducible advancements in code AI.

Title: Robust Iris Centre Localisation for Assistive Eye-Gaze Tracking

• Authors: Nipun Sandamal Ranasekara Pathiranage, Stefania Cristina, Kenneth P. Camilleri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04912
• Pdf URL: https://arxiv.org/pdf/2411.04912
• Copy Paste: [[2411.04912]] Robust Iris Centre Localisation for Assistive Eye-Gaze Tracking(https://arxiv.org/abs/2411.04912)
• Keywords: robust, segmentation
• Abstract: In this research work, we address the problem of robust iris centre localisation in unconstrained conditions as a core component of our eye-gaze tracking platform. We investigate the application of U-Net variants for segmentation-based and regression-based approaches to improve our iris centre localisation, which was previously based on Bayes' classification. The achieved results are comparable to or better than the state-of-the-art, offering a drastic improvement over those achieved by the Bayes' classifier, and without sacrificing the real-time performance of our eye-gaze tracking platform.

Title: GASE: Generatively Augmented Sentence Encoding

• Authors: Manuel Frank, Haithem Afli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04914
• Pdf URL: https://arxiv.org/pdf/2411.04914
• Copy Paste: [[2411.04914]] GASE: Generatively Augmented Sentence Encoding(https://arxiv.org/abs/2411.04914)
• Keywords: robust, generative
• Abstract: We propose an approach to enhance sentence embeddings by applying generative text models for data augmentation at inference time. Unlike conventional data augmentation that utilises synthetic training data, our approach does not require access to model parameters or the computational resources typically required for fine-tuning state-of-the-art models. Generatively Augmented Sentence Encoding uses diverse linguistic synthetic variants of input texts generated by paraphrasing, summarising, or extracting keywords, followed by pooling the original and synthetic embeddings. Experimental results on the Massive Text Embedding Benchmark for Semantic Textual Similarity (STS) demonstrate performance improvements across a range of embedding models using different generative models for augmentation. We find that generative augmentation leads to larger performance improvements for embedding models with lower baseline performance. These findings suggest that integrating generative augmentation at inference time adds semantic diversity and can enhance the robustness and generalizability of sentence embeddings for embedding models. Our results show that the degree to which generative augmentation can improve STS performance depends not only on the embedding model but also on the dataset. From a broader perspective, the approach allows trading training for inference compute.

Title: Evaluating Robustness of Reinforcement Learning Algorithms for Autonomous Shipping

• Authors: Bavo Lesy, Ali Anwar, Siegfried Mercelis
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04915
• Pdf URL: https://arxiv.org/pdf/2411.04915
• Copy Paste: [[2411.04915]] Evaluating Robustness of Reinforcement Learning Algorithms for Autonomous Shipping(https://arxiv.org/abs/2411.04915)
• Keywords: robust
• Abstract: Recently, there has been growing interest in autonomous shipping due to its potential to improve maritime efficiency and safety. The use of advanced technologies, such as artificial intelligence, can address the current navigational and operational challenges in autonomous shipping. In particular, inland waterway transport (IWT) presents a unique set of challenges, such as crowded waterways and variable environmental conditions. In such dynamic settings, the reliability and robustness of autonomous shipping solutions are critical factors for ensuring safe operations. This paper examines the robustness of benchmark deep reinforcement learning (RL) algorithms, implemented for IWT within an autonomous shipping simulator, and their ability to generate effective motion planning policies. We demonstrate that a model-free approach can achieve an adequate policy in the simulator, successfully navigating port environments never encountered during training. We focus particularly on Soft-Actor Critic (SAC), which we show to be inherently more robust to environmental disturbances compared to MuZero, a state-of-the-art model-based RL algorithm. In this paper, we take a significant step towards developing robust, applied RL frameworks that can be generalized to various vessel types and navigate complex port- and inland environments and scenarios.

Title: GPTKB: Building Very Large Knowledge Bases from Language Models

• Authors: Yujia Hu, Shrestha Ghosh, Tuan-Phong Nugyen, Simon Razniewski
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2411.04920
• Pdf URL: https://arxiv.org/pdf/2411.04920
• Copy Paste: [[2411.04920]] GPTKB: Building Very Large Knowledge Bases from Language Models(https://arxiv.org/abs/2411.04920)
• Keywords: large language model
• Abstract: General-domain knowledge bases (KB), in particular the "big three" -- Wikidata, Yago and DBpedia -- are the backbone of many intelligent applications. While these three have seen steady development, comprehensive KB construction at large has seen few fresh attempts. In this work, we propose to build a large general-domain KB entirely from a large language model (LLM). We demonstrate the feasibility of large-scale KB construction from LLMs, while highlighting specific challenges arising around entity recognition, entity and property canonicalization, and taxonomy construction. As a prototype, we use GPT-4o-mini to construct GPTKB, which contains 105 million triples for more than 2.9 million entities, at a cost 100x less than previous KBC projects. Our work is a landmark for two fields: For NLP, for the first time, it provides \textit{constructive} insights into the knowledge (or beliefs) of LLMs. For the Semantic Web, it shows novel ways forward for the long-standing challenge of general-domain KB construction. GPTKB is accessible at this https URL.

Title: VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos

• Authors: Shehan Munasinghe, Hanan Gani, Wenqi Zhu, Jiale Cao, Eric Xing, Fahad Shahbaz Khan, Salman Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04923
• Pdf URL: https://arxiv.org/pdf/2411.04923
• Copy Paste: [[2411.04923]] VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos(https://arxiv.org/abs/2411.04923)
• Keywords: large language model, segmentation
• Abstract: Fine-grained alignment between videos and text is challenging due to complex spatial and temporal dynamics in videos. Existing video-based Large Multimodal Models (LMMs) handle basic conversations but struggle with precise pixel-level grounding in videos. To address this, we introduce VideoGLaMM, a LMM designed for fine-grained pixel-level grounding in videos based on user-provided textual inputs. Our design seamlessly connects three key components: a Large Language Model, a dual vision encoder that emphasizes both spatial and temporal details, and a spatio-temporal decoder for accurate mask generation. This connection is facilitated via tunable V-L and L-V adapters that enable close Vision-Language (VL) alignment. The architecture is trained to synchronize both spatial and temporal elements of video content with textual instructions. To enable fine-grained grounding, we curate a multimodal dataset featuring detailed visually-grounded conversations using a semiautomatic annotation pipeline, resulting in a diverse set of 38k video-QA triplets along with 83k objects and 671k masks. We evaluate VideoGLaMM on three challenging tasks: Grounded Conversation Generation, Visual Grounding, and Referring Video Segmentation. Experimental results show that our model consistently outperforms existing approaches across all three tasks.

Title: MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views

• Authors: Yuedong Chen, Chuanxia Zheng, Haofei Xu, Bohan Zhuang, Andrea Vedaldi, Tat-Jen Cham, Jianfei Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04924
• Pdf URL: https://arxiv.org/pdf/2411.04924
• Copy Paste: [[2411.04924]] MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views(https://arxiv.org/abs/2411.04924)
• Keywords: diffusion
• Abstract: We introduce MVSplat360, a feed-forward approach for 360° novel view synthesis (NVS) of diverse real-world scenes, using only sparse observations. This setting is inherently ill-posed due to minimal overlap among input views and insufficient visual information provided, making it challenging for conventional methods to achieve high-quality results. Our MVSplat360 addresses this by effectively combining geometry-aware 3D reconstruction with temporally consistent video generation. Specifically, it refactors a feed-forward 3D Gaussian Splatting (3DGS) model to render features directly into the latent space of a pre-trained Stable Video Diffusion (SVD) model, where these features then act as pose and visual cues to guide the denoising process and produce photorealistic 3D-consistent views. Our model is end-to-end trainable and supports rendering arbitrary views with as few as 5 sparse input views. To evaluate MVSplat360's performance, we introduce a new benchmark using the challenging DL3DV-10K dataset, where MVSplat360 achieves superior visual quality compared to state-of-the-art methods on wide-sweeping or even 360° NVS tasks. Experiments on the existing benchmark RealEstate10K also confirm the effectiveness of our model. The video results are available on our project page: this https URL.

Title: DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion

• Authors: Wenqiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, Yikai Wang
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.04928
• Pdf URL: https://arxiv.org/pdf/2411.04928
• Copy Paste: [[2411.04928]] DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion(https://arxiv.org/abs/2411.04928)
• Keywords: diffusion
• Abstract: In this paper, we introduce \textbf{DimensionX}, a framework designed to generate photorealistic 3D and 4D scenes from just a single image with video diffusion. Our approach begins with the insight that both the spatial structure of a 3D scene and the temporal evolution of a 4D scene can be effectively represented through sequences of video frames. While recent video diffusion models have shown remarkable success in producing vivid visuals, they face limitations in directly recovering 3D/4D scenes due to limited spatial and temporal controllability during generation. To overcome this, we propose ST-Director, which decouples spatial and temporal factors in video diffusion by learning dimension-aware LoRAs from dimension-variant data. This controllable video diffusion approach enables precise manipulation of spatial structure and temporal dynamics, allowing us to reconstruct both 3D and 4D representations from sequential frames with the combination of spatial and temporal dimensions. Additionally, to bridge the gap between generated videos and real-world scenes, we introduce a trajectory-aware mechanism for 3D generation and an identity-preserving denoising strategy for 4D generation. Extensive experiments on various real-world and synthetic datasets demonstrate that DimensionX achieves superior results in controllable video generation, as well as in 3D and 4D scene generation, compared with previous methods.

Title: Fed-LDR: Federated Local Data-infused Graph Creation with Node-centric Model Refinement

• Authors: Jiechao Gao, Yuangang Li, Syeda Faiza Ahmed
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2411.04936
• Pdf URL: https://arxiv.org/pdf/2411.04936
• Copy Paste: [[2411.04936]] Fed-LDR: Federated Local Data-infused Graph Creation with Node-centric Model Refinement(https://arxiv.org/abs/2411.04936)
• Keywords: privacy, federate
• Abstract: The rapid acceleration of global urbanization has introduced novel challenges in enhancing urban infrastructure and services. Spatio-temporal data, integrating spatial and temporal dimensions, has emerged as a critical tool for understanding urban phenomena and promoting sustainability. In this context, Federated Learning (FL) has gained prominence as a distributed learning paradigm aligned with the privacy requirements of urban IoT environments. However, integrating traditional and deep learning models into the FL framework poses significant challenges, particularly in capturing complex spatio-temporal dependencies and adapting to diverse urban conditions. To address these challenges, we propose the Federated Local Data-Infused Graph Creation with Node-centric Model Refinement (Fed-LDR) algorithm. Fed-LDR leverages FL and Graph Convolutional Networks (GCN) to enhance spatio-temporal data analysis in urban environments. The algorithm comprises two key modules: (1) the Local Data-Infused Graph Creation (LDIGC) module, which dynamically reconfigures adjacency matrices to reflect evolving spatial relationships within urban environments, and (2) the Node-centric Model Refinement (NoMoR) module, which customizes model parameters for individual urban nodes to accommodate heterogeneity. Evaluations on the PeMSD4 and PeMSD8 datasets demonstrate Fed-LDR's superior performance over six baseline methods. Fed-LDR achieved the lowest Mean Absolute Error (MAE) values of 20.15 and 17.30, and the lowest Root Mean Square Error (RMSE) values of 32.30 and 27.15, respectively, while maintaining a high correlation coefficient of 0.96 across both datasets. Notably, on the PeMSD4 dataset, Fed-LDR reduced MAE and RMSE by up to 81\% and 78\%, respectively, compared to the best-performing baseline FedMedian.

Title: M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding

• Authors: Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, Mohit Bansal
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04952
• Pdf URL: https://arxiv.org/pdf/2411.04952
• Copy Paste: [[2411.04952]] M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding(https://arxiv.org/abs/2411.04952)
• Keywords: extraction
• Abstract: Document visual question answering (DocVQA) pipelines that answer questions from documents have broad applications. Existing methods focus on handling single-page documents with multi-modal language models (MLMs), or rely on text-based retrieval-augmented generation (RAG) that uses text extraction tools such as optical character recognition (OCR). However, there are difficulties in applying these methods in real-world scenarios: (a) questions often require information across different pages or documents, where MLMs cannot handle many long documents; (b) documents often have important information in visual elements such as figures, but text extraction tools ignore them. We introduce M3DocRAG, a novel multi-modal RAG framework that flexibly accommodates various document contexts (closed-domain and open-domain), question hops (single-hop and multi-hop), and evidence modalities (text, chart, figure, etc.). M3DocRAG finds relevant documents and answers questions using a multi-modal retriever and an MLM, so that it can efficiently handle single or many documents while preserving visual information. Since previous DocVQA datasets ask questions in the context of a specific document, we also present M3DocVQA, a new benchmark for evaluating open-domain DocVQA over 3,000+ PDF documents with 40,000+ pages. In three benchmarks (M3DocVQA/MMLongBench-Doc/MP-DocVQA), empirical results show that M3DocRAG with ColPali and Qwen2-VL 7B achieves superior performance than many strong baselines, including state-of-the-art performance in MP-DocVQA. We provide comprehensive analyses of different indexing, MLMs, and retrieval models. Lastly, we qualitatively show that M3DocRAG can successfully handle various scenarios, such as when relevant information exists across multiple pages and when answer evidence only exists in images.

Title: CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM

• Authors: Jingwei Xu, Chenyu Wang, Zibo Zhao, Wen Liu, Yi Ma, Shenghua Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04954
• Pdf URL: https://arxiv.org/pdf/2411.04954
• Copy Paste: [[2411.04954]] CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM(https://arxiv.org/abs/2411.04954)
• Keywords: robust, generative, large language model
• Abstract: This paper aims to design a unified Computer-Aided Design (CAD) generation system that can easily generate CAD models based on the user's inputs in the form of textual description, images, point clouds, or even a combination of them. Towards this goal, we introduce the CAD-MLLM, the first system capable of generating parametric CAD models conditioned on the multimodal input. Specifically, within the CAD-MLLM framework, we leverage the command sequences of CAD models and then employ advanced large language models (LLMs) to align the feature space across these diverse multi-modalities data and CAD models' vectorized representations. To facilitate the model training, we design a comprehensive data construction and annotation pipeline that equips each CAD model with corresponding multimodal data. Our resulting dataset, named Omni-CAD, is the first multimodal CAD dataset that contains textual description, multi-view images, points, and command sequence for each CAD model. It contains approximately 450K instances and their CAD construction sequences. To thoroughly evaluate the quality of our generated CAD models, we go beyond current evaluation metrics that focus on reconstruction quality by introducing additional metrics that assess topology quality and surface enclosure extent. Extensive experimental results demonstrate that CAD-MLLM significantly outperforms existing conditional generative methods and remains highly robust to noises and missing points. The project page and more visualizations can be found at: this https URL

Title: Uncovering Hidden Subspaces in Video Diffusion Models Using Re-Identification

• Authors: Mischa Dombrowski, Hadrien Reynaud, Bernhard Kainz
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04956
• Pdf URL: https://arxiv.org/pdf/2411.04956
• Copy Paste: [[2411.04956]] Uncovering Hidden Subspaces in Video Diffusion Models Using Re-Identification(https://arxiv.org/abs/2411.04956)
• Keywords: privacy, diffusion, generative
• Abstract: Latent Video Diffusion Models can easily deceive casual observers and domain experts alike thanks to the produced image quality and temporal consistency. Beyond entertainment, this creates opportunities around safe data sharing of fully synthetic datasets, which are crucial in healthcare, as well as other domains relying on sensitive personal information. However, privacy concerns with this approach have not fully been addressed yet, and models trained on synthetic data for specific downstream tasks still perform worse than those trained on real data. This discrepancy may be partly due to the sampling space being a subspace of the training videos, effectively reducing the training data size for downstream models. Additionally, the reduced temporal consistency when generating long videos could be a contributing factor. In this paper, we first show that training privacy-preserving models in latent space is computationally more efficient and generalize better. Furthermore, to investigate downstream degradation factors, we propose to use a re-identification model, previously employed as a privacy preservation filter. We demonstrate that it is sufficient to train this model on the latent space of the video generator. Subsequently, we use these models to evaluate the subspace covered by synthetic video datasets and thus introduce a new way to measure the faithfulness of generative machine learning models. We focus on a specific application in healthcare echocardiography to illustrate the effectiveness of our novel methods. Our findings indicate that only up to 30.8% of the training videos are learned in latent video diffusion models, which could explain the lack of performance when training downstream tasks on synthetic data.

Title: VAIR: Visuo-Acoustic Implicit Representations for Low-Cost, Multi-Modal Transparent Surface Reconstruction in Indoor Scenes

• Authors: Advaith V. Sethuraman, Onur Bagoren, Harikrishnan Seetharaman, Dalton Richardson, Joseph Taylor, Katherine A. Skinner
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.04963
• Pdf URL: https://arxiv.org/pdf/2411.04963
• Copy Paste: [[2411.04963]] VAIR: Visuo-Acoustic Implicit Representations for Low-Cost, Multi-Modal Transparent Surface Reconstruction in Indoor Scenes(https://arxiv.org/abs/2411.04963)
• Keywords: generative
• Abstract: Mobile robots operating indoors must be prepared to navigate challenging scenes that contain transparent surfaces. This paper proposes a novel method for the fusion of acoustic and visual sensing modalities through implicit neural representations to enable dense reconstruction of transparent surfaces in indoor scenes. We propose a novel model that leverages generative latent optimization to learn an implicit representation of indoor scenes consisting of transparent surfaces. We demonstrate that we can query the implicit representation to enable volumetric rendering in image space or 3D geometry reconstruction (point clouds or mesh) with transparent surface prediction. We evaluate our method's effectiveness qualitatively and quantitatively on a new dataset collected using a custom, low-cost sensing platform featuring RGB-D cameras and ultrasonic sensors. Our method exhibits significant improvement over state-of-the-art for transparent surface reconstruction.

Title: BitNet a4.8: 4-bit Activations for 1-bit LLMs

• Authors: Hongyu Wang, Shuming Ma, Furu Wei
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04965
• Pdf URL: https://arxiv.org/pdf/2411.04965
• Copy Paste: [[2411.04965]] BitNet a4.8: 4-bit Activations for 1-bit LLMs(https://arxiv.org/abs/2411.04965)
• Keywords: large language model
• Abstract: Recent research on the 1-bit Large Language Models (LLMs), such as BitNet b1.58, presents a promising direction for reducing the inference cost of LLMs while maintaining their performance. In this work, we introduce BitNet a4.8, enabling 4-bit activations for 1-bit LLMs. BitNet a4.8 employs a hybrid quantization and sparsification strategy to mitigate the quantization errors introduced by the outlier channels. Specifically, we utilize 4-bit activations for inputs to the attention and feed-forward network layers, while sparsifying intermediate states followed with 8-bit quantization. Extensive experiments demonstrate that BitNet a4.8 achieves performance comparable to BitNet b1.58 with equivalent training costs, while being faster in inference with enabling 4-bit (INT4/FP4) kernels. Additionally, BitNet a4.8 activates only 55% of parameters and supports 3-bit KV cache, further enhancing the efficiency of large-scale LLM deployment and inference.

Title: AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation

• Authors: Anil Kag, Huseyin Coskun, Jierun Chen, Junli Cao, Willi Menapace, Aliaksandr Siarohin, Sergey Tulyakov, Jian Ren
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04967
• Pdf URL: https://arxiv.org/pdf/2411.04967
• Copy Paste: [[2411.04967]] AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation(https://arxiv.org/abs/2411.04967)
• Keywords: transformer, segmentation
• Abstract: Neural network architecture design requires making many crucial decisions. The common desiderata is that similar decisions, with little modifications, can be reused in a variety of tasks and applications. To satisfy that, architectures must provide promising latency and performance trade-offs, support a variety of tasks, scale efficiently with respect to the amounts of data and compute, leverage available data from other tasks, and efficiently support various hardware. To this end, we introduce AsCAN -- a hybrid architecture, combining both convolutional and transformer blocks. We revisit the key design principles of hybrid architectures and propose a simple and effective \emph{asymmetric} architecture, where the distribution of convolutional and transformer blocks is \emph{asymmetric}, containing more convolutional blocks in the earlier stages, followed by more transformer blocks in later stages. AsCAN supports a variety of tasks: recognition, segmentation, class-conditional image generation, and features a superior trade-off between performance and latency. We then scale the same architecture to solve a large-scale text-to-image task and show state-of-the-art performance compared to the most recent public and commercial models. Notably, even without any computation optimization for transformer blocks, our models still yield faster inference speed than existing works featuring efficient attention mechanisms, highlighting the advantages and the value of our approach.

Title: SuffixDecoding: A Model-Free Approach to Speeding Up Large Language Model Inference

• Authors: Gabriele Oliaro, Zhihao Jia, Daniel Campos, Aurick Qiao
• Subjects: cs.CL, cs.AI, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04975
• Pdf URL: https://arxiv.org/pdf/2411.04975
• Copy Paste: [[2411.04975]] SuffixDecoding: A Model-Free Approach to Speeding Up Large Language Model Inference(https://arxiv.org/abs/2411.04975)
• Keywords: large language model
• Abstract: We present SuffixDecoding, a novel model-free approach to accelerating large language model (LLM) inference through speculative decoding. Unlike existing methods that rely on draft models or specialized decoding heads, SuffixDecoding leverages suffix trees built from previously generated outputs to efficiently predict candidate token sequences. Our approach enables flexible tree-structured speculation without the overhead of maintaining and orchestrating additional models. SuffixDecoding builds and dynamically updates suffix trees to capture patterns in the generated text, using them to construct speculation trees through a principled scoring mechanism based on empirical token frequencies. SuffixDecoding requires only CPU memory which is plentiful and underutilized on typical LLM serving nodes. We demonstrate that SuffixDecoding achieves competitive speedups compared to model-based approaches across diverse workloads including open-domain chat, code generation, and text-to-SQL tasks. For open-ended chat and code generation tasks, SuffixDecoding achieves up to $1.4\times$ higher output throughput than SpecInfer and up to $1.1\times$ lower time-per-token (TPOT) latency. For a proprietary multi-LLM text-to-SQL application, SuffixDecoding achieves up to $2.9\times$ higher output throughput and $3\times$ lower latency than speculative decoding. Our evaluation shows that SuffixDecoding maintains high acceptance rates even with small reference corpora of 256 examples, while continuing to improve performance as more historical outputs are incorporated.

Title: Enhancing Reverse Engineering: Investigating and Benchmarking Large Language Models for Vulnerability Analysis in Decompiled Binaries

• Authors: Dylan Manuel, Nafis Tanveer Islam, Joseph Khoury, Ana Nunez, Elias Bou-Harb, Peyman Najafirad
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.04981
• Pdf URL: https://arxiv.org/pdf/2411.04981
• Copy Paste: [[2411.04981]] Enhancing Reverse Engineering: Investigating and Benchmarking Large Language Models for Vulnerability Analysis in Decompiled Binaries(https://arxiv.org/abs/2411.04981)
• Keywords: security, large language model
• Abstract: Security experts reverse engineer (decompile) binary code to identify critical security vulnerabilities. The limited access to source code in vital systems - such as firmware, drivers, and proprietary software used in Critical Infrastructures (CI) - makes this analysis even more crucial on the binary level. Even with available source code, a semantic gap persists after compilation between the source and the binary code executed by the processor. This gap may hinder the detection of vulnerabilities in source code. That being said, current research on Large Language Models (LLMs) overlooks the significance of decompiled binaries in this area by focusing solely on source code. In this work, we are the first to empirically uncover the substantial semantic limitations of state-of-the-art LLMs when it comes to analyzing vulnerabilities in decompiled binaries, largely due to the absence of relevant datasets. To bridge the gap, we introduce DeBinVul, a novel decompiled binary code vulnerability dataset. Our dataset is multi-architecture and multi-optimization, focusing on C/C++ due to their wide usage in CI and association with numerous vulnerabilities. Specifically, we curate 150,872 samples of vulnerable and non-vulnerable decompiled binary code for the task of (i) identifying; (ii) classifying; (iii) describing vulnerabilities; and (iv) recovering function names in the domain of decompiled binaries. Subsequently, we fine-tune state-of-the-art LLMs using DeBinVul and report on a performance increase of 19%, 24%, and 21% in the capabilities of CodeLlama, Llama3, and CodeGen2 respectively, in detecting binary code vulnerabilities. Additionally, using DeBinVul, we report a high performance of 80-90% on the vulnerability classification task. Furthermore, we report improved performance in function name recovery and vulnerability description tasks.

Title: SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation

• Authors: Koichi Namekata, Sherwin Bahmani, Ziyi Wu, Yash Kant, Igor Gilitschenski, David B. Lindell
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04989
• Pdf URL: https://arxiv.org/pdf/2411.04989
• Copy Paste: [[2411.04989]] SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation(https://arxiv.org/abs/2411.04989)
• Keywords: diffusion
• Abstract: Methods for image-to-video generation have achieved impressive, photo-realistic quality. However, adjusting specific elements in generated videos, such as object motion or camera movement, is often a tedious process of trial and error, e.g., involving re-generating videos with different random seeds. Recent techniques address this issue by fine-tuning a pre-trained model to follow conditioning signals, such as bounding boxes or point trajectories. Yet, this fine-tuning procedure can be computationally expensive, and it requires datasets with annotated object motion, which can be difficult to procure. In this work, we introduce SG-I2V, a framework for controllable image-to-video generation that is self-guided$\unicode{x2013}$offering zero-shot control by relying solely on the knowledge present in a pre-trained image-to-video diffusion model without the need for fine-tuning or external knowledge. Our zero-shot method outperforms unsupervised baselines while being competitive with supervised models in terms of visual quality and motion fidelity.

Title: Clustering in Causal Attention Masking

• Authors: Nikita Karagodin, Yury Polyanskiy, Philippe Rigollet
• Subjects: cs.LG, cs.AI, math.AP, math.DS
• Abstract URL: https://arxiv.org/abs/2411.04990
• Pdf URL: https://arxiv.org/pdf/2411.04990
• Copy Paste: [[2411.04990]] Clustering in Causal Attention Masking(https://arxiv.org/abs/2411.04990)
• Keywords: transformer, generative
• Abstract: This work presents a modification of the self-attention dynamics proposed by Geshkovski et al. (arXiv:2312.10794) to better reflect the practically relevant, causally masked attention used in transformer architectures for generative AI. This modification translates into an interacting particle system that cannot be interpreted as a mean-field gradient flow. Despite this loss of structure, we significantly strengthen the results of Geshkovski et al. (arXiv:2312.10794) in this context: While previous rigorous results focused on cases where all three matrices (Key, Query, and Value) were scaled identities, we prove asymptotic convergence to a single cluster for arbitrary key-query matrices and a value matrix equal to the identity. Additionally, we establish a connection to the classical Rényi parking problem from combinatorial geometry to make initial theoretical steps towards demonstrating the existence of meta-stable states.

Title: Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models

• Authors: Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, Xi Victoria Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04996
• Pdf URL: https://arxiv.org/pdf/2411.04996
• Copy Paste: [[2411.04996]] Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models(https://arxiv.org/abs/2411.04996)
• Keywords: transformer, large language model
• Abstract: The development of large language models (LLMs) has expanded to multi-modal systems capable of processing text, images, and speech within a unified framework. Training these models demands significantly larger datasets and computational resources compared to text-only LLMs. To address the scaling challenges, we introduce Mixture-of-Transformers (MoT), a sparse multi-modal transformer architecture that significantly reduces pretraining computational costs. MoT decouples non-embedding parameters of the model by modality -- including feed-forward networks, attention matrices, and layer normalization -- enabling modality-specific processing with global self-attention over the full input sequence. We evaluate MoT across multiple settings and model scales. In the Chameleon 7B setting (autoregressive text-and-image generation), MoT matches the dense baseline's performance using only 55.8\% of the FLOPs. When extended to include speech, MoT reaches speech performance comparable to the dense baseline with only 37.2\% of the FLOPs. In the Transfusion setting, where text and image are trained with different objectives, a 7B MoT model matches the image modality performance of the dense baseline with one third of the FLOPs, and a 760M MoT model outperforms a 1.4B dense baseline across key image generation metrics. System profiling further highlights MoT's practical benefits, achieving dense baseline image quality in 47.2\% of the wall-clock time and text quality in 75.6\% of the wall-clock time (measured on AWS p4de.24xlarge instances with NVIDIA A100 GPUs).

Title: LLM2CLIP: Powerful Language Model Unlock Richer Visual Representation

• Authors: Weiquan Huang, Aoqi Wu, Yifan Yang, Xufang Luo, Yuqing Yang, Liang Hu, Qi Dai, Xiyang Dai, Dongdong Chen, Chong Luo, Lili Qiu
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04997
• Pdf URL: https://arxiv.org/pdf/2411.04997
• Copy Paste: [[2411.04997]] LLM2CLIP: Powerful Language Model Unlock Richer Visual Representation(https://arxiv.org/abs/2411.04997)
• Keywords: large language model
• Abstract: CLIP is one of the most important multimodal foundational models today. What powers CLIP's capabilities? The rich supervision signals provided by natural language, the carrier of human knowledge, shape a powerful cross-modal representation space. However, with the rapid advancements in large language models LLMs like GPT-4 and LLaMA, the boundaries of language comprehension and generation are continually being pushed. This raises an intriguing question: can the capabilities of LLMs be harnessed to further improve multimodal representation learning? The potential benefits of incorporating LLMs into CLIP are clear. LLMs' strong textual understanding can fundamentally improve CLIP's ability to handle image captions, drastically enhancing its ability to process long and complex texts, a well-known limitation of vanilla CLIP. Moreover, LLMs are trained on a vast corpus of text, possessing open-world knowledge. This allows them to expand on caption information during training, increasing the efficiency of the learning process. In this paper, we propose LLM2CLIP, a novel approach that embraces the power of LLMs to unlock CLIP's potential. By fine-tuning the LLM in the caption space with contrastive learning, we extract its textual capabilities into the output embeddings, significantly improving the output layer's textual discriminability. We then design an efficient training process where the fine-tuned LLM acts as a powerful teacher for CLIP's visual encoder. Thanks to the LLM's presence, we can now incorporate longer and more complex captions without being restricted by vanilla CLIP's text encoder's context window and ability limitations. Our experiments demonstrate that this approach brings substantial improvements in cross-modal tasks.

Title: Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?

• Authors: Jonathan Roberts, Kai Han, Samuel Albanie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.05000
• Pdf URL: https://arxiv.org/pdf/2411.05000
• Copy Paste: [[2411.05000]] Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?(https://arxiv.org/abs/2411.05000)
• Keywords: large language model
• Abstract: As the context limits of Large Language Models (LLMs) increase, the range of possible applications and downstream functions broadens. In many real-world tasks, decisions depend on details scattered across collections of often disparate documents containing mostly irrelevant information. Long-context LLMs appear well-suited to this form of complex information retrieval and reasoning, which has traditionally proven costly and time-consuming. However, although the development of longer context models has seen rapid gains in recent years, our understanding of how effectively LLMs use their context has not kept pace. To address this, we conduct a set of retrieval experiments designed to evaluate the capabilities of 17 leading LLMs, such as their ability to follow threads of information through the context window. Strikingly, we find that many models are remarkably threadsafe: capable of simultaneously following multiple threads without significant loss in performance. Still, for many models, we find the effective context limit is significantly shorter than the supported context length, with accuracy decreasing as the context window grows. Our study also highlights the important point that token counts from different tokenizers should not be directly compared -- they often correspond to substantially different numbers of written characters. We release our code and long-context experimental data.

Title: Analyzing The Language of Visual Tokens

• Authors: David M. Chan, Rodolfo Corona, Joonyong Park, Cheol Jun Cho, Yutong Bai, Trevor Darrell
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05001
• Pdf URL: https://arxiv.org/pdf/2411.05001
• Copy Paste: [[2411.05001]] Analyzing The Language of Visual Tokens(https://arxiv.org/abs/2411.05001)
• Keywords: transformer
• Abstract: With the introduction of transformer-based models for vision and language tasks, such as LLaVA and Chameleon, there has been renewed interest in the discrete tokenized representation of images. These models often treat image patches as discrete tokens, analogous to words in natural language, learning joint alignments between visual and human languages. However, little is known about the statistical behavior of these visual languages - whether they follow similar frequency distributions, grammatical structures, or topologies as natural languages. In this paper, we take a natural-language-centric approach to analyzing discrete visual languages and uncover striking similarities and fundamental differences. We demonstrate that, although visual languages adhere to Zipfian distributions, higher token innovation drives greater entropy and lower compression, with tokens predominantly representing object parts, indicating intermediate granularity. We also show that visual languages lack cohesive grammatical structures, leading to higher perplexity and weaker hierarchical organization compared to natural languages. Finally, we demonstrate that, while vision models align more closely with natural languages than other models, this alignment remains significantly weaker than the cohesion found within natural languages. Through these experiments, we demonstrate how understanding the statistical properties of discrete visual languages can inform the design of more effective computer vision models.

Title: ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning

• Authors: David Junhao Zhang, Roni Paiss, Shiran Zada, Nikhil Karnad, David E. Jacobs, Yael Pritch, Inbar Mosseri, Mike Zheng Shou, Neal Wadhwa, Nataniel Ruiz
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05003
• Pdf URL: https://arxiv.org/pdf/2411.05003
• Copy Paste: [[2411.05003]] ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning(https://arxiv.org/abs/2411.05003)
• Keywords: diffusion, generative
• Abstract: Recently, breakthroughs in video modeling have allowed for controllable camera trajectories in generated videos. However, these methods cannot be directly applied to user-provided videos that are not generated by a video model. In this paper, we present ReCapture, a method for generating new videos with novel camera trajectories from a single user-provided video. Our method allows us to re-generate the reference video, with all its existing scene motion, from vastly different angles and with cinematic camera motion. Notably, using our method we can also plausibly hallucinate parts of the scene that were not observable in the reference video. Our method works by (1) generating a noisy anchor video with a new camera trajectory using multiview diffusion models or depth-based point cloud rendering and then (2) regenerating the anchor video into a clean and temporally consistent reangled video using our proposed masked video fine-tuning technique.

Title: Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models

• Authors: Shuhong Zheng, Zhipeng Bao, Ruoyu Zhao, Martial Hebert, Yu-Xiong Wang
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.05005
• Pdf URL: https://arxiv.org/pdf/2411.05005
• Copy Paste: [[2411.05005]] Diff-2-in-1: Bridging Generation and Dense Perception with Diffusion Models(https://arxiv.org/abs/2411.05005)
• Keywords: diffusion
• Abstract: Beyond high-fidelity image synthesis, diffusion models have recently exhibited promising results in dense visual perception tasks. However, most existing work treats diffusion models as a standalone component for perception tasks, employing them either solely for off-the-shelf data augmentation or as mere feature extractors. In contrast to these isolated and thus sub-optimal efforts, we introduce a unified, versatile, diffusion-based framework, Diff-2-in-1, that can simultaneously handle both multi-modal data generation and dense visual perception, through a unique exploitation of the diffusion-denoising process. Within this framework, we further enhance discriminative visual perception via multi-modal generation, by utilizing the denoising network to create multi-modal data that mirror the distribution of the original training set. Importantly, Diff-2-in-1 optimizes the utilization of the created diverse and faithful data by leveraging a novel self-improving learning mechanism. Comprehensive experimental evaluations validate the effectiveness of our framework, showcasing consistent performance improvements across various discriminative backbones and high-quality multi-modal data generation characterized by both realism and usefulness.

Title: ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing

• Authors: Jun-Kun Chen, Yu-Xiong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.05006
• Pdf URL: https://arxiv.org/pdf/2411.05006
• Copy Paste: [[2411.05006]] ProEdit: Simple Progression is All You Need for High-Quality 3D Scene Editing(https://arxiv.org/abs/2411.05006)
• Keywords: diffusion
• Abstract: This paper proposes ProEdit - a simple yet effective framework for high-quality 3D scene editing guided by diffusion distillation in a novel progressive manner. Inspired by the crucial observation that multi-view inconsistency in scene editing is rooted in the diffusion model's large feasible output space (FOS), our framework controls the size of FOS and reduces inconsistency by decomposing the overall editing task into several subtasks, which are then executed progressively on the scene. Within this framework, we design a difficulty-aware subtask decomposition scheduler and an adaptive 3D Gaussian splatting (3DGS) training strategy, ensuring high quality and efficiency in performing each subtask. Extensive evaluation shows that our ProEdit achieves state-of-the-art results in various scenes and challenging editing tasks, all through a simple framework without any expensive or sophisticated add-ons like distillation losses, components, or training procedures. Notably, ProEdit also provides a new way to control, preview, and select the "aggressivity" of editing operation during the editing process.

Title: SVDQunat: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

• Authors: Muyang Li, Yujun Lin, Zhekai Zhang, Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng, Jun-Yan Zhu, Song Han
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.05007
• Pdf URL: https://arxiv.org/pdf/2411.05007
• Copy Paste: [[2411.05007]] SVDQunat: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models(https://arxiv.org/abs/2411.05007)
• Keywords: diffusion, large language model
• Abstract: Diffusion models have been proven highly effective at generating high-quality images. However, as these models grow larger, they require significantly more memory and suffer from higher latency, posing substantial challenges for deployment. In this work, we aim to accelerate diffusion models by quantizing their weights and activations to 4 bits. At such an aggressive level, both weights and activations are highly sensitive, where conventional post-training quantization methods for large language models like smoothing become insufficient. To overcome this limitation, we propose SVDQuant, a new 4-bit quantization paradigm. Different from smoothing which redistributes outliers between weights and activations, our approach absorbs these outliers using a low-rank branch. We first consolidate the outliers by shifting them from activations to weights, then employ a high-precision low-rank branch to take in the weight outliers with Singular Value Decomposition (SVD). This process eases the quantization on both sides. However, na\"ıvely running the low-rank branch independently incurs significant overhead due to extra data movement of activations, negating the quantization speedup. To address this, we co-design an inference engine Nunchaku that fuses the kernels of the low-rank branch into those of the low-bit branch to cut off redundant memory access. It can also seamlessly support off-the-shelf low-rank adapters (LoRAs) without the need for re-quantization. Extensive experiments on SDXL, PixArt-$\Sigma$, and FLUX.1 validate the effectiveness of SVDQuant in preserving image quality. We reduce the memory usage for the 12B FLUX.1 models by 3.5$\times$, achieving 3.0$\times$ speedup over the 4-bit weight-only quantized baseline on the 16GB laptop 4090 GPU, paving the way for more interactive applications on PCs. Our quantization library and inference engine are open-sourced.