2024-12-19

Title: Training Verification-Friendly Neural Networks via Neuron Behavior Consistency

• Authors: Zongxin Liu, Zhe Zhao, Fu Song, Jun Sun, Pengfei Yang, Xiaowei Huang, Lijun Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13229
• Pdf URL: https://arxiv.org/pdf/2412.13229
• Copy Paste: [[2412.13229]] Training Verification-Friendly Neural Networks via Neuron Behavior Consistency(https://arxiv.org/abs/2412.13229)
• Keywords: security, robust
• Abstract: Formal verification provides critical security assurances for neural networks, yet its practical application suffers from the long verification time. This work introduces a novel method for training verification-friendly neural networks, which are robust, easy to verify, and relatively accurate. Our method integrates neuron behavior consistency into the training process, making neuron activation states consistent across different inputs in a local neighborhood, reducing the number of unstable neurons and tightening the bounds of neurons thereby enhancing neural network verifiability. We evaluated our method using the MNIST, Fashion-MNIST, and CIFAR-10 datasets across various network architectures. The results of the experiment demonstrate that networks trained using our method are verification-friendly across different radii and different model architectures, whereas other tools fail to maintain verifiability as the radius increases. We also show that our method can be combined with existing methods to further improve the verifiability of networks.

Title: Content-aware Balanced Spectrum Encoding in Masked Modeling for Time Series Classification

• Authors: Yudong Han, Haocong Wang, Yupeng Hu, Yongshun Gong, Xuemeng Song, Weili Guan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13232
• Pdf URL: https://arxiv.org/pdf/2412.13232
• Copy Paste: [[2412.13232]] Content-aware Balanced Spectrum Encoding in Masked Modeling for Time Series Classification(https://arxiv.org/abs/2412.13232)
• Keywords: transformer
• Abstract: Due to the superior ability of global dependency, transformer and its variants have become the primary choice in Masked Time-series Modeling (MTM) towards time-series classification task. In this paper, we experimentally analyze that existing transformer-based MTM methods encounter with two under-explored issues when dealing with time series data: (1) they encode features by performing long-dependency ensemble averaging, which easily results in rank collapse and feature homogenization as the layer goes deeper; (2) they exhibit distinct priorities in fitting different frequency components contained in the time-series, inevitably leading to spectrum energy imbalance of encoded feature. To tackle these issues, we propose an auxiliary content-aware balanced decoder (CBD) to optimize the encoding quality in the spectrum space within masked modeling scheme. Specifically, the CBD iterates on a series of fundamental blocks, and thanks to two tailored units, each block could progressively refine the masked representation via adjusting the interaction pattern based on local content variations of time-series and learning to recalibrate the energy distribution across different frequency components. Moreover, a dual-constraint loss is devised to enhance the mutual optimization of vanilla decoder and our CBD. Extensive experimental results on ten time-series classification datasets show that our method nearly surpasses a bunch of baselines. Meanwhile, a series of explanatory results are showcased to sufficiently demystify the behaviors of our method.

Title: Enhancing Internet of Things Security throughSelf-Supervised Graph Neural Networks

• Authors: Safa Ben Atitallah, Maha Driss, Wadii Boulila, Anis Koubaa
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13240
• Pdf URL: https://arxiv.org/pdf/2412.13240
• Copy Paste: [[2412.13240]] Enhancing Internet of Things Security throughSelf-Supervised Graph Neural Networks(https://arxiv.org/abs/2412.13240)
• Keywords: security, attack, robust
• Abstract: With the rapid rise of the Internet of Things (IoT), ensuring the security of IoT devices has become essential. One of the primary challenges in this field is that new types of attacks often have significantly fewer samples than more common attacks, leading to unbalanced datasets. Existing research on detecting intrusions in these unbalanced labeled datasets primarily employs Convolutional Neural Networks (CNNs) or conventional Machine Learning (ML) models, which result in incomplete detection, especially for new attacks. To handle these challenges, we suggest a new approach to IoT intrusion detection using Self-Supervised Learning (SSL) with a Markov Graph Convolutional Network (MarkovGCN). Graph learning excels at modeling complex relationships within data, while SSL mitigates the issue of limited labeled data for emerging attacks. Our approach leverages the inherent structure of IoT networks to pre-train a GCN, which is then fine-tuned for the intrusion detection task. The integration of Markov chains in GCN uncovers network structures and enriches node and edge features with contextual information. Experimental results demonstrate that our approach significantly improves detection accuracy and robustness compared to conventional supervised learning methods. Using the EdgeIIoT-set dataset, we attained an accuracy of 98.68\%, a precision of 98.18%, a recall of 98.35%, and an F1-Score of 98.40%.

Title: TETRIS: Composing FHE Techniques for Private Functional Exploration Over Large Datasets

• Authors: Malika Izabachène, Jean-Philippe Bossuat
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.13269
• Pdf URL: https://arxiv.org/pdf/2412.13269
• Copy Paste: [[2412.13269]] TETRIS: Composing FHE Techniques for Private Functional Exploration Over Large Datasets(https://arxiv.org/abs/2412.13269)
• Keywords: secure, privacy
• Abstract: To derive valuable insights from statistics, machine learning applications frequently analyze substantial amounts of data. In this work, we address the problem of designing efficient secure techniques to probe large datasets which allow a scientist to conduct large-scale medical studies over specific attributes of patients' records, while maintaining the privacy of his model. We introduce a set of composable homomorphic operations and show how to combine private functions evaluation with private thresholds via approximate fully homomorphic encryption. This allows us to design a new system named TETRIS, which solves the real-world use case of private functional exploration of large databases, where the statistical criteria remain private to the server owning the patients' records. Our experiments show that TETRIS achieves practical performance over a large dataset of patients even for the evaluation of elaborate statements composed of linear and nonlinear functions. It is possible to extract private insights from a database of hundreds of thousands of patient records within only a few minutes on a single thread, with an amortized time per database entry smaller than 2ms.

Title: Enhancing Persona Classification in Dialogue Systems: A Graph Neural Network Approach

• Authors: Konstantin Zaitsev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13283
• Pdf URL: https://arxiv.org/pdf/2412.13283
• Copy Paste: [[2412.13283]] Enhancing Persona Classification in Dialogue Systems: A Graph Neural Network Approach(https://arxiv.org/abs/2412.13283)
• Keywords: large language model
• Abstract: In recent years, Large Language Models (LLMs) gain considerable attention for their potential to enhance personalized experiences in virtual assistants and chatbots. A key area of interest is the integration of personas into LLMs to improve dialogue naturalness and user engagement. This study addresses the challenge of persona classification, a crucial component in dialogue understanding, by proposing a framework that combines text embeddings with Graph Neural Networks (GNNs) for effective persona classification. Given the absence of dedicated persona classification datasets, we create a manually annotated dataset to facilitate model training and evaluation. Our method involves extracting semantic features from persona statements using text embeddings and constructing a graph where nodes represent personas and edges capture their similarities. The GNN component uses this graph structure to propagate relevant information, thereby improving classification performance. Experimental results show that our approach, in particular the integration of GNNs, significantly improves classification performance, especially with limited data. Our contributions include the development of a persona classification framework and the creation of a dataset.

Title: Posterior Mean Matching: Generative Modeling through Online Bayesian Inference

• Authors: Sebastian Salazar, Michal Kucer, Yixin Wang, Emily Casleton, David Blei
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.13286
• Pdf URL: https://arxiv.org/pdf/2412.13286
• Copy Paste: [[2412.13286]] Posterior Mean Matching: Generative Modeling through Online Bayesian Inference(https://arxiv.org/abs/2412.13286)
• Keywords: diffusion, generative
• Abstract: This paper introduces posterior mean matching (PMM), a new method for generative modeling that is grounded in Bayesian inference. PMM uses conjugate pairs of distributions to model complex data of various modalities like images and text, offering a flexible alternative to existing methods like diffusion models. PMM models iteratively refine noisy approximations of the target distribution using updates from online Bayesian inference. PMM is flexible because its mechanics are based on general Bayesian models. We demonstrate this flexibility by developing specialized examples: a generative PMM model of real-valued data using the Normal-Normal model, a generative PMM model of count data using a Gamma-Poisson model, and a generative PMM model of discrete data using a Dirichlet-Categorical model. For the Normal-Normal PMM model, we establish a direct connection to diffusion models by showing that its continuous-time formulation converges to a stochastic differential equation (SDE). Additionally, for the Gamma-Poisson PMM, we derive a novel SDE driven by a Cox process, which is a significant departure from traditional Brownian motion-based generative models. PMMs achieve performance that is competitive with generative models for language modeling and image generation.

Title: Hint Marginalization for Improved Reasoning in Large Language Models

• Authors: Soumyasundar Pal, Didier Chételat, Yingxue Zhang, Mark Coates
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13292
• Pdf URL: https://arxiv.org/pdf/2412.13292
• Copy Paste: [[2412.13292]] Hint Marginalization for Improved Reasoning in Large Language Models(https://arxiv.org/abs/2412.13292)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have exhibited an impressive capability to perform reasoning tasks, especially if they are encouraged to generate a sequence of intermediate steps. Reasoning performance can be improved by suitably combining multiple LLM responses, generated either in parallel in a single query, or via sequential interactions with LLMs throughout the reasoning process. Existing strategies for combination, such as self-consistency and progressive-hint-prompting, make inefficient usage of the LLM responses. We present Hint Marginalization, a novel and principled algorithmic framework to enhance the reasoning capabilities of LLMs. Our approach can be viewed as an iterative sampling strategy for forming a Monte Carlo approximation of an underlying distribution of answers, with the goal of identifying the mode the most likely answer. Empirical evaluation on several benchmark datasets for arithmetic reasoning demonstrates the superiority of the proposed approach.

Title: BadSAD: Clean-Label Backdoor Attacks against Deep Semi-Supervised Anomaly Detection

• Authors: He Cheng, Depeng Xu, Shuhan Yuan
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.13324
• Pdf URL: https://arxiv.org/pdf/2412.13324
• Copy Paste: [[2412.13324]] BadSAD: Clean-Label Backdoor Attacks against Deep Semi-Supervised Anomaly Detection(https://arxiv.org/abs/2412.13324)
• Keywords: security, attack
• Abstract: Image anomaly detection (IAD) is essential in applications such as industrial inspection, medical imaging, and security. Despite the progress achieved with deep learning models like Deep Semi-Supervised Anomaly Detection (DeepSAD), these models remain susceptible to backdoor attacks, presenting significant security challenges. In this paper, we introduce BadSAD, a novel backdoor attack framework specifically designed to target DeepSAD models. Our approach involves two key phases: trigger injection, where subtle triggers are embedded into normal images, and latent space manipulation, which positions and clusters the poisoned images near normal images to make the triggers appear benign. Extensive experiments on benchmark datasets validate the effectiveness of our attack strategy, highlighting the severe risks that backdoor attacks pose to deep learning-based anomaly detection systems.

Title: Experience of Training a 1.7B-Parameter LLaMa Model From Scratch

• Authors: Miles Q. Li, Benjamin C. M. Fung, Shih-Chia Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13335
• Pdf URL: https://arxiv.org/pdf/2412.13335
• Copy Paste: [[2412.13335]] Experience of Training a 1.7B-Parameter LLaMa Model From Scratch(https://arxiv.org/abs/2412.13335)
• Keywords: large language model
• Abstract: Pretraining large language models is a complex endeavor influenced by multiple factors, including model architecture, data quality, training continuity, and hardware constraints. In this paper, we share insights gained from the experience of training DMaS-LLaMa-Lite, a fully open source, 1.7-billion-parameter, LLaMa-based model, on approximately 20 billion tokens of carefully curated data. We chronicle the full training trajectory, documenting how evolving validation loss levels and downstream benchmarks reflect transitions from incoherent text to fluent, contextually grounded output. Beyond standard quantitative metrics, we highlight practical considerations such as the importance of restoring optimizer states when resuming from checkpoints, and the impact of hardware changes on training stability and throughput. While qualitative evaluation provides an intuitive understanding of model improvements, our analysis extends to various performance benchmarks, demonstrating how high-quality data and thoughtful scaling enable competitive results with significantly fewer training tokens. By detailing these experiences and offering training logs, checkpoints, and sample outputs, we aim to guide future researchers and practitioners in refining their pretraining strategies. The training script is available on Github at this https URL. The model checkpoints are available on Huggingface at this https URL.

Title: Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs

• Authors: Aldo Pareja, Nikhil Shivakumar Nayak, Hao Wang, Krishnateja Killamsetty, Shivchander Sudalairaj, Wenlong Zhao, Seungwook Han, Abhishek Bhandwaldar, Guangxuan Xu, Kai Xu, Ligong Han, Luke Inglis, Akash Srivastava
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.13337
• Pdf URL: https://arxiv.org/pdf/2412.13337
• Copy Paste: [[2412.13337]] Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs(https://arxiv.org/abs/2412.13337)
• Keywords: robust, large language model
• Abstract: The rise of large language models (LLMs) has created a significant disparity: industrial research labs with their computational resources, expert teams, and advanced infrastructures, can effectively fine-tune LLMs, while individual developers and small organizations face barriers due to limited resources. In this paper, we aim to bridge this gap by presenting a comprehensive study on supervised fine-tuning of LLMs using instruction-tuning datasets spanning diverse knowledge domains and skills. We focus on small-sized LLMs (3B to 7B parameters) for their cost-efficiency and accessibility. We explore various training configurations and strategies across four open-source pre-trained models. We provide detailed documentation of these configurations, revealing findings that challenge several common training practices, including hyperparameter recommendations from TULU and phased training recommended by Orca. Key insights from our work include: (i) larger batch sizes paired with lower learning rates lead to improved model performance on benchmarks such as MMLU, MTBench, and Open LLM Leaderboard; (ii) early-stage training dynamics, such as lower gradient norms and higher loss values, are strong indicators of better final model performance, enabling early termination of sub-optimal runs and significant computational savings; (iii) through a thorough exploration of hyperparameters like warmup steps and learning rate schedules, we provide guidance for practitioners and find that certain simplifications do not compromise performance; and (iv) we observed no significant difference in performance between phased and stacked training strategies, but stacked training is simpler and more sample efficient. With these findings holding robustly across datasets and models, we hope this study serves as a guide for practitioners fine-tuning small LLMs and promotes a more inclusive environment for LLM research.

Title: Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing

• Authors: Keltin Grimes, Marco Christiani, David Shriver, Marissa Connor
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13341
• Pdf URL: https://arxiv.org/pdf/2412.13341
• Copy Paste: [[2412.13341]] Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing(https://arxiv.org/abs/2412.13341)
• Keywords: attack, large language model
• Abstract: Model editing methods modify specific behaviors of Large Language Models by altering a small, targeted set of network weights and require very little data and compute. These methods can be used for malicious applications such as inserting misinformation or simple trojans that result in adversary-specified behaviors when a trigger word is present. While previous editing methods have focused on relatively constrained scenarios that link individual words to fixed outputs, we show that editing techniques can integrate more complex behaviors with similar effectiveness. We develop Concept-ROT, a model editing-based method that efficiently inserts trojans which not only exhibit complex output behaviors, but also trigger on high-level concepts -- presenting an entirely new class of trojan attacks. Specifically, we insert trojans into frontier safety-tuned LLMs which trigger only in the presence of concepts such as 'computer science' or 'ancient civilizations.' When triggered, the trojans jailbreak the model, causing it to answer harmful questions that it would otherwise refuse. Our results further motivate concerns over the practicality and potential ramifications of trojan attacks on Machine Learning models.

Title: Key Exchange in the Quantum Era: Evaluating a Hybrid System of Public-Key Cryptography and Physical-Layer Security

• Authors: Paul Staat, Meik Dörpinghaus, Azadeh Sheikholeslami, Christof Paar, Gerhard Fettweis, Dennis Goeckel
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.13352
• Pdf URL: https://arxiv.org/pdf/2412.13352
• Copy Paste: [[2412.13352]] Key Exchange in the Quantum Era: Evaluating a Hybrid System of Public-Key Cryptography and Physical-Layer Security(https://arxiv.org/abs/2412.13352)
• Keywords: security, attack
• Abstract: Today's information society relies on cryptography to achieve security goals such as confidentiality, integrity, authentication, and non-repudiation for digital communications. Here, public-key cryptosystems play a pivotal role to share encryption keys and create digital signatures. However, quantum computers threaten the security of traditional public-key cryptosystems as they can tame computational problems underlying the schemes, i.e., discrete logarithm and integer factorization. The prospective arrival of capable-enough quantum computers already threatens today's secret communication in terms of their long-term secrecy when stored to be later decrypted. Therefore, researchers strive to develop and deploy alternative schemes. In this work, evaluate a key exchange protocol based on combining public-key schemes with physical-layer security, anticipating the prospect of quantum attacks. If powerful quantum attackers cannot immediately obtain private keys, legitimate parties have a window of short-term secrecy to perform a physical-layer jamming key exchange (JKE) to establish a long-term shared secret. Thereby, the protocol constraints the computation time available to the attacker to break the employed public-key cryptography. In this paper, we outline the protocol, discuss its security, and point out challenges to be resolved.

Title: Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation

• Authors: Samin Mahdizadeh Sani, Pouya Sadeghi, Thuy-Trang Vu, Yadollah Yaghoobzadeh, Gholamreza Haffari
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13375
• Pdf URL: https://arxiv.org/pdf/2412.13375
• Copy Paste: [[2412.13375]] Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation(https://arxiv.org/abs/2412.13375)
• Keywords: large language model
• Abstract: Large language models (LLMs) have made great progress in classification and text generation tasks. However, they are mainly trained on English data and often struggle with low-resource languages. In this study, we explore adding a new language, i.e., Persian, to Llama (a model with a limited understanding of Persian) using parameter-efficient fine-tuning. We employ a multi-stage approach involving pretraining on monolingual Persian data, aligning representations through bilingual pretraining and instruction datasets, and instruction-tuning with task-specific datasets. We evaluate the model's performance at each stage on generation and classification tasks. Our findings suggest that incorporating the Persian language, through bilingual data alignment, can enhance classification accuracy for Persian tasks, with no adverse impact and sometimes even improvements on English tasks. Additionally, the results highlight the model's initial strength as a critical factor when working with limited training data, with cross-lingual alignment offering minimal benefits for the low-resource language. Knowledge transfer from English to Persian has a marginal effect, primarily benefiting simple classification tasks.

Title: Targeted View-Invariant Adversarial Perturbations for 3D Object Recognition

• Authors: Christian Green, Mehmet Ergezer, Abdurrahman Zeybey
• Subjects: cs.CV, cs.AI, cs.CR, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.13376
• Pdf URL: https://arxiv.org/pdf/2412.13376
• Copy Paste: [[2412.13376]] Targeted View-Invariant Adversarial Perturbations for 3D Object Recognition(https://arxiv.org/abs/2412.13376)
• Keywords: attack, robust
• Abstract: Adversarial attacks pose significant challenges in 3D object recognition, especially in scenarios involving multi-view analysis where objects can be observed from varying angles. This paper introduces View-Invariant Adversarial Perturbations (VIAP), a novel method for crafting robust adversarial examples that remain effective across multiple viewpoints. Unlike traditional methods, VIAP enables targeted attacks capable of manipulating recognition systems to classify objects as specific, pre-determined labels, all while using a single universal perturbation. Leveraging a dataset of 1,210 images across 121 diverse rendered 3D objects, we demonstrate the effectiveness of VIAP in both targeted and untargeted settings. Our untargeted perturbations successfully generate a singular adversarial noise robust to 3D transformations, while targeted attacks achieve exceptional results, with top-1 accuracies exceeding 95% across various epsilon values. These findings highlight VIAPs potential for real-world applications, such as testing the robustness of 3D recognition systems. The proposed method sets a new benchmark for view-invariant adversarial robustness, advancing the field of adversarial machine learning for 3D object recognition.

Title: DateLogicQA: Benchmarking Temporal Biases in Large Language Models

• Authors: Gagan Bhatia, MingZe Tang, Cristina Mahanta, Madiha Kazi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13377
• Pdf URL: https://arxiv.org/pdf/2412.13377
• Copy Paste: [[2412.13377]] DateLogicQA: Benchmarking Temporal Biases in Large Language Models(https://arxiv.org/abs/2412.13377)
• Keywords: large language model
• Abstract: This paper introduces DateLogicQA, a benchmark with 190 questions covering diverse date formats, temporal contexts, and reasoning types. We propose the Semantic Integrity Metric to assess tokenization quality and analyse two biases: Representation-Level Bias, affecting embeddings, and Logical-Level Bias, influencing reasoning outputs. Our findings provide a comprehensive evaluation of LLMs' capabilities and limitations in temporal reasoning, highlighting key challenges in handling temporal data accurately. The GitHub repository for our work is available at this https URL

Title: SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits

• Authors: Onkar Thorat, Philippe Laban, Chien-Sheng Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13378
• Pdf URL: https://arxiv.org/pdf/2412.13378
• Copy Paste: [[2412.13378]] SummExecEdit: A Factual Consistency Benchmark in Summarization with Executable Edits(https://arxiv.org/abs/2412.13378)
• Keywords: robust, interpretability
• Abstract: Detecting factual inconsistencies in summarization is critical, yet existing benchmarks lack the necessary challenge and interpretability for robust evaluation. In this paper, we introduce SummExecEdit, a novel benchmark leveraging executable edits to assess models on their ability to both detect factual errors and provide accurate explanations. The top-performing model, Claude3-Opus, achieves a joint detection and explanation score of only 0.49 in our benchmark, with individual scores of 0.67 for detection and 0.73 for explanation. Furthermore, we identify four primary types of explanation errors, with 45.4% of errors focusing on completely unrelated parts of the summary.

Title: An Automated Explainable Educational Assessment System Built on LLMs

• Authors: Jiazheng Li, Artem Bobrov, David West, Cesare Aloisi, Yulan He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13381
• Pdf URL: https://arxiv.org/pdf/2412.13381
• Copy Paste: [[2412.13381]] An Automated Explainable Educational Assessment System Built on LLMs(https://arxiv.org/abs/2412.13381)
• Keywords: robust, explainability, large language model
• Abstract: In this demo, we present AERA Chat, an automated and explainable educational assessment system designed for interactive and visual evaluations of student responses. This system leverages large language models (LLMs) to generate automated marking and rationale explanations, addressing the challenge of limited explainability in automated educational assessment and the high costs associated with annotation. Our system allows users to input questions and student answers, providing educators and researchers with insights into assessment accuracy and the quality of LLM-assessed rationales. Additionally, it offers advanced visualization and robust evaluation tools, enhancing the usability for educational assessment and facilitating efficient rationale verification. Our demo video can be found at this https URL.

Title: Marigold-DC: Zero-Shot Monocular Depth Completion with Guided Diffusion

• Authors: Massimiliano Viola, Kevin Qu, Nando Metzger, Bingxin Ke, Alexander Becker, Konrad Schindler, Anton Obukhov
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13389
• Pdf URL: https://arxiv.org/pdf/2412.13389
• Copy Paste: [[2412.13389]] Marigold-DC: Zero-Shot Monocular Depth Completion with Guided Diffusion(https://arxiv.org/abs/2412.13389)
• Keywords: robust, diffusion
• Abstract: Depth completion upgrades sparse depth measurements into dense depth maps guided by a conventional image. Existing methods for this highly ill-posed task operate in tightly constrained settings and tend to struggle when applied to images outside the training domain or when the available depth measurements are sparse, irregularly distributed, or of varying density. Inspired by recent advances in monocular depth estimation, we reframe depth completion as an image-conditional depth map generation guided by sparse measurements. Our method, Marigold-DC, builds on a pretrained latent diffusion model for monocular depth estimation and injects the depth observations as test-time guidance via an optimization scheme that runs in tandem with the iterative inference of denoising diffusion. The method exhibits excellent zero-shot generalization across a diverse range of environments and handles even extremely sparse guidance effectively. Our results suggest that contemporary monocular depth priors greatly robustify depth completion: it may be better to view the task as recovering dense depth from (dense) image pixels, guided by sparse depth; rather than as inpainting (sparse) depth, guided by an image. Project website: this https URL

Title: MMHMR: Generative Masked Modeling for Hand Mesh Recovery

• Authors: Muhammad Usama Saleem, Ekkasit Pinyoanuntapong, Mayur Jagdishbhai Patel, Hongfei Xue, Ahmed Helmy, Srijan Das, Pu Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13393
• Pdf URL: https://arxiv.org/pdf/2412.13393
• Copy Paste: [[2412.13393]] MMHMR: Generative Masked Modeling for Hand Mesh Recovery(https://arxiv.org/abs/2412.13393)
• Keywords: robust, transformer, generative
• Abstract: Reconstructing a 3D hand mesh from a single RGB image is challenging due to complex articulations, self-occlusions, and depth ambiguities. Traditional discriminative methods, which learn a deterministic mapping from a 2D image to a single 3D mesh, often struggle with the inherent ambiguities in 2D-to-3D mapping. To address this challenge, we propose MMHMR, a novel generative masked model for hand mesh recovery that synthesizes plausible 3D hand meshes by learning and sampling from the probabilistic distribution of the ambiguous 2D-to-3D mapping process. MMHMR consists of two key components: (1) a VQ-MANO, which encodes 3D hand articulations as discrete pose tokens in a latent space, and (2) a Context-Guided Masked Transformer that randomly masks out pose tokens and learns their joint distribution, conditioned on corrupted token sequences, image context, and 2D pose cues. This learned distribution facilitates confidence-guided sampling during inference, producing mesh reconstructions with low uncertainty and high precision. Extensive evaluations on benchmark and real-world datasets demonstrate that MMHMR achieves state-of-the-art accuracy, robustness, and realism in 3D hand mesh reconstruction. Project website: this https URL

Title: Distribution Shifts at Scale: Out-of-distribution Detection in Earth Observation

• Authors: Burak Ekim, Girmaw Abebe Tadesse, Caleb Robinson, Gilles Hacheme, Michael Schmitt, Rahul Dodhia, Juan M. Lavista Ferres
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13394
• Pdf URL: https://arxiv.org/pdf/2412.13394
• Copy Paste: [[2412.13394]] Distribution Shifts at Scale: Out-of-distribution Detection in Earth Observation(https://arxiv.org/abs/2412.13394)
• Keywords: robust
• Abstract: Training robust deep learning models is critical in Earth Observation, where globally deployed models often face distribution shifts that degrade performance, especially in low-data regions. Out-of-distribution (OOD) detection addresses this challenge by identifying inputs that differ from in-distribution (ID) data. However, existing methods either assume access to OOD data or compromise primary task performance, making them unsuitable for real-world deployment. We propose TARDIS, a post-hoc OOD detection method for scalable geospatial deployments. The core novelty lies in generating surrogate labels by integrating information from ID data and unknown distributions, enabling OOD detection at scale. Our method takes a pre-trained model, ID data, and WILD samples, disentangling the latter into surrogate ID and surrogate OOD labels based on internal activations, and fits a binary classifier as an OOD detector. We validate TARDIS on EuroSAT and xBD datasets, across 17 experimental setups covering covariate and semantic shifts, showing that it performs close to the theoretical upper bound in assigning surrogate ID and OOD samples in 13 cases. To demonstrate scalability, we deploy TARDIS on the Fields of the World dataset, offering actionable insights into pre-trained model behavior for large-scale deployments. The code is publicly available at this https URL.

Title: Zero-Shot Low Light Image Enhancement with Diffusion Prior

• Authors: Joshua Cho, Sara Aghajanzadeh, Zhen Zhu, D. A. Forsyth
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13401
• Pdf URL: https://arxiv.org/pdf/2412.13401
• Copy Paste: [[2412.13401]] Zero-Shot Low Light Image Enhancement with Diffusion Prior(https://arxiv.org/abs/2412.13401)
• Keywords: diffusion, generative
• Abstract: Balancing aesthetic quality with fidelity when enhancing images from challenging, degraded sources is a core objective in computational photography. In this paper, we address low light image enhancement (LLIE), a task in which dark images often contain limited visible information. Diffusion models, known for their powerful image enhancement capacities, are a natural choice for this problem. However, their deep generative priors can also lead to hallucinations, introducing non-existent elements or substantially altering the visual semantics of the original scene. In this work, we introduce a novel zero-shot method for controlling and refining the generative behavior of diffusion models for dark-to-light image conversion tasks. Our method demonstrates superior performance over existing state-of-the-art methods in the task of low-light image enhancement, as evidenced by both quantitative metrics and qualitative analysis.

Title: Safeguarding System Prompts for LLMs

• Authors: Zhifeng Jiang, Zhihua Jin, Guoliang He
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13426
• Pdf URL: https://arxiv.org/pdf/2412.13426
• Copy Paste: [[2412.13426]] Safeguarding System Prompts for LLMs(https://arxiv.org/abs/2412.13426)
• Keywords: privacy, protect, defense, attack, robust, extraction, large language model
• Abstract: Large language models (LLMs) are increasingly utilized in applications where system prompts, which guide model outputs, play a crucial role. These prompts often contain business logic and sensitive information, making their protection essential. However, adversarial and even regular user queries can exploit LLM vulnerabilities to expose these hidden prompts. To address this issue, we present PromptKeeper, a novel defense mechanism for system prompt privacy. By reliably detecting worst-case leakage and regenerating outputs without the system prompt when necessary, PromptKeeper ensures robust protection against prompt extraction attacks via either adversarial or regular queries, while preserving conversational capability and runtime efficiency during benign user interactions.

Title: Lightweight Safety Classification Using Pruned Language Models

• Authors: Mason Sawtell, Tula Masterman, Sandi Besen, Jim Brown
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13435
• Pdf URL: https://arxiv.org/pdf/2412.13435
• Copy Paste: [[2412.13435]] Lightweight Safety Classification Using Pruned Language Models(https://arxiv.org/abs/2412.13435)
• Keywords: robust, extraction, transformer, large language model
• Abstract: In this paper, we introduce a novel technique for content safety and prompt injection classification for Large Language Models. Our technique, Layer Enhanced Classification (LEC), trains a Penalized Logistic Regression (PLR) classifier on the hidden state of an LLM's optimal intermediate transformer layer. By combining the computational efficiency of a streamlined PLR classifier with the sophisticated language understanding of an LLM, our approach delivers superior performance surpassing GPT-4o and special-purpose models fine-tuned for each task. We find that small general-purpose models (Qwen 2.5 sizes 0.5B, 1.5B, and 3B) and other transformer-based architectures like DeBERTa v3 are robust feature extractors allowing simple classifiers to be effectively trained on fewer than 100 high-quality examples. Importantly, the intermediate transformer layers of these models typically outperform the final layer across both classification tasks. Our results indicate that a single general-purpose LLM can be used to classify content safety, detect prompt injections, and simultaneously generate output tokens. Alternatively, these relatively small LLMs can be pruned to the optimal intermediate layer and used exclusively as robust feature extractors. Since our results are consistent on different transformer architectures, we infer that robust feature extraction is an inherent capability of most, if not all, LLMs.

Title: Rare Event Detection in Imbalanced Multi-Class Datasets Using an Optimal MIP-Based Ensemble Weighting Approach

• Authors: Georgios Tertytchny, Georgios L. Stavrinides, Maria K. Michael
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13439
• Pdf URL: https://arxiv.org/pdf/2412.13439
• Copy Paste: [[2412.13439]] Rare Event Detection in Imbalanced Multi-Class Datasets Using an Optimal MIP-Based Ensemble Weighting Approach(https://arxiv.org/abs/2412.13439)
• Keywords: robust
• Abstract: To address the challenges of imbalanced multi-class datasets typically used for rare event detection in critical cyber-physical systems, we propose an optimal, efficient, and adaptable mixed integer programming (MIP) ensemble weighting scheme. Our approach leverages the diverse capabilities of the classifier ensemble on a granular per class basis, while optimizing the weights of classifier-class pairs using elastic net regularization for improved robustness and generalization. Additionally, it seamlessly and optimally selects a predefined number of classifiers from a given set. We evaluate and compare our MIP-based method against six well-established weighting schemes, using representative datasets and suitable metrics, under various ensemble sizes. The experimental results reveal that MIP outperforms all existing approaches, achieving an improvement in balanced accuracy ranging from 0.99% to 7.31%, with an overall average of 4.53% across all datasets and ensemble sizes. Furthermore, it attains an overall average increase of 4.63%, 4.60%, and 4.61% in macro-averaged precision, recall, and F1-score, respectively, while maintaining computational efficiency.

Title: Safeguarding Virtual Healthcare: A Novel Attacker-Centric Model for Data Security and Privacy

• Authors: Suvineetha Herath, Haywood Gelman, John Hastings, Yong Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.13440
• Pdf URL: https://arxiv.org/pdf/2412.13440
• Copy Paste: [[2412.13440]] Safeguarding Virtual Healthcare: A Novel Attacker-Centric Model for Data Security and Privacy(https://arxiv.org/abs/2412.13440)
• Keywords: secure, security, privacy, protect, attack
• Abstract: The rapid growth of remote healthcare delivery has introduced significant security and privacy risks to protected health information (PHI). Analysis of a comprehensive healthcare security breach dataset covering 2009-2023 reveals their significant prevalence and impact. This study investigates the root causes of such security incidents and introduces the Attacker-Centric Approach (ACA), a novel threat model tailored to protect PHI. ACA addresses limitations in existing threat models and regulatory frameworks by adopting a holistic attacker-focused perspective, examining threats from the viewpoint of cyber adversaries, their motivations, tactics, and potential attack vectors. Leveraging established risk management frameworks, ACA provides a multi-layered approach to threat identification, risk assessment, and proactive mitigation strategies. A comprehensive threat library classifies physical, third-party, external, and internal threats. ACA's iterative nature and feedback mechanisms enable continuous adaptation to emerging threats, ensuring sustained effectiveness. ACA allows healthcare providers to proactively identify and mitigate vulnerabilities, fostering trust and supporting the secure adoption of virtual care technologies.

Title: Communication-Efficient Personalized Federal Graph Learning via Low-Rank Decomposition

• Authors: Ruyue Liu, Rong Yin, Xiangzhen Bo, Xiaoshuai Hao, Xingrui Zhou, Yong Liu, Can Ma, Weiping Wang
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.13442
• Pdf URL: https://arxiv.org/pdf/2412.13442
• Copy Paste: [[2412.13442]] Communication-Efficient Personalized Federal Graph Learning via Low-Rank Decomposition(https://arxiv.org/abs/2412.13442)
• Keywords: privacy, federate
• Abstract: Federated graph learning (FGL) has gained significant attention for enabling heterogeneous clients to process their private graph data locally while interacting with a centralized server, thus maintaining privacy. However, graph data on clients are typically non-IID, posing a challenge for a single model to perform well across all clients. Another major bottleneck of FGL is the high cost of communication. To address these challenges, we propose a communication-efficient personalized federated graph learning algorithm, CEFGL. Our method decomposes the model parameters into low-rank generic and sparse private models. We employ a dual-channel encoder to learn sparse local knowledge in a personalized manner and low-rank global knowledge in a shared manner. Additionally, we perform multiple local stochastic gradient descent iterations between communication phases and integrate efficient compression techniques into the algorithm. The advantage of CEFGL lies in its ability to capture common and individual knowledge more precisely. By utilizing low-rank and sparse parameters along with compression techniques, CEFGL significantly reduces communication complexity. Extensive experiments demonstrate that our method achieves optimal classification accuracy in a variety of heterogeneous environments across sixteen datasets. Specifically, compared to the state-of-the-art method FedStar, the proposed method (with GIN as the base model) improves accuracy by 5.64\% on cross-datasets setting CHEM, reduces communication bits by a factor of 18.58, and reduces the communication time by a factor of 1.65.

Title: DarkIR: Robust Low-Light Image Restoration

• Authors: Daniel Feijoo, Juan C. Benito, Alvaro Garcia, Marcos V. Conde
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.13443
• Pdf URL: https://arxiv.org/pdf/2412.13443
• Copy Paste: [[2412.13443]] DarkIR: Robust Low-Light Image Restoration(https://arxiv.org/abs/2412.13443)
• Keywords: robust, transformer
• Abstract: Photography during night or in dark conditions typically suffers from noise, low light and blurring issues due to the dim environment and the common use of long exposure. Although Deblurring and Low-light Image Enhancement (LLIE) are related under these conditions, most approaches in image restoration solve these tasks separately. In this paper, we present an efficient and robust neural network for multi-task low-light image restoration. Instead of following the current tendency of Transformer-based models, we propose new attention mechanisms to enhance the receptive field of efficient CNNs. Our method reduces the computational costs in terms of parameters and MAC operations compared to previous methods. Our model, DarkIR, achieves new state-of-the-art results on the popular LOLBlur, LOLv2 and Real-LOLBlur datasets, being able to generalize on real-world night and dark images. Code and models at this https URL

Title: ConDo: Continual Domain Expansion for Absolute Pose Regression

• Authors: Zijun Li, Zhipeng Cai, Bochun Yang, Xuelun Shen, Siqi Shen, Xiaoliang Fan, Michael Paulitsch, Cheng Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13452
• Pdf URL: https://arxiv.org/pdf/2412.13452
• Copy Paste: [[2412.13452]] ConDo: Continual Domain Expansion for Absolute Pose Regression(https://arxiv.org/abs/2412.13452)
• Keywords: robust
• Abstract: Visual localization is a fundamental machine learning problem. Absolute Pose Regression (APR) trains a scene-dependent model to efficiently map an input image to the camera pose in a pre-defined scene. However, many applications have continually changing environments, where inference data at novel poses or scene conditions (weather, geometry) appear after deployment. Training APR on a fixed dataset leads to overfitting, making it fail catastrophically on challenging novel data. This work proposes Continual Domain Expansion (ConDo), which continually collects unlabeled inference data to update the deployed APR. Instead of applying standard unsupervised domain adaptation methods which are ineffective for APR, ConDo effectively learns from unlabeled data by distilling knowledge from scene-agnostic localization methods. By sampling data uniformly from historical and newly collected data, ConDo can effectively expand the generalization domain of APR. Large-scale benchmarks with various scene types are constructed to evaluate models under practical (long-term) data changes. ConDo consistently and significantly outperforms baselines across architectures, scene types, and data changes. On challenging scenes (Fig.1), it reduces the localization error by >7x (14.8m vs 1.7m). Analysis shows the robustness of ConDo against compute budgets, replay buffer sizes and teacher prediction noise. Comparing to model re-training, ConDo achieves similar performance up to 25x faster.

Title: Pre-training a Density-Aware Pose Transformer for Robust LiDAR-based 3D Human Pose Estimation

• Authors: Xiaoqi An, Lin Zhao, Chen Gong, Jun Li, Jian Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13454
• Pdf URL: https://arxiv.org/pdf/2412.13454
• Copy Paste: [[2412.13454]] Pre-training a Density-Aware Pose Transformer for Robust LiDAR-based 3D Human Pose Estimation(https://arxiv.org/abs/2412.13454)
• Keywords: robust, transformer
• Abstract: With the rapid development of autonomous driving, LiDAR-based 3D Human Pose Estimation (3D HPE) is becoming a research focus. However, due to the noise and sparsity of LiDAR-captured point clouds, robust human pose estimation remains challenging. Most of the existing methods use temporal information, multi-modal fusion, or SMPL optimization to correct biased results. In this work, we try to obtain sufficient information for 3D HPE only by modeling the intrinsic properties of low-quality point clouds. Hence, a simple yet powerful method is proposed, which provides insights both on modeling and augmentation of point clouds. Specifically, we first propose a concise and effective density-aware pose transformer (DAPT) to get stable keypoint representations. By using a set of joint anchors and a carefully designed exchange module, valid information is extracted from point clouds with different densities. Then 1D heatmaps are utilized to represent the precise locations of the keypoints. Secondly, a comprehensive LiDAR human synthesis and augmentation method is proposed to pre-train the model, enabling it to acquire a better human body prior. We increase the diversity of point clouds by randomly sampling human positions and orientations and by simulating occlusions through the addition of laser-level masks. Extensive experiments have been conducted on multiple datasets, including IMU-annotated LidarHuman26M, SLOPER4D, and manually annotated Waymo Open Dataset v2.0 (Waymo), HumanM3. Our method demonstrates SOTA performance in all scenarios. In particular, compared with LPFormer on Waymo, we reduce the average MPJPE by $10.0mm$. Compared with PRN on SLOPER4D, we notably reduce the average MPJPE by $20.7mm$.

Title: 4.5 Million (Suspected) Fake Stars in GitHub: A Growing Spiral of Popularity Contests, Scams, and Malware

• Authors: Hao He, Haoqin Yang, Philipp Burckhardt, Alexandros Kapravelos, Bogdan Vasilescu, Christian Kästner
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.13459
• Pdf URL: https://arxiv.org/pdf/2412.13459
• Copy Paste: [[2412.13459]] 4.5 Million (Suspected) Fake Stars in GitHub: A Growing Spiral of Popularity Contests, Scams, and Malware(https://arxiv.org/abs/2412.13459)
• Keywords: security
• Abstract: GitHub, the de-facto platform for open-source software development, provides a set of social-media-like features to signal high-quality repositories. Among them, the star count is the most widely used popularity signal, but it is also at risk of being artificially inflated (i.e., faked), decreasing its value as a decision-making signal and posing a security risk to all GitHub users. In this paper, we present a systematic, global, and longitudinal measurement study of fake stars in GitHub. To this end, we build StarScout, a scalable tool able to detect anomalous starring behaviors (i.e., low activity and lockstep) across the entire GitHub metadata. Analyzing the data collected using StarScout, we find that: (1) fake-star-related activities have rapidly surged since 2024; (2) the user profile characteristics of fake stargazers are not distinct from average GitHub users, but many of them have highly abnormal activity patterns; (3) the majority of fake stars are used to promote short-lived malware repositories masquerading as pirating software, game cheats, or cryptocurrency bots; (4) some repositories may have acquired fake stars for growth hacking, but fake stars only have a promotion effect in the short term (i.e., less than two months) and become a burden in the long term. Our study has implications for platform moderators, open-source practitioners, and supply chain security researchers.

Title: Look Inside for More: Internal Spatial Modality Perception for 3D Anomaly Detection

• Authors: Hanzhe Liang, Guoyang Xie, Chengbin Hou, Bingshu Wang, Can Gao, Jinbao Wang
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.13461
• Pdf URL: https://arxiv.org/pdf/2412.13461
• Copy Paste: [[2412.13461]] Look Inside for More: Internal Spatial Modality Perception for 3D Anomaly Detection(https://arxiv.org/abs/2412.13461)
• Keywords: extraction, segmentation
• Abstract: 3D anomaly detection has recently become a significant focus in computer vision. Several advanced methods have achieved satisfying anomaly detection performance. However, they typically concentrate on the external structure of 3D samples and struggle to leverage the internal information embedded within samples. Inspired by the basic intuition of why not look inside for more, we introduce a straightforward method named Internal Spatial Modality Perception (ISMP) to explore the feature representation from internal views fully. Specifically, our proposed ISMP consists of a critical perception module, Spatial Insight Engine (SIE), which abstracts complex internal information of point clouds into essential global features. Besides, to better align structural information with point data, we propose an enhanced key point feature extraction module for amplifying spatial structure feature representation. Simultaneously, a novel feature filtering module is incorporated to reduce noise and redundant features for further aligning precise spatial structure. Extensive experiments validate the effectiveness of our proposed method, achieving object-level and pixel-level AUROC improvements of 4.2% and 13.1%, respectively, on the Real3D-AD benchmarks. Note that the strong generalization ability of SIE has been theoretically proven and is verified in both classification and segmentation tasks.

Title: FlexPose: Pose Distribution Adaptation with Limited Guidance

• Authors: Zixiao Wang, Junwu Weng, Mengyuan Liu, Bei Yu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13463
• Pdf URL: https://arxiv.org/pdf/2412.13463
• Copy Paste: [[2412.13463]] FlexPose: Pose Distribution Adaptation with Limited Guidance(https://arxiv.org/abs/2412.13463)
• Keywords: generative
• Abstract: Numerous well-annotated human key-point datasets are publicly available to date. However, annotating human poses for newly collected images is still a costly and time-consuming progress. Pose distributions from different datasets share similar pose hinge-structure priors with different geometric transformations, such as pivot orientation, joint rotation, and bone length ratio. The difference between Pose distributions is essentially the difference between the transformation distributions. Inspired by this fact, we propose a method to calibrate a pre-trained pose generator in which the pose prior has already been learned to an adapted one following a new pose distribution. We treat the representation of human pose joint coordinates as skeleton image and transfer a pre-trained pose annotation generator with only a few annotation guidance. By fine-tuning a limited number of linear layers that closely related to the pose transformation, the adapted generator is able to produce any number of pose annotations that are similar to the target poses. We evaluate our proposed method, FlexPose, on several cross-dataset settings both qualitatively and quantitatively, which demonstrates that our approach achieves state-of-the-art performance compared to the existing generative-model-based transfer learning methods when given limited annotation guidance.

Title: Federated Unlearning Model Recovery in Data with Skewed Label Distributions

• Authors: Xinrui Yu, Wenbin Pei, Bing Xue, Qiang Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13466
• Pdf URL: https://arxiv.org/pdf/2412.13466
• Copy Paste: [[2412.13466]] Federated Unlearning Model Recovery in Data with Skewed Label Distributions(https://arxiv.org/abs/2412.13466)
• Keywords: federate
• Abstract: In federated learning, federated unlearning is a technique that provides clients with a rollback mechanism that allows them to withdraw their data contribution without training from scratch. However, existing research has not considered scenarios with skewed label distributions. Unfortunately, the unlearning of a client with skewed data usually results in biased models and makes it difficult to deliver high-quality service, complicating the recovery process. This paper proposes a recovery method of federated unlearning with skewed label distributions. Specifically, we first adopt a strategy that incorporates oversampling with deep learning to supplement the skewed class data for clients to perform recovery training, therefore enhancing the completeness of their local datasets. Afterward, a density-based denoising method is applied to remove noise from the generated data, further improving the quality of the remaining clients' datasets. Finally, all the remaining clients leverage the enhanced local datasets and engage in iterative training to effectively restore the performance of the unlearning model. Extensive evaluations on commonly used federated learning datasets with varying degrees of skewness show that our method outperforms baseline methods in restoring the performance of the unlearning model, particularly regarding accuracy on the skewed class.

Title: SocialED: A Python Library for Social Event Detection

• Authors: Kun Zhang, Xiaoyan Yu, Pu Li, Hao Peng, Philip S. Yu
• Subjects: cs.LG, cs.DL
• Abstract URL: https://arxiv.org/abs/2412.13472
• Pdf URL: https://arxiv.org/pdf/2412.13472
• Copy Paste: [[2412.13472]] SocialED: A Python Library for Social Event Detection(https://arxiv.org/abs/2412.13472)
• Keywords: robust
• Abstract: SocialED is a comprehensive, open-source Python library designed to support social event detection (SED) tasks, integrating 19 detection algorithms and 14 diverse datasets. It provides a unified API with detailed documentation, offering researchers and practitioners a complete solution for event detection in social media. The library is designed with modularity in mind, allowing users to easily adapt and extend components for various use cases. SocialED supports a wide range of preprocessing techniques, such as graph construction and tokenization, and includes standardized interfaces for training models and making predictions. By integrating popular deep learning frameworks, SocialED ensures high efficiency and scalability across both CPU and GPU environments. The library is built adhering to high code quality standards, including unit testing, continuous integration, and code coverage, ensuring that SocialED delivers robust, maintainable software. SocialED is publicly available at \url{this https URL} and can be installed via PyPI.

Title: A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models

• Authors: Bowen Chen, Namgi Han, Yusuke Miyao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13475
• Pdf URL: https://arxiv.org/pdf/2412.13475
• Copy Paste: [[2412.13475]] A Statistical and Multi-Perspective Revisiting of the Membership Inference Attack in Large Language Models(https://arxiv.org/abs/2412.13475)
• Keywords: attack, membership infer, large language model
• Abstract: The lack of data transparency in Large Language Models (LLMs) has highlighted the importance of Membership Inference Attack (MIA), which differentiates trained (member) and untrained (non-member) data. Though it shows success in previous studies, recent research reported a near-random performance in different settings, highlighting a significant performance inconsistency. We assume that a single setting doesn't represent the distribution of the vast corpora, causing members and non-members with different distributions to be sampled and causing inconsistency. In this study, instead of a single setting, we statistically revisit MIA methods from various settings with thousands of experiments for each MIA method, along with study in text feature, embedding, threshold decision, and decoding dynamics of members and non-members. We found that (1) MIA performance improves with model size and varies with domains, while most methods do not statistically outperform baselines, (2) Though MIA performance is generally low, a notable amount of differentiable member and non-member outliers exists and vary across MIA methods, (3) Deciding a threshold to separate members and non-members is an overlooked challenge, (4) Text dissimilarity and long text benefit MIA performance, (5) Differentiable or not is reflected in the LLM embedding, (6) Member and non-members show different decoding dynamics.

Title: Efficient Fine-Tuning of Single-Cell Foundation Models Enables Zero-Shot Molecular Perturbation Prediction

• Authors: Sepideh Maleki, Jan-Christian Huetter, Kangway V. Chuang, Gabriele Scalia, Tommaso Biancalani
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2412.13478
• Pdf URL: https://arxiv.org/pdf/2412.13478
• Copy Paste: [[2412.13478]] Efficient Fine-Tuning of Single-Cell Foundation Models Enables Zero-Shot Molecular Perturbation Prediction(https://arxiv.org/abs/2412.13478)
• Keywords: robust
• Abstract: Predicting transcriptional responses to novel drugs provides a unique opportunity to accelerate biomedical research and advance drug discovery efforts. However, the inherent complexity and high dimensionality of cellular responses, combined with the extremely limited available experimental data, makes the task challenging. In this study, we leverage single-cell foundation models (FMs) pre-trained on tens of millions of single cells, encompassing multiple cell types, states, and disease annotations, to address molecular perturbation prediction. We introduce a drug-conditional adapter that allows efficient fine-tuning by training less than 1% of the original foundation model, thus enabling molecular conditioning while preserving the rich biological representation learned during pre-training. The proposed strategy allows not only the prediction of cellular responses to novel drugs, but also the zero-shot generalization to unseen cell lines. We establish a robust evaluation framework to assess model performance across different generalization tasks, demonstrating state-of-the-art results across all settings, with significant improvements in the few-shot and zero-shot generalization to new cell lines compared to existing baselines.

Title: Real-time One-Step Diffusion-based Expressive Portrait Videos Generation

• Authors: Hanzhong Guo, Hongwei Yi, Daquan Zhou, Alexander William Bergman, Michael Lingelbach, Yizhou Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13479
• Pdf URL: https://arxiv.org/pdf/2412.13479
• Copy Paste: [[2412.13479]] Real-time One-Step Diffusion-based Expressive Portrait Videos Generation(https://arxiv.org/abs/2412.13479)
• Keywords: diffusion
• Abstract: Latent diffusion models have made great strides in generating expressive portrait videos with accurate lip-sync and natural motion from a single reference image and audio input. However, these models are far from real-time, often requiring many sampling steps that take minutes to generate even one second of video-significantly limiting practical use. We introduce OSA-LCM (One-Step Avatar Latent Consistency Model), paving the way for real-time diffusion-based avatars. Our method achieves comparable video quality to existing methods but requires only one sampling step, making it more than 10x faster. To accomplish this, we propose a novel avatar discriminator design that guides lip-audio consistency and motion expressiveness to enhance video quality in limited sampling steps. Additionally, we employ a second-stage training architecture using an editing fine-tuned method (EFT), transforming video generation into an editing task during training to effectively address the temporal gap challenge in single-step generation. Experiments demonstrate that OSA-LCM outperforms existing open-source portrait video generation models while operating more efficiently with a single sampling step.

Title: T$^3$-S2S: Training-free Triplet Tuning for Sketch to Scene Generation

• Authors: Zhenhong Sun, Yifu Wang, Yonhon Ng, Yunfei Duan, Daoyi Dong, Hongdong Li, Pan Ji
• Subjects: cs.CV, cs.CL, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.13486
• Pdf URL: https://arxiv.org/pdf/2412.13486
• Copy Paste: [[2412.13486]] T$^3$-S2S: Training-free Triplet Tuning for Sketch to Scene Generation(https://arxiv.org/abs/2412.13486)
• Keywords: generative
• Abstract: Scene generation is crucial to many computer graphics applications. Recent advances in generative AI have streamlined sketch-to-image workflows, easing the workload for artists and designers in creating scene concept art. However, these methods often struggle for complex scenes with multiple detailed objects, sometimes missing small or uncommon instances. In this paper, we propose a Training-free Triplet Tuning for Sketch-to-Scene (T3-S2S) generation after reviewing the entire cross-attention mechanism. This scheme revitalizes the existing ControlNet model, enabling effective handling of multi-instance generations, involving prompt balance, characteristics prominence, and dense tuning. Specifically, this approach enhances keyword representation via the prompt balance module, reducing the risk of missing critical instances. It also includes a characteristics prominence module that highlights TopK indices in each channel, ensuring essential features are better represented based on token sketches. Additionally, it employs dense tuning to refine contour details in the attention map, compensating for instance-related regions. Experiments validate that our triplet tuning approach substantially improves the performance of existing sketch-to-image models. It consistently generates detailed, multi-instance 2D images, closely adhering to the input prompts and enhancing visual quality in complex multi-instance scenes. Code is available at this https URL.

Title: Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models

• Authors: Xinxin Liu, Aaron Thomas, Cheng Zhang, Jianyi Cheng, Yiren Zhao, Xitong Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13488
• Pdf URL: https://arxiv.org/pdf/2412.13488
• Copy Paste: [[2412.13488]] Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models(https://arxiv.org/abs/2412.13488)
• Keywords: robust
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) has gained prominence through low-rank adaptation methods like LoRA. In this paper, we focus on sparsity-based PEFT (SPEFT), which introduces trainable sparse adaptations to the weight matrices in the model, offering greater flexibility in selecting fine-tuned parameters compared to low-rank methods. We conduct the first systematic evaluation of salience metrics for SPEFT, inspired by zero-cost NAS proxies, and identify simple gradient-based metrics is reliable, and results are on par with the best alternatives, offering both computational efficiency and robust performance. Additionally, we compare static and dynamic masking strategies, finding that static masking, which predetermines non-zero entries before training, delivers efficiency without sacrificing performance, while dynamic masking offers no substantial benefits. Across NLP tasks, a simple gradient-based, static SPEFT consistently outperforms other fine-tuning methods for LLMs, providing a simple yet effective baseline for SPEFT. Our work challenges the notion that complexity is necessary for effective PEFT. Our work is open source and available to the community at [this https URL].

Title: Federated t-SNE and UMAP for Distributed Data Visualization

• Authors: Dong Qiao, Xinxian Ma, Jicong Fan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13495
• Pdf URL: https://arxiv.org/pdf/2412.13495
• Copy Paste: [[2412.13495]] Federated t-SNE and UMAP for Distributed Data Visualization(https://arxiv.org/abs/2412.13495)
• Keywords: security, privacy, protect, federate
• Abstract: High-dimensional data visualization is crucial in the big data era and these techniques such as t-SNE and UMAP have been widely used in science and engineering. Big data, however, is often distributed across multiple data centers and subject to security and privacy concerns, which leads to difficulties for the standard algorithms of t-SNE and UMAP. To tackle the challenge, this work proposes Fed-tSNE and Fed-UMAP, which provide high-dimensional data visualization under the framework of federated learning, without exchanging data across clients or sending data to the central server. The main idea of Fed-tSNE and Fed-UMAP is implicitly learning the distribution information of data in a manner of federated learning and then estimating the global distance matrix for t-SNE and UMAP. To further enhance the protection of data privacy, we propose Fed-tSNE+ and Fed-UMAP+. We also extend our idea to federated spectral clustering, yielding algorithms of clustering distributed data. In addition to these new algorithms, we offer theoretical guarantees of optimization convergence, distance and similarity estimation, and differential privacy. Experiments on multiple datasets demonstrate that, compared to the original algorithms, the accuracy drops of our federated algorithms are tiny.

Title: VaeDiff-DocRE: End-to-end Data Augmentation Framework for Document-level Relation Extraction

• Authors: Khai Phan Tran, Wen Hua, Xue Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13503
• Pdf URL: https://arxiv.org/pdf/2412.13503
• Copy Paste: [[2412.13503]] VaeDiff-DocRE: End-to-end Data Augmentation Framework for Document-level Relation Extraction(https://arxiv.org/abs/2412.13503)
• Keywords: extraction, diffusion, generative
• Abstract: Document-level Relation Extraction (DocRE) aims to identify relationships between entity pairs within a document. However, most existing methods assume a uniform label distribution, resulting in suboptimal performance on real-world, imbalanced datasets. To tackle this challenge, we propose a novel data augmentation approach using generative models to enhance data from the embedding space. Our method leverages the Variational Autoencoder (VAE) architecture to capture all relation-wise distributions formed by entity pair representations and augment data for underrepresented relations. To better capture the multi-label nature of DocRE, we parameterize the VAE's latent space with a Diffusion Model. Additionally, we introduce a hierarchical training framework to integrate the proposed VAE-based augmentation module into DocRE systems. Experiments on two benchmark datasets demonstrate that our method outperforms state-of-the-art models, effectively addressing the long-tail distribution problem in DocRE.

Title: Urban Air Temperature Prediction using Conditional Diffusion Models

• Authors: Siyang Dai, Jun Liu, Ngai-Man Cheung
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.13504
• Pdf URL: https://arxiv.org/pdf/2412.13504
• Copy Paste: [[2412.13504]] Urban Air Temperature Prediction using Conditional Diffusion Models(https://arxiv.org/abs/2412.13504)
• Keywords: diffusion
• Abstract: Urbanization as a global trend has led to many environmental challenges, including the urban heat island (UHI) effect. The increase in temperature has a significant impact on the well-being of urban residents. Air temperature ($T_a$) at 2m above the surface is a key indicator of the UHI effect. How land use land cover (LULC) affects $T_a$ is a critical research question which requires high-resolution (HR) $T_a$ data at neighborhood scale. However, weather stations providing $T_a$ measurements are sparsely distributed e.g. more than 10km apart; and numerical models are impractically slow and computationally expensive. In this work, we propose a novel method to predict HR $T_a$ at 100m ground separation distance (gsd) using land surface temperature (LST) and other LULC related features which can be easily obtained from satellite imagery. Our method leverages diffusion models for the first time to generate accurate and visually realistic HR $T_a$ maps, which outperforms prior methods. We pave the way for meteorological research using computer vision techniques by providing a dataset of an extended spatial and temporal coverage, and a high spatial resolution as a benchmark for future research. Furthermore, we show that our model can be applied to urban planning by simulating the impact of different urban designs on $T_a$.

Title: Novel AI Camera Camouflage: Face Cloaking Without Full Disguise

• Authors: David Noever, Forrest McKee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13507
• Pdf URL: https://arxiv.org/pdf/2412.13507
• Copy Paste: [[2412.13507]] Novel AI Camera Camouflage: Face Cloaking Without Full Disguise(https://arxiv.org/abs/2412.13507)
• Keywords: attack
• Abstract: This study demonstrates a novel approach to facial camouflage that combines targeted cosmetic perturbations and alpha transparency layer manipulation to evade modern facial recognition systems. Unlike previous methods -- such as CV dazzle, adversarial patches, and theatrical disguises -- this work achieves effective obfuscation through subtle modifications to key-point regions, particularly the brow, nose bridge, and jawline. Empirical testing with Haar cascade classifiers and commercial systems like BetaFaceAPI and Microsoft Bing Visual Search reveals that vertical perturbations near dense facial key points significantly disrupt detection without relying on overt disguises. Additionally, leveraging alpha transparency attacks in PNG images creates a dual-layer effect: faces remain visible to human observers but disappear in machine-readable RGB layers, rendering them unidentifiable during reverse image searches. The results highlight the potential for creating scalable, low-visibility facial obfuscation strategies that balance effectiveness and subtlety, opening pathways for defeating surveillance while maintaining plausible anonymity.

Title: Privacy-Preserving Cyberattack Detection in Blockchain-Based IoT Systems Using AI and Homomorphic Encryption

• Authors: Bui Duc Manh, Chi-Hieu Nguyen, Dinh Thai Hoang, Diep N. Nguyen, Ming Zeng, Quoc-Viet Pham
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.13522
• Pdf URL: https://arxiv.org/pdf/2412.13522
• Copy Paste: [[2412.13522]] Privacy-Preserving Cyberattack Detection in Blockchain-Based IoT Systems Using AI and Homomorphic Encryption(https://arxiv.org/abs/2412.13522)
• Keywords: privacy, attack
• Abstract: This work proposes a novel privacy-preserving cyberattack detection framework for blockchain-based Internet-of-Things (IoT) systems. In our approach, artificial intelligence (AI)-driven detection modules are strategically deployed at blockchain nodes to identify real-time attacks, ensuring high accuracy and minimal delay. To achieve this efficiency, the model training is conducted by a cloud service provider (CSP). Accordingly, blockchain nodes send their data to the CSP for training, but to safeguard privacy, the data is encrypted using homomorphic encryption (HE) before transmission. This encryption method allows the CSP to perform computations directly on encrypted data without the need for decryption, preserving data privacy throughout the learning process. To handle the substantial volume of encrypted data, we introduce an innovative packing algorithm in a Single-Instruction-Multiple-Data (SIMD) manner, enabling efficient training on HE-encrypted data. Building on this, we develop a novel deep neural network training algorithm optimized for encrypted data. We further propose a privacy-preserving distributed learning approach based on the FedAvg algorithm, which parallelizes the training across multiple workers, significantly improving computation time. Upon completion, the CSP distributes the trained model to the blockchain nodes, enabling them to perform real-time, privacy-preserved detection. Our simulation results demonstrate that our proposed method can not only mitigate the training time but also achieve detection accuracy that is approximately identical to the approach without encryption, with a gap of around 0.01%. Additionally, our real implementations on various blockchain consensus algorithms and hardware configurations show that our proposed framework can also be effectively adapted to real-world systems.

Title: Hybrid Data-Free Knowledge Distillation

• Authors: Jialiang Tang, Shuo Chen, Chen Gong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13525
• Pdf URL: https://arxiv.org/pdf/2412.13525
• Copy Paste: [[2412.13525]] Hybrid Data-Free Knowledge Distillation(https://arxiv.org/abs/2412.13525)
• Keywords: data-free, generative
• Abstract: Data-free knowledge distillation aims to learn a compact student network from a pre-trained large teacher network without using the original training data of the teacher network. Existing collection-based and generation-based methods train student networks by collecting massive real examples and generating synthetic examples, respectively. However, they inevitably become weak in practical scenarios due to the difficulties in gathering or emulating sufficient real-world data. To solve this problem, we propose a novel method called \textbf{H}ybr\textbf{i}d \textbf{D}ata-\textbf{F}ree \textbf{D}istillation (HiDFD), which leverages only a small amount of collected data as well as generates sufficient examples for training student networks. Our HiDFD comprises two primary modules, \textit{i.e.}, the teacher-guided generation and student distillation. The teacher-guided generation module guides a Generative Adversarial Network (GAN) by the teacher network to produce high-quality synthetic examples from very few real-world collected examples. Specifically, we design a feature integration mechanism to prevent the GAN from overfitting and facilitate the reliable representation learning from the teacher network. Meanwhile, we drive a category frequency smoothing technique via the teacher network to balance the generative training of each category. In the student distillation module, we explore a data inflation strategy to properly utilize a blend of real and synthetic data to train the student network via a classifier-sharing-based feature alignment technique. Intensive experiments across multiple benchmarks demonstrate that our HiDFD can achieve state-of-the-art performance using 120 times less collected data than existing methods. Code is available at this https URL.

Title: Quantum Machine Learning in Log-based Anomaly Detection: Challenges and Opportunities

• Authors: Jiaxing Qi, Chang Zeng, Zhongzhi Luan, Shaohan Huang, Shu Yang, Yao Lu, Bin Han, Hailong Yang, Depei Qian
• Subjects: cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2412.13529
• Pdf URL: https://arxiv.org/pdf/2412.13529
• Copy Paste: [[2412.13529]] Quantum Machine Learning in Log-based Anomaly Detection: Challenges and Opportunities(https://arxiv.org/abs/2412.13529)
• Keywords: robust
• Abstract: Log-based anomaly detection (LogAD) is the main component of Artificial Intelligence for IT Operations (AIOps), which can detect anomalous that occur during the system on-the-fly. Existing methods commonly extract log sequence features using classical machine learning techniques to identify whether a new sequence is an anomaly or not. However, these classical approaches often require trade-offs between efficiency and accuracy. The advent of quantum machine learning (QML) offers a promising alternative. By transforming parts of classical machine learning computations into parameterized quantum circuits (PQCs), QML can significantly reduce the number of trainable parameters while maintaining accuracy comparable to classical counterparts. In this work, we introduce a unified framework, \ourframework{}, for evaluating QML models in the context of LogAD. This framework incorporates diverse log data, integrated QML models, and comprehensive evaluation metrics. State-of-the-art methods such as DeepLog, LogAnomaly, and LogRobust, along with their quantum-transformed counterparts, are included in our this http URL standard metrics like F1 score, precision, and recall, our evaluation extends to factors critical to QML performance, such as specificity, the number of circuits, circuit design, and quantum state encoding. Using \ourframework{}, we conduct extensive experiments to assess the performance of these models and their quantum counterparts, uncovering valuable insights and paving the way for future research in QML model selection and design for LogAD.

Title: Language-guided Medical Image Segmentation with Target-informed Multi-level Contrastive Alignments

• Authors: Mingjian Li, Mingyuan Meng, Shuchang Ye, David Dagan Feng, Lei Bi, Jinman Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13533
• Pdf URL: https://arxiv.org/pdf/2412.13533
• Copy Paste: [[2412.13533]] Language-guided Medical Image Segmentation with Target-informed Multi-level Contrastive Alignments(https://arxiv.org/abs/2412.13533)
• Keywords: segmentation
• Abstract: Medical image segmentation is crucial in modern medical image analysis, which can aid into diagnosis of various disease conditions. Recently, language-guided segmentation methods have shown promising results in automating image segmentation where text reports are incorporated as guidance. These text reports, containing image impressions and insights given by clinicians, provides auxiliary guidance. However, these methods neglect the inherent pattern gaps between the two distinct modalities, which leads to sub-optimal image-text feature fusion without proper cross-modality feature alignments. Contrastive alignments are widely used to associate image-text semantics in representation learning; however, it has not been exploited to bridge the pattern gaps in language-guided segmentation that relies on subtle low level image details to represent diseases. Existing contrastive alignment methods typically algin high-level global image semantics without involving low-level, localized target information, and therefore fails to explore fine-grained text guidance for language-guided segmentation. In this study, we propose a language-guided segmentation network with Target-informed Multi-level Contrastive Alignments (TMCA). TMCA enables target-informed cross-modality alignments and fine-grained text guidance to bridge the pattern gaps in language-guided segmentation. Specifically, we introduce: 1) a target-sensitive semantic distance module that enables granular image-text alignment modelling, and 2) a multi-level alignment strategy that directs text guidance on low-level image features. In addition, a language-guided target enhancement module is proposed to leverage the aligned text to redirect attention to focus on critical localized image features. Extensive experiments on 4 image-text datasets, involving 3 medical imaging modalities, demonstrated that our TMCA achieved superior performances.

Title: Information-Theoretic Generative Clustering of Documents

• Authors: Xin Du, Kumiko Tanaka-Ishii
• Subjects: cs.LG, cs.CL, cs.IR, cs.IT
• Abstract URL: https://arxiv.org/abs/2412.13534
• Pdf URL: https://arxiv.org/pdf/2412.13534
• Copy Paste: [[2412.13534]] Information-Theoretic Generative Clustering of Documents(https://arxiv.org/abs/2412.13534)
• Keywords: generative, large language model
• Abstract: We present {\em generative clustering} (GC) for clustering a set of documents, $\mathrm{X}$, by using texts $\mathrm{Y}$ generated by large language models (LLMs) instead of by clustering the original documents $\mathrm{X}$. Because LLMs provide probability distributions, the similarity between two documents can be rigorously defined in an information-theoretic manner by the KL divergence. We also propose a natural, novel clustering algorithm by using importance sampling. We show that GC achieves the state-of-the-art performance, outperforming any previous clustering method often by a large margin. Furthermore, we show an application to generative document retrieval in which documents are indexed via hierarchical clustering and our method improves the retrieval accuracy.

Title: MetaRuleGPT: Recursive Numerical Reasoning of Language Models Trained with Simple Rules

• Authors: Kejie Chen, Lin Wang, Qinghai Zhang, Renjun Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13536
• Pdf URL: https://arxiv.org/pdf/2412.13536
• Copy Paste: [[2412.13536]] MetaRuleGPT: Recursive Numerical Reasoning of Language Models Trained with Simple Rules(https://arxiv.org/abs/2412.13536)
• Keywords: transformer, large language model
• Abstract: Recent studies have highlighted the limitations of large language models in mathematical reasoning, particularly their inability to capture the underlying logic. Inspired by meta-learning, we propose that models should acquire not only task-specific knowledge but also transferable problem-solving skills. We introduce MetaRuleGPT, a novel Transformer-based architecture that performs precise numerical calculations and complex logical operations by learning and combining different rules. In contrast with traditional training sets, which are heavily composed of massive raw instance data, MetaRuleGPT is pre-trained on much less abstract datasets containing basic, compound, and iterative rules for mathematical reasoning. Extensive experimental results demonstrate MetaRuleGPT can mimic human's rule-following capabilities, break down complexity, and iteratively derive accurate results for complex mathematical problems. These findings prove the potential of rule learning to enhance the numerical reasoning abilities of language models.

Title: Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning

• Authors: Yingjie Zhu, Xuefeng Bai, Kehai Chen, Yang Xiang, Min Zhang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13540
• Pdf URL: https://arxiv.org/pdf/2412.13540
• Copy Paste: [[2412.13540]] Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning(https://arxiv.org/abs/2412.13540)
• Keywords: robust
• Abstract: Large Vision-Language Models (LVLMs) have demonstrated remarkable performance across diverse tasks. Despite great success, recent studies show that LVLMs encounter substantial limitations when engaging with visual graphs. To study the reason behind these limitations, we propose VGCure, a comprehensive benchmark covering 22 tasks for examining the fundamental graph understanding and reasoning capacities of LVLMs. Extensive evaluations conducted on 14 LVLMs reveal that LVLMs are weak in basic graph understanding and reasoning tasks, particularly those concerning relational or structurally complex information. Based on this observation, we propose a structure-aware fine-tuning framework to enhance LVLMs with structure learning abilities through 3 self-supervised learning tasks. Experiments validate the effectiveness of our method in improving LVLMs' zero-shot performance on fundamental graph learning tasks, as well as enhancing the robustness of LVLMs against complex visual graphs.

Title: Spatio-Temporal Fuzzy-oriented Multi-Modal Meta-Learning for Fine-grained Emotion Recognition

• Authors: Jingyao Wang, Yuxuan Yang, Wenwen Qiang, Changwen Zheng, Hui Xiong
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13541
• Pdf URL: https://arxiv.org/pdf/2412.13541
• Copy Paste: [[2412.13541]] Spatio-Temporal Fuzzy-oriented Multi-Modal Meta-Learning for Fine-grained Emotion Recognition(https://arxiv.org/abs/2412.13541)
• Keywords: robust
• Abstract: Fine-grained emotion recognition (FER) plays a vital role in various fields, such as disease diagnosis, personalized recommendations, and multimedia mining. However, existing FER methods face three key challenges in real-world applications: (i) they rely on large amounts of continuously annotated data to ensure accuracy since emotions are complex and ambiguous in reality, which is costly and time-consuming; (ii) they cannot capture the temporal heterogeneity caused by changing emotion patterns, because they usually assume that the temporal correlation within sampling periods is the same; (iii) they do not consider the spatial heterogeneity of different FER scenarios, that is, the distribution of emotion information in different data may have bias or interference. To address these challenges, we propose a Spatio-Temporal Fuzzy-oriented Multi-modal Meta-learning framework (ST-F2M). Specifically, ST-F2M first divides the multi-modal videos into multiple views, and each view corresponds to one modality of one emotion. Multiple randomly selected views for the same emotion form a meta-training task. Next, ST-F2M uses an integrated module with spatial and temporal convolutions to encode the data of each task, reflecting the spatial and temporal heterogeneity. Then it adds fuzzy semantic information to each task based on generalized fuzzy rules, which helps handle the complexity and ambiguity of emotions. Finally, ST-F2M learns emotion-related general meta-knowledge through meta-recurrent neural networks to achieve fast and robust fine-grained emotion recognition. Extensive experiments show that ST-F2M outperforms various state-of-the-art methods in terms of accuracy and model efficiency. In addition, we construct ablation studies and further analysis to explore why ST-F2M performs well.

Title: Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning

• Authors: Yunbin Tu, Liang Li, Li Su, Qingming Huang
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13543
• Pdf URL: https://arxiv.org/pdf/2412.13543
• Copy Paste: [[2412.13543]] Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning(https://arxiv.org/abs/2412.13543)
• Keywords: segmentation
• Abstract: Video has emerged as a favored multimedia format on the internet. To better gain video contents, a new topic HIREST is presented, including video retrieval, moment retrieval, moment segmentation, and step-captioning. The pioneering work chooses the pre-trained CLIP-based model for video retrieval, and leverages it as a feature extractor for other three challenging tasks solved in a multi-task learning paradigm. Nevertheless, this work struggles to learn the comprehensive cognition of user-preferred content, due to disregarding the hierarchies and association relations across modalities. In this paper, guided by the shallow-to-deep principle, we propose a query-centric audio-visual cognition (QUAG) network to construct a reliable multi-modal representation for moment retrieval, segmentation and step-captioning. Specifically, we first design the modality-synergistic perception to obtain rich audio-visual content, by modeling global contrastive alignment and local fine-grained interaction between visual and audio modalities. Then, we devise the query-centric cognition that uses the deep-level query to perform the temporal-channel filtration on the shallow-level audio-visual representation. This can cognize user-preferred content and thus attain a query-centric audio-visual representation for three tasks. Extensive experiments show QUAG achieves the SOTA results on HIREST. Further, we test QUAG on the query-based video summarization task and verify its good generalization.

Title: EscapeBench: Pushing Language Models to Think Outside the Box

• Authors: Cheng Qian, Peixuan Han, Qinyu Luo, Bingxiang He, Xiusi Chen, Yuji Zhang, Hongyi Du, Jiarui Yao, Xiaocheng Yang, Denghui Zhang, Yunzhu Li, Heng Ji
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13549
• Pdf URL: https://arxiv.org/pdf/2412.13549
• Copy Paste: [[2412.13549]] EscapeBench: Pushing Language Models to Think Outside the Box(https://arxiv.org/abs/2412.13549)
• Keywords: robust
• Abstract: Language model agents excel in long-session planning and reasoning, but existing benchmarks primarily focus on goal-oriented tasks with explicit objectives, neglecting creative adaptation in unfamiliar environments. To address this, we introduce EscapeBench, a benchmark suite of room escape game environments designed to challenge agents with creative reasoning, unconventional tool use, and iterative problem-solving to uncover implicit goals. Our results show that current LM models, despite employing working memory and Chain-of-Thought reasoning, achieve only 15% average progress without hints, highlighting their limitations in creativity. To bridge this gap, we propose EscapeAgent, a framework designed to enhance creative reasoning through Foresight (innovative tool use) and Reflection (identifying unsolved tasks). Experiments show that EscapeAgent can execute action chains over 1,000 steps while maintaining logical coherence. It navigates and completes games with up to 40% fewer steps and hints, performs robustly across varying difficulty levels, and achieves higher action success rates with more efficient and innovative puzzle-solving strategies. All the data and codes are released.

Title: Large Language Model Federated Learning with Blockchain and Unlearning for Cross-Organizational Collaboration

• Authors: Xuhan Zuo, Minghao Wang, Tianqing Zhu, Shui Yu, Wanlei Zhou
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.13551
• Pdf URL: https://arxiv.org/pdf/2412.13551
• Copy Paste: [[2412.13551]] Large Language Model Federated Learning with Blockchain and Unlearning for Cross-Organizational Collaboration(https://arxiv.org/abs/2412.13551)
• Keywords: privacy, protect, federate, large language model
• Abstract: Large language models (LLMs) have transformed the way computers understand and process human language, but using them effectively across different organizations remains still difficult. When organizations work together to improve LLMs, they face several main challenges. First, organizations hesitate to share their valuable data with others. Second, competition between organizations creates trust problems during collaboration. Third, new privacy laws require organizations to be able to delete specific data when requested, which is especially difficult when multiple organizations are learning from shared data. Traditional federated learning approaches do not address these interconnected challenges, particularly in scenarios where participants cannot fully trust each other or the central aggregator. To overcome these limitations, we propose a hybrid blockchain-based federated learning framework that uniquely combines public and private blockchain architectures with multi-agent reinforcement learning. Our framework enables transparent sharing of model update through the public blockchain while protecting sensitive computations in private chains. Each organization operates as an intelligent agent, using Q-learning to optimize its participation strategy and resource allocation, thus aligning individual incentives with collective goals. Notably, we introduce an efficient unlearning mechanism based on Low-Rank Adaptation (LoRA) that enables selective removal of specific data contributions without compromising the model's overall performance. Through extensive experimentation on real-world datasets, we demonstrate that our framework effectively balances privacy protection, trust establishment, and regulatory compliance while maintaining high model performance.

Title: Indirect Query Bayesian Optimization with Integrated Feedback

• Authors: Mengyan Zhang, Shahine Bouabid, Cheng Soon Ong, Seth Flaxman, Dino Sejdinovic
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13559
• Pdf URL: https://arxiv.org/pdf/2412.13559
• Copy Paste: [[2412.13559]] Indirect Query Bayesian Optimization with Integrated Feedback(https://arxiv.org/abs/2412.13559)
• Keywords: privacy
• Abstract: We develop the framework of Indirect Query Bayesian Optimization (IQBO), a new class of Bayesian optimization problems where the integrated feedback is given via a conditional expectation of the unknown function $f$ to be optimized. The underlying conditional distribution can be unknown and learned from data. The goal is to find the global optimum of $f$ by adaptively querying and observing in the space transformed by the conditional distribution. This is motivated by real-world applications where one cannot access direct feedback due to privacy, hardware or computational constraints. We propose the Conditional Max-Value Entropy Search (CMES) acquisition function to address this novel setting, and propose a hierarchical search algorithm to address the multi-resolution setting and improve the computational efficiency. We show regret bounds for our proposed methods and demonstrate the effectiveness of our approaches on simulated optimization tasks.

Title: Multi-View Pedestrian Occupancy Prediction with a Novel Synthetic Dataset

• Authors: Sithu Aung, Min-Cheol Sagong, Junghyun Cho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13569
• Pdf URL: https://arxiv.org/pdf/2412.13569
• Copy Paste: [[2412.13569]] Multi-View Pedestrian Occupancy Prediction with a Novel Synthetic Dataset(https://arxiv.org/abs/2412.13569)
• Keywords: robust
• Abstract: We address an advanced challenge of predicting pedestrian occupancy as an extension of multi-view pedestrian detection in urban traffic. To support this, we have created a new synthetic dataset called MVP-Occ, designed for dense pedestrian scenarios in large-scale scenes. Our dataset provides detailed representations of pedestrians using voxel structures, accompanied by rich semantic scene understanding labels, facilitating visual navigation and insights into pedestrian spatial information. Furthermore, we present a robust baseline model, termed OmniOcc, capable of predicting both the voxel occupancy state and panoptic labels for the entire scene from multi-view images. Through in-depth analysis, we identify and evaluate the key elements of our proposed model, highlighting their specific contributions and importance.

Title: Bridge then Begin Anew: Generating Target-relevant Intermediate Model for Source-free Visual Emotion Adaptation

• Authors: Jiankun Zhu, Sicheng Zhao, Jing Jiang, Wenbo Tang, Zhaopan Xu, Tingting Han, Pengfei Xu, Hongxun Yao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13577
• Pdf URL: https://arxiv.org/pdf/2412.13577
• Copy Paste: [[2412.13577]] Bridge then Begin Anew: Generating Target-relevant Intermediate Model for Source-free Visual Emotion Adaptation(https://arxiv.org/abs/2412.13577)
• Keywords: privacy
• Abstract: Visual emotion recognition (VER), which aims at understanding humans' emotional reactions toward different visual stimuli, has attracted increasing attention. Given the subjective and ambiguous characteristics of emotion, annotating a reliable large-scale dataset is hard. For reducing reliance on data labeling, domain adaptation offers an alternative solution by adapting models trained on labeled source data to unlabeled target data. Conventional domain adaptation methods require access to source data. However, due to privacy concerns, source emotional data may be inaccessible. To address this issue, we propose an unexplored task: source-free domain adaptation (SFDA) for VER, which does not have access to source data during the adaptation process. To achieve this, we propose a novel framework termed Bridge then Begin Anew (BBA), which consists of two steps: domain-bridged model generation (DMG) and target-related model adaptation (TMA). First, the DMG bridges cross-domain gaps by generating an intermediate model, avoiding direct alignment between two VER datasets with significant differences. Then, the TMA begins training the target model anew to fit the target structure, avoiding the influence of source-specific knowledge. Extensive experiments are conducted on six SFDA settings for VER. The results demonstrate the effectiveness of BBA, which achieves remarkable performance gains compared with state-of-the-art SFDA methods and outperforms representative unsupervised domain adaptation approaches.

Title: Socio-Culturally Aware Evaluation Framework for LLM-Based Content Moderation

• Authors: Shanu Kumar, Gauri Kholkar, Saish Mendke, Anubhav Sadana, Parag Agrawal, Sandipan Dandapat
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13578
• Pdf URL: https://arxiv.org/pdf/2412.13578
• Copy Paste: [[2412.13578]] Socio-Culturally Aware Evaluation Framework for LLM-Based Content Moderation(https://arxiv.org/abs/2412.13578)
• Keywords: large language model
• Abstract: With the growth of social media and large language models, content moderation has become crucial. Many existing datasets lack adequate representation of different groups, resulting in unreliable assessments. To tackle this, we propose a socio-culturally aware evaluation framework for LLM-driven content moderation and introduce a scalable method for creating diverse datasets using persona-based generation. Our analysis reveals that these datasets provide broader perspectives and pose greater challenges for LLMs than diversity-focused generation methods without personas. This challenge is especially pronounced in smaller LLMs, emphasizing the difficulties they encounter in moderating such diverse content.

Title: EvoWiki: Evaluating LLMs on Evolving Knowledge

• Authors: Wei Tang, Yixin Cao, Yang Deng, Jiahao Ying, Bo Wang, Yizhe Yang, Yuyue Zhao, Qi Zhang, Xuanjing Huang, Yugang Jiang, Yong Liao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13582
• Pdf URL: https://arxiv.org/pdf/2412.13582
• Copy Paste: [[2412.13582]] EvoWiki: Evaluating LLMs on Evolving Knowledge(https://arxiv.org/abs/2412.13582)
• Keywords: robust, large language model
• Abstract: Knowledge utilization is a critical aspect of LLMs, and understanding how they adapt to evolving knowledge is essential for their effective deployment. However, existing benchmarks are predominantly static, failing to capture the evolving nature of LLMs and knowledge, leading to inaccuracies and vulnerabilities such as contamination. In this paper, we introduce EvoWiki, an evolving dataset designed to reflect knowledge evolution by categorizing information into stable, evolved, and uncharted states. EvoWiki is fully auto-updatable, enabling precise evaluation of continuously changing knowledge and newly released LLMs. Through experiments with Retrieval-Augmented Generation (RAG) and Contunual Learning (CL), we evaluate how effectively LLMs adapt to evolving knowledge. Our results indicate that current models often struggle with evolved knowledge, frequently providing outdated or incorrect responses. Moreover, the dataset highlights a synergistic effect between RAG and CL, demonstrating their potential to better adapt to evolving knowledge. EvoWiki provides a robust benchmark for advancing future research on the knowledge evolution capabilities of large language models.

Title: SemiDFL: A Semi-Supervised Paradigm for Decentralized Federated Learning

• Authors: Xinyang Liu, Pengchao Han, Xuan Li, Bo Liu
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.13589
• Pdf URL: https://arxiv.org/pdf/2412.13589
• Copy Paste: [[2412.13589]] SemiDFL: A Semi-Supervised Paradigm for Decentralized Federated Learning(https://arxiv.org/abs/2412.13589)
• Keywords: federate, diffusion
• Abstract: Decentralized federated learning (DFL) realizes cooperative model training among connected clients without relying on a central server, thereby mitigating communication bottlenecks and eliminating the single-point failure issue present in centralized federated learning (CFL). Most existing work on DFL focuses on supervised learning, assuming each client possesses sufficient labeled data for local training. However, in real-world applications, much of the data is unlabeled. We address this by considering a challenging yet practical semisupervised learning (SSL) scenario in DFL, where clients may have varying data sources: some with few labeled samples, some with purely unlabeled data, and others with both. In this work, we propose SemiDFL, the first semi-supervised DFL method that enhances DFL performance in SSL scenarios by establishing a consensus in both data and model spaces. Specifically, we utilize neighborhood information to improve the quality of pseudo-labeling, which is crucial for effectively leveraging unlabeled data. We then design a consensusbased diffusion model to generate synthesized data, which is used in combination with pseudo-labeled data to create mixed datasets. Additionally, we develop an adaptive aggregation method that leverages the model accuracy of synthesized data to further enhance SemiDFL performance. Through extensive experimentation, we demonstrate the remarkable performance superiority of the proposed DFL-Semi method over existing CFL and DFL schemes in both IID and non-IID SSL scenarios.

Title: Beyond Outcomes: Transparent Assessment of LLM Reasoning in Games

• Authors: Wenye Lin, Jonathan Roberts, Yunhan Yang, Samuel Albanie, Zongqing Lu, Kai Han
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13602
• Pdf URL: https://arxiv.org/pdf/2412.13602
• Copy Paste: [[2412.13602]] Beyond Outcomes: Transparent Assessment of LLM Reasoning in Games(https://arxiv.org/abs/2412.13602)
• Keywords: robust, interpretability, large language model
• Abstract: Large Language Models (LLMs) are increasingly deployed in real-world applications that demand complex reasoning. To track progress, robust benchmarks are required to evaluate their capabilities beyond superficial pattern recognition. However, current LLM reasoning benchmarks often face challenges such as insufficient interpretability, performance saturation or data contamination. To address these challenges, we introduce GAMEBoT, a gaming arena designed for rigorous and transparent assessment of LLM reasoning capabilities. GAMEBoT decomposes complex reasoning in games into predefined modular subproblems. This decomposition allows us to design a suite of Chain-of-Thought (CoT) prompts that leverage domain knowledge to guide LLMs in addressing these subproblems before action selection. Furthermore, we develop a suite of rule-based algorithms to generate ground truth for these subproblems, enabling rigorous validation of the LLMs' intermediate reasoning steps. This approach facilitates evaluation of both the quality of final actions and the accuracy of the underlying reasoning process. GAMEBoT also naturally alleviates the risk of data contamination through dynamic games and head-to-head LLM competitions. We benchmark 17 prominent LLMs across eight games, encompassing various strategic abilities and game characteristics. Our results suggest that GAMEBoT presents a significant challenge, even when LLMs are provided with detailed CoT prompts. Project page: \url{this https URL}

Title: Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production

• Authors: Shengeng Tang, Jiayi He, Dan Guo, Yanyan Wei, Feng Li, Richang Hong
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.13609
• Pdf URL: https://arxiv.org/pdf/2412.13609
• Copy Paste: [[2412.13609]] Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production(https://arxiv.org/abs/2412.13609)
• Keywords: diffusion
• Abstract: Sign Language Production (SLP) aims to generate semantically consistent sign videos from textual statements, where the conversion from textual glosses to sign poses (G2P) is a crucial step. Existing G2P methods typically treat sign poses as discrete three-dimensional coordinates and directly fit them, which overlooks the relative positional relationships among joints. To this end, we provide a new perspective, constraining joint associations and gesture details by modeling the limb bones to improve the accuracy and naturalness of the generated poses. In this work, we propose a pioneering iconicity disentangled diffusion framework, termed Sign-IDD, specifically designed for SLP. Sign-IDD incorporates a novel Iconicity Disentanglement (ID) module to bridge the gap between relative positions among joints. The ID module disentangles the conventional 3D joint representation into a 4D bone representation, comprising the 3D spatial direction vector and 1D spatial distance vector between adjacent joints. Additionally, an Attribute Controllable Diffusion (ACD) module is introduced to further constrain joint associations, in which the attribute separation layer aims to separate the bone direction and length attributes, and the attribute control layer is designed to guide the pose generation by leveraging the above attributes. The ACD module utilizes the gloss embeddings as semantic conditions and finally generates sign poses from noise embeddings. Extensive experiments on PHOENIX14T and USTC-CSL datasets validate the effectiveness of our method. The code is available at: this https URL.

Title: Robust Tracking via Mamba-based Context-aware Token Learning

• Authors: Jinxia Xie, Bineng Zhong, Qihua Liang, Ning Li, Zhiyi Mo, Shuxiang Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13611
• Pdf URL: https://arxiv.org/pdf/2412.13611
• Copy Paste: [[2412.13611]] Robust Tracking via Mamba-based Context-aware Token Learning(https://arxiv.org/abs/2412.13611)
• Keywords: robust
• Abstract: How to make a good trade-off between performance and computational cost is crucial for a tracker. However, current famous methods typically focus on complicated and time-consuming learning that combining temporal and appearance information by input more and more images (or features). Consequently, these methods not only increase the model's computational source and learning burden but also introduce much useless and potentially interfering information. To alleviate the above issues, we propose a simple yet robust tracker that separates temporal information learning from appearance modeling and extracts temporal relations from a set of representative tokens rather than several images (or features). Specifically, we introduce one track token for each frame to collect the target's appearance information in the backbone. Then, we design a mamba-based Temporal Module for track tokens to be aware of context by interacting with other track tokens within a sliding window. This module consists of a mamba layer with autoregressive characteristic and a cross-attention layer with strong global perception ability, ensuring sufficient interaction for track tokens to perceive the appearance changes and movement trends of the target. Finally, track tokens serve as a guidance to adjust the appearance feature for the final prediction in the head. Experiments show our method is effective and achieves competitive performance on multiple benchmarks at a real-time speed. Code and trained models will be available at this https URL.

Title: Are LLMs Good Literature Review Writers? Evaluating the Literature Review Writing Ability of Large Language Models

• Authors: Xuemei Tang, Xufeng Duan, Zhenguang G. Cai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13612
• Pdf URL: https://arxiv.org/pdf/2412.13612
• Copy Paste: [[2412.13612]] Are LLMs Good Literature Review Writers? Evaluating the Literature Review Writing Ability of Large Language Models(https://arxiv.org/abs/2412.13612)
• Keywords: large language model
• Abstract: The literature review is a crucial form of academic writing that involves complex processes of literature collection, organization, and summarization. The emergence of large language models (LLMs) has introduced promising tools to automate these processes. However, their actual capabilities in writing comprehensive literature reviews remain underexplored, such as whether they can generate accurate and reliable references. To address this gap, we propose a framework to assess the literature review writing ability of LLMs automatically. We evaluate the performance of LLMs across three tasks: generating references, writing abstracts, and writing literature reviews. We employ external tools for a multidimensional evaluation, which includes assessing hallucination rates in references, semantic coverage, and factual consistency with human-written context. By analyzing the experimental results, we find that, despite advancements, even the most sophisticated models still cannot avoid generating hallucinated references. Additionally, different models exhibit varying performance in literature review writing across different disciplines.

Title: MambaLCT: Boosting Tracking via Long-term Context State Space Model

• Authors: Xiaohai Li, Bineng Zhong, Qihua Liang, Guorong Li, Zhiyi Mo, Shuxiang Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13615
• Pdf URL: https://arxiv.org/pdf/2412.13615
• Copy Paste: [[2412.13615]] MambaLCT: Boosting Tracking via Long-term Context State Space Model(https://arxiv.org/abs/2412.13615)
• Keywords: robust
• Abstract: Effectively constructing context information with long-term dependencies from video sequences is crucial for object tracking. However, the context length constructed by existing work is limited, only considering object information from adjacent frames or video clips, leading to insufficient utilization of contextual information. To address this issue, we propose MambaLCT, which constructs and utilizes target variation cues from the first frame to the current frame for robust tracking. First, a novel unidirectional Context Mamba module is designed to scan frame features along the temporal dimension, gathering target change cues throughout the entire sequence. Specifically, target-related information in frame features is compressed into a hidden state space through selective scanning mechanism. The target information across the entire video is continuously aggregated into target variation cues. Next, we inject the target change cues into the attention mechanism, providing temporal information for modeling the relationship between the template and search frames. The advantage of MambaLCT is its ability to continuously extend the length of the context, capturing complete target change cues, which enhances the stability and robustness of the tracker. Extensive experiments show that long-term context information enhances the model's ability to perceive targets in complex scenarios. MambaLCT achieves new SOTA performance on six benchmarks while maintaining real-time running speeds.

Title: LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning

• Authors: Yansheng Mao, Jiaqi Li, Fanxu Meng, Jing Xiong, Zilong Zheng, Muhan Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13626
• Pdf URL: https://arxiv.org/pdf/2412.13626
• Copy Paste: [[2412.13626]] LIFT: Improving Long Context Understanding Through Long Input Fine-Tuning(https://arxiv.org/abs/2412.13626)
• Keywords: large language model
• Abstract: Long context understanding remains challenging for large language models due to their limited context windows. This paper introduces Long Input Fine-Tuning (LIFT) for long context modeling, a novel framework that enhances LLM performance on long-context tasks by adapting model parameters to the context at test time. LIFT enables efficient processing of lengthy inputs without the computational burden of offline long-context adaptation, and can improve the long-context capabilities of arbitrary short-context models. The framework is further enhanced by integrating in-context learning and pre-LIFT supervised fine-tuning. The combination of in-context learning and LIFT enables short-context models like Llama 3 to handle arbitrarily long contexts and consistently improves their performance on popular long-context benchmarks like LooGLE and LongBench. We also provide a comprehensive analysis of the strengths and limitations of LIFT on long context understanding, offering valuable directions for future research.

Title: TAUDiff: Improving statistical downscaling for extreme weather events using generative diffusion models

• Authors: Rahul Sundar, Nishant Parashar, Antoine Blanchard, Boyko Dodov
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13627
• Pdf URL: https://arxiv.org/pdf/2412.13627
• Copy Paste: [[2412.13627]] TAUDiff: Improving statistical downscaling for extreme weather events using generative diffusion models(https://arxiv.org/abs/2412.13627)
• Keywords: diffusion, generative
• Abstract: Deterministic regression-based downscaling models for climate variables often suffer from spectral bias, which can be mitigated by generative models like diffusion models. To enable efficient and reliable simulation of extreme weather events, it is crucial to achieve rapid turnaround, dynamical consistency, and accurate spatio-temporal spectral recovery. We propose an efficient correction diffusion model, TAUDiff, that combines a deterministic spatio-temporal model for mean field downscaling with a smaller generative diffusion model for recovering the fine-scale stochastic features. We demonstrate the efficacy of this approach on downscaling atmospheric wind velocity fields obtained from coarse GCM simulations. Our approach can not only ensure quicker simulation of extreme events but also reduce overall carbon footprint due to low inference times.

Title: Self-control: A Better Conditional Mechanism for Masked Autoregressive Model

• Authors: Qiaoying Qu, Shiyu Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13635
• Pdf URL: https://arxiv.org/pdf/2412.13635
• Copy Paste: [[2412.13635]] Self-control: A Better Conditional Mechanism for Masked Autoregressive Model(https://arxiv.org/abs/2412.13635)
• Keywords: generative
• Abstract: Autoregressive conditional image generation algorithms are capable of generating photorealistic images that are consistent with given textual or image conditions, and have great potential for a wide range of applications. Nevertheless, the majority of popular autoregressive image generation methods rely heavily on vector quantization, and the inherent discrete characteristic of codebook presents a considerable challenge to achieving high-quality image generation. To address this limitation, this paper introduces a novel conditional introduction network for continuous masked autoregressive models. The proposed self-control network serves to mitigate the negative impact of vector quantization on the quality of the generated images, while simultaneously enhancing the conditional control during the generation process. In particular, the self-control network is constructed upon a continuous mask autoregressive generative model, which incorporates multimodal conditional information, including text and images, into a unified autoregressive sequence in a serial manner. Through a self-attention mechanism, the network is capable of generating images that are controllable based on specific conditions. The self-control network discards the conventional cross-attention-based conditional fusion mechanism and effectively unifies the conditional and generative information within the same space, thereby facilitating more seamless learning and fusion of multimodal features.

Title: G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o

• Authors: Tony Cheng Tong, Sirui He, Zhiwen Shao, Dit-Yan Yeung
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13647
• Pdf URL: https://arxiv.org/pdf/2412.13647
• Copy Paste: [[2412.13647]] G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o(https://arxiv.org/abs/2412.13647)
• Keywords: large language model
• Abstract: Evaluation metric of visual captioning is important yet not thoroughly explored. Traditional metrics like BLEU, METEOR, CIDEr, and ROUGE often miss semantic depth, while trained metrics such as CLIP-Score, PAC-S, and Polos are limited in zero-shot scenarios. Advanced Language Model-based metrics also struggle with aligning to nuanced human preferences. To address these issues, we introduce G-VEval, a novel metric inspired by G-Eval and powered by the new GPT-4o. G-VEval uses chain-of-thought reasoning in large multimodal models and supports three modes: reference-free, reference-only, and combined, accommodating both video and image inputs. We also propose MSVD-Eval, a new dataset for video captioning evaluation, to establish a more transparent and consistent framework for both human experts and evaluation metrics. It is designed to address the lack of clear criteria in existing datasets by introducing distinct dimensions of Accuracy, Completeness, Conciseness, and Relevance (ACCR). Extensive results show that G-VEval outperforms existing methods in correlation with human annotations, as measured by Kendall tau-b and Kendall tau-c. This provides a flexible solution for diverse captioning tasks and suggests a straightforward yet effective approach for large language models to understand video content, paving the way for advancements in automated captioning. Codes are available at this https URL

Title: RelationField: Relate Anything in Radiance Fields

• Authors: Sebastian Koch, Johanna Wald, Mirco Colosi, Narunas Vaskevicius, Pedro Hermosilla, Federico Tombari, Timo Ropinski
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13652
• Pdf URL: https://arxiv.org/pdf/2412.13652
• Copy Paste: [[2412.13652]] RelationField: Relate Anything in Radiance Fields(https://arxiv.org/abs/2412.13652)
• Keywords: segmentation
• Abstract: Neural radiance fields are an emerging 3D scene representation and recently even been extended to learn features for scene understanding by distilling open-vocabulary features from vision-language models. However, current method primarily focus on object-centric representations, supporting object segmentation or detection, while understanding semantic relationships between objects remains largely unexplored. To address this gap, we propose RelationField, the first method to extract inter-object relationships directly from neural radiance fields. RelationField represents relationships between objects as pairs of rays within a neural radiance field, effectively extending its formulation to include implicit relationship queries. To teach RelationField complex, open-vocabulary relationships, relationship knowledge is distilled from multi-modal LLMs. To evaluate RelationField, we solve open-vocabulary 3D scene graph generation tasks and relationship-guided instance segmentation, achieving state-of-the-art performance in both tasks. See the project website at this https URL.

Title: GAGS: Granularity-Aware Feature Distillation for Language Gaussian Splatting

• Authors: Yuning Peng, Haiping Wang, Yuan Liu, Chenglu Wen, Zhen Dong, Bisheng Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13654
• Pdf URL: https://arxiv.org/pdf/2412.13654
• Copy Paste: [[2412.13654]] GAGS: Granularity-Aware Feature Distillation for Language Gaussian Splatting(https://arxiv.org/abs/2412.13654)
• Keywords: segmentation
• Abstract: 3D open-vocabulary scene understanding, which accurately perceives complex semantic properties of objects in space, has gained significant attention in recent years. In this paper, we propose GAGS, a framework that distills 2D CLIP features into 3D Gaussian splatting, enabling open-vocabulary queries for renderings on arbitrary viewpoints. The main challenge of distilling 2D features for 3D fields lies in the multiview inconsistency of extracted 2D features, which provides unstable supervision for the 3D feature field. GAGS addresses this challenge with two novel strategies. First, GAGS associates the prompt point density of SAM with the camera distances, which significantly improves the multiview consistency of segmentation results. Second, GAGS further decodes a granularity factor to guide the distillation process and this granularity factor can be learned in a unsupervised manner to only select the multiview consistent 2D features in the distillation process. Experimental results on two datasets demonstrate significant performance and stability improvements of GAGS in visual grounding and semantic segmentation, with an inference speed 2$\times$ faster than baseline methods. The code and additional results are available at this https URL .

Title: VIIS: Visible and Infrared Information Synthesis for Severe Low-light Image Enhancement

• Authors: Chen Zhao, Mengyuan Yu, Fan Yang, Peiguang Jing
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13655
• Pdf URL: https://arxiv.org/pdf/2412.13655
• Copy Paste: [[2412.13655]] VIIS: Visible and Infrared Information Synthesis for Severe Low-light Image Enhancement(https://arxiv.org/abs/2412.13655)
• Keywords: diffusion
• Abstract: Images captured in severe low-light circumstances often suffer from significant information absence. Existing singular modality image enhancement methods struggle to restore image regions lacking valid information. By leveraging light-impervious infrared images, visible and infrared image fusion methods have the potential to reveal information hidden in darkness. However, they primarily emphasize inter-modal complementation but neglect intra-modal enhancement, limiting the perceptual quality of output images. To address these limitations, we propose a novel task, dubbed visible and infrared information synthesis (VIIS), which aims to achieve both information enhancement and fusion of the two modalities. Given the difficulty in obtaining ground truth in the VIIS task, we design an information synthesis pretext task (ISPT) based on image augmentation. We employ a diffusion model as the framework and design a sparse attention-based dual-modalities residual (SADMR) conditioning mechanism to enhance information interaction between the two modalities. This mechanism enables features with prior knowledge from both modalities to adaptively and iteratively attend to each modality's information during the denoising process. Our extensive experiments demonstrate that our model qualitatively and quantitatively outperforms not only the state-of-the-art methods in relevant fields but also the newly designed baselines capable of both information enhancement and fusion. The code is available at this https URL.

Title: PsyDT: Using LLMs to Construct the Digital Twin of Psychological Counselor with Personalized Counseling Style for Psychological Counseling

• Authors: Haojie Xie, Yirong Chen, Xiaofen Xing, Jingkai Lin, Xiangmin Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13660
• Pdf URL: https://arxiv.org/pdf/2412.13660
• Copy Paste: [[2412.13660]] PsyDT: Using LLMs to Construct the Digital Twin of Psychological Counselor with Personalized Counseling Style for Psychological Counseling(https://arxiv.org/abs/2412.13660)
• Keywords: large language model
• Abstract: Currently, large language models (LLMs) have made significant progress in the field of psychological counseling. However, existing mental health LLMs overlook a critical issue where they do not consider the fact that different psychological counselors exhibit different personal styles, including linguistic style and therapy techniques, etc. As a result, these LLMs fail to satisfy the individual needs of clients who seek different counseling styles. To help bridge this gap, we propose PsyDT, a novel framework using LLMs to construct the Digital Twin of Psychological counselor with personalized counseling style. Compared to the time-consuming and costly approach of collecting a large number of real-world counseling cases to create a specific counselor's digital twin, our framework offers a faster and more cost-effective solution. To construct PsyDT, we utilize dynamic one-shot learning by using GPT-4 to capture counselor's unique counseling style, mainly focusing on linguistic style and therapy techniques. Subsequently, using existing single-turn long-text dialogues with client's questions, GPT-4 is guided to synthesize multi-turn dialogues of specific counselor. Finally, we fine-tune the LLMs on the synthetic dataset, PsyDTCorpus, to achieve the digital twin of psychological counselor with personalized counseling style. Experimental results indicate that our proposed PsyDT framework can synthesize multi-turn dialogues that closely resemble real-world counseling cases and demonstrate better performance compared to other baselines, thereby show that our framework can effectively construct the digital twin of psychological counselor with a specific counseling style.

Title: Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference

• Authors: Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, Iacopo Poli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13663
• Pdf URL: https://arxiv.org/pdf/2412.13663
• Copy Paste: [[2412.13663]] Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference(https://arxiv.org/abs/2412.13663)
• Keywords: transformer
• Abstract: Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks with respect to larger decoder-only models. Despite being the workhorse of numerous production pipelines, there have been limited Pareto improvements to BERT since its release. In this paper, we introduce ModernBERT, bringing modern model optimizations to encoder-only models and representing a major Pareto improvement over older encoders. Trained on 2 trillion tokens with a native 8192 sequence length, ModernBERT models exhibit state-of-the-art results on a large pool of evaluations encompassing diverse classification tasks and both single and multi-vector retrieval on different domains (including code). In addition to strong downstream performance, ModernBERT is also the most speed and memory efficient encoder and is designed for inference on common GPUs.

Title: Evaluation of LLM Vulnerabilities to Being Misused for Personalized Disinformation Generation

• Authors: Aneta Zugecova, Dominik Macko, Ivan Srba, Robert Moro, Jakub Kopal, Katarina Marcincinova, Matus Mesarcik
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.13666
• Pdf URL: https://arxiv.org/pdf/2412.13666
• Copy Paste: [[2412.13666]] Evaluation of LLM Vulnerabilities to Being Misused for Personalized Disinformation Generation(https://arxiv.org/abs/2412.13666)
• Keywords: large language model
• Abstract: The capabilities of recent large language models (LLMs) to generate high-quality content indistinguishable by humans from human-written texts rises many concerns regarding their misuse. Previous research has shown that LLMs can be effectively misused for generating disinformation news articles following predefined narratives. Their capabilities to generate personalized (in various aspects) content have also been evaluated and mostly found usable. However, a combination of personalization and disinformation abilities of LLMs has not been comprehensively studied yet. Such a dangerous combination should trigger integrated safety filters of the LLMs, if there are some. This study fills this gap by evaluation of vulnerabilities of recent open and closed LLMs, and their willingness to generate personalized disinformation news articles in English. We further explore whether the LLMs can reliably meta-evaluate the personalization quality and whether the personalization affects the generated-texts detectability. Our results demonstrate the need for stronger safety-filters and disclaimers, as those are not properly functioning in most of the evaluated LLMs. Additionally, our study revealed that the personalization actually reduces the safety-filter activations; thus effectively functioning as a jailbreak. Such behavior must be urgently addressed by LLM developers and service providers.

Title: Exploring Multi-Modal Integration with Tool-Augmented LLM Agents for Precise Causal Discovery

• Authors: ChengAo Shen, Zhengzhang Chen, Dongsheng Luo, Dongkuan Xu, Haifeng Chen, Jingchao Ni
• Subjects: cs.LG, cs.AI, stat.ME
• Abstract URL: https://arxiv.org/abs/2412.13667
• Pdf URL: https://arxiv.org/pdf/2412.13667
• Copy Paste: [[2412.13667]] Exploring Multi-Modal Integration with Tool-Augmented LLM Agents for Precise Causal Discovery(https://arxiv.org/abs/2412.13667)
• Keywords: large language model
• Abstract: Causal inference is an imperative foundation for decision-making across domains, such as smart health, AI for drug discovery and AIOps. Traditional statistical causal discovery methods, while well-established, predominantly rely on observational data and often overlook the semantic cues inherent in cause-and-effect relationships. The advent of Large Language Models (LLMs) has ushered in an affordable way of leveraging the semantic cues for knowledge-driven causal discovery, but the development of LLMs for causal discovery lags behind other areas, particularly in the exploration of multi-modality data. To bridge the gap, we introduce MATMCD, a multi-agent system powered by tool-augmented LLMs. MATMCD has two key agents: a Data Augmentation agent that retrieves and processes modality-augmented data, and a Causal Constraint agent that integrates multi-modal data for knowledge-driven inference. Delicate design of the inner-workings ensures successful cooperation of the agents. Our empirical study across seven datasets suggests the significant potential of multi-modality enhanced causal discovery.

Title: AntiLeak-Bench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge

• Authors: Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu, William Yang Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13670
• Pdf URL: https://arxiv.org/pdf/2412.13670
• Copy Paste: [[2412.13670]] AntiLeak-Bench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledge(https://arxiv.org/abs/2412.13670)
• Keywords: fair
• Abstract: Data contamination hinders fair LLM evaluation by introducing test data into newer models' training sets. Existing studies solve this challenge by updating benchmarks with newly collected data. However, they fail to guarantee contamination-free evaluation as the newly collected data may contain pre-existing knowledge, and their benchmark updates rely on intensive human labor. To address these issues, we in this paper propose AntiLeak-Bench, an automated anti-leakage benchmarking framework. Instead of simply using newly collected data, we construct samples with explicitly new knowledge absent from LLMs' training sets, which thus ensures strictly contamination-free evaluation. We further design a fully automated workflow to build and update our benchmark without human labor. This significantly reduces the cost of benchmark maintenance to accommodate emerging LLMs. Through extensive experiments, we highlight that data contamination likely exists before LLMs' cutoff time and demonstrate AntiLeak-Bench effectively overcomes this challenge.

Title: On Enhancing Root Cause Analysis with SQL Summaries for Failures in Database Workload Replays at SAP HANA

• Authors: Neetha Jambigi, Joshua Hammesfahr, Moritz Mueller, Thomas Bach, Michael Felderer
• Subjects: cs.LG, cs.DB
• Abstract URL: https://arxiv.org/abs/2412.13679
• Pdf URL: https://arxiv.org/pdf/2412.13679
• Copy Paste: [[2412.13679]] On Enhancing Root Cause Analysis with SQL Summaries for Failures in Database Workload Replays at SAP HANA(https://arxiv.org/abs/2412.13679)
• Keywords: privacy, robust, large language model
• Abstract: Capturing the workload of a database and replaying this workload for a new version of the database can be an effective approach for regression testing. However, false positive errors caused by many factors such as data privacy limitations, time dependency or non-determinism in multi-threaded environment can negatively impact the effectiveness. Therefore, we employ a machine learning based framework to automate the root cause analysis of failures found during replays. However, handling unseen novel issues not found in the training data is one general challenge of machine learning approaches with respect to generalizability of the learned model. We describe how we continue to address this challenge for more robust long-term solutions. From our experience, retraining with new failures is inadequate due to features overlapping across distinct root causes. Hence, we leverage a large language model (LLM) to analyze failed SQL statements and extract concise failure summaries as an additional feature to enhance the classification process. Our experiments show the F1-Macro score improved by 4.77% for our data. We consider our approach beneficial for providing end users with additional information to gain more insights into the found issues and to improve the assessment of the replay results.

Title: MMO-IG: Multi-Class and Multi-Scale Object Image Generation for Remote Sensing

• Authors: Chuang Yang, Bingxuan Zhao, Qing Zhou, Qi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13684
• Pdf URL: https://arxiv.org/pdf/2412.13684
• Copy Paste: [[2412.13684]] MMO-IG: Multi-Class and Multi-Scale Object Image Generation for Remote Sensing(https://arxiv.org/abs/2412.13684)
• Keywords: diffusion, generative
• Abstract: The rapid advancement of deep generative models (DGMs) has significantly advanced research in computer vision, providing a cost-effective alternative to acquiring vast quantities of expensive imagery. However, existing methods predominantly focus on synthesizing remote sensing (RS) images aligned with real images in a global layout view, which limits their applicability in RS image object detection (RSIOD) research. To address these challenges, we propose a multi-class and multi-scale object image generator based on DGMs, termed MMO-IG, designed to generate RS images with supervised object labels from global and local aspects simultaneously. Specifically, from the local view, MMO-IG encodes various RS instances using an iso-spacing instance map (ISIM). During the generation process, it decodes each instance region with iso-spacing value in ISIM-corresponding to both background and foreground instances-to produce RS images through the denoising process of diffusion models. Considering the complex interdependencies among MMOs, we construct a spatial-cross dependency knowledge graph (SCDKG). This ensures a realistic and reliable multidirectional distribution among MMOs for region embedding, thereby reducing the discrepancy between source and target domains. Besides, we propose a structured object distribution instruction (SODI) to guide the generation of synthesized RS image content from a global aspect with SCDKG-based ISIM together. Extensive experimental results demonstrate that our MMO-IG exhibits superior generation capabilities for RS images with dense MMO-supervised labels, and RS detectors pre-trained with MMO-IG show excellent performance on real-world datasets.

Title: Optical aberrations in autonomous driving: Physics-informed parameterized temperature scaling for neural network uncertainty calibration

• Authors: Dominik Werner Wolf, Alexander Braun, Markus Ulrich
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13695
• Pdf URL: https://arxiv.org/pdf/2412.13695
• Copy Paste: [[2412.13695]] Optical aberrations in autonomous driving: Physics-informed parameterized temperature scaling for neural network uncertainty calibration(https://arxiv.org/abs/2412.13695)
• Keywords: robust, segmentation
• Abstract: 'A trustworthy representation of uncertainty is desirable and should be considered as a key feature of any machine learning method' (Huellermeier and Waegeman, 2021). This conclusion of Huellermeier et al. underpins the importance of calibrated uncertainties. Since AI-based algorithms are heavily impacted by dataset shifts, the automotive industry needs to safeguard its system against all possible contingencies. One important but often neglected dataset shift is caused by optical aberrations induced by the windshield. For the verification of the perception system performance, requirements on the AI performance need to be translated into optical metrics by a bijective mapping (Braun, 2023). Given this bijective mapping it is evident that the optical system characteristics add additional information about the magnitude of the dataset shift. As a consequence, we propose to incorporate a physical inductive bias into the neural network calibration architecture to enhance the robustness and the trustworthiness of the AI target application, which we demonstrate by using a semantic segmentation task as an example. By utilizing the Zernike coefficient vector of the optical system as a physical prior we can significantly reduce the mean expected calibration error in case of optical aberrations. As a result, we pave the way for a trustworthy uncertainty representation and for a holistic verification strategy of the perception chain.

Title: Towards Efficient and Explainable Hate Speech Detection via Model Distillation

• Authors: Paloma Piot, Javier Parapar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13698
• Pdf URL: https://arxiv.org/pdf/2412.13698
• Copy Paste: [[2412.13698]] Towards Efficient and Explainable Hate Speech Detection via Model Distillation(https://arxiv.org/abs/2412.13698)
• Keywords: interpretability, explainability, large language model
• Abstract: Automatic detection of hate and abusive language is essential to combat its online spread. Moreover, recognising and explaining hate speech serves to educate people about its negative effects. However, most current detection models operate as black boxes, lacking interpretability and explainability. In this context, Large Language Models (LLMs) have proven effective for hate speech detection and to promote interpretability. Nevertheless, they are computationally costly to run. In this work, we propose distilling big language models by using Chain-of-Thought to extract explanations that support the hate speech classification task. Having small language models for these tasks will contribute to their use in operational settings. In this paper, we demonstrate that distilled models deliver explanations of the same quality as larger models while surpassing them in classification performance. This dual capability, classifying and explaining, advances hate speech detection making it more affordable, understandable and actionable.

Title: Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Models

• Authors: Kunat Pipatanakul, Potsawee Manakul, Natapong Nitarach, Warit Sirichotedumrong, Surapon Nonesung, Teetouch Jaknamon, Parinthapat Pengpun, Pittawat Taveekitworachai, Adisai Na-Thalang, Sittipong Sripaisarnmongkol, Krisanapong Jirayoot, Kasima Tharnpipitchai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13702
• Pdf URL: https://arxiv.org/pdf/2412.13702
• Copy Paste: [[2412.13702]] Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Models(https://arxiv.org/abs/2412.13702)
• Keywords: large language model
• Abstract: This paper introduces Typhoon 2, a series of text and multimodal large language models optimized for the Thai language. The series includes models for text, vision, and audio. Typhoon2-Text builds on state-of-the-art open models, such as Llama 3 and Qwen2, and we perform continual pre-training on a mixture of English and Thai data. We employ various post-training techniques to enhance Thai language performance while preserving the base models' original capabilities. We release text models across a range of sizes, from 1 to 70 billion parameters, available in both base and instruction-tuned variants. Typhoon2-Vision improves Thai document understanding while retaining general visual capabilities, such as image captioning. Typhoon2-Audio introduces an end-to-end speech-to-speech model architecture capable of processing audio, speech, and text inputs and generating both text and speech outputs simultaneously.

Title: Mitigating Adversarial Attacks in LLMs through Defensive Suffix Generation

• Authors: Minkyoung Kim, Yunha Kim, Hyeram Seo, Heejung Choi, Jiye Han, Gaeun Kee, Soyoung Ko, HyoJe Jung, Byeolhee Kim, Young-Hak Kim, Sanghyun Park, Tae Joon Jun
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13705
• Pdf URL: https://arxiv.org/pdf/2412.13705
• Copy Paste: [[2412.13705]] Mitigating Adversarial Attacks in LLMs through Defensive Suffix Generation(https://arxiv.org/abs/2412.13705)
• Keywords: security, attack, robust, large language model
• Abstract: Large language models (LLMs) have exhibited outstanding performance in natural language processing tasks. However, these models remain susceptible to adversarial attacks in which slight input perturbations can lead to harmful or misleading outputs. A gradient-based defensive suffix generation algorithm is designed to bolster the robustness of LLMs. By appending carefully optimized defensive suffixes to input prompts, the algorithm mitigates adversarial influences while preserving the models' utility. To enhance adversarial understanding, a novel total loss function ($L_{\text{total}}$) combining defensive loss ($L_{\text{def}}$) and adversarial loss ($L_{\text{adv}}$) generates defensive suffixes more effectively. Experimental evaluations conducted on open-source LLMs such as Gemma-7B, mistral-7B, Llama2-7B, and Llama2-13B show that the proposed method reduces attack success rates (ASR) by an average of 11\% compared to models without defensive suffixes. Additionally, the perplexity score of Gemma-7B decreased from 6.57 to 3.93 when applying the defensive suffix generated by openELM-270M. Furthermore, TruthfulQA evaluations demonstrate consistent improvements with Truthfulness scores increasing by up to 10\% across tested configurations. This approach significantly enhances the security of LLMs in critical applications without requiring extensive retraining.

Title: JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts

• Authors: Taein Son, Soo Won Seo, Jisong Kim, Seok Hwan Lee, Jun Won Choi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13708
• Pdf URL: https://arxiv.org/pdf/2412.13708
• Copy Paste: [[2412.13708]] JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts(https://arxiv.org/abs/2412.13708)
• Keywords: transformer
• Abstract: Video Action Detection (VAD) involves localizing and categorizing action instances in videos. Videos inherently contain various information sources, including audio, visual cues, and surrounding scene contexts. Effectively leveraging this multi-modal information for VAD is challenging, as the model must accurately focus on action-relevant cues. In this study, we introduce a novel multi-modal VAD architecture called the Joint Actor-centric Visual, Audio, Language Encoder (JoVALE). JoVALE is the first VAD method to integrate audio and visual features with scene descriptive context derived from large image captioning models. The core principle of JoVALE is the actor-centric aggregation of audio, visual, and scene descriptive contexts, where action-related cues from each modality are identified and adaptively combined. We propose a specialized module called the Actor-centric Multi-modal Fusion Network, designed to capture the joint interactions among actors and multi-modal contexts through Transformer architecture. Our evaluation conducted on three popular VAD benchmarks, AVA, UCF101-24, and JHMDB51-21, demonstrates that incorporating multi-modal information leads to significant performance gains. JoVALE achieves state-of-the-art performances. The code will be available at \texttt{this https URL}.

Title: Physics-Based Adversarial Attack on Near-Infrared Human Detector for Nighttime Surveillance Camera Systems

• Authors: Muyao Niu, Zhuoxiao Li, Yifan Zhan, Huy H. Nguyen, Isao Echizen, Yinqiang Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13709
• Pdf URL: https://arxiv.org/pdf/2412.13709
• Copy Paste: [[2412.13709]] Physics-Based Adversarial Attack on Near-Infrared Human Detector for Nighttime Surveillance Camera Systems(https://arxiv.org/abs/2412.13709)
• Keywords: security, attack
• Abstract: Many surveillance cameras switch between daytime and nighttime modes based on illuminance levels. During the day, the camera records ordinary RGB images through an enabled IR-cut filter. At night, the filter is disabled to capture near-infrared (NIR) light emitted from NIR LEDs typically mounted around the lens. While RGB-based AI algorithm vulnerabilities have been widely reported, the vulnerabilities of NIR-based AI have rarely been investigated. In this paper, we identify fundamental vulnerabilities in NIR-based image understanding caused by color and texture loss due to the intrinsic characteristics of clothes' reflectance and cameras' spectral sensitivity in the NIR range. We further show that the nearly co-located configuration of illuminants and cameras in existing surveillance systems facilitates concealing and fully passive attacks in the physical world. Specifically, we demonstrate how retro-reflective and insulation plastic tapes can manipulate the intensity distribution of NIR images. We showcase an attack on the YOLO-based human detector using binary patterns designed in the digital space (via black-box query and searching) and then physically realized using tapes pasted onto clothes. Our attack highlights significant reliability concerns for nighttime surveillance systems, which are intended to enhance security. Codes Available: this https URL

Title: AnchorInv: Few-Shot Class-Incremental Learning of Physiological Signals via Representation Space Guided Inversion

• Authors: Chenqi Li, Boyan Gao, Gabriel Jones, Timothy Denison, Tingting Zhu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13714
• Pdf URL: https://arxiv.org/pdf/2412.13714
• Copy Paste: [[2412.13714]] AnchorInv: Few-Shot Class-Incremental Learning of Physiological Signals via Representation Space Guided Inversion(https://arxiv.org/abs/2412.13714)
• Keywords: privacy, protect
• Abstract: Deep learning models have demonstrated exceptional performance in a variety of real-world applications. These successes are often attributed to strong base models that can generalize to novel tasks with limited supporting data while keeping prior knowledge intact. However, these impressive results are based on the availability of a large amount of high-quality data, which is often lacking in specialized biomedical applications. In such fields, models are usually developed with limited data that arrive incrementally with novel categories. This requires the model to adapt to new information while preserving existing knowledge. Few-Shot Class-Incremental Learning (FSCIL) methods offer a promising approach to addressing these challenges, but they also depend on strong base models that face the same aforementioned limitations. To overcome these constraints, we propose AnchorInv following the straightforward and efficient buffer-replay strategy. Instead of selecting and storing raw data, AnchorInv generates synthetic samples guided by anchor points in the feature space. This approach protects privacy and regularizes the model for adaptation. When evaluated on three public physiological time series datasets, AnchorInv exhibits efficient knowledge forgetting prevention and improved adaptation to novel classes, surpassing state-of-the-art baselines.

Title: Towards Automatic Evaluation for Image Transcreation

• Authors: Simran Khanuja, Vivek Iyer, Claire He, Graham Neubig
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13717
• Pdf URL: https://arxiv.org/pdf/2412.13717
• Copy Paste: [[2412.13717]] Towards Automatic Evaluation for Image Transcreation(https://arxiv.org/abs/2412.13717)
• Keywords: robust
• Abstract: Beyond conventional paradigms of translating speech and text, recently, there has been interest in automated transcreation of images to facilitate localization of visual content across different cultures. Attempts to define this as a formal Machine Learning (ML) problem have been impeded by the lack of automatic evaluation mechanisms, with previous work relying solely on human evaluation. In this paper, we seek to close this gap by proposing a suite of automatic evaluation metrics inspired by machine translation (MT) metrics, categorized into: a) Object-based, b) Embedding-based, and c) VLM-based. Drawing on theories from translation studies and real-world transcreation practices, we identify three critical dimensions of image transcreation: cultural relevance, semantic equivalence and visual similarity, and design our metrics to evaluate systems along these axes. Our results show that proprietary VLMs best identify cultural relevance and semantic equivalence, while vision-encoder representations are adept at measuring visual similarity. Meta-evaluation across 7 countries shows our metrics agree strongly with human ratings, with average segment-level correlations ranging from 0.55-0.87. Finally, through a discussion of the merits and demerits of each metric, we offer a robust framework for automated image transcreation evaluation, grounded in both theoretical foundations and practical application. Our code can be found here: this https URL

Title: Federated Learning and RAG Integration: A Scalable Approach for Medical Large Language Models

• Authors: Jincheol Jung, Hongju Jeong, Eui-Nam Huh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13720
• Pdf URL: https://arxiv.org/pdf/2412.13720
• Copy Paste: [[2412.13720]] Federated Learning and RAG Integration: A Scalable Approach for Medical Large Language Models(https://arxiv.org/abs/2412.13720)
• Keywords: privacy, federate, large language model
• Abstract: This study analyzes the performance of domain-specific Large Language Models (LLMs) for the medical field by integrating Retrieval-Augmented Generation (RAG) systems within a federated learning framework. Leveraging the inherent advantages of federated learning, such as preserving data privacy and enabling distributed computation, this research explores the integration of RAG systems with models trained under varying client configurations to optimize performance. Experimental results demonstrate that the federated learning-based models integrated with RAG systems consistently outperform their non-integrated counterparts across all evaluation metrics. This study highlights the potential of combining federated learning and RAG systems for developing domain-specific LLMs in the medical field, providing a scalable and privacy-preserving solution for enhancing text generation capabilities.

Title: Text2Relight: Creative Portrait Relighting with Text Guidance

• Authors: Junuk Cha, Mengwei Ren, Krishna Kumar Singh, He Zhang, Yannick Hold-Geoffroy, Seunghyun Yoon, HyunJoon Jung, Jae Shin Yoon, Seungryul Baek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13734
• Pdf URL: https://arxiv.org/pdf/2412.13734
• Copy Paste: [[2412.13734]] Text2Relight: Creative Portrait Relighting with Text Guidance(https://arxiv.org/abs/2412.13734)
• Keywords: diffusion, generative, large language model
• Abstract: We present a lighting-aware image editing pipeline that, given a portrait image and a text prompt, performs single image relighting. Our model modifies the lighting and color of both the foreground and background to align with the provided text description. The unbounded nature in creativeness of a text allows us to describe the lighting of a scene with any sensory features including temperature, emotion, smell, time, and so on. However, the modeling of such mapping between the unbounded text and lighting is extremely challenging due to the lack of dataset where there exists no scalable data that provides large pairs of text and relighting, and therefore, current text-driven image editing models does not generalize to lighting-specific use cases. We overcome this problem by introducing a novel data synthesis pipeline: First, diverse and creative text prompts that describe the scenes with various lighting are automatically generated under a crafted hierarchy using a large language model (*e.g.,* ChatGPT). A text-guided image generation model creates a lighting image that best matches the text. As a condition of the lighting images, we perform image-based relighting for both foreground and background using a single portrait image or a set of OLAT (One-Light-at-A-Time) images captured from lightstage system. Particularly for the background relighting, we represent the lighting image as a set of point lights and transfer them to other background images. A generative diffusion model learns the synthesized large-scale data with auxiliary task augmentation (*e.g.,* portrait delighting and light positioning) to correlate the latent text and lighting distribution for text-guided portrait relighting.

Title: MedCoT: Medical Chain of Thought via Hierarchical Expert

• Authors: Jiaxiang Liu, Yuan Wang, Jiawei Du, Joey Tianyi Zhou, Zuozhu Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13736
• Pdf URL: https://arxiv.org/pdf/2412.13736
• Copy Paste: [[2412.13736]] MedCoT: Medical Chain of Thought via Hierarchical Expert(https://arxiv.org/abs/2412.13736)
• Keywords: robust, interpretability
• Abstract: Artificial intelligence has advanced in Medical Visual Question Answering (Med-VQA), but prevalent research tends to focus on the accuracy of the answers, often overlooking the reasoning paths and interpretability, which are crucial in clinical settings. Besides, current Med-VQA algorithms, typically reliant on singular models, lack the robustness needed for real-world medical diagnostics which usually require collaborative expert evaluation. To address these shortcomings, this paper presents MedCoT, a novel hierarchical expert verification reasoning chain method designed to enhance interpretability and accuracy in biomedical imaging inquiries. MedCoT is predicated on two principles: The necessity for explicit reasoning paths in Med-VQA and the requirement for multi-expert review to formulate accurate conclusions. The methodology involves an Initial Specialist proposing diagnostic rationales, followed by a Follow-up Specialist who validates these rationales, and finally, a consensus is reached through a vote among a sparse Mixture of Experts within the locally deployed Diagnostic Specialist, which then provides the definitive diagnosis. Experimental evaluations on four standard Med-VQA datasets demonstrate that MedCoT surpasses existing state-of-the-art approaches, providing significant improvements in performance and interpretability.

Title: Uncertainty separation via ensemble quantile regression

• Authors: Navid Ansari, Hans-Peter Seidel, Vahid Babaei
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2412.13738
• Pdf URL: https://arxiv.org/pdf/2412.13738
• Copy Paste: [[2412.13738]] Uncertainty separation via ensemble quantile regression(https://arxiv.org/abs/2412.13738)
• Keywords: robust
• Abstract: This paper introduces a novel and scalable framework for uncertainty estimation and separation with applications in data driven modeling in science and engineering tasks where reliable uncertainty quantification is critical. Leveraging an ensemble of quantile regression (E-QR) models, our approach enhances aleatoric uncertainty estimation while preserving the quality of epistemic uncertainty, surpassing competing methods, such as Deep Ensembles (DE) and Monte Carlo (MC) dropout. To address challenges in separating uncertainty types, we propose an algorithm that iteratively improves separation through progressive sampling in regions of high uncertainty. Our framework is scalable to large datasets and demonstrates superior performance on synthetic benchmarks, offering a robust tool for uncertainty quantification in data-driven applications.

Title: Learnable Prompting SAM-induced Knowledge Distillation for Semi-supervised Medical Image Segmentation

• Authors: Kaiwen Huang, Tao Zhou, Huazhu Fu, Yizhe Zhang, Yi Zhou, Chen Gong, Dong Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13742
• Pdf URL: https://arxiv.org/pdf/2412.13742
• Copy Paste: [[2412.13742]] Learnable Prompting SAM-induced Knowledge Distillation for Semi-supervised Medical Image Segmentation(https://arxiv.org/abs/2412.13742)
• Keywords: robust, segmentation
• Abstract: The limited availability of labeled data has driven advancements in semi-supervised learning for medical image segmentation. Modern large-scale models tailored for general segmentation, such as the Segment Anything Model (SAM), have revealed robust generalization capabilities. However, applying these models directly to medical image segmentation still exposes performance degradation. In this paper, we propose a learnable prompting SAM-induced Knowledge distillation framework (KnowSAM) for semi-supervised medical image segmentation. Firstly, we propose a Multi-view Co-training (MC) strategy that employs two distinct sub-networks to employ a co-teaching paradigm, resulting in more robust outcomes. Secondly, we present a Learnable Prompt Strategy (LPS) to dynamically produce dense prompts and integrate an adapter to fine-tune SAM specifically for medical image segmentation tasks. Moreover, we propose SAM-induced Knowledge Distillation (SKD) to transfer useful knowledge from SAM to two sub-networks, enabling them to learn from SAM's predictions and alleviate the effects of incorrect pseudo-labels during training. Notably, the predictions generated by our subnets are used to produce mask prompts for SAM, facilitating effective inter-module information exchange. Extensive experimental results on various medical segmentation tasks demonstrate that our model outperforms the state-of-the-art semi-supervised segmentation approaches. Crucially, our SAM distillation framework can be seamlessly integrated into other semi-supervised segmentation methods to enhance performance. The code will be released upon acceptance of this manuscript at: this https URL

Title: RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment

• Authors: Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.13746
• Pdf URL: https://arxiv.org/pdf/2412.13746
• Copy Paste: [[2412.13746]] RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment(https://arxiv.org/abs/2412.13746)
• Keywords: robust
• Abstract: Despite the significant progress made by existing retrieval augmented language models (RALMs) in providing trustworthy responses and grounding in reliable sources, they often overlook effective alignment with human preferences. In the alignment process, reward models (RMs) act as a crucial proxy for human values to guide optimization. However, it remains unclear how to evaluate and select a reliable RM for preference alignment in RALMs. To this end, we propose RAG-RewardBench, the first benchmark for evaluating RMs in RAG settings. First, we design four crucial and challenging RAG-specific scenarios to assess RMs, including multi-hop reasoning, fine-grained citation, appropriate abstain, and conflict robustness. Then, we incorporate 18 RAG subsets, six retrievers, and 24 RALMs to increase the diversity of data sources. Finally, we adopt an LLM-as-a-judge approach to improve preference annotation efficiency and effectiveness, exhibiting a strong correlation with human annotations. Based on the RAG-RewardBench, we conduct a comprehensive evaluation of 45 RMs and uncover their limitations in RAG scenarios. Additionally, we also reveal that existing trained RALMs show almost no improvement in preference alignment, highlighting the need for a shift towards preference-aligned this http URL release our benchmark and code publicly at this https URL for future work.

Title: Multi-Exposure Image Fusion via Distilled 3D LUT Grid with Editable Mode

• Authors: Xin Su, Zhuoran Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13749
• Pdf URL: https://arxiv.org/pdf/2412.13749
• Copy Paste: [[2412.13749]] Multi-Exposure Image Fusion via Distilled 3D LUT Grid with Editable Mode(https://arxiv.org/abs/2412.13749)
• Keywords: robust
• Abstract: With the rising imaging resolution of handheld devices, existing multi-exposure image fusion algorithms struggle to generate a high dynamic range image with ultra-high resolution in real-time. Apart from that, there is a trend to design a manageable and editable algorithm as the different needs of real application scenarios. To tackle these issues, we introduce 3D LUT technology, which can enhance images with ultra-high-definition (UHD) resolution in real time on resource-constrained devices. However, since the fusion of information from multiple images with different exposure rates is uncertain, and this uncertainty significantly trials the generalization power of the 3D LUT grid. To address this issue and ensure a robust learning space for the model, we propose using a teacher-student network to model the uncertainty on the 3D LUT this http URL, we provide an editable mode for the multi-exposure image fusion algorithm by using the implicit representation function to match the requirements in different scenarios. Extensive experiments demonstrate that our proposed method is highly competitive in efficiency and accuracy.

Title: Mesoscopic Insights: Orchestrating Multi-scale & Hybrid Architecture for Image Manipulation Localization

• Authors: Xuekang Zhu, Xiaochen Ma, Lei Su, Zhuohang Jiang, Bo Du, Xiwen Wang, Zeyu Lei, Wentao Feng, Chi-Man Pun, Jizhe Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13753
• Pdf URL: https://arxiv.org/pdf/2412.13753
• Copy Paste: [[2412.13753]] Mesoscopic Insights: Orchestrating Multi-scale & Hybrid Architecture for Image Manipulation Localization(https://arxiv.org/abs/2412.13753)
• Keywords: robust, transformer
• Abstract: The mesoscopic level serves as a bridge between the macroscopic and microscopic worlds, addressing gaps overlooked by both. Image manipulation localization (IML), a crucial technique to pursue truth from fake images, has long relied on low-level (microscopic-level) traces. However, in practice, most tampering aims to deceive the audience by altering image semantics. As a result, manipulation commonly occurs at the object level (macroscopic level), which is equally important as microscopic traces. Therefore, integrating these two levels into the mesoscopic level presents a new perspective for IML research. Inspired by this, our paper explores how to simultaneously construct mesoscopic representations of micro and macro information for IML and introduces the Mesorch architecture to orchestrate both. Specifically, this architecture i) combines Transformers and CNNs in parallel, with Transformers extracting macro information and CNNs capturing micro details, and ii) explores across different scales, assessing micro and macro information seamlessly. Additionally, based on the Mesorch architecture, the paper introduces two baseline models aimed at solving IML tasks through mesoscopic representation. Extensive experiments across four datasets have demonstrated that our models surpass the current state-of-the-art in terms of performance, computational complexity, and robustness.

Title: Federated Source-free Domain Adaptation for Classification: Weighted Cluster Aggregation for Unlabeled Data

• Authors: Junki Mori, Kosuke Kihara, Taiki Miyagawa, Akinori F. Ebihara, Isamu Teranishi, Hisashi Kashima
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13757
• Pdf URL: https://arxiv.org/pdf/2412.13757
• Copy Paste: [[2412.13757]] Federated Source-free Domain Adaptation for Classification: Weighted Cluster Aggregation for Unlabeled Data(https://arxiv.org/abs/2412.13757)
• Keywords: privacy, federate, segmentation
• Abstract: Federated learning (FL) commonly assumes that the server or some clients have labeled data, which is often impractical due to annotation costs and privacy concerns. Addressing this problem, we focus on a source-free domain adaptation task, where (1) the server holds a pre-trained model on labeled source domain data, (2) clients possess only unlabeled data from various target domains, and (3) the server and clients cannot access the source data in the adaptation phase. This task is known as Federated source-Free Domain Adaptation (FFREEDA). Specifically, we focus on classification tasks, while the previous work solely studies semantic segmentation. Our contribution is the novel Federated learning with Weighted Cluster Aggregation (FedWCA) method, designed to mitigate both domain shifts and privacy concerns with only unlabeled data. FedWCA comprises three phases: private and parameter-free clustering of clients to obtain domain-specific global models on the server, weighted aggregation of the global models for the clustered clients, and local domain adaptation with pseudo-labeling. Experimental results show that FedWCA surpasses several existing methods and baselines in FFREEDA, establishing its effectiveness and practicality.

Title: Cultivating Archipelago of Forests: Evolving Robust Decision Trees through Island Coevolution

• Authors: Adam Żychowski, Andrew Perrault, Jacek Mańdziuk
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2412.13762
• Pdf URL: https://arxiv.org/pdf/2412.13762
• Copy Paste: [[2412.13762]] Cultivating Archipelago of Forests: Evolving Robust Decision Trees through Island Coevolution(https://arxiv.org/abs/2412.13762)
• Keywords: attack, robust, interpretability
• Abstract: Decision trees are widely used in machine learning due to their simplicity and interpretability, but they often lack robustness to adversarial attacks and data perturbations. The paper proposes a novel island-based coevolutionary algorithm (ICoEvoRDF) for constructing robust decision tree ensembles. The algorithm operates on multiple islands, each containing populations of decision trees and adversarial perturbations. The populations on each island evolve independently, with periodic migration of top-performing decision trees between islands. This approach fosters diversity and enhances the exploration of the solution space, leading to more robust and accurate decision tree ensembles. ICoEvoRDF utilizes a popular game theory concept of mixed Nash equilibrium for ensemble weighting, which further leads to improvement in results. ICoEvoRDF is evaluated on 20 benchmark datasets, demonstrating its superior performance compared to state-of-the-art methods in optimizing both adversarial accuracy and minimax regret. The flexibility of ICoEvoRDF allows for the integration of decision trees from various existing methods, providing a unified framework for combining diverse solutions. Our approach offers a promising direction for developing robust and interpretable machine learning models

Title: LLM-SEM: A Sentiment-Based Student Engagement Metric Using LLMS for E-Learning Platforms

• Authors: Ali Hamdi, Ahmed Abdelmoneim Mazrou, Mohamed Shaltout
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13765
• Pdf URL: https://arxiv.org/pdf/2412.13765
• Copy Paste: [[2412.13765]] LLM-SEM: A Sentiment-Based Student Engagement Metric Using LLMS for E-Learning Platforms(https://arxiv.org/abs/2412.13765)
• Keywords: large language model
• Abstract: Current methods for analyzing student engagement in e-learning platforms, including automated systems, often struggle with challenges such as handling fuzzy sentiment in text comments and relying on limited metadata. Traditional approaches, such as surveys and questionnaires, also face issues like small sample sizes and scalability. In this paper, we introduce LLM-SEM (Language Model-Based Student Engagement Metric), a novel approach that leverages video metadata and sentiment analysis of student comments to measure engagement. By utilizing recent Large Language Models (LLMs), we generate high-quality sentiment predictions to mitigate text fuzziness and normalize key features such as views and likes. Our holistic method combines comprehensive metadata with sentiment polarity scores to gauge engagement at both the course and lesson levels. Extensive experiments were conducted to evaluate various LLM models, demonstrating the effectiveness of LLM-SEM in providing a scalable and accurate measure of student engagement. We fine-tuned LLMs, including AraBERT, TXLM-RoBERTa, LLama 3B and Gemma 9B from Ollama, using human-annotated sentiment datasets to enhance prediction accuracy.

Title: Data sharing in the metaverse with key abuse resistance based on decentralized CP-ABE

• Authors: Liang Zhang, Zhanrong Ou, Changhui Hu, Haibin Kan, Jiheng Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.13770
• Pdf URL: https://arxiv.org/pdf/2412.13770
• Copy Paste: [[2412.13770]] Data sharing in the metaverse with key abuse resistance based on decentralized CP-ABE(https://arxiv.org/abs/2412.13770)
• Keywords: secure
• Abstract: Data sharing is ubiquitous in the metaverse, which adopts blockchain as its foundation. Blockchain is employed because it enables data transparency, achieves tamper resis- tance, and supports smart contracts. However, securely shar- ing data based on blockchain necessitates further considera- tion. Ciphertext-policy attribute-based encryption (CP-ABE) is a promising primitive to provide confidentiality and fine-grained access control. Nonetheless, authority accountability and key abuse are critical issues that practical applications must address. Few studies have considered CP-ABE key confidentiality and authority accountability simultaneously. To our knowledge, we are the first to fill this gap by integrating non-interactive zero- knowledge (NIZK) proofs into CP-ABE keys and outsourcing the verification process to a smart contract. To meet the decen- tralization requirement, we incorporate a decentralized CP-ABE scheme into the proposed data sharing system. Additionally, we provide an implementation based on smart contract to determine whether an access control policy is satisfied by a set of CP- ABE keys. We also introduce an open incentive mechanism to encourage honest participation in data sharing. Hence, the key abuse issue is resolved through the NIZK proof and the incentive mechanism. We provide a theoretical analysis and conduct comprehensive experiments to demonstrate the feasibility and efficiency of the data sharing system. Based on the proposed accountable approach, we further illustrate an application in GameFi, where players can play to earn or contribute to an accountable DAO, fostering a thriving metaverse ecosystem.

Title: Rehearsal-Free Continual Federated Learning with Synergistic Regularization

• Authors: Yichen Li, Yuying Wang, Tianzhe Xiao, Haozhao Wang, Yining Qi, Ruixuan Li
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.13779
• Pdf URL: https://arxiv.org/pdf/2412.13779
• Copy Paste: [[2412.13779]] Rehearsal-Free Continual Federated Learning with Synergistic Regularization(https://arxiv.org/abs/2412.13779)
• Keywords: privacy, federate
• Abstract: Continual Federated Learning (CFL) allows distributed devices to collaboratively learn novel concepts from continuously shifting training data while avoiding knowledge forgetting of previously seen tasks. To tackle this challenge, most current CFL approaches rely on extensive rehearsal of previous data. Despite effectiveness, rehearsal comes at a cost to memory, and it may also violate data privacy. Considering these, we seek to apply regularization techniques to CFL by considering their cost-efficient properties that do not require sample caching or rehearsal. Specifically, we first apply traditional regularization techniques to CFL and observe that existing regularization techniques, especially synaptic intelligence, can achieve promising results under homogeneous data distribution but fail when the data is heterogeneous. Based on this observation, we propose a simple yet effective regularization algorithm for CFL named FedSSI, which tailors the synaptic intelligence for the CFL with heterogeneous data settings. FedSSI can not only reduce computational overhead without rehearsal but also address the data heterogeneity issue. Extensive experiments show that FedSSI achieves superior performance compared to state-of-the-art methods.

Title: Meta-Reflection: A Feedback-Free Reflection Learning Framework

• Authors: Yaoke Wang, Yun Zhu, Xintong Bao, Wenqiao Zhang, Suyang Dai, Kehan Chen, Wenqiang Li, Gang Huang, Siliang Tang, Yueting Zhuang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13781
• Pdf URL: https://arxiv.org/pdf/2412.13781
• Copy Paste: [[2412.13781]] Meta-Reflection: A Feedback-Free Reflection Learning Framework(https://arxiv.org/abs/2412.13781)
• Keywords: large language model
• Abstract: Despite the remarkable capabilities of large language models (LLMs) in natural language understanding and reasoning, they often display undesirable behaviors, such as generating hallucinations and unfaithful reasoning. A prevalent strategy to mitigate these issues is the use of reflection, which refines responses through an iterative process. However, while promising, reflection heavily relies on high-quality external feedback and requires iterative multi-agent inference processes, thus hindering its practical application. In this paper, we propose Meta-Reflection, a novel feedback-free reflection mechanism that necessitates only a single inference pass without external feedback. Motivated by the human ability to remember and retrieve reflections from past experiences when encountering similar problems, Meta-Reflection integrates reflective insights into a codebook, allowing the historical insights to be stored, retrieved, and used to guide LLMs in problem-solving. To thoroughly investigate and evaluate the practicality of Meta-Reflection in real-world scenarios, we introduce an industrial e-commerce benchmark named E-commerce Customer Intent Detection (ECID). Extensive experiments conducted on both public datasets and the ECID benchmark highlight the effectiveness and efficiency of our proposed approach.

Title: Knowledge Editing with Dynamic Knowledge Graphs for Multi-hop Question Answering

• Authors: Yifan Lu, Yigeng Zhou, Jing Li, Yequan Wang, Xuebo Liu, Daojing He, Fangming Liu, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13782
• Pdf URL: https://arxiv.org/pdf/2412.13782
• Copy Paste: [[2412.13782]] Knowledge Editing with Dynamic Knowledge Graphs for Multi-hop Question Answering(https://arxiv.org/abs/2412.13782)
• Keywords: large language model
• Abstract: Multi-hop question answering (MHQA) poses a significant challenge for large language models (LLMs) due to the extensive knowledge demands involved. Knowledge editing, which aims to precisely modify the LLMs to incorporate specific knowledge without negatively impacting other unrelated knowledge, offers a potential solution for addressing MHQA challenges with LLMs. However, current solutions struggle to effectively resolve issues of knowledge conflicts. Most parameter-preserving editing methods are hindered by inaccurate retrieval and overlook secondary editing issues, which can introduce noise into the reasoning process of LLMs. In this paper, we introduce KEDKG, a novel knowledge editing method that leverages a dynamic knowledge graph for MHQA, designed to ensure the reliability of answers. KEDKG involves two primary steps: dynamic knowledge graph construction and knowledge graph augmented generation. Initially, KEDKG autonomously constructs a dynamic knowledge graph to store revised information while resolving potential knowledge conflicts. Subsequently, it employs a fine-grained retrieval strategy coupled with an entity and relation detector to enhance the accuracy of graph retrieval for LLM generation. Experimental results on benchmarks show that KEDKG surpasses previous state-of-the-art models, delivering more accurate and reliable answers in environments with dynamic information.

Title: Open Universal Arabic ASR Leaderboard

• Authors: Yingzhi Wang, Anas Alhmoud, Muhammad Alqurishi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13788
• Pdf URL: https://arxiv.org/pdf/2412.13788
• Copy Paste: [[2412.13788]] Open Universal Arabic ASR Leaderboard(https://arxiv.org/abs/2412.13788)
• Keywords: robust
• Abstract: In recent years, the enhanced capabilities of ASR models and the emergence of multi-dialect datasets have increasingly pushed Arabic ASR model development toward an all-dialect-in-one direction. This trend highlights the need for benchmarking studies that evaluate model performance on multiple dialects, providing the community with insights into models' generalization capabilities. In this paper, we introduce Open Universal Arabic ASR Leaderboard, a continuous benchmark project for open-source general Arabic ASR models across various multi-dialect datasets. We also provide a comprehensive analysis of the model's robustness, speaker adaptation, inference efficiency, and memory consumption. This work aims to offer the Arabic ASR community a reference for models' general performance and also establish a common evaluation framework for multi-dialectal Arabic ASR models.

Title: Toward Efficient Data-Free Unlearning

• Authors: Chenhao Zhang, Shaofei Shen, Weitong Chen, Miao Xu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13790
• Pdf URL: https://arxiv.org/pdf/2412.13790
• Copy Paste: [[2412.13790]] Toward Efficient Data-Free Unlearning(https://arxiv.org/abs/2412.13790)
• Keywords: data-free
• Abstract: Machine unlearning without access to real data distribution is challenging. The existing method based on data-free distillation achieved unlearning by filtering out synthetic samples containing forgetting information but struggled to distill the retaining-related knowledge efficiently. In this work, we analyze that such a problem is due to over-filtering, which reduces the synthesized retaining-related information. We propose a novel method, Inhibited Synthetic PostFilter (ISPF), to tackle this challenge from two perspectives: First, the Inhibited Synthetic, by reducing the synthesized forgetting information; Second, the PostFilter, by fully utilizing the retaining-related information in synthesized samples. Experimental results demonstrate that the proposed ISPF effectively tackles the challenge and outperforms existing methods.

Title: Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models

• Authors: Xinyu Pang, Ruixin Hong, Zhanke Zhou, Fangrui Lv, Xinwei Yang, Zhilong Liang, Bo Han, Changshui Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13791
• Pdf URL: https://arxiv.org/pdf/2412.13791
• Copy Paste: [[2412.13791]] Physics Reasoner: Knowledge-Augmented Reasoning for Solving Physics Problems with Large Language Models(https://arxiv.org/abs/2412.13791)
• Keywords: large language model
• Abstract: Physics problems constitute a significant aspect of reasoning, necessitating complicated reasoning ability and abundant physics knowledge. However, existing large language models (LLMs) frequently fail due to a lack of knowledge or incorrect knowledge application. To mitigate these issues, we propose Physics Reasoner, a knowledge-augmented framework to solve physics problems with LLMs. Specifically, the proposed framework constructs a comprehensive formula set to provide explicit physics knowledge and utilizes checklists containing detailed instructions to guide effective knowledge application. Namely, given a physics problem, Physics Reasoner solves it through three stages: problem analysis, formula retrieval, and guided reasoning. During the process, checklists are employed to enhance LLMs' self-improvement in the analysis and reasoning stages. Empirically, Physics Reasoner mitigates the issues of insufficient knowledge and incorrect application, achieving state-of-the-art performance on SciBench with an average accuracy improvement of 5.8%.

Title: MATCHED: Multimodal Authorship-Attribution To Combat Human Trafficking in Escort-Advertisement Data

• Authors: Vageesh Saxena, Benjamin Bashpole, Gijs Van Dijck, Gerasimos Spanakis
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.13794
• Pdf URL: https://arxiv.org/pdf/2412.13794
• Copy Paste: [[2412.13794]] MATCHED: Multimodal Authorship-Attribution To Combat Human Trafficking in Escort-Advertisement Data(https://arxiv.org/abs/2412.13794)
• Keywords: robust
• Abstract: Human trafficking (HT) remains a critical issue, with traffickers increasingly leveraging online escort advertisements (ads) to advertise victims anonymously. Existing detection methods, including Authorship Attribution (AA), often center on text-based analyses and neglect the multimodal nature of online escort ads, which typically pair text with images. To address this gap, we introduce MATCHED, a multimodal dataset of 27,619 unique text descriptions and 55,115 unique images collected from the Backpage escort platform across seven U.S. cities in four geographical regions. Our study extensively benchmarks text-only, vision-only, and multimodal baselines for vendor identification and verification tasks, employing multitask (joint) training objectives that achieve superior classification and retrieval performance on in-distribution and out-of-distribution (OOD) datasets. Integrating multimodal features further enhances this performance, capturing complementary patterns across text and images. While text remains the dominant modality, visual data adds stylistic cues that enrich model performance. Moreover, text-image alignment strategies like CLIP and BLIP2 struggle due to low semantic overlap and vague connections between the modalities of escort ads, with end-to-end multimodal training proving more robust. Our findings emphasize the potential of multimodal AA (MAA) to combat HT, providing LEAs with robust tools to link ads and disrupt trafficking networks.

Title: Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN

• Authors: Pengxiang Li, Lu Yin, Shiwei Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13795
• Pdf URL: https://arxiv.org/pdf/2412.13795
• Copy Paste: [[2412.13795]] Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN(https://arxiv.org/abs/2412.13795)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have achieved remarkable success, yet recent findings reveal that their deeper layers often contribute minimally and can be pruned without affecting overall performance. While some view this as an opportunity for model compression, we identify it as a training shortfall rooted in the widespread use of Pre-Layer Normalization (Pre-LN). We demonstrate that Pre-LN, commonly employed in models like GPT and LLaMA, leads to diminished gradient norms in its deeper layers, reducing their effectiveness. In contrast, Post-Layer Normalization (Post-LN) preserves larger gradient norms in deeper layers but suffers from vanishing gradients in earlier layers. To address this, we introduce Mix-LN, a novel normalization technique that combines the strengths of Pre-LN and Post-LN within the same model. Mix-LN applies Post-LN to the earlier layers and Pre-LN to the deeper layers, ensuring more uniform gradients across layers. This allows all parts of the network--both shallow and deep layers--to contribute effectively to training. Extensive experiments with various model sizes from 70M to 7B demonstrate that Mix-LN consistently outperforms both Pre-LN and Post-LN, promoting more balanced, healthier gradient norms throughout the network, and enhancing the overall quality of LLM pre-training. Furthermore, we demonstrate that models pre-trained with Mix-LN learn better compared to those using Pre-LN or Post-LN during supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), highlighting the critical importance of high-quality deep layers. By effectively addressing the inefficiencies of deep layers in current LLMs, Mix-LN unlocks their potential, enhancing model capacity without increasing model size. Our code is available at this https URL.

Title: M$^3$-VOS: Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation

• Authors: Zixuan Chen, Jiaxin Li, Liming Tan, Yejie Guo, Junxuan Liang, Cewu Lu, Yonglu Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13803
• Pdf URL: https://arxiv.org/pdf/2412.13803
• Copy Paste: [[2412.13803]] M$^3$-VOS: Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation(https://arxiv.org/abs/2412.13803)
• Keywords: segmentation
• Abstract: Intelligent robots need to interact with diverse objects across various environments. The appearance and state of objects frequently undergo complex transformations depending on the object properties, e.g., phase transitions. However, in the vision community, segmenting dynamic objects with phase transitions is overlooked. In light of this, we introduce the concept of phase in segmentation, which categorizes real-world objects based on their visual characteristics and potential morphological and appearance changes. Then, we present a new benchmark, Multi-Phase, Multi-Transition, and Multi-Scenery Video Object Segmentation (M3-VOS), to verify the ability of models to understand object phases, which consists of 479 high-resolution videos spanning over 10 distinct everyday scenarios. It provides dense instance mask annotations that capture both object phases and their transitions. We evaluate state-of-the-art methods on M3-VOS, yielding several key insights. Notably, current appearance based approaches show significant room for improvement when handling objects with phase transitions. The inherent changes in disorder suggest that the predictive performance of the forward entropy-increasing process can be improved through a reverse entropy-reducing process. These findings lead us to propose ReVOS, a new plug-and-play model that improves its performance by reversal refinement. Our data and code will be publicly available

Title: Extreme Multi-label Completion for Semantic Document Labelling with Taxonomy-Aware Parallel Learning

• Authors: Julien Audiffren, Christophe Broillet, Ljiljana Dolamic, Philippe Cudré-Mauroux
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13809
• Pdf URL: https://arxiv.org/pdf/2412.13809
• Copy Paste: [[2412.13809]] Extreme Multi-label Completion for Semantic Document Labelling with Taxonomy-Aware Parallel Learning(https://arxiv.org/abs/2412.13809)
• Keywords: transformer
• Abstract: In Extreme Multi Label Completion (XMLCo), the objective is to predict the missing labels of a collection of documents. Together with XML Classification, XMLCo is arguably one of the most challenging document classification tasks, as the very high number of labels (at least ten of thousands) is generally very large compared to the number of available labelled documents in the training dataset. Such a task is often accompanied by a taxonomy that encodes the labels organic relationships, and many methods have been proposed to leverage this hierarchy to improve the results of XMLCo algorithms. In this paper, we propose a new approach to this problem, TAMLEC (Taxonomy-Aware Multi-task Learning for Extreme multi-label Completion). TAMLEC divides the problem into several Taxonomy-Aware Tasks, i.e. subsets of labels adapted to the hierarchical paths of the taxonomy, and trains on these tasks using a dynamic Parallel Feature sharing approach, where some parts of the model are shared between tasks while others are task-specific. Then, at inference time, TAMLEC uses the labels available in a document to infer the appropriate tasks and to predict missing labels. To achieve this result, TAMLEC uses a modified transformer architecture that predicts ordered sequences of labels on a Weak-Semilattice structure that is naturally induced by the tasks. This approach yields multiple advantages. First, our experiments on real-world datasets show that TAMLEC outperforms state-of-the-art methods for various XMLCo problems. Second, TAMLEC is by construction particularly suited for few-shots XML tasks, where new tasks or labels are introduced with only few examples, and extensive evaluations highlight its strong performance compared to existing methods.

Title: CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers?

• Authors: Dimitrios Mallis, Ahmet Serdar Karadeniz, Sebastian Cavada, Danila Rukhovich, Niki Foteinopoulou, Kseniya Cherenkova, Anis Kacem, Djamila Aouada
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.13810
• Pdf URL: https://arxiv.org/pdf/2412.13810
• Copy Paste: [[2412.13810]] CAD-Assistant: Tool-Augmented VLLMs as Generic CAD Task Solvers?(https://arxiv.org/abs/2412.13810)
• Keywords: large language model
• Abstract: We propose CAD-Assistant, a general-purpose CAD agent for AI-assisted design. Our approach is based on a powerful Vision and Large Language Model (VLLM) as a planner and a tool-augmentation paradigm using CAD-specific modules. CAD-Assistant addresses multimodal user queries by generating actions that are iteratively executed on a Python interpreter equipped with the FreeCAD software, accessed via its Python API. Our framework is able to assess the impact of generated CAD commands on geometry and adapts subsequent actions based on the evolving state of the CAD design. We consider a wide range of CAD-specific tools including Python libraries, modules of the FreeCAD Python API, helpful routines, rendering functions and other specialized modules. We evaluate our method on multiple CAD benchmarks and qualitatively demonstrate the potential of tool-augmented VLLMs as generic CAD task solvers across diverse CAD workflows.

Title: Object Style Diffusion for Generalized Object Detection in Urban Scene

• Authors: Hao Li, Xiangyuan Yang, Mengzhu Wang, Long Lan, Ke Liang, Xinwang Liu, Kenli Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13815
• Pdf URL: https://arxiv.org/pdf/2412.13815
• Copy Paste: [[2412.13815]] Object Style Diffusion for Generalized Object Detection in Urban Scene(https://arxiv.org/abs/2412.13815)
• Keywords: robust, diffusion
• Abstract: Object detection is a critical task in computer vision, with applications in various domains such as autonomous driving and urban scene monitoring. However, deep learning-based approaches often demand large volumes of annotated data, which are costly and difficult to acquire, particularly in complex and unpredictable real-world environments. This dependency significantly hampers the generalization capability of existing object detection techniques. To address this issue, we introduce a novel single-domain object detection generalization method, named GoDiff, which leverages a pre-trained model to enhance generalization in unseen domains. Central to our approach is the Pseudo Target Data Generation (PTDG) module, which employs a latent diffusion model to generate pseudo-target domain data that preserves source domain characteristics while introducing stylistic variations. By integrating this pseudo data with source domain data, we diversify the training dataset. Furthermore, we introduce a cross-style instance normalization technique to blend style features from different domains generated by the PTDG module, thereby increasing the detector's robustness. Experimental results demonstrate that our method not only enhances the generalization ability of existing detectors but also functions as a plug-and-play enhancement for other single-domain generalization methods, achieving state-of-the-art performance in autonomous driving scenarios.

Title: Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection

• Authors: Le Yang, Ziwei Zheng, Boxu Chen, Zhengyu Zhao, Chenhao Lin, Chao Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13817
• Pdf URL: https://arxiv.org/pdf/2412.13817
• Copy Paste: [[2412.13817]] Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection(https://arxiv.org/abs/2412.13817)
• Keywords: large language model
• Abstract: Recent studies have shown that large vision-language models (LVLMs) often suffer from the issue of object hallucinations (OH). To mitigate this issue, we introduce an efficient method that edits the model weights based on an unsafe subspace, which we call HalluSpace in this paper. With truthful and hallucinated text prompts accompanying the visual content as inputs, the HalluSpace can be identified by extracting the hallucinated embedding features and removing the truthful representations in LVLMs. By orthogonalizing the model weights, input features will be projected into the Null space of the HalluSpace to reduce OH, based on which we name our method Nullu. We reveal that HalluSpaces generally contain statistical bias and unimodal priors of the large language models (LLMs) applied to build LVLMs, which have been shown as essential causes of OH in previous studies. Therefore, null space projection suppresses the LLMs' priors to filter out the hallucinated features, resulting in contextually accurate outputs. Experiments show that our method can effectively mitigate OH across different LVLM families without extra inference costs and also show strong performance in general LVLM benchmarks. Code is released at \url{this https URL}.

Title: Fed-AugMix: Balancing Privacy and Utility via Data Augmentation

• Authors: Haoyang Li, Wei Chen, Xiaojin Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.13818
• Pdf URL: https://arxiv.org/pdf/2412.13818
• Copy Paste: [[2412.13818]] Fed-AugMix: Balancing Privacy and Utility via Data Augmentation(https://arxiv.org/abs/2412.13818)
• Keywords: privacy, protect, attack, robust, federate
• Abstract: Gradient leakage attacks pose a significant threat to the privacy guarantees of federated learning. While distortion-based protection mechanisms are commonly employed to mitigate this issue, they often lead to notable performance degradation. Existing methods struggle to preserve model performance while ensuring privacy. To address this challenge, we propose a novel data augmentation-based framework designed to achieve a favorable privacy-utility trade-off, with the potential to enhance model performance in certain cases. Our framework incorporates the AugMix algorithm at the client level, enabling data augmentation with controllable severity. By integrating the Jensen-Shannon divergence into the loss function, we embed the distortion introduced by AugMix into the model gradients, effectively safeguarding privacy against deep leakage attacks. Moreover, the JS divergence promotes model consistency across different augmentations of the same image, enhancing both robustness and performance. Extensive experiments on benchmark datasets demonstrate the effectiveness and stability of our method in protecting privacy. Furthermore, our approach maintains, and in some cases improves, model performance, showcasing its ability to achieve a robust privacy-utility trade-off.

Title: Prompt Categories Cluster for Weakly Supervised Semantic Segmentation

• Authors: Wangyu Wu, Xianglin Qiu, Siqi Song, Xiaowei Huang, Fei Ma, Jimin Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13823
• Pdf URL: https://arxiv.org/pdf/2412.13823
• Copy Paste: [[2412.13823]] Prompt Categories Cluster for Weakly Supervised Semantic Segmentation(https://arxiv.org/abs/2412.13823)
• Keywords: large language model, segmentation
• Abstract: Weakly Supervised Semantic Segmentation (WSSS), which leverages image-level labels, has garnered significant attention due to its cost-effectiveness. The previous methods mainly strengthen the inter-class differences to avoid class semantic ambiguity which may lead to erroneous activation. However, they overlook the positive function of some shared information between similar classes. Categories within the same cluster share some similar features. Allowing the model to recognize these features can further relieve the semantic ambiguity between these classes. To effectively identify and utilize this shared information, in this paper, we introduce a novel WSSS framework called Prompt Categories Clustering (PCC). Specifically, we explore the ability of Large Language Models (LLMs) to derive category clusters through prompts. These clusters effectively represent the intrinsic relationships between categories. By integrating this relational information into the training network, our model is able to better learn the hidden connections between categories. Experimental results demonstrate the effectiveness of our approach, showing its ability to enhance performance on the PASCAL VOC 2012 dataset and surpass existing state-of-the-art methods in WSSS.

Title: RACQUET: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs

• Authors: Alberto Testoni, Barbara Plank, Raquel Fernández
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13835
• Pdf URL: https://arxiv.org/pdf/2412.13835
• Copy Paste: [[2412.13835]] RACQUET: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs(https://arxiv.org/abs/2412.13835)
• Keywords: robust
• Abstract: Ambiguity resolution is key to effective communication. While humans effortlessly address ambiguity through conversational grounding strategies, the extent to which current language models can emulate these strategies remains unclear. In this work, we examine referential ambiguity in image-based question answering by introducing RACQUET, a carefully curated dataset targeting distinct aspects of ambiguity. Through a series of evaluations, we reveal significant limitations and problems of overconfidence of state-of-the-art large multimodal language models in addressing ambiguity in their responses. The overconfidence issue becomes particularly relevant for RACQUET-BIAS, a subset designed to analyze a critical yet underexplored problem: failing to address ambiguity leads to stereotypical, socially biased responses. Our results underscore the urgency of equipping models with robust strategies to deal with uncertainty without resorting to undesirable stereotypes.

Title: Unleashing the Power of Continual Learning on Non-Centralized Devices: A Survey

• Authors: Yichen Li, Haozhao Wang, Wenchao Xu, Tianzhe Xiao, Hong Liu, Minzhu Tu, Yuying Wang, Xin Yang, Rui Zhang, Shui Yu, Song Guo, Ruixuan Li
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.13840
• Pdf URL: https://arxiv.org/pdf/2412.13840
• Copy Paste: [[2412.13840]] Unleashing the Power of Continual Learning on Non-Centralized Devices: A Survey(https://arxiv.org/abs/2412.13840)
• Keywords: security, privacy
• Abstract: Non-Centralized Continual Learning (NCCL) has become an emerging paradigm for enabling distributed devices such as vehicles and servers to handle streaming data from a joint non-stationary environment. To achieve high reliability and scalability in deploying this paradigm in distributed systems, it is essential to conquer challenges stemming from both spatial and temporal dimensions, manifesting as distribution shifts, catastrophic forgetting, heterogeneity, and privacy issues. This survey focuses on a comprehensive examination of the development of the non-centralized continual learning algorithms and the real-world deployment across distributed devices. We begin with an introduction to the background and fundamentals of non-centralized learning and continual learning. Then, we review existing solutions from three levels to represent how existing techniques alleviate the catastrophic forgetting and distribution shift. Additionally, we delve into the various types of heterogeneity issues, security, and privacy attributes, as well as real-world applications across three prevalent scenarios. Furthermore, we establish a large-scale benchmark to revisit this problem and analyze the performance of the state-of-the-art NCCL approaches. Finally, we discuss the important challenges and future research directions in NCCL.

Title: Graph Coarsening via Supervised Granular-Ball for Scalable Graph Neural Network Training

• Authors: Shuyin Xia, Xinjun Ma, Zhiyuan Liu, Cheng Liu, Sen Zhao, Guoyin Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13842
• Pdf URL: https://arxiv.org/pdf/2412.13842
• Copy Paste: [[2412.13842]] Graph Coarsening via Supervised Granular-Ball for Scalable Graph Neural Network Training(https://arxiv.org/abs/2412.13842)
• Keywords: robust
• Abstract: Graph Neural Networks (GNNs) have demonstrated significant achievements in processing graph data, yet scalability remains a substantial challenge. To address this, numerous graph coarsening methods have been developed. However, most existing coarsening methods are training-dependent, leading to lower efficiency, and they all require a predefined coarsening rate, lacking an adaptive approach. In this paper, we employ granular-ball computing to effectively compress graph data. We construct a coarsened graph network by iteratively splitting the graph into granular-balls based on a purity threshold and using these granular-balls as super vertices. This granulation process significantly reduces the size of the original graph, thereby greatly enhancing the training efficiency and scalability of GNNs. Additionally, our algorithm can adaptively perform splitting without requiring a predefined coarsening rate. Experimental results demonstrate that our method achieves accuracy comparable to training on the original graph. Noise injection experiments further indicate that our method exhibits robust performance. Moreover, our approach can reduce the graph size by up to 20 times without compromising test accuracy, substantially enhancing the scalability of GNNs.

Title: Do Language Models Understand Time?

• Authors: Xi Ding, Lei Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13845
• Pdf URL: https://arxiv.org/pdf/2412.13845
• Copy Paste: [[2412.13845]] Do Language Models Understand Time?(https://arxiv.org/abs/2412.13845)
• Keywords: large language model
• Abstract: Large language models (LLMs) have revolutionized video-based computer vision applications, including action recognition, anomaly detection, and video summarization. Videos inherently pose unique challenges, combining spatial complexity with temporal dynamics that are absent in static images or textual data. Current approaches to video understanding with LLMs often rely on pretrained video encoders to extract spatiotemporal features and text encoders to capture semantic meaning. These representations are integrated within LLM frameworks, enabling multimodal reasoning across diverse video tasks. However, the critical question persists: Can LLMs truly understand the concept of time, and how effectively can they reason about temporal relationships in videos? This work critically examines the role of LLMs in video processing, with a specific focus on their temporal reasoning capabilities. We identify key limitations in the interaction between LLMs and pretrained encoders, revealing gaps in their ability to model long-term dependencies and abstract temporal concepts such as causality and event progression. Furthermore, we analyze challenges posed by existing video datasets, including biases, lack of temporal annotations, and domain-specific limitations that constrain the temporal understanding of LLMs. To address these gaps, we explore promising future directions, including the co-evolution of LLMs and encoders, the development of enriched datasets with explicit temporal labels, and innovative architectures for integrating spatial, temporal, and semantic reasoning. By addressing these challenges, we aim to advance the temporal comprehension of LLMs, unlocking their full potential in video analysis and beyond.

Title: MobiFuse: A High-Precision On-device Depth Perception System with Multi-Data Fusion

• Authors: Jinrui Zhang, Deyu Zhang, Tingting Long, Wenxin Chen, Ju Ren, Yunxin Liu, Yudong Zhao, Yaoxue Zhang, Youngki Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13848
• Pdf URL: https://arxiv.org/pdf/2412.13848
• Copy Paste: [[2412.13848]] MobiFuse: A High-Precision On-device Depth Perception System with Multi-Data Fusion(https://arxiv.org/abs/2412.13848)
• Keywords: segmentation
• Abstract: We present MobiFuse, a high-precision depth perception system on mobile devices that combines dual RGB and Time-of-Flight (ToF) cameras. To achieve this, we leverage physical principles from various environmental factors to propose the Depth Error Indication (DEI) modality, characterizing the depth error of ToF and stereo-matching. Furthermore, we employ a progressive fusion strategy, merging geometric features from ToF and stereo depth maps with depth error features from the DEI modality to create precise depth maps. Additionally, we create a new ToF-Stereo depth dataset, RealToF, to train and validate our model. Our experiments demonstrate that MobiFuse excels over baselines by significantly reducing depth measurement errors by up to 77.7%. It also showcases strong generalization across diverse datasets and proves effectiveness in two downstream tasks: 3D reconstruction and 3D segmentation. The demo video of MobiFuse in real-life scenarios is available at the de-identified YouTube link(this https URL).

Title: RadField3D: A Data Generator and Data Format for Deep Learning in Radiation-Protection Dosimetry for Medical Applications

• Authors: Felix Lehner, Pasquale Lombardo, Susana Castillo, Oliver Hupe, Marcus Magnor
• Subjects: cs.LG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2412.13852
• Pdf URL: https://arxiv.org/pdf/2412.13852
• Copy Paste: [[2412.13852]] RadField3D: A Data Generator and Data Format for Deep Learning in Radiation-Protection Dosimetry for Medical Applications(https://arxiv.org/abs/2412.13852)
• Keywords: protect
• Abstract: In this research work, we present our open-source Geant4-based Monte-Carlo simulation application, called RadField3D, for generating threedimensional radiation field datasets for dosimetry. Accompanying, we introduce a fast, machine-interpretable data format with a Python API for easy integration into neural network research, that we call RadFiled3D. Both developments are intended to be used to research alternative radiation simulation methods using deep learning.

Title: A Systematic Analysis of Input Modalities for Fracture Classification of the Paediatric Wrist

• Authors: Ron Keuth, Maren Balks, Sebastian Tschauner, Ludger Tüshaus, Mattias Heinrich
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13856
• Pdf URL: https://arxiv.org/pdf/2412.13856
• Copy Paste: [[2412.13856]] A Systematic Analysis of Input Modalities for Fracture Classification of the Paediatric Wrist(https://arxiv.org/abs/2412.13856)
• Keywords: segmentation
• Abstract: Fractures, particularly in the distal forearm, are among the most common injuries in children and adolescents, with approximately 800 000 cases treated annually in Germany. The AO/OTA system provides a structured fracture type classification, which serves as the foundation for treatment decisions. Although accurately classifying fractures can be challenging, current deep learning models have demonstrated performance comparable to that of experienced radiologists. While most existing approaches rely solely on radiographs, the potential impact of incorporating other additional modalities, such as automatic bone segmentation, fracture location, and radiology reports, remains underexplored. In this work, we systematically analyse the contribution of these three additional information types, finding that combining them with radiographs increases the AUROC from 91.71 to 93.25. Our code is available on GitHub.

Title: Zero-Shot Prompting and Few-Shot Fine-Tuning: Revisiting Document Image Classification Using Large Language Models

• Authors: Anna Scius-Bertrand, Michael Jungo, Lars Vögtlin, Jean-Marc Spat, Andreas Fischer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13859
• Pdf URL: https://arxiv.org/pdf/2412.13859
• Copy Paste: [[2412.13859]] Zero-Shot Prompting and Few-Shot Fine-Tuning: Revisiting Document Image Classification Using Large Language Models(https://arxiv.org/abs/2412.13859)
• Keywords: large language model
• Abstract: Classifying scanned documents is a challenging problem that involves image, layout, and text analysis for document understanding. Nevertheless, for certain benchmark datasets, notably RVL-CDIP, the state of the art is closing in to near-perfect performance when considering hundreds of thousands of training samples. With the advent of large language models (LLMs), which are excellent few-shot learners, the question arises to what extent the document classification problem can be addressed with only a few training samples, or even none at all. In this paper, we investigate this question in the context of zero-shot prompting and few-shot model fine-tuning, with the aim of reducing the need for human-annotated training samples as much as possible.

Title: Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepali

• Authors: Sharad Duwal, Suraj Prasai, Suresh Manandhar
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13860
• Pdf URL: https://arxiv.org/pdf/2412.13860
• Copy Paste: [[2412.13860]] Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepali(https://arxiv.org/abs/2412.13860)
• Keywords: large language model
• Abstract: Continual learning has emerged as an important research direction due to the infeasibility of retraining large language models (LLMs) from scratch in the event of new data availability. Of great interest is the domain-adaptive pre-training (DAPT) paradigm, which focuses on continually training a pre-trained language model to adapt it to a domain it was not originally trained on. In this work, we evaluate the feasibility of DAPT in a low-resource setting, namely the Nepali language. We use synthetic data to continue training Llama 3 8B to adapt it to the Nepali language in a 4-bit QLoRA setting. We evaluate the adapted model on its performance, forgetting, and knowledge acquisition. We compare the base model and the final model on their Nepali generation abilities, their performance on popular benchmarks, and run case-studies to probe their linguistic knowledge in Nepali. We see some unsurprising forgetting in the final model, but also surprisingly find that increasing the number of shots during evaluation yields better percent increases in the final model (as high as 19.29% increase) compared to the base model (4.98%), suggesting latent retention. We also explore layer-head self-attention heatmaps to establish dependency resolution abilities of the final model in Nepali.

Title: Towards an identity management solution on Arweave

• Authors: Andreea Elena Dragnoiu, Ruxandra F. Olimid
• Subjects: cs.CR, cs.ET
• Abstract URL: https://arxiv.org/abs/2412.13865
• Pdf URL: https://arxiv.org/pdf/2412.13865
• Copy Paste: [[2412.13865]] Towards an identity management solution on Arweave(https://arxiv.org/abs/2412.13865)
• Keywords: security, privacy, protect
• Abstract: Traditional identity management systems, often centralized, face challenges around privacy, data security, and user control, leaving users vulnerable to data breaches and misuse. This paper explores the potential of using the Arweave network to develop an identity management solution. By harnessing Arweave's permanent storage, our solution offers the users a Self-Sovereign Identity (SSI) framework, that uses Decentralized Identifiers (DIDs) and Verifiable Credentials (VCs) to allow individuals and other entities to create, own, and manage their digital identities. Further, the solution integrates privacy-preserving technologies, including zero-knowledge proofs and the BBS(+) signature scheme, enabling selective disclosure. This approach ultimately enhances user privacy and supports compliance with European Union legislation and regulatory standards like the General Data Protection Regulation (GDPR) by design.

Title: SHAP scores fail pervasively even when Lipschitz succeeds

• Authors: Olivier Letoffe, Xuanxiang Huang, Joao Marques-Silva
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13866
• Pdf URL: https://arxiv.org/pdf/2412.13866
• Copy Paste: [[2412.13866]] SHAP scores fail pervasively even when Lipschitz succeeds(https://arxiv.org/abs/2412.13866)
• Keywords: robust
• Abstract: The ubiquitous use of Shapley values in eXplainable AI (XAI) has been triggered by the tool SHAP, and as a result are commonly referred to as SHAP scores. Recent work devised examples of machine learning (ML) classifiers for which the computed SHAP scores are thoroughly unsatisfactory, by allowing human decision-makers to be misled. Nevertheless, such examples could be perceived as somewhat artificial, since the selected classes must be interpreted as numeric. Furthermore, it was unclear how general were the issues identified with SHAP scores. This paper answers these criticisms. First, the paper shows that for Boolean classifiers there are arbitrarily many examples for which the SHAP scores must be deemed unsatisfactory. Second, the paper shows that the issues with SHAP scores are also observed in the case of regression models. In addition, the paper studies the class of regression models that respect Lipschitz continuity, a measure of a function's rate of change that finds important recent uses in ML, including model robustness. Concretely, the paper shows that the issues with SHAP scores occur even for regression models that respect Lipschitz continuity. Finally, the paper shows that the same issues are guaranteed to exist for arbitrarily differentiable regression models.

Title: LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via Hierarchical Window Transformer

• Authors: Yipeng Zhang, Yifan Liu, Zonghao Guo, Yidan Zhang, Xuesong Yang, Chi Chen, Jun Song, Bo Zheng, Yuan Yao, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13871
• Pdf URL: https://arxiv.org/pdf/2412.13871
• Copy Paste: [[2412.13871]] LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via Hierarchical Window Transformer(https://arxiv.org/abs/2412.13871)
• Keywords: transformer, large language model
• Abstract: In multimodal large language models (MLLMs), vision transformers (ViTs) are widely employed for visual encoding. However, their performance in solving universal MLLM tasks is not satisfactory. We attribute it to a lack of information from diverse visual levels, impeding alignment with the various semantic granularity required for language generation. To address this issue, we present LLaVA-UHD v2, an advanced MLLM centered around a Hierarchical window transformer that enables capturing diverse visual granularity by constructing and integrating a high-resolution feature pyramid. As a vision-language projector, Hiwin transformer comprises two primary modules: (i) an inverse feature pyramid, constructed by a ViT-derived feature up-sampling process utilizing high-frequency details from an image pyramid, and (ii) hierarchical window attention, focusing on a set of key sampling features within cross-scale windows to condense multi-level feature maps. Extensive experiments demonstrate that LLaVA-UHD v2 achieves superior performance over existing MLLMs on popular benchmarks. Notably, our design brings an average boost of 3.7% across 14 benchmarks compared with the baseline method, 9.3% on DocVQA for instance. We make all the data, model checkpoint, and code publicly available to facilitate future research.

Title: Crabs: Consuming Resrouce via Auto-generation for LLM-DoS Attack under Black-box Settings

• Authors: Yuanhe Zhang, Zhenhong Zhou, Wei Zhang, Xinyue Wang, Xiaojun Jia, Yang Liu, Sen Su
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.13879
• Pdf URL: https://arxiv.org/pdf/2412.13879
• Copy Paste: [[2412.13879]] Crabs: Consuming Resrouce via Auto-generation for LLM-DoS Attack under Black-box Settings(https://arxiv.org/abs/2412.13879)
• Keywords: defense, attack, steal, large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across diverse tasks. LLMs continue to be vulnerable to external threats, particularly Denial-of-Service (DoS) attacks. Specifically, LLM-DoS attacks aim to exhaust computational resources and block services. However, prior works tend to focus on performing white-box attacks, overlooking black-box settings. In this work, we propose an automated algorithm designed for black-box LLMs, called Auto-Generation for LLM-DoS Attack (AutoDoS). AutoDoS introduces DoS Attack Tree and optimizes the prompt node coverage to enhance effectiveness under black-box conditions. Our method can bypass existing defense with enhanced stealthiness via semantic improvement of prompt nodes. Furthermore, we reveal that implanting Length Trojan in Basic DoS Prompt aids in achieving higher attack efficacy. Experimental results show that AutoDoS amplifies service response latency by over 250 $\times \uparrow$, leading to severe resource consumption in terms of GPU utilization and memory usage. Our code is available at \url{this https URL}.

Title: A Review of the Duality of Adversarial Learning in Network Intrusion: Attacks and Countermeasures

• Authors: Shalini Saini, Anitha Chennamaneni, Babatunde Sawyerr
• Subjects: cs.CR, cs.ET
• Abstract URL: https://arxiv.org/abs/2412.13880
• Pdf URL: https://arxiv.org/pdf/2412.13880
• Copy Paste: [[2412.13880]] A Review of the Duality of Adversarial Learning in Network Intrusion: Attacks and Countermeasures(https://arxiv.org/abs/2412.13880)
• Keywords: security, privacy, defense, attack
• Abstract: Deep learning solutions are instrumental in cybersecurity, harnessing their ability to analyze vast datasets, identify complex patterns, and detect anomalies. However, malevolent actors can exploit these capabilities to orchestrate sophisticated attacks, posing significant challenges to defenders and traditional security measures. Adversarial attacks, particularly those targeting vulnerabilities in deep learning models, present a nuanced and substantial threat to cybersecurity. Our study delves into adversarial learning threats such as Data Poisoning, Test Time Evasion, and Reverse Engineering, specifically impacting Network Intrusion Detection Systems. Our research explores the intricacies and countermeasures of attacks to deepen understanding of network security challenges amidst adversarial threats. In our study, we present insights into the dynamic realm of adversarial learning and its implications for network intrusion. The intersection of adversarial attacks and defenses within network traffic data, coupled with advances in machine learning and deep learning techniques, represents a relatively underexplored domain. Our research lays the groundwork for strengthening defense mechanisms to address the potential breaches in network security and privacy posed by adversarial attacks. Through our in-depth analysis, we identify domain-specific research gaps, such as the scarcity of real-life attack data and the evaluation of AI-based solutions for network traffic. Our focus on these challenges aims to stimulate future research efforts toward the development of resilient network defense strategies.

Title: Understanding and Analyzing Model Robustness and Knowledge-Transfer in Multilingual Neural Machine Translation using TX-Ray

• Authors: Vageesh Saxena, Sharid Loáiciga, Nils Rethmeier
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13881
• Pdf URL: https://arxiv.org/pdf/2412.13881
• Copy Paste: [[2412.13881]] Understanding and Analyzing Model Robustness and Knowledge-Transfer in Multilingual Neural Machine Translation using TX-Ray(https://arxiv.org/abs/2412.13881)
• Keywords: robust
• Abstract: Neural networks have demonstrated significant advancements in Neural Machine Translation (NMT) compared to conventional phrase-based approaches. However, Multilingual Neural Machine Translation (MNMT) in extremely low-resource settings remains underexplored. This research investigates how knowledge transfer across languages can enhance MNMT in such scenarios. Using the Tatoeba translation challenge dataset from Helsinki NLP, we perform English-German, English-French, and English-Spanish translations, leveraging minimal parallel data to establish cross-lingual mappings. Unlike conventional methods relying on extensive pre-training for specific language pairs, we pre-train our model on English-English translations, setting English as the source language for all tasks. The model is fine-tuned on target language pairs using joint multi-task and sequential transfer learning strategies. Our work addresses three key questions: (1) How can knowledge transfer across languages improve MNMT in extremely low-resource scenarios? (2) How does pruning neuron knowledge affect model generalization, robustness, and catastrophic forgetting? (3) How can TX-Ray interpret and quantify knowledge transfer in trained models? Evaluation using BLEU-4 scores demonstrates that sequential transfer learning outperforms baselines on a 40k parallel sentence corpus, showcasing its efficacy. However, pruning neuron knowledge degrades performance, increases catastrophic forgetting, and fails to improve robustness or generalization. Our findings provide valuable insights into the potential and limitations of knowledge transfer and pruning in MNMT for extremely low-resource settings.

Title: T-Edge: Trusted Heterogeneous Edge Computing

• Authors: Jiamin Shen, Yao Chen, Weng-Fai Wong, Ee-Chien Chang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.13905
• Pdf URL: https://arxiv.org/pdf/2412.13905
• Copy Paste: [[2412.13905]] T-Edge: Trusted Heterogeneous Edge Computing(https://arxiv.org/abs/2412.13905)
• Keywords: security, privacy
• Abstract: Heterogeneous computing, which incorporates GPUs, NPUs, and FPGAs, is increasingly utilized to improve the efficiency of computer systems. However, this shift has given rise to significant security and privacy concerns, especially when the execution platform is remote. One way to tackle these challenges is to establish a trusted and isolated environment for remote program execution, while maintaining minimal overhead and flexibility. While CPU-based trusted execution has been extensively explored and found commercial success, extension to heterogeneous computing systems remains a challenge. This paper proposes a practical trusted execution environment design for ARM/FPGA System-on-Chip platforms, leveraging TrustZone's unique characteristics. The design features a dedicated security controller within the ARM TrustZone, overseeing FPGA reconfiguration and managing communication between CPU cores and FPGA fabrics. This design involves a provisioning service that enables application users to establish trust in the FPGA fabric within cloud-based computing resources provided by the platform owner, running applications developed by third-party developers and hardware manufactured by the device manufacturer. To ensure the security of our proposed system, we employ an automated protocol verifier, ProVerif, to validate its compliance with essential security requirements. Furthermore, we demonstrate the practicality of our system model by implementing a prototype application on the Xilinx MPSoC development board.

Title: Memorizing SAM: 3D Medical Segment Anything Model with Memorizing Transformer

• Authors: Xinyuan Shao, Yiqing Shen, Mathias Unberath
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13908
• Pdf URL: https://arxiv.org/pdf/2412.13908
• Copy Paste: [[2412.13908]] Memorizing SAM: 3D Medical Segment Anything Model with Memorizing Transformer(https://arxiv.org/abs/2412.13908)
• Keywords: transformer, segmentation
• Abstract: Segment Anything Models (SAMs) have gained increasing attention in medical image analysis due to their zero-shot generalization capability in segmenting objects of unseen classes and domains when provided with appropriate user prompts. Addressing this performance gap is important to fully leverage the pre-trained weights of SAMs, particularly in the domain of volumetric medical image segmentation, where accuracy is important but well-annotated 3D medical data for fine-tuning is limited. In this work, we investigate whether introducing the memory mechanism as a plug-in, specifically the ability to memorize and recall internal representations of past inputs, can improve the performance of SAM with limited computation cost. To this end, we propose Memorizing SAM, a novel 3D SAM architecture incorporating a memory Transformer as a plug-in. Unlike conventional memorizing Transformers that save the internal representation during training or inference, our Memorizing SAM utilizes existing highly accurate internal representation as the memory source to ensure the quality of memory. We evaluate the performance of Memorizing SAM in 33 categories from the TotalSegmentator dataset, which indicates that Memorizing SAM can outperform state-of-the-art 3D SAM variant i.e., FastSAM3D with an average Dice increase of 11.36% at the cost of only 4.38 millisecond increase in inference time. The source code is publicly available at this https URL

Title: A Black-Box Evaluation Framework for Semantic Robustness in Bird's Eye View Detection

• Authors: Fu Wang, Yanghao Zhang, Xiangyu Yin, Guangliang Cheng, Zeyu Fu, Xiaowei Huang, Wenjie Ruan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13913
• Pdf URL: https://arxiv.org/pdf/2412.13913
• Copy Paste: [[2412.13913]] A Black-Box Evaluation Framework for Semantic Robustness in Bird's Eye View Detection(https://arxiv.org/abs/2412.13913)
• Keywords: robust
• Abstract: Camera-based Bird's Eye View (BEV) perception models receive increasing attention for their crucial role in autonomous driving, a domain where concerns about the robustness and reliability of deep learning have been raised. While only a few works have investigated the effects of randomly generated semantic perturbations, aka natural corruptions, on the multi-view BEV detection task, we develop a black-box robustness evaluation framework that adversarially optimises three common semantic perturbations: geometric transformation, colour shifting, and motion blur, to deceive BEV models, serving as the first approach in this emerging field. To address the challenge posed by optimising the semantic perturbation, we design a smoothed, distance-based surrogate function to replace the mAP metric and introduce SimpleDIRECT, a deterministic optimisation algorithm that utilises observed slopes to guide the optimisation process. By comparing with randomised perturbation and two optimisation baselines, we demonstrate the effectiveness of the proposed framework. Additionally, we provide a benchmark on the semantic robustness of ten recent BEV models. The results reveal that PolarFormer, which emphasises geometric information from multi-view images, exhibits the highest robustness, whereas BEVDet is fully compromised, with its precision reduced to zero.

Title: Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basque

• Authors: Ander Corral, Ixak Sarasua, Xabier Saralegi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13922
• Pdf URL: https://arxiv.org/pdf/2412.13922
• Copy Paste: [[2412.13922]] Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basque(https://arxiv.org/abs/2412.13922)
• Keywords: large language model
• Abstract: Large language models (LLMs) are typically optimized for resource-rich languages like English, exacerbating the gap between high-resource and underrepresented languages. This work presents a detailed analysis of strategies for developing a model capable of following instructions in a low-resource language, specifically Basque, by focusing on three key stages: pre-training, instruction tuning, and alignment with human preferences. Our findings demonstrate that continual pre-training with a high-quality Basque corpus of around 600 million words improves natural language understanding (NLU) of the foundational model by over 12 points. Moreover, instruction tuning and human preference alignment using automatically translated datasets proved highly effective, resulting in a 24-point improvement in instruction-following performance. The resulting models, Llama-eus-8B and Llama-eus-8B-instruct, establish a new state-of-the-art for Basque in the sub-10B parameter category.

Title: Spatio-Temporal Forecasting of PM2.5 via Spatial-Diffusion guided Encoder-Decoder Architecture

• Authors: Malay Pandey, Vaishali Jain, Nimit Godhani, Sachchida Nand Tripathi, Piyush Rai
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13935
• Pdf URL: https://arxiv.org/pdf/2412.13935
• Copy Paste: [[2412.13935]] Spatio-Temporal Forecasting of PM2.5 via Spatial-Diffusion guided Encoder-Decoder Architecture(https://arxiv.org/abs/2412.13935)
• Keywords: diffusion, transformer
• Abstract: In many problem settings that require spatio-temporal forecasting, the values in the time-series not only exhibit spatio-temporal correlations but are also influenced by spatial diffusion across locations. One such example is forecasting the concentration of fine particulate matter (PM2.5) in the atmosphere which is influenced by many complex factors, the most important ones being diffusion due to meteorological factors as well as transport across vast distances over a period of time. We present a novel Spatio-Temporal Graph Neural Network architecture, that specifically captures these dependencies to forecast the PM2.5 concentration. Our model is based on an encoder-decoder architecture where the encoder and decoder parts leverage gated recurrent units (GRU) augmented with a graph neural network (TransformerConv) to account for spatial diffusion. Our model can also be seen as a generalization of various existing models for time-series or spatio-temporal forecasting. We demonstrate the model's effectiveness on two real-world PM2.5 datasets: (1) data collected by us using a recently deployed network of low-cost PM$_{2.5}$ sensors from 511 locations spanning the entirety of the Indian state of Bihar over a period of one year, and (2) another publicly available dataset that covers severely polluted regions from China for a period of 4 years. Our experimental results show our model's impressive ability to account for both spatial as well as temporal dependencies precisely.

Title: A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI

• Authors: Beiduo Chen, Siyao Peng, Anna Korhonen, Barbara Plank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13942
• Pdf URL: https://arxiv.org/pdf/2412.13942
• Copy Paste: [[2412.13942]] A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI(https://arxiv.org/abs/2412.13942)
• Keywords: large language model
• Abstract: Disagreement in human labeling is ubiquitous, and can be captured in human judgment distributions (HJDs). Recent research has shown that explanations provide valuable information for understanding human label variation (HLV) and large language models (LLMs) can approximate HJD from a few human-provided label-explanation pairs. However, collecting explanations for every label is still time-consuming. This paper examines whether LLMs can be used to replace humans in generating explanations for approximating HJD. Specifically, we use LLMs as annotators to generate model explanations for a few given human labels. We test ways to obtain and combine these label-explanations with the goal to approximate human judgment distribution. We further compare the resulting human with model-generated explanations, and test automatic and human explanation selection. Our experiments show that LLM explanations are promising for NLI: to estimate HJD, generated explanations yield comparable results to human's when provided with human labels. Importantly, our results generalize from datasets with human explanations to i) datasets where they are not available and ii) challenging out-of-distribution test sets.

Title: Real Classification by Description: Extending CLIP's Limits of Part Attributes Recognition

• Authors: Ethan Baron, Idan Tankel, Peter Tu, Guy Ben-Yosef
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13947
• Pdf URL: https://arxiv.org/pdf/2412.13947
• Copy Paste: [[2412.13947]] Real Classification by Description: Extending CLIP's Limits of Part Attributes Recognition(https://arxiv.org/abs/2412.13947)
• Keywords: large language model
• Abstract: In this study, we define and tackle zero shot "real" classification by description, a novel task that evaluates the ability of Vision-Language Models (VLMs) like CLIP to classify objects based solely on descriptive attributes, excluding object class names. This approach highlights the current limitations of VLMs in understanding intricate object descriptions, pushing these models beyond mere object recognition. To facilitate this exploration, we introduce a new challenge and release description data for six popular fine-grained benchmarks, which omit object names to encourage genuine zero-shot learning within the research community. Additionally, we propose a method to enhance CLIP's attribute detection capabilities through targeted training using ImageNet21k's diverse object categories, paired with rich attribute descriptions generated by large language models. Furthermore, we introduce a modified CLIP architecture that leverages multiple resolutions to improve the detection of fine-grained part attributes. Through these efforts, we broaden the understanding of part-attribute recognition in CLIP, improving its performance in fine-grained classification tasks across six popular benchmarks, as well as in the PACO dataset, a widely used benchmark for object-attribute recognition. Code is available at: this https URL.

Title: Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence

• Authors: Jinghan He, Kuan Zhu, Haiyun Guo, Junfeng Fang, Zhenglin Hua, Yuheng Jia, Ming Tang, Tat-Seng Chua, Jinqiao Wang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13949
• Pdf URL: https://arxiv.org/pdf/2412.13949
• Copy Paste: [[2412.13949]] Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence(https://arxiv.org/abs/2412.13949)
• Keywords: large language model
• Abstract: Large vision-language models (LVLMs) have made substantial progress in integrating large language models (LLMs) with visual inputs, enabling advanced multimodal reasoning. Despite their success, a persistent challenge is hallucination-where generated text fails to accurately reflect visual content-undermining both accuracy and reliability. Existing methods focus on alignment training or decoding refinements but primarily address symptoms at the generation stage without probing the underlying causes. In this work, we investigate the internal mechanisms driving hallucination in LVLMs, with an emphasis on the multi-head attention module. Specifically, we introduce Vision-aware Head Divergence (VHD), a metric that quantifies the sensitivity of attention head outputs to visual context. Based on this, our findings reveal the presence of vision-aware attention heads that are more attuned to visual information; however, the model's overreliance on its prior language patterns is closely related to hallucinations. Building on these insights, we propose Vision-aware Head Reinforcement (VHR), a training-free approach to mitigate hallucination by enhancing the role of vision-aware attention heads. Extensive experiments demonstrate that our method achieves superior performance compared to state-of-the-art approaches in mitigating hallucinations, while maintaining high efficiency with negligible additional time overhead.

Title: Prompting Strategies for Enabling Large Language Models to Infer Causation from Correlation

• Authors: Eleni Sgouritsa, Virginia Aglietti, Yee Whye Teh, Arnaud Doucet, Arthur Gretton, Silvia Chiappa
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13952
• Pdf URL: https://arxiv.org/pdf/2412.13952
• Copy Paste: [[2412.13952]] Prompting Strategies for Enabling Large Language Models to Infer Causation from Correlation(https://arxiv.org/abs/2412.13952)
• Keywords: robust, large language model
• Abstract: The reasoning abilities of Large Language Models (LLMs) are attracting increasing attention. In this work, we focus on causal reasoning and address the task of establishing causal relationships based on correlation information, a highly challenging problem on which several LLMs have shown poor performance. We introduce a prompting strategy for this problem that breaks the original task into fixed subquestions, with each subquestion corresponding to one step of a formal causal discovery algorithm, the PC algorithm. The proposed prompting strategy, PC-SubQ, guides the LLM to follow these algorithmic steps, by sequentially prompting it with one subquestion at a time, augmenting the next subquestion's prompt with the answer to the previous one(s). We evaluate our approach on an existing causal benchmark, Corr2Cause: our experiments indicate a performance improvement across five LLMs when comparing PC-SubQ to baseline prompting strategies. Results are robust to causal query perturbations, when modifying the variable names or paraphrasing the expressions.

Title: Self-attentive Transformer for Fast and Accurate Postprocessing of Temperature and Wind Speed Forecasts

• Authors: Aaron Van Poecke, Tobias Sebastian Finn, Ruoke Meng, Joris Van den Bergh, Geert Smet, Jonathan Demaeyer, Piet Termonia, Hossein Tabari, Peter Hellinckx
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2412.13957
• Pdf URL: https://arxiv.org/pdf/2412.13957
• Copy Paste: [[2412.13957]] Self-attentive Transformer for Fast and Accurate Postprocessing of Temperature and Wind Speed Forecasts(https://arxiv.org/abs/2412.13957)
• Keywords: transformer
• Abstract: Current postprocessing techniques often require separate models for each lead time and disregard possible inter-ensemble relationships by either correcting each member separately or by employing distributional approaches. In this work, we tackle these shortcomings with an innovative, fast and accurate Transformer which postprocesses each ensemble member individually while allowing information exchange across variables, spatial dimensions and lead times by means of multi-headed self-attention. Weather foreacasts are postprocessed over 20 lead times simultaneously while including up to twelve meteorological predictors. We use the EUPPBench dataset for training which contains ensemble predictions from the European Center for Medium-range Weather Forecasts' integrated forecasting system alongside corresponding observations. The work presented here is the first to postprocess the ten and one hundred-meter wind speed forecasts within this benchmark dataset, while also correcting the two-meter temperature. Our approach significantly improves the original forecasts, as measured by the CRPS, with 17.5 % for two-meter temperature, nearly 5% for ten-meter wind speed and 5.3 % for one hundred-meter wind speed, outperforming a classical member-by-member approach employed as competitive benchmark. Furthermore, being up to 75 times faster, it fulfills the demand for rapid operational weather forecasts in various downstream applications, including renewable energy forecasting.

Title: Harvesting energy from turbulent winds with Reinforcement Learning

• Authors: Lorenzo Basile, Maria Grazia Berni, Antonio Celani
• Subjects: cs.LG, eess.SY, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2412.13961
• Pdf URL: https://arxiv.org/pdf/2412.13961
• Copy Paste: [[2412.13961]] Harvesting energy from turbulent winds with Reinforcement Learning(https://arxiv.org/abs/2412.13961)
• Keywords: robust
• Abstract: Airborne Wind Energy (AWE) is an emerging technology designed to harness the power of high-altitude winds, offering a solution to several limitations of conventional wind turbines. AWE is based on flying devices (usually gliders or kites) that, tethered to a ground station and driven by the wind, convert its mechanical energy into electrical energy by means of a generator. Such systems are usually controlled by manoeuvering the kite so as to follow a predefined path prescribed by optimal control techniques, such as model-predictive control. These methods are strongly dependent on the specific model at use and difficult to generalize, especially in unpredictable conditions such as the turbulent atmospheric boundary layer. Our aim is to explore the possibility of replacing these techniques with an approach based on Reinforcement Learning (RL). Unlike traditional methods, RL does not require a predefined model, making it robust to variability and uncertainty. Our experimental results in complex simulated environments demonstrate that AWE agents trained with RL can effectively extract energy from turbulent flows, relying on minimal local information about the kite orientation and speed relative to the wind.

Title: Comparative Analysis of Machine Learning-Based Imputation Techniques for Air Quality Datasets with High Missing Data Rates

• Authors: Sen Yan, David J. O'Connor, Xiaojun Wang, Noel E. O'Connor, Alan. F. Smeaton, Mingming Liu
• Subjects: cs.LG, physics.data-an
• Abstract URL: https://arxiv.org/abs/2412.13966
• Pdf URL: https://arxiv.org/pdf/2412.13966
• Copy Paste: [[2412.13966]] Comparative Analysis of Machine Learning-Based Imputation Techniques for Air Quality Datasets with High Missing Data Rates(https://arxiv.org/abs/2412.13966)
• Keywords: protect, diffusion
• Abstract: Urban pollution poses serious health risks, particularly in relation to traffic-related air pollution, which remains a major concern in many cities. Vehicle emissions contribute to respiratory and cardiovascular issues, especially for vulnerable and exposed road users like pedestrians and cyclists. Therefore, accurate air quality monitoring with high spatial resolution is vital for good urban environmental management. This study aims to provide insights for processing spatiotemporal datasets with high missing data rates. In this study, the challenge of high missing data rates is a result of the limited data available and the fine granularity required for precise classification of PM2.5 levels. The data used for analysis and imputation were collected from both mobile sensors and fixed stations by Dynamic Parcel Distribution, the Environmental Protection Agency, and Google in Dublin, Ireland, where the missing data rate was approximately 82.42%, making accurate Particulate Matter 2.5 level predictions particularly difficult. Various imputation and prediction approaches were evaluated and compared, including ensemble methods, deep learning models, and diffusion models. External features such as traffic flow, weather conditions, and data from the nearest stations were incorporated to enhance model performance. The results indicate that diffusion methods with external features achieved the highest F1 score, reaching 0.9486 (Accuracy: 94.26%, Precision: 94.42%, Recall: 94.82%), with ensemble models achieving the highest accuracy of 94.82%, illustrating that good performance can be obtained despite a high missing data rate.

Title: RAG for Effective Supply Chain Security Questionnaire Automation

• Authors: Zaynab Batool Reza, Abdul Rafay Syed, Omer Iqbal, Ethel Mensah, Qian Liu, Maxx Richard Rahman, Wolfgang Maass
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13988
• Pdf URL: https://arxiv.org/pdf/2412.13988
• Copy Paste: [[2412.13988]] RAG for Effective Supply Chain Security Questionnaire Automation(https://arxiv.org/abs/2412.13988)
• Keywords: secure, security, large language model
• Abstract: In an era where digital security is crucial, efficient processing of security-related inquiries through supply chain security questionnaires is imperative. This paper introduces a novel approach using Natural Language Processing (NLP) and Retrieval-Augmented Generation (RAG) to automate these responses. We developed QuestSecure, a system that interprets diverse document formats and generates precise responses by integrating large language models (LLMs) with an advanced retrieval system. Our experiments show that QuestSecure significantly improves response accuracy and operational efficiency. By employing advanced NLP techniques and tailored retrieval mechanisms, the system consistently produces contextually relevant and semantically rich responses, reducing cognitive load on security teams and minimizing potential errors. This research offers promising avenues for automating complex security management tasks, enhancing organizational security processes.

Title: Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes

• Authors: Katarzyna Kobalczyk, Claudio Fanconi, Hao Sun, Mihaela van der Schaar
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13998
• Pdf URL: https://arxiv.org/pdf/2412.13998
• Copy Paste: [[2412.13998]] Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes(https://arxiv.org/abs/2412.13998)
• Keywords: large language model
• Abstract: As large language models (LLMs) become increasingly embedded in everyday applications, ensuring their alignment with the diverse preferences of individual users has become a critical challenge. Currently deployed approaches typically assume homogeneous user objectives and rely on single-objective fine-tuning. However, human preferences are inherently heterogeneous, influenced by various unobservable factors, leading to conflicting signals in preference data. Existing solutions addressing this diversity often require costly datasets labelled for specific objectives and involve training multiple reward models or LLM policies, which is computationally expensive and impractical. In this work, we present a novel framework for few-shot steerable alignment, where users' underlying preferences are inferred from a small sample of their choices. To achieve this, we extend the Bradley-Terry-Luce model to handle heterogeneous preferences with unobserved variability factors and propose its practical implementation for reward modelling and LLM fine-tuning. Thanks to our proposed approach of functional parameter-space conditioning, LLMs trained with our framework can be adapted to individual preferences at inference time, generating outputs over a continuum of behavioural modes. We empirically validate the effectiveness of methods, demonstrating their ability to capture and align with diverse human preferences in a data-efficient manner. Our code is made available at: this https URL.

Title: InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models

• Authors: Cong Wei, Yujie Zhong, Haoxian Tan, Yingsen Zeng, Yong Liu, Zheng Zhao, Yujiu Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14006
• Pdf URL: https://arxiv.org/pdf/2412.14006
• Copy Paste: [[2412.14006]] InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models(https://arxiv.org/abs/2412.14006)
• Keywords: large language model, segmentation
• Abstract: Boosted by Multi-modal Large Language Models (MLLMs), text-guided universal segmentation models for the image and video domains have made rapid progress recently. However, these methods are often developed separately for specific domains, overlooking the similarities in task settings and solutions across these two areas. In this paper, we define the union of referring segmentation and reasoning segmentation at both the image and video levels as Instructed Visual Segmentation (IVS). Correspondingly, we propose InstructSeg, an end-to-end segmentation pipeline equipped with MLLMs for IVS. Specifically, we employ an object-aware video perceiver to extract temporal and object information from reference frames, facilitating comprehensive video understanding. Additionally, we introduce vision-guided multi-granularity text fusion to better integrate global and detailed text information with fine-grained visual guidance. By leveraging multi-task and end-to-end training, InstructSeg demonstrates superior performance across diverse image and video segmentation tasks, surpassing both segmentation specialists and MLLM-based methods with a single model. Our code is available at this https URL.

Title: FarExStance: Explainable Stance Detection for Farsi

• Authors: Majid Zarharan, Maryam Hashemi, Malika Behroozrazegh, Sauleh Eetemadi, Mohammad Taher Pilehvar, Jennifer Foster
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.14008
• Pdf URL: https://arxiv.org/pdf/2412.14008
• Copy Paste: [[2412.14008]] FarExStance: Explainable Stance Detection for Farsi(https://arxiv.org/abs/2412.14008)
• Keywords: large language model
• Abstract: We introduce FarExStance, a new dataset for explainable stance detection in Farsi. Each instance in this dataset contains a claim, the stance of an article or social media post towards that claim, and an extractive explanation which provides evidence for the stance label. We compare the performance of a fine-tuned multilingual RoBERTa model to several large language models in zero-shot, few-shot, and parameter-efficient fine-tuned settings on our new dataset. On stance detection, the most accurate models are the fine-tuned RoBERTa model, the LLM Aya-23-8B which has been fine-tuned using parameter-efficient fine-tuning, and few-shot Claude-3.5-Sonnet. Regarding the quality of the explanations, our automatic evaluation metrics indicate that few-shot GPT-4o generates the most coherent explanations, while our human evaluation reveals that the best Overall Explanation Score (OES) belongs to few-shot Claude-3.5-Sonnet. The fine-tuned Aya-32-8B model produced explanations most closely aligned with the reference explanations.

Title: Towards an optimised evaluation of teachers' discourse: The case of engaging messages

• Authors: Samuel Falcon, Jaime Leon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.14011
• Pdf URL: https://arxiv.org/pdf/2412.14011
• Copy Paste: [[2412.14011]] Towards an optimised evaluation of teachers' discourse: The case of engaging messages(https://arxiv.org/abs/2412.14011)
• Keywords: large language model
• Abstract: Evaluating teachers' skills is crucial for enhancing education quality and student outcomes. Teacher discourse, significantly influencing student performance, is a key component. However, coding this discourse can be laborious. This study addresses this issue by introducing a new methodology for optimising the assessment of teacher discourse. The research consisted of two studies, both within the framework of engaging messages used by secondary education teachers. The first study involved training two large language models on real-world examples from audio-recorded lessons over two academic years to identify and classify the engaging messages from the lessons' transcripts. This resulted in sensitivities of 84.31% and 91.11%, and specificities of 97.69% and 86.36% in identification and classification, respectively. The second study applied these models to transcripts of audio-recorded lessons from a third academic year to examine the frequency and distribution of message types by educational level and moment of the academic year. Results showed teachers predominantly use messages emphasising engagement benefits, linked to improved outcomes, while one-third highlighted non-engagement disadvantages, associated with increased anxiety. The use of engaging messages declined in Grade 12 and towards the academic year's end. These findings suggest potential interventions to optimise engaging message use, enhancing teaching quality and student outcomes.

Title: SurgSora: Decoupled RGBD-Flow Diffusion Model for Controllable Surgical Video Generation

• Authors: Tong Chen, Shuya Yang, Junyi Wang, Long Bai, Hongliang Ren, Luping Zhou
• Subjects: cs.CV, cs.AI, cs.MM, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.14018
• Pdf URL: https://arxiv.org/pdf/2412.14018
• Copy Paste: [[2412.14018]] SurgSora: Decoupled RGBD-Flow Diffusion Model for Controllable Surgical Video Generation(https://arxiv.org/abs/2412.14018)
• Keywords: diffusion, segmentation
• Abstract: Medical video generation has transformative potential for enhancing surgical understanding and pathology insights through precise and controllable visual representations. However, current models face limitations in controllability and authenticity. To bridge this gap, we propose SurgSora, a motion-controllable surgical video generation framework that uses a single input frame and user-controllable motion cues. SurgSora consists of three key modules: the Dual Semantic Injector (DSI), which extracts object-relevant RGB and depth features from the input frame and integrates them with segmentation cues to capture detailed spatial features of complex anatomical structures; the Decoupled Flow Mapper (DFM), which fuses optical flow with semantic-RGB-D features at multiple scales to enhance temporal understanding and object spatial dynamics; and the Trajectory Controller (TC), which allows users to specify motion directions and estimates sparse optical flow, guiding the video generation process. The fused features are used as conditions for a frozen Stable Diffusion model to produce realistic, temporally coherent surgical videos. Extensive evaluations demonstrate that SurgSora outperforms state-of-the-art methods in controllability and authenticity, showing its potential to advance surgical video generation for medical education, training, and research.

Title: Machine learning in wastewater treatment: insights from modelling a pilot denitrification reactor

• Authors: Eivind Bøhn, Sølve Eidnes, Kjell Rune Jonassen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.14030
• Pdf URL: https://arxiv.org/pdf/2412.14030
• Copy Paste: [[2412.14030]] Machine learning in wastewater treatment: insights from modelling a pilot denitrification reactor(https://arxiv.org/abs/2412.14030)
• Keywords: robust
• Abstract: Wastewater treatment plants are increasingly recognized as promising candidates for machine learning applications, due to their societal importance and high availability of data. However, their varied designs, operational conditions, and influent characteristics hinder straightforward automation. In this study, we use data from a pilot reactor at the Veas treatment facility in Norway to explore how machine learning can be used to optimize biological nitrate ($\mathrm{NO_3^-}$) reduction to molecular nitrogen ($\mathrm{N_2}$) in the biogeochemical process known as \textit{denitrification}. Rather than focusing solely on predictive accuracy, our approach prioritizes understanding the foundational requirements for effective data-driven modelling of wastewater treatment. Specifically, we aim to identify which process parameters are most critical, the necessary data quantity and quality, how to structure data effectively, and what properties are required by the models. We find that nonlinear models perform best on the training and validation data sets, indicating nonlinear relationships to be learned, but linear models transfer better to the unseen test data, which comes later in time. The variable measuring the water temperature has a particularly detrimental effect on the models, owing to a significant change in distributions between training and test data. We therefore conclude that multiple years of data is necessary to learn robust machine learning models. By addressing foundational elements, particularly in the context of the climatic variability faced by northern regions, this work lays the groundwork for a more structured and tailored approach to machine learning for wastewater treatment. We share publicly both the data and code used to produce the results in the paper.

Title: Hansel: Output Length Controlling Framework for Large Language Models

• Authors: Seoha Song, Junhyun Lee, Hyeonmok Ko
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.14033
• Pdf URL: https://arxiv.org/pdf/2412.14033
• Copy Paste: [[2412.14033]] Hansel: Output Length Controlling Framework for Large Language Models(https://arxiv.org/abs/2412.14033)
• Keywords: large language model
• Abstract: Despite the great success of large language models (LLMs), efficiently controlling the length of the output sequence still remains a challenge. In this paper, we propose Hansel, an efficient framework for length control in LLMs without affecting its generation ability. Hansel utilizes periodically outputted hidden special tokens to keep track of the remaining target length of the output sequence. Together with techniques to avoid abrupt termination of the output, this seemingly simple method proved to be efficient and versatile, while not harming the coherency and fluency of the generated text. The framework can be applied to any pre-trained LLMs during the finetuning stage of the model, regardless of its original positional encoding method. We demonstrate this by finetuning four different LLMs with Hansel and show that the mean absolute error of the output sequence decreases significantly in every model and dataset compared to the prompt-based length control finetuning. Moreover, the framework showed a substantially improved ability to extrapolate to target lengths unseen during finetuning, such as long dialog responses or extremely short summaries. This indicates that the model learns the general means of length control, rather than learning to match output lengths to those seen during training.

Title: CAD-Recode: Reverse Engineering CAD Code from Point Clouds

• Authors: Danila Rukhovich, Elona Dupont, Dimitrios Mallis, Kseniya Cherenkova, Anis Kacem, Djamila Aouada
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14042
• Pdf URL: https://arxiv.org/pdf/2412.14042
• Copy Paste: [[2412.14042]] CAD-Recode: Reverse Engineering CAD Code from Point Clouds(https://arxiv.org/abs/2412.14042)
• Keywords: large language model
• Abstract: Computer-Aided Design (CAD) models are typically constructed by sequentially drawing parametric sketches and applying CAD operations to obtain a 3D model. The problem of 3D CAD reverse engineering consists of reconstructing the sketch and CAD operation sequences from 3D representations such as point clouds. In this paper, we address this challenge through novel contributions across three levels: CAD sequence representation, network design, and dataset. In particular, we represent CAD sketch-extrude sequences as Python code. The proposed CAD-Recode translates a point cloud into Python code that, when executed, reconstructs the CAD model. Taking advantage of the exposure of pre-trained Large Language Models (LLMs) to Python code, we leverage a relatively small LLM as a decoder for CAD-Recode and combine it with a lightweight point cloud projector. CAD-Recode is trained solely on a proposed synthetic dataset of one million diverse CAD sequences. CAD-Recode significantly outperforms existing methods across three datasets while requiring fewer input points. Notably, it achieves 10 times lower mean Chamfer distance than state-of-the-art methods on DeepCAD and Fusion360 datasets. Furthermore, we show that our CAD Python code output is interpretable by off-the-shelf LLMs, enabling CAD editing and CAD-specific question answering from point clouds.

Title: Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation

• Authors: Vera Neplenbroek, Arianna Bisazza, Raquel Fernández
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.14050
• Pdf URL: https://arxiv.org/pdf/2412.14050
• Copy Paste: [[2412.14050]] Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation(https://arxiv.org/abs/2412.14050)
• Keywords: generative, large language model
• Abstract: Recent generative large language models (LLMs) show remarkable performance in non-English languages, but when prompted in those languages they tend to express higher harmful social biases and toxicity levels. Prior work has shown that finetuning on specialized datasets can mitigate this behavior, and doing so in English can transfer to other languages. In this work, we investigate the impact of different finetuning methods on the model's bias and toxicity, but also on its ability to produce fluent and diverse text. Our results show that finetuning on curated non-harmful text is more effective for mitigating bias, and finetuning on direct preference optimization (DPO) datasets is more effective for mitigating toxicity. The mitigation caused by applying these methods in English also transfers to non-English languages. We find evidence that the extent to which transfer takes place can be predicted by the amount of data in a given language present in the model's pretraining data. However, this transfer of bias and toxicity mitigation often comes at the expense of decreased language generation ability in non-English languages, highlighting the importance of developing language-specific bias and toxicity mitigation methods.

Title: Digestion Algorithm in Hierarchical Symbolic Forests: A Fast Text Normalization Algorithm and Semantic Parsing Framework for Specific Scenarios and Lightweight Deployment

• Authors: Kevin You
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.14054
• Pdf URL: https://arxiv.org/pdf/2412.14054
• Copy Paste: [[2412.14054]] Digestion Algorithm in Hierarchical Symbolic Forests: A Fast Text Normalization Algorithm and Semantic Parsing Framework for Specific Scenarios and Lightweight Deployment(https://arxiv.org/abs/2412.14054)
• Keywords: interpretability, large language model
• Abstract: Text Normalization and Semantic Parsing have numerous applications in natural language processing, such as natural language programming, paraphrasing, data augmentation, constructing expert systems, text matching, and more. Despite the prominent achievements of deep learning in Large Language Models (LLMs), the interpretability of neural network architectures is still poor, which affects their credibility and hence limits the deployments of risk-sensitive scenarios. In certain scenario-specific domains with scarce data, rapidly obtaining a large number of supervised learning labels is challenging, and the workload of manually labeling data would be enormous. Catastrophic forgetting in neural networks further leads to low data utilization rates. In situations where swift responses are vital, the density of the model makes local deployment difficult and the response time long, which is not conducive to local applications of these fields. Inspired by the multiplication rule, a principle of combinatorial mathematics, and human thinking patterns, a multilayer framework along with its algorithm, the Digestion Algorithm in Hierarchical Symbolic Forests (DAHSF), is proposed to address these above issues, combining text normalization and semantic parsing workflows. The Chinese Scripting Language "Fire Bunny Intelligent Development Platform V2.0" is an important test and application of the technology discussed in this paper. DAHSF can run locally in scenario-specific domains on little datasets, with model size and memory usage optimized by at least two orders of magnitude, thus improving the execution speed, and possessing a promising optimization outlook.

Title: A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future

• Authors: Shilin Sun, Wenbin An, Feng Tian, Fang Nan, Qidong Liu, Jun Liu, Nazaraf Shah, Ping Chen
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.14056
• Pdf URL: https://arxiv.org/pdf/2412.14056
• Copy Paste: [[2412.14056]] A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future(https://arxiv.org/abs/2412.14056)
• Keywords: fair, interpretability, generative, large language model
• Abstract: Artificial intelligence (AI) has rapidly developed through advancements in computational power and the growth of massive datasets. However, this progress has also heightened challenges in interpreting the "black-box" nature of AI models. To address these concerns, eXplainable AI (XAI) has emerged with a focus on transparency and interpretability to enhance human understanding and trust in AI decision-making processes. In the context of multimodal data fusion and complex reasoning scenarios, the proposal of Multimodal eXplainable AI (MXAI) integrates multiple modalities for prediction and explanation tasks. Meanwhile, the advent of Large Language Models (LLMs) has led to remarkable breakthroughs in natural language processing, yet their complexity has further exacerbated the issue of MXAI. To gain key insights into the development of MXAI methods and provide crucial guidance for building more transparent, fair, and trustworthy AI systems, we review the MXAI methods from a historical perspective and categorize them across four eras: traditional machine learning, deep learning, discriminative foundation models, and generative LLMs. We also review evaluation metrics and datasets used in MXAI research, concluding with a discussion of future challenges and directions. A project related to this review has been created at this https URL.

Title: Online MDP with Transition Prototypes: A Robust Adaptive Approach

• Authors: Shuo Sun, Meng Qi, Zuo-jun Max Shen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.14075
• Pdf URL: https://arxiv.org/pdf/2412.14075
• Copy Paste: [[2412.14075]] Online MDP with Transition Prototypes: A Robust Adaptive Approach(https://arxiv.org/abs/2412.14075)
• Keywords: robust
• Abstract: In this work, we consider an online robust Markov Decision Process (MDP) where we have the information of finitely many prototypes of the underlying transition kernel. We consider an adaptively updated ambiguity set of the prototypes and propose an algorithm that efficiently identifies the true underlying transition kernel while guaranteeing the performance of the corresponding robust policy. To be more specific, we provide a sublinear regret of the subsequent optimal robust policy. We also provide an early stopping mechanism and a worst-case performance bound of the value function. In numerical experiments, we demonstrate that our method outperforms existing approaches, particularly in the early stage with limited data. This work contributes to robust MDPs by considering possible prior information about the underlying transition probability and online learning, offering both theoretical insights and practical algorithms for improved decision-making under uncertainty.

Title: On the Robustness of Distributed Machine Learning against Transfer Attacks

• Authors: Sébastien Andreina, Pascal Zimmer, Ghassan Karame
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.14080
• Pdf URL: https://arxiv.org/pdf/2412.14080
• Copy Paste: [[2412.14080]] On the Robustness of Distributed Machine Learning against Transfer Attacks(https://arxiv.org/abs/2412.14080)
• Keywords: attack, robust, federate
• Abstract: Although distributed machine learning (distributed ML) is gaining considerable attention in the community, prior works have independently looked at instances of distributed ML in either the training or the inference phase. No prior work has examined the combined robustness stemming from distributing both the learning and the inference process. In this work, we explore, for the first time, the robustness of distributed ML models that are fully heterogeneous in training data, architecture, scheduler, optimizer, and other model parameters. Supported by theory and extensive experimental validation using CIFAR10 and FashionMNIST, we show that such properly distributed ML instantiations achieve across-the-board improvements in accuracy-robustness tradeoffs against state-of-the-art transfer-based attacks that could otherwise not be realized by current ensemble or federated learning instantiations. For instance, our experiments on CIFAR10 show that for the Common Weakness attack, one of the most powerful state-of-the-art transfer-based attacks, our method improves robust accuracy by up to 40%, with a minimal impact on clean task accuracy.

Title: Future Research Avenues for Artificial Intelligence in Digital Gaming: An Exploratory Report

• Authors: Markus Dablander
• Subjects: cs.LG, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.14085
• Pdf URL: https://arxiv.org/pdf/2412.14085
• Copy Paste: [[2412.14085]] Future Research Avenues for Artificial Intelligence in Digital Gaming: An Exploratory Report(https://arxiv.org/abs/2412.14085)
• Keywords: generative, large language model
• Abstract: Video games are a natural and synergistic application domain for artificial intelligence (AI) systems, offering both the potential to enhance player experience and immersion, as well as providing valuable benchmarks and virtual environments to advance AI technologies in general. This report presents a high-level overview of five promising research pathways for applying state-of-the-art AI methods, particularly deep learning, to digital gaming within the context of the current research landscape. The objective of this work is to outline a curated, non-exhaustive list of encouraging research directions at the intersection of AI and video games that may serve to inspire more rigorous and comprehensive research efforts in the future. We discuss (i) investigating large language models as core engines for game agent modelling, (ii) using neural cellular automata for procedural game content generation, (iii) accelerating computationally expensive in-game simulations via deep surrogate modelling, (iv) leveraging self-supervised learning to obtain useful video game state embeddings, and (v) training generative models of interactive worlds using unlabelled video data. We also briefly address current technical challenges associated with the integration of advanced deep learning systems into video game development, and indicate key areas where further progress is likely to be beneficial.

Title: SEKE: Specialised Experts for Keyword Extraction

• Authors: Matej Martinc, Hanh Thi Hong Tran, Senja Pollak, Boshko Koloski
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.14087
• Pdf URL: https://arxiv.org/pdf/2412.14087
• Copy Paste: [[2412.14087]] SEKE: Specialised Experts for Keyword Extraction(https://arxiv.org/abs/2412.14087)
• Keywords: extraction, explainability
• Abstract: Keyword extraction involves identifying the most descriptive words in a document, allowing automatic categorisation and summarisation of large quantities of diverse textual data. Relying on the insight that real-world keyword detection often requires handling of diverse content, we propose a novel supervised keyword extraction approach based on the mixture of experts (MoE) technique. MoE uses a learnable routing sub-network to direct information to specialised experts, allowing them to specialize in distinct regions of the input space. SEKE, a mixture of Specialised Experts for supervised Keyword Extraction, uses DeBERTa as the backbone model and builds on the MoE framework, where experts attend to each token, by integrating it with a recurrent neural network (RNN), to allow successful extraction even on smaller corpora, where specialisation is harder due to lack of training data. The MoE framework also provides an insight into inner workings of individual experts, enhancing the explainability of the approach. We benchmark SEKE on multiple English datasets, achieving state-of-the-art performance compared to strong supervised and unsupervised baselines. Our analysis reveals that depending on data size and type, experts specialize in distinct syntactic and semantic components, such as punctuation, stopwords, parts-of-speech, or named entities. Code is available at: this https URL

Title: Adaptive Concept Bottleneck for Foundation Models Under Distribution Shifts

• Authors: Jihye Choi, Jayaram Raghuram, Yixuan Li, Somesh Jha
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14097
• Pdf URL: https://arxiv.org/pdf/2412.14097
• Copy Paste: [[2412.14097]] Adaptive Concept Bottleneck for Foundation Models Under Distribution Shifts(https://arxiv.org/abs/2412.14097)
• Keywords: security
• Abstract: Advancements in foundation models (FMs) have led to a paradigm shift in machine learning. The rich, expressive feature representations from these pre-trained, large-scale FMs are leveraged for multiple downstream tasks, usually via lightweight fine-tuning of a shallow fully-connected network following the representation. However, the non-interpretable, black-box nature of this prediction pipeline can be a challenge, especially in critical domains such as healthcare, finance, and security. In this paper, we explore the potential of Concept Bottleneck Models (CBMs) for transforming complex, non-interpretable foundation models into interpretable decision-making pipelines using high-level concept vectors. Specifically, we focus on the test-time deployment of such an interpretable CBM pipeline "in the wild", where the input distribution often shifts from the original training distribution. We first identify the potential failure modes of such a pipeline under different types of distribution shifts. Then we propose an adaptive concept bottleneck framework to address these failure modes, that dynamically adapts the concept-vector bank and the prediction layer based solely on unlabeled data from the target domain, without access to the source (training) dataset. Empirical evaluations with various real-world distribution shifts show that our adaptation method produces concept-based interpretations better aligned with the test data and boosts post-deployment accuracy by up to 28%, aligning the CBM performance with that of non-interpretable classification.

Title: Foundation Models Meet Low-Cost Sensors: Test-Time Adaptation for Rescaling Disparity for Zero-Shot Metric Depth Estimation

• Authors: Rémi Marsal, Alexandre Chapoutot, Philippe Xu, David Filliat
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14103
• Pdf URL: https://arxiv.org/pdf/2412.14103
• Copy Paste: [[2412.14103]] Foundation Models Meet Low-Cost Sensors: Test-Time Adaptation for Rescaling Disparity for Zero-Shot Metric Depth Estimation(https://arxiv.org/abs/2412.14103)
• Keywords: robust
• Abstract: The recent development of foundation models for monocular depth estimation such as Depth Anything paved the way to zero-shot monocular depth estimation. Since it returns an affine-invariant disparity map, the favored technique to recover the metric depth consists in fine-tuning the model. However, this stage is costly to perform because of the training but also due to the creation of the dataset. It must contain images captured by the camera that will be used at test time and the corresponding ground truth. Moreover, the fine-tuning may also degrade the generalizing capacity of the original model. Instead, we propose in this paper a new method to rescale Depth Anything predictions using 3D points provided by low-cost sensors or techniques such as low-resolution LiDAR, stereo camera, structure-from-motion where poses are given by an IMU. Thus, this approach avoids fine-tuning and preserves the generalizing power of the original depth estimation model while being robust to the noise of the sensor or of the depth model. Our experiments highlight improvements relative to other metric depth estimation methods and competitive results compared to fine-tuned approaches. Code available at this https URL.

Title: Adversarial Hubness in Multi-Modal Retrieval

• Authors: Tingwei Zhang, Fnu Suya, Rishi Jha, Collin Zhang, Vitaly Shmatikov
• Subjects: cs.CR, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.14113
• Pdf URL: https://arxiv.org/pdf/2412.14113
• Copy Paste: [[2412.14113]] Adversarial Hubness in Multi-Modal Retrieval(https://arxiv.org/abs/2412.14113)
• Keywords: defense, attack
• Abstract: Hubness is a phenomenon in high-dimensional vector spaces where a single point from the natural distribution is unusually close to many other points. This is a well-known problem in information retrieval that causes some items to accidentally (and incorrectly) appear relevant to many queries. In this paper, we investigate how attackers can exploit hubness to turn any image or audio input in a multi-modal retrieval system into an adversarial hub. Adversarial hubs can be used to inject universal adversarial content (e.g., spam) that will be retrieved in response to thousands of different queries, as well as for targeted attacks on queries related to specific, attacker-chosen concepts. We present a method for creating adversarial hubs and evaluate the resulting hubs on benchmark multi-modal retrieval datasets and an image-to-image retrieval system based on a tutorial from Pinecone, a popular vector database. For example, in text-caption-to-image retrieval, a single adversarial hub is retrieved as the top-1 most relevant image for more than 21,000 out of 25,000 test queries (by contrast, the most common natural hub is the top-1 response to only 102 queries). We also investigate whether techniques for mitigating natural hubness are an effective defense against adversarial hubs, and show that they are not effective against hubs that target queries related to specific concepts.

Title: Trustworthy Transfer Learning: A Survey

• Authors: Jun Wu, Jingrui He
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.14116
• Pdf URL: https://arxiv.org/pdf/2412.14116
• Copy Paste: [[2412.14116]] Trustworthy Transfer Learning: A Survey(https://arxiv.org/abs/2412.14116)
• Keywords: privacy, robust, fair
• Abstract: Transfer learning aims to transfer knowledge or information from a source domain to a relevant target domain. In this paper, we understand transfer learning from the perspectives of knowledge transferability and trustworthiness. This involves two research questions: How is knowledge transferability quantitatively measured and enhanced across domains? Can we trust the transferred knowledge in the transfer learning process? To answer these questions, this paper provides a comprehensive review of trustworthy transfer learning from various aspects, including problem definitions, theoretical analysis, empirical algorithms, and real-world applications. Specifically, we summarize recent theories and algorithms for understanding knowledge transferability under (within-domain) IID and non-IID assumptions. In addition to knowledge transferability, we review the impact of trustworthiness on transfer learning, e.g., whether the transferred knowledge is adversarially robust or algorithmically fair, how to transfer the knowledge under privacy-preserving constraints, etc. Beyond discussing the current advancements, we highlight the open questions and future directions for understanding transfer learning in a reliable and trustworthy manner.

Title: GaraMoSt: Parallel Multi-Granularity Motion and Structural Modeling for Efficient Multi-Frame Interpolation in DSA Images

• Authors: Ziyang Xu, Huangxuan Zhao, Wenyu Liu, Xinggang Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14118
• Pdf URL: https://arxiv.org/pdf/2412.14118
• Copy Paste: [[2412.14118]] GaraMoSt: Parallel Multi-Granularity Motion and Structural Modeling for Efficient Multi-Frame Interpolation in DSA Images(https://arxiv.org/abs/2412.14118)
• Keywords: robust
• Abstract: The rapid and accurate direct multi-frame interpolation method for Digital Subtraction Angiography (DSA) images is crucial for reducing radiation and providing real-time assistance to physicians for precise diagnostics and treatment. DSA images contain complex vascular structures and various motions. Applying natural scene Video Frame Interpolation (VFI) methods results in motion artifacts, structural dissipation, and blurriness. Recently, MoSt-DSA has specifically addressed these issues for the first time and achieved SOTA results. However, MoSt-DSA's focus on real-time performance leads to insufficient suppression of high-frequency noise and incomplete filtering of low-frequency noise in the generated images. To address these issues within the same computational time scale, we propose GaraMoSt. Specifically, we optimize the network pipeline with a parallel design and propose a module named MG-MSFE. MG-MSFE extracts frame-relative motion and structural features at various granularities in a fully convolutional parallel manner and supports independent, flexible adjustment of context-aware granularity at different scales, thus enhancing computational efficiency and accuracy. Extensive experiments demonstrate that GaraMoSt achieves the SOTA performance in accuracy, robustness, visual effects, and noise suppression, comprehensively surpassing MoSt-DSA and other natural scene VFI methods. The code and models are available at this https URL.

Title: AnySat: An Earth Observation Model for Any Resolutions, Scales, and Modalities

• Authors: Guillaume Astruc, Nicolas Gonthier, Clement Mallet, Loic Landrieu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14123
• Pdf URL: https://arxiv.org/pdf/2412.14123
• Copy Paste: [[2412.14123]] AnySat: An Earth Observation Model for Any Resolutions, Scales, and Modalities(https://arxiv.org/abs/2412.14123)
• Keywords: segmentation
• Abstract: Geospatial models must adapt to the diversity of Earth observation data in terms of resolutions, scales, and modalities. However, existing approaches expect fixed input configurations, which limits their practical applicability. We propose AnySat, a multimodal model based on joint embedding predictive architecture (JEPA) and resolution-adaptive spatial encoders, allowing us to train a single model on highly heterogeneous data in a self-supervised manner. To demonstrate the advantages of this unified approach, we compile GeoPlex, a collection of $5$ multimodal datasets with varying characteristics and $11$ distinct sensors. We then train a single powerful model on these diverse datasets simultaneously. Once fine-tuned, we achieve better or near state-of-the-art results on the datasets of GeoPlex and $4$ additional ones for $5$ environment monitoring tasks: land cover mapping, tree species identification, crop type classification, change detection, and flood segmentation. The code and models are available at this https URL.

Title: Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models

• Authors: Ido Cohen, Daniela Gottesman, Mor Geva, Raja Giryes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.14133
• Pdf URL: https://arxiv.org/pdf/2412.14133
• Copy Paste: [[2412.14133]] Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models(https://arxiv.org/abs/2412.14133)
• Keywords: extraction, interpretability
• Abstract: Vision-language models (VLMs) excel at extracting and reasoning about information from images. Yet, their capacity to leverage internal knowledge about specific entities remains underexplored. This work investigates the disparity in model performance when answering factual questions about an entity described in text versus depicted in an image. Our results reveal a significant accuracy drop --averaging 19%-- when the entity is presented visually instead of textually. We hypothesize that this decline arises from limitations in how information flows from image tokens to query tokens. We use mechanistic interpretability tools to reveal that, although image tokens are preprocessed by the vision encoder, meaningful information flow from these tokens occurs only in the much deeper layers. Furthermore, critical image processing happens in the language model's middle layers, allowing few layers for consecutive reasoning, highlighting a potential inefficiency in how the model utilizes its layers for reasoning. These insights shed light on the internal mechanics of VLMs and offer pathways for enhancing their reasoning capabilities.

Title: GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking

• Authors: Darshan Deshpande, Selvan Sunitha Ravi, Sky CH-Wang, Bartosz Mielczarek, Anand Kannappan, Rebecca Qian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.14140
• Pdf URL: https://arxiv.org/pdf/2412.14140
• Copy Paste: [[2412.14140]] GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking(https://arxiv.org/abs/2412.14140)
• Keywords: explainability
• Abstract: The LLM-as-judge paradigm is increasingly being adopted for automated evaluation of model outputs. While LLM judges have shown promise on constrained evaluation tasks, closed source LLMs display critical shortcomings when deployed in real world applications due to challenges of fine grained metrics and explainability, while task specific evaluation models lack cross-domain generalization. We introduce GLIDER, a powerful 3B evaluator LLM that can score any text input and associated context on arbitrary user defined criteria. GLIDER shows higher Pearson's correlation than GPT-4o on FLASK and greatly outperforms prior evaluation models, achieving comparable performance to LLMs 17x its size. GLIDER supports fine-grained scoring, multilingual reasoning, span highlighting and was trained on 685 domains and 183 criteria. Extensive qualitative analysis shows that GLIDER scores are highly correlated with human judgments, with 91.3% human agreement. We have open-sourced GLIDER to facilitate future research.

Title: On Calibration in Multi-Distribution Learning

• Authors: Rajeev Verma, Volker Fischer, Eric Nalisnick
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.14142
• Pdf URL: https://arxiv.org/pdf/2412.14142
• Copy Paste: [[2412.14142]] On Calibration in Multi-Distribution Learning(https://arxiv.org/abs/2412.14142)
• Keywords: robust, fair
• Abstract: Modern challenges of robustness, fairness, and decision-making in machine learning have led to the formulation of multi-distribution learning (MDL) frameworks in which a predictor is optimized across multiple distributions. We study the calibration properties of MDL to better understand how the predictor performs uniformly across the multiple distributions. Through classical results on decomposing proper scoring losses, we first derive the Bayes optimal rule for MDL, demonstrating that it maximizes the generalized entropy of the associated loss function. Our analysis reveals that while this approach ensures minimal worst-case loss, it can lead to non-uniform calibration errors across the multiple distributions and there is an inherent calibration-refinement trade-off, even at Bayes optimality. Our results highlight a critical limitation: despite the promise of MDL, one must use caution when designing predictors tailored to multiple distributions so as to minimize disparity.

Title: Incorporating Feature Pyramid Tokenization and Open Vocabulary Semantic Segmentation

• Authors: Jianyu Zhang, Li Zhang, Shijian Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14145
• Pdf URL: https://arxiv.org/pdf/2412.14145
• Copy Paste: [[2412.14145]] Incorporating Feature Pyramid Tokenization and Open Vocabulary Semantic Segmentation(https://arxiv.org/abs/2412.14145)
• Keywords: segmentation
• Abstract: The visual understanding are often approached from 3 granular levels: image, patch and pixel. Visual Tokenization, trained by self-supervised reconstructive learning, compresses visual data by codebook in patch-level with marginal information loss, but the visual tokens does not have semantic meaning. Open Vocabulary semantic segmentation benefits from the evolving Vision-Language models (VLMs) with strong image zero-shot capability, but transferring image-level to pixel-level understanding remains an imminent challenge. In this paper, we treat segmentation as tokenizing pixels and study a united perceptual and semantic token compression for all granular understanding and consequently facilitate open vocabulary semantic segmentation. Referring to the cognitive process of pretrained VLM where the low-level features are progressively composed to high-level semantics, we propose Feature Pyramid Tokenization (PAT) to cluster and represent multi-resolution feature by learnable codebooks and then decode them by joint learning pixel reconstruction and semantic segmentation. We design loosely coupled pixel and semantic learning branches. The pixel branch simulates bottom-up composition and top-down visualization of codebook tokens, while the semantic branch collectively fuse hierarchical codebooks as auxiliary segmentation guidance. Our experiments show that PAT enhances the semantic intuition of VLM feature pyramid, improves performance over the baseline segmentation model and achieves competitive performance on open vocabulary semantic segmentation benchmark. Our model is parameter-efficient for VLM integration and flexible for the independent tokenization. We hope to give inspiration not only on improving segmentation but also on semantic visual token utilization.

Title: MCMat: Multiview-Consistent and Physically Accurate PBR Material Generation

• Authors: Shenhao Zhu, Lingteng Qiu, Xiaodong Gu, Zhengyi Zhao, Chao Xu, Yuxiao He, Zhe Li, Xiaoguang Han, Yao Yao, Xun Cao, Siyu Zhu, Weihao Yuan, Zilong Dong, Hao Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14148
• Pdf URL: https://arxiv.org/pdf/2412.14148
• Copy Paste: [[2412.14148]] MCMat: Multiview-Consistent and Physically Accurate PBR Material Generation(https://arxiv.org/abs/2412.14148)
• Keywords: diffusion, transformer
• Abstract: Existing 2D methods utilize UNet-based diffusion models to generate multi-view physically-based rendering (PBR) maps but struggle with multi-view inconsistency, while some 3D methods directly generate UV maps, encountering generalization issues due to the limited 3D data. To address these problems, we propose a two-stage approach, including multi-view generation and UV materials refinement. In the generation stage, we adopt a Diffusion Transformer (DiT) model to generate PBR materials, where both the specially designed multi-branch DiT and reference-based DiT blocks adopt a global attention mechanism to promote feature interaction and fusion between different views, thereby improving multi-view consistency. In addition, we adopt a PBR-based diffusion loss to ensure that the generated materials align with realistic physical principles. In the refinement stage, we propose a material-refined DiT that performs inpainting in empty areas and enhances details in UV space. Except for the normal condition, this refinement also takes the material map from the generation stage as an additional condition to reduce the learning difficulty and improve generalization. Extensive experiments show that our method achieves state-of-the-art performance in texturing 3D objects with PBR materials and provides significant advantages for graphics relighting applications. Project Page: this https URL

Title: AKiRa: Augmentation Kit on Rays for optical video generation

• Authors: Xi Wang, Robin Courant, Marc Christie, Vicky Kalogeiton
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.14158
• Pdf URL: https://arxiv.org/pdf/2412.14158
• Copy Paste: [[2412.14158]] AKiRa: Augmentation Kit on Rays for optical video generation(https://arxiv.org/abs/2412.14158)
• Keywords: diffusion
• Abstract: Recent advances in text-conditioned video diffusion have greatly improved video quality. However, these methods offer limited or sometimes no control to users on camera aspects, including dynamic camera motion, zoom, distorted lens and focus shifts. These motion and optical aspects are crucial for adding controllability and cinematic elements to generation frameworks, ultimately resulting in visual content that draws focus, enhances mood, and guides emotions according to filmmakers' controls. In this paper, we aim to close the gap between controllable video generation and camera optics. To achieve this, we propose AKiRa (Augmentation Kit on Rays), a novel augmentation framework that builds and trains a camera adapter with a complex camera model over an existing video generation backbone. It enables fine-tuned control over camera motion as well as complex optical parameters (focal length, distortion, aperture) to achieve cinematic effects such as zoom, fisheye effect, and bokeh. Extensive experiments demonstrate AKiRa's effectiveness in combining and composing camera optics while outperforming all state-of-the-art methods. This work sets a new landmark in controlled and optically enhanced video generation, paving the way for future optical video generation methods.

Title: TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

• Authors: Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, Graham Neubig
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.14161
• Pdf URL: https://arxiv.org/pdf/2412.14161
• Copy Paste: [[2412.14161]] TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks(https://arxiv.org/abs/2412.14161)
• Keywords: large language model
• Abstract: We interact with computers on an everyday basis, be it in everyday life or work, and many aspects of work can be done entirely with access to a computer and the Internet. At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and affect change in their surrounding environments. But how performant are AI agents at helping to accelerate or even autonomously perform work-related tasks? The answer to this question has important implications for both industry looking to adopt AI into their workflows, and for economic policy to understand the effects that adoption of AI may have on the labor market. To measure the progress of these LLM agents' performance on performing real-world professional tasks, in this paper, we introduce TheAgentCompany, an extensible benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker: by browsing the Web, writing code, running programs, and communicating with other coworkers. We build a self-contained environment with internal web sites and data that mimics a small software company environment, and create a variety of tasks that may be performed by workers in such a company. We test baseline agents powered by both closed API-based and open-weights language models (LMs), and find that with the most competitive agent, 24% of the tasks can be completed autonomously. This paints a nuanced picture on task automation with LM agents -- in a setting simulating a real workplace, a good portion of simpler tasks could be solved autonomously, but more difficult long-horizon tasks are still beyond the reach of current systems.

Title: VideoDPO: Omni-Preference Alignment for Video Diffusion Generation

• Authors: Runtao Liu, Haoyu Wu, Zheng Ziqiang, Chen Wei, Yingqing He, Renjie Pi, Qifeng Chen
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.14167
• Pdf URL: https://arxiv.org/pdf/2412.14167
• Copy Paste: [[2412.14167]] VideoDPO: Omni-Preference Alignment for Video Diffusion Generation(https://arxiv.org/abs/2412.14167)
• Keywords: diffusion, generative
• Abstract: Recent progress in generative diffusion models has greatly advanced text-to-video generation. While text-to-video models trained on large-scale, diverse datasets can produce varied outputs, these generations often deviate from user preferences, highlighting the need for preference alignment on pre-trained models. Although Direct Preference Optimization (DPO) has demonstrated significant improvements in language and image generation, we pioneer its adaptation to video diffusion models and propose a VideoDPO pipeline by making several key adjustments. Unlike previous image alignment methods that focus solely on either (i) visual quality or (ii) semantic alignment between text and videos, we comprehensively consider both dimensions and construct a preference score accordingly, which we term the OmniScore. We design a pipeline to automatically collect preference pair data based on the proposed OmniScore and discover that re-weighting these pairs based on the score significantly impacts overall preference alignment. Our experiments demonstrate substantial improvements in both visual quality and semantic alignment, ensuring that no preference aspect is neglected. Code and data will be shared at this https URL.

Title: FashionComposer: Compositional Fashion Image Generation

• Authors: Sihui Ji, Yiyang Wang, Xi Chen, Xiaogang Xu, Hao Luo, Hengshuang Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14168
• Pdf URL: https://arxiv.org/pdf/2412.14168
• Copy Paste: [[2412.14168]] FashionComposer: Compositional Fashion Image Generation(https://arxiv.org/abs/2412.14168)
• Keywords: robust
• Abstract: We present FashionComposer for compositional fashion image generation. Unlike previous methods, FashionComposer is highly flexible. It takes multi-modal input (i.e., text prompt, parametric human model, garment image, and face image) and supports personalizing the appearance, pose, and figure of the human and assigning multiple garments in one pass. To achieve this, we first develop a universal framework capable of handling diverse input modalities. We construct scaled training data to enhance the model's robust compositional capabilities. To accommodate multiple reference images (garments and faces) seamlessly, we organize these references in a single image as an "asset library" and employ a reference UNet to extract appearance features. To inject the appearance features into the correct pixels in the generated result, we propose subject-binding attention. It binds the appearance features from different "assets" with the corresponding text features. In this way, the model could understand each asset according to their semantics, supporting arbitrary numbers and types of reference images. As a comprehensive solution, FashionComposer also supports many other applications like human album generation, diverse virtual try-on tasks, etc.

Title: Autoregressive Video Generation without Vector Quantization

• Authors: Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, Xinlong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14169
• Pdf URL: https://arxiv.org/pdf/2412.14169
• Copy Paste: [[2412.14169]] Autoregressive Video Generation without Vector Quantization(https://arxiv.org/abs/2412.14169)
• Keywords: diffusion
• Abstract: This paper presents a novel approach that enables autoregressive video generation with high efficiency. We propose to reformulate the video generation problem as a non-quantized autoregressive modeling of temporal frame-by-frame prediction and spatial set-by-set prediction. Unlike raster-scan prediction in prior autoregressive models or joint distribution modeling of fixed-length tokens in diffusion models, our approach maintains the causal property of GPT-style models for flexible in-context capabilities, while leveraging bidirectional modeling within individual frames for efficiency. With the proposed approach, we train a novel video autoregressive model without vector quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior autoregressive video models in data efficiency, inference speed, visual fidelity, and video fluency, even with a much smaller model capacity, i.e., 0.6B parameters. NOVA also outperforms state-of-the-art image diffusion models in text-to-image generation tasks, with a significantly lower training cost. Additionally, NOVA generalizes well across extended video durations and enables diverse zero-shot applications in one unified model. Code and models are publicly available at this https URL.

Title: E-CAR: Efficient Continuous Autoregressive Image Generation via Multistage Modeling

• Authors: Zhihang Yuan, Yuzhang Shang, Hanling Zhang, Tongcheng Fang, Rui Xie, Bingxin Xu, Yan Yan, Shengen Yan, Guohao Dai, Yu Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.14170
• Pdf URL: https://arxiv.org/pdf/2412.14170
• Copy Paste: [[2412.14170]] E-CAR: Efficient Continuous Autoregressive Image Generation via Multistage Modeling(https://arxiv.org/abs/2412.14170)
• Keywords: diffusion
• Abstract: Recent advances in autoregressive (AR) models with continuous tokens for image generation show promising results by eliminating the need for discrete tokenization. However, these models face efficiency challenges due to their sequential token generation nature and reliance on computationally intensive diffusion-based sampling. We present ECAR (Efficient Continuous Auto-Regressive Image Generation via Multistage Modeling), an approach that addresses these limitations through two intertwined innovations: (1) a stage-wise continuous token generation strategy that reduces computational complexity and provides progressively refined token maps as hierarchical conditions, and (2) a multistage flow-based distribution modeling method that transforms only partial-denoised distributions at each stage comparing to complete denoising in normal diffusion models. Holistically, ECAR operates by generating tokens at increasing resolutions while simultaneously denoising the image at each stage. This design not only reduces token-to-image transformation cost by a factor of the stage number but also enables parallel processing at the token level. Our approach not only enhances computational efficiency but also aligns naturally with image generation principles by operating in continuous token space and following a hierarchical generation process from coarse to fine details. Experimental results demonstrate that ECAR achieves comparable image quality to DiT Peebles & Xie [2023] while requiring 10$\times$ FLOPs reduction and 5$\times$ speedup to generate a 256$\times$256 image.

Title: Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces

• Authors: Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, Saining Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14171
• Pdf URL: https://arxiv.org/pdf/2412.14171
• Copy Paste: [[2412.14171]] Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces(https://arxiv.org/abs/2412.14171)
• Keywords: large language model
• Abstract: Humans possess the visual-spatial intelligence to remember spaces from sequential visual observations. However, can Multimodal Large Language Models (MLLMs) trained on million-scale video datasets also ``think in space'' from videos? We present a novel video-based visual-spatial intelligence benchmark (VSI-Bench) of over 5,000 question-answer pairs, and find that MLLMs exhibit competitive - though subhuman - visual-spatial intelligence. We probe models to express how they think in space both linguistically and visually and find that while spatial reasoning capabilities remain the primary bottleneck for MLLMs to reach higher benchmark performance, local world models and spatial awareness do emerge within these models. Notably, prevailing linguistic reasoning techniques (e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve performance, whereas explicitly generating cognitive maps during question-answering enhances MLLMs' spatial distance ability.

Title: AniDoc: Animation Creation Made Easier

• Authors: Yihao Meng, Hao Ouyang, Hanlin Wang, Qiuyu Wang, Wen Wang, Ka Leong Cheng, Zhiheng Liu, Yujun Shen, Huamin Qu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14173
• Pdf URL: https://arxiv.org/pdf/2412.14173
• Copy Paste: [[2412.14173]] AniDoc: Animation Creation Made Easier(https://arxiv.org/abs/2412.14173)
• Keywords: robust, diffusion, generative
• Abstract: The production of 2D animation follows an industry-standard workflow, encompassing four essential stages: character design, keyframe animation, in-betweening, and coloring. Our research focuses on reducing the labor costs in the above process by harnessing the potential of increasingly powerful generative AI. Using video diffusion models as the foundation, AniDoc emerges as a video line art colorization tool, which automatically converts sketch sequences into colored animations following the reference character specification. Our model exploits correspondence matching as an explicit guidance, yielding strong robustness to the variations (e.g., posture) between the reference character and each line art frame. In addition, our model could even automate the in-betweening process, such that users can easily create a temporally consistent animation by simply providing a character image as well as the start and end sketches. Our code is available at: this https URL.