2025-01-13

Title: LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Models

• Authors: Hang Yang, Hao Chen, Hui Guo, Yineng Chen, Ching-Sheng Lin, Shu Hu, Jinrong Hu, Xi Wu, Xin Wang
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.05464
• Pdf URL: https://arxiv.org/pdf/2501.05464
• Copy Paste: [[2501.05464]] LLM-MedQA: Enhancing Medical Question Answering through Case Studies in Large Language Models(https://arxiv.org/abs/2501.05464)
• Keywords: robust, interpretability, large language model
• Abstract: Accurate and efficient question-answering systems are essential for delivering high-quality patient care in the medical field. While Large Language Models (LLMs) have made remarkable strides across various domains, they continue to face significant challenges in medical question answering, particularly in understanding domain-specific terminologies and performing complex reasoning. These limitations undermine their effectiveness in critical medical applications. To address these issues, we propose a novel approach incorporating similar case generation within a multi-agent medical question-answering (MedQA) system. Specifically, we leverage the Llama3.1:70B model, a state-of-the-art LLM, in a multi-agent architecture to enhance performance on the MedQA dataset using zero-shot learning. Our method capitalizes on the model's inherent medical knowledge and reasoning capabilities, eliminating the need for additional training data. Experimental results show substantial performance gains over existing benchmark models, with improvements of 7% in both accuracy and F1-score across various medical QA tasks. Furthermore, we examine the model's interpretability and reliability in addressing complex medical queries. This research not only offers a robust solution for medical question answering but also establishes a foundation for broader applications of LLMs in the medical domain.

Title: LatteReview: A Multi-Agent Framework for Systematic Review Automation Using Large Language Models

• Authors: Pouria Rouzrokh, Moein Shariatnia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05468
• Pdf URL: https://arxiv.org/pdf/2501.05468
• Copy Paste: [[2501.05468]] LatteReview: A Multi-Agent Framework for Systematic Review Automation Using Large Language Models(https://arxiv.org/abs/2501.05468)
• Keywords: extraction, large language model
• Abstract: Systematic literature reviews and meta-analyses are essential for synthesizing research insights, but they remain time-intensive and labor-intensive due to the iterative processes of screening, evaluation, and data extraction. This paper introduces and evaluates LatteReview, a Python-based framework that leverages large language models (LLMs) and multi-agent systems to automate key elements of the systematic review process. Designed to streamline workflows while maintaining rigor, LatteReview utilizes modular agents for tasks such as title and abstract screening, relevance scoring, and structured data extraction. These agents operate within orchestrated workflows, supporting sequential and parallel review rounds, dynamic decision-making, and iterative refinement based on user feedback. LatteReview's architecture integrates LLM providers, enabling compatibility with both cloud-based and locally hosted models. The framework supports features such as Retrieval-Augmented Generation (RAG) for incorporating external context, multimodal reviews, Pydantic-based validation for structured inputs and outputs, and asynchronous programming for handling large-scale datasets. The framework is available on the GitHub repository, with detailed documentation and an installable package.

Title: Found in Translation: semantic approaches for enhancing AI interpretability in face verification

• Authors: Miriam Doh (UMONS, ULB), Caroline Mazini Rodrigues (LRDE, LIGM), N. Boutry (LRDE), L. Najman (LIGM), Matei Mancas (UMONS), Bernard Gosselin (UMONS)
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05471
• Pdf URL: https://arxiv.org/pdf/2501.05471
• Copy Paste: [[2501.05471]] Found in Translation: semantic approaches for enhancing AI interpretability in face verification(https://arxiv.org/abs/2501.05471)
• Keywords: interpretability, large language model
• Abstract: The increasing complexity of machine learning models in computer vision, particularly in face verification, requires the development of explainable artificial intelligence (XAI) to enhance interpretability and transparency. This study extends previous work by integrating semantic concepts derived from human cognitive processes into XAI frameworks to bridge the comprehension gap between model outputs and human understanding. We propose a novel approach combining global and local explanations, using semantic features defined by user-selected facial landmarks to generate similarity maps and textual explanations via large language models (LLMs). The methodology was validated through quantitative experiments and user feedback, demonstrating improved interpretability. Results indicate that our semantic-based approach, particularly the most detailed set, offers a more nuanced understanding of model decisions than traditional methods. User studies highlight a preference for our semantic explanations over traditional pixelbased heatmaps, emphasizing the benefits of human-centric interpretability in AI. This work contributes to the ongoing efforts to create XAI frameworks that align AI models behaviour with human cognitive processes, fostering trust and acceptance in critical applications.

Title: The 2nd Place Solution from the 3D Semantic Segmentation Track in the 2024 Waymo Open Dataset Challenge

• Authors: Qing Wu
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.05472
• Pdf URL: https://arxiv.org/pdf/2501.05472
• Copy Paste: [[2501.05472]] The 2nd Place Solution from the 3D Semantic Segmentation Track in the 2024 Waymo Open Dataset Challenge(https://arxiv.org/abs/2501.05472)
• Keywords: segmentation
• Abstract: 3D semantic segmentation is one of the most crucial tasks in driving perception. The ability of a learning-based model to accurately perceive dense 3D surroundings often ensures the safe operation of autonomous vehicles. However, existing LiDAR-based 3D semantic segmentation databases consist of sequentially acquired LiDAR scans that are long-tailed and lack training diversity. In this report, we introduce MixSeg3D, a sophisticated combination of the strong point cloud segmentation model with advanced 3D data mixing strategies. Specifically, our approach integrates the MinkUNet family with LaserMix and PolarMix, two scene-scale data augmentation methods that blend LiDAR point clouds along the ego-scene's inclination and azimuth directions. Through empirical experiments, we demonstrate the superiority of MixSeg3D over the baseline and prior arts. Our team achieved 2nd place in the 3D semantic segmentation track of the 2024 Waymo Open Dataset Challenge.

Title: Implicit Guidance and Explicit Representation of Semantic Information in Points Cloud: A Survey

• Authors: Jingyuan Tang, Yuhuan Zhao, Songlin Sun, Yangang Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05473
• Pdf URL: https://arxiv.org/pdf/2501.05473
• Copy Paste: [[2501.05473]] Implicit Guidance and Explicit Representation of Semantic Information in Points Cloud: A Survey(https://arxiv.org/abs/2501.05473)
• Keywords: extraction
• Abstract: Point clouds, a prominent method of 3D representation, are extensively utilized across industries such as autonomous driving, surveying, electricity, architecture, and gaming, and have been rigorously investigated for their accuracy and resilience. The extraction of semantic information from scenes enhances both human understanding and machine perception. By integrating semantic information from two-dimensional scenes with three-dimensional point clouds, researchers aim to improve the precision and efficiency of various tasks. This paper provides a comprehensive review of the diverse applications and recent advancements in the integration of semantic information within point clouds. We explore the dual roles of semantic information in point clouds, encompassing both implicit guidance and explicit representation, across traditional and emerging tasks. Additionally, we offer a comparative analysis of publicly available datasets tailored to specific tasks and present notable observations. In conclusion, we discuss several challenges and potential issues that may arise in the future when fully utilizing semantic information in point clouds, providing our perspectives on these obstacles. The classified and organized articles related to semantic based point cloud tasks, and continuously followed up on relevant achievements in different fields, which can be accessed through this https URL.

Title: Modality-Invariant Bidirectional Temporal Representation Distillation Network for Missing Multimodal Sentiment Analysis

• Authors: Xincheng Wang, Liejun Wang, Yinfeng Yu, Xinxin Jiao
• Subjects: cs.CL, cs.AI, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.05474
• Pdf URL: https://arxiv.org/pdf/2501.05474
• Copy Paste: [[2501.05474]] Modality-Invariant Bidirectional Temporal Representation Distillation Network for Missing Multimodal Sentiment Analysis(https://arxiv.org/abs/2501.05474)
• Keywords: robust
• Abstract: Multimodal Sentiment Analysis (MSA) integrates diverse modalities(text, audio, and video) to comprehensively analyze and understand individuals' emotional states. However, the real-world prevalence of incomplete data poses significant challenges to MSA, mainly due to the randomness of modality missing. Moreover, the heterogeneity issue in multimodal data has yet to be effectively addressed. To tackle these challenges, we introduce the Modality-Invariant Bidirectional Temporal Representation Distillation Network (MITR-DNet) for Missing Multimodal Sentiment Analysis. MITR-DNet employs a distillation approach, wherein a complete modality teacher model guides a missing modality student model, ensuring robustness in the presence of modality missing. Simultaneously, we developed the Modality-Invariant Bidirectional Temporal Representation Learning Module (MIB-TRL) to mitigate heterogeneity.

Title: IntegrityAI at GenAI Detection Task 2: Detecting Machine-Generated Academic Essays in English and Arabic Using ELECTRA and Stylometry

• Authors: Mohammad AL-Smadi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05476
• Pdf URL: https://arxiv.org/pdf/2501.05476
• Copy Paste: [[2501.05476]] IntegrityAI at GenAI Detection Task 2: Detecting Machine-Generated Academic Essays in English and Arabic Using ELECTRA and Stylometry(https://arxiv.org/abs/2501.05476)
• Keywords: transformer
• Abstract: Recent research has investigated the problem of detecting machine-generated essays for academic purposes. To address this challenge, this research utilizes pre-trained, transformer-based models fine-tuned on Arabic and English academic essays with stylometric features. Custom models based on ELECTRA for English and AraELECTRA for Arabic were trained and evaluated using a benchmark dataset. Proposed models achieved excellent results with an F1-score of 99.7%, ranking 2nd among of 26 teams in the English subtask, and 98.4%, finishing 1st out of 23 teams in the Arabic one.

Title: Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Models

• Authors: Malak Mansour, Ahmed Aly, Bahey Tharwat, Sarim Hashmi, Dong An, Ian Reid
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.05478
• Pdf URL: https://arxiv.org/pdf/2501.05478
• Copy Paste: [[2501.05478]] Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Models(https://arxiv.org/abs/2501.05478)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) such as GPT-4, trained on huge amount of datasets spanning multiple domains, exhibit significant reasoning, understanding, and planning capabilities across various tasks. This study presents the first-ever work in Arabic language integration within the Vision-and-Language Navigation (VLN) domain in robotics, an area that has been notably underexplored in existing research. We perform a comprehensive evaluation of state-of-the-art multi-lingual Small Language Models (SLMs), including GPT-4o mini, Llama 3 8B, and Phi-3 medium 14B, alongside the Arabic-centric LLM, Jais. Our approach utilizes the NavGPT framework, a pure LLM-based instruction-following navigation agent, to assess the impact of language on navigation reasoning through zero-shot sequential action prediction using the R2R dataset. Through comprehensive experiments, we demonstrate that our framework is capable of high-level planning for navigation tasks when provided with instructions in both English and Arabic. However, certain models struggled with reasoning and planning in the Arabic language due to inherent limitations in their capabilities, sub-optimal performance, and parsing issues. These findings highlight the importance of enhancing planning and reasoning capabilities in language models for effective navigation, emphasizing this as a key area for further development while also unlocking the potential of Arabic-language models for impactful real-world applications.

Title: Practical Design and Benchmarking of Generative AI Applications for Surgical Billing and Coding

• Authors: John C. Rollman (1), Bruce Rogers (1), Hamed Zaribafzadeh (1), Daniel Buckland (2), Ursula Rogers (1), Jennifer Gagnon (1), Ozanan Meireles (1), Lindsay Jennings (3), Jim Bennett (1), Jennifer Nicholson (3), Nandan Lad (4), Linda Cendales (1), Andreas Seas (4,5,6), Alessandro Martinino (6), E. Shelley Hwang (1), Allan D. Kirk (1)
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05479
• Pdf URL: https://arxiv.org/pdf/2501.05479
• Copy Paste: [[2501.05479]] Practical Design and Benchmarking of Generative AI Applications for Surgical Billing and Coding(https://arxiv.org/abs/2501.05479)
• Keywords: security, privacy, generative, large language model
• Abstract: Background: Healthcare has many manual processes that can benefit from automation and augmentation with Generative Artificial Intelligence (AI), the medical billing and coding process. However, current foundational Large Language Models (LLMs) perform poorly when tasked with generating accurate International Classification of Diseases, 10th edition, Clinical Modification (ICD-10-CM) and Current Procedural Terminology (CPT) codes. Additionally, there are many security and financial challenges in the application of generative AI to healthcare. We present a strategy for developing generative AI tools in healthcare, specifically for medical billing and coding, that balances accuracy, accessibility, and patient privacy. Methods: We fine tune the PHI-3 Mini and PHI-3 Medium LLMs using institutional data and compare the results against the PHI-3 base model, a PHI-3 RAG application, and GPT-4o. We use the post operative surgical report as input and the patients billing claim the associated ICD-10, CPT, and Modifier codes as the target result. Performance is measured by accuracy of code generation, proportion of invalid codes, and the fidelity of the billing claim format. Results: Both fine-tuned models performed better or as well as GPT-4o. The Phi-3 Medium fine-tuned model showed the best performance (ICD-10 Recall and Precision: 72%, 72%; CPT Recall and Precision: 77%, 79%; Modifier Recall and Precision: 63%, 64%). The Phi-3 Medium fine-tuned model only fabricated 1% of ICD-10 codes and 0.6% of CPT codes generated. Conclusions: Our study shows that a small model that is fine-tuned on domain-specific data for specific tasks using a simple set of open-source tools and minimal technological and monetary requirements performs as well as the larger contemporary consumer models.

Title: HP-BERT: A framework for longitudinal study of Hinduphobia on social media via LLMs

• Authors: Ashutosh Singh, Rohitash Chandra
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2501.05482
• Pdf URL: https://arxiv.org/pdf/2501.05482
• Copy Paste: [[2501.05482]] HP-BERT: A framework for longitudinal study of Hinduphobia on social media via LLMs(https://arxiv.org/abs/2501.05482)
• Keywords: large language model
• Abstract: During the COVID-19 pandemic, community tensions intensified, fuelling Hinduphobic sentiments and discrimination against individuals of Hindu descent within India and worldwide. Large language models (LLMs) have become prominent in natural language processing (NLP) tasks and social media analysis, enabling longitudinal studies of platforms like X (formerly Twitter) for specific issues during COVID-19. We present an abuse detection and sentiment analysis framework that offers a longitudinal analysis of Hinduphobia on X (Twitter) during and after the COVID-19 pandemic. This framework assesses the prevalence and intensity of Hinduphobic discourse, capturing elements such as derogatory jokes and racist remarks through sentiment analysis and abuse detection from pre-trained and fine-tuned LLMs. Additionally, we curate and publish a "Hinduphobic COVID-19 X (Twitter) Dataset" of 8,000 tweets annotated for Hinduphobic abuse detection, which is used to fine-tune a BERT model, resulting in the development of the Hinduphobic BERT (HP-BERT) model. We then further fine-tune HP-BERT using the SenWave dataset for multi-label sentiment analysis. Our study encompasses approximately 27.4 million tweets from six countries, including Australia, Brazil, India, Indonesia, Japan, and the United Kingdom. Our findings reveal a strong correlation between spikes in COVID-19 cases and surges in Hinduphobic rhetoric, highlighting how political narratives, misinformation, and targeted jokes contributed to communal polarisation. These insights provide valuable guidance for developing strategies to mitigate communal tensions in future crises, both locally and globally. We advocate implementing automated monitoring and removal of such content on social media to curb divisive discourse.

Title: Tuning-Free Long Video Generation via Global-Local Collaborative Diffusion

• Authors: Yongjia Ma, Junlin Chen, Donglin Di, Qi Xie, Lei Fan, Wei Chen, Xiaofei Gou, Na Zhao, Xun Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05484
• Pdf URL: https://arxiv.org/pdf/2501.05484
• Copy Paste: [[2501.05484]] Tuning-Free Long Video Generation via Global-Local Collaborative Diffusion(https://arxiv.org/abs/2501.05484)
• Keywords: diffusion
• Abstract: Creating high-fidelity, coherent long videos is a sought-after aspiration. While recent video diffusion models have shown promising potential, they still grapple with spatiotemporal inconsistencies and high computational resource demands. We propose GLC-Diffusion, a tuning-free method for long video generation. It models the long video denoising process by establishing denoising trajectories through Global-Local Collaborative Denoising to ensure overall content consistency and temporal coherence between frames. Additionally, we introduce a Noise Reinitialization strategy which combines local noise shuffling with frequency fusion to improve global content consistency and visual diversity. Further, we propose a Video Motion Consistency Refinement (VMCR) module that computes the gradient of pixel-wise and frequency-wise losses to enhance visual consistency and temporal smoothness. Extensive experiments, including quantitative and qualitative evaluations on videos of varying lengths (\textit{e.g.}, 3\times and 6\times longer), demonstrate that our method effectively integrates with existing video diffusion models, producing coherent, high-fidelity long videos superior to previous approaches.

Title: S2 Chunking: A Hybrid Framework for Document Segmentation Through Integrated Spatial and Semantic Analysis

• Authors: Prashant Verma
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05485
• Pdf URL: https://arxiv.org/pdf/2501.05485
• Copy Paste: [[2501.05485]] S2 Chunking: A Hybrid Framework for Document Segmentation Through Integrated Spatial and Semantic Analysis(https://arxiv.org/abs/2501.05485)
• Keywords: segmentation
• Abstract: Document chunking is a critical task in natural language processing (NLP) that involves dividing a document into meaningful segments. Traditional methods often rely solely on semantic analysis, ignoring the spatial layout of elements, which is crucial for understanding relationships in complex documents. This paper introduces a novel hybrid approach that combines layout structure, semantic analysis, and spatial relationships to enhance the cohesion and accuracy of document chunks. By leveraging bounding box information (bbox) and text embeddings, our method constructs a weighted graph representation of document elements, which is then clustered using spectral clustering. Experimental results demonstrate that this approach outperforms traditional methods, particularly in documents with diverse layouts such as reports, articles, and multi-column designs. The proposed method also ensures that no chunk exceeds a specified token length, making it suitable for use cases where token limits are critical (e.g., language models with input size limitations)

Title: The Future of AI: Exploring the Potential of Large Concept Models

• Authors: Hussain Ahmad, Diksha Goel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05487
• Pdf URL: https://arxiv.org/pdf/2501.05487
• Copy Paste: [[2501.05487]] The Future of AI: Exploring the Potential of Large Concept Models(https://arxiv.org/abs/2501.05487)
• Keywords: generative, large language model
• Abstract: The field of Artificial Intelligence (AI) continues to drive transformative innovations, with significant progress in conversational interfaces, autonomous vehicles, and intelligent content creation. Since the launch of ChatGPT in late 2022, the rise of Generative AI has marked a pivotal era, with the term Large Language Models (LLMs) becoming a ubiquitous part of daily life. LLMs have demonstrated exceptional capabilities in tasks such as text summarization, code generation, and creative writing. However, these models are inherently limited by their token-level processing, which restricts their ability to perform abstract reasoning, conceptual understanding, and efficient generation of long-form content. To address these limitations, Meta has introduced Large Concept Models (LCMs), representing a significant shift from traditional token-based frameworks. LCMs use concepts as foundational units of understanding, enabling more sophisticated semantic reasoning and context-aware decision-making. Given the limited academic research on this emerging technology, our study aims to bridge the knowledge gap by collecting, analyzing, and synthesizing existing grey literature to provide a comprehensive understanding of LCMs. Specifically, we (i) identify and describe the features that distinguish LCMs from LLMs, (ii) explore potential applications of LCMs across multiple domains, and (iii) propose future research directions and practical strategies to advance LCM development and adoption.

Title: Mathematical Modeling and Machine Learning for Predicting Shade-Seeking Behavior in Cows Under Heat Stress

• Authors: S. Sanjuan, D. A. Méndez, R. Arnau, J. M. Calabuig, X. Díaz de Otálora Aguirre, F. Estellés
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05494
• Pdf URL: https://arxiv.org/pdf/2501.05494
• Copy Paste: [[2501.05494]] Mathematical Modeling and Machine Learning for Predicting Shade-Seeking Behavior in Cows Under Heat Stress(https://arxiv.org/abs/2501.05494)
• Keywords: robust, interpretability, explainability
• Abstract: In this paper we develop a mathematical model combined with machine learning techniques to predict shade-seeking behavior in cows exposed to heat stress. The approach integrates advanced mathematical features, such as time-averaged thermal indices and accumulated heat stress metrics, obtained by mathematical analysis of data from a farm in Titaguas (Valencia, Spain), collected during the summer of 2023. Two predictive models, Random Forests and Neural Networks, are compared for accuracy, robustness, and interpretability. The Random Forest model is highlighted for its balance between precision and explainability, achieving an RMSE of $14.97$. The methodology also employs $5-$fold cross-validation to ensure robustness under real-world conditions. This work not only advances the mathematical modeling of animal behavior but also provides useful insights for mitigating heat stress in livestock through data-driven tools.

Title: FedSA: A Unified Representation Learning via Semantic Anchors for Prototype-based Federated Learning

• Authors: Yanbing Zhou, Xiangmou Qu, Chenlong You, Jiyang Zhou, Jingyue Tang, Xin Zheng, Chunmao Cai, Yingbo Wu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05496
• Pdf URL: https://arxiv.org/pdf/2501.05496
• Copy Paste: [[2501.05496]] FedSA: A Unified Representation Learning via Semantic Anchors for Prototype-based Federated Learning(https://arxiv.org/abs/2501.05496)
• Keywords: robust, federate
• Abstract: Prototype-based federated learning has emerged as a promising approach that shares lightweight prototypes to transfer knowledge among clients with data heterogeneity in a model-agnostic manner. However, existing methods often collect prototypes directly from local models, which inevitably introduce inconsistencies into representation learning due to the biased data distributions and differing model architectures among clients. In this paper, we identify that both statistical and model heterogeneity create a vicious cycle of representation inconsistency, classifier divergence, and skewed prototype alignment, which negatively impacts the performance of clients. To break the vicious cycle, we propose a novel framework named Federated Learning via Semantic Anchors (FedSA) to decouple the generation of prototypes from local representation learning. We introduce a novel perspective that uses simple yet effective semantic anchors serving as prototypes to guide local models in learning consistent representations. By incorporating semantic anchors, we further propose anchor-based regularization with margin-enhanced contrastive learning and anchor-based classifier calibration to correct feature extractors and calibrate classifiers across clients, achieving intra-class compactness and inter-class separability of prototypes while ensuring consistent decision boundaries. We then update the semantic anchors with these consistent and discriminative prototypes, which iteratively encourage clients to collaboratively learn a unified data representation with robust generalization. Extensive experiments under both statistical and model heterogeneity settings show that FedSA significantly outperforms existing prototype-based FL methods on various classification tasks.

Title: Spatial Information Integration in Small Language Models for Document Layout Generation and Classification

• Authors: Pablo Melendez, Clemens Havas
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.05497
• Pdf URL: https://arxiv.org/pdf/2501.05497
• Copy Paste: [[2501.05497]] Spatial Information Integration in Small Language Models for Document Layout Generation and Classification(https://arxiv.org/abs/2501.05497)
• Keywords: transformer
• Abstract: Document layout understanding is a field of study that analyzes the spatial arrangement of information in a document hoping to understand its structure and layout. Models such as LayoutLM (and its subsequent iterations) can understand semi-structured documents with SotA results; however, the lack of open semi-structured data is a limitation in itself. While semi-structured data is common in everyday life (balance sheets, purchase orders, receipts), there is a lack of public datasets for training machine learning models for this type of document. In this investigation we propose a method to generate new, synthetic, layout information that can help overcoming this data shortage. According to our results, the proposed method performs better than LayoutTransformer, another popular layout generation method. We also show that, in some scenarios, text classification can improve when supported by bounding box information.

Title: Generative Flow Networks: Theory and Applications to Structure Learning

• Authors: Tristan Deleu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05498
• Pdf URL: https://arxiv.org/pdf/2501.05498
• Copy Paste: [[2501.05498]] Generative Flow Networks: Theory and Applications to Structure Learning(https://arxiv.org/abs/2501.05498)
• Keywords: generative
• Abstract: Without any assumptions about data generation, multiple causal models may explain our observations equally well. To avoid selecting a single arbitrary model that could result in unsafe decisions if it does not match reality, it is therefore essential to maintain a notion of epistemic uncertainty about our possible candidates. This thesis studies the problem of structure learning from a Bayesian perspective, approximating the posterior distribution over the structure of a causal model, represented as a directed acyclic graph (DAG), given data. It introduces Generative Flow Networks (GFlowNets), a novel class of probabilistic models designed for modeling distributions over discrete and compositional objects such as graphs. They treat generation as a sequential decision making problem, constructing samples of a target distribution defined up to a normalization constant piece by piece. In the first part of this thesis, we present the mathematical foundations of GFlowNets, their connections to existing domains of machine learning and statistics such as variational inference and reinforcement learning, and their extensions beyond discrete problems. In the second part of this thesis, we show how GFlowNets can approximate the posterior distribution over DAG structures of causal Bayesian Networks, along with the parameters of its causal mechanisms, given observational and experimental data.

Title: Shrink the longest: improving latent space isotropy with symplicial geometry

• Authors: Sergei Kudriashov, Olesya Karpik, Eduard Klyshinsky
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05502
• Pdf URL: https://arxiv.org/pdf/2501.05502
• Copy Paste: [[2501.05502]] Shrink the longest: improving latent space isotropy with symplicial geometry(https://arxiv.org/abs/2501.05502)
• Keywords: transformer
• Abstract: Although transformer-based models have been dominating the field of deep learning, various studies of their embedding space have shown that they suffer from "representation degeneration problem": embeddings tend to be distributed in a narrow cone, making the latent space highly anisotropic. Increasing the isotropy has shown to improve performance in downstream tasks both in static and contextual language models. However, most of approaches either add inference overhead or require substantial amount of data for model reparametrization. We propose a novel regularization technique based on simplicial geometry to improve the isotropy of latent representations. The core idea of our method is based on maximizing the persistent entropy of barcodes obtained using Vietoris-Rips filtration from contextual embeddings in the underlying latent space. We demonstrate that the method leads to an increase in downstream performance while significantly lowering the anisotropy during fine-tuning by exploiting existing geometric structures instead of reparametrization.

Title: On Fair Ordering and Differential Privacy

• Authors: Shir Cohen, Neel Basu, Soumya Basu, Lorenzo Alvisi
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.05535
• Pdf URL: https://arxiv.org/pdf/2501.05535
• Copy Paste: [[2501.05535]] On Fair Ordering and Differential Privacy(https://arxiv.org/abs/2501.05535)
• Keywords: privacy, fair
• Abstract: In blockchain systems, fair transaction ordering is crucial for a trusted and regulation-compliant economic ecosystem. Unlike traditional State Machine Replication (SMR) systems, which focus solely on liveness and safety, blockchain systems also require a fairness property. This paper examines these properties and aims to eliminate algorithmic bias in transaction ordering services. We build on the notion of equal opportunity. We characterize transactions in terms of relevant and irrelevant features, requiring that the order be determined solely by the relevant ones. Specifically, transactions with identical relevant features should have an equal chance of being ordered before one another. We extend this framework to define a property where the greater the distance in relevant features between transactions, the higher the probability of prioritizing one over the other. We reveal a surprising link between equal opportunity in SMR and Differential Privacy (DP), showing that any DP mechanism can be used to ensure fairness in SMR. This connection not only enhances our understanding of the interplay between privacy and fairness in distributed computing but also opens up new opportunities for designing fair distributed protocols using well-established DP techniques.

Title: NSChat: A Chatbot System To Rule Them All

• Authors: Zenon Lamprou, Yashar Moshfeghi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05541
• Pdf URL: https://arxiv.org/pdf/2501.05541
• Copy Paste: [[2501.05541]] NSChat: A Chatbot System To Rule Them All(https://arxiv.org/abs/2501.05541)
• Keywords: large language model
• Abstract: The rapid advancement of artificial intelligence has resulted in the advent of large language models (LLMs) with the capacity to produce text that closely resembles human communication. These models have been seamlessly integrated into diverse applications, enabling interactive and responsive communication across multiple platforms. The potential utility of chatbots transcends these traditional applications, particularly in research contexts, wherein they can offer valuable insights and facilitate the design of innovative experiments. In this study, we present NSChat, a web-based chatbot system designed to assist in neuroscience research. The system is meticulously designed to function as an experimental instrument rather than a conventional chatbot, necessitating users to input a username and experiment code upon access. This setup facilitates precise data cross-referencing, thereby augmenting the integrity and applicability of the data collected for research purposes. It can be easily expanded to accommodate new basic events as needed; and it allows researchers to integrate their own logging events without the necessity of implementing a separate logging mechanism. It is worth noting that our system was built to assist primarily neuroscience research but is not limited to it, it can easily be adapted to assist information retrieval research or interacting with chat bot agents in general.

Title: Infecting Generative AI With Viruses

• Authors: David Noever, Forrest McKee
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.05542
• Pdf URL: https://arxiv.org/pdf/2501.05542
• Copy Paste: [[2501.05542]] Infecting Generative AI With Viruses(https://arxiv.org/abs/2501.05542)
• Keywords: security, extraction, generative, large language model
• Abstract: This study demonstrates a novel approach to testing the security boundaries of Vision-Large Language Model (VLM/ LLM) using the EICAR test file embedded within JPEG images. We successfully executed four distinct protocols across multiple LLM platforms, including OpenAI GPT-4o, Microsoft Copilot, Google Gemini 1.5 Pro, and Anthropic Claude 3.5 Sonnet. The experiments validated that a modified JPEG containing the EICAR signature could be uploaded, manipulated, and potentially executed within LLM virtual workspaces. Key findings include: 1) consistent ability to mask the EICAR string in image metadata without detection, 2) successful extraction of the test file using Python-based manipulation within LLM environments, and 3) demonstration of multiple obfuscation techniques including base64 encoding and string reversal. This research extends Microsoft Research's "Penetration Testing Rules of Engagement" framework to evaluate cloud-based generative AI and LLM security boundaries, particularly focusing on file handling and execution capabilities within containerized environments.

Title: Emergent weight morphologies in deep neural networks

• Authors: Pascal de Jong, Felix Meigel, Steffen Rulands
• Subjects: cs.LG, cond-mat.dis-nn
• Abstract URL: https://arxiv.org/abs/2501.05550
• Pdf URL: https://arxiv.org/pdf/2501.05550
• Copy Paste: [[2501.05550]] Emergent weight morphologies in deep neural networks(https://arxiv.org/abs/2501.05550)
• Keywords: security
• Abstract: Whether deep neural networks can exhibit emergent behaviour is not only relevant for understanding how deep learning works, it is also pivotal for estimating potential security risks of increasingly capable artificial intelligence systems. Here, we show that training deep neural networks gives rise to emergent weight morphologies independent of the training data. Specifically, in analogy to condensed matter physics, we derive a theory that predict that the homogeneous state of deep neural networks is unstable in a way that leads to the emergence of periodic channel structures. We verified these structures by performing numerical experiments on a variety of data sets. Our work demonstrates emergence in the training of deep neural networks, which impacts the achievable performance of deep neural networks.

Title: The dynamics of meaning through time: Assessment of Large Language Models

• Authors: Mohamed Taher Alrefaie, Fatty Salem, Nour Eldin Morsy, Nada Samir, Mohamed Medhat Gaber
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05552
• Pdf URL: https://arxiv.org/pdf/2501.05552
• Copy Paste: [[2501.05552]] The dynamics of meaning through time: Assessment of Large Language Models(https://arxiv.org/abs/2501.05552)
• Keywords: large language model
• Abstract: Understanding how large language models (LLMs) grasp the historical context of concepts and their semantic evolution is essential in advancing artificial intelligence and linguistic studies. This study aims to evaluate the capabilities of various LLMs in capturing temporal dynamics of meaning, specifically how they interpret terms across different time periods. We analyze a diverse set of terms from multiple domains, using tailored prompts and measuring responses through both objective metrics (e.g., perplexity and word count) and subjective human expert evaluations. Our comparative analysis includes prominent models like ChatGPT, GPT-4, Claude, Bard, Gemini, and Llama. Findings reveal marked differences in each model's handling of historical context and semantic shifts, highlighting both strengths and limitations in temporal semantic understanding. These insights offer a foundation for refining LLMs to better address the evolving nature of language, with implications for historical text analysis, AI design, and applications in digital humanities.

Title: LLMQuoter: Enhancing RAG Capabilities Through Efficient Quote Extraction From Large Contexts

• Authors: Yuri Facanha Bezerra, Li Weigang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05554
• Pdf URL: https://arxiv.org/pdf/2501.05554
• Copy Paste: [[2501.05554]] LLMQuoter: Enhancing RAG Capabilities Through Efficient Quote Extraction From Large Contexts(https://arxiv.org/abs/2501.05554)
• Keywords: extraction, large language model
• Abstract: We introduce LLMQuoter, a lightweight, distillation-based model designed to enhance Retrieval Augmented Generation (RAG) by extracting the most relevant textual evidence for downstream reasoning tasks. Built on the LLaMA-3B architecture and fine-tuned with Low-Rank Adaptation (LoRA) on a 15,000-sample subset of HotpotQA, LLMQuoter adopts a "quote-first-then-answer" strategy, efficiently identifying key quotes before passing curated snippets to reasoning models. This workflow reduces cognitive overhead and outperforms full-context approaches like Retrieval-Augmented Fine-Tuning (RAFT), achieving over 20-point accuracy gains across both small and large language models. By leveraging knowledge distillation from a high-performing teacher model, LLMQuoter achieves competitive results in a resource-efficient fine-tuning setup. It democratizes advanced RAG capabilities, delivering significant performance improvements without requiring extensive model retraining. Our results highlight the potential of distilled quote-based reasoning to streamline complex workflows, offering a scalable and practical solution for researchers and practitioners alike.

Title: Vision-Language Models for Autonomous Driving: CLIP-Based Dynamic Scene Understanding

• Authors: Mohammed Elhenawy, Huthaifa I. Ashqar, Andry Rakotonirainy, Taqwa I. Alhadidi, Ahmed Jaber, Mohammad Abu Tami
• Subjects: cs.CV, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.05566
• Pdf URL: https://arxiv.org/pdf/2501.05566
• Copy Paste: [[2501.05566]] Vision-Language Models for Autonomous Driving: CLIP-Based Dynamic Scene Understanding(https://arxiv.org/abs/2501.05566)
• Keywords: robust
• Abstract: Scene understanding is essential for enhancing driver safety, generating human-centric explanations for Automated Vehicle (AV) decisions, and leveraging Artificial Intelligence (AI) for retrospective driving video analysis. This study developed a dynamic scene retrieval system using Contrastive Language-Image Pretraining (CLIP) models, which can be optimized for real-time deployment on edge devices. The proposed system outperforms state-of-the-art in-context learning methods, including the zero-shot capabilities of GPT-4o, particularly in complex scenarios. By conducting frame-level analysis on the Honda Scenes Dataset, which contains a collection of about 80 hours of annotated driving videos capturing diverse real-world road and weather conditions, our study highlights the robustness of CLIP models in learning visual concepts from natural language supervision. Results also showed that fine-tuning the CLIP models, such as ViT-L/14 and ViT-B/32, significantly improved scene classification, achieving a top F1 score of 91.1%. These results demonstrate the ability of the system to deliver rapid and precise scene recognition, which can be used to meet the critical requirements of Advanced Driver Assistance Systems (ADAS). This study shows the potential of CLIP models to provide scalable and efficient frameworks for dynamic scene understanding and classification. Furthermore, this work lays the groundwork for advanced autonomous vehicle technologies by fostering a deeper understanding of driver behavior, road conditions, and safety-critical scenarios, marking a significant step toward smarter, safer, and more context-aware autonomous driving systems.

Title: Enforcing Fundamental Relations via Adversarial Attacks on Input Parameter Correlations

• Authors: Timo Saala, Lucie Flek, Alexander Jung, Akbar Karimi, Alexander Schmidt, Matthias Schott, Philipp Soldin, Christopher Wiebusch
• Subjects: cs.LG, hep-ex
• Abstract URL: https://arxiv.org/abs/2501.05588
• Pdf URL: https://arxiv.org/pdf/2501.05588
• Copy Paste: [[2501.05588]] Enforcing Fundamental Relations via Adversarial Attacks on Input Parameter Correlations(https://arxiv.org/abs/2501.05588)
• Keywords: attack
• Abstract: Correlations between input parameters play a crucial role in many scientific classification tasks, since these are often related to fundamental laws of nature. For example, in high energy physics, one of the common deep learning use-cases is the classification of signal and background processes in particle collisions. In many such cases, the fundamental principles of the correlations between observables are often better understood than the actual distributions of the observables themselves. In this work, we present a new adversarial attack algorithm called Random Distribution Shuffle Attack (RDSA), emphasizing the correlations between observables in the network rather than individual feature characteristics. Correct application of the proposed novel attack can result in a significant improvement in classification performance - particularly in the context of data augmentation - when using the generated adversaries within adversarial training. Given that correlations between input features are also crucial in many other disciplines. We demonstrate the RDSA effectiveness on six classification tasks, including two particle collision challenges (using CERN Open Data), hand-written digit recognition (MNIST784), human activity recognition (HAR), weather forecasting (Rain in Australia), and ICU patient mortality (MIMIC-IV), demonstrating a general use case beyond fundamental physics for this new type of adversarial attack algorithms.

Title: Session-Level Dynamic Ad Load Optimization using Offline Robust Reinforcement Learning

• Authors: Tao Liu, Qi Xu, Wei Shi, Zhigang Hua, Shuang Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05591
• Pdf URL: https://arxiv.org/pdf/2501.05591
• Copy Paste: [[2501.05591]] Session-Level Dynamic Ad Load Optimization using Offline Robust Reinforcement Learning(https://arxiv.org/abs/2501.05591)
• Keywords: robust
• Abstract: Session-level dynamic ad load optimization aims to personalize the density and types of delivered advertisements in real time during a user's online session by dynamically balancing user experience quality and ad monetization. Traditional causal learning-based approaches struggle with key technical challenges, especially in handling confounding bias and distribution shifts. In this paper, we develop an offline deep Q-network (DQN)-based framework that effectively mitigates confounding bias in dynamic systems and demonstrates more than 80% offline gains compared to the best causal learning-based production baseline. Moreover, to improve the framework's robustness against unanticipated distribution shifts, we further enhance our framework with a novel offline robust dueling DQN approach. This approach achieves more stable rewards on multiple OpenAI-Gym datasets as perturbations increase, and provides an additional 5% offline gains on real-world ad delivery data. Deployed across multiple production systems, our approach has achieved outsized topline gains. Post-launch online A/B tests have shown double-digit improvements in the engagement-ad score trade-off efficiency, significantly enhancing our platform's capability to serve both consumers and advertisers.

Title: Exploring Large Language Models for Translating Romanian Computational Problems into English

• Authors: Adrian Marius Dumitran, Adrian-Catalin Badea, Stefan-Gabriel Muscalu, Angela-Liliana Dumitran, Stefan-Cosmin Dascalescu, Radu-Sebastian Amarie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05601
• Pdf URL: https://arxiv.org/pdf/2501.05601
• Copy Paste: [[2501.05601]] Exploring Large Language Models for Translating Romanian Computational Problems into English(https://arxiv.org/abs/2501.05601)
• Keywords: robust, large language model
• Abstract: Recent studies have suggested that large language models (LLMs) underperform on mathematical and computer science tasks when these problems are translated from Romanian into English, compared to their original Romanian format. Accurate translation is critical for applications ranging from automatic translations in programming competitions to the creation of high-quality educational materials, as well as minimizing errors or fraud in human translations. This study shows that robust large language models (LLMs) can maintain or even enhance their performance in translating less common languages when given well-structured prompts. Our findings suggest that LLMs, with appropriate supervision, can be reliably used for the automatic translation of IOI (International Olympiad in Informatics)-style tasks. We evaluate several translation methods across multiple LLMs, including OpenRoLLM, Llama 3.1 8B, Llama 3.2 3B and GPT-4o, assessing their translation accuracy and performance stability through repeated runs. Additionally, we augment the OJI (Romanian County-Level Informatics Olympiad) Romanian dataset with accurate English translations, enhancing its utility for future LLM training and evaluation. Through detailed syntactic and semantic analyses, we confirm that with human oversight, LLMs can serve as a viable solution for multilingual problem-solving. We also compare the translation quality of LLMs against human translators, as evaluated by a certified expert, underscoring the potential of LLMs in realworld scenarios.

Title: Harmonizing Metadata of Language Resources for Enhanced Querying and Accessibility

• Authors: Zixuan Liang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.05606
• Pdf URL: https://arxiv.org/pdf/2501.05606
• Copy Paste: [[2501.05606]] Harmonizing Metadata of Language Resources for Enhanced Querying and Accessibility(https://arxiv.org/abs/2501.05606)
• Keywords: extraction
• Abstract: This paper addresses the harmonization of metadata from diverse repositories of language resources (LRs). Leveraging linked data and RDF techniques, we integrate data from multiple sources into a unified model based on DCAT and META-SHARE OWL ontology. Our methodology supports text-based search, faceted browsing, and advanced SPARQL queries through Linghub, a newly developed portal. Real user queries from the Corpora Mailing List (CML) were evaluated to assess Linghub capability to satisfy actual user needs. Results indicate that while some limitations persist, many user requests can be successfully addressed. The study highlights significant metadata issues and advocates for adherence to open vocabularies and standards to enhance metadata harmonization. This initial research underscores the importance of API-based access to LRs, promoting machine usability and data subset extraction for specific purposes, paving the way for more efficient and standardized LR utilization.

Title: Watermarking Graph Neural Networks via Explanations for Ownership Protection

• Authors: Jane Downer, Ren Wang, Binghui Wang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05614
• Pdf URL: https://arxiv.org/pdf/2501.05614
• Copy Paste: [[2501.05614]] Watermarking Graph Neural Networks via Explanations for Ownership Protection(https://arxiv.org/abs/2501.05614)
• Keywords: protect, attack, robust, watermark
• Abstract: Graph Neural Networks (GNNs) are the mainstream method to learn pervasive graph data and are widely deployed in industry, making their intellectual property valuable. However, protecting GNNs from unauthorized use remains a challenge. Watermarking, which embeds ownership information into a model, is a potential solution. However, existing watermarking methods have two key limitations: First, almost all of them focus on non-graph data, with watermarking GNNs for complex graph data largely unexplored. Second, the de facto backdoor-based watermarking methods pollute training data and induce ownership ambiguity through intentional misclassification. Our explanation-based watermarking inherits the strengths of backdoor-based methods (e.g., robust to watermark removal attacks), but avoids data pollution and eliminates intentional misclassification. In particular, our method learns to embed the watermark in GNN explanations such that this unique watermark is statistically distinct from other potential solutions, and ownership claims must show statistical significance to be verified. We theoretically prove that, even with full knowledge of our method, locating the watermark is an NP-hard problem. Empirically, our method manifests robustness to removal attacks like fine-tuning and pruning. By addressing these challenges, our approach marks a significant advancement in protecting GNN intellectual property.

Title: Kite: How to Delegate Voting Power Privately

• Authors: Kamilla Nazirkhanova, Vrushank Gunjur, X. Pilli Cruz-De Jesus, Dan Boneh
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.05626
• Pdf URL: https://arxiv.org/pdf/2501.05626
• Copy Paste: [[2501.05626]] Kite: How to Delegate Voting Power Privately(https://arxiv.org/abs/2501.05626)
• Keywords: security, privacy
• Abstract: Ensuring the privacy of votes in an election is crucial for the integrity of a democratic process. Often, voting power is delegated to representatives (e.g., in congress) who subsequently vote on behalf of voters on specific issues. This delegation model is also widely used in Decentralized Autonomous Organizations (DAOs). Although several existing voting systems used in DAOs support private voting, they only offer public delegation. In this paper, we introduce Kite, a new protocol that enables $\textit{private}$ delegation of voting power for DAO members. Voters can freely delegate, revoke, and re-delegate their power without revealing any information about who they delegated to. Even the delegate does not learn who delegated to them. The only information that is recorded publicly is that the voter delegated or re-delegated their vote to someone. Kite accommodates both public and private voting for the delegates themselves. We analyze the security of our protocol within the Universal Composability (UC) framework. We implement Kite as an extension to the existing Governor Bravo smart contract on the Ethereum blockchain, that is widely used for DAO governance. Furthermore, we provide an evaluation of our implementation that demonstrates the practicality of the protocol. The most expensive operation is delegation due to the required zero-knowledge proofs. On a consumer-grade laptop, delegation takes between 7 and 167 seconds depending on the requested level of privacy.

Title: An Efficient Key Expansion Method Applied to Security Credential Management System

• Authors: Abel C. H. Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.05627
• Pdf URL: https://arxiv.org/pdf/2501.05627
• Copy Paste: [[2501.05627]] An Efficient Key Expansion Method Applied to Security Credential Management System(https://arxiv.org/abs/2501.05627)
• Keywords: security, privacy
• Abstract: In recent years, U.S. Department of Transportation has adopts Institute of Electrical and Electronics Engineers (IEEE) 1609 series to build the security credential management system (SCMS) for being the standard of connected cars in U.S. Furthermore, a butterfly key expansion (BKE) method in SCMS has been designed to provide pseudonym certificates for improving the privacy of connected cars. However, the BKE method is designed based on elliptic curve cryptography (ECC) in the standard of IEEE 1609.2.1, but more execution time is required for key expansion. Therefore, this study proposes an original efficient key expansion method, and the mathematical principles have been proposed to prove the encryption/decryption feasibility, car privacy, and method efficiency. In a practical environment, the proposed method improves the efficiency of key expansion method in IEEE 1609.2.1-2022 with the same security strength thousands of times.

Title: The Impact of Model Scaling on Seen and Unseen Language Performance

• Authors: Rhitabrat Pokharel, Sina Bagheri Nezhad, Ameeta Agrawal, Suresh Singh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05629
• Pdf URL: https://arxiv.org/pdf/2501.05629
• Copy Paste: [[2501.05629]] The Impact of Model Scaling on Seen and Unseen Language Performance(https://arxiv.org/abs/2501.05629)
• Keywords: large language model
• Abstract: The rapid advancement of Large Language Models (LLMs), particularly those trained on multilingual corpora, has intensified the need for a deeper understanding of their performance across a diverse range of languages and model sizes. Our research addresses this critical need by studying the performance and scaling behavior of multilingual LLMs in text classification and machine translation tasks across 204 languages. We systematically examine both seen and unseen languages across three model families of varying sizes in zero-shot and few-shot settings. Our findings show significant differences in scaling behavior between zero-shot and two-shot scenarios, with striking disparities in performance between seen and unseen languages. Model scale has little effect on zero-shot performance, which remains mostly flat. However, in two-shot settings, larger models show clear linear improvements in multilingual text classification. For translation tasks, however, only the instruction-tuned model showed clear benefits from scaling. Our analysis also suggests that overall resource levels, not just the proportions of pretraining languages, are better predictors of model performance, shedding light on what drives multilingual LLM effectiveness.

Title: HFMF: Hierarchical Fusion Meets Multi-Stream Models for Deepfake Detection

• Authors: Anant Mehta, Bryant McArthur, Nagarjuna Kolloju, Zhengzhong Tu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05631
• Pdf URL: https://arxiv.org/pdf/2501.05631
• Copy Paste: [[2501.05631]] HFMF: Hierarchical Fusion Meets Multi-Stream Models for Deepfake Detection(https://arxiv.org/abs/2501.05631)
• Keywords: robust, extraction, diffusion, transformer, generative
• Abstract: The rapid progress in deep generative models has led to the creation of incredibly realistic synthetic images that are becoming increasingly difficult to distinguish from real-world data. The widespread use of Variational Models, Diffusion Models, and Generative Adversarial Networks has made it easier to generate convincing fake images and videos, which poses significant challenges for detecting and mitigating the spread of misinformation. As a result, developing effective methods for detecting AI-generated fakes has become a pressing concern. In our research, we propose HFMF, a comprehensive two-stage deepfake detection framework that leverages both hierarchical cross-modal feature fusion and multi-stream feature extraction to enhance detection performance against imagery produced by state-of-the-art generative AI models. The first component of our approach integrates vision Transformers and convolutional nets through a hierarchical feature fusion mechanism. The second component of our framework combines object-level information and a fine-tuned convolutional net model. We then fuse the outputs from both components via an ensemble deep neural net, enabling robust classification performances. We demonstrate that our architecture achieves superior performance across diverse dataset benchmarks while maintaining calibration and interoperability.

Title: Automating Date Format Detection for Data Visualization

• Authors: Zixuan Liang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05640
• Pdf URL: https://arxiv.org/pdf/2501.05640
• Copy Paste: [[2501.05640]] Automating Date Format Detection for Data Visualization(https://arxiv.org/abs/2501.05640)
• Keywords: extraction
• Abstract: Data preparation, specifically date parsing, is a significant bottleneck in analytic workflows. To address this, we present two algorithms, one based on minimum entropy and the other on natural language modeling that automatically derive date formats from string data. These algorithms achieve over 90% accuracy on a large corpus of data columns, streamlining the data preparation process within visualization environments. The minimal entropy approach is particularly fast, providing interactive feedback. Our methods simplify date format extraction, making them suitable for integration into data visualization tools and databases.

Title: Iconicity in Large Language Models

• Authors: Anna Marklová, Jiří Milička, Leonid Ryvkin, Ľudmila Lacková Bennet, Libuše Kormaníková
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05643
• Pdf URL: https://arxiv.org/pdf/2501.05643
• Copy Paste: [[2501.05643]] Iconicity in Large Language Models(https://arxiv.org/abs/2501.05643)
• Keywords: large language model
• Abstract: Lexical iconicity, a direct relation between a word's meaning and its form, is an important aspect of every natural language, most commonly manifesting through sound-meaning associations. Since Large language models' (LLMs') access to both meaning and sound of text is only mediated (meaning through textual context, sound through written representation, further complicated by tokenization), we might expect that the encoding of iconicity in LLMs would be either insufficient or significantly different from human processing. This study addresses this hypothesis by having GPT-4 generate highly iconic pseudowords in artificial languages. To verify that these words actually carry iconicity, we had their meanings guessed by Czech and German participants (n=672) and subsequently by LLM-based participants (generated by GPT-4 and Claude 3.5 Sonnet). The results revealed that humans can guess the meanings of pseudowords in the generated iconic language more accurately than words in distant natural languages and that LLM-based participants are even more successful than humans in this task. This core finding is accompanied by several additional analyses concerning the universality of the generated language and the cues that both human and LLM-based participants utilize.

Title: Efficient Representations for High-Cardinality Categorical Variables in Machine Learning

• Authors: Zixuan Liang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05646
• Pdf URL: https://arxiv.org/pdf/2501.05646
• Copy Paste: [[2501.05646]] Efficient Representations for High-Cardinality Categorical Variables in Machine Learning(https://arxiv.org/abs/2501.05646)
• Keywords: robust, interpretability
• Abstract: High\-cardinality categorical variables pose significant challenges in machine learning, particularly in terms of computational efficiency and model interpretability. Traditional one\-hot encoding often results in high\-dimensional sparse feature spaces, increasing the risk of overfitting and reducing scalability. This paper introduces novel encoding techniques, including means encoding, low\-rank encoding, and multinomial logistic regression encoding, to address these challenges. These methods leverage sufficient representations to generate compact and informative embeddings of categorical data. We conduct rigorous theoretical analyses and empirical validations on diverse datasets, demonstrating significant improvements in model performance and computational efficiency compared to baseline methods. The proposed techniques are particularly effective in domains requiring scalable solutions for large datasets, paving the way for more robust and efficient applications in machine learning.

Title: Cascaded Self-Evaluation Augmented Training for Efficient Multimodal Large Language Models

• Authors: Zheqi Lv, Wenkai Wang, Jiawei Wang, Shengyu Zhang, Fei Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05662
• Pdf URL: https://arxiv.org/pdf/2501.05662
• Copy Paste: [[2501.05662]] Cascaded Self-Evaluation Augmented Training for Efficient Multimodal Large Language Models(https://arxiv.org/abs/2501.05662)
• Keywords: large language model
• Abstract: Efficient Multimodal Large Language Models (EMLLMs) have rapidly advanced recently. Incorporating Chain-of-Thought (CoT) reasoning and step-by-step self-evaluation has improved their performance. However, limited parameters often hinder EMLLMs from effectively using self-evaluation during inference. Key challenges include synthesizing evaluation data, determining its quantity, optimizing training and inference strategies, and selecting appropriate prompts. To address these issues, we introduce Self-Evaluation Augmented Training (SEAT). SEAT uses more powerful EMLLMs for CoT reasoning, data selection, and evaluation generation, then trains EMLLMs with the synthesized data. However, handling long prompts and maintaining CoT reasoning quality are problematic. Therefore, we propose Cascaded Self-Evaluation Augmented Training (Cas-SEAT), which breaks down lengthy prompts into shorter, task-specific cascaded prompts and reduces costs for resource-limited settings. During data synthesis, we employ open-source 7B-parameter EMLLMs and annotate a small dataset with short prompts. Experiments demonstrate that Cas-SEAT significantly boosts EMLLMs' self-evaluation abilities, improving performance by 19.68%, 55.57%, and 46.79% on the MathVista, Math-V, and We-Math datasets, respectively. Additionally, our Cas-SEAT Dataset serves as a valuable resource for future research in enhancing EMLLM self-evaluation.

Title: LPRnet: A self-supervised registration network for LiDAR and photogrammetric point clouds

• Authors: Chen Wang, Yanfeng Gu, Xian Li
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.05669
• Pdf URL: https://arxiv.org/pdf/2501.05669
• Copy Paste: [[2501.05669]] LPRnet: A self-supervised registration network for LiDAR and photogrammetric point clouds(https://arxiv.org/abs/2501.05669)
• Keywords: robust, extraction, transformer
• Abstract: LiDAR and photogrammetry are active and passive remote sensing techniques for point cloud acquisition, respectively, offering complementary advantages and heterogeneous. Due to the fundamental differences in sensing mechanisms, spatial distributions and coordinate systems, their point clouds exhibit significant discrepancies in density, precision, noise, and overlap. Coupled with the lack of ground truth for large-scale scenes, integrating the heterogeneous point clouds is a highly challenging task. This paper proposes a self-supervised registration network based on a masked autoencoder, focusing on heterogeneous LiDAR and photogrammetric point clouds. At its core, the method introduces a multi-scale masked training strategy to extract robust features from heterogeneous point clouds under self-supervision. To further enhance registration performance, a rotation-translation embedding module is designed to effectively capture the key features essential for accurate rigid transformations. Building upon the robust representations, a transformer-based architecture seamlessly integrates local and global features, fostering precise alignment across diverse point cloud datasets. The proposed method demonstrates strong feature extraction capabilities for both LiDAR and photogrammetric point clouds, addressing the challenges of acquiring ground truth at the scene level. Experiments conducted on two real-world datasets validate the effectiveness of the proposed method in solving heterogeneous point cloud registration problems.

Title: UniQ: Unified Decoder with Task-specific Queries for Efficient Scene Graph Generation

• Authors: Xinyao Liao, Wei Wei, Dangyang Chen, Yuanyuan Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05687
• Pdf URL: https://arxiv.org/pdf/2501.05687
• Copy Paste: [[2501.05687]] UniQ: Unified Decoder with Task-specific Queries for Efficient Scene Graph Generation(https://arxiv.org/abs/2501.05687)
• Keywords: robust, extraction
• Abstract: Scene Graph Generation(SGG) is a scene understanding task that aims at identifying object entities and reasoning their relationships within a given image. In contrast to prevailing two-stage methods based on a large object detector (e.g., Faster R-CNN), one-stage methods integrate a fixed-size set of learnable queries to jointly reason relational triplets . This paradigm demonstrates robust performance with significantly reduced parameters and computational overhead. However, the challenge in one-stage methods stems from the issue of weak entanglement, wherein entities involved in relationships require both coupled features shared within triplets and decoupled visual features. Previous methods either adopt a single decoder for coupled triplet feature modeling or multiple decoders for separate visual feature extraction but fail to consider both. In this paper, we introduce UniQ, a Unified decoder with task-specific Queries architecture, where task-specific queries generate decoupled visual features for subjects, objects, and predicates respectively, and unified decoder enables coupled feature modeling within relational triplets. Experimental results on the Visual Genome dataset demonstrate that UniQ has superior performance to both one-stage and two-stage methods.

Title: eKalibr: Dynamic Intrinsic Calibration for Event Cameras From First Principles of Events

• Authors: Shuolong Chen, Xingxing Li, Liu Yuan, Ziao Liu
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.05688
• Pdf URL: https://arxiv.org/pdf/2501.05688
• Copy Paste: [[2501.05688]] eKalibr: Dynamic Intrinsic Calibration for Event Cameras From First Principles of Events(https://arxiv.org/abs/2501.05688)
• Keywords: extraction
• Abstract: The bio-inspired event camera has garnered extensive research attention in recent years, owing to its significant potential derived from its high dynamic range and low latency characteristics. Similar to the standard camera, the event camera requires precise intrinsic calibration to facilitate further high-level visual applications, such as pose estimation and mapping. While several calibration methods for event cameras have been proposed, most of them are either (i) engineering-driven, heavily relying on conventional image-based calibration pipelines, or (ii) inconvenient, requiring complex instrumentation. To this end, we propose an accurate and convenient intrinsic calibration method for event cameras, named eKalibr, which builds upon a carefully designed event-based circle grid pattern recognition algorithm. To extract target patterns from events, we perform event-based normal flow estimation to identify potential events generated by circle edges, and cluster them spatially. Subsequently, event clusters associated with the same grid circles are matched and grouped using normal flows, for subsequent time-varying ellipse estimation. Fitted ellipse centers are time-synchronized, for final grid pattern recognition. We conducted extensive experiments to evaluate the performance of eKalibr in terms of pattern extraction and intrinsic calibration. The implementation of eKalibr is open-sourced at (this https URL) to benefit the research community.

Title: Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains

• Authors: Vighnesh Subramaniam, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Shuang Li, Igor Mordatch
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05707
• Pdf URL: https://arxiv.org/pdf/2501.05707
• Copy Paste: [[2501.05707]] Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains(https://arxiv.org/abs/2501.05707)
• Keywords: large language model
• Abstract: Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data. To improve models beyond the training data, recent works have explored how LLMs can be used to generate synthetic data for autonomous self-improvement. However, successive steps of self-improvement can reach a point of diminishing returns. In this work, we propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models. A group of language models, all starting from the same base model, are independently specialized by updating each one using data generated through multiagent interactions among the models. By training each model on independent sets of data, we illustrate how this approach enables specialization across models and diversification over the set of models. As a result, our overall system is able to preserve diverse reasoning chains and autonomously improve over many more rounds of fine-tuning than single-agent self-improvement methods. We quantitatively illustrate the efficacy of the approach across a wide suite of reasoning tasks.

Title: Multi-Step Reasoning in Korean and the Emergent Mirage

• Authors: Guijin Son, Hyunwoo Ko, Dasol Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05712
• Pdf URL: https://arxiv.org/pdf/2501.05712
• Copy Paste: [[2501.05712]] Multi-Step Reasoning in Korean and the Emergent Mirage(https://arxiv.org/abs/2501.05712)
• Keywords: large language model
• Abstract: We introduce HRMCR (HAE-RAE Multi-Step Commonsense Reasoning), a benchmark designed to evaluate large language models' ability to perform multi-step reasoning in culturally specific contexts, focusing on Korean. The questions are automatically generated via templates and algorithms, requiring LLMs to integrate Korean cultural knowledge into sequential reasoning steps. Consistent with prior observations on emergent abilities, our experiments reveal that models trained on fewer than \(2 \cdot 10^{25}\) training FLOPs struggle to solve any questions, showing near-zero performance. Beyond this threshold, performance improves sharply. State-of-the-art models (e.g., O1) still score under 50\%, underscoring the difficulty of our tasks. Notably, stepwise analysis suggests the observed emergent behavior may stem from compounding errors across multiple steps rather than reflecting a genuinely new capability. We publicly release the benchmark and commit to regularly updating the dataset to prevent contamination.

Title: How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond

• Authors: Chen Huang, Yang Deng, Wenqiang Lei, Jiancheng Lv, Tat-Seng Chua, Jimmy Xiangji Huang
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.05714
• Pdf URL: https://arxiv.org/pdf/2501.05714
• Copy Paste: [[2501.05714]] How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond(https://arxiv.org/abs/2501.05714)
• Keywords: large language model
• Abstract: With the advancement of large language models (LLMs), intelligent models have evolved from mere tools to autonomous agents with their own goals and strategies for cooperating with humans. This evolution has birthed a novel paradigm in NLP, i.e., human-model cooperation, that has yielded remarkable progress in numerous NLP tasks in recent years. In this paper, we take the first step to present a thorough review of human-model cooperation, exploring its principles, formalizations, and open challenges. In particular, we introduce a new taxonomy that provides a unified perspective to summarize existing approaches. Also, we discuss potential frontier areas and their corresponding challenges. We regard our work as an entry point, paving the way for more breakthrough research in this regard.

Title: Zero-shot Shark Tracking and Biometrics from Aerial Imagery

• Authors: Chinmay K Lalgudi, Mark E Leone, Jaden V Clark, Sergio Madrigal-Mora, Mario Espinoza
• Subjects: cs.CV, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2501.05717
• Pdf URL: https://arxiv.org/pdf/2501.05717
• Copy Paste: [[2501.05717]] Zero-shot Shark Tracking and Biometrics from Aerial Imagery(https://arxiv.org/abs/2501.05717)
• Keywords: biometric, segmentation
• Abstract: The recent widespread adoption of drones for studying marine animals provides opportunities for deriving biological information from aerial imagery. The large scale of imagery data acquired from drones is well suited for machine learning (ML) analysis. Development of ML models for analyzing marine animal aerial imagery has followed the classical paradigm of training, testing, and deploying a new model for each dataset, requiring significant time, human effort, and ML expertise. We introduce Frame Level ALIgment and tRacking (FLAIR), which leverages the video understanding of Segment Anything Model 2 (SAM2) and the vision-language capabilities of Contrastive Language-Image Pre-training (CLIP). FLAIR takes a drone video as input and outputs segmentation masks of the species of interest across the video. Notably, FLAIR leverages a zero-shot approach, eliminating the need for labeled data, training a new model, or fine-tuning an existing model to generalize to other species. With a dataset of 18,000 drone images of Pacific nurse sharks, we trained state-of-the-art object detection models to compare against FLAIR. We show that FLAIR massively outperforms these object detectors and performs competitively against two human-in-the-loop methods for prompting SAM2, achieving a Dice score of 0.81. FLAIR readily generalizes to other shark species without additional human effort and can be combined with novel heuristics to automatically extract relevant information including length and tailbeat frequency. FLAIR has significant potential to accelerate aerial imagery analysis workflows, requiring markedly less human effort and expertise than traditional machine learning workflows, while achieving superior accuracy. By reducing the effort required for aerial imagery analysis, FLAIR allows scientists to spend more time interpreting results and deriving insights about marine ecosystems.

Title: An Efficiency Firmware Verification Framework for Public Key Infrastructure with Smart Grid and Energy Storage System

• Authors: Jhih-Zen Shih, Cheng-Che Chuang, Hong-Sheng Huang, Hsuan-Tung Chen, Hung-Min Sun
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.05722
• Pdf URL: https://arxiv.org/pdf/2501.05722
• Copy Paste: [[2501.05722]] An Efficiency Firmware Verification Framework for Public Key Infrastructure with Smart Grid and Energy Storage System(https://arxiv.org/abs/2501.05722)
• Keywords: secure, security, attack, robust
• Abstract: As a critical component of electrical energy infrastructure, the smart grid system has become indispensable to the energy sector. However, the rapid evolution of smart grids has attracted numerous nation-state actors seeking to disrupt the power infrastructure of adversarial nations. This development underscores the urgent need to establish secure mechanisms for firmware updates, with firmware signing and verification serving as pivotal elements in safeguarding system integrity. In this work, we propose a digital signing and verification framework grounded in Public Key Infrastructure (PKI), specifically tailored for resource-constrained devices such as smart meters. The framework utilizes the Concise Binary Object Representation (CBOR) and Object Signing and Encryption (COSE) formats to achieve efficient da-ta encapsulation and robust security features. Our approach not only en-sures the secure deployment of firmware updates against the convergence of information technology (IT) and operational technology (OT) attacks but also addresses performance bottlenecks stemming from device limitations, thereby enhancing the overall reliability and stability of the smart grid sys-tem.

Title: Enabling Scalable Oversight via Self-Evolving Critic

• Authors: Zhengyang Tang, Ziniu Li, Zhenyang Xiao, Tian Ding, Ruoyu Sun, Benyou Wang, Dayiheng Liu, Fei Huang, Tianyu Liu, Bowen Yu, Junyang Lin
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05727
• Pdf URL: https://arxiv.org/pdf/2501.05727
• Copy Paste: [[2501.05727]] Enabling Scalable Oversight via Self-Evolving Critic(https://arxiv.org/abs/2501.05727)
• Keywords: large language model
• Abstract: Despite their remarkable performance, the development of Large Language Models (LLMs) faces a critical challenge in scalable oversight: providing effective feedback for tasks where human evaluation is difficult or where LLMs outperform humans. While there is growing interest in using LLMs for critique, current approaches still rely on human annotations or more powerful models, leaving the issue of enhancing critique capabilities without external supervision unresolved. We introduce SCRIT (Self-evolving CRITic), a framework that enables genuine self-evolution of critique abilities. Technically, SCRIT self-improves by training on synthetic data, generated by a contrastive-based self-critic that uses reference solutions for step-by-step critique, and a self-validation mechanism that ensures critique quality through correction outcomes. Implemented with Qwen2.5-72B-Instruct, one of the most powerful LLMs, SCRIT achieves up to a 10.3\% improvement on critique-correction and error identification benchmarks. Our analysis reveals that SCRIT's performance scales positively with data and model size, outperforms alternative approaches, and benefits critically from its self-validation component.

Title: Super-class guided Transformer for Zero-Shot Attribute Classification

• Authors: Sehyung Kim, Chanhyeong Yang, Jihwan Park, Taehoon Song, Hyunwoo J. Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05728
• Pdf URL: https://arxiv.org/pdf/2501.05728
• Copy Paste: [[2501.05728]] Super-class guided Transformer for Zero-Shot Attribute Classification(https://arxiv.org/abs/2501.05728)
• Keywords: transformer
• Abstract: Attribute classification is crucial for identifying specific characteristics within image regions. Vision-Language Models (VLMs) have been effective in zero-shot tasks by leveraging their general knowledge from large-scale datasets. Recent studies demonstrate that transformer-based models with class-wise queries can effectively address zero-shot multi-label classification. However, poor utilization of the relationship between seen and unseen attributes makes the model lack generalizability. Additionally, attribute classification generally involves many attributes, making maintaining the model's scalability difficult. To address these issues, we propose Super-class guided transFormer (SugaFormer), a novel framework that leverages super-classes to enhance scalability and generalizability for zero-shot attribute classification. SugaFormer employs Super-class Query Initialization (SQI) to reduce the number of queries, utilizing common semantic information from super-classes, and incorporates Multi-context Decoding (MD) to handle diverse visual cues. To strengthen generalizability, we introduce two knowledge transfer strategies that utilize VLMs. During training, Super-class guided Consistency Regularization (SCR) aligns SugaFormer's features with VLMs using region-specific prompts, and during inference, Zero-shot Retrieval-based Score Enhancement (ZRSE) refines predictions for unseen attributes. Extensive experiments demonstrate that SugaFormer achieves state-of-the-art performance across three widely-used attribute classification benchmarks under zero-shot, and cross-dataset transfer settings. Our code is available at this https URL.

Title: TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videos

• Authors: Korawat Charoenpitaks, Van-Quang Nguyen, Masanori Suganuma, Kentaro Arai, Seiji Totsuka, Hiroshi Ino, Takayuki Okatani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05733
• Pdf URL: https://arxiv.org/pdf/2501.05733
• Copy Paste: [[2501.05733]] TB-Bench: Training and Testing Multi-Modal AI for Understanding Spatio-Temporal Traffic Behaviors from Dashcam Images/Videos(https://arxiv.org/abs/2501.05733)
• Keywords: large language model
• Abstract: The application of Multi-modal Large Language Models (MLLMs) in Autonomous Driving (AD) faces significant challenges due to their limited training on traffic-specific data and the absence of dedicated benchmarks for spatiotemporal understanding. This study addresses these issues by proposing TB-Bench, a comprehensive benchmark designed to evaluate MLLMs on understanding traffic behaviors across eight perception tasks from ego-centric views. We also introduce vision-language instruction tuning datasets, TB-100k and TB-250k, along with simple yet effective baselines for the tasks. Through extensive experiments, we show that existing MLLMs underperform in these tasks, with even a powerful model like GPT-4o achieving less than 35% accuracy on average. In contrast, when fine-tuned with TB-100k or TB-250k, our baseline models achieve average accuracy up to 85%, significantly enhancing performance on the tasks. Additionally, we demonstrate performance transfer by co-training TB-100k with another traffic dataset, leading to improved performance on the latter. Overall, this study represents a step forward by introducing a comprehensive benchmark, high-quality datasets, and baselines, thus supporting the gradual integration of MLLMs into the perception, prediction, and planning stages of AD.

Title: LLVD: LSTM-based Explicit Motion Modeling in Latent Space for Blind Video Denoising

• Authors: Loay Rashid, Siddharth Roheda, Amit Unde
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05744
• Pdf URL: https://arxiv.org/pdf/2501.05744
• Copy Paste: [[2501.05744]] LLVD: LSTM-based Explicit Motion Modeling in Latent Space for Blind Video Denoising(https://arxiv.org/abs/2501.05744)
• Keywords: extraction
• Abstract: Video restoration plays a pivotal role in revitalizing degraded video content by rectifying imperfections caused by various degradations introduced during capturing (sensor noise, motion blur, etc.), saving/sharing (compression, resizing, etc.) and editing. This paper introduces a novel algorithm designed for scenarios where noise is introduced during video capture, aiming to enhance the visual quality of videos by reducing unwanted noise artifacts. We propose the Latent space LSTM Video Denoiser (LLVD), an end-to-end blind denoising model. LLVD uniquely combines spatial and temporal feature extraction, employing Long Short Term Memory (LSTM) within the encoded feature domain. This integration of LSTM layers is crucial for maintaining continuity and minimizing flicker in the restored video. Moreover, processing frames in the encoded feature domain significantly reduces computations, resulting in a very lightweight architecture. LLVD's blind nature makes it versatile for real, in-the-wild denoising scenarios where prior information about noise characteristics is not available. Experiments reveal that LLVD demonstrates excellent performance for both synthetic and captured noise. Specifically, LLVD surpasses the current State-Of-The-Art (SOTA) in RAW denoising by 0.3dB, while also achieving a 59\% reduction in computational complexity.

Title: StarGen: A Spatiotemporal Autoregression Framework with Video Diffusion Model for Scalable and Controllable Scene Generation

• Authors: Shangjin Zhai, Zhichao Ye, Jialin Liu, Weijian Xie, Jiaqi Hu, Zhen Peng, Hua Xue, Danpeng Chen, Xiaomeng Wang, Lei Yang, Nan Wang, Haomin Liu, Guofeng Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05763
• Pdf URL: https://arxiv.org/pdf/2501.05763
• Copy Paste: [[2501.05763]] StarGen: A Spatiotemporal Autoregression Framework with Video Diffusion Model for Scalable and Controllable Scene Generation(https://arxiv.org/abs/2501.05763)
• Keywords: diffusion, generative
• Abstract: Recent advances in large reconstruction and generative models have significantly improved scene reconstruction and novel view generation. However, due to compute limitations, each inference with these large models is confined to a small area, making long-range consistent scene generation challenging. To address this, we propose StarGen, a novel framework that employs a pre-trained video diffusion model in an autoregressive manner for long-range scene generation. The generation of each video clip is conditioned on the 3D warping of spatially adjacent images and the temporally overlapping image from previously generated clips, improving spatiotemporal consistency in long-range scene generation with precise pose control. The spatiotemporal condition is compatible with various input conditions, facilitating diverse tasks, including sparse view interpolation, perpetual view generation, and layout-conditioned city generation. Quantitative and qualitative evaluations demonstrate StarGen's superior scalability, fidelity, and pose accuracy compared to state-of-the-art methods.

Title: Controlling Large Language Models Through Concept Activation Vectors

• Authors: Hanyu Zhang, Xiting Wang, Chengao Li, Xiang Ao, Qing He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05764
• Pdf URL: https://arxiv.org/pdf/2501.05764
• Copy Paste: [[2501.05764]] Controlling Large Language Models Through Concept Activation Vectors(https://arxiv.org/abs/2501.05764)
• Keywords: large language model
• Abstract: As large language models (LLMs) are widely deployed across various domains, the ability to control their generated outputs has become more critical. This control involves aligning LLMs outputs with human values and ethical principles or customizing LLMs on specific topics or styles for individual users. Existing controlled generation methods either require significant computational resources and extensive trial-and-error or provide coarse-grained control. In this paper, we propose Generation with Concept Activation Vector (GCAV), a lightweight model control framework that ensures accurate control without requiring resource-extensive fine-tuning. Specifically, GCAV first trains a concept activation vector for specified concepts to be controlled, such as toxicity. During inference, GCAV steers the concept vector in LLMs, for example, by removing the toxicity concept vector from the activation layers. Control experiments from different perspectives, including toxicity reduction, sentiment control, linguistic style, and topic control, demonstrate that our framework achieves state-of-the-art performance with granular control, allowing for fine-grained adjustments of both the steering layers and the steering magnitudes for individual samples.

Title: Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models

• Authors: You Li, Heyu Huang, Chi Chen, Kaiyu Huang, Chao Huang, Zonghao Guo, Zhiyuan Liu, Jinan Xu, Yuhua Li, Ruixuan Li, Maosong Sun
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05767
• Pdf URL: https://arxiv.org/pdf/2501.05767
• Copy Paste: [[2501.05767]] Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models(https://arxiv.org/abs/2501.05767)
• Keywords: large language model
• Abstract: The recent advancement of Multimodal Large Language Models (MLLMs) has significantly improved their fine-grained perception of single images and general comprehension across multiple images. However, existing MLLMs still face challenges in achieving precise grounding in complex multi-image scenarios. To address this, we first explore a Chain-of-Thought (CoT) framework that integrates single-image grounding with multi-image comprehension. While partially effective, it remains unstable and struggles to capture abstract visual information due to its non-end-to-end nature. Therefore, we introduce Migician, the first multi-image grounding model capable of performing free-form and accurate grounding across multiple images. To support this, we present the MGrounding-630k dataset, which comprises data for several multi-image grounding tasks derived from existing datasets, along with newly generated free-form grounding instruction-following data. Furthermore, we propose MIG-Bench, a comprehensive benchmark specifically designed for evaluating multi-image grounding capabilities. Experimental results demonstrate that our model achieves significantly superior multi-image grounding capabilities, outperforming the best existing MLLMs by 21.61% and even surpassing much larger 70B models. Our code, model, dataset, and benchmark are fully open-sourced.

Title: Halal or Not: Knowledge Graph Completion for Predicting Cultural Appropriateness of Daily Products

• Authors: Van Thuy Hoang, Tien-Bach-Thanh Do, Jinho Seo, Seung Charlie Kim, Luong Vuong Nguyen, Duong Nguyen Minh Huy, Hyeon-Ju Jeon, O-Joun Lee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05768
• Pdf URL: https://arxiv.org/pdf/2501.05768
• Copy Paste: [[2501.05768]] Halal or Not: Knowledge Graph Completion for Predicting Cultural Appropriateness of Daily Products(https://arxiv.org/abs/2501.05768)
• Keywords: robust
• Abstract: The growing demand for halal cosmetic products has exposed significant challenges, especially in Muslim-majority countries. Recently, various machine learning-based strategies, e.g., image-based methods, have shown remarkable success in predicting the halal status of cosmetics. However, these methods mainly focus on analyzing the discrete and specific ingredients within separate cosmetics, which ignore the high-order and complex relations between cosmetics and ingredients. To address this problem, we propose a halal cosmetic recommendation framework, namely HaCKG, that leverages a knowledge graph of cosmetics and their ingredients to explicitly model and capture the relationships between cosmetics and their components. By representing cosmetics and ingredients as entities within the knowledge graph, HaCKG effectively learns the high-order and complex relations between entities, offering a robust method for predicting halal status. Specifically, we first construct a cosmetic knowledge graph representing the relations between various cosmetics, ingredients, and their properties. We then propose a pre-trained relational graph attention network model with residual connections to learn the structural relation between entities in the knowledge graph. The pre-trained model is then fine-tuned on downstream cosmetic data to predict halal status. Extensive experiments on the cosmetic dataset over halal prediction tasks demonstrate the superiority of our model over state-of-the-art baselines.

Title: Conditional Diffusion Model for Electrical Impedance Tomography

• Authors: Duanpeng Shi, Wendong Zheng, Di Guo, Huaping Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05769
• Pdf URL: https://arxiv.org/pdf/2501.05769
• Copy Paste: [[2501.05769]] Conditional Diffusion Model for Electrical Impedance Tomography(https://arxiv.org/abs/2501.05769)
• Keywords: robust, diffusion
• Abstract: Electrical impedance tomography (EIT) is a non-invasive imaging technique, which has been widely used in the fields of industrial inspection, medical monitoring and tactile sensing. However, due to the inherent non-linearity and ill-conditioned nature of the EIT inverse problem, the reconstructed image is highly sensitive to the measured data, and random noise artifacts often appear in the reconstructed image, which greatly limits the application of EIT. To address this issue, a conditional diffusion model with voltage consistency (CDMVC) is proposed in this study. The method consists of a pre-imaging module, a conditional diffusion model for reconstruction, a forward voltage constraint network and a scheme of voltage consistency constraint during sampling process. The pre-imaging module is employed to generate the initial reconstruction. This serves as a condition for training the conditional diffusion model. Finally, based on the forward voltage constraint network, a voltage consistency constraint is implemented in the sampling phase to incorporate forward information of EIT, thereby enhancing imaging quality. A more complete dataset, including both common and complex concave shapes, is generated. The proposed method is validated using both simulation and physical experiments. Experimental results demonstrate that our method can significantly improves the quality of reconstructed images. In addition, experimental results also demonstrate that our method has good robustness and generalization performance.

Title: rmlnomogram: An R package to construct an explainable nomogram for any machine learning algorithms

• Authors: Herdiantri Sufriyana, Emily Chia-Yu Su
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.05772
• Pdf URL: https://arxiv.org/pdf/2501.05772
• Copy Paste: [[2501.05772]] rmlnomogram: An R package to construct an explainable nomogram for any machine learning algorithms(https://arxiv.org/abs/2501.05772)
• Keywords: fair, explainability
• Abstract: Background: Current nomogram can only be created for regression algorithm. Providing nomogram for any machine learning (ML) algorithms may accelerate model deployment in clinical settings or improve model availability. We developed an R package and web application to construct nomogram with model explainability of any ML algorithms. Methods: We formulated a function to transform an ML prediction model into a nomogram, requiring datasets with: (1) all possible combinations of predictor values; (2) the corresponding outputs of the model; and (3) the corresponding explainability values for each predictor (optional). Web application was also created. Results: Our R package could create 5 types of nomograms for categorical predictors and binary outcome without probability (1), categorical predictors and binary outcome with probability (2) or continuous outcome (3), and categorical with single numerical predictors and binary outcome with probability (4) or continuous outcome (5). Respectively, the first and remaining types optimally allowed maximum 15 and 5 predictors with maximum 3,200 combinations. Web application is provided with such limits. The explainability values were possible for types 2 to 5. Conclusions: Our R package and web application could construct nomogram with model explainability of any ML algorithms using a fair number of predictors.

Title: STHFL: Spatio-Temporal Heterogeneous Federated Learning

• Authors: Shunxin Guo, Hongsong Wang, Shuxia Lin, Xu Yang, Xin Geng
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.05775
• Pdf URL: https://arxiv.org/pdf/2501.05775
• Copy Paste: [[2501.05775]] STHFL: Spatio-Temporal Heterogeneous Federated Learning(https://arxiv.org/abs/2501.05775)
• Keywords: privacy, protect, federate
• Abstract: Federated learning is a new framework that protects data privacy and allows multiple devices to cooperate in training machine learning models. Previous studies have proposed multiple approaches to eliminate the challenges posed by non-iid data and inter-domain heterogeneity issues. However, they ignore the \textbf{spatio-temporal} heterogeneity formed by different data distributions of increasing task data in the intra-domain. Moreover, the global data is generally a long-tailed distribution rather than assuming the global data is balanced in practical applications. To tackle the \textbf{spatio-temporal} dilemma, we propose a novel setting named \textbf{Spatio-Temporal Heterogeneity} Federated Learning (STHFL). Specially, the Global-Local Dynamic Prototype (GLDP) framework is designed for STHFL. In GLDP, the model in each client contains personalized layers which can dynamically adapt to different data distributions. For long-tailed data distribution, global prototypes are served as complementary knowledge for the training on classes with few samples in clients without leaking privacy. As tasks increase in clients, the knowledge of local prototypes generated in previous tasks guides for training in the current task to solve catastrophic forgetting. Meanwhile, the global-local prototypes are updated through the moving average method after training local prototypes in clients. Finally, we evaluate the effectiveness of GLDP, which achieves remarkable results compared to state-of-the-art methods in STHFL scenarios.

Title: StructSR: Refuse Spurious Details in Real-World Image Super-Resolution

• Authors: Yachao Li, Dong Liang, Tianyu Ding, Sheng-Jun Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05777
• Pdf URL: https://arxiv.org/pdf/2501.05777
• Copy Paste: [[2501.05777]] StructSR: Refuse Spurious Details in Real-World Image Super-Resolution(https://arxiv.org/abs/2501.05777)
• Keywords: diffusion
• Abstract: Diffusion-based models have shown great promise in real-world image super-resolution (Real-ISR), but often generate content with structural errors and spurious texture details due to the empirical priors and illusions of these models. To address this issue, we introduce StructSR, a simple, effective, and plug-and-play method that enhances structural fidelity and suppresses spurious details for diffusion-based Real-ISR. StructSR operates without the need for additional fine-tuning, external model priors, or high-level semantic knowledge. At its core is the Structure-Aware Screening (SAS) mechanism, which identifies the image with the highest structural similarity to the low-resolution (LR) input in the early inference stage, allowing us to leverage it as a historical structure knowledge to suppress the generation of spurious details. By intervening in the diffusion inference process, StructSR seamlessly integrates with existing diffusion-based Real-ISR models. Our experimental results demonstrate that StructSR significantly improves the fidelity of structure and texture, improving the PSNR and SSIM metrics by an average of 5.27% and 9.36% on a synthetic dataset (DIV2K-Val) and 4.13% and 8.64% on two real-world datasets (RealSR and DRealSR) when integrated with four state-of-the-art diffusion-based Real-ISR methods.

Title: UV-Attack: Physical-World Adversarial Attacks for Person Detection via Dynamic-NeRF-based UV Mapping

• Authors: Yanjie Li, Wenxuan Zhang, Kaisheng Liang, Bin Xiao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05783
• Pdf URL: https://arxiv.org/pdf/2501.05783
• Copy Paste: [[2501.05783]] UV-Attack: Physical-World Adversarial Attacks for Person Detection via Dynamic-NeRF-based UV Mapping(https://arxiv.org/abs/2501.05783)
• Keywords: attack
• Abstract: In recent research, adversarial attacks on person detectors using patches or static 3D model-based texture modifications have struggled with low success rates due to the flexible nature of human movement. Modeling the 3D deformations caused by various actions has been a major challenge. Fortunately, advancements in Neural Radiance Fields (NeRF) for dynamic human modeling offer new possibilities. In this paper, we introduce UV-Attack, a groundbreaking approach that achieves high success rates even with extensive and unseen human actions. We address the challenge above by leveraging dynamic-NeRF-based UV mapping. UV-Attack can generate human images across diverse actions and viewpoints, and even create novel actions by sampling from the SMPL parameter space. While dynamic NeRF models are capable of modeling human bodies, modifying clothing textures is challenging because they are embedded in neural network parameters. To tackle this, UV-Attack generates UV maps instead of RGB images and modifies the texture stacks. This approach enables real-time texture edits and makes the attack more practical. We also propose a novel Expectation over Pose Transformation loss (EoPT) to improve the evasion success rate on unseen poses and views. Our experiments show that UV-Attack achieves a 92.75% attack success rate against the FastRCNN model across varied poses in dynamic video settings, significantly outperforming the state-of-the-art AdvCamou attack, which only had a 28.50% ASR. Moreover, we achieve 49.5% ASR on the latest YOLOv8 detector in black-box settings. This work highlights the potential of dynamic NeRF-based UV mapping for creating more effective adversarial attacks on person detectors, addressing key challenges in modeling human movement and texture modification.

Title: Cryptanalysis of Cancelable Biometrics Vault

• Authors: Patrick Lacharme, Kevin Thiry-Atighehchi
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05786
• Pdf URL: https://arxiv.org/pdf/2501.05786
• Copy Paste: [[2501.05786]] Cryptanalysis of Cancelable Biometrics Vault(https://arxiv.org/abs/2501.05786)
• Keywords: secure, protect, attack, biometric
• Abstract: Cancelable Biometrics (CB) stands for a range of biometric transformation schemes combining biometrics with user specific tokens to generate secure templates. Required properties are the irreversibility, unlikability and recognition accuracy of templates while making their revocation possible. In biometrics, a key-binding scheme is used for protecting a cryptographic key using a biometric data. The key can be recomputed only if a correct biometric data is acquired during authentication. Applications of key-binding schemes are typically disk encryption, where the cryptographic key is used to encrypt and decrypt the disk. In this paper, we cryptanalyze a recent key-binding scheme, called Cancelable Biometrics Vault (CBV) based on cancelable biometrics. More precisely, the introduced cancelable transformation, called BioEncoding scheme, for instantiating the CBV framework is attacked in terms of reversibility and linkability of templates. Subsequently, our linkability attack enables to recover the key in the vault without additional assumptions. Our cryptanalysis introduces a new perspective by uncovering the CBV scheme's revocability and linkability vulnerabilities, which were not previously identified in comparable biometric-based key-binding schemes.

Title: ActMiner: Applying Causality Tracking and Increment Aligning for Graph-based Cyber Threat Hunting

• Authors: Mingjun Ma, Tiantian Zhu, Tieming Chen, Shuang Li, Jie Ying, Chunlin Xiong, Mingqi Lv, Yan Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.05793
• Pdf URL: https://arxiv.org/pdf/2501.05793
• Copy Paste: [[2501.05793]] ActMiner: Applying Causality Tracking and Increment Aligning for Graph-based Cyber Threat Hunting(https://arxiv.org/abs/2501.05793)
• Keywords: security, attack
• Abstract: To defend against Advanced Persistent Threats on the endpoint, threat hunting employs security knowledge such as cyber threat intelligence to continuously analyze system audit logs through retrospective scanning, querying, or pattern matching, aiming to uncover attack patterns/graphs that traditional detection methods (e.g., recognition for Point of Interest) fail to capture. However, existing threat hunting systems based on provenance graphs face challenges of high false negatives, high false positives, and low efficiency when confronted with diverse attack tactics and voluminous audit logs. To address these issues, we propose a system called Actminer, which constructs query graphs from descriptive relationships in cyber threat intelligence reports for precise threat hunting (i.e., graph alignment) on provenance graphs. First, we present a heuristic search strategy based on equivalent semantic transfer to reduce false negatives. Second, we establish a filtering mechanism based on causal relationships of attack behaviors to mitigate false positives. Finally, we design a tree structure to incrementally update the alignment results, significantly improving hunting efficiency. Evaluation on the DARPA Engagement dataset demonstrates that compared to the SOTA POIROT, Actminer reduces false positives by 39.1%, eliminates all false negatives, and effectively counters adversarial attacks.

Title: Robust Counterfactual Explanations under Model Multiplicity Using Multi-Objective Optimization

• Authors: Keita Kinjo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05795
• Pdf URL: https://arxiv.org/pdf/2501.05795
• Copy Paste: [[2501.05795]] Robust Counterfactual Explanations under Model Multiplicity Using Multi-Objective Optimization(https://arxiv.org/abs/2501.05795)
• Keywords: robust, explainability
• Abstract: In recent years, explainability in machine learning has gained importance. In this context, counterfactual explanation (CE), which is an explanation method that uses examples, has attracted attention. However, it has been pointed out that CE is not robust when there are multiple machine-learning models. These problems are important when using machine learning to make safe decisions. In this paper, we propose robust CEs that introduce a new viewpoint - Pareto improvement - and a method that uses multi-objective optimization to generate it. To evaluate the proposed method, we conducted experiments using both simulated and actual data. The results demonstrate that the proposed method is robust and useful. We believe that this research will contribute to a wide range of research areas, such as explainability in machine learning, decision-making, and action planning based on machine learning.

Title: Alignment without Over-optimization: Training-Free Solution for Diffusion Models

• Authors: Sunwoo Kim, Minkyu Kim, Dongmin Park
• Subjects: cs.LG, cs.AI, cs.CV, math.ST
• Abstract URL: https://arxiv.org/abs/2501.05803
• Pdf URL: https://arxiv.org/pdf/2501.05803
• Copy Paste: [[2501.05803]] Alignment without Over-optimization: Training-Free Solution for Diffusion Models(https://arxiv.org/abs/2501.05803)
• Keywords: robust, diffusion, generative
• Abstract: Diffusion models excel in generative tasks, but aligning them with specific objectives while maintaining their versatility remains challenging. Existing fine-tuning methods often suffer from reward over-optimization, while approximate guidance approaches fail to optimize target rewards effectively. Addressing these limitations, we propose a training-free sampling method based on Sequential Monte Carlo (SMC) to sample from the reward-aligned target distribution. Our approach, tailored for diffusion sampling and incorporating tempering techniques, achieves comparable or superior target rewards to fine-tuning methods while preserving diversity and cross-reward generalization. We demonstrate its effectiveness in single-reward optimization, multi-objective scenarios, and online black-box optimization. This work offers a robust solution for aligning diffusion models with diverse downstream objectives without compromising their general capabilities. Code is available at this https URL .

Title: AdaPRL: Adaptive Pairwise Regression Learning with Uncertainty Estimation for Universal Regression Tasks

• Authors: Fuhang Liang, Rucong Xu, Deng Lin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05809
• Pdf URL: https://arxiv.org/pdf/2501.05809
• Copy Paste: [[2501.05809]] AdaPRL: Adaptive Pairwise Regression Learning with Uncertainty Estimation for Universal Regression Tasks(https://arxiv.org/abs/2501.05809)
• Keywords: robust, interpretability
• Abstract: Current deep regression models usually learn in point-wise way that treat each sample as an independent input, neglecting the relative ordering among different data. Consequently, the regression model could neglect the data 's interrelationships, potentially resulting in suboptimal performance. Moreover, the existence of aleatoric uncertainty in the training data may drive the model to capture non-generalizable patterns, contributing to increased overfitting. To address these issues, we propose a novel adaptive pairwise learning framework (AdaPRL) for regression tasks which leverages the relative differences between data points and integrates with deep probabilistic models to quantify the uncertainty associated with the predictions. Additionally, we adapt AdaPRL for applications in multi-task learning and multivariate time series forecasting. Extensive experiments with several real-world regression datasets including recommendation systems, age estimation, time series forecasting, natural language understanding, finance, and industry datasets show that AdaPRL is compatible with different backbone networks in various tasks and achieves state-of-the-art performance on the vast majority of tasks, highlighting its notable potential including enhancing prediction accuracy and ranking ability, increasing generalization capability, improving robustness to noisy data, improving resilience to reduced data, and enhancing interpretability, etc.

Title: Diffusion Models for Smarter UAVs: Decision-Making and Modeling

• Authors: Yousef Emami, Hao Zhou, Luis Almeida, Kai Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05819
• Pdf URL: https://arxiv.org/pdf/2501.05819
• Copy Paste: [[2501.05819]] Diffusion Models for Smarter UAVs: Decision-Making and Modeling(https://arxiv.org/abs/2501.05819)
• Keywords: robust, diffusion, generative
• Abstract: Unmanned Aerial Vehicles (UAVs) are increasingly adopted in modern communication networks. However, challenges in decision-making and digital modeling continue to impede their rapid advancement. Reinforcement Learning (RL) algorithms face limitations such as low sample efficiency and limited data versatility, further magnified in UAV communication scenarios. Moreover, Digital Twin (DT) modeling introduces substantial decision-making and data management complexities. RL models, often integrated into DT frameworks, require extensive training data to achieve accurate predictions. In contrast to traditional approaches that focus on class boundaries, Diffusion Models (DMs), a new class of generative AI, learn the underlying probability distribution from the training data and can generate trustworthy new patterns based on this learned distribution. This paper explores the integration of DMs with RL and DT to effectively address these challenges. By combining the data generation capabilities of DMs with the decision-making framework of RL and the modeling accuracy of DT, the integration improves the adaptability and real-time performance of UAV communication. Moreover, the study shows how DMs can alleviate data scarcity, improve policy networks, and optimize dynamic modeling, providing a robust solution for complex UAV communication scenarios.

Title: PersonaHOI: Effortlessly Improving Personalized Face with Human-Object Interaction Generation

• Authors: Xinting Hu, Haoran Wang, Jan Eric Lenssen, Bernt Schiele
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05823
• Pdf URL: https://arxiv.org/pdf/2501.05823
• Copy Paste: [[2501.05823]] PersonaHOI: Effortlessly Improving Personalized Face with Human-Object Interaction Generation(https://arxiv.org/abs/2501.05823)
• Keywords: diffusion
• Abstract: We introduce PersonaHOI, a training- and tuning-free framework that fuses a general StableDiffusion model with a personalized face diffusion (PFD) model to generate identity-consistent human-object interaction (HOI) images. While existing PFD models have advanced significantly, they often overemphasize facial features at the expense of full-body coherence, PersonaHOI introduces an additional StableDiffusion (SD) branch guided by HOI-oriented text inputs. By incorporating cross-attention constraints in the PFD branch and spatial merging at both latent and residual levels, PersonaHOI preserves personalized facial details while ensuring interactive non-facial regions. Experiments, validated by a novel interaction alignment metric, demonstrate the superior realism and scalability of PersonaHOI, establishing a new standard for practical personalized face with HOI generation. Our code will be available at this https URL

Title: Fine-tuning is Not Fine: Mitigating Backdoor Attacks in GNNs with Limited Clean Data

• Authors: Jiale Zhang, Bosen Rao, Chengcheng Zhu, Xiaobing Sun, Qingming Li, Haibo Hu, Xiapu Luo, Qingqing Ye, Shouling Ji
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.05835
• Pdf URL: https://arxiv.org/pdf/2501.05835
• Copy Paste: [[2501.05835]] Fine-tuning is Not Fine: Mitigating Backdoor Attacks in GNNs with Limited Clean Data(https://arxiv.org/abs/2501.05835)
• Keywords: defense, attack
• Abstract: Graph Neural Networks (GNNs) have achieved remarkable performance through their message-passing mechanism. However, recent studies have highlighted the vulnerability of GNNs to backdoor attacks, which can lead the model to misclassify graphs with attached triggers as the target class. The effectiveness of recent promising defense techniques, such as fine-tuning or distillation, is heavily contingent on having comprehensive knowledge of the sufficient training dataset. Empirical studies have shown that fine-tuning methods require a clean dataset of 20% to reduce attack accuracy to below 25%, while distillation methods require a clean dataset of 15%. However, obtaining such a large amount of clean data is commonly impractical. In this paper, we propose a practical backdoor mitigation framework, denoted as GRAPHNAD, which can capture high-quality intermediate-layer representations in GNNs to enhance the distillation process with limited clean data. To achieve this, we address the following key questions: How to identify the appropriate attention representations in graphs for distillation? How to enhance distillation with limited data? By adopting the graph attention transfer method, GRAPHNAD can effectively align the intermediate-layer attention representations of the backdoored model with that of the teacher model, forcing the backdoor neurons to transform into benign ones. Besides, we extract the relation maps from intermediate-layer transformation and enforce the relation maps of the backdoored model to be consistent with that of the teacher model, thereby ensuring model accuracy while further reducing the influence of backdoors. Extensive experimental results show that by fine-tuning a teacher model with only 3% of the clean data, GRAPHNAD can reduce the attack success rate to below 5%.

Title: Poetry in Pixels: Prompt Tuning for Poem Image Generation via Diffusion Models

• Authors: Sofia Jamil, Bollampalli Areen Reddy, Raghvendra Kumar, Sriparna Saha, K J Joseph, Koustava Goswami
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05839
• Pdf URL: https://arxiv.org/pdf/2501.05839
• Copy Paste: [[2501.05839]] Poetry in Pixels: Prompt Tuning for Poem Image Generation via Diffusion Models(https://arxiv.org/abs/2501.05839)
• Keywords: diffusion
• Abstract: The task of text-to-image generation has encountered significant challenges when applied to literary works, especially poetry. Poems are a distinct form of literature, with meanings that frequently transcend beyond the literal words. To address this shortcoming, we propose a PoemToPixel framework designed to generate images that visually represent the inherent meanings of poems. Our approach incorporates the concept of prompt tuning in our image generation framework to ensure that the resulting images closely align with the poetic content. In addition, we propose the PoeKey algorithm, which extracts three key elements in the form of emotions, visual elements, and themes from poems to form instructions which are subsequently provided to a diffusion model for generating corresponding images. Furthermore, to expand the diversity of the poetry dataset across different genres and ages, we introduce MiniPo, a novel multimodal dataset comprising 1001 children's poems and images. Leveraging this dataset alongside PoemSum, we conducted both quantitative and qualitative evaluations of image generation using our PoemToPixel framework. This paper demonstrates the effectiveness of our approach and offers a fresh perspective on generating images from literary sources.

Title: Orthogonal projection-based regularization for efficient model augmentation

• Authors: Bendegúz M. Györök, Jan H. Hoekstra, Johan Kon, Tamás Péni, Maarten Schoukens, Roland Tóth
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2501.05842
• Pdf URL: https://arxiv.org/pdf/2501.05842
• Copy Paste: [[2501.05842]] Orthogonal projection-based regularization for efficient model augmentation(https://arxiv.org/abs/2501.05842)
• Keywords: interpretability
• Abstract: Deep-learning-based nonlinear system identification has shown the ability to produce reliable and highly accurate models in practice. However, these black-box models lack physical interpretability, and often a considerable part of the learning effort is spent on capturing already expected/known behavior due to first-principles-based understanding of some aspects of the system. A potential solution is to integrate prior physical knowledge directly into the model structure, combining the strengths of physics-based modeling and deep-learning-based identification. The most common approach is to use an additive model augmentation structure, where the physics-based and the machine-learning (ML) components are connected in parallel. However, such models are overparametrized, training them is challenging, potentially causing the physics-based part to lose interpretability. To overcome this challenge, this paper proposes an orthogonal projection-based regularization technique to enhance parameter learning, convergence, and even model accuracy in learning-based augmentation of nonlinear baseline models.

Title: ConSim: Measuring Concept-Based Explanations' Effectiveness with Automated Simulatability

• Authors: Antonin Poché, Alon Jacovi, Agustin Martin Picard, Victor Boutin (CERCO, ANITI), Fanny Jourdan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05855
• Pdf URL: https://arxiv.org/pdf/2501.05855
• Copy Paste: [[2501.05855]] ConSim: Measuring Concept-Based Explanations' Effectiveness with Automated Simulatability(https://arxiv.org/abs/2501.05855)
• Keywords: large language model
• Abstract: Concept-based explanations work by mapping complex model computations to human-understandable concepts. Evaluating such explanations is very difficult, as it includes not only the quality of the induced space of possible concepts but also how effectively the chosen concepts are communicated to users. Existing evaluation metrics often focus solely on the former, neglecting the latter. We introduce an evaluation framework for measuring concept explanations via automated simulatability: a simulator's ability to predict the explained model's outputs based on the provided explanations. This approach accounts for both the concept space and its interpretation in an end-to-end evaluation. Human studies for simulatability are notoriously difficult to enact, particularly at the scale of a wide, comprehensive empirical evaluation (which is the subject of this work). We propose using large language models (LLMs) as simulators to approximate the evaluation and report various analyses to make such approximations reliable. Our method allows for scalable and consistent evaluation across various models and datasets. We report a comprehensive empirical evaluation using this framework and show that LLMs provide consistent rankings of explanation methods. Code available at this https URL

Title: Text-to-Edit: Controllable End-to-End Video Ad Creation via Multimodal LLMs

• Authors: Dabing Cheng, Haosen Zhan, Xingchen Zhao, Guisheng Liu, Zemin Li, Jinghui Xie, Zhao Song, Weiguo Feng, Bingyue Peng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05884
• Pdf URL: https://arxiv.org/pdf/2501.05884
• Copy Paste: [[2501.05884]] Text-to-Edit: Controllable End-to-End Video Ad Creation via Multimodal LLMs(https://arxiv.org/abs/2501.05884)
• Keywords: extraction, large language model
• Abstract: The exponential growth of short-video content has ignited a surge in the necessity for efficient, automated solutions to video editing, with challenges arising from the need to understand videos and tailor the editing according to user requirements. Addressing this need, we propose an innovative end-to-end foundational framework, ultimately actualizing precise control over the final video content editing. Leveraging the flexibility and generalizability of Multimodal Large Language Models (MLLMs), we defined clear input-output mappings for efficient video creation. To bolster the model's capability in processing and comprehending video content, we introduce a strategic combination of a denser frame rate and a slow-fast processing technique, significantly enhancing the extraction and understanding of both temporal and spatial video information. Furthermore, we introduce a text-to-edit mechanism that allows users to achieve desired video outcomes through textual input, thereby enhancing the quality and controllability of the edited videos. Through comprehensive experimentation, our method has not only showcased significant effectiveness within advertising datasets, but also yields universally applicable conclusions on public datasets.

Title: EDNet: Edge-Optimized Small Target Detection in UAV Imagery -- Faster Context Attention, Better Feature Fusion, and Hardware Acceleration

• Authors: Zhifan Song, Yuan Zhang, Abd Al Rahman M. Abu Ebayyeh
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05885
• Pdf URL: https://arxiv.org/pdf/2501.05885
• Copy Paste: [[2501.05885]] EDNet: Edge-Optimized Small Target Detection in UAV Imagery -- Faster Context Attention, Better Feature Fusion, and Hardware Acceleration(https://arxiv.org/abs/2501.05885)
• Keywords: privacy, extraction
• Abstract: Detecting small targets in drone imagery is challenging due to low resolution, complex backgrounds, and dynamic scenes. We propose EDNet, a novel edge-target detection framework built on an enhanced YOLOv10 architecture, optimized for real-time applications without post-processing. EDNet incorporates an XSmall detection head and a Cross Concat strategy to improve feature fusion and multi-scale context awareness for detecting tiny targets in diverse environments. Our unique C2f-FCA block employs Faster Context Attention to enhance feature extraction while reducing computational complexity. The WIoU loss function is employed for improved bounding box regression. With seven model sizes ranging from Tiny to XL, EDNet accommodates various deployment environments, enabling local real-time inference and ensuring data privacy. Notably, EDNet achieves up to a 5.6% gain in mAP@50 with significantly fewer parameters. On an iPhone 12, EDNet variants operate at speeds ranging from 16 to 55 FPS, providing a scalable and efficient solution for edge-based object detection in challenging drone imagery. The source code and pre-trained models are available at: this https URL.

Title: Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs

• Authors: Bianca Raimondi, Saverio Giallorenzo, Maurizio Gabbrielli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05891
• Pdf URL: https://arxiv.org/pdf/2501.05891
• Copy Paste: [[2501.05891]] Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs(https://arxiv.org/abs/2501.05891)
• Keywords: large language model
• Abstract: In education, the capability of generating human-like text of Large Language Models (LLMs) inspired work on how they can increase the efficiency of learning and teaching. We study the affordability of these models for educators and students by investigating how LLMs answer multiple-choice questions (MCQs) with respect to hardware constraints and refinement techniques. We explore this space by using generic pre-trained LLMs (the 7B, 13B, and 70B variants of LLaMA-2) to answer 162 undergraduate-level MCQs from a course on Programming Languages (PL) -- the MCQ dataset is a contribution of this work, which we make publicly available. Specifically, we dissect how different factors, such as using readily-available material -- (parts of) the course's textbook -- for fine-tuning and quantisation (to decrease resource usage) can change the accuracy of the responses. The main takeaway is that smaller textbook-based fine-tuned models outperform generic larger ones (whose pre-training requires conspicuous resources), making the usage of LLMs for answering MCQs resource- and material-wise affordable.

Title: Beyond Flat Text: Dual Self-inherited Guidance for Visual Text Generation

• Authors: Minxing Luo, Zixun Xia, Liaojun Chen, Zhenhang Li, Weichao Zeng, Jianye Wang, Wentao Cheng, Yaxing Wang, Yu Zhou, Jian Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05892
• Pdf URL: https://arxiv.org/pdf/2501.05892
• Copy Paste: [[2501.05892]] Beyond Flat Text: Dual Self-inherited Guidance for Visual Text Generation(https://arxiv.org/abs/2501.05892)
• Keywords: diffusion, generative
• Abstract: In real-world images, slanted or curved texts, especially those on cans, banners, or badges, appear as frequently, if not more so, than flat texts due to artistic design or layout constraints. While high-quality visual text generation has become available with the advanced generative capabilities of diffusion models, these models often produce distorted text and inharmonious text background when given slanted or curved text layouts due to training data limitation. In this paper, we introduce a new training-free framework, STGen, which accurately generates visual texts in challenging scenarios (\eg, slanted or curved text layouts) while harmonizing them with the text background. Our framework decomposes the visual text generation process into two branches: (i) \textbf{Semantic Rectification Branch}, which leverages the ability in generating flat but accurate visual texts of the model to guide the generation of challenging scenarios. The generated latent of flat text is abundant in accurate semantic information related both to the text itself and its background. By incorporating this, we rectify the semantic information of the texts and harmonize the integration of the text with its background in complex layouts. (ii) \textbf{Structure Injection Branch}, which reinforces the visual text structure during inference. We incorporate the latent information of the glyph image, rich in glyph structure, as a new condition to further strengthen the text structure. To enhance image harmony, we also apply an effective combination method to merge the priors, providing a solid foundation for generation. Extensive experiments across a variety of visual text layouts demonstrate that our framework achieves superior accuracy and outstanding quality.

Title: Valley2: Exploring Multimodal Models with Scalable Vision-Language Design

• Authors: Ziheng Wu, Zhenghao Chen, Ruipu Luo, Can Zhang, Yuan Gao, Zhentao He, Xian Wang, Haoran Lin, Minghui Qiu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05901
• Pdf URL: https://arxiv.org/pdf/2501.05901
• Copy Paste: [[2501.05901]] Valley2: Exploring Multimodal Models with Scalable Vision-Language Design(https://arxiv.org/abs/2501.05901)
• Keywords: large language model
• Abstract: Recently, vision-language models have made remarkable progress, demonstrating outstanding capabilities in various tasks such as image captioning and video understanding. We introduce Valley2, a novel multimodal large language model designed to enhance performance across all domains and extend the boundaries of practical applications in e-commerce and short video scenarios. Notably, Valley2 achieves state-of-the-art (SOTA) performance on e-commerce benchmarks, surpassing open-source models of similar size by a large margin (79.66 vs. 72.76). Additionally, Valley2 ranks second on the OpenCompass leaderboard among models with fewer than 10B parameters, with an impressive average score of 67.4. The code and model weights are open-sourced at this https URL.

Title: Binary Event-Driven Spiking Transformer

• Authors: Honglin Cao, Zijian Zhou, Wenjie Wei, Ammar Belatreche, Yu Liang, Dehao Zhang, Malu Zhang, Yang Yang, Haizhou Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05904
• Pdf URL: https://arxiv.org/pdf/2501.05904
• Copy Paste: [[2501.05904]] Binary Event-Driven Spiking Transformer(https://arxiv.org/abs/2501.05904)
• Keywords: transformer
• Abstract: Transformer-based Spiking Neural Networks (SNNs) introduce a novel event-driven self-attention paradigm that combines the high performance of Transformers with the energy efficiency of SNNs. However, the larger model size and increased computational demands of the Transformer structure limit their practicality in resource-constrained scenarios. In this paper, we integrate binarization techniques into Transformer-based SNNs and propose the Binary Event-Driven Spiking Transformer, i.e. BESTformer. The proposed BESTformer can significantly reduce storage and computational demands by representing weights and attention maps with a mere 1-bit. However, BESTformer suffers from a severe performance drop from its full-precision counterpart due to the limited representation capability of binarization. To address this issue, we propose a Coupled Information Enhancement (CIE) method, which consists of a reversible framework and information enhancement distillation. By maximizing the mutual information between the binary model and its full-precision counterpart, the CIE method effectively mitigates the performance degradation of the BESTformer. Extensive experiments on static and neuromorphic datasets demonstrate that our method achieves superior performance to other binary SNNs, showcasing its potential as a compact yet high-performance model for resource-limited edge devices.

Title: Security Testing Framework for Web Applications: Benchmarking ZAP V2.12.0 and V2.13.0 by OWASP as an example

• Authors: Usha-Sri Potti, Hong-Sheng Huang, Hsuan-Tung Chen, Hung-Min Sun
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.05907
• Pdf URL: https://arxiv.org/pdf/2501.05907
• Copy Paste: [[2501.05907]] Security Testing Framework for Web Applications: Benchmarking ZAP V2.12.0 and V2.13.0 by OWASP as an example(https://arxiv.org/abs/2501.05907)
• Keywords: secure, security, robust
• Abstract: The Huge growth in the usage of web applications has raised concerns regarding their security vulnerabilities, which in turn pushes toward robust security testing tools. This study compares OWASP ZAP, the leading open-source web application vulnerability scanner, across its two most recent iterations. While comparing their performance to the OWASP Benchmark, the study evaluates their efficiency in spotting vulnerabilities in the purposefully vulnerable application, OWASP Benchmark project. The research methodology involves conducting systematic scans of OWASP Benchmark using both v2.12.0 and v2.13.0 of OWASP ZAP. The OWASP Benchmark provides a standardized framework to evaluate the scanner's abilities in identifying security flaws, Insecure Cookies, Path traversal, SQL injection, and more. Results obtained from this benchmark comparison offer valuable insights into the strengths and weaknesses of each version of the tool. This study aids in web application security testing by shedding light on how well-known scanners work at spotting vulnerabilities. The knowledge gained from this study can assist security professionals and developers in making informed decisions to support their web application security status. In conclusion, this study comprehensively analyzes ZAP's capabilities in detecting security flaws using OWASP Benchmark v1.2. The findings add to the continuing debates about online application security tools and establish the framework for future studies and developments in the research field of web application security testing.

Title: From Balance to Breach: Cyber Threats to Battery Energy Storage Systems

• Authors: Frans Öhrström, Joakim Oscarsson, Zeeshan Afzal, János Dani, Mikael Asplund
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.05923
• Pdf URL: https://arxiv.org/pdf/2501.05923
• Copy Paste: [[2501.05923]] From Balance to Breach: Cyber Threats to Battery Energy Storage Systems(https://arxiv.org/abs/2501.05923)
• Keywords: security, attack
• Abstract: Battery energy storage systems are an important part of modern power systems as a solution to maintain grid balance. However, such systems are often remotely managed using cloud-based control systems. This exposes them to cyberattacks that could result in catastrophic consequences for the electrical grid and the connected infrastructure. This paper takes a step towards advancing understanding of these systems and investigates the effects of cyberattacks targeting them. We propose a reference model for an electrical grid cloud-controlled load-balancing system connected to remote battery energy storage systems. The reference model is evaluated from a cybersecurity perspective by implementing and simulating various cyberattacks. The results reveal the system's attack surface and demonstrate the impact of cyberattacks that can criticaly threaten the security and stability of the electrical grid.

Title: Navigating Tomorrow: Reliably Assessing Large Language Models Performance on Future Event Prediction

• Authors: Petraq Nako, Adam Jatowt
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.05925
• Pdf URL: https://arxiv.org/pdf/2501.05925
• Copy Paste: [[2501.05925]] Navigating Tomorrow: Reliably Assessing Large Language Models Performance on Future Event Prediction(https://arxiv.org/abs/2501.05925)
• Keywords: large language model
• Abstract: Predicting future events is an important activity with applications across multiple fields and domains. For example, the capacity to foresee stock market trends, natural disasters, business developments, or political events can facilitate early preventive measures and uncover new opportunities. Multiple diverse computational methods for attempting future predictions, including predictive analysis, time series forecasting, and simulations have been proposed. This study evaluates the performance of several large language models (LLMs) in supporting future prediction tasks, an under-explored domain. We assess the models across three scenarios: Affirmative vs. Likelihood questioning, Reasoning, and Counterfactual analysis. For this, we create a dataset1 by finding and categorizing news articles based on entity type and its popularity. We gather news articles before and after the LLMs training cutoff date in order to thoroughly test and compare model performance. Our research highlights LLMs potential and limitations in predictive modeling, providing a foundation for future improvements.

Title: LLMs Reproduce Stereotypes of Sexual and Gender Minorities

• Authors: Ruby Ostrow, Adam Lopez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05926
• Pdf URL: https://arxiv.org/pdf/2501.05926
• Copy Paste: [[2501.05926]] LLMs Reproduce Stereotypes of Sexual and Gender Minorities(https://arxiv.org/abs/2501.05926)
• Keywords: large language model
• Abstract: A large body of research has found substantial gender bias in NLP systems. Most of this research takes a binary, essentialist view of gender: limiting its variation to the categories _men_ and _women_, conflating gender with sex, and ignoring different sexual identities. But gender and sexuality exist on a spectrum, so in this paper we study the biases of large language models (LLMs) towards sexual and gender minorities beyond binary categories. Grounding our study in a widely used psychological framework -- the Stereotype Content Model -- we demonstrate that English-language survey questions about social perceptions elicit more negative stereotypes of sexual and gender minorities from LLMs, just as they do from humans. We then extend this framework to a more realistic use case: text generation. Our analysis shows that LLMs generate stereotyped representations of sexual and gender minorities in this setting, raising concerns about their capacity to amplify representational harms in creative writing, a widely promoted use case.

Title: Towards Backdoor Stealthiness in Model Parameter Space

• Authors: Xiaoyun Xu, Zhuoran Liu, Stefanos Koffas, Stjepan Picek
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05928
• Pdf URL: https://arxiv.org/pdf/2501.05928
• Copy Paste: [[2501.05928]] Towards Backdoor Stealthiness in Model Parameter Space(https://arxiv.org/abs/2501.05928)
• Keywords: defense, attack, steal
• Abstract: Recent research on backdoor stealthiness focuses mainly on indistinguishable triggers in input space and inseparable backdoor representations in feature space, aiming to circumvent backdoor defenses that examine these respective spaces. However, existing backdoor attacks are typically designed to resist a specific type of backdoor defense without considering the diverse range of defense mechanisms. Based on this observation, we pose a natural question: Are current backdoor attacks truly a real-world threat when facing diverse practical defenses? To answer this question, we examine 12 common backdoor attacks that focus on input-space or feature-space stealthiness and 17 diverse representative defenses. Surprisingly, we reveal a critical blind spot: Backdoor attacks designed to be stealthy in input and feature spaces can be mitigated by examining backdoored models in parameter space. To investigate the underlying causes behind this common vulnerability, we study the characteristics of backdoor attacks in the parameter space. Notably, we find that input- and feature-space attacks introduce prominent backdoor-related neurons in parameter space, which are not thoroughly considered by current backdoor attacks. Taking comprehensive stealthiness into account, we propose a novel supply-chain attack called Grond. Grond limits the parameter changes by a simple yet effective module, Adversarial Backdoor Injection (ABI), which adaptively increases the parameter-space stealthiness during the backdoor injection. Extensive experiments demonstrate that Grond outperforms all 12 backdoor attacks against state-of-the-art (including adaptive) defenses on CIFAR-10, GTSRB, and a subset of ImageNet. In addition, we show that ABI consistently improves the effectiveness of common backdoor attacks.

Title: DiffuSETS: 12-lead ECG Generation Conditioned on Clinical Text Reports and Patient-Specific Information

• Authors: Yongfan Lai, Jiabo Chen, Deyun Zhang, Yue Wang, Shijia Geng, Hongyan Li, Shenda Hong
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05932
• Pdf URL: https://arxiv.org/pdf/2501.05932
• Copy Paste: [[2501.05932]] DiffuSETS: 12-lead ECG Generation Conditioned on Clinical Text Reports and Patient-Specific Information(https://arxiv.org/abs/2501.05932)
• Keywords: privacy, generative
• Abstract: Heart disease remains a significant threat to human health. As a non-invasive diagnostic tool, the electrocardiogram (ECG) is one of the most widely used methods for cardiac screening. However, the scarcity of high-quality ECG data, driven by privacy concerns and limited medical resources, creates a pressing need for effective ECG signal generation. Existing approaches for generating ECG signals typically rely on small training datasets, lack comprehensive evaluation frameworks, and overlook potential applications beyond data augmentation. To address these challenges, we propose DiffuSETS, a novel framework capable of generating ECG signals with high semantic alignment and fidelity. DiffuSETS accepts various modalities of clinical text reports and patient-specific information as inputs, enabling the creation of clinically meaningful ECG signals. Additionally, to address the lack of standardized evaluation in ECG generation, we introduce a comprehensive benchmarking methodology to assess the effectiveness of generative models in this domain. Our model achieve excellent results in tests, proving its superiority in the task of ECG generation. Furthermore, we showcase its potential to mitigate data scarcity while exploring novel applications in cardiology education and medical knowledge discovery, highlighting the broader impact of our work.

Title: Weakly Supervised Segmentation of Hyper-Reflective Foci with Compact Convolutional Transformers and SAM2

• Authors: Olivier Morelle (1 and 2), Justus Bisten (1), Maximilian W. M. Wintergerst (2 and 5), Robert P. Finger (2 and 4), Thomas Schultz (1 and 3) ((1) B-IT and Department of Computer Science, University of Bonn, (2) Department of Ophthalmology, University Hospital Bonn, (3) Lamarr Institute for Machine Learning and Artificial Intelligence, (4) Department of Ophthalmology, University Medical Center Mannheim, Heidelberg University, (5) Augenzentrum Grischun, Chur, Switzerland)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05933
• Pdf URL: https://arxiv.org/pdf/2501.05933
• Copy Paste: [[2501.05933]] Weakly Supervised Segmentation of Hyper-Reflective Foci with Compact Convolutional Transformers and SAM2(https://arxiv.org/abs/2501.05933)
• Keywords: transformer, segmentation
• Abstract: Weakly supervised segmentation has the potential to greatly reduce the annotation effort for training segmentation models for small structures such as hyper-reflective foci (HRF) in optical coherence tomography (OCT). However, most weakly supervised methods either involve a strong downsampling of input images, or only achieve localization at a coarse resolution, both of which are unsatisfactory for small structures. We propose a novel framework that increases the spatial resolution of a traditional attention-based Multiple Instance Learning (MIL) approach by using Layer-wise Relevance Propagation (LRP) to prompt the Segment Anything Model (SAM~2), and increases recall with iterative inference. Moreover, we demonstrate that replacing MIL with a Compact Convolutional Transformer (CCT), which adds a positional encoding, and permits an exchange of information between different regions of the OCT image, leads to a further and substantial increase in segmentation accuracy.

Title: Encoded Spatial Attribute in Multi-Tier Federated Learning

• Authors: Asfia Kawnine, Francis Palma, Seyed Alireza Rahimi Azghadi, Hung Cao
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.05934
• Pdf URL: https://arxiv.org/pdf/2501.05934
• Copy Paste: [[2501.05934]] Encoded Spatial Attribute in Multi-Tier Federated Learning(https://arxiv.org/abs/2501.05934)
• Keywords: federate
• Abstract: This research presents an Encoded Spatial Multi-Tier Federated Learning approach for a comprehensive evaluation of aggregated models for geospatial data. In the client tier, encoding spatial information is introduced to better predict the target outcome. The research aims to assess the performance of these models across diverse datasets and spatial attributes, highlighting variations in predictive accuracy. Using evaluation metrics such as accuracy, our research reveals insights into the complexities of spatial granularity and the challenges of capturing underlying patterns in the data. We extended the scope of federated learning (FL) by having multi-tier along with the functionality of encoding spatial attributes. Our N-tier FL approach used encoded spatial data to aggregate in different tiers. We obtained multiple models that predicted the different granularities of spatial data. Our findings underscore the need for further research to improve predictive accuracy and model generalization, with potential avenues including incorporating additional features, refining model architectures, and exploring alternative modeling approaches. Our experiments have several tiers representing different levels of spatial aspects. We obtained accuracy of 75.62% and 89.52% for the global model without having to train the model using the data constituted with the designated tier. The research also highlights the importance of the proposed approach in real-time applications.

Title: A Multimodal Dataset for Enhancing Industrial Task Monitoring and Engagement Prediction

• Authors: Naval Kishore Mehta, Arvind, Himanshu Kumar, Abeer Banerjee, Sumeet Saurav, Sanjay Singh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05936
• Pdf URL: https://arxiv.org/pdf/2501.05936
• Copy Paste: [[2501.05936]] A Multimodal Dataset for Enhancing Industrial Task Monitoring and Engagement Prediction(https://arxiv.org/abs/2501.05936)
• Keywords: robust
• Abstract: Detecting and interpreting operator actions, engagement, and object interactions in dynamic industrial workflows remains a significant challenge in human-robot collaboration research, especially within complex, real-world environments. Traditional unimodal methods often fall short of capturing the intricacies of these unstructured industrial settings. To address this gap, we present a novel Multimodal Industrial Activity Monitoring (MIAM) dataset that captures realistic assembly and disassembly tasks, facilitating the evaluation of key meta-tasks such as action localization, object interaction, and engagement prediction. The dataset comprises multi-view RGB, depth, and Inertial Measurement Unit (IMU) data collected from 22 sessions, amounting to 290 minutes of untrimmed video, annotated in detail for task performance and operator behavior. Its distinctiveness lies in the integration of multiple data modalities and its emphasis on real-world, untrimmed industrial workflows-key for advancing research in human-robot collaboration and operator monitoring. Additionally, we propose a multimodal network that fuses RGB frames, IMU data, and skeleton sequences to predict engagement levels during industrial tasks. Our approach improves the accuracy of recognizing engagement states, providing a robust solution for monitoring operator performance in dynamic industrial environments. The dataset and code can be accessed from this https URL.

Title: Soft regression trees: a model variant and a decomposition training algorithm

• Authors: Antonio Consolo, Edoardo Amaldi, Andrea Manno
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2501.05942
• Pdf URL: https://arxiv.org/pdf/2501.05942
• Copy Paste: [[2501.05942]] Soft regression trees: a model variant and a decomposition training algorithm(https://arxiv.org/abs/2501.05942)
• Keywords: robust, interpretability
• Abstract: Decision trees are widely used for classification and regression tasks in a variety of application fields due to their interpretability and good accuracy. During the past decade, growing attention has been devoted to globally optimized decision trees with deterministic or soft splitting rules at branch nodes, which are trained by optimizing the error function over all the tree parameters. In this work, we propose a new variant of soft multivariate regression trees (SRTs) where, for every input vector, the prediction is defined as the linear regression associated to a single leaf node, namely, the leaf node obtained by routing the input vector from the root along the branches with higher probability. SRTs exhibit the conditional computational property, i.e., each prediction depends on a small number of nodes (parameters), and our nonlinear optimization formulation for training them is amenable to decomposition. After showing a universal approximation result for SRTs, we present a decomposition training algorithm including a clustering-based initialization procedure and a heuristic for reassigning the input vectors along the tree. Under mild assumptions, we establish asymptotic convergence guarantees. Experiments on 15 wellknown datasets indicate that our SRTs and decomposition algorithm yield higher accuracy and robustness compared with traditional soft regression trees trained using the nonlinear optimization formulation of Blanquero et al., and a significant reduction in training times as well as a slightly better average accuracy compared with the mixed-integer optimization approach of Bertsimas and Dunn. We also report a comparison with the Random Forest ensemble method.

Title: Universal-2-TF: Robust All-Neural Text Formatting for ASR

• Authors: Yash Khare, Taufiquzzaman Peyash, Andrea Vanzo, Takuya Yoshioka
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05948
• Pdf URL: https://arxiv.org/pdf/2501.05948
• Copy Paste: [[2501.05948]] Universal-2-TF: Robust All-Neural Text Formatting for ASR(https://arxiv.org/abs/2501.05948)
• Keywords: robust
• Abstract: This paper introduces an all-neural text formatting (TF) model designed for commercial automatic speech recognition (ASR) systems, encompassing punctuation restoration (PR), truecasing, and inverse text normalization (ITN). Unlike traditional rule-based or hybrid approaches, this method leverages a two-stage neural architecture comprising a multi-objective token classifier and a sequence-to-sequence (seq2seq) model. This design minimizes computational costs and reduces hallucinations while ensuring flexibility and robustness across diverse linguistic entities and text domains. Developed as part of the Universal-2 ASR system, the proposed method demonstrates superior performance in TF accuracy, computational efficiency, and perceptual quality, as validated through comprehensive evaluations using both objective and subjective methods. This work underscores the importance of holistic TF models in enhancing ASR usability in practical settings.

Title: Swin-X2S: Reconstructing 3D Shape from 2D Biplanar X-ray with Swin Transformers

• Authors: Kuan Liu, Zongyuan Ying, Jie Jin, Dongyan Li, Ping Huang, Wenjian Wu, Zhe Chen, Jin Qi, Yong Lu, Lianfu Deng, Bo Chen
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.05961
• Pdf URL: https://arxiv.org/pdf/2501.05961
• Copy Paste: [[2501.05961]] Swin-X2S: Reconstructing 3D Shape from 2D Biplanar X-ray with Swin Transformers(https://arxiv.org/abs/2501.05961)
• Keywords: extraction, transformer, segmentation
• Abstract: The conversion from 2D X-ray to 3D shape holds significant potential for improving diagnostic efficiency and safety. However, existing reconstruction methods often rely on hand-crafted features, manual intervention, and prior knowledge, resulting in unstable shape errors and additional processing costs. In this paper, we introduce Swin-X2S, an end-to-end deep learning method for directly reconstructing 3D segmentation and labeling from 2D biplanar orthogonal X-ray images. Swin-X2S employs an encoder-decoder architecture: the encoder leverages 2D Swin Transformer for X-ray information extraction, while the decoder employs 3D convolution with cross-attention to integrate structural features from orthogonal views. A dimension-expanding module is introduced to bridge the encoder and decoder, ensuring a smooth conversion from 2D pixels to 3D voxels. We evaluate proposed method through extensive qualitative and quantitative experiments across nine publicly available datasets covering four anatomies (femur, hip, spine, and rib), with a total of 54 categories. Significant improvements over previous methods have been observed not only in the segmentation and labeling metrics but also in the clinically relevant parameters that are of primary concern in practical applications, which demonstrates the promise of Swin-X2S to provide an effective option for anatomical shape reconstruction in clinical scenarios. Code implementation is available at: \url{this https URL}.

Title: Effective faking of verbal deception detection with target-aligned adversarial attacks

• Authors: Bennett Kleinberg, Riccardo Loconte, Bruno Verschuere
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05962
• Pdf URL: https://arxiv.org/pdf/2501.05962
• Copy Paste: [[2501.05962]] Effective faking of verbal deception detection with target-aligned adversarial attacks(https://arxiv.org/abs/2501.05962)
• Keywords: attack, robust, large language model
• Abstract: Background: Deception detection through analysing language is a promising avenue using both human judgments and automated machine learning judgments. For both forms of credibility assessment, automated adversarial attacks that rewrite deceptive statements to appear truthful pose a serious threat. Methods: We used a dataset of 243 truthful and 262 fabricated autobiographical stories in a deception detection task for humans and machine learning models. A large language model was tasked to rewrite deceptive statements so that they appear truthful. In Study 1, humans who made a deception judgment or used the detailedness heuristic and two machine learning models (a fine-tuned language model and a simple n-gram model) judged original or adversarial modifications of deceptive statements. In Study 2, we manipulated the target alignment of the modifications, i.e. tailoring the attack to whether the statements would be assessed by humans or computer models. Results: When adversarial modifications were aligned with their target, human (d=-0.07 and d=-0.04) and machine judgments (51% accuracy) dropped to the chance level. When the attack was not aligned with the target, both human heuristics judgments (d=0.30 and d=0.36) and machine learning predictions (63-78%) were significantly better than chance. Conclusions: Easily accessible language models can effectively help anyone fake deception detection efforts both by humans and machine learning models. Robustness against adversarial modifications for humans and machines depends on that target alignment. We close with suggestions on advancing deception research with adversarial attack designs.

Title: Model Inversion in Split Learning for Personalized LLMs: New Insights from Information Bottleneck Theory

• Authors: Yunmeng Shu, Shaofeng Li, Tian Dong, Yan Meng, Haojin Zhu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05965
• Pdf URL: https://arxiv.org/pdf/2501.05965
• Copy Paste: [[2501.05965]] Model Inversion in Split Learning for Personalized LLMs: New Insights from Information Bottleneck Theory(https://arxiv.org/abs/2501.05965)
• Keywords: secure, privacy, defense, attack, transformer, generative, large language model
• Abstract: Personalized Large Language Models (LLMs) have become increasingly prevalent, showcasing the impressive capabilities of models like GPT-4. This trend has also catalyzed extensive research on deploying LLMs on mobile devices. Feasible approaches for such edge-cloud deployment include using split learning. However, previous research has largely overlooked the privacy leakage associated with intermediate representations transmitted from devices to servers. This work is the first to identify model inversion attacks in the split learning framework for LLMs, emphasizing the necessity of secure defense. For the first time, we introduce mutual information entropy to understand the information propagation of Transformer-based LLMs and assess privacy attack performance for LLM blocks. To address the issue of representations being sparser and containing less information than embeddings, we propose a two-stage attack system in which the first part projects representations into the embedding space, and the second part uses a generative model to recover text from these embeddings. This design breaks down the complexity and achieves attack scores of 38%-75% in various scenarios, with an over 60% improvement over the SOTA. This work comprehensively highlights the potential privacy risks during the deployment of personalized LLMs on the edge side.

Title: Hermit Kingdom Through the Lens of Multiple Perspectives: A Case Study of LLM Hallucination on North Korea

• Authors: Eunjung Cho, Won Ik Cho, Soomin Seo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.05981
• Pdf URL: https://arxiv.org/pdf/2501.05981
• Copy Paste: [[2501.05981]] Hermit Kingdom Through the Lens of Multiple Perspectives: A Case Study of LLM Hallucination on North Korea(https://arxiv.org/abs/2501.05981)
• Keywords: security, large language model
• Abstract: Hallucination in large language models (LLMs) remains a significant challenge for their safe deployment, particularly due to its potential to spread misinformation. Most existing solutions address this challenge by focusing on aligning the models with credible sources or by improving how models communicate their confidence (or lack thereof) in their outputs. While these measures may be effective in most contexts, they may fall short in scenarios requiring more nuanced approaches, especially in situations where access to accurate data is limited or determining credible sources is challenging. In this study, we take North Korea - a country characterised by an extreme lack of reliable sources and the prevalence of sensationalist falsehoods - as a case study. We explore and evaluate how some of the best-performing multilingual LLMs and specific language-based models generate information about North Korea in three languages spoken in countries with significant geo-political interests: English (United States, United Kingdom), Korean (South Korea), and Mandarin Chinese (China). Our findings reveal significant differences, suggesting that the choice of model and language can lead to vastly different understandings of North Korea, which has important implications given the global security challenges the country poses.

Title: Comparing Self-Supervised Learning Models Pre-Trained on Human Speech and Animal Vocalizations for Bioacoustics Processing

• Authors: Eklavya Sarkar, Mathew Magimai.-Doss
• Subjects: cs.LG, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.05987
• Pdf URL: https://arxiv.org/pdf/2501.05987
• Copy Paste: [[2501.05987]] Comparing Self-Supervised Learning Models Pre-Trained on Human Speech and Animal Vocalizations for Bioacoustics Processing(https://arxiv.org/abs/2501.05987)
• Keywords: robust
• Abstract: Self-supervised learning (SSL) foundation models have emerged as powerful, domain-agnostic, general-purpose feature extractors applicable to a wide range of tasks. Such models pre-trained on human speech have demonstrated high transferability for bioacoustic processing. This paper investigates (i) whether SSL models pre-trained directly on animal vocalizations offer a significant advantage over those pre-trained on speech, and (ii) whether fine-tuning speech-pretrained models on automatic speech recognition (ASR) tasks can enhance bioacoustic classification. We conduct a comparative analysis using three diverse bioacoustic datasets and two different bioacoustic tasks. Results indicate that pre-training on bioacoustic data provides only marginal improvements over speech-pretrained models, with comparable performance in most scenarios. Fine-tuning on ASR tasks yields mixed outcomes, suggesting that the general-purpose representations learned during SSL pre-training are already well-suited for bioacoustic tasks. These findings highlight the robustness of speech-pretrained SSL models for bioacoustics and imply that extensive fine-tuning may not be necessary for optimal performance.

Title: Addressing speaker gender bias in large scale speech translation systems

• Authors: Shubham Bansal, Vikas Joshi, Harveen Chadha, Rupeshkumar Mehta, Jinyu Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05989
• Pdf URL: https://arxiv.org/pdf/2501.05989
• Copy Paste: [[2501.05989]] Addressing speaker gender bias in large scale speech translation systems(https://arxiv.org/abs/2501.05989)
• Keywords: large language model
• Abstract: This study addresses the issue of speaker gender bias in Speech Translation (ST) systems, which can lead to offensive and inaccurate translations. The masculine bias often found in large-scale ST systems is typically perpetuated through training data derived from Machine Translation (MT) systems. Our approach involves two key steps. First, we employ Large Language Models (LLMs) to rectify translations based on the speaker's gender in a cost-effective manner. Second, we fine-tune the ST model with the corrected data, enabling the model to generate gender-specific translations directly from audio cues, without the need for explicit gender input. Additionally, we propose a three-mode fine-tuned model for scenarios where the speaker's gender is either predefined or should not be inferred from speech cues. We demonstrate a 70% improvement in translations for female speakers compared to our baseline and other large-scale ST systems, such as Seamless M4T and Canary, on the MuST-SHE test set.

Title: Minimizing Occlusion Effect on Multi-View Camera Perception in BEV with Multi-Sensor Fusion

• Authors: Sanjay Kumar, Hiep Truong, Sushil Sharma, Ganesh Sistu, Tony Scanlan, Eoin Grua, Ciarán Eising
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05997
• Pdf URL: https://arxiv.org/pdf/2501.05997
• Copy Paste: [[2501.05997]] Minimizing Occlusion Effect on Multi-View Camera Perception in BEV with Multi-Sensor Fusion(https://arxiv.org/abs/2501.05997)
• Keywords: robust, segmentation
• Abstract: Autonomous driving technology is rapidly evolving, offering the potential for safer and more efficient transportation. However, the performance of these systems can be significantly compromised by the occlusion on sensors due to environmental factors like dirt, dust, rain, and fog. These occlusions severely affect vision-based tasks such as object detection, vehicle segmentation, and lane recognition. In this paper, we investigate the impact of various kinds of occlusions on camera sensor by projecting their effects from multi-view camera images of the nuScenes dataset into the Bird's-Eye View (BEV) domain. This approach allows us to analyze how occlusions spatially distribute and influence vehicle segmentation accuracy within the BEV domain. Despite significant advances in sensor technology and multi-sensor fusion, a gap remains in the existing literature regarding the specific effects of camera occlusions on BEV-based perception systems. To address this gap, we use a multi-sensor fusion technique that integrates LiDAR and radar sensor data to mitigate the performance degradation caused by occluded cameras. Our findings demonstrate that this approach significantly enhances the accuracy and robustness of vehicle segmentation tasks, leading to more reliable autonomous driving systems.

Title: Self-Supervised Partial Cycle-Consistency for Multi-View Matching

• Authors: Fedor Taggenbrock, Gertjan Burghouts, Ronald Poppe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06000
• Pdf URL: https://arxiv.org/pdf/2501.06000
• Copy Paste: [[2501.06000]] Self-Supervised Partial Cycle-Consistency for Multi-View Matching(https://arxiv.org/abs/2501.06000)
• Keywords: robust, extraction
• Abstract: Matching objects across partially overlapping camera views is crucial in multi-camera systems and requires a view-invariant feature extraction network. Training such a network with cycle-consistency circumvents the need for labor-intensive labeling. In this paper, we extend the mathematical formulation of cycle-consistency to handle partial overlap. We then introduce a pseudo-mask which directs the training loss to take partial overlap into account. We additionally present several new cycle variants that complement each other and present a time-divergent scene sampling scheme that improves the data input for this self-supervised setting. Cross-camera matching experiments on the challenging DIVOTrack dataset show the merits of our approach. Compared to the self-supervised state-of-the-art, we achieve a 4.3 percentage point higher F1 score with our combined contributions. Our improvements are robust to reduced overlap in the training data, with substantial improvements in challenging scenes that need to make few matches between many people. Self-supervised feature networks trained with our method are effective at matching objects in a range of multi-camera settings, providing opportunities for complex tasks like large-scale multi-camera scene understanding.

Title: Learning to generate feasible graphs using graph grammars

• Authors: Stefan Mautner, Rolf Backofen, Fabrizio Costa
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06003
• Pdf URL: https://arxiv.org/pdf/2501.06003
• Copy Paste: [[2501.06003]] Learning to generate feasible graphs using graph grammars(https://arxiv.org/abs/2501.06003)
• Keywords: generative
• Abstract: Generative methods for graphs need to be sufficiently flexible to model complex dependencies between sets of nodes. At the same time, the generated graphs need to satisfy domain-dependent feasibility conditions, that is, they should not violate certain constraints that would make their interpretation impossible within the given application domain (e.g. a molecular graph where an atom has a very large number of chemical bounds). Crucially, constraints can involve not only local but also long-range dependencies: for example, the maximal length of a cycle can be bounded. Currently, a large class of generative approaches for graphs, such as methods based on artificial neural networks, is based on message passing schemes. These approaches suffer from information 'dilution' issues that severely limit the maximal range of the dependencies that can be modeled. To address this problem, we propose a generative approach based on the notion of graph grammars. The key novel idea is to introduce a domain-dependent coarsening procedure to provide short-cuts for long-range dependencies. We show the effectiveness of our proposal in two domains: 1) small drugs and 2) RNA secondary structures. In the first case, we compare the quality of the generated molecular graphs via the Molecular Sets (MOSES) benchmark suite, which evaluates the distance between generated and real molecules, their lipophilicity, synthesizability, and drug-likeness. In the second case, we show that the approach can generate very large graphs (with hundreds of nodes) that are accepted as valid examples for a desired RNA family by the "Infernal" covariance model, a state-of-the-art RNA classifier. Our implementation is available on github: this http URL

Title: CamCtrl3D: Single-Image Scene Exploration with Precise 3D Camera Control

• Authors: Stefan Popov, Amit Raj, Michael Krainin, Yuanzhen Li, William T. Freeman, Michael Rubinstein
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06006
• Pdf URL: https://arxiv.org/pdf/2501.06006
• Copy Paste: [[2501.06006]] CamCtrl3D: Single-Image Scene Exploration with Precise 3D Camera Control(https://arxiv.org/abs/2501.06006)
• Keywords: diffusion, transformer
• Abstract: We propose a method for generating fly-through videos of a scene, from a single image and a given camera trajectory. We build upon an image-to-video latent diffusion model. We condition its UNet denoiser on the camera trajectory, using four techniques. (1) We condition the UNet's temporal blocks on raw camera extrinsics, similar to MotionCtrl. (2) We use images containing camera rays and directions, similar to CameraCtrl. (3) We reproject the initial image to subsequent frames and use the resulting video as a condition. (4) We use 2D<=>3D transformers to introduce a global 3D representation, which implicitly conditions on the camera poses. We combine all conditions in a ContolNet-style architecture. We then propose a metric that evaluates overall video quality and the ability to preserve details with view changes, which we use to analyze the trade-offs of individual and combined conditions. Finally, we identify an optimal combination of conditions. We calibrate camera positions in our datasets for scale consistency across scenes, and we train our scene exploration model, CamCtrl3D, demonstrating state-of-theart results.

Title: RPKI-Based Location-Unaware Tor Guard Relay Selection Algorithms

• Authors: Zhifan Lu, Siyang Sun, Yixin Sun
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.06010
• Pdf URL: https://arxiv.org/pdf/2501.06010
• Copy Paste: [[2501.06010]] RPKI-Based Location-Unaware Tor Guard Relay Selection Algorithms(https://arxiv.org/abs/2501.06010)
• Keywords: secure, security, privacy, attack
• Abstract: Tor is a well-known anonymous communication tool, used by people with various privacy and security needs. Prior works have exploited routing attacks to observe Tor traffic and deanonymize Tor users. Subsequently, location-aware relay selection algorithms have been proposed to defend against such attacks on Tor. However, location-aware relay selection algorithms are known to be vulnerable to information leakage on client locations and guard placement attacks. Can we design a new location-unaware approach to relay selection while achieving the similar goal of defending against routing attacks? Towards this end, we leverage the Resource Public Key Infrastructure (RPKI) in designing new guard relay selection algorithms. We develop a lightweight Discount Selection algorithm by only incorporating Route Origin Authorization (ROA) information, and a more secure Matching Selection algorithm by incorporating both ROA and Route Origin Validation (ROV) information. Our evaluation results show an increase in the number of ROA-ROV matched client-relay pairs using our Matching Selection algorithm, reaching 48.47% with minimal performance overhead through custom Shadow simulations and benchmarking.

Title: BRIGHT: A globally distributed multimodal building damage assessment dataset with very-high-resolution for all-weather disaster response

• Authors: Hongruixuan Chen, Jian Song, Olivier Dietrich, Clifford Broni-Bediako, Weihao Xuan, Junjue Wang, Xinlei Shao, Yimin Wei, Junshi Xia, Cuiling Lan, Konrad Schindler, Naoto Yokoya
• Subjects: cs.CV, cs.AI, eess.IV, eess.SP
• Abstract URL: https://arxiv.org/abs/2501.06019
• Pdf URL: https://arxiv.org/pdf/2501.06019
• Copy Paste: [[2501.06019]] BRIGHT: A globally distributed multimodal building damage assessment dataset with very-high-resolution for all-weather disaster response(https://arxiv.org/abs/2501.06019)
• Keywords: robust
• Abstract: Disaster events occur around the world and cause significant damage to human life and property. Earth observation (EO) data enables rapid and comprehensive building damage assessment (BDA), an essential capability in the aftermath of a disaster to reduce human casualties and to inform disaster relief efforts. Recent research focuses on the development of AI models to achieve accurate mapping of unseen disaster events, mostly using optical EO data. However, solutions based on optical data are limited to clear skies and daylight hours, preventing a prompt response to disasters. Integrating multimodal (MM) EO data, particularly the combination of optical and SAR imagery, makes it possible to provide all-weather, day-and-night disaster responses. Despite this potential, the development of robust multimodal AI models has been constrained by the lack of suitable benchmark datasets. In this paper, we present a BDA dataset using veRy-hIGH-resoluTion optical and SAR imagery (BRIGHT) to support AI-based all-weather disaster response. To the best of our knowledge, BRIGHT is the first open-access, globally distributed, event-diverse MM dataset specifically curated to support AI-based disaster response. It covers five types of natural disasters and two types of man-made disasters across 12 regions worldwide, with a particular focus on developing countries where external assistance is most needed. The optical and SAR imagery in BRIGHT, with a spatial resolution between 0.3-1 meters, provides detailed representations of individual buildings, making it ideal for precise BDA. In our experiments, we have tested seven advanced AI models trained with our BRIGHT to validate the transferability and robustness. The dataset and code are available at this https URL. BRIGHT also serves as the official dataset for the 2025 IEEE GRSS Data Fusion Contest.

Title: Geometric-Based Nail Segmentation for Clinical Measurements

• Authors: Bernat Galmés, Gabriel Moyà-Alcover, Pedro Bibiloni, Javier Varona, Antoni Jaume-i-Capó
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06027
• Pdf URL: https://arxiv.org/pdf/2501.06027
• Copy Paste: [[2501.06027]] Geometric-Based Nail Segmentation for Clinical Measurements(https://arxiv.org/abs/2501.06027)
• Keywords: robust, segmentation
• Abstract: A robust segmentation method that can be used to perform measurements on toenails is presented. The proposed method is used as the first step in a clinical trial to objectively quantify the incidence of a particular pathology. For such an assessment, it is necessary to distinguish a nail, which locally appears to be similar to the skin. Many algorithms have been used, each of which leverages different aspects of toenail appearance. We used the Hough transform to locate the tip of the toe and estimate the nail location and size. Subsequently, we classified the super-pixels of the image based on their geometric and photometric information. Thereafter, the watershed transform delineated the border of the nail. The method was validated using a 348-image medical dataset, achieving an accuracy of 0.993 and an F-measure of 0.925. The proposed method is considerably robust across samples, with respect to factors such as nail shape, skin pigmentation, illumination conditions, and appearance of large regions affected by a medical condition

Title: IoT Firmware Version Identification Using Transfer Learning with Twin Neural Networks

• Authors: Ashley Andrews, George Oikonomou, Simon Armour, Paul Thomas, Thomas Cattermole
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.06033
• Pdf URL: https://arxiv.org/pdf/2501.06033
• Copy Paste: [[2501.06033]] IoT Firmware Version Identification Using Transfer Learning with Twin Neural Networks(https://arxiv.org/abs/2501.06033)
• Keywords: security
• Abstract: As the Internet of Things (IoT) becomes more embedded within our daily lives, there is growing concern about the risk `smart' devices pose to network security. To address this, one avenue of research has focused on automated IoT device identification. Research has however largely neglected the identification of IoT device firmware versions. There is strong evidence that IoT security relies on devices being on the latest version patched for known vulnerabilities. Identifying when a device has updated (has changed version) or not (is on a stable version) is therefore useful for IoT security. Version identification involves challenges beyond those for identifying the model, type, and manufacturer of IoT devices, and traditional machine learning algorithms are ill-suited for effective version identification due to being limited by the availability of data for training. In this paper, we introduce an effective technique for identifying IoT device versions based on transfer learning. This technique relies on the idea that we can use a Twin Neural Network (TNN) - trained at distinguishing devices - to detect differences between a device on different versions. This facilitates real-world implementation by requiring relatively little training data. We extract statistical features from on-wire packet flows, convert these features into greyscale images, pass these images into a TNN, and determine version changes based on the Hedges' g effect size of the similarity scores. This allows us to detect the subtle changes present in on-wire traffic when a device changes version. To evaluate our technique, we set up a lab containing 12 IoT devices and recorded their on-wire packet captures for 11 days across multiple firmware versions. For testing data held out from training, our best performing model is shown to be 95.83% and 84.38% accurate at identifying stable versions and version changes respectively.

Title: Nonisotropic Gaussian Diffusion for Realistic 3D Human Motion Prediction

• Authors: Cecilia Curreli, Dominik Muhle, Abhishek Saroha, Zhenzhang Ye, Riccardo Marin, Daniel Cremers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06035
• Pdf URL: https://arxiv.org/pdf/2501.06035
• Copy Paste: [[2501.06035]] Nonisotropic Gaussian Diffusion for Realistic 3D Human Motion Prediction(https://arxiv.org/abs/2501.06035)
• Keywords: diffusion
• Abstract: Probabilistic human motion prediction aims to forecast multiple possible future movements from past observations. While current approaches report high diversity and realism, they often generate motions with undetected limb stretching and jitter. To address this, we introduce SkeletonDiffusion, a latent diffusion model that embeds an explicit inductive bias on the human body within its architecture and training. Our model is trained with a novel nonisotropic Gaussian diffusion formulation that aligns with the natural kinematic structure of the human skeleton. Results show that our approach outperforms conventional isotropic alternatives, consistently generating realistic predictions while avoiding artifacts such as limb distortion. Additionally, we identify a limitation in commonly used diversity metrics, which may inadvertently favor models that produce inconsistent limb lengths within the same sequence. SkeletonDiffusion sets a new benchmark on three real-world datasets, outperforming various baselines across multiple evaluation metrics. Visit our project page: this https URL

Title: A Holistically Point-guided Text Framework for Weakly-Supervised Camouflaged Object Detection

• Authors: Tsui Qin Mok, Shuyong Gao, Haozhe Xing, Miaoyang He, Yan Wang, Wenqiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06038
• Pdf URL: https://arxiv.org/pdf/2501.06038
• Copy Paste: [[2501.06038]] A Holistically Point-guided Text Framework for Weakly-Supervised Camouflaged Object Detection(https://arxiv.org/abs/2501.06038)
• Keywords: transformer
• Abstract: Weakly-Supervised Camouflaged Object Detection (WSCOD) has gained popularity for its promise to train models with weak labels to segment objects that visually blend into their surroundings. Recently, some methods using sparsely-annotated supervision shown promising results through scribbling in WSCOD, while point-text supervision remains underexplored. Hence, this paper introduces a novel holistically point-guided text framework for WSCOD by decomposing into three phases: segment, choose, train. Specifically, we propose Point-guided Candidate Generation (PCG), where the point's foreground serves as a correction for the text path to explicitly correct and rejuvenate the loss detection object during the mask generation process (SEGMENT). We also introduce a Qualified Candidate Discriminator (QCD) to choose the optimal mask from a given text prompt using CLIP (CHOOSE), and employ the chosen pseudo mask for training with a self-supervised Vision Transformer (TRAIN). Additionally, we developed a new point-supervised dataset (P2C-COD) and a text-supervised dataset (T-COD). Comprehensive experiments on four benchmark datasets demonstrate our method outperforms state-of-the-art methods by a large margin, and also outperforms some existing fully-supervised camouflaged object detection methods.

Title: MSCViT: A Small-size ViT architecture with Multi-Scale Self-Attention Mechanism for Tiny Datasets

• Authors: Bowei Zhang, Yi Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06040
• Pdf URL: https://arxiv.org/pdf/2501.06040
• Copy Paste: [[2501.06040]] MSCViT: A Small-size ViT architecture with Multi-Scale Self-Attention Mechanism for Tiny Datasets(https://arxiv.org/abs/2501.06040)
• Keywords: extraction, transformer
• Abstract: Vision Transformer (ViT) has demonstrated significant potential in various vision tasks due to its strong ability in modelling long-range dependencies. However, such success is largely fueled by training on massive samples. In real applications, the large-scale datasets are not always available, and ViT performs worse than Convolutional Neural Networks (CNNs) if it is only trained on small scale dataset (called tiny dataset), since it requires large amount of training data to ensure its representational capacity. In this paper, a small-size ViT architecture with multi-scale self-attention mechanism and convolution blocks is presented (dubbed MSCViT) to model different scales of attention at each layer. Firstly, we introduced wavelet convolution, which selectively combines the high-frequency components obtained by frequency division with our convolution channel to extract local features. Then, a lightweight multi-head attention module is developed to reduce the number of tokens and computational costs. Finally, the positional encoding (PE) in the backbone is replaced by a local feature extraction module. Compared with the original ViT, it is parameter-efficient and is particularly suitable for tiny datasets. Extensive experiments have been conducted on tiny datasets, in which our model achieves an accuracy of 84.68% on CIFAR-100 with 14.0M parameters and 2.5 GFLOPs, without pre-training on large datasets.

Title: Benchmarking Rotary Position Embeddings for Automatic Speech Recognition

• Authors: Shucong Zhang, Titouan Parcollet, Rogier van Dalen, Sourav Bhattacharya
• Subjects: cs.CL, cs.AI, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.06051
• Pdf URL: https://arxiv.org/pdf/2501.06051
• Copy Paste: [[2501.06051]] Benchmarking Rotary Position Embeddings for Automatic Speech Recognition(https://arxiv.org/abs/2501.06051)
• Keywords: transformer
• Abstract: Rotary Position Embedding (RoPE) encodes relative and absolute positional information in Transformer-based models through rotation matrices applied to input vectors within sequences. While RoPE has demonstrated superior performance compared to other positional embedding technologies in natural language processing tasks, its effectiveness in speech processing applications remains understudied. In this work, we conduct a comprehensive evaluation of RoPE across diverse automatic speech recognition (ASR) tasks. Our experimental results demonstrate that for ASR tasks, RoPE consistently achieves lower error rates compared to the currently widely used relative positional embedding. To facilitate further research, we release the implementation and all experimental recipes through the SpeechBrain toolkit.

Title: Enhancing, Refining, and Fusing: Towards Robust Multi-Scale and Dense Ship Detection

• Authors: Congxia Zhao, Xiongjun Fu, Jian Dong, Shen Cao, Chunyan Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06053
• Pdf URL: https://arxiv.org/pdf/2501.06053
• Copy Paste: [[2501.06053]] Enhancing, Refining, and Fusing: Towards Robust Multi-Scale and Dense Ship Detection(https://arxiv.org/abs/2501.06053)
• Keywords: robust
• Abstract: Synthetic aperture radar (SAR) imaging, celebrated for its high resolution, all-weather capability, and day-night operability, is indispensable for maritime applications. However, ship detection in SAR imagery faces significant challenges, including complex backgrounds, densely arranged targets, and large scale variations. To address these issues, we propose a novel framework, Center-Aware SAR Ship Detector (CASS-Det), designed for robust multi-scale and densely packed ship detection. CASS-Det integrates three key innovations: (1) a center enhancement module (CEM) that employs rotational convolution to emphasize ship centers, improving localization while suppressing background interference; (2) a neighbor attention module (NAM) that leverages cross-layer dependencies to refine ship boundaries in densely populated scenes; and (3) a cross-connected feature pyramid network (CC-FPN) that enhances multi-scale feature fusion by integrating shallow and deep features. Extensive experiments on the SSDD, HRSID, and LS-SSDD-v1.0 datasets demonstrate the state-of-the-art performance of CASS-Det, excelling at detecting multi-scale and densely arranged ships.

Title: COMIX: Compositional Explanations using Prototypes

• Authors: Sarath Sivaprasad, Dmitry Kangin, Plamen Angelov, Mario Fritz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06059
• Pdf URL: https://arxiv.org/pdf/2501.06059
• Copy Paste: [[2501.06059]] COMIX: Compositional Explanations using Prototypes(https://arxiv.org/abs/2501.06059)
• Keywords: interpretability
• Abstract: Aligning machine representations with human understanding is key to improving interpretability of machine learning (ML) models. When classifying a new image, humans often explain their decisions by decomposing the image into concepts and pointing to corresponding regions in familiar images. Current ML explanation techniques typically either trace decision-making processes to reference prototypes, generate attribution maps highlighting feature importance, or incorporate intermediate bottlenecks designed to align with human-interpretable concepts. The proposed method, named COMIX, classifies an image by decomposing it into regions based on learned concepts and tracing each region to corresponding ones in images from the training dataset, assuring that explanations fully represent the actual decision-making process. We dissect the test image into selected internal representations of a neural network to derive prototypical parts (primitives) and match them with the corresponding primitives derived from the training data. In a series of qualitative and quantitative experiments, we theoretically prove and demonstrate that our method, in contrast to post hoc analysis, provides fidelity of explanations and shows that the efficiency is competitive with other inherently interpretable architectures. Notably, it shows substantial improvements in fidelity and sparsity metrics, including 48.82% improvement in the C-insertion score on the ImageNet dataset over the best state-of-the-art baseline.

Title: Unveiling Malware Patterns: A Self-analysis Perspective

• Authors: Fangtian Zhong, Qin Hu, Yili Jiang, Jiaqi Huang, Xiuzhen Cheng
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.06071
• Pdf URL: https://arxiv.org/pdf/2501.06071
• Copy Paste: [[2501.06071]] Unveiling Malware Patterns: A Self-analysis Perspective(https://arxiv.org/abs/2501.06071)
• Keywords: security, privacy, defense, attack
• Abstract: The widespread usage of Microsoft Windows has unfortunately led to a surge in malware, posing a serious threat to the security and privacy of millions of users. In response, the research community has mobilized, with numerous efforts dedicated to strengthening defenses against these threats. The primary goal of these techniques is to detect malicious software early, preventing attacks before any damage occurs. However, many of these methods either claim that packing has minimal impact on malware detection or fail to address the reliability of their approaches when applied to packed samples. Consequently, they are not capable of assisting victims in handling packed programs or recovering from the damages caused by untimely malware detection. In light of these challenges, we propose VisUnpack, a static analysis-based data visualization framework for bolstering attack prevention while aiding recovery post-attack by unveiling malware patterns and offering more detailed information including both malware class and family. Our method includes unpacking packed malware programs, calculating local similarity descriptors based on basic blocks, enhancing correlations between descriptors, and refining them by minimizing noises to obtain self-analysis descriptors. Moreover, we employ machine learning to learn the correlations of self-analysis descriptors through architectural learning for final classification. Our comprehensive evaluation of VisUnpack based on a freshly gathered dataset with over 27,106 samples confirms its capability in accurately classifying malware programs with a precision of 99.7%. Additionally, VisUnpack reveals that most antivirus products in VirusTotal can not handle packed samples properly or provide precise malware classification information. We also achieve over 97% space savings compared to existing data visualization based methods.

Title: A monthly sub-national Harmonized Food Insecurity Dataset for comprehensive analysis and predictive modeling

• Authors: Machefer Mélissande, Michele Ronco, Anne-Claire Thomas, Michael Assouline, Melanie Rabier, Christina Corbane, Felix Rembold
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06076
• Pdf URL: https://arxiv.org/pdf/2501.06076
• Copy Paste: [[2501.06076]] A monthly sub-national Harmonized Food Insecurity Dataset for comprehensive analysis and predictive modeling(https://arxiv.org/abs/2501.06076)
• Keywords: security
• Abstract: Food security is a complex, multidimensional concept challenging to measure comprehensively. Effective anticipation, monitoring, and mitigation of food crises require timely and comprehensive global data. This paper introduces the Harmonized Food Insecurity Dataset (HFID), an open-source resource consolidating four key data sources: the Integrated Food Security Phase Classification (IPC)/Cadre Harmonisé (CH) phases, the Famine Early Warning Systems Network (FEWS NET) IPC-compatible phases, and the World Food Program's (WFP) Food Consumption Score (FCS) and reduced Coping Strategy Index (rCSI). Updated monthly and using a common reference system for administrative units, the HFID offers extensive spatial and temporal coverage. It serves as a vital tool for food security experts and humanitarian agencies, providing a unified resource for analyzing food security conditions and highlighting global data disparities. The scientific community can also leverage the HFID to develop data-driven predictive models, enhancing the capacity to forecast and prevent future food crises.

Title: Explainable Federated Bayesian Causal Inference and Its Application in Advanced Manufacturing

• Authors: Xiaofeng Xiao, Khawlah Alharbi, Pengyu Zhang, Hantang Qin, Xubo Yue
• Subjects: cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2501.06077
• Pdf URL: https://arxiv.org/pdf/2501.06077
• Copy Paste: [[2501.06077]] Explainable Federated Bayesian Causal Inference and Its Application in Advanced Manufacturing(https://arxiv.org/abs/2501.06077)
• Keywords: federate
• Abstract: Causal inference has recently gained notable attention across various fields like biology, healthcare, and environmental science, especially within explainable artificial intelligence (xAI) systems, for uncovering the causal relationships among multiple variables and outcomes. Yet, it has not been fully recognized and deployed in the manufacturing systems. In this paper, we introduce an explainable, scalable, and flexible federated Bayesian learning framework, \texttt{xFBCI}, designed to explore causality through treatment effect estimation in distributed manufacturing systems. By leveraging federated Bayesian learning, we efficiently estimate posterior of local parameters to derive the propensity score for each client without accessing local private data. These scores are then used to estimate the treatment effect using propensity score matching (PSM). Through simulations on various datasets and a real-world Electrohydrodynamic (EHD) printing data, we demonstrate that our approach outperforms standard Bayesian causal inference methods and several state-of-the-art federated learning benchmarks.

Title: Explaining k-Nearest Neighbors: Abductive and Counterfactual Explanations

• Authors: Pablo Barceló, Alexander Kozachinskiy, Miguel Romero Orth, Bernardo Subercaseaux, José Verschae
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06078
• Pdf URL: https://arxiv.org/pdf/2501.06078
• Copy Paste: [[2501.06078]] Explaining k-Nearest Neighbors: Abductive and Counterfactual Explanations(https://arxiv.org/abs/2501.06078)
• Keywords: interpretability, explainability
• Abstract: Despite the wide use of $k$-Nearest Neighbors as classification models, their explainability properties remain poorly understood from a theoretical perspective. While nearest neighbors classifiers offer interpretability from a "data perspective", in which the classification of an input vector $\bar{x}$ is explained by identifying the vectors $\bar{v}_1, \ldots, \bar{v}_k$ in the training set that determine the classification of $\bar{x}$, we argue that such explanations can be impractical in high-dimensional applications, where each vector has hundreds or thousands of features and it is not clear what their relative importance is. Hence, we focus on understanding nearest neighbor classifications through a "feature perspective", in which the goal is to identify how the values of the features in $\bar{x}$ affect its classification. Concretely, we study abductive explanations such as "minimum sufficient reasons", which correspond to sets of features in $\bar{x}$ that are enough to guarantee its classification, and "counterfactual explanations" based on the minimum distance feature changes one would have to perform in $\bar{x}$ to change its classification. We present a detailed landscape of positive and negative complexity results for counterfactual and abductive explanations, distinguishing between discrete and continuous feature spaces, and considering the impact of the choice of distance function involved. Finally, we show that despite some negative complexity results, Integer Quadratic Programming and SAT solving allow for computing explanations in practice.

Title: Scale-up Unlearnable Examples Learning with High-Performance Computing

• Authors: Yanfan Zhu, Issac Lyngaas, Murali Gopalakrishnan Meena, Mary Ellen I. Koran, Bradley Malin, Daniel Moyer, Shunxing Bao, Anuj Kapadia, Xiao Wang, Bennett Landman, Yuankai Huo
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2501.06080
• Pdf URL: https://arxiv.org/pdf/2501.06080
• Copy Paste: [[2501.06080]] Scale-up Unlearnable Examples Learning with High-Performance Computing(https://arxiv.org/abs/2501.06080)
• Keywords: security, privacy, protect, robust
• Abstract: Recent advancements in AI models are structured to retain user interactions, which could inadvertently include sensitive healthcare data. In the healthcare field, particularly when radiologists use AI-driven diagnostic tools hosted on online platforms, there is a risk that medical imaging data may be repurposed for future AI training without explicit consent, spotlighting critical privacy and intellectual property concerns around healthcare data usage. Addressing these privacy challenges, a novel approach known as Unlearnable Examples (UEs) has been introduced, aiming to make data unlearnable to deep learning models. A prominent method within this area, called Unlearnable Clustering (UC), has shown improved UE performance with larger batch sizes but was previously limited by computational resources. To push the boundaries of UE performance with theoretically unlimited resources, we scaled up UC learning across various datasets using Distributed Data Parallel (DDP) training on the Summit supercomputer. Our goal was to examine UE efficacy at high-performance computing (HPC) levels to prevent unauthorized learning and enhance data security, particularly exploring the impact of batch size on UE's unlearnability. Utilizing the robust computational capabilities of the Summit, extensive experiments were conducted on diverse datasets such as Pets, MedMNist, Flowers, and Flowers102. Our findings reveal that both overly large and overly small batch sizes can lead to performance instability and affect accuracy. However, the relationship between batch size and unlearnability varied across datasets, highlighting the necessity for tailored batch size strategies to achieve optimal data protection. Our results underscore the critical role of selecting appropriate batch sizes based on the specific characteristics of each dataset to prevent learning and ensure data security in deep learning applications.

Title: Multi-layered Authentication and Key Management Scheme for Secure IoV

• Authors: Morteza Azmoudeh Afshar, Nesrine Benchoubane, Busra Cayoren, Gunes Karabulut Kurt, Enver Ozdemir
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.06087
• Pdf URL: https://arxiv.org/pdf/2501.06087
• Copy Paste: [[2501.06087]] Multi-layered Authentication and Key Management Scheme for Secure IoV(https://arxiv.org/abs/2501.06087)
• Keywords: secure, security, privacy, protect, attack, robust
• Abstract: The rapid development of Vehicular Ad-hoc Networks (VANETs) within the Internet of Vehicles (IoV) necessitates efficient and secure authentication methods to support high-speed, high-density environments. Current group authentication schemes provide user identity protection and unlinkability but face limitations, such as reliance on a central group manager and vulnerability to collaborative attacks. This paper presents a privacy-preserving authentication scheme that incorporates batch authentication, mutual authentication, and secure key establishment, enabling users to authenticate one another without a central authority. Our proposed scheme facilitates simultaneous multi-user authentication, significantly enhancing scalability and security in dynamic IoV networks. Results from realistic implementations show that our method achieves average authentication and verification times of 10.61 ms and 1.78 ms, respectively, for a fleet of 100 vehicles, outperforming existing methods. Scalability tests demonstrate efficient processing for larger groups of up to 500 vehicles, where average authentication times remain low, establishing our scheme as a robust solution for secure communication in IoV systems.

Title: Explaining Deep Learning-based Anomaly Detection in Energy Consumption Data by Focusing on Contextually Relevant Data

• Authors: Mohammad Noorchenarboo, Katarina Grolinger
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06099
• Pdf URL: https://arxiv.org/pdf/2501.06099
• Copy Paste: [[2501.06099]] Explaining Deep Learning-based Anomaly Detection in Energy Consumption Data by Focusing on Contextually Relevant Data(https://arxiv.org/abs/2501.06099)
• Keywords: robust, explainability
• Abstract: Detecting anomalies in energy consumption data is crucial for identifying energy waste, equipment malfunction, and overall, for ensuring efficient energy management. Machine learning, and specifically deep learning approaches, have been greatly successful in anomaly detection; however, they are black-box approaches that do not provide transparency or explanations. SHAP and its variants have been proposed to explain these models, but they suffer from high computational complexity (SHAP) or instability and inconsistency (e.g., Kernel SHAP). To address these challenges, this paper proposes an explainability approach for anomalies in energy consumption data that focuses on context-relevant information. The proposed approach leverages existing explainability techniques, focusing on SHAP variants, together with global feature importance and weighted cosine similarity to select background dataset based on the context of each anomaly point. By focusing on the context and most relevant features, this approach mitigates the instability of explainability algorithms. Experimental results across 10 different machine learning models, five datasets, and five XAI techniques, demonstrate that our method reduces the variability of explanations providing consistent explanations. Statistical analyses confirm the robustness of our approach, showing an average reduction in variability of approximately 38% across multiple datasets.

Title: From Conversation to Automation: Leveraging Large Language Models to Analyze Strategies in Problem Solving Therapy

• Authors: Elham Aghakhani, Lu Wang, Karla T. Washington, George Demiris, Jina Huh-Yoo, Rezvaneh Rezapour
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.06101
• Pdf URL: https://arxiv.org/pdf/2501.06101
• Copy Paste: [[2501.06101]] From Conversation to Automation: Leveraging Large Language Models to Analyze Strategies in Problem Solving Therapy(https://arxiv.org/abs/2501.06101)
• Keywords: transformer, large language model
• Abstract: Problem-solving therapy (PST) is a structured psychological approach that helps individuals manage stress and resolve personal issues by guiding them through problem identification, solution brainstorming, decision-making, and outcome evaluation. As mental health care increasingly integrates technologies like chatbots and large language models (LLMs), understanding how PST can be effectively automated is important. This study leverages anonymized therapy transcripts to analyze and classify therapeutic interventions using various LLMs and transformer-based models. Our results show that GPT-4o achieved the highest accuracy (0.76) in identifying PST strategies, outperforming other models. Additionally, we introduced a new dimension of communication strategies that enhances the current PST framework, offering deeper insights into therapist-client interactions. This research demonstrates the potential of LLMs to automate complex therapeutic dialogue analysis, providing a scalable, efficient tool for mental health interventions. Our annotation framework can enhance the accessibility, effectiveness, and personalization of PST, supporting therapists in real-time with more precise, targeted interventions.

Title: Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding

• Authors: Fabian David Schmidt, Ivan Vulić, Goran Glavaš, David Ifeoluwa Adelani
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06117
• Pdf URL: https://arxiv.org/pdf/2501.06117
• Copy Paste: [[2501.06117]] Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding(https://arxiv.org/abs/2501.06117)
• Keywords: robust, large language model
• Abstract: While recent multilingual automatic speech recognition models claim to support thousands of languages, ASR for low-resource languages remains highly unreliable due to limited bimodal speech and text training data. Better multilingual spoken language understanding (SLU) can strengthen massively the robustness of multilingual ASR by levering language semantics to compensate for scarce training data, such as disambiguating utterances via context or exploiting semantic similarities across languages. Even more so, SLU is indispensable for inclusive speech technology in roughly half of all living languages that lack a formal writing system. However, the evaluation of multilingual SLU remains limited to shallower tasks such as intent classification or language identification. To address this, we present Fleurs-SLU, a multilingual SLU benchmark that encompasses topical speech classification in 102 languages and multiple-choice question answering through listening comprehension in 92 languages. We extensively evaluate both end-to-end speech classification models and cascaded systems that combine speech-to-text transcription with subsequent classification by large language models on Fleurs-SLU. Our results show that cascaded systems exhibit greater robustness in multilingual SLU tasks, though speech encoders can achieve competitive performance in topical speech classification when appropriately pre-trained. We further find a strong correlation between robust multilingual ASR, effective speech-to-text translation, and strong multilingual SLU, highlighting the mutual benefits between acoustic and semantic speech representations.

Title: Merging Feed-Forward Sublayers for Compressed Transformers

• Authors: Neha Verma, Kenton Murray, Kevin Duh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06126
• Pdf URL: https://arxiv.org/pdf/2501.06126
• Copy Paste: [[2501.06126]] Merging Feed-Forward Sublayers for Compressed Transformers(https://arxiv.org/abs/2501.06126)
• Keywords: transformer
• Abstract: With the rise and ubiquity of larger deep learning models, the need for high-quality compression techniques is growing in order to deploy these models widely. The sheer parameter count of these models makes it difficult to fit them into the memory constraints of different hardware. In this work, we present a novel approach to model compression by merging similar parameter groups within a model, rather than pruning away less important parameters. Specifically, we select, align, and merge separate feed-forward sublayers in Transformer models, and test our method on language modeling, image classification, and machine translation. With our method, we demonstrate performance comparable to the original models while combining more than a third of model feed-forward sublayers, and demonstrate improved performance over a strong layer-pruning baseline. For instance, we can remove over 21% of total parameters from a Vision Transformer, while maintaining 99% of its original performance. Additionally, we observe that some groups of feed-forward sublayers exhibit high activation similarity, which may help explain their surprising mergeability.

Title: Contextual ASR Error Handling with LLMs Augmentation for Goal-Oriented Conversational AI

• Authors: Yuya Asano, Sabit Hassan, Paras Sharma, Anthony Sicilia, Katherine Atwell, Diane Litman, Malihe Alikhani
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.06129
• Pdf URL: https://arxiv.org/pdf/2501.06129
• Copy Paste: [[2501.06129]] Contextual ASR Error Handling with LLMs Augmentation for Goal-Oriented Conversational AI(https://arxiv.org/abs/2501.06129)
• Keywords: large language model
• Abstract: General-purpose automatic speech recognition (ASR) systems do not always perform well in goal-oriented dialogue. Existing ASR correction methods rely on prior user data or named entities. We extend correction to tasks that have no prior user data and exhibit linguistic flexibility such as lexical and syntactic variations. We propose a novel context augmentation with a large language model and a ranking strategy that incorporates contextual information from the dialogue states of a goal-oriented conversational AI and its tasks. Our method ranks (1) n-best ASR hypotheses by their lexical and semantic similarity with context and (2) context by phonetic correspondence with ASR hypotheses. Evaluated in home improvement and cooking domains with real-world users, our method improves recall and F1 of correction by 34% and 16%, respectively, while maintaining precision and false positive rate. Users rated .8-1 point (out of 5) higher when our correction method worked properly, with no decrease due to false positives.

Title: MS-Temba : Multi-Scale Temporal Mamba for Efficient Temporal Action Detection

• Authors: Arkaprava Sinha, Monish Soundar Raj, Pu Wang, Ahmed Helmy, Srijan Das
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06138
• Pdf URL: https://arxiv.org/pdf/2501.06138
• Copy Paste: [[2501.06138]] MS-Temba : Multi-Scale Temporal Mamba for Efficient Temporal Action Detection(https://arxiv.org/abs/2501.06138)
• Keywords: transformer
• Abstract: Action detection in real-world scenarios is particularly challenging due to densely distributed actions in hour-long untrimmed videos. It requires modeling both short- and long-term temporal relationships while handling significant intra-class temporal variations. Previous state-of-the-art (SOTA) Transformer-based architectures, though effective, are impractical for real-world deployment due to their high parameter count, GPU memory usage, and limited throughput, making them unsuitable for very long videos. In this work, we innovatively adapt the Mamba architecture for action detection and propose Multi-scale Temporal Mamba (MS-Temba), comprising two key components: Temporal Mamba (Temba) Blocks and the Temporal Mamba Fuser. Temba Blocks include the Temporal Local Module (TLM) for short-range temporal modeling and the Dilated Temporal SSM (DTS) for long-range dependencies. By introducing dilations, a novel concept for Mamba, TLM and DTS capture local and global features at multiple scales. The Temba Fuser aggregates these scale-specific features using Mamba to learn comprehensive multi-scale representations of untrimmed videos. MS-Temba is validated on three public datasets, outperforming SOTA methods on long videos and matching prior methods on short videos while using only one-eighth of the parameters.

Title: Emergent Symbol-like Number Variables in Artificial Neural Networks

• Authors: Satchel Grant, Noah D. Goodman, James L. McClelland
• Subjects: cs.LG, cs.AI, cs.SC
• Abstract URL: https://arxiv.org/abs/2501.06141
• Pdf URL: https://arxiv.org/pdf/2501.06141
• Copy Paste: [[2501.06141]] Emergent Symbol-like Number Variables in Artificial Neural Networks(https://arxiv.org/abs/2501.06141)
• Keywords: transformer
• Abstract: What types of numeric representations emerge in Neural Networks (NNs)? To what degree do NNs induce abstract, mutable, slot-like numeric variables, and in what situations do these representations emerge? How do these representations change over learning, and how can we understand the neural implementations in ways that are unified across different NNs? In this work, we approach these questions by first training sequence based neural systems using Next Token Prediction (NTP) objectives on numeric tasks. We then seek to understand the neural solutions through the lens of causal abstractions or symbolic algorithms. We use a combination of causal interventions and visualization methods to find that artificial neural models do indeed develop analogs of interchangeable, mutable, latent number variables purely from the NTP objective. We then ask how variations on the tasks and model architectures affect the models' learned solutions to find that these symbol-like numeric representations do not form for every variant of the task, and transformers solve the problem in a notably different way than their recurrent counterparts. We then show how the symbol-like variables change over the course of training to find a strong correlation between the models' task performance and the alignment of their symbol-like representations. Lastly, we show that in all cases, some degree of gradience exists in these neural symbols, highlighting the difficulty of finding simple, interpretable symbolic stories of how neural networks perform numeric tasks. Taken together, our results are consistent with the view that neural networks can approximate interpretable symbolic programs of number cognition, but the particular program they approximate and the extent to which they approximate it can vary widely, depending on the network architecture, training data, extent of training, and network size.

Title: From discrete-time policies to continuous-time diffusion samplers: Asymptotic equivalences and faster training

• Authors: Julius Berner, Lorenz Richter, Marcin Sendera, Jarrid Rector-Brooks, Nikolay Malkin
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.06148
• Pdf URL: https://arxiv.org/pdf/2501.06148
• Copy Paste: [[2501.06148]] From discrete-time policies to continuous-time diffusion samplers: Asymptotic equivalences and faster training(https://arxiv.org/abs/2501.06148)
• Keywords: diffusion, generative
• Abstract: We study the problem of training neural stochastic differential equations, or diffusion models, to sample from a Boltzmann distribution without access to target samples. Existing methods for training such models enforce time-reversal of the generative and noising processes, using either differentiable simulation or off-policy reinforcement learning (RL). We prove equivalences between families of objectives in the limit of infinitesimal discretization steps, linking entropic RL methods (GFlowNets) with continuous-time objects (partial differential equations and path space measures). We further show that an appropriate choice of coarse time discretization during training allows greatly improved sample efficiency and the use of time-local objectives, achieving competitive performance on standard sampling benchmarks with reduced computational cost.

Title: GenMol: A Drug Discovery Generalist with Discrete Diffusion

• Authors: Seul Lee, Karsten Kreis, Srimukh Prasad Veccham, Meng Liu, Danny Reidenbach, Yuxing Peng, Saee Paliwal, Weili Nie, Arash Vahdat
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.06158
• Pdf URL: https://arxiv.org/pdf/2501.06158
• Copy Paste: [[2501.06158]] GenMol: A Drug Discovery Generalist with Discrete Diffusion(https://arxiv.org/abs/2501.06158)
• Keywords: diffusion, generative
• Abstract: Drug discovery is a complex process that involves multiple scenarios and stages, such as fragment-constrained molecule generation, hit generation and lead optimization. However, existing molecular generative models can only tackle one or two of these scenarios and lack the flexibility to address various aspects of the drug discovery pipeline. In this paper, we present Generalist Molecular generative model (GenMol), a versatile framework that addresses these limitations by applying discrete diffusion to the Sequential Attachment-based Fragment Embedding (SAFE) molecular representation. GenMol generates SAFE sequences through non-autoregressive bidirectional parallel decoding, thereby allowing utilization of a molecular context that does not rely on the specific token ordering and enhanced computational efficiency. Moreover, under the discrete diffusion framework, we introduce fragment remasking, a strategy that optimizes molecules by replacing fragments with masked tokens and regenerating them, enabling effective exploration of chemical space. GenMol significantly outperforms the previous GPT-based model trained on SAFE representations in de novo generation and fragment-constrained generation, and achieves state-of-the-art performance in goal-directed hit generation and lead optimization. These experimental results demonstrate that GenMol can tackle a wide range of drug discovery tasks, providing a unified and versatile approach for molecular design.

Title: RIOT-based smart metering system for privacy-preserving data aggregation using watermarking and encryption

• Authors: David Megias, Farzana Kabir, Krzysztof Cabaj
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2501.06161
• Pdf URL: https://arxiv.org/pdf/2501.06161
• Copy Paste: [[2501.06161]] RIOT-based smart metering system for privacy-preserving data aggregation using watermarking and encryption(https://arxiv.org/abs/2501.06161)
• Keywords: security, privacy, attack, watermark
• Abstract: The remarkable advancement of smart grid technology in the IoT sector has raised concerns over the privacy and security of the data collected and transferred in real-time. Smart meters generate detailed information about consumers' energy consumption patterns, increasing the risks of data breaches, identity theft, and other forms of cyber attacks. This study proposes a privacy-preserving data aggregation protocol that uses reversible watermarking and AES cryptography to ensure the security and privacy of the data. There are two versions of the protocol: one for low-frequency smart meters that uses LSB-shifting-based reversible watermarking (RLS) and another for high-frequency smart meters that uses difference expansion-based reversible watermarking (RDE). This enables the aggregation of smart meter data, maintaining confidentiality, integrity, and authenticity. The proposed protocol significantly enhances privacy-preserving measures for smart metering systems, conducting an experimental evaluation with real hardware implementation using Nucleo microcontroller boards and the RIOT operating system and comparing the results to existing security schemes.

Title: PEACE: Empowering Geologic Map Holistic Understanding with MLLMs

• Authors: Yangyu Huang, Tianyi Gao, Haoran Xu, Qihao Zhao, Yang Song, Zhipeng Gui, Tengchao Lv, Hao Chen, Lei Cui, Scarlett Li, Furu Wei
• Subjects: cs.CV, cs.MA
• Abstract URL: https://arxiv.org/abs/2501.06184
• Pdf URL: https://arxiv.org/pdf/2501.06184
• Copy Paste: [[2501.06184]] PEACE: Empowering Geologic Map Holistic Understanding with MLLMs(https://arxiv.org/abs/2501.06184)
• Keywords: extraction, large language model
• Abstract: Geologic map, as a fundamental diagram in geology science, provides critical insights into the structure and composition of Earth's subsurface and surface. These maps are indispensable in various fields, including disaster detection, resource exploration, and civil engineering. Despite their significance, current Multimodal Large Language Models (MLLMs) often fall short in geologic map understanding. This gap is primarily due to the challenging nature of cartographic generalization, which involves handling high-resolution map, managing multiple associated components, and requiring domain-specific knowledge. To quantify this gap, we construct GeoMap-Bench, the first-ever benchmark for evaluating MLLMs in geologic map understanding, which assesses the full-scale abilities in extracting, referring, grounding, reasoning, and analyzing. To bridge this gap, we introduce GeoMap-Agent, the inaugural agent designed for geologic map understanding, which features three modules: Hierarchical Information Extraction (HIE), Domain Knowledge Injection (DKI), and Prompt-enhanced Question Answering (PEQA). Inspired by the interdisciplinary collaboration among human scientists, an AI expert group acts as consultants, utilizing a diverse tool pool to comprehensively analyze questions. Through comprehensive experiments, GeoMap-Agent achieves an overall score of 0.811 on GeoMap-Bench, significantly outperforming 0.369 of GPT-4o. Our work, emPowering gEologic mAp holistiC undErstanding (PEACE) with MLLMs, paves the way for advanced AI applications in geology, enhancing the efficiency and accuracy of geological investigations.

Title: LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs

• Authors: Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, Hisham Cholakkal, Ivan Laptev, Mubarak Shah, Fahad Shahbaz Khan, Salman Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06186
• Pdf URL: https://arxiv.org/pdf/2501.06186
• Copy Paste: [[2501.06186]] LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs(https://arxiv.org/abs/2501.06186)
• Keywords: robust, large language model
• Abstract: Reasoning is a fundamental capability for solving complex multi-step problems, particularly in visual contexts where sequential step-wise understanding is essential. Existing approaches lack a comprehensive framework for evaluating visual reasoning and do not emphasize step-wise problem-solving. To this end, we propose a comprehensive framework for advancing step-by-step visual reasoning in large language models (LMMs) through three key contributions. First, we introduce a visual reasoning benchmark specifically designed to evaluate multi-step reasoning tasks. The benchmark presents a diverse set of challenges with eight different categories ranging from complex visual perception to scientific reasoning with over 4k reasoning steps in total, enabling robust evaluation of LLMs' abilities to perform accurate and interpretable visual reasoning across multiple steps. Second, we propose a novel metric that assesses visual reasoning quality at the granularity of individual steps, emphasizing both correctness and logical coherence. The proposed metric offers deeper insights into reasoning performance compared to traditional end-task accuracy metrics. Third, we present a new multimodal visual reasoning model, named LlamaV-o1, trained using a multi-step curriculum learning approach, where tasks are progressively organized to facilitate incremental skill acquisition and problem-solving. The proposed LlamaV-o1 is designed for multi-step reasoning and learns step-by-step through a structured training paradigm. Extensive experiments show that our LlamaV-o1 outperforms existing open-source models and performs favorably against close-source proprietary models. Compared to the recent Llava-CoT, our LlamaV-o1 achieves an average score of 67.3 with an absolute gain of 3.8\% across six benchmarks while being 5 times faster during inference scaling. Our benchmark, model, and code are publicly available.

Title: Multi-subject Open-set Personalization in Video Generation

• Authors: Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Yuwei Fang, Kwot Sin Lee, Ivan Skorokhodov, Kfir Aberman, Jun-Yan Zhu, Ming-Hsuan Yang, Sergey Tulyakov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.06187
• Pdf URL: https://arxiv.org/pdf/2501.06187
• Copy Paste: [[2501.06187]] Multi-subject Open-set Personalization in Video Generation(https://arxiv.org/abs/2501.06187)
• Keywords: diffusion, transformer
• Abstract: Video personalization methods allow us to synthesize videos with specific concepts such as people, pets, and places. However, existing methods often focus on limited domains, require time-consuming optimization per subject, or support only a single subject. We present Video Alchemist $-$ a video model with built-in multi-subject, open-set personalization capabilities for both foreground objects and background, eliminating the need for time-consuming test-time optimization. Our model is built on a new Diffusion Transformer module that fuses each conditional reference image and its corresponding subject-level text prompt with cross-attention layers. Developing such a large model presents two main challenges: dataset and evaluation. First, as paired datasets of reference images and videos are extremely hard to collect, we sample selected video frames as reference images and synthesize a clip of the target video. However, while models can easily denoise training videos given reference frames, they fail to generalize to new contexts. To mitigate this issue, we design a new automatic data construction pipeline with extensive image augmentations. Second, evaluating open-set video personalization is a challenge in itself. To address this, we introduce a personalization benchmark that focuses on accurate subject fidelity and supports diverse personalization scenarios. Finally, our extensive experiments show that our method significantly outperforms existing personalization methods in both quantitative and qualitative evaluations.