2024-12-11

Title: Poison Attacks and Adversarial Prompts Against an Informed University Virtual Assistant

• Authors: Ivan A. Fernandez, Subash Neupane, Sudip Mittal, Shahram Rahimi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.06788
• Pdf URL: https://arxiv.org/pdf/2412.06788
• Copy Paste: [[2412.06788]] Poison Attacks and Adversarial Prompts Against an Informed University Virtual Assistant(https://arxiv.org/abs/2412.06788)
• Keywords: security, attack, generative, large language model
• Abstract: Recent research has shown that large language models (LLMs) are particularly vulnerable to adversarial attacks. Since the release of ChatGPT, various industries are adopting LLM-based chatbots and virtual assistants in their data workflows. The rapid development pace of AI-based systems is being driven by the potential of Generative AI (GenAI) to assist humans in decision making. The immense optimism behind GenAI often overshadows the adversarial risks associated with these technologies. A threat actor can use security gaps, poor safeguards, and limited data governance to carry out attacks that grant unauthorized access to the system and its data. As a proof-of-concept, we assess the performance of BarkPlug, the Mississippi State University chatbot, against data poison attacks from a red team perspective.

Title: Federated Block-Term Tensor Regression for decentralised data analysis in healthcare

• Authors: Axel Faes, Ashkan Pirmani, Yves Moreau, Liesbet M. Peeters
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06815
• Pdf URL: https://arxiv.org/pdf/2412.06815
• Copy Paste: [[2412.06815]] Federated Block-Term Tensor Regression for decentralised data analysis in healthcare(https://arxiv.org/abs/2412.06815)
• Keywords: privacy, federate
• Abstract: Block-Term Tensor Regression (BTTR) has proven to be a powerful tool for modeling complex, high-dimensional data by leveraging multilinear relationships, making it particularly well-suited for applications in healthcare and neuroscience. However, traditional implementations of BTTR rely on centralized datasets, which pose significant privacy risks and hinder collaboration across institutions. To address these challenges, we introduce Federated Block-Term Tensor Regression (FBTTR), an extension of BTTR designed for federated learning scenarios. FBTTR enables decentralized data analysis, allowing institutions to collaboratively build predictive models while preserving data privacy and complying with regulations. FBTTR represents a major step forward in applying tensor regression to federated learning environments. Its performance is evaluated in two case studies: finger movement decoding from Electrocorticography (ECoG) signals and heart disease prediction. In the first case study, using the BCI Competition IV dataset, FBTTR outperforms non-multilinear models, demonstrating superior accuracy in decoding finger movements. For the dataset, for subject 3, the thumb obtained a performance of 0.76 $\pm$ .05 compared to 0.71 $\pm$ 0.05 for centralised BTTR. In the second case study, FBTTR is applied to predict heart disease using real-world clinical datasets, outperforming both standard federated learning approaches and centralized BTTR models. In the Fed-Heart-Disease Dataset, an AUC-ROC was obtained of 0.872 $\pm$ 0.02 and an accuracy of 0.772 $\pm$ 0.02 compared to 0.812 $\pm$ 0.003 and 0.753 $\pm$ 0.007 for the centralized model.

Title: Guidance is All You Need: Temperature-Guided Reasoning in Large Language Models

• Authors: Eyad Gomaa, Gomaa Salah
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06822
• Pdf URL: https://arxiv.org/pdf/2412.06822
• Copy Paste: [[2412.06822]] Guidance is All You Need: Temperature-Guided Reasoning in Large Language Models(https://arxiv.org/abs/2412.06822)
• Keywords: large language model
• Abstract: We present Quasar-1, a novel architecture that introduces temperature-guided reasoning to large language models through the Token Temperature Mechanism (TTM) and Guided Sequence of Thought (GSoT). Our approach leverages the concept of hot and cold tokens, where hot tokens are prioritized for their contextual relevance, while cold tokens provide supplementary information. This dynamic modulation of token importance enables the model to achieve superior logical reasoning capabilities compared to traditional chain-of-thought approaches. Through rigorous mathematical analysis, we prove that our temperature-guided attention mechanism converges to optimal reasoning paths with exponential guarantees. Empirical results show significant improvements in reasoning accuracy and computational efficiency across a wide range of tasks, making advanced AI reasoning accessible to a broader range of applications.

Title: Feature Group Tabular Transformer: A Novel Approach to Traffic Crash Modeling and Causality Analysis

• Authors: Oscar Lares, Hao Zhen, Jidong J. Yang
• Subjects: cs.LG, cs.AI, stat.AP
• Abstract URL: https://arxiv.org/abs/2412.06825
• Pdf URL: https://arxiv.org/pdf/2412.06825
• Copy Paste: [[2412.06825]] Feature Group Tabular Transformer: A Novel Approach to Traffic Crash Modeling and Causality Analysis(https://arxiv.org/abs/2412.06825)
• Keywords: transformer
• Abstract: Reliable and interpretable traffic crash modeling is essential for understanding causality and improving road safety. This study introduces a novel approach to predicting collision types by utilizing a comprehensive dataset fused from multiple sources, including weather data, crash reports, high-resolution traffic information, pavement geometry, and facility characteristics. Central to our approach is the development of a Feature Group Tabular Transformer (FGTT) model, which organizes disparate data into meaningful feature groups, represented as tokens. These group-based tokens serve as rich semantic components, enabling effective identification of collision patterns and interpretation of causal mechanisms. The FGTT model is benchmarked against widely used tree ensemble models, including Random Forest, XGBoost, and CatBoost, demonstrating superior predictive performance. Furthermore, model interpretation reveals key influential factors, providing fresh insights into the underlying causality of distinct crash types.

Title: Enhancing LLMs for Physics Problem-Solving using Reinforcement Learning with Human-AI Feedback

• Authors: Avinash Anand, Kritarth Prasad, Chhavi Kirtani, Ashwin R Nair, Mohit Gupta, Saloni Garg, Anurag Gautam, Snehal Buldeo, Rajiv Ratn Shah
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06827
• Pdf URL: https://arxiv.org/pdf/2412.06827
• Copy Paste: [[2412.06827]] Enhancing LLMs for Physics Problem-Solving using Reinforcement Learning with Human-AI Feedback(https://arxiv.org/abs/2412.06827)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated strong capabilities in text-based tasks but struggle with the complex reasoning required for physics problems, particularly in advanced arithmetic and conceptual understanding. While some research has explored ways to enhance LLMs in physics education using techniques such as prompt engineering and Retrieval Augmentation Generation (RAG), not enough effort has been made in addressing their limitations in physics reasoning. This paper presents a novel approach to improving LLM performance on physics questions using Reinforcement Learning with Human and Artificial Intelligence Feedback (RLHAIF). We evaluate several reinforcement learning methods, including Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), and Remax optimization. These methods are chosen to investigate RL policy performance with different settings on the PhyQA dataset, which includes challenging physics problems from high school textbooks. Our RLHAIF model, tested on leading LLMs like LLaMA2 and Mistral, achieved superior results, notably with the MISTRAL-PPO model, demonstrating marked improvements in reasoning and accuracy. It achieved high scores, with a 58.67 METEOR score and a 0.74 Reasoning score, making it a strong example for future physics reasoning research in this area.

Title: Enhancing LLMs for Impression Generation in Radiology Reports through a Multi-Agent System

• Authors: Fang Zeng, Zhiliang Lyu, Quanzheng Li, Xiang Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06828
• Pdf URL: https://arxiv.org/pdf/2412.06828
• Copy Paste: [[2412.06828]] Enhancing LLMs for Impression Generation in Radiology Reports through a Multi-Agent System(https://arxiv.org/abs/2412.06828)
• Keywords: robust, large language model
• Abstract: This study introduces "RadCouncil," a multi-agent Large Language Model (LLM) framework designed to enhance the generation of impressions in radiology reports from the finding section. RadCouncil comprises three specialized agents: 1) a "Retrieval" Agent that identifies and retrieves similar reports from a vector database, 2) a "Radiologist" Agent that generates impressions based on the finding section of the given report plus the exemplar reports retrieved by the Retrieval Agent, and 3) a "Reviewer" Agent that evaluates the generated impressions and provides feedback. The performance of RadCouncil was evaluated using both quantitative metrics (BLEU, ROUGE, BERTScore) and qualitative criteria assessed by GPT-4, using chest X-ray as a case study. Experiment results show improvements in RadCouncil over the single-agent approach across multiple dimensions, including diagnostic accuracy, stylistic concordance, and clarity. This study highlights the potential of utilizing multiple interacting LLM agents, each with a dedicated task, to enhance performance in specialized medical tasks and the development of more robust and adaptable healthcare AI solutions.

Title: TransitGPT: A Generative AI-based framework for interacting with GTFS data using Large Language Models

• Authors: Saipraneeth Devunuri, Lewis Lehe
• Subjects: cs.CL, cs.AI, stat.AP
• Abstract URL: https://arxiv.org/abs/2412.06831
• Pdf URL: https://arxiv.org/pdf/2412.06831
• Copy Paste: [[2412.06831]] TransitGPT: A Generative AI-based framework for interacting with GTFS data using Large Language Models(https://arxiv.org/abs/2412.06831)
• Keywords: generative, large language model
• Abstract: This paper introduces a framework that leverages Large Language Models (LLMs) to answer natural language queries about General Transit Feed Specification (GTFS) data. The framework is implemented in a chatbot called TransitGPT with open-source code. TransitGPT works by guiding LLMs to generate Python code that extracts and manipulates GTFS data relevant to a query, which is then executed on a server where the GTFS feed is stored. It can accomplish a wide range of tasks, including data retrieval, calculations, and interactive visualizations, without requiring users to have extensive knowledge of GTFS or programming. The LLMs that produce the code are guided entirely by prompts, without fine-tuning or access to the actual GTFS feeds. We evaluate TransitGPT using GPT-4o and Claude-3.5-Sonnet LLMs on a benchmark dataset of 100 tasks, to demonstrate its effectiveness and versatility. The results show that TransitGPT can significantly enhance the accessibility and usability of transit data.

Title: APS-LSTM: Exploiting Multi-Periodicity and Diverse Spatial Dependencies for Flood Forecasting

• Authors: Jun Feng, Xueyi Liu, Jiamin Lu, Pingping Shao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06835
• Pdf URL: https://arxiv.org/pdf/2412.06835
• Copy Paste: [[2412.06835]] APS-LSTM: Exploiting Multi-Periodicity and Diverse Spatial Dependencies for Flood Forecasting(https://arxiv.org/abs/2412.06835)
• Keywords: extraction
• Abstract: Accurate flood prediction is crucial for disaster prevention and mitigation. Hydrological data exhibit highly nonlinear temporal patterns and encompass complex spatial relationships between rainfall and flow. Existing flood prediction models struggle to capture these intricate temporal features and spatial dependencies. This paper presents an adaptive periodic and spatial self-attention method based on LSTM (APS-LSTM) to address these challenges. The APS-LSTM learns temporal features from a multi-periodicity perspective and captures diverse spatial dependencies from different period divisions. The APS-LSTM consists of three main stages, (i) Multi-Period Division, that utilizes Fast Fourier Transform (FFT) to divide various periodic patterns; (ii) Spatio-Temporal Information Extraction, that performs periodic and spatial self-attention focusing on intra- and inter-periodic temporal patterns and spatial dependencies; (iii) Adaptive Aggregation, that relies on amplitude strength to aggregate the computational results from each periodic division. The abundant experiments on two real-world datasets demonstrate the superiority of APS-LSTM. The code is available: this https URL.

Title: Innovative Sentiment Analysis and Prediction of Stock Price Using FinBERT, GPT-4 and Logistic Regression: A Data-Driven Approach

• Authors: Olamilekan Shobayo, Sidikat Adeyemi-Longe, Olusogo Popoola, Bayode Ogunleye
• Subjects: cs.LG, cs.AI, q-fin.ST, stat.AP, stat.CO
• Abstract URL: https://arxiv.org/abs/2412.06837
• Pdf URL: https://arxiv.org/pdf/2412.06837
• Copy Paste: [[2412.06837]] Innovative Sentiment Analysis and Prediction of Stock Price Using FinBERT, GPT-4 and Logistic Regression: A Data-Driven Approach(https://arxiv.org/abs/2412.06837)
• Keywords: robust, transformer
• Abstract: This study explores the comparative performance of cutting-edge AI models, i.e., Finaance Bidirectional Encoder representations from Transsformers (FinBERT), Generatice Pre-trained Transformer GPT-4, and Logistic Regression, for sentiment analysis and stock index prediction using financial news and the NGX All-Share Index data label. By leveraging advanced natural language processing models like GPT-4 and FinBERT, alongside a traditional machine learning model, Logistic Regression, we aim to classify market sentiment, generate sentiment scores, and predict market price movements. This research highlights global AI advancements in stock markets, showcasing how state-of-the-art language models can contribute to understanding complex financial data. The models were assessed using metrics such as accuracy, precision, recall, F1 score, and ROC AUC. Results indicate that Logistic Regression outperformed the more computationally intensive FinBERT and predefined approach of versatile GPT-4, with an accuracy of 81.83% and a ROC AUC of 89.76%. The GPT-4 predefined approach exhibited a lower accuracy of 54.19% but demonstrated strong potential in handling complex data. FinBERT, while offering more sophisticated analysis, was resource-demanding and yielded a moderate performance. Hyperparameter optimization using Optuna and cross-validation techniques ensured the robustness of the models. This study highlights the strengths and limitations of the practical applications of AI approaches in stock market prediction and presents Logistic Regression as the most efficient model for this task, with FinBERT and GPT-4 representing emerging tools with potential for future exploration and innovation in AI-driven financial analytics

Title: MDiFF: Exploiting Multimodal Score-based Diffusion Models for New Fashion Product Performance Forecasting

• Authors: Andrea Avogaro, Luigi Capogrosso, Franco Fummi, Marco Cristani
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06840
• Pdf URL: https://arxiv.org/pdf/2412.06840
• Copy Paste: [[2412.06840]] MDiFF: Exploiting Multimodal Score-based Diffusion Models for New Fashion Product Performance Forecasting(https://arxiv.org/abs/2412.06840)
• Keywords: diffusion
• Abstract: The fast fashion industry suffers from significant environmental impacts due to overproduction and unsold inventory. Accurately predicting sales volumes for unreleased products could significantly improve efficiency and resource utilization. However, predicting performance for entirely new items is challenging due to the lack of historical data and rapidly changing trends, and existing deterministic models often struggle with domain shifts when encountering items outside the training data distribution. The recently proposed diffusion models address this issue using a continuous-time diffusion process. This allows us to simulate how new items are adopted, reducing the impact of domain shift challenges faced by deterministic models. As a result, in this paper, we propose MDiFF: a novel two-step multimodal diffusion models-based pipeline for New Fashion Product Performance Forecasting (NFPPF). First, we use a score-based diffusion model to predict multiple future sales for different clothes over time. Then, we refine these multiple predictions with a lightweight Multi-layer Perceptron (MLP) to get the final forecast. MDiFF leverages the strengths of both architectures, resulting in the most accurate and efficient forecasting system for the fast-fashion industry at the state-of-the-art. The code can be found at this https URL.

Title: Semantic loss guided data efficient supervised fine tuning for Safe Responses in LLMs

• Authors: Yuxiao Lu, Arunesh Sinha, Pradeep Varakantham
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06843
• Pdf URL: https://arxiv.org/pdf/2412.06843
• Copy Paste: [[2412.06843]] Semantic loss guided data efficient supervised fine tuning for Safe Responses in LLMs(https://arxiv.org/abs/2412.06843)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) generating unsafe responses to toxic prompts is a significant issue in their applications. While various efforts aim to address this safety concern, previous approaches often demand substantial human data collection or rely on the less dependable option of using another LLM to generate corrective data. In this paper, we aim to take this problem and overcome limitations of requiring significant high-quality human data. Our method requires only a small set of unsafe responses to toxic prompts, easily obtained from the unsafe LLM itself. By employing a semantic cost combined with a negative Earth Mover Distance (EMD) loss, we guide the LLM away from generating unsafe responses. Additionally, we propose a novel lower bound for EMD loss, enabling more efficient optimization. Our results demonstrate superior performance and data efficiency compared to baselines, and we further examine the nuanced effects of over-alignment and potential degradation of language capabilities when using contrastive data.

Title: Fully Open Source Moxin-7B Technical Report

• Authors: Pu Zhao, Xuan Shen, Zhenglun Kong, Yixin Shen, Sung-En Chang, Timothy Rupprecht, Lei Lu, Enfu Nan, Changdi Yang, Yumei He, Xingchen Xu, Yu Huang, Wei Wang, Yue Chen, Yong He, Yanzhi Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06845
• Pdf URL: https://arxiv.org/pdf/2412.06845
• Copy Paste: [[2412.06845]] Fully Open Source Moxin-7B Technical Report(https://arxiv.org/abs/2412.06845)
• Keywords: large language model
• Abstract: Recently, Large Language Models (LLMs) have undergone a significant transformation, marked by a rapid rise in both their popularity and capabilities. Leading this evolution are proprietary LLMs like GPT-4 and GPT-o1, which have captured widespread attention in the AI community due to their remarkable performance and versatility. Simultaneously, open-source LLMs, such as LLaMA and Mistral, have made great contributions to the ever-increasing popularity of LLMs due to the ease to customize and deploy the models across diverse applications. Although open-source LLMs present unprecedented opportunities for innovation and research, the commercialization of LLMs has raised concerns about transparency, reproducibility, and safety. Many open-source LLMs fail to meet fundamental transparency requirements by withholding essential components like training code and data, and some use restrictive licenses whilst claiming to be "open-source," which may hinder further innovations on LLMs. To mitigate this issue, we introduce Moxin 7B, a fully open-source LLM developed in accordance with the Model Openness Framework (MOF), a ranked classification system that evaluates AI models based on model completeness and openness, adhering to principles of open science, open source, open data, and open access. Our model achieves the highest MOF classification level of "open science" through the comprehensive release of pre-training code and configurations, training and fine-tuning datasets, and intermediate and final checkpoints. Experiments show that our model achieves superior performance in zero-shot evaluation compared with popular 7B models and performs competitively in few-shot evaluation.

Title: GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model

• Authors: Haotong Yang, Xiyuan Wang, Qian Tao, Shuxian Hu, Zhouchen Lin, Muhan Zhang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06849
• Pdf URL: https://arxiv.org/pdf/2412.06849
• Copy Paste: [[2412.06849]] GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model(https://arxiv.org/abs/2412.06849)
• Keywords: transformer, large language model
• Abstract: Recent research on integrating Large Language Models (LLMs) with Graph Neural Networks (GNNs) typically follows two approaches: LLM-centered models, which convert graph data into tokens for LLM processing, and GNN-centered models, which use LLMs to encode text features into node and edge representations for GNN input. LLM-centered models often struggle to capture graph structures effectively, while GNN-centered models compress variable-length textual data into fixed-size vectors, limiting their ability to understand complex semantics. Additionally, GNN-centered approaches require converting tasks into a uniform, manually-designed format, restricting them to classification tasks and preventing language output. To address these limitations, we introduce a new architecture that deeply integrates GNN with LLM, featuring three key innovations: (1) Structure-Aware Transformers, which incorporate GNN's message-passing capabilities directly into LLM's transformer layers, allowing simultaneous processing of textual and structural information and generating outputs from both GNN and LLM; (2) Graph-Text Cross-Attention, which processes full, uncompressed text from graph nodes and edges, ensuring complete semantic integration; and (3) GNN-LLM Twin Predictor, enabling LLM's flexible autoregressive generation alongside GNN's scalable one-pass prediction. GL-Fusion achieves outstand performance on various tasks. Notably, it achieves state-of-the-art performance on OGBN-Arxiv and OGBG-Code2.

Title: EGEAN: An Exposure-Guided Embedding Alignment Network for Post-Click Conversion Estimation

• Authors: Huajian Feng, Guoxiao Zhang, Yadong Zhang, Yi We, Qiang Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06852
• Pdf URL: https://arxiv.org/pdf/2412.06852
• Copy Paste: [[2412.06852]] EGEAN: An Exposure-Guided Embedding Alignment Network for Post-Click Conversion Estimation(https://arxiv.org/abs/2412.06852)
• Keywords: robust
• Abstract: Accurate post-click conversion rate (CVR) estimation is crucial for online advertising systems. Despite significant advances in causal approaches designed to address the Sample Selection Bias problem, CVR estimation still faces challenges due to Covariate Shift. Given the intrinsic connection between the distribution of covariates in the click and non-click spaces, this study proposes an Exposure-Guided Embedding Alignment Network (EGEAN) to address estimation bias caused by covariate shift. Additionally, we propose a Parameter Varying Doubly Robust Estimator with steady-state control to handle small propensities better. Online A/B tests conducted on the Meituan advertising system demonstrate that our method significantly outperforms baseline models with respect to CVR and GMV, validating its effectiveness. Code is available: this https URL.

Title: Comb Tensor Networks vs. Matrix Product States: Enhanced Efficiency in High-Dimensional Spaces

• Authors: Danylo Kolesnyk, Yelyzaveta Vodovozova
• Subjects: cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2412.06857
• Pdf URL: https://arxiv.org/pdf/2412.06857
• Copy Paste: [[2412.06857]] Comb Tensor Networks vs. Matrix Product States: Enhanced Efficiency in High-Dimensional Spaces(https://arxiv.org/abs/2412.06857)
• Keywords: generative
• Abstract: Modern approaches to generative modeling of continuous data using tensor networks incorporate compression layers to capture the most meaningful features of high-dimensional inputs. These methods, however, rely on traditional Matrix Product States (MPS) architectures. Here, we demonstrate that beyond a certain threshold in data and bond dimensions, a comb-shaped tensor network architecture can yield more efficient contractions than a standard MPS. This finding suggests that for continuous and high-dimensional data distributions, transitioning from MPS to a comb tensor network representation can substantially reduce computational overhead while maintaining accuracy.

Title: Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization

• Authors: Dongwei Wang, Huanrui Yang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06858
• Pdf URL: https://arxiv.org/pdf/2412.06858
• Copy Paste: [[2412.06858]] Taming Sensitive Weights : Noise Perturbation Fine-tuning for Robust LLM Quantization(https://arxiv.org/abs/2412.06858)
• Keywords: robust
• Abstract: Quantization is a critical step to enable efficient LLM serving under limited resource. However, previous research observes that certain weights in the LLM, known as outliers, are significantly sensitive to quantization noises. Existing quantization methods leave these outliers as floating points or higher precisions to retain performance, posting challenges on the efficient hardware deployment of the mixed-precision model. This work investigates an alternative way to tame the sensitive weights' impact on the quantization error, by reducing the loss Hessian trace with respect to outliers through an efficient fine-tuning process. We propose Noise Perturbation Fine-tuning (NPFT), which identifies outlier weights and add random weight perturbations on the outliers as the model going through a PEFT optimization. NPFT tames the sensitivity of outlier weights so that the quantized model performance can be improved without special treatment to the outliers. When applied to OPT and LLaMA models, our NPFT method achieves stable performance improvements for both uniform and non-uniform quantizers, while also offering better inference efficiency. Notably, the simplest RTN can achieve performance on par with GPTQ using our NPFT on LLaMA2-7B-4bits benchmark.

Title: Generating floorplans for various building functionalities via latent diffusion model

• Authors: Mohamed R. Ibrahim, Josef Musil, Irene Gallou
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06859
• Pdf URL: https://arxiv.org/pdf/2412.06859
• Copy Paste: [[2412.06859]] Generating floorplans for various building functionalities via latent diffusion model(https://arxiv.org/abs/2412.06859)
• Keywords: diffusion, generative
• Abstract: In the domain of architectural design, the foundational essence of creativity and human intelligence lies in the mastery of solving floorplans, a skill demanding distinctive expertise and years of experience. Traditionally, the architectural design process of creating floorplans often requires substantial manual labour and architectural expertise. Even when relying on parametric design approaches, the process is limited based on the designer's ability to build a complex set of parameters to iteratively explore design alternatives. As a result, these approaches hinder creativity and limit discovery of an optimal solution. Here, we present a generative latent diffusion model that learns to generate floorplans for various building types based on building footprints and design briefs. The introduced model learns from the complexity of the inter-connections between diverse building types and the mutations of architectural designs. By harnessing the power of latent diffusion models, this research surpasses conventional limitations in the design process. The model's ability to learn from diverse building types means that it cannot only replicate existing designs but also produce entirely new configurations that fuse design elements in unexpected ways. This innovation introduces a new dimension of creativity into architectural design, allowing architects, urban planners and even individuals without specialised expertise to explore uncharted territories of form and function with speed and cost-effectiveness.

Title: Balancing Efficiency and Effectiveness: An LLM-Infused Approach for Optimized CTR Prediction

• Authors: Guoxiao Zhang, Yi Wei, Yadong Zhang, Huajian Feng, Qiang Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06860
• Pdf URL: https://arxiv.org/pdf/2412.06860
• Copy Paste: [[2412.06860]] Balancing Efficiency and Effectiveness: An LLM-Infused Approach for Optimized CTR Prediction(https://arxiv.org/abs/2412.06860)
• Keywords: large language model
• Abstract: Click-Through Rate (CTR) prediction is essential in online advertising, where semantic information plays a pivotal role in shaping user decisions and enhancing CTR effectiveness. Capturing and modeling deep semantic information, such as a user's preference for "Häagen-Dazs' HEAVEN strawberry light ice cream" due to its health-conscious and premium attributes, is challenging. Traditional semantic modeling often overlooks these intricate details at the user and item levels. To bridge this gap, we introduce a novel approach that models deep semantic information end-to-end, leveraging the comprehensive world knowledge capabilities of Large Language Models (LLMs). Our proposed LLM-infused CTR prediction framework(Multi-level Deep Semantic Information Infused CTR model via Distillation, MSD) is designed to uncover deep semantic insights by utilizing LLMs to extract and distill critical information into a smaller, more efficient model, enabling seamless end-to-end training and inference. Importantly, our framework is carefully designed to balance efficiency and effectiveness, ensuring that the model not only achieves high performance but also operates with optimal resource utilization. Online A/B tests conducted on the Meituan sponsored-search system demonstrate that our method significantly outperforms baseline models in terms of Cost Per Mile (CPM) and CTR, validating its effectiveness, scalability, and balanced approach in real-world applications.

Title: Stock Type Prediction Model Based on Hierarchical Graph Neural Network

• Authors: Jianhua Yao, Yuxin Dong, Jiajing Wang, Bingxing Wang, Hongye Zheng, Honglin Qin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06862
• Pdf URL: https://arxiv.org/pdf/2412.06862
• Copy Paste: [[2412.06862]] Stock Type Prediction Model Based on Hierarchical Graph Neural Network(https://arxiv.org/abs/2412.06862)
• Keywords: extraction
• Abstract: This paper introduces a novel approach to stock data analysis by employing a Hierarchical Graph Neural Network (HGNN) model that captures multi-level information and relational structures in the stock market. The HGNN model integrates stock relationship data and hierarchical attributes to predict stock types effectively. The paper discusses the construction of a stock industry relationship graph and the extraction of temporal information from historical price sequences. It also highlights the design of a graph convolution operation and a temporal attention aggregator to model the macro market state. The integration of these features results in a comprehensive stock prediction model that addresses the challenges of utilizing stock relationship data and modeling hierarchical attributes in the stock market.

Title: Political-LLM: Large Language Models in Political Science

• Authors: Lincan Li, Jiaqi Li, Catherine Chen, Fred Gui, Hongjia Yang, Chenxiao Yu, Zhengguang Wang, Jianing Cai, Junlong Aaron Zhou, Bolin Shen, Alex Qian, Weixin Chen, Zhongkai Xue, Lichao Sun, Lifang He, Hanjie Chen, Kaize Ding, Zijian Du, Fangzhou Mu, Jiaxin Pei, Jieyu Zhao, Swabha Swayamdipta, Willie Neiswanger, Hua Wei, Xiyang Hu, Shixiang Zhu, Tianlong Chen, Yingzhou Lu, Yang Shi, Lianhui Qin, Tianfan Fu, Zhengzhong Tu, Yuzhe Yang, Jaemin Yoo, Jiaheng Zhang, Ryan Rossi, Liang Zhan, Liang Zhao, Emilio Ferrara, Yan Liu, Furong Huang, Xiangliang Zhang, Lawrence Rothenberg, Shuiwang Ji, Philip S. Yu, Yue Zhao, Yushun Dong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06864
• Pdf URL: https://arxiv.org/pdf/2412.06864
• Copy Paste: [[2412.06864]] Political-LLM: Large Language Models in Political Science(https://arxiv.org/abs/2412.06864)
• Keywords: fair, generative, large language model
• Abstract: In recent years, large language models (LLMs) have been widely adopted in political science tasks such as election prediction, sentiment analysis, policy impact assessment, and misinformation detection. Meanwhile, the need to systematically understand how LLMs can further revolutionize the field also becomes urgent. In this work, we--a multidisciplinary team of researchers spanning computer science and political science--present the first principled framework termed Political-LLM to advance the comprehensive understanding of integrating LLMs into computational political science. Specifically, we first introduce a fundamental taxonomy classifying the existing explorations into two perspectives: political science and computational methodologies. In particular, from the political science perspective, we highlight the role of LLMs in automating predictive and generative tasks, simulating behavior dynamics, and improving causal inference through tools like counterfactual generation; from a computational perspective, we introduce advancements in data preparation, fine-tuning, and evaluation methods for LLMs that are tailored to political contexts. We identify key challenges and future directions, emphasizing the development of domain-specific datasets, addressing issues of bias and fairness, incorporating human expertise, and redefining evaluation criteria to align with the unique requirements of computational political science. Political-LLM seeks to serve as a guidebook for researchers to foster an informed, ethical, and impactful use of Artificial Intelligence in political science. Our online resource is available at: this http URL.

Title: Lossless Model Compression via Joint Low-Rank Factorization Optimization

• Authors: Boyang Zhang, Daning Cheng, Yunquan Zhang, Fangmin Liu, Jiake Tian
• Subjects: cs.LG, cs.AI, cs.CC
• Abstract URL: https://arxiv.org/abs/2412.06867
• Pdf URL: https://arxiv.org/pdf/2412.06867
• Copy Paste: [[2412.06867]] Lossless Model Compression via Joint Low-Rank Factorization Optimization(https://arxiv.org/abs/2412.06867)
• Keywords: robust
• Abstract: Low-rank factorization is a popular model compression technique that minimizes the error $\delta$ between approximated and original weight matrices. Despite achieving performances close to the original models when $\delta$ is optimized, a performance discrepancy remains due to the separate optimization processes for low-rank factorization and model performance, resulting in unavoidable losses. We address this issue by introducing a novel joint optimization strategy for lossless low-rank weight factorization, which, for the first time, enhances the model's performance beyond the original. Our approach begins with a theoretical analysis of the relationship between low-rank factorization and model optimization objectives, establishing a precise perturbation range for matrix factorization errors on model performance. This challenge is then reformulated as a numerical rank deficiency problem with inequality constraints and develop a joint objective that simultaneously addresses factorization error and model performance. Based on the above analysis, we propose two optimization algorithms: \textbf{a lossless optimization algorithm} that maximizes model accuracy while ensuring compression, and \textbf{a compact optimization algorithm} that minimizes model size while preserving performance. These algorithms do not require fine-tuning and can directly compress numerous deep models to achieve lossless results. Our methods demonstrate robust efficacy across various vision and language tasks. For example, the compressed model reduced by 70\% on ResNext50 outperforms the original. Our code will be made public.

Title: Predicting Subway Passenger Flows under Incident Situation with Causality

• Authors: Xiannan Huang, Shuhan Qiu, Quan Yuan, Chao Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06871
• Pdf URL: https://arxiv.org/pdf/2412.06871
• Copy Paste: [[2412.06871]] Predicting Subway Passenger Flows under Incident Situation with Causality(https://arxiv.org/abs/2412.06871)
• Keywords: interpretability
• Abstract: In the context of rail transit operations, real-time passenger flow prediction is essential; however, most models primarily focus on normal conditions, with limited research addressing incident situations. There are several intrinsic challenges associated with prediction during incidents, such as a lack of interpretability and data scarcity. To address these challenges, we propose a two-stage method that separates predictions under normal conditions and the causal effects of incidents. First, a normal prediction model is trained using data from normal situations. Next, the synthetic control method is employed to identify the causal effects of incidents, combined with placebo tests to determine significant levels of these effects. The significant effects are then utilized to train a causal effect prediction model, which can forecast the impact of incidents based on features of the incidents and passenger flows. During the prediction phase, the results from both the normal situation model and the causal effect prediction model are integrated to generate final passenger flow predictions during incidents. Our approach is validated using real-world data, demonstrating improved accuracy. Furthermore, the two-stage methodology enhances interpretability. By analyzing the causal effect prediction model, we can identify key influencing factors related to the effects of incidents and gain insights into their underlying mechanisms. Our work can assist subway system managers in estimating passenger flow affected by incidents and enable them to take proactive measures. Additionally, it can deepen researchers' understanding of the impact of incidents on subway passenger flows.

Title: LLMs for Generalizable Language-Conditioned Policy Learning under Minimal Data Requirements

• Authors: Thomas Pouplin, Katarzyna Kobalczyk, Hao Sun, Mihaela van der Schaar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06877
• Pdf URL: https://arxiv.org/pdf/2412.06877
• Copy Paste: [[2412.06877]] LLMs for Generalizable Language-Conditioned Policy Learning under Minimal Data Requirements(https://arxiv.org/abs/2412.06877)
• Keywords: large language model
• Abstract: To develop autonomous agents capable of executing complex, multi-step decision-making tasks as specified by humans in natural language, existing reinforcement learning approaches typically require expensive labeled datasets or access to real-time experimentation. Moreover, conventional methods often face difficulties in generalizing to unseen goals and states, thereby limiting their practical applicability. This paper presents TEDUO, a novel training pipeline for offline language-conditioned policy learning. TEDUO operates on easy-to-obtain, unlabeled datasets and is suited for the so-called in-the-wild evaluation, wherein the agent encounters previously unseen goals and states. To address the challenges posed by such data and evaluation settings, our method leverages the prior knowledge and instruction-following capabilities of large language models (LLMs) to enhance the fidelity of pre-collected offline data and enable flexible generalization to new goals and states. Empirical results demonstrate that the dual role of LLMs in our framework-as data enhancers and generalizers-facilitates both effective and data-efficient learning of generalizable language-conditioned policies.

Title: SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations

• Authors: Zhaorun Chen, Francesco Pinto, Minzhou Pan, Bo Li
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06878
• Pdf URL: https://arxiv.org/pdf/2412.06878
• Copy Paste: [[2412.06878]] SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations(https://arxiv.org/abs/2412.06878)
• Keywords: security, generative, large language model
• Abstract: With the rise of generative AI and rapid growth of high-quality video generation, video guardrails have become more crucial than ever to ensure safety and security across platforms. Current video guardrails, however, are either overly simplistic, relying on pure classification models trained on simple policies with limited unsafe categories, which lack detailed explanations, or prompting multimodal large language models (MLLMs) with long safety guidelines, which are inefficient and impractical for guardrailing real-world content. To bridge this gap, we propose SafeWatch, an efficient MLLM-based video guardrail model designed to follow customized safety policies and provide multi-label video guardrail outputs with content-specific explanations in a zero-shot manner. In particular, unlike traditional MLLM-based guardrails that encode all safety policies autoregressively, causing inefficiency and bias, SafeWatch uniquely encodes each policy chunk in parallel and eliminates their position bias such that all policies are attended simultaneously with equal importance. In addition, to improve efficiency and accuracy, SafeWatch incorporates a policy-aware visual token pruning algorithm that adaptively selects the most relevant video tokens for each policy, discarding noisy or irrelevant information. This allows for more focused, policy-compliant guardrail with significantly reduced computational overhead. Considering the limitations of existing video guardrail benchmarks, we propose SafeWatch-Bench, a large-scale video guardrail benchmark comprising over 2M videos spanning six safety categories which covers over 30 tasks to ensure a comprehensive coverage of all potential safety scenarios. SafeWatch outperforms SOTA by 28.2% on SafeWatch-Bench, 13.6% on benchmarks, cuts costs by 10%, and delivers top-tier explanations validated by LLM and human reviews.

Title: Efficient user history modeling with amortized inference for deep learning recommendation models

• Authors: Lars Hertel, Neil Daftary, Fedor Borisyuk, Aman Gupta, Rahul Mazumder
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.06924
• Pdf URL: https://arxiv.org/pdf/2412.06924
• Copy Paste: [[2412.06924]] Efficient user history modeling with amortized inference for deep learning recommendation models(https://arxiv.org/abs/2412.06924)
• Keywords: transformer
• Abstract: We study user history modeling via Transformer encoders in deep learning recommendation models (DLRM). Such architectures can significantly improve recommendation quality, but usually incur high latency cost necessitating infrastructure upgrades or very small Transformer models. An important part of user history modeling is early fusion of the candidate item and various methods have been studied. We revisit early fusion and compare concatenation of the candidate to each history item against appending it to the end of the list as a separate item. Using the latter method, allows us to reformulate the recently proposed amortized history inference algorithm M-FALCON \cite{zhai2024actions} for the case of DLRM models. We show via experimental results that appending with cross-attention performs on par with concatenation and that amortization significantly reduces inference costs. We conclude with results from deploying this model on the LinkedIn Feed and Ads surfaces, where amortization reduces latency by 30\% compared to non-amortized inference.

Title: When Every Token Counts: Optimal Segmentation for Low-Resource Language Models

• Authors: Bharath Raj S, Garvit Suri, Vikrant Dewangan, Raghav Sonavane
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06926
• Pdf URL: https://arxiv.org/pdf/2412.06926
• Copy Paste: [[2412.06926]] When Every Token Counts: Optimal Segmentation for Low-Resource Language Models(https://arxiv.org/abs/2412.06926)
• Keywords: segmentation
• Abstract: Traditional greedy tokenization methods have been a critical step in Natural Language Processing (NLP), influencing how text is converted into tokens and directly impacting model performance. While subword tokenizers like Byte-Pair Encoding (BPE) are widely used, questions remain about their optimality across model scales and languages. In this work, we demonstrate through extensive experiments that an optimal BPE configuration significantly reduces token count compared to greedy segmentation, yielding improvements in token-saving percentages and performance benefits, particularly for smaller models. We evaluate tokenization performance across various intrinsic and extrinsic tasks, including generation and classification. Our findings suggest that compression-optimized tokenization strategies could provide substantial advantages for multilingual and low-resource language applications, highlighting a promising direction for further research and inclusive NLP.

Title: Gradient-based facial encoding for key generation to encrypt and decrypt multimedia data

• Authors: Ankit Kumar Patel, Dewanshi Paul, Sneha Chaudhary, Sarthak Giri
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06927
• Pdf URL: https://arxiv.org/pdf/2412.06927
• Copy Paste: [[2412.06927]] Gradient-based facial encoding for key generation to encrypt and decrypt multimedia data(https://arxiv.org/abs/2412.06927)
• Keywords: secure, security, protect, attack, biometric
• Abstract: Password-based security is prone to forgetting, guessing, and hacking. Similarly, standalone biometric-based security is susceptible to template spoofing and replay attacks. This paper proposes a biocryptosystem based on face recognition technique to bridge this gap such that it can encrypt and decrypt any kind of file using the Advanced Encryption Standard (AES). The biocryptosystem uses a combination of biometric identification and cryptographic methods to protect sensitive information in a secure and effective manner. To verify a user's identity, our proposed system first captures an image of their face and extracts facial traits. The Histogram of Oriented Gradients (HOG) detects all the unique facial traits because HOG effectively captures edge-based features even in dim lighting. Every data type, including text, audio, and video files, can be encrypted and decrypted using this system. Biometric evidence is inherently tied to an individual, so it is almost impossible for attackers to access the user's data. This method also offers a high level of security by employing biometric data as an element in the 2-factor authentication process. The precision, efficiency, and security of this biocryptosystem are experimentally proven by different metrics like entropy and avalanche effect. Applications for the proposed system include safe file sharing, online transactions, and data archiving. Hence, it offers a strong and dependable option for safeguarding sensitive data.

Title: Analysing Public Transport User Sentiment on Low Resource Multilingual Data

• Authors: Rozina L. Myoya, Vukosi Marivate, Idris Abdulmumin
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06951
• Pdf URL: https://arxiv.org/pdf/2412.06951
• Copy Paste: [[2412.06951]] Analysing Public Transport User Sentiment on Low Resource Multilingual Data(https://arxiv.org/abs/2412.06951)
• Keywords: extraction
• Abstract: Public transport systems in many Sub-Saharan countries often receive less attention compared to other sectors, underscoring the need for innovative solutions to improve the Quality of Service (QoS) and overall user experience. This study explored commuter opinion mining to understand sentiments toward existing public transport systems in Kenya, Tanzania, and South Africa. We used a qualitative research design, analysing data from X (formerly Twitter) to assess sentiments across rail, mini-bus taxis, and buses. By leveraging Multilingual Opinion Mining techniques, we addressed the linguistic diversity and code-switching present in our dataset, thus demonstrating the application of Natural Language Processing (NLP) in extracting insights from under-resourced languages. We employed PLMs such as AfriBERTa, AfroXLMR, AfroLM, and PuoBERTa to conduct the sentiment analysis. The results revealed predominantly negative sentiments in South Africa and Kenya, while the Tanzanian dataset showed mainly positive sentiments due to the advertising nature of the tweets. Furthermore, feature extraction using the Word2Vec model and K-Means clustering illuminated semantic relationships and primary themes found within the different datasets. By prioritising the analysis of user experiences and sentiments, this research paves the way for developing more responsive, user-centered public transport systems in Sub-Saharan countries, contributing to the broader goal of improving urban mobility and sustainability.

Title: Enhancing operational wind downscaling capabilities over Canada: Application of a Conditional Wasserstein GAN methodology

• Authors: Jorge Guevara, Victor Nascimento, Johannes Schmude, Daniel Salles, Simon Corbeil-Létourneau, Madalina Surcel, Dominique Brunet
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06958
• Pdf URL: https://arxiv.org/pdf/2412.06958
• Copy Paste: [[2412.06958]] Enhancing operational wind downscaling capabilities over Canada: Application of a Conditional Wasserstein GAN methodology(https://arxiv.org/abs/2412.06958)
• Keywords: robust, generative
• Abstract: Wind downscaling is essential for improving the spatial resolution of weather forecasts, particularly in operational Numerical Weather Prediction (NWP). This study advances wind downscaling by extending the DownGAN framework introduced by Annau et al.,to operational datasets from the Global Deterministic Prediction System (GDPS) and High-Resolution Deterministic Prediction System (HRDPS), covering the entire Canadian domain. We enhance the model by incorporating high-resolution static covariates, such as HRDPS-derived topography, into a Conditional Wasserstein Generative Adversarial Network with Gradient Penalty, implemented using a UNET-based generator. Following the DownGAN framework, our methodology integrates low-resolution GDPS forecasts (15 km, 10-day horizon) and high-resolution HRDPS forecasts (2.5 km, 48-hour horizon) with Frequency Separation techniques adapted from computer vision. Through robust training and inference over the Canadian region, we demonstrate the operational scalability of our approach, achieving significant improvements in wind downscaling accuracy. Statistical validation highlights reductions in root mean square error (RMSE) and log spectral distance (LSD) metrics compared to the original DownGAN. High-resolution conditioning covariates and Frequency Separation strategies prove instrumental in enhancing model performance. This work underscores the potential for extending high-resolution wind forecasts beyond the 48-hour horizon, bridging the gap to the 10-day low resolution global forecast window.

Title: Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice

• Authors: A. Feder Cooper, Christopher A. Choquette-Choo, Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alexandra Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, Ilia Shumailov, Eleni Triantafillou, Peter Kairouz, Nicole Mitchell, Percy Liang, Daniel E. Ho, Yejin Choi, Sanmi Koyejo, Fernando Delgado, James Grimmelmann, Vitaly Shmatikov, Christopher De Sa, Solon Barocas, Amy Cyphert, Mark Lemley, danah boyd, Jennifer Wortman Vaughan, Miles Brundage, David Bau, Seth Neel, Abigail Z. Jacobs, Andreas Terzis, Hanna Wallach, Nicolas Papernot, Katherine Lee
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.06966
• Pdf URL: https://arxiv.org/pdf/2412.06966
• Copy Paste: [[2412.06966]] Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice(https://arxiv.org/abs/2412.06966)
• Keywords: privacy, generative
• Abstract: We articulate fundamental mismatches between technical methods for machine unlearning in Generative AI, and documented aspirations for broader impact that these methods could have for law and policy. These aspirations are both numerous and varied, motivated by issues that pertain to privacy, copyright, safety, and more. For example, unlearning is often invoked as a solution for removing the effects of targeted information from a generative-AI model's parameters, e.g., a particular individual's personal data or in-copyright expression of Spiderman that was included in the model's training data. Unlearning is also proposed as a way to prevent a model from generating targeted types of information in its outputs, e.g., generations that closely resemble a particular individual's data or reflect the concept of "Spiderman." Both of these goals--the targeted removal of information from a model and the targeted suppression of information from a model's outputs--present various technical and substantive challenges. We provide a framework for thinking rigorously about these challenges, which enables us to be clear about why unlearning is not a general-purpose solution for circumscribing generative-AI model behavior in service of broader positive impact. We aim for conceptual clarity and to encourage more thoughtful communication among machine learning (ML), law, and policy experts who seek to develop and apply technical methods for compliance with policy objectives.

Title: Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning

• Authors: Yingyi Ma, Zhe Liu, Ozlem Kalinli
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.06967
• Pdf URL: https://arxiv.org/pdf/2412.06967
• Copy Paste: [[2412.06967]] Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning(https://arxiv.org/abs/2412.06967)
• Keywords: large language model
• Abstract: The advent of Large Language Models (LLM) has reformed the Automatic Speech Recognition (ASR). Prompting LLM with audio embeddings to generate transcriptions becomes the new state-of-the-art ASR. Despite LLMs being trained with an extensive amount of text corpora, high-quality domain-specific text data can still significantly enhance ASR performance on domain adaptation tasks. Although LLM-based ASR can naturally incorporate more text corpora by fine-tuning the LLM decoder, fine-tuning such ASR on text-only data without paired prompts may diminish the effectiveness of domain-specific knowledge. To mitigate this issue, we propose a two-step soft prompt fine-tuning strategy that enhances domain-specific text adaptation. Experimental results show that text adaptation with our proposed method achieved a relative up to 9% Word Error Rate (WER) reduction and up to 18% Entity Error Rate (EER) reduction on the target domain compared to the baseline ASR. Combining this with domain-specific Language Model (LM) fusion can further improve the EER by a relative 2-5%

Title: SphereUFormer: A U-Shaped Transformer for Spherical 360 Perception

• Authors: Yaniv Benny, Lior Wolf
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06968
• Pdf URL: https://arxiv.org/pdf/2412.06968
• Copy Paste: [[2412.06968]] SphereUFormer: A U-Shaped Transformer for Spherical 360 Perception(https://arxiv.org/abs/2412.06968)
• Keywords: transformer, segmentation
• Abstract: This paper proposes a novel method for omnidirectional 360$\degree$ perception. Most common previous methods relied on equirectangular projection. This representation is easily applicable to 2D operation layers but introduces distortions into the image. Other methods attempted to remove the distortions by maintaining a sphere representation but relied on complicated convolution kernels that failed to show competitive results. In this work, we introduce a transformer-based architecture that, by incorporating a novel ``Spherical Local Self-Attention'' and other spherically-oriented modules, successfully operates in the spherical domain and outperforms the state-of-the-art in 360$\degree$ perception benchmarks for depth estimation and semantic segmentation.

Title: MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds

• Authors: Zhenggang Tang, Yuchen Fan, Dilin Wang, Hongyu Xu, Rakesh Ranjan, Alexander Schwing, Zhicheng Yan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06974
• Pdf URL: https://arxiv.org/pdf/2412.06974
• Copy Paste: [[2412.06974]] MV-DUSt3R+: Single-Stage Scene Reconstruction from Sparse Views In 2 Seconds(https://arxiv.org/abs/2412.06974)
• Keywords: robust
• Abstract: Recent sparse multi-view scene reconstruction advances like DUSt3R and MASt3R no longer require camera calibration and camera pose estimation. However, they only process a pair of views at a time to infer pixel-aligned pointmaps. When dealing with more than two views, a combinatorial number of error prone pairwise reconstructions are usually followed by an expensive global optimization, which often fails to rectify the pairwise reconstruction errors. To handle more views, reduce errors, and improve inference time, we propose the fast single-stage feed-forward network MV-DUSt3R. At its core are multi-view decoder blocks which exchange information across any number of views while considering one reference view. To make our method robust to reference view selection, we further propose MV-DUSt3R+, which employs cross-reference-view blocks to fuse information across different reference view choices. To further enable novel view synthesis, we extend both by adding and jointly training Gaussian splatting heads. Experiments on multi-view stereo reconstruction, multi-view pose estimation, and novel view synthesis confirm that our methods improve significantly upon prior art. Code will be released.

Title: AutoReason: Automatic Few-Shot Reasoning Decomposition

• Authors: Arda Sevinc, Abdurrahman Gumus
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06975
• Pdf URL: https://arxiv.org/pdf/2412.06975
• Copy Paste: [[2412.06975]] AutoReason: Automatic Few-Shot Reasoning Decomposition(https://arxiv.org/abs/2412.06975)
• Keywords: interpretability, large language model
• Abstract: Chain of Thought (CoT) was introduced in recent research as a method for improving step-by-step reasoning in Large Language Models. However, CoT has limited applications such as its need for hand-crafted few-shot exemplar prompts and no capability to adjust itself to different queries. In this work, we propose a system to automatically generate rationales using CoT. Our method improves multi-step implicit reasoning capabilities by decomposing the implicit query into several explicit questions. This provides interpretability for the model, improving reasoning in weaker LLMs. We test our approach with two Q\&A datasets: StrategyQA and HotpotQA. We show an increase in accuracy with both, especially on StrategyQA. To facilitate further research in this field, the complete source code for this study has been made publicly available on GitHub: this https URL.

Title: Edge-SD-SR: Low Latency and Parameter Efficient On-device Super-Resolution with Stable Diffusion via Bidirectional Conditioning

• Authors: Mehdi Noroozi, Isma Hadji, Victor Escorcia, Anestis Zaganidis, Brais Martinez, Georgios Tzimiropoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06978
• Pdf URL: https://arxiv.org/pdf/2412.06978
• Copy Paste: [[2412.06978]] Edge-SD-SR: Low Latency and Parameter Efficient On-device Super-Resolution with Stable Diffusion via Bidirectional Conditioning(https://arxiv.org/abs/2412.06978)
• Keywords: diffusion
• Abstract: There has been immense progress recently in the visual quality of Stable Diffusion-based Super Resolution (SD-SR). However, deploying large diffusion models on computationally restricted devices such as mobile phones remains impractical due to the large model size and high latency. This is compounded for SR as it often operates at high res (e.g. 4Kx3K). In this work, we introduce Edge-SD-SR, the first parameter efficient and low latency diffusion model for image super-resolution. Edge-SD-SR consists of ~169M parameters, including UNet, encoder and decoder, and has a complexity of only ~142 GFLOPs. To maintain a high visual quality on such low compute budget, we introduce a number of training strategies: (i) A novel conditioning mechanism on the low resolution input, coined bidirectional conditioning, which tailors the SD model for the SR task. (ii) Joint training of the UNet and encoder, while decoupling the encodings of the HR and LR images and using a dedicated schedule. (iii) Finetuning the decoder using the UNet's output to directly tailor the decoder to the latents obtained at inference time. Edge-SD-SR runs efficiently on device, e.g. it can upscale a 128x128 patch to 512x512 in 38 msec while running on a Samsung S24 DSP, and of a 512x512 to 2048x2048 (requiring 25 model evaluations) in just ~1.1 sec. Furthermore, we show that Edge-SD-SR matches or even outperforms state-of-the-art SR approaches on the most established SR benchmarks.

Title: Diffusing Differentiable Representations

• Authors: Yash Savani, Marc Finzi, J. Zico Kolter
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06981
• Pdf URL: https://arxiv.org/pdf/2412.06981
• Copy Paste: [[2412.06981]] Diffusing Differentiable Representations(https://arxiv.org/abs/2412.06981)
• Keywords: diffusion
• Abstract: We introduce a novel, training-free method for sampling differentiable representations (diffreps) using pretrained diffusion models. Rather than merely mode-seeking, our method achieves sampling by "pulling back" the dynamics of the reverse-time process--from the image space to the diffrep parameter space--and updating the parameters according to this pulled-back process. We identify an implicit constraint on the samples induced by the diffrep and demonstrate that addressing this constraint significantly improves the consistency and detail of the generated objects. Our method yields diffreps with substantially improved quality and diversity for images, panoramas, and 3D NeRFs compared to existing techniques. Our approach is a general-purpose method for sampling diffreps, expanding the scope of problems that diffusion models can tackle.

Title: In-Application Defense Against Evasive Web Scans through Behavioral Analysis

• Authors: Behzad Ousat, Mahshad Shariatnasab, Esteban Schafir, Farhad Shirani Chaharsooghi, Amin Kharraz
• Subjects: cs.LG, cs.CR, cs.IT
• Abstract URL: https://arxiv.org/abs/2412.07005
• Pdf URL: https://arxiv.org/pdf/2412.07005
• Copy Paste: [[2412.07005]] In-Application Defense Against Evasive Web Scans through Behavioral Analysis(https://arxiv.org/abs/2412.07005)
• Keywords: security, defense, robust
• Abstract: Web traffic has evolved to include both human users and automated agents, ranging from benign web crawlers to adversarial scanners such as those capable of credential stuffing, command injection, and account hijacking at the web scale. The estimated financial costs of these adversarial activities are estimated to exceed tens of billions of dollars in 2023. In this work, we introduce WebGuard, a low-overhead in-application forensics engine, to enable robust identification and monitoring of automated web scanners, and help mitigate the associated security risks. WebGuard focuses on the following design criteria: (i) integration into web applications without any changes to the underlying software components or infrastructure, (ii) minimal communication overhead, (iii) capability for real-time detection, e.g., within hundreds of milliseconds, and (iv) attribution capability to identify new behavioral patterns and detect emerging agent categories. To this end, we have equipped WebGuard with multi-modal behavioral monitoring mechanisms, such as monitoring spatio-temporal data and browser events. We also design supervised and unsupervised learning architectures for real-time detection and offline attribution of human and automated agents, respectively. Information theoretic analysis and empirical evaluations are provided to show that multi-modal data analysis, as opposed to uni-modal analysis which relies solely on mouse movement dynamics, significantly improves time-to-detection and attribution accuracy. Various numerical evaluations using real-world data collected via WebGuard are provided achieving high accuracy in hundreds of milliseconds, with a communication overhead below 10 KB per second.

Title: TAE: A Model-Constrained Tikhonov Autoencoder Approach for Forward and Inverse Problems

• Authors: Hai V. Nguyen, Tan Bui-Thanh
• Subjects: cs.LG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2412.07010
• Pdf URL: https://arxiv.org/pdf/2412.07010
• Copy Paste: [[2412.07010]] TAE: A Model-Constrained Tikhonov Autoencoder Approach for Forward and Inverse Problems(https://arxiv.org/abs/2412.07010)
• Keywords: robust, generative
• Abstract: Efficient real-time solvers for forward and inverse problems are essential in engineering and science applications. Machine learning surrogate models have emerged as promising alternatives to traditional methods, offering substantially reduced computational time. Nevertheless, these models typically demand extensive training datasets to achieve robust generalization across diverse scenarios. While physics-based approaches can partially mitigate this data dependency and ensure physics-interpretable solutions, addressing scarce data regimes remains a challenge. Both purely data-driven and physics-based machine learning approaches demonstrate severe overfitting issues when trained with insufficient data. We propose a novel Tikhonov autoencoder model-constrained framework, called TAE, capable of learning both forward and inverse surrogate models using a single arbitrary observation sample. We develop comprehensive theoretical foundations including forward and inverse inference error bounds for the proposed approach for linear cases. For comparative analysis, we derive equivalent formulations for pure data-driven and model-constrained approach counterparts. At the heart of our approach is a data randomization strategy, which functions as a generative mechanism for exploring the training data space, enabling effective training of both forward and inverse surrogate models from a single observation, while regularizing the learning process. We validate our approach through extensive numerical experiments on two challenging inverse problems: 2D heat conductivity inversion and initial condition reconstruction for time-dependent 2D Navier-Stokes equations. Results demonstrate that TAE achieves accuracy comparable to traditional Tikhonov solvers and numerical forward solvers for both inverse and forward problems, respectively, while delivering orders of magnitude computational speedups.

Title: ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models

• Authors: Jieyu Zhang, Le Xue, Linxin Song, Jun Wang, Weikai Huang, Manli Shu, An Yan, Zixian Ma, Juan Carlos Niebles, silvio savarese, Caiming Xiong, Zeyuan Chen, Ranjay Krishna, Ran Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07012
• Pdf URL: https://arxiv.org/pdf/2412.07012
• Copy Paste: [[2412.07012]] ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models(https://arxiv.org/abs/2412.07012)
• Keywords: interpretability, large language model
• Abstract: With the rise of multimodal applications, instruction data has become critical for training multimodal language models capable of understanding complex image-based queries. Existing practices rely on powerful but costly large language models (LLMs) or multimodal language models (MLMs) to produce instruction data. These are often prone to hallucinations, licensing issues and the generation process is often hard to scale and interpret. In this work, we present a programmatic approach that employs scene graphs as symbolic representations of images and human-written programs to systematically synthesize vision-centric instruction data. Our approach ensures the interpretability and controllability of the data generation process and scales efficiently while maintaining factual accuracy. By implementing a suite of 24 single-image, 14 multi-image instruction generators, and a scene graph generation pipeline, we build a scalable, cost-effective system: ProVision which produces diverse question-answer pairs concerning objects, attributes, relations, depth, etc., for any given image. Applied to Visual Genome and DataComp datasets, we generate over 10 million instruction data points, ProVision-10M, and leverage them in both pretraining and instruction tuning stages of MLMs. When adopted in the instruction tuning stage, our single-image instruction data yields up to a 7% improvement on the 2D split and 8% on the 3D split of CVBench, along with a 3% increase in performance on QBench2, RealWorldQA, and MMMU. Our multi-image instruction data leads to an 8% improvement on Mantis-Eval. Incorporation of our data in both pre-training and fine-tuning stages of xGen-MM-4B leads to an averaged improvement of 1.6% across 11 benchmarks.

Title: Asynchronous LLM Function Calling

• Authors: In Gim, Seung-seob Lee, Lin Zhong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07017
• Pdf URL: https://arxiv.org/pdf/2412.07017
• Copy Paste: [[2412.07017]] Asynchronous LLM Function Calling(https://arxiv.org/abs/2412.07017)
• Keywords: large language model
• Abstract: Large language models (LLMs) use function calls to interface with external tools and data source. However, the current approach to LLM function calling is inherently synchronous, where each call blocks LLM inference, limiting LLM operation and concurrent function execution. In this work, we propose AsyncLM, a system for asynchronous LLM function calling. AsyncLM improves LLM's operational efficiency by enabling LLMs to generate and execute function calls concurrently. Instead of waiting for each call's completion, AsyncLM introduces an interrupt mechanism to asynchronously notify the LLM in-flight when function calls return. We design an in-context protocol for function calls and interrupts, provide fine-tuning strategy to adapt LLMs to the interrupt semantics, and implement these mechanisms efficiently on LLM inference process. We demonstrate that AsyncLM can reduce end-to-end task completion latency from 1.6x-5.4x compared to synchronous function calling on a set of benchmark tasks in the Berkeley function calling leaderboard (BFCL). Furthermore, we discuss how interrupt mechanisms can be extended to enable novel human-LLM or LLM-LLM interactions.

Title: Assessing the Impact of Conspiracy Theories Using Large Language Models

• Authors: Bohan Jiang, Dawei Li, Zhen Tan, Xinyi Zhou, Ashwin Rao, Kristina Lerman, H. Russell Bernard, Huan Liu
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.07019
• Pdf URL: https://arxiv.org/pdf/2412.07019
• Copy Paste: [[2412.07019]] Assessing the Impact of Conspiracy Theories Using Large Language Models(https://arxiv.org/abs/2412.07019)
• Keywords: large language model
• Abstract: Measuring the relative impact of CTs is important for prioritizing responses and allocating resources effectively, especially during crises. However, assessing the actual impact of CTs on the public poses unique challenges. It requires not only the collection of CT-specific knowledge but also diverse information from social, psychological, and cultural dimensions. Recent advancements in large language models (LLMs) suggest their potential utility in this context, not only due to their extensive knowledge from large training corpora but also because they can be harnessed for complex reasoning. In this work, we develop datasets of popular CTs with human-annotated impacts. Borrowing insights from human impact assessment processes, we then design tailored strategies to leverage LLMs for performing human-like CT impact assessments. Through rigorous experiments, we textit{discover that an impact assessment mode using multi-step reasoning to analyze more CT-related evidence critically produces accurate results; and most LLMs demonstrate strong bias, such as assigning higher impacts to CTs presented earlier in the prompt, while generating less accurate impact assessments for emotionally charged and verbose CTs.

Title: Sequential Compression Layers for Efficient Federated Learning in Foundational Models

• Authors: Navyansh Mahla, Sunny Gupta, Amit Sethi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07021
• Pdf URL: https://arxiv.org/pdf/2412.07021
• Copy Paste: [[2412.07021]] Sequential Compression Layers for Efficient Federated Learning in Foundational Models(https://arxiv.org/abs/2412.07021)
• Keywords: federate, transformer, large language model
• Abstract: Federated Learning (FL) has gained popularity for fine-tuning large language models (LLMs) across multiple nodes, each with its own private data. While LoRA has been widely adopted for parameter efficient federated fine-tuning, recent theoretical and empirical studies highlight its suboptimal performance in the federated learning context. In response, we propose a novel, simple, and more effective parameter-efficient fine-tuning method that does not rely on LoRA. Our approach introduces a small multi-layer perceptron (MLP) layer between two existing MLP layers the up proj (the FFN projection layer following the self-attention module) and down proj within the feed forward network of the transformer block. This solution addresses the bottlenecks associated with LoRA in federated fine tuning and outperforms recent LoRA-based approaches, demonstrating superior performance for both language models and vision encoders.

Title: Dense Cross-Connected Ensemble Convolutional Neural Networks for Enhanced Model Robustness

• Authors: Longwei Wang, Xueqian Li, Zheng Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07022
• Pdf URL: https://arxiv.org/pdf/2412.07022
• Copy Paste: [[2412.07022]] Dense Cross-Connected Ensemble Convolutional Neural Networks for Enhanced Model Robustness(https://arxiv.org/abs/2412.07022)
• Keywords: attack, robust
• Abstract: The resilience of convolutional neural networks against input variations and adversarial attacks remains a significant challenge in image recognition tasks. Motivated by the need for more robust and reliable image recognition systems, we propose the Dense Cross-Connected Ensemble Convolutional Neural Network (DCC-ECNN). This novel architecture integrates the dense connectivity principle of DenseNet with the ensemble learning strategy, incorporating intermediate cross-connections between different DenseNet paths to facilitate extensive feature sharing and integration. The DCC-ECNN architecture leverages DenseNet's efficient parameter usage and depth while benefiting from the robustness of ensemble learning, ensuring a richer and more resilient feature representation.

Title: GenAI4UQ: A Software for Inverse Uncertainty Quantification Using Conditional Generative Models

• Authors: Ming Fan, Zezhong Zhang, Dan Lu, Guannan Zhang
• Subjects: cs.LG, physics.geo-ph
• Abstract URL: https://arxiv.org/abs/2412.07026
• Pdf URL: https://arxiv.org/pdf/2412.07026
• Copy Paste: [[2412.07026]] GenAI4UQ: A Software for Inverse Uncertainty Quantification Using Conditional Generative Models(https://arxiv.org/abs/2412.07026)
• Keywords: robust, generative
• Abstract: We introduce GenAI4UQ, a software package for inverse uncertainty quantification in model calibration, parameter estimation, and ensemble forecasting in scientific applications. GenAI4UQ leverages a generative artificial intelligence (AI) based conditional modeling framework to address the limitations of traditional inverse modeling techniques, such as Markov Chain Monte Carlo methods. By replacing computationally intensive iterative processes with a direct, learned mapping, GenAI4UQ enables efficient calibration of model input parameters and generation of output predictions directly from observations. The software's design allows for rapid ensemble forecasting with robust uncertainty quantification, while maintaining high computational and storage efficiency. GenAI4UQ simplifies the model training process through built-in auto-tuning of hyperparameters, making it accessible to users with varying levels of expertise. Its conditional generative framework ensures versatility, enabling applicability across a wide range of scientific domains. At its core, GenAI4UQ transforms the paradigm of inverse modeling by providing a fast, reliable, and user-friendly solution. It empowers researchers and practitioners to quickly estimate parameter distributions and generate model predictions for new observations, facilitating efficient decision-making and advancing the state of uncertainty quantification in computational modeling. (The code and data are available at this https URL).

Title: Static Key Attention in Vision

• Authors: Zizhao Hu, Xiaolin Zhou, Mohammad Rostami
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07049
• Pdf URL: https://arxiv.org/pdf/2412.07049
• Copy Paste: [[2412.07049]] Static Key Attention in Vision(https://arxiv.org/abs/2412.07049)
• Keywords: transformer
• Abstract: The success of vision transformers is widely attributed to the expressive power of their dynamically parameterized multi-head self-attention mechanism. We examine the impact of substituting the dynamic parameterized key with a static key within the standard attention mechanism in Vision Transformers. Our findings reveal that static key attention mechanisms can match or even exceed the performance of standard self-attention. Integrating static key attention modules into a Metaformer backbone, we find that it serves as a better intermediate stage in hierarchical hybrid architectures, balancing the strengths of depth-wise convolution and self-attention. Experiments on several vision tasks underscore the effectiveness of the static key mechanism, indicating that the typical two-step dynamic parameterization in attention can be streamlined to a single step without impacting performance under certain circumstances.

Title: Advancing clinical trial outcomes using deep learning and predictive modelling: bridging precision medicine and patient-centered care

• Authors: Sydney Anuyah, Mallika K Singh, Hope Nyavor
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07050
• Pdf URL: https://arxiv.org/pdf/2412.07050
• Copy Paste: [[2412.07050]] Advancing clinical trial outcomes using deep learning and predictive modelling: bridging precision medicine and patient-centered care(https://arxiv.org/abs/2412.07050)
• Keywords: robust, transformer
• Abstract: The integration of artificial intelligence [AI] into clinical trials has revolutionized the process of drug development and personalized medicine. Among these advancements, deep learning and predictive modelling have emerged as transformative tools for optimizing clinical trial design, patient recruitment, and real-time monitoring. This study explores the application of deep learning techniques, such as convolutional neural networks [CNNs] and transformerbased models, to stratify patients, forecast adverse events, and personalize treatment plans. Furthermore, predictive modelling approaches, including survival analysis and time-series forecasting, are employed to predict trial outcomes, enhancing efficiency and reducing trial failure rates. To address challenges in analysing unstructured clinical data, such as patient notes and trial protocols, natural language processing [NLP] techniques are utilized for extracting actionable insights. A custom dataset comprising structured patient demographics, genomic data, and unstructured text is curated for training and validating these models. Key metrics, including precision, recall, and F1 scores, are used to evaluate model performance, while trade-offs between accuracy and computational efficiency are examined to identify the optimal model for clinical deployment. This research underscores the potential of AI-driven methods to streamline clinical trial workflows, improve patient-centric outcomes, and reduce costs associated with trial inefficiencies. The findings provide a robust framework for integrating predictive analytics into precision medicine, paving the way for more adaptive and efficient clinical trials. By bridging the gap between technological innovation and real-world applications, this study contributes to advancing the role of AI in healthcare, particularly in fostering personalized care and improving overall trial success rates.

Title: Optimizing Personalized Federated Learning through Adaptive Layer-Wise Learning

• Authors: Weihang Chen, Jie Ren, Zhiqiang Li, Ling Gao, Zheng Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07062
• Pdf URL: https://arxiv.org/pdf/2412.07062
• Copy Paste: [[2412.07062]] Optimizing Personalized Federated Learning through Adaptive Layer-Wise Learning(https://arxiv.org/abs/2412.07062)
• Keywords: federate
• Abstract: Real-life deployment of federated Learning (FL) often faces non-IID data, which leads to poor accuracy and slow convergence. Personalized FL (pFL) tackles these issues by tailoring local models to individual data sources and using weighted aggregation methods for client-specific learning. However, existing pFL methods often fail to provide each local model with global knowledge on demand while maintaining low computational overhead. Additionally, local models tend to over-personalize their data during the training process, potentially dropping previously acquired global information. We propose FLAYER, a novel layer-wise learning method for pFL that optimizes local model personalization performance. FLAYER considers the different roles and learning abilities of neural network layers of individual local models. It incorporates global information for each local model as needed to initialize the local model cost-effectively. It then dynamically adjusts learning rates for each layer during local training, optimizing the personalized learning process for each local model while preserving global knowledge. Additionally, to enhance global representation in pFL, FLAYER selectively uploads parameters for global aggregation in a layer-wise manner. We evaluate FLAYER on four representative datasets in computer vision and natural language processing domains. Compared to six state-of-the-art pFL methods, FLAYER improves the inference accuracy, on average, by 5.42% (up to 14.29%).

Title: MoE-CAP: Cost-Accuracy-Performance Benchmarking for Mixture-of-Experts Systems

• Authors: Yao Fu, Yinsicheng Jiang, Yeqi Huang, Ping Nie, Zhan Lu, Leyang Xue, Congjie He, Man-Kit Sit, Jilong Xue, Li Dong, Ziming Miao, Kai Zou, Edoardo Ponti, Luo Mai
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.07067
• Pdf URL: https://arxiv.org/pdf/2412.07067
• Copy Paste: [[2412.07067]] MoE-CAP: Cost-Accuracy-Performance Benchmarking for Mixture-of-Experts Systems(https://arxiv.org/abs/2412.07067)
• Keywords: large language model
• Abstract: The sparse Mixture-of-Experts (MoE) architecture is increasingly favored for scaling Large Language Models (LLMs) efficiently; however, MoE systems rely on heterogeneous compute and memory resources. These factors collectively influence the system's Cost, Accuracy, and Performance (CAP), creating a challenging trade-off. Current benchmarks often fail to provide precise estimates of these effects, complicating practical considerations for deploying MoE systems. To bridge this gap, we introduce MoE-CAP, a benchmark specifically designed to evaluate MoE systems. Our findings highlight the difficulty of achieving an optimal balance of cost, accuracy, and performance with existing hardware capabilities. MoE systems often necessitate compromises on one factor to optimize the other two, a dynamic we term the MoE-CAP trade-off. To identify the best trade-off, we propose novel performance evaluation metrics - Sparse Memory Bandwidth Utilization (S-MBU) and Sparse Model FLOPS Utilization (S-MFU) - and develop cost models that account for the heterogeneous compute and memory hardware integral to MoE systems. This benchmark is publicly available on HuggingFace: this https URL.

Title: Enhancing radioisotope identification in gamma spectra with transfer learning

• Authors: Peter Lalor
• Subjects: cs.LG, nucl-th
• Abstract URL: https://arxiv.org/abs/2412.07069
• Pdf URL: https://arxiv.org/pdf/2412.07069
• Copy Paste: [[2412.07069]] Enhancing radioisotope identification in gamma spectra with transfer learning(https://arxiv.org/abs/2412.07069)
• Keywords: transformer
• Abstract: Machine learning methods in gamma spectroscopy have the potential to provide accurate, real-time classification of unknown radioactive samples. However, obtaining sufficient experimental training data is often prohibitively expensive and time-consuming, and models trained solely on synthetic data can struggle to generalize to the unpredictable range of real-world operating scenarios. In this work, we pretrain a model using physically derived synthetic data and subsequently leverage transfer learning techniques to fine-tune the model for a specific target domain. This paradigm enables us to embed physical principles during the pretraining step, thus requiring less data from the target domain compared to classical machine learning methods. Results of this analysis indicate that fine-tuned models significantly outperform those trained exclusively on synthetic data or solely on target-domain data, particularly in the intermediate data regime (${\approx} 10^4$ training samples). This conclusion is consistent across four different machine learning architectures (MLP, CNN, Transformer, and LSTM) considered in this study. This research serves as proof of concept for applying transfer learning techniques to application scenarios where access to experimental data is limited.

Title: Stable Mean Teacher for Semi-supervised Video Action Detection

• Authors: Akash Kumar, Sirshapan Mitra, Yogesh Singh Rawat
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07072
• Pdf URL: https://arxiv.org/pdf/2412.07072
• Copy Paste: [[2412.07072]] Stable Mean Teacher for Semi-supervised Video Action Detection(https://arxiv.org/abs/2412.07072)
• Keywords: segmentation
• Abstract: In this work, we focus on semi-supervised learning for video action detection. Video action detection requires spatiotemporal localization in addition to classification, and a limited amount of labels makes the model prone to unreliable predictions. We present Stable Mean Teacher, a simple end-to-end teacher-based framework that benefits from improved and temporally consistent pseudo labels. It relies on a novel Error Recovery (EoR) module, which learns from students' mistakes on labeled samples and transfers this knowledge to the teacher to improve pseudo labels for unlabeled samples. Moreover, existing spatiotemporal losses do not take temporal coherency into account and are prone to temporal inconsistencies. To address this, we present Difference of Pixels (DoP), a simple and novel constraint focused on temporal consistency, leading to coherent temporal detections. We evaluate our approach on four different spatiotemporal detection benchmarks: UCF101-24, JHMDB21, AVA, and YouTube-VOS. Our approach outperforms the supervised baselines for action detection by an average margin of 23.5% on UCF101-24, 16% on JHMDB21, and 3.3% on AVA. Using merely 10% and 20% of data, it provides competitive performance compared to the supervised baseline trained on 100% annotations on UCF101-24 and JHMDB21, respectively. We further evaluate its effectiveness on AVA for scaling to large-scale datasets and YouTube-VOS for video object segmentation, demonstrating its generalization capability to other tasks in the video domain. Code and models are publicly available.

Title: Retaining and Enhancing Pre-trained Knowledge in Vision-Language Models with Prompt Ensembling

• Authors: Donggeun Kim, Yujin Jo, Myungjoo Lee, Taesup Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07077
• Pdf URL: https://arxiv.org/pdf/2412.07077
• Copy Paste: [[2412.07077]] Retaining and Enhancing Pre-trained Knowledge in Vision-Language Models with Prompt Ensembling(https://arxiv.org/abs/2412.07077)
• Keywords: robust
• Abstract: The advancement of vision-language models, particularly the Contrastive Language-Image Pre-training (CLIP) model, has revolutionized the field of machine learning by enabling robust zero-shot learning capabilities. These capabilities allow models to understand and respond to previously unseen data without task-specific training. However, adapting CLIP to integrate specialized knowledge from various domains while retaining its zero-shot capabilities remains a significant challenge. To address this, we introduce a novel prompt ensemble learning approach called Group-wise Prompt Ensemble (GPE). This method aims to enhance CLIP's zero-shot capabilities by incorporating new domain knowledge while improving its adaptability and robustness against data distribution shifts. Our approach hinges on three main strategies: prompt grouping with masked attention to optimize CLIP's adaptability while safeguarding its zero-shot capabilities; the incorporation of auxiliary prompts for the seamless integration of new domain insights without disrupting the original model's representation; and an ensemble learning strategy that effectively merges original and new knowledge. Through rigorous experimentation, including more challenging cross-dataset transfer evaluations, our GPE method redefines the benchmarks for the adaptability and efficiency of vision-language models, surpassing existing models across various scenarios.

Title: Defensive Dual Masking for Robust Adversarial Defense

• Authors: Wangli Yang, Jie Yang, Yi Guo, Johan Barthelemy
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07078
• Pdf URL: https://arxiv.org/pdf/2412.07078
• Copy Paste: [[2412.07078]] Defensive Dual Masking for Robust Adversarial Defense(https://arxiv.org/abs/2412.07078)
• Keywords: defense, attack, robust, large language model
• Abstract: The field of textual adversarial defenses has gained considerable attention in recent years due to the increasing vulnerability of natural language processing (NLP) models to adversarial attacks, which exploit subtle perturbations in input text to deceive models. This paper introduces the Defensive Dual Masking (DDM) algorithm, a novel approach designed to enhance model robustness against such attacks. DDM utilizes a unique adversarial training strategy where [MASK] tokens are strategically inserted into training samples to prepare the model to handle adversarial perturbations more effectively. During inference, potentially adversarial tokens are dynamically replaced with [MASK] tokens to neutralize potential threats while preserving the core semantics of the input. The theoretical foundation of our approach is explored, demonstrating how the selective masking mechanism strengthens the model's ability to identify and mitigate adversarial manipulations. Our empirical evaluation across a diverse set of benchmark datasets and attack mechanisms consistently shows that DDM outperforms state-of-the-art defense techniques, improving model accuracy and robustness. Moreover, when applied to Large Language Models (LLMs), DDM also enhances their resilience to adversarial attacks, providing a scalable defense mechanism for large-scale NLP applications.

Title: Creative Portraiture: Exploring Creative Adversarial Networks and Conditional Creative Adversarial Networks

• Authors: Sebastian Hereu, Qianfei Hu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07091
• Pdf URL: https://arxiv.org/pdf/2412.07091
• Copy Paste: [[2412.07091]] Creative Portraiture: Exploring Creative Adversarial Networks and Conditional Creative Adversarial Networks(https://arxiv.org/abs/2412.07091)
• Keywords: generative
• Abstract: Convolutional neural networks (CNNs) have been combined with generative adversarial networks (GANs) to create deep convolutional generative adversarial networks (DCGANs) with great success. DCGANs have been used for generating images and videos from creative domains such as fashion design and painting. A common critique of the use of DCGANs in creative applications is that they are limited in their ability to generate creative products because the generator simply learns to copy the training distribution. We explore an extension of DCGANs, creative adversarial networks (CANs). Using CANs, we generate novel, creative portraits, using the WikiArt dataset to train the network. Moreover, we introduce our extension of CANs, conditional creative adversarial networks (CCANs), and demonstrate their potential to generate creative portraits conditioned on a style label. We argue that generating products that are conditioned, or inspired, on a style label closely emulates real creative processes in which humans produce imaginative work that is still rooted in previous styles.

Title: Streaming Private Continual Counting via Binning

• Authors: Joel Daniel Andersson, Rasmus Pagh
• Subjects: cs.LG, cs.CR, cs.DS
• Abstract URL: https://arxiv.org/abs/2412.07093
• Pdf URL: https://arxiv.org/pdf/2412.07093
• Copy Paste: [[2412.07093]] Streaming Private Continual Counting via Binning(https://arxiv.org/abs/2412.07093)
• Keywords: privacy
• Abstract: In differential privacy, $\textit{continual observation}$ refers to problems in which we wish to continuously release a function of a dataset that is revealed one element at a time. The challenge is to maintain a good approximation while keeping the combined output over all time steps differentially private. In the special case of $\textit{continual counting}$ we seek to approximate a sum of binary input elements. This problem has received considerable attention lately, in part due to its relevance in implementations of differentially private stochastic gradient descent. $\textit{Factorization mechanisms}$ are the leading approach to continual counting, but the best such mechanisms do not work well in $\textit{streaming}$ settings since they require space proportional to the size of the input. In this paper, we present a simple approach to approximating factorization mechanisms in low space via $\textit{binning}$, where adjacent matrix entries with similar values are changed to be identical in such a way that a matrix-vector product can be maintained in sublinear space. Our approach has provable sublinear space guarantees for a class of lower triangular matrices whose entries are monotonically decreasing away from the diagonal. We show empirically that even with very low space usage we are able to closely match, and sometimes surpass, the performance of asymptotically optimal factorization mechanisms. Recently, and independently of our work, Dvijotham et al. have also suggested an approach to implementing factorization mechanisms in a streaming setting. Their work differs from ours in several respects: It only addresses factorization into $\textit{Toeplitz}$ matrices, only considers $\textit{maximum}$ error, and uses a different technique based on rational function approximation that seems less versatile than our binning approach.

Title: On Evaluating the Durability of Safeguards for Open-Weight LLMs

• Authors: Xiangyu Qi, Boyi Wei, Nicholas Carlini, Yangsibo Huang, Tinghao Xie, Luxi He, Matthew Jagielski, Milad Nasr, Prateek Mittal, Peter Henderson
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07097
• Pdf URL: https://arxiv.org/pdf/2412.07097
• Copy Paste: [[2412.07097]] On Evaluating the Durability of Safeguards for Open-Weight LLMs(https://arxiv.org/abs/2412.07097)
• Keywords: defense, large language model
• Abstract: Stakeholders -- from model developers to policymakers -- seek to minimize the dual-use risks of large language models (LLMs). An open challenge to this goal is whether technical safeguards can impede the misuse of LLMs, even when models are customizable via fine-tuning or when model weights are fully open. In response, several recent studies have proposed methods to produce durable LLM safeguards for open-weight LLMs that can withstand adversarial modifications of the model's weights via fine-tuning. This holds the promise of raising adversaries' costs even under strong threat models where adversaries can directly fine-tune model weights. However, in this paper, we urge for more careful characterization of the limits of these approaches. Through several case studies, we demonstrate that even evaluating these defenses is exceedingly difficult and can easily mislead audiences into thinking that safeguards are more durable than they really are. We draw lessons from the evaluation pitfalls that we identify and suggest future research carefully cabin claims to more constrained, well-defined, and rigorously examined threat models, which can provide more useful and candid assessments to stakeholders.

Title: Improving the Natural Language Inference robustness to hard dataset by data augmentation and preprocessing

• Authors: Zijiang Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07108
• Pdf URL: https://arxiv.org/pdf/2412.07108
• Copy Paste: [[2412.07108]] Improving the Natural Language Inference robustness to hard dataset by data augmentation and preprocessing(https://arxiv.org/abs/2412.07108)
• Keywords: robust, transformer
• Abstract: Natural Language Inference (NLI) is the task of inferring whether the hypothesis can be justified by the given premise. Basically, we classify the hypothesis into three labels(entailment, neutrality and contradiction) given the premise. NLI was well studied by the previous researchers. A number of models, especially the transformer based ones, have achieved significant improvement on these tasks. However, it is reported that these models are suffering when they are dealing with hard datasets. Particularly, they perform much worse when dealing with unseen out-of-distribution premise and hypothesis. They may not understand the semantic content but learn the spurious correlations. In this work, we propose the data augmentation and preprocessing methods to solve the word overlap, numerical reasoning and length mismatch problems. These methods are general methods that do not rely on the distribution of the testing data and they help improve the robustness of the models.

Title: Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need

• Authors: Bo-Wen Zhang, Yan Yan, Boxiang Yang, Yifei Xue, Guang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07111
• Pdf URL: https://arxiv.org/pdf/2412.07111
• Copy Paste: [[2412.07111]] Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need(https://arxiv.org/abs/2412.07111)
• Keywords: robust, large language model
• Abstract: While scaling laws optimize training configurations for large language models (LLMs) through experiments on smaller or early-stage models, they fail to predict emergent abilities due to the absence of such capabilities in these models. To address this, we propose a method that predicts emergent abilities by leveraging proxy tasks. We begin by establishing relevance metrics between the target task and candidate tasks based on performance differences across multiple models. These candidate tasks are then validated for robustness with small model ensembles, leading to the selection of the most appropriate proxy tasks. The predicted performance on the target task is then derived by integrating the evaluation results of these proxies. In a case study on tool utilization capabilities, our method demonstrated a strong correlation between predicted and actual performance, confirming its effectiveness.

Title: Exploring Coding Spot: Understanding Parametric Contributions to LLM Coding Performance

• Authors: Dongjun Kim, Minhyuk Kim, YongChan Chun, Chanjun Park, Heuiseok Lim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07113
• Pdf URL: https://arxiv.org/pdf/2412.07113
• Copy Paste: [[2412.07113]] Exploring Coding Spot: Understanding Parametric Contributions to LLM Coding Performance(https://arxiv.org/abs/2412.07113)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated notable proficiency in both code generation and comprehension across multiple programming languages. However, the mechanisms underlying this proficiency remain underexplored, particularly with respect to whether distinct programming languages are processed independently or within a shared parametric region. Drawing an analogy to the specialized regions of the brain responsible for distinct cognitive functions, we introduce the concept of Coding Spot, a specialized parametric region within LLMs that facilitates coding capabilities. Our findings identify this Coding Spot and show that targeted modifications to this subset significantly affect performance on coding tasks, while largely preserving non-coding functionalities. This compartmentalization mirrors the functional specialization observed in cognitive neuroscience, where specific brain regions are dedicated to distinct tasks, suggesting that LLMs may similarly employ specialized parameter regions for different knowledge domains.

Title: TT-MPD: Test Time Model Pruning and Distillation

• Authors: Haihang Wu, Wei Wang, Tamasha Malepathirana, Sachith Seneviratne, Denny Oetomo, Saman Halgamuge
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07114
• Pdf URL: https://arxiv.org/pdf/2412.07114
• Copy Paste: [[2412.07114]] TT-MPD: Test Time Model Pruning and Distillation(https://arxiv.org/abs/2412.07114)
• Keywords: privacy
• Abstract: Pruning can be an effective method of compressing large pre-trained models for inference speed acceleration. Previous pruning approaches rely on access to the original training dataset for both pruning and subsequent fine-tuning. However, access to the training data can be limited due to concerns such as data privacy and commercial confidentiality. Furthermore, with covariate shift (disparities between test and training data distributions), pruning and finetuning with training datasets can hinder the generalization of the pruned model to test data. To address these issues, pruning and finetuning the model with test time samples becomes essential. However, test-time model pruning and fine-tuning incur additional computation costs and slow down the model's prediction speed, thus posing efficiency issues. Existing pruning methods are not efficient enough for test time model pruning setting, since finetuning the pruned model is needed to evaluate the importance of removable components. To address this, we propose two variables to approximate the fine-tuned accuracy. We then introduce an efficient pruning method that considers the approximated finetuned accuracy and potential inference latency saving. To enhance fine-tuning efficiency, we propose an efficient knowledge distillation method that only needs to generate pseudo labels for a small set of finetuning samples one time, thereby reducing the expensive pseudo-label generation cost. Experimental results demonstrate that our method achieves a comparable or superior tradeoff between test accuracy and inference latency, with a 32% relative reduction in pruning and finetuning time compared to the best existing method.

Title: A Review of Human Emotion Synthesis Based on Generative Technology

• Authors: Fei Ma, Yukan Li, Yifan Xie, Ying He, Yi Zhang, Hongwei Ren, Zhou Liu, Wei Yao, Fuji Ren, Fei Richard Yu, Shiguang Ni
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07116
• Pdf URL: https://arxiv.org/pdf/2412.07116
• Copy Paste: [[2412.07116]] A Review of Human Emotion Synthesis Based on Generative Technology(https://arxiv.org/abs/2412.07116)
• Keywords: diffusion, generative, large language model
• Abstract: Human emotion synthesis is a crucial aspect of affective computing. It involves using computational methods to mimic and convey human emotions through various modalities, with the goal of enabling more natural and effective human-computer interactions. Recent advancements in generative models, such as Autoencoders, Generative Adversarial Networks, Diffusion Models, Large Language Models, and Sequence-to-Sequence Models, have significantly contributed to the development of this field. However, there is a notable lack of comprehensive reviews in this field. To address this problem, this paper aims to address this gap by providing a thorough and systematic overview of recent advancements in human emotion synthesis based on generative models. Specifically, this review will first present the review methodology, the emotion models involved, the mathematical principles of generative models, and the datasets used. Then, the review covers the application of different generative models to emotion synthesis based on a variety of modalities, including facial images, speech, and text. It also examines mainstream evaluation metrics. Additionally, the review presents some major findings and suggests future research directions, providing a comprehensive understanding of the role of generative technology in the nuanced domain of emotion synthesis.

Title: DiffCLIP: Few-shot Language-driven Multimodal Classifier

• Authors: Jiaqing Zhang, Mingxiang Cao, Xue Yang, Kai Jiang, Yunsong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07119
• Pdf URL: https://arxiv.org/pdf/2412.07119
• Copy Paste: [[2412.07119]] DiffCLIP: Few-shot Language-driven Multimodal Classifier(https://arxiv.org/abs/2412.07119)
• Keywords: diffusion
• Abstract: Visual language models like Contrastive Language-Image Pretraining (CLIP) have shown impressive performance in analyzing natural images with language information. However, these models often encounter challenges when applied to specialized domains such as remote sensing due to the limited availability of image-text pairs for training. To tackle this issue, we introduce DiffCLIP, a novel framework that extends CLIP to effectively convey comprehensive language-driven semantic information for accurate classification of high-dimensional multimodal remote sensing images. DiffCLIP is a few-shot learning method that leverages unlabeled images for pretraining. It employs unsupervised mask diffusion learning to capture the distribution of diverse modalities without requiring labels. The modality-shared image encoder maps multimodal data into a unified subspace, extracting shared features with consistent parameters across modalities. A well-trained image encoder further enhances learning by aligning visual representations with class-label text information from CLIP. By integrating these approaches, DiffCLIP significantly boosts CLIP performance using a minimal number of image-text pairs. We evaluate DiffCLIP on widely used high-dimensional multimodal datasets, demonstrating its effectiveness in addressing few-shot annotated classification tasks. DiffCLIP achieves an overall accuracy improvement of 10.65% across three remote sensing datasets compared with CLIP, while utilizing only 2-shot image-text pairs. The code has been released at this https URL.

Title: Bridging the Gap for Test-Time Multimodal Sentiment Analysis

• Authors: Zirun Guo, Tao Jin, Wenlong Xu, Wang Lin, Yangyang Wu
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07121
• Pdf URL: https://arxiv.org/pdf/2412.07121
• Copy Paste: [[2412.07121]] Bridging the Gap for Test-Time Multimodal Sentiment Analysis(https://arxiv.org/abs/2412.07121)
• Keywords: privacy
• Abstract: Multimodal sentiment analysis (MSA) is an emerging research topic that aims to understand and recognize human sentiment or emotions through multiple modalities. However, in real-world dynamic scenarios, the distribution of target data is always changing and different from the source data used to train the model, which leads to performance degradation. Common adaptation methods usually need source data, which could pose privacy issues or storage overheads. Therefore, test-time adaptation (TTA) methods are introduced to improve the performance of the model at inference time. Existing TTA methods are always based on probabilistic models and unimodal learning, and thus can not be applied to MSA which is often considered as a multimodal regression task. In this paper, we propose two strategies: Contrastive Adaptation and Stable Pseudo-label generation (CASP) for test-time adaptation for multimodal sentiment analysis. The two strategies deal with the distribution shifts for MSA by enforcing consistency and minimizing empirical risk, respectively. Extensive experiments show that CASP brings significant and consistent improvements to the performance of the model across various distribution shift settings and with different backbones, demonstrating its effectiveness and versatility. Our codes are available at this https URL.

Title: Deep Learning-Enhanced Preconditioning for Efficient Conjugate Gradient Solvers in Large-Scale PDE Systems

• Authors: Rui Li, Song Wang, Chen Wang
• Subjects: cs.LG, cs.AI, math.NA
• Abstract URL: https://arxiv.org/abs/2412.07127
• Pdf URL: https://arxiv.org/pdf/2412.07127
• Copy Paste: [[2412.07127]] Deep Learning-Enhanced Preconditioning for Efficient Conjugate Gradient Solvers in Large-Scale PDE Systems(https://arxiv.org/abs/2412.07127)
• Keywords: robust
• Abstract: Preconditioning techniques are crucial for enhancing the efficiency of solving large-scale linear equation systems that arise from partial differential equation (PDE) discretization. These techniques, such as Incomplete Cholesky factorization (IC) and data-driven neural network methods, accelerate the convergence of iterative solvers like Conjugate Gradient (CG) by approximating the original matrices. This paper introduces a novel approach that integrates Graph Neural Network (GNN) with traditional IC, addressing the shortcomings of direct generation methods based on GNN and achieving significant improvements in computational efficiency and scalability. Experimental results demonstrate an average reduction in iteration counts by 24.8% compared to IC and a two-order-of-magnitude increase in training scale compared to previous methods. A three-dimensional static structural analysis utilizing finite element methods was validated on training sparse matrices of up to 5 million dimensions and inference scales of up to 10 million. Furthermore, the approach demon-strates robust generalization capabilities across scales, facilitating the effective acceleration of CG solvers for large-scale linear equations using small-scale data on modest hardware. The method's robustness and scalability make it a practical solution for computational science.

Title: StyleMark: A Robust Watermarking Method for Art Style Images Against Black-Box Arbitrary Style Transfer

• Authors: Yunming Zhang, Dengpan Ye, Sipeng Shen, Jun Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07129
• Pdf URL: https://arxiv.org/pdf/2412.07129
• Copy Paste: [[2412.07129]] StyleMark: A Robust Watermarking Method for Art Style Images Against Black-Box Arbitrary Style Transfer(https://arxiv.org/abs/2412.07129)
• Keywords: secure, attack, robust, watermark
• Abstract: Arbitrary Style Transfer (AST) achieves the rendering of real natural images into the painting styles of arbitrary art style images, promoting art communication. However, misuse of unauthorized art style images for AST may infringe on artists' copyrights. One countermeasure is robust watermarking, which tracks image propagation by embedding copyright watermarks into carriers. Unfortunately, AST-generated images lose the structural and semantic information of the original style image, hindering end-to-end robust tracking by watermarks. To fill this gap, we propose StyleMark, the first robust watermarking method for black-box AST, which can be seamlessly applied to art style images achieving precise attribution of artistic styles after AST. Specifically, we propose a new style watermark network that adjusts the mean activations of style features through multi-scale watermark embedding, thereby planting watermark traces into the shared style feature space of style images. Furthermore, we design a distribution squeeze loss, which constrain content statistical feature distortion, forcing the reconstruction network to focus on integrating style features with watermarks, thus optimizing the intrinsic watermark distribution. Finally, based on solid end-to-end training, StyleMark mitigates the optimization conflict between robustness and watermark invisibility through decoder fine-tuning under random noise. Experimental results demonstrate that StyleMark exhibits significant robustness against black-box AST and common pixel-level distortions, while also securely defending against malicious adaptive attacks.

Title: Oreo: Protecting ASLR Against Microarchitectural Attacks (Extended Version)

• Authors: Shixin Song, Joseph Zhang, Mengjia Yan
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2412.07135
• Pdf URL: https://arxiv.org/pdf/2412.07135
• Copy Paste: [[2412.07135]] Oreo: Protecting ASLR Against Microarchitectural Attacks (Extended Version)(https://arxiv.org/abs/2412.07135)
• Keywords: protect, attack
• Abstract: Address Space Layout Randomization (ASLR) is one of the most prominently deployed mitigations against memory corruption attacks. ASLR randomly shuffles program virtual addresses to prevent attackers from knowing the location of program contents in memory. Microarchitectural side channels have been shown to defeat ASLR through various hardware mechanisms. We systematically analyze existing microarchitectural attacks and identify multiple leakage paths. Given the vast attack surface exposed by ASLR, it is challenging to effectively prevent leaking the ASLR secret against microarchitectural attacks. Motivated by this, we present Oreo, a software-hardware co-design mitigation that strengthens ASLR against these attacks. Oreo uses a new memory mapping interface to remove secret randomized bits in virtual addresses before translating them to their corresponding physical addresses. This extra step hides randomized virtual addresses from microarchitecture structures, preventing side channels from leaking ASLR secrets. Oreo is transparent to user programs and incurs low overhead. We prototyped and evaluated our design on Linux using the hardware simulator gem5.

Title: Unlocking TriLevel Learning with Level-Wise Zeroth Order Constraints: Distributed Algorithms and Provable Non-Asymptotic Convergence

• Authors: Yang Jiao, Kai Yang, Chengtao Jian
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2412.07138
• Pdf URL: https://arxiv.org/pdf/2412.07138
• Copy Paste: [[2412.07138]] Unlocking TriLevel Learning with Level-Wise Zeroth Order Constraints: Distributed Algorithms and Provable Non-Asymptotic Convergence(https://arxiv.org/abs/2412.07138)
• Keywords: privacy, robust
• Abstract: Trilevel learning (TLL) found diverse applications in numerous machine learning applications, ranging from robust hyperparameter optimization to domain adaptation. However, existing researches primarily focus on scenarios where TLL can be addressed with first order information available at each level, which is inadequate in many situations involving zeroth order constraints, such as when black-box models are employed. Moreover, in trilevel learning, data may be distributed across various nodes, necessitating strategies to address TLL problems without centralizing data on servers to uphold data privacy. To this end, an effective distributed trilevel zeroth order learning framework DTZO is proposed in this work to address the TLL problems with level-wise zeroth order constraints in a distributed manner. The proposed DTZO is versatile and can be adapted to a wide range of (grey-box) TLL problems with partial zeroth order constraints. In DTZO, the cascaded polynomial approximation can be constructed without relying on gradients or sub-gradients, leveraging a novel cut, i.e., zeroth order cut. Furthermore, we theoretically carry out the non-asymptotic convergence rate analysis for the proposed DTZO in achieving the $\epsilon$-stationary point. Extensive experiments have been conducted to demonstrate and validate the superior performance of the proposed DTZO, e.g., it approximately achieves up to a 40$\%$ improvement in performance.

Title: FIRE: Robust Detection of Diffusion-Generated Images via Frequency-Guided Reconstruction Error

• Authors: Beilin Chu, Xuan Xu, Xin Wang, Yufei Zhang, Weike You, Linna Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07140
• Pdf URL: https://arxiv.org/pdf/2412.07140
• Copy Paste: [[2412.07140]] FIRE: Robust Detection of Diffusion-Generated Images via Frequency-Guided Reconstruction Error(https://arxiv.org/abs/2412.07140)
• Keywords: robust, diffusion
• Abstract: The rapid advancement of diffusion models has significantly improved high-quality image generation, making generated content increasingly challenging to distinguish from real images and raising concerns about potential misuse. In this paper, we observe that diffusion models struggle to accurately reconstruct mid-band frequency information in real images, suggesting the limitation could serve as a cue for detecting diffusion model generated images. Motivated by this observation, we propose a novel method called Frequency-guided Reconstruction Error (FIRE), which, to the best of our knowledge, is the first to investigate the influence of frequency decomposition on reconstruction error. FIRE assesses the variation in reconstruction error before and after the frequency decomposition, offering a robust method for identifying diffusion model generated images. Extensive experiments show that FIRE generalizes effectively to unseen diffusion models and maintains robustness against diverse perturbations.

Title: RAP-SR: RestorAtion Prior Enhancement in Diffusion Models for Realistic Image Super-Resolution

• Authors: Jiangang Wang, Qingnan Fan, Jinwei Chen, Hong Gu, Feng Huang, Wenqi Ren
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07149
• Pdf URL: https://arxiv.org/pdf/2412.07149
• Copy Paste: [[2412.07149]] RAP-SR: RestorAtion Prior Enhancement in Diffusion Models for Realistic Image Super-Resolution(https://arxiv.org/abs/2412.07149)
• Keywords: diffusion, generative
• Abstract: Benefiting from their powerful generative capabilities, pretrained diffusion models have garnered significant attention for real-world image super-resolution (Real-SR). Existing diffusion-based SR approaches typically utilize semantic information from degraded images and restoration prompts to activate prior for producing realistic high-resolution images. However, general-purpose pretrained diffusion models, not designed for restoration tasks, often have suboptimal prior, and manually defined prompts may fail to fully exploit the generated potential. To address these limitations, we introduce RAP-SR, a novel restoration prior enhancement approach in pretrained diffusion models for Real-SR. First, we develop the High-Fidelity Aesthetic Image Dataset (HFAID), curated through a Quality-Driven Aesthetic Image Selection Pipeline (QDAISP). Our dataset not only surpasses existing ones in fidelity but also excels in aesthetic quality. Second, we propose the Restoration Priors Enhancement Framework, which includes Restoration Priors Refinement (RPR) and Restoration-Oriented Prompt Optimization (ROPO) modules. RPR refines the restoration prior using the HFAID, while ROPO optimizes the unique restoration identifier, improving the quality of the resulting images. RAP-SR effectively bridges the gap between general-purpose models and the demands of Real-SR by enhancing restoration prior. Leveraging the plug-and-play nature of RAP-SR, our approach can be seamlessly integrated into existing diffusion-based SR methods, boosting their performance. Extensive experiments demonstrate its broad applicability and state-of-the-art results. Codes and datasets will be available upon acceptance.

Title: Hero-SR: One-Step Diffusion for Super-Resolution with Human Perception Priors

• Authors: Jiangang Wang, Qingnan Fan, Qi Zhang, Haigen Liu, Yuhang Yu, Jinwei Chen, Wenqi Ren
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07152
• Pdf URL: https://arxiv.org/pdf/2412.07152
• Copy Paste: [[2412.07152]] Hero-SR: One-Step Diffusion for Super-Resolution with Human Perception Priors(https://arxiv.org/abs/2412.07152)
• Keywords: robust, diffusion
• Abstract: Owing to the robust priors of diffusion models, recent approaches have shown promise in addressing real-world super-resolution (Real-SR). However, achieving semantic consistency and perceptual naturalness to meet human perception demands remains difficult, especially under conditions of heavy degradation and varied input complexities. To tackle this, we propose Hero-SR, a one-step diffusion-based SR framework explicitly designed with human perception priors. Hero-SR consists of two novel modules: the Dynamic Time-Step Module (DTSM), which adaptively selects optimal diffusion steps for flexibly meeting human perceptual standards, and the Open-World Multi-modality Supervision (OWMS), which integrates guidance from both image and text domains through CLIP to improve semantic consistency and perceptual naturalness. Through these modules, Hero-SR generates high-resolution images that not only preserve intricate details but also reflect human perceptual preferences. Extensive experiments validate that Hero-SR achieves state-of-the-art performance in Real-SR. The code will be publicly available upon paper acceptance.

Title: Fast Occupancy Network

• Authors: Mingjie Lu, Yuanxian Huang, Ji Liu, Xingliang Huang, Dong Li, Jinzhang Peng, Lu Tian, Emad Barsoum
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07163
• Pdf URL: https://arxiv.org/pdf/2412.07163
• Copy Paste: [[2412.07163]] Fast Occupancy Network(https://arxiv.org/abs/2412.07163)
• Keywords: segmentation
• Abstract: Occupancy Network has recently attracted much attention in autonomous driving. Instead of monocular 3D detection and recent bird's eye view(BEV) models predicting 3D bounding box of obstacles, Occupancy Network predicts the category of voxel in specified 3D space around the ego vehicle via transforming 3D detection task into 3D voxel segmentation task, which has much superiority in tackling category outlier obstacles and providing fine-grained 3D representation. However, existing methods usually require huge computation resources than previous methods, which hinder the Occupancy Network solution applying in intelligent driving systems. To address this problem, we make an analysis of the bottleneck of Occupancy Network inference cost, and present a simple and fast Occupancy Network model, which adopts a deformable 2D convolutional layer to lift BEV feature to 3D voxel feature and presents an efficient voxel feature pyramid network (FPN) module to improve performance with few computational cost. Further, we present a cost-free 2D segmentation branch in perspective view after feature extractors for Occupancy Network during inference phase to improve accuracy. Experimental results demonstrate that our method consistently outperforms existing methods in both accuracy and inference speed, which surpasses recent state-of-the-art (SOTA) OCCNet by 1.7% with ResNet50 backbone with about 3X inference speedup. Furthermore, our method can be easily applied to existing BEV models to transform them into Occupancy Network models.

Title: Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models

• Authors: Haoran Lian, Junmin Chen, Wei Huang, Yizhe Xiong, Wenping Hu, Guiguang Ding, Hui Chen, Jianwei Niu, Zijia Lin, Fuzheng Zhang, Di Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07171
• Pdf URL: https://arxiv.org/pdf/2412.07171
• Copy Paste: [[2412.07171]] Breaking the Stage Barrier: A Novel Single-Stage Approach to Long Context Extension for Large Language Models(https://arxiv.org/abs/2412.07171)
• Keywords: large language model
• Abstract: Recently, Large language models (LLMs) have revolutionized Natural Language Processing (NLP). Pretrained LLMs, due to limited training context size, struggle with handling long token sequences, limiting their performance on various downstream tasks. Current solutions toward long context modeling often employ multi-stage continual pertaining, which progressively increases the effective context length through several continual pretraining stages. However, those approaches require extensive manual tuning and human expertise. In this paper, we introduce a novel single-stage continual pretraining method, Head-Adaptive Rotary Position Encoding (HARPE), to equip LLMs with long context modeling capabilities while simplifying the training process. Our HARPE leverages different Rotary Position Encoding (RoPE) base frequency values across different attention heads and directly trains LLMs on the target context length. Extensive experiments on 4 language modeling benchmarks, including the latest RULER benchmark, demonstrate that HARPE excels in understanding and integrating long-context tasks with single-stage training, matching and even outperforming existing multi-stage methods. Our results highlight that HARPE successfully breaks the stage barrier for training LLMs with long context modeling capabilities.

Title: Post-Training Statistical Calibration for Higher Activation Sparsity

• Authors: Vui Seng Chua, Yujie Pan, Nilesh Jain
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07174
• Pdf URL: https://arxiv.org/pdf/2412.07174
• Copy Paste: [[2412.07174]] Post-Training Statistical Calibration for Higher Activation Sparsity(https://arxiv.org/abs/2412.07174)
• Keywords: robust, transformer
• Abstract: We present Statistical Calibrated Activation Pruning (SCAP), a post-training activation pruning framework that (1) generalizes sparsification by input activations of Fully-Connected layers for generic and flexible application across Transformers, and (2) features a simple Mode-Centering technique to pre-calibrate activation distributions for maximizing post-training sparsity. Our results demonstrate robust Pareto efficiency compared to prior methods, translating to a 1.5x additional LLM decoding speedup against CATS at iso model quality. SCAP effectiveness is empirically verified across a wide range of models, including recent Transformer Decoders, MoE, Mamba2, Encoding Transformer, and pre-quantized models, highlighting its practicality and scalability. The code is available at: this https URL.

Title: An Enhancement of CNN Algorithm for Rice Leaf Disease Image Classification in Mobile Applications

• Authors: Kayne Uriel K. Rodrigo, Jerriane Hillary Heart S. Marcial, Samuel C. Brillo, Khatalyn E. Mata, Jonathan C. Morano
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07182
• Pdf URL: https://arxiv.org/pdf/2412.07182
• Copy Paste: [[2412.07182]] An Enhancement of CNN Algorithm for Rice Leaf Disease Image Classification in Mobile Applications(https://arxiv.org/abs/2412.07182)
• Keywords: robust, extraction, interpretability, transformer
• Abstract: This study focuses on enhancing rice leaf disease image classification algorithms, which have traditionally relied on Convolutional Neural Network (CNN) models. We employed transfer learning with MobileViTV2_050 using ImageNet-1k weights, a lightweight model that integrates CNN's local feature extraction with Vision Transformers' global context learning through a separable self-attention mechanism. Our approach resulted in a significant 15.66% improvement in classification accuracy for MobileViTV2_050-A, our first enhanced model trained on the baseline dataset, achieving 93.14%. Furthermore, MobileViTV2_050-B, our second enhanced model trained on a broader rice leaf dataset, demonstrated a 22.12% improvement, reaching 99.6% test accuracy. Additionally, MobileViTV2-A attained an F1-score of 93% across four rice labels and a Receiver Operating Characteristic (ROC) curve ranging from 87% to 97%. In terms of resource consumption, our enhanced models reduced the total parameters of the baseline CNN model by up to 92.50%, from 14 million to 1.1 million. These results indicate that MobileViTV2_050 not only improves computational efficiency through its separable self-attention mechanism but also enhances global context learning. Consequently, it offers a lightweight and robust solution suitable for mobile deployment, advancing the interpretability and practicality of models in precision agriculture.

Title: Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomaly

• Authors: Hang Du, Guoshun Nan, Jiawen Qian, Wangchenhui Wu, Wendi Deng, Hanqing Mu, Zhenyan Chen, Pengxuan Mao, Xiaofeng Tao, Jun Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07183
• Pdf URL: https://arxiv.org/pdf/2412.07183
• Copy Paste: [[2412.07183]] Exploring What Why and How: A Multifaceted Benchmark for Causation Understanding of Video Anomaly(https://arxiv.org/abs/2412.07183)
• Keywords: large language model
• Abstract: Recent advancements in video anomaly understanding (VAU) have opened the door to groundbreaking applications in various fields, such as traffic monitoring and industrial automation. While the current benchmarks in VAU predominantly emphasize the detection and localization of anomalies. Here, we endeavor to delve deeper into the practical aspects of VAU by addressing the essential questions: "what anomaly occurred?", "why did it happen?", and "how severe is this abnormal event?". In pursuit of these answers, we introduce a comprehensive benchmark for Exploring the Causation of Video Anomalies (ECVA). Our benchmark is meticulously designed, with each video accompanied by detailed human annotations. Specifically, each instance of our ECVA involves three sets of human annotations to indicate "what", "why" and "how" of an anomaly, including 1) anomaly type, start and end times, and event descriptions, 2) natural language explanations for the cause of an anomaly, and 3) free text reflecting the effect of the abnormality. Building upon this foundation, we propose a novel prompt-based methodology that serves as a baseline for tackling the intricate challenges posed by ECVA. We utilize "hard prompt" to guide the model to focus on the critical parts related to video anomaly segments, and "soft prompt" to establish temporal and spatial relationships within these anomaly segments. Furthermore, we propose AnomEval, a specialized evaluation metric crafted to align closely with human judgment criteria for ECVA. This metric leverages the unique features of the ECVA dataset to provide a more comprehensive and reliable assessment of various video large language models. We demonstrate the efficacy of our approach through rigorous experimental analysis and delineate possible avenues for further investigation into the comprehension of video anomaly causation.

Title: A New Federated Learning Framework Against Gradient Inversion Attacks

• Authors: Pengxin Guo, Shuang Zeng, Wenhao Chen, Xiaodan Zhang, Weihong Ren, Yuyin Zhou, Liangqiong Qu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.07187
• Pdf URL: https://arxiv.org/pdf/2412.07187
• Copy Paste: [[2412.07187]] A New Federated Learning Framework Against Gradient Inversion Attacks(https://arxiv.org/abs/2412.07187)
• Keywords: secure, privacy, protect, attack, federate
• Abstract: Federated Learning (FL) aims to protect data privacy by enabling clients to collectively train machine learning models without sharing their raw data. However, recent studies demonstrate that information exchanged during FL is subject to Gradient Inversion Attacks (GIA) and, consequently, a variety of privacy-preserving methods have been integrated into FL to thwart such attacks, such as Secure Multi-party Computing (SMC), Homomorphic Encryption (HE), and Differential Privacy (DP). Despite their ability to protect data privacy, these approaches inherently involve substantial privacy-utility trade-offs. By revisiting the key to privacy exposure in FL under GIA, which lies in the frequent sharing of model gradients that contain private data, we take a new perspective by designing a novel privacy preserve FL framework that effectively ``breaks the direct connection'' between the shared parameters and the local private data to defend against GIA. Specifically, we propose a Hypernetwork Federated Learning (HyperFL) framework that utilizes hypernetworks to generate the parameters of the local model and only the hypernetwork parameters are uploaded to the server for aggregation. Theoretical analyses demonstrate the convergence rate of the proposed HyperFL, while extensive experimental results show the privacy-preserving capability and comparable performance of HyperFL. Code is available at this https URL.

Title: A Step towards Automated and Generalizable Tactile Map Generation using Generative Adversarial Networks

• Authors: David G Hobson, Majid Komeili
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07191
• Pdf URL: https://arxiv.org/pdf/2412.07191
• Copy Paste: [[2412.07191]] A Step towards Automated and Generalizable Tactile Map Generation using Generative Adversarial Networks(https://arxiv.org/abs/2412.07191)
• Keywords: generative
• Abstract: Blindness and visual impairments affect many people worldwide. For help with navigation, people with visual impairments often rely on tactile maps that utilize raised surfaces and edges to convey information through touch. Although these maps are helpful, they are often not widely available and current tools to automate their production have similar limitations including only working at certain scales, for particular world regions, or adhering to specific tactile map standards. To address these shortcomings, we train a proof-of-concept model as a first step towards applying computer vision techniques to help automate the generation of tactile maps. We create a first-of-its-kind tactile maps dataset of street-views from Google Maps spanning 6500 locations and including different tactile line- and area-like features. Generative adversarial network (GAN) models trained on a single zoom successfully identify key map elements, remove extraneous ones, and perform inpainting with median F1 and intersection-over-union (IoU) scores of better than 0.97 across all features. Models trained on two zooms experience only minor drops in performance, and generalize well both to unseen map scales and world regions. Finally, we discuss future directions towards a full implementation of a tactile map solution that builds on our results.

Title: PrisonBreak: Jailbreaking Large Language Models with Fewer Than Twenty-Five Targeted Bit-flips

• Authors: Zachary Coalson, Jeonghyun Woo, Shiyang Chen, Yu Sun, Lishan Yang, Prashant Nair, Bo Fang, Sanghyun Hong
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07192
• Pdf URL: https://arxiv.org/pdf/2412.07192
• Copy Paste: [[2412.07192]] PrisonBreak: Jailbreaking Large Language Models with Fewer Than Twenty-Five Targeted Bit-flips(https://arxiv.org/abs/2412.07192)
• Keywords: secure, protect, attack, large language model
• Abstract: We introduce a new class of attacks on commercial-scale (human-aligned) language models that induce jailbreaking through targeted bitwise corruptions in model parameters. Our adversary can jailbreak billion-parameter language models with fewer than 25 bit-flips in all cases$-$and as few as 5 in some$-$using up to 40$\times$ less bit-flips than existing attacks on computer vision models at least 100$\times$ smaller. Unlike prompt-based jailbreaks, our attack renders these models in memory 'uncensored' at runtime, allowing them to generate harmful responses without any input modifications. Our attack algorithm efficiently identifies target bits to flip, offering up to 20$\times$ more computational efficiency than previous methods. This makes it practical for language models with billions of parameters. We show an end-to-end exploitation of our attack using software-induced fault injection, Rowhammer (RH). Our work examines 56 DRAM RH profiles from DDR4 and LPDDR4X devices with different RH vulnerabilities. We show that our attack can reliably induce jailbreaking in systems similar to those affected by prior bit-flip attacks. Moreover, our approach remains effective even against highly RH-secure systems (e.g., 46$\times$ more secure than previously tested systems). Our analyses further reveal that: (1) models with less post-training alignment require fewer bit flips to jailbreak; (2) certain model components, such as value projection layers, are substantially more vulnerable than others; and (3) our method is mechanistically different than existing jailbreaks. Our findings highlight a pressing, practical threat to the language model ecosystem and underscore the need for research to protect these models from bit-flip attacks.

Title: A Progressive Image Restoration Network for High-order Degradation Imaging in Remote Sensing

• Authors: Yujie Feng, Yin Yang, Xiaohong Fan, Zhengpeng Zhang, Lijing Bu, Jianping Zhang
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.07195
• Pdf URL: https://arxiv.org/pdf/2412.07195
• Copy Paste: [[2412.07195]] A Progressive Image Restoration Network for High-order Degradation Imaging in Remote Sensing(https://arxiv.org/abs/2412.07195)
• Keywords: interpretability
• Abstract: Recently, deep learning methods have gained remarkable achievements in the field of image restoration for remote sensing (RS). However, most existing RS image restoration methods focus mainly on conventional first-order degradation models, which may not effectively capture the imaging mechanisms of remote sensing images. Furthermore, many RS image restoration approaches that use deep learning are often criticized for their lacks of architecture transparency and model interpretability. To address these problems, we propose a novel progressive restoration network for high-order degradation imaging (HDI-PRNet), to progressively restore different image degradation. HDI-PRNet is developed based on the theoretical framework of degradation imaging, offering the benefit of mathematical interpretability within the unfolding network. The framework is composed of three main components: a module for image denoising that relies on proximal mapping prior learning, a module for image deblurring that integrates Neumann series expansion with dual-domain degradation learning, and a module for super-resolution. Extensive experiments demonstrate that our method achieves superior performance on both synthetic and real remote sensing images.

Title: Fine-grained Text to Image Synthesis

• Authors: Xu Ouyang, Ying Chen, Kaiyue Zhu, Gady Agam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07196
• Pdf URL: https://arxiv.org/pdf/2412.07196
• Copy Paste: [[2412.07196]] Fine-grained Text to Image Synthesis(https://arxiv.org/abs/2412.07196)
• Keywords: generative
• Abstract: Fine-grained text to image synthesis involves generating images from texts that belong to different categories. In contrast to general text to image synthesis, in fine-grained synthesis there is high similarity between images of different subclasses, and there may be linguistic discrepancy among texts describing the same image. Recent Generative Adversarial Networks (GAN), such as the Recurrent Affine Transformation (RAT) GAN model, are able to synthesize clear and realistic images from texts. However, GAN models ignore fine-grained level information. In this paper we propose an approach that incorporates an auxiliary classifier in the discriminator and a contrastive learning method to improve the accuracy of fine-grained details in images synthesized by RAT GAN. The auxiliary classifier helps the discriminator classify the class of images, and helps the generator synthesize more accurate fine-grained images. The contrastive learning method minimizes the similarity between images from different subclasses and maximizes the similarity between images from the same subclass. We evaluate on several state-of-the-art methods on the commonly used CUB-200-2011 bird dataset and Oxford-102 flower dataset, and demonstrated superior performance.

Title: Hierarchical Split Federated Learning: Convergence Analysis and System Optimization

• Authors: Zheng Lin, Wei Wei, Zhe Chen, Chan-Tong Lam, Xianhao Chen, Yue Gao, Jun Luo
• Subjects: cs.LG, cs.AI, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2412.07197
• Pdf URL: https://arxiv.org/pdf/2412.07197
• Copy Paste: [[2412.07197]] Hierarchical Split Federated Learning: Convergence Analysis and System Optimization(https://arxiv.org/abs/2412.07197)
• Keywords: federate
• Abstract: As AI models expand in size, it has become increasingly challenging to deploy federated learning (FL) on resource-constrained edge devices. To tackle this issue, split federated learning (SFL) has emerged as an FL framework with reduced workload on edge devices via model splitting; it has received extensive attention from the research community in recent years. Nevertheless, most prior works on SFL focus only on a two-tier architecture without harnessing multi-tier cloudedge computing resources. In this paper, we intend to analyze and optimize the learning performance of SFL under multi-tier systems. Specifically, we propose the hierarchical SFL (HSFL) framework and derive its convergence bound. Based on the theoretical results, we formulate a joint optimization problem for model splitting (MS) and model aggregation (MA). To solve this rather hard problem, we then decompose it into MS and MA subproblems that can be solved via an iterative descending algorithm. Simulation results demonstrate that the tailored algorithm can effectively optimize MS and MA for SFL within virtually any multi-tier system.

Title: A Parametric Approach to Adversarial Augmentation for Cross-Domain Iris Presentation Attack Detection

• Authors: Debasmita Pal, Redwan Sony, Arun Ross
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07199
• Pdf URL: https://arxiv.org/pdf/2412.07199
• Copy Paste: [[2412.07199]] A Parametric Approach to Adversarial Augmentation for Cross-Domain Iris Presentation Attack Detection(https://arxiv.org/abs/2412.07199)
• Keywords: attack, biometric
• Abstract: Iris-based biometric systems are vulnerable to presentation attacks (PAs), where adversaries present physical artifacts (e.g., printed iris images, textured contact lenses) to defeat the system. This has led to the development of various presentation attack detection (PAD) algorithms, which typically perform well in intra-domain settings. However, they often struggle to generalize effectively in cross-domain scenarios, where training and testing employ different sensors, PA instruments, and datasets. In this work, we use adversarial training samples of both bonafide irides and PAs to improve the cross-domain performance of a PAD classifier. The novelty of our approach lies in leveraging transformation parameters from classical data augmentation schemes (e.g., translation, rotation) to generate adversarial samples. We achieve this through a convolutional autoencoder, ADV-GEN, that inputs original training samples along with a set of geometric and photometric transformations. The transformation parameters act as regularization variables, guiding ADV-GEN to generate adversarial samples in a constrained search space. Experiments conducted on the LivDet-Iris 2017 database, comprising four datasets, and the LivDet-Iris 2020 dataset, demonstrate the efficacy of our proposed method. The code is available at this https URL.

Title: A Review on the Applications of Transformer-based language models for Nucleotide Sequence Analysis

• Authors: Nimisha Ghosh, Daniele Santoni, Indrajit Saha, Giovanni Felici
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07201
• Pdf URL: https://arxiv.org/pdf/2412.07201
• Copy Paste: [[2412.07201]] A Review on the Applications of Transformer-based language models for Nucleotide Sequence Analysis(https://arxiv.org/abs/2412.07201)
• Keywords: transformer
• Abstract: In recent times, Transformer-based language models are making quite an impact in the field of natural language processing. As relevant parallels can be drawn between biological sequences and natural languages, the models used in NLP can be easily extended and adapted for various applications in bioinformatics. In this regard, this paper introduces the major developments of Transformer-based models in the recent past in the context of nucleotide sequences. We have reviewed and analysed a large number of application-based papers on this subject, giving evidence of the main characterizing features and to different approaches that may be adopted to customize such powerful computational machines. We have also provided a structured description of the functioning of Transformers, that may enable even first time users to grab the essence of such complex architectures. We believe this review will help the scientific community in understanding the various applications of Transformer-based language models to nucleotide sequences. This work will motivate the readers to build on these methodologies to tackle also various other problems in the field of bioinformatics.

Title: BrokerChain: A Blockchain Sharding Protocol by Exploiting Broker Accounts

• Authors: Huawei Huang, Zhaokang Yin, Qinde Chen, Guang Ye, Xiaowen Peng, Yue Lin, Zibin Zheng, Song Guo
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.07202
• Pdf URL: https://arxiv.org/pdf/2412.07202
• Copy Paste: [[2412.07202]] BrokerChain: A Blockchain Sharding Protocol by Exploiting Broker Accounts(https://arxiv.org/abs/2412.07202)
• Keywords: security, segmentation
• Abstract: State-of-the-art blockchain sharding solutions such as Monoxide, can cause severely imbalanced distribution of transaction (TX) workloads across all blockchain shards due to the deployment policy of their accounts. Imbalanced TX distributions then produce hot shards, in which the cross-shard TXs may experience an unlimited confirmation latency. Thus, how to address the hot-shard issue and how to reduce crossshard TXs become significant challenges of blockchain sharding. Through reviewing the related studies, we find that a crossshard TX protocol that can achieve workload balance among all shards and simultaneously reduce the quantity of crossshard TXs is still absent from the literature. To this end, we propose BrokerChain, which is a cross-shard blockchain protocol dedicated to account-based state sharding. Essentially, BrokerChain exploits fine-grained state partition and account segmentation. We also elaborate on how BrokerChain handles cross-shard TXs through broker accounts. The security issues and other properties of BrokerChain are analyzed rigorously. Finally, we conduct comprehensive evaluations using an opensource blockchain sharding prototype named BlockEmulator. The evaluation results show that BrokerChain outperforms other baselines in terms of transaction throughput, transaction confirmation latency, the queue size of the transaction pool, and workload balance.

Title: Crack-EdgeSAM Self-Prompting Crack Segmentation System for Edge Devices

• Authors: Yingchu Wang, Ji He, Shijie Yu
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.07205
• Pdf URL: https://arxiv.org/pdf/2412.07205
• Copy Paste: [[2412.07205]] Crack-EdgeSAM Self-Prompting Crack Segmentation System for Edge Devices(https://arxiv.org/abs/2412.07205)
• Keywords: segmentation
• Abstract: Structural health monitoring (SHM) is essential for the early detection of infrastructure defects, such as cracks in concrete bridge pier. but often faces challenges in efficiency and accuracy in complex environments. Although the Segment Anything Model (SAM) achieves excellent segmentation performance, its computational demands limit its suitability for real-time applications on edge devices. To address these challenges, this paper proposes Crack-EdgeSAM, a self-prompting crack segmentation system that integrates YOLOv8 for generating prompt boxes and a fine-tuned EdgeSAM model for crack segmentation. To ensure computational efficiency, the method employs ConvLoRA, a Parameter-Efficient Fine-Tuning (PEFT) technique, along with DiceFocalLoss to fine-tune the EdgeSAM model. Our experimental results on public datasets and the climbing robot automatic inspections demonstrate that the system achieves high segmentation accuracy and significantly enhanced inference speed compared to the most recent methods. Notably, the system processes 1024 x 1024 pixels images at 46 FPS on our PC and 8 FPS on Jetson Orin Nano.

Title: MAPLE: A Framework for Active Preference Learning Guided by Large Language Models

• Authors: Saaduddin Mahmud, Mason Nakamura, Shlomo Zilberstein
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07207
• Pdf URL: https://arxiv.org/pdf/2412.07207
• Copy Paste: [[2412.07207]] MAPLE: A Framework for Active Preference Learning Guided by Large Language Models(https://arxiv.org/abs/2412.07207)
• Keywords: interpretability, large language model
• Abstract: The advent of large language models (LLMs) has sparked significant interest in using natural language for preference learning. However, existing methods often suffer from high computational burdens, taxing human supervision, and lack of interpretability. To address these issues, we introduce MAPLE, a framework for large language model-guided Bayesian active preference learning. MAPLE leverages LLMs to model the distribution over preference functions, conditioning it on both natural language feedback and conventional preference learning feedback, such as pairwise trajectory rankings. MAPLE also employs active learning to systematically reduce uncertainty in this distribution and incorporates a language-conditioned active query selection mechanism to identify informative and easy-to-answer queries, thus reducing human burden. We evaluate MAPLE's sample efficiency and preference inference quality across two benchmarks, including a real-world vehicle route planning benchmark using OpenStreetMap data. Our results demonstrate that MAPLE accelerates the learning process and effectively improves humans' ability to answer queries.

Title: Comateformer: Combined Attention Transformer for Semantic Sentence Matching

• Authors: Bo Li, Di Liang, Zixin Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07220
• Pdf URL: https://arxiv.org/pdf/2412.07220
• Copy Paste: [[2412.07220]] Comateformer: Combined Attention Transformer for Semantic Sentence Matching(https://arxiv.org/abs/2412.07220)
• Keywords: robust, transformer
• Abstract: The Transformer-based model have made significant strides in semantic matching tasks by capturing connections between phrase pairs. However, to assess the relevance of sentence pairs, it is insufficient to just examine the general similarity between the sentences. It is crucial to also consider the tiny subtleties that differentiate them from each other. Regrettably, attention softmax operations in transformers tend to miss these subtle differences. To this end, in this work, we propose a novel semantic sentence matching model named Combined Attention Network based on Transformer model (Comateformer). In Comateformer model, we design a novel transformer-based quasi-attention mechanism with compositional properties. Unlike traditional attention mechanisms that merely adjust the weights of input tokens, our proposed method learns how to combine, subtract, or resize specific vectors when building a representation. Moreover, our proposed approach builds on the intuition of similarity and dissimilarity (negative affinity) when calculating dual affinity scores. This allows for a more meaningful representation of relationships between sentences. To evaluate the performance of our proposed model, we conducted extensive experiments on ten public real-world datasets and robustness testing. Experimental results show that our method achieves consistent improvements.

Title: EchoIR: Advancing Image Restoration with Echo Upsampling and Bi-Level Optimization

• Authors: Yuhan He, Yuchun He
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.07225
• Pdf URL: https://arxiv.org/pdf/2412.07225
• Copy Paste: [[2412.07225]] EchoIR: Advancing Image Restoration with Echo Upsampling and Bi-Level Optimization(https://arxiv.org/abs/2412.07225)
• Keywords: transformer
• Abstract: Image restoration represents a fundamental challenge in low-level vision, focusing on reconstructing high-quality images from their degraded counterparts. With the rapid advancement of deep learning technologies, transformer-based methods with pyramid structures have advanced the field by capturing long-range cross-scale spatial interaction. Despite its popularity, the degradation of essential features during the upsampling process notably compromised the restoration performance, resulting in suboptimal reconstruction outcomes. We introduce the EchoIR, an UNet-like image restoration network with a bilateral learnable upsampling mechanism to bridge this gap. Specifically, we proposed the Echo-Upsampler that optimizes the upsampling process by learning from the bilateral intermediate features of U-Net, the "Echo", aiming for a more refined restoration by minimizing the degradation during upsampling. In pursuit of modeling a hierarchical model of image restoration and upsampling tasks, we propose the Approximated Sequential Bi-level Optimization (AS-BLO), an advanced bi-level optimization model establishing a relationship between upsampling learning and image restoration tasks. Extensive experiments against the state-of-the-art (SOTA) methods demonstrate the proposed EchoIR surpasses the existing methods, achieving SOTA performance in image restoration tasks.

Title: Moderating the Generalization of Score-based Generative Model

• Authors: Wan Jiang, He Wang, Xin Zhang, Dan Guo, Zhaoxin Fan, Yunfeng Diao, Richang Hong
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07229
• Pdf URL: https://arxiv.org/pdf/2412.07229
• Copy Paste: [[2412.07229]] Moderating the Generalization of Score-based Generative Model(https://arxiv.org/abs/2412.07229)
• Keywords: diffusion, generative
• Abstract: Score-based Generative Models (SGMs) have demonstrated remarkable generalization abilities, e.g. generating unseen, but natural data. However, the greater the generalization power, the more likely the unintended generalization, and the more dangerous the abuse. Research on moderated generalization in SGMs remains limited. To fill this gap, we first examine the current 'gold standard' in Machine Unlearning (MU), i.e., re-training the model after removing the undesirable training data, and find it does not work in SGMs. Further analysis of score functions reveals that the MU 'gold standard' does not alter the original score function, which explains its ineffectiveness. Based on this insight, we propose the first Moderated Score-based Generative Model (MSGM), which introduces a novel score adjustment strategy that redirects the score function away from undesirable data during the continuous-time stochastic differential equation process. Extensive experimental results demonstrate that MSGM significantly reduces the likelihood of generating undesirable content while preserving high visual quality for normal image generation. Albeit designed for SGMs, MSGM is a general and flexible MU framework that is compatible with diverse diffusion architectures (SGM and DDPM) and training strategies (re-training and fine-tuning), and enables zero-shot transfer of the pre-trained models to downstream tasks, e.g. image inpainting and reconstruction. The code will be shared upon acceptance.

Title: Repetitive Action Counting with Hybrid Temporal Relation Modeling

• Authors: Kun Li, Xinge Peng, Dan Guo, Xun Yang, Meng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07233
• Pdf URL: https://arxiv.org/pdf/2412.07233
• Copy Paste: [[2412.07233]] Repetitive Action Counting with Hybrid Temporal Relation Modeling(https://arxiv.org/abs/2412.07233)
• Keywords: robust
• Abstract: Repetitive Action Counting (RAC) aims to count the number of repetitive actions occurring in videos. In the real world, repetitive actions have great diversity and bring numerous challenges (e.g., viewpoint changes, non-uniform periods, and action interruptions). Existing methods based on the temporal self-similarity matrix (TSSM) for RAC are trapped in the bottleneck of insufficient capturing action periods when applied to complicated daily videos. To tackle this issue, we propose a novel method named Hybrid Temporal Relation Modeling Network (HTRM-Net) to build diverse TSSM for RAC. The HTRM-Net mainly consists of three key components: bi-modal temporal self-similarity matrix modeling, random matrix dropping, and local temporal context modeling. Specifically, we construct temporal self-similarity matrices by bi-modal (self-attention and dual-softmax) operations, yielding diverse matrix representations from the combination of row-wise and column-wise correlations. To further enhance matrix representations, we propose incorporating a random matrix dropping module to guide channel-wise learning of the matrix explicitly. After that, we inject the local temporal context of video frames and the learned matrix into temporal correlation modeling, which can make the model robust enough to cope with error-prone situations, such as action interruption. Finally, a multi-scale matrix fusion module is designed to aggregate temporal correlations adaptively in multi-scale matrices. Extensive experiments across intra- and cross-datasets demonstrate that the proposed method not only outperforms current state-of-the-art methods but also exhibits robust capabilities in accurately counting repetitive actions in unseen action categories. Notably, our method surpasses the classical TransRAC method by 20.04\% in MAE and 22.76\% in OBO.

Title: ArtFormer: Controllable Generation of Diverse 3D Articulated Objects

• Authors: Jiayi Su, Youhe Feng, Zheng Li, Jinhua Song, Yangfan He, Botao Ren, Botian Xu
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.07237
• Pdf URL: https://arxiv.org/pdf/2412.07237
• Copy Paste: [[2412.07237]] ArtFormer: Controllable Generation of Diverse 3D Articulated Objects(https://arxiv.org/abs/2412.07237)
• Keywords: transformer
• Abstract: This paper presents a novel framework for modeling and conditional generation of 3D articulated objects. Troubled by flexibility-quality tradeoffs, existing methods are often limited to using predefined structures or retrieving shapes from static datasets. To address these challenges, we parameterize an articulated object as a tree of tokens and employ a transformer to generate both the object's high-level geometry code and its kinematic relations. Subsequently, each sub-part's geometry is further decoded using a signed-distance-function (SDF) shape prior, facilitating the synthesis of high-quality 3D shapes. Our approach enables the generation of diverse objects with high-quality geometry and varying number of parts. Comprehensive experiments on conditional generation from text descriptions demonstrate the effectiveness and flexibility of our method.

Title: Filling Memory Gaps: Enhancing Continual Semantic Parsing via SQL Syntax Variance-Guided LLMs without Real Data Replay

• Authors: Ruiheng Liu, Jinyu Zhang, Yanqi Song, Yu Zhang, Bailong Yang
• Subjects: cs.CL, cs.DB
• Abstract URL: https://arxiv.org/abs/2412.07246
• Pdf URL: https://arxiv.org/pdf/2412.07246
• Copy Paste: [[2412.07246]] Filling Memory Gaps: Enhancing Continual Semantic Parsing via SQL Syntax Variance-Guided LLMs without Real Data Replay(https://arxiv.org/abs/2412.07246)
• Keywords: privacy, large language model
• Abstract: Continual Semantic Parsing (CSP) aims to train parsers to convert natural language questions into SQL across tasks with limited annotated examples, adapting to the real-world scenario of dynamically updated databases. Previous studies mitigate this challenge by replaying historical data or employing parameter-efficient tuning (PET), but they often violate data privacy or rely on ideal continual learning settings. To address these problems, we propose a new Large Language Model (LLM)-Enhanced Continuous Semantic Parsing method, named LECSP, which alleviates forgetting while encouraging generalization, without requiring real data replay or ideal settings. Specifically, it first analyzes the commonalities and differences between tasks from the SQL syntax perspective to guide LLMs in reconstructing key memories and improving memory accuracy through a calibration strategy. Then, it uses a task-aware dual-teacher distillation framework to promote the accumulation and transfer of knowledge during sequential training. Experimental results on two CSP benchmarks show that our method significantly outperforms existing methods, even those utilizing data replay or ideal settings. Additionally, we achieve generalization performance beyond the upper limits, better adapting to unseen tasks.

Title: Buster: Incorporating Backdoor Attacks into Text Encoder to Mitigate NSFW Content Generation

• Authors: Xin Zhao, Xiaojun Chen, Yuexin Xuan, Zhendong Zhao
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07249
• Pdf URL: https://arxiv.org/pdf/2412.07249
• Copy Paste: [[2412.07249]] Buster: Incorporating Backdoor Attacks into Text Encoder to Mitigate NSFW Content Generation(https://arxiv.org/abs/2412.07249)
• Keywords: defense, attack
• Abstract: In the digital age, the proliferation of deep learning models has led to significant concerns about the generation of Not Safe for Work (NSFW) content. Existing defense methods primarily involve model fine-tuning and post-hoc content moderation. However, these approaches often lack scalability in eliminating harmful content, degrade the quality of benign image generation, or incur high inference costs. To tackle these challenges, we propose an innovative framework called \textbf{Buster}, which injects backdoor attacks into the text encoder to prevent NSFW content generation. Specifically, Buster leverages deep semantic information rather than explicit prompts as triggers, redirecting NSFW prompts towards targeted benign prompts. This approach demonstrates exceptional resilience and scalability in mitigating NSFW content. Remarkably, Buster fine-tunes the text encoder of Text-to-Image models within just five minutes, showcasing high efficiency. Our extensive experiments reveal that Buster outperforms all other baselines, achieving superior NSFW content removal rate while preserving the quality of harmless images.

Title: KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context

• Authors: Xiaonan Wang, Jinyoung Yeo, Joon-Ho Lim, Hansaem Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07251
• Pdf URL: https://arxiv.org/pdf/2412.07251
• Copy Paste: [[2412.07251]] KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context(https://arxiv.org/abs/2412.07251)
• Keywords: large language model
• Abstract: Large language models have exhibited significant enhancements in performance across various tasks. However, the complexity of their evaluation increases as these models generate more fluent and coherent content. Current multilingual benchmarks often use translated English versions, which may incorporate Western cultural biases that do not accurately assess other languages and cultures. To address this research gap, we introduce KULTURE Bench, an evaluation framework specifically designed for Korean culture that features datasets of cultural news, idioms, and poetry. It is designed to assess language models' cultural comprehension and reasoning capabilities at the word, sentence, and paragraph levels. Using the KULTURE Bench, we assessed the capabilities of models trained with different language corpora and analyzed the results comprehensively. The results show that there is still significant room for improvement in the models' understanding of texts related to the deeper aspects of Korean culture.

Title: CapGen:An Environment-Adaptive Generator of Adversarial Patches

• Authors: Chaoqun Li, Zhuodong Liu, Huanqian Yan, Hang Su
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07253
• Pdf URL: https://arxiv.org/pdf/2412.07253
• Copy Paste: [[2412.07253]] CapGen:An Environment-Adaptive Generator of Adversarial Patches(https://arxiv.org/abs/2412.07253)
• Keywords: protect, attack, robust, steal
• Abstract: Adversarial patches, often used to provide physical stealth protection for critical assets and assess perception algorithm robustness, usually neglect the need for visual harmony with the background environment, making them easily noticeable. Moreover, existing methods primarily concentrate on improving attack performance, disregarding the intricate dynamics of adversarial patch elements. In this work, we introduce the Camouflaged Adversarial Pattern Generator (CAPGen), a novel approach that leverages specific base colors from the surrounding environment to produce patches that seamlessly blend with their background for superior visual stealthiness while maintaining robust adversarial performance. We delve into the influence of both patterns (i.e., color-agnostic texture information) and colors on the effectiveness of attacks facilitated by patches, discovering that patterns exert a more pronounced effect on performance than colors. Based on these findings, we propose a rapid generation strategy for adversarial patches. This involves updating the colors of high-performance adversarial patches to align with those of the new environment, ensuring visual stealthiness without compromising adversarial impact. This paper is the first to comprehensively examine the roles played by patterns and colors in the context of adversarial patches.

Title: Label-Confidence-Aware Uncertainty Estimation in Natural Language Generation

• Authors: Qinhong Lin, Linna Zhou, Zhongliang Yang, Yuang Cai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07255
• Pdf URL: https://arxiv.org/pdf/2412.07255
• Copy Paste: [[2412.07255]] Label-Confidence-Aware Uncertainty Estimation in Natural Language Generation(https://arxiv.org/abs/2412.07255)
• Keywords: robust, generative, large language model
• Abstract: Large Language Models (LLMs) display formidable capabilities in generative tasks but also pose potential risks due to their tendency to generate hallucinatory responses. Uncertainty Quantification (UQ), the evaluation of model output reliability, is crucial for ensuring the safety and robustness of AI systems. Recent studies have concentrated on model uncertainty by analyzing the relationship between output entropy under various sampling conditions and the corresponding labels. However, these methods primarily focus on measuring model entropy with precision to capture response characteristics, often neglecting the uncertainties associated with greedy decoding results-the sources of model labels, which can lead to biased classification outcomes. In this paper, we explore the biases introduced by greedy decoding and propose a label-confidence-aware (LCA) uncertainty estimation based on Kullback-Leibler (KL) divergence bridging between samples and label source, thus enhancing the reliability and stability of uncertainty assessments. Our empirical evaluations across a range of popular LLMs and NLP datasets reveal that different label sources can indeed affect classification, and that our approach can effectively capture differences in sampling results and label sources, demonstrating more effective uncertainty estimation.

Title: DFREC: DeepFake Identity Recovery Based on Identity-aware Masked Autoencoder

• Authors: Peipeng Yu, Hui Gao, Zhitao Huang, Zhihua Xia, Chip-Hong Chang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07260
• Pdf URL: https://arxiv.org/pdf/2412.07260
• Copy Paste: [[2412.07260]] DFREC: DeepFake Identity Recovery Based on Identity-aware Masked Autoencoder(https://arxiv.org/abs/2412.07260)
• Keywords: attack, interpretability, segmentation
• Abstract: Recent advances in deepfake forensics have primarily focused on improving the classification accuracy and generalization performance. Despite enormous progress in detection accuracy across a wide variety of forgery algorithms, existing algorithms lack intuitive interpretability and identity traceability to help with forensic investigation. In this paper, we introduce a novel DeepFake Identity Recovery scheme (DFREC) to fill this gap. DFREC aims to recover the pair of source and target faces from a deepfake image to facilitate deepfake identity tracing and reduce the risk of deepfake attack. It comprises three key components: an Identity Segmentation Module (ISM), a Source Identity Reconstruction Module (SIRM), and a Target Identity Reconstruction Module (TIRM). The ISM segments the input face into distinct source and target face information, and the SIRM reconstructs the source face and extracts latent target identity features with the segmented source information. The background context and latent target identity features are synergetically fused by a Masked Autoencoder in the TIRM to reconstruct the target face. We evaluate DFREC on six different high-fidelity face-swapping attacks on FaceForensics++, CelebaMegaFS and FFHQ-E4S datasets, which demonstrate its superior recovery performance over state-of-the-art deepfake recovery algorithms. In addition, DFREC is the only scheme that can recover both pristine source and target faces directly from the forgery image with high fadelity.

Title: MemHunter: Automated and Verifiable Memorization Detection at Dataset-scale in LLMs

• Authors: Zhenpeng Wu, Jian Lou, Zibin Zheng, Chuan Chen
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07261
• Pdf URL: https://arxiv.org/pdf/2412.07261
• Copy Paste: [[2412.07261]] MemHunter: Automated and Verifiable Memorization Detection at Dataset-scale in LLMs(https://arxiv.org/abs/2412.07261)
• Keywords: privacy, large language model
• Abstract: Large language models (LLMs) have been shown to memorize and reproduce content from their training data, raising significant privacy concerns, especially with web-scale datasets. Existing methods for detecting memorization are largely sample-specific, relying on manually crafted or discretely optimized memory-inducing prompts generated on a per-sample basis, which become impractical for dataset-level detection due to the prohibitive computational cost of iterating over all samples. In real-world scenarios, data owners may need to verify whether a susceptible LLM has memorized their dataset, particularly if the LLM may have collected the data from the web without authorization. To address this, we introduce \textit{MemHunter}, which trains a memory-inducing LLM and employs hypothesis testing to efficiently detect memorization at the dataset level, without requiring sample-specific memory inducing. Experiments on models such as Pythia and Llama-2 demonstrate that \textit{MemHunter} can extract up to 40\% more training data than existing methods under constrained time resources and reduce search time by up to 80\% when integrated as a plug-in. Crucially, \textit{MemHunter} is the first method capable of dataset-level memorization detection, providing an indispensable tool for assessing privacy risks in LLMs that are powered by vast web-sourced datasets.

Title: A Generative Victim Model for Segmentation

• Authors: Aixuan Li, Jing Zhang, Jiawei Shi, Yiran Zhong, Yuchao Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07274
• Pdf URL: https://arxiv.org/pdf/2412.07274
• Copy Paste: [[2412.07274]] A Generative Victim Model for Segmentation(https://arxiv.org/abs/2412.07274)
• Keywords: attack, robust, generative, segmentation
• Abstract: We find that the well-trained victim models (VMs), against which the attacks are generated, serve as fundamental prerequisites for adversarial attacks, i.e. a segmentation VM is needed to generate attacks for segmentation. In this context, the victim model is assumed to be robust to achieve effective adversarial perturbation generation. Instead of focusing on improving the robustness of the task-specific victim models, we shift our attention to image generation. From an image generation perspective, we derive a novel VM for segmentation, aiming to generate adversarial perturbations for segmentation tasks without requiring models explicitly designed for image segmentation. Our approach to adversarial attack generation diverges from conventional white-box or black-box attacks, offering a fresh outlook on adversarial attack strategies. Experiments show that our attack method is able to generate effective adversarial attacks with good transferability.

Title: Backdoor Attacks against No-Reference Image Quality Assessment Models via A Scalable Trigger

• Authors: Yi Yu, Song Xia, Xun Lin, Wenhan Yang, Shijian Lu, Yap-peng Tan, Alex Kot
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.07277
• Pdf URL: https://arxiv.org/pdf/2412.07277
• Copy Paste: [[2412.07277]] Backdoor Attacks against No-Reference Image Quality Assessment Models via A Scalable Trigger(https://arxiv.org/abs/2412.07277)
• Keywords: attack
• Abstract: No-Reference Image Quality Assessment (NR-IQA), responsible for assessing the quality of a single input image without using any reference, plays a critical role in evaluating and optimizing computer vision systems, e.g., low-light enhancement. Recent research indicates that NR-IQA models are susceptible to adversarial attacks, which can significantly alter predicted scores with visually imperceptible perturbations. Despite revealing vulnerabilities, these attack methods have limitations, including high computational demands, untargeted manipulation, limited practical utility in white-box scenarios, and reduced effectiveness in black-box scenarios. To address these challenges, we shift our focus to another significant threat and present a novel poisoning-based backdoor attack against NR-IQA (BAIQA), allowing the attacker to manipulate the IQA model's output to any desired target value by simply adjusting a scaling coefficient $\alpha$ for the trigger. We propose to inject the trigger in the discrete cosine transform (DCT) domain to improve the local invariance of the trigger for countering trigger diminishment in NR-IQA models due to widely adopted data augmentations. Furthermore, the universal adversarial perturbations (UAP) in the DCT space are designed as the trigger, to increase IQA model susceptibility to manipulation and improve attack effectiveness. In addition to the heuristic method for poison-label BAIQA (P-BAIQA), we explore the design of clean-label BAIQA (C-BAIQA), focusing on $\alpha$ sampling and image data refinement, driven by theoretical insights we reveal. Extensive experiments on diverse datasets and various NR-IQA models demonstrate the effectiveness of our attacks. Code will be released at this https URL.

Title: HARP: Hesitation-Aware Reframing in Transformer Inference Pass

• Authors: Romain Storaï, Seung-won Hwang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07282
• Pdf URL: https://arxiv.org/pdf/2412.07282
• Copy Paste: [[2412.07282]] HARP: Hesitation-Aware Reframing in Transformer Inference Pass(https://arxiv.org/abs/2412.07282)
• Keywords: transformer, large language model
• Abstract: This paper aims to improve the performance of large language models by addressing the variable computational demands in inference steps, where some tokens require more computational resources than others. We present HARP, a simple modification to "off-the-shelf" Transformer forward pass. Drawing from hesitation and the framing effect in decision-making, HARP selectively applies additional computation when the model encounters uncertainty during token generation. Our method mimics human cognitive processes by pausing at difficult decision points and reframing inputs for a different perspective. Unlike other approaches, HARP is model-agnostic, training-free, and easy to implement. We thoroughly evaluate our method across various downstream tasks and model sizes, demonstrating performance improvements up to +5.16%. Notably, HARP achieves these gains while maintaining inference times twice faster than beam search. Simple and yet with significant gains, HARP offers a practical solution for enhancing the performance of Transformer-based language models with minimal computational impact.

Title: Image Classification Using Singular Value Decomposition and Optimization

• Authors: Isabela M. Yepes, Manasvi Goyal
• Subjects: cs.CV, math.NA
• Abstract URL: https://arxiv.org/abs/2412.07288
• Pdf URL: https://arxiv.org/pdf/2412.07288
• Copy Paste: [[2412.07288]] Image Classification Using Singular Value Decomposition and Optimization(https://arxiv.org/abs/2412.07288)
• Keywords: robust
• Abstract: This study investigates the applicability of Singular Value Decomposition for the image classification of specific breeds of cats and dogs using fur color as the primary identifying feature. Sequential Quadratic Programming (SQP) is employed to construct optimally weighted templates. The proposed method achieves 69% accuracy using the Frobenius norm at rank 10. The results partially validate the assumption that dominant features, such as fur color, can be effectively captured through low-rank approximations. However, the accuracy suggests that additional features or methods may be required for more robust classification, highlighting the trade-off between simplicity and performance in resource-constrained environments.

Title: Enhancing Relation Extraction via Supervised Rationale Verification and Feedback

• Authors: Yongqi Li, Xin Miao, Shen Zhou, Mayi Xu, Yuyang Ren, Tieyun Qian
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07289
• Pdf URL: https://arxiv.org/pdf/2412.07289
• Copy Paste: [[2412.07289]] Enhancing Relation Extraction via Supervised Rationale Verification and Feedback(https://arxiv.org/abs/2412.07289)
• Keywords: extraction, large language model
• Abstract: Despite the rapid progress that existing automated feedback methods have made in correcting the output of large language models (LLMs), these methods cannot be well applied to the relation extraction (RE) task due to their designated feedback objectives and correction manner. To address this problem, we propose a novel automated feedback framework for RE, which presents a rationale supervisor to verify the rationale and provide re-selected demonstrations as feedback to correct the initial prediction. Specifically, we first design a causal intervention and observation method for to collect biased/unbiased rationales for contrastive training the rationale supervisor. Then, we present a verification-feedback-correction procedure to iteratively enhance LLMs' capability of handling the RE task. Extensive experiments prove that our proposed framework significantly outperforms existing methods.

Title: The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model

• Authors: Jiawei Chen, Wentao Chen, Jing Su, Jingjing Xu, Hongyu Lin, Mengjie Ren, Yaojie Lu, Xianpei Han, Le Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07298
• Pdf URL: https://arxiv.org/pdf/2412.07298
• Copy Paste: [[2412.07298]] The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model(https://arxiv.org/abs/2412.07298)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown significant multilingual capabilities. However, the mechanisms underlying the development of these capabilities during pre-training are not well understood. In this paper, we use code LLMs as an experimental platform to explore the evolution of multilingual capabilities in LLMs during the pre-training process. Based on our observations, we propose the Babel Tower Hypothesis, which describes the entire process of LLMs acquiring new language capabilities. During the learning process, multiple languages initially share a single knowledge system dominated by the primary language and gradually develop language-specific knowledge systems. We then validate the above hypothesis by tracking the internal states of the LLMs through identifying working languages and language transferring neurons. Experimental results show that the internal state changes of the LLM are consistent with our Babel Tower Hypothesis. Building on these insights, we propose a novel method to construct an optimized pre-training corpus for multilingual code LLMs, which significantly outperforms LLMs trained on the original corpus. The proposed Babel Tower Hypothesis provides new insights into designing pre-training data distributions to achieve optimal multilingual capabilities in LLMs.

Title: Compression of Large-Scale 3D Point Clouds Based on Joint Optimization of Point Sampling and Feature Extraction

• Authors: Jae-Young Yim, Jae-Young Sim
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.07302
• Pdf URL: https://arxiv.org/pdf/2412.07302
• Copy Paste: [[2412.07302]] Compression of Large-Scale 3D Point Clouds Based on Joint Optimization of Point Sampling and Feature Extraction(https://arxiv.org/abs/2412.07302)
• Keywords: extraction
• Abstract: Large-scale 3D point clouds (LS3DPC) obtained by LiDAR scanners require huge storage space and transmission bandwidth due to a large amount of data. The existing methods of LS3DPC compression separately perform rule-based point sampling and learnable feature extraction, and hence achieve limited compression performance. In this paper, we propose a fully end-to-end training framework for LS3DPC compression where the point sampling and the feature extraction are jointly optimized in terms of the rate and distortion losses. To this end, we first make the point sampling module to be trainable such that an optimal position of the downsampled point is estimated via aggregation with learnable weights. We also develop a reliable point reconstruction scheme that adaptively aggregates the expanded candidate points to refine the positions of upsampled points. Experimental results evaluated on the SemanticKITTI and nuScenes datasets show that the proposed method achieves significantly higher compression ratios compared with the existing state-of-the-art methods.

Title: FaceX: Understanding Face Attribute Classifiers through Summary Model Explanations

• Authors: Ioannis Sarridis, Christos Koutlis, Symeon Papadopoulos, Christos Diou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07313
• Pdf URL: https://arxiv.org/pdf/2412.07313
• Copy Paste: [[2412.07313]] FaceX: Understanding Face Attribute Classifiers through Summary Model Explanations(https://arxiv.org/abs/2412.07313)
• Keywords: fair, interpretability
• Abstract: EXplainable Artificial Intelligence (XAI) approaches are widely applied for identifying fairness issues in Artificial Intelligence (AI) systems. However, in the context of facial analysis, existing XAI approaches, such as pixel attribution methods, offer explanations for individual images, posing challenges in assessing the overall behavior of a model, which would require labor-intensive manual inspection of a very large number of instances and leaving to the human the task of drawing a general impression of the model behavior from the individual outputs. Addressing this limitation, we introduce FaceX, the first method that provides a comprehensive understanding of face attribute classifiers through summary model explanations. Specifically, FaceX leverages the presence of distinct regions across all facial images to compute a region-level aggregation of model activations, allowing for the visualization of the model's region attribution across 19 predefined regions of interest in facial images, such as hair, ears, or skin. Beyond spatial explanations, FaceX enhances interpretability by visualizing specific image patches with the highest impact on the model's decisions for each facial region within a test benchmark. Through extensive evaluation in various experimental setups, including scenarios with or without intentional biases and mitigation efforts on four benchmarks, namely CelebA, FairFace, CelebAMask-HQ, and Racial Faces in the Wild, FaceX demonstrates high effectiveness in identifying the models' biases.

Title: CoMA: Compositional Human Motion Generation with Multi-modal Agents

• Authors: Shanlin Sun, Gabriel De Araujo, Jiaqi Xu, Shenghan Zhou, Hanwen Zhang, Ziheng Huang, Chenyu You, Xiaohui Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07320
• Pdf URL: https://arxiv.org/pdf/2412.07320
• Copy Paste: [[2412.07320]] CoMA: Compositional Human Motion Generation with Multi-modal Agents(https://arxiv.org/abs/2412.07320)
• Keywords: transformer
• Abstract: 3D human motion generation has seen substantial advancement in recent years. While state-of-the-art approaches have improved performance significantly, they still struggle with complex and detailed motions unseen in training data, largely due to the scarcity of motion datasets and the prohibitive cost of generating new training examples. To address these challenges, we introduce CoMA, an agent-based solution for complex human motion generation, editing, and comprehension. CoMA leverages multiple collaborative agents powered by large language and vision models, alongside a mask transformer-based motion generator featuring body part-specific encoders and codebooks for fine-grained control. Our framework enables generation of both short and long motion sequences with detailed instructions, text-guided motion editing, and self-correction for improved quality. Evaluations on the HumanML3D dataset demonstrate competitive performance against state-of-the-art methods. Additionally, we create a set of context-rich, compositional, and long text prompts, where user studies show our method significantly outperforms existing approaches.

Title: ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC)

• Authors: Kartik Singhal, Gautam Shroff
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07322
• Pdf URL: https://arxiv.org/pdf/2412.07322
• Copy Paste: [[2412.07322]] ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC)(https://arxiv.org/abs/2412.07322)
• Keywords: large language model
• Abstract: The Abstraction and Reasoning Corpus (ARC) poses a significant challenge to artificial intelligence, demanding broad generalization and few-shot learning capabilities that remain elusive for current deep learning methods, including large language models (LLMs). While LLMs excel in program synthesis, their direct application to ARC yields limited success. To address this, we introduce ConceptSearch, a novel function-search algorithm that leverages LLMs for program generation and employs a concept-based scoring method to guide the search efficiently. Unlike simplistic pixel-based metrics like Hamming distance, ConceptSearch evaluates programs on their ability to capture the underlying transformation concept reflected in the input-output examples. We explore three scoring functions: Hamming distance, a CNN-based scoring function, and an LLM-based natural language scoring function. Experimental results demonstrate the effectiveness of ConceptSearch, achieving a significant performance improvement over direct prompting with GPT-4. Moreover, our novel concept-based scoring exhibits up to 30% greater efficiency compared to Hamming distance, measured in terms of the number of iterations required to reach the correct solution. These findings highlight the potential of LLM-driven program search when integrated with concept-based guidance for tackling challenging generalization problems like ARC. Code: this https URL

Title: Addressing Key Challenges of Adversarial Attacks and Defenses in the Tabular Domain: A Methodological Framework for Coherence and Consistency

• Authors: Yael Itzhakev, Amit Giloni, Yuval Elovici, Asaf Shabtai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07326
• Pdf URL: https://arxiv.org/pdf/2412.07326
• Copy Paste: [[2412.07326]] Addressing Key Challenges of Adversarial Attacks and Defenses in the Tabular Domain: A Methodological Framework for Coherence and Consistency(https://arxiv.org/abs/2412.07326)
• Keywords: defense, attack, explainability
• Abstract: Machine learning models trained on tabular data are vulnerable to adversarial attacks, even in realistic scenarios where attackers have access only to the model's outputs. Researchers evaluate such attacks by considering metrics like success rate, perturbation magnitude, and query count. However, unlike other data domains, the tabular domain contains complex interdependencies among features, presenting a unique aspect that should be evaluated: the need for the attack to generate coherent samples and ensure feature consistency for indistinguishability. Currently, there is no established methodology for evaluating adversarial samples based on these criteria. In this paper, we address this gap by proposing new evaluation criteria tailored for tabular attacks' quality; we defined anomaly-based framework to assess the distinguishability of adversarial samples and utilize the SHAP explainability technique to identify inconsistencies in the model's decision-making process caused by adversarial samples. These criteria could form the basis for potential detection methods and be integrated into established evaluation metrics for assessing attack's quality Additionally, we introduce a novel technique for perturbing dependent features while maintaining coherence and feature consistency within the sample. We compare different attacks' strategies, examining black-box query-based attacks and transferability-based gradient attacks across four target models. Our experiments, conducted on benchmark tabular datasets, reveal significant differences between the examined attacks' strategies in terms of the attacker's risk and effort and the attacks' quality. The findings provide valuable insights on the strengths, limitations, and trade-offs of various adversarial attacks in the tabular domain, laying a foundation for future research on attacks and defense development.

Title: Fusion Embedding for Pose-Guided Person Image Synthesis with Diffusion Model

• Authors: Donghwna Lee, Kyungha Min, Kirok Kim, Seyoung Jeong, Jiwoo Jeong, Wooju Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07333
• Pdf URL: https://arxiv.org/pdf/2412.07333
• Copy Paste: [[2412.07333]] Fusion Embedding for Pose-Guided Person Image Synthesis with Diffusion Model(https://arxiv.org/abs/2412.07333)
• Keywords: diffusion, generative
• Abstract: Pose-Guided Person Image Synthesis (PGPIS) aims to synthesize high-quality person images corresponding to target poses while preserving the appearance of the source image. Recently, PGPIS methods that use diffusion models have achieved competitive performance. Most approaches involve extracting representations of the target pose and source image and learning their relationships in the generative model's training process. This approach makes it difficult to learn the semantic relationships between the input and target images and complicates the model structure needed to enhance generation results. To address these issues, we propose Fusion embedding for PGPIS using a Diffusion Model (FPDM). Inspired by the successful application of pre-trained CLIP models in text-to-image diffusion models, our method consists of two stages. The first stage involves training the fusion embedding of the source image and target pose to align with the target image's embedding. In the second stage, the generative model uses this fusion embedding as a condition to generate the target image. We applied the proposed method to the benchmark datasets DeepFashion and RWTH-PHOENIX-Weather 2014T, and conducted both quantitative and qualitative evaluations, demonstrating state-of-the-art (SOTA) performance. An ablation study of the model structure showed that even a model using only the second stage achieved performance close to the other PGPIS SOTA models. The code is available at this https URL.

Title: Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generation

• Authors: Pedro H. V. Valois, Lincon S. Souza, Erica K. Shimomoto, Kazuhiro Fukui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07334
• Pdf URL: https://arxiv.org/pdf/2412.07334
• Copy Paste: [[2412.07334]] Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generation(https://arxiv.org/abs/2412.07334)
• Keywords: robust, interpretability, large language model
• Abstract: Interpretability is a key challenge in fostering trust for Large Language Models (LLMs), which stems from the complexity of extracting reasoning from model's parameters. We present the Frame Representation Hypothesis, a theoretically robust framework grounded in the Linear Representation Hypothesis (LRH) to interpret and control LLMs by modeling multi-token words. Prior research explored LRH to connect LLM representations with linguistic concepts, but was limited to single token analysis. As most words are composed of several tokens, we extend LRH to multi-token words, thereby enabling usage on any textual data with thousands of concepts. To this end, we propose words can be interpreted as frames, ordered sequences of vectors that better capture token-word relationships. Then, concepts can be represented as the average of word frames sharing a common concept. We showcase these tools through Top-k Concept-Guided Decoding, which can intuitively steer text generation using concepts of choice. We verify said ideas on Llama 3.1, Gemma 2, and Phi 3 families, demonstrating gender and language biases, exposing harmful content, but also potential to remediate them, leading to safer and more transparent LLMs. Code is available at this https URL

Title: Efficient 3D Recognition with Event-driven Spike Sparse Convolution

• Authors: Xuerui Qiu, Man Yao, Jieyuan Zhang, Yuhong Chou, Ning Qiao, Shibo Zhou, Bo Xu, Guoqi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07360
• Pdf URL: https://arxiv.org/pdf/2412.07360
• Copy Paste: [[2412.07360]] Efficient 3D Recognition with Event-driven Spike Sparse Convolution(https://arxiv.org/abs/2412.07360)
• Keywords: extraction, segmentation
• Abstract: Spiking Neural Networks (SNNs) provide an energy-efficient way to extract 3D spatio-temporal features. Point clouds are sparse 3D spatial data, which suggests that SNNs should be well-suited for processing them. However, when applying SNNs to point clouds, they often exhibit limited performance and fewer application scenarios. We attribute this to inappropriate preprocessing and feature extraction methods. To address this issue, we first introduce the Spike Voxel Coding (SVC) scheme, which encodes the 3D point clouds into a sparse spike train space, reducing the storage requirements and saving time on point cloud preprocessing. Then, we propose a Spike Sparse Convolution (SSC) model for efficiently extracting 3D sparse point cloud features. Combining SVC and SSC, we design an efficient 3D SNN backbone (E-3DSNN), which is friendly with neuromorphic hardware. For instance, SSC can be implemented on neuromorphic chips with only minor modifications to the addressing function of vanilla spike convolution. Experiments on ModelNet40, KITTI, and Semantic KITTI datasets demonstrate that E-3DSNN achieves state-of-the-art (SOTA) results with remarkable efficiency. Notably, our E-3DSNN (1.87M) obtained 91.7\% top-1 accuracy on ModelNet40, surpassing the current best SNN baselines (14.3M) by 3.0\%. To our best knowledge, it is the first direct training 3D SNN backbone that can simultaneously handle various 3D computer vision tasks (e.g., classification, detection, and segmentation) with an event-driven nature. Code is available: this https URL.

Title: My Words Imply Your Opinion: Reader Agent-Based Propagation Enhancement for Personalized Implicit Emotion Analysis

• Authors: Jian Liao, Yu Feng, Xiaoyu Wang, Suge Wang, Jianxing Zheng, Deyu Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07367
• Pdf URL: https://arxiv.org/pdf/2412.07367
• Copy Paste: [[2412.07367]] My Words Imply Your Opinion: Reader Agent-Based Propagation Enhancement for Personalized Implicit Emotion Analysis(https://arxiv.org/abs/2412.07367)
• Keywords: large language model
• Abstract: In implicit emotion analysis (IEA), the subtlety of emotional expressions makes it particularly sensitive to user-specific characteristics. Existing studies often inject personalization into the analysis by focusing on the authorial dimension of the emotional text. However, these methods overlook the potential influence of the intended reader on the reaction of implicit emotions. In this paper, we refine the IEA task to Personalized Implicit Emotion Analysis (PIEA) and introduce the RAPPIE model, a novel framework designed to address the issue of missing user information within this task. In particular, 1) we create reader agents based on the Large Language Model to simulate reader reactions, to address challenges of the spiral of silence and data incompleteness encountered when acquiring reader feedback information. 2) We establish a reader propagation role system and develop a role-aware emotion propagation multi-view graph learning model, which effectively deals with the sparsity of reader information by utilizing the distribution of propagation roles. 3) We annotate two Chinese PIEA datasets with detailed user metadata, thereby addressing the limitation of prior datasets that primarily focus on textual content annotation. Extensive experiments on these datasets indicate that the RAPPIE model outperforms current state-of-the-art baselines, highlighting the significance and efficacy of incorporating reader feedback into the PIEA process.

Title: PRM: Photometric Stereo based Large Reconstruction Model

• Authors: Wenhang Ge, Jiantao Lin, Guibao Shen, Jiawei Feng, Tao Hu, Xinli Xu, Ying-Cong Chen
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.07371
• Pdf URL: https://arxiv.org/pdf/2412.07371
• Copy Paste: [[2412.07371]] PRM: Photometric Stereo based Large Reconstruction Model(https://arxiv.org/abs/2412.07371)
• Keywords: robust
• Abstract: We propose PRM, a novel photometric stereo based large reconstruction model to reconstruct high-quality meshes with fine-grained local details. Unlike previous large reconstruction models that prepare images under fixed and simple lighting as both input and supervision, PRM renders photometric stereo images by varying materials and lighting for the purposes, which not only improves the precise local details by providing rich photometric cues but also increases the model robustness to variations in the appearance of input images. To offer enhanced flexibility of images rendering, we incorporate a real-time physically-based rendering (PBR) method and mesh rasterization for online images rendering. Moreover, in employing an explicit mesh as our 3D representation, PRM ensures the application of differentiable PBR, which supports the utilization of multiple photometric supervisions and better models the specular color for high-quality geometry optimization. Our PRM leverages photometric stereo images to achieve high-quality reconstructions with fine-grained local details, even amidst sophisticated image appearances. Extensive experiments demonstrate that PRM significantly outperforms other models.

Title: CADSpotting: Robust Panoptic Symbol Spotting on Large-Scale CAD Drawings

• Authors: Jiazuo Mu, Fuyi Yang, Yanshun Zhang, Junxiong Zhang, Yongjian Luo, Lan Xu, Yujiao Shi, Jingyi Yu, Yingliang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07377
• Pdf URL: https://arxiv.org/pdf/2412.07377
• Copy Paste: [[2412.07377]] CADSpotting: Robust Panoptic Symbol Spotting on Large-Scale CAD Drawings(https://arxiv.org/abs/2412.07377)
• Keywords: robust, segmentation
• Abstract: We introduce CADSpotting, an efficient method for panoptic symbol spotting in large-scale architectural CAD drawings. Existing approaches struggle with the diversity of symbols, scale variations, and overlapping elements in CAD designs. CADSpotting overcomes these challenges by representing each primitive with dense points instead of a single primitive point, described by essential attributes like coordinates and color. Building upon a unified 3D point cloud model for joint semantic, instance, and panoptic segmentation, CADSpotting learns robust feature representations. To enable accurate segmentation in large, complex drawings, we further propose a novel Sliding Window Aggregation (SWA) technique, combining weighted voting and Non-Maximum Suppression (NMS). Moreover, we introduce a large-scale CAD dataset named LS-CAD to support our experiments. Each floorplan in LS-CAD has an average coverage of 1,000 square meter(versus 100 square meter in the existing dataset), providing a valuable benchmark for symbol spotting research. Experimental results on FloorPlanCAD and LS-CAD datasets demonstrate that CADSpotting outperforms existing methods, showcasing its robustness and scalability for real-world CAD applications.

Title: SpecFuse: Ensembling Large Language Models via Next-Segment Prediction

• Authors: Bo Lv, Chen Tang, Yanan Zhang, Xin Liu, Yue Yu, Ping Luo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07380
• Pdf URL: https://arxiv.org/pdf/2412.07380
• Copy Paste: [[2412.07380]] SpecFuse: Ensembling Large Language Models via Next-Segment Prediction(https://arxiv.org/abs/2412.07380)
• Keywords: generative, large language model
• Abstract: Ensembles of generative large language models (LLMs) can integrate the strengths of different LLMs to compensate for the limitations of individual models. However, recent work has focused on training an additional fusion model to combine complete responses from multiple LLMs, failing to tap into their collaborative potential to generate higher-quality responses. Moreover, as the additional fusion model is trained on a specialized dataset, these methods struggle with generalizing to open-domain queries from online users. In this paper, we propose SpecFuse, a novel ensemble framework that outputs the fused result by iteratively producing the next segment through collaboration among LLMs. This is achieved through cyclic execution of its inference and verification components. In each round, the inference component invokes each base LLM to generate candidate segments in parallel, and the verify component calls these LLMs again to predict the ranking of the segments. The top-ranked segment is then broadcast to all LLMs, encouraging them to generate higher-quality segments in the next round. This approach also allows the base LLMs to be plug-and-play, without any training or adaptation, avoiding generalization limitations. Furthermore, to conserve computational resources, we propose a model exit mechanism that dynamically excludes models exhibiting poor performance in previous rounds during each query response. In this way, it effectively reduces the number of model calls while maintaining overall performance.

Title: LOGen: Toward Lidar Object Generation by Point Diffusion

• Authors: Ellington Kirby, Mickael Chen, Renaud Marlet, Nermin Samet
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07385
• Pdf URL: https://arxiv.org/pdf/2412.07385
• Copy Paste: [[2412.07385]] LOGen: Toward Lidar Object Generation by Point Diffusion(https://arxiv.org/abs/2412.07385)
• Keywords: diffusion, segmentation
• Abstract: A common strategy to improve lidar segmentation results on rare semantic classes consists of pasting objects from one lidar scene into another. While this augments the quantity of instances seen at training time and varies their context, the instances fundamentally remain the same. In this work, we explore how to enhance instance diversity using a lidar object generator. We introduce a novel diffusion-based method to produce lidar point clouds of dataset objects, including reflectance, and with an extensive control of the generation via conditioning information. Our experiments on nuScenes show the quality of our object generations measured with new 3D metrics developed to suit lidar objects.

Title: Algorithmic Phase Transitions in Language Models: A Mechanistic Case Study of Arithmetic

• Authors: Alan Sun, Ethan Sun, Warren Shepard
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07386
• Pdf URL: https://arxiv.org/pdf/2412.07386
• Copy Paste: [[2412.07386]] Algorithmic Phase Transitions in Language Models: A Mechanistic Case Study of Arithmetic(https://arxiv.org/abs/2412.07386)
• Keywords: large language model
• Abstract: Zero-shot capabilities of large language models make them powerful tools for solving a range of tasks without explicit training. It remains unclear, however, how these models achieve such performance, or why they can zero-shot some tasks but not others. In this paper, we shed some light on this phenomenon by defining and investigating algorithmic stability in language models -- changes in problem-solving strategy employed by the model as a result of changes in task specification. We focus on a task where algorithmic stability is needed for generalization: two-operand arithmetic. Surprisingly, we find that Gemma-2-2b employs substantially different computational models on closely related subtasks, i.e. four-digit versus eight-digit addition. Our findings suggest that algorithmic instability may be a contributing factor to language models' poor zero-shot performance across certain logical reasoning tasks, as they struggle to abstract different problem-solving strategies and smoothly transition between them.

Title: Post-Training Non-Uniform Quantization for Convolutional Neural Networks

• Authors: Ahmed Luqman, Khuzemah Qazi, Imdadullah Khan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07391
• Pdf URL: https://arxiv.org/pdf/2412.07391
• Copy Paste: [[2412.07391]] Post-Training Non-Uniform Quantization for Convolutional Neural Networks(https://arxiv.org/abs/2412.07391)
• Keywords: segmentation
• Abstract: Despite the success of CNN models on a variety of Image classification and segmentation tasks, their extensive computational and storage demands pose considerable challenges for real-world deployment on resource constrained devices. Quantization is one technique that aims to alleviate these large storage requirements and speed up the inference process by reducing the precision of model parameters to lower-bit representations. In this paper, we introduce a novel post-training quantization method for model weights. Our method finds optimal clipping thresholds and scaling factors along with mathematical guarantees that our method minimizes quantization noise. Empirical results on Real World Datasets demonstrate that our quantization scheme significantly reduces model size and computational requirements while preserving model accuracy.

Title: Benchmarking Vision-Based Object Tracking for USVs in Complex Maritime Environments

• Authors: Muhayy Ud Din, Ahsan B. Bakht, Waseem Akram, Yihao Dong, Lakmal Seneviratne, Irfan Hussain
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.07392
• Pdf URL: https://arxiv.org/pdf/2412.07392
• Copy Paste: [[2412.07392]] Benchmarking Vision-Based Object Tracking for USVs in Complex Maritime Environments(https://arxiv.org/abs/2412.07392)
• Keywords: robust, transformer
• Abstract: Vision-based target tracking is crucial for unmanned surface vehicles (USVs) to perform tasks such as inspection, monitoring, and surveillance. However, real-time tracking in complex maritime environments is challenging due to dynamic camera movement, low visibility, and scale variation. Typically, object detection methods combined with filtering techniques are commonly used for tracking, but they often lack robustness, particularly in the presence of camera motion and missed detections. Although advanced tracking methods have been proposed recently, their application in maritime scenarios is limited. To address this gap, this study proposes a vision-guided object-tracking framework for USVs, integrating state-of-the-art tracking algorithms with low-level control systems to enable precise tracking in dynamic maritime environments. We benchmarked the performance of seven distinct trackers, developed using advanced deep learning techniques such as Siamese Networks and Transformers, by evaluating them on both simulated and real-world maritime datasets. In addition, we evaluated the robustness of various control algorithms in conjunction with these tracking systems. The proposed framework was validated through simulations and real-world sea experiments, demonstrating its effectiveness in handling dynamic maritime conditions. The results show that SeqTrack, a Transformer-based tracker, performed best in adverse conditions, such as dust storms. Among the control algorithms evaluated, the linear quadratic regulator controller (LQR) demonstrated the most robust and smooth control, allowing for stable tracking of the USV.

Title: CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models

• Authors: Dongfang Li, Zetian Sun, Xinshuo Hu, Baotian Hu, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07393
• Pdf URL: https://arxiv.org/pdf/2412.07393
• Copy Paste: [[2412.07393]] CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Models(https://arxiv.org/abs/2412.07393)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) need to adapt to the continuous changes in data, tasks, and user preferences. Due to their massive size and the high costs associated with training, LLMs are not suitable for frequent retraining. However, updates are necessary to keep them in sync with rapidly evolving human knowledge. To address these challenges, this paper proposes the Compression Memory Training (CMT) method, an efficient and effective online adaptation framework for LLMs that features robust knowledge retention capabilities. Inspired by human memory mechanisms, CMT compresses and extracts information from new documents to be stored in a memory bank. When answering to queries related to these new documents, the model aggregates these document memories from the memory bank to better answer user questions. The parameters of the LLM itself do not change during training and inference, reducing the risk of catastrophic forgetting. To enhance the encoding, retrieval, and aggregation of memory, we further propose three new general and flexible techniques, including memory-aware objective, self-matching and top-aggregation. Extensive experiments conducted on three continual learning datasets (i.e., StreamingQA, SQuAD and ArchivalQA) demonstrate that the proposed method improves model adaptability and robustness across multiple base LLMs (e.g., +4.07 EM & +4.19 F1 in StreamingQA with Llama-2-7b).

Title: Explainability of Deep Learning-Based Plant Disease Classifiers Through Automated Concept Identification

• Authors: Jihen Amara, Birgitta König-Ries, Sheeba Samuel
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07408
• Pdf URL: https://arxiv.org/pdf/2412.07408
• Copy Paste: [[2412.07408]] Explainability of Deep Learning-Based Plant Disease Classifiers Through Automated Concept Identification(https://arxiv.org/abs/2412.07408)
• Keywords: robust, explainability
• Abstract: While deep learning has significantly advanced automatic plant disease detection through image-based classification, improving model explainability remains crucial for reliable disease detection. In this study, we apply the Automated Concept-based Explanation (ACE) method to plant disease classification using the widely adopted InceptionV3 model and the PlantVillage dataset. ACE automatically identifies the visual concepts found in the image data and provides insights about the critical features influencing the model predictions. This approach reveals both effective disease-related patterns and incidental biases, such as those from background or lighting that can compromise model robustness. Through systematic experiments, ACE helped us to identify relevant features and pinpoint areas for targeted model improvement. Our findings demonstrate the potential of ACE to improve the explainability of plant disease classification based on deep learning, which is essential for producing transparent tools for plant disease management in agriculture.

Title: Generating Knowledge Graphs from Large Language Models: A Comparative Study of GPT-4, LLaMA 2, and BERT

• Authors: Ahan Bhatt, Nandan Vaghela, Kush Dudhia
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2412.07412
• Pdf URL: https://arxiv.org/pdf/2412.07412
• Copy Paste: [[2412.07412]] Generating Knowledge Graphs from Large Language Models: A Comparative Study of GPT-4, LLaMA 2, and BERT(https://arxiv.org/abs/2412.07412)
• Keywords: generative, large language model
• Abstract: Knowledge Graphs (KGs) are essential for the functionality of GraphRAGs, a form of Retrieval-Augmented Generative Systems (RAGs) that excel in tasks requiring structured reasoning and semantic understanding. However, creating KGs for GraphRAGs remains a significant challenge due to accuracy and scalability limitations of traditional methods. This paper introduces a novel approach leveraging large language models (LLMs) like GPT-4, LLaMA 2 (13B), and BERT to generate KGs directly from unstructured data, bypassing traditional pipelines. Using metrics such as Precision, Recall, F1-Score, Graph Edit Distance, and Semantic Similarity, we evaluate the models' ability to generate high-quality KGs. Results demonstrate that GPT-4 achieves superior semantic fidelity and structural accuracy, LLaMA 2 excels in lightweight, domain-specific graphs, and BERT provides insights into challenges in entity-relationship modeling. This study underscores the potential of LLMs to streamline KG creation and enhance GraphRAG accessibility for real-world applications, while setting a foundation for future advancements.

Title: Machine Learning Algorithms for Detecting Mental Stress in College Students

• Authors: Ashutosh Singh, Khushdeep Singh, Amit Kumar, Abhishek Shrivastava, Santosh Kumar
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.07415
• Pdf URL: https://arxiv.org/pdf/2412.07415
• Copy Paste: [[2412.07415]] Machine Learning Algorithms for Detecting Mental Stress in College Students(https://arxiv.org/abs/2412.07415)
• Keywords: attack
• Abstract: In today's world, stress is a big problem that affects people's health and happiness. More and more people are feeling stressed out, which can lead to lots of health issues like breathing problems, feeling overwhelmed, heart attack, diabetes, etc. This work endeavors to forecast stress and non-stress occurrences among college students by applying various machine learning algorithms: Decision Trees, Random Forest, Support Vector Machines, AdaBoost, Naive Bayes, Logistic Regression, and K-nearest Neighbors. The primary objective of this work is to leverage a research study to predict and mitigate stress and non-stress based on the collected questionnaire dataset. We conducted a workshop with the primary goal of studying the stress levels found among the students. This workshop was attended by Approximately 843 students aged between 18 to 21 years old. A questionnaire was given to the students validated under the guidance of the experts from the All India Institute of Medical Sciences (AIIMS) Raipur, Chhattisgarh, India, on which our dataset is based. The survey consists of 28 questions, aiming to comprehensively understand the multidimensional aspects of stress, including emotional well-being, physical health, academic performance, relationships, and leisure. This work finds that Support Vector Machines have a maximum accuracy for Stress, reaching 95\%. The study contributes to a deeper understanding of stress determinants. It aims to improve college student's overall quality of life and academic success, addressing the multifaceted nature of stress.

Title: Optimizing Alignment with Less: Leveraging Data Augmentation for Personalized Evaluation

• Authors: Javad Seraj, Mohammad Mahdi Mohajeri, Mohammad Javad Dousti, Majid Nili Ahmadabadi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07429
• Pdf URL: https://arxiv.org/pdf/2412.07429
• Copy Paste: [[2412.07429]] Optimizing Alignment with Less: Leveraging Data Augmentation for Personalized Evaluation(https://arxiv.org/abs/2412.07429)
• Keywords: large language model
• Abstract: Automatic evaluation by large language models (LLMs) is a prominent topic today; however, judgment and evaluation tasks are often subjective and influenced by various factors, making adaptation challenging. While many studies demonstrate the capabilities of state-of-the-art proprietary LLMs in comparison to human evaluators, they often struggle to adapt to reference evaluators over time, a requirement for achieving personalized judgment. Additionally, numerous works have attempted to apply open LLMs as judges or evaluators, but these efforts frequently overlook the limitations of working with scarce data. Personalized judgment is inherently associated with limited data scenarios, which are common in many real-world problems. Our work aims to present a data augmentation technique to select a more effective sample from limited data in order to align an open LLM with human preference. Our work achieves approximately 7% improvements in Pearson correlation with a reference judge over the baseline,and 30% improvement over the base model (Llama3.1-8B-Instruct) in the mathematical reasoning evaluation task. demonstrating that augmenting selecting more effective preference data enables our approach to surpass baseline methods.

Title: BENet: A Cross-domain Robust Network for Detecting Face Forgeries via Bias Expansion and Latent-space Attention

• Authors: Weihua Liu, Jianhua Qiu, Said Boumaraf, Chaochao lin, Pan liyuan, Lin Li, Mohammed Bennamoun, Naoufel Werghi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07431
• Pdf URL: https://arxiv.org/pdf/2412.07431
• Copy Paste: [[2412.07431]] BENet: A Cross-domain Robust Network for Detecting Face Forgeries via Bias Expansion and Latent-space Attention(https://arxiv.org/abs/2412.07431)
• Keywords: defense, robust
• Abstract: In response to the growing threat of deepfake technology, we introduce BENet, a Cross-Domain Robust Bias Expansion Network. BENet enhances the detection of fake faces by addressing limitations in current detectors related to variations across different types of fake face generation techniques, where ``cross-domain" refers to the diverse range of these deepfakes, each considered a separate domain. BENet's core feature is a bias expansion module based on autoencoders. This module maintains genuine facial features while enhancing differences in fake reconstructions, creating a reliable bias for detecting fake faces across various deepfake domains. We also introduce a Latent-Space Attention (LSA) module to capture inconsistencies related to fake faces at different scales, ensuring robust defense against advanced deepfake techniques. The enriched LSA feature maps are multiplied with the expanded bias to create a versatile feature space optimized for subtle forgeries detection. To improve its ability to detect fake faces from unknown sources, BENet integrates a cross-domain detector module that enhances recognition accuracy by verifying the facial domain during inference. We train our network end-to-end with a novel bias expansion loss, adopted for the first time, in face forgery detection. Extensive experiments covering both intra and cross-dataset demonstrate BENet's superiority over current state-of-the-art solutions.

Title: Impact of Sampling Techniques and Data Leakage on XGBoost Performance in Credit Card Fraud Detection

• Authors: Siyaxolisa Kabane
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07437
• Pdf URL: https://arxiv.org/pdf/2412.07437
• Copy Paste: [[2412.07437]] Impact of Sampling Techniques and Data Leakage on XGBoost Performance in Credit Card Fraud Detection(https://arxiv.org/abs/2412.07437)
• Keywords: security
• Abstract: Credit card fraud detection remains a critical challenge in financial security, with machine learning models like XGBoost(eXtreme gradient boosting) emerging as powerful tools for identifying fraudulent transactions. However, the inherent class imbalance in credit card transaction datasets poses significant challenges for model performance. Although sampling techniques are commonly used to address this imbalance, their implementation sometimes precedes the train-test split, potentially introducing data leakage. This study presents a comparative analysis of XGBoost's performance in credit card fraud detection under three scenarios: Firstly without any imbalance handling techniques, secondly with sampling techniques applied only to the training set after the train-test split, and third with sampling techniques applied before the train-test split. We utilized a dataset from Kaggle of 284,807 credit card transactions, containing 0.172\% fraudulent cases, to evaluate these approaches. Our findings show that although sampling strategies enhance model performance, the reliability of results is greatly impacted by when they are applied. Due to a data leakage issue that frequently occurs in machine learning models during the sampling phase, XGBoost models trained on data where sampling was applied prior to the train-test split may have displayed artificially inflated performance metrics. Surprisingly, models trained with sampling techniques applied solely to the training set demonstrated significantly lower results than those with pre-split sampling, all the while preserving the integrity of the evaluation process.

Title: Tazza: Shuffling Neural Network Parameters for Secure and Private Federated Learning

• Authors: Kichang Lee, Jaeho Jin, JaeYeon Park, JeongGil Ko
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07454
• Pdf URL: https://arxiv.org/pdf/2412.07454
• Copy Paste: [[2412.07454]] Tazza: Shuffling Neural Network Parameters for Secure and Private Federated Learning(https://arxiv.org/abs/2412.07454)
• Keywords: secure, security, privacy, defense, attack, robust, federate
• Abstract: Federated learning enables decentralized model training without sharing raw data, preserving data privacy. However, its vulnerability towards critical security threats, such as gradient inversion and model poisoning by malicious clients, remain unresolved. Existing solutions often address these issues separately, sacrificing either system robustness or model accuracy. This work introduces Tazza, a secure and efficient federated learning framework that simultaneously addresses both challenges. By leveraging the permutation equivariance and invariance properties of neural networks via weight shuffling and shuffled model validation, Tazza enhances resilience against diverse poisoning attacks, while ensuring data confidentiality and high model accuracy. Comprehensive evaluations on various datasets and embedded platforms show that Tazza achieves robust defense with up to 6.7x improved computational efficiency compared to alternative schemes, without compromising performance.

Title: AHSG: Adversarial Attacks on High-level Semantics in Graph Neural Networks

• Authors: Kai Yuan, Xiaobing Pei, Haoran Yang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07468
• Pdf URL: https://arxiv.org/pdf/2412.07468
• Copy Paste: [[2412.07468]] AHSG: Adversarial Attacks on High-level Semantics in Graph Neural Networks(https://arxiv.org/abs/2412.07468)
• Keywords: defense, attack
• Abstract: Graph Neural Networks (GNNs) have garnered significant interest among researchers due to their impressive performance in graph learning tasks. However, like other deep neural networks, GNNs are also vulnerable to adversarial attacks. In existing adversarial attack methods for GNNs, the metric between the attacked graph and the original graph is usually the attack budget or a measure of global graph properties. However, we have found that it is possible to generate attack graphs that disrupt the primary semantics even within these constraints. To address this problem, we propose a Adversarial Attacks on High-level Semantics in Graph Neural Networks (AHSG), which is a graph structure attack model that ensures the retention of primary semantics. The latent representations of each node can extract rich semantic information by applying convolutional operations on graph data. These representations contain both task-relevant primary semantic information and task-irrelevant secondary semantic information. The latent representations of same-class nodes with the same primary semantics can fulfill the objective of modifying secondary semantics while preserving the primary semantics. Finally, the latent representations with attack effects is mapped to an attack graph using Projected Gradient Descent (PGD) algorithm. By attacking graph deep learning models with some advanced defense strategies, we validate that AHSG has superior attack effectiveness compared to other attack methods. Additionally, we employ Contextual Stochastic Block Models (CSBMs) as a proxy for the primary semantics to detect the attacked graph, confirming that AHSG almost does not disrupt the original primary semantics of the graph.

Title: Manta: Enhancing Mamba for Few-Shot Action Recognition of Long Sub-Sequence

• Authors: Wenbo Huang, Jinghui Zhang, Guang Li, Lei Zhang, Shuoyuan Wang, Fang Dong, Jiahui Jin, Takahiro Ogawa, Miki Haseyama
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07481
• Pdf URL: https://arxiv.org/pdf/2412.07481
• Copy Paste: [[2412.07481]] Manta: Enhancing Mamba for Few-Shot Action Recognition of Long Sub-Sequence(https://arxiv.org/abs/2412.07481)
• Keywords: transformer
• Abstract: In few-shot action recognition~(FSAR), long sub-sequences of video naturally express entire actions more effectively. However, the computational complexity of mainstream Transformer-based methods limits their application. Recent Mamba demonstrates efficiency in modeling long sequences, but directly applying Mamba to FSAR overlooks the importance of local feature modeling and alignment. Moreover, long sub-sequences within the same class accumulate intra-class variance, which adversely impacts FSAR performance. To solve these challenges, we propose a \underline{\textbf{M}}atryoshka M\underline{\textbf{A}}mba and Co\underline{\textbf{N}}tras\underline{\textbf{T}}ive Le\underline{\textbf{A}}rning framework~(\textbf{Manta}). Firstly, the Matryoshka Mamba introduces multiple Inner Modules to enhance local feature representation, rather than directly modeling global features. An Outer Module captures dependencies of timeline between these local features for implicit temporal alignment. Secondly, a hybrid contrastive learning paradigm, combining both supervised and unsupervised methods, is designed to mitigate the negative effects of intra-class variance accumulation. The Matryoshka Mamba and the hybrid contrastive learning paradigm operate in parallel branches within Manta, enhancing Mamba for FSAR of long sub-sequence. Manta achieves new state-of-the-art performance on prominent benchmarks, including SSv2, Kinetics, UCF101, and HMDB51. Extensive empirical studies prove that Manta significantly improves FSAR of long sub-sequence from multiple perspectives. The code is released at this https URL.

Title: ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learning

• Authors: Hongshu Guo, Zeyuan Ma, Jiacheng Chen, Yining Ma, Zhiguang Cao, Xinglin Zhang, Yue-Jiao Gong
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2412.07507
• Pdf URL: https://arxiv.org/pdf/2412.07507
• Copy Paste: [[2412.07507]] ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learning(https://arxiv.org/abs/2412.07507)
• Keywords: robust, transformer
• Abstract: Recent advances in Meta-learning for Black-Box Optimization (MetaBBO) have shown the potential of using neural networks to dynamically configure evolutionary algorithms (EAs), enhancing their performance and adaptability across various BBO instances. However, they are often tailored to a specific EA, which limits their generalizability and necessitates retraining or redesigns for different EAs and optimization problems. To address this limitation, we introduce ConfigX, a new paradigm of the MetaBBO framework that is capable of learning a universal configuration agent (model) for boosting diverse EAs. To achieve so, our ConfigX first leverages a novel modularization system that enables the flexible combination of various optimization sub-modules to generate diverse EAs during training. Additionally, we propose a Transformer-based neural network to meta-learn a universal configuration policy through multitask reinforcement learning across a designed joint optimization task space. Extensive experiments verify that, our ConfigX, after large-scale pre-training, achieves robust zero-shot generalization to unseen tasks and outperforms state-of-the-art baselines. Moreover, ConfigX exhibits strong lifelong learning capabilities, allowing efficient adaptation to new tasks through fine-tuning. Our proposed ConfigX represents a significant step toward an automatic, all-purpose configuration agent for EAs.

Title: Stealthy and Robust Backdoor Attack against 3D Point Clouds through Additional Point Features

• Authors: Xiaoyang Ning, Qing Xie, Jinyu Xu, Wenbo Jiang, Jiachen Li, Yanchun Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07511
• Pdf URL: https://arxiv.org/pdf/2412.07511
• Copy Paste: [[2412.07511]] Stealthy and Robust Backdoor Attack against 3D Point Clouds through Additional Point Features(https://arxiv.org/abs/2412.07511)
• Keywords: security, defense, attack, robust, steal
• Abstract: Recently, 3D backdoor attacks have posed a substantial threat to 3D Deep Neural Networks (3D DNNs) designed for 3D point clouds, which are extensively deployed in various security-critical applications. Although the existing 3D backdoor attacks achieved high attack performance, they remain vulnerable to preprocessing-based defenses (e.g., outlier removal and rotation augmentation) and are prone to detection by human inspection. In pursuit of a more challenging-to-defend and stealthy 3D backdoor attack, this paper introduces the Stealthy and Robust Backdoor Attack (SRBA), which ensures robustness and stealthiness through intentional design considerations. The key insight of our attack involves applying a uniform shift to the additional point features of point clouds (e.g., reflection intensity) widely utilized as part of inputs for 3D DNNs as the trigger. Without altering the geometric information of the point clouds, our attack ensures visual consistency between poisoned and benign samples, and demonstrate robustness against preprocessing-based defenses. In addition, to automate our attack, we employ Bayesian Optimization (BO) to identify the suitable trigger. Extensive experiments suggest that SRBA achieves an attack success rate (ASR) exceeding 94% in all cases, and significantly outperforms previous SOTA methods when multiple preprocessing operations are applied during training.

Title: CoPrUS: Consistency Preserving Utterance Synthesis towards more realistic benchmark dialogues

• Authors: Sebastian Steindl, Ulrich Schäfer, Bernd Ludwig
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07515
• Pdf URL: https://arxiv.org/pdf/2412.07515
• Copy Paste: [[2412.07515]] CoPrUS: Consistency Preserving Utterance Synthesis towards more realistic benchmark dialogues(https://arxiv.org/abs/2412.07515)
• Keywords: large language model
• Abstract: Large-scale Wizard-Of-Oz dialogue datasets have enabled the training of deep learning-based dialogue systems. While they are successful as benchmark datasets, they lack certain types of utterances, which would make them more realistic. In this work, we investigate the creation of synthetic communication errors in an automatic pipeline. Based on linguistic theory, we propose and follow a simple error taxonomy. We focus on three types of miscommunications that could happen in real-world dialogues but are underrepresented in the benchmark dataset: misunderstandings, non-understandings and vaguely related questions. Our two-step approach uses a state-of-the-art Large Language Model (LLM) to first create the error and secondly the repairing utterance. We perform Language Model-based evaluation to ensure the quality of the generated utterances. We apply the method to the MultiWOZ dataset and evaluate it both qualitatively and empirically as well as with human judges. Our results indicate that current LLMs can aid in adding post-hoc miscommunications to benchmark datasets as a form of data augmentation. We publish the resulting dataset, in which nearly 1900 dialogues have been modified, as CoPrUS-MultiWOZ to facilitate future work on dialogue systems.

Title: Quantifying the Prediction Uncertainty of Machine Learning Models for Individual Data

• Authors: Koby Bibas
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2412.07520
• Pdf URL: https://arxiv.org/pdf/2412.07520
• Copy Paste: [[2412.07520]] Quantifying the Prediction Uncertainty of Machine Learning Models for Individual Data(https://arxiv.org/abs/2412.07520)
• Keywords: attack, robust
• Abstract: Machine learning models have exhibited exceptional results in various domains. The most prevalent approach for learning is the empirical risk minimizer (ERM), which adapts the model's weights to reduce the loss on a training set and subsequently leverages these weights to predict the label for new test data. Nonetheless, ERM makes the assumption that the test distribution is similar to the training distribution, which may not always hold in real-world situations. In contrast, the predictive normalized maximum likelihood (pNML) was proposed as a min-max solution for the individual setting where no assumptions are made on the distribution of the tested input. This study investigates pNML's learnability for linear regression and neural networks, and demonstrates that pNML can improve the performance and robustness of these models on various tasks. Moreover, the pNML provides an accurate confidence measure for its output, showcasing state-of-the-art results for out-of-distribution detection, resistance to adversarial attacks, and active learning.

Title: ReCap: Better Gaussian Relighting with Cross-Environment Captures

• Authors: Jingzhi Li, Zongwei Wu, Eduard Zamfir, Radu Timofte
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07534
• Pdf URL: https://arxiv.org/pdf/2412.07534
• Copy Paste: [[2412.07534]] ReCap: Better Gaussian Relighting with Cross-Environment Captures(https://arxiv.org/abs/2412.07534)
• Keywords: robust
• Abstract: Accurate 3D objects relighting in diverse unseen environments is crucial for realistic virtual object placement. Due to the albedo-lighting ambiguity, existing methods often fall short in producing faithful relights. Without proper constraints, observed training views can be explained by numerous combinations of lighting and material attributes, lacking physical correspondence with the actual environment maps used for relighting. In this work, we present ReCap, treating cross-environment captures as multi-task target to provide the missing supervision that cuts through the entanglement. Specifically, ReCap jointly optimizes multiple lighting representations that share a common set of material attributes. This naturally harmonizes a coherent set of lighting representations around the mutual material attributes, exploiting commonalities and differences across varied object appearances. Such coherence enables physically sound lighting reconstruction and robust material estimation - both essential for accurate relighting. Together with a streamlined shading function and effective post-processing, ReCap outperforms the leading competitor by 3.4 dB in PSNR on an expanded relighting benchmark.

Title: Can Neural Decompilation Assist Vulnerability Prediction on Binary Code?

• Authors: D. Cotroneo, F. C. Grasso, R. Natella, V. Orbinato
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07538
• Pdf URL: https://arxiv.org/pdf/2412.07538
• Copy Paste: [[2412.07538]] Can Neural Decompilation Assist Vulnerability Prediction on Binary Code?(https://arxiv.org/abs/2412.07538)
• Keywords: security
• Abstract: Vulnerability prediction is valuable in identifying security issues more efficiently, even though it requires the source code of the target software system, which is a restrictive hypothesis. This paper presents an experimental study to predict vulnerabilities in binary code without source code or complex representations of the binary, leveraging the pivotal idea of decompiling the binary file through neural decompilation and predicting vulnerabilities through deep learning on the decompiled source code. The results outperform the state-of-the-art in both neural decompilation and vulnerability prediction, showing that it is possible to identify vulnerable programs with this approach concerning bi-class (vulnerable/non-vulnerable) and multi-class (type of vulnerability) analysis.

Title: Anomaly detection using Diffusion-based methods

• Authors: Aryan Bhosale, Samrat Mukherjee, Biplab Banerjee, Fabio Cuzzolin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07539
• Pdf URL: https://arxiv.org/pdf/2412.07539
• Copy Paste: [[2412.07539]] Anomaly detection using Diffusion-based methods(https://arxiv.org/abs/2412.07539)
• Keywords: robust, diffusion, transformer
• Abstract: This paper explores the utility of diffusion-based models for anomaly detection, focusing on their efficacy in identifying deviations in both compact and high-resolution datasets. Diffusion-based architectures, including Denoising Diffusion Probabilistic Models (DDPMs) and Diffusion Transformers (DiTs), are evaluated for their performance using reconstruction objectives. By leveraging the strengths of these models, this study benchmarks their performance against traditional anomaly detection methods such as Isolation Forests, One-Class SVMs, and COPOD. The results demonstrate the superior adaptability, scalability, and robustness of diffusion-based methods in handling complex real-world anomaly detection tasks. Key findings highlight the role of reconstruction error in enhancing detection accuracy and underscore the scalability of these models to high-dimensional datasets. Future directions include optimizing encoder-decoder architectures and exploring multi-modal datasets to further advance diffusion-based anomaly detection.

Title: Adaptive Epsilon Adversarial Training for Robust Gravitational Wave Parameter Estimation Using Normalizing Flows

• Authors: Yiqian Yang, Xihua Zhu, Fan Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07559
• Pdf URL: https://arxiv.org/pdf/2412.07559
• Copy Paste: [[2412.07559]] Adaptive Epsilon Adversarial Training for Robust Gravitational Wave Parameter Estimation Using Normalizing Flows(https://arxiv.org/abs/2412.07559)
• Keywords: attack, robust
• Abstract: Adversarial training with Normalizing Flow (NF) models is an emerging research area aimed at improving model robustness through adversarial samples. In this study, we focus on applying adversarial training to NF models for gravitational wave parameter estimation. We propose an adaptive epsilon method for Fast Gradient Sign Method (FGSM) adversarial training, which dynamically adjusts perturbation strengths based on gradient magnitudes using logarithmic scaling. Our hybrid architecture, combining ResNet and Inverse Autoregressive Flow, reduces the Negative Log Likelihood (NLL) loss by 47\% under FGSM attacks compared to the baseline model, while maintaining an NLL of 4.2 on clean data (only 5\% higher than the baseline). For perturbation strengths between 0.01 and 0.1, our model achieves an average NLL of 5.8, outperforming both fixed-epsilon (NLL: 6.7) and progressive-epsilon (NLL: 7.2) methods. Under stronger Projected Gradient Descent attacks with perturbation strength of 0.05, our model maintains an NLL of 6.4, demonstrating superior robustness while avoiding catastrophic overfitting.

Title: Defending Against Neural Network Model Inversion Attacks via Data Poisoning

• Authors: Shuai Zhou, Dayong Ye, Tianqing Zhu, Wanlei Zhou
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.07575
• Pdf URL: https://arxiv.org/pdf/2412.07575
• Copy Paste: [[2412.07575]] Defending Against Neural Network Model Inversion Attacks via Data Poisoning(https://arxiv.org/abs/2412.07575)
• Keywords: privacy, protect, defense, attack, robust
• Abstract: Model inversion attacks pose a significant privacy threat to machine learning models by reconstructing sensitive data from their outputs. While various defenses have been proposed to counteract these attacks, they often come at the cost of the classifier's utility, thus creating a challenging trade-off between privacy protection and model utility. Moreover, most existing defenses require retraining the classifier for enhanced robustness, which is impractical for large-scale, well-established models. This paper introduces a novel defense mechanism to better balance privacy and utility, particularly against adversaries who employ a machine learning model (i.e., inversion model) to reconstruct private data. Drawing inspiration from data poisoning attacks, which can compromise the performance of machine learning models, we propose a strategy that leverages data poisoning to contaminate the training data of inversion models, thereby preventing model inversion attacks. Two defense methods are presented. The first, termed label-preserving poisoning attacks for all output vectors (LPA), involves subtle perturbations to all output vectors while preserving their labels. Our findings demonstrate that these minor perturbations, introduced through a data poisoning approach, significantly increase the difficulty of data reconstruction without compromising the utility of the classifier. Subsequently, we introduce a second method, label-flipping poisoning for partial output vectors (LFP), which selectively perturbs a small subset of output vectors and alters their labels during the process. Empirical results indicate that LPA is notably effective, outperforming the current state-of-the-art defenses. Our data poisoning-based defense provides a new retraining-free defense paradigm that preserves the victim classifier's utility.

Title: Mobile Video Diffusion

• Authors: Haitam Ben Yahia, Denis Korzhenkov, Ioannis Lelekas, Amir Ghodrati, Amirhossein Habibian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07583
• Pdf URL: https://arxiv.org/pdf/2412.07583
• Copy Paste: [[2412.07583]] Mobile Video Diffusion(https://arxiv.org/abs/2412.07583)
• Keywords: diffusion
• Abstract: Video diffusion models have achieved impressive realism and controllability but are limited by high computational demands, restricting their use on mobile devices. This paper introduces the first mobile-optimized video diffusion model. Starting from a spatio-temporal UNet from Stable Video Diffusion (SVD), we reduce memory and computational cost by reducing the frame resolution, incorporating multi-scale temporal representations, and introducing two novel pruning schema to reduce the number of channels and temporal blocks. Furthermore, we employ adversarial finetuning to reduce the denoising to a single step. Our model, coined as MobileVD, is 523x more efficient (1817.2 vs. 4.34 TFLOPs) with a slight quality drop (FVD 149 vs. 171), generating latents for a 14x512x256 px clip in 1.7 seconds on a Xiaomi-14 Pro. Our results are available at this https URL

Title: Scaling Sequential Recommendation Models with Transformers

• Authors: Pablo Zivic, Hernan Vazquez, Jorge Sanchez
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07585
• Pdf URL: https://arxiv.org/pdf/2412.07585
• Copy Paste: [[2412.07585]] Scaling Sequential Recommendation Models with Transformers(https://arxiv.org/abs/2412.07585)
• Keywords: transformer, large language model
• Abstract: Modeling user preferences has been mainly addressed by looking at users' interaction history with the different elements available in the system. Tailoring content to individual preferences based on historical data is the main goal of sequential recommendation. The nature of the problem, as well as the good performance observed across various domains, has motivated the use of the transformer architecture, which has proven effective in leveraging increasingly larger amounts of training data when accompanied by an increase in the number of model parameters. This scaling behavior has brought a great deal of attention, as it provides valuable guidance in the design and training of even larger models. Taking inspiration from the scaling laws observed in training large language models, we explore similar principles for sequential recommendation. We use the full Amazon Product Data dataset, which has only been partially explored in other studies, and reveal scaling behaviors similar to those found in language models. Compute-optimal training is possible but requires a careful analysis of the compute-performance trade-offs specific to the application. We also show that performance scaling translates to downstream tasks by fine-tuning larger pre-trained models on smaller task-specific domains. Our approach and findings provide a strategic roadmap for model training and deployment in real high-dimensional preference spaces, facilitating better training and inference efficiency. We hope this paper bridges the gap between the potential of transformers and the intrinsic complexities of high-dimensional sequential recommendation in real-world recommender systems. Code and models can be found at this https URL

Title: Paired Wasserstein Autoencoders for Conditional Sampling

• Authors: Moritz Piening, Matthias Chung
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.07586
• Pdf URL: https://arxiv.org/pdf/2412.07586
• Copy Paste: [[2412.07586]] Paired Wasserstein Autoencoders for Conditional Sampling(https://arxiv.org/abs/2412.07586)
• Keywords: generative
• Abstract: Wasserstein distances greatly influenced and coined various types of generative neural network models. Wasserstein autoencoders are particularly notable for their mathematical simplicity and straight-forward implementation. However, their adaptation to the conditional case displays theoretical difficulties. As a remedy, we propose the use of two paired autoencoders. Under the assumption of an optimal autoencoder pair, we leverage the pairwise independence condition of our prescribed Gaussian latent distribution to overcome this theoretical hurdle. We conduct several experiments to showcase the practical applicability of the resulting paired Wasserstein autoencoders. Here, we consider imaging tasks and enable conditional sampling for denoising, inpainting, and unsupervised image translation. Moreover, we connect our image translation model to the Monge map behind Wasserstein-2 distances.

Title: DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation

• Authors: Jianzong Wu, Chao Tang, Jingbo Wang, Yanhong Zeng, Xiangtai Li, Yunhai Tong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07589
• Pdf URL: https://arxiv.org/pdf/2412.07589
• Copy Paste: [[2412.07589]] DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation(https://arxiv.org/abs/2412.07589)
• Keywords: diffusion, large language model
• Abstract: Story visualization, the task of creating visual narratives from textual descriptions, has seen progress with text-to-image generation models. However, these models often lack effective control over character appearances and interactions, particularly in multi-character scenes. To address these limitations, we propose a new task: \textbf{customized manga generation} and introduce \textbf{DiffSensei}, an innovative framework specifically designed for generating manga with dynamic multi-character control. DiffSensei integrates a diffusion-based image generator with a multimodal large language model (MLLM) that acts as a text-compatible identity adapter. Our approach employs masked cross-attention to seamlessly incorporate character features, enabling precise layout control without direct pixel transfer. Additionally, the MLLM-based adapter adjusts character features to align with panel-specific text cues, allowing flexible adjustments in character expressions, poses, and actions. We also introduce \textbf{MangaZero}, a large-scale dataset tailored to this task, containing 43,264 manga pages and 427,147 annotated panels, supporting the visualization of varied character interactions and movements across sequential frames. Extensive experiments demonstrate that DiffSensei outperforms existing models, marking a significant advancement in manga generation by enabling text-adaptable character customization. The project page is this https URL.

Title: ViewDelta: Text-Prompted Change Detection in Unaligned Images

• Authors: Subin Varghese, Joshua Gao, Vedhus Hoskere
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07612
• Pdf URL: https://arxiv.org/pdf/2412.07612
• Copy Paste: [[2412.07612]] ViewDelta: Text-Prompted Change Detection in Unaligned Images(https://arxiv.org/abs/2412.07612)
• Keywords: segmentation
• Abstract: Detecting changes between images is a fundamental problem in computer vision with broad applications in situational awareness, infrastructure assessment, environment monitoring, and industrial automation. Existing supervised models are typically limited to detecting specific types of changes, necessitating retraining for new tasks. To address these limitations with a single approach, we propose a novel change detection method that is the first to utilize unaligned images and textual prompts to output a binary segmentation of changes relevant to user-provided text. Our architecture not only enables flexible detection across diverse change detection use cases, but also yields state-of-the art performance on established benchmarks. Additionally, we release an accompanying dataset comprising of 100,311 pairs of images with text prompts and the corresponding change detection labels. We demonstrate the effectiveness of our method both quantitatively and qualitatively on datasets with a wide variety of viewpoints in indoor, outdoor, street level, synthetic, and satellite images.

Title: DRUM: Learning Demonstration Retriever for Large MUlti-modal Models

• Authors: Ellen Yi-Ge, Jiechao Gao, Wei Han, Wei Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07619
• Pdf URL: https://arxiv.org/pdf/2412.07619
• Copy Paste: [[2412.07619]] DRUM: Learning Demonstration Retriever for Large MUlti-modal Models(https://arxiv.org/abs/2412.07619)
• Keywords: large language model
• Abstract: Recently, large language models (LLMs) have demonstrated impressive capabilities in dealing with new tasks with the help of in-context learning (ICL). In the study of Large Vision-Language Models (LVLMs), when implementing ICL, researchers usually adopts the naive strategies like fixed demonstrations across different samples, or selecting demonstrations directly via a visual-language embedding model. These methods does not guarantee the configured demonstrations fit the need of the LVLMs. To address this issue, we now propose a novel framework, \underline{d}emonstration \underline{r}etriever for large m\underline{u}lti-modal \underline{m}odel (DRUM), which fine-tunes the visual-language embedding model to better meet the LVLM's needs. First, we discuss the retrieval strategies for a visual-language task, assuming an embedding model is given. And we propose to concate the image and text embeddings to enhance the retrieval performance. Second, we propose to re-rank the demonstrations retrieved by the embedding model via the LVLM's feedbacks, and calculate a list-wise ranking loss for training the embedding model. Third, we propose an iterative demonstration mining strategy to improve the training of the embedding model. Through extensive experiments on 3 types of visual-language tasks, 7 benchmark datasets, our DRUM framework is proven to be effective in boosting the LVLM's in-context learning performance via retrieving more proper demonstrations.

Title: OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations

• Authors: Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongying Tu, Conghui He
• Subjects: cs.CV, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.07626
• Pdf URL: https://arxiv.org/pdf/2412.07626
• Copy Paste: [[2412.07626]] OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations(https://arxiv.org/abs/2412.07626)
• Keywords: robust, extraction, fair, large language model
• Abstract: Document content extraction is crucial in computer vision, especially for meeting the high-quality data needs of large language models (LLMs) and retrieval-augmented generation (RAG) technologies. However, current document parsing methods suffer from significant limitations in terms of diversity and comprehensive evaluation. To address these challenges, we introduce OmniDocBench, a novel multi-source benchmark designed to advance automated document content extraction. OmniDocBench includes a meticulously curated and annotated high-quality evaluation dataset comprising nine diverse document types, such as academic papers, textbooks, slides, among others. Our benchmark provides a flexible and comprehensive evaluation framework with 19 layout category labels and 14 attribute labels, enabling multi-level assessments across entire datasets, individual modules, or specific data types. Using OmniDocBench, we perform an exhaustive comparative analysis of existing modular pipelines and multimodal end-to-end methods, highlighting their limitations in handling document diversity and ensuring fair evaluation. OmniDocBench establishes a robust, diverse, and fair evaluation standard for the document content extraction field, offering crucial insights for future advancements and fostering the development of document parsing technologies. The codes and dataset is available in this https URL.

Title: ChocoLlama: Lessons Learned From Teaching Llamas Dutch

• Authors: Matthieu Meeus, Anthony Rathé, François Remy, Pieter Delobelle, Jens-Joris Decorte, Thomas Demeester
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07633
• Pdf URL: https://arxiv.org/pdf/2412.07633
• Copy Paste: [[2412.07633]] ChocoLlama: Lessons Learned From Teaching Llamas Dutch(https://arxiv.org/abs/2412.07633)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) have shown remarkable capabilities in natural language understanding and generation, their performance often lags in lower-resource, non-English languages due to biases in the training data. In this work, we explore strategies for adapting the primarily English LLMs (Llama-2 and Llama-3) to Dutch, a language spoken by 30 million people worldwide yet often underrepresented in LLM development. We collect 104GB of Dutch text ($32$B tokens) from various sources to first apply continued pretraining using low-rank adaptation (LoRA), complemented with Dutch posttraining strategies provided by prior work. For Llama-2, we consider using (i) the tokenizer of the original model, and (ii) training a new, Dutch-specific tokenizer combined with embedding reinitialization. We evaluate our adapted models, ChocoLlama-2, both on standard benchmarks and a novel Dutch benchmark, ChocoLlama-Bench. Our results demonstrate that LoRA can effectively scale for language adaptation, and that tokenizer modification with careful weight reinitialization can improve performance. Notably, Llama-3 was released during the course of this project and, upon evaluation, demonstrated superior Dutch capabilities compared to our Dutch-adapted versions of Llama-2. We hence apply the same adaptation technique to Llama-3, using its original tokenizer. While our adaptation methods enhanced Llama-2's Dutch capabilities, we found limited gains when applying the same techniques to Llama-3. This suggests that for ever improving, multilingual foundation models, language adaptation techniques may benefit more from focusing on language-specific posttraining rather than on continued pretraining. We hope this work contributes to the broader understanding of adapting LLMs to lower-resource languages, and to the development of Dutch LLMs in particular.

Title: TrojanWhisper: Evaluating Pre-trained LLMs to Detect and Localize Hardware Trojans

• Authors: Md Omar Faruque, Peter Jamieson, Ahmad Patooghy, Abdel-Hameed A. Badawy
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07636
• Pdf URL: https://arxiv.org/pdf/2412.07636
• Copy Paste: [[2412.07636]] TrojanWhisper: Evaluating Pre-trained LLMs to Detect and Localize Hardware Trojans(https://arxiv.org/abs/2412.07636)
• Keywords: security, large language model
• Abstract: Existing Hardware Trojans (HT) detection methods face several critical limitations: logic testing struggles with scalability and coverage for large designs, side-channel analysis requires golden reference chips, and formal verification methods suffer from state-space explosion. The emergence of Large Language Models (LLMs) offers a promising new direction for HT detection by leveraging their natural language understanding and reasoning capabilities. For the first time, this paper explores the potential of general-purpose LLMs in detecting various HTs inserted in Register Transfer Level (RTL) designs, including SRAM, AES, and UART modules. We propose a novel tool for this goal that systematically assesses state-of-the-art LLMs (GPT-4o, Gemini 1.5 pro, and Llama 3.1) in detecting HTs without prior fine-tuning. To address potential training data bias, the tool implements perturbation techniques, i.e., variable name obfuscation, and design restructuring, that make the cases more sophisticated for the used LLMs. Our experimental evaluation demonstrates perfect detection rates by GPT-4o and Gemini 1.5 pro in baseline scenarios (100%/100% precision/recall), with both models achieving better trigger line coverage (TLC: 0.82-0.98) than payload line coverage (PLC: 0.32-0.46). Under code perturbation, while Gemini 1.5 pro maintains perfect detection performance (100%/100%), GPT-4o (100%/85.7%) and Llama 3.1 (66.7%/85.7%) show some degradation in detection rates, and all models experience decreased accuracy in localizing both triggers and payloads. This paper validates the potential of LLM approaches for hardware security applications, highlighting areas for future improvement.

Title: Searching for Structure: Investigating Emergent Communication with Large Language Models

• Authors: Tom Kouwenhoven, Max Peeperkorn, Tessa Verhoef
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07646
• Pdf URL: https://arxiv.org/pdf/2412.07646
• Copy Paste: [[2412.07646]] Searching for Structure: Investigating Emergent Communication with Large Language Models(https://arxiv.org/abs/2412.07646)
• Keywords: large language model
• Abstract: Human languages have evolved to be structured through repeated language learning and use. These processes introduce biases that operate during language acquisition and shape linguistic systems toward communicative efficiency. In this paper, we investigate whether the same happens if artificial languages are optimised for implicit biases of Large Language Models (LLMs). To this end, we simulate a classical referential game in which LLMs learn and use artificial languages. Our results show that initially unstructured holistic languages are indeed shaped to have some structural properties that allow two LLM agents to communicate successfully. Similar to observations in human experiments, generational transmission increases the learnability of languages, but can at the same time result in non-humanlike degenerate vocabularies. Taken together, this work extends experimental findings, shows that LLMs can be used as tools in simulations of language evolution, and opens possibilities for future human-machine experiments in this field.

Title: TraSCE: Trajectory Steering for Concept Erasure

• Authors: Anubhav Jain, Yuya Kobayashi, Takashi Shibuya, Yuhta Takida, Nasir Memon, Julian Togelius, Yuki Mitsufuji
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07658
• Pdf URL: https://arxiv.org/pdf/2412.07658
• Copy Paste: [[2412.07658]] TraSCE: Trajectory Steering for Concept Erasure(https://arxiv.org/abs/2412.07658)
• Keywords: diffusion
• Abstract: Recent advancements in text-to-image diffusion models have brought them to the public spotlight, becoming widely accessible and embraced by everyday users. However, these models have been shown to generate harmful content such as not-safe-for-work (NSFW) images. While approaches have been proposed to erase such abstract concepts from the models, jail-breaking techniques have succeeded in bypassing such safety measures. In this paper, we propose TraSCE, an approach to guide the diffusion trajectory away from generating harmful content. Our approach is based on negative prompting, but as we show in this paper, conventional negative prompting is not a complete solution and can easily be bypassed in some corner cases. To address this issue, we first propose a modification of conventional negative prompting. Furthermore, we introduce a localized loss-based guidance that enhances the modified negative prompting technique by steering the diffusion trajectory. We demonstrate that our proposed method achieves state-of-the-art results on various benchmarks in removing harmful content including ones proposed by red teams; and erasing artistic styles and objects. Our proposed approach does not require any training, weight modifications, or training data (both image or prompt), making it easier for model owners to erase new concepts.

Title: Proc-GS: Procedural Building Generation for City Assembly with 3D Gaussians

• Authors: Yixuan Li, Xingjian Ran, Linning Xu, Tao Lu, Mulin Yu, Zhenzhi Wang, Yuanbo Xiangli, Dahua Lin, Bo Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07660
• Pdf URL: https://arxiv.org/pdf/2412.07660
• Copy Paste: [[2412.07660]] Proc-GS: Procedural Building Generation for City Assembly with 3D Gaussians(https://arxiv.org/abs/2412.07660)
• Keywords: generative
• Abstract: Buildings are primary components of cities, often featuring repeated elements such as windows and doors. Traditional 3D building asset creation is labor-intensive and requires specialized skills to develop design rules. Recent generative models for building creation often overlook these patterns, leading to low visual fidelity and limited scalability. Drawing inspiration from procedural modeling techniques used in the gaming and visual effects industry, our method, Proc-GS, integrates procedural code into the 3D Gaussian Splatting (3D-GS) framework, leveraging their advantages in high-fidelity rendering and efficient asset management from both worlds. By manipulating procedural code, we can streamline this process and generate an infinite variety of buildings. This integration significantly reduces model size by utilizing shared foundational assets, enabling scalable generation with precise control over building assembly. We showcase the potential for expansive cityscape generation while maintaining high rendering fidelity and precise control on both real and synthetic cases.

Title: Multimodal Instruction Disassembly with Covariate Shift Adaptation and Real-time Implementation

• Authors: Yunkai Bai, Jungmin Park, Domenic Forte
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.07671
• Pdf URL: https://arxiv.org/pdf/2412.07671
• Copy Paste: [[2412.07671]] Multimodal Instruction Disassembly with Covariate Shift Adaptation and Real-time Implementation(https://arxiv.org/abs/2412.07671)
• Keywords: security
• Abstract: Side-channel based instruction disassembly has been proposed as a low-cost and non-invasive approach for security applications such as IP infringement detection, code flow analysis, malware detection, and reconstructing unknown code from obsolete systems. However, existing approaches to side-channel based disassembly rely on setups to collect and process side-channel traces that make them impractical for real-time applications. In addition, they rely on fixed classifiers that cannot adapt to statistical deviations in side-channels caused by different operating environments. In this article, we advance the state of the art in side-channel based disassembly in multiple ways. First, we introduce a new miniature platform, RASCv3, that can simultaneously collect power and EM measurements from a target device and subsequently process them for instruction disassembly in real time. Second, we devise a new approach to combine and select features from power and EM traces using information theory that improves classification accuracy and avoids the curse of dimensionality. Third, we explore covariate shift adjustment techniques that further improve accuracy over time and in response to statistical changes. The proposed methodology is demonstrated on six benchmarks, and the recognition rates of offline and real-time instruction disassemblers are compared for single- and multi-modal cases with a variety of classifiers and over time. Since the proposed approach is only applied to an 8-bit Arduino UNO, we also discuss challenges of extending to more complex targets.

Title: FlexLLM: Exploring LLM Customization for Moving Target Defense on Black-Box LLMs Against Jailbreak Attacks

• Authors: Bocheng Chen, Hanqing Guo, Qiben Yan
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07672
• Pdf URL: https://arxiv.org/pdf/2412.07672
• Copy Paste: [[2412.07672]] FlexLLM: Exploring LLM Customization for Moving Target Defense on Black-Box LLMs Against Jailbreak Attacks(https://arxiv.org/abs/2412.07672)
• Keywords: protect, defense, attack, robust, large language model
• Abstract: Defense in large language models (LLMs) is crucial to counter the numerous attackers exploiting these systems to generate harmful content through manipulated prompts, known as jailbreak attacks. Although many defense strategies have been proposed, they often require access to the model's internal structure or need additional training, which is impractical for service providers using LLM APIs, such as OpenAI APIs or Claude APIs. In this paper, we propose a moving target defense approach that alters decoding hyperparameters to enhance model robustness against various jailbreak attacks. Our approach does not require access to the model's internal structure and incurs no additional training costs. The proposed defense includes two key components: (1) optimizing the decoding strategy by identifying and adjusting decoding hyperparameters that influence token generation probabilities, and (2) transforming the decoding hyperparameters and model system prompts into dynamic targets, which are continuously altered during each runtime. By continuously modifying decoding strategies and prompts, the defense effectively mitigates the existing attacks. Our results demonstrate that our defense is the most effective against jailbreak attacks in three of the models tested when using LLMs as black-box APIs. Moreover, our defense offers lower inference costs and maintains comparable response quality, making it a potential layer of protection when used alongside other defense methods.

Title: FiVA: Fine-grained Visual Attribute Dataset for Text-to-Image Diffusion Models

• Authors: Tong Wu, Yinghao Xu, Ryan Po, Mengchen Zhang, Guandao Yang, Jiaqi Wang, Ziwei Liu, Dahua Lin, Gordon Wetzstein
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07674
• Pdf URL: https://arxiv.org/pdf/2412.07674
• Copy Paste: [[2412.07674]] FiVA: Fine-grained Visual Attribute Dataset for Text-to-Image Diffusion Models(https://arxiv.org/abs/2412.07674)
• Keywords: diffusion
• Abstract: Recent advances in text-to-image generation have enabled the creation of high-quality images with diverse applications. However, accurately describing desired visual attributes can be challenging, especially for non-experts in art and photography. An intuitive solution involves adopting favorable attributes from the source images. Current methods attempt to distill identity and style from source images. However, "style" is a broad concept that includes texture, color, and artistic elements, but does not cover other important attributes such as lighting and dynamics. Additionally, a simplified "style" adaptation prevents combining multiple attributes from different sources into one generated image. In this work, we formulate a more effective approach to decompose the aesthetics of a picture into specific visual attributes, allowing users to apply characteristics such as lighting, texture, and dynamics from different images. To achieve this goal, we constructed the first fine-grained visual attributes dataset (FiVA) to the best of our knowledge. This FiVA dataset features a well-organized taxonomy for visual attributes and includes around 1 M high-quality generated images with visual attribute annotations. Leveraging this dataset, we propose a fine-grained visual attribute adaptation framework (FiVA-Adapter), which decouples and adapts visual attributes from one or more source images into a generated one. This approach enhances user-friendly customization, allowing users to selectively apply desired attributes to create images that meet their unique preferences and specific content requirements.

Title: RAZOR: Sharpening Knowledge by Cutting Bias with Unsupervised Text Rewriting

• Authors: Shuo Yang, Bardh Prenkaj, Gjergji Kasneci
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07675
• Pdf URL: https://arxiv.org/pdf/2412.07675
• Copy Paste: [[2412.07675]] RAZOR: Sharpening Knowledge by Cutting Bias with Unsupervised Text Rewriting(https://arxiv.org/abs/2412.07675)
• Keywords: robust, fair
• Abstract: Despite the widespread use of LLMs due to their superior performance in various tasks, their high computational costs often lead potential users to opt for the pretraining-finetuning pipeline. However, biases prevalent in manually constructed datasets can introduce spurious correlations between tokens and labels, creating so-called shortcuts and hindering the generalizability of fine-tuned models. Existing debiasing methods often rely on prior knowledge of specific dataset biases, which is challenging to acquire a priori. We propose RAZOR (Rewriting And Zero-bias Optimization Refinement), a novel, unsupervised, and data-focused debiasing approach based on text rewriting for shortcut mitigation. RAZOR leverages LLMs to iteratively rewrite potentially biased text segments by replacing them with heuristically selected alternatives in a shortcut space defined by token statistics and positional information. This process aims to align surface-level text features more closely with diverse label distributions, thereby promoting the learning of genuine linguistic patterns. Compared with unsupervised SoTA models, RAZOR improves by 3.5% on the FEVER and 6.5% on MNLI and SNLI datasets according to the F1 score. Additionally, RAZOR effectively mitigates specific known biases, reducing bias-related terms by x2 without requiring prior bias information, a result that is on par with SoTA models that leverage prior information. Our work prioritizes data manipulation over architectural modifications, emphasizing the pivotal role of data quality in enhancing model performance and fairness. This research contributes to developing more robust evaluation benchmarks for debiasing methods by incorporating metrics for bias reduction and overall model efficacy.

Title: RADIO Amplified: Improved Baselines for Agglomerative Vision Foundation Models

• Authors: Greg Heinrich, Mike Ranzinger, Hongxu (Danny)Yin, Yao Lu, Jan Kautz, Andrew Tao, Bryan Catanzaro, Pavlo Molchanov
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07679
• Pdf URL: https://arxiv.org/pdf/2412.07679
• Copy Paste: [[2412.07679]] RADIO Amplified: Improved Baselines for Agglomerative Vision Foundation Models(https://arxiv.org/abs/2412.07679)
• Keywords: robust
• Abstract: Agglomerative models have recently emerged as a powerful approach to training vision foundation models, leveraging multi-teacher distillation from existing models such as CLIP, DINO, and SAM. This strategy enables the efficient creation of robust models, combining the strengths of individual teachers while significantly reducing computational and resource demands. In this paper, we thoroughly analyze state-of-the-art agglomerative models, identifying critical challenges including resolution mode shifts, teacher imbalance, idiosyncratic teacher artifacts, and an excessive number of output tokens. To address these issues, we propose several novel solutions: multi-resolution training, mosaic augmentation, and improved balancing of teacher loss functions. Specifically, in the context of Vision Language Models, we introduce a token compression technique to maintain high-resolution information within a fixed token count. We release our top-performing models, available in multiple scales (-B, -L, -H, and -g), alongside inference code and pretrained weights.

Title: TRIM: Token Reduction and Inference Modeling for Cost-Effective Language Generation

• Authors: Alfredo Garrachón Ruiz, Tomás de la Rosa, Daniel Borrajo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07682
• Pdf URL: https://arxiv.org/pdf/2412.07682
• Copy Paste: [[2412.07682]] TRIM: Token Reduction and Inference Modeling for Cost-Effective Language Generation(https://arxiv.org/abs/2412.07682)
• Keywords: large language model
• Abstract: The inference cost of Large Language Models (LLMs) is a significant challenge due to their computational demands, specially on tasks requiring long outputs. However, natural language often contains redundancy, which presents an opportunity for optimization. We have observed that LLMs can generate distilled language-concise outputs that retain essential meaning, when prompted appropriately. We propose a framework for saving computational cost, in which a shorter distilled output from the LLM is reconstructed into a full narrative by a smaller model with lower inference costs. Our experiments show promising results, particularly in general knowledge domains with 20.58% saved tokens on average with tiny decrease in evaluation metrics, hinting that this approach can effectively balance efficiency and accuracy in language processing tasks.

Title: The Pitfalls of Memorization: When Memorization Hurts Generalization

• Authors: Reza Bayat, Mohammad Pezeshki, Elvis Dohmatob, David Lopez-Paz, Pascal Vincent
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.07684
• Pdf URL: https://arxiv.org/pdf/2412.07684
• Copy Paste: [[2412.07684]] The Pitfalls of Memorization: When Memorization Hurts Generalization(https://arxiv.org/abs/2412.07684)
• Keywords: robust
• Abstract: Neural networks often learn simple explanations that fit the majority of the data while memorizing exceptions that deviate from these this http URL behavior leads to poor generalization when the learned explanations rely on spurious correlations. In this work, we formalize the interplay between memorization and generalization, showing that spurious correlations would particularly lead to poor generalization when are combined with memorization. Memorization can reduce training loss to zero, leaving no incentive to learn robust, generalizable patterns. To address this, we propose memorization-aware training (MAT), which uses held-out predictions as a signal of memorization to shift a model's logits. MAT encourages learning robust patterns invariant across distributions, improving generalization under distribution shifts.

Title: Privacy-Preserving Customer Support: A Framework for Secure and Scalable Interactions

• Authors: Anant Prakash Awasthi, Chandraketu Singh, Rakshit Varma, Sanchit Sharma
• Subjects: cs.LG, cs.CR, stat.AP
• Abstract URL: https://arxiv.org/abs/2412.07687
• Pdf URL: https://arxiv.org/pdf/2412.07687
• Copy Paste: [[2412.07687]] Privacy-Preserving Customer Support: A Framework for Secure and Scalable Interactions(https://arxiv.org/abs/2412.07687)
• Keywords: secure, privacy, protect, robust, federate, large language model
• Abstract: The growing reliance on artificial intelligence (AI) in customer support has significantly improved operational efficiency and user experience. However, traditional machine learning (ML) approaches, which require extensive local training on sensitive datasets, pose substantial privacy risks and compliance challenges with regulations like the General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA). Existing privacy-preserving techniques, such as anonymization, differential privacy, and federated learning, address some concerns but face limitations in utility, scalability, and complexity. This paper introduces the Privacy-Preserving Zero-Shot Learning (PP-ZSL) framework, a novel approach leveraging large language models (LLMs) in a zero-shot learning mode. Unlike conventional ML methods, PP-ZSL eliminates the need for local training on sensitive data by utilizing pre-trained LLMs to generate responses directly. The framework incorporates real-time data anonymization to redact or mask sensitive information, retrieval-augmented generation (RAG) for domain-specific query resolution, and robust post-processing to ensure compliance with regulatory standards. This combination reduces privacy risks, simplifies compliance, and enhances scalability and operational efficiency. Empirical analysis demonstrates that the PP-ZSL framework provides accurate, privacy-compliant responses while significantly lowering the costs and complexities of deploying AI-driven customer support systems. The study highlights potential applications across industries, including financial services, healthcare, e-commerce, legal support, telecommunications, and government services. By addressing the dual challenges of privacy and performance, this framework establishes a foundation for secure, efficient, and regulatory-compliant AI applications in customer interactions.

Title: DriveMM: All-in-One Large Multimodal Model for Autonomous Driving

• Authors: Zhijian Huang, Chengjian Feng, Feng Yan, Baihui Xiao, Zequn Jie, Yujie Zhong, Xiaodan Liang, Lin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07689
• Pdf URL: https://arxiv.org/pdf/2412.07689
• Copy Paste: [[2412.07689]] DriveMM: All-in-One Large Multimodal Model for Autonomous Driving(https://arxiv.org/abs/2412.07689)
• Keywords: large language model
• Abstract: Large Multimodal Models (LMMs) have demonstrated exceptional comprehension and interpretation capabilities in Autonomous Driving (AD) by incorporating large language models. Despite the advancements, current data-driven AD approaches tend to concentrate on a single dataset and specific tasks, neglecting their overall capabilities and ability to generalize. To bridge these gaps, we propose DriveMM, a general large multimodal model designed to process diverse data inputs, such as images and multi-view videos, while performing a broad spectrum of AD tasks, including perception, prediction, and planning. Initially, the model undergoes curriculum pre-training to process varied visual signals and perform basic visual comprehension and perception tasks. Subsequently, we augment and standardize various AD-related datasets to fine-tune the model, resulting in an all-in-one LMM for autonomous driving. To assess the general capabilities and generalization ability, we conduct evaluations on six public benchmarks and undertake zero-shot transfer on an unseen dataset, where DriveMM achieves state-of-the-art performance across all tasks. We hope DriveMM as a promising solution for future end-toend autonomous driving applications in the real world.

Title: SimVS: Simulating World Inconsistencies for Robust View Synthesis

• Authors: Alex Trevithick, Roni Paiss, Philipp Henzler, Dor Verbin, Rundi Wu, Hadi Alzayer, Ruiqi Gao, Ben Poole, Jonathan T. Barron, Aleksander Holynski, Ravi Ramamoorthi, Pratul P. Srinivasan
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07696
• Pdf URL: https://arxiv.org/pdf/2412.07696
• Copy Paste: [[2412.07696]] SimVS: Simulating World Inconsistencies for Robust View Synthesis(https://arxiv.org/abs/2412.07696)
• Keywords: robust, generative
• Abstract: Novel-view synthesis techniques achieve impressive results for static scenes but struggle when faced with the inconsistencies inherent to casual capture settings: varying illumination, scene motion, and other unintended effects that are difficult to model explicitly. We present an approach for leveraging generative video models to simulate the inconsistencies in the world that can occur during capture. We use this process, along with existing multi-view datasets, to create synthetic data for training a multi-view harmonization network that is able to reconcile inconsistent observations into a consistent 3D scene. We demonstrate that our world-simulation strategy significantly outperforms traditional augmentation methods in handling real-world scene variations, thereby enabling highly accurate static 3D reconstructions in the presence of a variety of challenging inconsistencies. Project page: this https URL

Title: ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer

• Authors: Jinyi Hu, Shengding Hu, Yuxuan Song, Yufei Huang, Mingxuan Wang, Hao Zhou, Zhiyuan Liu, Wei-Ying Ma, Maosong Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07720
• Pdf URL: https://arxiv.org/pdf/2412.07720
• Copy Paste: [[2412.07720]] ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer(https://arxiv.org/abs/2412.07720)
• Keywords: diffusion, transformer
• Abstract: The recent surge of interest in comprehensive multimodal models has necessitated the unification of diverse modalities. However, the unification suffers from disparate methodologies. Continuous visual generation necessitates the full-sequence diffusion-based approach, despite its divergence from the autoregressive modeling in the text domain. We posit that autoregressive modeling, i.e., predicting the future based on past deterministic experience, remains crucial in developing both a visual generation model and a potential unified multimodal model. In this paper, we explore an interpolation between the autoregressive modeling and full-parameters diffusion to model visual information. At its core, we present ACDiT, an Autoregressive blockwise Conditional Diffusion Transformer, where the block size of diffusion, i.e., the size of autoregressive units, can be flexibly adjusted to interpolate between token-wise autoregression and full-sequence diffusion. ACDiT is easy to implement, as simple as creating a Skip-Causal Attention Mask (SCAM) during training. During inference, the process iterates between diffusion denoising and autoregressive decoding that can make full use of KV-Cache. We verify the effectiveness of ACDiT on image and video generation tasks. We also demonstrate that benefitted from autoregressive modeling, ACDiT can be seamlessly used in visual understanding tasks despite being trained on the diffusion objective. The analysis of the trade-off between autoregressive modeling and diffusion demonstrates the potential of ACDiT to be used in long-horizon visual generation tasks. These strengths make it promising as the backbone of future unified models.

Title: Granite Guardian

• Authors: Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchia, Subhajit Chaudhury, Tejaswini Pedapati, Pierre Dognin, Keerthiram Murugesan, Erik Miehling, Martín Santillán Cooper, Kieran Fraser, Giulio Zizzo, Muhammad Zaid Hameed, Mark Purcell, Michael Desmond, Qian Pan, Inge Vejsbjerg, Elizabeth M. Daly, Michael Hind, Werner Geyer, Ambrish Rawat, Kush R. Varshney, Prasanna Sattigeri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07724
• Pdf URL: https://arxiv.org/pdf/2412.07724
• Copy Paste: [[2412.07724]] Granite Guardian(https://arxiv.org/abs/2412.07724)
• Keywords: large language model
• Abstract: We introduce the Granite Guardian models, a suite of safeguards designed to provide risk detection for prompts and responses, enabling safe and responsible use in combination with any large language model (LLM). These models offer comprehensive coverage across multiple risk dimensions, including social bias, profanity, violence, sexual content, unethical behavior, jailbreaking, and hallucination-related risks such as context relevance, groundedness, and answer relevance for retrieval-augmented generation (RAG). Trained on a unique dataset combining human annotations from diverse sources and synthetic data, Granite Guardian models address risks typically overlooked by traditional risk detection models, such as jailbreaks and RAG-specific issues. With AUC scores of 0.871 and 0.854 on harmful content and RAG-hallucination-related benchmarks respectively, Granite Guardian is the most generalizable and competitive model available in the space. Released as open-source, Granite Guardian aims to promote responsible AI development across the community. this https URL

Title: STIV: Scalable Text and Image Conditioned Video Generation

• Authors: Zongyu Lin, Wei Liu, Chen Chen, Jiasen Lu, Wenze Hu, Tsu-Jui Fu, Jesse Allardice, Zhengfeng Lai, Liangchen Song, Bowen Zhang, Cha Chen, Yiran Fei, Yifan Jiang, Lezhi Li, Yizhou Sun, Kai-Wei Chang, Yinfei Yang
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.07730
• Pdf URL: https://arxiv.org/pdf/2412.07730
• Copy Paste: [[2412.07730]] STIV: Scalable Text and Image Conditioned Video Generation(https://arxiv.org/abs/2412.07730)
• Keywords: robust, diffusion, transformer
• Abstract: The field of video generation has made remarkable advancements, yet there remains a pressing need for a clear, systematic recipe that can guide the development of robust and scalable models. In this work, we present a comprehensive study that systematically explores the interplay of model architectures, training recipes, and data curation strategies, culminating in a simple and scalable text-image-conditioned video generation method, named STIV. Our framework integrates image condition into a Diffusion Transformer (DiT) through frame replacement, while incorporating text conditioning via a joint image-text conditional classifier-free guidance. This design enables STIV to perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks simultaneously. Additionally, STIV can be easily extended to various applications, such as video prediction, frame interpolation, multi-view generation, and long video generation, etc. With comprehensive ablation studies on T2I, T2V, and TI2V, STIV demonstrate strong performance, despite its simple design. An 8.7B model with 512 resolution achieves 83.1 on VBench T2V, surpassing both leading open and closed-source models like CogVideoX-5B, Pika, Kling, and Gen-3. The same-sized model also achieves a state-of-the-art result of 90.1 on VBench I2V task at 512 resolution. By providing a transparent and extensible recipe for building cutting-edge video generation models, we aim to empower future research and accelerate progress toward more versatile and reliable video generation solutions.

Title: Zero-Shot ATC Coding with Large Language Models for Clinical Assessments

• Authors: Zijian Chen, John-Michael Gamble, Micaela Jantzi, John P. Hirdes, Jimmy Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07743
• Pdf URL: https://arxiv.org/pdf/2412.07743
• Copy Paste: [[2412.07743]] Zero-Shot ATC Coding with Large Language Models for Clinical Assessments(https://arxiv.org/abs/2412.07743)
• Keywords: privacy, extraction, large language model
• Abstract: Manual assignment of Anatomical Therapeutic Chemical (ATC) codes to prescription records is a significant bottleneck in healthcare research and operations at Ontario Health and InterRAI Canada, requiring extensive expert time and effort. To automate this process while maintaining data privacy, we develop a practical approach using locally deployable large language models (LLMs). Inspired by recent advances in automatic International Classification of Diseases (ICD) coding, our method frames ATC coding as a hierarchical information extraction task, guiding LLMs through the ATC ontology level by level. We evaluate our approach using GPT-4o as an accuracy ceiling and focus development on open-source Llama models suitable for privacy-sensitive deployment. Testing across Health Canada drug product data, the RABBITS benchmark, and real clinical notes from Ontario Health, our method achieves 78% exact match accuracy with GPT-4o and 60% with Llama 3.1 70B. We investigate knowledge grounding through drug definitions, finding modest improvements in accuracy. Further, we show that fine-tuned Llama 3.1 8B matches zero-shot Llama 3.1 70B accuracy, suggesting that effective ATC coding is feasible with smaller models. Our results demonstrate the feasibility of automatic ATC coding in privacy-sensitive healthcare environments, providing a foundation for future deployments.

Title: StyleMaster: Stylize Your Video with Artistic Generation and Translation

• Authors: Zixuan Ye, Huijuan Huang, Xintao Wang, Pengfei Wan, Di Zhang, Wenhan Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07744
• Pdf URL: https://arxiv.org/pdf/2412.07744
• Copy Paste: [[2412.07744]] StyleMaster: Stylize Your Video with Artistic Generation and Translation(https://arxiv.org/abs/2412.07744)
• Keywords: extraction
• Abstract: Style control has been popular in video generation models. Existing methods often generate videos far from the given style, cause content leakage, and struggle to transfer one video to the desired style. Our first observation is that the style extraction stage matters, whereas existing methods emphasize global style but ignore local textures. In order to bring texture features while preventing content leakage, we filter content-related patches while retaining style ones based on prompt-patch similarity; for global style extraction, we generate a paired style dataset through model illusion to facilitate contrastive learning, which greatly enhances the absolute style consistency. Moreover, to fill in the image-to-video gap, we train a lightweight motion adapter on still videos, which implicitly enhances stylization extent, and enables our image-trained model to be seamlessly applied to videos. Benefited from these efforts, our approach, StyleMaster, not only achieves significant improvement in both style resemblance and temporal coherence, but also can easily generalize to video style transfer with a gray tile ControlNet. Extensive experiments and visualizations demonstrate that StyleMaster significantly outperforms competitors, effectively generating high-quality stylized videos that align with textual content and closely resemble the style of reference images. Our project page is at this https URL

Title: LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation Models

• Authors: Ziqi Lu, Heng Yang, Danfei Xu, Boyi Li, Boris Ivanovic, Marco Pavone, Yue Wang
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.07746
• Pdf URL: https://arxiv.org/pdf/2412.07746
• Copy Paste: [[2412.07746]] LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation Models(https://arxiv.org/abs/2412.07746)
• Keywords: robust
• Abstract: Emerging 3D geometric foundation models, such as DUSt3R, offer a promising approach for in-the-wild 3D vision tasks. However, due to the high-dimensional nature of the problem space and scarcity of high-quality 3D data, these pre-trained models still struggle to generalize to many challenging circumstances, such as limited view overlap or low lighting. To address this, we propose LoRA3D, an efficient self-calibration pipeline to $\textit{specialize}$ the pre-trained models to target scenes using their own multi-view predictions. Taking sparse RGB images as input, we leverage robust optimization techniques to refine multi-view predictions and align them into a global coordinate frame. In particular, we incorporate prediction confidence into the geometric optimization process, automatically re-weighting the confidence to better reflect point estimation accuracy. We use the calibrated confidence to generate high-quality pseudo labels for the calibrating views and use low-rank adaptation (LoRA) to fine-tune the models on the pseudo-labeled data. Our method does not require any external priors or manual labels. It completes the self-calibration process on a $\textbf{single standard GPU within just 5 minutes}$. Each low-rank adapter requires only $\textbf{18MB}$ of storage. We evaluated our method on $\textbf{more than 160 scenes}$ from the Replica, TUM and Waymo Open datasets, achieving up to $\textbf{88% performance improvement}$ on 3D reconstruction, multi-view pose estimation and novel-view rendering.

Title: Multi-Shot Character Consistency for Text-to-Video Generation

• Authors: Yuval Atzmon, Rinon Gal, Yoad Tewel, Yoni Kasten, Gal Chechik
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07750
• Pdf URL: https://arxiv.org/pdf/2412.07750
• Copy Paste: [[2412.07750]] Multi-Shot Character Consistency for Text-to-Video Generation(https://arxiv.org/abs/2412.07750)
• Keywords: diffusion
• Abstract: Text-to-video models have made significant strides in generating short video clips from textual descriptions. Yet, a significant challenge remains: generating several video shots of the same characters, preserving their identity without hurting video quality, dynamics, and responsiveness to text prompts. We present Video Storyboarding, a training-free method to enable pretrained text-to-video models to generate multiple shots with consistent characters, by sharing features between them. Our key insight is that self-attention query features (Q) encode both motion and identity. This creates a hard-to-avoid trade-off between preserving character identity and making videos dynamic, when features are shared. To address this issue, we introduce a novel query injection strategy that balances identity preservation and natural motion retention. This approach improves upon naive consistency techniques applied to videos, which often struggle to maintain this delicate equilibrium. Our experiments demonstrate significant improvements in character consistency across scenes while maintaining high-quality motion and text alignment. These results offer insights into critical stages of video generation and the interplay of structure and motion in video diffusion models.

Title: FlashRNN: Optimizing Traditional RNNs on Modern Hardware

• Authors: Korbinian Pöppel, Maximilian Beck, Sepp Hochreiter
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07752
• Pdf URL: https://arxiv.org/pdf/2412.07752
• Copy Paste: [[2412.07752]] FlashRNN: Optimizing Traditional RNNs on Modern Hardware(https://arxiv.org/abs/2412.07752)
• Keywords: transformer
• Abstract: While Transformers and other sequence-parallelizable neural network architectures seem like the current state of the art in sequence modeling, they specifically lack state-tracking capabilities. These are important for time-series tasks and logical reasoning. Traditional RNNs like LSTMs and GRUs, as well as modern variants like sLSTM do have these capabilities at the cost of strictly sequential processing. While this is often seen as a strong limitation, we show how fast these networks can get with our hardware-optimization FlashRNN in Triton and CUDA, optimizing kernels to the register level on modern GPUs. We extend traditional RNNs with a parallelization variant that processes multiple RNNs of smaller hidden state in parallel, similar to the head-wise processing in Transformers. To enable flexibility on different GPU variants, we introduce a new optimization framework for hardware-internal cache sizes, memory and compute handling. It models the hardware in a setting using polyhedral-like constraints, including the notion of divisibility. This speeds up the solution process in our ConstrINT library for general integer constraint satisfaction problems (integer CSPs). We show that our kernels can achieve 50x speed-ups over a vanilla PyTorch implementation and allow 40x larger hidden sizes compared to our Triton implementation. Our open-source kernels and the optimization library are released here to boost research in the direction of state-tracking enabled RNNs and sequence modeling: \url{this https URL}

Title: PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation

• Authors: Fatemeh Nazarieh, Zhenhua Feng, Diptesh Kanojia, Muhammad Awais, Josef Kittler
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07754
• Pdf URL: https://arxiv.org/pdf/2412.07754
• Copy Paste: [[2412.07754]] PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation(https://arxiv.org/abs/2412.07754)
• Keywords: diffusion
• Abstract: Audio-driven talking face generation is a challenging task in digital communication. Despite significant progress in the area, most existing methods concentrate on audio-lip synchronization, often overlooking aspects such as visual quality, customization, and generalization that are crucial to producing realistic talking faces. To address these limitations, we introduce a novel, customizable one-shot audio-driven talking face generation framework, named PortraitTalk. Our proposed method utilizes a latent diffusion framework consisting of two main components: IdentityNet and AnimateNet. IdentityNet is designed to preserve identity features consistently across the generated video frames, while AnimateNet aims to enhance temporal coherence and motion consistency. This framework also integrates an audio input with the reference images, thereby reducing the reliance on reference-style videos prevalent in existing approaches. A key innovation of PortraitTalk is the incorporation of text prompts through decoupled cross-attention mechanisms, which significantly expands creative control over the generated videos. Through extensive experiments, including a newly developed evaluation metric, our model demonstrates superior performance over the state-of-the-art methods, setting a new standard for the generation of customizable realistic talking faces suitable for real-world applications.

Title: 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation

• Authors: Xiao Fu, Xian Liu, Xintao Wang, Sida Peng, Menghan Xia, Xiaoyu Shi, Ziyang Yuan, Pengfei Wan, Di Zhang, Dahua Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07759
• Pdf URL: https://arxiv.org/pdf/2412.07759
• Copy Paste: [[2412.07759]] 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation(https://arxiv.org/abs/2412.07759)
• Keywords: robust, diffusion
• Abstract: This paper aims to manipulate multi-entity 3D motions in video generation. Previous methods on controllable video generation primarily leverage 2D control signals to manipulate object motions and have achieved remarkable synthesis results. However, 2D control signals are inherently limited in expressing the 3D nature of object motions. To overcome this problem, we introduce 3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D space, given user-desired 6DoF pose (location and rotation) sequences of entities. At the core of our approach is a plug-and-play 3D-motion grounded object injector that fuses multiple input entities with their respective 3D trajectories through a gated self-attention mechanism. In addition, we exploit an injector architecture to preserve the video diffusion prior, which is crucial for generalization ability. To mitigate video quality degradation, we introduce a domain adaptor during training and employ an annealed sampling strategy during inference. To address the lack of suitable training data, we construct a 360-Motion Dataset, which first correlates collected 3D human and animal assets with GPT-generated trajectory and then captures their motion with 12 evenly-surround cameras on diverse 3D UE platforms. Extensive experiments show that 3DTrajMaster sets a new state-of-the-art in both accuracy and generalization for controlling multi-entity 3D motions. Project page: this http URL

Title: SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints

• Authors: Jianhong Bai, Menghan Xia, Xintao Wang, Ziyang Yuan, Xiao Fu, Zuozhu Liu, Haoji Hu, Pengfei Wan, Di Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07760
• Pdf URL: https://arxiv.org/pdf/2412.07760
• Copy Paste: [[2412.07760]] SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints(https://arxiv.org/abs/2412.07760)
• Keywords: diffusion
• Abstract: Recent advancements in video diffusion models have shown exceptional abilities in simulating real-world dynamics and maintaining 3D consistency. This progress inspires us to investigate the potential of these models to ensure dynamic consistency across various viewpoints, a highly desirable feature for applications such as virtual filming. Unlike existing methods focused on multi-view generation of single objects for 4D reconstruction, our interest lies in generating open-world videos from arbitrary viewpoints, incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation, ensuring consistent content across different viewpoints. Specifically, we introduce a multi-view synchronization module to maintain appearance and geometry consistency across these viewpoints. Given the scarcity of high-quality training data, we design a hybrid training scheme that leverages multi-camera images and monocular videos to supplement Unreal Engine-rendered multi-camera videos. Furthermore, our method enables intriguing extensions, such as re-rendering a video from novel viewpoints. We also release a multi-view synchronized video dataset, named SynCamVideo-Dataset. Project page: this https URL.

Title: Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation

• Authors: Jingxi Chen, Brandon Y. Feng, Haoming Cai, Tianfu Wang, Levi Burner, Dehao Yuan, Cornelia Fermuller, Christopher A. Metzler, Yiannis Aloimonos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07761
• Pdf URL: https://arxiv.org/pdf/2412.07761
• Copy Paste: [[2412.07761]] Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation(https://arxiv.org/abs/2412.07761)
• Keywords: diffusion
• Abstract: Video Frame Interpolation aims to recover realistic missing frames between observed frames, generating a high-frame-rate video from a low-frame-rate video. However, without additional guidance, the large motion between frames makes this problem ill-posed. Event-based Video Frame Interpolation (EVFI) addresses this challenge by using sparse, high-temporal-resolution event measurements as motion guidance. This guidance allows EVFI methods to significantly outperform frame-only methods. However, to date, EVFI methods have relied on a limited set of paired event-frame training data, severely limiting their performance and generalization capabilities. In this work, we overcome the limited data challenge by adapting pre-trained video diffusion models trained on internet-scale datasets to EVFI. We experimentally validate our approach on real-world EVFI datasets, including a new one that we introduce. Our method outperforms existing methods and generalizes across cameras far better than existing approaches.

Title: Make-A-Texture: Fast Shape-Aware Texture Generation in 3 Seconds

• Authors: Xiaoyu Xiang, Liat Sless Gorelik, Yuchen Fan, Omri Armstrong, Forrest Iandola, Yilei Li, Ita Lifshitz, Rakesh Ranjan
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.07766
• Pdf URL: https://arxiv.org/pdf/2412.07766
• Copy Paste: [[2412.07766]] Make-A-Texture: Fast Shape-Aware Texture Generation in 3 Seconds(https://arxiv.org/abs/2412.07766)
• Keywords: diffusion
• Abstract: We present Make-A-Texture, a new framework that efficiently synthesizes high-resolution texture maps from textual prompts for given 3D geometries. Our approach progressively generates textures that are consistent across multiple viewpoints with a depth-aware inpainting diffusion model, in an optimized sequence of viewpoints determined by an automatic view selection algorithm. A significant feature of our method is its remarkable efficiency, achieving a full texture generation within an end-to-end runtime of just 3.07 seconds on a single NVIDIA H100 GPU, significantly outperforming existing methods. Such an acceleration is achieved by optimizations in the diffusion model and a specialized backprojection method. Moreover, our method reduces the artifacts in the backprojection phase, by selectively masking out non-frontal faces, and internal faces of open-surfaced objects. Experimental results demonstrate that Make-A-Texture matches or exceeds the quality of other state-of-the-art methods. Our work significantly improves the applicability and practicality of texture generation models for real-world 3D content creation, including interactive creation and text-guided texture editing.

Title: Learning Visual Generative Priors without Text

• Authors: Shuailei Ma, Kecheng Zheng, Ying Wei, Wei Wu, Fan Lu, Yifei Zhang, Chen-wei Xie, Jiapeng Zhu, Yujun Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07767
• Pdf URL: https://arxiv.org/pdf/2412.07767
• Copy Paste: [[2412.07767]] Learning Visual Generative Priors without Text(https://arxiv.org/abs/2412.07767)
• Keywords: generative
• Abstract: Although text-to-image (T2I) models have recently thrived as visual generative priors, their reliance on high-quality text-image pairs makes scaling up expensive. We argue that grasping the cross-modality alignment is not a necessity for a sound visual generative prior, whose focus should be on texture modeling. Such a philosophy inspires us to study image-to-image (I2I) generation, where models can learn from in-the-wild images in a self-supervised manner. We first develop a pure vision-based training framework, Lumos, and confirm the feasibility and the scalability of learning I2I models. We then find that, as an upstream task of T2I, our I2I model serves as a more foundational visual prior and achieves on-par or better performance than existing T2I models using only 1/10 text-image pairs for fine-tuning. We further demonstrate the superiority of I2I priors over T2I priors on some text-irrelevant visual generative tasks, like image-to-3D and image-to-video.

Title: From an Image to a Scene: Learning to Imagine the World from a Million 360 Videos

• Authors: Matthew Wallingford, Anand Bhattad, Aditya Kusupati, Vivek Ramanujan, Matt Deitke, Sham Kakade, Aniruddha Kembhavi, Roozbeh Mottaghi, Wei-Chiu Ma, Ali Farhadi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07770
• Pdf URL: https://arxiv.org/pdf/2412.07770
• Copy Paste: [[2412.07770]] From an Image to a Scene: Learning to Imagine the World from a Million 360 Videos(https://arxiv.org/abs/2412.07770)
• Keywords: diffusion
• Abstract: Three-dimensional (3D) understanding of objects and scenes play a key role in humans' ability to interact with the world and has been an active area of research in computer vision, graphics, and robotics. Large scale synthetic and object-centric 3D datasets have shown to be effective in training models that have 3D understanding of objects. However, applying a similar approach to real-world objects and scenes is difficult due to a lack of large-scale data. Videos are a potential source for real-world 3D data, but finding diverse yet corresponding views of the same content has shown to be difficult at scale. Furthermore, standard videos come with fixed viewpoints, determined at the time of capture. This restricts the ability to access scenes from a variety of more diverse and potentially useful perspectives. We argue that large scale 360 videos can address these limitations to provide: scalable corresponding frames from diverse views. In this paper, we introduce 360-1M, a 360 video dataset, and a process for efficiently finding corresponding frames from diverse viewpoints at scale. We train our diffusion-based model, Odin, on 360-1M. Empowered by the largest real-world, multi-view dataset to date, Odin is able to freely generate novel views of real-world scenes. Unlike previous methods, Odin can move the camera through the environment, enabling the model to infer the geometry and layout of the scene. Additionally, we show improved performance on standard novel view synthesis and 3D reconstruction benchmarks.

Title: From Slow Bidirectional to Fast Causal Video Generators

• Authors: Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Fredo Durand, Eli Shechtman, Xun Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07772
• Pdf URL: https://arxiv.org/pdf/2412.07772
• Copy Paste: [[2412.07772]] From Slow Bidirectional to Fast Causal Video Generators(https://arxiv.org/abs/2412.07772)
• Keywords: diffusion, transformer
• Abstract: Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies. The generation of a single frame requires the model to process the entire sequence, including the future. We address this limitation by adapting a pretrained bidirectional diffusion transformer to a causal transformer that generates frames on-the-fly. To further reduce latency, we extend distribution matching distillation (DMD) to videos, distilling 50-step diffusion model into a 4-step generator. To enable stable and high-quality distillation, we introduce a student initialization scheme based on teacher's ODE trajectories, as well as an asymmetric distillation strategy that supervises a causal student model with a bidirectional teacher. This approach effectively mitigates error accumulation in autoregressive generation, allowing long-duration video synthesis despite training on short clips. Our model supports fast streaming generation of high quality videos at 9.4 FPS on a single GPU thanks to KV caching. Our approach also enables streaming video-to-video translation, image-to-video, and dynamic prompting in a zero-shot manner. We will release the code based on an open-source model in the future.

Title: Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets

• Authors: Zhen Liu, Tim Z. Xiao, Weiyang Liu, Yoshua Bengio, Dinghuai Zhang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07775
• Pdf URL: https://arxiv.org/pdf/2412.07775
• Copy Paste: [[2412.07775]] Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets(https://arxiv.org/abs/2412.07775)
• Keywords: diffusion, generative
• Abstract: While one commonly trains large diffusion models by collecting datasets on target downstream tasks, it is often desired to align and finetune pretrained diffusion models on some reward functions that are either designed by experts or learned from small-scale datasets. Existing methods for finetuning diffusion models typically suffer from lack of diversity in generated samples, lack of prior preservation, and/or slow convergence in finetuning. Inspired by recent successes in generative flow networks (GFlowNets), a class of probabilistic models that sample with the unnormalized density of a reward function, we propose a novel GFlowNet method dubbed Nabla-GFlowNet (abbreviated as $\nabla$-GFlowNet), the first GFlowNet method that leverages the rich signal in reward gradients, together with an objective called $\nabla$-DB plus its variant residual $\nabla$-DB designed for prior-preserving diffusion alignment. We show that our proposed method achieves fast yet diversity- and prior-preserving alignment of Stable Diffusion, a large-scale text-conditioned image diffusion model, on different realistic reward functions.

Title: Video Motion Transfer with Diffusion Transformers

• Authors: Alexander Pondaven, Aliaksandr Siarohin, Sergey Tulyakov, Philip Torr, Fabio Pizzati
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07776
• Pdf URL: https://arxiv.org/pdf/2412.07776
• Copy Paste: [[2412.07776]] Video Motion Transfer with Diffusion Transformers(https://arxiv.org/abs/2412.07776)
• Keywords: diffusion, transformer
• Abstract: We propose DiTFlow, a method for transferring the motion of a reference video to a newly synthesized one, designed specifically for Diffusion Transformers (DiT). We first process the reference video with a pre-trained DiT to analyze cross-frame attention maps and extract a patch-wise motion signal called the Attention Motion Flow (AMF). We guide the latent denoising process in an optimization-based, training-free, manner by optimizing latents with our AMF loss to generate videos reproducing the motion of the reference one. We also apply our optimization strategy to transformer positional embeddings, granting us a boost in zero-shot motion transfer capabilities. We evaluate DiTFlow against recently published methods, outperforming all across multiple metrics and human evaluation.