2024-12-24

Title: Superposition through Active Learning lens

• Authors: Akanksha Devkar
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16168
• Pdf URL: https://arxiv.org/pdf/2412.16168
• Copy Paste: [[2412.16168]] Superposition through Active Learning lens(https://arxiv.org/abs/2412.16168)
• Keywords: interpretability
• Abstract: Superposition or Neuron Polysemanticity are important concepts in the field of interpretability and one might say they are these most intricately beautiful blockers in our path of decoding the Machine Learning black-box. The idea behind this paper is to examine whether it is possible to decode Superposition using Active Learning methods. While it seems that Superposition is an attempt to arrange more features in smaller space to better utilize the limited resources, it might be worth inspecting if Superposition is dependent on any other factors. This paper uses CIFAR-10 and Tiny ImageNet image datasets and the ResNet18 model and compares Baseline and Active Learning models and the presence of Superposition in them is inspected across multiple criteria, including t-SNE visualizations, cosine similarity histograms, Silhouette Scores, and Davies-Bouldin Indexes. Contrary to our expectations, the active learning model did not significantly outperform the baseline in terms of feature separation and overall accuracy. This suggests that non-informative sample selection and potential overfitting to uncertain samples may have hindered the active learning model's ability to generalize better suggesting more sophisticated approaches might be needed to decode superposition and potentially reduce it.

Title: Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs

• Authors: Michael Wornow, Suhana Bedi, Miguel Angel Fuentes Hernandez, Ethan Steinberg, Jason Alan Fries, Christopher Ré, Sanmi Koyejo, Nigam H. Shah
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2412.16178
• Pdf URL: https://arxiv.org/pdf/2412.16178
• Copy Paste: [[2412.16178]] Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs(https://arxiv.org/abs/2412.16178)
• Keywords: robust
• Abstract: Foundation Models (FMs) trained on Electronic Health Records (EHRs) have achieved state-of-the-art results on numerous clinical prediction tasks. However, most existing EHR FMs have context windows of <1k tokens. This prevents them from modeling full patient EHRs which can exceed 10k's of events. Recent advancements in subquadratic long-context architectures (e.g., Mamba) offer a promising solution. However, their application to EHR data has not been well-studied. We address this gap by presenting the first systematic evaluation of the effect of context length on modeling EHR data. We find that longer context models improve predictive performance -- our Mamba-based model surpasses the prior state-of-the-art on 9/14 tasks on the EHRSHOT prediction benchmark. For clinical applications, however, model performance alone is insufficient -- robustness to the unique properties of EHR is crucial. Thus, we also evaluate models across three previously underexplored properties of EHR data: (1) the prevalence of "copy-forwarded" diagnoses which creates artificial repetition of tokens within EHR sequences; (2) the irregular time intervals between EHR events which can lead to a wide range of timespans within a context window; and (3) the natural increase in disease complexity over time which makes later tokens in the EHR harder to predict than earlier ones. Stratifying our EHRSHOT results, we find that higher levels of each property correlate negatively with model performance, but that longer context models are more robust to more extreme levels of these properties. Our work highlights the potential for using long-context architectures to model EHR data, and offers a case study for identifying new challenges in modeling sequential data motivated by domains outside of natural language. We release our models and code at: this https URL

Title: HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing

• Authors: Minghui Liu, Tahseen Rabbani, Tony O'Halloran, Ananth Sankaralingam, Mary-Anne Hartley, Brian Gravelle, Furong Huang, Cornelia Fermüller, Yiannis Aloimonos
• Subjects: cs.LG, cs.AI, cs.CL, cs.DS, cs.PF
• Abstract URL: https://arxiv.org/abs/2412.16187
• Pdf URL: https://arxiv.org/pdf/2412.16187
• Copy Paste: [[2412.16187]] HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing(https://arxiv.org/abs/2412.16187)
• Keywords: transformer, large language model
• Abstract: Transformer-based large language models (LLMs) use the key-value (KV) cache to significantly accelerate inference by storing the key and value embeddings of past tokens. However, this cache consumes significant GPU memory. In this work, we introduce LSH-E, an algorithm that uses locality-sensitive hashing (LSH) to compress the KV cache. LSH-E quickly locates tokens in the cache that are cosine dissimilar to the current query token. This is achieved by computing the Hamming distance between binarized Gaussian projections of the current token query and cached token keys, with a projection length much smaller than the embedding dimension. We maintain a lightweight binary structure in GPU memory to facilitate these calculations. Unlike existing compression strategies that compute attention to determine token retention, LSH-E makes these decisions pre-attention, thereby reducing computational costs. Additionally, LSH-E is dynamic - at every decoding step, the key and value of the current token replace the embeddings of a token expected to produce the lowest attention score. We demonstrate that LSH-E can compress the KV cache by 30%-70% while maintaining high performance across reasoning, multiple-choice, long-context retrieval and summarization tasks.

Title: A Decade of Deep Learning: A Survey on The Magnificent Seven

• Authors: Dilshod Azizov, Muhammad Arslan Manzoor, Velibor Bojkovic, Yingxu Wang, Zixiao Wang, Zangir Iklassov, Kailong Zhao, Liang Li, Siwei Liu, Yu Zhong, Wei Liu, Shangsong Liang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16188
• Pdf URL: https://arxiv.org/pdf/2412.16188
• Copy Paste: [[2412.16188]] A Decade of Deep Learning: A Survey on The Magnificent Seven(https://arxiv.org/abs/2412.16188)
• Keywords: extraction, diffusion, transformer, generative
• Abstract: Deep learning has fundamentally reshaped the landscape of artificial intelligence over the past decade, enabling remarkable achievements across diverse domains. At the heart of these developments lie multi-layered neural network architectures that excel at automatic feature extraction, leading to significant improvements in machine learning tasks. To demystify these advances and offer accessible guidance, we present a comprehensive overview of the most influential deep learning algorithms selected through a broad-based survey of the field. Our discussion centers on pivotal architectures, including Residual Networks, Transformers, Generative Adversarial Networks, Variational Autoencoders, Graph Neural Networks, Contrastive Language-Image Pre-training, and Diffusion models. We detail their historical context, highlight their mathematical foundations and algorithmic principles, and examine subsequent variants, extensions, and practical considerations such as training methodologies, normalization techniques, and learning rate schedules. Beyond historical and technical insights, we also address their applications, challenges, and potential research directions. This survey aims to serve as a practical manual for both newcomers seeking an entry point into cutting-edge deep learning methods and experienced researchers transitioning into this rapidly evolving domain.

Title: Resilient Cloud cluster with DevSecOps security model, automates a data analysis, vulnerability search and risk calculation

• Authors: Abed Saif Ahmed Alghawli, Tamara Radivilova
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16190
• Pdf URL: https://arxiv.org/pdf/2412.16190
• Copy Paste: [[2412.16190]] Resilient Cloud cluster with DevSecOps security model, automates a data analysis, vulnerability search and risk calculation(https://arxiv.org/abs/2412.16190)
• Keywords: secure, security, fair
• Abstract: Automated, secure software development is an important task of digitalization, which is solved with the DevSecOps approach. An important part of the DevSecOps approach is continuous risk assessment, which is necessary to identify and evaluate risk factors. Combining the development cycle with continuous risk assessment creates synergies in software development and operation and minimizes vulnerabilities. The article presents the main methods of deploying web applications, ways to increase the level of information security at all stages of product development, compares different types of infrastructures and cloud computing providers, and analyzes modern tools used to automate processes. The cloud cluster was deployed using Terraform and the Jenkins pipeline, which is written in the Groovy programming language, which checks program code for vulnerabilities and allows you to fix violations at the earliest stages of developing secure web applications. The developed cluster implements the proposed algorithm for automated risk assessment based on the calculation (modeling) of threats and vulnerabilities of cloud infrastructure, which operates in real time, periodically collecting all information and adjusting the system in accordance with the risk and applied controls. The algorithm for calculating risk and losses is based on statistical data and the concept of the FAIR information risk assessment methodology. The risk value obtained using the proposed method is quantitative, which allows more efficient forecasting of information security costs in software development.

Title: AgroXAI: Explainable AI-Driven Crop Recommendation System for Agriculture 4.0

• Authors: Ozlem Turgut, Ibrahim Kok, Suat Ozdemir
• Subjects: cs.LG, cs.AI, cs.NI
• Abstract URL: https://arxiv.org/abs/2412.16196
• Pdf URL: https://arxiv.org/pdf/2412.16196
• Copy Paste: [[2412.16196]] AgroXAI: Explainable AI-Driven Crop Recommendation System for Agriculture 4.0(https://arxiv.org/abs/2412.16196)
• Keywords: explainability
• Abstract: Today, crop diversification in agriculture is a critical issue to meet the increasing demand for food and improve food safety and quality. This issue is considered to be the most important challenge for the next generation of agriculture due to the diminishing natural resources, the limited arable land, and unpredictable climatic conditions caused by climate change. In this paper, we employ emerging technologies such as the Internet of Things (IoT), machine learning (ML), and explainable artificial intelligence (XAI) to improve operational efficiency and productivity in the agricultural sector. Specifically, we propose an edge computing-based explainable crop recommendation system, AgroXAI, which suggests suitable crops for a region based on weather and soil conditions. In this system, we provide local and global explanations of ML model decisions with methods such as ELI5, LIME, SHAP, which we integrate into ML models. More importantly, we provide regional alternative crop recommendations with the counterfactual explainability method. In this way, we envision that our proposed AgroXAI system will be a platform that provides regional crop diversity in the next generation agriculture.

Title: Stabilizing Machine Learning for Reproducible and Explainable Results: A Novel Validation Approach to Subject-Specific Insights

• Authors: Gideon Vos, Liza van Eijk, Zoltan Sarnyai, Mostafa Rahimi Azghadi
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.16199
• Pdf URL: https://arxiv.org/pdf/2412.16199
• Copy Paste: [[2412.16199]] Stabilizing Machine Learning for Reproducible and Explainable Results: A Novel Validation Approach to Subject-Specific Insights(https://arxiv.org/abs/2412.16199)
• Keywords: robust
• Abstract: Machine Learning is transforming medical research by improving diagnostic accuracy and personalizing treatments. General ML models trained on large datasets identify broad patterns across populations, but their effectiveness is often limited by the diversity of human biology. This has led to interest in subject-specific models that use individual data for more precise predictions. However, these models are costly and challenging to develop. To address this, we propose a novel validation approach that uses a general ML model to ensure reproducible performance and robust feature importance analysis at both group and subject-specific levels. We tested a single Random Forest (RF) model on nine datasets varying in domain, sample size, and demographics. Different validation techniques were applied to evaluate accuracy and feature importance consistency. To introduce variability, we performed up to 400 trials per subject, randomly seeding the ML algorithm for each trial. This generated 400 feature sets per subject, from which we identified top subject-specific features. A group-specific feature importance set was then derived from all subject-specific results. We compared our approach to conventional validation methods in terms of performance and feature importance consistency. Our repeated trials approach, with random seed variation, consistently identified key features at the subject level and improved group-level feature importance analysis using a single general model. Subject-specific models address biological variability but are resource-intensive. Our novel validation technique provides consistent feature importance and improved accuracy within a general ML model, offering a practical and explainable alternative for clinical research.

Title: Robust Spectral Anomaly Detection in EELS Spectral Images via Three Dimensional Convolutional Variational Autoencoders

• Authors: Seyfal Sultanov, James P Buban, Robert F Klie
• Subjects: cs.CV, cond-mat.mtrl-sci, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16200
• Pdf URL: https://arxiv.org/pdf/2412.16200
• Copy Paste: [[2412.16200]] Robust Spectral Anomaly Detection in EELS Spectral Images via Three Dimensional Convolutional Variational Autoencoders(https://arxiv.org/abs/2412.16200)
• Keywords: robust
• Abstract: We introduce a Three-Dimensional Convolutional Variational Autoencoder (3D-CVAE) for automated anomaly detection in Electron Energy Loss Spectroscopy Spectrum Imaging (EELS-SI) data. Our approach leverages the full three-dimensional structure of EELS-SI data to detect subtle spectral anomalies while preserving both spatial and spectral correlations across the datacube. By employing negative log-likelihood loss and training on bulk spectra, the model learns to reconstruct bulk features characteristic of the defect-free material. In exploring methods for anomaly detection, we evaluated both our 3D-CVAE approach and Principal Component Analysis (PCA), testing their performance using Fe L-edge peak shifts designed to simulate material defects. Our results show that 3D-CVAE achieves superior anomaly detection and maintains consistent performance across various shift magnitudes. The method demonstrates clear bimodal separation between normal and anomalous spectra, enabling reliable classification. Further analysis verifies that lower dimensional representations are robust to anomalies in the data. While performance advantages over PCA diminish with decreasing anomaly concentration, our method maintains high reconstruction quality even in challenging, noise-dominated spectral regions. This approach provides a robust framework for unsupervised automated detection of spectral anomalies in EELS-SI data, particularly valuable for analyzing complex material systems.

Title: Saliency Methods are Encoders: Analysing Logical Relations Towards Interpretation

• Authors: Leonid Schwenke, Martin Atzmueller
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16204
• Pdf URL: https://arxiv.org/pdf/2412.16204
• Copy Paste: [[2412.16204]] Saliency Methods are Encoders: Analysing Logical Relations Towards Interpretation(https://arxiv.org/abs/2412.16204)
• Keywords: fair, interpretability, explainability
• Abstract: With their increase in performance, neural network architectures also become more complex, necessitating explainability. Therefore, many new and improved methods are currently emerging, which often generate so-called saliency maps in order to improve interpretability. Those methods are often evaluated by visual expectations, yet this typically leads towards a confirmation bias. Due to a lack of a general metric for explanation quality, non-accessible ground truth data about the model's reasoning and the large amount of involved assumptions, multiple works claim to find flaws in those methods. However, this often leads to unfair comparison metrics. Additionally, the complexity of most datasets (mostly images or text) is often so high, that approximating all possible explanations is not feasible. For those reasons, this paper introduces a test for saliency map evaluation: proposing controlled experiments based on all possible model reasonings over multiple simple logical datasets. Using the contained logical relationships, we aim to understand how different saliency methods treat information in different class discriminative scenarios (e.g. via complementary and redundant information). By introducing multiple new metrics, we analyse propositional logical patterns towards a non-informative attribution score baseline to find deviations of typical expectations. Our results show that saliency methods can encode classification relevant information into the ordering of saliency scores.

Title: Synthetic Time Series Data Generation for Healthcare Applications: A PCG Case Study

• Authors: Ainaz Jamshidi, Muhammad Arif, Sabir Ali Kalhoro, Alexander Gelbukh
• Subjects: cs.LG, cs.CE, eess.SP
• Abstract URL: https://arxiv.org/abs/2412.16207
• Pdf URL: https://arxiv.org/pdf/2412.16207
• Copy Paste: [[2412.16207]] Synthetic Time Series Data Generation for Healthcare Applications: A PCG Case Study(https://arxiv.org/abs/2412.16207)
• Keywords: privacy, robust, generative
• Abstract: The generation of high-quality medical time series data is essential for advancing healthcare diagnostics and safeguarding patient privacy. Specifically, synthesizing realistic phonocardiogram (PCG) signals offers significant potential as a cost-effective and efficient tool for cardiac disease pre-screening. Despite its potential, the synthesis of PCG signals for this specific application received limited attention in research. In this study, we employ and compare three state-of-the-art generative models from different categories - WaveNet, DoppelGANger, and DiffWave - to generate high-quality PCG data. We use data from the George B. Moody PhysioNet Challenge 2022. Our methods are evaluated using various metrics widely used in the previous literature in the domain of time series data generation, such as mean absolute error and maximum mean discrepancy. Our results demonstrate that the generated PCG data closely resembles the original datasets, indicating the effectiveness of our generative models in producing realistic synthetic PCG data. In our future work, we plan to incorporate this method into a data augmentation pipeline to synthesize abnormal PCG signals with heart murmurs, in order to address the current scarcity of abnormal data. We hope to improve the robustness and accuracy of diagnostic tools in cardiology, enhancing their effectiveness in detecting heart murmurs.

Title: Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation

• Authors: Yiping Wang, Xuehai He, Kuan Wang, Luyao Ma, Jianwei Yang, Shuohang Wang, Simon Shaolei Du, Yelong Shen
• Subjects: cs.CV, cs.CL, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.16211
• Pdf URL: https://arxiv.org/pdf/2412.16211
• Copy Paste: [[2412.16211]] Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation(https://arxiv.org/abs/2412.16211)
• Keywords: generative
• Abstract: The current state-of-the-art video generative models can produce commercial-grade videos with highly realistic details. However, they still struggle to coherently present multiple sequential events in the stories specified by the prompts, which is foreseeable an essential capability for future long video generation scenarios. For example, top T2V generative models still fail to generate a video of the short simple story 'how to put an elephant into a refrigerator.' While existing detail-oriented benchmarks primarily focus on fine-grained metrics like aesthetic quality and spatial-temporal consistency, they fall short of evaluating models' abilities to handle event-level story presentation. To address this gap, we introduce StoryEval, a story-oriented benchmark specifically designed to assess text-to-video (T2V) models' story-completion capabilities. StoryEval features 423 prompts spanning 7 classes, each representing short stories composed of 2-4 consecutive events. We employ advanced vision-language models, such as GPT-4V and LLaVA-OV-Chat-72B, to verify the completion of each event in the generated videos, applying a unanimous voting method to enhance reliability. Our methods ensure high alignment with human evaluations, and the evaluation of 11 models reveals its challenge, with none exceeding an average story-completion rate of 50%. StoryEval provides a new benchmark for advancing T2V models and highlights the challenges and opportunities in developing next-generation solutions for coherent story-driven video generation.

Title: AdvIRL: Reinforcement Learning-Based Adversarial Attacks on 3D NeRF Models

• Authors: Tommy Nguyen, Mehmet Ergezer, Christian Green
• Subjects: cs.CV, cs.AI, cs.CY, cs.GR, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.16213
• Pdf URL: https://arxiv.org/pdf/2412.16213
• Copy Paste: [[2412.16213]] AdvIRL: Reinforcement Learning-Based Adversarial Attacks on 3D NeRF Models(https://arxiv.org/abs/2412.16213)
• Keywords: attack, robust, generative
• Abstract: The increasing deployment of AI models in critical applications has exposed them to significant risks from adversarial attacks. While adversarial vulnerabilities in 2D vision models have been extensively studied, the threat landscape for 3D generative models, such as Neural Radiance Fields (NeRF), remains underexplored. This work introduces \textit{AdvIRL}, a novel framework for crafting adversarial NeRF models using Instant Neural Graphics Primitives (Instant-NGP) and Reinforcement Learning. Unlike prior methods, \textit{AdvIRL} generates adversarial noise that remains robust under diverse 3D transformations, including rotations and scaling, enabling effective black-box attacks in real-world scenarios. Our approach is validated across a wide range of scenes, from small objects (e.g., bananas) to large environments (e.g., lighthouses). Notably, targeted attacks achieved high-confidence misclassifications, such as labeling a banana as a slug and a truck as a cannon, demonstrating the practical risks posed by adversarial NeRFs. Beyond attacking, \textit{AdvIRL}-generated adversarial models can serve as adversarial training data to enhance the robustness of vision systems. The implementation of \textit{AdvIRL} is publicly available at \url{this https URL}, ensuring reproducibility and facilitating future research.

Title: FairTP: A Prolonged Fairness Framework for Traffic Prediction

• Authors: Jiangnan Xia, Yu Yang, Jiaxing Shen, Senzhang Wang, Jiannong Cao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16214
• Pdf URL: https://arxiv.org/pdf/2412.16214
• Copy Paste: [[2412.16214]] FairTP: A Prolonged Fairness Framework for Traffic Prediction(https://arxiv.org/abs/2412.16214)
• Keywords: fair
• Abstract: Traffic prediction plays a crucial role in intelligent transportation systems. Existing approaches primarily focus on improving overall accuracy, often neglecting a critical issue: whether predictive models lead to biased decisions by transportation authorities. In practice, the uneven deployment of traffic sensors across urban areas results in imbalanced data, causing prediction models to perform poorly in certain regions and leading to unfair decision-making. This imbalance ultimately harms the equity and quality of life for residents. Moreover, current fairness-aware machine learning models only ensure fairness at specific time points, failing to maintain fairness over extended periods. As traffic conditions change, such static fairness approaches become ineffective. To address this gap, we propose FairTP, a framework for prolonged fair traffic prediction. We introduce two new fairness definitions tailored for dynamic traffic scenarios. Fairness in traffic prediction is not static; it varies over time and across regions. Each sensor or urban area can alternate between two states: "sacrifice" (low prediction accuracy) and "benefit" (high prediction accuracy). Prolonged fairness is achieved when the overall states of sensors remain similar over a given period. We define two types of fairness: region-based static fairness and sensor-based dynamic fairness. To implement this, FairTP incorporates a state identification module to classify sensors' states as either "sacrifice" or "benefit," enabling prolonged fairness-aware predictions. Additionally, we introduce a state-guided balanced sampling strategy to further enhance fairness, addressing performance disparities among regions with uneven sensor distributions. Extensive experiments on two real-world datasets demonstrate that FairTP significantly improves prediction fairness while minimizing accuracy degradation.

Title: Zero-Shot Image Moderation in Google Ads with LLM-Assisted Textual Descriptions and Cross-modal Co-embeddings

• Authors: Enming Luo, Wei Qiao, Katie Warren, Jingxiang Li, Eric Xiao, Krishna Viswanathan, Yuan Wang, Yintao Liu, Jimin Li, Ariel Fuxman
• Subjects: cs.CV, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.16215
• Pdf URL: https://arxiv.org/pdf/2412.16215
• Copy Paste: [[2412.16215]] Zero-Shot Image Moderation in Google Ads with LLM-Assisted Textual Descriptions and Cross-modal Co-embeddings(https://arxiv.org/abs/2412.16215)
• Keywords: large language model
• Abstract: We present a scalable and agile approach for ads image content moderation at Google, addressing the challenges of moderating massive volumes of ads with diverse content and evolving policies. The proposed method utilizes human-curated textual descriptions and cross-modal text-image co-embeddings to enable zero-shot classification of policy violating ads images, bypassing the need for extensive supervised training data and human labeling. By leveraging large language models (LLMs) and user expertise, the system generates and refines a comprehensive set of textual descriptions representing policy guidelines. During inference, co-embedding similarity between incoming images and the textual descriptions serves as a reliable signal for policy violation detection, enabling efficient and adaptable ads content moderation. Evaluation results demonstrate the efficacy of this framework in significantly boosting the detection of policy violating content.

Title: GNN-Transformer Cooperative Architecture for Trustworthy Graph Contrastive Learning

• Authors: Jianqing Liang, Xinkai Wei, Min Chen, Yuan Liu, Zhiqiang Wang, Jiye Liang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16218
• Pdf URL: https://arxiv.org/pdf/2412.16218
• Copy Paste: [[2412.16218]] GNN-Transformer Cooperative Architecture for Trustworthy Graph Contrastive Learning(https://arxiv.org/abs/2412.16218)
• Keywords: transformer
• Abstract: Graph contrastive learning (GCL) has become a hot topic in the field of graph representation learning. In contrast to traditional supervised learning relying on a large number of labels, GCL exploits augmentation strategies to generate multiple views and positive/negative pairs, both of which greatly influence the performance. Unfortunately, commonly used random augmentations may disturb the underlying semantics of graphs. Moreover, traditional GNNs, a type of widely employed encoders in GCL, are inevitably confronted with over-smoothing and over-squashing problems. To address these issues, we propose GNN-Transformer Cooperative Architecture for Trustworthy Graph Contrastive Learning (GTCA), which inherits the advantages of both GNN and Transformer, incorporating graph topology to obtain comprehensive graph representations. Theoretical analysis verifies the trustworthiness of the proposed method. Extensive experiments on benchmark datasets demonstrate state-of-the-art empirical performance.

Title: Adaptive Calibration: A Unified Conversion Framework of Spiking Neural Network

• Authors: Ziqing Wang, Yuetong Fang, Jiahang Cao, Hongwei Ren, Renjing Xu
• Subjects: cs.CV, cs.NE
• Abstract URL: https://arxiv.org/abs/2412.16219
• Pdf URL: https://arxiv.org/pdf/2412.16219
• Copy Paste: [[2412.16219]] Adaptive Calibration: A Unified Conversion Framework of Spiking Neural Network(https://arxiv.org/abs/2412.16219)
• Keywords: segmentation
• Abstract: Spiking Neural Networks (SNNs) are seen as an energy-efficient alternative to traditional Artificial Neural Networks (ANNs), but the performance gap remains a challenge. While this gap is narrowing through ANN-to-SNN conversion, substantial computational resources are still needed, and the energy efficiency of converted SNNs cannot be ensured. To address this, we present a unified training-free conversion framework that significantly enhances both the performance and efficiency of converted SNNs. Inspired by the biological nervous system, we propose a novel Adaptive-Firing Neuron Model (AdaFire), which dynamically adjusts firing patterns across different layers to substantially reduce the Unevenness Error - the primary source of error of converted SNNs within limited inference timesteps. We further introduce two efficiency-enhancing techniques: the Sensitivity Spike Compression (SSC) technique for reducing spike operations, and the Input-aware Adaptive Timesteps (IAT) technique for decreasing latency. These methods collectively enable our approach to achieve state-of-the-art performance while delivering significant energy savings of up to 70.1%, 60.3%, and 43.1% on CIFAR-10, CIFAR-100, and ImageNet datasets, respectively. Extensive experiments across 2D, 3D, event-driven classification tasks, object detection, and segmentation tasks, demonstrate the effectiveness of our method in various domains. The code is available at: this https URL.

Title: Formal Verification of Permission Voucher

• Authors: Khan Reaz, Gerhard Wunder
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16224
• Pdf URL: https://arxiv.org/pdf/2412.16224
• Copy Paste: [[2412.16224]] Formal Verification of Permission Voucher(https://arxiv.org/abs/2412.16224)
• Keywords: secure, security, attack, robust
• Abstract: Formal verification is a critical process in ensuring the security and correctness of cryptographic protocols, particularly in high-assurance domains. This paper presents a comprehensive formal analysis of the Permission Voucher Protocol, a system designed for secure and authenticated access control in distributed environments. The analysis employs the Tamarin Prover, a state-of-the-art tool for symbolic verification, to evaluate key security properties such as authentication, confidentiality, integrity, mutual authentication, and replay prevention. We model the protocol's components, including trust relationships, secure channels, and adversary capabilities under the Dolev-Yao model. Verification results confirm the protocol's robustness against common attacks such as message tampering, impersonation, and replay. Additionally, dependency graphs and detailed proofs demonstrate the successful enforcement of security properties like voucher authenticity, data confidentiality, and key integrity. The study identifies potential enhancements, such as incorporating timestamp-based validity checks and augmenting mutual authentication mechanisms to address insider threats and key management challenges. This work highlights the advantages and limitations of using the Tamarin Prover for formal security verification and proposes strategies to mitigate scalability and performance constraints in complex systems.

Title: GALOT: Generative Active Learning via Optimizable Zero-shot Text-to-image Generation

• Authors: Hanbin Hong, Shenao Yan, Shuya Feng, Yan Yan, Yuan Hong
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16227
• Pdf URL: https://arxiv.org/pdf/2412.16227
• Copy Paste: [[2412.16227]] GALOT: Generative Active Learning via Optimizable Zero-shot Text-to-image Generation(https://arxiv.org/abs/2412.16227)
• Keywords: generative
• Abstract: Active Learning (AL) represents a crucial methodology within machine learning, emphasizing the identification and utilization of the most informative samples for efficient model training. However, a significant challenge of AL is its dependence on the limited labeled data samples and data distribution, resulting in limited performance. To address this limitation, this paper integrates the zero-shot text-to-image (T2I) synthesis and active learning by designing a novel framework that can efficiently train a machine learning (ML) model sorely using the text description. Specifically, we leverage the AL criteria to optimize the text inputs for generating more informative and diverse data samples, annotated by the pseudo-label crafted from text, then served as a synthetic dataset for active learning. This approach reduces the cost of data collection and annotation while increasing the efficiency of model training by providing informative training samples, enabling a novel end-to-end ML task from text description to vision models. Through comprehensive evaluations, our framework demonstrates consistent and significant improvements over traditional AL methods.

Title: Is AI Robust Enough for Scientific Research?

• Authors: Jun-Jie Zhang, Jiahao Song, Xiu-Cheng Wang, Fu-Peng Li, Zehan Liu, Jian-Nan Chen, Haoning Dang, Shiyao Wang, Yiyan Zhang, Jianhui Xu, Chunxiang Shi, Fei Wang, Long-Gang Pang, Nan Cheng, Weiwei Zhang, Duo Zhang, Deyu Meng
• Subjects: cs.LG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2412.16234
• Pdf URL: https://arxiv.org/pdf/2412.16234
• Copy Paste: [[2412.16234]] Is AI Robust Enough for Scientific Research?(https://arxiv.org/abs/2412.16234)
• Keywords: security, robust
• Abstract: We uncover a phenomenon largely overlooked by the scientific community utilizing AI: neural networks exhibit high susceptibility to minute perturbations, resulting in significant deviations in their outputs. Through an analysis of five diverse application areas -- weather forecasting, chemical energy and force calculations, fluid dynamics, quantum chromodynamics, and wireless communication -- we demonstrate that this vulnerability is a broad and general characteristic of AI systems. This revelation exposes a hidden risk in relying on neural networks for essential scientific computations, calling further studies on their reliability and security.

Title: Utilizing Causal Network Markers to Identify Tipping Points ahead of Critical Transition

• Authors: Shirui Bian, Zezhou Wang, Siyang Leng, Wei Lin, Jifan Shi
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.16235
• Pdf URL: https://arxiv.org/pdf/2412.16235
• Copy Paste: [[2412.16235]] Utilizing Causal Network Markers to Identify Tipping Points ahead of Critical Transition(https://arxiv.org/abs/2412.16235)
• Keywords: robust
• Abstract: Early-warning signals of delicate design are always used to predict critical transitions in complex systems, which makes it possible to render the systems far away from the catastrophic state by introducing timely interventions. Traditional signals including the dynamical network biomarker (DNB), based on statistical properties such as variance and autocorrelation of nodal dynamics, overlook directional interactions and thus have limitations in capturing underlying mechanisms and simultaneously sustaining robustness against noise perturbations. This paper therefore introduces a framework of causal network markers (CNMs) by incorporating causality indicators, which reflect the directional influence between variables. Actually, to detect and identify the tipping points ahead of critical transition, two markers are designed: CNM-GC for linear causality and CNM-TE for non-linear causality, as well as a functional representation of different causality indicators and a clustering technique to verify the system's dominant group. Through demonstrations using benchmark models and real-world datasets of epileptic seizure, the framework of CNMs shows higher predictive power and accuracy than the traditional DNB indicator. It is believed that, due to the versatility and scalability, the CNMs are suitable for comprehensively evaluating the systems. The most possible direction for application includes the identification of tipping points in clinical disease.

Title: Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Data

• Authors: Zhiqiang Tang, Zihan Zhong, Tong He, Gerald Friedland
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16243
• Pdf URL: https://arxiv.org/pdf/2412.16243
• Copy Paste: [[2412.16243]] Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Data(https://arxiv.org/abs/2412.16243)
• Keywords: robust
• Abstract: This paper studies the best practices for automatic machine learning (AutoML). While previous AutoML efforts have predominantly focused on unimodal data, the multimodal aspect remains under-explored. Our study delves into classification and regression problems involving flexible combinations of image, text, and tabular data. We curate a benchmark comprising 22 multimodal datasets from diverse real-world applications, encompassing all 4 combinations of the 3 modalities. Across this benchmark, we scrutinize design choices related to multimodal fusion strategies, multimodal data augmentation, converting tabular data into text, cross-modal alignment, and handling missing modalities. Through extensive experimentation and analysis, we distill a collection of effective strategies and consolidate them into a unified pipeline, achieving robust performance on diverse datasets.

Title: Web Privacy based on Contextual Integrity: Measuring the Collapse of Online Contexts

• Authors: Ido Sivan-Sevilla, Parthav Poudel
• Subjects: cs.CR, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.16246
• Pdf URL: https://arxiv.org/pdf/2412.16246
• Copy Paste: [[2412.16246]] Web Privacy based on Contextual Integrity: Measuring the Collapse of Online Contexts(https://arxiv.org/abs/2412.16246)
• Keywords: privacy, protect
• Abstract: The collapse of social contexts has been amplified by digital infrastructures but surprisingly received insufficient attention from Web privacy scholars. Users are persistently identified within and across distinct web contexts, in varying degrees, through and by different websites and trackers, losing the ability to maintain a fragmented identity. To systematically evaluate this structural privacy harm we operationalize the theory of Privacy as Contextual Integrity and measure persistent user identification within and between distinct Web contexts. We crawl the top-700 popular websites across the contexts of health, finance, news & media, LGBTQ, eCommerce, adult, and education websites, for 27 days, to learn how persistent browser identification via third-party cookies and JavaScript fingerprinting is diffused within and between web contexts. Past work measured Web tracking in bulk, highlighting the volume of trackers and tracking techniques. These measurements miss a crucial privacy implication of Web tracking - the collapse of online contexts. Our findings reveal how persistent browser identification varies between and within contexts, diffusing user IDs to different distances, contrasting known tracking distributions across websites, and conducted as a joint or separate effort via cookie IDs and JS fingerprinting. Our network analysis can inform the construction of browser storage containers to protect users against real-time context collapse. This is a first modest step in measuring Web privacy as contextual integrity, opening new avenues for contextual Web privacy research.

Title: Towards scientific discovery with dictionary learning: Extracting biological concepts from microscopy foundation models

• Authors: Konstantin Donhauser, Kristina Ulicna, Gemma Elyse Moran, Aditya Ravuri, Kian Kenyon-Dean, Cian Eastwood, Jason Hartford
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.16247
• Pdf URL: https://arxiv.org/pdf/2412.16247
• Copy Paste: [[2412.16247]] Towards scientific discovery with dictionary learning: Extracting biological concepts from microscopy foundation models(https://arxiv.org/abs/2412.16247)
• Keywords: interpretability, large language model
• Abstract: Dictionary learning (DL) has emerged as a powerful interpretability tool for large language models. By extracting known concepts (e.g., Golden-Gate Bridge) from human-interpretable data (e.g., text), sparse DL can elucidate a model's inner workings. In this work, we ask if DL can also be used to discover unknown concepts from less human-interpretable scientific data (e.g., cell images), ultimately enabling modern approaches to scientific discovery. As a first step, we use DL algorithms to study microscopy foundation models trained on multi-cell image data, where little prior knowledge exists regarding which high-level concepts should arise. We show that sparse dictionaries indeed extract biologically-meaningful concepts such as cell type and genetic perturbation type. We also propose a new DL algorithm, Iterative Codebook Feature Learning~(ICFL), and combine it with a pre-processing step that uses PCA whitening from a control dataset. In our experiments, we demonstrate that both ICFL and PCA improve the selectivity of extracted features compared to TopK sparse autoencoders.

Title: Decoding fairness: a reinforcement learning perspective

• Authors: Guozhong Zheng, Jiqiang Zhang, Xin Ou, Shengfeng Deng, Li Chen
• Subjects: cs.LG, cond-mat.dis-nn, nlin.AO, physics.soc-ph, q-bio.PE
• Abstract URL: https://arxiv.org/abs/2412.16249
• Pdf URL: https://arxiv.org/pdf/2412.16249
• Copy Paste: [[2412.16249]] Decoding fairness: a reinforcement learning perspective(https://arxiv.org/abs/2412.16249)
• Keywords: robust, fair
• Abstract: Behavioral experiments on the ultimatum game (UG) reveal that we humans prefer fair acts, which contradicts the prediction made in orthodox Economics. Existing explanations, however, are mostly attributed to exogenous factors within the imitation learning framework. Here, we adopt the reinforcement learning paradigm, where individuals make their moves aiming to maximize their accumulated rewards. Specifically, we apply Q-learning to UG, where each player is assigned two Q-tables to guide decisions for the roles of proposer and responder. In a two-player scenario, fairness emerges prominently when both experiences and future rewards are appreciated. In particular, the probability of successful deals increases with higher offers, which aligns with observations in behavioral experiments. Our mechanism analysis reveals that the system undergoes two phases, eventually stabilizing into fair or rational strategies. These results are robust when the rotating role assignment is replaced by a random or fixed manner, or the scenario is extended to a latticed population. Our findings thus conclude that the endogenous factor is sufficient to explain the emergence of fairness, exogenous factors are not needed.

Title: Know2Vec: A Black-Box Proxy for Neural Network Retrieval

• Authors: Zhuoyi Shang, Yanwei Liu, Jinxia Liu, Xiaoyan Gu, Ying Ding, Xiangyang Ji
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16251
• Pdf URL: https://arxiv.org/pdf/2412.16251
• Copy Paste: [[2412.16251]] Know2Vec: A Black-Box Proxy for Neural Network Retrieval(https://arxiv.org/abs/2412.16251)
• Keywords: privacy
• Abstract: For general users, training a neural network from scratch is usually challenging and labor-intensive. Fortunately, neural network zoos enable them to find a well-performing model for directly use or fine-tuning it in their local environments. Although current model retrieval solutions attempt to convert neural network models into vectors to avoid complex multiple inference processes required for model selection, it is still difficult to choose a suitable model due to inaccurate vectorization and biased correlation alignment between the query dataset and models. From the perspective of knowledge consistency, i.e., whether the knowledge possessed by the model can meet the needs of query tasks, we propose a model retrieval scheme, named Know2Vec, that acts as a black-box retrieval proxy for model zoo. Know2Vec first accesses to models via a black-box interface in advance, capturing vital decision knowledge from models while ensuring their privacy. Next, it employs an effective encoding technique to transform the knowledge into precise model vectors. Secondly, it maps the user's query task to a knowledge vector by probing the semantic relationships within query samples. Furthermore, the proxy ensures the knowledge-consistency between query vector and model vectors within their alignment space, which is optimized through the supervised learning with diverse loss functions, and finally it can identify the most suitable model for a given task during the inference stage. Extensive experiments show that our Know2Vec achieves superior retrieval accuracy against the state-of-the-art methods in diverse neural network retrieval tasks.

Title: Post-hoc Interpretability Illumination for Scientific Interaction Discovery

• Authors: Ling Zhang, Zhichao Hou, Tingxiang Ji, Yuanyuan Xu, Runze Li
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.16252
• Pdf URL: https://arxiv.org/pdf/2412.16252
• Copy Paste: [[2412.16252]] Post-hoc Interpretability Illumination for Scientific Interaction Discovery(https://arxiv.org/abs/2412.16252)
• Keywords: interpretability, explainability
• Abstract: Model interpretability and explainability have garnered substantial attention in recent years, particularly in decision-making applications. However, existing interpretability tools often fall short in delivering satisfactory performance due to limited capabilities or efficiency issues. To address these challenges, we propose a novel post-hoc method: Iterative Kings' Forests (iKF), designed to uncover complex multi-order interactions among variables. iKF iteratively selects the next most important variable, the "King", and constructs King's Forests by placing it at the root node of each tree to identify variables that interact with the "King". It then generates ranked short lists of important variables and interactions of varying orders. Additionally, iKF provides inference metrics to analyze the patterns of the selected interactions and classify them into one of three interaction types: Accompanied Interaction, Synergistic Interaction, and Hierarchical Interaction. Extensive experiments demonstrate the strong interpretive power of our proposed iKF, highlighting its great potential for explainable modeling and scientific discovery across diverse scientific fields.

Title: Interactive Scene Authoring with Specialized Generative Primitives

• Authors: Clément Jambon (1), Changwoon Choi (2), Dongsu Zhang (2), Olga Sorkine-Hornung (1), Young Min Kim (2) ((1) ETH Zurich, (2) Seoul National University)
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.16253
• Pdf URL: https://arxiv.org/pdf/2412.16253
• Copy Paste: [[2412.16253]] Interactive Scene Authoring with Specialized Generative Primitives(https://arxiv.org/abs/2412.16253)
• Keywords: generative
• Abstract: Generating high-quality 3D digital assets often requires expert knowledge of complex design tools. We introduce Specialized Generative Primitives, a generative framework that allows non-expert users to author high-quality 3D scenes in a seamless, lightweight, and controllable manner. Each primitive is an efficient generative model that captures the distribution of a single exemplar from the real world. With our framework, users capture a video of an environment, which we turn into a high-quality and explicit appearance model thanks to 3D Gaussian Splatting. Users then select regions of interest guided by semantically-aware features. To create a generative primitive, we adapt Generative Cellular Automata to single-exemplar training and controllable generation. We decouple the generative task from the appearance model by operating on sparse voxels and we recover a high-quality output with a subsequent sparse patch consistency step. Each primitive can be trained within 10 minutes and used to author new scenes interactively in a fully compositional manner. We showcase interactive sessions where various primitives are extracted from real-world scenes and controlled to create 3D assets and scenes in a few minutes. We also demonstrate additional capabilities of our primitives: handling various 3D representations to control generation, transferring appearances, and editing geometries.

Title: Adversarial Robustness through Dynamic Ensemble Learning

• Authors: Hetvi Waghela, Jaydip Sen, Sneha Rakshit
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16254
• Pdf URL: https://arxiv.org/pdf/2412.16254
• Copy Paste: [[2412.16254]] Adversarial Robustness through Dynamic Ensemble Learning(https://arxiv.org/abs/2412.16254)
• Keywords: secure, attack, robust
• Abstract: Adversarial attacks pose a significant threat to the reliability of pre-trained language models (PLMs) such as GPT, BERT, RoBERTa, and T5. This paper presents Adversarial Robustness through Dynamic Ensemble Learning (ARDEL), a novel scheme designed to enhance the robustness of PLMs against such attacks. ARDEL leverages the diversity of multiple PLMs and dynamically adjusts the ensemble configuration based on input characteristics and detected adversarial patterns. Key components of ARDEL include a meta-model for dynamic weighting, an adversarial pattern detection module, and adversarial training with regularization techniques. Comprehensive evaluations using standardized datasets and various adversarial attack scenarios demonstrate that ARDEL significantly improves robustness compared to existing methods. By dynamically reconfiguring the ensemble to prioritize the most robust models for each input, ARDEL effectively reduces attack success rates and maintains higher accuracy under adversarial conditions. This work contributes to the broader goal of developing more secure and trustworthy AI systems for real-world NLP applications, offering a practical and scalable solution to enhance adversarial resilience in PLMs.

Title: PromptLA: Towards Integrity Verification of Black-box Text-to-Image Diffusion Models

• Authors: Zhuomeng Zhang, Fangqi Li, Chong Di, Shilin Wang
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16257
• Pdf URL: https://arxiv.org/pdf/2412.16257
• Copy Paste: [[2412.16257]] PromptLA: Towards Integrity Verification of Black-box Text-to-Image Diffusion Models(https://arxiv.org/abs/2412.16257)
• Keywords: protect, diffusion, generative
• Abstract: Current text-to-image (T2I) diffusion models can produce high-quality images, and malicious users who are authorized to use the model only for benign purposes might modify their models to generate images that result in harmful social impacts. Therefore, it is essential to verify the integrity of T2I diffusion models, especially when they are deployed as black-box services. To this end, considering the randomness within the outputs of generative models and the high costs in interacting with them, we capture modifications to the model through the differences in the distributions of the features of generated images. We propose a novel prompt selection algorithm based on learning automaton for efficient and accurate integrity verification of T2I diffusion models. Extensive experiments demonstrate the effectiveness, stability, accuracy and generalization of our algorithm against existing integrity violations compared with baselines. To the best of our knowledge, this paper is the first work addressing the integrity verification of T2I diffusion models, which paves the way to copyright discussions and protections for artificial intelligence applications in practice.

Title: Inference Scaling vs Reasoning: An Empirical Analysis of Compute-Optimal LLM Problem-Solving

• Authors: Marwan AbdElhameed, Pavly Halim
• Subjects: cs.LG, cs.CC, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16260
• Pdf URL: https://arxiv.org/pdf/2412.16260
• Copy Paste: [[2412.16260]] Inference Scaling vs Reasoning: An Empirical Analysis of Compute-Optimal LLM Problem-Solving(https://arxiv.org/abs/2412.16260)
• Keywords: large language model
• Abstract: Recent advances in large language models (LLMs) have predominantly focused on maximizing accuracy and reasoning capabilities, often overlooking crucial computational efficiency considerations. While this approach has yielded impressive accuracy improvements, it has led to methods that may be impractical for real-world deployment due to computational overhead and latency constraints. This paper investigates the potential synergy between reasoning enhancement and computational efficiency by analyzing the integration of two contrasting approaches: Quiet-STaR (Self-Taught Reasoner) and REBASE (REward BAlanced SEarch). Through comprehensive empirical analysis using the Mistral-7B model on the GSM8K dataset, we demonstrate that while each method excels in its primary objective-Quiet-STaR achieving superior accuracy (32.03%) despite high computational cost (554.66s runtime, 12.73T FLOPs), and REBASE providing exceptional efficiency (8.47s runtime, 2.35T FLOPs) while maintaining baseline-comparable accuracy (10.94%)-their integration reveals fundamental challenges in reconciling reasoning depth with computational efficiency. The combined approach unexpectedly results in degraded performance (9.38% accuracy, 143.66s runtime), highlighting critical insights about the complex interplay between reasoning enhancement and efficiency optimization in LLMs. Our findings illuminate the need for novel architectures and algorithms specifically designed to bridge the gap between these competing objectives, while providing concrete directions for future research in compute-efficient reasoning methods.

Title: HybGRAG: Hybrid Retrieval-Augmented Generation on Textual and Relational Knowledge Bases

• Authors: Meng-Chieh Lee, Qi Zhu, Costas Mavromatis, Zhen Han, Soji Adeshina, Vassilis N. Ioannidis, Huzefa Rangwala, Christos Faloutsos
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.16311
• Pdf URL: https://arxiv.org/pdf/2412.16311
• Copy Paste: [[2412.16311]] HybGRAG: Hybrid Retrieval-Augmented Generation on Textual and Relational Knowledge Bases(https://arxiv.org/abs/2412.16311)
• Keywords: large language model
• Abstract: Given a semi-structured knowledge base (SKB), where text documents are interconnected by relations, how can we effectively retrieve relevant information to answer user questions? Retrieval-Augmented Generation (RAG) retrieves documents to assist large language models (LLMs) in question answering; while Graph RAG (GRAG) uses structured knowledge bases as its knowledge source. However, many questions require both textual and relational information from SKB - referred to as "hybrid" questions - which complicates the retrieval process and underscores the need for a hybrid retrieval method that leverages both information. In this paper, through our empirical analysis, we identify key insights that show why existing methods may struggle with hybrid question answering (HQA) over SKB. Based on these insights, we propose HybGRAG for HQA consisting of a retriever bank and a critic module, with the following advantages: (1) Agentic, it automatically refines the output by incorporating feedback from the critic module, (2) Adaptive, it solves hybrid questions requiring both textual and relational information with the retriever bank, (3) Interpretable, it justifies decision making with intuitive refinement path, and (4) Effective, it surpasses all baselines on HQA benchmarks. In experiments on the STaRK benchmark, HybGRAG achieves significant performance gains, with an average relative improvement in Hit@1 of 51%.

Title: Principal-Agent Bandit Games with Self-Interested and Exploratory Learning Agents

• Authors: Junyan Liu, Lillian J. Ratliff
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.16318
• Pdf URL: https://arxiv.org/pdf/2412.16318
• Copy Paste: [[2412.16318]] Principal-Agent Bandit Games with Self-Interested and Exploratory Learning Agents(https://arxiv.org/abs/2412.16318)
• Keywords: robust
• Abstract: We study the repeated principal-agent bandit game, where the principal indirectly interacts with the unknown environment by proposing incentives for the agent to play arms. Most existing work assumes the agent has full knowledge of the reward means and always behaves greedily, but in many online marketplaces, the agent needs to learn the unknown environment and sometimes explore. Motivated by such settings, we model a self-interested learning agent with exploration behaviors who iteratively updates reward estimates and either selects an arm that maximizes the estimated reward plus incentive or explores arbitrarily with a certain probability. As a warm-up, we first consider a self-interested learning agent without exploration. We propose algorithms for both i.i.d. and linear reward settings with bandit feedback in a finite horizon $T$, achieving regret bounds of $\widetilde{O}(\sqrt{T})$ and $\widetilde{O}( T^{2/3} )$, respectively. Specifically, these algorithms are established upon a novel elimination framework coupled with newly-developed search algorithms which accommodate the uncertainty arising from the learning behavior of the agent. We then extend the framework to handle the exploratory learning agent and develop an algorithm to achieve a $\widetilde{O}(T^{2/3})$ regret bound in i.i.d. reward setup by enhancing the robustness of our elimination framework to the potential agent exploration. Finally, when reducing our agent behaviors to the one studied in (Dogan et al., 2023a), we propose an algorithm based on our robust framework, which achieves a $\widetilde{O}(\sqrt{T})$ regret bound, significantly improving upon their $\widetilde{O}(T^{11/12})$ bound.

Title: When Worse is Better: Navigating the compression-generation tradeoff in visual tokenization

• Authors: Vivek Ramanujan, Kushal Tirumala, Armen Aghajanyan, Luke Zettlemoyer, Ali Farhadi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16326
• Pdf URL: https://arxiv.org/pdf/2412.16326
• Copy Paste: [[2412.16326]] When Worse is Better: Navigating the compression-generation tradeoff in visual tokenization(https://arxiv.org/abs/2412.16326)
• Keywords: diffusion, generative
• Abstract: Current image generation methods, such as latent diffusion and discrete token-based generation, depend on a two-stage training approach. In stage 1, an auto-encoder is trained to compress an image into a latent space; in stage 2, a generative model is trained to learn a distribution over that latent space. Most work focuses on maximizing stage 1 performance independent of stage 2, assuming better reconstruction always leads to better generation. However, we show this is not strictly true. Smaller stage 2 models can benefit from more compressed stage 1 latents even if reconstruction performance worsens, showing a fundamental trade-off between compression and generation modeling capacity. To better optimize this trade-off, we introduce Causally Regularized Tokenization (CRT), which uses knowledge of the stage 2 generation modeling procedure to embed useful inductive biases in stage 1 latents. This regularization makes stage 1 reconstruction performance worse, but makes stage 2 generation performance better by making the tokens easier to model: we are able to improve compute efficiency 2-3$\times$ over baseline and match state-of-the-art discrete autoregressive ImageNet generation (2.18 FID) with less than half the tokens per image (256 vs. 576) and a fourth the total model parameters (775M vs. 3.1B) as the previous SOTA (LlamaGen).

Title: Optimizing Fintech Marketing: A Comparative Study of Logistic Regression and XGBoost

• Authors: Sahar Yarmohammadtoosky Dinesh Chowdary Attota
• Subjects: cs.LG, cs.AI, q-fin.ST, stat.AP
• Abstract URL: https://arxiv.org/abs/2412.16333
• Pdf URL: https://arxiv.org/pdf/2412.16333
• Copy Paste: [[2412.16333]] Optimizing Fintech Marketing: A Comparative Study of Logistic Regression and XGBoost(https://arxiv.org/abs/2412.16333)
• Keywords: robust
• Abstract: As several studies have shown, predicting credit risk is still a major concern for the financial services industry and is receiving a lot of scholarly interest. This area of study is crucial because it aids financial organizations in determining the probability that borrowers would default, which has a direct bearing on lending choices and risk management tactics. Despite the progress made in this domain, there is still a substantial knowledge gap concerning consumer actions that take place prior to the filing of credit card applications. The objective of this study is to predict customer responses to mail campaigns and assess the likelihood of default among those who engage. This research employs advanced machine learning techniques, specifically logistic regression and XGBoost, to analyze consumer behavior and predict responses to direct mail campaigns. By integrating different data preprocessing strategies, including imputation and binning, we enhance the robustness and accuracy of our predictive models. The results indicate that XGBoost consistently outperforms logistic regression across various metrics, particularly in scenarios using categorical binning and custom imputation. These findings suggest that XGBoost is particularly effective in handling complex data structures and provides a strong predictive capability in assessing credit risk.

Title: DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment

• Authors: Cijo Jose, Théo Moutakanni, Dahyun Kang, Federico Baldassarre, Timothée Darcet, Hu Xu, Daniel Li, Marc Szafraniec, Michaël Ramamonjisoa, Maxime Oquab, Oriane Siméoni, Huy V. Vo, Patrick Labatut, Piotr Bojanowski
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16334
• Pdf URL: https://arxiv.org/pdf/2412.16334
• Copy Paste: [[2412.16334]] DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment(https://arxiv.org/abs/2412.16334)
• Keywords: segmentation
• Abstract: Self-supervised visual foundation models produce powerful embeddings that achieve remarkable performance on a wide range of downstream tasks. However, unlike vision-language models such as CLIP, self-supervised visual features are not readily aligned with language, hindering their adoption in open-vocabulary tasks. Our method, named this http URL, unlocks this new ability for DINOv2, a widely used self-supervised visual encoder. We build upon the LiT training strategy, which trains a text encoder to align with a frozen vision model but leads to unsatisfactory results on dense tasks. We propose several key ingredients to improve performance on both global and dense tasks, such as concatenating the [CLS] token with the patch average to train the alignment and curating data using both text and image modalities. With these, we successfully train a CLIP-like model with only a fraction of the computational cost compared to CLIP while achieving state-of-the-art results in zero-shot classification and open-vocabulary semantic segmentation.

Title: Improving Equity in Health Modeling with GPT4-Turbo Generated Synthetic Data: A Comparative Study

• Authors: Daniel Smolyak, Arshana Welivita, Margrét V. Bjarnadóttir, Ritu Agarwal
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.16335
• Pdf URL: https://arxiv.org/pdf/2412.16335
• Copy Paste: [[2412.16335]] Improving Equity in Health Modeling with GPT4-Turbo Generated Synthetic Data: A Comparative Study(https://arxiv.org/abs/2412.16335)
• Keywords: fair
• Abstract: Objective. Demographic groups are often represented at different rates in medical datasets. These differences can create bias in machine learning algorithms, with higher levels of performance for better-represented groups. One promising solution to this problem is to generate synthetic data to mitigate potential adverse effects of non-representative data sets. Methods. We build on recent advances in LLM-based synthetic data generation to create a pipeline where the synthetic data is generated separately for each demographic group. We conduct our study using MIMIC-IV and Framingham "Offspring and OMNI-1 Cohorts" datasets. We prompt GPT4-Turbo to create group-specific data, providing training examples and the dataset context. An exploratory analysis is conducted to ascertain the quality of the generated data. We then evaluate the utility of the synthetic data for augmentation of a training dataset in a downstream machine learning task, focusing specifically on model performance metrics across groups. Results. The performance of GPT4-Turbo augmentation is generally superior but not always. In the majority of experiments our method outperforms standard modeling baselines, however, prompting GPT-4-Turbo to produce data specific to a group provides little to no additional benefit over a prompt that does not specify the group. Conclusion. We developed a method for using LLMs out-of-the-box to synthesize group-specific data to address imbalances in demographic representation in medical datasets. As another "tool in the toolbox", this method can improve model fairness and thus health equity. More research is needed to understand the conditions under which LLM generated synthetic data is useful for non-representative medical data sets.

Title: Deliberative Alignment: Reasoning Enables Safer Language Models

• Authors: Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Heylar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, Amelia Glaese
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16339
• Pdf URL: https://arxiv.org/pdf/2412.16339
• Copy Paste: [[2412.16339]] Deliberative Alignment: Reasoning Enables Safer Language Models(https://arxiv.org/abs/2412.16339)
• Keywords: robust
• Abstract: As large-scale language models increasingly impact safety-critical domains, ensuring their reliable adherence to well-defined principles remains a fundamental challenge. We introduce Deliberative Alignment, a new paradigm that directly teaches the model safety specifications and trains it to explicitly recall and accurately reason over the specifications before answering. We used this approach to align OpenAI's o-series models, and achieved highly precise adherence to OpenAI's safety policies, without requiring human-written chain-of-thoughts or answers. Deliberative Alignment pushes the Pareto frontier by simultaneously increasing robustness to jailbreaks while decreasing overrefusal rates, and also improves out-of-distribution generalization. We demonstrate that reasoning over explicitly specified policies enables more scalable, trustworthy, and interpretable alignment.

Title: A Machine Learning Approach for Emergency Detection in Medical Scenarios Using Large Language Models

• Authors: Ferit Akaybicen, Aaron Cummings, Lota Iwuagwu, Xinyue Zhang, Modupe Adewuyi
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16341
• Pdf URL: https://arxiv.org/pdf/2412.16341
• Copy Paste: [[2412.16341]] A Machine Learning Approach for Emergency Detection in Medical Scenarios Using Large Language Models(https://arxiv.org/abs/2412.16341)
• Keywords: large language model
• Abstract: The rapid identification of medical emergencies through digital communication channels remains a critical challenge in modern healthcare delivery, particularly with the increasing prevalence of telemedicine. This paper presents a novel approach leveraging large language models (LLMs) and prompt engineering techniques for automated emergency detection in medical communications. We developed and evaluated a comprehensive system using multiple LLaMA model variants (1B, 3B, and 7B parameters) to classify medical scenarios as emergency or non-emergency situations. Our methodology incorporated both system prompts and in-prompt training approaches, evaluated across different hardware configurations. The results demonstrate exceptional performance, with the LLaMA 2 (7B) model achieving 99.7% accuracy and the LLaMA 3.2 (3B) model reaching 99.6% accuracy with optimal prompt engineering. Through systematic testing of training examples within the prompts, we identified that including 10 example scenarios in the model prompts yielded optimal classification performance. Processing speeds varied significantly between platforms, ranging from 0.05 to 2.2 seconds per request. The system showed particular strength in minimizing high-risk false negatives in emergency scenarios, which is crucial for patient safety. The code implementation and evaluation framework are publicly available on GitHub, facilitating further research and development in this crucial area of healthcare technology.

Title: Do we still need canaries in the coal mine? Measuring shadow stack effectiveness in countering stack smashing

• Authors: Hugo Depuydt, Merve Gülmez, Thomas Nyman, Jan Tobias Mühlberg
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16343
• Pdf URL: https://arxiv.org/pdf/2412.16343
• Copy Paste: [[2412.16343]] Do we still need canaries in the coal mine? Measuring shadow stack effectiveness in countering stack smashing(https://arxiv.org/abs/2412.16343)
• Keywords: protect
• Abstract: Stack canaries and shadow stacks are widely deployed mitigations to memory-safety vulnerabilities. While stack canaries are introduced by the compiler and rely on sentry values placed between variables and control data, shadow stack implementations protect return addresses explicitly and rely on hardware features available in modern processor designs for efficiency. In this paper we hypothesize that stack canaries and shadow stacks provide similar levels of protections against sequential stack-based overflows. Based on the Juliet test suite, we evaluate whether 64-bit x86 (x86-64) systems benefit from enabling stack canaries in addition to the x86-64 shadow stack enforcement. We observe divergence in overflow detection rates between the GCC and Clang compilers and across optimization levels, which we attribute to differences in stack layouts generated by the compilers. We also find that x86-64 shadow stack implementations are more effective and outperform stack canaries when combined with a stack-protector-like stack layout. We implement and evaluate an enhancement to the Clang x86-64 shadow stack instrumentation that improves the shadow stack detection accuracy based on this observation.

Title: Texture- and Shape-based Adversarial Attacks for Vehicle Detection in Synthetic Overhead Imagery

• Authors: Mikael Yeghiazaryan, Sai Abhishek Siddhartha Namburu, Emily Kim, Stanislav Panev, Celso de Melo, Brent Lance, Fernando De la Torre, Jessica K. Hodgins
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16358
• Pdf URL: https://arxiv.org/pdf/2412.16358
• Copy Paste: [[2412.16358]] Texture- and Shape-based Adversarial Attacks for Vehicle Detection in Synthetic Overhead Imagery(https://arxiv.org/abs/2412.16358)
• Keywords: attack
• Abstract: Detecting vehicles in aerial images can be very challenging due to complex backgrounds, small resolution, shadows, and occlusions. Despite the effectiveness of SOTA detectors such as YOLO, they remain vulnerable to adversarial attacks (AAs), compromising their reliability. Traditional AA strategies often overlook the practical constraints of physical implementation, focusing solely on attack performance. Our work addresses this issue by proposing practical implementation constraints for AA in texture and/or shape. These constraints include pixelation, masking, limiting the color palette of the textures, and constraining the shape modifications. We evaluated the proposed constraints through extensive experiments using three widely used object detector architectures, and compared them to previous works. The results demonstrate the effectiveness of our solutions and reveal a trade-off between practicality and performance. Additionally, we introduce a labeled dataset of overhead images featuring vehicles of various categories. We will make the code/dataset public upon paper acceptance.

Title: Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Context

• Authors: Nilanjana Das, Edward Raff, Manas Gaur
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16359
• Pdf URL: https://arxiv.org/pdf/2412.16359
• Copy Paste: [[2412.16359]] Human-Readable Adversarial Prompts: An Investigation into LLM Vulnerabilities Using Situational Context(https://arxiv.org/abs/2412.16359)
• Keywords: attack, robust
• Abstract: Previous research on LLM vulnerabilities often relied on nonsensical adversarial prompts, which were easily detectable by automated methods. We address this gap by focusing on human-readable adversarial prompts, a more realistic and potent threat. Our key contributions are situation-driven attacks leveraging movie scripts to create contextually relevant, human-readable prompts that successfully deceive LLMs, adversarial suffix conversion to transform nonsensical adversarial suffixes into meaningful text, and AdvPrompter with p-nucleus sampling, a method to generate diverse, human-readable adversarial suffixes, improving attack efficacy in models like GPT-3.5 and Gemma 7B. Our findings demonstrate that LLMs can be tricked by sophisticated adversaries into producing harmful responses with human-readable adversarial prompts and that there exists a scope for improvement when it comes to robust LLMs.

Title: Toward Robust Neural Reconstruction from Sparse Point Sets

• Authors: Amine Ouasfi, Shubhendu Jena, Eric Marchand, Adnane Boukhayma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16361
• Pdf URL: https://arxiv.org/pdf/2412.16361
• Copy Paste: [[2412.16361]] Toward Robust Neural Reconstruction from Sparse Point Sets(https://arxiv.org/abs/2412.16361)
• Keywords: robust
• Abstract: We consider the challenging problem of learning Signed Distance Functions (SDF) from sparse and noisy 3D point clouds. In contrast to recent methods that depend on smoothness priors, our method, rooted in a distributionally robust optimization (DRO) framework, incorporates a regularization term that leverages samples from the uncertainty regions of the model to improve the learned SDFs. Thanks to tractable dual formulations, we show that this framework enables a stable and efficient optimization of SDFs in the absence of ground truth supervision. Using a variety of synthetic and real data evaluations from different modalities, we show that our DRO based learning framework can improve SDF learning with respect to baselines and the state-of-the-art methods.

Title: A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation

• Authors: Shijie Zhou, Ruiyi Zhang, Yufan Zhou, Changyou Chen
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16364
• Pdf URL: https://arxiv.org/pdf/2412.16364
• Copy Paste: [[2412.16364]] A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation(https://arxiv.org/abs/2412.16364)
• Keywords: large language model
• Abstract: Large multimodal models still struggle with text-rich images because of inadequate training data. Self-Instruct provides an annotation-free way for generating instruction data, but its quality is poor, as multimodal alignment remains a hurdle even for the largest models. In this work, we propose LLaVAR-2, to enhance multimodal alignment for text-rich images through hybrid instruction generation between human annotators and large language models. Specifically, it involves detailed image captions from human annotators, followed by the use of these annotations in tailored text prompts for GPT-4o to curate a dataset. It also implements several mechanisms to filter out low-quality data, and the resulting dataset comprises 424k high-quality pairs of instructions. Empirical results show that models fine-tuned on this dataset exhibit impressive enhancements over those trained with self-instruct data.

Title: FairREAD: Re-fusing Demographic Attributes after Disentanglement for Fair Medical Image Classification

• Authors: Yicheng Gao, Jinkui Hao, Bo Zhou
• Subjects: cs.CV, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.16373
• Pdf URL: https://arxiv.org/pdf/2412.16373
• Copy Paste: [[2412.16373]] FairREAD: Re-fusing Demographic Attributes after Disentanglement for Fair Medical Image Classification(https://arxiv.org/abs/2412.16373)
• Keywords: fair
• Abstract: Recent advancements in deep learning have shown transformative potential in medical imaging, yet concerns about fairness persist due to performance disparities across demographic subgroups. Existing methods aim to address these biases by mitigating sensitive attributes in image data; however, these attributes often carry clinically relevant information, and their removal can compromise model performance-a highly undesirable outcome. To address this challenge, we propose Fair Re-fusion After Disentanglement (FairREAD), a novel, simple, and efficient framework that mitigates unfairness by re-integrating sensitive demographic attributes into fair image representations. FairREAD employs orthogonality constraints and adversarial training to disentangle demographic information while using a controlled re-fusion mechanism to preserve clinically relevant details. Additionally, subgroup-specific threshold adjustments ensure equitable performance across demographic groups. Comprehensive evaluations on a large-scale clinical X-ray dataset demonstrate that FairREAD significantly reduces unfairness metrics while maintaining diagnostic accuracy, establishing a new benchmark for fairness and performance in medical image classification.

Title: Iterative Encoding-Decoding VAEs Anomaly Detection in NOAA's DART Time Series: A Machine Learning Approach for Enhancing Data Integrity for NASA's GRACE-FO Verification and Validation

• Authors: Kevin Lee
• Subjects: cs.LG, cs.AI, physics.geo-ph
• Abstract URL: https://arxiv.org/abs/2412.16375
• Pdf URL: https://arxiv.org/pdf/2412.16375
• Copy Paste: [[2412.16375]] Iterative Encoding-Decoding VAEs Anomaly Detection in NOAA's DART Time Series: A Machine Learning Approach for Enhancing Data Integrity for NASA's GRACE-FO Verification and Validation(https://arxiv.org/abs/2412.16375)
• Keywords: robust, interpretability
• Abstract: NOAA's Deep-ocean Assessment and Reporting of Tsunamis (DART) data are critical for NASA-JPL's tsunami detection, real-time operations, and oceanographic research. However, these time-series data often contain spikes, steps, and drifts that degrade data quality and obscure essential oceanographic features. To address these anomalies, the work introduces an Iterative Encoding-Decoding Variational Autoencoders (Iterative Encoding-Decoding VAEs) model to improve the quality of DART time series. Unlike traditional filtering and thresholding methods that risk distorting inherent signal characteristics, Iterative Encoding-Decoding VAEs progressively remove anomalies while preserving the data's latent structure. A hybrid thresholding approach further retains genuine oceanographic features near boundaries. Applied to complex DART datasets, this approach yields reconstructions that better maintain key oceanic properties compared to classical statistical techniques, offering improved robustness against spike removal and subtle step changes. The resulting high-quality data supports critical verification and validation efforts for the GRACE-FO mission at NASA-JPL, where accurate surface measurements are essential to modeling Earth's gravitational field and global water dynamics. Ultimately, this data processing method enhances tsunami detection and underpins future climate modeling with improved interpretability and reliability.

Title: VerSe: Integrating Multiple Queries as Prompts for Versatile Cardiac MRI Segmentation

• Authors: Bangwei Guo, Meng Ye, Yunhe Gao, Bingyu Xin, Leon Axel, Dimitris Metaxas
• Subjects: cs.CV, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.16381
• Pdf URL: https://arxiv.org/pdf/2412.16381
• Copy Paste: [[2412.16381]] VerSe: Integrating Multiple Queries as Prompts for Versatile Cardiac MRI Segmentation(https://arxiv.org/abs/2412.16381)
• Keywords: segmentation
• Abstract: Despite the advances in learning-based image segmentation approach, the accurate segmentation of cardiac structures from magnetic resonance imaging (MRI) remains a critical challenge. While existing automatic segmentation methods have shown promise, they still require extensive manual corrections of the segmentation results by human experts, particularly in complex regions such as the basal and apical parts of the heart. Recent efforts have been made on developing interactive image segmentation methods that enable human-in-the-loop learning. However, they are semi-automatic and inefficient, due to their reliance on click-based prompts, especially for 3D cardiac MRI volumes. To address these limitations, we propose VerSe, a Versatile Segmentation framework to unify automatic and interactive segmentation through mutiple queries. Our key innovation lies in the joint learning of object and click queries as prompts for a shared segmentation backbone. VerSe supports both fully automatic segmentation, through object queries, and interactive mask refinement, by providing click queries when needed. With the proposed integrated prompting scheme, VerSe demonstrates significant improvement in performance and efficiency over existing methods, on both cardiac MRI and out-of-distribution medical imaging datasets. The code is available at this https URL.

Title: Application of Multimodal Large Language Models in Autonomous Driving

• Authors: Md Robiul Islam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16410
• Pdf URL: https://arxiv.org/pdf/2412.16410
• Copy Paste: [[2412.16410]] Application of Multimodal Large Language Models in Autonomous Driving(https://arxiv.org/abs/2412.16410)
• Keywords: large language model
• Abstract: In this era of technological advancements, several cutting-edge techniques are being implemented to enhance Autonomous Driving (AD) systems, focusing on improving safety, efficiency, and adaptability in complex driving environments. However, AD still faces some problems including performance limitations. To address this problem, we conducted an in-depth study on implementing the Multi-modal Large Language Model. We constructed a Virtual Question Answering (VQA) dataset to fine-tune the model and address problems with the poor performance of MLLM on AD. We then break down the AD decision-making process by scene understanding, prediction, and decision-making. Chain of Thought has been used to make the decision more perfectly. Our experiments and detailed analysis of Autonomous Driving give an idea of how important MLLM is for AD.

Title: InfoTech Assistant : A Multimodal Conversational Agent for InfoTechnology Web Portal Queries

• Authors: Sai Surya Gadiraju, Duoduo Liao, Akhila Kudupudi, Santosh Kasula, Charitha Chalasani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16412
• Pdf URL: https://arxiv.org/pdf/2412.16412
• Copy Paste: [[2412.16412]] InfoTech Assistant : A Multimodal Conversational Agent for InfoTechnology Web Portal Queries(https://arxiv.org/abs/2412.16412)
• Keywords: large language model
• Abstract: This pilot study presents the development of the InfoTech Assistant, a domain-specific, multimodal chatbot engineered to address queries in bridge evaluation and infrastructure technology. By integrating web data scraping, large language models (LLMs), and Retrieval-Augmented Generation (RAG), the InfoTech Assistant provides accurate and contextually relevant responses. Data, including textual descriptions and images, are sourced from publicly available documents on the InfoTechnology website and organized in JSON format to facilitate efficient querying. The architecture of the system includes an HTML-based interface and a Flask back end connected to the Llama 3.1 model via LLM Studio. Evaluation results show approximately 95 percent accuracy on domain-specific tasks, with high similarity scores confirming the quality of response matching. This RAG-enhanced setup enables the InfoTech Assistant to handle complex, multimodal queries, offering both textual and visual information in its responses. The InfoTech Assistant demonstrates strong potential as a dependable tool for infrastructure professionals, delivering high accuracy and relevance in its domain-specific outputs.

Title: Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities

• Authors: Huan Liu, Lingyu Xiao, Jiangjiang Liu, Xiaofan Li, Ze Feng, Sen Yang, Jingdong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16418
• Pdf URL: https://arxiv.org/pdf/2412.16418
• Copy Paste: [[2412.16418]] Revisiting MLLMs: An In-Depth Analysis of Image Classification Abilities(https://arxiv.org/abs/2412.16418)
• Keywords: large language model
• Abstract: With the rapid advancement of Multimodal Large Language Models (MLLMs), a variety of benchmarks have been introduced to evaluate their capabilities. While most evaluations have focused on complex tasks such as scientific comprehension and visual reasoning, little attention has been given to assessing their fundamental image classification abilities. In this paper, we address this gap by thoroughly revisiting the MLLMs with an in-depth analysis of image classification. Specifically, building on established datasets, we examine a broad spectrum of scenarios, from general classification tasks (e.g., ImageNet, ObjectNet) to more fine-grained categories such as bird and food classification. Our findings reveal that the most recent MLLMs can match or even outperform CLIP-style vision-language models on several datasets, challenging the previous assumption that MLLMs are bad at image classification \cite{VLMClassifier}. To understand the factors driving this improvement, we conduct an in-depth analysis of the network architecture, data selection, and training recipe used in public MLLMs. Our results attribute this success to advancements in language models and the diversity of training data sources. Based on these observations, we further analyze and attribute the potential reasons to conceptual knowledge transfer and enhanced exposure of target concepts, respectively. We hope our findings will offer valuable insights for future research on MLLMs and their evaluation in image classification tasks.

Title: Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding

• Authors: Junyi Ye, Ankan Dash, Wenpeng Yin, Guiling Wang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16420
• Pdf URL: https://arxiv.org/pdf/2412.16420
• Copy Paste: [[2412.16420]] Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding(https://arxiv.org/abs/2412.16420)
• Keywords: robust, explainability
• Abstract: Flowcharts are typically presented as images, driving the trend of using vision-language models (VLMs) for end-to-end flowchart understanding. However, two key challenges arise: (i) Limited controllability--users have minimal influence over the downstream task, as they can only modify input images, while the training of VLMs is often out of reach for most researchers. (ii) Lack of explainability--it is difficult to trace VLM errors to specific causes, such as failures in visual encoding or reasoning. We propose TextFlow, addressing aforementioned issues with two stages: (i) Vision Textualizer--which generates textual representations from flowchart images; and (ii) Textual Reasoner--which performs question-answering based on the text representations. TextFlow offers three key advantages: (i) users can select the type of text representations (e.g., Graphviz, Mermaid, PlantUML), or further convert them into executable graph object to call tools, enhancing performance and controllability; (ii) it improves explainability by helping to attribute errors more clearly to visual or textual processing components; and (iii) it promotes the modularization of the solution, such as allowing advanced LLMs to be used in the Reasoner stage when VLMs underperform in end-to-end fashion. Experiments on the FlowVQA and FlowLearn benchmarks demonstrate TextFlow's state-of-the-art performance as well as its robustness. All code is publicly available.

Title: Technical Report: Small Language Model for Japanese Clinical and Medicine

• Authors: Shogo Watanabe
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16423
• Pdf URL: https://arxiv.org/pdf/2412.16423
• Copy Paste: [[2412.16423]] Technical Report: Small Language Model for Japanese Clinical and Medicine(https://arxiv.org/abs/2412.16423)
• Keywords: large language model
• Abstract: This report presents a small language model (SLM) for Japanese clinical and medicine, named NCVC-slm-1. This 1B parameters model was trained using Japanese text classified to be of high-quality. Moreover, NCVC-slm-1 was augmented with respect to clinical and medicine content that includes the variety of diseases, drugs, and examinations. Using a carefully designed pre-processing, a specialized morphological analyzer and tokenizer, this small and light-weight model performed not only to generate text but also indicated the feasibility of understanding clinical and medicine text. In comparison to other large language models, a fine-tuning NCVC-slm-1 demonstrated the highest scores on 6 tasks of total 8 on JMED-LLM. According to this result, SLM indicated the feasibility of performing several downstream tasks in the field of clinical and medicine. Hopefully, NCVC-slm-1 will be contributed to develop and accelerate the field of clinical and medicine for a bright future.

Title: Deepfake detection, image manipulation detection, fairness, generalization

• Authors: Uzoamaka Ezeakunne, Chrisantus Eze, Xiuwen Liu
• Subjects: cs.LG, cs.AI, cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.16428
• Pdf URL: https://arxiv.org/pdf/2412.16428
• Copy Paste: [[2412.16428]] Deepfake detection, image manipulation detection, fairness, generalization(https://arxiv.org/abs/2412.16428)
• Keywords: robust, fair
• Abstract: Despite the progress made in deepfake detection research, recent studies have shown that biases in the training data for these detectors can result in varying levels of performance across different demographic groups, such as race and gender. These disparities can lead to certain groups being unfairly targeted or excluded. Traditional methods often rely on fair loss functions to address these issues, but they under-perform when applied to unseen datasets, hence, fairness generalization remains a challenge. In this work, we propose a data-driven framework for tackling the fairness generalization problem in deepfake detection by leveraging synthetic datasets and model optimization. Our approach focuses on generating and utilizing synthetic data to enhance fairness across diverse demographic groups. By creating a diverse set of synthetic samples that represent various demographic groups, we ensure that our model is trained on a balanced and representative dataset. This approach allows us to generalize fairness more effectively across different domains. We employ a comprehensive strategy that leverages synthetic data, a loss sharpness-aware optimization pipeline, and a multi-task learning framework to create a more equitable training environment, which helps maintain fairness across both intra-dataset and cross-dataset evaluations. Extensive experiments on benchmark deepfake detection datasets demonstrate the efficacy of our approach, surpassing state-of-the-art approaches in preserving fairness during cross-dataset evaluation. Our results highlight the potential of synthetic datasets in achieving fairness generalization, providing a robust solution for the challenges faced in deepfake detection.

Title: WiP: Deception-in-Depth Using Multiple Layers of Deception

• Authors: Jason Landsborough, Neil C. Rowe, Thuy D. Nguyen, Sunny Fugate
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16430
• Pdf URL: https://arxiv.org/pdf/2412.16430
• Copy Paste: [[2412.16430]] WiP: Deception-in-Depth Using Multiple Layers of Deception(https://arxiv.org/abs/2412.16430)
• Keywords: protect, defense, attack
• Abstract: Deception is being increasingly explored as a cyberdefense strategy to protect operational systems. We are studying implementation of deception-in-depth strategies with initially three logical layers: network, host, and data. We draw ideas from military deception, network orchestration, software deception, file deception, fake honeypots, and moving-target defenses. We are building a prototype representing our ideas and will be testing it in several adversarial environments. We hope to show that deploying a broad range of deception techniques can be more effective in protecting systems than deploying single techniques. Unlike traditional deception methods that try to encourage active engagement from attackers to collect intelligence, we focus on deceptions that can be used on real machines to discourage attacks.

Title: Object Detection Approaches to Identifying Hand Images with High Forensic Values

• Authors: Thanh Thi Nguyen, Campbell Wilson, Imad Khan, Janis Dalins
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16431
• Pdf URL: https://arxiv.org/pdf/2412.16431
• Copy Paste: [[2412.16431]] Object Detection Approaches to Identifying Hand Images with High Forensic Values(https://arxiv.org/abs/2412.16431)
• Keywords: transformer
• Abstract: Forensic science plays a crucial role in legal investigations, and the use of advanced technologies, such as object detection based on machine learning methods, can enhance the efficiency and accuracy of forensic analysis. Human hands are unique and can leave distinct patterns, marks, or prints that can be utilized for forensic examinations. This paper compares various machine learning approaches to hand detection and presents the application results of employing the best-performing model to identify images of significant importance in forensic contexts. We fine-tune YOLOv8 and vision transformer-based object detection models on four hand image datasets, including the 11k hands dataset with our own bounding boxes annotated by a semi-automatic approach. Two YOLOv8 variants, i.e., YOLOv8 nano (YOLOv8n) and YOLOv8 extra-large (YOLOv8x), and two vision transformer variants, i.e., DEtection TRansformer (DETR) and Detection Transformers with Assignment (DETA), are employed for the experiments. Experimental results demonstrate that the YOLOv8 models outperform DETR and DETA on all datasets. The experiments also show that YOLOv8 approaches result in superior performance compared with existing hand detection methods, which were based on YOLOv3 and YOLOv4 models. Applications of our fine-tuned YOLOv8 models for identifying hand images (or frames in a video) with high forensic values produce excellent results, significantly reducing the time required by forensic experts. This implies that our approaches can be implemented effectively for real-world applications in forensics or related fields.

Title: Has LLM Reached the Scaling Ceiling Yet? Unified Insights into LLM Regularities and Constraints

• Authors: Charles Luo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16443
• Pdf URL: https://arxiv.org/pdf/2412.16443
• Copy Paste: [[2412.16443]] Has LLM Reached the Scaling Ceiling Yet? Unified Insights into LLM Regularities and Constraints(https://arxiv.org/abs/2412.16443)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their scalability raises a critical question: Have we reached the scaling ceiling? This paper addresses this pivotal question by developing a unified theoretical framework that integrates mathematical and statistical insights to explain the scaling dynamics of LLMs. We present: 1. Central Limit Theorem (CLT) for Hidden Representations: We show that noise in hidden representations scales inversely with context size, explaining stabilization effects and the limits of context length improvements. 2. Bias-Variance Decomposition: We decompose next-token prediction loss into irreducible entropy, capacity-driven bias, and finite sample variance, revealing trade-offs where scaling yields diminishing returns. 3. Emergent SNR Thresholds: By defining signal-to-noise ratio (SNR), we quantify how capabilities emerge abruptly once SNR surpasses a threshold, offering insights into when scaling becomes less effective. Through this framework, we conclude that while LLMs have not reached an absolute scaling ceiling, practical constraints are increasingly prominent: diminishing returns, resource inefficiencies, and data limitations. Future progress will require a shift from brute-force scaling to innovations in architecture, data quality, and training paradigms. This work provides a roadmap for guiding the efficient development of next-generation LLMs and advancing the field beyond traditional scaling strategies. Keywords: Large Language Models; Scaling Ceiling; Central Limit Theorem; Bias-Variance Trade-Off; Signal-to-Noise Ratio; Emergent Capabilities

Title: Sensitive Image Classification by Vision Transformers

• Authors: Hanxian He, Campbell Wilson, Thanh Thi Nguyen, Janis Dalins
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16446
• Pdf URL: https://arxiv.org/pdf/2412.16446
• Copy Paste: [[2412.16446]] Sensitive Image Classification by Vision Transformers(https://arxiv.org/abs/2412.16446)
• Keywords: transformer
• Abstract: When it comes to classifying child sexual abuse images, managing similar inter-class correlations and diverse intra-class correlations poses a significant challenge. Vision transformer models, unlike conventional deep convolutional network models, leverage a self-attention mechanism to capture global interactions among contextual local elements. This allows them to navigate through image patches effectively, avoiding incorrect correlations and reducing ambiguity in attention maps, thus proving their efficacy in computer vision tasks. Rather than directly analyzing child sexual abuse data, we constructed two datasets: one comprising clean and pornographic images and another with three classes, which additionally include images indicative of pornography, sourced from Reddit and Google Open Images data. In our experiments, we also employ an adult content image benchmark dataset. These datasets served as a basis for assessing the performance of vision transformer models in pornographic image classification. In our study, we conducted a comparative analysis between various popular vision transformer models and traditional pre-trained ResNet models. Furthermore, we compared them with established methods for sensitive image detection such as attention and metric learning based CNN and Bumble. The findings demonstrated that vision transformer networks surpassed the benchmark pre-trained models, showcasing their superior classification and detection capabilities in this task.

Title: A Generalizable Anomaly Detection Method in Dynamic Graphs

• Authors: Xiao Yang, Xuejiao Zhao, Zhiqi Shen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16447
• Pdf URL: https://arxiv.org/pdf/2412.16447
• Copy Paste: [[2412.16447]] A Generalizable Anomaly Detection Method in Dynamic Graphs(https://arxiv.org/abs/2412.16447)
• Keywords: security
• Abstract: Anomaly detection aims to identify deviations from normal patterns within data. This task is particularly crucial in dynamic graphs, which are common in applications like social networks and cybersecurity, due to their evolving structures and complex relationships. Although recent deep learning-based methods have shown promising results in anomaly detection on dynamic graphs, they often lack of generalizability. In this study, we propose GeneralDyG, a method that samples temporal ego-graphs and sequentially extracts structural and temporal features to address the three key challenges in achieving generalizability: Data Diversity, Dynamic Feature Capture, and Computational Cost. Extensive experimental results demonstrate that our proposed GeneralDyG significantly outperforms state-of-the-art methods on four real-world datasets.

Title: CBNN: 3-Party Secure Framework for Customized Binary Neural Networks Inference

• Authors: Benchang Dong, Zhili Chen, Xin Chen, Shiwen Wei, Jie Fu, Huifa Li
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16449
• Pdf URL: https://arxiv.org/pdf/2412.16449
• Copy Paste: [[2412.16449]] CBNN: 3-Party Secure Framework for Customized Binary Neural Networks Inference(https://arxiv.org/abs/2412.16449)
• Keywords: secure, security, privacy
• Abstract: Binarized Neural Networks (BNN) offer efficient implementations for machine learning tasks and facilitate Privacy-Preserving Machine Learning (PPML) by simplifying operations with binary values. Nevertheless, challenges persist in terms of communication and accuracy in their application scenarios. In this work, we introduce CBNN, a three-party secure computation framework tailored for efficient BNN inference. Leveraging knowledge distillation and separable convolutions, CBNN transforms standard BNNs into MPC-friendly customized BNNs, maintaining high utility. It performs secure inference using optimized protocols for basic operations. Specifically, CBNN enhances linear operations with replicated secret sharing and MPC-friendly convolutions, while introducing a novel secure activation function to optimize non-linear operations. We demonstrate the effectiveness of CBNN by transforming and securely implementing several typical BNN models. Experimental results indicate that CBNN maintains impressive performance even after customized binarization and security measures

Title: Correcting Large Language Model Behavior via Influence Function

• Authors: Han Zhang, Zhuo Zhang, Yi Zhang, Yuanzhao Zhai, Hanyang Peng, Yu Lei, Yue Yu, Hui Wang, Bin Liang, Lin Gui, Ruifeng Xu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16451
• Pdf URL: https://arxiv.org/pdf/2412.16451
• Copy Paste: [[2412.16451]] Correcting Large Language Model Behavior via Influence Function(https://arxiv.org/abs/2412.16451)
• Keywords: interpretability, large language model
• Abstract: Recent advancements in AI alignment techniques have significantly improved the alignment of large language models (LLMs) with static human preferences. However, the dynamic nature of human preferences can render some prior training data outdated or even erroneous, ultimately causing LLMs to deviate from contemporary human preferences and societal norms. Existing methodologies, whether they involve the curation of new data for continual alignment or the manual correction of outdated data for re-alignment, demand costly human resources. To address this challenge, we propose a novel approach, Large Language Model Behavior Correction with Influence Function Recall and Post-Training (LANCET), which requires no human involvement. LANCET consists of two phases: (1) using influence functions to identify the training data that significantly impact undesirable model outputs, and (2) applying an Influence function-driven Bregman Optimization (IBO) technique to adjust the model's behavior based on these influence distributions. Our experiments demonstrate that LANCET effectively and efficiently correct inappropriate behaviors of LLMs. Furthermore, LANCET can outperform methods that rely on collecting human preferences, and it enhances the interpretability of learning human preferences within LLMs.

Title: FACTS: Fine-Grained Action Classification for Tactical Sports

• Authors: Christopher Lai, Jason Mo, Haotian Xia, Yuan-fang Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16454
• Pdf URL: https://arxiv.org/pdf/2412.16454
• Copy Paste: [[2412.16454]] FACTS: Fine-Grained Action Classification for Tactical Sports(https://arxiv.org/abs/2412.16454)
• Keywords: transformer
• Abstract: Classifying fine-grained actions in fast-paced, close-combat sports such as fencing and boxing presents unique challenges due to the complexity, speed, and nuance of movements. Traditional methods reliant on pose estimation or fancy sensor data often struggle to capture these dynamics accurately. We introduce FACTS, a novel transformer-based approach for fine-grained action recognition that processes raw video data directly, eliminating the need for pose estimation and the use of cumbersome body markers and sensors. FACTS achieves state-of-the-art performance, with 90% accuracy on fencing actions and 83.25% on boxing actions. Additionally, we present a new publicly available dataset featuring 8 detailed fencing actions, addressing critical gaps in sports analytics resources. Our findings enhance training, performance analysis, and spectator engagement, setting a new benchmark for action classification in tactical sports.

Title: Positive2Negative: Breaking the Information-Lossy Barrier in Self-Supervised Single Image Denoising

• Authors: Tong Li, Lizhi Wang, Zhiyuan Xu, Lin Zhu, Wanxuan Lu, Hua Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16460
• Pdf URL: https://arxiv.org/pdf/2412.16460
• Copy Paste: [[2412.16460]] Positive2Negative: Breaking the Information-Lossy Barrier in Self-Supervised Single Image Denoising(https://arxiv.org/abs/2412.16460)
• Keywords: robust
• Abstract: Image denoising enhances image quality, serving as a foundational technique across various computational photography applications. The obstacle to clean image acquisition in real scenarios necessitates the development of self-supervised image denoising methods only depending on noisy images, especially a single noisy image. Existing self-supervised image denoising paradigms (Noise2Noise and Noise2Void) rely heavily on information-lossy operations, such as downsampling and masking, culminating in low quality denoising performance. In this paper, we propose a novel self-supervised single image denoising paradigm, Positive2Negative, to break the information-lossy barrier. Our paradigm involves two key steps: Renoised Data Construction (RDC) and Denoised Consistency Supervision (DCS). RDC renoises the predicted denoised image by the predicted noise to construct multiple noisy images, preserving all the information of the original image. DCS ensures consistency across the multiple denoised images, supervising the network to learn robust denoising. Our Positive2Negative paradigm achieves state-of-the-art performance in self-supervised single image denoising with significant speed improvements. The code will be released to the public.

Title: Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognition

• Authors: Keqi Deng, Jinxi Guo, Yingyi Ma, Niko Moritz, Philip C. Woodland, Ozlem Kalinli, Mike Seltzer
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.16464
• Pdf URL: https://arxiv.org/pdf/2412.16464
• Copy Paste: [[2412.16464]] Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognition(https://arxiv.org/abs/2412.16464)
• Keywords: large language model
• Abstract: While large language models (LLMs) have been applied to automatic speech recognition (ASR), the task of making the model streamable remains a challenge. This paper proposes a novel model architecture, Transducer-Llama, that integrates LLMs into a Factorized Transducer (FT) model, naturally enabling streaming capabilities. Furthermore, given that the large vocabulary of LLMs can cause data sparsity issue and increased training costs for spoken language systems, this paper introduces an efficient vocabulary adaptation technique to align LLMs with speech system vocabularies. The results show that directly optimizing the FT model with a strong pre-trained LLM-based predictor using the RNN-T loss yields some but limited improvements over a smaller pre-trained LM predictor. Therefore, this paper proposes a weak-to-strong LM swap strategy, using a weak LM predictor during RNN-T loss training and then replacing it with a strong LLM. After LM replacement, the minimum word error rate (MWER) loss is employed to finetune the integration of the LLM predictor with the Transducer-Llama model. Experiments on the LibriSpeech and large-scale multi-lingual LibriSpeech corpora show that the proposed streaming Transducer-Llama approach gave a 17% relative WER reduction (WERR) over a strong FT baseline and a 32% WERR over an RNN-T baseline.

Title: The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignment

• Authors: HyunJin Kim, Xiaoyuan Yi, Jing Yao, Jianxun Lian, Muhua Huang, Shitong Duan, JinYeong Bak, Xing Xie
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16468
• Pdf URL: https://arxiv.org/pdf/2412.16468
• Copy Paste: [[2412.16468]] The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignment(https://arxiv.org/abs/2412.16468)
• Keywords: robust, large language model
• Abstract: The emergence of large language models (LLMs) has sparked the possibility of about Artificial Superintelligence (ASI), a hypothetical AI system surpassing human intelligence. However, existing alignment paradigms struggle to guide such advanced AI systems. Superalignment, the alignment of AI systems with human values and safety requirements at superhuman levels of capability aims to addresses two primary goals -- scalability in supervision to provide high-quality guidance signals and robust governance to ensure alignment with human values. In this survey, we examine scalable oversight methods and potential solutions for superalignment. Specifically, we explore the concept of ASI, the challenges it poses, and the limitations of current alignment paradigms in addressing the superalignment problem. Then we review scalable oversight methods for superalignment. Finally, we discuss the key challenges and propose pathways for the safe and continual improvement of ASI systems. By comprehensively reviewing the current literature, our goal is provide a systematical introduction of existing methods, analyze their strengths and limitations, and discuss potential future directions.

Title: Chained Tuning Leads to Biased Forgetting

• Authors: Megan Ung, Alicia Sun, Samuel J. Bell, Bhaktipriya Radharapu, Levent Sagun, Adina Williams
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16469
• Pdf URL: https://arxiv.org/pdf/2412.16469
• Copy Paste: [[2412.16469]] Chained Tuning Leads to Biased Forgetting(https://arxiv.org/abs/2412.16469)
• Keywords: large language model
• Abstract: Large language models (LLMs) are often fine-tuned for use on downstream tasks, though this can degrade capabilities learned during previous training. This phenomenon, often referred to as catastrophic forgetting, has important potential implications for the safety of deployed models. In this work, we first show that models trained on downstream tasks forget their safety tuning to a greater extent than models trained in the opposite this http URL, we show that forgetting disproportionately impacts safety information about certain groups. To quantify this phenomenon, we define a new metric we term biased forgetting. We conduct a systematic evaluation of the effects of task ordering on forgetting and apply mitigations that can help the model recover from the forgetting observed. We hope our findings can better inform methods for chaining the finetuning of LLMs in continual learning settings to enable training of safer and less toxic models.

Title: When Can Proxies Improve the Sample Complexity of Preference Learning?

• Authors: Yuchen Zhu, Daniel Augusto de Souza, Zhengyan Shi, Mengyue Yang, Pasquale Minervini, Alexander D'Amour, Matt J. Kusner
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.16475
• Pdf URL: https://arxiv.org/pdf/2412.16475
• Copy Paste: [[2412.16475]] When Can Proxies Improve the Sample Complexity of Preference Learning?(https://arxiv.org/abs/2412.16475)
• Keywords: large language model
• Abstract: We address the problem of reward hacking, where maximising a proxy reward does not necessarily increase the true reward. This is a key concern for Large Language Models (LLMs), as they are often fine-tuned on human preferences that may not accurately reflect a true objective. Existing work uses various tricks such as regularisation, tweaks to the reward model, and reward hacking detectors, to limit the influence that such proxy preferences have on a model. Luckily, in many contexts such as medicine, education, and law, a sparse amount of expert data is often available. In these cases, it is often unclear whether the addition of proxy data can improve policy learning. We outline a set of sufficient conditions on proxy feedback that, if satisfied, indicate that proxy data can provably improve the sample complexity of learning the ground truth policy. These conditions can inform the data collection process for specific tasks. The result implies a parameterisation for LLMs that achieves this improved sample complexity. We detail how one can adapt existing architectures to yield this improved sample complexity.

Title: Enhancing Nighttime Vehicle Detection with Day-to-Night Style Transfer and Labeling-Free Augmentation

• Authors: Yunxiang Yang, Hao Zhen, Yongcan Huang, Jidong J. Yang
• Subjects: cs.CV, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.16478
• Pdf URL: https://arxiv.org/pdf/2412.16478
• Copy Paste: [[2412.16478]] Enhancing Nighttime Vehicle Detection with Day-to-Night Style Transfer and Labeling-Free Augmentation(https://arxiv.org/abs/2412.16478)
• Keywords: generative
• Abstract: Existing deep learning-based object detection models perform well under daytime conditions but face significant challenges at night, primarily because they are predominantly trained on daytime images. Additionally, training with nighttime images presents another challenge: even human annotators struggle to accurately label objects in low-light conditions. This issue is particularly pronounced in transportation applications, such as detecting vehicles and other objects of interest on rural roads at night, where street lighting is often absent, and headlights may introduce undesirable glare. This study addresses these challenges by introducing a novel framework for labeling-free data augmentation, leveraging CARLA-generated synthetic data for day-to-night image style transfer. Specifically, the framework incorporates the Efficient Attention Generative Adversarial Network for realistic day-to-night style transfer and uses CARLA-generated synthetic nighttime images to help the model learn vehicle headlight effects. To evaluate the efficacy of the proposed framework, we fine-tuned the YOLO11 model with an augmented dataset specifically curated for rural nighttime environments, achieving significant improvements in nighttime vehicle detection. This novel approach is simple yet effective, offering a scalable solution to enhance AI-based detection systems in low-visibility environments and extend the applicability of object detection models to broader real-world contexts.

Title: Flash3D: Super-scaling Point Transformers through Joint Hardware-Geometry Locality

• Authors: Liyan Chen, Gregory P. Meyer, Zaiwei Zhang, Eric M. Wolff, Paul Vernaza
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16481
• Pdf URL: https://arxiv.org/pdf/2412.16481
• Copy Paste: [[2412.16481]] Flash3D: Super-scaling Point Transformers through Joint Hardware-Geometry Locality(https://arxiv.org/abs/2412.16481)
• Keywords: transformer
• Abstract: Recent efforts recognize the power of scale in 3D learning (e.g. PTv3) and attention mechanisms (e.g. FlashAttention). However, current point cloud backbones fail to holistically unify geometric locality, attention mechanisms, and GPU architectures in one view. In this paper, we introduce Flash3D Transformer, which aligns geometric locality and GPU tiling through a principled locality mechanism based on Perfect Spatial Hashing (PSH). The common alignment with GPU tiling naturally fuses our PSH locality mechanism with FlashAttention at negligible extra cost. This mechanism affords flexible design choices throughout the backbone that result in superior downstream task results. Flash3D outperforms state-of-the-art PTv3 results on benchmark datasets, delivering a 2.25x speed increase and 2.4x memory efficiency boost. This efficiency enables scaling to wider attention scopes and larger models without additional overhead. Such scaling allows Flash3D to achieve even higher task accuracies than PTv3 under the same compute budget.

Title: MOL-Mamba: Enhancing Molecular Representation with Structural & Electronic Insights

• Authors: Jingjing Hu, Dan Guo, Zhan Si, Deguang Liu, Yunfeng Diao, Jing Zhang, Jinxing Zhou, Meng Wang
• Subjects: cs.LG, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2412.16483
• Pdf URL: https://arxiv.org/pdf/2412.16483
• Copy Paste: [[2412.16483]] MOL-Mamba: Enhancing Molecular Representation with Structural & Electronic Insights(https://arxiv.org/abs/2412.16483)
• Keywords: transformer
• Abstract: Molecular representation learning plays a crucial role in various downstream tasks, such as molecular property prediction and drug design. To accurately represent molecules, Graph Neural Networks (GNNs) and Graph Transformers (GTs) have shown potential in the realm of self-supervised pretraining. However, existing approaches often overlook the relationship between molecular structure and electronic information, as well as the internal semantic reasoning within molecules. This omission of fundamental chemical knowledge in graph semantics leads to incomplete molecular representations, missing the integration of structural and electronic data. To address these issues, we introduce MOL-Mamba, a framework that enhances molecular representation by combining structural and electronic insights. MOL-Mamba consists of an Atom & Fragment Mamba-Graph (MG) for hierarchical structural reasoning and a Mamba-Transformer (MT) fuser for integrating molecular structure and electronic correlation learning. Additionally, we propose a Structural Distribution Collaborative Training and E-semantic Fusion Training framework to further enhance molecular representation learning. Extensive experiments demonstrate that MOL-Mamba outperforms state-of-the-art baselines across eleven chemical-biological molecular datasets. Code is available at this https URL.

Title: Automated CVE Analysis: Harnessing Machine Learning In Designing Question-Answering Models For Cybersecurity Information Extraction

• Authors: Tanjim Bin Faruk
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16484
• Pdf URL: https://arxiv.org/pdf/2412.16484
• Copy Paste: [[2412.16484]] Automated CVE Analysis: Harnessing Machine Learning In Designing Question-Answering Models For Cybersecurity Information Extraction(https://arxiv.org/abs/2412.16484)
• Keywords: security, attack, robust, extraction
• Abstract: The vast majority of cybersecurity information is unstructured text, including critical data within databases such as CVE, NVD, CWE, CAPEC, and the MITRE ATT&CK Framework. These databases are invaluable for analyzing attack patterns and understanding attacker behaviors. Creating a knowledge graph by integrating this information could unlock significant insights. However, processing this large amount of data requires advanced deep-learning techniques. A crucial step towards building such a knowledge graph is developing a robust mechanism for automating the extraction of answers to specific questions from the unstructured text. Question Answering (QA) systems play a pivotal role in this process by pinpointing and extracting precise information, facilitating the mapping of relationships between various data points. In the cybersecurity context, QA systems encounter unique challenges due to the need to interpret and answer questions based on a wide array of domain-specific information. To tackle these challenges, it is necessary to develop a cybersecurity-specific dataset and train a machine learning model on it, aimed at enhancing the understanding and retrieval of domain-specific information. This paper presents a novel dataset and describes a machine learning model trained on this dataset for the QA task. It also discusses the model's performance and key findings in a manner that maintains a balance between formality and accessibility.

Title: Evaluating the Performance of Large Language Models in Scientific Claim Detection and Classification

• Authors: Tanjim Bin Faruk
• Subjects: cs.CL, cs.CY, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2412.16486
• Pdf URL: https://arxiv.org/pdf/2412.16486
• Copy Paste: [[2412.16486]] Evaluating the Performance of Large Language Models in Scientific Claim Detection and Classification(https://arxiv.org/abs/2412.16486)
• Keywords: large language model
• Abstract: The pervasive influence of social media during the COVID-19 pandemic has been a double-edged sword, enhancing communication while simultaneously propagating misinformation. This \textit{Digital Infodemic} has highlighted the urgent need for automated tools capable of discerning and disseminating factual content. This study evaluates the efficacy of Large Language Models (LLMs) as innovative solutions for mitigating misinformation on platforms like Twitter. LLMs, such as OpenAI's GPT and Meta's LLaMA, offer a pre-trained, adaptable approach that bypasses the extensive training and overfitting issues associated with traditional machine learning models. We assess the performance of LLMs in detecting and classifying COVID-19-related scientific claims, thus facilitating informed decision-making. Our findings indicate that LLMs have significant potential as automated fact-checking tools, though research in this domain is nascent and further exploration is required. We present a comparative analysis of LLMs' performance using a specialized dataset and propose a framework for their application in public health communication.

Title: ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition

• Authors: Seungdong Yoa, Seungjun Lee, Hyeseung Cho, Bumsoo Kim, Woohyung Lim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16491
• Pdf URL: https://arxiv.org/pdf/2412.16491
• Copy Paste: [[2412.16491]] ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition(https://arxiv.org/abs/2412.16491)
• Keywords: transformer
• Abstract: Vision Transformers (ViTs) have achieved remarkable success in various computer vision tasks. However, ViTs have a huge computational cost due to their inherent reliance on multi-head self-attention (MHSA), prompting efforts to accelerate ViTs for practical applications. To this end, recent works aim to reduce the number of tokens, mainly focusing on how to effectively prune or merge them. Nevertheless, since ViT tokens are generated from non-overlapping grid patches, they usually do not convey sufficient semantics, making it incompatible with efficient ViTs. To address this, we propose ImagePiece, a novel re-tokenization strategy for Vision Transformers. Following the MaxMatch strategy of NLP tokenization, ImagePiece groups semantically insufficient yet locally coherent tokens until they convey meaning. This simple retokenization is highly compatible with previous token reduction methods, being able to drastically narrow down relevant tokens, enhancing the inference speed of DeiT-S by 54% (nearly 1.5$\times$ faster) while achieving a 0.39% improvement in ImageNet classification accuracy. For hyper-speed inference scenarios (with 251% acceleration), our approach surpasses other baselines by an accuracy over 8%.

Title: STKDRec: Spatial-Temporal Knowledge Distillation for Takeaway Recommendation

• Authors: Shuyuan Zhao, Wei Chen, Boyan Shi, Liyong Zhou, Shuohao Lin, Huaiyu Wan
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.16502
• Pdf URL: https://arxiv.org/pdf/2412.16502
• Copy Paste: [[2412.16502]] STKDRec: Spatial-Temporal Knowledge Distillation for Takeaway Recommendation(https://arxiv.org/abs/2412.16502)
• Keywords: transformer
• Abstract: The takeaway recommendation system is designed to recommend users' future takeaway purchases based on their historical purchase behaviors, thereby improving user satisfaction and increasing merchant sales. Existing methods focus on incorporating auxiliary information or leveraging knowledge graphs to alleviate the sparsity issue of user purchase sequence data. However, two main challenges limit the performance of these approaches: (1) how to capture dynamic user preferences on complex geospatial information and (2) how to efficiently integrate spatial-temporal knowledge from graphs and sequence data with low calculation costs. In this paper, we propose a novel spatial-temporal knowledge distillation for takeaway recommendation model (STKDRec) based on the two-stage training process. Specifically, during the first pre-training stage, a spatial-temporal knowledge graph (STKG) encoder is pre-trained to extract the high-order spatial-temporal and collaborative associations within the STKG. During the second STKD stage, a spatial-temporal Transformer is employed to comprehensively model dynamic user preferences on various types of fine-grained geospatial information from a sequence perspective. Furthermore, the STKD strategy is introduced to adaptively fuse the rich spatial-temporal knowledge from the pre-trained STKG encoder and the spatial-temporal transformer while reducing the cost of model training. Extensive experiments on three real-world datasets show that our STKDRec significantly outperforms the state-of-the-art baselines. Our code is available at:this https URL.

Title: First-frame Supervised Video Polyp Segmentation via Propagative and Semantic Dual-teacher Network

• Authors: Qiang Hu, Mei Liu, Qiang Li, Zhiwei Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16503
• Pdf URL: https://arxiv.org/pdf/2412.16503
• Copy Paste: [[2412.16503]] First-frame Supervised Video Polyp Segmentation via Propagative and Semantic Dual-teacher Network(https://arxiv.org/abs/2412.16503)
• Keywords: segmentation
• Abstract: Automatic video polyp segmentation plays a critical role in gastrointestinal cancer screening, but the cost of frameby-frame annotations is prohibitively high. While sparse-frame supervised methods have reduced this burden proportionately, the cost remains overwhelming for long-duration videos and large-scale datasets. In this paper, we, for the first time, reduce the annotation cost to just a single frame per polyp video, regardless of the video's length. To this end, we introduce a new task, First-Frame Supervised Video Polyp Segmentation (FSVPS), and propose a novel Propagative and Semantic Dual-Teacher Network (PSDNet). Specifically, PSDNet adopts a teacher-student framework but employs two distinct types of teachers: the propagative teacher and the semantic teacher. The propagative teacher is a universal object tracker that propagates the first-frame annotation to subsequent frames as pseudo labels. However, tracking errors may accumulate over time, gradually degrading the pseudo labels and misguiding the student model. To address this, we introduce the semantic teacher, an exponential moving average of the student model, which produces more stable and time-invariant pseudo labels. PSDNet merges the pseudo labels from both teachers using a carefully-designed back-propagation strategy. This strategy assesses the quality of the pseudo labels by tracking them backward to the first frame. High-quality pseudo labels are more likely to spatially align with the firstframe annotation after this backward tracking, ensuring more accurate teacher-to-student knowledge transfer and improved segmentation performance. Benchmarking on SUN-SEG, the largest VPS dataset, demonstrates the competitive performance of PSDNet compared to fully-supervised approaches, and its superiority over sparse-frame supervised state-of-the-arts with a minimum improvement of 4.5% in Dice score.

Title: Context-Aware Outlier Rejection for Robust Multi-View 3D Tracking of Similar Small Birds in An Outdoor Aviary

• Authors: Keon Moradi, Ethan Haque, Jasmeen Kaur, Alexandra B. Bentz, Eli S. Bridge, Golnaz Habibi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16511
• Pdf URL: https://arxiv.org/pdf/2412.16511
• Copy Paste: [[2412.16511]] Context-Aware Outlier Rejection for Robust Multi-View 3D Tracking of Similar Small Birds in An Outdoor Aviary(https://arxiv.org/abs/2412.16511)
• Keywords: robust
• Abstract: This paper presents a novel approach for robust 3D tracking of multiple birds in an outdoor aviary using a multi-camera system. Our method addresses the challenges of visually similar birds and their rapid movements by leveraging environmental landmarks for enhanced feature matching and 3D reconstruction. In our approach, outliers are rejected based on their nearest landmark. This enables precise 3D-modeling and simultaneous tracking of multiple birds. By utilizing environmental context, our approach significantly improves the differentiation between visually similar birds, a key obstacle in existing tracking systems. Experimental results demonstrate the effectiveness of our method, showing a $20\%$ elimination of outliers in the 3D reconstruction process, with a $97\%$ accuracy in matching. This remarkable accuracy in 3D modeling translates to robust and reliable tracking of multiple birds, even in challenging outdoor conditions. Our work not only advances the field of computer vision but also provides a valuable tool for studying bird behavior and movement patterns in natural settings. We also provide a large annotated dataset of 80 birds residing in four enclosures for 20 hours of footage which provides a rich testbed for researchers in computer vision, ornithologists, and ecologists. Code and the link to the dataset is available at this https URL

Title: TrojFlow: Flow Models are Natural Targets for Trojan Attacks

• Authors: Zhengyang Qi, Xiaohua Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16512
• Pdf URL: https://arxiv.org/pdf/2412.16512
• Copy Paste: [[2412.16512]] TrojFlow: Flow Models are Natural Targets for Trojan Attacks(https://arxiv.org/abs/2412.16512)
• Keywords: defense, attack, diffusion, generative
• Abstract: Flow-based generative models (FMs) have rapidly advanced as a method for mapping noise to data, its efficient training and sampling process makes it widely applicable in various fields. FMs can be viewed as a variant of diffusion models (DMs). At the same time, previous studies have shown that DMs are vulnerable to Trojan/Backdoor attacks, a type of output manipulation attack triggered by a maliciously embedded pattern at model input. We found that Trojan attacks on generative models are essentially equivalent to image transfer tasks from the backdoor distribution to the target distribution, the unique ability of FMs to fit any two arbitrary distributions significantly simplifies the training and sampling setups for attacking FMs, making them inherently natural targets for backdoor attacks. In this paper, we propose TrojFlow, exploring the vulnerabilities of FMs through Trojan attacks. In particular, we consider various attack settings and their combinations and thoroughly explore whether existing defense methods for DMs can effectively defend against our proposed attack scenarios. We evaluate TrojFlow on CIFAR-10 and CelebA datasets, our experiments show that our method can compromise FMs with high utility and specificity, and can easily break through existing defense mechanisms.

Title: VSFormer: Value and Shape-Aware Transformer with Prior-Enhanced Self-Attention for Multivariate Time Series Classification

• Authors: Wenjie Xi, Rundong Zuo, Alejandro Alvarez, Jie Zhang, Byron Choi, Jessica Lin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16515
• Pdf URL: https://arxiv.org/pdf/2412.16515
• Copy Paste: [[2412.16515]] VSFormer: Value and Shape-Aware Transformer with Prior-Enhanced Self-Attention for Multivariate Time Series Classification(https://arxiv.org/abs/2412.16515)
• Keywords: transformer
• Abstract: Multivariate time series classification is a crucial task in data mining, attracting growing research interest due to its broad applications. While many existing methods focus on discovering discriminative patterns in time series, real-world data does not always present such patterns, and sometimes raw numerical values can also serve as discriminative features. Additionally, the recent success of Transformer models has inspired many studies. However, when applying to time series classification, the self-attention mechanisms in Transformer models could introduce classification-irrelevant features, thereby compromising accuracy. To address these challenges, we propose a novel method, VSFormer, that incorporates both discriminative patterns (shape) and numerical information (value). In addition, we extract class-specific prior information derived from supervised information to enrich the positional encoding and provide classification-oriented self-attention learning, thereby enhancing its effectiveness. Extensive experiments on all 30 UEA archived datasets demonstrate the superior performance of our method compared to SOTA models. Through ablation studies, we demonstrate the effectiveness of the improved encoding layer and the proposed self-attention mechanism. Finally, We provide a case study on a real-world time series dataset without discriminative patterns to interpret our model.

Title: HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Device Scenarios

• Authors: Jun Wang, Jiamu Zhou, Muning Wen, Xiaoyun Mo, Haoyu Zhang, Qiqiang Lin, Cheng Jin, Xihuai Wang, Weinan Zhang, Qiuying Peng, Jun Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16516
• Pdf URL: https://arxiv.org/pdf/2412.16516
• Copy Paste: [[2412.16516]] HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Device Scenarios(https://arxiv.org/abs/2412.16516)
• Keywords: large language model
• Abstract: Evaluating the capabilities of large language models (LLMs) in human-LLM interactions remains challenging due to the inherent complexity and openness of dialogue processes. This paper introduces HammerBench, a novel benchmarking framework designed to assess the function-calling ability of LLMs more effectively in such interactions. We model a wide range of real-world user scenarios on mobile devices, encompassing imperfect instructions, diverse question-answer trajectories, intent/argument shifts, and the use of external individual information through pronouns. To construct the corresponding datasets, we propose a comprehensive pipeline that involves LLM-generated data and multiple rounds of human validation, ensuring high data quality. Additionally, we decompose the conversations into function-calling snapshots, enabling a fine-grained evaluation of each turn. We evaluate several popular LLMs using HammerBench and highlight different performance aspects. Our empirical findings reveal that errors in parameter naming constitute the primary factor behind conversation failures across different data types.

Title: Physics-Guided Fair Graph Sampling for Water Temperature Prediction in River Networks

• Authors: Erhu He, Declan Kutscher, Yiqun Xie, Jacob Zwart, Zhe Jiang, Huaxiu Yao, Xiaowei Jia
• Subjects: cs.LG, cs.CY, physics.soc-ph, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.16523
• Pdf URL: https://arxiv.org/pdf/2412.16523
• Copy Paste: [[2412.16523]] Physics-Guided Fair Graph Sampling for Water Temperature Prediction in River Networks(https://arxiv.org/abs/2412.16523)
• Keywords: fair
• Abstract: This work introduces a novel graph neural networks (GNNs)-based method to predict stream water temperature and reduce model bias across locations of different income and education levels. Traditional physics-based models often have limited accuracy because they are necessarily approximations of reality. Recently, there has been an increasing interest of using GNNs in modeling complex water dynamics in stream networks. Despite their promise in improving the accuracy, GNNs can bring additional model bias through the aggregation process, where node features are updated by aggregating neighboring nodes. The bias can be especially pronounced when nodes with similar sensitive attributes are frequently connected. We introduce a new method that leverages physical knowledge to represent the node influence in GNNs, and then utilizes physics-based influence to refine the selection and weights over the neighbors. The objective is to facilitate equitable treatment over different sensitive groups in the graph aggregation, which helps reduce spatial bias over locations, especially for those in underprivileged groups. The results on the Delaware River Basin demonstrate the effectiveness of the proposed method in preserving equitable performance across locations in different sensitive groups.

Title: LLaVA-SLT: Visual Language Tuning for Sign Language Translation

• Authors: Han Liang, Chengyu Huang, Yuecheng Xu, Cheng Tang, Weicai Ye, Juze Zhang, Xin Chen, Jingyi Yu, Lan Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16524
• Pdf URL: https://arxiv.org/pdf/2412.16524
• Copy Paste: [[2412.16524]] LLaVA-SLT: Visual Language Tuning for Sign Language Translation(https://arxiv.org/abs/2412.16524)
• Keywords: robust, large language model
• Abstract: In the realm of Sign Language Translation (SLT), reliance on costly gloss-annotated datasets has posed a significant barrier. Recent advancements in gloss-free SLT methods have shown promise, yet they often largely lag behind gloss-based approaches in terms of translation accuracy. To narrow this performance gap, we introduce LLaVA-SLT, a pioneering Large Multimodal Model (LMM) framework designed to leverage the power of Large Language Models (LLMs) through effectively learned visual language embeddings. Our model is trained through a trilogy. First, we propose linguistic continued pretraining. We scale up the LLM and adapt it to the sign language domain using an extensive corpus dataset, effectively enhancing its textual linguistic knowledge about sign language. Then, we adopt visual contrastive pretraining to align the visual encoder with a large-scale pretrained text encoder. We propose hierarchical visual encoder that learns a robust word-level intermediate representation that is compatible with LLM token embeddings. Finally, we propose visual language tuning. We freeze pretrained models and employ a lightweight trainable MLP connector. It efficiently maps the pretrained visual language embeddings into the LLM token embedding space, enabling downstream SLT task. Our comprehensive experiments demonstrate that LLaVA-SLT outperforms the state-of-the-art methods. By using extra annotation-free data, it even closes to the gloss-based accuracy.

Title: Accelerating Private Large Transformers Inference through Fine-grained Collaborative Computation

• Authors: Yuntian Chen, Zhanyong Tang, Tianpei Lu, Bingsheng Zhang, Zhiying Shi, Zheng Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16537
• Pdf URL: https://arxiv.org/pdf/2412.16537
• Copy Paste: [[2412.16537]] Accelerating Private Large Transformers Inference through Fine-grained Collaborative Computation(https://arxiv.org/abs/2412.16537)
• Keywords: privacy, transformer
• Abstract: Homomorphic encryption (HE) and secret sharing (SS) enable computations on encrypted data, providing significant privacy benefits for large transformer-based models (TBM) in sensitive sectors like medicine and finance. However, private TBM inference incurs significant costs due to the coarse-grained application of HE and SS. We present FASTLMPI, a new approach to accelerate private TBM inference through fine-grained computation optimization. Specifically, through the fine-grained co-design of homomorphic encryption and secret sharing, FASTLMPI achieves efficient protocols for matrix multiplication, SoftMax, LayerNorm, and GeLU. In addition, FASTLMPI introduces a precise segmented approximation technique for differentiable non-linear, improving its fitting accuracy while maintaining a low polynomial degree. Compared to solution BOLT (S\&P'24), \SystemName shows a remarkable 54\% to 64\% decrease in runtime and an impressive 72.2\% reduction in communication costs.

Title: Towards Environmentally Equitable AI

• Authors: Mohammad Hajiesmaili, Shaolei Ren, Ramesh K. Sitaraman, Adam Wierman
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.16539
• Pdf URL: https://arxiv.org/pdf/2412.16539
• Copy Paste: [[2412.16539]] Towards Environmentally Equitable AI(https://arxiv.org/abs/2412.16539)
• Keywords: fair
• Abstract: The skyrocketing demand for artificial intelligence (AI) has created an enormous appetite for globally deployed power-hungry servers. As a result, the environmental footprint of AI systems has come under increasing scrutiny. More crucially, the current way that we exploit AI workloads' flexibility and manage AI systems can lead to wildly different environmental impacts across locations, increasingly raising environmental inequity concerns and creating unintended sociotechnical consequences. In this paper, we advocate environmental equity as a priority for the management of future AI systems, advancing the boundaries of existing resource management for sustainable AI and also adding a unique dimension to AI fairness. Concretely, we uncover the potential of equity-aware geographical load balancing to fairly re-distribute the environmental cost across different regions, followed by algorithmic challenges. We conclude by discussing a few future directions to exploit the full potential of system management approaches to mitigate AI's environmental inequity.

Title: FairDD: Enhancing Fairness with domain-incremental learning in dermatological disease diagnosis

• Authors: Yiqin Luo, Tianlong Gu
• Subjects: cs.LG, cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.16542
• Pdf URL: https://arxiv.org/pdf/2412.16542
• Copy Paste: [[2412.16542]] FairDD: Enhancing Fairness with domain-incremental learning in dermatological disease diagnosis(https://arxiv.org/abs/2412.16542)
• Keywords: robust, fair
• Abstract: With the rapid advancement of deep learning technologies, artificial intelligence has become increasingly prevalent in the research and application of dermatological disease diagnosis. However, this data-driven approach often faces issues related to decision bias. Existing fairness enhancement techniques typically come at a substantial cost to accuracy. This study aims to achieve a better trade-off between accuracy and fairness in dermatological diagnostic models. To this end, we propose a novel fair dermatological diagnosis network, named FairDD, which leverages domain incremental learning to balance the learning of different groups by being sensitive to changes in data distribution. Additionally, we incorporate the mixup data augmentation technique and supervised contrastive learning to enhance the network's robustness and generalization. Experimental validation on two dermatological datasets demonstrates that our proposed method excels in both fairness criteria and the trade-off between fairness and performance.

Title: Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models

• Authors: Zhisong Zhang, Yan Wang, Xinting Huang, Tianqing Fang, Hongming Zhang, Chenlong Deng, Shuaiyi Li, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16545
• Pdf URL: https://arxiv.org/pdf/2412.16545
• Copy Paste: [[2412.16545]] Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models(https://arxiv.org/abs/2412.16545)
• Keywords: transformer, large language model
• Abstract: Large language models have shown remarkable performance across a wide range of language tasks, owing to their exceptional capabilities in context modeling. The most commonly used method of context modeling is full self-attention, as seen in standard decoder-only Transformers. Although powerful, this method can be inefficient for long sequences and may overlook inherent input structures. To address these problems, an alternative approach is parallel context encoding, which splits the context into sub-pieces and encodes them parallelly. Because parallel patterns are not encountered during training, naively applying parallel encoding leads to performance degradation. However, the underlying reasons and potential mitigations are unclear. In this work, we provide a detailed analysis of this issue and identify that unusually high attention entropy can be a key factor. Furthermore, we adopt two straightforward methods to reduce attention entropy by incorporating attention sinks and selective mechanisms. Experiments on various tasks reveal that these methods effectively lower irregular attention entropy and narrow performance gaps. We hope this study can illuminate ways to enhance context modeling mechanisms.

Title: Diffusion Prior Interpolation for Flexibility Real-World Face Super-Resolution

• Authors: Jiarui Yang, Tao Dai, Yufei Zhu, Naiqi Li, Jinmin Li, Shutao Xia
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16552
• Pdf URL: https://arxiv.org/pdf/2412.16552
• Copy Paste: [[2412.16552]] Diffusion Prior Interpolation for Flexibility Real-World Face Super-Resolution(https://arxiv.org/abs/2412.16552)
• Keywords: diffusion, generative
• Abstract: Diffusion models represent the state-of-the-art in generative modeling. Due to their high training costs, many works leverage pre-trained diffusion models' powerful representations for downstream tasks, such as face super-resolution (FSR), through fine-tuning or prior-based methods. However, relying solely on priors without supervised training makes it challenging to meet the pixel-level accuracy requirements of discrimination task. Although prior-based methods can achieve high fidelity and high-quality results, ensuring consistency remains a significant challenge. In this paper, we propose a masking strategy with strong and weak constraints and iterative refinement for real-world FSR, termed Diffusion Prior Interpolation (DPI). We introduce conditions and constraints on consistency by masking different sampling stages based on the structural characteristics of the face. Furthermore, we propose a condition Corrector (CRT) to establish a reciprocal posterior sampling process, enhancing FSR performance by mutual refinement of conditions and samples. DPI can balance consistency and diversity and can be seamlessly integrated into pre-trained models. In extensive experiments conducted on synthetic and real datasets, along with consistency validation in face recognition, DPI demonstrates superiority over SOTA FSR methods. The code is available at \url{this https URL}.

Title: Semantics Prompting Data-Free Quantization for Low-Bit Vision Transformers

• Authors: Yunshan Zhong, Yuyao Zhou, Yuxin Zhang, Shen Li, Yong Li, Fei Chao, Zhanpeng Zeng, Rongrong Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16553
• Pdf URL: https://arxiv.org/pdf/2412.16553
• Copy Paste: [[2412.16553]] Semantics Prompting Data-Free Quantization for Low-Bit Vision Transformers(https://arxiv.org/abs/2412.16553)
• Keywords: security, data-free, transformer
• Abstract: Data-free quantization (DFQ), which facilitates model quantization without real data to address increasing concerns about data security, has garnered significant attention within the model compression community. Recently, the unique architecture of vision transformers (ViTs) has driven the development of specialized DFQ techniques. However, we observe that the synthetic images from existing methods suffer from the deficient semantics issue compared to real images, thereby compromising performance. Motivated by this, we propose SPDFQ, a Semantics Prompting Data-Free Quantization method for ViTs. First, SPDFQ incorporates Attention Priors Alignment (APA), which uses randomly generated attention priors to enhance the semantics of synthetic images. Second, SPDFQ introduces Multi-Semantic Reinforcement (MSR), which utilizes localized patch optimization to prompt efficient parameterization and diverse semantics in synthetic images. Finally, SPDFQ employs Softlabel Learning (SL), where soft learning targets are adapted to encourage more complex semantics and accommodate images augmented by MSR. Experimental results demonstrate that SPDFQ significantly outperforms existing methods. For instance, SPDFQ achieves a 15.52% increase in top-1 accuracy on ImageNet for W4A4 ViT-B

Title: Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models

• Authors: Yanxu Mao, Peipei Liu, Tiehan Cui, Congying Liu, Datao You
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16555
• Pdf URL: https://arxiv.org/pdf/2412.16555
• Copy Paste: [[2412.16555]] Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models(https://arxiv.org/abs/2412.16555)
• Keywords: security, attack, large language model
• Abstract: Large language models (LLMs) are widely applied in various fields of society due to their powerful reasoning, understanding, and generation capabilities. However, the security issues associated with these models are becoming increasingly severe. Jailbreaking attacks, as an important method for detecting vulnerabilities in LLMs, have been explored by researchers who attempt to induce these models to generate harmful content through various attack methods. Nevertheless, existing jailbreaking methods face numerous limitations, such as excessive query counts, limited coverage of jailbreak modalities, low attack success rates, and simplistic evaluation methods. To overcome these constraints, this paper proposes a multimodal jailbreaking method: JMLLM. This method integrates multiple strategies to perform comprehensive jailbreak attacks across text, visual, and auditory modalities. Additionally, we contribute a new and comprehensive dataset for multimodal jailbreaking research: TriJail, which includes jailbreak prompts for all three modalities. Experiments on the TriJail dataset and the benchmark dataset AdvBench, conducted on 13 popular LLMs, demonstrate advanced attack success rates and significant reduction in time overhead.

Title: Learning for Cross-Layer Resource Allocation in MEC-Aided Cell-Free Networks

• Authors: Chong Zheng, Shiwen He, Yongming Huang, Tony Q. S. Quek
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16565
• Pdf URL: https://arxiv.org/pdf/2412.16565
• Copy Paste: [[2412.16565]] Learning for Cross-Layer Resource Allocation in MEC-Aided Cell-Free Networks(https://arxiv.org/abs/2412.16565)
• Keywords: robust
• Abstract: Cross-layer resource allocation over mobile edge computing (MEC)-aided cell-free networks can sufficiently exploit the transmitting and computing resources to promote the data rate. However, the technical bottlenecks of traditional methods pose significant challenges to cross-layer optimization. In this paper, joint subcarrier allocation and beamforming optimization are investigated for the MEC-aided cell-free network from the perspective of deep learning to maximize the weighted sum rate. Specifically, we convert the underlying problem into a joint multi-task optimization problem and then propose a centralized multi-task self-supervised learning algorithm to solve the problem so as to avoid costly manual labeling. Therein, two novel and general loss functions, i.e., negative fraction linear loss and exponential linear loss whose advantages in robustness and target domain have been proved and discussed, are designed to enable self-supervised learning. Moreover, we further design a MEC-enabled distributed multi-task self-supervised learning (DMTSSL) algorithm, with low complexity and high scalability to address the challenge of dimensional disaster. Finally, we develop the distance-aware transfer learning algorithm based on the DMTSSL algorithm to handle the dynamic scenario with negligible computation cost. Simulation results under $3$rd generation partnership project 38.901 urban-macrocell scenario demonstrate the superiority of the proposed algorithms over the baseline algorithms.

Title: FedGA: Federated Learning with Gradient Alignment for Error Asymmetry Mitigation

• Authors: Chenguang Xiao, Zheming Zuo, Shuo Wang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16582
• Pdf URL: https://arxiv.org/pdf/2412.16582
• Copy Paste: [[2412.16582]] FedGA: Federated Learning with Gradient Alignment for Error Asymmetry Mitigation(https://arxiv.org/abs/2412.16582)
• Keywords: federate
• Abstract: Federated learning (FL) triggers intra-client and inter-client class imbalance, with the latter compared to the former leading to biased client updates and thus deteriorating the distributed models. Such a bias is exacerbated during the server aggregation phase and has yet to be effectively addressed by conventional re-balancing methods. To this end, different from the off-the-shelf label or loss-based approaches, we propose a gradient alignment (GA)-informed FL method, dubbed as FedGA, where the importance of error asymmetry (EA) in bias is observed and its linkage to the gradient of the loss to raw logits is explored. Concretely, GA, implemented by label calibration during the model backpropagation process, prevents catastrophic forgetting of rate and missing classes, hence boosting model convergence and accuracy. Experimental results on five benchmark datasets demonstrate that GA outperforms the pioneering counterpart FedAvg and its four variants in minimizing EA and updating bias, and accordingly yielding higher F1 score and accuracy margins when the Dirichlet distribution sampling factor $\alpha$ increases. The code and more details are available at \url{this https URL}.

Title: REO-VLM: Transforming VLM to Meet Regression Challenges in Earth Observation

• Authors: Xizhe Xue, Guoting Wei, Hao Chen, Haokui Zhang, Feng Lin, Chunhua Shen, Xiao Xiang Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16583
• Pdf URL: https://arxiv.org/pdf/2412.16583
• Copy Paste: [[2412.16583]] REO-VLM: Transforming VLM to Meet Regression Challenges in Earth Observation(https://arxiv.org/abs/2412.16583)
• Keywords: generative
• Abstract: The rapid evolution of Vision Language Models (VLMs) has catalyzed significant advancements in artificial intelligence, expanding research across various disciplines, including Earth Observation (EO). While VLMs have enhanced image understanding and data processing within EO, their applications have predominantly focused on image content description. This limited focus overlooks their potential in geographic and scientific regression tasks, which are essential for diverse EO applications. To bridge this gap, this paper introduces a novel benchmark dataset, called \textbf{REO-Instruct} to unify regression and generation tasks specifically for the EO domain. Comprising 1.6 million multimodal EO imagery and language pairs, this dataset is designed to support both biomass regression and image content interpretation tasks. Leveraging this dataset, we develop \textbf{REO-VLM}, a groundbreaking model that seamlessly integrates regression capabilities with traditional generative functions. By utilizing language-driven reasoning to incorporate scientific domain knowledge, REO-VLM goes beyond solely relying on EO imagery, enabling comprehensive interpretation of complex scientific attributes from EO data. This approach establishes new performance benchmarks and significantly enhances the capabilities of environmental monitoring and resource management.

Title: Leveraging Contrastive Learning for Semantic Segmentation with Consistent Labels Across Varying Appearances

• Authors: Javier Montalvo, Roberto Alcover-Couso, Pablo Carballeira, Álvaro García-Martín, Juan C. SanMiguel, Marcos Escudero-Viñolo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16592
• Pdf URL: https://arxiv.org/pdf/2412.16592
• Copy Paste: [[2412.16592]] Leveraging Contrastive Learning for Semantic Segmentation with Consistent Labels Across Varying Appearances(https://arxiv.org/abs/2412.16592)
• Keywords: segmentation
• Abstract: This paper introduces a novel synthetic dataset that captures urban scenes under a variety of weather conditions, providing pixel-perfect, ground-truth-aligned images to facilitate effective feature alignment across domains. Additionally, we propose a method for domain adaptation and generalization that takes advantage of the multiple versions of each scene, enforcing feature consistency across different weather scenarios. Our experimental results demonstrate the impact of our dataset in improving performance across several alignment metrics, addressing key challenges in domain adaptation and generalization for segmentation tasks. This research also explores critical aspects of synthetic data generation, such as optimizing the balance between the volume and variability of generated images to enhance segmentation performance. Ultimately, this work sets forth a new paradigm for synthetic data generation and domain adaptation.

Title: Fingerprinting of Machines in Critical Systems for Integrity Monitoring and Verification

• Authors: Prakhar Paliwal, Arjun Sable, Manjesh K. Hanawal
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16595
• Pdf URL: https://arxiv.org/pdf/2412.16595
• Copy Paste: [[2412.16595]] Fingerprinting of Machines in Critical Systems for Integrity Monitoring and Verification(https://arxiv.org/abs/2412.16595)
• Keywords: secure, security, robust
• Abstract: As cyber threats continue to evolve and diversify, it has become increasingly challenging to identify the root causes of security breaches that occur between periodic security assessments. This paper explores the fundamental importance of system fingerprinting as a proactive and effective approach to addressing this issue. By capturing a comprehensive host's fingerprint, including hardware-related details, file hashes, and kernel-level information, during periods of system cleanliness, a historical record is established. This historical record provides valuable insights into system changes and assists in understanding the factors contributing to a security breach. We develop a tool to capture and store these fingerprints securely, leveraging the advanced security features. Our approach presents a robust solution to address the constantly evolving cyber threat landscape, thereby safeguarding the integrity and security of critical systems.

Title: V"Mean"ba: Visual State Space Models only need 1 hidden dimension

• Authors: Tien-Yu Chi, Hung-Yueh Chiang, Chi-Chih Chang, Ning-Chi Huang, Kai-Chiang Wu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16602
• Pdf URL: https://arxiv.org/pdf/2412.16602
• Copy Paste: [[2412.16602]] V"Mean"ba: Visual State Space Models only need 1 hidden dimension(https://arxiv.org/abs/2412.16602)
• Keywords: transformer, segmentation
• Abstract: Vision transformers dominate image processing tasks due to their superior performance. However, the quadratic complexity of self-attention limits the scalability of these systems and their deployment on resource-constrained devices. State Space Models (SSMs) have emerged as a solution by introducing a linear recurrence mechanism, which reduces the complexity of sequence modeling from quadratic to linear. Recently, SSMs have been extended to high-resolution vision tasks. Nonetheless, the linear recurrence mechanism struggles to fully utilize matrix multiplication units on modern hardware, resulting in a computational bottleneck. We address this issue by introducing \textit{VMeanba}, a training-free compression method that eliminates the channel dimension in SSMs using mean operations. Our key observation is that the output activations of SSM blocks exhibit low variances across channels. Our \textit{VMeanba} leverages this property to optimize computation by averaging activation maps across the channel to reduce the computational overhead without compromising accuracy. Evaluations on image classification and semantic segmentation tasks demonstrate that \textit{VMeanba} achieves up to a 1.12x speedup with less than a 3\% accuracy loss. When combined with 40\% unstructured pruning, the accuracy drop remains under 3\%.

Title: OmniSplat: Taming Feed-Forward 3D Gaussian Splatting for Omnidirectional Images with Editable Capabilities

• Authors: Suyoung Lee, Jaeyoung Chung, Kihoon Kim, Jaeyoo Huh, Gunhee Lee, Minsoo Lee, Kyoung Mu Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16604
• Pdf URL: https://arxiv.org/pdf/2412.16604
• Copy Paste: [[2412.16604]] OmniSplat: Taming Feed-Forward 3D Gaussian Splatting for Omnidirectional Images with Editable Capabilities(https://arxiv.org/abs/2412.16604)
• Keywords: segmentation
• Abstract: Feed-forward 3D Gaussian Splatting (3DGS) models have gained significant popularity due to their ability to generate scenes immediately without needing per-scene optimization. Although omnidirectional images are getting more popular since they reduce the computation for image stitching to composite a holistic scene, existing feed-forward models are only designed for perspective images. The unique optical properties of omnidirectional images make it difficult for feature encoders to correctly understand the context of the image and make the Gaussian non-uniform in space, which hinders the image quality synthesized from novel views. We propose OmniSplat, a pioneering work for fast feed-forward 3DGS generation from a few omnidirectional images. We introduce Yin-Yang grid and decompose images based on it to reduce the domain gap between omnidirectional and perspective images. The Yin-Yang grid can use the existing CNN structure as it is, but its quasi-uniform characteristic allows the decomposed image to be similar to a perspective image, so it can exploit the strong prior knowledge of the learned feed-forward network. OmniSplat demonstrates higher reconstruction accuracy than existing feed-forward networks trained on perspective images. Furthermore, we enhance the segmentation consistency between omnidirectional images by leveraging attention from the encoder of OmniSplat, providing fast and clean 3DGS editing results.

Title: Improving Discovery of Known Software Vulnerability For Enhanced Cybersecurity

• Authors: Devesh Sawant, Manjesh K. Hanawal, Atul Kabra
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16607
• Pdf URL: https://arxiv.org/pdf/2412.16607
• Copy Paste: [[2412.16607]] Improving Discovery of Known Software Vulnerability For Enhanced Cybersecurity(https://arxiv.org/abs/2412.16607)
• Keywords: security, attack
• Abstract: Software vulnerabilities are commonly exploited as attack vectors in cyberattacks. Hence, it is crucial to identify vulnerable software configurations early to apply preventive measures. Effective vulnerability detection relies on identifying software vulnerabilities through standardized identifiers such as Common Platform Enumeration (CPE) strings. However, non-standardized CPE strings issued by software vendors create a significant challenge. Inconsistent formats, naming conventions, and versioning practices lead to mismatches when querying databases like the National Vulnerability Database (NVD), hindering accurate vulnerability detection. Failure to properly identify and prioritize vulnerable software complicates the patching process and causes delays in updating the vulnerable software, thereby giving attackers a window of opportunity. To address this, we present a method to enhance CPE string consistency by implementing a multi-layered sanitization process combined with a fuzzy matching algorithm on data collected using Osquery. Our method includes a union query with priority weighting, which assigns relevance to various attribute combinations, followed by a fuzzy matching process with threshold-based similarity scoring, yielding higher confidence in accurate matches. Comparative analysis with open-source tools such as FleetDM demonstrates that our approach improves detection accuracy by 40%.

Title: Concept Guided Co-saliency Objection Detection

• Authors: Jiayi Zhu, Qing Guo, Felix Juefei-Xu, Yihao Huang, Yang Liu, Geguang Pu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16609
• Pdf URL: https://arxiv.org/pdf/2412.16609
• Copy Paste: [[2412.16609]] Concept Guided Co-saliency Objection Detection(https://arxiv.org/abs/2412.16609)
• Keywords: segmentation
• Abstract: The task of co-saliency object detection (Co-SOD) seeks to identify common, salient objects across a collection of images by examining shared visual features. However, traditional Co-SOD methods often encounter limitations when faced with diverse object variations (e.g., different postures) and irrelevant background elements that introduce noise. To address these challenges, we propose ConceptCoSOD, a novel concept-guided approach that leverages text semantic information to enhance Co-SOD performance by guiding the model to focus on consistent object features. Through rethinking Co-SOD as an (image-text)-to-image task instead of an image-to-image task, ConceptCoSOD first captures shared semantic concepts within an image group and then uses them as guidance for precise object segmentation in complex scenarios. Experimental results on three benchmark datasets and six corruptions reveal that ConceptCoSOD significantly improves detection accuracy, especially in challenging settings with considerable background distractions and object variability.

Title: Automated Classification of Cybercrime Complaints using Transformer-based Language Models for Hinglish Texts

• Authors: Nanda Rani, Divyanshu Singh, Bikash Saha, Sandeep Kumar Shukla
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16614
• Pdf URL: https://arxiv.org/pdf/2412.16614
• Copy Paste: [[2412.16614]] Automated Classification of Cybercrime Complaints using Transformer-based Language Models for Hinglish Texts(https://arxiv.org/abs/2412.16614)
• Keywords: security, privacy, robust, transformer
• Abstract: The rise in cybercrime and the complexity of multilingual and code-mixed complaints present significant challenges for law enforcement and cybersecurity agencies. These organizations need automated, scalable methods to identify crime types, enabling efficient processing and prioritization of large complaint volumes. Manual triaging is inefficient, and traditional machine learning methods fail to capture the semantic and contextual nuances of textual cybercrime complaints. Moreover, the lack of publicly available datasets and privacy concerns hinder the research to present robust solutions. To address these challenges, we propose a framework for automated cybercrime complaint classification. The framework leverages Hinglish-adapted transformers, such as HingBERT and HingRoBERTa, to handle code-mixed inputs effectively. We employ the real-world dataset provided by Indian Cybercrime Coordination Centre (I4C) during CyberGuard AI Hackathon 2024. We employ GenAI open source model-based data augmentation method to address class imbalance. We also employ privacy-aware preprocessing to ensure compliance with ethical standards while maintaining data integrity. Our solution achieves significant performance improvements, with HingRoBERTa attaining an accuracy of 74.41% and an F1-score of 71.49%. We also develop ready-to-use tool by integrating Django REST backend with a modern frontend. The developed tool is scalable and ready for real-world deployment in platforms like the National Cyber Crime Reporting Portal. This work bridges critical gaps in cybercrime complaint management, offering a scalable, privacy-conscious, and adaptable solution for modern cybersecurity challenges.

Title: Automated Bleeding Detection and Classification in Wireless Capsule Endoscopy with YOLOv8-X

• Authors: Pavan C Shekar, Vivek Kanhangad, Shishir Maheshwari, T Sunil Kumar
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16624
• Pdf URL: https://arxiv.org/pdf/2412.16624
• Copy Paste: [[2412.16624]] Automated Bleeding Detection and Classification in Wireless Capsule Endoscopy with YOLOv8-X(https://arxiv.org/abs/2412.16624)
• Keywords: robust
• Abstract: Gastrointestinal (GI) bleeding, a critical indicator of digestive system disorders, re quires efficient and accurate detection methods. This paper presents our solution to the Auto-WCEBleedGen Version V1 Challenge, where we achieved the consolation position. We developed a unified YOLOv8-X model for both detection and classification of bleeding regions in Wireless Capsule Endoscopy (WCE) images. Our approach achieved 96.10% classification accuracy and 76.8% mean Average Precision (mAP) at 0.5 IoU on the val idation dataset. Through careful dataset curation and annotation, we assembled and trained on 6,345 diverse images to ensure robust model performance. Our implementa tion code and trained models are publicly available at this https URL.

Title: Fractional Spending: VRF&Ring Signatures As Efficient Primitives For Secret Quorums

• Authors: Maxence Perion, Sara Tucci-Piergiovanni, Rida Bazzi
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.16648
• Pdf URL: https://arxiv.org/pdf/2412.16648
• Copy Paste: [[2412.16648]] Fractional Spending: VRF&Ring Signatures As Efficient Primitives For Secret Quorums(https://arxiv.org/abs/2412.16648)
• Keywords: protect
• Abstract: Digital currencies have emerged as a significant evolution in the financial system, yet they face challenges in distributed settings, particularly regarding double spending. Traditional approaches, such as Bitcoin, use consensus to establish a total order of transactions, ensuring that no more than the currency held by an account is spent in the order. However, consensus protocols are costly, especially when coping with Byzantine faults. It was shown that solving Consensus is not needed to perform currency's transfer, for instance using byzantine quorum systems but validation remains per-account sequential. Recent research also introduced the fractional spending problem, which enables concurrent but non-conflicting transactions i.e., transactions that spend from the same account but cannot lead to a double spending because each is only spending a small fraction of the balance. A solution was proposed based on a new quorum system and specific cryptographic primitives to protect against an adaptive adversary. The quorum system, called (k1, k2)-quorum system, guarantees that at least k1 transactions can be validated concurrently but that no more than k2 can. Employing such quorums, a payer can validate concurrently multiple fractional spending transactions in parallel with high probability. Subsequently, the payer reclaims any remaining sum through a settlement. This paper enhances such solution by integrating different cryptographic primitives, VRF and Ring Signatures, into a similar protocol. But contrarily, these tools ensure quorums to remain secret during settlements, allowing to reduces its communication costs from cubic to quadratic in messages. We also achieve payment transaction with 3 message delays rather then 5. Additionally, we propose a refined formalization of the fractional spending problem, introducing coupons, which simplifies the theoretical framework and proof structure.

Title: PB-UAP: Hybrid Universal Adversarial Attack For Image Segmentation

• Authors: Yufei Song, Ziqi Zhou, Minghui Li, Xianlong Wang, Menghao Deng, Wei Wan, Shengshan Hu, Leo Yu Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16651
• Pdf URL: https://arxiv.org/pdf/2412.16651
• Copy Paste: [[2412.16651]] PB-UAP: Hybrid Universal Adversarial Attack For Image Segmentation(https://arxiv.org/abs/2412.16651)
• Keywords: attack, robust, segmentation
• Abstract: With the rapid advancement of deep learning, the model robustness has become a significant research hotspot, \ie, adversarial attacks on deep neural networks. Existing works primarily focus on image classification tasks, aiming to alter the model's predicted labels. Due to the output complexity and deeper network architectures, research on adversarial examples for segmentation models is still limited, particularly for universal adversarial perturbations. In this paper, we propose a novel universal adversarial attack method designed for segmentation models, which includes dual feature separation and low-frequency scattering modules. The two modules guide the training of adversarial examples in the pixel and frequency space, respectively. Experiments demonstrate that our method achieves high attack success rates surpassing the state-of-the-art methods, and exhibits strong transferability across different models.

Title: IV-tuning: Parameter-Efficient Transfer Learning for Infrared-Visible Tasks

• Authors: Yaming Zhang, Chenqiang Gao, Fangcen Liu, Junjie Guo, Lan Wang, Xinggan Peng, Deyu Meng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16654
• Pdf URL: https://arxiv.org/pdf/2412.16654
• Copy Paste: [[2412.16654]] IV-tuning: Parameter-Efficient Transfer Learning for Infrared-Visible Tasks(https://arxiv.org/abs/2412.16654)
• Keywords: segmentation
• Abstract: Infrared-visible (IR-VIS) tasks, such as semantic segmentation and object detection, greatly benefit from the advantage of combining infrared and visible modalities. To inherit the general representations of the Vision Foundation Models (VFMs), task-specific dual-branch networks are designed and fully fine-tuned on downstream datasets. Although effective, this manner lacks generality and is sub-optimal due to the scarcity of downstream infrared-visible datasets and limited transferability. In this paper, we propose a novel and general fine-tuning approach, namely "IV-tuning", to parameter-efficiently harness VFMs for various infrared-visible downstream tasks. At its core, IV-tuning freezes pre-trained visible-based VFMs and integrates modal-specific prompts with adapters within the backbone, bridging the gap between VFMs and downstream infrared-visible tasks while simultaneously learning the complementarity between different modalities. By fine-tuning approximately 3% of the backbone parameters, IV-tuning outperforms full fine-tuning across various baselines in infrared-visible semantic segmentation and object detection, as well as previous state-of-the-art methods. Extensive experiments across various settings demonstrate that IV-tuning achieves superior performance with fewer training parameters, providing a good alternative to full fine-tuning and a novel method of extending visible-based models for infrared-visible tasks. The code is available at this https URL.

Title: Generalizable Articulated Object Perception with Superpoints

• Authors: Qiaojun Yu, Ce Hao, Xibin Yuan, Li Zhang, Liu Liu, Yukang Huo, Rohit Agarwal, Cewu Lu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16656
• Pdf URL: https://arxiv.org/pdf/2412.16656
• Copy Paste: [[2412.16656]] Generalizable Articulated Object Perception with Superpoints(https://arxiv.org/abs/2412.16656)
• Keywords: transformer, segmentation
• Abstract: Manipulating articulated objects with robotic arms is challenging due to the complex kinematic structure, which requires precise part segmentation for efficient manipulation. In this work, we introduce a novel superpoint-based perception method designed to improve part segmentation in 3D point clouds of articulated objects. We propose a learnable, part-aware superpoint generation technique that efficiently groups points based on their geometric and semantic similarities, resulting in clearer part boundaries. Furthermore, by leveraging the segmentation capabilities of the 2D foundation model SAM, we identify the centers of pixel regions and select corresponding superpoints as candidate query points. Integrating a query-based transformer decoder further enhances our method's ability to achieve precise part segmentation. Experimental results on the GAPartNet dataset show that our method outperforms existing state-of-the-art approaches in cross-category part segmentation, achieving AP50 scores of 77.9% for seen categories (4.4% improvement) and $39.3\%$ for unseen categories (11.6% improvement), with superior results in 5 out of 9 part categories for seen objects and outperforming all previous methods across all part categories for unseen objects.

Title: Adversarial Attack Against Images Classification based on Generative Adversarial Networks

• Authors: Yahe Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16662
• Pdf URL: https://arxiv.org/pdf/2412.16662
• Copy Paste: [[2412.16662]] Adversarial Attack Against Images Classification based on Generative Adversarial Networks(https://arxiv.org/abs/2412.16662)
• Keywords: security, privacy, attack, generative
• Abstract: Adversarial attacks on image classification systems have always been an important problem in the field of machine learning, and generative adversarial networks (GANs), as popular models in the field of image generation, have been widely used in various novel scenarios due to their powerful generative capabilities. However, with the popularity of generative adversarial networks, the misuse of fake image technology has raised a series of security problems, such as malicious tampering with other people's photos and videos, and invasion of personal privacy. Inspired by the generative adversarial networks, this work proposes a novel adversarial attack method, aiming to gain insight into the weaknesses of the image classification system and improve its anti-attack ability. Specifically, the generative adversarial networks are used to generate adversarial samples with small perturbations but enough to affect the decision-making of the classifier, and the adversarial samples are generated through the adversarial learning of the training generator and the classifier. From extensive experiment analysis, we evaluate the effectiveness of the method on a classical image classification dataset, and the results show that our model successfully deceives a variety of advanced classifiers while maintaining the naturalness of adversarial samples.

Title: Transformer-based toxin-protein interaction analysis prioritizes airborne particulate matter components with potential adverse health effects

• Authors: Yan Zhu, Shihao Wang, Yong Han, Yao Lu, Shulan Qiu, Ling Jin, Xiangdong Li, Weixiong Zhang
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2412.16664
• Pdf URL: https://arxiv.org/pdf/2412.16664
• Copy Paste: [[2412.16664]] Transformer-based toxin-protein interaction analysis prioritizes airborne particulate matter components with potential adverse health effects(https://arxiv.org/abs/2412.16664)
• Keywords: transformer
• Abstract: Air pollution, particularly airborne particulate matter (PM), poses a significant threat to public health globally. It is crucial to comprehend the association between PM-associated toxic components and their cellular targets in humans to understand the mechanisms by which air pollution impacts health and to establish causal relationships between air pollution and public health consequences. Although many studies have explored the impact of PM on human health, the understanding of the association between toxins and the associated targets remain limited. Leveraging cutting-edge deep learning technologies, we developed tipFormer (toxin-protein interaction prediction based on transformer), a novel deep-learning tool for identifying toxic components capable of penetrating human cells and instigating pathogenic biological activities and signaling cascades. Experimental results show that tipFormer effectively captures interactions between proteins and toxic components. It incorporates dual pre-trained language models to encode protein sequences and chemicals. It employs a convolutional encoder to assimilate the sequential attributes of proteins and chemicals. It then introduces a learning module with a cross-attention mechanism to decode and elucidate the multifaceted interactions pivotal for the hotspots binding proteins and chemicals. Experimental results show that tipFormer effectively captures interactions between proteins and toxic components. This approach offers significant value to air quality and toxicology researchers by allowing high-throughput identification and prioritization of hazards. It supports more targeted laboratory studies and field measurements, ultimately enhancing our understanding of how air pollution impacts human health.

Title: Label Privacy in Split Learning for Large Models with Parameter-Efficient Training

• Authors: Philip Zmushko, Marat Mansurov, Ruslan Svirschevski, Denis Kuznedelev, Max Ryabinin, Aleksandr Beznosikov
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16669
• Pdf URL: https://arxiv.org/pdf/2412.16669
• Copy Paste: [[2412.16669]] Label Privacy in Split Learning for Large Models with Parameter-Efficient Training(https://arxiv.org/abs/2412.16669)
• Keywords: privacy, federate
• Abstract: As deep learning models become larger and more expensive, many practitioners turn to fine-tuning APIs. These web services allow fine-tuning a model between two parties: the client that provides the data, and the server that hosts the model. While convenient, these APIs raise a new concern: the data of the client is at risk of privacy breach during the training procedure. This challenge presents an important practical case of vertical federated learning, where the two parties perform parameter-efficient fine-tuning (PEFT) of a large model. In this study, we systematically search for a way to fine-tune models over an API while keeping the labels private. We analyze the privacy of LoRA, a popular approach for parameter-efficient fine-tuning when training over an API. Using this analysis, we propose P$^3$EFT, a multi-party split learning algorithm that takes advantage of existing PEFT properties to maintain privacy at a lower performance overhead. To validate our algorithm, we fine-tune DeBERTa-v2-XXLarge, Flan-T5 Large and LLaMA-2 7B using LoRA adapters on a range of NLP tasks. We find that P$^3$EFT is competitive with existing privacy-preserving methods in multi-party and two-party setups while having higher accuracy.

Title: Two-in-One: Unified Multi-Person Interactive Motion Generation by Latent Diffusion Transformer

• Authors: Boyuan Li, Xihua Wang, Ruihua Song, Wenbing Huang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.16670
• Pdf URL: https://arxiv.org/pdf/2412.16670
• Copy Paste: [[2412.16670]] Two-in-One: Unified Multi-Person Interactive Motion Generation by Latent Diffusion Transformer(https://arxiv.org/abs/2412.16670)
• Keywords: diffusion, transformer
• Abstract: Multi-person interactive motion generation, a critical yet under-explored domain in computer character animation, poses significant challenges such as intricate modeling of inter-human interactions beyond individual motions and generating two motions with huge differences from one text condition. Current research often employs separate module branches for individual motions, leading to a loss of interaction information and increased computational demands. To address these challenges, we propose a novel, unified approach that models multi-person motions and their interactions within a single latent space. Our approach streamlines the process by treating interactive motions as an integrated data point, utilizing a Variational AutoEncoder (VAE) for compression into a unified latent space, and performing a diffusion process within this space, guided by the natural language conditions. Experimental results demonstrate our method's superiority over existing approaches in generation quality, performing text condition in particular when motions have significant asymmetry, and accelerating the generation efficiency while preserving high quality.

Title: The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents

• Authors: Feiran Jia, Tong Wu, Xin Qin, Anna Squicciarini
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16682
• Pdf URL: https://arxiv.org/pdf/2412.16682
• Copy Paste: [[2412.16682]] The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents(https://arxiv.org/abs/2412.16682)
• Keywords: security, defense, attack, robust, large language model
• Abstract: Large Language Model (LLM) agents are increasingly being deployed as conversational assistants capable of performing complex real-world tasks through tool integration. This enhanced ability to interact with external systems and process various data sources, while powerful, introduces significant security vulnerabilities. In particular, indirect prompt injection attacks pose a critical threat, where malicious instructions embedded within external data sources can manipulate agents to deviate from user intentions. While existing defenses based on rule constraints, source spotlighting, and authentication protocols show promise, they struggle to maintain robust security while preserving task functionality. We propose a novel and orthogonal perspective that reframes agent security from preventing harmful actions to ensuring task alignment, requiring every agent action to serve user objectives. Based on this insight, we develop Task Shield, a test-time defense mechanism that systematically verifies whether each instruction and tool call contributes to user-specified goals. Through experiments on the AgentDojo benchmark, we demonstrate that Task Shield reduces attack success rates (2.07\%) while maintaining high task utility (69.79\%) on GPT-4o.

Title: NILE: Internal Consistency Alignment in Large Language Models

• Authors: Minda Hu, Qiyuan Zhang, Yufei Wang, Bowei He, Hongru Wang, Jingyan Zhou, Liangyou Li, Yasheng Wang, Chen Ma, Irwin King
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16686
• Pdf URL: https://arxiv.org/pdf/2412.16686
• Copy Paste: [[2412.16686]] NILE: Internal Consistency Alignment in Large Language Models(https://arxiv.org/abs/2412.16686)
• Keywords: large language model
• Abstract: As a crucial step to enhance LLMs alignment with human intentions, Instruction Fine-Tuning (IFT) has a high demand on dataset quality. However, existing IFT datasets often contain knowledge that is inconsistent with LLMs' internal knowledge learned from the pre-training phase, which can greatly affect the efficacy of IFT. To address this issue, we introduce NILE (iNternal consIstency aLignmEnt) framework, aimed at optimizing IFT datasets to unlock LLMs' capability further. NILE operates by eliciting target pre-trained LLM's internal knowledge corresponding to instruction data. The internal knowledge is leveraged to revise the answer in IFT datasets. Additionally, we propose a novel Internal Consistency Filtering (ICF) method to filter training samples, ensuring its high consistency with LLM's internal knowledge. Our experiments demonstrate that NILE-aligned IFT datasets sharply boost LLM performance across multiple LLM ability evaluation datasets, achieving up to 66.6% gain on Arena-Hard and 68.5% on Alpaca-Eval V2. Further analysis confirms that each component of the NILE}framework contributes to these substantial performance improvements, and provides compelling evidence that dataset consistency with pre-trained internal knowledge is pivotal for maximizing LLM potential.

Title: Subgoal Discovery Using a Free Energy Paradigm and State Aggregations

• Authors: Amirhossein Mesbah, Reshad Hosseini, Seyed Pooya Shariatpanahi, Majid Nili Ahmadabadi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16687
• Pdf URL: https://arxiv.org/pdf/2412.16687
• Copy Paste: [[2412.16687]] Subgoal Discovery Using a Free Energy Paradigm and State Aggregations(https://arxiv.org/abs/2412.16687)
• Keywords: robust
• Abstract: Reinforcement learning (RL) plays a major role in solving complex sequential decision-making tasks. Hierarchical and goal-conditioned RL are promising methods for dealing with two major problems in RL, namely sample inefficiency and difficulties in reward shaping. These methods tackle the mentioned problems by decomposing a task into simpler subtasks and temporally abstracting a task in the action space. One of the key components for task decomposition of these methods is subgoal discovery. We can use the subgoal states to define hierarchies of actions and also use them in decomposing complex tasks. Under the assumption that subgoal states are more unpredictable, we propose a free energy paradigm to discover them. This is achieved by using free energy to select between two spaces, the main space and an aggregation space. The $model \; changes$ from neighboring states to a given state shows the unpredictability of a given state, and therefore it is used in this paper for subgoal discovery. Our empirical results on navigation tasks like grid-world environments show that our proposed method can be applied for subgoal discovery without prior knowledge of the task. Our proposed method is also robust to the stochasticity of environments.

Title: CyberSentinel: Efficient Anomaly Detection in Programmable Switch using Knowledge Distillation

• Authors: Sankalp Mittal
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2412.16693
• Pdf URL: https://arxiv.org/pdf/2412.16693
• Copy Paste: [[2412.16693]] CyberSentinel: Efficient Anomaly Detection in Programmable Switch using Knowledge Distillation(https://arxiv.org/abs/2412.16693)
• Keywords: attack
• Abstract: The increasing volume of traffic (especially from IoT devices) is posing a challenge to the current anomaly detection systems. Existing systems are forced to take the support of the control plane for a more thorough and accurate detection of malicious traffic (anomalies). This introduces latency in making decisions regarding fast incoming traffic and therefore, existing systems are unable to scale to such growing rates of traffic. In this paper, we propose CyberSentinel, a high throughput and accurate anomaly detection system deployed entirely in the programmable switch data plane; making it the first work to accurately detect anomalies at line speed. To detect unseen network attacks, CyberSentinel uses a novel knowledge distillation scheme that incorporates "learned" knowledge of deep unsupervised ML models (\textit{e.g.}, autoencoders) to develop an iForest model that is then installed in the data plane in the form of whitelist rules. We implement a prototype of CyberSentinel on a testbed with an Intel Tofino switch and evaluate it on various real-world use cases. CyberSentinel yields similar detection performance compared to the state-of-the-art control plane solutions but with an increase in packet-processing throughput by $66.47\%$ on a $40$ Gbps link, and a reduction in average per-packet latency by $50\%$.

Title: DragonVerseQA: Open-Domain Long-Form Context-Aware Question-Answering

• Authors: Aritra Kumar Lahiri, Qinmin Vivian Hu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.16694
• Pdf URL: https://arxiv.org/pdf/2412.16694
• Copy Paste: [[2412.16694]] DragonVerseQA: Open-Domain Long-Form Context-Aware Question-Answering(https://arxiv.org/abs/2412.16694)
• Keywords: extraction
• Abstract: This paper proposes a novel approach to develop an open-domain and long-form Over-The-Top (OTT) Question-Answering (QA) dataset, DragonVerseQA, specifically oriented to the fantasy universe of "House of the Dragon" and "Game Of Thrones" TV series. Most existing QA datasets focus on short, fact-based answers sourced almost solely from Wikipedia articles, devoid of depth and contextual richness for sophisticated narrative understanding. We curate a dataset that combines full episode summaries sourced from HBO and fandom wiki websites, user reviews from sources like IMDb and Rotten Tomatoes, and high-quality, open-domain, legally admissible sources, and structured data from repositories like WikiData into one dataset. The dataset provides a multi-dimensional context, reflecting complex character dynamics and plot developments from these varied sources. That means, on equal footing, only after heavy data preprocessing and filtering methods will meaningful, non-spam unbiased reviews be available in this enriched dataset. The comprehensive insights are given through the long-form answers generated from this enriched context. This is what makes this valuable dataset for improving conversational AI, narrative analysis, sentiment analysis, summarization techniques, and relation extraction. A comparative analysis with state-of-the-art QA datasets such as SQuAD 2.0, TriviaQA, and Natural Questions brings to light the unique advantages of our dataset in terms of contextual complexity and answer length. Detailed reviews add layers to audience sentiment and narrative interpretation, raising the bar for domain-specific QA with a new quality benchmark. Our work also allows a deeper understanding of entertainment-industry content and opens the door to more knowledgeable and creative AI-driven interactions within digital media environments.

Title: TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models

• Authors: Haocheng Huang, Jiaxin Chen, Jinyang Guo, Ruiyi Zhan, Yunhong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16700
• Pdf URL: https://arxiv.org/pdf/2412.16700
• Copy Paste: [[2412.16700]] TCAQ-DM: Timestep-Channel Adaptive Quantization for Diffusion Models(https://arxiv.org/abs/2412.16700)
• Keywords: diffusion
• Abstract: Diffusion models have achieved remarkable success in the image and video generation tasks. Nevertheless, they often require a large amount of memory and time overhead during inference, due to the complex network architecture and considerable number of timesteps for iterative diffusion. Recently, the post-training quantization (PTQ) technique has proved a promising way to reduce the inference cost by quantizing the float-point operations to low-bit ones. However, most of them fail to tackle with the large variations in the distribution of activations across distinct channels and timesteps, as well as the inconsistent of input between quantization and inference on diffusion models, thus leaving much room for improvement. To address the above issues, we propose a novel method dubbed Timestep-Channel Adaptive Quantization for Diffusion Models (TCAQ-DM). Specifically, we develop a timestep-channel joint reparameterization (TCR) module to balance the activation range along both the timesteps and channels, facilitating the successive reconstruction procedure. Subsequently, we employ a dynamically adaptive quantization (DAQ) module that mitigate the quantization error by selecting an optimal quantizer for each post-Softmax layers according to their specific types of distributions. Moreover, we present a progressively aligned reconstruction (PAR) strategy to mitigate the bias caused by the input mismatch. Extensive experiments on various benchmarks and distinct diffusion models demonstrate that the proposed method substantially outperforms the state-of-the-art approaches in most cases, especially yielding comparable FID metrics to the full precision model on CIFAR-10 in the W6A6 setting, while enabling generating available images in the W4A4 settings.

Title: From Histopathology Images to Cell Clouds: Learning Slide Representations with Hierarchical Cell Transformer

• Authors: Zijiang Yang, Zhongwei Qiu, Tiancheng Lin, Hanqing Chao, Wanxing Chang, Yelin Yang, Yunshuo Zhang, Wenpei Jiao, Yixuan Shen, Wenbin Liu, Dongmei Fu, Dakai Jin, Ke Yan, Le Lu, Hui Jiang, Yun Bian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16715
• Pdf URL: https://arxiv.org/pdf/2412.16715
• Copy Paste: [[2412.16715]] From Histopathology Images to Cell Clouds: Learning Slide Representations with Hierarchical Cell Transformer(https://arxiv.org/abs/2412.16715)
• Keywords: transformer
• Abstract: It is clinically crucial and potentially very beneficial to be able to analyze and model directly the spatial distributions of cells in histopathology whole slide images (WSI). However, most existing WSI datasets lack cell-level annotations, owing to the extremely high cost over giga-pixel images. Thus, it remains an open question whether deep learning models can directly and effectively analyze WSIs from the semantic aspect of cell distributions. In this work, we construct a large-scale WSI dataset with more than 5 billion cell-level annotations, termed WSI-Cell5B, and a novel hierarchical Cell Cloud Transformer (CCFormer) to tackle these challenges. WSI-Cell5B is based on 6,998 WSIs of 11 cancers from The Cancer Genome Atlas Program, and all WSIs are annotated per cell by coordinates and types. To the best of our knowledge, WSI-Cell5B is the first WSI-level large-scale dataset integrating cell-level annotations. On the other hand, CCFormer formulates the collection of cells in each WSI as a cell cloud and models cell spatial distribution. Specifically, Neighboring Information Embedding (NIE) is proposed to characterize the distribution of cells within the neighborhood of each cell, and a novel Hierarchical Spatial Perception (HSP) module is proposed to learn the spatial relationship among cells in a bottom-up manner. The clinical analysis indicates that WSI-Cell5B can be used to design clinical evaluation metrics based on counting cells that effectively assess the survival risk of patients. Extensive experiments on survival prediction and cancer staging show that learning from cell spatial distribution alone can already achieve state-of-the-art (SOTA) performance, i.e., CCFormer strongly outperforms other competing methods.

Title: GANFusion: Feed-Forward Text-to-3D with Diffusion in GAN Space

• Authors: Souhaib Attaiki, Paul Guerrero, Duygu Ceylan, Niloy J. Mitra, Maks Ovsjanikov
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16717
• Pdf URL: https://arxiv.org/pdf/2412.16717
• Copy Paste: [[2412.16717]] GANFusion: Feed-Forward Text-to-3D with Diffusion in GAN Space(https://arxiv.org/abs/2412.16717)
• Keywords: diffusion, generative
• Abstract: We train a feed-forward text-to-3D diffusion generator for human characters using only single-view 2D data for supervision. Existing 3D generative models cannot yet match the fidelity of image or video generative models. State-of-the-art 3D generators are either trained with explicit 3D supervision and are thus limited by the volume and diversity of existing 3D data. Meanwhile, generators that can be trained with only 2D data as supervision typically produce coarser results, cannot be text-conditioned, or must revert to test-time optimization. We observe that GAN- and diffusion-based generators have complementary qualities: GANs can be trained efficiently with 2D supervision to produce high-quality 3D objects but are hard to condition on text. In contrast, denoising diffusion models can be conditioned efficiently but tend to be hard to train with only 2D supervision. We introduce GANFusion, which starts by generating unconditional triplane features for 3D data using a GAN architecture trained with only single-view 2D data. We then generate random samples from the GAN, caption them, and train a text-conditioned diffusion model that directly learns to sample from the space of good triplane features that can be decoded into 3D objects.

Title: Large Language Models Compression via Low-Rank Feature Distillation

• Authors: Yaya Sy, Christophe Cerisara, Irina Illina
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16719
• Pdf URL: https://arxiv.org/pdf/2412.16719
• Copy Paste: [[2412.16719]] Large Language Models Compression via Low-Rank Feature Distillation(https://arxiv.org/abs/2412.16719)
• Keywords: transformer, large language model
• Abstract: Current LLM structured pruning methods involve two steps: (1) compressing with calibration data and (2) continued pretraining on billions of tokens to recover the lost performance. This costly second step is needed as the first step significantly impacts performance. Previous studies have found that pretrained Transformer weights aren't inherently low-rank, unlike their activations, which may explain this performance drop. Based on this observation, we introduce a one-shot compression method that locally distills low-rank weights. We accelerate convergence by initializing the low-rank weights with SVD and using a joint loss that combines teacher and student activations. We reduce memory requirements by applying local gradient updates only. Our approach can compress Mixtral-8x7B within minutes on a single A100 GPU, removing 10 billion parameters while maintaining over 95% of the original performance. Phi-2 3B can be compressed by 40% using only 13 million calibration tokens into a small model that competes with recent models of similar size. We show our method generalizes well to non-transformer architectures: Mamba-3B can be compressed by 20% while maintaining 99% of its performance.

Title: Divide and Conquer: Grounding a Bleeding Areas in Gastrointestinal Image with Two-Stage Model

• Authors: Yu-Fan Lin, Bo-Cheng Qiu, Chia-Ming Lee, Chih-Chung Hsu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16723
• Pdf URL: https://arxiv.org/pdf/2412.16723
• Copy Paste: [[2412.16723]] Divide and Conquer: Grounding a Bleeding Areas in Gastrointestinal Image with Two-Stage Model(https://arxiv.org/abs/2412.16723)
• Keywords: robust, segmentation
• Abstract: Accurate detection and segmentation of gastrointestinal bleeding are critical for diagnosing diseases such as peptic ulcers and colorectal cancer. This study proposes a two-stage framework that decouples classification and grounding to address the inherent challenges posed by traditional Multi-Task Learning models, which jointly optimizes classification and segmentation. Our approach separates these tasks to achieve targeted optimization for each. The model first classifies images as bleeding or non-bleeding, thereby isolating subsequent grounding from inter-task interference and label heterogeneity. To further enhance performance, we incorporate Stochastic Weight Averaging and Test-Time Augmentation, which improve model robustness against domain shifts and annotation inconsistencies. Our method is validated on the Auto-WCEBleedGen Challenge V2 Challenge dataset and achieving second place. Experimental results demonstrate significant improvements in classification accuracy and segmentation precision, especially on sequential datasets with consistent visual patterns. This study highlights the practical benefits of a two-stage strategy for medical image analysis and sets a new standard for GI bleeding detection and segmentation. Our code is publicly available at this GitHub repository.

Title: LUCES-MV: A Multi-View Dataset for Near-Field Point Light Source Photometric Stereo

• Authors: Fotios Logothetis, Ignas Budvytis, Stephan Liwicki, Roberto Cipolla
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16737
• Pdf URL: https://arxiv.org/pdf/2412.16737
• Copy Paste: [[2412.16737]] LUCES-MV: A Multi-View Dataset for Near-Field Point Light Source Photometric Stereo(https://arxiv.org/abs/2412.16737)
• Keywords: robust
• Abstract: The biggest improvements in Photometric Stereo (PS) field has recently come from adoption of differentiable volumetric rendering techniques such as NeRF or Neural SDF achieving impressive reconstruction error of 0.2mm on DiLiGenT-MV benchmark. However, while there are sizeable datasets for environment lit objects such as Digital Twin Catalogue (DTS), there are only several small Photometric Stereo datasets which often lack challenging objects (simple, smooth, untextured) and practical, small form factor (near-field) light setup. To address this, we propose LUCES-MV, the first real-world, multi-view dataset designed for near-field point light source photometric stereo. Our dataset includes 15 objects with diverse materials, each imaged under varying light conditions from an array of 15 LEDs positioned 30 to 40 centimeters from the camera center. To facilitate transparent end-to-end evaluation, our dataset provides not only ground truth normals and ground truth object meshes and poses but also light and camera calibration images. We evaluate state-of-the-art near-field photometric stereo algorithms, highlighting their strengths and limitations across different material and shape complexities. LUCES-MV dataset offers an important benchmark for developing more robust, accurate and scalable real-world Photometric Stereo based 3D reconstruction methods.

Title: KKANs: Kurkova-Kolmogorov-Arnold Networks and Their Learning Dynamics

• Authors: Juan Diego Toscano, Li-Lian Wang, George Em Karniadakis
• Subjects: cs.LG, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.16738
• Pdf URL: https://arxiv.org/pdf/2412.16738
• Copy Paste: [[2412.16738]] KKANs: Kurkova-Kolmogorov-Arnold Networks and Their Learning Dynamics(https://arxiv.org/abs/2412.16738)
• Keywords: robust, diffusion
• Abstract: Inspired by the Kolmogorov-Arnold representation theorem and Kurkova's principle of using approximate representations, we propose the Kurkova-Kolmogorov-Arnold Network (KKAN), a new two-block architecture that combines robust multi-layer perceptron (MLP) based inner functions with flexible linear combinations of basis functions as outer functions. We first prove that KKAN is a universal approximator, and then we demonstrate its versatility across scientific machine-learning applications, including function regression, physics-informed machine learning (PIML), and operator-learning frameworks. The benchmark results show that KKANs outperform MLPs and the original Kolmogorov-Arnold Networks (KANs) in function approximation and operator learning tasks and achieve performance comparable to fully optimized MLPs for PIML. To better understand the behavior of the new representation models, we analyze their geometric complexity and learning dynamics using information bottleneck theory, identifying three universal learning stages, fitting, transition, and diffusion, across all types of architectures. We find a strong correlation between geometric complexity and signal-to-noise ratio (SNR), with optimal generalization achieved during the diffusion stage. Additionally, we propose self-scaled residual-based attention weights to maintain high SNR dynamically, ensuring uniform convergence and prolonged learning.

Title: Solving Inverse Problems via Diffusion Optimal Control

• Authors: Henry Li, Marcus Pereira
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16748
• Pdf URL: https://arxiv.org/pdf/2412.16748
• Copy Paste: [[2412.16748]] Solving Inverse Problems via Diffusion Optimal Control(https://arxiv.org/abs/2412.16748)
• Keywords: diffusion, generative
• Abstract: Existing approaches to diffusion-based inverse problem solvers frame the signal recovery task as a probabilistic sampling episode, where the solution is drawn from the desired posterior distribution. This framework suffers from several critical drawbacks, including the intractability of the conditional likelihood function, strict dependence on the score network approximation, and poor $\mathbf{x}_0$ prediction quality. We demonstrate that these limitations can be sidestepped by reframing the generative process as a discrete optimal control episode. We derive a diffusion-based optimal controller inspired by the iterative Linear Quadratic Regulator (iLQR) algorithm. This framework is fully general and able to handle any differentiable forward measurement operator, including super-resolution, inpainting, Gaussian deblurring, nonlinear deblurring, and even highly nonlinear neural classifiers. Furthermore, we show that the idealized posterior sampling equation can be recovered as a special case of our algorithm. We then evaluate our method against a selection of neural inverse problem solvers, and establish a new baseline in image reconstruction with inverse problems.

Title: SoK: Understanding the Attack Surface in Device Driver Isolation Frameworks

• Authors: Yongzhe Huang, Kaiming Huang, Matthew Ennis, Vikram Narayanan, Anton Burtsev, Trent Jaeger, Gang Tan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16754
• Pdf URL: https://arxiv.org/pdf/2412.16754
• Copy Paste: [[2412.16754]] SoK: Understanding the Attack Surface in Device Driver Isolation Frameworks(https://arxiv.org/abs/2412.16754)
• Keywords: security, protect, attack
• Abstract: Device driver isolation is a promising approach for protecting the kernel from faulty or malicious drivers, but the actual security provided by such frameworks is often not well understood. Recent research has identified Compartment Interface Vulnerabilities (CIVs) in userspace compartmentalized applications, yet their impact on driver isolation frameworks remains poorly understood. This paper provides a comprehensive survey of the design and security guarantees of existing driver isolation frameworks and systemizes existing CIV classifications, evaluating them under driver isolation. The analysis shows that different classes of CIVs are prevalent across the studied drivers under a baseline threat model, with large drivers having more than 100 instances of different CIVs and an average of 33 instances across the studied drivers. Enforcing extra security properties, such as CFI, can reduce the number of CIVs to around 28 instances on average. This study provides insights for understanding existing driver isolation security and the prevalence of CIVs in the driver isolation context, and extracts useful insights that can provide security guidance for future driver isolation systems.

Title: Paraformer: Parameterization of Sub-grid Scale Processes Using Transformers

• Authors: Shuochen Wang, Nishant Yadav, Auroop R. Ganguly
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2412.16763
• Pdf URL: https://arxiv.org/pdf/2412.16763
• Copy Paste: [[2412.16763]] Paraformer: Parameterization of Sub-grid Scale Processes Using Transformers(https://arxiv.org/abs/2412.16763)
• Keywords: transformer
• Abstract: One of the major sources of uncertainty in the current generation of Global Climate Models (GCMs) is the representation of sub-grid scale physical processes. Over the years, a series of deep-learning-based parameterization schemes have been developed and tested on both idealized and real-geography GCMs. However, datasets on which previous deep-learning models were trained either contain limited variables or have low spatial-temporal coverage, which can not fully simulate the parameterization process. Additionally, these schemes rely on classical architectures while the latest attention mechanism used in Transformer models remains unexplored in this field. In this paper, we propose Paraformer, a "memory-aware" Transformer-based model on ClimSim, the largest dataset ever created for climate parameterization. Our results demonstrate that the proposed model successfully captures the complex non-linear dependencies in the sub-grid scale variables and outperforms classical deep-learning architectures. This work highlights the applicability of the attenuation mechanism in this field and provides valuable insights for developing future deep-learning-based climate parameterization schemes.

Title: Does calibration mean what they say it means; or, the reference class problem rises again

• Authors: Lily Hu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16769
• Pdf URL: https://arxiv.org/pdf/2412.16769
• Copy Paste: [[2412.16769]] Does calibration mean what they say it means; or, the reference class problem rises again(https://arxiv.org/abs/2412.16769)
• Keywords: fair
• Abstract: Discussions of statistical criteria for fairness commonly convey the normative significance of calibration within groups by invoking what risk scores "mean." On the Same Meaning picture, group-calibrated scores "mean the same thing" (on average) across individuals from different groups and accordingly, guard against disparate treatment of individuals based on group membership. My contention is that calibration guarantees no such thing. Since concrete actual people belong to many groups, calibration cannot ensure the kind of consistent score interpretation that the Same Meaning picture implies matters for fairness, unless calibration is met within every group to which an individual belongs. Alas only perfect predictors may meet this bar. The Same Meaning picture thus commits a reference class fallacy by inferring from calibration within some group to the "meaning" or evidential value of an individual's score, because they are a member of that group. Furthermore, the reference class answer it presumes is almost surely wrong. I then show that the reference class problem besets not just calibration but all group statistical facts that claim a close connection to fairness. Reflecting on the origins of this error opens a wider lens onto the predominant methodology in algorithmic fairness based on stylized cases.

Title: RoomPainter: View-Integrated Diffusion for Consistent Indoor Scene Texturing

• Authors: Zhipeng Huang, Wangbo Yu, Xinhua Cheng, ChengShu Zhao, Yunyang Ge, Mingyi Guo, Li Yuan, Yonghong Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16778
• Pdf URL: https://arxiv.org/pdf/2412.16778
• Copy Paste: [[2412.16778]] RoomPainter: View-Integrated Diffusion for Consistent Indoor Scene Texturing(https://arxiv.org/abs/2412.16778)
• Keywords: diffusion
• Abstract: Indoor scene texture synthesis has garnered significant interest due to its important potential applications in virtual reality, digital media, and creative arts. Existing diffusion model-based researches either rely on per-view inpainting techniques, which are plagued by severe cross-view inconsistencies and conspicuous seams, or they resort to optimization-based approaches that entail substantial computational overhead. In this work, we present RoomPainter, a framework that seamlessly integrates efficiency and consistency to achieve high-fidelity texturing of indoor scenes. The core of RoomPainter features a zero-shot technique that effectively adapts a 2D diffusion model for 3D-consistent texture synthesis, along with a two-stage generation strategy that ensures both global and local consistency. Specifically, we introduce Attention-Guided Multi-View Integrated Sampling (MVIS) combined with a neighbor-integrated attention mechanism for zero-shot texture map generation. Using the MVIS, we firstly generate texture map for the entire room to ensure global consistency, then adopt its variant, namely an attention-guided multi-view integrated repaint sampling (MVRS) to repaint individual instances within the room, thereby further enhancing local consistency. Experiments demonstrate that RoomPainter achieves superior performance for indoor scene texture synthesis in visual quality, global consistency, and generation efficiency.

Title: Fed-ZOE: Communication-Efficient Over-the-Air Federated Learning via Zeroth-Order Estimation

• Authors: Jonggyu Jang, Hyeonsu Lyu, David J. Love, Hyun Jong Yang
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2412.16779
• Pdf URL: https://arxiv.org/pdf/2412.16779
• Copy Paste: [[2412.16779]] Fed-ZOE: Communication-Efficient Over-the-Air Federated Learning via Zeroth-Order Estimation(https://arxiv.org/abs/2412.16779)
• Keywords: secure, federate
• Abstract: As 6G and beyond networks grow increasingly complex and interconnected, federated learning (FL) emerges as an indispensable paradigm for securely and efficiently leveraging decentralized edge data for AI. By virtue of the superposition property of communication signals, over-the-air FL (OtA-FL) achieves constant communication overhead irrespective of the number of edge devices (EDs). However, training neural networks over the air still incurs substantial communication costs, as the number of transmitted symbols equals the number of trainable parameters. To alleviate this issue, the most straightforward approach is to reduce the number of transmitted symbols by 1) gradient compression and 2) gradient sparsification. Unfortunately, these methods are incompatible with OtA-FL due to the loss of its superposition property. In this work, we introduce federated zeroth-order estimation (Fed-ZOE), an efficient framework inspired by the randomized gradient estimator (RGE) commonly used in zeroth-order optimization (ZOO). In FedZOE, EDs perform local weight updates as in standard FL, but instead of transmitting full gradient vectors, they send compressed local model update vectors in the form of several scalar-valued inner products between the local model update vectors and random vectors. These scalar values enable the parameter server (PS) to reconstruct the gradient using the RGE trick with highly reduced overhead, as well as preserving the superposition property. Unlike conventional ZOO leveraging RGE for step-wise gradient descent, Fed-ZOE compresses local model update vectors before transmission, thereby achieving higher accuracy and computational efficiency. Numerical evaluations using ResNet-18 on datasets such as CIFAR-10, TinyImageNet, SVHN, CIFAR-100, and Brain-CT demonstrate that Fed-ZOE achieves performance comparable to Fed-OtA while drastically reducing communication costs.

Title: SubData: A Python Library to Collect and Combine Datasets for Evaluating LLM Alignment on Downstream Tasks

• Authors: Leon Fröhling, Pietro Bernardelle, Gianluca Demartini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16783
• Pdf URL: https://arxiv.org/pdf/2412.16783
• Copy Paste: [[2412.16783]] SubData: A Python Library to Collect and Combine Datasets for Evaluating LLM Alignment on Downstream Tasks(https://arxiv.org/abs/2412.16783)
• Keywords: large language model
• Abstract: With the release of ever more capable large language models (LLMs), researchers in NLP and related disciplines have started to explore the usability of LLMs for a wide variety of different annotation tasks. Very recently, a lot of this attention has shifted to tasks that are subjective in nature. Given that the latest generations of LLMs have digested and encoded extensive knowledge about different human subpopulations and individuals, the hope is that these models can be trained, tuned or prompted to align with a wide range of different human perspectives. While researchers already evaluate the success of this alignment via surveys and tests, there is a lack of resources to evaluate the alignment on what oftentimes matters the most in NLP; the actual downstream tasks. To fill this gap we present SubData, a Python library that offers researchers working on topics related to subjectivity in annotation tasks a convenient way of collecting, combining and using a range of suitable datasets.

Title: Enhancing web traffic attacks identification through ensemble methods and feature selection

• Authors: Daniel Urda, Branly Martínez, Nuño Basurto, Meelis Kull, Ángel Arroyo, Álvaro Herrero
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16791
• Pdf URL: https://arxiv.org/pdf/2412.16791
• Copy Paste: [[2412.16791]] Enhancing web traffic attacks identification through ensemble methods and feature selection(https://arxiv.org/abs/2412.16791)
• Keywords: attack, robust
• Abstract: Websites, as essential digital assets, are highly vulnerable to cyberattacks because of their high traffic volume and the significant impact of breaches. This study aims to enhance the identification of web traffic attacks by leveraging machine learning techniques. A methodology was proposed to extract relevant features from HTTP traces using the CSIC2010 v2 dataset, which simulates e-commerce web traffic. Ensemble methods, such as Random Forest and Extreme Gradient Boosting, were employed and compared against baseline classifiers, including k-nearest Neighbor, LASSO, and Support Vector Machines. The results demonstrate that the ensemble methods outperform baseline classifiers by approximately 20% in predictive accuracy, achieving an Area Under the ROC Curve (AUC) of 0.989. Feature selection methods such as Information Gain, LASSO, and Random Forest further enhance the robustness of these models. This study highlights the efficacy of ensemble models in improving attack detection while minimizing performance variability, offering a practical framework for securing web traffic in diverse application contexts.

Title: Balls-and-Bins Sampling for DP-SGD

• Authors: Lynn Chua, Badih Ghazi, Charlie Harrison, Ethan Leeman, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang
• Subjects: cs.LG, cs.CR, cs.DS, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.16802
• Pdf URL: https://arxiv.org/pdf/2412.16802
• Copy Paste: [[2412.16802]] Balls-and-Bins Sampling for DP-SGD(https://arxiv.org/abs/2412.16802)
• Keywords: privacy
• Abstract: We introduce the Balls-and-Bins sampling for differentially private (DP) optimization methods such as DP-SGD. While it has been common practice to use some form of shuffling in DP-SGD implementations, privacy accounting algorithms have typically assumed that Poisson subsampling is used instead. Recent work by Chua et al. (ICML 2024) however pointed out that shuffling based DP-SGD can have a much larger privacy cost in practical regimes of parameters. We show that the Balls-and-Bins sampling achieves the "best-of-both" samplers, namely, the implementation of Balls-and-Bins sampling is similar to that of Shuffling and models trained using DP-SGD with Balls-and-Bins sampling achieve utility comparable to those trained using DP-SGD with Shuffling at the same noise multiplier, and yet, Balls-and-Bins sampling enjoys similar-or-better privacy amplification as compared to Poisson subsampling in practical regimes.

Title: Quantum-Like Contextuality in Large Language Models

• Authors: Kin Ian Lo, Mehrnoosh Sadrzadeh, Shane Mansfield
• Subjects: cs.CL, quant-ph
• Abstract URL: https://arxiv.org/abs/2412.16806
• Pdf URL: https://arxiv.org/pdf/2412.16806
• Copy Paste: [[2412.16806]] Quantum-Like Contextuality in Large Language Models(https://arxiv.org/abs/2412.16806)
• Keywords: large language model
• Abstract: Contextuality is a distinguishing feature of quantum mechanics and there is growing evidence that it is a necessary condition for quantum advantage. In order to make use of it, researchers have been asking whether similar phenomena arise in other domains. The answer has been yes, e.g. in behavioural sciences. However, one has to move to frameworks that take some degree of signalling into account. Two such frameworks exist: (1) a signalling-corrected sheaf theoretic model, and (2) the Contextuality-by-Default (CbD) framework. This paper provides the first large scale experimental evidence for a yes answer in natural language. We construct a linguistic schema modelled over a contextual quantum scenario, instantiate it in the Simple English Wikipedia and extract probability distributions for the instances using the large language model BERT. This led to the discovery of 77,118 sheaf-contextual and 36,938,948 CbD contextual instances. We proved that the contextual instances came from semantically similar words, by deriving an equation between degrees of contextuality and Euclidean distances of BERT's embedding vectors. A regression model further reveals that Euclidean distance is indeed the best statistical predictor of contextuality. Our linguistic schema is a variant of the co-reference resolution challenge. These results are an indication that quantum methods may be advantageous in language tasks.

Title: Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers

• Authors: Haoran You, Connelly Barnes, Yuqian Zhou, Yan Kang, Zhenbang Du, Wei Zhou, Lingzhi Zhang, Yotam Nitzan, Xiaoyang Liu, Zhe Lin, Eli Shechtman, Sohrab Amirghodsi, Yingyan Celine Lin
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16822
• Pdf URL: https://arxiv.org/pdf/2412.16822
• Copy Paste: [[2412.16822]] Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers(https://arxiv.org/abs/2412.16822)
• Keywords: diffusion, transformer
• Abstract: Diffusion Transformers (DiTs) have achieved state-of-the-art (SOTA) image generation quality but suffer from high latency and memory inefficiency, making them difficult to deploy on resource-constrained devices. One key efficiency bottleneck is that existing DiTs apply equal computation across all regions of an image. However, not all image tokens are equally important, and certain localized areas require more computation, such as objects. To address this, we propose DiffRatio-MoD, a dynamic DiT inference framework with differentiable compression ratios, which automatically learns to dynamically route computation across layers and timesteps for each image token, resulting in Mixture-of-Depths (MoD) efficient DiT models. Specifically, DiffRatio-MoD integrates three features: (1) A token-level routing scheme where each DiT layer includes a router that is jointly fine-tuned with model weights to predict token importance scores. In this way, unimportant tokens bypass the entire layer's computation; (2) A layer-wise differentiable ratio mechanism where different DiT layers automatically learn varying compression ratios from a zero initialization, resulting in large compression ratios in redundant layers while others remain less compressed or even uncompressed; (3) A timestep-wise differentiable ratio mechanism where each denoising timestep learns its own compression ratio. The resulting pattern shows higher ratios for noisier timesteps and lower ratios as the image becomes clearer. Extensive experiments on both text-to-image and inpainting tasks show that DiffRatio-MoD effectively captures dynamism across token, layer, and timestep axes, achieving superior trade-offs between generation quality and efficiency compared to prior works.

Title: RealisID: Scale-Robust and Fine-Controllable Identity Customization via Local and Global Complementation

• Authors: Zhaoyang Sun, Fei Du, Weihua Chen, Fan Wang, Yaxiong Chen, Yi Rong, Shengwu Xiong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16832
• Pdf URL: https://arxiv.org/pdf/2412.16832
• Copy Paste: [[2412.16832]] RealisID: Scale-Robust and Fine-Controllable Identity Customization via Local and Global Complementation(https://arxiv.org/abs/2412.16832)
• Keywords: robust
• Abstract: Recently, the success of text-to-image synthesis has greatly advanced the development of identity customization techniques, whose main goal is to produce realistic identity-specific photographs based on text prompts and reference face images. However, it is difficult for existing identity customization methods to simultaneously meet the various requirements of different real-world applications, including the identity fidelity of small face, the control of face location, pose and expression, as well as the customization of multiple persons. To this end, we propose a scale-robust and fine-controllable method, namely RealisID, which learns different control capabilities through the cooperation between a pair of local and global branches. Specifically, by using cropping and up-sampling operations to filter out face-irrelevant information, the local branch concentrates the fine control of facial details and the scale-robust identity fidelity within the face region. Meanwhile, the global branch manages the overall harmony of the entire image. It also controls the face location by taking the location guidance as input. As a result, RealisID can benefit from the complementarity of these two branches. Finally, by implementing our branches with two different variants of ControlNet, our method can be easily extended to handle multi-person customization, even only trained on single-person datasets. Extensive experiments and ablation studies indicate the effectiveness of RealisID and verify its ability in fulfilling all the requirements mentioned above.

Title: Ask-Before-Detection: Identifying and Mitigating Conformity Bias in LLM-Powered Error Detector for Math Word Problem Solutions

• Authors: Hang Li, Tianlong Xu, Kaiqi Yang, Yucheng Chu, Yanling Chen, Yichi Song, Qingsong Wen, Hui Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16838
• Pdf URL: https://arxiv.org/pdf/2412.16838
• Copy Paste: [[2412.16838]] Ask-Before-Detection: Identifying and Mitigating Conformity Bias in LLM-Powered Error Detector for Math Word Problem Solutions(https://arxiv.org/abs/2412.16838)
• Keywords: large language model
• Abstract: The rise of large language models (LLMs) offers new opportunities for automatic error detection in education, particularly for math word problems (MWPs). While prior studies demonstrate the promise of LLMs as error detectors, they overlook the presence of multiple valid solutions for a single MWP. Our preliminary analysis reveals a significant performance gap between conventional and alternative solutions in MWPs, a phenomenon we term conformity bias in this work. To mitigate this bias, we introduce the Ask-Before-Detect (AskBD) framework, which generates adaptive reference solutions using LLMs to enhance error detection. Experiments on 200 examples of GSM8K show that AskBD effectively mitigates bias and improves performance, especially when combined with reasoning-enhancing techniques like chain-of-thought prompting.

Title: Human-Guided Image Generation for Expanding Small-Scale Training Image Datasets

• Authors: Changjian Chen, Fei Lv, Yalong Guan, Pengcheng Wang, Shengjie Yu, Yifan Zhang, Zhuo Tang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16839
• Pdf URL: https://arxiv.org/pdf/2412.16839
• Copy Paste: [[2412.16839]] Human-Guided Image Generation for Expanding Small-Scale Training Image Datasets(https://arxiv.org/abs/2412.16839)
• Keywords: generative
• Abstract: The performance of computer vision models in certain real-world applications (e.g., rare wildlife observation) is limited by the small number of available this http URL datasets using pre-trained generative models is an effective way to address this limitation. However, since the automatic generation process is uncontrollable, the generated images are usually limited in diversity, and some of them are undesired. In this paper, we propose a human-guided image generation method for more controllable dataset expansion. We develop a multi-modal projection method with theoretical guarantees to facilitate the exploration of both the original and generated images. Based on the exploration, users refine the prompts and re-generate images for better performance. Since directly refining the prompts is challenging for novice users, we develop a sample-level prompt refinement method to make it easier. With this method, users only need to provide sample-level feedback (e.g., which samples are undesired) to obtain better prompts. The effectiveness of our method is demonstrated through the quantitative evaluation of the multi-modal projection method, improved model performance in the case study for both classification and object detection tasks, and positive feedback from the experts.

Title: Seamless Detection: Unifying Salient Object Detection and Camouflaged Object Detection

• Authors: Yi Liu, Chengxin Li, Xiaohui Dong, Lei Li, Dingwen Zhang, Shoukun Xu, Jungong Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16840
• Pdf URL: https://arxiv.org/pdf/2412.16840
• Copy Paste: [[2412.16840]] Seamless Detection: Unifying Salient Object Detection and Camouflaged Object Detection(https://arxiv.org/abs/2412.16840)
• Keywords: segmentation
• Abstract: Achieving joint learning of Salient Object Detection (SOD) and Camouflaged Object Detection (COD) is extremely challenging due to their distinct object characteristics, i.e., saliency and camouflage. The only preliminary research treats them as two contradictory tasks, training models on large-scale labeled data alternately for each task and assessing them independently. However, such task-specific mechanisms fail to meet real-world demands for addressing unknown tasks effectively. To address this issue, in this paper, we pioneer a task-agnostic framework to unify SOD and COD. To this end, inspired by the agreeable nature of binary segmentation for SOD and COD, we propose a Contrastive Distillation Paradigm (CDP) to distil the foreground from the background, facilitating the identification of salient and camouflaged objects amidst their surroundings. To probe into the contribution of our CDP, we design a simple yet effective contextual decoder involving the interval-layer and global context, which achieves an inference speed of 67 fps. Besides the supervised setting, our CDP can be seamlessly integrated into unsupervised settings, eliminating the reliance on extensive human annotations. Experiments on public SOD and COD datasets demonstrate the superiority of our proposed framework in both supervised and unsupervised settings, compared with existing state-of-the-art approaches. Code is available on this https URL.

Title: Sim911: Towards Effective and Equitable 9-1-1 Dispatcher Training with an LLM-Enabled Simulation

• Authors: Zirong Chen, Elizabeth Chason, Noah Mladenovski, Erin Wilson, Kristin Mullen, Stephen Martini, Meiyi Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16844
• Pdf URL: https://arxiv.org/pdf/2412.16844
• Copy Paste: [[2412.16844]] Sim911: Towards Effective and Equitable 9-1-1 Dispatcher Training with an LLM-Enabled Simulation(https://arxiv.org/abs/2412.16844)
• Keywords: large language model
• Abstract: Emergency response services are vital for enhancing public safety by safeguarding the environment, property, and human lives. As frontline members of these services, 9-1-1 dispatchers have a direct impact on response times and the overall effectiveness of emergency operations. However, traditional dispatcher training methods, which rely on role-playing by experienced personnel, are labor-intensive, time-consuming, and often neglect the specific needs of underserved communities. To address these challenges, we introduce Sim911, the first training simulation for 9-1-1 dispatchers powered by Large Language Models (LLMs). Sim911 enhances training through three key technical innovations: (1) knowledge construction, which utilizes archived 9-1-1 call data to generate simulations that closely mirror real-world scenarios; (2) context-aware controlled generation, which employs dynamic prompts and vector bases to ensure that LLM behavior aligns with training objectives; and (3) validation with looped correction, which filters out low-quality responses and refines the system performance.

Title: GME: Improving Universal Multimodal Retrieval by Multimodal LLMs

• Authors: Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, Min Zhang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.16855
• Pdf URL: https://arxiv.org/pdf/2412.16855
• Copy Paste: [[2412.16855]] GME: Improving Universal Multimodal Retrieval by Multimodal LLMs(https://arxiv.org/abs/2412.16855)
• Keywords: large language model
• Abstract: Universal Multimodal Retrieval (UMR) aims to enable search across various modalities using a unified model, where queries and candidates can consist of pure text, images, or a combination of both. Previous work has attempted to adopt multimodal large language models (MLLMs) to realize UMR using only text data. However, our preliminary experiments demonstrate that more diverse multimodal training data can further unlock the potential of MLLMs. Despite its effectiveness, the existing multimodal training data is highly imbalanced in terms of modality, which motivates us to develop a training data synthesis pipeline and construct a large-scale, high-quality fused-modal training dataset. Based on the synthetic training data, we develop the General Multimodal Embedder (GME), an MLLM-based dense retriever designed for UMR. Furthermore, we construct a comprehensive UMR Benchmark (UMRB) to evaluate the effectiveness of our approach. Experimental results show that our method achieves state-of-the-art performance among existing UMR methods. Last, we provide in-depth analyses of model scaling, training strategies, and perform ablation studies on both the model and synthetic data.

Title: Adversarial Diffusion Model for Unsupervised Domain-Adaptive Semantic Segmentation

• Authors: Jongmin Yu, Zhongtian Sun, Shan Luo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16859
• Pdf URL: https://arxiv.org/pdf/2412.16859
• Copy Paste: [[2412.16859]] Adversarial Diffusion Model for Unsupervised Domain-Adaptive Semantic Segmentation(https://arxiv.org/abs/2412.16859)
• Keywords: diffusion, segmentation
• Abstract: Semantic segmentation requires labour-intensive labelling tasks to obtain the supervision signals, and because of this issue, it is encouraged that using domain adaptation, which transfers information from the existing labelled source domains to unlabelled or weakly labelled target domains, is essential. However, it is intractable to find a well-generalised representation which can describe two domains due to probabilistic or geometric difference between the two domains. This paper presents a novel method, the Conditional and Inter-coder Connected Latent Diffusion (CICLD) based Semantic Segmentation Model, to advance unsupervised domain adaptation (UDA) for semantic segmentation tasks. Leveraging the strengths of latent diffusion models and adversarial learning, our method effectively bridges the gap between synthetic and real-world imagery. CICLD incorporates a conditioning mechanism to improve contextual understanding during segmentation and an inter-coder connection to preserve fine-grained details and spatial hierarchies. Additionally, adversarial learning aligns latent feature distributions across source, mixed, and target domains, further enhancing generalisation. Extensive experiments are conducted across three benchmark datasets-GTA5, Synthia, and Cityscape-shows that CICLD outperforms state-of-the-art UDA methods. Notably, the proposed method achieves a mean Intersection over Union (mIoU) of 74.4 for the GTA5 to Cityscape UDA setting and 67.2 mIoU for the Synthia to Cityscape UDA setting. This project is publicly available on 'this https URL.

Title: CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models

• Authors: Yeyuan Wang, Dehong Gao, Bin Li, Rujiao Long, Lei Yi, Xiaoyan Cai, Libin Yang, Jinxia Zhang, Shanqing Yu, Qi Xuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16869
• Pdf URL: https://arxiv.org/pdf/2412.16869
• Copy Paste: [[2412.16869]] CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models(https://arxiv.org/abs/2412.16869)
• Keywords: large language model
• Abstract: The impressive performance of Large Language Model (LLM) has prompted researchers to develop Multi-modal LLM (MLLM), which has shown great potential for various multi-modal tasks. However, current MLLM often struggles to effectively address fine-grained multi-modal challenges. We argue that this limitation is closely linked to the models' visual grounding capabilities. The restricted spatial awareness and perceptual acuity of visual encoders frequently lead to interference from irrelevant background information in images, causing the models to overlook subtle but crucial details. As a result, achieving fine-grained regional visual comprehension becomes difficult. In this paper, we break down multi-modal understanding into two stages, from Coarse to Fine (CoF). In the first stage, we prompt the MLLM to locate the approximate area of the answer. In the second stage, we further enhance the model's focus on relevant areas within the image through visual prompt engineering, adjusting attention weights of pertinent regions. This, in turn, improves both visual grounding and overall performance in downstream tasks. Our experiments show that this approach significantly boosts the performance of baseline models, demonstrating notable generalization and effectiveness. Our CoF approach is available online at this https URL.

Title: Teaching LLMs to Refine with Tools

• Authors: Dian Yu, Yuheng Zhang, Jiahao Xu, Tian Liang, Linfeng Song, Zhaopeng Tu, Haitao Mi, Dong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16871
• Pdf URL: https://arxiv.org/pdf/2412.16871
• Copy Paste: [[2412.16871]] Teaching LLMs to Refine with Tools(https://arxiv.org/abs/2412.16871)
• Keywords: large language model
• Abstract: Large language models (LLMs) can refine their responses based on feedback, enabling self-improvement through iterative training or test-time refinement. However, existing methods predominantly focus on refinement within the same reasoning format, which may lead to non-correcting behaviors. We propose CaP, a novel approach that uses external tools to refine chain-of-thought (CoT) responses generated by the same or other LLMs. CaP employs a two-stage training process: supervised fine-tuning followed by preference optimization with DPO variants. Our observations highlight the critical role of preference optimization in enabling effective refinement. Additionally, we compare several sampling strategies to leverage CoT and tools at inference time. Experimental results demonstrate CaP's potential for effective cross-reasoning refinement and efficient inference.

Title: MAGIC++: Efficient and Resilient Modality-Agnostic Semantic Segmentation via Hierarchical Modality Selection

• Authors: Xu Zheng, Yuanhuiyi Lyu, Lutao Jiang, Jiazhou Zhou, Lin Wang, Xuming Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16876
• Pdf URL: https://arxiv.org/pdf/2412.16876
• Copy Paste: [[2412.16876]] MAGIC++: Efficient and Resilient Modality-Agnostic Semantic Segmentation via Hierarchical Modality Selection(https://arxiv.org/abs/2412.16876)
• Keywords: robust, segmentation
• Abstract: In this paper, we address the challenging modality-agnostic semantic segmentation (MaSS), aiming at centering the value of every modality at every feature granularity. Training with all available visual modalities and effectively fusing an arbitrary combination of them is essential for robust multi-modal fusion in semantic segmentation, especially in real-world scenarios, yet remains less explored to date. Existing approaches often place RGB at the center, treating other modalities as secondary, resulting in an asymmetric architecture. However, RGB alone can be limiting in scenarios like nighttime, where modalities such as event data excel. Therefore, a resilient fusion model must dynamically adapt to each modality's strengths while compensating for weaker this http URL this end, we introduce the MAGIC++ framework, which comprises two key plug-and-play modules for effective multi-modal fusion and hierarchical modality selection that can be equipped with various backbone models. Firstly, we introduce a multi-modal interaction module to efficiently process features from the input multi-modal batches and extract complementary scene information with channel-wise and spatial-wise guidance. On top, a unified multi-scale arbitrary-modal selection module is proposed to utilize the aggregated features as the benchmark to rank the multi-modal features based on the similarity scores at hierarchical feature spaces. This way, our method can eliminate the dependence on RGB modality at every feature granularity and better overcome sensor failures and environmental noises while ensuring the segmentation performance. Under the common multi-modal setting, our method achieves state-of-the-art performance on both real-world and synthetic benchmarks. Moreover, our method is superior in the novel modality-agnostic setting, where it outperforms prior arts by a large margin.

Title: Reconsidering SMT Over NMT for Closely Related Languages: A Case Study of Persian-Hindi Pair

• Authors: Waisullah Yousofi, Pushpak Bhattacharyya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16877
• Pdf URL: https://arxiv.org/pdf/2412.16877
• Copy Paste: [[2412.16877]] Reconsidering SMT Over NMT for Closely Related Languages: A Case Study of Persian-Hindi Pair(https://arxiv.org/abs/2412.16877)
• Keywords: transformer
• Abstract: This paper demonstrates that Phrase-Based Statistical Machine Translation (PBSMT) can outperform Transformer-based Neural Machine Translation (NMT) in moderate-resource scenarios, specifically for structurally similar languages, like the Persian-Hindi pair. Despite the Transformer architecture's typical preference for large parallel corpora, our results show that PBSMT achieves a BLEU score of 66.32, significantly exceeding the Transformer-NMT score of 53.7 on the same dataset. Additionally, we explore variations of the SMT architecture, including training on Romanized text and modifying the word order of Persian sentences to match the left-to-right (LTR) structure of Hindi. Our findings highlight the importance of choosing the right architecture based on language pair characteristics and advocate for SMT as a high-performing alternative, even in contexts commonly dominated by NMT.

Title: Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model

• Authors: Songjun Tu, Jingbo Sun, Qichao Zhang, Xiangyuan Lan, Dongbin Zhao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16878
• Pdf URL: https://arxiv.org/pdf/2412.16878
• Copy Paste: [[2412.16878]] Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Model(https://arxiv.org/abs/2412.16878)
• Keywords: large language model
• Abstract: Preference-based reinforcement learning (PbRL) provides a powerful paradigm to avoid meticulous reward engineering by learning rewards based on human preferences. However, real-time human feedback is hard to obtain in online tasks. Most work suppose there is a "scripted teacher" that utilizes privileged predefined reward to provide preference feedback. In this paper, we propose a RL Self-augmented Large Language Model Feedback (RL-SaLLM-F) technique that does not rely on privileged information for online PbRL. RL-SaLLM-F leverages the reflective and discriminative capabilities of LLM to generate self-augmented trajectories and provide preference labels for reward learning. First, we identify an failure issue in LLM-based preference discrimination, specifically "query ambiguity", in online PbRL. Then LLM is employed to provide preference labels and generate self-augmented imagined trajectories that better achieve the task goal, thereby enhancing the quality and efficiency of feedback. Additionally, a double-check mechanism is introduced to mitigate randomness in the preference labels, improving the reliability of LLM feedback. The experiment across multiple tasks in the MetaWorld benchmark demonstrates the specific contributions of each proposed module in RL-SaLLM-F, and shows that self-augmented LLM feedback can effectively replace the impractical "scripted teacher" feedback. In summary, RL-SaLLM-F introduces a new direction of feedback acquisition in online PbRL that does not rely on any online privileged information, offering an efficient and lightweight solution with LLM-driven feedback.

Title: Preventing Non-intrusive Load Monitoring Privacy Invasion: A Precise Adversarial Attack Scheme for Networked Smart Meters

• Authors: Jialing He, Jiacheng Wang, Ning Wang, Shangwei Guo, Liehuang Zhu, Dusit Niyato, Tao Xiang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16893
• Pdf URL: https://arxiv.org/pdf/2412.16893
• Copy Paste: [[2412.16893]] Preventing Non-intrusive Load Monitoring Privacy Invasion: A Precise Adversarial Attack Scheme for Networked Smart Meters(https://arxiv.org/abs/2412.16893)
• Keywords: privacy, attack
• Abstract: Smart grid, through networked smart meters employing the non-intrusive load monitoring (NILM) technique, can considerably discern the usage patterns of residential appliances. However, this technique also incurs privacy leakage. To address this issue, we propose an innovative scheme based on adversarial attack in this paper. The scheme effectively prevents NILM models from violating appliance-level privacy, while also ensuring accurate billing calculation for users. To achieve this objective, we overcome two primary challenges. First, as NILM models fall under the category of time-series regression models, direct application of traditional adversarial attacks designed for classification tasks is not feasible. To tackle this issue, we formulate a novel adversarial attack problem tailored specifically for NILM and providing a theoretical foundation for utilizing the Jacobian of the NILM model to generate imperceptible perturbations. Leveraging the Jacobian, our scheme can produce perturbations, which effectively misleads the signal prediction of NILM models to safeguard users' appliance-level privacy. The second challenge pertains to fundamental utility requirements, where existing adversarial attack schemes struggle to achieve accurate billing calculation for users. To handle this problem, we introduce an additional constraint, mandating that the sum of added perturbations within a billing period must be precisely zero. Experimental validation on real-world power datasets REDD and UK-DALE demonstrates the efficacy of our proposed solutions, which can significantly amplify the discrepancy between the output of the targeted NILM model and the actual power signal of appliances, and enable accurate billing at the same time. Additionally, our solutions exhibit transferability, making the generated perturbation signal from one target model applicable to other diverse NILM models.

Title: A Backdoor Attack Scheme with Invisible Triggers Based on Model Architecture Modification

• Authors: Yuan Ma, Xu Ma, Jiankang Wei, Jinmeng Tang, Xiaoyu Zhang, Yilun Lyu, Kehao Chen, Jingtong Huang
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16905
• Pdf URL: https://arxiv.org/pdf/2412.16905
• Copy Paste: [[2412.16905]] A Backdoor Attack Scheme with Invisible Triggers Based on Model Architecture Modification(https://arxiv.org/abs/2412.16905)
• Keywords: attack, steal
• Abstract: Machine learning systems are vulnerable to backdoor attacks, where attackers manipulate model behavior through data tampering or architectural modifications. Traditional backdoor attacks involve injecting malicious samples with specific triggers into the training data, causing the model to produce targeted incorrect outputs in the presence of the corresponding triggers. More sophisticated attacks modify the model's architecture directly, embedding backdoors that are harder to detect as they evade traditional data-based detection methods. However, the drawback of the architectural modification based backdoor attacks is that the trigger must be visible in order to activate the backdoor. To further strengthen the invisibility of the backdoor attacks, a novel backdoor attack method is presented in the paper. To be more specific, this method embeds the backdoor within the model's architecture and has the capability to generate inconspicuous and stealthy triggers. The attack is implemented by modifying pre-trained models, which are then redistributed, thereby posing a potential threat to unsuspecting users. Comprehensive experiments conducted on standard computer vision benchmarks validate the effectiveness of this attack and highlight the stealthiness of its triggers, which remain undetectable through both manual visual inspection and advanced detection tools.

Title: Self-Corrected Flow Distillation for Consistent One-Step and Few-Step Text-to-Image Generation

• Authors: Quan Dao, Hao Phung, Trung Dao, Dimitris Metaxas, Anh Tran
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16906
• Pdf URL: https://arxiv.org/pdf/2412.16906
• Copy Paste: [[2412.16906]] Self-Corrected Flow Distillation for Consistent One-Step and Few-Step Text-to-Image Generation(https://arxiv.org/abs/2412.16906)
• Keywords: diffusion, generative
• Abstract: Flow matching has emerged as a promising framework for training generative models, demonstrating impressive empirical performance while offering relative ease of training compared to diffusion-based models. However, this method still requires numerous function evaluations in the sampling process. To address these limitations, we introduce a self-corrected flow distillation method that effectively integrates consistency models and adversarial training within the flow-matching framework. This work is a pioneer in achieving consistent generation quality in both few-step and one-step sampling. Our extensive experiments validate the effectiveness of our method, yielding superior results both quantitatively and qualitatively on CelebA-HQ and zero-shot benchmarks on the COCO dataset. Our implementation is released at this https URL

Title: FADA: Fast Diffusion Avatar Synthesis with Mixed-Supervised Multi-CFG Distillation

• Authors: Tianyun Zhong, Chao Liang, Jianwen Jiang, Gaojie Lin, Jiaqi Yang, Zhou Zhao
• Subjects: cs.CV, cs.AI, cs.GR, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.16915
• Pdf URL: https://arxiv.org/pdf/2412.16915
• Copy Paste: [[2412.16915]] FADA: Fast Diffusion Avatar Synthesis with Mixed-Supervised Multi-CFG Distillation(https://arxiv.org/abs/2412.16915)
• Keywords: robust, diffusion
• Abstract: Diffusion-based audio-driven talking avatar methods have recently gained attention for their high-fidelity, vivid, and expressive results. However, their slow inference speed limits practical applications. Despite the development of various distillation techniques for diffusion models, we found that naive diffusion distillation methods do not yield satisfactory results. Distilled models exhibit reduced robustness with open-set input images and a decreased correlation between audio and video compared to teacher models, undermining the advantages of diffusion models. To address this, we propose FADA (Fast Diffusion Avatar Synthesis with Mixed-Supervised Multi-CFG Distillation). We first designed a mixed-supervised loss to leverage data of varying quality and enhance the overall model capability as well as robustness. Additionally, we propose a multi-CFG distillation with learnable tokens to utilize the correlation between audio and reference image conditions, reducing the threefold inference runs caused by multi-CFG with acceptable quality degradation. Extensive experiments across multiple datasets show that FADA generates vivid videos comparable to recent diffusion model-based methods while achieving an NFE speedup of 4.17-12.5 times. Demos are available at our webpage this http URL.

Title: On the Differential Privacy and Interactivity of Privacy Sandbox Reports

• Authors: Badih Ghazi, Charlie Harrison, Arpana Hosabettu, Pritish Kamath, Alexander Knop, Ravi Kumar, Ethan Leeman, Pasin Manurangsi, Vikas Sahu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.16916
• Pdf URL: https://arxiv.org/pdf/2412.16916
• Copy Paste: [[2412.16916]] On the Differential Privacy and Interactivity of Privacy Sandbox Reports(https://arxiv.org/abs/2412.16916)
• Keywords: privacy
• Abstract: The Privacy Sandbox initiative from Google includes APIs for enabling privacy-preserving advertising functionalities as part of the effort around limiting third-party cookies. In particular, the Private Aggregation API (PAA) and the Attribution Reporting API (ARA) can be used for ad measurement while providing different guardrails for safeguarding user privacy, including a framework for satisfying differential privacy (DP). In this work, we provide a formal model for analyzing the privacy of these APIs and show that they satisfy a formal DP guarantee under certain assumptions. Our analysis handles the case where both the queries and database can change interactively based on previous responses from the API.

Title: Detect Changes like Humans: Incorporating Semantic Priors for Improved Change Detection

• Authors: Yuhang Gan, Wenjie Xuan, Zhiming Luo, Lei Fang, Zengmao Wang, Juhua Liu, Bo Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16918
• Pdf URL: https://arxiv.org/pdf/2412.16918
• Copy Paste: [[2412.16918]] Detect Changes like Humans: Incorporating Semantic Priors for Improved Change Detection(https://arxiv.org/abs/2412.16918)
• Keywords: segmentation
• Abstract: When given two similar images, humans identify their differences by comparing the appearance ({\it e.g., color, texture}) with the help of semantics ({\it e.g., objects, relations}). However, mainstream change detection models adopt a supervised training paradigm, where the annotated binary change map is the main constraint. Thus, these methods primarily emphasize the difference-aware features between bi-temporal images and neglect the semantic understanding of the changed landscapes, which undermines the accuracy in the presence of noise and illumination variations. To this end, this paper explores incorporating semantic priors to improve the ability to detect changes. Firstly, we propose a Semantic-Aware Change Detection network, namely SA-CDNet, which transfers the common knowledge of the visual foundation models ({\it i.e., FastSAM}) to change detection. Inspired by the human visual paradigm, a novel dual-stream feature decoder is derived to distinguish changes by combining semantic-aware features and difference-aware features. Secondly, we design a single-temporal semantic pre-training strategy to enhance the semantic understanding of landscapes, which brings further increments. Specifically, we construct pseudo-change detection data from public single-temporal remote sensing segmentation datasets for large-scale pre-training, where an extra branch is also introduced for the proxy semantic segmentation task. Experimental results on five challenging benchmarks demonstrate the superiority of our method over the existing state-of-the-art methods. The code is available at \href{this https URL}{SA-CD}.

Title: TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction

• Authors: Xuying Zhang, Yutong Liu, Yangguang Li, Renrui Zhang, Yufei Liu, Kai Wang, Wanli Ouyang, Zhiwei Xiong, Peng Gao, Qibin Hou, Ming-Ming Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16919
• Pdf URL: https://arxiv.org/pdf/2412.16919
• Copy Paste: [[2412.16919]] TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction(https://arxiv.org/abs/2412.16919)
• Keywords: transformer, generative
• Abstract: We present TAR3D, a novel framework that consists of a 3D-aware Vector Quantized-Variational AutoEncoder (VQ-VAE) and a Generative Pre-trained Transformer (GPT) to generate high-quality 3D assets. The core insight of this work is to migrate the multimodal unification and promising learning capabilities of the next-token prediction paradigm to conditional 3D object generation. To achieve this, the 3D VQ-VAE first encodes a wide range of 3D shapes into a compact triplane latent space and utilizes a set of discrete representations from a trainable codebook to reconstruct fine-grained geometries under the supervision of query point occupancy. Then, the 3D GPT, equipped with a custom triplane position embedding called TriPE, predicts the codebook index sequence with prefilling prompt tokens in an autoregressive manner so that the composition of 3D geometries can be modeled part by part. Extensive experiments on ShapeNet and Objaverse demonstrate that TAR3D can achieve superior generation quality over existing methods in text-to-3D and image-to-3D tasks

Title: Leveraging Consistent Spatio-Temporal Correspondence for Robust Visual Odometry

• Authors: Zhaoxing Zhang, Junda Cheng, Gangwei Xu, Xiaoxiang Wang, Can Zhang, Xin Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16923
• Pdf URL: https://arxiv.org/pdf/2412.16923
• Copy Paste: [[2412.16923]] Leveraging Consistent Spatio-Temporal Correspondence for Robust Visual Odometry(https://arxiv.org/abs/2412.16923)
• Keywords: robust
• Abstract: Recent approaches to VO have significantly improved performance by using deep networks to predict optical flow between video frames. However, existing methods still suffer from noisy and inconsistent flow matching, making it difficult to handle challenging scenarios and long-sequence estimation. To overcome these challenges, we introduce Spatio-Temporal Visual Odometry (STVO), a novel deep network architecture that effectively leverages inherent spatio-temporal cues to enhance the accuracy and consistency of multi-frame flow matching. With more accurate and consistent flow matching, STVO can achieve better pose estimation through the bundle adjustment (BA). Specifically, STVO introduces two innovative components: 1) the Temporal Propagation Module that utilizes multi-frame information to extract and propagate temporal cues across adjacent frames, maintaining temporal consistency; 2) the Spatial Activation Module that utilizes geometric priors from the depth maps to enhance spatial consistency while filtering out excessive noise and incorrect matches. Our STVO achieves state-of-the-art performance on TUM-RGBD, EuRoc MAV, ETH3D and KITTI Odometry benchmarks. Notably, it improves accuracy by 77.8% on ETH3D benchmark and 38.9% on KITTI Odometry benchmark over the previous best methods.

Title: Prompting Large Language Models with Rationale Heuristics for Knowledge-based Visual Question Answering

• Authors: Zhongjian Hu, Peng Yang, Bing Li, Fengyuan Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16936
• Pdf URL: https://arxiv.org/pdf/2412.16936
• Copy Paste: [[2412.16936]] Prompting Large Language Models with Rationale Heuristics for Knowledge-based Visual Question Answering(https://arxiv.org/abs/2412.16936)
• Keywords: large language model
• Abstract: Recently, Large Language Models (LLMs) have been used for knowledge-based Visual Question Answering (VQA). Despite the encouraging results of previous studies, prior methods prompt LLMs to predict answers directly, neglecting intermediate thought processes. We argue that prior methods do not sufficiently activate the capacities of LLMs. We propose a framework called PLRH that Prompts LLMs with Rationale Heuristics for knowledge-based VQA. The PLRH prompts LLMs with Chain of Thought (CoT) to generate rationale heuristics, i.e., intermediate thought processes, and then leverages the rationale heuristics to inspire LLMs to predict answers. Experiments show that our approach outperforms the existing baselines by more than 2.2 and 2.1 on OK-VQA and A-OKVQA, respectively.

Title: PINN-EMFNet: PINN-based and Enhanced Multi-Scale Feature Fusion Network for Breast Ultrasound Images Segmentation

• Authors: Jiajun Ding, Beiyao Zhu, Wenjie Wang, Shurong Zhang, Dian Zhua, Zhao Liua
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16937
• Pdf URL: https://arxiv.org/pdf/2412.16937
• Copy Paste: [[2412.16937]] PINN-EMFNet: PINN-based and Enhanced Multi-Scale Feature Fusion Network for Breast Ultrasound Images Segmentation(https://arxiv.org/abs/2412.16937)
• Keywords: robust, segmentation
• Abstract: With the rapid development of deep learning and computer vision technologies, medical image segmentation plays a crucial role in the early diagnosis of breast cancer. However, due to the characteristics of breast ultrasound images, such as low contrast, speckle noise, and the highly diverse morphology of tumors, existing segmentation methods exhibit significant limitations in terms of accuracy and robustness. To address these challenges, this study proposes a PINN-based and Enhanced Multi-Scale Feature Fusion Network. The network introduces a Hierarchical Aggregation Encoder in the backbone, which efficiently integrates and globally models multi-scale features through several structural innovations and a novel PCAM module. In the decoder section, a Multi-Scale Feature Refinement Decoder is employed, which, combined with a Multi-Scale Supervision Mechanism and a correction module, significantly improves segmentation accuracy and adaptability. Additionally, the loss function incorporating the PINN mechanism introduces physical constraints during the segmentation process, enhancing the model's ability to accurately delineate tumor boundaries. Comprehensive evaluations on two publicly available breast ultrasound datasets, BUSIS and BUSI, demonstrate that the proposed method outperforms previous segmentation approaches in terms of segmentation accuracy and robustness, particularly under conditions of complex noise and low contrast, effectively improving the accuracy and reliability of tumor segmentation. This method provides a more precise and robust solution for computer-aided diagnosis of breast ultrasound images.

Title: Image Quality Assessment: Investigating Causal Perceptual Effects with Abductive Counterfactual Inference

• Authors: Wenhao Shen, Mingliang Zhou, Yu Chen, Xuekai Wei, Jun Luo, Huayan Pu, Weijia Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16939
• Pdf URL: https://arxiv.org/pdf/2412.16939
• Copy Paste: [[2412.16939]] Image Quality Assessment: Investigating Causal Perceptual Effects with Abductive Counterfactual Inference(https://arxiv.org/abs/2412.16939)
• Keywords: robust, interpretability
• Abstract: Existing full-reference image quality assessment (FR-IQA) methods often fail to capture the complex causal mechanisms that underlie human perceptual responses to image distortions, limiting their ability to generalize across diverse scenarios. In this paper, we propose an FR-IQA method based on abductive counterfactual inference to investigate the causal relationships between deep network features and perceptual distortions. First, we explore the causal effects of deep features on perception and integrate causal reasoning with feature comparison, constructing a model that effectively handles complex distortion types across different IQA scenarios. Second, the analysis of the perceptual causal correlations of our proposed method is independent of the backbone architecture and thus can be applied to a variety of deep networks. Through abductive counterfactual experiments, we validate the proposed causal relationships, confirming the model's superior perceptual relevance and interpretability of quality scores. The experimental results demonstrate the robustness and effectiveness of the method, providing competitive quality predictions across multiple benchmarks. The source code is available at this https URL.

Title: A Career Interview Dialogue System using Large Language Model-based Dynamic Slot Generation

• Authors: Ekai Hashimoto, Mikio Nakano, Takayoshi Sakurai, Shun Shiramatsu, Toshitake Komazaki, Shiho Tsuchiya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16943
• Pdf URL: https://arxiv.org/pdf/2412.16943
• Copy Paste: [[2412.16943]] A Career Interview Dialogue System using Large Language Model-based Dynamic Slot Generation(https://arxiv.org/abs/2412.16943)
• Keywords: large language model
• Abstract: This study aims to improve the efficiency and quality of career interviews conducted by nursing managers. To this end, we have been developing a slot-filling dialogue system that engages in pre-interviews to collect information on staff careers as a preparatory step before the actual interviews. Conventional slot-filling-based interview dialogue systems have limitations in the flexibility of information collection because the dialogue progresses based on predefined slot sets. We therefore propose a method that leverages large language models (LLMs) to dynamically generate new slots according to the flow of the dialogue, achieving more natural conversations. Furthermore, we incorporate abduction into the slot generation process to enable more appropriate and effective slot generation. To validate the effectiveness of the proposed method, we conducted experiments using a user simulator. The results suggest that the proposed method using abduction is effective in enhancing both information-collecting capabilities and the naturalness of the dialogue.

Title: Linguistics-Vision Monotonic Consistent Network for Sign Language Production

• Authors: Xu Wang, Shengeng Tang, Peipei Song, Shuo Wang, Dan Guo, Richang Hong
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.16944
• Pdf URL: https://arxiv.org/pdf/2412.16944
• Copy Paste: [[2412.16944]] Linguistics-Vision Monotonic Consistent Network for Sign Language Production(https://arxiv.org/abs/2412.16944)
• Keywords: transformer
• Abstract: Sign Language Production (SLP) aims to generate sign videos corresponding to spoken language sentences, where the conversion of sign Glosses to Poses (G2P) is the key step. Due to the cross-modal semantic gap and the lack of word-action correspondence labels for strong supervision alignment, the SLP suffers huge challenges in linguistics-vision consistency. In this work, we propose a Transformer-based Linguistics-Vision Monotonic Consistent Network (LVMCN) for SLP, which constrains fine-grained cross-modal monotonic alignment and coarse-grained multimodal semantic consistency in language-visual cues through Cross-modal Semantic Aligner (CSA) and Multimodal Semantic Comparator (MSC). In the CSA, we constrain the implicit alignment between corresponding gloss and pose sequences by computing the cosine similarity association matrix between cross-modal feature sequences (i.e., the order consistency of fine-grained sign glosses and actions). As for MSC, we construct multimodal triplets based on paired and unpaired samples in batch data. By pulling closer the corresponding text-visual pairs and pushing apart the non-corresponding text-visual pairs, we constrain the semantic co-occurrence degree between corresponding gloss and pose sequences (i.e., the semantic consistency of coarse-grained textual sentences and sign videos). Extensive experiments on the popular PHOENIX14T benchmark show that the LVMCN outperforms the state-of-the-art.

Title: Separating Drone Point Clouds From Complex Backgrounds by Cluster Filter -- Technical Report for CVPR 2024 UG2 Challenge

• Authors: Hanfang Liang, Jinming Hu, Xiaohuan Ling, Bing Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16947
• Pdf URL: https://arxiv.org/pdf/2412.16947
• Copy Paste: [[2412.16947]] Separating Drone Point Clouds From Complex Backgrounds by Cluster Filter -- Technical Report for CVPR 2024 UG2 Challenge(https://arxiv.org/abs/2412.16947)
• Keywords: segmentation
• Abstract: The increasing deployment of small drones as tools of conflict and disruption has amplified their threat, highlighting the urgent need for effective anti-drone measures. However, the compact size of most drones presents a significant challenge, as traditional supervised point cloud or image-based object detection methods often fail to identify such small objects effectively. This paper proposes a simple UAV detection method using an unsupervised pipeline. It uses spatial-temporal sequence processing to fuse multiple lidar datasets effectively, tracking and determining the position of UAVs, so as to detect and track UAVs in challenging environments. Our method performs front and rear background segmentation of point clouds through a global-local sequence clusterer and parses point cloud data from both the spatial-temporal density and spatial-temporal voxels of the point cloud. Furthermore, a scoring mechanism for point cloud moving targets is proposed, using time series detection to improve accuracy and efficiency. We used the MMAUD dataset, and our method achieved 4th place in the CVPR 2024 UG2+ Challenge, confirming the effectiveness of our method in practical applications.

Title: DTSGAN: Learning Dynamic Textures via Spatiotemporal Generative Adversarial Network

• Authors: Xiangtian Li, Xiaobo Wang, Zhen Qi, Han Cao, Zhaoyang Zhang, Ao Xiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16948
• Pdf URL: https://arxiv.org/pdf/2412.16948
• Copy Paste: [[2412.16948]] DTSGAN: Learning Dynamic Textures via Spatiotemporal Generative Adversarial Network(https://arxiv.org/abs/2412.16948)
• Keywords: generative
• Abstract: Dynamic texture synthesis aims to generate sequences that are visually similar to a reference video texture and exhibit specific stationary properties in time. In this paper, we introduce a spatiotemporal generative adversarial network (DTSGAN) that can learn from a single dynamic texture by capturing its motion and content distribution. With the pipeline of DTSGAN, a new video sequence is generated from the coarsest scale to the finest one. To avoid mode collapse, we propose a novel strategy for data updates that helps improve the diversity of generated results. Qualitative and quantitative experiments show that our model is able to generate high quality dynamic textures and natural motion.

Title: Aristotle: Mastering Logical Reasoning with A Logic-Complete Decompose-Search-Resolve Framework

• Authors: Jundong Xu, Hao Fei, Meng Luo, Qian Liu, Liangming Pan, William Yang Wang, Preslav Nakov, Mong-Li Lee, Wynne Hsu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.16953
• Pdf URL: https://arxiv.org/pdf/2412.16953
• Copy Paste: [[2412.16953]] Aristotle: Mastering Logical Reasoning with A Logic-Complete Decompose-Search-Resolve Framework(https://arxiv.org/abs/2412.16953)
• Keywords: large language model
• Abstract: In the context of large language models (LLMs), current advanced reasoning methods have made impressive strides in various reasoning tasks. However, when it comes to logical reasoning tasks, major challenges remain in both efficacy and efficiency. This is rooted in the fact that these systems fail to fully leverage the inherent structure of logical tasks throughout the reasoning processes such as decomposition, search, and resolution. To address this, we propose a logic-complete reasoning framework, Aristotle, with three key components: Logical Decomposer, Logical Search Router, and Logical Resolver. In our framework, symbolic expressions and logical rules are comprehensively integrated into the entire reasoning process, significantly alleviating the bottlenecks of logical reasoning, i.e., reducing sub-task complexity, minimizing search errors, and resolving logical contradictions. The experimental results on several datasets demonstrate that Aristotle consistently outperforms state-of-the-art reasoning frameworks in both accuracy and efficiency, particularly excelling in complex logical reasoning scenarios. We will open-source all our code at this https URL.

Title: NumbOD: A Spatial-Frequency Fusion Attack Against Object Detectors

• Authors: Ziqi Zhou, Bowen Li, Yufei Song, Zhifei Yu, Shengshan Hu, Wei Wan, Leo Yu Zhang, Dezhong Yao, Hai Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16955
• Pdf URL: https://arxiv.org/pdf/2412.16955
• Copy Paste: [[2412.16955]] NumbOD: A Spatial-Frequency Fusion Attack Against Object Detectors(https://arxiv.org/abs/2412.16955)
• Keywords: attack, steal
• Abstract: With the advancement of deep learning, object detectors (ODs) with various architectures have achieved significant success in complex scenarios like autonomous driving. Previous adversarial attacks against ODs have been focused on designing customized attacks targeting their specific structures (e.g., NMS and RPN), yielding some results but simultaneously constraining their scalability. Moreover, most efforts against ODs stem from image-level attacks originally designed for classification tasks, resulting in redundant computations and disturbances in object-irrelevant areas (e.g., background). Consequently, how to design a model-agnostic efficient attack to comprehensively evaluate the vulnerabilities of ODs remains challenging and unresolved. In this paper, we propose NumbOD, a brand-new spatial-frequency fusion attack against various ODs, aimed at disrupting object detection within images. We directly leverage the features output by the OD without relying on its internal structures to craft adversarial examples. Specifically, we first design a dual-track attack target selection strategy to select high-quality bounding boxes from OD outputs for targeting. Subsequently, we employ directional perturbations to shift and compress predicted boxes and change classification results to deceive ODs. Additionally, we focus on manipulating the high-frequency components of images to confuse ODs' attention on critical objects, thereby enhancing the attack efficiency. Our extensive experiments on nine ODs and two datasets show that NumbOD achieves powerful attack performance and high stealthiness.

Title: Semantic Hierarchical Prompt Tuning for Parameter-Efficient Fine-Tuning

• Authors: Haowei Zhu, Fangyuan Zhang, Rui Qin, Tianxiang Pan, Junhai Yong, Bin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16956
• Pdf URL: https://arxiv.org/pdf/2412.16956
• Copy Paste: [[2412.16956]] Semantic Hierarchical Prompt Tuning for Parameter-Efficient Fine-Tuning(https://arxiv.org/abs/2412.16956)
• Keywords: robust
• Abstract: As the scale of vision models continues to grow, Visual Prompt Tuning (VPT) has emerged as a parameter-efficient transfer learning technique, noted for its superior performance compared to full fine-tuning. However, indiscriminately applying prompts to every layer without considering their inherent correlations, can cause significant disturbances, leading to suboptimal transferability. Additionally, VPT disrupts the original self-attention structure, affecting the aggregation of visual features, and lacks a mechanism for explicitly mining discriminative visual features, which are crucial for classification. To address these issues, we propose a Semantic Hierarchical Prompt (SHIP) fine-tuning strategy. We adaptively construct semantic hierarchies and use semantic-independent and semantic-shared prompts to learn hierarchical representations. We also integrate attribute prompts and a prompt matching loss to enhance feature discrimination and employ decoupled attention for robustness and reduced inference costs. SHIP significantly improves performance, achieving a 4.8\% gain in accuracy over VPT with a ViT-B/16 backbone on VTAB-1k tasks. Our code is available at this https URL.

Title: Breaking Barriers in Physical-World Adversarial Examples: Improving Robustness and Transferability via Robust Feature

• Authors: Yichen Wang, Yuxuan Chou, Ziqi Zhou, Hangtao Zhang, Wei Wan, Shengshan Hu, Minghui Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16958
• Pdf URL: https://arxiv.org/pdf/2412.16958
• Copy Paste: [[2412.16958]] Breaking Barriers in Physical-World Adversarial Examples: Improving Robustness and Transferability via Robust Feature(https://arxiv.org/abs/2412.16958)
• Keywords: attack, robust, steal
• Abstract: As deep neural networks (DNNs) are widely applied in the physical world, many researches are focusing on physical-world adversarial examples (PAEs), which introduce perturbations to inputs and cause the model's incorrect outputs. However, existing PAEs face two challenges: unsatisfactory attack performance (i.e., poor transferability and insufficient robustness to environment conditions), and difficulty in balancing attack effectiveness with stealthiness, where better attack effectiveness often makes PAEs more perceptible. In this paper, we explore a novel perturbation-based method to overcome the challenges. For the first challenge, we introduce a strategy Deceptive RF injection based on robust features (RFs) that are predictive, robust to perturbations, and consistent across different models. Specifically, it improves the transferability and robustness of PAEs by covering RFs of other classes onto the predictive features in clean images. For the second challenge, we introduce another strategy Adversarial Semantic Pattern Minimization, which removes most perturbations and retains only essential adversarial patterns in AEsBased on the two strategies, we design our method Robust Feature Coverage Attack (RFCoA), comprising Robust Feature Disentanglement and Adversarial Feature Fusion. In the first stage, we extract target class RFs in feature space. In the second stage, we use attention-based feature fusion to overlay these RFs onto predictive features of clean images and remove unnecessary perturbations. Experiments show our method's superior transferability, robustness, and stealthiness compared to existing state-of-the-art methods. Additionally, our method's effectiveness can extend to Large Vision-Language Models (LVLMs), indicating its potential applicability to more complex tasks.

Title: FedCross: Intertemporal Federated Learning Under Evolutionary Games

• Authors: Jianfeng Lu, Ying Zhang, Riheng Jia, Shuqin Cao, Jing Liu, Hao Fu
• Subjects: cs.LG, cs.DC, cs.GT
• Abstract URL: https://arxiv.org/abs/2412.16968
• Pdf URL: https://arxiv.org/pdf/2412.16968
• Copy Paste: [[2412.16968]] FedCross: Intertemporal Federated Learning Under Evolutionary Games(https://arxiv.org/abs/2412.16968)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) mitigates privacy leakage in decentralized machine learning by allowing multiple clients to train collaboratively locally. However, dynamic mobile networks with high mobility, intermittent connectivity, and bandwidth limitation severely hinder model updates to the cloud server. Although previous studies have typically addressed user mobility issue through task reassignment or predictive modeling, frequent migrations may result in high communication overhead. Overcoming this obstacle involves not only dealing with resource constraints, but also finding ways to mitigate the challenges posed by user migrations. We therefore propose an intertemporal incentive framework, FedCross, which ensures the continuity of FL tasks by migrating interrupted training tasks to feasible mobile devices. Specifically, FedCross comprises two distinct stages. In Stage 1, we address the task allocation problem across regions under resource constraints by employing a multi-objective migration algorithm to quantify the optimal task receivers. Moreover, we adopt evolutionary game theory to capture the dynamic decision-making of users, forecasting the evolution of user proportions across different regions to mitigate frequent migrations. In Stage 2, we utilize a procurement auction mechanism to allocate rewards among base stations, ensuring that those providing high-quality models receive optimal compensation. This approach incentivizes sustained user participation, thereby ensuring the overall feasibility of FedCross. Finally, experimental results validate the theoretical soundness of FedCross and demonstrate its significant reduction in communication overhead.

Title: On Fusing ChatGPT and Ensemble Learning in Discon-tinuous Named Entity Recognition in Health Corpora

• Authors: Tzu-Chieh Chen, Wen-Yang Lin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16976
• Pdf URL: https://arxiv.org/pdf/2412.16976
• Copy Paste: [[2412.16976]] On Fusing ChatGPT and Ensemble Learning in Discon-tinuous Named Entity Recognition in Health Corpora(https://arxiv.org/abs/2412.16976)
• Keywords: robust, large language model
• Abstract: Named Entity Recognition has traditionally been a key task in natural language processing, aiming to identify and extract important terms from unstructured text data. However, a notable challenge for contemporary deep-learning NER models has been identifying discontinuous entities, which are often fragmented within the text. To date, methods to address Discontinuous Named Entity Recognition have not been explored using ensemble learning to the best of our knowledge. Furthermore, the rise of large language models, such as ChatGPT in recent years, has shown significant effectiveness across many NLP tasks. Most existing approaches, however, have primarily utilized ChatGPT as a problem-solving tool rather than exploring its potential as an integrative element within ensemble learning algorithms. In this study, we investigated the integration of ChatGPT as an arbitrator within an ensemble method, aiming to enhance performance on DNER tasks. Our method combines five state-of-the-art NER models with ChatGPT using custom prompt engineering to assess the robustness and generalization capabilities of the ensemble algorithm. We conducted experiments on three benchmark medical datasets, comparing our method against the five SOTA models, individual applications of GPT-3.5 and GPT-4, and a voting ensemble method. The results indicate that our proposed fusion of ChatGPT with the ensemble learning algorithm outperforms the SOTA results in the CADEC, ShARe13, and ShARe14 datasets, showcasing its potential to enhance NLP applications in the healthcare domain.

Title: PromptDresser: Improving the Quality and Controllability of Virtual Try-On via Generative Textual Prompt and Prompt-aware Mask

• Authors: Jeongho Kim, Hoiyeong Jin, Sunghyun Park, Jaegul Choo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16978
• Pdf URL: https://arxiv.org/pdf/2412.16978
• Copy Paste: [[2412.16978]] PromptDresser: Improving the Quality and Controllability of Virtual Try-On via Generative Textual Prompt and Prompt-aware Mask(https://arxiv.org/abs/2412.16978)
• Keywords: diffusion, generative
• Abstract: Recent virtual try-on approaches have advanced by fine-tuning the pre-trained text-to-image diffusion models to leverage their powerful generative ability. However, the use of text prompts in virtual try-on is still underexplored. This paper tackles a text-editable virtual try-on task that changes the clothing item based on the provided clothing image while editing the wearing style (e.g., tucking style, fit) according to the text descriptions. In the text-editable virtual try-on, three key aspects exist: (i) designing rich text descriptions for paired person-clothing data to train the model, (ii) addressing the conflicts where textual information of the existing person's clothing interferes the generation of the new clothing, and (iii) adaptively adjust the inpainting mask aligned with the text descriptions, ensuring proper editing areas while preserving the original person's appearance irrelevant to the new clothing. To address these aspects, we propose PromptDresser, a text-editable virtual try-on model that leverages large multimodal model (LMM) assistance to enable high-quality and versatile manipulation based on generative text prompts. Our approach utilizes LMMs via in-context learning to generate detailed text descriptions for person and clothing images independently, including pose details and editing attributes using minimal human cost. Moreover, to ensure the editing areas, we adjust the inpainting mask depending on the text prompts adaptively. We found that our approach, utilizing detailed text prompts, not only enhances text editability but also effectively conveys clothing details that are difficult to capture through images alone, thereby enhancing image quality. Our code is available at this https URL.

Title: A Conditional Diffusion Model for Electrical Impedance Tomography Image Reconstruction

• Authors: Shuaikai Shi, Ruiyuan Kang, Panos Liatsis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16979
• Pdf URL: https://arxiv.org/pdf/2412.16979
• Copy Paste: [[2412.16979]] A Conditional Diffusion Model for Electrical Impedance Tomography Image Reconstruction(https://arxiv.org/abs/2412.16979)
• Keywords: diffusion
• Abstract: Electrical impedance tomography (EIT) is a non-invasive imaging technique, capable of reconstructing images of the electrical conductivity of tissues and materials. It is popular in diverse application areas, from medical imaging to industrial process monitoring and tactile sensing, due to its low cost, real-time capabilities and non-ionizing nature. EIT visualizes the conductivity distribution within a body by measuring the boundary voltages, given a current injection. However, EIT image reconstruction is ill-posed due to the mismatch between the under-sampled voltage data and the high-resolution conductivity image. A variety of approaches, both conventional and deep learning-based, have been proposed, capitalizing on the use of spatial regularizers, and the paradigm of image regression. In this research, a novel method based on the conditional diffusion model for EIT reconstruction is proposed, termed CDEIT. Specifically, CDEIT consists of the forward diffusion process, which first gradually adds Gaussian noise to the clean conductivity images, and a reverse denoising process, which learns to predict the original conductivity image from its noisy version, conditioned on the boundary voltages. Following model training, CDEIT applies the conditional reverse process on test voltage data to generate the desired conductivities. Moreover, we provide the details of a normalization procedure, which demonstrates how EIT image reconstruction models trained on simulated datasets can be applied on real datasets with varying sizes, excitation currents and background conductivities. Experiments conducted on a synthetic dataset and two real datasets demonstrate that the proposed model outperforms state-of-the-art methods. The CDEIT software is available as open-source (this https URL) for reproducibility purposes.

Title: InterDance:Reactive 3D Dance Generation with Realistic Duet Interactions

• Authors: Ronghui Li, Youliang Zhang, Yachao Zhang, Yuxiang Zhang, Mingyang Su, Jie Guo, Ziwei Liu, Yebin Liu, Xiu Li
• Subjects: cs.CV, cs.GR, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.16982
• Pdf URL: https://arxiv.org/pdf/2412.16982
• Copy Paste: [[2412.16982]] InterDance:Reactive 3D Dance Generation with Realistic Duet Interactions(https://arxiv.org/abs/2412.16982)
• Keywords: diffusion, generative
• Abstract: Humans perform a variety of interactive motions, among which duet dance is one of the most challenging interactions. However, in terms of human motion generative models, existing works are still unable to generate high-quality interactive motions, especially in the field of duet dance. On the one hand, it is due to the lack of large-scale high-quality datasets. On the other hand, it arises from the incomplete representation of interactive motion and the lack of fine-grained optimization of interactions. To address these challenges, we propose, InterDance, a large-scale duet dance dataset that significantly enhances motion quality, data scale, and the variety of dance genres. Built upon this dataset, we propose a new motion representation that can accurately and comprehensively describe interactive motion. We further introduce a diffusion-based framework with an interaction refinement guidance strategy to optimize the realism of interactions progressively. Extensive experiments demonstrate the effectiveness of our dataset and algorithm.

Title: Pinwheel-shaped Convolution and Scale-based Dynamic Loss for Infrared Small Target Detection

• Authors: Jiangnan Yang, Shuangli Liu, Jingjun Wu, Xinyu Su, Nan Hai, Xueli Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16986
• Pdf URL: https://arxiv.org/pdf/2412.16986
• Copy Paste: [[2412.16986]] Pinwheel-shaped Convolution and Scale-based Dynamic Loss for Infrared Small Target Detection(https://arxiv.org/abs/2412.16986)
• Keywords: extraction
• Abstract: These recent years have witnessed that convolutional neural network (CNN)-based methods for detecting infrared small targets have achieved outstanding performance. However, these methods typically employ standard convolutions, neglecting to consider the spatial characteristics of the pixel distribution of infrared small targets. Therefore, we propose a novel pinwheel-shaped convolution (PConv) as a replacement for standard convolutions in the lower layers of the backbone network. PConv better aligns with the pixel Gaussian spatial distribution of dim small targets, enhances feature extraction, significantly increases the receptive field, and introduces only a minimal increase in parameters. Additionally, while recent loss functions combine scale and location losses, they do not adequately account for the varying sensitivity of these losses across different target scales, limiting detection performance on dim-small targets. To overcome this, we propose a scale-based dynamic (SD) Loss that dynamically adjusts the influence of scale and location losses based on target size, improving the network's ability to detect targets of varying scales. We construct a new benchmark, SIRST-UAVB, which is the largest and most challenging dataset to date for real-shot single-frame infrared small target detection. Lastly, by integrating PConv and SD Loss into the latest small target detection algorithms, we achieved significant performance improvements on IRSTD-1K and our SIRST-UAVB dataset, validating the effectiveness and generalizability of our approach. Code -- this https URL

Title: Multi-Scale Foreground-Background Confidence for Out-of-Distribution Segmentation

• Authors: Samuel Marschall, Kira Maag
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16990
• Pdf URL: https://arxiv.org/pdf/2412.16990
• Copy Paste: [[2412.16990]] Multi-Scale Foreground-Background Confidence for Out-of-Distribution Segmentation(https://arxiv.org/abs/2412.16990)
• Keywords: segmentation
• Abstract: Deep neural networks have shown outstanding performance in computer vision tasks such as semantic segmentation and have defined the state-of-the-art. However, these segmentation models are trained on a closed and predefined set of semantic classes, which leads to significant prediction failures in open-world scenarios on unknown objects. As this behavior prevents the application in safety-critical applications such as automated driving, the detection and segmentation of these objects from outside their predefined semantic space (out-of-distribution (OOD) objects) is of the utmost importance. In this work, we present a multi-scale OOD segmentation method that exploits the confidence information of a foreground-background segmentation model. While semantic segmentation models are trained on specific classes, this restriction does not apply to foreground-background methods making them suitable for OOD segmentation. We consider the per pixel confidence score of the model prediction which is close to 1 for a pixel in a foreground object. By aggregating these confidence values for different sized patches, objects of various sizes can be identified in a single image. Our experiments show improved performance of our method in OOD segmentation compared to comparable baselines in the SegmentMeIfYouCan benchmark.

Title: Where am I? Cross-View Geo-localization with Natural Language Descriptions

• Authors: Junyan Ye, Honglin Lin, Leyan Ou, Dairong Chen, Zihao Wang, Conghui He, Weijia Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17007
• Pdf URL: https://arxiv.org/pdf/2412.17007
• Copy Paste: [[2412.17007]] Where am I? Cross-View Geo-localization with Natural Language Descriptions(https://arxiv.org/abs/2412.17007)
• Keywords: explainability
• Abstract: Cross-view geo-localization identifies the locations of street-view images by matching them with geo-tagged satellite images or OSM. However, most studies focus on image-to-image retrieval, with fewer addressing text-guided retrieval, a task vital for applications like pedestrian navigation and emergency response. In this work, we introduce a novel task for cross-view geo-localization with natural language descriptions, which aims to retrieve corresponding satellite images or OSM database based on scene text. To support this task, we construct the CVG-Text dataset by collecting cross-view data from multiple cities and employing a scene text generation approach that leverages the annotation capabilities of Large Multimodal Models to produce high-quality scene text descriptions with localization this http URL, we propose a novel text-based retrieval localization method, CrossText2Loc, which improves recall by 10% and demonstrates excellent long-text retrieval capabilities. In terms of explainability, it not only provides similarity scores but also offers retrieval reasons. More information can be found at this https URL.

Title: Data value estimation on private gradients

• Authors: Zijian Zhou, Xinyi Xu, Daniela Rus, Bryan Kian Hsiang Low
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.17008
• Pdf URL: https://arxiv.org/pdf/2412.17008
• Copy Paste: [[2412.17008]] Data value estimation on private gradients(https://arxiv.org/abs/2412.17008)
• Keywords: privacy, federate
• Abstract: For gradient-based machine learning (ML) methods commonly adopted in practice such as stochastic gradient descent, the de facto differential privacy (DP) technique is perturbing the gradients with random Gaussian noise. Data valuation attributes the ML performance to the training data and is widely used in privacy-aware applications that require enforcing DP such as data pricing, collaborative ML, and federated learning (FL). Can existing data valuation methods still be used when DP is enforced via gradient perturbations? We show that the answer is no with the default approach of injecting i.i.d.~random noise to the gradients because the estimation uncertainty of the data value estimation paradoxically linearly scales with more estimation budget, producing estimates almost like random guesses. To address this issue, we propose to instead inject carefully correlated noise to provably remove the linear scaling of estimation uncertainty w.r.t.~the budget. We also empirically demonstrate that our method gives better data value estimates on various ML tasks and is applicable to use cases including dataset valuation and~FL.

Title: Robustness of Large Language Models Against Adversarial Attacks

• Authors: Yiyi Tao, Yixian Shen, Hang Zhang, Yanxin Shen, Lun Wang, Chuanqi Shi, Shaoshuai Du
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17011
• Pdf URL: https://arxiv.org/pdf/2412.17011
• Copy Paste: [[2412.17011]] Robustness of Large Language Models Against Adversarial Attacks(https://arxiv.org/abs/2412.17011)
• Keywords: attack, robust, large language model
• Abstract: The increasing deployment of Large Language Models (LLMs) in various applications necessitates a rigorous evaluation of their robustness against adversarial attacks. In this paper, we present a comprehensive study on the robustness of GPT LLM family. We employ two distinct evaluation methods to assess their resilience. The first method introduce character-level text attack in input prompts, testing the models on three sentiment classification datasets: StanfordNLP/IMDB, Yelp Reviews, and SST-2. The second method involves using jailbreak prompts to challenge the safety mechanisms of the LLMs. Our experiments reveal significant variations in the robustness of these models, demonstrating their varying degrees of vulnerability to both character-level and semantic-level adversarial attacks. These findings underscore the necessity for improved adversarial training and enhanced safety mechanisms to bolster the robustness of LLMs.

Title: Reversed Attention: On The Gradient Descent Of Attention Layers In GPT

• Authors: Shahar Katz, Lior Wolf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17019
• Pdf URL: https://arxiv.org/pdf/2412.17019
• Copy Paste: [[2412.17019]] Reversed Attention: On The Gradient Descent Of Attention Layers In GPT(https://arxiv.org/abs/2412.17019)
• Keywords: explainability, transformer
• Abstract: The success of Transformer-based Language Models (LMs) stems from their attention mechanism. While this mechanism has been extensively studied in explainability research, particularly through the attention values obtained during the forward pass of LMs, the backward pass of attention has been largely overlooked. In this work, we study the mathematics of the backward pass of attention, revealing that it implicitly calculates an attention matrix we refer to as "Reversed Attention". We examine the properties of Reversed Attention and demonstrate its ability to elucidate the models' behavior and edit dynamics. In an experimental setup, we showcase the ability of Reversed Attention to directly alter the forward pass of attention, without modifying the model's weights, using a novel method called "attention patching". In addition to enhancing the comprehension of how LM configure attention layers during backpropagation, Reversed Attention maps contribute to a more interpretable backward pass.

Title: FriendsQA: A New Large-Scale Deep Video Understanding Dataset with Fine-grained Topic Categorization for Story Videos

• Authors: Zhengqian Wu, Ruizhe Li, Zijun Xu, Zhongyuan Wang, Chunxia Xiao, Chao Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17022
• Pdf URL: https://arxiv.org/pdf/2412.17022
• Copy Paste: [[2412.17022]] FriendsQA: A New Large-Scale Deep Video Understanding Dataset with Fine-grained Topic Categorization for Story Videos(https://arxiv.org/abs/2412.17022)
• Keywords: large language model
• Abstract: Video question answering (VideoQA) aims to answer natural language questions according to the given videos. Although existing models perform well in the factoid VideoQA task, they still face challenges in deep video understanding (DVU) task, which focuses on story videos. Compared to factoid videos, the most significant feature of story videos is storylines, which are composed of complex interactions and long-range evolvement of core story topics including characters, actions and locations. Understanding these topics requires models to possess DVU capability. However, existing DVU datasets rarely organize questions according to these story topics, making them difficult to comprehensively assess VideoQA models' DVU capability of complex storylines. Additionally, the question quantity and video length of these dataset are limited by high labor costs of handcrafted dataset building method. In this paper, we devise a large language model based multi-agent collaboration framework, StoryMind, to automatically generate a new large-scale DVU dataset. The dataset, FriendsQA, derived from the renowned sitcom Friends with an average episode length of 1,358 seconds, contains 44.6K questions evenly distributed across 14 fine-grained topics. Finally, We conduct comprehensive experiments on 10 state-of-the-art VideoQA models using the FriendsQA dataset.

Title: MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge

• Authors: Jie He, Nan Hu, Wanqiu Long, Jiaoyan Chen, Jeff Z. Pan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17032
• Pdf URL: https://arxiv.org/pdf/2412.17032
• Copy Paste: [[2412.17032]] MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge(https://arxiv.org/abs/2412.17032)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks but face significant challenges with complex, knowledge-intensive multi-hop queries, particularly those involving new or long-tail knowledge. Existing benchmarks often fail to fully address these challenges. To bridge this gap, we introduce MINTQA (Multi-hop Question Answering on New and Tail Knowledge), a comprehensive benchmark to evaluate LLMs' capabilities in multi-hop reasoning across four critical dimensions: question handling strategy, sub-question generation, retrieval-augmented generation, and iterative or dynamic decomposition and retrieval. MINTQA comprises 10,479 question-answer pairs for evaluating new knowledge and 17,887 pairs for assessing long-tail knowledge, with each question equipped with corresponding sub-questions and answers. Our systematic evaluation of 22 state-of-the-art LLMs on MINTQA reveals significant limitations in their ability to handle complex knowledge base queries, particularly in handling new or unpopular knowledge. Our findings highlight critical challenges and offer insights for advancing multi-hop reasoning capabilities. The MINTQA benchmark is available at this https URL.

Title: Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models

• Authors: Lang Gao, Xiangliang Zhang, Preslav Nakov, Xiuying Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17034
• Pdf URL: https://arxiv.org/pdf/2412.17034
• Copy Paste: [[2412.17034]] Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Models(https://arxiv.org/abs/2412.17034)
• Keywords: security, defense, attack, large language model
• Abstract: Jailbreaking in Large Language Models (LLMs) is a major security concern as it can deceive LLMs to generate harmful text. Yet, there is still insufficient understanding of how jailbreaking works, which makes it hard to develop effective defense strategies. We aim to shed more light into this issue: we conduct a detailed large-scale analysis of seven different jailbreak methods and find that these disagreements stem from insufficient observation samples. In particular, we introduce \textit{safety boundary}, and we find that jailbreaks shift harmful activations outside that safety boundary, where LLMs are less sensitive to harmful information. We also find that the low and the middle layers are critical in such shifts, while deeper layers have less impact. Leveraging on these insights, we propose a novel defense called \textbf{Activation Boundary Defense} (ABD), which adaptively constrains the activations within the safety boundary. We further use Bayesian optimization to selectively apply the defense method to the low and the middle layers. Our experiments on several benchmarks show that ABD achieves an average DSR of over 98\% against various forms of jailbreak attacks, with less than 2\% impact on the model's general capabilities.

Title: ErasableMask: A Robust and Erasable Privacy Protection Scheme against Black-box Face Recognition Models

• Authors: Sipeng Shen, Yunming Zhang, Dengpan Ye, Xiuwen Shi, Long Tang, Haoran Duan, Ziyi Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17038
• Pdf URL: https://arxiv.org/pdf/2412.17038
• Copy Paste: [[2412.17038]] ErasableMask: A Robust and Erasable Privacy Protection Scheme against Black-box Face Recognition Models(https://arxiv.org/abs/2412.17038)
• Keywords: privacy, protect, attack, robust
• Abstract: While face recognition (FR) models have brought remarkable convenience in face verification and identification, they also pose substantial privacy risks to the public. Existing facial privacy protection schemes usually adopt adversarial examples to disrupt face verification of FR models. However, these schemes often suffer from weak transferability against black-box FR models and permanently damage the identifiable information that cannot fulfill the requirements of authorized operations such as forensics and authentication. To address these limitations, we propose ErasableMask, a robust and erasable privacy protection scheme against black-box FR models. Specifically, via rethinking the inherent relationship between surrogate FR models, ErasableMask introduces a novel meta-auxiliary attack, which boosts black-box transferability by learning more general features in a stable and balancing optimization strategy. It also offers a perturbation erasion mechanism that supports the erasion of semantic perturbations in protected face without degrading image quality. To further improve performance, ErasableMask employs a curriculum learning strategy to mitigate optimization conflicts between adversarial attack and perturbation erasion. Extensive experiments on the CelebA-HQ and FFHQ datasets demonstrate that ErasableMask achieves the state-of-the-art performance in transferability, achieving over 72% confidence on average in commercial FR systems. Moreover, ErasableMask also exhibits outstanding perturbation erasion performance, achieving over 90% erasion success rate.

Title: HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories

• Authors: Eric Hedlin, Munawar Hayat, Fatih Porikli, Kwang Moo Yi, Shweta Mahajan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17040
• Pdf URL: https://arxiv.org/pdf/2412.17040
• Copy Paste: [[2412.17040]] HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories(https://arxiv.org/abs/2412.17040)
• Keywords: generative
• Abstract: To efficiently adapt large models or to train generative models of neural representations, Hypernetworks have drawn interest. While hypernetworks work well, training them is cumbersome, and often requires ground truth optimized weights for each sample. However, obtaining each of these weights is a training problem of its own-one needs to train, e.g., adaptation weights or even an entire neural field for hypernetworks to regress to. In this work, we propose a method to train hypernetworks, without the need for any per-sample ground truth. Our key idea is to learn a Hypernetwork `Field` and estimate the entire trajectory of network weight training instead of simply its converged state. In other words, we introduce an additional input to the Hypernetwork, the convergence state, which then makes it act as a neural field that models the entire convergence pathway of a task network. A critical benefit in doing so is that the gradient of the estimated weights at any convergence state must then match the gradients of the original task -- this constraint alone is sufficient to train the Hypernetwork Field. We demonstrate the effectiveness of our method through the task of personalized image generation and 3D shape reconstruction from images and point clouds, demonstrating competitive results without any per-sample ground truth.

Title: Adapting Image-to-Video Diffusion Models for Large-Motion Frame Interpolation

• Authors: Luoxu Jin, Hiroshi Watanabe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17042
• Pdf URL: https://arxiv.org/pdf/2412.17042
• Copy Paste: [[2412.17042]] Adapting Image-to-Video Diffusion Models for Large-Motion Frame Interpolation(https://arxiv.org/abs/2412.17042)
• Keywords: diffusion, generative
• Abstract: The development of video generation models has advanced significantly in recent years. For video frame interpolation, we adopt a pre-trained large-scale image-to-video diffusion model. To enable this adaptation, we propose a conditional encoder, which serves as a simple yet effective trainable module. By leveraging the first and last frames, we extract spatial and temporal features and input them into the conditional encoder. The computed features of the conditional encoder guide the video diffusion model in generating keyframe-guided video sequences. Our method demonstrates superior performance on the Fréchet Video Distance (FVD) metric compared to previous deterministic approaches in handling large-motion cases, highlighting advancements in generative-based methodologies.

Title: DR-Encoder: Encode Low-rank Gradients with Random Prior for Large Language Models Differentially Privately

• Authors: Huiwen Wu, Deyi Zhang, Xiaohan Li, Xiaogang Xu, Jiafei Wu, Zhe Liu
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.17053
• Pdf URL: https://arxiv.org/pdf/2412.17053
• Copy Paste: [[2412.17053]] DR-Encoder: Encode Low-rank Gradients with Random Prior for Large Language Models Differentially Privately(https://arxiv.org/abs/2412.17053)
• Keywords: privacy, federate, transformer, large language model
• Abstract: The emergence of the Large Language Model (LLM) has shown their superiority in a wide range of disciplines, including language understanding and translation, relational logic reasoning, and even partial differential equations solving. The transformer is the pervasive backbone architecture for the foundation model construction. It is vital to research how to adjust the Transformer architecture to achieve an end-to-end privacy guarantee in LLM fine-tuning. In this paper, we investigate three potential information leakage during a federated fine-tuning procedure for LLM (FedLLM). Based on the potential information leakage, we provide an end-to-end privacy guarantee solution for FedLLM by inserting two-stage randomness. The first stage is to train a gradient auto-encoder with a Gaussian random prior based on the statistical information of the gradients generated by local clients. The second stage is to fine-tune the overall LLM with a differential privacy guarantee by adopting appropriate Gaussian noises. We show the efficiency and accuracy gains of our proposed method with several foundation models and two popular evaluation benchmarks. Furthermore, we present a comprehensive privacy analysis with Gaussian Differential Privacy (GDP) and Renyi Differential Privacy (RDP).

Title: The HalluRAG Dataset: Detecting Closed-Domain Hallucinations in RAG Applications Using an LLM's Internal States

• Authors: Fabian Ridder, Malte Schilling
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17056
• Pdf URL: https://arxiv.org/pdf/2412.17056
• Copy Paste: [[2412.17056]] The HalluRAG Dataset: Detecting Closed-Domain Hallucinations in RAG Applications Using an LLM's Internal States(https://arxiv.org/abs/2412.17056)
• Keywords: large language model
• Abstract: Detecting hallucinations in large language models (LLMs) is critical for enhancing their reliability and trustworthiness. Most research focuses on hallucinations as deviations from information seen during training. However, the opaque nature of an LLM's parametric knowledge complicates the understanding of why generated texts appear ungrounded: The LLM might not have picked up the necessary knowledge from large and often inaccessible datasets, or the information might have been changed or contradicted during further training. Our focus is on hallucinations involving information not used in training, which we determine by using recency to ensure the information emerged after a cut-off date. This study investigates these hallucinations by detecting them at sentence level using different internal states of various LLMs. We present HalluRAG, a dataset designed to train classifiers on these hallucinations. Depending on the model and quantization, MLPs trained on HalluRAG detect hallucinations with test accuracies ranging up to 75 %, with Mistral-7B-Instruct-v0.1 achieving the highest test accuracies. Our results show that IAVs detect hallucinations as effectively as CEVs and reveal that answerable and unanswerable prompts are encoded differently as separate classifiers for these categories improved accuracy. However, HalluRAG showed some limited generalizability, advocating for more diversity in datasets on hallucinations.

Title: Interactive Classification Metrics: A graphical application to build robust intuition for classification model evaluation

• Authors: David H. Brown, Davide Chicco
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17066
• Pdf URL: https://arxiv.org/pdf/2412.17066
• Copy Paste: [[2412.17066]] Interactive Classification Metrics: A graphical application to build robust intuition for classification model evaluation(https://arxiv.org/abs/2412.17066)
• Keywords: robust
• Abstract: Machine learning continues to grow in popularity in academia, in industry, and is increasingly used in other fields. However, most of the common metrics used to evaluate even simple binary classification models have shortcomings that are neither immediately obvious nor consistently taught to practitioners. Here we present Interactive Classification Metrics (ICM), an application to visualize and explore the relationships between different evaluation metrics. The user changes the distribution statistics and explores corresponding changes across a suite of evaluation metrics. The interactive, graphical nature of this tool emphasizes the tradeoffs of each metric without the overhead of data wrangling and model training. The goals of this application are: (1) to aid practitioners in the ever-expanding machine learning field to choose the most appropriate evaluation metrics for their classification problem; (2) to promote careful attention to interpretation that is required even in the simplest scenarios like binary classification. Our application is publicly available for free under the MIT license as a Python package on PyPI at this https URL and on GitHub at this https URL.

Title: MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizes

• Authors: Igor Sokolov, Peter Richtárik
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.17082
• Pdf URL: https://arxiv.org/pdf/2412.17082
• Copy Paste: [[2412.17082]] MARINA-P: Superior Performance in Non-smooth Federated Optimization with Adaptive Stepsizes(https://arxiv.org/abs/2412.17082)
• Keywords: federate
• Abstract: Non-smooth communication-efficient federated optimization is crucial for many machine learning applications, yet remains largely unexplored theoretically. Recent advancements have primarily focused on smooth convex and non-convex regimes, leaving a significant gap in understanding the non-smooth convex setting. Additionally, existing literature often overlooks efficient server-to-worker communication (downlink), focusing primarily on worker-to-server communication (uplink). We consider a setup where uplink costs are negligible and focus on optimizing downlink communication by improving state-of-the-art schemes like EF21-P (arXiv:2209.15218) and MARINA-P (arXiv:2402.06412) in the non-smooth convex setting. We extend the non-smooth convex theory of EF21-P [Anonymous, 2024], originally developed for single-node scenarios, to the distributed setting, and extend MARINA-P to the non-smooth convex setting. For both algorithms, we prove an optimal $O(1/\sqrt{T})$ convergence rate and establish communication complexity bounds matching classical subgradient methods. We provide theoretical guarantees under constant, decreasing, and adaptive (Polyak-type) stepsizes. Our experiments demonstrate that MARINA-P with correlated compressors outperforms other methods in both smooth non-convex and non-smooth convex settings. This work presents the first theoretical results for distributed non-smooth optimization with server-to-worker compression, along with comprehensive analysis for various stepsize schemes.

Title: SAIL: Sample-Centric In-Context Learning for Document Information Extraction

• Authors: Jinyu Zhang, Zhiyuan You, Jize Wang, Xinyi Le
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17092
• Pdf URL: https://arxiv.org/pdf/2412.17092
• Copy Paste: [[2412.17092]] SAIL: Sample-Centric In-Context Learning for Document Information Extraction(https://arxiv.org/abs/2412.17092)
• Keywords: extraction, large language model
• Abstract: Document Information Extraction (DIE) aims to extract structured information from Visually Rich Documents (VRDs). Previous full-training approaches have demonstrated strong performance but may struggle with generalization to unseen data. In contrast, training-free methods leverage powerful pre-trained models like Large Language Models (LLMs) to address various downstream tasks with only a few examples. Nonetheless, training-free methods for DIE encounter two primary challenges: (1) understanding the complex relationship between layout and textual elements in VRDs, and (2) providing accurate guidance to pre-trained models. To address these challenges, we propose Sample-centric In-context Learning (SAIL) for DIE. SAIL introduces a fine-grained entity-level textual similarity to facilitate in-depth text analysis by LLMs and incorporates layout similarity to enhance the analysis of layouts in VRDs. Additionally, SAIL formulates a unified In-Context Learning (ICL) prompt template for various sample-centric examples, enabling tailored prompts that deliver precise guidance to pre-trained models for each sample. Extensive experiments on FUNSD, CORD, and SROIE benchmarks with various base models (e.g., LLMs) indicate that our method outperforms training-free baselines, even closer to the full-training methods. The results show the superiority and generalization of our method.

Title: DreamOmni: Unified Image Generation and Editing

• Authors: Bin Xia, Yuechen Zhang, Jingyao Li, Chengyao Wang, Yitong Wang, Xinglong Wu, Bei Yu, Jiaya Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17098
• Pdf URL: https://arxiv.org/pdf/2412.17098
• Copy Paste: [[2412.17098]] DreamOmni: Unified Image Generation and Editing(https://arxiv.org/abs/2412.17098)
• Keywords: large language model
• Abstract: Currently, the success of large language models (LLMs) illustrates that a unified multitasking approach can significantly enhance model usability, streamline deployment, and foster synergistic benefits across different tasks. However, in computer vision, while text-to-image (T2I) models have significantly improved generation quality through scaling up, their framework design did not initially consider how to unify with downstream tasks, such as various types of editing. To address this, we introduce DreamOmni, a unified model for image generation and editing. We begin by analyzing existing frameworks and the requirements of downstream tasks, proposing a unified framework that integrates both T2I models and various editing tasks. Furthermore, another key challenge is the efficient creation of high-quality editing data, particularly for instruction-based and drag-based editing. To this end, we develop a synthetic data pipeline using sticker-like elements to synthesize accurate, high-quality datasets efficiently, which enables editing data scaling up for unified model training. For training, DreamOmni jointly trains T2I generation and downstream tasks. T2I training enhances the model's understanding of specific concepts and improves generation quality, while editing training helps the model grasp the nuances of the editing task. This collaboration significantly boosts editing performance. Extensive experiments confirm the effectiveness of DreamOmni. The code and model will be released.

Title: Refining CNN-based Heatmap Regression with Gradient-based Corner Points for Electrode Localization

• Authors: Lin Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17105
• Pdf URL: https://arxiv.org/pdf/2412.17105
• Copy Paste: [[2412.17105]] Refining CNN-based Heatmap Regression with Gradient-based Corner Points for Electrode Localization(https://arxiv.org/abs/2412.17105)
• Keywords: extraction
• Abstract: We propose a method for detecting the electrode positions in lithium-ion batteries. The process begins by identifying the region of interest (ROI) in the battery's X-ray image through corner point detection. A convolutional neural network is then used to regress the pole positions within this ROI. Finally, the regressed positions are optimized and corrected using corner point priors, significantly mitigating the loss of localization accuracy caused by operations such as feature map down-sampling and padding during network training. Our findings show that combining traditional pixel gradient analysis with CNN-based heatmap regression for keypoint extraction enhances both accuracy and efficiency, resulting in significant performance improvements.

Title: Similarity Trajectories: Linking Sampling Process to Artifacts in Diffusion-Generated Images

• Authors: Dennis Menn, Feng Liang, Hung-Yueh Chiang, Diana Marculescu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17109
• Pdf URL: https://arxiv.org/pdf/2412.17109
• Copy Paste: [[2412.17109]] Similarity Trajectories: Linking Sampling Process to Artifacts in Diffusion-Generated Images(https://arxiv.org/abs/2412.17109)
• Keywords: diffusion
• Abstract: Artifact detection algorithms are crucial to correcting the output generated by diffusion models. However, because of the variety of artifact forms, existing methods require substantial annotated data for training. This requirement limits their scalability and efficiency, which restricts their wide application. This paper shows that the similarity of denoised images between consecutive time steps during the sampling process is related to the severity of artifacts in images generated by diffusion models. Building on this observation, we introduce the concept of Similarity Trajectory to characterize the sampling process and its correlation with the image artifacts presented. Using an annotated data set of 680 images, which is only 0.1% of the amount of data used in the prior work, we trained a classifier on these trajectories to predict the presence of artifacts in images. By performing 10-fold validation testing on the balanced annotated data set, the classifier can achieve an accuracy of 72.35%, highlighting the connection between the Similarity Trajectory and the occurrence of artifacts. This approach enables differentiation between artifact-exhibiting and natural-looking images using limited training data.

Title: Fair and Accurate Regression: Strong Formulations and Algorithms

• Authors: Anna Deza, Andrés Gómez, Alper Atamtürk
• Subjects: cs.LG, cs.CY, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.17116
• Pdf URL: https://arxiv.org/pdf/2412.17116
• Copy Paste: [[2412.17116]] Fair and Accurate Regression: Strong Formulations and Algorithms(https://arxiv.org/abs/2412.17116)
• Keywords: fair
• Abstract: This paper introduces mixed-integer optimization methods to solve regression problems that incorporate fairness metrics. We propose an exact formulation for training fair regression models. To tackle this computationally hard problem, we study the polynomially-solvable single-factor and single-observation subproblems as building blocks and derive their closed convex hull descriptions. Strong formulations obtained for the general fair regression problem in this manner are utilized to solve the problem with a branch-and-bound algorithm exactly or as a relaxation to produce fair and accurate models rapidly. Moreover, to handle large-scale instances, we develop a coordinate descent algorithm motivated by the convex-hull representation of the single-factor fair regression problem to improve a given solution efficiently. Numerical experiments conducted on fair least squares and fair logistic regression problems show competitive statistical performance with state-of-the-art methods while significantly reducing training times.

Title: Fairness in Reinforcement Learning with Bisimulation Metrics

• Authors: Sahand Rezaei-Shoshtari, Hanna Yurchyk, Scott Fujimoto, Doina Precup, David Meger
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.17123
• Pdf URL: https://arxiv.org/pdf/2412.17123
• Copy Paste: [[2412.17123]] Fairness in Reinforcement Learning with Bisimulation Metrics(https://arxiv.org/abs/2412.17123)
• Keywords: fair
• Abstract: Ensuring long-term fairness is crucial when developing automated decision making systems, specifically in dynamic and sequential environments. By maximizing their reward without consideration of fairness, AI agents can introduce disparities in their treatment of groups or individuals. In this paper, we establish the connection between bisimulation metrics and group fairness in reinforcement learning. We propose a novel approach that leverages bisimulation metrics to learn reward functions and observation dynamics, ensuring that learners treat groups fairly while reflecting the original problem. We demonstrate the effectiveness of our method in addressing disparities in sequential decision making problems through empirical evaluation on a standard fairness benchmark consisting of lending and college admission scenarios.

Title: Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Models

• Authors: Cameron R. Jones, Benjamin K. Bergen
• Subjects: cs.CL, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.17128
• Pdf URL: https://arxiv.org/pdf/2412.17128
• Copy Paste: [[2412.17128]] Lies, Damned Lies, and Distributional Language Statistics: Persuasion and Deception with Large Language Models(https://arxiv.org/abs/2412.17128)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) can generate content that is as persuasive as human-written text and appear capable of selectively producing deceptive outputs. These capabilities raise concerns about potential misuse and unintended consequences as these systems become more widely deployed. This review synthesizes recent empirical work examining LLMs' capacity and proclivity for persuasion and deception, analyzes theoretical risks that could arise from these capabilities, and evaluates proposed mitigations. While current persuasive effects are relatively small, various mechanisms could increase their impact, including fine-tuning, multimodality, and social factors. We outline key open questions for future research, including how persuasive AI systems might become, whether truth enjoys an inherent advantage over falsehoods, and how effective different mitigation strategies may be in practice.

Title: Hate Speech Detection and Target Identification in Devanagari Languages via Parameter Efficient Fine-Tuning of LLMs

• Authors: Rushendra Sidibomma, Pransh Patwa, Parth Patwa, Aman Chadha, Vinija Jain, Amitava Das
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17131
• Pdf URL: https://arxiv.org/pdf/2412.17131
• Copy Paste: [[2412.17131]] Hate Speech Detection and Target Identification in Devanagari Languages via Parameter Efficient Fine-Tuning of LLMs(https://arxiv.org/abs/2412.17131)
• Keywords: large language model
• Abstract: The detection of hate speech has become increasingly important in combating online hostility and its real-world consequences. Despite recent advancements, there is limited research addressing hate speech detection in Devanagari-scripted languages, where resources and tools are scarce. While large language models (LLMs) have shown promise in language-related tasks, traditional fine-tuning approaches are often infeasible given the size of the models. In this paper, we propose a Parameter Efficient Fine tuning (PEFT) based solution for hate speech detection and target identification. We evaluate multiple LLMs on the Devanagari dataset provided by (Thapa et al., 2025), which contains annotated instances in 2 languages - Hindi and Nepali. The results demonstrate the efficacy of our approach in handling Devanagari-scripted content.

Title: A Multi-AI Agent System for Autonomous Optimization of Agentic AI Solutions via Iterative Refinement and LLM-Driven Feedback Loops

• Authors: Kamer Ali Yuksel, Hassan Sawaf
• Subjects: cs.CL, cs.AI, cs.ET, cs.MA, cs.NE
• Abstract URL: https://arxiv.org/abs/2412.17149
• Pdf URL: https://arxiv.org/pdf/2412.17149
• Copy Paste: [[2412.17149]] A Multi-AI Agent System for Autonomous Optimization of Agentic AI Solutions via Iterative Refinement and LLM-Driven Feedback Loops(https://arxiv.org/abs/2412.17149)
• Keywords: robust
• Abstract: Agentic AI systems use specialized agents to handle tasks within complex workflows, enabling automation and efficiency. However, optimizing these systems often requires labor-intensive, manual adjustments to refine roles, tasks, and interactions. This paper introduces a framework for autonomously optimizing Agentic AI solutions across industries, such as NLP-driven enterprise applications. The system employs agents for Refinement, Execution, Evaluation, Modification, and Documentation, leveraging iterative feedback loops powered by an LLM (Llama 3.2-3B). The framework achieves optimal performance without human input by autonomously generating and testing hypotheses to improve system configurations. This approach enhances scalability and adaptability, offering a robust solution for real-world applications in dynamic environments. Case studies across diverse domains illustrate the transformative impact of this framework, showcasing significant improvements in output quality, relevance, and actionability. All data for these case studies, including original and evolved agent codes, along with their outputs, are here: this https URL

Title: SplitFedZip: Learned Compression for Data Transfer Reduction in Split-Federated Learning

• Authors: Chamani Shiranthika, Hadi Hadizadeh, Parvaneh Saeedi, Ivan V. Bajić
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17150
• Pdf URL: https://arxiv.org/pdf/2412.17150
• Copy Paste: [[2412.17150]] SplitFedZip: Learned Compression for Data Transfer Reduction in Split-Federated Learning(https://arxiv.org/abs/2412.17150)
• Keywords: privacy, federate, segmentation
• Abstract: Federated Learning (FL) enables multiple clients to train a collaborative model without sharing their local data. Split Learning (SL) allows a model to be trained in a split manner across different locations. Split-Federated (SplitFed) learning is a more recent approach that combines the strengths of FL and SL. SplitFed minimizes the computational burden of FL by balancing computation across clients and servers, while still preserving data privacy. This makes it an ideal learning framework across various domains, especially in healthcare, where data privacy is of utmost importance. However, SplitFed networks encounter numerous communication challenges, such as latency, bandwidth constraints, synchronization overhead, and a large amount of data that needs to be transferred during the learning process. In this paper, we propose SplitFedZip -- a novel method that employs learned compression to reduce data transfer in SplitFed learning. Through experiments on medical image segmentation, we show that learned compression can provide a significant data communication reduction in SplitFed learning, while maintaining the accuracy of the final trained model. The implementation is available at: \url{this https URL}.

Title: Generative Diffusion Modeling: A Practical Handbook

• Authors: Zihan Ding, Chi Jin
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17162
• Pdf URL: https://arxiv.org/pdf/2412.17162
• Copy Paste: [[2412.17162]] Generative Diffusion Modeling: A Practical Handbook(https://arxiv.org/abs/2412.17162)
• Keywords: robust, fair, diffusion, generative
• Abstract: This handbook offers a unified perspective on diffusion models, encompassing diffusion probabilistic models, score-based generative models, consistency models, rectified flow, and related methods. By standardizing notations and aligning them with code implementations, it aims to bridge the "paper-to-code" gap and facilitate robust implementations and fair comparisons. The content encompasses the fundamentals of diffusion models, the pre-training process, and various post-training methods. Post-training techniques include model distillation and reward-based fine-tuning. Designed as a practical guide, it emphasizes clarity and usability over theoretical depth, focusing on widely adopted approaches in generative modeling with diffusion models.

Title: Where Did Your Model Learn That? Label-free Influence for Self-supervised Learning

• Authors: Nidhin Harilal, Amit Kiran Rege, Reza Akbarian Bafghi, Maziar Raissi, Claire Monteleoni
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17170
• Pdf URL: https://arxiv.org/pdf/2412.17170
• Copy Paste: [[2412.17170]] Where Did Your Model Learn That? Label-free Influence for Self-supervised Learning(https://arxiv.org/abs/2412.17170)
• Keywords: fair
• Abstract: Self-supervised learning (SSL) has revolutionized learning from large-scale unlabeled datasets, yet the intrinsic relationship between pretraining data and the learned representations remains poorly understood. Traditional supervised learning benefits from gradient-based data attribution tools like influence functions that measure the contribution of an individual data point to model predictions. However, existing definitions of influence rely on labels, making them unsuitable for SSL settings. We address this gap by introducing Influence-SSL, a novel and label-free approach for defining influence functions tailored to SSL. Our method harnesses the stability of learned representations against data augmentations to identify training examples that help explain model predictions. We provide both theoretical foundations and empirical evidence to show the utility of Influence-SSL in analyzing pre-trained SSL models. Our analysis reveals notable differences in how SSL models respond to influential data compared to supervised models. Finally, we validate the effectiveness of Influence-SSL through applications in duplicate detection, outlier identification and fairness analysis. Code is available at: \url{this https URL}.

Title: Enhancing Item Tokenization for Generative Recommendation through Self-Improvement

• Authors: Runjin Chen, Mingxuan Ju, Ngoc Bui, Dimosthenis Antypas, Stanley Cai, Xiaopeng Wu, Leonardo Neves, Zhangyang Wang, Neil Shah, Tong Zhao
• Subjects: cs.LG, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.17171
• Pdf URL: https://arxiv.org/pdf/2412.17171
• Copy Paste: [[2412.17171]] Enhancing Item Tokenization for Generative Recommendation through Self-Improvement(https://arxiv.org/abs/2412.17171)
• Keywords: generative, large language model
• Abstract: Generative recommendation systems, driven by large language models (LLMs), present an innovative approach to predicting user preferences by modeling items as token sequences and generating recommendations in a generative manner. A critical challenge in this approach is the effective tokenization of items, ensuring that they are represented in a form compatible with LLMs. Current item tokenization methods include using text descriptions, numerical strings, or sequences of discrete tokens. While text-based representations integrate seamlessly with LLM tokenization, they are often too lengthy, leading to inefficiencies and complicating accurate generation. Numerical strings, while concise, lack semantic depth and fail to capture meaningful item relationships. Tokenizing items as sequences of newly defined tokens has gained traction, but it often requires external models or algorithms for token assignment. These external processes may not align with the LLM's internal pretrained tokenization schema, leading to inconsistencies and reduced model performance. To address these limitations, we propose a self-improving item tokenization method that allows the LLM to refine its own item tokenizations during training process. Our approach starts with item tokenizations generated by any external model and periodically adjusts these tokenizations based on the LLM's learned patterns. Such alignment process ensures consistency between the tokenization and the LLM's internal understanding of the items, leading to more accurate recommendations. Furthermore, our method is simple to implement and can be integrated as a plug-and-play enhancement into existing generative recommendation systems. Experimental results on multiple datasets and using various initial tokenization strategies demonstrate the effectiveness of our method, with an average improvement of 8\% in recommendation performance.

Title: WPMixer: Efficient Multi-Resolution Mixing for Long-Term Time Series Forecasting

• Authors: Md Mahmuddun Nabi Murad, Mehmet Aktukmak, Yasin Yilmaz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17176
• Pdf URL: https://arxiv.org/pdf/2412.17176
• Copy Paste: [[2412.17176]] WPMixer: Efficient Multi-Resolution Mixing for Long-Term Time Series Forecasting(https://arxiv.org/abs/2412.17176)
• Keywords: transformer
• Abstract: Time series forecasting is crucial for various applications, such as weather forecasting, power load forecasting, and financial analysis. In recent studies, MLP-mixer models for time series forecasting have been shown as a promising alternative to transformer-based models. However, the performance of these models is still yet to reach its potential. In this paper, we propose Wavelet Patch Mixer (WPMixer), a novel MLP-based model, for long-term time series forecasting, which leverages the benefits of patching, multi-resolution wavelet decomposition, and mixing. Our model is based on three key components: (i) multi-resolution wavelet decomposition, (ii) patching and embedding, and (iii) MLP mixing. Multi-resolution wavelet decomposition efficiently extracts information in both the frequency and time domains. Patching allows the model to capture an extended history with a look-back window and enhances capturing local information while MLP mixing incorporates global information. Our model significantly outperforms state-of-the-art MLP-based and transformer-based models for long-term time series forecasting in a computationally efficient way, demonstrating its efficacy and potential for practical applications.

Title: Dual Conditioned Motion Diffusion for Pose-Based Video Anomaly Detection

• Authors: Andi Xu, Hongsong Wang, Pinle Ding, Jie Gui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17210
• Pdf URL: https://arxiv.org/pdf/2412.17210
• Copy Paste: [[2412.17210]] Dual Conditioned Motion Diffusion for Pose-Based Video Anomaly Detection(https://arxiv.org/abs/2412.17210)
• Keywords: diffusion, transformer
• Abstract: Video Anomaly Detection (VAD) is essential for computer vision research. Existing VAD methods utilize either reconstruction-based or prediction-based frameworks. The former excels at detecting irregular patterns or structures, whereas the latter is capable of spotting abnormal deviations or trends. We address pose-based video anomaly detection and introduce a novel framework called Dual Conditioned Motion Diffusion (DCMD), which enjoys the advantages of both approaches. The DCMD integrates conditioned motion and conditioned embedding to comprehensively utilize the pose characteristics and latent semantics of observed movements, respectively. In the reverse diffusion process, a motion transformer is proposed to capture potential correlations from multi-layered characteristics within the spectrum space of human motion. To enhance the discriminability between normal and abnormal instances, we design a novel United Association Discrepancy (UAD) regularization that primarily relies on a Gaussian kernel-based time association and a self-attention-based global association. Finally, a mask completion strategy is introduced during the inference stage of the reverse diffusion process to enhance the utilization of conditioned motion for the prediction branch of anomaly detection. Extensive experiments on four datasets demonstrate that our method dramatically outperforms state-of-the-art methods and exhibits superior generalization performance.

Title: Attack by Yourself: Effective and Unnoticeable Multi-Category Graph Backdoor Attacks with Subgraph Triggers Pool

• Authors: Jiangtong Li, Dungy Liu, Dawei Cheng, Changchun Jiang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.17213
• Pdf URL: https://arxiv.org/pdf/2412.17213
• Copy Paste: [[2412.17213]] Attack by Yourself: Effective and Unnoticeable Multi-Category Graph Backdoor Attacks with Subgraph Triggers Pool(https://arxiv.org/abs/2412.17213)
• Keywords: defense, attack
• Abstract: \textbf{G}raph \textbf{N}eural \textbf{N}etworks~(GNNs) have achieved significant success in various real-world applications, including social networks, finance systems, and traffic management. Recent researches highlight their vulnerability to backdoor attacks in node classification, where GNNs trained on a poisoned graph misclassify a test node only when specific triggers are attached. These studies typically focus on single attack categories and use adaptive trigger generators to create node-specific triggers. However, adaptive trigger generators typically have a simple structure, limited parameters, and lack category-aware graph knowledge, which makes them struggle to handle backdoor attacks across multiple categories as the number of target categories increases. We address this gap by proposing a novel approach for \textbf{E}ffective and \textbf{U}nnoticeable \textbf{M}ulti-\textbf{C}ategory~(EUMC) graph backdoor attacks, leveraging subgraph from the attacked graph as category-aware triggers to precisely control the target category. To ensure the effectiveness of our method, we construct a \textbf{M}ulti-\textbf{C}ategory \textbf{S}ubgraph \textbf{T}riggers \textbf{P}ool~(MC-STP) using the subgraphs of the attacked graph as triggers. We then exploit the attachment probability shifts of each subgraph trigger as category-aware priors for target category determination. Moreover, we develop a ``select then attach'' strategy that connects suitable category-aware trigger to attacked nodes for unnoticeability. Extensive experiments across different real-world datasets confirm the efficacy of our method in conducting multi-category graph backdoor attacks on various GNN models and defense strategies.

Title: Discriminative Image Generation with Diffusion Models for Zero-Shot Learning

• Authors: Dingjie Fu, Wenjin Hou, Shiming Chen, Shuhuang Chen, Xinge You, Salman Khan, Fahad Shahbaz Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17219
• Pdf URL: https://arxiv.org/pdf/2412.17219
• Copy Paste: [[2412.17219]] Discriminative Image Generation with Diffusion Models for Zero-Shot Learning(https://arxiv.org/abs/2412.17219)
• Keywords: diffusion, generative
• Abstract: Generative Zero-Shot Learning (ZSL) methods synthesize class-related features based on predefined class semantic prototypes, showcasing superior performance. However, this feature generation paradigm falls short of providing interpretable insights. In addition, existing approaches rely on semantic prototypes annotated by human experts, which exhibit a significant limitation in their scalability to generalized scenes. To overcome these deficiencies, a natural solution is to generate images for unseen classes using text prompts. To this end, We present DIG-ZSL, a novel Discriminative Image Generation framework for Zero-Shot Learning. Specifically, to ensure the generation of discriminative images for training an effective ZSL classifier, we learn a discriminative class token (DCT) for each unseen class under the guidance of a pre-trained category discrimination model (CDM). Harnessing DCTs, we can generate diverse and high-quality images, which serve as informative unseen samples for ZSL tasks. In this paper, the extensive experiments and visualizations on four datasets show that our DIG-ZSL: (1) generates diverse and high-quality images, (2) outperforms previous state-of-the-art nonhuman-annotated semantic prototype-based methods by a large margin, and (3) achieves comparable or better performance than baselines that leverage human-annotated semantic prototypes. The codes will be made available upon acceptance of the paper.

Title: CharGen: High Accurate Character-Level Visual Text Generation Model with MultiModal Encoder

• Authors: Lichen Ma, Tiezhu Yue, Pei Fu, Yujie Zhong, Kai Zhou, Xiaoming Wei, Jie Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17225
• Pdf URL: https://arxiv.org/pdf/2412.17225
• Copy Paste: [[2412.17225]] CharGen: High Accurate Character-Level Visual Text Generation Model with MultiModal Encoder(https://arxiv.org/abs/2412.17225)
• Keywords: diffusion
• Abstract: Recently, significant advancements have been made in diffusion-based visual text generation models. Although the effectiveness of these methods in visual text rendering is rapidly improving, they still encounter challenges such as inaccurate characters and strokes when rendering complex visual text. In this paper, we propose CharGen, a highly accurate character-level visual text generation and editing model. Specifically, CharGen employs a character-level multimodal encoder that not only extracts character-level text embeddings but also encodes glyph images character by character. This enables it to capture fine-grained cross-modality features more effectively. Additionally, we introduce a new perceptual loss in CharGen to enhance character shape supervision and address the issue of inaccurate strokes in generated text. It is worth mentioning that CharGen can be integrated into existing diffusion models to generate visual text with high accuracy. CharGen significantly improves text rendering accuracy, outperforming recent methods in public benchmarks such as AnyText-benchmark and MARIO-Eval, with improvements of more than 8% and 6%, respectively. Notably, CharGen achieved a 5.5% increase in accuracy on Chinese test sets.

Title: OLiDM: Object-aware LiDAR Diffusion Models for Autonomous Driving

• Authors: Tianyi Yan, Junbo Yin, Xianpeng Lang, Ruigang Yang, Cheng-Zhong Xu, Jianbing Shen
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.17226
• Pdf URL: https://arxiv.org/pdf/2412.17226
• Copy Paste: [[2412.17226]] OLiDM: Object-aware LiDAR Diffusion Models for Autonomous Driving(https://arxiv.org/abs/2412.17226)
• Keywords: diffusion
• Abstract: To enhance autonomous driving safety in complex scenarios, various methods have been proposed to simulate LiDAR point cloud data. Nevertheless, these methods often face challenges in producing high-quality, diverse, and controllable foreground objects. To address the needs of object-aware tasks in 3D perception, we introduce OLiDM, a novel framework capable of generating high-fidelity LiDAR data at both the object and the scene levels. OLiDM consists of two pivotal components: the Object-Scene Progressive Generation (OPG) module and the Object Semantic Alignment (OSA) module. OPG adapts to user-specific prompts to generate desired foreground objects, which are subsequently employed as conditions in scene generation, ensuring controllable outputs at both the object and scene levels. This also facilitates the association of user-defined object-level annotations with the generated LiDAR scenes. Moreover, OSA aims to rectify the misalignment between foreground objects and background scenes, enhancing the overall quality of the generated objects. The broad effectiveness of OLiDM is demonstrated across various LiDAR generation tasks, as well as in 3D perception tasks. Specifically, on the KITTI-360 dataset, OLiDM surpasses prior state-of-the-art methods such as UltraLiDAR by 17.5 in FPD. Additionally, in sparse-to-dense LiDAR completion, OLiDM achieves a significant improvement over LiDARGen, with a 57.47\% increase in semantic IoU. Moreover, OLiDM enhances the performance of mainstream 3D detectors by 2.4\% in mAP and 1.9\% in NDS, underscoring its potential in advancing object-aware 3D tasks. Code is available at: this https URL.

Title: Brain-to-Text Benchmark '24: Lessons Learned

• Authors: Francis R. Willett, Jingyuan Li, Trung Le, Chaofei Fan, Mingfei Chen, Eli Shlizerman, Yue Chen, Xin Zheng, Tatsuo S. Okubo, Tyler Benster, Hyun Dong Lee, Maxwell Kounga, E. Kelly Buchanan, David Zoltowski, Scott W. Linderman, Jaimie M. Henderson
• Subjects: cs.CL, cs.LG, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2412.17227
• Pdf URL: https://arxiv.org/pdf/2412.17227
• Copy Paste: [[2412.17227]] Brain-to-Text Benchmark '24: Lessons Learned(https://arxiv.org/abs/2412.17227)
• Keywords: transformer, large language model
• Abstract: Speech brain-computer interfaces aim to decipher what a person is trying to say from neural activity alone, restoring communication to people with paralysis who have lost the ability to speak intelligibly. The Brain-to-Text Benchmark '24 and associated competition was created to foster the advancement of decoding algorithms that convert neural activity to text. Here, we summarize the lessons learned from the competition ending on June 1, 2024 (the top 4 entrants also presented their experiences in a recorded webinar). The largest improvements in accuracy were achieved using an ensembling approach, where the output of multiple independent decoders was merged using a fine-tuned large language model (an approach used by all 3 top entrants). Performance gains were also found by improving how the baseline recurrent neural network (RNN) model was trained, including by optimizing learning rate scheduling and by using a diphone training objective. Improving upon the model architecture itself proved more difficult, however, with attempts to use deep state space models or transformers not yet appearing to offer a benefit over the RNN baseline. The benchmark will remain open indefinitely to support further work towards increasing the accuracy of brain-to-text algorithms.

Title: FedMeld: A Model-dispersal Federated Learning Framework for Space-ground Integrated Networks

• Authors: Qian Chen, Xianhao Chen, Kaibin Huang
• Subjects: cs.LG, cs.IT, cs.NI
• Abstract URL: https://arxiv.org/abs/2412.17231
• Pdf URL: https://arxiv.org/pdf/2412.17231
• Copy Paste: [[2412.17231]] FedMeld: A Model-dispersal Federated Learning Framework for Space-ground Integrated Networks(https://arxiv.org/abs/2412.17231)
• Keywords: federate
• Abstract: To bridge the digital divide, the space-ground integrated networks (SGINs), which will be a key component of the six-generation (6G) mobile networks, are expected to deliver artificial intelligence (AI) services to every corner of the world. One mission of SGINs is to support federated learning (FL) at a global scale. However, existing space-ground integrated FL frameworks involve ground stations or costly inter-satellite links, entailing excessive training latency and communication costs. To overcome these limitations, we propose an infrastructure-free federated learning framework based on a model dispersal (FedMeld) strategy, which exploits periodic movement patterns and store-carry-forward capabilities of satellites to enable parameter mixing across large-scale geographical regions. We theoretically show that FedMeld leads to global model convergence and quantify the effects of round interval and mixing ratio between adjacent areas on its learning performance. Based on the theoretical results, we formulate a joint optimization problem to design the staleness control and mixing ratio (SC-MR) for minimizing the training loss. By decomposing the problem into sequential SC and MR subproblems without compromising the optimality, we derive the round interval solution in a closed form and the mixing ratio in a semi-closed form to achieve the \textit{optimal} latency-accuracy tradeoff. Experiments using various datasets demonstrate that FedMeld achieves superior model accuracy while significantly reducing communication costs as compared with traditional FL schemes for SGINs.

Title: Unity is Strength: Unifying Convolutional and Transformeral Features for Better Person Re-Identification

• Authors: Yuhao Wang, Pingping Zhang, Xuehu Liu, Zhengzheng Tu, Huchuan Lu
• Subjects: cs.CV, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.17239
• Pdf URL: https://arxiv.org/pdf/2412.17239
• Copy Paste: [[2412.17239]] Unity is Strength: Unifying Convolutional and Transformeral Features for Better Person Re-Identification(https://arxiv.org/abs/2412.17239)
• Keywords: extraction, transformer
• Abstract: Person Re-identification (ReID) aims to retrieve the specific person across non-overlapping cameras, which greatly helps intelligent transportation systems. As we all know, Convolutional Neural Networks (CNNs) and Transformers have the unique strengths to extract local and global features, respectively. Considering this fact, we focus on the mutual fusion between them to learn more comprehensive representations for persons. In particular, we utilize the complementary integration of deep features from different model structures. We propose a novel fusion framework called FusionReID to unify the strengths of CNNs and Transformers for image-based person ReID. More specifically, we first deploy a Dual-branch Feature Extraction (DFE) to extract features through CNNs and Transformers from a single image. Moreover, we design a novel Dual-attention Mutual Fusion (DMF) to achieve sufficient feature fusions. The DMF comprises Local Refinement Units (LRU) and Heterogenous Transmission Modules (HTM). LRU utilizes depth-separable convolutions to align deep features in channel dimensions and spatial sizes. HTM consists of a Shared Encoding Unit (SEU) and two Mutual Fusion Units (MFU). Through the continuous stacking of HTM, deep features after LRU are repeatedly utilized to generate more discriminative features. Extensive experiments on three public ReID benchmarks demonstrate that our method can attain superior performances than most state-of-the-arts. The source code is available at this https URL.

Title: QTSeg: A Query Token-Based Architecture for Efficient 2D Medical Image Segmentation

• Authors: Phuong-Nam Tran, Nhat Truong Pham, Duc Ngoc Minh Dang, Eui-Nam Huh, Choong Seon Hong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17241
• Pdf URL: https://arxiv.org/pdf/2412.17241
• Copy Paste: [[2412.17241]] QTSeg: A Query Token-Based Architecture for Efficient 2D Medical Image Segmentation(https://arxiv.org/abs/2412.17241)
• Keywords: transformer, segmentation
• Abstract: Medical image segmentation is crucial in assisting medical doctors in making diagnoses and enabling accurate automatic diagnosis. While advanced convolutional neural networks (CNNs) excel in segmenting regions of interest with pixel-level precision, they often struggle with long-range dependencies, which is crucial for enhancing model performance. Conversely, transformer architectures leverage attention mechanisms to excel in handling long-range dependencies. However, the computational complexity of transformers grows quadratically, posing resource-intensive challenges, especially with high-resolution medical images. Recent research aims to combine CNN and transformer architectures to mitigate their drawbacks and enhance performance while keeping resource demands low. Nevertheless, existing approaches have not fully leveraged the strengths of both architectures to achieve high accuracy with low computational requirements. To address this gap, we propose a novel architecture for 2D medical image segmentation (QTSeg) that leverages a feature pyramid network (FPN) as the image encoder, a multi-level feature fusion (MLFF) as the adaptive module between encoder and decoder and a multi-query mask decoder (MQM Decoder) as the mask decoder. In the first step, an FPN model extracts pyramid features from the input image. Next, MLFF is incorporated between the encoder and decoder to adapt features from different encoder stages to the decoder. Finally, an MQM Decoder is employed to improve mask generation by integrating query tokens with pyramid features at all stages of the mask decoder. Our experimental results show that QTSeg outperforms state-of-the-art methods across all metrics with lower computational demands than the baseline and the existing methods. Code is available at this https URL (v0.1.0)

Title: STeInFormer: Spatial-Temporal Interaction Transformer Architecture for Remote Sensing Change Detection

• Authors: Xiaowen Ma, Zhenkai Wu, Mengting Ma, Mengjiao Zhao, Fan Yang, Zhenhong Du, Wei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17247
• Pdf URL: https://arxiv.org/pdf/2412.17247
• Copy Paste: [[2412.17247]] STeInFormer: Spatial-Temporal Interaction Transformer Architecture for Remote Sensing Change Detection(https://arxiv.org/abs/2412.17247)
• Keywords: extraction, transformer
• Abstract: Convolutional neural networks and attention mechanisms have greatly benefited remote sensing change detection (RSCD) because of their outstanding discriminative ability. Existent RSCD methods often follow a paradigm of using a non-interactive Siamese neural network for multi-temporal feature extraction and change detection heads for feature fusion and change representation. However, this paradigm lacks the contemplation of the characteristics of RSCD in temporal and spatial dimensions, and causes the drawback on spatial-temporal interaction that hinders high-quality feature extraction. To address this problem, we present STeInFormer, a spatial-temporal interaction Transformer architecture for multi-temporal feature extraction, which is the first general backbone network specifically designed for RSCD. In addition, we propose a parameter-free multi-frequency token mixer to integrate frequency-domain features that provide spectral information for RSCD. Experimental results on three datasets validate the effectiveness of the proposed method, which can outperform the state-of-the-art methods and achieve the most satisfactory efficiency-accuracy trade-off. Code is available at this https URL.

Title: GCS-M3VLT: Guided Context Self-Attention based Multi-modal Medical Vision Language Transformer for Retinal Image Captioning

• Authors: Teja Krishna Cherukuri, Nagur Shareef Shaik, Jyostna Devi Bodapati, Dong Hye Ye
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.17251
• Pdf URL: https://arxiv.org/pdf/2412.17251
• Copy Paste: [[2412.17251]] GCS-M3VLT: Guided Context Self-Attention based Multi-modal Medical Vision Language Transformer for Retinal Image Captioning(https://arxiv.org/abs/2412.17251)
• Keywords: transformer
• Abstract: Retinal image analysis is crucial for diagnosing and treating eye diseases, yet generating accurate medical reports from images remains challenging due to variability in image quality and pathology, especially with limited labeled data. Previous Transformer-based models struggled to integrate visual and textual information under limited supervision. In response, we propose a novel vision-language model for retinal image captioning that combines visual and textual features through a guided context self-attention mechanism. This approach captures both intricate details and the global clinical context, even in data-scarce scenarios. Extensive experiments on the DeepEyeNet dataset demonstrate a 0.023 BLEU@4 improvement, along with significant qualitative advancements, highlighting the effectiveness of our model in generating comprehensive medical captions.

Title: A Coalition Game for On-demand Multi-modal 3D Automated Delivery System

• Authors: Farzan Moosavi, Bilal Farooq
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2412.17252
• Pdf URL: https://arxiv.org/pdf/2412.17252
• Copy Paste: [[2412.17252]] A Coalition Game for On-demand Multi-modal 3D Automated Delivery System(https://arxiv.org/abs/2412.17252)
• Keywords: robust, transformer
• Abstract: We introduce a multi-modal autonomous delivery optimization framework as a coalition game for a fleet of UAVs and ADRs operating in two overlaying networks to address last-mile delivery in urban environments, including high-density areas, road-based routing, and real-world operational challenges. The problem is defined as multiple depot pickup and delivery with time windows constrained over operational restrictions, such as vehicle battery limitation, precedence time window, and building obstruction. Subsequently, the coalition game theory is applied to investigate cooperation structures among the modes to capture how strategic collaboration among vehicles can improve overall routing efficiency. To do so, a generalized reinforcement learning model is designed to evaluate the cost-sharing and allocation to different coalitions for which sub-additive property and non-empty core exist. Our methodology leverages an end-to-end deep multi-agent policy gradient method augmented by a novel spatio-temporal adjacency neighbourhood graph attention network and transformer architecture using a heterogeneous edge-enhanced attention model. Conducting several numerical experiments on last-mile delivery applications, the result from the case study in the city of Mississauga shows that despite the incorporation of an extensive network in the graph for two modes and a complex training structure, the model addresses realistic operational constraints and achieves high-quality solutions compared with the existing transformer-based and heuristics methods and can perform well on non-homogeneous data distribution, generalizes well on the different scale and configuration, and demonstrate a robust performance under stochastic scenarios subject to wind speed and direction.

Title: Enhancing Multi-Text Long Video Generation Consistency without Tuning: Time-Frequency Analysis, Prompt Alignment, and Theory

• Authors: Xingyao Li, Fengzhuo Zhang, Jiachun Pan, Yunlong Hou, Vincent Y. F. Tan, Zhuoran Yang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17254
• Pdf URL: https://arxiv.org/pdf/2412.17254
• Copy Paste: [[2412.17254]] Enhancing Multi-Text Long Video Generation Consistency without Tuning: Time-Frequency Analysis, Prompt Alignment, and Theory(https://arxiv.org/abs/2412.17254)
• Keywords: diffusion
• Abstract: Despite the considerable progress achieved in the long video generation problem, there is still significant room to improve the consistency of the videos, particularly in terms of smoothness and transitions between scenes. We address these issues to enhance the consistency and coherence of videos generated with either single or multiple prompts. We propose the Time-frequency based temporal Attention Reweighting Algorithm (TiARA), which meticulously edits the attention score matrix based on the Discrete Short-Time Fourier Transform. Our method is supported by a theoretical guarantee, the first-of-its-kind for frequency-based methods in diffusion models. For videos generated by multiple prompts, we further investigate key factors affecting prompt interpolation quality and propose PromptBlend, an advanced prompt interpolation pipeline. The efficacy of our proposed method is validated via extensive experimental results, exhibiting consistent and impressive improvements over baseline methods. The code will be released upon acceptance.

Title: Unlocking Cross-Lingual Sentiment Analysis through Emoji Interpretation: A Multimodal Generative AI Approach

• Authors: Rafid Ishrak Jahan, Heng Fan, Haihua Chen, Yunhe Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17255
• Pdf URL: https://arxiv.org/pdf/2412.17255
• Copy Paste: [[2412.17255]] Unlocking Cross-Lingual Sentiment Analysis through Emoji Interpretation: A Multimodal Generative AI Approach(https://arxiv.org/abs/2412.17255)
• Keywords: generative, large language model
• Abstract: Emojis have become ubiquitous in online communication, serving as a universal medium to convey emotions and decorative elements. Their widespread use transcends language and cultural barriers, enhancing understanding and fostering more inclusive interactions. While existing work gained valuable insight into emojis understanding, exploring emojis' capability to serve as a universal sentiment indicator leveraging large language models (LLMs) has not been thoroughly examined. Our study aims to investigate the capacity of emojis to serve as reliable sentiment markers through LLMs across languages and cultures. We leveraged the multimodal capabilities of ChatGPT to explore the sentiments of various representations of emojis and evaluated how well emoji-conveyed sentiment aligned with text sentiment on a multi-lingual dataset collected from 32 countries. Our analysis reveals that the accuracy of LLM-based emoji-conveyed sentiment is 81.43%, underscoring emojis' significant potential to serve as a universal sentiment marker. We also found a consistent trend that the accuracy of sentiment conveyed by emojis increased as the number of emojis grew in text. The results reinforce the potential of emojis to serve as global sentiment indicators, offering insight into fields such as cross-lingual and cross-cultural sentiment analysis on social media platforms. Code: this https URL.

Title: An Intrinsically Explainable Approach to Detecting Vertebral Compression Fractures in CT Scans via Neurosymbolic Modeling

• Authors: Blanca Inigo, Yiqing Shen, Benjamin D. Killeen, Michelle Song, Axel Krieger, Christopher Bradley, Mathias Unberath
• Subjects: cs.LG, cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.17258
• Pdf URL: https://arxiv.org/pdf/2412.17258
• Copy Paste: [[2412.17258]] An Intrinsically Explainable Approach to Detecting Vertebral Compression Fractures in CT Scans via Neurosymbolic Modeling(https://arxiv.org/abs/2412.17258)
• Keywords: interpretability, segmentation
• Abstract: Vertebral compression fractures (VCFs) are a common and potentially serious consequence of osteoporosis. Yet, they often remain undiagnosed. Opportunistic screening, which involves automated analysis of medical imaging data acquired primarily for other purposes, is a cost-effective method to identify undiagnosed VCFs. In high-stakes scenarios like opportunistic medical diagnosis, model interpretability is a key factor for the adoption of AI recommendations. Rule-based methods are inherently explainable and closely align with clinical guidelines, but they are not immediately applicable to high-dimensional data such as CT scans. To address this gap, we introduce a neurosymbolic approach for VCF detection in CT volumes. The proposed model combines deep learning (DL) for vertebral segmentation with a shape-based algorithm (SBA) that analyzes vertebral height distributions in salient anatomical regions. This allows for the definition of a rule set over the height distributions to detect VCFs. Evaluation of VerSe19 dataset shows that our method achieves an accuracy of 96% and a sensitivity of 91% in VCF detection. In comparison, a black box model, DenseNet, achieved an accuracy of 95% and sensitivity of 91% in the same dataset. Our results demonstrate that our intrinsically explainable approach can match or surpass the performance of black box deep neural networks while providing additional insights into why a prediction was made. This transparency can enhance clinician's trust thus, supporting more informed decision-making in VCF diagnosis and treatment planning.

Title: Multi-view Fuzzy Graph Attention Networks for Enhanced Graph Learning

• Authors: Jinming Xing, Dongwen Luo, Qisen Cheng, Chang Xue, Ruilin Xing
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17271
• Pdf URL: https://arxiv.org/pdf/2412.17271
• Copy Paste: [[2412.17271]] Multi-view Fuzzy Graph Attention Networks for Enhanced Graph Learning(https://arxiv.org/abs/2412.17271)
• Keywords: robust
• Abstract: Fuzzy Graph Attention Network (FGAT), which combines Fuzzy Rough Sets and Graph Attention Networks, has shown promise in tasks requiring robust graph-based learning. However, existing models struggle to effectively capture dependencies from multiple perspectives, limiting their ability to model complex data. To address this gap, we propose the Multi-view Fuzzy Graph Attention Network (MFGAT), a novel framework that constructs and aggregates multi-view information using a specially designed Transformation Block. This block dynamically transforms data from multiple aspects and aggregates the resulting representations via a weighted sum mechanism, enabling comprehensive multi-view modeling. The aggregated information is fed into FGAT to enhance fuzzy graph convolutions. Additionally, we introduce a simple yet effective learnable global pooling mechanism for improved graph-level understanding. Extensive experiments on graph classification tasks demonstrate that MFGAT outperforms state-of-the-art baselines, underscoring its effectiveness and versatility.

Title: Learning from Mistakes: Self-correct Adversarial Training for Chinese Unnatural Text Correction

• Authors: Xuan Feng, Tianlong Gu, Xiaoli Liu, Liang Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17279
• Pdf URL: https://arxiv.org/pdf/2412.17279
• Copy Paste: [[2412.17279]] Learning from Mistakes: Self-correct Adversarial Training for Chinese Unnatural Text Correction(https://arxiv.org/abs/2412.17279)
• Keywords: defense
• Abstract: Unnatural text correction aims to automatically detect and correct spelling errors or adversarial perturbation errors in sentences. Existing methods typically rely on fine-tuning or adversarial training to correct errors, which have achieved significant success. However, these methods exhibit poor generalization performance due to the difference in data distribution between training data and real-world scenarios, known as the exposure bias problem. In this paper, we propose a self-correct adversarial training framework for \textbf{L}earn\textbf{I}ng from \textbf{MI}s\textbf{T}akes (\textbf{LIMIT}), which is a task- and model-independent framework to correct unnatural errors or mistakes. Specifically, we fully utilize errors generated by the model that are actively exposed during the inference phase, i.e., predictions that are inconsistent with the target. This training method not only simulates potential errors in real application scenarios, but also mitigates the exposure bias of the traditional training process. Meanwhile, we design a novel decoding intervention strategy to maintain semantic consistency. Extensive experimental results on Chinese unnatural text error correction datasets show that our proposed method can correct multiple forms of errors and outperforms the state-of-the-art text correction methods. In addition, extensive results on Chinese and English datasets validate that LIMIT can serve as a plug-and-play defense module and can extend to new models and datasets without further training.

Title: Free-viewpoint Human Animation with Pose-correlated Reference Selection

• Authors: Fa-Ting Hong, Zhan Xu, Haiyang Liu, Qinjie Lin, Luchuan Song, Zhixin Shu, Yang Zhou, Duygu Ceylan, Dan Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17290
• Pdf URL: https://arxiv.org/pdf/2412.17290
• Copy Paste: [[2412.17290]] Free-viewpoint Human Animation with Pose-correlated Reference Selection(https://arxiv.org/abs/2412.17290)
• Keywords: diffusion, generative
• Abstract: Diffusion-based human animation aims to animate a human character based on a source human image as well as driving signals such as a sequence of poses. Leveraging the generative capacity of diffusion model, existing approaches are able to generate high-fidelity poses, but struggle with significant viewpoint changes, especially in zoom-in/zoom-out scenarios where camera-character distance varies. This limits the applications such as cinematic shot type plan or camera control. We propose a pose-correlated reference selection diffusion network, supporting substantial viewpoint variations in human animation. Our key idea is to enable the network to utilize multiple reference images as input, since significant viewpoint changes often lead to missing appearance details on the human body. To eliminate the computational cost, we first introduce a novel pose correlation module to compute similarities between non-aligned target and source poses, and then propose an adaptive reference selection strategy, utilizing the attention map to identify key regions for animation generation. To train our model, we curated a large dataset from public TED talks featuring varied shots of the same character, helping the model learn synthesis for different perspectives. Our experimental results show that with the same number of reference images, our model performs favorably compared to the current SOTA methods under large viewpoint change. We further show that the adaptive reference selection is able to choose the most relevant reference regions to generate humans under free viewpoints.

Title: Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding

• Authors: Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Qun Liu, Dongyan Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17295
• Pdf URL: https://arxiv.org/pdf/2412.17295
• Copy Paste: [[2412.17295]] Friends-MMC: A Dataset for Multi-modal Multi-party Conversation Understanding(https://arxiv.org/abs/2412.17295)
• Keywords: generative
• Abstract: Multi-modal multi-party conversation (MMC) is a less studied yet important topic of research due to that it well fits real-world scenarios and thus potentially has more widely-used applications. Compared with the traditional multi-modal conversations, MMC requires stronger character-centered understanding abilities as there are many interlocutors appearing in both the visual and textual context. To facilitate the study of this problem, we present Friends-MMC in this paper, an MMC dataset that contains 24,000+ unique utterances paired with video context. To explore the character-centered understanding of the dialogue, we also annotate the speaker of each utterance, the names and bounding bboxes of faces that appear in the video. Based on this Friends-MMC dataset, we further study two fundamental MMC tasks: conversation speaker identification and conversation response prediction, both of which have the multi-party nature with the video or image as visual context. For conversation speaker identification, we demonstrate the inefficiencies of existing methods such as pre-trained models, and propose a simple yet effective baseline method that leverages an optimization solver to utilize the context of two modalities to achieve better performance. For conversation response prediction, we fine-tune generative dialogue models on Friend-MMC, and analyze the benefits of speaker information. The code and dataset is publicly available at this https URL and thus we call for more attention on modeling speaker information when understanding conversations.

Title: When Focus Enhances Utility: Target Range LDP Frequency Estimation and Unknown Item Discovery

• Authors: Bo Jiang, Wanrong Zhang, Donghang Lu, Jian Du, Qiang Yan
• Subjects: cs.CR, cs.DB
• Abstract URL: https://arxiv.org/abs/2412.17303
• Pdf URL: https://arxiv.org/pdf/2412.17303
• Copy Paste: [[2412.17303]] When Focus Enhances Utility: Target Range LDP Frequency Estimation and Unknown Item Discovery(https://arxiv.org/abs/2412.17303)
• Keywords: privacy
• Abstract: Local Differential Privacy (LDP) protocols enable the collection of randomized client messages for data analysis, without the necessity of a trusted data curator. Such protocols have been successfully deployed in real-world scenarios by major tech companies like Google, Apple, and Microsoft. In this paper, we propose a Generalized Count Mean Sketch (GCMS) protocol that captures many existing frequency estimation protocols. Our method significantly improves the three-way trade-offs between communication, privacy, and accuracy. We also introduce a general utility analysis framework that enables optimizing parameter designs. {Based on that, we propose an Optimal Count Mean Sketch (OCMS) framework that minimizes the variance for collecting items with targeted frequencies.} Moreover, we present a novel protocol for collecting data within unknown domain, as our frequency estimation protocols only work effectively with known data domain. Leveraging the stability-based histogram technique alongside the Encryption-Shuffling-Analysis (ESA) framework, our approach employs an auxiliary server to construct histograms without accessing original data messages. This protocol achieves accuracy akin to the central DP model while offering local-like privacy guarantees and substantially lowering computational costs.

Title: FedLEC: Effective Federated Learning Algorithm with Spiking Neural Networks Under Label Skews

• Authors: Di Yu, Xin Du, Linshan Jiang, Shunwen Bai, Wentao Tong, Shuiguang Deng
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17305
• Pdf URL: https://arxiv.org/pdf/2412.17305
• Copy Paste: [[2412.17305]] FedLEC: Effective Federated Learning Algorithm with Spiking Neural Networks Under Label Skews(https://arxiv.org/abs/2412.17305)
• Keywords: federate
• Abstract: With the advancement of neuromorphic chips, implementing Federated Learning (FL) with Spiking Neural Networks (SNNs) potentially offers a more energy-efficient schema for collaborative learning across various resource-constrained edge devices. However, one significant challenge in the FL systems is that the data from different clients are often non-independently and identically distributed (non-IID), with label skews presenting substantial difficulties in various federated SNN learning tasks. In this study, we propose a practical post-hoc framework named FedLEC to address the challenge. This framework penalizes the corresponding local logits for locally missing labels to enhance each local model's generalization ability. Additionally, it leverages the pertinent label distribution information distilled from the global model to mitigate label bias. Extensive experiments with three different structured SNNs across five datasets (i.e., three non-neuromorphic and two neuromorphic datasets) demonstrate the efficiency of FedLEC. Compared to seven state-of-the-art FL algorithms, FedLEC achieves an average accuracy improvement of approximately 11.59\% under various label skew distribution settings.

Title: Improving Pareto Set Learning for Expensive Multi-objective Optimization via Stein Variational Hypernetworks

• Authors: Minh-Duc Nguyen, Phuong Mai Dinh, Quang-Huy Nguyen, Long P. Hoang, Dung D. Le
• Subjects: cs.LG, cs.NE, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.17312
• Pdf URL: https://arxiv.org/pdf/2412.17312
• Copy Paste: [[2412.17312]] Improving Pareto Set Learning for Expensive Multi-objective Optimization via Stein Variational Hypernetworks(https://arxiv.org/abs/2412.17312)
• Keywords: robust
• Abstract: Expensive multi-objective optimization problems (EMOPs) are common in real-world scenarios where evaluating objective functions is costly and involves extensive computations or physical experiments. Current Pareto set learning methods for such problems often rely on surrogate models like Gaussian processes to approximate the objective functions. These surrogate models can become fragmented, resulting in numerous small uncertain regions between explored solutions. When using acquisition functions such as the Lower Confidence Bound (LCB), these uncertain regions can turn into pseudo-local optima, complicating the search for globally optimal solutions. To address these challenges, we propose a novel approach called SVH-PSL, which integrates Stein Variational Gradient Descent (SVGD) with Hypernetworks for efficient Pareto set learning. Our method addresses the issues of fragmented surrogate models and pseudo-local optima by collectively moving particles in a manner that smooths out the solution space. The particles interact with each other through a kernel function, which helps maintain diversity and encourages the exploration of underexplored regions. This kernel-based interaction prevents particles from clustering around pseudo-local optima and promotes convergence towards globally optimal solutions. Our approach aims to establish robust relationships between trade-off reference vectors and their corresponding true Pareto solutions, overcoming the limitations of existing methods. Through extensive experiments across both synthetic and real-world MOO benchmarks, we demonstrate that SVH-PSL significantly improves the quality of the learned Pareto set, offering a promising solution for expensive multi-objective optimization problems.

Title: Collaborative Optimization in Financial Data Mining Through Deep Learning and ResNeXt

• Authors: Pengbin Feng, Yankaiqi Li, Yijiashun Qi, Xiaojun Guo, Zhenghao Lin
• Subjects: cs.LG, q-fin.CP
• Abstract URL: https://arxiv.org/abs/2412.17314
• Pdf URL: https://arxiv.org/pdf/2412.17314
• Copy Paste: [[2412.17314]] Collaborative Optimization in Financial Data Mining Through Deep Learning and ResNeXt(https://arxiv.org/abs/2412.17314)
• Keywords: robust, extraction
• Abstract: This study proposes a multi-task learning framework based on ResNeXt, aiming to solve the problem of feature extraction and task collaborative optimization in financial data mining. Financial data usually has the complex characteristics of high dimensionality, nonlinearity, and time series, and is accompanied by potential correlations between multiple tasks, making it difficult for traditional methods to meet the needs of data mining. This study introduces the ResNeXt model into the multi-task learning framework and makes full use of its group convolution mechanism to achieve efficient extraction of local patterns and global features of financial data. At the same time, through the design of task sharing layers and dedicated layers, it is established between multiple related tasks. Deep collaborative optimization relationships. Through flexible multi-task loss weight design, the model can effectively balance the learning needs of different tasks and improve overall performance. Experiments are conducted on a real S&P 500 financial data set, verifying the significant advantages of the proposed framework in classification and regression tasks. The results indicate that, when compared to other conventional deep learning models, the proposed method delivers superior performance in terms of accuracy, F1 score, root mean square error, and other metrics, highlighting its outstanding effectiveness and robustness in handling complex financial data. This research provides an efficient and adaptable solution for financial data mining, and at the same time opens up a new research direction for the combination of multi-task learning and deep learning, which has important theoretical significance and practical application value.

Title: Fast Gradient Computation for RoPE Attention in Almost Linear Time

• Authors: Yifang Chen, Jiayan Huo, Xiaoyu Li, Yingyu Liang, Zhenmei Shi, Zhao Song
• Subjects: cs.LG, cs.AI, cs.CC, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17316
• Pdf URL: https://arxiv.org/pdf/2412.17316
• Copy Paste: [[2412.17316]] Fast Gradient Computation for RoPE Attention in Almost Linear Time(https://arxiv.org/abs/2412.17316)
• Keywords: transformer
• Abstract: The Rotary Position Embedding (RoPE) mechanism has become a powerful enhancement to the Transformer architecture, which enables models to capture token relationships when encoding positional information. However, the RoPE mechanisms make the computations of attention mechanisms more complicated, which makes efficient algorithms challenging. Earlier research introduced almost linear time, i.e., $n^{1+o(1)}$ where $n$ is the number of input tokens, algorithms for the forward computation under specific parameter settings. However, achieving a subquadratic time algorithm for other parameter regimes remains impossible unless the widely accepted Strong Exponential Time Hypothesis (SETH) is disproven. In this work, we develop the first almost linear time algorithm for backward computations in the RoPE-based attention under bounded entries. Our approach builds on recent advancements in fast RoPE attention computations, utilizing a novel combination of the polynomial method and the Fast Fourier Transform. Furthermore, we show that with lower bounds derived from the SETH, the bounded entry condition is necessary for subquadratic performance.

Title: Better Knowledge Enhancement for Privacy-Preserving Cross-Project Defect Prediction

• Authors: Yuying Wang, Yichen Li, Haozhao Wang, Lei Zhao, Xiaofang Zhang
• Subjects: cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.17317
• Pdf URL: https://arxiv.org/pdf/2412.17317
• Copy Paste: [[2412.17317]] Better Knowledge Enhancement for Privacy-Preserving Cross-Project Defect Prediction(https://arxiv.org/abs/2412.17317)
• Keywords: privacy, federate
• Abstract: Cross-Project Defect Prediction (CPDP) poses a non-trivial challenge to construct a reliable defect predictor by leveraging data from other projects, particularly when data owners are concerned about data privacy. In recent years, Federated Learning (FL) has become an emerging paradigm to guarantee privacy information by collaborative training a global model among multiple parties without sharing raw data. While the direct application of FL to the CPDP task offers a promising solution to address privacy concerns, the data heterogeneity arising from proprietary projects across different companies or organizations will bring troubles for model training. In this paper, we study the privacy-preserving cross-project defect prediction with data heterogeneity under the federated learning framework. To address this problem, we propose a novel knowledge enhancement approach named FedDP with two simple but effective solutions: 1. Local Heterogeneity Awareness and 2. Global Knowledge Distillation. Specifically, we employ open-source project data as the distillation dataset and optimize the global model with the heterogeneity-aware local model ensemble via knowledge distillation. Experimental results on 19 projects from two datasets demonstrate that our method significantly outperforms baselines.

Title: Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance

• Authors: Nicolas Devatine, Louis Abraham
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17321
• Pdf URL: https://arxiv.org/pdf/2412.17321
• Copy Paste: [[2412.17321]] Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance(https://arxiv.org/abs/2412.17321)
• Keywords: large language model
• Abstract: Assessing the extent of human edits on texts generated by Large Language Models (LLMs) is crucial to understanding the human-AI interactions and improving the quality of automated text generation systems. Existing edit distance metrics, such as Levenshtein, BLEU, ROUGE, and TER, often fail to accurately measure the effort required for post-editing, especially when edits involve substantial modifications, such as block operations. In this paper, we introduce a novel compression-based edit distance metric grounded in the Lempel-Ziv-77 algorithm, designed to quantify the amount of post-editing applied to LLM-generated texts. Our method leverages the properties of text compression to measure the informational difference between the original and edited texts. Through experiments on real-world human edits datasets, we demonstrate that our proposed metric is highly correlated with actual edit time and effort. We also show that LLMs exhibit an implicit understanding of editing speed, that aligns well with our metric. Furthermore, we compare our metric with existing ones, highlighting its advantages in capturing complex edits with linear computational efficiency. Our code and data are available at: this https URL

Title: xPatch: Dual-Stream Time Series Forecasting with Exponential Seasonal-Trend Decomposition

• Authors: Artyom Stitsyuk, Jaesik Choi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17323
• Pdf URL: https://arxiv.org/pdf/2412.17323
• Copy Paste: [[2412.17323]] xPatch: Dual-Stream Time Series Forecasting with Exponential Seasonal-Trend Decomposition(https://arxiv.org/abs/2412.17323)
• Keywords: robust, transformer
• Abstract: In recent years, the application of transformer-based models in time-series forecasting has received significant attention. While often demonstrating promising results, the transformer architecture encounters challenges in fully exploiting the temporal relations within time series data due to its attention mechanism. In this work, we design eXponential Patch (xPatch for short), a novel dual-stream architecture that utilizes exponential decomposition. Inspired by the classical exponential smoothing approaches, xPatch introduces the innovative seasonal-trend exponential decomposition module. Additionally, we propose a dual-flow architecture that consists of an MLP-based linear stream and a CNN-based non-linear stream. This model investigates the benefits of employing patching and channel-independence techniques within a non-transformer model. Finally, we develop a robust arctangent loss function and a sigmoid learning rate adjustment scheme, which prevent overfitting and boost forecasting performance. The code is available at the following repository: this https URL.

Title: Feature Based Methods Domain Adaptation for Object Detection: A Review Paper

• Authors: Helia Mohamadi, Mohammad Ali Keyvanrad, Mohammad Reza Mohammadi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17325
• Pdf URL: https://arxiv.org/pdf/2412.17325
• Copy Paste: [[2412.17325]] Feature Based Methods Domain Adaptation for Object Detection: A Review Paper(https://arxiv.org/abs/2412.17325)
• Keywords: robust
• Abstract: Domain adaptation, a pivotal branch of transfer learning, aims to enhance the performance of machine learning models when deployed in target domains with distinct data distributions. This is particularly critical for object detection tasks, where domain shifts (caused by factors such as lighting conditions, viewing angles, and environmental variations) can lead to significant performance degradation. This review delves into advanced methodologies for domain adaptation, including adversarial learning, discrepancy-based, multi-domain, teacher-student, ensemble, and VLM techniques, emphasizing their efficacy in reducing domain gaps and enhancing model robustness. Feature-based methods have emerged as powerful tools for addressing these challenges by harmonizing feature representations across domains. These techniques, such as Feature Alignment, Feature Augmentation/Reconstruction, and Feature Transformation, are employed alongside or as integral parts of other domain adaptation strategies to minimize domain gaps and improve model performance. Special attention is given to strategies that minimize the reliance on extensive labeled data and using unlabeled data, particularly in scenarios involving synthetic-to-real domain shifts. Applications in fields such as autonomous driving and medical imaging are explored, showcasing the potential of these methods to ensure reliable object detection in diverse and complex settings. By providing a thorough analysis of state-of-the-art techniques, challenges, and future directions, this work offers a valuable reference for researchers striving to develop resilient and adaptable object detection frameworks, advancing the seamless deployment of artificial intelligence in dynamic environments.

Title: SoK: The Design Paradigm of Safe and Secure Defaults

• Authors: Jukka Ruohonen
• Subjects: cs.CR, cs.NI, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.17329
• Pdf URL: https://arxiv.org/pdf/2412.17329
• Copy Paste: [[2412.17329]] SoK: The Design Paradigm of Safe and Secure Defaults(https://arxiv.org/abs/2412.17329)
• Keywords: secure, security
• Abstract: In security engineering, including software security engineering, there is a well-known design paradigm telling to prefer safe and secure defaults. The paper presents a systematization of knowledge (SoK) of this paradigm by the means of a systematic mapping study and a scoping review of relevant literature. According to the mapping and review, the paradigm has been extensively discussed, used, and developed further since the late 1990s. Partially driven by the insecurity of the Internet of things, the volume of publications has accelerated from the circa mid-2010s onward. The publications reviewed indicate that the paradigm has been adopted in numerous different contexts. It has also been expanded with security design principles not originally considered when the paradigm was initiated in the mid-1970s. Among the newer principles are an "off by default" principle, various overriding and fallback principles, as well as those related to the zero trust model. The review also indicates obstacles developers and others have faced with the~paradigm.

Title: Uncertainty-Participation Context Consistency Learning for Semi-supervised Semantic Segmentation

• Authors: Jianjian Yin, Yi Chen, Zhichao Zheng, Junsheng Zhou, Yanhui Gu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17331
• Pdf URL: https://arxiv.org/pdf/2412.17331
• Copy Paste: [[2412.17331]] Uncertainty-Participation Context Consistency Learning for Semi-supervised Semantic Segmentation(https://arxiv.org/abs/2412.17331)
• Keywords: segmentation
• Abstract: Semi-supervised semantic segmentation has attracted considerable attention for its ability to mitigate the reliance on extensive labeled data. However, existing consistency regularization methods only utilize high certain pixels with prediction confidence surpassing a fixed threshold for training, failing to fully leverage the potential supervisory information within the network. Therefore, this paper proposes the Uncertainty-participation Context Consistency Learning (UCCL) method to explore richer supervisory signals. Specifically, we first design the semantic backpropagation update (SBU) strategy to fully exploit the knowledge from uncertain pixel regions, enabling the model to learn consistent pixel-level semantic information from those areas. Furthermore, we propose the class-aware knowledge regulation (CKR) module to facilitate the regulation of class-level semantic features across different augmented views, promoting consistent learning of class-level semantic information within the encoder. Experimental results on two public benchmarks demonstrate that our proposed method achieves state-of-the-art performance. Our code is available at this https URL.

Title: A Dual-Perspective Metaphor Detection Framework Using Large Language Models

• Authors: Yujie Lin, Jingyao Liu, Yan Gao, Ante Wang, Jinsong Su
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17332
• Pdf URL: https://arxiv.org/pdf/2412.17332
• Copy Paste: [[2412.17332]] A Dual-Perspective Metaphor Detection Framework Using Large Language Models(https://arxiv.org/abs/2412.17332)
• Keywords: large language model
• Abstract: Metaphor detection, a critical task in natural language processing, involves identifying whether a particular word in a sentence is used metaphorically. Traditional approaches often rely on supervised learning models that implicitly encode semantic relationships based on metaphor theories. However, these methods often suffer from a lack of transparency in their decision-making processes, which undermines the reliability of their predictions. Recent research indicates that LLMs (large language models) exhibit significant potential in metaphor detection. Nevertheless, their reasoning capabilities are constrained by predefined knowledge graphs. To overcome these limitations, we propose DMD, a novel dual-perspective framework that harnesses both implicit and explicit applications of metaphor theories to guide LLMs in metaphor detection and adopts a self-judgment mechanism to validate the responses from the aforementioned forms of guidance. In comparison to previous methods, our framework offers more transparent reasoning processes and delivers more reliable predictions. Experimental results prove the effectiveness of DMD, demonstrating state-of-the-art performance across widely-used datasets.

Title: Broadband Ground Motion Synthesis by Diffusion Model with Minimal Condition

• Authors: Jaeheun Jung, Jaehyuk Lee, Chang-Hae Jung, Hanyoung Kim, Bosung Jung, Donghun Lee
• Subjects: cs.LG, cs.AI, physics.geo-ph
• Abstract URL: https://arxiv.org/abs/2412.17333
• Pdf URL: https://arxiv.org/pdf/2412.17333
• Copy Paste: [[2412.17333]] Broadband Ground Motion Synthesis by Diffusion Model with Minimal Condition(https://arxiv.org/abs/2412.17333)
• Keywords: diffusion
• Abstract: Earthquakes are rare. Hence there is a fundamental call for reliable methods to generate realistic ground motion data for data-driven approaches in seismology. Recent GAN-based methods fall short of the call, as the methods either require special information such as geological traits or generate subpar waveforms that fail to satisfy seismological constraints such as phase arrival times. We propose a specialized Latent Diffusion Model (LDM) that reliably generates realistic waveforms after learning from real earthquake data with minimal conditions: location and magnitude. We also design a domain-specific training method that exploits the traits of earthquake dataset: multiple observed waveforms time-aligned and paired to each earthquake source that are tagged with seismological metadata comprised of earthquake magnitude, depth of focus, and the locations of epicenter and seismometers. We construct the time-aligned earthquake dataset using Southern California Earthquake Data Center (SCEDC) API, and train our model with the dataset and our proposed training method for performance evaluation. Our model surpasses all comparable data-driven methods in various test criteria not only from waveform generation domain but also from seismology such as phase arrival time, GMPE analysis, and spectrum analysis. Our result opens new future research directions for deep learning applications in seismology.

Title: APEX$^2$: Adaptive and Extreme Summarization for Personalized Knowledge Graphs

• Authors: Zihao Li, Dongqi Fu, Mengting Ai, Jingrui He
• Subjects: cs.LG, cs.AI, cs.DB, cs.SC
• Abstract URL: https://arxiv.org/abs/2412.17336
• Pdf URL: https://arxiv.org/pdf/2412.17336
• Copy Paste: [[2412.17336]] APEX$^2$: Adaptive and Extreme Summarization for Personalized Knowledge Graphs(https://arxiv.org/abs/2412.17336)
• Keywords: robust
• Abstract: Knowledge graphs (KGs), which store an extensive number of relational facts, serve various applications. Recently, personalized knowledge graphs (PKGs) have emerged as a solution to optimize storage costs by customizing their content to align with users' specific interests within particular domains. In the real world, on one hand, user queries and their underlying interests are inherently evolving, requiring PKGs to adapt continuously; on the other hand, the summarization is constantly expected to be as small as possible in terms of storage cost. However, the existing PKG summarization methods implicitly assume that the user's interests are constant and do not shift. Furthermore, when the size constraint of PKG is extremely small, the existing methods cannot distinguish which facts are more of immediate interest and guarantee the utility of the summarized PKG. To address these limitations, we propose APEX$^2$, a highly scalable PKG summarization framework designed with robust theoretical guarantees to excel in adaptive summarization tasks with extremely small size constraints. To be specific, after constructing an initial PKG, APEX$^2$ continuously tracks the interest shift and adjusts the previous summary. We evaluate APEX$^2$ under an evolving query setting on benchmark KGs containing up to 12 million triples, summarizing with compression ratios $\leq 0.1\%$. The experiments show that APEX outperforms state-of-the-art baselines in terms of both query-answering accuracy and efficiency.

Title: FFA Sora, video generation as fundus fluorescein angiography simulator

• Authors: Xinyuan Wu, Lili Wang, Ruoyu Chen, Bowen Liu, Weiyi Zhang, Xi Yang, Yifan Feng, Mingguang He, Danli Shi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17346
• Pdf URL: https://arxiv.org/pdf/2412.17346
• Copy Paste: [[2412.17346]] FFA Sora, video generation as fundus fluorescein angiography simulator(https://arxiv.org/abs/2412.17346)
• Keywords: privacy, diffusion, transformer
• Abstract: Fundus fluorescein angiography (FFA) is critical for diagnosing retinal vascular diseases, but beginners often struggle with image interpretation. This study develops FFA Sora, a text-to-video model that converts FFA reports into dynamic videos via a Wavelet-Flow Variational Autoencoder (WF-VAE) and a diffusion transformer (DiT). Trained on an anonymized dataset, FFA Sora accurately simulates disease features from the input text, as confirmed by objective metrics: Frechet Video Distance (FVD) = 329.78, Learned Perceptual Image Patch Similarity (LPIPS) = 0.48, and Visual-question-answering Score (VQAScore) = 0.61. Specific evaluations showed acceptable alignment between the generated videos and textual prompts, with BERTScore of 0.35. Additionally, the model demonstrated strong privacy-preserving performance in retrieval evaluations, achieving an average Recall@K of 0.073. Human assessments indicated satisfactory visual quality, with an average score of 1.570(scale: 1 = best, 5 = worst). This model addresses privacy concerns associated with sharing large-scale FFA data and enhances medical education.

Title: Three-Class Text Sentiment Analysis Based on LSTM

• Authors: Yin Qixuan
• Subjects: cs.CL, cs.CE, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17347
• Pdf URL: https://arxiv.org/pdf/2412.17347
• Copy Paste: [[2412.17347]] Three-Class Text Sentiment Analysis Based on LSTM(https://arxiv.org/abs/2412.17347)
• Keywords: extraction
• Abstract: Sentiment analysis is a crucial task in natural language processing (NLP) with applications in public opinion monitoring, market research, and beyond. This paper introduces a three-class sentiment classification method for Weibo comments using Long Short-Term Memory (LSTM) networks to discern positive, neutral, and negative sentiments. LSTM, as a deep learning model, excels at capturing long-distance dependencies in text data, providing significant advantages over traditional machine learning approaches. Through preprocessing and feature extraction from Weibo comment texts, our LSTM model achieves precise sentiment prediction. Experimental results demonstrate superior performance, achieving an accuracy of 98.31% and an F1 score of 98.28%, notably outperforming conventional models and other deep learning methods. This underscores the effectiveness of LSTM in capturing nuanced sentiment information within text, thereby enhancing classification accuracy. Despite its strengths, the LSTM model faces challenges such as high computational complexity and slower processing times for lengthy texts. Moreover, complex emotional expressions like sarcasm and humor pose additional difficulties. Future work could explore combining pre-trained models or advancing feature engineering techniques to further improve both accuracy and practicality. Overall, this study provides an effective solution for sentiment analysis on Weibo comments.

Title: ORIGAMI: A generative transformer architecture for predictions from semi-structured data

• Authors: Thomas Rückstieß, Alana Huang, Robin Vujanic
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17348
• Pdf URL: https://arxiv.org/pdf/2412.17348
• Copy Paste: [[2412.17348]] ORIGAMI: A generative transformer architecture for predictions from semi-structured data(https://arxiv.org/abs/2412.17348)
• Keywords: robust, transformer, generative
• Abstract: Despite the popularity and widespread use of semi-structured data formats such as JSON, end-to-end supervised learning applied directly to such data remains underexplored. We present ORIGAMI (Object RepresentatIon via Generative Autoregressive ModellIng), a transformer-based architecture that directly processes nested key/value pairs while preserving their hierarchical semantics. Our key technical contributions include: (1) a structure-preserving tokenizer, (2) a novel key/value position encoding scheme, and (3) a grammar-constrained training and inference framework that ensures valid outputs and accelerates training convergence. These enhancements enable efficient end-to-end modeling of semi-structured data. By reformulating classification as next-token prediction, ORIGAMI naturally handles both single-label and multi-label tasks without architectural modifications. Empirical evaluation across diverse domains demonstrates ORIGAMI's effectiveness: On standard tabular benchmarks converted to JSON, ORIGAMI remains competitive with classical and state-of-the-art approaches. On native JSON datasets, we outperform baselines on multi-label classification and specialized models such as convolutional and graph neural networks on a code classification task. Through extensive ablation studies, we validate the impact of each architectural component and establish ORIGAMI as a robust framework for end-to-end learning on semi-structured data.

Title: DiffFormer: a Differential Spatial-Spectral Transformer for Hyperspectral Image Classification

• Authors: Muhammad Ahmad, Manuel Mazzara, Salvatore Distefano, Adil Mehmood Khan, Silvia Liberata Ullo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17350
• Pdf URL: https://arxiv.org/pdf/2412.17350
• Copy Paste: [[2412.17350]] DiffFormer: a Differential Spatial-Spectral Transformer for Hyperspectral Image Classification(https://arxiv.org/abs/2412.17350)
• Keywords: robust, extraction, transformer
• Abstract: Hyperspectral image classification (HSIC) has gained significant attention because of its potential in analyzing high-dimensional data with rich spectral and spatial information. In this work, we propose the Differential Spatial-Spectral Transformer (DiffFormer), a novel framework designed to address the inherent challenges of HSIC, such as spectral redundancy and spatial discontinuity. The DiffFormer leverages a Differential Multi-Head Self-Attention (DMHSA) mechanism, which enhances local feature discrimination by introducing differential attention to accentuate subtle variations across neighboring spectral-spatial patches. The architecture integrates Spectral-Spatial Tokenization through three-dimensional (3D) convolution-based patch embeddings, positional encoding, and a stack of transformer layers equipped with the SWiGLU activation function for efficient feature extraction (SwiGLU is a variant of the Gated Linear Unit (GLU) activation function). A token-based classification head further ensures robust representation learning, enabling precise labeling of hyperspectral pixels. Extensive experiments on benchmark hyperspectral datasets demonstrate the superiority of DiffFormer in terms of classification accuracy, computational efficiency, and generalizability, compared to existing state-of-the-art (SOTA) methods. In addition, this work provides a detailed analysis of computational complexity, showcasing the scalability of the model for large-scale remote sensing applications. The source code will be made available at \url{this https URL} after the first round of revision.

Title: A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motions

• Authors: Youliang Zhang, Ronghui Li, Yachao Zhang, Liang Pan, Jingbo Wang, Yebin Liu, Xiu Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17377
• Pdf URL: https://arxiv.org/pdf/2412.17377
• Copy Paste: [[2412.17377]] A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motions(https://arxiv.org/abs/2412.17377)
• Keywords: segmentation
• Abstract: Extracting physically plausible 3D human motion from videos is a critical task. Although existing simulation-based motion imitation methods can enhance the physical quality of daily motions estimated from monocular video capture, extending this capability to high-difficulty motions remains an open challenge. This can be attributed to some flawed motion clips in video-based motion capture results and the inherent complexity in modeling high-difficulty motions. Therefore, sensing the advantage of segmentation in localizing human body, we introduce a mask-based motion correction module (MCM) that leverages motion context and video mask to repair flawed motions, producing imitation-friendly motions; and propose a physics-based motion transfer module (PTM), which employs a pretrain and adapt approach for motion imitation, improving physical plausibility with the ability to handle in-the-wild and challenging motions. Our approach is designed as a plug-and-play module to physically refine the video motion capture results, including high-difficulty in-the-wild motions. Finally, to validate our approach, we collected a challenging in-the-wild test set to establish a benchmark, and our method has demonstrated effectiveness on both the new benchmark and existing public this http URL://physicalmotionrestoration.this http URL

Title: Interweaving Memories of a Siamese Large Language Model

• Authors: Xin Song, Zhikai Xue, Guoxiu He, Jiawei Liu, Wei Lu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17383
• Pdf URL: https://arxiv.org/pdf/2412.17383
• Copy Paste: [[2412.17383]] Interweaving Memories of a Siamese Large Language Model(https://arxiv.org/abs/2412.17383)
• Keywords: large language model
• Abstract: Parameter-efficient fine-tuning (PEFT) methods optimize large language models (LLMs) by modifying or introducing a small number of parameters to enhance alignment with downstream tasks. However, they can result in catastrophic forgetting, where LLMs prioritize new knowledge at the expense of comprehensive world knowledge. A promising approach to mitigate this issue is to recall prior memories based on the original knowledge. To this end, we propose a model-agnostic PEFT framework, IMSM, which Interweaves Memories of a Siamese Large Language Model. Specifically, our siamese LLM is equipped with an existing PEFT method. Given an incoming query, it generates two distinct memories based on the pre-trained and fine-tuned parameters. IMSM then incorporates an interweaving mechanism that regulates the contributions of both original and enhanced memories when generating the next token. This framework is theoretically applicable to all open-source LLMs and existing PEFT methods. We conduct extensive experiments across various benchmark datasets, evaluating the performance of popular open-source LLMs using the proposed IMSM, in comparison to both classical and leading PEFT methods. Our findings indicate that IMSM maintains comparable time and space efficiency to backbone PEFT methods while significantly improving performance and effectively mitigating catastrophic forgetting.

Title: Singular Value Scaling: Efficient Generative Model Compression via Pruned Weights Refinement

• Authors: Hyeonjin Kim, Jaejun Yoo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17387
• Pdf URL: https://arxiv.org/pdf/2412.17387
• Copy Paste: [[2412.17387]] Singular Value Scaling: Efficient Generative Model Compression via Pruned Weights Refinement(https://arxiv.org/abs/2412.17387)
• Keywords: diffusion, generative
• Abstract: While pruning methods effectively maintain model performance without extra training costs, they often focus solely on preserving crucial connections, overlooking the impact of pruned weights on subsequent fine-tuning or distillation, leading to inefficiencies. Moreover, most compression techniques for generative models have been developed primarily for GANs, tailored to specific architectures like StyleGAN, and research into compressing Diffusion models has just begun. Even more, these methods are often applicable only to GANs or Diffusion models, highlighting the need for approaches that work across both model types. In this paper, we introduce Singular Value Scaling (SVS), a versatile technique for refining pruned weights, applicable to both model types. Our analysis reveals that pruned weights often exhibit dominant singular vectors, hindering fine-tuning efficiency and leading to suboptimal performance compared to random initialization. Our method enhances weight initialization by minimizing the disparities between singular values of pruned weights, thereby improving the fine-tuning process. This approach not only guides the compressed model toward superior solutions but also significantly speeds up fine-tuning. Extensive experiments on StyleGAN2, StyleGAN3 and DDPM demonstrate that SVS improves compression performance across model types without additional training costs. Our code is available at: this https URL.

Title: PointVoxelFormer -- Reviving point cloud networks for 3D medical imaging

• Authors: Mattias Paul Heinrich
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17390
• Pdf URL: https://arxiv.org/pdf/2412.17390
• Copy Paste: [[2412.17390]] PointVoxelFormer -- Reviving point cloud networks for 3D medical imaging(https://arxiv.org/abs/2412.17390)
• Keywords: privacy, robust, transformer, segmentation
• Abstract: Point clouds are a very efficient way to represent volumetric data in medical imaging. First, they do not occupy resources for empty spaces and therefore can avoid trade-offs between resolution and field-of-view for voxel-based 3D convolutional networks (CNNs) - leading to smaller and robust models. Second, they provide a modality agnostic representation of anatomical surfaces and shapes to avoid domain gaps for generic geometric models. Third, they remove identifiable patient-specific information and may increase privacy preservation when publicly sharing data. Despite their benefits, point clouds are still underexplored in medical imaging compared to volumetric 3D CNNs and vision transformers. To date both datasets and stringent studies on comparative strengths and weaknesses of methodological choices are missing. Interactions and information exchange of spatially close points - e.g. through k-nearest neighbour graphs in edge convolutions or point transformations - within points clouds are crucial for learning geometrically meaningful features but may incur computational bottlenecks. This work presents a hybrid approach that combines point-wise operations with intermediate differentiable rasterisation and dense localised CNNs. For deformable point cloud registration, we devise an early fusion scheme for coordinate features that joins both clouds within a common reference frame and is coupled with an inverse consistent, two-step alignment architecture. Our extensive experiments on three different datasets for segmentation and registration demonstrate that our method, PointVoxelFormer, enables very compact models that excel with threefold speed-ups, fivefold memory reduction and over 30% registration error reduction against edge convolutions and other state-of-the-art models in geometric deep learning.

Title: WarriorCoder: Learning from Expert Battles to Augment Code Large Language Models

• Authors: Huawen Feng, Pu Zhao, Qingfeng Sun, Can Xu, Fangkai Yang, Lu Wang, Qianli Ma, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17395
• Pdf URL: https://arxiv.org/pdf/2412.17395
• Copy Paste: [[2412.17395]] WarriorCoder: Learning from Expert Battles to Augment Code Large Language Models(https://arxiv.org/abs/2412.17395)
• Keywords: large language model
• Abstract: Despite recent progress achieved by code large language models (LLMs), their remarkable abilities are largely dependent on fine-tuning on the high-quality data, posing challenges for data collection and annotation. To address this, current methods often design various data flywheels to gather complex code instructions, enabling models to handle more intricate tasks. However, these approaches typically rely on off-the-shelf datasets and data augmentation from the limited pool of proprietary LLMs (e.g., Claude, GPT4, and so on), which limits the diversity of the constructed data and makes it prone to systemic biases. In this paper, we propose WarriorCoder which learns from expert battles to address these limitations. Specifically, we create an arena for current expert code LLMs, where each model challenges and responds to others' challenges, with evaluations conducted by uninvolved judge models. This competitive framework generates novel training data constructed from scratch, harnessing the strengths of all participants. Experimental results demonstrate that WarriorCoder achieves competitive performance compared to previous methods, even without relying on proprietary LLMs.

Title: Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learning

• Authors: Huchen Jiang, Yangyang Ma, Chaofan Ding, Kexin Luan, Xinhan Di
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17397
• Pdf URL: https://arxiv.org/pdf/2412.17397
• Copy Paste: [[2412.17397]] Towards Intrinsic Self-Correction Enhancement in Monte Carlo Tree Search Boosted Reasoning via Iterative Preference Learning(https://arxiv.org/abs/2412.17397)
• Keywords: large language model
• Abstract: With current state-of-the-art approaches aimed at enhancing the reasoning capabilities of Large Language Models(LLMs) through iterative preference learning inspired by AlphaZero, we propose to further enhance the step-wise reasoning capabilities through intrinsic self-correction to some extent. Our work leverages step-wise preference learning to enhance self-verification via reinforcement learning. We initially conduct our work through a two-stage training procedure. At the first stage, the self-correction reasoning ability of an LLM is enhanced through its own predictions, relying entirely on self-generated data within the intrinsic self-correction to some extent. At the second stage, the baseline step-wise preference learning is leveraged via the application of the enhanced self-correct policy achieved at the first stage. In the evaluation of arithmetic reasoning tasks, our approach outperforms OpenMath2-Llama3.1-8B, dart-math-mistral-7b-uniform on MATH with increases in accuracy to 71.34%(+4.18%) and 48.06%(+4.94%) and LLama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.1 on GSM8K with increases in accuracy to 86.76%(+2.00%) and 38.06%(+2.28%).

Title: Learning Dynamic Local Context Representations for Infrared Small Target Detection

• Authors: Guoyi Zhang, Guangsheng Xu, Han Wang, Siyang Chen, Yunxiao Shan, Xiaohu Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17401
• Pdf URL: https://arxiv.org/pdf/2412.17401
• Copy Paste: [[2412.17401]] Learning Dynamic Local Context Representations for Infrared Small Target Detection(https://arxiv.org/abs/2412.17401)
• Keywords: transformer
• Abstract: Infrared small target detection (ISTD) is challenging due to complex backgrounds, low signal-to-clutter ratios, and varying target sizes and shapes. Effective detection relies on capturing local contextual information at the appropriate scale. However, small-kernel CNNs have limited receptive fields, leading to false alarms, while transformer models, with global receptive fields, often treat small targets as noise, resulting in miss-detections. Hybrid models struggle to bridge the semantic gap between CNNs and transformers, causing high this http URL address these challenges, we propose LCRNet, a novel method that learns dynamic local context representations for ISTD. The model consists of three components: (1) C2FBlock, inspired by PDE solvers, for efficient small target information capture; (2) DLC-Attention, a large-kernel attention mechanism that dynamically builds context and reduces feature redundancy; and (3) HLKConv, a hierarchical convolution operator based on large-kernel decomposition that preserves sparsity and mitigates the drawbacks of dilated convolutions. Despite its simplicity, with only 1.65M parameters, LCRNet achieves state-of-the-art (SOTA) this http URL on multiple datasets, comparing LCRNet with 33 SOTA methods, demonstrate its superior performance and efficiency.

Title: Just What You Desire: Constrained Timeline Summarization with Self-Reflection for Enhanced Relevance

• Authors: Muhammad Reza Qorib, Qisheng Hu, Hwee Tou Ng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17408
• Pdf URL: https://arxiv.org/pdf/2412.17408
• Copy Paste: [[2412.17408]] Just What You Desire: Constrained Timeline Summarization with Self-Reflection for Enhanced Relevance(https://arxiv.org/abs/2412.17408)
• Keywords: large language model
• Abstract: Given news articles about an entity, such as a public figure or organization, timeline summarization (TLS) involves generating a timeline that summarizes the key events about the entity. However, the TLS task is too underspecified, since what is of interest to each reader may vary, and hence there is not a single ideal or optimal timeline. In this paper, we introduce a novel task, called Constrained Timeline Summarization (CTLS), where a timeline is generated in which all events in the timeline meet some constraint. An example of a constrained timeline concerns the legal battles of Tiger Woods, where only events related to his legal problems are selected to appear in the timeline. We collected a new human-verified dataset of constrained timelines involving 47 entities and 5 constraints per entity. We propose an approach that employs a large language model (LLM) to summarize news articles according to a specified constraint and cluster them to identify key events to include in a constrained timeline. In addition, we propose a novel self-reflection method during summary generation, demonstrating that this approach successfully leads to improved performance.

Title: VidCtx: Context-aware Video Question Answering with Image Models

• Authors: Andreas Goulas, Vasileios Mezaris, Ioannis Patras
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.17415
• Pdf URL: https://arxiv.org/pdf/2412.17415
• Copy Paste: [[2412.17415]] VidCtx: Context-aware Video Question Answering with Image Models(https://arxiv.org/abs/2412.17415)
• Keywords: large language model
• Abstract: To address computational and memory limitations of Large Multimodal Models in the Video Question-Answering task, several recent methods extract textual representations per frame (e.g., by captioning) and feed them to a Large Language Model (LLM) that processes them to produce the final response. However, in this way, the LLM does not have access to visual information and often has to process repetitive textual descriptions of nearby frames. To address those shortcomings, in this paper, we introduce VidCtx, a novel training-free VideoQA framework which integrates both modalities, i.e. both visual information from input frames and textual descriptions of others frames that give the appropriate context. More specifically, in the proposed framework a pre-trained Large Multimodal Model (LMM) is prompted to extract at regular intervals, question-aware textual descriptions (captions) of video frames. Those will be used as context when the same LMM will be prompted to answer the question at hand given as input a) a certain frame, b) the question and c) the context/caption of an appropriate frame. To avoid redundant information, we chose as context the descriptions of distant frames. Finally, a simple yet effective max pooling mechanism is used to aggregate the frame-level decisions. This methodology enables the model to focus on the relevant segments of the video and scale to a high number of frames. Experiments show that VidCtx achieves competitive performance among approaches that rely on open models on three public Video QA benchmarks, NExT-QA, IntentQA and STAR.

Title: Multimodal Preference Data Synthetic Alignment with Reward Model

• Authors: Robert Wijaya, Ngoc-Bao Nguyen, Ngai-Man Cheung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17417
• Pdf URL: https://arxiv.org/pdf/2412.17417
• Copy Paste: [[2412.17417]] Multimodal Preference Data Synthetic Alignment with Reward Model(https://arxiv.org/abs/2412.17417)
• Keywords: generative, large language model
• Abstract: Multimodal large language models (MLLMs) have significantly advanced tasks like caption generation and visual question answering by integrating visual and textual data. However, they sometimes produce misleading or hallucinate content due to discrepancies between their pre-training data and real user prompts. Existing approaches using Direct Preference Optimization (DPO) in vision-language tasks often rely on strong models like GPT-4 or CLIP to determine positive and negative responses. Here, we propose a new framework in generating synthetic data using a reward model as a proxy of human preference for effective multimodal alignment with DPO training. The resulting DPO dataset ranges from 2K to 9K image-text pairs, was evaluated on LLaVA-v1.5-7B, where our approach demonstrated substantial improvements in both the trustworthiness and reasoning capabilities of the base model across multiple hallucination and vision-language benchmark. The experiment results indicate that integrating selected synthetic data, such as from generative and rewards models can effectively reduce reliance on human-annotated data while enhancing MLLMs' alignment capability, offering a scalable solution for safer deployment.

Title: Measuring Contextual Informativeness in Child-Directed Text

• Authors: Maria Valentini, Téa Wright, Ali Marashian, Jennifer Weber, Eliana Colunga, Katharina von der Wense
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17427
• Pdf URL: https://arxiv.org/pdf/2412.17427
• Copy Paste: [[2412.17427]] Measuring Contextual Informativeness in Child-Directed Text(https://arxiv.org/abs/2412.17427)
• Keywords: large language model
• Abstract: To address an important gap in creating children's stories for vocabulary enrichment, we investigate the automatic evaluation of how well stories convey the semantics of target vocabulary words, a task with substantial implications for generating educational content. We motivate this task, which we call measuring contextual informativeness in children's stories, and provide a formal task definition as well as a dataset for the task. We further propose a method for automating the task using a large language model (LLM). Our experiments show that our approach reaches a Spearman correlation of 0.4983 with human judgments of informativeness, while the strongest baseline only obtains a correlation of 0.3534. An additional analysis shows that the LLM-based approach is able to generalize to measuring contextual informativeness in adult-directed text, on which it also outperforms all baselines.

Title: Applying LLM and Topic Modelling in Psychotherapeutic Contexts

• Authors: Alexander Vanin, Vadim Bolshev, Anastasia Panfilova
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17449
• Pdf URL: https://arxiv.org/pdf/2412.17449
• Copy Paste: [[2412.17449]] Applying LLM and Topic Modelling in Psychotherapeutic Contexts(https://arxiv.org/abs/2412.17449)
• Keywords: large language model
• Abstract: This study explores the use of Large language models to analyze therapist remarks in a psychotherapeutic setting. The paper focuses on the application of BERTopic, a machine learning-based topic modeling tool, to the dialogue of two different groups of therapists (classical and modern), which makes it possible to identify and describe a set of topics that consistently emerge across these groups. The paper describes in detail the chosen algorithm for BERTopic, which included creating a vector space from a corpus of therapist remarks, reducing its dimensionality, clustering the space, and creating and optimizing topic representation. Along with the automatic topical modeling by the BERTopic, the research involved an expert assessment of the findings and manual topic structure optimization. The topic modeling results highlighted the most common and stable topics in therapists speech, offering insights into how language patterns in therapy develop and remain stable across different therapeutic styles. This work contributes to the growing field of machine learning in psychotherapy by demonstrating the potential of automated methods to improve both the practice and training of therapists. The study highlights the value of topic modeling as a tool for gaining a deeper understanding of therapeutic dialogue and offers new opportunities for improving therapeutic effectiveness and clinical supervision.

Title: Diving into Self-Evolving Training for Multimodal Reasoning

• Authors: Wei Liu, Junlong Li, Xiwen Zhang, Fan Zhou, Yu Cheng, Junxian He
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17451
• Pdf URL: https://arxiv.org/pdf/2412.17451
• Copy Paste: [[2412.17451]] Diving into Self-Evolving Training for Multimodal Reasoning(https://arxiv.org/abs/2412.17451)
• Keywords: robust
• Abstract: Reasoning ability is essential for Large Multimodal Models (LMMs). In the absence of multimodal chain-of-thought annotated data, self-evolving training, where the model learns from its own outputs, has emerged as an effective and scalable approach for enhancing reasoning abilities. Despite its growing usage, a comprehensive understanding of self-evolving training, particularly in the context of multimodal reasoning, remains limited. In this paper, we delve into the intricacies of self-evolving training for multimodal reasoning, pinpointing three key factors: Training Method, Reward Model, and Prompt Variation. We systematically examine each factor and explore how various configurations affect the training's effectiveness. Our analysis leads to a set of best practices for each factor, aimed at optimizing multimodal reasoning. Furthermore, we explore the Self-Evolution Dynamics during training and the impact of automatic balancing mechanisms in boosting performance. After all the investigations, we present a final recipe for self-evolving training in multimodal reasoning, encapsulating these design choices into a framework we call MSTaR (Multimodal Self-evolving Training for Reasoning), which is universally effective for models with different sizes on various benchmarks, e.g., surpassing the pre-evolved model significantly on 5 multimodal reasoning benchmarks without using additional human annotations, as demonstrated on MiniCPM-V-2.5 (8B), Phi-3.5-Vision (4B) and InternVL2 (2B). We believe this study fills a significant gap in the understanding of self-evolving training for multimodal reasoning and offers a robust framework for future research. Our policy and reward models, as well as the collected data, is released to facilitate further investigation in multimodal reasoning.

Title: A Temporal Convolutional Network-based Approach for Network Intrusion Detection

• Authors: Rukmini Nazre, Rujuta Budke, Omkar Oak, Suraj Sawant, Amit Joshi
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17452
• Pdf URL: https://arxiv.org/pdf/2412.17452
• Copy Paste: [[2412.17452]] A Temporal Convolutional Network-based Approach for Network Intrusion Detection(https://arxiv.org/abs/2412.17452)
• Keywords: attack
• Abstract: Network intrusion detection is critical for securing modern networks, yet the complexity of network traffic poses significant challenges to traditional methods. This study proposes a Temporal Convolutional Network(TCN) model featuring a residual block architecture with dilated convolutions to capture dependencies in network traffic data while ensuring training stability. The TCN's ability to process sequences in parallel enables faster, more accurate sequence modeling than Recurrent Neural Networks. Evaluated on the Edge-IIoTset dataset, which includes 15 classes with normal traffic and 14 cyberattack types, the proposed model achieved an accuracy of 96.72% and a loss of 0.0688, outperforming 1D CNN, CNN-LSTM, CNN-GRU, CNN-BiLSTM, and CNN-GRU-LSTM models. A class-wise classification report, encompassing metrics such as recall, precision, accuracy, and F1-score, demonstrated the TCN model's superior performance across varied attack categories, including Malware, Injection, and DDoS. These results underscore the model's potential in addressing the complexities of network intrusion detection effectively.

Title: CALLIC: Content Adaptive Learning for Lossless Image Compression

• Authors: Daxin Li, Yuanchao Bai, Kai Wang, Junjun Jiang, Xianming Liu, Wen Gao
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.17464
• Pdf URL: https://arxiv.org/pdf/2412.17464
• Copy Paste: [[2412.17464]] CALLIC: Content Adaptive Learning for Lossless Image Compression(https://arxiv.org/abs/2412.17464)
• Keywords: generative
• Abstract: Learned lossless image compression has achieved significant advancements in recent years. However, existing methods often rely on training amortized generative models on massive datasets, resulting in sub-optimal probability distribution estimation for specific testing images during encoding process. To address this challenge, we explore the connection between the Minimum Description Length (MDL) principle and Parameter-Efficient Transfer Learning (PETL), leading to the development of a novel content-adaptive approach for learned lossless image compression, dubbed CALLIC. Specifically, we first propose a content-aware autoregressive self-attention mechanism by leveraging convolutional gating operations, termed Masked Gated ConvFormer (MGCF), and pretrain MGCF on training dataset. Cache then Crop Inference (CCI) is proposed to accelerate the coding process. During encoding, we decompose pre-trained layers, including depth-wise convolutions, using low-rank matrices and then adapt the incremental weights on testing image by Rate-guided Progressive Fine-Tuning (RPFT). RPFT fine-tunes with gradually increasing patches that are sorted in descending order by estimated entropy, optimizing learning process and reducing adaptation time. Extensive experiments across diverse datasets demonstrate that CALLIC sets a new state-of-the-art (SOTA) for learned lossless image compression.

Title: A Survey on Multi-Generative Agent System: Recent Advances and New Frontiers

• Authors: Shuaihang Chen, Yuanxing Liu, Wei Han, Weinan Zhang, Ting Liu
• Subjects: cs.CL, cs.MA
• Abstract URL: https://arxiv.org/abs/2412.17481
• Pdf URL: https://arxiv.org/pdf/2412.17481
• Copy Paste: [[2412.17481]] A Survey on Multi-Generative Agent System: Recent Advances and New Frontiers(https://arxiv.org/abs/2412.17481)
• Keywords: generative, large language model
• Abstract: Multi-generative agent systems (MGASs) have become a research hotspot since the rise of large language models (LLMs). However, with the continuous influx of new related works, the existing reviews struggle to capture them comprehensively. This paper presents a comprehensive survey of these studies. We first discuss the definition of MGAS, a framework encompassing much of previous work. We provide an overview of the various applications of MGAS in (i) solving complex tasks, (ii) simulating specific scenarios, and (iii) evaluating generative agents. Building on previous studies, we also highlight several challenges and propose future directions for research in this field.

Title: A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression

• Authors: Chenlong Deng, Zhisong Zhang, Kelong Mao, Shuaiyi Li, Xinting Huang, Dong Yu, Zhicheng Dou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17483
• Pdf URL: https://arxiv.org/pdf/2412.17483
• Copy Paste: [[2412.17483]] A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression(https://arxiv.org/abs/2412.17483)
• Keywords: large language model
• Abstract: In this work, we provide a thorough investigation of gist-based context compression methods to improve long-context processing in large language models. We focus on two key questions: (1) How well can these methods replace full attention models? and (2) What potential failure patterns arise due to compression? Through extensive experiments, we show that while gist-based compression can achieve near-lossless performance on tasks like retrieval-augmented generation and long-document QA, it faces challenges in tasks like synthetic recall. Furthermore, we identify three key failure patterns: lost by the boundary, lost if surprise, and lost along the way. To mitigate these issues, we propose two effective strategies: fine-grained autoencoding, which enhances the reconstruction of original token information, and segment-wise token importance estimation, which adjusts optimization based on token dependencies. Our work provides valuable insights into the understanding of gist token-based context compression and offers practical strategies for improving compression capabilities.

Title: Improving the Noise Estimation of Latent Neural Stochastic Differential Equations

• Authors: Linus Heck, Maximilian Gelbrecht, Michael T. Schaub, Niklas Boers
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.17499
• Pdf URL: https://arxiv.org/pdf/2412.17499
• Copy Paste: [[2412.17499]] Improving the Noise Estimation of Latent Neural Stochastic Differential Equations(https://arxiv.org/abs/2412.17499)
• Keywords: diffusion, generative
• Abstract: Latent neural stochastic differential equations (SDEs) have recently emerged as a promising approach for learning generative models from stochastic time series data. However, they systematically underestimate the noise level inherent in such data, limiting their ability to capture stochastic dynamics accurately. We investigate this underestimation in detail and propose a straightforward solution: by including an explicit additional noise regularization in the loss function, we are able to learn a model that accurately captures the diffusion component of the data. We demonstrate our results on a conceptual model system that highlights the improved latent neural SDE's capability to model stochastic bistable dynamics.

Title: An Evaluation Framework for Product Images Background Inpainting based on Human Feedback and Product Consistency

• Authors: Yuqi Liang, Jun Luo, Xiaoxi Guo, Jianqi Bi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17504
• Pdf URL: https://arxiv.org/pdf/2412.17504
• Copy Paste: [[2412.17504]] An Evaluation Framework for Product Images Background Inpainting based on Human Feedback and Product Consistency(https://arxiv.org/abs/2412.17504)
• Keywords: segmentation
• Abstract: In product advertising applications, the automated inpainting of backgrounds utilizing AI techniques in product images has emerged as a significant task. However, the techniques still suffer from issues such as inappropriate background and inconsistent product in generated product images, and existing approaches for evaluating the quality of generated product images are mostly inconsistent with human feedback causing the evaluation for this task to depend on manual annotation. To relieve the issues above, this paper proposes Human Feedback and Product Consistency (HFPC), which can automatically assess the generated product images based on two modules. Firstly, to solve inappropriate backgrounds, human feedback on 44,000 automated inpainting product images is collected to train a reward model based on multi-modal features extracted from BLIP and comparative learning. Secondly, to filter generated product images containing inconsistent products, a fine-tuned segmentation model is employed to segment the product of the original and generated product images and then compare the differences between the above two. Extensive experiments have demonstrated that HFPC can effectively evaluate the quality of generated product images and significantly reduce the expense of manual annotation. Moreover, HFPC achieves state-of-the-art(96.4% in precision) in comparison to other open-source visual-quality-assessment models. Dataset and code are available at: this https URL inpainting products dataset/.

Title: BEE: Metric-Adapted Explanations via Baseline Exploration-Exploitation

• Authors: Oren Barkan, Yehonatan Elisha, Jonathan Weill, Noam Koenigstein
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17512
• Pdf URL: https://arxiv.org/pdf/2412.17512
• Copy Paste: [[2412.17512]] BEE: Metric-Adapted Explanations via Baseline Exploration-Exploitation(https://arxiv.org/abs/2412.17512)
• Keywords: explainability
• Abstract: Two prominent challenges in explainability research involve 1) the nuanced evaluation of explanations and 2) the modeling of missing information through baseline representations. The existing literature introduces diverse evaluation metrics, each scrutinizing the quality of explanations through distinct lenses. Additionally, various baseline representations have been proposed, each modeling the notion of missingness differently. Yet, a consensus on the ultimate evaluation metric and baseline representation remains elusive. This work acknowledges the diversity in explanation metrics and baselines, demonstrating that different metrics exhibit preferences for distinct explanation maps resulting from the utilization of different baseline representations and distributions. To address the diversity in metrics and accommodate the variety of baseline representations in a unified manner, we propose Baseline Exploration-Exploitation (BEE) - a path-integration method that introduces randomness to the integration process by modeling the baseline as a learned random tensor. This tensor follows a learned mixture of baseline distributions optimized through a contextual exploration-exploitation procedure to enhance performance on the specific metric of interest. By resampling the baseline from the learned distribution, BEE generates a comprehensive set of explanation maps, facilitating the selection of the best-performing explanation map in this broad set for the given metric. Extensive evaluations across various model architectures showcase the superior performance of BEE in comparison to state-of-the-art explanation methods on a variety of objective evaluation metrics.

Title: Dataset for Real-World Human Action Detection Using FMCW mmWave Radar

• Authors: Dylan jayabahu, Parthipan Siva
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17517
• Pdf URL: https://arxiv.org/pdf/2412.17517
• Copy Paste: [[2412.17517]] Dataset for Real-World Human Action Detection Using FMCW mmWave Radar(https://arxiv.org/abs/2412.17517)
• Keywords: privacy
• Abstract: Human action detection using privacy-preserving mmWave radar sensors is studied for its applications in healthcare and home automation. Unlike existing research, limited to simulations in controlled environments, we present a real-world mmWave radar dataset with baseline results for human action detection.

Title: DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak

• Authors: Hao Wang, Hao Li, Junda Zhu, Xinyuan Wang, Chengwei Pan, MinLie Huang, Lei Sha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17522
• Pdf URL: https://arxiv.org/pdf/2412.17522
• Copy Paste: [[2412.17522]] DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreak(https://arxiv.org/abs/2412.17522)
• Keywords: security, attack, diffusion, generative, large language model
• Abstract: Large Language Models (LLMs) are susceptible to generating harmful content when prompted with carefully crafted inputs, a vulnerability known as LLM jailbreaking. As LLMs become more powerful, studying jailbreak methods is critical to enhancing security and aligning models with human values. Traditionally, jailbreak techniques have relied on suffix addition or prompt templates, but these methods suffer from limited attack diversity. This paper introduces DiffusionAttacker, an end-to-end generative approach for jailbreak rewriting inspired by diffusion models. Our method employs a sequence-to-sequence (seq2seq) text diffusion model as a generator, conditioning on the original prompt and guiding the denoising process with a novel attack loss. Unlike previous approaches that use autoregressive LLMs to generate jailbreak prompts, which limit the modification of already generated tokens and restrict the rewriting space, DiffusionAttacker utilizes a seq2seq diffusion model, allowing more flexible token modifications. This approach preserves the semantic content of the original prompt while producing harmful content. Additionally, we leverage the Gumbel-Softmax technique to make the sampling process from the diffusion model's output distribution differentiable, eliminating the need for iterative token search. Extensive experiments on Advbench and Harmbench demonstrate that DiffusionAttacker outperforms previous methods across various evaluation metrics, including attack success rate (ASR), fluency, and diversity.

Title: Constructing Fair Latent Space for Intersection of Fairness and Explainability

• Authors: Hyungjun Joo, Hyeonggeun Han, Sehwan Kim, Sangwoo Hong, Jungwoo Lee
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17523
• Pdf URL: https://arxiv.org/pdf/2412.17523
• Copy Paste: [[2412.17523]] Constructing Fair Latent Space for Intersection of Fairness and Explainability(https://arxiv.org/abs/2412.17523)
• Keywords: fair, explainability, generative
• Abstract: As the use of machine learning models has increased, numerous studies have aimed to enhance fairness. However, research on the intersection of fairness and explainability remains insufficient, leading to potential issues in gaining the trust of actual users. Here, we propose a novel module that constructs a fair latent space, enabling faithful explanation while ensuring fairness. The fair latent space is constructed by disentangling and redistributing labels and sensitive attributes, allowing the generation of counterfactual explanations for each type of information. Our module is attached to a pretrained generative model, transforming its biased latent space into a fair latent space. Additionally, since only the module needs to be trained, there are advantages in terms of time and cost savings, without the need to train the entire generative model. We validate the fair latent space with various fairness metrics and demonstrate that our approach can effectively provide explanations for biased decisions and assurances of fairness.

Title: Double Landmines: Invisible Textual Backdoor Attacks based on Dual-Trigger

• Authors: Yang Hou, Qiuling Yue, Lujia Chai, Guozhao Liao, Wenbao Han, Wei Ou
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17531
• Pdf URL: https://arxiv.org/pdf/2412.17531
• Copy Paste: [[2412.17531]] Double Landmines: Invisible Textual Backdoor Attacks based on Dual-Trigger(https://arxiv.org/abs/2412.17531)
• Keywords: security, protect, defense, attack, robust
• Abstract: At present, all textual backdoor attack methods are based on single triggers: for example, inserting specific content into the text to activate the backdoor; or changing the abstract text features. The former is easier to be identified by existing defense strategies due to its obvious characteristics; the latter, although improved in invisibility, has certain shortcomings in terms of attack performance, construction of poisoned datasets, and selection of the final poisoning rate. On this basis, this paper innovatively proposes a Dual-Trigger backdoor attack based on syntax and mood, and optimizes the construction of the poisoned dataset and the selection strategy of the final poisoning rate. A large number of experimental results show that this method significantly outperforms the previous methods based on abstract features in attack performance, and achieves comparable attack performance (almost 100% attack success rate) with the insertion-based method. In addition, the two trigger mechanisms included in this method can be activated independently in the application phase of the model, which not only improves the flexibility of the trigger style, but also enhances its robustness against defense strategies. These results profoundly reveal that textual backdoor attacks are extremely harmful and provide a new perspective for security protection in this field.

Title: Behind Closed Words: Creating and Investigating the forePLay Annotated Dataset for Polish Erotic Discourse

• Authors: Anna Kołos, Katarzyna Lorenc, Emilia Wiśnios, Agnieszka Karlińska
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17533
• Pdf URL: https://arxiv.org/pdf/2412.17533
• Copy Paste: [[2412.17533]] Behind Closed Words: Creating and Investigating the forePLay Annotated Dataset for Polish Erotic Discourse(https://arxiv.org/abs/2412.17533)
• Keywords: robust, transformer
• Abstract: The surge in online content has created an urgent demand for robust detection systems, especially in non-English contexts where current tools demonstrate significant limitations. We present forePLay, a novel Polish language dataset for erotic content detection, featuring over 24k annotated sentences with a multidimensional taxonomy encompassing ambiguity, violence, and social unacceptability dimensions. Our comprehensive evaluation demonstrates that specialized Polish language models achieve superior performance compared to multilingual alternatives, with transformer-based architectures showing particular strength in handling imbalanced categories. The dataset and accompanying analysis establish essential frameworks for developing linguistically-aware content moderation systems, while highlighting critical considerations for extending such capabilities to morphologically complex languages.

Title: WildPPG: A Real-World PPG Dataset of Long Continuous Recordings

• Authors: Manuel Meier, Berken Utku Demirel, Christian Holz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17540
• Pdf URL: https://arxiv.org/pdf/2412.17540
• Copy Paste: [[2412.17540]] WildPPG: A Real-World PPG Dataset of Long Continuous Recordings(https://arxiv.org/abs/2412.17540)
• Keywords: robust
• Abstract: Reflective photoplethysmography (PPG) has become the default sensing technique in wearable devices to monitor cardiac activity via a person's heart rate (HR). However, PPG-based HR estimates can be substantially impacted by factors such as the wearer's activities, sensor placement and resulting motion artifacts, as well as environmental characteristics such as temperature and ambient light. These and other factors can significantly impact and decrease HR prediction reliability. In this paper, we show that state-of-the-art HR estimation methods struggle when processing \emph{representative} data from everyday activities in outdoor environments, likely because they rely on existing datasets that captured controlled conditions. We introduce a novel multimodal dataset and benchmark results for continuous PPG recordings during outdoor activities from 16 participants over 13.5 hours, captured from four wearable sensors, each worn at a different location on the body, totaling 216\,hours. Our recordings include accelerometer, temperature, and altitude data, as well as a synchronized Lead I-based electrocardiogram for ground-truth HR references. Participants completed a round trip from Zurich to Jungfraujoch, a tall mountain in Switzerland over the course of one day. The trip included outdoor and indoor activities such as walking, hiking, stair climbing, eating, drinking, and resting at various temperatures and altitudes (up to 3,571\,m above sea level) as well as using cars, trains, cable cars, and lifts for transport -- all of which impacted participants' physiological dynamics. We also present a novel method that estimates HR values more robustly in such real-world scenarios than existing baselines.

Title: Concept Discovery in Deep Neural Networks for Explainable Face Anti-Spoofing

• Authors: Haoyuan Zhang, Xiangyu Zhu, Li Gao, Jiawei Pan, Kai Pang, Guoying Zhao, Stan Z. Li, Zhen Lei
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17541
• Pdf URL: https://arxiv.org/pdf/2412.17541
• Copy Paste: [[2412.17541]] Concept Discovery in Deep Neural Networks for Explainable Face Anti-Spoofing(https://arxiv.org/abs/2412.17541)
• Keywords: attack
• Abstract: With the rapid growth usage of face recognition in people's daily life, face anti-spoofing becomes increasingly important to avoid malicious attacks. Recent face anti-spoofing models can reach a high classification accuracy on multiple datasets but these models can only tell people ``this face is fake'' while lacking the explanation to answer ``why it is fake''. Such a system undermines trustworthiness and causes user confusion, as it denies their requests without providing any explanations. In this paper, we incorporate XAI into face anti-spoofing and propose a new problem termed X-FAS (eXplainable Face Anti-Spoofing) empowering face anti-spoofing models to provide an explanation. We propose SPED (SPoofing Evidence Discovery), an X-FAS method which can discover spoof concepts and provide reliable explanations on the basis of discovered concepts. To evaluate the quality of X-FAS methods, we propose an X-FAS benchmark with annotated spoofing evidence by experts. We analyze SPED explanations on face anti-spoofing dataset and compare SPED quantitatively and qualitatively with previous XAI methods on proposed X-FAS benchmark. Experimental results demonstrate SPED's ability to generate reliable explanations.

Title: Resource-Aware Arabic LLM Creation: Model Adaptation, Integration, and Multi-Domain Testing

• Authors: Prakash Aryan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17548
• Pdf URL: https://arxiv.org/pdf/2412.17548
• Copy Paste: [[2412.17548]] Resource-Aware Arabic LLM Creation: Model Adaptation, Integration, and Multi-Domain Testing(https://arxiv.org/abs/2412.17548)
• Keywords: robust, large language model
• Abstract: This paper presents a novel approach to fine-tuning the Qwen2-1.5B model for Arabic language processing using Quantized Low-Rank Adaptation (QLoRA) on a system with only 4GB VRAM. We detail the process of adapting this large language model to the Arabic domain, using diverse datasets including Bactrian, OpenAssistant, and Wikipedia Arabic corpora. Our methodology involves custom data preprocessing, model configuration, and training optimization techniques such as gradient accumulation and mixed-precision training. We address specific challenges in Arabic NLP, including morphological complexity, dialectal variations, and diacritical mark handling. Experimental results over 10,000 training steps show significant performance improvements, with the final loss converging to 0.1083. We provide comprehensive analysis of GPU memory usage, training dynamics, and model evaluation across various Arabic language tasks, including text classification, question answering, and dialect identification. The fine-tuned model demonstrates robustness to input perturbations and improved handling of Arabic-specific linguistic phenomena. This research contributes to multilingual AI by demonstrating a resource-efficient approach for creating specialized language models, potentially democratizing access to advanced NLP technologies for diverse linguistic communities. Our work paves the way for future research in low-resource language adaptation and efficient fine-tuning of large language models.

Title: A Survey of Query Optimization in Large Language Models

• Authors: Mingyang Song, Mao Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17558
• Pdf URL: https://arxiv.org/pdf/2412.17558
• Copy Paste: [[2412.17558]] A Survey of Query Optimization in Large Language Models(https://arxiv.org/abs/2412.17558)
• Keywords: large language model
• Abstract: \textit{Query Optimization} (QO) refers to techniques aimed at enhancing the efficiency and quality of Large Language Models (LLMs) in understanding and answering queries, especially complex ones in scenarios like Retrieval-Augmented Generation (RAG). Specifically, RAG mitigates the limitations of LLMs by dynamically retrieving and leveraging up-to-date relevant information, which provides a cost-effective solution to the challenge of LLMs producing plausible but potentially inaccurate responses. Recently, as RAG evolves and incorporates multiple components that influence its performance, QO has emerged as a critical element, playing a pivotal role in determining the effectiveness of RAG's retrieval stage in accurately sourcing the necessary multiple pieces of evidence to answer queries correctly. In this paper, we trace the evolution of QO techniques by summarizing and analyzing significant studies. Through an organized framework and categorization, we aim to consolidate existing QO techniques in RAG, elucidate their technological foundations, and highlight their potential to enhance the versatility and applications of LLMs.

Title: GQSA: Group Quantization and Sparsity for Accelerating Large Language Model Inference

• Authors: Chao Zeng, Songwei Liu, Shu Yang, Fangmin Chen, Xing Mei, Lean Fu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17560
• Pdf URL: https://arxiv.org/pdf/2412.17560
• Copy Paste: [[2412.17560]] GQSA: Group Quantization and Sparsity for Accelerating Large Language Model Inference(https://arxiv.org/abs/2412.17560)
• Keywords: large language model
• Abstract: With the rapid growth in the scale and complexity of large language models (LLMs), the costs of training and inference have risen substantially. Model compression has emerged as a mainstream solution to reduce memory usage and computational overhead. This paper presents Group Quantization and Sparse Acceleration (\textbf{GQSA}), a novel compression technique tailored for LLMs. Traditional methods typically focus exclusively on either quantization or sparsification, but relying on a single strategy often results in significant performance loss at high compression rates. In contrast, GQSA integrates quantization and sparsification in a tightly coupled manner, leveraging GPU-friendly structured group sparsity and quantization for efficient acceleration. The proposed method consists of three key steps. First, GQSA applies group structured pruning to adhere to GPU-friendly sparse pattern constraints. Second, a two-stage sparsity-aware training process is employed to maximize performance retention after compression. Finally, the framework adopts the Block Sparse Row (BSR) format to enable practical deployment and efficient execution. Experimental results on the LLaMA model family show that GQSA achieves an excellent balance between model speed and accuracy. Furthermore, on the latest LLaMA-3 and LLaMA-3.1 models, GQSA outperforms existing LLM compression techniques significantly.

Title: S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Field

• Authors: Zixi Liang, Guowei Xu, Haifeng Wu, Ye Huang, Wen Li, Lixin Duan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17561
• Pdf URL: https://arxiv.org/pdf/2412.17561
• Copy Paste: [[2412.17561]] S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Field(https://arxiv.org/abs/2412.17561)
• Keywords: generative
• Abstract: Learning-based methods have become increasingly popular in 3D indoor scene synthesis (ISS), showing superior performance over traditional optimization-based approaches. These learning-based methods typically model distributions on simple yet explicit scene representations using generative models. However, due to the oversimplified explicit representations that overlook detailed information and the lack of guidance from multimodal relationships within the scene, most learning-based methods struggle to generate indoor scenes with realistic object arrangements and styles. In this paper, we introduce a new method, Scene Implicit Neural Field (S-INF), for indoor scene synthesis, aiming to learn meaningful representations of multimodal relationships, to enhance the realism of indoor scene synthesis. S-INF assumes that the scene layout is often related to the object-detailed information. It disentangles the multimodal relationships into scene layout relationships and detailed object relationships, fusing them later through implicit neural fields (INFs). By learning specialized scene layout relationships and projecting them into S-INF, we achieve a realistic generation of scene layout. Additionally, S-INF captures dense and detailed object relationships through differentiable rendering, ensuring stylistic consistency across objects. Through extensive experiments on the benchmark 3D-FRONT dataset, we demonstrate that our method consistently achieves state-of-the-art performance under different types of ISS.

Title: Evaluation of Bio-Inspired Models under Different Learning Settings For Energy Efficiency in Network Traffic Prediction

• Authors: Theodoros Tsiolakis, Nikolaos Pavlidis, Vasileios Perifanis, Pavlos Efraimidis
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17565
• Pdf URL: https://arxiv.org/pdf/2412.17565
• Copy Paste: [[2412.17565]] Evaluation of Bio-Inspired Models under Different Learning Settings For Energy Efficiency in Network Traffic Prediction(https://arxiv.org/abs/2412.17565)
• Keywords: privacy, federate
• Abstract: Cellular traffic forecasting is a critical task that enables network operators to efficiently allocate resources and address anomalies in rapidly evolving environments. The exponential growth of data collected from base stations poses significant challenges to processing and analysis. While machine learning (ML) algorithms have emerged as powerful tools for handling these large datasets and providing accurate predictions, their environmental impact, particularly in terms of energy consumption, is often overlooked in favor of their predictive capabilities. This study investigates the potential of two bio-inspired models: Spiking Neural Networks (SNNs) and Reservoir Computing through Echo State Networks (ESNs) for cellular traffic forecasting. The evaluation focuses on both their predictive performance and energy efficiency. These models are implemented in both centralized and federated settings to analyze their effectiveness and energy consumption in decentralized systems. Additionally, we compare bio-inspired models with traditional architectures, such as Convolutional Neural Networks (CNNs) and Multi-Layer Perceptrons (MLPs), to provide a comprehensive evaluation. Using data collected from three diverse locations in Barcelona, Spain, we examine the trade-offs between predictive accuracy and energy demands across these approaches. The results indicate that bio-inspired models, such as SNNs and ESNs, can achieve significant energy savings while maintaining predictive accuracy comparable to traditional architectures. Furthermore, federated implementations were tested to evaluate their energy efficiency in decentralized settings compared to centralized systems, particularly in combination with bio-inspired models. These findings offer valuable insights into the potential of bio-inspired models for sustainable and privacy-preserving cellular traffic forecasting.

Title: HPCNeuroNet: A Neuromorphic Approach Merging SNN Temporal Dynamics with Transformer Attention for FPGA-based Particle Physics

• Authors: Murat Isik, Hiruna Vishwamith, Jonathan Naoukin, I. Can Dikmen
• Subjects: cs.LG, cs.AR, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2412.17571
• Pdf URL: https://arxiv.org/pdf/2412.17571
• Copy Paste: [[2412.17571]] HPCNeuroNet: A Neuromorphic Approach Merging SNN Temporal Dynamics with Transformer Attention for FPGA-based Particle Physics(https://arxiv.org/abs/2412.17571)
• Keywords: robust, transformer
• Abstract: This paper presents the innovative HPCNeuroNet model, a pioneering fusion of Spiking Neural Networks (SNNs), Transformers, and high-performance computing tailored for particle physics, particularly in particle identification from detector responses. Our approach leverages SNNs' intrinsic temporal dynamics and Transformers' robust attention mechanisms to enhance performance when discerning intricate particle interactions. At the heart of HPCNeuroNet lies the integration of the sequential dynamism inherent in SNNs with the context-aware attention capabilities of Transformers, enabling the model to precisely decode and interpret complex detector data. HPCNeuroNet is realized through the HLS4ML framework and optimized for deployment in FPGA environments. The model accuracy and scalability are also enhanced by this architectural choice. Benchmarked against machine learning models, HPCNeuroNet showcases better performance metrics, underlining its transformative potential in high-energy physics. We demonstrate that the combination of SNNs, Transformers, and FPGA-based high-performance computing in particle physics signifies a significant step forward and provides a strong foundation for future research.

Title: URoadNet: Dual Sparse Attentive U-Net for Multiscale Road Network Extraction

• Authors: Jie Song, Yue Sun, Ziyun Cai, Liang Xiao, Yawen Huang, Yefeng Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17573
• Pdf URL: https://arxiv.org/pdf/2412.17573
• Copy Paste: [[2412.17573]] URoadNet: Dual Sparse Attentive U-Net for Multiscale Road Network Extraction(https://arxiv.org/abs/2412.17573)
• Keywords: extraction, transformer, segmentation
• Abstract: The challenges of road network segmentation demand an algorithm capable of adapting to the sparse and irregular shapes, as well as the diverse context, which often leads traditional encoding-decoding methods and simple Transformer embeddings to failure. We introduce a computationally efficient and powerful framework for elegant road-aware segmentation. Our method, called URoadNet, effectively encodes fine-grained local road connectivity and holistic global topological semantics while decoding multiscale road network information. URoadNet offers a novel alternative to the U-Net architecture by integrating connectivity attention, which can exploit intra-road interactions across multi-level sampling features with reduced computational complexity. This local interaction serves as valuable prior information for learning global interactions between road networks and the background through another integrality attention mechanism. The two forms of sparse attention are arranged alternatively and complementarily, and trained jointly, resulting in performance improvements without significant increases in computational complexity. Extensive experiments on various datasets with different resolutions, including Massachusetts, DeepGlobe, SpaceNet, and Large-Scale remote sensing images, demonstrate that URoadNet outperforms state-of-the-art techniques. Our approach represents a significant advancement in the field of road network extraction, providing a computationally feasible solution that achieves high-quality segmentation results.

Title: HumanVBench: Exploring Human-Centric Video Understanding Capabilities of MLLMs with Synthetic Benchmark Data

• Authors: Ting Zhou, Daoyuan Chen, Qirui Jiao, Bolin Ding, Yaliang Li, Ying Shen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17574
• Pdf URL: https://arxiv.org/pdf/2412.17574
• Copy Paste: [[2412.17574]] HumanVBench: Exploring Human-Centric Video Understanding Capabilities of MLLMs with Synthetic Benchmark Data(https://arxiv.org/abs/2412.17574)
• Keywords: large language model
• Abstract: In the domain of Multimodal Large Language Models (MLLMs), achieving human-centric video understanding remains a formidable challenge. Existing benchmarks primarily emphasize object and action recognition, often neglecting the intricate nuances of human emotions, behaviors, and speech visual alignment within video content. We present HumanVBench, an innovative benchmark meticulously crafted to bridge these gaps in the evaluation of video MLLMs. HumanVBench comprises 17 carefully designed tasks that explore two primary dimensions: inner emotion and outer manifestations, spanning static and dynamic, basic and complex, as well as single-modal and cross-modal aspects. With two advanced automated pipelines for video annotation and distractor-included QA generation, HumanVBench utilizes diverse state-of-the-art (SOTA) techniques to streamline benchmark data synthesis and quality assessment, minimizing human annotation dependency tailored to human-centric multimodal attributes. A comprehensive evaluation across 16 SOTA video MLLMs reveals notable limitations in current performance, especially in cross-modal and temporal alignment, underscoring the necessity for further refinement toward achieving more human-like understanding. HumanVBench is open-sourced to facilitate future advancements and real-world applications in video MLLMs.

Title: Investigating Length Issues in Document-level Machine Translation

• Authors: Ziqian Peng, Rachel Bawden, François Yvon
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17592
• Pdf URL: https://arxiv.org/pdf/2412.17592
• Copy Paste: [[2412.17592]] Investigating Length Issues in Document-level Machine Translation(https://arxiv.org/abs/2412.17592)
• Keywords: transformer
• Abstract: Transformer architectures are increasingly effective at processing and generating very long chunks of texts, opening new perspectives for document-level machine translation (MT). In this work, we challenge the ability of MT systems to handle texts comprising up to several thousands of tokens. We design and implement a new approach designed to precisely measure the effect of length increments on MT outputs. Our experiments with two representative architectures unambiguously show that (a)~translation performance decreases with the length of the input text; (b)~the position of sentences within the document matters and translation quality is higher for sentences occurring earlier in a document. We further show that manipulating the distribution of document lengths and of positional embeddings only marginally mitigates such problems. Our results suggest that even though document-level MT is computationally feasible, it does not yet match the performance of sentence-based MT.

Title: V$^2$-SfMLearner: Learning Monocular Depth and Ego-motion for Multimodal Wireless Capsule Endoscopy

• Authors: Long Bai, Beilei Cui, Liangyu Wang, Yanheng Li, Shilong Yao, Sishen Yuan, Yanan Wu, Yang Zhang, Max Q.-H. Meng, Zhen Li, Weiping Ding, Hongliang Ren
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.17595
• Pdf URL: https://arxiv.org/pdf/2412.17595
• Copy Paste: [[2412.17595]] V$^2$-SfMLearner: Learning Monocular Depth and Ego-motion for Multimodal Wireless Capsule Endoscopy(https://arxiv.org/abs/2412.17595)
• Keywords: robust
• Abstract: Deep learning can predict depth maps and capsule ego-motion from capsule endoscopy videos, aiding in 3D scene reconstruction and lesion localization. However, the collisions of the capsule endoscopies within the gastrointestinal tract cause vibration perturbations in the training data. Existing solutions focus solely on vision-based processing, neglecting other auxiliary signals like vibrations that could reduce noise and improve performance. Therefore, we propose V$^2$-SfMLearner, a multimodal approach integrating vibration signals into vision-based depth and capsule motion estimation for monocular capsule endoscopy. We construct a multimodal capsule endoscopy dataset containing vibration and visual signals, and our artificial intelligence solution develops an unsupervised method using vision-vibration signals, effectively eliminating vibration perturbations through multimodal learning. Specifically, we carefully design a vibration network branch and a Fourier fusion module, to detect and mitigate vibration noises. The fusion framework is compatible with popular vision-only algorithms. Extensive validation on the multimodal dataset demonstrates superior performance and robustness against vision-only algorithms. Without the need for large external equipment, our V$^2$-SfMLearner has the potential for integration into clinical capsule robots, providing real-time and dependable digestive examination tools. The findings show promise for practical implementation in clinical settings, enhancing the diagnostic capabilities of doctors.

Title: LiveIdeaBench: Evaluating LLMs' Scientific Creativity and Idea Generation with Minimal Context

• Authors: Kai Ruan, Xuan Wang, Jixiang Hong, Hao Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17596
• Pdf URL: https://arxiv.org/pdf/2412.17596
• Copy Paste: [[2412.17596]] LiveIdeaBench: Evaluating LLMs' Scientific Creativity and Idea Generation with Minimal Context(https://arxiv.org/abs/2412.17596)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) have demonstrated remarkable capabilities in scientific tasks, existing evaluation frameworks primarily assess their performance using rich contextual inputs, overlooking their ability to generate novel ideas from minimal information. We introduce LiveIdeaBench, a comprehensive benchmark that evaluates LLMs' scientific creativity and divergent thinking capabilities using single-keyword prompts. Drawing from Guilford's creativity theory, our framework employs a dynamic panel of state-of-the-art LLMs to assess generated ideas across four key dimensions: originality, feasibility, fluency, and flexibility. Through extensive experimentation with 20 leading models across 1,180 keywords spanning 18 scientific domains, we reveal that scientific creative ability shows distinct patterns from general intelligence metrics. Notably, our results demonstrate that models like QwQ-32B-preview achieve comparable creative performance to top-tier models like o1-preview, despite significant gaps in their general intelligence scores. These findings highlight the importance of specialized evaluation frameworks for scientific creativity and suggest that the development of creative capabilities in LLMs may follow different trajectories than traditional problem-solving abilities.

Title: AFANet: Adaptive Frequency-Aware Network for Weakly-Supervised Few-Shot Semantic Segmentation

• Authors: Jiaqi Ma, Guo-Sen Xie, Fang Zhao, Zechao Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17601
• Pdf URL: https://arxiv.org/pdf/2412.17601
• Copy Paste: [[2412.17601]] AFANet: Adaptive Frequency-Aware Network for Weakly-Supervised Few-Shot Semantic Segmentation(https://arxiv.org/abs/2412.17601)
• Keywords: segmentation
• Abstract: Few-shot learning aims to recognize novel concepts by leveraging prior knowledge learned from a few samples. However, for visually intensive tasks such as few-shot semantic segmentation, pixel-level annotations are time-consuming and costly. Therefore, in this paper, we utilize the more challenging image-level annotations and propose an adaptive frequency-aware network (AFANet) for weakly-supervised few-shot semantic segmentation (WFSS). Specifically, we first propose a cross-granularity frequency-aware module (CFM) that decouples RGB images into high-frequency and low-frequency distributions and further optimizes semantic structural information by realigning them. Unlike most existing WFSS methods using the textual information from the multi-modal language-vision model, e.g., CLIP, in an offline learning manner, we further propose a CLIP-guided spatial-adapter module (CSM), which performs spatial domain adaptive transformation on textual information through online learning, thus providing enriched cross-modal semantic information for CFM. Extensive experiments on the Pascal-5\textsuperscript{i} and COCO-20\textsuperscript{i} datasets demonstrate that AFANet has achieved state-of-the-art performance. The code is available at this https URL.

Title: EasyTime: Time Series Forecasting Made Easy

• Authors: Xiangfei Qiu, Xiuwen Li, Ruiyang Pang, Zhicheng Pan, Xingjian Wu, Liu Yang, Jilin Hu, Yang Shu, Xuesong Lu, Chengcheng Yang, Chenjuan Guo, Aoying Zhou, Christian S. Jensen, Bin Yang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.17603
• Pdf URL: https://arxiv.org/pdf/2412.17603
• Copy Paste: [[2412.17603]] EasyTime: Time Series Forecasting Made Easy(https://arxiv.org/abs/2412.17603)
• Keywords: large language model
• Abstract: Time series forecasting has important applications across diverse domains. EasyTime, the system we demonstrate, facilitates easy use of time-series forecasting methods by researchers and practitioners alike. First, EasyTime enables one-click evaluation, enabling researchers to evaluate new forecasting methods using the suite of diverse time series datasets collected in the preexisting time series forecasting benchmark (TFB). This is achieved by leveraging TFB's flexible and consistent evaluation pipeline. Second, when practitioners must perform forecasting on a new dataset, a nontrivial first step is often to find an appropriate forecasting method. EasyTime provides an Automated Ensemble module that combines the promising forecasting methods to yield superior forecasting accuracy compared to individual methods. Third, EasyTime offers a natural language Q&A module leveraging large language models. Given a question like "Which method is best for long term forecasting on time series with strong seasonality?", EasyTime converts the question into SQL queries on the database of results obtained by TFB and then returns an answer in natural language and charts. By demonstrating EasyTime, we intend to show how it is possible to simplify the use of time series forecasting and to offer better support for the development of new generations of time series forecasting methods.

Title: Emerging Security Challenges of Large Language Models

• Authors: Herve Debar, Sven Dietrich, Pavel Laskov, Emil C. Lupu, Eirini Ntoutsi
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17614
• Pdf URL: https://arxiv.org/pdf/2412.17614
• Copy Paste: [[2412.17614]] Emerging Security Challenges of Large Language Models(https://arxiv.org/abs/2412.17614)
• Keywords: security, defense, attack, large language model
• Abstract: Large language models (LLMs) have achieved record adoption in a short period of time across many different sectors including high importance areas such as education [4] and healthcare [23]. LLMs are open-ended models trained on diverse data without being tailored for specific downstream tasks, enabling broad applicability across various domains. They are commonly used for text generation, but also widely used to assist with code generation [3], and even analysis of security information, as Microsoft Security Copilot demonstrates [18]. Traditional Machine Learning (ML) models are vulnerable to adversarial attacks [9]. So the concerns on the potential security implications of such wide scale adoption of LLMs have led to the creation of this working group on the security of LLMs. During the Dagstuhl seminar on "Network Attack Detection and Defense - AI-Powered Threats and Responses", the working group discussions focused on the vulnerability of LLMs to adversarial attacks, rather than their potential use in generating malware or enabling cyberattacks. Although we note the potential threat represented by the latter, the role of the LLMs in such uses is mostly as an accelerator for development, similar to what it is in benign use. To make the analysis more specific, the working group employed ChatGPT as a concrete example of an LLM and addressed the following points, which also form the structure of this report: 1. How do LLMs differ in vulnerabilities from traditional ML models? 2. What are the attack objectives in LLMs? 3. How complex it is to assess the risks posed by the vulnerabilities of LLMs? 4. What is the supply chain in LLMs, how data flow in and out of systems and what are the security implications? We conclude with an overview of open challenges and outlook.

Title: Be More Diverse than the Most Diverse: Online Selection of Diverse Mixtures of Generative Models

• Authors: Parham Rezaei, Farzan Farnia, Cheuk Ting Li
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.17622
• Pdf URL: https://arxiv.org/pdf/2412.17622
• Copy Paste: [[2412.17622]] Be More Diverse than the Most Diverse: Online Selection of Diverse Mixtures of Generative Models(https://arxiv.org/abs/2412.17622)
• Keywords: generative
• Abstract: The availability of multiple training algorithms and architectures for generative models requires a selection mechanism to form a single model over a group of well-trained generation models. The selection task is commonly addressed by identifying the model that maximizes an evaluation score based on the diversity and quality of the generated data. However, such a best-model identification approach overlooks the possibility that a mixture of available models can outperform each individual model. In this work, we explore the selection of a mixture of multiple generative models and formulate a quadratic optimization problem to find an optimal mixture model achieving the maximum of kernel-based evaluation scores including kernel inception distance (KID) and Rényi kernel entropy (RKE). To identify the optimal mixture of the models using the fewest possible sample queries, we propose an online learning approach called Mixture Upper Confidence Bound (Mixture-UCB). Specifically, our proposed online learning method can be extended to every convex quadratic function of the mixture weights, for which we prove a concentration bound to enable the application of the UCB approach. We prove a regret bound for the proposed Mixture-UCB algorithm and perform several numerical experiments to show the success of the proposed Mixture-UCB method in finding the optimal mixture of text-based and image-based generative models. The codebase is available at this https URL .

Title: Tracking the Feature Dynamics in LLM Training: A Mechanistic Study

• Authors: Yang Xu, Yi Wang, Hao Wang
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17626
• Pdf URL: https://arxiv.org/pdf/2412.17626
• Copy Paste: [[2412.17626]] Tracking the Feature Dynamics in LLM Training: A Mechanistic Study(https://arxiv.org/abs/2412.17626)
• Keywords: interpretability, large language model
• Abstract: Understanding training dynamics and feature evolution is crucial for the mechanistic interpretability of large language models (LLMs). Although sparse autoencoders (SAEs) have been used to identify features within LLMs, a clear picture of how these features evolve during training remains elusive. In this study, we: (1) introduce SAE-Track, a method to efficiently obtain a continual series of SAEs; (2) formulate the process of feature formation and conduct a mechanistic analysis; and (3) analyze and visualize feature drift during training. Our work provides new insights into the dynamics of features in LLMs, enhancing our understanding of training mechanisms and feature evolution.

Title: Detail-Preserving Latent Diffusion for Stable Shadow Removal

• Authors: Jiamin Xu, Yuxin Zheng, Zelong Li, Chi Wang, Renshu Gu, Weiwei Xu, Gang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17630
• Pdf URL: https://arxiv.org/pdf/2412.17630
• Copy Paste: [[2412.17630]] Detail-Preserving Latent Diffusion for Stable Shadow Removal(https://arxiv.org/abs/2412.17630)
• Keywords: diffusion
• Abstract: Achieving high-quality shadow removal with strong generalizability is challenging in scenes with complex global illumination. Due to the limited diversity in shadow removal datasets, current methods are prone to overfitting training data, often leading to reduced performance on unseen cases. To address this, we leverage the rich visual priors of a pre-trained Stable Diffusion (SD) model and propose a two-stage fine-tuning pipeline to adapt the SD model for stable and efficient shadow removal. In the first stage, we fix the VAE and fine-tune the denoiser in latent space, which yields substantial shadow removal but may lose some high-frequency details. To resolve this, we introduce a second stage, called the detail injection stage. This stage selectively extracts features from the VAE encoder to modulate the decoder, injecting fine details into the final results. Experimental results show that our method outperforms state-of-the-art shadow removal techniques. The cross-dataset evaluation further demonstrates that our method generalizes effectively to unseen data, enhancing the applicability of shadow removal methods.

Title: LangSurf: Language-Embedded Surface Gaussians for 3D Scene Understanding

• Authors: Hao Li, Roy Qin, Zhengyu Zou, Diqi He, Bohan Li, Bingquan Dai, Dingewn Zhang, Junwei Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17635
• Pdf URL: https://arxiv.org/pdf/2412.17635
• Copy Paste: [[2412.17635]] LangSurf: Language-Embedded Surface Gaussians for 3D Scene Understanding(https://arxiv.org/abs/2412.17635)
• Keywords: extraction, segmentation
• Abstract: Applying Gaussian Splatting to perception tasks for 3D scene understanding is becoming increasingly popular. Most existing works primarily focus on rendering 2D feature maps from novel viewpoints, which leads to an imprecise 3D language field with outlier languages, ultimately failing to align objects in 3D space. By utilizing masked images for feature extraction, these approaches also lack essential contextual information, leading to inaccurate feature representation. To this end, we propose a Language-Embedded Surface Field (LangSurf), which accurately aligns the 3D language fields with the surface of objects, facilitating precise 2D and 3D segmentation with text query, widely expanding the downstream tasks such as removal and editing. The core of LangSurf is a joint training strategy that flattens the language Gaussian on the object surfaces using geometry supervision and contrastive losses to assign accurate language features to the Gaussians of objects. In addition, we also introduce the Hierarchical-Context Awareness Module to extract features at the image level for contextual information then perform hierarchical mask pooling using masks segmented by SAM to obtain fine-grained language features in different hierarchies. Extensive experiments on open-vocabulary 2D and 3D semantic segmentation demonstrate that LangSurf outperforms the previous state-of-the-art method LangSplat by a large margin. As shown in Fig.~\ref{fig:teaser}, our method is capable of segmenting objects in 3D space, thus boosting the effectiveness of our approach in instance recognition, removal, and editing, which is also supported by comprehensive experiments. \url{this https URL}{Project Page}.

Title: SCBench: A Sports Commentary Benchmark for Video LLMs

• Authors: Kuangzhi Ge, Lingjun Chen, Kevin Zhang, Yulin Luo, Tianyu Shi, Liaoyuan Fan, Xiang Li, Guanqun Wang, Shanghang Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17637
• Pdf URL: https://arxiv.org/pdf/2412.17637
• Copy Paste: [[2412.17637]] SCBench: A Sports Commentary Benchmark for Video LLMs(https://arxiv.org/abs/2412.17637)
• Keywords: large language model
• Abstract: Recently, significant advances have been made in Video Large Language Models (Video LLMs) in both academia and industry. However, methods to evaluate and benchmark the performance of different Video LLMs, especially their fine-grained, temporal visual capabilities, remain very limited. On one hand, current benchmarks use relatively simple videos (e.g., subtitled movie clips) where the model can understand the entire video by processing just a few frames. On the other hand, their datasets lack diversity in task format, comprising only QA or multi-choice QA, which overlooks the models' capacity for generating in-depth and precise texts. Sports videos, which feature intricate visual information, sequential events, and emotionally charged commentary, present a critical challenge for Video LLMs, making sports commentary an ideal benchmarking task. Inspired by these challenges, we propose a novel task: sports video commentary generation, developed $\textbf{SCBench}$ for Video LLMs. To construct such a benchmark, we introduce (1) $\textbf{SCORES}$, a six-dimensional metric specifically designed for our task, upon which we propose a GPT-based evaluation method, and (2) $\textbf{CommentarySet}$, a dataset consisting of 5,775 annotated video clips and ground-truth labels tailored to our metric. Based on SCBench, we conduct comprehensive evaluations on multiple Video LLMs (e.g. VILA, Video-LLaVA, etc.) and chain-of-thought baseline methods. Our results found that InternVL-Chat-2 achieves the best performance with 5.44, surpassing the second-best by 1.04. Our work provides a fresh perspective for future research, aiming to enhance models' overall capabilities in complex visual understanding tasks. Our dataset will be released soon.

Title: Hierarchical Vector Quantization for Unsupervised Action Segmentation

• Authors: Federico Spurio, Emad Bahrami, Gianpiero Francesca, Juergen Gall
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17640
• Pdf URL: https://arxiv.org/pdf/2412.17640
• Copy Paste: [[2412.17640]] Hierarchical Vector Quantization for Unsupervised Action Segmentation(https://arxiv.org/abs/2412.17640)
• Keywords: segmentation
• Abstract: In this work, we address unsupervised temporal action segmentation, which segments a set of long, untrimmed videos into semantically meaningful segments that are consistent across videos. While recent approaches combine representation learning and clustering in a single step for this task, they do not cope with large variations within temporal segments of the same class. To address this limitation, we propose a novel method, termed Hierarchical Vector Quantization (\ours), that consists of two subsequent vector quantization modules. This results in a hierarchical clustering where the additional subclusters cover the variations within a cluster. We demonstrate that our approach captures the distribution of segment lengths much better than the state of the art. To this end, we introduce a new metric based on the Jensen-Shannon Distance (JSD) for unsupervised temporal action segmentation. We evaluate our approach on three public datasets, namely Breakfast, YouTube Instructional and IKEA ASM. Our approach outperforms the state of the art in terms of F1 score, recall and JSD.

Title: DreamFit: Garment-Centric Human Generation via a Lightweight Anything-Dressing Encoder

• Authors: Ente Lin, Xujie Zhang, Fuwei Zhao, Yuxuan Luo, Xin Dong, Long Zeng, Xiaodan Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17644
• Pdf URL: https://arxiv.org/pdf/2412.17644
• Copy Paste: [[2412.17644]] DreamFit: Garment-Centric Human Generation via a Lightweight Anything-Dressing Encoder(https://arxiv.org/abs/2412.17644)
• Keywords: diffusion
• Abstract: Diffusion models for garment-centric human generation from text or image prompts have garnered emerging attention for their great application potential. However, existing methods often face a dilemma: lightweight approaches, such as adapters, are prone to generate inconsistent textures; while finetune-based methods involve high training costs and struggle to maintain the generalization capabilities of pretrained diffusion models, limiting their performance across diverse scenarios. To address these challenges, we propose DreamFit, which incorporates a lightweight Anything-Dressing Encoder specifically tailored for the garment-centric human generation. DreamFit has three key advantages: (1) \textbf{Lightweight training}: with the proposed adaptive attention and LoRA modules, DreamFit significantly minimizes the model complexity to 83.4M trainable parameters. (2)\textbf{Anything-Dressing}: Our model generalizes surprisingly well to a wide range of (non-)garments, creative styles, and prompt instructions, consistently delivering high-quality results across diverse scenarios. (3) \textbf{Plug-and-play}: DreamFit is engineered for smooth integration with any community control plugins for diffusion models, ensuring easy compatibility and minimizing adoption barriers. To further enhance generation quality, DreamFit leverages pretrained large multi-modal models (LMMs) to enrich the prompt with fine-grained garment descriptions, thereby reducing the prompt gap between training and inference. We conduct comprehensive experiments on both $768 \times 512$ high-resolution benchmarks and in-the-wild images. DreamFit surpasses all existing methods, highlighting its state-of-the-art capabilities of garment-centric human generation.

Title: Benchmarking Generative AI Models for Deep Learning Test Input Generation

• Authors: Maryam, Matteo Biagiola, Andrea Stocco, Vincenzo Riccio
• Subjects: cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.17652
• Pdf URL: https://arxiv.org/pdf/2412.17652
• Copy Paste: [[2412.17652]] Benchmarking Generative AI Models for Deep Learning Test Input Generation(https://arxiv.org/abs/2412.17652)
• Keywords: diffusion, generative
• Abstract: Test Input Generators (TIGs) are crucial to assess the ability of Deep Learning (DL) image classifiers to provide correct predictions for inputs beyond their training and test sets. Recent advancements in Generative AI (GenAI) models have made them a powerful tool for creating and manipulating synthetic images, although these advancements also imply increased complexity and resource demands for training. In this work, we benchmark and combine different GenAI models with TIGs, assessing their effectiveness, efficiency, and quality of the generated test images, in terms of domain validity and label preservation. We conduct an empirical study involving three different GenAI architectures (VAEs, GANs, Diffusion Models), five classification tasks of increasing complexity, and 364 human evaluations. Our results show that simpler architectures, such as VAEs, are sufficient for less complex datasets like MNIST. However, when dealing with feature-rich datasets, such as ImageNet, more sophisticated architectures like Diffusion Models achieve superior performance by generating a higher number of valid, misclassification-inducing inputs.

Title: Generating Completions for Fragmented Broca's Aphasic Sentences Using Large Language Models

• Authors: Sijbren van Vaals, Yevgen Matusevych, Frank Tsiwah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17669
• Pdf URL: https://arxiv.org/pdf/2412.17669
• Copy Paste: [[2412.17669]] Generating Completions for Fragmented Broca's Aphasic Sentences Using Large Language Models(https://arxiv.org/abs/2412.17669)
• Keywords: large language model
• Abstract: Broca's aphasia is a type of aphasia characterized by non-fluent, effortful and fragmented speech production with relatively good comprehension. Since traditional aphasia treatment methods are often time-consuming, labour-intensive, and do not reflect real-world conversations, applying natural language processing based approaches such as Large Language Models (LLMs) could potentially contribute to improving existing treatment approaches. To address this issue, we explore the use of sequence-to-sequence LLMs for completing fragmented Broca's aphasic sentences. We first generate synthetic Broca's aphasic data using a rule-based system designed to mirror the linguistic characteristics of Broca's aphasic speech. Using this synthetic data, we then fine-tune four pre-trained LLMs on the task of completing fragmented sentences. We evaluate our fine-tuned models on both synthetic and authentic Broca's aphasic data. We demonstrate LLMs' capability for reconstructing fragmented sentences, with the models showing improved performance with longer input utterances. Our result highlights the LLMs' potential in advancing communication aids for individuals with Broca's aphasia and possibly other clinical populations.

Title: A Bias-Free Training Paradigm for More General AI-generated Image Detection

• Authors: Fabrizio Guillaro, Giada Zingarini, Ben Usman, Avneesh Sud, Davide Cozzolino, Luisa Verdoliva
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17671
• Pdf URL: https://arxiv.org/pdf/2412.17671
• Copy Paste: [[2412.17671]] A Bias-Free Training Paradigm for More General AI-generated Image Detection(https://arxiv.org/abs/2412.17671)
• Keywords: robust, diffusion, generative
• Abstract: Successful forensic detectors can produce excellent results in supervised learning benchmarks but struggle to transfer to real-world applications. We believe this limitation is largely due to inadequate training data quality. While most research focuses on developing new algorithms, less attention is given to training data selection, despite evidence that performance can be strongly impacted by spurious correlations such as content, format, or resolution. A well-designed forensic detector should detect generator specific artifacts rather than reflect data biases. To this end, we propose B-Free, a bias-free training paradigm, where fake images are generated from real ones using the conditioning procedure of stable diffusion models. This ensures semantic alignment between real and fake images, allowing any differences to stem solely from the subtle artifacts introduced by AI generation. Through content-based augmentation, we show significant improvements in both generalization and robustness over state-of-the-art detectors and more calibrated results across 27 different generative models, including recent releases, like FLUX and Stable Diffusion 3.5. Our findings emphasize the importance of a careful dataset curation, highlighting the need for further research in dataset design. Code and data will be publicly available at this https URL

Title: FedTLU: Federated Learning with Targeted Layer Updates

• Authors: Jong-Ik Park, Carlee Joe-Wong
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.17692
• Pdf URL: https://arxiv.org/pdf/2412.17692
• Copy Paste: [[2412.17692]] FedTLU: Federated Learning with Targeted Layer Updates(https://arxiv.org/abs/2412.17692)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) addresses privacy concerns in language modeling by enabling multiple clients to contribute to training language models. However, non-IID (identically and independently distributed) data across clients often limits FL's performance. This issue is especially challenging during model fine-tuning, as noise due to variations in clients' data distributions can harm model convergence near the optimum. This paper proposes a targeted layer update strategy for fine-tuning in FL. Instead of randomly updating layers of the language model, as often done in practice, we use a scoring mechanism to identify and update the most critical layers, avoiding excessively noisy or even poisoned updates by freezing the parameters in other layers. We show in extensive experiments that our method improves convergence and performance in non-IID settings, offering a more efficient approach to fine-tuning federated language models.

Title: Understanding the Logic of Direct Preference Alignment through Logic

• Authors: Kyle Richardson, Vivek Srikumar, Ashish Sabharwal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17696
• Pdf URL: https://arxiv.org/pdf/2412.17696
• Copy Paste: [[2412.17696]] Understanding the Logic of Direct Preference Alignment through Logic(https://arxiv.org/abs/2412.17696)
• Keywords: large language model
• Abstract: Recent direct preference alignment algorithms (DPA), such as DPO, have shown great promise in aligning large language models to human preferences. While this has motivated the development of many new variants of the original DPO loss, understanding the differences between these recent proposals, as well as developing new DPA loss functions, remains difficult given the lack of a technical and conceptual framework for reasoning about the underlying semantics of these algorithms. In this paper, we attempt to remedy this by formalizing DPA losses in terms of discrete reasoning problems. Specifically, we ask: Given an existing DPA loss, can we systematically derive a symbolic expression that characterizes its semantics? How do the semantics of two losses relate to each other? We propose a novel formalism for characterizing preference losses for single model and reference model based approaches, and identify symbolic forms for a number of commonly used DPA variants. Further, we show how this formal view of preference learning sheds new light on both the size and structure of the DPA loss landscape, making it possible to not only rigorously characterize the relationships between recent loss proposals but also to systematically explore the landscape and derive new loss functions from first principles. We hope our framework and findings will help provide useful guidance to those working on human AI alignment.

Title: GaussianPainter: Painting Point Cloud into 3D Gaussians with Normal Guidance

• Authors: Jingqiu Zhou, Lue Fan, Xuesong Chen, Linjiang Huang, Si Liu, Hongsheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17715
• Pdf URL: https://arxiv.org/pdf/2412.17715
• Copy Paste: [[2412.17715]] GaussianPainter: Painting Point Cloud into 3D Gaussians with Normal Guidance(https://arxiv.org/abs/2412.17715)
• Keywords: robust
• Abstract: In this paper, we present GaussianPainter, the first method to paint a point cloud into 3D Gaussians given a reference image. GaussianPainter introduces an innovative feed-forward approach to overcome the limitations of time-consuming test-time optimization in 3D Gaussian splatting. Our method addresses a critical challenge in the field: the non-uniqueness problem inherent in the large parameter space of 3D Gaussian splatting. This space, encompassing rotation, anisotropic scales, and spherical harmonic coefficients, introduces the challenge of rendering similar images from substantially different Gaussian fields. As a result, feed-forward networks face instability when attempting to directly predict high-quality Gaussian fields, struggling to converge on consistent parameters for a given output. To address this issue, we propose to estimate a surface normal for each point to determine its Gaussian rotation. This strategy enables the network to effectively predict the remaining Gaussian parameters in the constrained space. We further enhance our approach with an appearance injection module, incorporating reference image appearance into Gaussian fields via a multiscale triplane representation. Our method successfully balances efficiency and fidelity in 3D Gaussian generation, achieving high-quality, diverse, and robust 3D content creation from point clouds in a single forward pass.

Title: Asynchronous Federated Learning: A Scalable Approach for Decentralized Machine Learning

• Authors: Ali Forootani, Raffaele Iervolino
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2412.17723
• Pdf URL: https://arxiv.org/pdf/2412.17723
• Copy Paste: [[2412.17723]] Asynchronous Federated Learning: A Scalable Approach for Decentralized Machine Learning(https://arxiv.org/abs/2412.17723)
• Keywords: privacy, robust, federate
• Abstract: Federated Learning (FL) has emerged as a powerful paradigm for decentralized machine learning, enabling collaborative model training across diverse clients without sharing raw data. However, traditional FL approaches often face limitations in scalability and efficiency due to their reliance on synchronous client updates, which can result in significant delays and increased communication overhead, particularly in heterogeneous and dynamic environments. To address these challenges in this paper, we propose an Asynchronous Federated Learning (AFL) algorithm, which allows clients to update the global model independently and asynchronously. Our key contributions include a comprehensive convergence analysis of AFL in the presence of client delays and model staleness. By leveraging martingale difference sequence theory and variance bounds, we ensure robust convergence despite asynchronous updates. Assuming strongly convex local objective functions, we establish bounds on gradient variance under random client sampling and derive a recursion formula quantifying the impact of client delays on convergence. Furthermore, we demonstrate the practical applicability of AFL by training a decentralized Long Short-Term Memory (LSTM)-based deep learning model on the CMIP6 climate dataset, effectively handling non-IID and geographically distributed data. The proposed AFL algorithm addresses key limitations of traditional FL methods, such as inefficiency due to global synchronization and susceptibility to client drift. It enhances scalability, robustness, and efficiency in real-world settings with heterogeneous client populations and dynamic network conditions. Our results underscore the potential of AFL to drive advancements in distributed learning systems, particularly for large-scale, privacy-preserving applications in resource-constrained environments.

Title: VidTwin: Video VAE with Decoupled Structure and Dynamics

• Authors: Yuchi Wang, Junliang Guo, Xinyi Xie, Tianyu He, Xu Sun, Jiang Bian
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17726
• Pdf URL: https://arxiv.org/pdf/2412.17726
• Copy Paste: [[2412.17726]] VidTwin: Video VAE with Decoupled Structure and Dynamics(https://arxiv.org/abs/2412.17726)
• Keywords: explainability, generative
• Abstract: Recent advancements in video autoencoders (Video AEs) have significantly improved the quality and efficiency of video generation. In this paper, we propose a novel and compact video autoencoder, VidTwin, that decouples video into two distinct latent spaces: Structure latent vectors, which capture overall content and global movement, and Dynamics latent vectors, which represent fine-grained details and rapid movements. Specifically, our approach leverages an Encoder-Decoder backbone, augmented with two submodules for extracting these latent spaces, respectively. The first submodule employs a Q-Former to extract low-frequency motion trends, followed by downsampling blocks to remove redundant content details. The second averages the latent vectors along the spatial dimension to capture rapid motion. Extensive experiments show that VidTwin achieves a high compression rate of 0.20% with high reconstruction quality (PSNR of 28.14 on the MCL-JCV dataset), and performs efficiently and effectively in downstream generative tasks. Moreover, our model demonstrates explainability and scalability, paving the way for future research in video latent representation and generation. Our code has been released at this https URL.

Title: Knowledge Editing through Chain-of-Thought

• Authors: Changyue Wang, Weihang Su, Qingyao Ai, Yiqun Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17727
• Pdf URL: https://arxiv.org/pdf/2412.17727
• Copy Paste: [[2412.17727]] Knowledge Editing through Chain-of-Thought(https://arxiv.org/abs/2412.17727)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated exceptional capabilities across a wide range of natural language processing (NLP) tasks. However, keeping these models up-to-date with evolving world knowledge remains a significant challenge due to the high costs of frequent retraining. To address this challenge, knowledge editing techniques have emerged to update LLMs with new information without rebuilding the model from scratch. Among these, the in-context editing paradigm stands out for its effectiveness in integrating new knowledge while preserving the model's original capabilities. Despite its potential, existing in-context knowledge editing methods are often task-specific, focusing primarily on multi-hop QA tasks using structured knowledge triples. Moreover, their reliance on few-shot prompting for task decomposition makes them unstable and less effective in generalizing across diverse tasks. In response to these limitations, we propose EditCoT, a novel knowledge editing framework that flexibly and efficiently updates LLMs across various tasks without retraining. EditCoT works by generating a chain-of-thought (CoT) for a given input and then iteratively refining this CoT process using a CoT editor based on updated knowledge. We evaluate EditCoT across a diverse range of benchmarks, covering multiple languages and tasks. The results demonstrate that our approach achieves state-of-the-art performance while offering superior generalization, effectiveness, and stability compared to existing methods, marking a significant advancement in the field of knowledge updating. Code and data are available at: this https URL.

Title: LASE: Learned Adjacency Spectral Embeddings

• Authors: Sofía Pérez Casulo, Marcelo Fiori, Federico Larroca, Gonzalo Mateos
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.17734
• Pdf URL: https://arxiv.org/pdf/2412.17734
• Copy Paste: [[2412.17734]] LASE: Learned Adjacency Spectral Embeddings(https://arxiv.org/abs/2412.17734)
• Keywords: robust
• Abstract: We put forth a principled design of a neural architecture to learn nodal Adjacency Spectral Embeddings (ASE) from graph inputs. By bringing to bear the gradient descent (GD) method and leveraging the principle of algorithm unrolling, we truncate and re-interpret each GD iteration as a layer in a graph neural network (GNN) that is trained to approximate the ASE. Accordingly, we call the resulting embeddings and our parametric model Learned ASE (LASE), which is interpretable, parameter efficient, robust to inputs with unobserved edges, and offers controllable complexity during inference. LASE layers combine Graph Convolutional Network (GCN) and fully-connected Graph Attention Network (GAT) modules, which is intuitively pleasing since GCN-based local aggregations alone are insufficient to express the sought graph eigenvectors. We propose several refinements to the unrolled LASE architecture (such as sparse attention in the GAT module and decoupled layerwise parameters) that offer favorable approximation error versus computation tradeoffs; even outperforming heavily-optimized eigendecomposition routines from scientific computing libraries. Because LASE is a differentiable function with respect to its parameters as well as its graph input, we can seamlessly integrate it as a trainable module within a larger (semi-)supervised graph representation learning pipeline. The resulting end-to-end system effectively learns ``discriminative ASEs'' that exhibit competitive performance in supervised link prediction and node classification tasks, outperforming a GNN even when the latter is endowed with open loop, meaning task-agnostic, precomputed spectral positional encodings.

Title: Contextual Backpropagation Loops: Amplifying Deep Reasoning with Iterative Top-Down Feedback

• Authors: Jacob Fein-Ashley
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17737
• Pdf URL: https://arxiv.org/pdf/2412.17737
• Copy Paste: [[2412.17737]] Contextual Backpropagation Loops: Amplifying Deep Reasoning with Iterative Top-Down Feedback(https://arxiv.org/abs/2412.17737)
• Keywords: robust
• Abstract: Deep neural networks typically rely on a single forward pass for inference, which can limit their capacity to resolve ambiguous inputs. We introduce Contextual Backpropagation Loops (CBLs) as an iterative mechanism that incorporates top-down feedback to refine intermediate representations, thereby improving accuracy and robustness. This repeated process mirrors how humans continuously re-interpret sensory information in daily life-by checking and re-checking our perceptions using contextual cues. Our results suggest that CBLs can offer a straightforward yet powerful way to incorporate such contextual reasoning in modern deep learning architectures.

Title: Sensitivity Curve Maximization: Attacking Robust Aggregators in Distributed Learning

• Authors: Christian A. Schroth, Stefan Vlaski, Abdelhak M. Zoubir
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2412.17740
• Pdf URL: https://arxiv.org/pdf/2412.17740
• Copy Paste: [[2412.17740]] Sensitivity Curve Maximization: Attacking Robust Aggregators in Distributed Learning(https://arxiv.org/abs/2412.17740)
• Keywords: attack, robust
• Abstract: In distributed learning agents aim at collaboratively solving a global learning problem. It becomes more and more likely that individual agents are malicious or faulty with an increasing size of the network. This leads to a degeneration or complete breakdown of the learning process. Classical aggregation schemes are prone to breakdown at small contamination rates, therefore robust aggregation schemes are sought for. While robust aggregation schemes can generally tolerate larger contamination rates, many have been shown to be susceptible to carefully crafted malicious attacks. In this work, we show how the sensitivity curve (SC), a classical tool from robust statistics, can be used to systematically derive optimal attack patterns against arbitrary robust aggregators, in most cases rendering them ineffective. We show the effectiveness of the proposed attack in multiple simulations.

Title: Reasoning to Attend: Try to Understand How Token Works

• Authors: Rui Qian, Xin Yin, Dejing Dou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17741
• Pdf URL: https://arxiv.org/pdf/2412.17741
• Copy Paste: [[2412.17741]] Reasoning to Attend: Try to Understand How Token Works(https://arxiv.org/abs/2412.17741)
• Keywords: large language model
• Abstract: Current Large Multimodal Models (LMMs) empowered visual grounding typically rely on $\texttt{}$ token as a text prompt to jointly optimize the vision-language model (e.g., LLaVA) and the downstream task-specified model (\eg, SAM). However, we observe that little research has looked into how it this http URL this work, we first visualize the similarity maps, which are obtained by computing the semantic similarity between the $\texttt{}$ token and the image token embeddings derived from the last hidden layer in both the LLaVA encoder and SAM decoder. Intriguingly, we have found that a striking consistency holds in terms of activation responses in the similarity map,which reveals that what $\texttt{}$ token contributes to is the semantic similarity within image-text pairs. Specifically, $\texttt{}$ token, a placeholder expanded in text vocabulary, extensively queries among individual tokenized image patches to match the semantics of an object from text to the paired image while the Large Language Models (LLMs) are being fine-tuned. Upon the above findings, we present READ, which facilitates LMMs' resilient $\textbf{REA}$soning capability of where to atten$\textbf{D}$ under the guidance of highly activated points borrowed from similarity maps. Remarkably, READ features an intuitive design, Similarity as Points module (SasP), which can be seamlessly applied to $\texttt{}$-like paradigms in a plug-and-play this http URL, extensive experiments have been conducted on the ReasonSeg and RefCOCO(+/g) datasets. To validate whether READ suffers from catastrophic forgetting of previous skills after fine-tuning, we further assess its generation ability on an augmented FP-RefCOCO(+/g) dataset. All codes and models are publicly available at this https URL.

Title: YuLan-Mini: An Open Data-efficient Language Model

• Authors: Yiwen Hu, Huatong Song, Jia Deng, Jiapeng Wang, Jie Chen, Kun Zhou, Yutao Zhu, Jinhao Jiang, Zican Dong, Wayne Xin Zhao, Ji-Rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17743
• Pdf URL: https://arxiv.org/pdf/2412.17743
• Copy Paste: [[2412.17743]] YuLan-Mini: An Open Data-efficient Language Model(https://arxiv.org/abs/2412.17743)
• Keywords: robust, large language model
• Abstract: Effective pre-training of large language models (LLMs) has been challenging due to the immense resource demands and the complexity of the technical processes involved. This paper presents a detailed technical report on YuLan-Mini, a highly capable base model with 2.42B parameters that achieves top-tier performance among models of similar parameter scale. Our pre-training approach focuses on enhancing training efficacy through three key technical contributions: an elaborate data pipeline combines data cleaning with data schedule strategies, a robust optimization method to mitigate training instability, and an effective annealing approach that incorporates targeted data selection and long context training. Remarkably, YuLan-Mini, trained on 1.08T tokens, achieves performance comparable to industry-leading models that require significantly more data. To facilitate reproduction, we release the full details of the data composition for each training phase. Project details can be accessed at the following link: this https URL.

Title: Deliberation in Latent Space via Differentiable Cache Augmentation

• Authors: Luyang Liu, Jonas Pfeiffer, Jiaxing Wu, Jun Xie, Arthur Szlam
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17747
• Pdf URL: https://arxiv.org/pdf/2412.17747
• Copy Paste: [[2412.17747]] Deliberation in Latent Space via Differentiable Cache Augmentation(https://arxiv.org/abs/2412.17747)
• Keywords: large language model
• Abstract: Techniques enabling large language models (LLMs) to "think more" by generating and attending to intermediate reasoning steps have shown promise in solving complex problems. However, the standard approaches generate sequences of discrete tokens immediately before responding, and so they can incur significant latency costs and be challenging to optimize. In this work, we demonstrate that a frozen LLM can be augmented with an offline coprocessor that operates on the model's key-value (kv) cache. This coprocessor augments the cache with a set of latent embeddings designed to improve the fidelity of subsequent decoding. We train this coprocessor using the language modeling loss from the decoder on standard pretraining data, while keeping the decoder itself frozen. This approach enables the model to learn, in an end-to-end differentiable fashion, how to distill additional computation into its kv-cache. Because the decoder remains unchanged, the coprocessor can operate offline and asynchronously, and the language model can function normally if the coprocessor is unavailable or if a given cache is deemed not to require extra computation. We show experimentally that when a cache is augmented, the decoder achieves lower perplexity on numerous subsequent tokens. Furthermore, even without any task-specific training, our experiments demonstrate that cache augmentation consistently reduces perplexity and improves performance across a range of reasoning-intensive tasks.

Title: In Case You Missed It: ARC 'Challenge' Is Not That Challenging

• Authors: Łukasz Borchmann
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17758
• Pdf URL: https://arxiv.org/pdf/2412.17758
• Copy Paste: [[2412.17758]] In Case You Missed It: ARC 'Challenge' Is Not That Challenging(https://arxiv.org/abs/2412.17758)
• Keywords: fair
• Abstract: ARC Challenge appears more difficult than ARC Easy for modern LLMs primarily due to an evaluation setup that prevents direct comparison of answer choices rather than inherent complexity. Although some researchers have quietly shifted to a more appropriate scheme over the last year, the implications of this change have yet to be widely acknowledged. We highlight this overlooked shift, show how similar evaluation practices falsely imply reasoning deficits in other benchmarks, and demonstrate that fairer methods dramatically reduce performance gaps (e.g. on SIQA) and even yield superhuman results (OpenBookQA). In doing so, we reveal how evaluation shapes perceived difficulty and offer guidelines to ensure that multiple-choice evaluations accurately reflect actual model capabilities.

Title: The Superposition of Diffusion Models Using the It\^o Density Estimator

• Authors: Marta Skreta, Lazar Atanackovic, Avishek Joey Bose, Alexander Tong, Kirill Neklyudov
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17762
• Pdf URL: https://arxiv.org/pdf/2412.17762
• Copy Paste: [[2412.17762]] The Superposition of Diffusion Models Using the It\^o Density Estimator(https://arxiv.org/abs/2412.17762)
• Keywords: diffusion
• Abstract: The Cambrian explosion of easily accessible pre-trained diffusion models suggests a demand for methods that combine multiple different pre-trained diffusion models without incurring the significant computational burden of re-training a larger combined model. In this paper, we cast the problem of combining multiple pre-trained diffusion models at the generation stage under a novel proposed framework termed superposition. Theoretically, we derive superposition from rigorous first principles stemming from the celebrated continuity equation and design two novel algorithms tailor-made for combining diffusion models in SuperDiff. SuperDiff leverages a new scalable Itô density estimator for the log likelihood of the diffusion SDE which incurs no additional overhead compared to the well-known Hutchinson's estimator needed for divergence calculations. We demonstrate that SuperDiff is scalable to large pre-trained diffusion models as superposition is performed solely through composition during inference, and also enjoys painless implementation as it combines different pre-trained vector fields through an automated re-weighting scheme. Notably, we show that SuperDiff is efficient during inference time, and mimics traditional composition operators such as the logical OR and the logical AND. We empirically demonstrate the utility of using SuperDiff for generating more diverse images on CIFAR-10, more faithful prompt conditioned image editing using Stable Diffusion, and improved unconditional de novo structure design of proteins. this https URL

Title: ResearchTown: Simulator of Human Research Community

• Authors: Haofei Yu, Zhaochen Hong, Zirui Cheng, Kunlun Zhu, Keyang Xuan, Jinwei Yao, Tao Feng, Jiaxuan You
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17767
• Pdf URL: https://arxiv.org/pdf/2412.17767
• Copy Paste: [[2412.17767]] ResearchTown: Simulator of Human Research Community(https://arxiv.org/abs/2412.17767)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable potential in scientific domains, yet a fundamental question remains unanswered: Can we simulate human research communities with LLMs? Addressing this question can deepen our understanding of the processes behind idea brainstorming and inspire the automatic discovery of novel scientific insights. In this work, we propose ResearchTown, a multi-agent framework for research community simulation. Within this framework, the human research community is simplified and modeled as an agent-data graph, where researchers and papers are represented as agent-type and data-type nodes, respectively, and connected based on their collaboration relationships. We also introduce TextGNN, a text-based inference framework that models various research activities (e.g., paper reading, paper writing, and review writing) as special forms of a unified message-passing process on the agent-data graph. To evaluate the quality of the research simulation, we present ResearchBench, a benchmark that uses a node-masking prediction task for scalable and objective assessment based on similarity. Our experiments reveal three key findings: (1) ResearchTown can provide a realistic simulation of collaborative research activities, including paper writing and review writing; (2) ResearchTown can maintain robust simulation with multiple researchers and diverse papers; (3) ResearchTown can generate interdisciplinary research ideas that potentially inspire novel research directions.

Title: Examining Imbalance Effects on Performance and Demographic Fairness of Clinical Language Models

• Authors: Precious Jones, Weisi Liu, I-Chan Huang, Xiaolei Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17803
• Pdf URL: https://arxiv.org/pdf/2412.17803
• Copy Paste: [[2412.17803]] Examining Imbalance Effects on Performance and Demographic Fairness of Clinical Language Models(https://arxiv.org/abs/2412.17803)
• Keywords: robust, fair
• Abstract: Data imbalance is a fundamental challenge in applying language models to biomedical applications, particularly in ICD code prediction tasks where label and demographic distributions are uneven. While state-of-the-art language models have been increasingly adopted in biomedical tasks, few studies have systematically examined how data imbalance affects model performance and fairness across demographic groups. This study fills the gap by statistically probing the relationship between data imbalance and model performance in ICD code prediction. We analyze imbalances in a standard benchmark data across gender, age, ethnicity, and social determinants of health by state-of-the-art biomedical language models. By deploying diverse performance metrics and statistical analyses, we explore the influence of data imbalance on performance variations and demographic fairness. Our study shows that data imbalance significantly impacts model performance and fairness, but feature similarity to the majority class may be a more critical factor. We believe this study provides valuable insights for developing more equitable and robust language models in healthcare applications.

Title: GauSim: Registering Elastic Objects into Digital World by Gaussian Simulator

• Authors: Yidi Shao, Mu Huang, Chen Change Loy, Bo Dai
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.17804
• Pdf URL: https://arxiv.org/pdf/2412.17804
• Copy Paste: [[2412.17804]] GauSim: Registering Elastic Objects into Digital World by Gaussian Simulator(https://arxiv.org/abs/2412.17804)
• Keywords: robust
• Abstract: In this work, we introduce GauSim, a novel neural network-based simulator designed to capture the dynamic behaviors of real-world elastic objects represented through Gaussian kernels. Unlike traditional methods that treat kernels as particles within particle-based simulations, we leverage continuum mechanics, modeling each kernel as a continuous piece of matter to account for realistic deformations without idealized assumptions. To improve computational efficiency and fidelity, we employ a hierarchical structure that organizes kernels into Center of Mass Systems (CMS) with explicit formulations, enabling a coarse-to-fine simulation approach. This structure significantly reduces computational overhead while preserving detailed dynamics. In addition, GauSim incorporates explicit physics constraints, such as mass and momentum conservation, ensuring interpretable results and robust, physically plausible simulations. To validate our approach, we present a new dataset, READY, containing multi-view videos of real-world elastic deformations. Experimental results demonstrate that GauSim achieves superior performance compared to existing physics-driven baselines, offering a practical and accurate solution for simulating complex dynamic behaviors. Code and model will be released. Project page: this https URL .

Title: Large Motion Video Autoencoding with Cross-modal Video VAE

• Authors: Yazhou Xing, Yang Fei, Yingqing He, Jingye Chen, Jiaxin Xie, Xiaowei Chi, Qifeng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17805
• Pdf URL: https://arxiv.org/pdf/2412.17805
• Copy Paste: [[2412.17805]] Large Motion Video Autoencoding with Cross-modal Video VAE(https://arxiv.org/abs/2412.17805)
• Keywords: robust
• Abstract: Learning a robust video Variational Autoencoder (VAE) is essential for reducing video redundancy and facilitating efficient video generation. Directly applying image VAEs to individual frames in isolation can result in temporal inconsistencies and suboptimal compression rates due to a lack of temporal compression. Existing Video VAEs have begun to address temporal compression; however, they often suffer from inadequate reconstruction performance. In this paper, we present a novel and powerful video autoencoder capable of high-fidelity video encoding. First, we observe that entangling spatial and temporal compression by merely extending the image VAE to a 3D VAE can introduce motion blur and detail distortion artifacts. Thus, we propose temporal-aware spatial compression to better encode and decode the spatial information. Additionally, we integrate a lightweight motion compression model for further temporal compression. Second, we propose to leverage the textual information inherent in text-to-video datasets and incorporate text guidance into our model. This significantly enhances reconstruction quality, particularly in terms of detail preservation and temporal stability. Third, we further improve the versatility of our model through joint training on both images and videos, which not only enhances reconstruction quality but also enables the model to perform both image and video autoencoding. Extensive evaluations against strong recent baselines demonstrate the superior performance of our method. The project website can be found at~\href{this https URL}{this https URL}.

Title: Reconstructing People, Places, and Cameras

• Authors: Lea Müller, Hongsuk Choi, Anthony Zhang, Brent Yi, Jitendra Malik, Angjoo Kanazawa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17806
• Pdf URL: https://arxiv.org/pdf/2412.17806
• Copy Paste: [[2412.17806]] Reconstructing People, Places, and Cameras(https://arxiv.org/abs/2412.17806)
• Keywords: robust
• Abstract: We present "Humans and Structure from Motion" (HSfM), a method for jointly reconstructing multiple human meshes, scene point clouds, and camera parameters in a metric world coordinate system from a sparse set of uncalibrated multi-view images featuring people. Our approach combines data-driven scene reconstruction with the traditional Structure-from-Motion (SfM) framework to achieve more accurate scene reconstruction and camera estimation, while simultaneously recovering human meshes. In contrast to existing scene reconstruction and SfM methods that lack metric scale information, our method estimates approximate metric scale by leveraging a human statistical model. Furthermore, it reconstructs multiple human meshes within the same world coordinate system alongside the scene point cloud, effectively capturing spatial relationships among individuals and their positions in the environment. We initialize the reconstruction of humans, scenes, and cameras using robust foundational models and jointly optimize these elements. This joint optimization synergistically improves the accuracy of each component. We compare our method to existing approaches on two challenging benchmarks, EgoHumans and EgoExo4D, demonstrating significant improvements in human localization accuracy within the world coordinate frame (reducing error from 3.51m to 1.04m in EgoHumans and from 2.9m to 0.56m in EgoExo4D). Notably, our results show that incorporating human data into the SfM pipeline improves camera pose estimation (e.g., increasing RRA@15 by 20.3% on EgoHumans). Additionally, qualitative results show that our approach improves overall scene reconstruction quality. Our code is available at: this http URL.

Title: Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoders

• Authors: Rui Chen, Jianfeng Zhang, Yixun Liang, Guan Luo, Weiyu Li, Jiarui Liu, Xiu Li, Xiaoxiao Long, Jiashi Feng, Ping Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17808
• Pdf URL: https://arxiv.org/pdf/2412.17808
• Copy Paste: [[2412.17808]] Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoders(https://arxiv.org/abs/2412.17808)
• Keywords: diffusion
• Abstract: Recent 3D content generation pipelines commonly employ Variational Autoencoders (VAEs) to encode shapes into compact latent representations for diffusion-based generation. However, the widely adopted uniform point sampling strategy in Shape VAE training often leads to a significant loss of geometric details, limiting the quality of shape reconstruction and downstream generation tasks. We present Dora-VAE, a novel approach that enhances VAE reconstruction through our proposed sharp edge sampling strategy and a dual cross-attention mechanism. By identifying and prioritizing regions with high geometric complexity during training, our method significantly improves the preservation of fine-grained shape features. Such sampling strategy and the dual attention mechanism enable the VAE to focus on crucial geometric details that are typically missed by uniform sampling approaches. To systematically evaluate VAE reconstruction quality, we additionally propose Dora-bench, a benchmark that quantifies shape complexity through the density of sharp edges, introducing a new metric focused on reconstruction accuracy at these salient geometric features. Extensive experiments on the Dora-bench demonstrate that Dora-VAE achieves comparable reconstruction quality to the state-of-the-art dense XCube-VAE while requiring a latent space at least 8$\times$ smaller (1,280 vs. > 10,000 codes). We will release our code and benchmark dataset to facilitate future research in 3D shape modeling.

Title: Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction

• Authors: Ziyang Wu, Tianjiao Ding, Yifu Lu, Druv Pai, Jingyuan Zhang, Weida Wang, Yaodong Yu, Yi Ma, Benjamin D. Haeffele
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17810
• Pdf URL: https://arxiv.org/pdf/2412.17810
• Copy Paste: [[2412.17810]] Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction(https://arxiv.org/abs/2412.17810)
• Keywords: transformer
• Abstract: The attention operator is arguably the key distinguishing factor of transformer architectures, which have demonstrated state-of-the-art performance on a variety of tasks. However, transformer attention operators often impose a significant computational burden, with the computational complexity scaling quadratically with the number of tokens. In this work, we propose a novel transformer attention operator whose computational complexity scales linearly with the number of tokens. We derive our network architecture by extending prior work which has shown that a transformer style architecture naturally arises by "white-box" architecture design, where each layer of the network is designed to implement an incremental optimization step of a maximal coding rate reduction objective (MCR$^2$). Specifically, we derive a novel variational form of the MCR$^2$ objective and show that the architecture that results from unrolled gradient descent of this variational objective leads to a new attention module called Token Statistics Self-Attention (TSSA). TSSA has linear computational and memory complexity and radically departs from the typical attention architecture that computes pairwise similarities between tokens. Experiments on vision, language, and long sequence tasks show that simply swapping TSSA for standard self-attention, which we refer to as the Token Statistics Transformer (ToST), achieves competitive performance with conventional transformers while being significantly more computationally efficient and interpretable. Our results also somewhat call into question the conventional wisdom that pairwise similarity style attention mechanisms are critical to the success of transformer architectures. Code will be available at this https URL.

Title: ChatGarment: Garment Estimation, Generation and Editing via Large Language Models

• Authors: Siyuan Bian, Chenghao Xu, Yuliang Xiu, Artur Grigorev, Zhen Liu, Cewu Lu, Michael J. Black, Yao Feng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17811
• Pdf URL: https://arxiv.org/pdf/2412.17811
• Copy Paste: [[2412.17811]] ChatGarment: Garment Estimation, Generation and Editing via Large Language Models(https://arxiv.org/abs/2412.17811)
• Keywords: large language model
• Abstract: We introduce ChatGarment, a novel approach that leverages large vision-language models (VLMs) to automate the estimation, generation, and editing of 3D garments from images or text descriptions. Unlike previous methods that struggle in real-world scenarios or lack interactive editing capabilities, ChatGarment can estimate sewing patterns from in-the-wild images or sketches, generate them from text descriptions, and edit garments based on user instructions, all within an interactive dialogue. These sewing patterns can then be draped into 3D garments, which are easily animatable and simulatable. This is achieved by finetuning a VLM to directly generate a JSON file that includes both textual descriptions of garment types and styles, as well as continuous numerical attributes. This JSON file is then used to create sewing patterns through a programming parametric model. To support this, we refine the existing programming model, GarmentCode, by expanding its garment type coverage and simplifying its structure for efficient VLM fine-tuning. Additionally, we construct a large-scale dataset of image-to-sewing-pattern and text-to-sewing-pattern pairs through an automated data pipeline. Extensive evaluations demonstrate ChatGarment's ability to accurately reconstruct, generate, and edit garments from multimodal inputs, highlighting its potential to revolutionize workflows in fashion and gaming applications. Code and data will be available at this https URL.

Title: FaceLift: Single Image to 3D Head with View Generation and GS-LRM

• Authors: Weijie Lyu, Yi Zhou, Ming-Hsuan Yang, Zhixin Shu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.17812
• Pdf URL: https://arxiv.org/pdf/2412.17812
• Copy Paste: [[2412.17812]] FaceLift: Single Image to 3D Head with View Generation and GS-LRM(https://arxiv.org/abs/2412.17812)
• Keywords: robust, diffusion
• Abstract: We present FaceLift, a feed-forward approach for rapid, high-quality, 360-degree head reconstruction from a single image. Our pipeline begins by employing a multi-view latent diffusion model that generates consistent side and back views of the head from a single facial input. These generated views then serve as input to a GS-LRM reconstructor, which produces a comprehensive 3D representation using Gaussian splats. To train our system, we develop a dataset of multi-view renderings using synthetic 3D human head as-sets. The diffusion-based multi-view generator is trained exclusively on synthetic head images, while the GS-LRM reconstructor undergoes initial training on Objaverse followed by fine-tuning on synthetic head data. FaceLift excels at preserving identity and maintaining view consistency across views. Despite being trained solely on synthetic data, FaceLift demonstrates remarkable generalization to real-world images. Through extensive qualitative and quantitative evaluations, we show that FaceLift outperforms state-of-the-art methods in 3D head reconstruction, highlighting its practical applicability and robust performance on real-world images. In addition to single image reconstruction, FaceLift supports video inputs for 4D novel view synthesis and seamlessly integrates with 2D reanimation techniques to enable 3D facial animation. Project page: this https URL.