2024-12-05

Title: DYffCast: Regional Precipitation Nowcasting Using IMERG Satellite Data. A case study over South America

• Authors: Daniel Seal, Rossella Arcucci, Salva Rühling-Cachay, César Quilodrán-Casas
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02723
• Pdf URL: https://arxiv.org/pdf/2412.02723
• Copy Paste: [[2412.02723]] DYffCast: Regional Precipitation Nowcasting Using IMERG Satellite Data. A case study over South America(https://arxiv.org/abs/2412.02723)
• Keywords: generative
• Abstract: Climate change is increasing the frequency of extreme precipitation events, making weather disasters such as flooding and landslides more likely. The ability to accurately nowcast precipitation is therefore becoming more critical for safeguarding society by providing immediate, accurate information to decision makers. Motivated by the recent success of generative models at precipitation nowcasting, this paper: extends the DYffusion framework to this task and evaluates its performance at forecasting IMERG satellite precipitation data up to a 4-hour horizon; modifies the DYffusion framework to improve its ability to model rainfall data; and introduces a novel loss function that combines MSE, MAE and the LPIPS perceptual score. In a quantitative evaluation of forecasts up to a 4-hour horizon, the modified DYffusion framework trained with the novel loss outperforms four competitor models. It has the highest CSI scores for weak, moderate, and heavy rain thresholds and retains an LPIPS score $<$ 0.2 for the entire roll-out, degrading the least as lead-time increases. The proposed nowcasting model demonstrates visually stable and sharp forecasts up to a 2-hour horizon on a heavy rain case study. Code is available at this https URL.

Title: Mixture of Physical Priors Adapter for Parameter-Efficient Fine-Tuning

• Authors: Zhaozhi Wang, Conghu Li, Qixiang Ye, Tong Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02759
• Pdf URL: https://arxiv.org/pdf/2412.02759
• Copy Paste: [[2412.02759]] Mixture of Physical Priors Adapter for Parameter-Efficient Fine-Tuning(https://arxiv.org/abs/2412.02759)
• Keywords: diffusion, transformer
• Abstract: Most parameter-efficient fine-tuning (PEFT) methods rely on low-rank representations to adapt models. However, these approaches often oversimplify representations, particularly when the underlying data has high-rank or high-frequency components. This limitation hinders the model's ability to capture complex data interactions effectively. In this paper, we propose a novel approach that models network weights by leveraging a combination of physical priors, enabling more accurate approximations. We use three foundational equations -- heat diffusion, wave propagation, and Poisson's steady-state equation -- each contributing distinctive modeling properties: heat diffusion enforces local smoothness, wave propagation facilitates long-range interactions, and Poisson's equation captures global equilibrium. To combine these priors effectively, we introduce the Mixture of Physical Priors Adapter (MoPPA), using an efficient Discrete Cosine Transform (DCT) implementation. To dynamically balance these priors, a route regularization mechanism is designed to adaptively tune their contributions. MoPPA serves as a lightweight, plug-and-play module that seamlessly integrates into transformer architectures, with adaptable complexity depending on the local context. Specifically, using MAE pre-trained ViT-B, MoPPA improves PEFT accuracy by up to 2.1% on VTAB-1K image classification with a comparable number of trainable parameters, and advantages are further validated through experiments across various vision backbones, showcasing MoPPA's effectiveness and adaptability. The code will be made public available.

Title: Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training

• Authors: H. Toprak Kesgin, M. Kaan Yuce, Eren Dogan, M. Egemen Uzun, Atahan Uz, Elif Ince, Yusuf Erdem, Osama Shbib, Ahmed Zeer, M. Fatih Amasyali
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.02775
• Pdf URL: https://arxiv.org/pdf/2412.02775
• Copy Paste: [[2412.02775]] Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training(https://arxiv.org/abs/2412.02775)
• Keywords: large language model
• Abstract: In this study, we develop and assess new corpus selection and training methodologies to improve the effectiveness of Turkish language models. Specifically, we adapted Large Language Model generated datasets and translated English datasets into Turkish, integrating these resources into the training process. This approach led to substantial enhancements in model accuracy for both few-shot and zero-shot learning scenarios. Furthermore, the merging of these adapted models was found to markedly improve their performance. Human evaluative metrics, including task-specific performance assessments, further demonstrated that these adapted models possess a greater aptitude for comprehending the Turkish language and addressing logic-based queries. This research underscores the importance of refining corpus selection strategies to optimize the performance of multilingual models, particularly for under-resourced languages like Turkish.

Title: Hacking CTFs with Plain Agents

• Authors: Rustem Turtayev, Artem Petrov, Dmitrii Volkov, Denis Volk
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02776
• Pdf URL: https://arxiv.org/pdf/2412.02776
• Copy Paste: [[2412.02776]] Hacking CTFs with Plain Agents(https://arxiv.org/abs/2412.02776)
• Keywords: security
• Abstract: We saturate a high-school-level hacking benchmark with plain LLM agent design. Concretely, we obtain 95% performance on InterCode-CTF, a popular offensive security benchmark, using prompting, tool use, and multiple attempts. This beats prior work by Phuong et al. 2024 (29%) and Abramovich et al. 2024 (72%). Our results suggest that current LLMs have surpassed the high school level in offensive cybersecurity. Their hacking capabilities remain underelicited: our ReAct&Plan prompting strategy solves many challenges in 1-2 turns without complex engineering or advanced harnessing.

Title: Synergistic Development of Perovskite Memristors and Algorithms for Robust Analog Computing

• Authors: Nanyang Ye, Qiao Sun, Yifei Wang, Liujia Yang, Jundong Zhou, Lei Wang, Guang-Zhong Yang, Xinbing Wang, Chenghu Zhou, Huaqiang Wu, Qinying Gu
• Subjects: cs.LG, cs.AI, cs.ET
• Abstract URL: https://arxiv.org/abs/2412.02779
• Pdf URL: https://arxiv.org/pdf/2412.02779
• Copy Paste: [[2412.02779]] Synergistic Development of Perovskite Memristors and Algorithms for Robust Analog Computing(https://arxiv.org/abs/2412.02779)
• Keywords: robust
• Abstract: Analog computing using non-volatile memristors has emerged as a promising solution for energy-efficient deep learning. New materials, like perovskites-based memristors are recently attractive due to their cost-effectiveness, energy efficiency and flexibility. Yet, challenges in material diversity and immature fabrications require extensive experimentation for device development. Moreover, significant non-idealities in these memristors often impede them for computing. Here, we propose a synergistic methodology to concurrently optimize perovskite memristor fabrication and develop robust analog DNNs that effectively address the inherent non-idealities of these memristors. Employing Bayesian optimization (BO) with a focus on usability, we efficiently identify optimal materials and fabrication conditions for perovskite memristors. Meanwhile, we developed "BayesMulti", a DNN training strategy utilizing BO-guided noise injection to improve the resistance of analog DNNs to memristor imperfections. Our approach theoretically ensures that within a certain range of parameter perturbations due to memristor non-idealities, the prediction outcomes remain consistent. Our integrated approach enables use of analog computing in much deeper and wider networks, which significantly outperforms existing methods in diverse tasks like image classification, autonomous driving, species identification, and large vision-language models, achieving up to 100-fold improvements. We further validate our methodology on a 10$\times$10 optimized perovskite memristor crossbar, demonstrating high accuracy in a classification task and low energy consumption. This study offers a versatile solution for efficient optimization of various analog computing systems, encompassing both devices and algorithms.

Title: Hybrid-SQuAD: Hybrid Scholarly Question Answering Dataset

• Authors: Tilahun Abedissa Taffa, Debayan Baneerje, Yaregal Assabie, Ricardo Usbeck
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02788
• Pdf URL: https://arxiv.org/pdf/2412.02788
• Copy Paste: [[2412.02788]] Hybrid-SQuAD: Hybrid Scholarly Question Answering Dataset(https://arxiv.org/abs/2412.02788)
• Keywords: large language model
• Abstract: Existing Scholarly Question Answering (QA) methods typically target homogeneous data sources, relying solely on either text or Knowledge Graphs (KGs). However, scholarly information often spans heterogeneous sources, necessitating the development of QA systems that can integrate information from multiple heterogeneous data sources. To address this challenge, we introduce Hybrid-SQuAD (Hybrid Scholarly Question Answering Dataset), a novel large-scale QA dataset designed to facilitate answering questions incorporating both text and KG facts. The dataset consists of 10.5K question-answer pairs generated by a large language model, leveraging the KGs - DBLP and SemOpenAlex alongside corresponding text from Wikipedia. In addition, we propose a RAG-based baseline hybrid QA model, achieving an exact match score of 69.65 on the Hybrid-SQuAD test set.

Title: An Evolutionary Large Language Model for Hallucination Mitigation

• Authors: Abdennour Boulesnane, Abdelhakim Souilah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02790
• Pdf URL: https://arxiv.org/pdf/2412.02790
• Copy Paste: [[2412.02790]] An Evolutionary Large Language Model for Hallucination Mitigation(https://arxiv.org/abs/2412.02790)
• Keywords: robust, large language model
• Abstract: The emergence of LLMs, like ChatGPT and Gemini, has marked the modern era of artificial intelligence applications characterized by high-impact applications generating text, images, and videos. However, these models usually ensue with one critical challenge called hallucination: confident presentation of inaccurate or fabricated information. This problem attracts serious concern when these models are applied to specialized domains, including healthcare and law, where the accuracy and preciseness of information are absolute conditions. In this paper, we propose EvoLLMs, an innovative framework inspired by Evolutionary Computation, which automates the generation of high-quality Question-answering (QA) datasets while minimizing hallucinations. EvoLLMs employs genetic algorithms, mimicking evolutionary processes like selection, variation, and mutation, to guide LLMs in generating accurate, contextually relevant question-answer pairs. Comparative analysis shows that EvoLLMs consistently outperforms human-generated datasets in key metrics such as Depth, Relevance, and Coverage, while nearly matching human performance in mitigating hallucinations. These results highlight EvoLLMs as a robust and efficient solution for QA dataset generation, significantly reducing the time and resources required for manual curation.

Title: Hijacking Vision-and-Language Navigation Agents with Adversarial Environmental Attacks

• Authors: Zijiao Yang, Xiangxi Shi, Eric Slyman, Stefan Lee
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.02795
• Pdf URL: https://arxiv.org/pdf/2412.02795
• Copy Paste: [[2412.02795]] Hijacking Vision-and-Language Navigation Agents with Adversarial Environmental Attacks(https://arxiv.org/abs/2412.02795)
• Keywords: attack
• Abstract: Assistive embodied agents that can be instructed in natural language to perform tasks in open-world environments have the potential to significantly impact labor tasks like manufacturing or in-home care -- benefiting the lives of those who come to depend on them. In this work, we consider how this benefit might be hijacked by local modifications in the appearance of the agent's operating environment. Specifically, we take the popular Vision-and-Language Navigation (VLN) task as a representative setting and develop a whitebox adversarial attack that optimizes a 3D attack object's appearance to induce desired behaviors in pretrained VLN agents that observe it in the environment. We demonstrate that the proposed attack can cause VLN agents to ignore their instructions and execute alternative actions after encountering the attack object -- even for instructions and agent paths not considered when optimizing the attack. For these novel settings, we find our attacks can induce early-termination behaviors or divert an agent along an attacker-defined multi-step trajectory. Under both conditions, environmental attacks significantly reduce agent capabilities to successfully follow user instructions.

Title: Grayscale to Hyperspectral at Any Resolution Using a Phase-Only Lens

• Authors: Dean Hazineh, Federico Capasso, Todd Zickler
• Subjects: cs.CV, eess.IV, physics.optics
• Abstract URL: https://arxiv.org/abs/2412.02798
• Pdf URL: https://arxiv.org/pdf/2412.02798
• Copy Paste: [[2412.02798]] Grayscale to Hyperspectral at Any Resolution Using a Phase-Only Lens(https://arxiv.org/abs/2412.02798)
• Keywords: diffusion
• Abstract: We consider the problem of reconstructing a $H\times W\times 31$ hyperspectral image from a $H\times W$ grayscale snapshot measurement that is captured using a single diffractive optic and a filterless panchromatic photosensor. This problem is severely ill-posed, and we present the first model that is able to produce high-quality results. We train a conditional denoising diffusion model that maps a small grayscale measurement patch to a hyperspectral patch. We then deploy the model to many patches in parallel, using global physics-based guidance to synchronize the patch predictions. Our model can be trained using small hyperspectral datasets and then deployed to reconstruct hyperspectral images of arbitrary size. Also, by drawing multiple samples with different seeds, our model produces useful uncertainty maps. We show that our model achieves state-of-the-art performance on previous snapshot hyperspectral benchmarks where reconstruction is better conditioned. Our work lays the foundation for a new class of high-resolution hyperspectral imagers that are compact and light-efficient.

Title: Gaussian Splatting Under Attack: Investigating Adversarial Noise in 3D Objects

• Authors: Abdurrahman Zeybey, Mehmet Ergezer, Tommy Nguyen
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.02803
• Pdf URL: https://arxiv.org/pdf/2412.02803
• Copy Paste: [[2412.02803]] Gaussian Splatting Under Attack: Investigating Adversarial Noise in 3D Objects(https://arxiv.org/abs/2412.02803)
• Keywords: security, defense, attack, robust
• Abstract: 3D Gaussian Splatting has advanced radiance field reconstruction, enabling high-quality view synthesis and fast rendering in 3D modeling. While adversarial attacks on object detection models are well-studied for 2D images, their impact on 3D models remains underexplored. This work introduces the Masked Iterative Fast Gradient Sign Method (M-IFGSM), designed to generate adversarial noise targeting the CLIP vision-language model. M-IFGSM specifically alters the object of interest by focusing perturbations on masked regions, degrading the performance of CLIP's zero-shot object detection capability when applied to 3D models. Using eight objects from the Common Objects 3D (CO3D) dataset, we demonstrate that our method effectively reduces the accuracy and confidence of the model, with adversarial noise being nearly imperceptible to human observers. The top-1 accuracy in original model renders drops from 95.4\% to 12.5\% for train images and from 91.2\% to 35.4\% for test images, with confidence levels reflecting this shift from true classification to misclassification, underscoring the risks of adversarial attacks on 3D models in applications such as autonomous driving, robotics, and surveillance. The significance of this research lies in its potential to expose vulnerabilities in modern 3D vision models, including radiance fields, prompting the development of more robust defenses and security measures in critical real-world applications.

Title: Temporally Consistent Dynamic Scene Graphs: An End-to-End Approach for Action Tracklet Generation

• Authors: Raphael Ruschel, Md Awsafur Rahman, Hardik Prajapati, Suya You, B. S. Manjuanth
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.02808
• Pdf URL: https://arxiv.org/pdf/2412.02808
• Copy Paste: [[2412.02808]] Temporally Consistent Dynamic Scene Graphs: An End-to-End Approach for Action Tracklet Generation(https://arxiv.org/abs/2412.02808)
• Keywords: robust
• Abstract: Understanding video content is pivotal for advancing real-world applications like activity recognition, autonomous systems, and human-computer interaction. While scene graphs are adept at capturing spatial relationships between objects in individual frames, extending these representations to capture dynamic interactions across video sequences remains a significant challenge. To address this, we present TCDSG, Temporally Consistent Dynamic Scene Graphs, an innovative end-to-end framework that detects, tracks, and links subject-object relationships across time, generating action tracklets, temporally consistent sequences of entities and their interactions. Our approach leverages a novel bipartite matching mechanism, enhanced by adaptive decoder queries and feedback loops, ensuring temporal coherence and robust tracking over extended sequences. This method not only establishes a new benchmark by achieving over 60% improvement in temporal recall@k on the Action Genome, OpenPVSG, and MEVA datasets but also pioneers the augmentation of MEVA with persistent object ID annotations for comprehensive tracklet generation. By seamlessly integrating spatial and temporal dynamics, our work sets a new standard in multi-frame video analysis, opening new avenues for high-impact applications in surveillance, autonomous navigation, and beyond.

Title: Unleashing GHOST: An LLM-Powered Framework for Automated Hardware Trojan Design

• Authors: Md Omar Faruque, Peter Jamieson, Ahmad Patooghy, Abdel-Hameed A. Badawy
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.02816
• Pdf URL: https://arxiv.org/pdf/2412.02816
• Copy Paste: [[2412.02816]] Unleashing GHOST: An LLM-Powered Framework for Automated Hardware Trojan Design(https://arxiv.org/abs/2412.02816)
• Keywords: security, attack, steal, large language model
• Abstract: Traditionally, inserting realistic Hardware Trojans (HTs) into complex hardware systems has been a time-consuming and manual process, requiring comprehensive knowledge of the design and navigating intricate Hardware Description Language (HDL) codebases. Machine Learning (ML)-based approaches have attempted to automate this process but often face challenges such as the need for extensive training data, long learning times, and limited generalizability across diverse hardware design landscapes. This paper addresses these challenges by proposing GHOST (Generator for Hardware-Oriented Stealthy Trojans), an automated attack framework that leverages Large Language Models (LLMs) for rapid HT generation and insertion. Our study evaluates three state-of-the-art LLMs - GPT-4, Gemini-1.5-pro, and Llama-3-70B - across three hardware designs: SRAM, AES, and UART. According to our evaluations, GPT-4 demonstrates superior performance, with 88.88% of HT insertion attempts successfully generating functional and synthesizable HTs. This study also highlights the security risks posed by LLM-generated HTs, showing that 100% of GHOST-generated synthesizable HTs evaded detection by an ML-based HT detection tool. These results underscore the urgent need for advanced detection and prevention mechanisms in hardware security to address the emerging threat of LLM-generated HTs. The GHOST HT benchmarks are available at: this https URL

Title: CNNSum: Exploring Long-Conext Summarization with Large Language Models in Chinese Novels

• Authors: Lingxiao Wei, He Yan, Xiangju Lu, Junmin Zhu, Jun Wang, Wei Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02819
• Pdf URL: https://arxiv.org/pdf/2412.02819
• Copy Paste: [[2412.02819]] CNNSum: Exploring Long-Conext Summarization with Large Language Models in Chinese Novels(https://arxiv.org/abs/2412.02819)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have been well-researched in many long-context tasks. However, due to high annotation costs, high-quality long-context summary datasets for training or evaluation are scarce, limiting further research. In this work, we introduce CNNSum, a new multi-scale Chinese long-context novel summarization benchmark, including four subsets, length covering 16k\textasciitilde128k, 695 samples in total, the annotations are human-driven. We evaluate commercial and open-source models on CNNSum and conduct a detailed analysis. Based on the observations, we further conduct fine-tuning exploration with short-context summary data. In our study: (1) GPT-4o underperformed, due to excessive subjective commentary. (2) Currently, long-context summarization mainly relies on memory ability, small LLMs with stable longer context lengths are the most cost-effective. Using long data concatenated from short-context summaries makes a significant improvement. (3) Prompt templates may cause a large performance gap but can be mitigated through fine-tuning. (4) Fine-tuned Chat or Instruction versions may harm the Base model and further fine-tuning cannot bridge performance gap. (5) while models with RoPE base scaling exhibit strong extrapolation potential, their performance may vary significantly when combined with other interpolation methods and need careful selection. (6) CNNSum provides more reliable and insightful evaluation results than other benchmarks. We release CNNSum to advance research in this field.

Title: Minimization of Boolean Complexity in In-Context Concept Learning

• Authors: Leroy Z. Wang, R. Thomas McCoy, Shane Steinert-Threlkeld
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02823
• Pdf URL: https://arxiv.org/pdf/2412.02823
• Copy Paste: [[2412.02823]] Minimization of Boolean Complexity in In-Context Concept Learning(https://arxiv.org/abs/2412.02823)
• Keywords: large language model
• Abstract: What factors contribute to the relative success and corresponding difficulties of in-context learning for Large Language Models (LLMs)? Drawing on insights from the literature on human concept learning, we test LLMs on carefully designed concept learning tasks, and show that task performance highly correlates with the Boolean complexity of the concept. This suggests that in-context learning exhibits a learning bias for simplicity in a way similar to humans.

Title: Many-MobileNet: Multi-Model Augmentation for Robust Retinal Disease Classification

• Authors: Hao Wang, Wenhui Zhu, Xuanzhao Dong, Yanxi Chen, Xin Li, Peijie Qiu, Xiwen Chen, Vamsi Krishna Vasa, Yujian Xiong, Oana M. Dumitrascu, Abolfazl Razi, Yalin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02825
• Pdf URL: https://arxiv.org/pdf/2412.02825
• Copy Paste: [[2412.02825]] Many-MobileNet: Multi-Model Augmentation for Robust Retinal Disease Classification(https://arxiv.org/abs/2412.02825)
• Keywords: robust, extraction
• Abstract: In this work, we propose Many-MobileNet, an efficient model fusion strategy for retinal disease classification using lightweight CNN architecture. Our method addresses key challenges such as overfitting and limited dataset variability by training multiple models with distinct data augmentation strategies and different model complexities. Through this fusion technique, we achieved robust generalization in data-scarce domains while balancing computational efficiency with feature extraction capabilities.

Title: RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Models

• Authors: Hieu Tran, Zonghai Yao, Junda Wang, Yifan Zhang, Zhichao Yang, Hong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.02830
• Pdf URL: https://arxiv.org/pdf/2412.02830
• Copy Paste: [[2412.02830]] RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Models(https://arxiv.org/abs/2412.02830)
• Keywords: large language model
• Abstract: This work introduces RARE (Retrieval-Augmented Reasoning Enhancement), a versatile extension to the mutual reasoning framework (rStar), aimed at enhancing reasoning accuracy and factual integrity across large language models (LLMs) for complex, knowledge-intensive tasks such as commonsense and medical reasoning. RARE incorporates two innovative actions within the Monte Carlo Tree Search (MCTS) framework: A6, which generates search queries based on the initial problem statement, performs information retrieval using those queries, and augments reasoning with the retrieved data to formulate the final answer; and A7, which leverages information retrieval specifically for generated sub-questions and re-answers these sub-questions with the relevant contextual information. Additionally, a Retrieval-Augmented Factuality Scorer is proposed to replace the original discriminator, prioritizing reasoning paths that meet high standards of factuality. Experimental results with LLaMA 3.1 show that RARE enables open-source LLMs to achieve competitive performance with top open-source models like GPT-4 and GPT-4o. This research establishes RARE as a scalable solution for improving LLMs in domains where logical coherence and factual integrity are critical.

Title: FLAME 3 Dataset: Unleashing the Power of Radiometric Thermal UAV Imagery for Wildfire Management

• Authors: Bryce Hopkins, Leo ONeill, Michael Marinaccio, Eric Rowell, Russell Parsons, Sarah Flanary, Irtija Nazim, Carl Seielstad, Fatemeh Afghah
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02831
• Pdf URL: https://arxiv.org/pdf/2412.02831
• Copy Paste: [[2412.02831]] FLAME 3 Dataset: Unleashing the Power of Radiometric Thermal UAV Imagery for Wildfire Management(https://arxiv.org/abs/2412.02831)
• Keywords: segmentation
• Abstract: The increasing accessibility of radiometric thermal imaging sensors for unmanned aerial vehicles (UAVs) offers significant potential for advancing AI-driven aerial wildfire management. Radiometric imaging provides per-pixel temperature estimates, a valuable improvement over non-radiometric data that requires irradiance measurements to be converted into visible images using RGB color palettes. Despite its benefits, this technology has been underutilized largely due to a lack of available data for researchers. This study addresses this gap by introducing methods for collecting and processing synchronized visual spectrum and radiometric thermal imagery using UAVs at prescribed fires. The included imagery processing pipeline drastically simplifies and partially automates each step from data collection to neural network input. Further, we present the FLAME 3 dataset, the first comprehensive collection of side-by-side visual spectrum and radiometric thermal imagery of wildland fires. Building on our previous FLAME 1 and FLAME 2 datasets, FLAME 3 includes radiometric thermal Tag Image File Format (TIFFs) and nadir thermal plots, providing a new data type and collection method. This dataset aims to spur a new generation of machine learning models utilizing radiometric thermal imagery, potentially trivializing tasks such as aerial wildfire detection, segmentation, and assessment. A single-burn subset of FLAME 3 for computer vision applications is available on Kaggle with the full 6 burn set available to readers upon request.

Title: Enhancing Robustness of CLIP to Common Corruptions through Bimodal Test-Time Adaptation

• Authors: Sarthak Kumar Maharana, Baoming Zhang, Leonid Karlinsky, Rogerio Feris, Yunhui Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02837
• Pdf URL: https://arxiv.org/pdf/2412.02837
• Copy Paste: [[2412.02837]] Enhancing Robustness of CLIP to Common Corruptions through Bimodal Test-Time Adaptation(https://arxiv.org/abs/2412.02837)
• Keywords: robust, extraction
• Abstract: Although open-vocabulary classification models like Contrastive Language Image Pretraining (CLIP) have demonstrated strong zero-shot learning capabilities, their robustness to common image corruptions remains poorly understood. Through extensive experiments, we show that zero-shot CLIP lacks robustness to common image corruptions at increasing severity levels during test-time, necessitating the adaptation of CLIP to unlabeled corrupted images using test-time adaptation (TTA). However, we found that existing TTA methods have severe limitations in adapting CLIP due to their unimodal nature. To address these limitations, we propose \framework, a bimodal TTA method specially designed to improve CLIP's robustness to common image corruptions. The key insight of our approach is not only to adapt the visual encoders for better image feature extraction but also to strengthen the alignment between image and text features by promoting a stronger association between the image class prototype, computed using pseudo-labels, and the corresponding text feature. We evaluate our approach on benchmark image corruption datasets and achieve state-of-the-art results in TTA for CLIP, specifically for domains involving image corruption. Particularly, with a ViT-B/16 vision backbone, we obtain mean accuracy improvements of 9.7%, 5.94%, and 5.12% for CIFAR-10C, CIFAR-100C, and ImageNet-C, respectively.

Title: Optimized IoT Intrusion Detection using Machine Learning Technique

• Authors: Muhammad Zawad Mahmud, Samiha Islam, Shahran Rahman Alve, Al Jubayer Pial
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.02845
• Pdf URL: https://arxiv.org/pdf/2412.02845
• Copy Paste: [[2412.02845]] Optimized IoT Intrusion Detection using Machine Learning Technique(https://arxiv.org/abs/2412.02845)
• Keywords: security, privacy, protect, defense, attack, extraction
• Abstract: An application of software known as an Intrusion Detection System (IDS) employs machine algorithms to identify network intrusions. Selective logging, safeguarding privacy, reputation-based defense against numerous attacks, and dynamic response to threats are a few of the problems that intrusion identification is used to solve. The biological system known as IoT has seen a rapid increase in high dimensionality and information traffic. Self-protective mechanisms like intrusion detection systems (IDSs) are essential for defending against a variety of attacks. On the other hand, the functional and physical diversity of IoT IDS systems causes significant issues. These attributes make it troublesome and unrealistic to completely use all IoT elements and properties for IDS self-security. For peculiarity-based IDS, this study proposes and implements a novel component selection and extraction strategy (our strategy). A five-ML algorithm model-based IDS for machine learning-based networks with proper hyperparamater tuning is presented in this paper by examining how the most popular feature selection methods and classifiers are combined, such as K-Nearest Neighbors (KNN) Classifier, Decision Tree (DT) Classifier, Random Forest (RF) Classifier, Gradient Boosting Classifier, and Ada Boost Classifier. The Random Forest (RF) classifier had the highest accuracy of 99.39%. The K-Nearest Neighbor (KNN) classifier exhibited the lowest performance among the evaluated models, achieving an accuracy of 94.84%. This study's models have a significantly higher performance rate than those used in previous studies, indicating that they are more reliable.

Title: Effortless Efficiency: Low-Cost Pruning of Diffusion Models

• Authors: Yang Zhang, Er Jin, Yanfei Dong, Ashkan Khakzar, Philip Torr, Johannes Stegmaier, Kenji Kawaguchi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02852
• Pdf URL: https://arxiv.org/pdf/2412.02852
• Copy Paste: [[2412.02852]] Effortless Efficiency: Low-Cost Pruning of Diffusion Models(https://arxiv.org/abs/2412.02852)
• Keywords: diffusion, transformer
• Abstract: Diffusion models have achieved impressive advancements in various vision tasks. However, these gains often rely on increasing model size, which escalates computational complexity and memory demands, complicating deployment, raising inference costs, and causing environmental impact. While some studies have explored pruning techniques to improve the memory efficiency of diffusion models, most existing methods require extensive retraining to retain the model performance. Retraining a modern large diffusion model is extremely costly and resource-intensive, which limits the practicality of these methods. In this work, we achieve low-cost diffusion pruning without retraining by proposing a model-agnostic structural pruning framework for diffusion models that learns a differentiable mask to sparsify the model. To ensure effective pruning that preserves the quality of the final denoised latent, we design a novel end-to-end pruning objective that spans the entire diffusion process. As end-to-end pruning is memory-intensive, we further propose time step gradient checkpointing, a technique that significantly reduces memory usage during optimization, enabling end-to-end pruning within a limited memory budget. Results on state-of-the-art U-Net diffusion models SDXL and diffusion transformers (FLUX) demonstrate that our method can effectively prune up to 20% parameters with minimal perceptible performance degradation, and notably, without the need for model retraining. We also showcase that our method can still prune on top of time step distilled diffusion models.

Title: Is Large-Scale Pretraining the Secret to Good Domain Generalization?

• Authors: Piotr Teterwak, Kuniaki Saito, Theodoros Tsiligkaridis, Bryan A. Plummer, Kate Saenko
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.02856
• Pdf URL: https://arxiv.org/pdf/2412.02856
• Copy Paste: [[2412.02856]] Is Large-Scale Pretraining the Secret to Good Domain Generalization?(https://arxiv.org/abs/2412.02856)
• Keywords: robust
• Abstract: Multi-Source Domain Generalization (DG) is the task of training on multiple source domains and achieving high classification performance on unseen target domains. Recent methods combine robust features from web-scale pretrained backbones with new features learned from source data, and this has dramatically improved benchmark results. However, it remains unclear if DG finetuning methods are becoming better over time, or if improved benchmark performance is simply an artifact of stronger pre-training. Prior studies have shown that perceptual similarity to pre-training data correlates with zero-shot performance, but we find the effect limited in the DG setting. Instead, we posit that having perceptually similar data in pretraining is not enough; and that it is how well these data were learned that determines performance. This leads us to introduce the Alignment Hypothesis, which states that the final DG performance will be high if and only if alignment of image and class label text embeddings is high. Our experiments confirm the Alignment Hypothesis is true, and we use it as an analysis tool of existing DG methods evaluated on DomainBed datasets by splitting evaluation data into In-pretraining (IP) and Out-of-pretraining (OOP). We show that all evaluated DG methods struggle on DomainBed-OOP, while recent methods excel on DomainBed-IP. Put together, our findings highlight the need for DG methods which can generalize beyond pretraining alignment.

Title: Measuring Bias of Web-filtered Text Datasets and Bias Propagation Through Training

• Authors: Youssef Mansour, Reinhard Heckel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.02857
• Pdf URL: https://arxiv.org/pdf/2412.02857
• Copy Paste: [[2412.02857]] Measuring Bias of Web-filtered Text Datasets and Bias Propagation Through Training(https://arxiv.org/abs/2412.02857)
• Keywords: large language model
• Abstract: We investigate biases in pretraining datasets for large language models (LLMs) through dataset classification experiments. Building on prior work demonstrating the existence of biases in popular computer vision datasets, we analyze popular open-source pretraining datasets for LLMs derived from CommonCrawl including C4, RefinedWeb, DolmaCC, RedPajama-V2, FineWeb, and DCLM-Baseline. Despite those datasets being obtained with similar filtering and deduplication steps, neural networks can classify surprisingly well which dataset a single text sequence belongs to, significantly better than a human can. This indicates that popular pretraining datasets have their own unique biases or fingerprints. Those biases remain even when the text is rewritten with LLMs. Moreover, these biases propagate through training: Random sequences generated by models trained on those datasets can be classified well by a classifier trained on the original datasets.

Title: Memory-efficient Continual Learning with Neural Collapse Contrastive

• Authors: Trung-Anh Dang, Vincent Nguyen, Ngoc-Son Vu, Christel Vrain
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.02865
• Pdf URL: https://arxiv.org/pdf/2412.02865
• Copy Paste: [[2412.02865]] Memory-efficient Continual Learning with Neural Collapse Contrastive(https://arxiv.org/abs/2412.02865)
• Keywords: privacy
• Abstract: Contrastive learning has significantly improved representation quality, enhancing knowledge transfer across tasks in continual learning (CL). However, catastrophic forgetting remains a key challenge, as contrastive based methods primarily focus on "soft relationships" or "softness" between samples, which shift with changing data distributions and lead to representation overlap across tasks. Recently, the newly identified Neural Collapse phenomenon has shown promise in CL by focusing on "hard relationships" or "hardness" between samples and fixed prototypes. However, this approach overlooks "softness", crucial for capturing intra-class variability, and this rigid focus can also pull old class representations toward current ones, increasing forgetting. Building on these insights, we propose Focal Neural Collapse Contrastive (FNC2), a novel representation learning loss that effectively balances both soft and hard relationships. Additionally, we introduce the Hardness-Softness Distillation (HSD) loss to progressively preserve the knowledge gained from these relationships across tasks. Our method outperforms state-of-the-art approaches, particularly in minimizing memory reliance. Remarkably, even without the use of memory, our approach rivals rehearsal-based methods, offering a compelling solution for data privacy concerns.

Title: MAGMA: Manifold Regularization for MAEs

• Authors: Alin Dondera, Anuj Singh, Hadi Jamali-Rad
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02871
• Pdf URL: https://arxiv.org/pdf/2412.02871
• Copy Paste: [[2412.02871]] MAGMA: Manifold Regularization for MAEs(https://arxiv.org/abs/2412.02871)
• Keywords: transformer
• Abstract: Masked Autoencoders (MAEs) are an important divide in self-supervised learning (SSL) due to their independence from augmentation techniques for generating positive (and/or negative) pairs as in contrastive frameworks. Their masking and reconstruction strategy also nicely aligns with SSL approaches in natural language processing. Most MAEs are built upon Transformer-based architectures where visual features are not regularized as opposed to their convolutional neural network (CNN) based counterparts, which can potentially hinder their performance. To address this, we introduce MAGMA, a novel batch-wide layer-wise regularization loss applied to representations of different Transformer layers. We demonstrate that by plugging in the proposed regularization loss, one can significantly improve the performance of MAE-based models. We further demonstrate the impact of the proposed loss on optimizing other generic SSL approaches (such as VICReg and SimCLR), broadening the impact of the proposed approach. Our code base can be found at this https URL.

Title: Out-of-Distribution Detection for Neurosymbolic Autonomous Cyber Agents

• Authors: Ankita Samaddar, Nicholas Potteiger, Xenofon Koutsoukos
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.02875
• Pdf URL: https://arxiv.org/pdf/2412.02875
• Copy Paste: [[2412.02875]] Out-of-Distribution Detection for Neurosymbolic Autonomous Cyber Agents(https://arxiv.org/abs/2412.02875)
• Keywords: security, defense
• Abstract: Autonomous agents for cyber applications take advantage of modern defense techniques by adopting intelligent agents with conventional and learning-enabled components. These intelligent agents are trained via reinforcement learning (RL) algorithms, and can learn, adapt to, reason about and deploy security rules to defend networked computer systems while maintaining critical operational workflows. However, the knowledge available during training about the state of the operational network and its environment may be limited. The agents should be trustworthy so that they can reliably detect situations they cannot handle, and hand them over to cyber experts. In this work, we develop an out-of-distribution (OOD) Monitoring algorithm that uses a Probabilistic Neural Network (PNN) to detect anomalous or OOD situations of RL-based agents with discrete states and discrete actions. To demonstrate the effectiveness of the proposed approach, we integrate the OOD monitoring algorithm with a neurosymbolic autonomous cyber agent that uses behavior trees with learning-enabled components. We evaluate the proposed approach in a simulated cyber environment under different adversarial strategies. Experimental results over a large number of episodes illustrate the overall efficiency of our proposed approach.

Title: Pairwise Spatiotemporal Partial Trajectory Matching for Co-movement Analysis

• Authors: Maria Cardei, Sabit Ahmed, Gretchen Chapman, Afsaneh Doryab
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02879
• Pdf URL: https://arxiv.org/pdf/2412.02879
• Copy Paste: [[2412.02879]] Pairwise Spatiotemporal Partial Trajectory Matching for Co-movement Analysis(https://arxiv.org/abs/2412.02879)
• Keywords: interpretability
• Abstract: Spatiotemporal pairwise movement analysis involves identifying shared geographic-based behaviors between individuals within specific time frames. Traditionally, this task relies on sequence modeling and behavior analysis techniques applied to tabular or video-based data, but these methods often lack interpretability and struggle to capture partial matching. In this paper, we propose a novel method for pairwise spatiotemporal partial trajectory matching that transforms tabular spatiotemporal data into interpretable trajectory images based on specified time windows, allowing for partial trajectory analysis. This approach includes localization of trajectories, checking for spatial overlap, and pairwise matching using a Siamese Neural Network. We evaluate our method on a co-walking classification task, demonstrating its effectiveness in a novel co-behavior identification application. Our model surpasses established methods, achieving an F1-score up to 0.73. Additionally, we explore the method's utility for pair routine pattern analysis in real-world scenarios, providing insights into the frequency, timing, and duration of shared behaviors. This approach offers a powerful, interpretable framework for spatiotemporal behavior analysis, with potential applications in social behavior research, urban planning, and healthcare.

Title: Patchfinder: Leveraging Visual Language Models for Accurate Information Retrieval using Model Uncertainty

• Authors: Roman Colman, Minh Vu, Manish Bhattarai, Martin Ma, Hari Viswanathan, Daniel O'Malley, Javier E. Santos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02886
• Pdf URL: https://arxiv.org/pdf/2412.02886
• Copy Paste: [[2412.02886]] Patchfinder: Leveraging Visual Language Models for Accurate Information Retrieval using Model Uncertainty(https://arxiv.org/abs/2412.02886)
• Keywords: extraction, large language model
• Abstract: For decades, corporations and governments have relied on scanned documents to record vast amounts of information. However, extracting this information is a slow and tedious process due to the overwhelming amount of documents. The rise of vision language models presents a way to efficiently and accurately extract the information out of these documents. The current automated workflow often requires a two-step approach involving the extraction of information using optical character recognition software, and subsequent usage of large language models for processing this information. Unfortunately, these methods encounter significant challenges when dealing with noisy scanned documents. The high information density of such documents often necessitates using computationally expensive language models to effectively reduce noise. In this study, we propose PatchFinder, an algorithm that builds upon Vision Language Models (VLMs) to address the information extraction task. First, we devise a confidence-based score, called Patch Confidence, based on the Maximum Softmax Probability of the VLMs' output to measure the model's confidence in its predictions. Then, PatchFinder utilizes that score to determine a suitable patch size, partition the input document into overlapping patches of that size, and generate confidence-based predictions for the target information. Our experimental results show that PatchFinder can leverage Phi-3v, a 4.2 billion parameter vision language model, to achieve an accuracy of 94% on our dataset of 190 noisy scanned documents, surpassing the performance of ChatGPT-4o by 18.5 percentage points.

Title: EvRT-DETR: The Surprising Effectiveness of DETR-based Detection for Event Cameras

• Authors: Dmitrii Torbunov, Yihui Ren, Animesh Ghose, Odera Dim, Yonggang Cui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02890
• Pdf URL: https://arxiv.org/pdf/2412.02890
• Copy Paste: [[2412.02890]] EvRT-DETR: The Surprising Effectiveness of DETR-based Detection for Event Cameras(https://arxiv.org/abs/2412.02890)
• Keywords: transformer
• Abstract: Event-based cameras (EBCs) have emerged as a bio-inspired alternative to traditional cameras, offering advantages in power efficiency, temporal resolution, and high dynamic range. However, the development of image analysis methods for EBCs is challenging due to the sparse and asynchronous nature of the data. This work addresses the problem of object detection for the EBC cameras. The current approaches to EBC object detection focus on constructing complex data representations and rely on specialized architectures. Here, we demonstrate that the combination of a Real-Time DEtection TRansformer, or RT-DETR, a state-of-the-art natural image detector, with a simple image-like representation of the EBC data achieves remarkable performance, surpassing current state-of-the-art results. Specifically, we show that a properly trained RT-DETR model on the EBC data achieves performance comparable to the most advanced EBC object detection methods. Next, we propose a low-rank adaptation (LoRA)-inspired way to augment the RT-DETR model to handle temporal dynamics of the data. The designed EvRT-DETR model outperforms the current, most advanced results on standard benchmark datasets Gen1 (mAP $+2.3$) and Gen4 (mAP $+1.4$) while only using standard modules from natural image and video analysis. These results demonstrate that effective EBC object detection can be achieved through careful adaptation of mainstream object detection architectures without requiring specialized architectural engineering. The code is available at: this https URL

Title: Removing Spurious Correlation from Neural Network Interpretations

• Authors: Milad Fotouhi, Mohammad Taha Bahadori, Oluwaseyi Feyisetan, Payman Arabshahi, David Heckerman
• Subjects: cs.CL, cs.AI, cs.LG, stat.AP, stat.ME
• Abstract URL: https://arxiv.org/abs/2412.02893
• Pdf URL: https://arxiv.org/pdf/2412.02893
• Copy Paste: [[2412.02893]] Removing Spurious Correlation from Neural Network Interpretations(https://arxiv.org/abs/2412.02893)
• Keywords: large language model
• Abstract: The existing algorithms for identification of neurons responsible for undesired and harmful behaviors do not consider the effects of confounders such as topic of the conversation. In this work, we show that confounders can create spurious correlations and propose a new causal mediation approach that controls the impact of the topic. In experiments with two large language models, we study the localization hypothesis and show that adjusting for the effect of conversation topic, toxicity becomes less localized.

Title: GUESS: Generative Uncertainty Ensemble for Self Supervision

• Authors: Salman Mohamadi, Gianfranco Doretto, Donald A. Adjeroh
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02896
• Pdf URL: https://arxiv.org/pdf/2412.02896
• Copy Paste: [[2412.02896]] GUESS: Generative Uncertainty Ensemble for Self Supervision(https://arxiv.org/abs/2412.02896)
• Keywords: robust, fair, generative
• Abstract: Self-supervised learning (SSL) frameworks consist of pretext task, and loss function aiming to learn useful general features from unlabeled data. The basic idea of most SSL baselines revolves around enforcing the invariance to a variety of data augmentations via the loss function. However, one main issue is that, inattentive or deterministic enforcement of the invariance to any kind of data augmentation is generally not only inefficient, but also potentially detrimental to performance on the downstream tasks. In this work, we investigate the issue from the viewpoint of uncertainty in invariance representation. Uncertainty representation is fairly under-explored in the design of SSL architectures as well as loss functions. We incorporate uncertainty representation in both loss function as well as architecture design aiming for more data-dependent invariance enforcement. The former is represented in the form of data-derived uncertainty in SSL loss function resulting in a generative-discriminative loss function. The latter is achieved by feeding slightly different distorted versions of samples to the ensemble aiming for learning better and more robust representation. Specifically, building upon the recent methods that use hard and soft whitening (a.k.a redundancy reduction), we introduce a new approach GUESS, a pseudo-whitening framework, composed of controlled uncertainty injection, a new architecture, and a new loss function. We include detailed results and ablation analysis establishing GUESS as a new baseline.

Title: MLD-EA: Check and Complete Narrative Coherence by Introducing Emotions and Actions

• Authors: Jinming Zhang, Yunfei Long
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02897
• Pdf URL: https://arxiv.org/pdf/2412.02897
• Copy Paste: [[2412.02897]] MLD-EA: Check and Complete Narrative Coherence by Introducing Emotions and Actions(https://arxiv.org/abs/2412.02897)
• Keywords: large language model
• Abstract: Narrative understanding and story generation are critical challenges in natural language processing (NLP), with much of the existing research focused on summarization and question-answering tasks. While previous studies have explored predicting plot endings and generating extended narratives, they often neglect the logical coherence within stories, leaving a significant gap in the field. To address this, we introduce the Missing Logic Detector by Emotion and Action (MLD-EA) model, which leverages large language models (LLMs) to identify narrative gaps and generate coherent sentences that integrate seamlessly with the story's emotional and logical flow. The experimental results demonstrate that the MLD-EA model enhances narrative understanding and story generation, highlighting LLMs' potential as effective logic checkers in story writing with logical coherence and emotional consistency. This work fills a gap in NLP research and advances border goals of creating more sophisticated and reliable story-generation systems.

Title: Enhancing Trust in Large Language Models with Uncertainty-Aware Fine-Tuning

• Authors: Ranganath Krishnan, Piyush Khanna, Omesh Tickoo
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.02904
• Pdf URL: https://arxiv.org/pdf/2412.02904
• Copy Paste: [[2412.02904]] Enhancing Trust in Large Language Models with Uncertainty-Aware Fine-Tuning(https://arxiv.org/abs/2412.02904)
• Keywords: large language model
• Abstract: Large language models (LLMs) have revolutionized the field of natural language processing with their impressive reasoning and question-answering capabilities. However, these models are sometimes prone to generating credible-sounding but incorrect information, a phenomenon known as LLM hallucinations. Reliable uncertainty estimation in LLMs is essential for fostering trust in their generated responses and serves as a critical tool for the detection and prevention of erroneous or hallucinated outputs. To achieve reliable and well-calibrated uncertainty quantification in open-ended and free-form natural language generation, we propose an uncertainty-aware fine-tuning approach for LLMs. This approach enhances the model's ability to provide reliable uncertainty estimates without compromising accuracy, thereby guiding them to produce more trustworthy responses. We introduce a novel uncertainty-aware causal language modeling loss function, grounded in the principles of decision theory. Through rigorous evaluation on multiple free-form question-answering datasets and models, we demonstrate that our uncertainty-aware fine-tuning approach yields better calibrated uncertainty estimates in natural language generation tasks than fine-tuning with the standard causal language modeling loss. Furthermore, the experimental results show that the proposed method significantly improves the model's ability to detect hallucinations and identify out-of-domain prompts.

Title: Single-Cell Omics Arena: A Benchmark Study for Large Language Models on Cell Type Annotation Using Single-Cell Data

• Authors: Junhao Liu, Siwei Xu, Lei Zhang, Jing Zhang
• Subjects: cs.CL, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2412.02915
• Pdf URL: https://arxiv.org/pdf/2412.02915
• Copy Paste: [[2412.02915]] Single-Cell Omics Arena: A Benchmark Study for Large Language Models on Cell Type Annotation Using Single-Cell Data(https://arxiv.org/abs/2412.02915)
• Keywords: robust, large language model
• Abstract: Over the past decade, the revolution in single-cell sequencing has enabled the simultaneous molecular profiling of various modalities across thousands of individual cells, allowing scientists to investigate the diverse functions of complex tissues and uncover underlying disease mechanisms. Among all the analytical steps, assigning individual cells to specific types is fundamental for understanding cellular heterogeneity. However, this process is usually labor-intensive and requires extensive expert knowledge. Recent advances in large language models (LLMs) have demonstrated their ability to efficiently process and synthesize vast corpora of text to automatically extract essential biological knowledge, such as marker genes, potentially promoting more efficient and automated cell type annotations. To thoroughly evaluate the capability of modern instruction-tuned LLMs in automating the cell type identification process, we introduce SOAR, a comprehensive benchmarking study of LLMs for cell type annotation tasks in single-cell genomics. Specifically, we assess the performance of 8 instruction-tuned LLMs across 11 datasets, spanning multiple cell types and species. Our study explores the potential of LLMs to accurately classify and annotate cell types in single-cell RNA sequencing (scRNA-seq) data, while extending their application to multiomics data through cross-modality translation. Additionally, we evaluate the effectiveness of chain-of-thought (CoT) prompting techniques in generating detailed biological insights during the annotation process. The results demonstrate that LLMs can provide robust interpretations of single-cell data without requiring additional fine-tuning, advancing the automation of cell type annotation in genomics research.

Title: Higher Order Transformers: Efficient Attention Mechanism for Tensor Structured Data

• Authors: Soroush Omranpour, Guillaume Rabusseau, Reihaneh Rabbany
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02919
• Pdf URL: https://arxiv.org/pdf/2412.02919
• Copy Paste: [[2412.02919]] Higher Order Transformers: Efficient Attention Mechanism for Tensor Structured Data(https://arxiv.org/abs/2412.02919)
• Keywords: transformer
• Abstract: Transformers are now ubiquitous for sequence modeling tasks, but their extension to multi-dimensional data remains a challenge due to the quadratic cost of the attention mechanism. In this paper, we propose Higher-Order Transformers (HOT), a novel architecture designed to efficiently process data with more than two axes, i.e. higher-order tensors. To address the computational challenges associated with high-order tensor attention, we introduce a novel Kronecker factorized attention mechanism that reduces the attention cost to quadratic in each axis' dimension, rather than quadratic in the total size of the input tensor. To further enhance efficiency, HOT leverages kernelized attention, reducing the complexity to linear. This strategy maintains the model's expressiveness while enabling scalable attention computation. We validate the effectiveness of HOT on two high-dimensional tasks, including multivariate time series forecasting, and 3D medical image classification. Experimental results demonstrate that HOT achieves competitive performance while significantly improving computational efficiency, showcasing its potential for tackling a wide range of complex, multi-dimensional data.

Title: Panoptic Diffusion Models: co-generation of images and segmentation maps

• Authors: Yinghan Long, Kaushik Roy
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02929
• Pdf URL: https://arxiv.org/pdf/2412.02929
• Copy Paste: [[2412.02929]] Panoptic Diffusion Models: co-generation of images and segmentation maps(https://arxiv.org/abs/2412.02929)
• Keywords: diffusion, transformer, segmentation
• Abstract: Recently, diffusion models have demonstrated impressive capabilities in text-guided and image-conditioned image generation. However, existing diffusion models cannot simultaneously generate a segmentation map of objects and a corresponding image from the prompt. Previous attempts either generate segmentation maps based on the images or provide maps as input conditions to control image generation, limiting their functionality to given inputs. Incorporating an inherent understanding of the scene layouts can improve the creativity and realism of diffusion models. To address this limitation, we present Panoptic Diffusion Model (PDM), the first model designed to generate both images and panoptic segmentation maps concurrently. PDM bridges the gap between image and text by constructing segmentation layouts that provide detailed, built-in guidance throughout the generation process. This ensures the inclusion of categories mentioned in text prompts and enriches the diversity of segments within the background. We demonstrate the effectiveness of PDM across two architectures: a unified diffusion transformer and a two-stream transformer with a pretrained backbone. To facilitate co-generation with fewer sampling steps, we incorporate a fast diffusion solver into PDM. Additionally, when ground-truth maps are available, PDM can function as a text-guided image-to-image generation model. Finally, we propose a novel metric for evaluating the quality of generated maps and show that PDM achieves state-of-the-art results in image generation with implicit scene control.

Title: Video LLMs for Temporal Reasoning in Long Videos

• Authors: Fawad Javed Fateh, Umer Ahmed, Hamza Khan, M. Zeeshan Zia, Quoc-Huy Tran
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02930
• Pdf URL: https://arxiv.org/pdf/2412.02930
• Copy Paste: [[2412.02930]] Video LLMs for Temporal Reasoning in Long Videos(https://arxiv.org/abs/2412.02930)
• Keywords: large language model, segmentation
• Abstract: This paper introduces TemporalVLM, a video large language model capable of effective temporal reasoning and fine-grained understanding in long videos. At the core, our approach includes a visual encoder for mapping a long-term input video into features which are time-aware and contain both local and global cues. In particular, it first divides the input video into short-term clips, which are jointly encoded with their timestamps into time-sensitive local features. Next, the local features are passed through a bidirectional long short-term memory module for global feature aggregation. The extracted time-aware and multi-level features are important for accurate temporal reasoning and fine-grained understanding in long videos. Moreover, to facilitate the evaluation of TemporalVLM, we present a large-scale long video dataset of industry assembly processes, namely IndustryASM, which consists of videos recorded on factory floors with actions and timestamps annotated by industrial engineers for time and motion studies and temporal action segmentation evaluation. Finally, extensive experiments on datasets of long videos, including TimeIT and IndustryASM, show that TemporalVLM achieves superior performance than previous methods across temporal reasoning and fine-grained understanding tasks, namely dense video captioning, temporal video grounding, video highlight detection, and temporal action segmentation.

Title: BGTplanner: Maximizing Training Accuracy for Differentially Private Federated Recommenders via Strategic Privacy Budget Allocation

• Authors: Xianzhi Zhang, Yipeng Zhou, Miao Hu, Di Wu, Pengshan Liao, Mohsen Guizani, Michael Sheng
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.02934
• Pdf URL: https://arxiv.org/pdf/2412.02934
• Copy Paste: [[2412.02934]] BGTplanner: Maximizing Training Accuracy for Differentially Private Federated Recommenders via Strategic Privacy Budget Allocation(https://arxiv.org/abs/2412.02934)
• Keywords: privacy, federate
• Abstract: To mitigate the rising concern about privacy leakage, the federated recommender (FR) paradigm emerges, in which decentralized clients co-train the recommendation model without exposing their raw user-item rating data. The differentially private federated recommender (DPFR) further enhances FR by injecting differentially private (DP) noises into clients. Yet, current DPFRs, suffering from noise distortion, cannot achieve satisfactory accuracy. Various efforts have been dedicated to improving DPFRs by adaptively allocating the privacy budget over the learning process. However, due to the intricate relation between privacy budget allocation and model accuracy, existing works are still far from maximizing DPFR accuracy. To address this challenge, we develop BGTplanner (Budget Planner) to strategically allocate the privacy budget for each round of DPFR training, improving overall training performance. Specifically, we leverage the Gaussian process regression and historical information to predict the change in recommendation accuracy with a certain allocated privacy budget. Additionally, Contextual Multi-Armed Bandit (CMAB) is harnessed to make privacy budget allocation decisions by reconciling the current improvement and long-term privacy constraints. Our extensive experimental results on real datasets demonstrate that \emph{BGTplanner} achieves an average improvement of 6.76\% in training performance compared to state-of-the-art baselines.

Title: Curriculum-style Data Augmentation for LLM-based Metaphor Detection

• Authors: Kaidi Jia, Yanxia Wu, Rongsheng Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.02956
• Pdf URL: https://arxiv.org/pdf/2412.02956
• Copy Paste: [[2412.02956]] Curriculum-style Data Augmentation for LLM-based Metaphor Detection(https://arxiv.org/abs/2412.02956)
• Keywords: large language model
• Abstract: Recently, utilizing large language models (LLMs) for metaphor detection has achieved promising results. However, these methods heavily rely on the capabilities of closed-source LLMs, which come with relatively high inference costs and latency. To address this, we propose a method for metaphor detection by fine-tuning open-source LLMs, effectively reducing inference costs and latency with a single inference step. Furthermore, metaphor detection suffers from a severe data scarcity problem, which hinders effective fine-tuning of LLMs. To tackle this, we introduce Curriculum-style Data Augmentation (CDA). Specifically, before fine-tuning, we evaluate the training data to identify correctly predicted instances for fine-tuning, while incorrectly predicted instances are used as seed data for data augmentation. This approach enables the model to quickly learn simpler knowledge and progressively acquire more complex knowledge, thereby improving performance incrementally. Experimental results demonstrate that our method achieves state-of-the-art performance across all baselines. Additionally, we provide detailed ablation studies to validate the effectiveness of CDA.

Title: Semantic Segmentation Prior for Diffusion-Based Real-World Super-Resolution

• Authors: Jiahua Xiao, Jiawei Zhang, Dongqing Zou, Xiaodan Zhang, Jimmy Ren, Xing Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02960
• Pdf URL: https://arxiv.org/pdf/2412.02960
• Copy Paste: [[2412.02960]] Semantic Segmentation Prior for Diffusion-Based Real-World Super-Resolution(https://arxiv.org/abs/2412.02960)
• Keywords: diffusion, segmentation
• Abstract: Real-world image super-resolution (Real-ISR) has achieved a remarkable leap by leveraging large-scale text-to-image models, enabling realistic image restoration from given recognition textual prompts. However, these methods sometimes fail to recognize some salient objects, resulting in inaccurate semantic restoration in these regions. Additionally, the same region may have a strong response to more than one prompt and it will lead to semantic ambiguity for image super-resolution. To alleviate the above two issues, in this paper, we propose to consider semantic segmentation as an additional control condition into diffusion-based image super-resolution. Compared to textual prompt conditions, semantic segmentation enables a more comprehensive perception of salient objects within an image by assigning class labels to each pixel. It also mitigates the risks of semantic ambiguities by explicitly allocating objects to their respective spatial regions. In practice, inspired by the fact that image super-resolution and segmentation can benefit each other, we propose SegSR which introduces a dual-diffusion framework to facilitate interaction between the image super-resolution and segmentation diffusion models. Specifically, we develop a Dual-Modality Bridge module to enable updated information flow between these two diffusion models, achieving mutual benefit during the reverse diffusion process. Extensive experiments show that SegSR can generate realistic images while preserving semantic structures more effectively.

Title: Partially Conditioned Patch Parallelism for Accelerated Diffusion Model Inference

• Authors: XiuYu Zhang, Zening Luo, Michelle E. Lu
• Subjects: cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.02962
• Pdf URL: https://arxiv.org/pdf/2412.02962
• Copy Paste: [[2412.02962]] Partially Conditioned Patch Parallelism for Accelerated Diffusion Model Inference(https://arxiv.org/abs/2412.02962)
• Keywords: diffusion
• Abstract: Diffusion models have exhibited exciting capabilities in generating images and are also very promising for video creation. However, the inference speed of diffusion models is limited by the slow sampling process, restricting its use cases. The sequential denoising steps required for generating a single sample could take tens or hundreds of iterations and thus have become a significant bottleneck. This limitation is more salient for applications that are interactive in nature or require small latency. To address this challenge, we propose Partially Conditioned Patch Parallelism (PCPP) to accelerate the inference of high-resolution diffusion models. Using the fact that the difference between the images in adjacent diffusion steps is nearly zero, Patch Parallelism (PP) leverages multiple GPUs communicating asynchronously to compute patches of an image in multiple computing devices based on the entire image (all patches) in the previous diffusion step. PCPP develops PP to reduce computation in inference by conditioning only on parts of the neighboring patches in each diffusion step, which also decreases communication among computing devices. As a result, PCPP decreases the communication cost by around $70\%$ compared to DistriFusion (the state of the art implementation of PP) and achieves $2.36\sim 8.02\times$ inference speed-up using $4\sim 8$ GPUs compared to $2.32\sim 6.71\times$ achieved by DistriFusion depending on the computing device configuration and resolution of generation at the cost of a possible decrease in image quality. PCPP demonstrates the potential to strike a favorable trade-off, enabling high-quality image generation with substantially reduced latency.

Title: Theoretical limitations of multi-layer Transformer

• Authors: Lijie Chen, Binghui Peng, Hongxun Wu
• Subjects: cs.LG, cs.AI, cs.CC, cs.DS
• Abstract URL: https://arxiv.org/abs/2412.02975
• Pdf URL: https://arxiv.org/pdf/2412.02975
• Copy Paste: [[2412.02975]] Theoretical limitations of multi-layer Transformer(https://arxiv.org/abs/2412.02975)
• Keywords: transformer, large language model
• Abstract: Transformers, especially the decoder-only variants, are the backbone of most modern large language models; yet we do not have much understanding of their expressive power except for the simple $1$-layer case. Due to the difficulty of analyzing multi-layer models, all previous work relies on unproven complexity conjectures to show limitations for multi-layer Transformers. In this work, we prove the first $\textit{unconditional}$ lower bound against multi-layer decoder-only transformers. For any constant $L$, we prove that any $L$-layer decoder-only transformer needs a polynomial model dimension ($n^{\Omega(1)}$) to perform sequential composition of $L$ functions over an input of $n$ tokens. As a consequence, our results give: (1) the first depth-width trade-off for multi-layer transformers, exhibiting that the $L$-step composition task is exponentially harder for $L$-layer models compared to $(L+1)$-layer ones; (2) an unconditional separation between encoder and decoder, exhibiting a hard task for decoders that can be solved by an exponentially shallower and smaller encoder; (3) a provable advantage of chain-of-thought, exhibiting a task that becomes exponentially easier with chain-of-thought. On the technical side, we propose the multi-party $\textit{autoregressive}$ $\textit{communication}$ $\textit{model}$ that captures the computation of a decoder-only Transformer. We also introduce a new proof technique that finds a certain $\textit{indistinguishable}$ $\textit{decomposition}$ of all possible inputs iteratively for proving lower bounds in this model. We believe our new communication model and proof technique will be helpful to further understand the computational power of transformers.

Title: Progressive Vision-Language Prompt for Multi-Organ Multi-Class Cell Semantic Segmentation with Single Branch

• Authors: Qing Zhang, Hang Guo, Siyuan Yang, Qingli Li, Yan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02978
• Pdf URL: https://arxiv.org/pdf/2412.02978
• Copy Paste: [[2412.02978]] Progressive Vision-Language Prompt for Multi-Organ Multi-Class Cell Semantic Segmentation with Single Branch(https://arxiv.org/abs/2412.02978)
• Keywords: extraction, segmentation
• Abstract: Pathological cell semantic segmentation is a fundamental technology in computational pathology, essential for applications like cancer diagnosis and effective treatment. Given that multiple cell types exist across various organs, with subtle differences in cell size and shape, multi-organ, multi-class cell segmentation is particularly challenging. Most existing methods employ multi-branch frameworks to enhance feature extraction, but often result in complex architectures. Moreover, reliance on visual information limits performance in multi-class analysis due to intricate textural details. To address these challenges, we propose a Multi-OrgaN multi-Class cell semantic segmentation method with a single brancH (MONCH) that leverages vision-language input. Specifically, we design a hierarchical feature extraction mechanism to provide coarse-to-fine-grained features for segmenting cells of various shapes, including high-frequency, convolutional, and topological features. Inspired by the synergy of textual and multi-grained visual features, we introduce a progressive prompt decoder to harmonize multimodal information, integrating features from fine to coarse granularity for better context capture. Extensive experiments on the PanNuke dataset, which has significant class imbalance and subtle cell size and shape variations, demonstrate that MONCH outperforms state-of-the-art cell segmentation methods and vision-language models. Codes and implementations will be made publicly available.

Title: Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models

• Authors: Alex Havrilla, Andrew Dai, Laura O'Mahony, Koen Oostermeijer, Vera Zisler, Alon Albalak, Fabrizio Milo, Sharath Chandra Raparthy, Kanishk Gandhi, Baber Abbasi, Duy Phung, Maia Iyer, Dakota Mahan, Chase Blagden, Srishti Gureja, Mohammed Hamdy, Wen-Ding Li, Giovanni Paolini, Pawan Sasanka Ammanamanchi, Elliot Meyerson
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.02980
• Pdf URL: https://arxiv.org/pdf/2412.02980
• Copy Paste: [[2412.02980]] Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models(https://arxiv.org/abs/2412.02980)
• Keywords: large language model
• Abstract: Synthetic data generation with Large Language Models is a promising paradigm for augmenting natural data over a nearly infinite range of tasks. Given this variety, direct comparisons among synthetic data generation algorithms are scarce, making it difficult to understand where improvement comes from and what bottlenecks exist. We propose to evaluate algorithms via the makeup of synthetic data generated by each algorithm in terms of data quality, diversity, and complexity. We choose these three characteristics for their significance in open-ended processes and the impact each has on the capabilities of downstream models. We find quality to be essential for in-distribution model generalization, diversity to be essential for out-of-distribution generalization, and complexity to be beneficial for both. Further, we emphasize the existence of Quality-Diversity trade-offs in training data and the downstream effects on model performance. We then examine the effect of various components in the synthetic data pipeline on each data characteristic. This examination allows us to taxonomize and compare synthetic data generation algorithms through the components they utilize and the resulting effects on data QDC composition. This analysis extends into a discussion on the importance of balancing QDC in synthetic data for efficient reinforcement learning and self-improvement algorithms. Analogous to the QD trade-offs in training data, often there exist trade-offs between model output quality and output diversity which impact the composition of synthetic data. We observe that many models are currently evaluated and optimized only for output quality, thereby limiting output diversity and the potential for self-improvement. We argue that balancing these trade-offs is essential to the development of future self-improvement algorithms and highlight a number of works making progress in this direction.

Title: Is Foreground Prototype Sufficient? Few-Shot Medical Image Segmentation with Background-Fused Prototype

• Authors: Song Tang, Chunxiao Zu, Wenxin Su, Yuan Dong, Mao Ye, Yan Gan, Xiatian Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02983
• Pdf URL: https://arxiv.org/pdf/2412.02983
• Copy Paste: [[2412.02983]] Is Foreground Prototype Sufficient? Few-Shot Medical Image Segmentation with Background-Fused Prototype(https://arxiv.org/abs/2412.02983)
• Keywords: segmentation
• Abstract: Few-shot Semantic Segmentation(FSS)aim to adapt a pre-trained model to new classes with as few as a single labeled training sample per class. The existing prototypical work used in natural image scenarios biasedly focus on capturing foreground's discrimination while employing a simplistic representation for background, grounded on the inherent observation separation between foreground and background. However, this paradigm is not applicable to medical images where the foreground and background share numerous visual features, necessitating a more detailed description for background. In this paper, we present a new pluggable Background-fused prototype(Bro)approach for FSS in medical images. Instead of finding a commonality of background subjects in support image, Bro incorporates this background with two pivot designs. Specifically, Feature Similarity Calibration(FeaC)initially reduces noise in the support image by employing feature cross-attention with the query image. Subsequently, Hierarchical Channel Adversarial Attention(HiCA)merges the background into comprehensive prototypes. We achieve this by a channel groups-based attention mechanism, where an adversarial Mean-Offset structure encourages a coarse-to-fine fusion. Extensive experiments show that previous state-of-the-art methods, when paired with Bro, experience significant performance improvements. This demonstrates a more integrated way to represent backgrounds specifically for medical image.

Title: Advancing Conversational Psychotherapy: Integrating Privacy, Dual-Memory, and Domain Expertise with Large Language Models

• Authors: XiuYu Zhang, Zening Luo
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.02987
• Pdf URL: https://arxiv.org/pdf/2412.02987
• Copy Paste: [[2412.02987]] Advancing Conversational Psychotherapy: Integrating Privacy, Dual-Memory, and Domain Expertise with Large Language Models(https://arxiv.org/abs/2412.02987)
• Keywords: privacy, robust, large language model
• Abstract: Mental health has increasingly become a global issue that reveals the limitations of traditional conversational psychotherapy, constrained by location, time, expense, and privacy concerns. In response to these challenges, we introduce SoulSpeak, a Large Language Model (LLM)-enabled chatbot designed to democratize access to psychotherapy. SoulSpeak improves upon the capabilities of standard LLM-enabled chatbots by incorporating a novel dual-memory component that combines short-term and long-term context via Retrieval Augmented Generation (RAG) to offer personalized responses while ensuring the preservation of user privacy and intimacy through a dedicated privacy module. In addition, it leverages a counseling chat dataset of therapist-client interactions and various prompting techniques to align the generated responses with psychotherapeutic methods. We introduce two fine-tuned BERT models to evaluate the system against existing LLMs and human therapists: the Conversational Psychotherapy Preference Model (CPPM) to simulate human preference among responses and another to assess response relevance to user input. CPPM is useful for training and evaluating psychotherapy-focused language models independent from SoulSpeak, helping with the constrained resources available for psychotherapy. Furthermore, the effectiveness of the dual-memory component and the robustness of the privacy module are also examined. Our findings highlight the potential and challenge of enhancing mental health care by offering an alternative that combines the expertise of traditional therapy with the advantages of LLMs, providing a promising way to address the accessibility and personalization gap in current mental health services.

Title: EchoONE: Segmenting Multiple echocardiography Planes in One Model

• Authors: Jiongtong Hu, Wei Zhuo, Jun Cheng, Yingying Liu, Wufeng Xue, Dong Ni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.02993
• Pdf URL: https://arxiv.org/pdf/2412.02993
• Copy Paste: [[2412.02993]] EchoONE: Segmenting Multiple echocardiography Planes in One Model(https://arxiv.org/abs/2412.02993)
• Keywords: segmentation
• Abstract: In clinical practice of echocardiography examinations, multiple planes containing the heart structures of different view are usually required in screening, diagnosis and treatment of cardiac disease. AI models for echocardiography have to be tailored for each specific plane due to the dramatic structure differences, thus resulting in repetition development and extra complexity. Effective solution for such a multi-plane segmentation (MPS) problem is highly demanded for medical images, yet has not been well investigated. In this paper, we propose a novel solution, EchoONE, for this problem with a SAM-based segmentation architecture, a prior-composable mask learning (PC-Mask) module for semantic-aware dense prompt generation, and a learnable CNN-branch with a simple yet effective local feature fusion and adaption (LFFA) module for SAM adapting. We extensively evaluated our method on multiple internal and external echocardiography datasets, and achieved consistently state-of-the-art performance for multi-source datasets with different heart planes. This is the first time that the MPS problem is solved in one model for echocardiography data. The code will be available at this https URL.

Title: CLAS: A Machine Learning Enhanced Framework for Exploring Large 3D Design Datasets

• Authors: XiuYu Zhang, Xiaolei Ye, Jui-Che Chang, Yue Fang
• Subjects: cs.CV, cs.HC, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.02996
• Pdf URL: https://arxiv.org/pdf/2412.02996
• Copy Paste: [[2412.02996]] CLAS: A Machine Learning Enhanced Framework for Exploring Large 3D Design Datasets(https://arxiv.org/abs/2412.02996)
• Keywords: generative
• Abstract: Three-dimensional (3D) objects have wide applications. Despite the growing interest in 3D modeling in academia and industries, designing and/or creating 3D objects from scratch remains time-consuming and challenging. With the development of generative artificial intelligence (AI), designers discover a new way to create images for ideation. However, generative AIs are less useful in creating 3D objects with satisfying qualities. To allow 3D designers to access a wide range of 3D objects for creative activities based on their specific demands, we propose a machine learning (ML) enhanced framework CLAS - named after the four-step of capture, label, associate, and search - to enable fully automatic retrieval of 3D objects based on user specifications leveraging the existing datasets of 3D objects. CLAS provides an effective and efficient method for any person or organization to benefit from their existing but not utilized 3D datasets. In addition, CLAS may also be used to produce high-quality 3D object synthesis datasets for training and evaluating 3D generative models. As a proof of concept, we created and showcased a search system with a web user interface (UI) for retrieving 6,778 3D objects of chairs in the ShapeNet dataset powered by CLAS. In a close-set retrieval setting, our retrieval method achieves a mean reciprocal rank (MRR) of 0.58, top 1 accuracy of 42.27%, and top 10 accuracy of 89.64%.

Title: AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?

• Authors: Shouwei Ruan, Hanqin Liu, Yao Huang, Xiaoqi Wang, Caixin Kang, Hang Su, Yinpeng Dong, Xingxing Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03002
• Pdf URL: https://arxiv.org/pdf/2412.03002
• Copy Paste: [[2412.03002]] AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations?(https://arxiv.org/abs/2412.03002)
• Keywords: robust, generative
• Abstract: Vision Language Models (VLMs) have exhibited remarkable generalization capabilities, yet their robustness in dynamic real-world scenarios remains largely unexplored. To systematically evaluate VLMs' robustness to real-world 3D variations, we propose AdvDreamer, the first framework that generates physically reproducible adversarial 3D transformation (Adv-3DT) samples from single-view images. AdvDreamer integrates advanced generative techniques with two key innovations and aims to characterize the worst-case distributions of 3D variations from natural images. To ensure adversarial effectiveness and method generality, we introduce an Inverse Semantic Probability Objective that executes adversarial optimization on fundamental vision-text alignment spaces, which can be generalizable across different VLM architectures and downstream tasks. To mitigate the distribution discrepancy between generated and real-world samples while maintaining physical reproducibility, we design a Naturalness Reward Model that provides regularization feedback during adversarial optimization, preventing convergence towards hallucinated and unnatural elements. Leveraging AdvDreamer, we establish MM3DTBench, the first VQA dataset for benchmarking VLMs' 3D variations robustness. Extensive evaluations on representative VLMs with diverse architectures highlight that 3D variations in the real world may pose severe threats to model performance across various tasks.

Title: Data Acquisition for Improving Model Fairness using Reinforcement Learning

• Authors: Jahid Hasan, Romila Pradhan
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.03009
• Pdf URL: https://arxiv.org/pdf/2412.03009
• Copy Paste: [[2412.03009]] Data Acquisition for Improving Model Fairness using Reinforcement Learning(https://arxiv.org/abs/2412.03009)
• Keywords: fair
• Abstract: Machine learning systems are increasingly being used in critical decision making such as healthcare, finance, and criminal justice. Concerns around their fairness have resulted in several bias mitigation techniques that emphasize the need for high-quality data to ensure fairer decisions. However, the role of earlier stages of machine learning pipelines in mitigating model bias has not been explored well. In this paper, we focus on the task of acquiring additional labeled data points for training the downstream machine learning model to rapidly improve its fairness. Since not all data points in a data pool are equally beneficial to the task of fairness, we generate an ordering in which data points should be acquired. We present DataSift, a data acquisition framework based on the idea of data valuation that relies on partitioning and multi-armed bandits to determine the most valuable data points to acquire. Over several iterations, DataSift selects a partition and randomly samples a batch of data points from the selected partition, evaluates the benefit of acquiring the batch on model fairness, and updates the utility of partitions depending on the benefit. To further improve the effectiveness and efficiency of evaluating batches, we leverage influence functions that estimate the effect of acquiring a batch without retraining the model. We empirically evaluate DataSift on several real-world and synthetic datasets and show that the fairness of a machine learning model can be significantly improved even while acquiring a few data points.

Title: Human Multi-View Synthesis from a Single-View Model:Transferred Body and Face Representations

• Authors: Yu Feng, Shunsi Zhang, Jian Shu, Hanfeng Zhao, Guoliang Pang, Chi Zhang, Hao Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03011
• Pdf URL: https://arxiv.org/pdf/2412.03011
• Copy Paste: [[2412.03011]] Human Multi-View Synthesis from a Single-View Model:Transferred Body and Face Representations(https://arxiv.org/abs/2412.03011)
• Keywords: diffusion
• Abstract: Generating multi-view human images from a single view is a complex and significant challenge. Although recent advancements in multi-view object generation have shown impressive results with diffusion models, novel view synthesis for humans remains constrained by the limited availability of 3D human datasets. Consequently, many existing models struggle to produce realistic human body shapes or capture fine-grained facial details accurately. To address these issues, we propose an innovative framework that leverages transferred body and facial representations for multi-view human synthesis. Specifically, we use a single-view model pretrained on a large-scale human dataset to develop a multi-view body representation, aiming to extend the 2D knowledge of the single-view model to a multi-view diffusion model. Additionally, to enhance the model's detail restoration capability, we integrate transferred multimodal facial features into our trained human diffusion model. Experimental evaluations on benchmark datasets demonstrate that our approach outperforms the current state-of-the-art methods, achieving superior performance in multi-view human synthesis.

Title: Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach

• Authors: Lingchen Sun, Rongyuan Wu, Zhiyuan Ma, Shuaizheng Liu, Qiaosi Yi, Lei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03017
• Pdf URL: https://arxiv.org/pdf/2412.03017
• Copy Paste: [[2412.03017]] Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach(https://arxiv.org/abs/2412.03017)
• Keywords: diffusion
• Abstract: Diffusion prior-based methods have shown impressive results in real-world image super-resolution (SR). However, most existing methods entangle pixel-level and semantic-level SR objectives in the training process, struggling to balance pixel-wise fidelity and perceptual quality. Meanwhile, users have varying preferences on SR results, thus it is demanded to develop an adjustable SR model that can be tailored to different fidelity-perception preferences during inference without re-training. We present Pixel-level and Semantic-level Adjustable SR (PiSA-SR), which learns two LoRA modules upon the pre-trained stable-diffusion (SD) model to achieve improved and adjustable SR results. We first formulate the SD-based SR problem as learning the residual between the low-quality input and the high-quality output, then show that the learning objective can be decoupled into two distinct LoRA weight spaces: one is characterized by the $\ell_2$-loss for pixel-level regression, and another is characterized by the LPIPS and classifier score distillation losses to extract semantic information from pre-trained classification and SD models. In its default setting, PiSA-SR can be performed in a single diffusion step, achieving leading real-world SR results in both quality and efficiency. By introducing two adjustable guidance scales on the two LoRA modules to control the strengths of pixel-wise fidelity and semantic-level details during inference, PiSASR can offer flexible SR results according to user preference without re-training. Codes and models can be found at this https URL.

Title: Human Variability vs. Machine Consistency: A Linguistic Analysis of Texts Generated by Humans and Large Language Models

• Authors: Sergio E. Zanotto, Segun Aroyehun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03025
• Pdf URL: https://arxiv.org/pdf/2412.03025
• Copy Paste: [[2412.03025]] Human Variability vs. Machine Consistency: A Linguistic Analysis of Texts Generated by Humans and Large Language Models(https://arxiv.org/abs/2412.03025)
• Keywords: large language model
• Abstract: The rapid advancements in large language models (LLMs) have significantly improved their ability to generate natural language, making texts generated by LLMs increasingly indistinguishable from human-written texts. Recent research has predominantly focused on using LLMs to classify text as either human-written or machine-generated. In our study, we adopt a different approach by profiling texts spanning four domains based on 250 distinct linguistic features. We select the M4 dataset from the Subtask B of SemEval 2024 Task 8. We automatically calculate various linguistic features with the LFTK tool and additionally measure the average syntactic depth, semantic similarity, and emotional content for each document. We then apply a two-dimensional PCA reduction to all the calculated features. Our analyses reveal significant differences between human-written texts and those generated by LLMs, particularly in the variability of these features, which we find to be considerably higher in human-written texts. This discrepancy is especially evident in text genres with less rigid linguistic style constraints. Our findings indicate that humans write texts that are less cognitively demanding, with higher semantic content, and richer emotional content compared to texts generated by LLMs. These insights underscore the need for incorporating meaningful linguistic features to enhance the understanding of textual outputs of LLMs.

Title: Frequency-Guided Diffusion Model with Perturbation Training for Skeleton-Based Video Anomaly Detection

• Authors: Xiaofeng Tan, Hongsong Wang, Xin Geng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03044
• Pdf URL: https://arxiv.org/pdf/2412.03044
• Copy Paste: [[2412.03044]] Frequency-Guided Diffusion Model with Perturbation Training for Skeleton-Based Video Anomaly Detection(https://arxiv.org/abs/2412.03044)
• Keywords: robust, diffusion
• Abstract: Video anomaly detection is an essential yet challenging open-set task in computer vision, often addressed by leveraging reconstruction as a proxy task. However, existing reconstruction-based methods encounter challenges in two main aspects: (1) limited model robustness for open-set scenarios, (2) and an overemphasis on, but restricted capacity for, detailed motion reconstruction. To this end, we propose a novel frequency-guided diffusion model with perturbation training, which enhances the model robustness by perturbation training and emphasizes the principal motion components guided by motion frequencies. Specifically, we first use a trainable generator to produce perturbative samples for perturbation training of the diffusion model. During the perturbation training phase, the model robustness is enhanced and the domain of the reconstructed model is broadened by training against this generator. Subsequently, perturbative samples are introduced for inference, which impacts the reconstruction of normal and abnormal motions differentially, thereby enhancing their separability. Considering that motion details originate from high-frequency information, we propose a masking method based on 2D discrete cosine transform to separate high-frequency information and low-frequency information. Guided by the high-frequency information from observed motion, the diffusion model can focus on generating low-frequency information, and thus reconstructing the motion accurately. Experimental results on five video anomaly detection datasets, including human-related and open-set benchmarks, demonstrate the effectiveness of the proposed method. Our code is available at this https URL.

Title: Less is More: A Stealthy and Efficient Adversarial Attack Method for DRL-based Autonomous Driving Policies

• Authors: Junchao Fan, Xuyang Lei, Xiaolin Chang, Jelena Mišić, Vojislav B. Mišić
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03051
• Pdf URL: https://arxiv.org/pdf/2412.03051
• Copy Paste: [[2412.03051]] Less is More: A Stealthy and Efficient Adversarial Attack Method for DRL-based Autonomous Driving Policies(https://arxiv.org/abs/2412.03051)
• Keywords: protect, attack, robust, steal
• Abstract: Despite significant advancements in deep reinforcement learning (DRL)-based autonomous driving policies, these policies still exhibit vulnerability to adversarial attacks. This vulnerability poses a formidable challenge to the practical deployment of these policies in autonomous driving. Designing effective adversarial attacks is an indispensable prerequisite for enhancing the robustness of these policies. In view of this, we present a novel stealthy and efficient adversarial attack method for DRL-based autonomous driving policies. Specifically, we introduce a DRL-based adversary designed to trigger safety violations (e.g., collisions) by injecting adversarial samples at critical moments. We model the attack as a mixed-integer optimization problem and formulate it as a Markov decision process. Then, we train the adversary to learn the optimal policy for attacking at critical moments without domain knowledge. Furthermore, we introduce attack-related information and a trajectory clipping method to enhance the learning capability of the adversary. Finally, we validate our method in an unprotected left-turn scenario across different traffic densities. The experimental results show that our method achieves more than 90% collision rate within three attacks in most cases. Furthermore, our method achieves more than 130% improvement in attack efficiency compared to the unlimited attack method.

Title: Point-GR: Graph Residual Point Cloud Network for 3D Object Classification and Segmentation

• Authors: Md Meraz, Md Afzal Ansari, Mohammed Javed, Pavan Chakraborty
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.03052
• Pdf URL: https://arxiv.org/pdf/2412.03052
• Copy Paste: [[2412.03052]] Point-GR: Graph Residual Point Cloud Network for 3D Object Classification and Segmentation(https://arxiv.org/abs/2412.03052)
• Keywords: extraction, segmentation
• Abstract: In recent years, the challenge of 3D shape analysis within point cloud data has gathered significant attention in computer vision. Addressing the complexities of effective 3D information representation and meaningful feature extraction for classification tasks remains crucial. This paper presents Point-GR, a novel deep learning architecture designed explicitly to transform unordered raw point clouds into higher dimensions while preserving local geometric features. It introduces residual-based learning within the network to mitigate the point permutation issues in point cloud data. The proposed Point-GR network significantly reduced the number of network parameters in Classification and Part-Segmentation compared to baseline graph-based networks. Notably, the Point-GR model achieves a state-of-the-art scene segmentation mean IoU of 73.47% on the S3DIS benchmark dataset, showcasing its effectiveness. Furthermore, the model shows competitive results in Classification and Part-Segmentation tasks.

Title: Revisiting Energy-Based Model for Out-of-Distribution Detection

• Authors: Yifan Wu, Xichen Ye, Songmin Dai, Dengye Pan, Xiaoqiang Li, Weizhong Zhang, Yifan Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03058
• Pdf URL: https://arxiv.org/pdf/2412.03058
• Copy Paste: [[2412.03058]] Revisiting Energy-Based Model for Out-of-Distribution Detection(https://arxiv.org/abs/2412.03058)
• Keywords: robust
• Abstract: Out-of-distribution (OOD) detection is an essential approach to robustifying deep learning models, enabling them to identify inputs that fall outside of their trained distribution. Existing OOD detection methods usually depend on crafted data, such as specific outlier datasets or elaborate data augmentations. While this is reasonable, the frequent mismatch between crafted data and OOD data limits model robustness and generalizability. In response to this issue, we introduce Outlier Exposure by Simple Transformations (OEST), a framework that enhances OOD detection by leveraging "peripheral-distribution" (PD) data. Specifically, PD data are samples generated through simple data transformations, thus providing an efficient alternative to manually curated outliers. We adopt energy-based models (EBMs) to study PD data. We recognize the "energy barrier" in OOD detection, which characterizes the energy difference between in-distribution (ID) and OOD samples and eases detection. PD data are introduced to establish the energy barrier during training. Furthermore, this energy barrier concept motivates a theoretically grounded energy-barrier loss to replace the classical energy-bounded loss, leading to an improved paradigm, OEST*, which achieves a more effective and theoretically sound separation between ID and OOD samples. We perform empirical validation of our proposal, and extensive experiments across various benchmarks demonstrate that OEST* achieves better or similar accuracy compared with state-of-the-art methods.

Title: UTSD: Unified Time Series Diffusion Model

• Authors: Xiangkai Ma, Xiaobin Hong, Wenzhong Li, Sanglu Lu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03068
• Pdf URL: https://arxiv.org/pdf/2412.03068
• Copy Paste: [[2412.03068]] UTSD: Unified Time Series Diffusion Model(https://arxiv.org/abs/2412.03068)
• Keywords: diffusion, transformer
• Abstract: Transformer-based architectures have achieved unprecedented success in time series analysis. However, facing the challenge of across-domain modeling, existing studies utilize statistical prior as prompt engineering fails under the huge distribution shift among various domains. In this paper, a Unified Time Series Diffusion (UTSD) model is established for the first time to model the multi-domain probability distribution, utilizing the powerful probability distribution modeling ability of Diffusion. Unlike the autoregressive models that capture the conditional probabilities of the prediction horizon to the historical sequence, we use a diffusion denoising process to model the mixture distribution of the cross-domain data and generate the prediction sequence for the target domain directly utilizing conditional sampling. The proposed UTSD contains three pivotal designs: (1) The condition network captures the multi-scale fluctuation patterns from the observation sequence, which are utilized as context representations to guide the denoising network to generate the prediction sequence; (2) Adapter-based fine-tuning strategy, the multi-domain universal representation learned in the pretraining stage is utilized for downstream tasks in target domains; (3) The diffusion and denoising process on the actual sequence space, combined with the improved classifier free guidance as the conditional generation strategy, greatly improves the stability and accuracy of the downstream task. We conduct extensive experiments on mainstream benchmarks, and the pre-trained UTSD outperforms existing foundation models on all data domains, exhibiting superior zero-shot generalization ability. After training from scratch, UTSD achieves comparable performance against domain-specific proprietary models. The empirical results validate the potential of UTSD as a time series foundational model.

Title: ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction

• Authors: Victor Junqiu Wei, Weicheng Wang, Di Jiang, Yuanfeng Song, Lu Wang
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.03075
• Pdf URL: https://arxiv.org/pdf/2412.03075
• Copy Paste: [[2412.03075]] ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction(https://arxiv.org/abs/2412.03075)
• Keywords: large language model
• Abstract: Automatic speech Recognition (ASR) is a fundamental and important task in the field of speech and natural language processing. It is an inherent building block in many applications such as voice assistant, speech translation, etc. Despite the advancement of ASR technologies in recent years, it is still inevitable for modern ASR systems to have a substantial number of erroneous recognition due to environmental noise, ambiguity, etc. Therefore, the error correction in ASR is crucial. Motivated by this, this paper studies ASR error correction in the Chinese language, which is one of the most popular languages and enjoys a large number of users in the world. We first create a benchmark dataset named \emph{ASR-EC} that contains a wide spectrum of ASR errors generated by industry-grade ASR systems. To the best of our knowledge, it is the first Chinese ASR error correction benchmark. Then, inspired by the recent advances in \emph{large language models (LLMs)}, we investigate how to harness the power of LLMs to correct ASR errors. We apply LLMs to ASR error correction in three paradigms. The first paradigm is prompting, which is further categorized as zero-shot, few-shot, and multi-step. The second paradigm is finetuning, which finetunes LLMs with ASR error correction data. The third paradigm is multi-modal augmentation, which collectively utilizes the audio and ASR transcripts for error correction. Extensive experiments reveal that prompting is not effective for ASR error correction. Finetuning is effective only for a portion of LLMs. Multi-modal augmentation is the most effective method for error correction and achieves state-of-the-art performance.

Title: RoDyGS: Robust Dynamic Gaussian Splatting for Casual Videos

• Authors: Yoonwoo Jeong, Junmyeong Lee, Hoseung Choi, Minsu Cho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03077
• Pdf URL: https://arxiv.org/pdf/2412.03077
• Copy Paste: [[2412.03077]] RoDyGS: Robust Dynamic Gaussian Splatting for Casual Videos(https://arxiv.org/abs/2412.03077)
• Keywords: robust
• Abstract: Dynamic view synthesis (DVS) has advanced remarkably in recent years, achieving high-fidelity rendering while reducing computational costs. Despite the progress, optimizing dynamic neural fields from casual videos remains challenging, as these videos do not provide direct 3D information, such as camera trajectories or the underlying scene geometry. In this work, we present RoDyGS, an optimization pipeline for dynamic Gaussian Splatting from casual videos. It effectively learns motion and underlying geometry of scenes by separating dynamic and static primitives, and ensures that the learned motion and geometry are physically plausible by incorporating motion and geometric regularization terms. We also introduce a comprehensive benchmark, Kubric-MRig, that provides extensive camera and object motion along with simultaneous multi-view captures, features that are absent in previous benchmarks. Experimental results demonstrate that the proposed method significantly outperforms previous pose-free dynamic neural fields and achieves competitive rendering quality compared to existing pose-free static neural fields. The code and data are publicly available at this https URL.

Title: Align3R: Aligned Monocular Depth Estimation for Dynamic Videos

• Authors: Jiahao Lu, Tianyu Huang, Peng Li, Zhiyang Dou, Cheng Lin, Zhiming Cui, Zhen Dong, Sai-Kit Yeung, Wenping Wang, Yuan Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03079
• Pdf URL: https://arxiv.org/pdf/2412.03079
• Copy Paste: [[2412.03079]] Align3R: Aligned Monocular Depth Estimation for Dynamic Videos(https://arxiv.org/abs/2412.03079)
• Keywords: diffusion
• Abstract: Recent developments in monocular depth estimation methods enable high-quality depth estimation of single-view images but fail to estimate consistent video depth across different frames. Recent works address this problem by applying a video diffusion model to generate video depth conditioned on the input video, which is training-expensive and can only produce scale-invariant depth values without camera poses. In this paper, we propose a novel video-depth estimation method called Align3R to estimate temporal consistent depth maps for a dynamic video. Our key idea is to utilize the recent DUSt3R model to align estimated monocular depth maps of different timesteps. First, we fine-tune the DUSt3R model with additional estimated monocular depth as inputs for the dynamic scenes. Then, we apply optimization to reconstruct both depth maps and camera poses. Extensive experiments demonstrate that Align3R estimates consistent video depth and camera poses for a monocular video with superior performance than baseline methods.

Title: Mimir: Improving Video Diffusion Models for Precise Text Understanding

• Authors: Shuai Tan, Biao Gong, Yutong Feng, Kecheng Zheng, Dandan Zheng, Shuwei Shi, Yujun Shen, Jingdong Chen, Ming Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03085
• Pdf URL: https://arxiv.org/pdf/2412.03085
• Copy Paste: [[2412.03085]] Mimir: Improving Video Diffusion Models for Precise Text Understanding(https://arxiv.org/abs/2412.03085)
• Keywords: diffusion, transformer, large language model
• Abstract: Text serves as the key control signal in video generation due to its narrative nature. To render text descriptions into video clips, current video diffusion models borrow features from text encoders yet struggle with limited text comprehension. The recent success of large language models (LLMs) showcases the power of decoder-only transformers, which offers three clear benefits for text-to-video (T2V) generation, namely, precise text understanding resulting from the superior scalability, imagination beyond the input text enabled by next token prediction, and flexibility to prioritize user interests through instruction tuning. Nevertheless, the feature distribution gap emerging from the two different text modeling paradigms hinders the direct use of LLMs in established T2V models. This work addresses this challenge with Mimir, an end-to-end training framework featuring a carefully tailored token fuser to harmonize the outputs from text encoders and LLMs. Such a design allows the T2V model to fully leverage learned video priors while capitalizing on the text-related capability of LLMs. Extensive quantitative and qualitative results demonstrate the effectiveness of Mimir in generating high-quality videos with excellent text comprehension, especially when processing short captions and managing shifting motions. Project page: this https URL

Title: Revolve: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization

• Authors: Peiyan Zhang, Haibo Jin, Leyang Hu, Xinnuo Li, Liying Kang, Man Luo, Yangqiu Song, Haohan Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03092
• Pdf URL: https://arxiv.org/pdf/2412.03092
• Copy Paste: [[2412.03092]] Revolve: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization(https://arxiv.org/abs/2412.03092)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have significantly enhanced the ability of LLM-based systems to perform complex tasks through natural language processing and tool interaction. However, optimizing these LLM-based systems for specific tasks remains challenging, often requiring manual interventions like prompt engineering and hyperparameter tuning. Existing automatic optimization methods, such as textual feedback-based techniques (e.g., TextGrad), tend to focus on immediate feedback, analogous to using immediate derivatives in traditional numerical gradient descent. However, relying solely on such feedback can be limited when the adjustments made in response to this feedback are either too small or fluctuate irregularly, potentially slowing down or even stalling the optimization process. To overcome these challenges, more adaptive methods are needed, especially in situations where the system's response is evolving slowly or unpredictably. In this paper, we introduce REVOLVE, an optimization method that tracks how "R"esponses "EVOLVE" across iterations in LLM systems. By focusing on the evolution of responses over time, REVOLVE enables more stable and effective optimization by making thoughtful, progressive adjustments at each step. Experimental results demonstrate that REVOLVE outperforms competitive baselines, achieving a 7.8% improvement in prompt optimization, a 20.72% gain in solution refinement, and a 29.17% increase in code optimization. Additionally, REVOLVE converges in fewer iterations, resulting in significant computational savings. These advantages highlight its adaptability and efficiency, positioning REVOLVE as a valuable tool for optimizing LLM-based systems and accelerating the development of next-generation AI technologies. Code is available at: this https URL.

Title: Expanding Event Modality Applications through a Robust CLIP-Based Encoder

• Authors: Sungheon Jeong, Hanning Chen, Sanggeon Yun, Suhyeon Cho, Wenjun Huang, Xiangjian Liu, Mohsen Imani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03093
• Pdf URL: https://arxiv.org/pdf/2412.03093
• Copy Paste: [[2412.03093]] Expanding Event Modality Applications through a Robust CLIP-Based Encoder(https://arxiv.org/abs/2412.03093)
• Keywords: robust
• Abstract: This paper introduces a powerful encoder that transfers CLIP`s capabilities to event-based data, enhancing its utility and expanding its applicability across diverse domains. While large-scale datasets have significantly advanced image-based models, the scarcity of comprehensive event datasets has limited performance potential in event modality. To address this challenge, we adapt CLIP`s architecture to align event embeddings with image embeddings, supporting zero-shot learning and preserving text alignment while mitigating catastrophic forgetting. Our encoder achieves strong performance in object recognition, with competitive results in zero-shot and few-shot learning tasks. Notably, it generalizes effectively to events extracted from video data without requiring additional training, highlighting its versatility. Additionally, we integrate this encoder within a cross-modality framework that facilitates interaction across five modalities-Image, Event, Text, Sound, and Depth-expanding the possibilities for cross-modal applications. Overall, this work underscores the transformative potential of a robust event encoder, broadening the scope and utility of event-based data across various fields.

Title: TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLM

• Authors: Huiying Cao, Yiqun Zhang, Shi Feng, Xiaocui Yang, Daling Wang, Yifei Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03096
• Pdf URL: https://arxiv.org/pdf/2412.03096
• Copy Paste: [[2412.03096]] TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLM(https://arxiv.org/abs/2412.03096)
• Keywords: large language model
• Abstract: Empathetic conversation is a crucial characteristic in daily conversations between individuals. Nowadays, Large Language models (LLMs) have shown outstanding performance in generating empathetic responses. Knowledge bases like COMET can assist LLMs in mitigating illusions and enhancing the understanding of users' intentions and emotions. However, models remain heavily reliant on fixed knowledge bases and unrestricted incorporation of external knowledge can introduce noise. Tool learning is a flexible end-to-end approach that assists LLMs in handling complex problems. In this paper, we propose Emotional Knowledge Tool Calling (EKTC) framework, which encapsulates the commonsense knowledge bases as empathetic tools, enabling LLMs to integrate external knowledge flexibly through tool calling. In order to adapt the models to the new task, we construct a novel dataset TOOL-ED based on the EMPATHETICMPATHETIC DIALOGUE (ED) dataset. We validate EKTC on the ED dataset, and the experimental results demonstrate that our framework can enhance the ability of LLMs to generate empathetic responses effectively.

Title: MultiGO: Towards Multi-level Geometry Learning for Monocular 3D Textured Human Reconstruction

• Authors: Gangjian Zhang, Nanjie Yao, Shunsi Zhang, Hanfeng Zhao, Guoliang Pang, Jian Shu, Hao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03103
• Pdf URL: https://arxiv.org/pdf/2412.03103
• Copy Paste: [[2412.03103]] MultiGO: Towards Multi-level Geometry Learning for Monocular 3D Textured Human Reconstruction(https://arxiv.org/abs/2412.03103)
• Keywords: diffusion, generative
• Abstract: This paper investigates the research task of reconstructing the 3D clothed human body from a monocular image. Due to the inherent ambiguity of single-view input, existing approaches leverage pre-trained SMPL(-X) estimation models or generative models to provide auxiliary information for human reconstruction. However, these methods capture only the general human body geometry and overlook specific geometric details, leading to inaccurate skeleton reconstruction, incorrect joint positions, and unclear cloth wrinkles. In response to these issues, we propose a multi-level geometry learning framework. Technically, we design three key components: skeleton-level enhancement, joint-level augmentation, and wrinkle-level refinement modules. Specifically, we effectively integrate the projected 3D Fourier features into a Gaussian reconstruction model, introduce perturbations to improve joint depth estimation during training, and refine the human coarse wrinkles by resembling the de-noising process of diffusion model. Extensive quantitative and qualitative experiments on two out-of-distribution test sets show the superior performance of our approach compared to state-of-the-art (SOTA) methods.

Title: Few-Shot Learning with Adaptive Weight Masking in Conditional GANs

• Authors: Jiacheng Hu, Zhen Qi, Jianjun Wei, Jiajing Chen, Runyuan Bao, Xinyu Qiu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03105
• Pdf URL: https://arxiv.org/pdf/2412.03105
• Copy Paste: [[2412.03105]] Few-Shot Learning with Adaptive Weight Masking in Conditional GANs(https://arxiv.org/abs/2412.03105)
• Keywords: robust, generative
• Abstract: Deep learning has revolutionized various fields, yet its efficacy is hindered by overfitting and the requirement of extensive annotated data, particularly in few-shot learning scenarios where limited samples are available. This paper introduces a novel approach to few-shot learning by employing a Residual Weight Masking Conditional Generative Adversarial Network (RWM-CGAN) for data augmentation. The proposed model integrates residual units within the generator to enhance network depth and sample quality, coupled with a weight mask regularization technique in the discriminator to improve feature learning from small-sample categories. This method addresses the core issues of robustness and generalization in few-shot learning by providing a controlled and clear augmentation of the sample space. Extensive experiments demonstrate that RWM-CGAN not only expands the sample space effectively but also enriches the diversity and quality of generated samples, leading to significant improvements in detection and classification accuracy on public datasets. The paper contributes to the advancement of few-shot learning by offering a practical solution to the challenges posed by data scarcity and the need for rapid generalization to new tasks or categories.

Title: Splats in Splats: Embedding Invisible 3D Watermark within Gaussian Splatting

• Authors: Yijia Guo, Wenkai Huang, Yang Li, Gaolei Li, Hang Zhang, Liwen Hu, Jianhua Li, Tiejun Huang, Lei Ma
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.03121
• Pdf URL: https://arxiv.org/pdf/2412.03121
• Copy Paste: [[2412.03121]] Splats in Splats: Embedding Invisible 3D Watermark within Gaussian Splatting(https://arxiv.org/abs/2412.03121)
• Keywords: security, protect, robust, watermark
• Abstract: 3D Gaussian splatting (3DGS) has demonstrated impressive 3D reconstruction performance with explicit scene representations. Given the widespread application of 3DGS in 3D reconstruction and generation tasks, there is an urgent need to protect the copyright of 3DGS assets. However, existing copyright protection techniques for 3DGS overlook the usability of 3D assets, posing challenges for practical deployment. Here we describe WaterGS, the first 3DGS watermarking framework that embeds 3D content in 3DGS itself without modifying any attributes of the vanilla 3DGS. To achieve this, we take a deep insight into spherical harmonics (SH) and devise an importance-graded SH coefficient encryption strategy to embed the hidden SH coefficients. Furthermore, we employ a convolutional autoencoder to establish a mapping between the original Gaussian primitives' opacity and the hidden Gaussian primitives' opacity. Extensive experiments indicate that WaterGS significantly outperforms existing 3D steganography techniques, with 5.31% higher scene fidelity and 3X faster rendering speed, while ensuring security, robustness, and user experience. Codes and data will be released at this https URL.

Title: Unifying KV Cache Compression for Large Language Models with LeanKV

• Authors: Yanqi Zhang, Yuwei Hu, Runyuan Zhao, John C.S. Lui, Haibo Chen
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.03131
• Pdf URL: https://arxiv.org/pdf/2412.03131
• Copy Paste: [[2412.03131]] Unifying KV Cache Compression for Large Language Models with LeanKV(https://arxiv.org/abs/2412.03131)
• Keywords: large language model
• Abstract: Large language models (LLMs) demonstrate exceptional performance but incur high serving costs due to substantial memory demands, with the key-value (KV) cache being a primary bottleneck. Existing KV cache compression methods, including quantization and pruning, struggle with limitations such as uniform treatment of keys and values and static memory allocation across attention heads. To address these challenges, we introduce LeanKV, a unified KV cache compression framework that enhances LLM serving efficiency without compromising accuracy through three innovations: (1) Hetero-KV quantization, which stores keys at a higher precision than values to reflect their greater impact on attention computations; (2) per-head dynamic sparsity, which allocates memory based on token importance per head and per request; and (3) unified KV compression, integrating mixed-precision quantization and selective pruning to enable a smooth tradeoff between model accuracy and memory efficiency. To efficiently support these techniques, LeanKV introduces systems optimizations including unified paging and on-GPU parallel memory management. Implemented on vLLM, LeanKV compresses the KV cache by $3.0\times$ to $5.0\times$ without accuracy loss and up to $11.0\times$ with under 5% accuracy loss, enhancing throughput by $1.9\times$ to $2.5\times$, and up to $6.9\times$.

Title: Fine-Grained Behavior Simulation with Role-Playing Large Language Model on Social Media

• Authors: Kun Li, Chenwei Dai, Wei Zhou, Songlin Hu
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.03148
• Pdf URL: https://arxiv.org/pdf/2412.03148
• Copy Paste: [[2412.03148]] Fine-Grained Behavior Simulation with Role-Playing Large Language Model on Social Media(https://arxiv.org/abs/2412.03148)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated impressive capabilities in role-playing tasks. However, there is limited research on whether LLMs can accurately simulate user behavior in real-world scenarios, such as social media. This requires models to effectively analyze a user's history and simulate their role. In this paper, we introduce \textbf{FineRob}, a novel fine-grained behavior simulation dataset. We collect the complete behavioral history of 1,866 distinct users across three social media platforms. Each behavior is decomposed into three fine-grained elements: object, type, and content, resulting in 78.6k QA records. Based on FineRob, we identify two dominant reasoning patterns in LLMs' behavior simulation processes and propose the \textbf{OM-CoT} fine-tuning method to enhance the capability. Through comprehensive experiments, we conduct an in-depth analysis of key factors of behavior simulation and also demonstrate the effectiveness of OM-CoT approach\footnote{Code and dataset are available at \url{this https URL}}

Title: Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis

• Authors: Siyoon Jin, Jisu Nam, Jiyoung Kim, Dahyun Chung, Yeong-Seok Kim, Joonhyung Park, Heonjeong Chu, Seungryong Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03150
• Pdf URL: https://arxiv.org/pdf/2412.03150
• Copy Paste: [[2412.03150]] Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis(https://arxiv.org/abs/2412.03150)
• Keywords: diffusion, segmentation
• Abstract: Exemplar-based semantic image synthesis aims to generate images aligned with given semantic content while preserving the appearance of an exemplar image. Conventional structure-guidance models, such as ControlNet, are limited in that they cannot directly utilize exemplar images as input, relying instead solely on text prompts to control appearance. Recent tuning-free approaches address this limitation by transferring local appearance from the exemplar image to the synthesized image through implicit cross-image matching in the augmented self-attention mechanism of pre-trained diffusion models. However, these methods face challenges when applied to content-rich scenes with significant geometric deformations, such as driving scenes. In this paper, we propose the Appearance Matching Adapter (AM-Adapter), a learnable framework that enhances cross-image matching within augmented self-attention by incorporating semantic information from segmentation maps. To effectively disentangle generation and matching processes, we adopt a stage-wise training approach. Initially, we train the structure-guidance and generation networks, followed by training the AM-Adapter while keeping the other networks frozen. During inference, we introduce an automated exemplar retrieval method to efficiently select exemplar image-segmentation pairs. Despite utilizing a limited number of learnable parameters, our method achieves state-of-the-art performance, excelling in both semantic alignment preservation and local appearance fidelity. Extensive ablation studies further validate our design choices. Code and pre-trained weights will be publicly available.: this https URL

Title: A Measure of the System Dependence of Automated Metrics

• Authors: Pius von Däniken, Jan Deriu, Mark Cieliebak
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03152
• Pdf URL: https://arxiv.org/pdf/2412.03152
• Copy Paste: [[2412.03152]] A Measure of the System Dependence of Automated Metrics(https://arxiv.org/abs/2412.03152)
• Keywords: fair
• Abstract: Automated metrics for Machine Translation have made significant progress, with the goal of replacing expensive and time-consuming human evaluations. These metrics are typically assessed by their correlation with human judgments, which captures the monotonic relationship between human and metric scores. However, we argue that it is equally important to ensure that metrics treat all systems fairly and consistently. In this paper, we introduce a method to evaluate this aspect.

Title: Testing Neural Network Verifiers: A Soundness Benchmark with Hidden Counterexamples

• Authors: Xingjian Zhou, Hongji Xu, Andy Xu, Zhouxing Shi, Cho-Jui Hsieh, Huan Zhang
• Subjects: cs.LG, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.03154
• Pdf URL: https://arxiv.org/pdf/2412.03154
• Copy Paste: [[2412.03154]] Testing Neural Network Verifiers: A Soundness Benchmark with Hidden Counterexamples(https://arxiv.org/abs/2412.03154)
• Keywords: attack, robust
• Abstract: In recent years, many neural network (NN) verifiers have been developed to formally verify certain properties of neural networks such as robustness. Although many benchmarks have been constructed to evaluate the performance of NN verifiers, they typically lack a ground-truth for hard instances where no current verifier can verify and no counterexample can be found, which makes it difficult to check the soundness of a new verifier if it claims to verify hard instances which no other verifier can do. We propose to develop a soundness benchmark for NN verification. Our benchmark contains instances with deliberately inserted counterexamples while we also try to hide the counterexamples from regular adversarial attacks which can be used for finding counterexamples. We design a training method to produce neural networks with such hidden counterexamples. Our benchmark aims to be used for testing the soundness of NN verifiers and identifying falsely claimed verifiability when it is known that hidden counterexamples exist. We systematically construct our benchmark and generate instances across diverse model architectures, activation functions, input sizes, and perturbation radii. We demonstrate that our benchmark successfully identifies bugs in state-of-the-art NN verifiers, as well as synthetic bugs, providing a crucial step toward enhancing the reliability of testing NN verifiers. Our code is available at this https URL and our benchmark is available at this https URL.

Title: Byte BPE Tokenization as an Inverse string Homomorphism

• Authors: Saibo Geng, Sankalp Gambhir, Chris Wendler, Robert West
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03160
• Pdf URL: https://arxiv.org/pdf/2412.03160
• Copy Paste: [[2412.03160]] Byte BPE Tokenization as an Inverse string Homomorphism(https://arxiv.org/abs/2412.03160)
• Keywords: large language model
• Abstract: Tokenization is an important preprocessing step in the training and inference of large language models (LLMs). While there has been extensive research on the expressive power of the neural achitectures used in LLMs, the impact of tokenization has not been well understood. In this work, we demonstrate that tokenization, irrespective of the algorithm used, acts as an inverse homomorphism between strings and tokens. This suggests that the character space of the source language and the token space of the tokenized language are homomorphic, preserving the structural properties of the source language. Additionally, we explore the concept of proper tokenization, which refers to an unambiguous tokenization returned from the tokenizer. Our analysis reveals that the expressiveness of neural architectures in recognizing context-free languages is not affected by tokenization.

Title: Are Explanations Helpful? A Comparative Analysis of Explainability Methods in Skin Lesion Classifiers

• Authors: Rosa Y. G. Paccotacya-Yanque, Alceu Bissoto, Sandra Avila
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03166
• Pdf URL: https://arxiv.org/pdf/2412.03166
• Copy Paste: [[2412.03166]] Are Explanations Helpful? A Comparative Analysis of Explainability Methods in Skin Lesion Classifiers(https://arxiv.org/abs/2412.03166)
• Keywords: explainability
• Abstract: Deep Learning has shown outstanding results in computer vision tasks; healthcare is no exception. However, there is no straightforward way to expose the decision-making process of DL models. Good accuracy is not enough for skin cancer predictions. Understanding the model's behavior is crucial for clinical application and reliable outcomes. In this work, we identify desiderata for explanations in skin-lesion models. We analyzed seven methods, four based on pixel-attribution (Grad-CAM, Score-CAM, LIME, SHAP) and three on high-level concepts (ACE, ICE, CME), for a deep neural network trained on the International Skin Imaging Collaboration Archive. Our findings indicate that while these techniques reveal biases, there is room for improving the comprehensiveness of explanations to achieve transparency in skin-lesion models.

Title: Automatic detection of diseases in Spanish clinical notes combining medical language models and ontologies

• Authors: Leon-Paul Schaub Torre, Pelayo Quiros, Helena Garcia Mieres
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03176
• Pdf URL: https://arxiv.org/pdf/2412.03176
• Copy Paste: [[2412.03176]] Automatic detection of diseases in Spanish clinical notes combining medical language models and ontologies(https://arxiv.org/abs/2412.03176)
• Keywords: large language model
• Abstract: In this paper we present a hybrid method for the automatic detection of dermatological pathologies in medical reports. We use a large language model combined with medical ontologies to predict, given a first appointment or follow-up medical report, the pathology a person may suffer from. The results show that teaching the model to learn the type, severity and location on the body of a dermatological pathology, as well as in which order it has to learn these three features, significantly increases its accuracy. The article presents the demonstration of state-of-the-art results for classification of medical texts with a precision of 0.84, micro and macro F1-score of 0.82 and 0.75, and makes both the method and the data set used available to the community.

Title: Optimizing Dense Visual Predictions Through Multi-Task Coherence and Prioritization

• Authors: Maxime Fontana, Michael Spratling, Miaojing Shi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03179
• Pdf URL: https://arxiv.org/pdf/2412.03179
• Copy Paste: [[2412.03179]] Optimizing Dense Visual Predictions Through Multi-Task Coherence and Prioritization(https://arxiv.org/abs/2412.03179)
• Keywords: transformer
• Abstract: Multi-Task Learning (MTL) involves the concurrent training of multiple tasks, offering notable advantages for dense prediction tasks in computer vision. MTL not only reduces training and inference time as opposed to having multiple single-task models, but also enhances task accuracy through the interaction of multiple tasks. However, existing methods face limitations. They often rely on suboptimal cross-task interactions, resulting in task-specific predictions with poor geometric and predictive coherence. In addition, many approaches use inadequate loss weighting strategies, which do not address the inherent variability in task evolution during training. To overcome these challenges, we propose an advanced MTL model specifically designed for dense vision tasks. Our model leverages state-of-the-art vision transformers with task-specific decoders. To enhance cross-task coherence, we introduce a trace-back method that improves both cross-task geometric and predictive features. Furthermore, we present a novel dynamic task balancing approach that projects task losses onto a common scale and prioritizes more challenging tasks during training. Extensive experiments demonstrate the superiority of our method, establishing new state-of-the-art performance across two benchmark datasets. The code is available at:this https URL

Title: Biologically-inspired Semi-supervised Semantic Segmentation for Biomedical Imaging

• Authors: Luca Ciampi, Gabriele Lagani, Giuseppe Amato, Fabrizio Falchi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03192
• Pdf URL: https://arxiv.org/pdf/2412.03192
• Copy Paste: [[2412.03192]] Biologically-inspired Semi-supervised Semantic Segmentation for Biomedical Imaging(https://arxiv.org/abs/2412.03192)
• Keywords: segmentation
• Abstract: We propose a novel two-stage semi-supervised learning approach for training downsampling-upsampling semantic segmentation architectures. The first stage does not use backpropagation. Rather, it exploits the bio-inspired Hebbian principle "fire together, wire together" as a local learning rule for updating the weights of both convolutional and transpose-convolutional layers, allowing unsupervised discovery of data features. In the second stage, the model is fine-tuned with standard backpropagation on a small subset of labeled data. We evaluate our methodology through experiments conducted on several widely used biomedical datasets, deeming that this domain is paramount in computer vision and is notably impacted by data scarcity. Results show that our proposed method outperforms SOTA approaches across different levels of label availability. Furthermore, we show that using our unsupervised stage to initialize the SOTA approaches leads to performance improvements. The code to replicate our experiments can be found at: this https URL

Title: Fab-ME: A Vision State-Space and Attention-Enhanced Framework for Fabric Defect Detection

• Authors: Shuai Wang, Huiyan Kong, Baotian Li, Fa Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03200
• Pdf URL: https://arxiv.org/pdf/2412.03200
• Copy Paste: [[2412.03200]] Fab-ME: A Vision State-Space and Attention-Enhanced Framework for Fabric Defect Detection(https://arxiv.org/abs/2412.03200)
• Keywords: extraction
• Abstract: Effective defect detection is critical for ensuring the quality, functionality, and economic value of textile products. However, existing methods face challenges in achieving high accuracy, real-time performance, and efficient global information extraction. To address these issues, we propose Fab-ME, an advanced framework based on YOLOv8s, specifically designed for the accurate detection of 20 fabric defect types. Our contributions include the introduction of the cross-stage partial bottleneck with two convolutions (C2F) vision state-space (C2F-VMamba) module, which integrates visual state-space (VSS) blocks into the YOLOv8s feature fusion network neck, enhancing the capture of intricate details and global context while maintaining high processing speeds. Additionally, we incorporate an enhanced multi-scale channel attention (EMCA) module into the final layer of the feature extraction network, significantly improving sensitivity to small targets. Experimental results on the Tianchi fabric defect detection dataset demonstrate that Fab-ME achieves a 3.3\% improvement in mAP@0.5 compared to the original YOLOv8s, validating its effectiveness for precise and efficient fabric defect detection.

Title: Parametric Enhancement of PerceptNet: A Human-Inspired Approach for Image Quality Assessment

• Authors: Jorge Vila-Tomás, Pablo Hernández-Cámara, Valero Laparra, Jesús Malo
• Subjects: cs.CV, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2412.03210
• Pdf URL: https://arxiv.org/pdf/2412.03210
• Copy Paste: [[2412.03210]] Parametric Enhancement of PerceptNet: A Human-Inspired Approach for Image Quality Assessment(https://arxiv.org/abs/2412.03210)
• Keywords: interpretability
• Abstract: While deep learning models can learn human-like features at earlier levels, which suggests their utility in modeling human vision, few attempts exist to incorporate these features by design. Current approaches mostly optimize all parameters blindly, only constraining minor architectural aspects. This paper demonstrates how parametrizing neural network layers enables more biologically-plausible operations while reducing trainable parameters and improving interpretability. We constrain operations to functional forms present in human vision, optimizing only these functions' parameters rather than all convolutional tensor elements independently. We present two parametric model versions: one with hand-chosen biologically plausible parameters, and another fitted to human perception experimental data. We compare these with a non-parametric version. All models achieve comparable state-of-the-art results, with parametric versions showing orders of magnitude parameter reduction for minimal performance loss. The parametric models demonstrate improved interpretability and training behavior. Notably, the model fitted to human perception, despite biological initialization, converges to biologically incorrect results. This raises scientific questions and highlights the need for diverse evaluation methods to measure models' humanness, rather than assuming task performance correlates with human-like behavior.

Title: Semi-Supervised Transfer Boosting (SS-TrBoosting)

• Authors: Lingfei Deng, Changming Zhao, Zhenbang Du, Kun Xia, Dongrui Wu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03212
• Pdf URL: https://arxiv.org/pdf/2412.03212
• Copy Paste: [[2412.03212]] Semi-Supervised Transfer Boosting (SS-TrBoosting)(https://arxiv.org/abs/2412.03212)
• Keywords: privacy, protect
• Abstract: Semi-supervised domain adaptation (SSDA) aims at training a high-performance model for a target domain using few labeled target data, many unlabeled target data, and plenty of auxiliary data from a source domain. Previous works in SSDA mainly focused on learning transferable representations across domains. However, it is difficult to find a feature space where the source and target domains share the same conditional probability distribution. Additionally, there is no flexible and effective strategy extending existing unsupervised domain adaptation (UDA) approaches to SSDA settings. In order to solve the above two challenges, we propose a novel fine-tuning framework, semi-supervised transfer boosting (SS-TrBoosting). Given a well-trained deep learning-based UDA or SSDA model, we use it as the initial model, generate additional base learners by boosting, and then use all of them as an ensemble. More specifically, half of the base learners are generated by supervised domain adaptation, and half by semi-supervised learning. Furthermore, for more efficient data transmission and better data privacy protection, we propose a source data generation approach to extend SS-TrBoosting to semi-supervised source-free domain adaptation (SS-SFDA). Extensive experiments showed that SS-TrBoosting can be applied to a variety of existing UDA, SSDA and SFDA approaches to further improve their performance.

Title: ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression

• Authors: Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, Minyi Guo
• Subjects: cs.LG, cs.AI, cs.PF
• Abstract URL: https://arxiv.org/abs/2412.03213
• Pdf URL: https://arxiv.org/pdf/2412.03213
• Copy Paste: [[2412.03213]] ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression(https://arxiv.org/abs/2412.03213)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have been widely deployed in a variety of applications, and the context length is rapidly increasing to handle tasks such as long-document QA and complex logical reasoning. However, long context poses significant challenges for inference efficiency, including high memory costs of key-value (KV) cache and increased latency due to extensive memory accesses. Recent works have proposed compressing KV cache to approximate computation, but these methods either evict tokens permanently, never recalling them for later inference, or recall previous tokens at the granularity of pages divided by textual positions. Both approaches degrade the model accuracy and output quality. To achieve efficient and accurate recallable KV cache compression, we introduce ClusterKV, which recalls tokens at the granularity of semantic clusters. We design and implement efficient algorithms and systems for clustering, selection, indexing and caching. Experiment results show that ClusterKV attains negligible accuracy loss across various tasks with 32k context lengths, using only a 1k to 2k KV cache budget, and achieves up to a 2$\times$ speedup in latency and a 2.5$\times$ improvement in decoding throughput. Compared to SoTA recallable KV compression methods, ClusterKV demonstrates higher model accuracy and output quality, while maintaining or exceeding inference efficiency.

Title: Continual Low-Rank Scaled Dot-product Attention

• Authors: Ginés Carreto Picón, Illia Oleksiienko, Lukas Hedegaard, Arian Bakhtiarnia, Alexandros Iosifidis
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03214
• Pdf URL: https://arxiv.org/pdf/2412.03214
• Copy Paste: [[2412.03214]] Continual Low-Rank Scaled Dot-product Attention(https://arxiv.org/abs/2412.03214)
• Keywords: transformer
• Abstract: Transformers are widely used for their ability to capture data relations in sequence processing, with great success for a wide range of static tasks. However, the computational and memory footprint of their main component, i.e., the Scaled Dot-product Attention, is commonly overlooked. This makes their adoption in applications involving stream data processing with constraints in response latency, computational and memory resources infeasible. Some works have proposed methods to lower the computational cost of transformers, i.e. low-rank approximations, sparsity in attention, and efficient formulations for Continual Inference. In this paper, we introduce a new formulation of the Scaled Dot-product Attention based on the Nyström approximation that is suitable for Continual Inference. In experiments on Online Audio Classification and Online Action Detection tasks, the proposed Continual Scaled Dot-product Attention can lower the number of operations by up to three orders of magnitude compared to the original Transformers while retaining the predictive performance of competing models.

Title: Beyond [cls]: Exploring the true potential of Masked Image Modeling representations

• Authors: Marcin Przewięźlikowski, Randall Balestriero, Wojciech Jasiński, Marek Śmieja, Bartosz Zieliński
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03215
• Pdf URL: https://arxiv.org/pdf/2412.03215
• Copy Paste: [[2412.03215]] Beyond [cls]: Exploring the true potential of Masked Image Modeling representations(https://arxiv.org/abs/2412.03215)
• Keywords: transformer
• Abstract: Masked Image Modeling (MIM) has emerged as a popular method for Self-Supervised Learning (SSL) of visual representations. However, for high-level perception tasks, MIM-pretrained models offer lower out-of-the-box representation quality than the Joint-Embedding Architectures (JEA) - another prominent SSL paradigm. To understand this performance gap, we analyze the information flow in Vision Transformers (ViT) learned by both approaches. We reveal that whereas JEAs construct their representation on a selected set of relevant image fragments, MIM models aggregate nearly whole image content. Moreover, we demonstrate that MIM-trained ViTs retain valuable information within their patch tokens, which is not effectively captured by the global [cls] token representations. Therefore, selective aggregation of relevant patch tokens, without any fine-tuning, results in consistently higher-quality of MIM representations. To our knowledge, we are the first to highlight the lack of effective representation aggregation as an emergent issue of MIM and propose directions to address it, contributing to future advances in Self-Supervised Learning.

Title: Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

• Authors: Minghao Shao, Abdul Basit, Ramesh Karri, Muhammad Shafique
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03220
• Pdf URL: https://arxiv.org/pdf/2412.03220
• Copy Paste: [[2412.03220]] Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges(https://arxiv.org/abs/2412.03220)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs) represent a class of deep learning models adept at understanding natural language and generating coherent responses to various prompts or queries. These models far exceed the complexity of conventional neural networks, often encompassing dozens of neural network layers and containing billions to trillions of parameters. They are typically trained on vast datasets, utilizing architectures based on transformer blocks. Present-day LLMs are multi-functional, capable of performing a range of tasks from text generation and language translation to question answering, as well as code generation and analysis. An advanced subset of these models, known as Multimodal Large Language Models (MLLMs), extends LLM capabilities to process and interpret multiple data modalities, including images, audio, and video. This enhancement empowers MLLMs with capabilities like video editing, image comprehension, and captioning for visual content. This survey provides a comprehensive overview of the recent advancements in LLMs. We begin by tracing the evolution of LLMs and subsequently delve into the advent and nuances of MLLMs. We analyze emerging state-of-the-art MLLMs, exploring their technical features, strengths, and limitations. Additionally, we present a comparative analysis of these models and discuss their challenges, potential limitations, and prospects for future development.

Title: Linq-Embed-Mistral Technical Report

• Authors: Chanyeol Choi, Junseong Kim, Seolhwa Lee, Jihoon Kwon, Sangmo Gu, Yejin Kim, Minkyung Cho, Jy-yong Sohn
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03223
• Pdf URL: https://arxiv.org/pdf/2412.03223
• Copy Paste: [[2412.03223]] Linq-Embed-Mistral Technical Report(https://arxiv.org/abs/2412.03223)
• Keywords: large language model
• Abstract: This report explores the enhancement of text retrieval performance using advanced data refinement techniques. We develop Linq-Embed-Mistral\footnote{\url{this https URL}} by building on the E5-mistral and Mistral-7B-v0.1 models, focusing on sophisticated data crafting, data filtering, and negative mining methods, which are highly tailored to each task, applied to both existing benchmark dataset and highly tailored synthetic dataset generated via large language models (LLMs). Linq-Embed-Mistral excels in the MTEB benchmarks (as of May 29, 2024), achieving an average score of 68.2 across 56 datasets, and ranks 1st among all models for retrieval tasks on the MTEB leaderboard with a performance score of 60.2. This performance underscores its superior capability in enhancing search precision and reliability. Our contributions include advanced data refinement methods that significantly improve model performance on benchmark and synthetic datasets, techniques for homogeneous task ordering and mixed task fine-tuning to enhance model generalization and stability, and a streamlined evaluation process using 4-bit precision and a light retrieval evaluation set, which accelerates validation without sacrificing accuracy.

Title: MaterialPicker: Multi-Modal Material Generation with Diffusion Transformers

• Authors: Xiaohe Ma, Valentin Deschaintre, Miloš Hašan, Fujun Luan, Kun Zhou, Hongzhi Wu, Yiwei Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03225
• Pdf URL: https://arxiv.org/pdf/2412.03225
• Copy Paste: [[2412.03225]] MaterialPicker: Multi-Modal Material Generation with Diffusion Transformers(https://arxiv.org/abs/2412.03225)
• Keywords: diffusion, transformer
• Abstract: High-quality material generation is key for virtual environment authoring and inverse rendering. We propose MaterialPicker, a multi-modal material generator leveraging a Diffusion Transformer (DiT) architecture, improving and simplifying the creation of high-quality materials from text prompts and/or photographs. Our method can generate a material based on an image crop of a material sample, even if the captured surface is distorted, viewed at an angle or partially occluded, as is often the case in photographs of natural scenes. We further allow the user to specify a text prompt to provide additional guidance for the generation. We finetune a pre-trained DiT-based video generator into a material generator, where each material map is treated as a frame in a video sequence. We evaluate our approach both quantitatively and qualitatively and show that it enables more diverse material generation and better distortion correction than previous work.

Title: Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?

• Authors: Sravanti Addepalli, Yerram Varun, Arun Suggala, Karthikeyan Shanmugam, Prateek Jain
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03235
• Pdf URL: https://arxiv.org/pdf/2412.03235
• Copy Paste: [[2412.03235]] Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?(https://arxiv.org/abs/2412.03235)
• Keywords: defense, attack, large language model
• Abstract: Large Language Models (LLMs) are known to be susceptible to crafted adversarial attacks or jailbreaks that lead to the generation of objectionable content despite being aligned to human preferences using safety fine-tuning methods. While the large dimensionality of input token space makes it inevitable to find adversarial prompts that can jailbreak these models, we aim to evaluate whether safety fine-tuned LLMs are safe against natural prompts which are semantically related to toxic seed prompts that elicit safe responses after alignment. We surprisingly find that popular aligned LLMs such as GPT-4 can be compromised using naive prompts that are NOT even crafted with an objective of jailbreaking the model. Furthermore, we empirically show that given a seed prompt that elicits a toxic response from an unaligned model, one can systematically generate several semantically related natural prompts that can jailbreak aligned LLMs. Towards this, we propose a method of Response Guided Question Augmentation (ReG-QA) to evaluate the generalization of safety aligned LLMs to natural prompts, that first generates several toxic answers given a seed question using an unaligned LLM (Q to A), and further leverages an LLM to generate questions that are likely to produce these answers (A to Q). We interestingly find that safety fine-tuned LLMs such as GPT-4o are vulnerable to producing natural jailbreak questions from unsafe content (without denial) and can thus be used for the latter (A to Q) step. We obtain attack success rates that are comparable to/ better than leading adversarial attack methods on the JailbreakBench leaderboard, while being significantly more stable against defenses such as Smooth-LLM and Synonym Substitution, which are effective against existing all attacks on the leaderboard.

Title: Task-driven Image Fusion with Learnable Fusion Loss

• Authors: Haowen Bai, Jiangshe Zhang, Zixiang Zhao, Yichen Wu, Lilun Deng, Yukun Cui, Tao Feng, Shuang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03240
• Pdf URL: https://arxiv.org/pdf/2412.03240
• Copy Paste: [[2412.03240]] Task-driven Image Fusion with Learnable Fusion Loss(https://arxiv.org/abs/2412.03240)
• Keywords: segmentation
• Abstract: Multi-modal image fusion aggregates information from multiple sensor sources, achieving superior visual quality and perceptual characteristics compared to any single source, often enhancing downstream tasks. However, current fusion methods for downstream tasks still use predefined fusion objectives that potentially mismatch the downstream tasks, limiting adaptive guidance and reducing model flexibility. To address this, we propose Task-driven Image Fusion (TDFusion), a fusion framework incorporating a learnable fusion loss guided by task loss. Specifically, our fusion loss includes learnable parameters modeled by a neural network called the loss generation module. This module is supervised by the loss of downstream tasks in a meta-learning manner. The learning objective is to minimize the task loss of the fused images, once the fusion module has been optimized by the fusion loss. Iterative updates between the fusion module and the loss module ensure that the fusion network evolves toward minimizing task loss, guiding the fusion process toward the task objectives. TDFusion's training relies solely on the loss of downstream tasks, making it adaptable to any specific task. It can be applied to any architecture of fusion and task networks. Experiments demonstrate TDFusion's performance in both fusion and task-related applications, including four public fusion datasets, semantic segmentation, and object detection. The code will be released.

Title: Benchmarking terminology building capabilities of ChatGPT on an English-Russian Fashion Corpus

• Authors: Anastasiia Bezobrazova, Miriam Seghiri, Constantin Orasan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03242
• Pdf URL: https://arxiv.org/pdf/2412.03242
• Copy Paste: [[2412.03242]] Benchmarking terminology building capabilities of ChatGPT on an English-Russian Fashion Corpus(https://arxiv.org/abs/2412.03242)
• Keywords: extraction
• Abstract: This paper compares the accuracy of the terms extracted using SketchEngine, TBXTools and ChatGPT. In addition, it evaluates the quality of the definitions produced by ChatGPT for these terms. The research is carried out on a comparable corpus of fashion magazines written in English and Russian collected from the web. A gold standard for the fashion terminology was also developed by identifying web pages that can be harvested automatically and contain definitions of terms from the fashion domain in English and Russian. This gold standard was used to evaluate the quality of the extracted terms and of the definitions produced. Our evaluation shows that TBXTools and SketchEngine, while capable of high recall, suffer from reduced precision as the number of terms increases, which affects their overall performance. Conversely, ChatGPT demonstrates superior performance, maintaining or improving precision as more terms are considered. Analysis of the definitions produced by ChatGPT for 60 commonly used terms in English and Russian shows that ChatGPT maintains a reasonable level of accuracy and fidelity across languages, but sometimes the definitions in both languages miss crucial specifics and include unnecessary deviations. Our research reveals that no single tool excels universally; each has strengths suited to particular aspects of terminology extraction and application.

Title: AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning

• Authors: Yiwu Zhong, Zhuoming Liu, Yin Li, Liwei Wang
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03248
• Pdf URL: https://arxiv.org/pdf/2412.03248
• Copy Paste: [[2412.03248]] AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruning(https://arxiv.org/abs/2412.03248)
• Keywords: large language model
• Abstract: Large language models (LLMs) have enabled the creation of multi-modal LLMs that exhibit strong comprehension of visual data such as images and videos. However, these models usually rely on extensive visual tokens from visual encoders, leading to high computational demands, which limits their applicability in resource-constrained environments and for long-context tasks. In this work, we propose a training-free adaptive inference method for multi-modal LLMs that can accommodate a broad range of efficiency requirements with a minimum performance drop. Our method consists of a) iterative token merging based on embedding similarity before LLMs, and b) progressive token pruning within LLM layers based on multi-modal importance. With a minimalist design, our method can be applied to both video and image LLMs. Extensive experiments on diverse video and image benchmarks demonstrate that, our method substantially reduces computation load (e.g., a $\textbf{7-fold}$ reduction in FLOPs) while preserving the performance of video and image LLMs. Further, under a similar computational cost, our method outperforms the state-of-the-art methods in long video understanding (e.g., $\textbf{+4.6}$ on MLVU). Additionally, our in-depth analysis provides insights into token redundancy and LLM layer behaviors, offering guidance for future research in designing efficient multi-modal LLMs. Our code will be available at this https URL.

Title: Alignment at Pre-training! Towards Native Alignment for Arabic LLMs

• Authors: Juhao Liang, Zhenyang Cai, Jianqing Zhu, Huang Huang, Kewei Zong, Bang An, Mosen Alharthi, Juncai He, Lian Zhang, Haizhou Li, Benyou Wang, Jinchao Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03253
• Pdf URL: https://arxiv.org/pdf/2412.03253
• Copy Paste: [[2412.03253]] Alignment at Pre-training! Towards Native Alignment for Arabic LLMs(https://arxiv.org/abs/2412.03253)
• Keywords: large language model
• Abstract: The alignment of large language models (LLMs) is critical for developing effective and safe language models. Traditional approaches focus on aligning models during the instruction tuning or reinforcement learning stages, referred to in this paper as `post alignment'. We argue that alignment during the pre-training phase, which we term `native alignment', warrants investigation. Native alignment aims to prevent unaligned content from the beginning, rather than relying on post-hoc processing. This approach leverages extensively aligned pre-training data to enhance the effectiveness and usability of pre-trained models. Our study specifically explores the application of native alignment in the context of Arabic LLMs. We conduct comprehensive experiments and ablation studies to evaluate the impact of native alignment on model performance and alignment stability. Additionally, we release open-source Arabic LLMs that demonstrate state-of-the-art performance on various benchmarks, providing significant benefits to the Arabic LLM community.

Title: DynamicControl: Adaptive Condition Selection for Improved Text-to-Image Generation

• Authors: Qingdong He, Jinlong Peng, Pengcheng Xu, Boyuan Jiang, Xiaobin Hu, Donghao Luo, Yong Liu, Yabiao Wang, Chengjie Wang, Xiangtai Li, Jiangning Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03255
• Pdf URL: https://arxiv.org/pdf/2412.03255
• Copy Paste: [[2412.03255]] DynamicControl: Adaptive Condition Selection for Improved Text-to-Image Generation(https://arxiv.org/abs/2412.03255)
• Keywords: diffusion, large language model
• Abstract: To enhance the controllability of text-to-image diffusion models, current ControlNet-like models have explored various control signals to dictate image attributes. However, existing methods either handle conditions inefficiently or use a fixed number of conditions, which does not fully address the complexity of multiple conditions and their potential conflicts. This underscores the need for innovative approaches to manage multiple conditions effectively for more reliable and detailed image synthesis. To address this issue, we propose a novel framework, DynamicControl, which supports dynamic combinations of diverse control signals, allowing adaptive selection of different numbers and types of conditions. Our approach begins with a double-cycle controller that generates an initial real score sorting for all input conditions by leveraging pre-trained conditional generation models and discriminative models. This controller evaluates the similarity between extracted conditions and input conditions, as well as the pixel-level similarity with the source image. Then, we integrate a Multimodal Large Language Model (MLLM) to build an efficient condition evaluator. This evaluator optimizes the ordering of conditions based on the double-cycle controller's score ranking. Our method jointly optimizes MLLMs and diffusion models, utilizing MLLMs' reasoning capabilities to facilitate multi-condition text-to-image (T2I) tasks. The final sorted conditions are fed into a parallel multi-control adapter, which learns feature maps from dynamic visual conditions and integrates them to modulate ControlNet, thereby enhancing control over generated images. Through both quantitative and qualitative comparisons, DynamicControl demonstrates its superiority over existing methods in terms of controllability, generation quality and composability under various conditional controls.

Title: RFSR: Improving ISR Diffusion Models via Reward Feedback Learning

• Authors: Xiaopeng Sun, Qinwei Lin, Yu Gao, Yujie Zhong, Chengjian Feng, Dengjie Li, Zheng Zhao, Jie Hu, Lin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03268
• Pdf URL: https://arxiv.org/pdf/2412.03268
• Copy Paste: [[2412.03268]] RFSR: Improving ISR Diffusion Models via Reward Feedback Learning(https://arxiv.org/abs/2412.03268)
• Keywords: diffusion, generative
• Abstract: Generative diffusion models (DM) have been extensively utilized in image super-resolution (ISR). Most of the existing methods adopt the denoising loss from DDPMs for model optimization. We posit that introducing reward feedback learning to finetune the existing models can further improve the quality of the generated images. In this paper, we propose a timestep-aware training strategy with reward feedback learning. Specifically, in the initial denoising stages of ISR diffusion, we apply low-frequency constraints to super-resolution (SR) images to maintain structural stability. In the later denoising stages, we use reward feedback learning to improve the perceptual and aesthetic quality of the SR images. In addition, we incorporate Gram-KL regularization to alleviate stylization caused by reward hacking. Our method can be integrated into any diffusion-based ISR model in a plug-and-play manner. Experiments show that ISR diffusion models, when fine-tuned with our method, significantly improve the perceptual and aesthetic quality of SR images, achieving excellent subjective results. Code: this https URL

Title: Intent-driven In-context Learning for Few-shot Dialogue State Tracking

• Authors: Zihao Yi, Zhe Xu, Ying Shen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03270
• Pdf URL: https://arxiv.org/pdf/2412.03270
• Copy Paste: [[2412.03270]] Intent-driven In-context Learning for Few-shot Dialogue State Tracking(https://arxiv.org/abs/2412.03270)
• Keywords: large language model
• Abstract: Dialogue state tracking (DST) plays an essential role in task-oriented dialogue systems. However, user's input may contain implicit information, posing significant challenges for DST tasks. Additionally, DST data includes complex information, which not only contains a large amount of noise unrelated to the current turn, but also makes constructing DST datasets expensive. To address these challenges, we introduce Intent-driven In-context Learning for Few-shot DST (IDIC-DST). By extracting user's intent, we propose an Intent-driven Dialogue Information Augmentation module to augment the dialogue information, which can track dialogue states more effectively. Moreover, we mask noisy information from DST data and rewrite user's input in the Intent-driven Examples Retrieval module, where we retrieve similar examples. We then utilize a pre-trained large language model to update the dialogue state using the augmented dialogue information and examples. Experimental results demonstrate that IDIC-DST achieves state-of-the-art performance in few-shot settings on MultiWOZ 2.1 and MultiWOZ 2.4 datasets.

Title: AntLM: Bridging Causal and Masked Language Models

• Authors: Xinru Yu, Bin Guo, Shiwei Luo, Jie Wang, Tao Ji, Yuanbin Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03275
• Pdf URL: https://arxiv.org/pdf/2412.03275
• Copy Paste: [[2412.03275]] AntLM: Bridging Causal and Masked Language Models(https://arxiv.org/abs/2412.03275)
• Keywords: transformer
• Abstract: Causal Language Modeling (CLM) and Masked Language Modeling (MLM) are two mainstream learning paradigms based on Transformer networks, specifically the Decoder-only and Encoder-only architectures. The strengths of each paradigm in downstream tasks have shown a mix of advantages and disadvantages. In the past BabyLM Challenge 2023, although the MLM paradigm achieved the best average performance, the CLM paradigm demonstrated significantly faster convergence rates. For the BabyLM Challenge 2024, we propose a novel language modeling paradigm named $\textbf{AntLM}$, which integrates both CLM and MLM to leverage the advantages of these two classic paradigms. We chose the strict-small track and conducted experiments on two foundation models: BabyLlama, representing CLM, and LTG-BERT, representing MLM. During the training process for specific foundation models, we alternate between applying CLM or MLM training objectives and causal or bidirectional attention masks. Experimental results show that combining the two pretraining objectives leverages their strengths, enhancing overall training performance. Under the same epochs, $AntLM_{BabyLlama}$ improves Macro-average by 1%, and $AntLM_{LTG-BERT}$ achieves a 2.2% increase over the baselines.

Title: EAP-FIDO: A Novel EAP Method for Using FIDO2 Credentials for Network Authentication

• Authors: Martiño Rivera-Dourado, Christos Xenakis, Alejandro Pazos, Jose Vázquez-Naya
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.03277
• Pdf URL: https://arxiv.org/pdf/2412.03277
• Copy Paste: [[2412.03277]] EAP-FIDO: A Novel EAP Method for Using FIDO2 Credentials for Network Authentication(https://arxiv.org/abs/2412.03277)
• Keywords: security, protect
• Abstract: The adoption of FIDO2 authentication by major tech companies in web applications has grown significantly in recent years. However, we argue FIDO2 has broader potential applications. In this paper, we introduce EAP-FIDO, a novel Extensible Authentication Protocol (EAP) method for use in IEEE 802.1X-protected networks. This allows organisations with WPA2/3-Enterprise wireless networks or MACSec-enabled wired networks to leverage FIDO2's passwordless authentication in compliance with existing standards. Additionally, we provide a comprehensive security and performance analysis to support the feasibility of this approach.

Title: Black-Box Forgery Attacks on Semantic Watermarks for Diffusion Models

• Authors: Andreas Müller, Denis Lukovnikov, Jonas Thietke, Asja Fischer, Erwin Quiring
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03283
• Pdf URL: https://arxiv.org/pdf/2412.03283
• Copy Paste: [[2412.03283]] Black-Box Forgery Attacks on Semantic Watermarks for Diffusion Models(https://arxiv.org/abs/2412.03283)
• Keywords: security, attack, robust, watermark, diffusion
• Abstract: Integrating watermarking into the generation process of latent diffusion models (LDMs) simplifies detection and attribution of generated content. Semantic watermarks, such as Tree-Rings and Gaussian Shading, represent a novel class of watermarking techniques that are easy to implement and highly robust against various perturbations. However, our work demonstrates a fundamental security vulnerability of semantic watermarks. We show that attackers can leverage unrelated models, even with different latent spaces and architectures (UNet vs DiT), to perform powerful and realistic forgery attacks. Specifically, we design two watermark forgery attacks. The first imprints a targeted watermark into real images by manipulating the latent representation of an arbitrary image in an unrelated LDM to get closer to the latent representation of a watermarked image. We also show that this technique can be used for watermark removal. The second attack generates new images with the target watermark by inverting a watermarked image and re-generating it with an arbitrary prompt. Both attacks just need a single reference image with the target watermark. Overall, our findings question the applicability of semantic watermarks by revealing that attackers can easily forge or remove these watermarks under realistic conditions.

Title: Composed Image Retrieval for Training-Free Domain Conversion

• Authors: Nikos Efthymiadis, Bill Psomas, Zakaria Laskar, Konstantinos Karantzalos, Yannis Avrithis, Ondřej Chum, Giorgos Tolias
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03297
• Pdf URL: https://arxiv.org/pdf/2412.03297
• Copy Paste: [[2412.03297]] Composed Image Retrieval for Training-Free Domain Conversion(https://arxiv.org/abs/2412.03297)
• Keywords: robust
• Abstract: This work addresses composed image retrieval in the context of domain conversion, where the content of a query image is retrieved in the domain specified by the query text. We show that a strong vision-language model provides sufficient descriptive power without additional training. The query image is mapped to the text input space using textual inversion. Unlike common practice that invert in the continuous space of text tokens, we use the discrete word space via a nearest-neighbor search in a text vocabulary. With this inversion, the image is softly mapped across the vocabulary and is made more robust using retrieval-based augmentation. Database images are retrieved by a weighted ensemble of text queries combining mapped words with the domain text. Our method outperforms prior art by a large margin on standard and newly introduced benchmarks. Code: this https URL

Title: Equivariant Representation Learning for Augmentation-based Self-Supervised Learning via Image Reconstruction

• Authors: Qin Wang, Kai Krajsek, Hanno Scharr
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03314
• Pdf URL: https://arxiv.org/pdf/2412.03314
• Copy Paste: [[2412.03314]] Equivariant Representation Learning for Augmentation-based Self-Supervised Learning via Image Reconstruction(https://arxiv.org/abs/2412.03314)
• Keywords: robust
• Abstract: Augmentation-based self-supervised learning methods have shown remarkable success in self-supervised visual representation learning, excelling in learning invariant features but often neglecting equivariant ones. This limitation reduces the generalizability of foundation models, particularly for downstream tasks requiring equivariance. We propose integrating an image reconstruction task as an auxiliary component in augmentation-based self-supervised learning algorithms to facilitate equivariant feature learning without additional parameters. Our method implements a cross-attention mechanism to blend features learned from two augmented views, subsequently reconstructing one of them. This approach is adaptable to various datasets and augmented-pair based learning methods. We evaluate its effectiveness on learning equivariant features through multiple linear regression tasks and downstream applications on both artificial (3DIEBench) and natural (ImageNet) datasets. Results consistently demonstrate significant improvements over standard augmentation-based self-supervised learning methods and state-of-the-art approaches, particularly excelling in scenarios involving combined augmentations. Our method enhances the learning of both invariant and equivariant features, leading to more robust and generalizable visual representations for computer vision tasks.

Title: Geometry-guided Cross-view Diffusion for One-to-many Cross-view Image Synthesis

• Authors: Tao Jun Lin, Wenqing Wang, Yujiao Shi, Akhil Perincherry, Ankit Vora, Hongdong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03315
• Pdf URL: https://arxiv.org/pdf/2412.03315
• Copy Paste: [[2412.03315]] Geometry-guided Cross-view Diffusion for One-to-many Cross-view Image Synthesis(https://arxiv.org/abs/2412.03315)
• Keywords: diffusion
• Abstract: This paper presents a novel approach for cross-view synthesis aimed at generating plausible ground-level images from corresponding satellite imagery or vice versa. We refer to these tasks as satellite-to-ground (Sat2Grd) and ground-to-satellite (Grd2Sat) synthesis, respectively. Unlike previous works that typically focus on one-to-one generation, producing a single output image from a single input image, our approach acknowledges the inherent one-to-many nature of the problem. This recognition stems from the challenges posed by differences in illumination, weather conditions, and occlusions between the two views. To effectively model this uncertainty, we leverage recent advancements in diffusion models. Specifically, we exploit random Gaussian noise to represent the diverse possibilities learnt from the target view data. We introduce a Geometry-guided Cross-view Condition (GCC) strategy to establish explicit geometric correspondences between satellite and street-view features. This enables us to resolve the geometry ambiguity introduced by camera pose between image pairs, boosting the performance of cross-view image synthesis. Through extensive quantitative and qualitative analyses on three benchmark cross-view datasets, we demonstrate the superiority of our proposed geometry-guided cross-view condition over baseline methods, including recent state-of-the-art approaches in cross-view image synthesis. Our method generates images of higher quality, fidelity, and diversity than other state-of-the-art approaches.

Title: AI-Driven Day-to-Day Route Choice

• Authors: Leizhen Wang, Peibo Duan, Zhengbing He, Cheng Lyu, Xin Chen, Nan Zheng, Li Yao, Zhenliang Ma
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03338
• Pdf URL: https://arxiv.org/pdf/2412.03338
• Copy Paste: [[2412.03338]] AI-Driven Day-to-Day Route Choice(https://arxiv.org/abs/2412.03338)
• Keywords: large language model
• Abstract: Understanding travelers' route choices can help policymakers devise optimal operational and planning strategies for both normal and abnormal circumstances. However, existing choice modeling methods often rely on predefined assumptions and struggle to capture the dynamic and adaptive nature of travel behavior. Recently, Large Language Models (LLMs) have emerged as a promising alternative, demonstrating remarkable ability to replicate human-like behaviors across various fields. Despite this potential, their capacity to accurately simulate human route choice behavior in transportation contexts remains doubtful. To satisfy this curiosity, this paper investigates the potential of LLMs for route choice modeling by introducing an LLM-empowered agent, "LLMTraveler." This agent integrates an LLM as its core, equipped with a memory system that learns from past experiences and makes decisions by balancing retrieved data and personality traits. The study systematically evaluates the LLMTraveler's ability to replicate human-like decision-making through two stages: (1) analyzing its route-switching behavior in single origin-destination (OD) pair congestion game scenarios, where it demonstrates patterns align with laboratory data but are not fully explained by traditional models, and (2) testing its capacity to model day-to-day (DTD) adaptive learning behaviors on the Ortuzar and Willumsen (OW) network, producing results comparable to Multinomial Logit (MNL) and Reinforcement Learning (RL) models. These experiments demonstrate that the framework can partially replicate human-like decision-making in route choice while providing natural language explanations for its decisions. This capability offers valuable insights for transportation policymaking, such as simulating traveler responses to new policies or changes in the network.

Title: Improving Linguistic Diversity of Large Language Models with Possibility Exploration Fine-Tuning

• Authors: Long Mai, Julie Carson-Berndsen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03343
• Pdf URL: https://arxiv.org/pdf/2412.03343
• Copy Paste: [[2412.03343]] Improving Linguistic Diversity of Large Language Models with Possibility Exploration Fine-Tuning(https://arxiv.org/abs/2412.03343)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) have made significant strides in replicating human-like abilities, there are concerns about a reduction in the linguistic diversity of their outputs. This results in the homogenization of viewpoints and perspectives, as well as the underrepresentation of specific demographic groups. Although several fine-tuning and prompting techniques have been suggested to tackle the issue, they are often tailored to specific tasks or come with a substantial increase in computational cost and latency. This makes them challenging to apply to applications that demand very low latency, such as chatbots and virtual assistants. We propose Possibility Exploration Fine-Tuning (PEFT), a task-agnostic framework that enhances the text diversity of LLMs without increasing latency or computational cost. Given the same prompt, models fine-tuned with PEFT can simultaneously generate multiple diverse responses, each corresponding with a controllable possibility number. Experiments on dialogue and story generation tasks demonstrate that PEFT significantly enhances the diversity of LLM outputs, as evidenced by lower similarity between candidate responses. Since PEFT emphasizes semantic diversity over lexical diversity, it can also notably reduce demographic bias in dialogue systems. The implementations and datasets are available in our repository: this https URL

Title: DIVE: Taming DINO for Subject-Driven Video Editing

• Authors: Yi Huang, Wei Xiong, He Zhang, Chaoqi Chen, Jianzhuang Liu, Mingfu Yan, Shifeng Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03347
• Pdf URL: https://arxiv.org/pdf/2412.03347
• Copy Paste: [[2412.03347]] DIVE: Taming DINO for Subject-Driven Video Editing(https://arxiv.org/abs/2412.03347)
• Keywords: robust, diffusion
• Abstract: Building on the success of diffusion models in image generation and editing, video editing has recently gained substantial attention. However, maintaining temporal consistency and motion alignment still remains challenging. To address these issues, this paper proposes DINO-guided Video Editing (DIVE), a framework designed to facilitate subject-driven editing in source videos conditioned on either target text prompts or reference images with specific identities. The core of DIVE lies in leveraging the powerful semantic features extracted from a pretrained DINOv2 model as implicit correspondences to guide the editing process. Specifically, to ensure temporal motion consistency, DIVE employs DINO features to align with the motion trajectory of the source video. Extensive experiments on diverse real-world videos demonstrate that our framework can achieve high-quality editing results with robust motion consistency, highlighting the potential of DINO to contribute to video editing. For precise subject editing, DIVE incorporates the DINO features of reference images into a pretrained text-to-image model to learn Low-Rank Adaptations (LoRAs), effectively registering the target subject's identity. Project page: this https URL

Title: Fairer Analysis and Demographically Balanced Face Generation for Fairer Face Verification

• Authors: Alexandre Fournier-Montgieux, Michael Soumm, Adrian Popescu, Bertrand Luvison, Hervé Le Borgne
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03349
• Pdf URL: https://arxiv.org/pdf/2412.03349
• Copy Paste: [[2412.03349]] Fairer Analysis and Demographically Balanced Face Generation for Fairer Face Verification(https://arxiv.org/abs/2412.03349)
• Keywords: privacy, fair, generative
• Abstract: Face recognition and verification are two computer vision tasks whose performances have advanced with the introduction of deep representations. However, ethical, legal, and technical challenges due to the sensitive nature of face data and biases in real-world training datasets hinder their development. Generative AI addresses privacy by creating fictitious identities, but fairness problems remain. Using the existing DCFace SOTA framework, we introduce a new controlled generation pipeline that improves fairness. Through classical fairness metrics and a proposed in-depth statistical analysis based on logit models and ANOVA, we show that our generation pipeline improves fairness more than other bias mitigation approaches while slightly improving raw performance.

Title: Intuitive Axial Augmentation Using Polar-Sine-Based Piecewise Distortion for Medical Slice-Wise Segmentation

• Authors: Yiqin Zhang, Qingkui Chen, Chen Huang, Zhengjie Zhang, Meiling Chen, Zhibing Fu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03352
• Pdf URL: https://arxiv.org/pdf/2412.03352
• Copy Paste: [[2412.03352]] Intuitive Axial Augmentation Using Polar-Sine-Based Piecewise Distortion for Medical Slice-Wise Segmentation(https://arxiv.org/abs/2412.03352)
• Keywords: robust, segmentation
• Abstract: Most data-driven models for medical image analysis rely on universal augmentations to improve performance. Experimental evidence has confirmed their effectiveness, but the unclear mechanism underlying them poses a barrier to the widespread acceptance and trust in such methods within the medical community. We revisit and acknowledge the unique characteristics of medical images apart from traditional digital images, and consequently, proposed a medical-specific augmentation algorithm that is more elastic and aligns well with radiology scan procedure. The method performs piecewise affine with sinusoidal distorted ray according to radius on polar coordinates, thus simulating uncertain postures of human lying flat on the scanning table. Our method could generate human visceral distribution without affecting the fundamental relative position on axial plane. Two non-adaptive algorithms, namely Meta-based Scan Table Removal and Similarity-Guided Parameter Search, are introduced to bolster robustness of our augmentation method. Experiments show our method improves accuracy across multiple famous segmentation frameworks without requiring more data samples. Our preview code is available in: this https URL.

Title: TASR: Timestep-Aware Diffusion Model for Image Super-Resolution

• Authors: Qinwei Lin, Xiaopeng Sun, Yu Gao, Yujie Zhong, Dengjie Li, Zheng Zhao, Haoqian Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03355
• Pdf URL: https://arxiv.org/pdf/2412.03355
• Copy Paste: [[2412.03355]] TASR: Timestep-Aware Diffusion Model for Image Super-Resolution(https://arxiv.org/abs/2412.03355)
• Keywords: diffusion
• Abstract: Diffusion models have recently achieved outstanding results in the field of image super-resolution. These methods typically inject low-resolution (LR) images via this http URL this paper, we first explore the temporal dynamics of information infusion through ControlNet, revealing that the input from LR images predominantly influences the initial stages of the denoising process. Leveraging this insight, we introduce a novel timestep-aware diffusion model that adaptively integrates features from both ControlNet and the pre-trained Stable Diffusion (SD). Our method enhances the transmission of LR information in the early stages of diffusion to guarantee image fidelity and stimulates the generation ability of the SD model itself more in the later stages to enhance the detail of generated images. To train this method, we propose a timestep-aware training strategy that adopts distinct losses at varying timesteps and acts on disparate modules. Experiments on benchmark datasets demonstrate the effectiveness of our method. Code: this https URL

Title: Granular Ball Twin Support Vector Machine with Universum Data

• Authors: M. A. Ganaie, Vrushank Ahire
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03375
• Pdf URL: https://arxiv.org/pdf/2412.03375
• Copy Paste: [[2412.03375]] Granular Ball Twin Support Vector Machine with Universum Data(https://arxiv.org/abs/2412.03375)
• Keywords: robust, interpretability
• Abstract: Classification with support vector machines (SVM) often suffers from limited performance when relying solely on labeled data from target classes and is sensitive to noise and outliers. Incorporating prior knowledge from Universum data and more robust data representations can enhance accuracy and efficiency. Motivated by these findings, we propose a novel Granular Ball Twin Support Vector Machine with Universum Data (GBU-TSVM) that extends the TSVM framework to leverage both Universum samples and granular ball computing during model training. Unlike existing TSVM methods, the proposed GBU-TSVM represents data instances as hyper-balls rather than points in the feature space. This innovative approach improves the model's robustness and efficiency, particularly in handling noisy and large datasets. By grouping data points into granular balls, the model achieves superior computational efficiency, increased noise resistance, and enhanced interpretability. Additionally, the inclusion of Universum data, which consists of samples that are not strictly from the target classes, further refines the classification boundaries. This integration enriches the model with contextual information, refining classification boundaries and boosting overall accuracy. Experimental results on UCI benchmark datasets demonstrate that the GBU-TSVM outperforms existing TSVM models in both accuracy and computational efficiency. These findings highlight the potential of the GBU-TSVM model in setting a new standard in data representation and classification.

Title: Mapping using Transformers for Volumes -- Network for Super-Resolution with Long-Range Interactions

• Authors: August Leander Høeg, Sophia W. Bardenfleth, Hans Martin Kjer, Tim B. Dyrby, Vedrana Andersen Dahl, Anders Dahl
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.03379
• Pdf URL: https://arxiv.org/pdf/2412.03379
• Copy Paste: [[2412.03379]] Mapping using Transformers for Volumes -- Network for Super-Resolution with Long-Range Interactions(https://arxiv.org/abs/2412.03379)
• Keywords: transformer
• Abstract: Until now, it has been difficult for volumetric super-resolution to utilize the recent advances in transformer-based models seen in 2D super-resolution. The memory required for self-attention in 3D volumes limits the receptive field. Therefore, long-range interactions are not used in 3D to the extent done in 2D and the strength of transformers is not realized. We propose a multi-scale transformer-based model based on hierarchical attention blocks combined with carrier tokens at multiple scales to overcome this. Here information from larger regions at coarse resolution is sequentially carried on to finer-resolution regions to predict the super-resolved image. Using transformer layers at each resolution, our coarse-to-fine modeling limits the number of tokens at each scale and enables attention over larger regions than what has previously been possible. We experimentally compare our method, MTVNet, against state-of-the-art volumetric super-resolution models on five 3D datasets demonstrating the advantage of an increased receptive field. This advantage is especially pronounced for images that are larger than what is seen in popularly used 3D datasets. Our code is available at this https URL

Title: RedStone: Curating General, Code, Math, and QA Data for Large Language Models

• Authors: Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03398
• Pdf URL: https://arxiv.org/pdf/2412.03398
• Copy Paste: [[2412.03398]] RedStone: Curating General, Code, Math, and QA Data for Large Language Models(https://arxiv.org/abs/2412.03398)
• Keywords: large language model
• Abstract: Pre-training Large Language Models (LLMs) on high-quality, meticulously curated datasets is widely recognized as critical for enhancing their performance and generalization capabilities. This study explores the untapped potential of Common Crawl as a comprehensive and flexible resource for pre-training LLMs, addressing both general-purpose language understanding and specialized domain knowledge. We introduce RedStone, an innovative and scalable pipeline engineered to extract and process data from Common Crawl, facilitating the creation of extensive and varied pre-training datasets. Unlike traditional datasets, which often require expensive curation and domain-specific expertise, RedStone leverages the breadth of Common Crawl to deliver datasets tailored to a wide array of domains. In this work, we exemplify its capability by constructing pre-training datasets across multiple fields, including general language understanding, code, mathematics, and question-answering tasks. The flexibility of RedStone allows for easy adaptation to other specialized domains, significantly lowering the barrier to creating valuable domain-specific datasets. Our findings demonstrate that Common Crawl, when harnessed through effective pipelines like RedStone, can serve as a rich, renewable source of pre-training data, unlocking new avenues for domain adaptation and knowledge discovery in LLMs. This work also underscores the importance of innovative data acquisition strategies and highlights the role of web-scale data as a powerful resource in the continued evolution of LLMs. RedStone code and data samples will be publicly available at \url{this https URL}.

Title: Implicit Priors Editing in Stable Diffusion via Targeted Token Adjustment

• Authors: Feng He, Chao Zhang, Zhixue Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03400
• Pdf URL: https://arxiv.org/pdf/2412.03400
• Copy Paste: [[2412.03400]] Implicit Priors Editing in Stable Diffusion via Targeted Token Adjustment(https://arxiv.org/abs/2412.03400)
• Keywords: diffusion
• Abstract: Implicit assumptions and priors are often necessary in text-to-image generation tasks, especially when textual prompts lack sufficient context. However, these assumptions can sometimes reflect outdated concepts, inaccuracies, or societal bias embedded in the training data. We present Embedding-only Editing (Embedit), a method designed to efficiently adjust implict assumptions and priors in the model without affecting its interpretation of unrelated objects or overall performance. Given a "source" prompt (e.g., "rose") that elicits an implicit assumption (e.g., rose is red) and a "destination" prompt that specifies the desired attribute (e.g., "blue rose"), Embedit fine-tunes only the word token embedding (WTE) of the target object ("rose") to optimize the last hidden state of text encoder in Stable Diffusion, a SOTA text-to-image model. This targeted adjustment prevents unintended effects on other objects in the model's knowledge base, as the WTEs for unrelated objects and the model weights remain unchanged. Consequently, when a prompt does not contain the edited object, all representations, and the model outputs are identical to those of the original, unedited model. Our method is highly efficient, modifying only 768 parameters for Stable Diffusion 1.4 and 2048 for XL in a single edit, matching the WTE dimension of each respective model. This minimal scope, combined with rapid execution, makes Embedit highly practical for real-world applications. Additionally, changes are easily reversible by restoring the original WTE layers. Our experimental results demonstrate that Embedit consistently outperforms previous methods across various models, tasks, and editing scenarios (both single and sequential multiple edits), achieving at least a 6.01% improvement (from 87.17% to 93.18%).

Title: Benchmarking Pretrained Attention-based Models for Real-Time Recognition in Robot-Assisted Esophagectomy

• Authors: Ronald L.P.D. de Jong, Yasmina al Khalil, Tim J.M. Jaspers, Romy C. van Jaarsveld, Gino M. Kuiper, Yiping Li, Richard van Hillegersberg, Jelle P. Ruurda, Marcel Breeuwer, Fons van der Sommen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03401
• Pdf URL: https://arxiv.org/pdf/2412.03401
• Copy Paste: [[2412.03401]] Benchmarking Pretrained Attention-based Models for Real-Time Recognition in Robot-Assisted Esophagectomy(https://arxiv.org/abs/2412.03401)
• Keywords: segmentation
• Abstract: Esophageal cancer is among the most common types of cancer worldwide. It is traditionally treated using open esophagectomy, but in recent years, robot-assisted minimally invasive esophagectomy (RAMIE) has emerged as a promising alternative. However, robot-assisted surgery can be challenging for novice surgeons, as they often suffer from a loss of spatial orientation. Computer-aided anatomy recognition holds promise for improving surgical navigation, but research in this area remains limited. In this study, we developed a comprehensive dataset for semantic segmentation in RAMIE, featuring the largest collection of vital anatomical structures and surgical instruments to date. Handling this diverse set of classes presents challenges, including class imbalance and the recognition of complex structures such as nerves. This study aims to understand the challenges and limitations of current state-of-the-art algorithms on this novel dataset and problem. Therefore, we benchmarked eight real-time deep learning models using two pretraining datasets. We assessed both traditional and attention-based networks, hypothesizing that attention-based networks better capture global patterns and address challenges such as occlusion caused by blood or other tissues. The benchmark includes our RAMIE dataset and the publicly available CholecSeg8k dataset, enabling a thorough assessment of surgical segmentation tasks. Our findings indicate that pretraining on ADE20k, a dataset for semantic segmentation, is more effective than pretraining on ImageNet. Furthermore, attention-based models outperform traditional convolutional neural networks, with SegNeXt and Mask2Former achieving higher Dice scores, and Mask2Former additionally excelling in average symmetric surface distance.

Title: Skel3D: Skeleton Guided Novel View Synthesis

• Authors: Aron Fóthi, Bence Fazekas, Natabara Máté Gyöngyössy, Kristian Fenech
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03407
• Pdf URL: https://arxiv.org/pdf/2412.03407
• Copy Paste: [[2412.03407]] Skel3D: Skeleton Guided Novel View Synthesis(https://arxiv.org/abs/2412.03407)
• Keywords: diffusion, generative
• Abstract: In this paper, we present an approach for monocular open-set novel view synthesis (NVS) that leverages object skeletons to guide the underlying diffusion model. Building upon a baseline that utilizes a pre-trained 2D image generator, our method takes advantage of the Objaverse dataset, which includes animated objects with bone structures. By introducing a skeleton guide layer following the existing ray conditioning normalization (RCN) layer, our approach enhances pose accuracy and multi-view consistency. The skeleton guide layer provides detailed structural information for the generative model, improving the quality of synthesized views. Experimental results demonstrate that our skeleton-guided method significantly enhances consistency and accuracy across diverse object categories within the Objaverse dataset. Our method outperforms existing state-of-the-art NVS techniques both quantitatively and qualitatively, without relying on explicit 3D representations.

Title: SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model

• Authors: Yan Li, Ziya Zhou, Zhiqiang Wang, Wei Xue, Wenhan Luo, Yike Guo
• Subjects: cs.CV, cs.LG, cs.SD
• Abstract URL: https://arxiv.org/abs/2412.03430
• Pdf URL: https://arxiv.org/pdf/2412.03430
• Copy Paste: [[2412.03430]] SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model(https://arxiv.org/abs/2412.03430)
• Keywords: diffusion, generative
• Abstract: Recent advancements in generative models have significantly enhanced talking face video generation, yet singing video generation remains underexplored. The differences between human talking and singing limit the performance of existing talking face video generation models when applied to singing. The fundamental differences between talking and singing-specifically in audio characteristics and behavioral expressions-limit the effectiveness of existing models. We observe that the differences between singing and talking audios manifest in terms of frequency and amplitude. To address this, we have designed a multi-scale spectral module to help the model learn singing patterns in the spectral domain. Additionally, we develop a spectral-filtering module that aids the model in learning the human behaviors associated with singing audio. These two modules are integrated into the diffusion model to enhance singing video generation performance, resulting in our proposed model, SINGER. Furthermore, the lack of high-quality real-world singing face videos has hindered the development of the singing video generation community. To address this gap, we have collected an in-the-wild audio-visual singing dataset to facilitate research in this area. Our experiments demonstrate that SINGER is capable of generating vivid singing videos and outperforms state-of-the-art methods in both objective and subjective evaluations.

Title: CleanDIFT: Diffusion Features without Noise

• Authors: Nick Stracke, Stefan Andreas Baumann, Kolja Bauer, Frank Fundel, Björn Ommer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03439
• Pdf URL: https://arxiv.org/pdf/2412.03439
• Copy Paste: [[2412.03439]] CleanDIFT: Diffusion Features without Noise(https://arxiv.org/abs/2412.03439)
• Keywords: extraction, diffusion
• Abstract: Internal features from large-scale pre-trained diffusion models have recently been established as powerful semantic descriptors for a wide range of downstream tasks. Works that use these features generally need to add noise to images before passing them through the model to obtain the semantic features, as the models do not offer the most useful features when given images with little to no noise. We show that this noise has a critical impact on the usefulness of these features that cannot be remedied by ensembling with different random noises. We address this issue by introducing a lightweight, unsupervised fine-tuning method that enables diffusion backbones to provide high-quality, noise-free semantic features. We show that these features readily outperform previous diffusion features by a wide margin in a wide variety of extraction setups and downstream tasks, offering better performance than even ensemble-based methods at a fraction of the cost.

Title: PBP: Post-training Backdoor Purification for Malware Classifiers

• Authors: Dung Thuy Nguyen, Ngoc N. Tran, Taylor T. Johnson, Kevin Leach
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.03441
• Pdf URL: https://arxiv.org/pdf/2412.03441
• Copy Paste: [[2412.03441]] PBP: Post-training Backdoor Purification for Malware Classifiers(https://arxiv.org/abs/2412.03441)
• Keywords: security, defense, attack
• Abstract: In recent years, the rise of machine learning (ML) in cybersecurity has brought new challenges, including the increasing threat of backdoor poisoning attacks on ML malware classifiers. For instance, adversaries could inject malicious samples into public malware repositories, contaminating the training data and potentially misclassifying malware by the ML model. Current countermeasures predominantly focus on detecting poisoned samples by leveraging disagreements within the outputs of a diverse set of ensemble models on training data points. However, these methods are not suitable for scenarios where Machine Learning-as-a-Service (MLaaS) is used or when users aim to remove backdoors from a model after it has been trained. Addressing this scenario, we introduce PBP, a post-training defense for malware classifiers that mitigates various types of backdoor embeddings without assuming any specific backdoor embedding mechanism. Our method exploits the influence of backdoor attacks on the activation distribution of neural networks, independent of the trigger-embedding method. In the presence of a backdoor attack, the activation distribution of each layer is distorted into a mixture of distributions. By regulating the statistics of the batch normalization layers, we can guide a backdoored model to perform similarly to a clean one. Our method demonstrates substantial advantages over several state-of-the-art methods, as evidenced by experiments on two datasets, two types of backdoor methods, and various attack configurations. Notably, our approach requires only a small portion of the training data -- only 1\% -- to purify the backdoor and reduce the attack success rate from 100\% to almost 0\%, a 100-fold improvement over the baseline methods. Our code is available at \url{this https URL}.

Title: State Frequency Estimation for Anomaly Detection

• Authors: Clinton Cao, Agathe Blaise, Annibale Panichella, Sicco Verwer
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.03442
• Pdf URL: https://arxiv.org/pdf/2412.03442
• Copy Paste: [[2412.03442]] State Frequency Estimation for Anomaly Detection(https://arxiv.org/abs/2412.03442)
• Keywords: attack
• Abstract: Many works have studied the efficacy of state machines for detecting anomalies within NetFlows. These works typically learn a model from unlabeled data and compute anomaly scores for arbitrary traces based on their likelihood of occurrence or how well they fit within the model. However, these methods do not dynamically adapt their scores based on the traces seen at test time. This becomes a problem when an adversary produces seemingly common traces in their attack, causing the model to miss the detection by assigning low anomaly scores. We propose SEQUENT, a new approach that uses the state visit frequency to adapt its scoring for anomaly detection dynamically. SEQUENT subsequently uses the scores to generate root causes for anomalies. These allow the grouping of alarms and simplify the analysis of anomalies. Our evaluation of SEQUENT on three NetFlow datasets indicates that our approach outperforms existing methods, demonstrating its effectiveness in detecting anomalies.

Title: Pre-trained Multiple Latent Variable Generative Models are good defenders against Adversarial Attacks

• Authors: Dario Serez, Marco Cristani, Alessio Del Bue, Vittorio Murino, Pietro Morerio
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03453
• Pdf URL: https://arxiv.org/pdf/2412.03453
• Copy Paste: [[2412.03453]] Pre-trained Multiple Latent Variable Generative Models are good defenders against Adversarial Attacks(https://arxiv.org/abs/2412.03453)
• Keywords: attack, generative
• Abstract: Attackers can deliberately perturb classifiers' input with subtle noise, altering final predictions. Among proposed countermeasures, adversarial purification employs generative networks to preprocess input images, filtering out adversarial noise. In this study, we propose specific generators, defined Multiple Latent Variable Generative Models (MLVGMs), for adversarial purification. These models possess multiple latent variables that naturally disentangle coarse from fine features. Taking advantage of these properties, we autoencode images to maintain class-relevant information, while discarding and re-sampling any detail, including adversarial noise. The procedure is completely training-free, exploring the generalization abilities of pre-trained MLVGMs on the adversarial purification downstream task. Despite the lack of large models, trained on billions of samples, we show that smaller MLVGMs are already competitive with traditional methods, and can be used as foundation models. Official code released at this https URL.

Title: Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning

• Authors: Neale Ratzlaff, Man Luo, Xin Su, Vasudev Lal, Phillip Howard
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03467
• Pdf URL: https://arxiv.org/pdf/2412.03467
• Copy Paste: [[2412.03467]] Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning(https://arxiv.org/abs/2412.03467)
• Keywords: large language model
• Abstract: Multimodal models typically combine a powerful large language model (LLM) with a vision encoder and are then trained on multimodal data via instruction tuning. While this process adapts LLMs to multimodal settings, it remains unclear whether this adaptation compromises their original language reasoning capabilities. In this work, we explore the effects of multimodal instruction tuning on language reasoning performance. We focus on LLaVA, a leading multimodal framework that integrates LLMs such as Vicuna or Mistral with the CLIP vision encoder. We compare the performance of the original LLMs with their multimodal-adapted counterparts across eight language reasoning tasks. Our experiments yield several key insights. First, the impact of multimodal learning varies between Vicuna and Mistral: we observe a degradation in language reasoning for Mistral but improvements for Vicuna across most tasks. Second, while multimodal instruction learning consistently degrades performance on mathematical reasoning tasks (e.g., GSM8K), it enhances performance on commonsense reasoning tasks (e.g., CommonsenseQA). Finally, we demonstrate that a training-free model merging technique can effectively mitigate the language reasoning degradation observed in multimodal-adapted Mistral and even improve performance on visual tasks.

Title: Measure Anything: Real-time, Multi-stage Vision-based Dimensional Measurement using Segment Anything

• Authors: Yongkyu Lee, Shivam Kumar Panda, Wei Wang, Mohammad Khalid Jawed
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03472
• Pdf URL: https://arxiv.org/pdf/2412.03472
• Copy Paste: [[2412.03472]] Measure Anything: Real-time, Multi-stage Vision-based Dimensional Measurement using Segment Anything(https://arxiv.org/abs/2412.03472)
• Keywords: segmentation
• Abstract: We present Measure Anything, a comprehensive vision-based framework for dimensional measurement of objects with circular cross-sections, leveraging the Segment Anything Model (SAM). Our approach estimates key geometric features -- including diameter, length, and volume -- for rod-like geometries with varying curvature and general objects with constant skeleton slope. The framework integrates segmentation, mask processing, skeleton construction, and 2D-3D transformation, packaged in a user-friendly interface. We validate our framework by estimating the diameters of Canola stems -- collected from agricultural fields in North Dakota -- which are thin and non-uniform, posing challenges for existing methods. Measuring its diameters is critical, as it is a phenotypic traits that correlates with the health and yield of Canola crops. This application also exemplifies the potential of Measure Anything, where integrating intelligent models -- such as keypoint detection -- extends its scalability to fully automate the measurement process for high-throughput applications. Furthermore, we showcase its versatility in robotic grasping, leveraging extracted geometric features to identify optimal grasp points.

Title: Convolutional Neural Networks and Mixture of Experts for Intrusion Detection in 5G Networks and beyond

• Authors: Loukas Ilias, George Doukas, Vangelis Lamprou, Christos Ntanos, Dimitris Askounis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03483
• Pdf URL: https://arxiv.org/pdf/2412.03483
• Copy Paste: [[2412.03483]] Convolutional Neural Networks and Mixture of Experts for Intrusion Detection in 5G Networks and beyond(https://arxiv.org/abs/2412.03483)
• Keywords: security, attack
• Abstract: The advent of 6G/NextG networks comes along with a series of benefits, including extreme capacity, reliability, and efficiency. However, these networks may become vulnerable to new security threats. Therefore, 6G/NextG networks must be equipped with advanced Artificial Intelligence algorithms, in order to evade these attacks. Existing studies on the intrusion detection task rely on the train of shallow machine learning classifiers, including Logistic Regression, Decision Trees, and so on, yielding suboptimal performance. Others are based on deep neural networks consisting of static components, which are not conditional on the input. This limits their representation power and efficiency. To resolve these issues, we present the first study integrating Mixture of Experts (MoE) for identifying malicious traffic. Specifically, we use network traffic data and convert the 1D array of features into a 2D matrix. Next, we pass this matrix through convolutional neural network (CNN) layers followed by batch normalization and max pooling layers. After obtaining the representation vector via the CNN layers, a sparsely gated MoE layer is used. This layer consists of a set of experts (dense layers) and a router, where the router assigns weights to the output of each expert. Sparsity is achieved by choosing the most relevant experts of the total ones. Finally, we perform a series of ablation experiments to prove the effectiveness of our proposed model. Experiments are conducted on the 5G-NIDD dataset, a network intrusion detection dataset generated from a real 5G test network. Results show that our introduced approach reaches weighted F1-score up to 99.95% achieving comparable performance to existing approaches. Findings also show that our proposed model achieves multiple advantages over state-of-the-art approaches.

Title: Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective

• Authors: Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, Ricky T. Q. Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03487
• Pdf URL: https://arxiv.org/pdf/2412.03487
• Copy Paste: [[2412.03487]] Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspective(https://arxiv.org/abs/2412.03487)
• Keywords: diffusion, generative
• Abstract: The design space of discrete-space diffusion or flow generative models are significantly less well-understood than their continuous-space counterparts, with many works focusing only on a simple masked construction. In this work, we aim to take a holistic approach to the construction of discrete generative models based on continuous-time Markov chains, and for the first time, allow the use of arbitrary discrete probability paths, or colloquially, corruption processes. Through the lens of optimizing the symmetric kinetic energy, we propose velocity formulas that can be applied to any given probability path, completely decoupling the probability and velocity, and giving the user the freedom to specify any desirable probability path based on expert knowledge specific to the data domain. Furthermore, we find that a special construction of mixture probability paths optimizes the symmetric kinetic energy for the discrete case. We empirically validate the usefulness of this new design space across multiple modalities: text generation, inorganic material generation, and image generation. We find that we can outperform the mask construction even in text with kinetic-optimal mixture paths, while we can make use of domain-specific constructions of the probability path over the visual domain.

Title: A Bidirectional Siamese Recurrent Neural Network for Accurate Gait Recognition Using Body Landmarks

• Authors: Proma Hossain Progga, Md. Jobayer Rahman, Swapnil Biswas, Md. Shakil Ahmed, Arif Reza Anwary, Swakkhar Shatabda
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03498
• Pdf URL: https://arxiv.org/pdf/2412.03498
• Copy Paste: [[2412.03498]] A Bidirectional Siamese Recurrent Neural Network for Accurate Gait Recognition Using Body Landmarks(https://arxiv.org/abs/2412.03498)
• Keywords: biometric
• Abstract: Gait recognition is a significant biometric technique for person identification, particularly in scenarios where other physiological biometrics are impractical or ineffective. In this paper, we address the challenges associated with gait recognition and present a novel approach to improve its accuracy and reliability. The proposed method leverages advanced techniques, including sequential gait landmarks obtained through the Mediapipe pose estimation model, Procrustes analysis for alignment, and a Siamese biGRU-dualStack Neural Network architecture for capturing temporal dependencies. Extensive experiments were conducted on large-scale cross-view datasets to demonstrate the effectiveness of the approach, achieving high recognition accuracy compared to other models. The model demonstrated accuracies of 95.7%, 94.44%, 87.71%, and 86.6% on CASIA-B, SZU RGB-D, OU-MVLP, and Gait3D datasets respectively. The results highlight the potential applications of the proposed method in various practical domains, indicating its significant contribution to the field of gait recognition.

Title: Distillation of Diffusion Features for Semantic Correspondence

• Authors: Frank Fundel, Johannes Schusterbauer, Vincent Tao Hu, Björn Ommer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03512
• Pdf URL: https://arxiv.org/pdf/2412.03512
• Copy Paste: [[2412.03512]] Distillation of Diffusion Features for Semantic Correspondence(https://arxiv.org/abs/2412.03512)
• Keywords: diffusion, generative
• Abstract: Semantic correspondence, the task of determining relationships between different parts of images, underpins various applications including 3D reconstruction, image-to-image translation, object tracking, and visual place recognition. Recent studies have begun to explore representations learned in large generative image models for semantic correspondence, demonstrating promising results. Building on this progress, current state-of-the-art methods rely on combining multiple large models, resulting in high computational demands and reduced efficiency. In this work, we address this challenge by proposing a more computationally efficient approach. We propose a novel knowledge distillation technique to overcome the problem of reduced efficiency. We show how to use two large vision foundation models and distill the capabilities of these complementary models into one smaller model that maintains high accuracy at reduced computational cost. Furthermore, we demonstrate that by incorporating 3D data, we are able to further improve performance, without the need for human-annotated correspondences. Overall, our empirical results demonstrate that our distilled model with 3D data augmentation achieves performance superior to current state-of-the-art methods while significantly reducing computational load and enhancing practicality for real-world applications, such as semantic video correspondence. Our code and weights are publicly available on our project page.

Title: Distilling Diffusion Models to Efficient 3D LiDAR Scene Completion

• Authors: Shengyuan Zhang, An Zhao, Ling Yang, Zejian Li, Chenye Meng, Haoran Xu, Tianrun Chen, AnYang Wei, Perry Pengyun GU, Lingyun Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03515
• Pdf URL: https://arxiv.org/pdf/2412.03515
• Copy Paste: [[2412.03515]] Distilling Diffusion Models to Efficient 3D LiDAR Scene Completion(https://arxiv.org/abs/2412.03515)
• Keywords: diffusion
• Abstract: Diffusion models have been applied to 3D LiDAR scene completion due to their strong training stability and high completion quality. However, the slow sampling speed limits the practical application of diffusion-based scene completion models since autonomous vehicles require an efficient perception of surrounding environments. This paper proposes a novel distillation method tailored for 3D LiDAR scene completion models, dubbed $\textbf{ScoreLiDAR}$, which achieves efficient yet high-quality scene completion. ScoreLiDAR enables the distilled model to sample in significantly fewer steps after distillation. To improve completion quality, we also introduce a novel $\textbf{Structural Loss}$, which encourages the distilled model to capture the geometric structure of the 3D LiDAR scene. The loss contains a scene-wise term constraining the holistic structure and a point-wise term constraining the key landmark points and their relative configuration. Extensive experiments demonstrate that ScoreLiDAR significantly accelerates the completion time from 30.55 to 5.37 seconds per frame ($>$5$\times$) on SemanticKITTI and achieves superior performance compared to state-of-the-art 3D LiDAR scene completion models. Our code is publicly available at this https URL.

Title: NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images

• Authors: Lingen Li, Zhaoyang Zhang, Yaowei Li, Jiale Xu, Xiaoyu Li, Wenbo Hu, Weihao Cheng, Jinwei Gu, Tianfan Xue, Ying Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03517
• Pdf URL: https://arxiv.org/pdf/2412.03517
• Copy Paste: [[2412.03517]] NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images(https://arxiv.org/abs/2412.03517)
• Keywords: diffusion, generative
• Abstract: Recent advancements in generative models have significantly improved novel view synthesis (NVS) from multi-view data. However, existing methods depend on external multi-view alignment processes, such as explicit pose estimation or pre-reconstruction, which limits their flexibility and accessibility, especially when alignment is unstable due to insufficient overlap or occlusions between views. In this paper, we propose NVComposer, a novel approach that eliminates the need for explicit external alignment. NVComposer enables the generative model to implicitly infer spatial and geometric relationships between multiple conditional views by introducing two key components: 1) an image-pose dual-stream diffusion model that simultaneously generates target novel views and condition camera poses, and 2) a geometry-aware feature alignment module that distills geometric priors from dense stereo models during training. Extensive experiments demonstrate that NVComposer achieves state-of-the-art performance in generative multi-view NVS tasks, removing the reliance on external alignment and thus improving model accessibility. Our approach shows substantial improvements in synthesis quality as the number of unposed input views increases, highlighting its potential for more flexible and accessible generative NVS systems.

Title: Seeing Beyond Views: Multi-View Driving Scene Video Generation with Holistic Attention

• Authors: Hannan Lu, Xiaohe Wu, Shudong Wang, Xiameng Qin, Xinyu Zhang, Junyu Han, Wangmeng Zuo, Ji Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03520
• Pdf URL: https://arxiv.org/pdf/2412.03520
• Copy Paste: [[2412.03520]] Seeing Beyond Views: Multi-View Driving Scene Video Generation with Holistic Attention(https://arxiv.org/abs/2412.03520)
• Keywords: diffusion, transformer
• Abstract: Generating multi-view videos for autonomous driving training has recently gained much attention, with the challenge of addressing both cross-view and cross-frame consistency. Existing methods typically apply decoupled attention mechanisms for spatial, temporal, and view dimensions. However, these approaches often struggle to maintain consistency across dimensions, particularly when handling fast-moving objects that appear at different times and viewpoints. In this paper, we present CogDriving, a novel network designed for synthesizing high-quality multi-view driving videos. CogDriving leverages a Diffusion Transformer architecture with holistic-4D attention modules, enabling simultaneous associations across the spatial, temporal, and viewpoint dimensions. We also propose a lightweight controller tailored for CogDriving, i.e., Micro-Controller, which uses only 1.1% of the parameters of the standard ControlNet, enabling precise control over Bird's-Eye-View layouts. To enhance the generation of object instances crucial for autonomous driving, we propose a re-weighted learning objective, dynamically adjusting the learning weights for object instances during training. CogDriving demonstrates strong performance on the nuScenes validation set, achieving an FVD score of 37.8, highlighting its ability to generate realistic driving videos. The project can be found at this https URL.

Title: FANAL -- Financial Activity News Alerting Language Modeling Framework

• Authors: Urjitkumar Patel, Fang-Chun Yeh, Chinmay Gondhalekar, Hari Nalluri
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03527
• Pdf URL: https://arxiv.org/pdf/2412.03527
• Copy Paste: [[2412.03527]] FANAL -- Financial Activity News Alerting Language Modeling Framework(https://arxiv.org/abs/2412.03527)
• Keywords: large language model
• Abstract: In the rapidly evolving financial sector, the accurate and timely interpretation of market news is essential for stakeholders needing to navigate unpredictable events. This paper introduces FANAL (Financial Activity News Alerting Language Modeling Framework), a specialized BERT-based framework engineered for real-time financial event detection and analysis, categorizing news into twelve distinct financial categories. FANAL leverages silver-labeled data processed through XGBoost and employs advanced fine-tuning techniques, alongside ORBERT (Odds Ratio BERT), a novel variant of BERT fine-tuned with ORPO (Odds Ratio Preference Optimization) for superior class-wise probability calibration and alignment with financial event relevance. We evaluate FANAL's performance against leading large language models, including GPT-4o, Llama-3.1 8B, and Phi-3, demonstrating its superior accuracy and cost efficiency. This framework sets a new standard for financial intelligence and responsiveness, significantly outstripping existing models in both performance and affordability.

Title: A Review on Scientific Knowledge Extraction using Large Language Models in Biomedical Sciences

• Authors: Gabriel Lino Garcia, João Renato Ribeiro Manesco, Pedro Henrique Paiola, Lucas Miranda, Maria Paola de Salvo, João Paulo Papa
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03531
• Pdf URL: https://arxiv.org/pdf/2412.03531
• Copy Paste: [[2412.03531]] A Review on Scientific Knowledge Extraction using Large Language Models in Biomedical Sciences(https://arxiv.org/abs/2412.03531)
• Keywords: extraction, large language model
• Abstract: The rapid advancement of large language models (LLMs) has opened new boundaries in the extraction and synthesis of medical knowledge, particularly within evidence synthesis. This paper reviews the state-of-the-art applications of LLMs in the biomedical domain, exploring their effectiveness in automating complex tasks such as evidence synthesis and data extraction from a biomedical corpus of documents. While LLMs demonstrate remarkable potential, significant challenges remain, including issues related to hallucinations, contextual understanding, and the ability to generalize across diverse medical tasks. We highlight critical gaps in the current research literature, particularly the need for unified benchmarks to standardize evaluations and ensure reliability in real-world applications. In addition, we propose directions for future research, emphasizing the integration of state-of-the-art techniques such as retrieval-augmented generation (RAG) to enhance LLM performance in evidence synthesis. By addressing these challenges and utilizing the strengths of LLMs, we aim to improve access to medical literature and facilitate meaningful discoveries in healthcare.

Title: Evaluating Gender Bias Transfer between Pre-trained and Prompt-Adapted Language Models

• Authors: Natalie Mackraz, Nivedha Sivakumar, Samira Khorshidi, Krishna Patel, Barry-John Theobald, Luca Zappella, Nicholas Apostoloff
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03537
• Pdf URL: https://arxiv.org/pdf/2412.03537
• Copy Paste: [[2412.03537]] Evaluating Gender Bias Transfer between Pre-trained and Prompt-Adapted Language Models(https://arxiv.org/abs/2412.03537)
• Keywords: fair, large language model
• Abstract: Large language models (LLMs) are increasingly being adapted to achieve task-specificity for deployment in real-world decision systems. Several previous works have investigated the bias transfer hypothesis (BTH) by studying the effect of the fine-tuning adaptation strategy on model fairness to find that fairness in pre-trained masked language models have limited effect on the fairness of models when adapted using fine-tuning. In this work, we expand the study of BTH to causal models under prompt adaptations, as prompting is an accessible, and compute-efficient way to deploy models in real-world systems. In contrast to previous works, we establish that intrinsic biases in pre-trained Mistral, Falcon and Llama models are strongly correlated (rho >= 0.94) with biases when the same models are zero- and few-shot prompted, using a pronoun co-reference resolution task. Further, we find that bias transfer remains strongly correlated even when LLMs are specifically prompted to exhibit fair or biased behavior (rho >= 0.92), and few-shot length and stereotypical composition are varied (rho >= 0.97). Our findings highlight the importance of ensuring fairness in pre-trained LLMs, especially when they are later used to perform downstream tasks via prompt adaptation.

Title: NODE-AdvGAN: Improving the transferability and perceptual similarity of adversarial examples by dynamic-system-driven adversarial generative model

• Authors: Xinheng Xie, Yue Wu, Cuiyu He
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03539
• Pdf URL: https://arxiv.org/pdf/2412.03539
• Copy Paste: [[2412.03539]] NODE-AdvGAN: Improving the transferability and perceptual similarity of adversarial examples by dynamic-system-driven adversarial generative model(https://arxiv.org/abs/2412.03539)
• Keywords: attack, robust, generative
• Abstract: Understanding adversarial examples is crucial for improving the model's robustness, as they introduce imperceptible perturbations that deceive models. Effective adversarial examples, therefore, offer the potential to train more robust models by removing their singularities. We propose NODE-AdvGAN, a novel approach that treats adversarial generation as a continuous process and employs a Neural Ordinary Differential Equation (NODE) for simulating the dynamics of the generator. By mimicking the iterative nature of traditional gradient-based methods, NODE-AdvGAN generates smoother and more precise perturbations that preserve high perceptual similarity when added to benign images. We also propose a new training strategy, NODE-AdvGAN-T, which enhances transferability in black-box attacks by effectively tuning noise parameters during training. Experiments demonstrate that NODE-AdvGAN and NODE-AdvGAN-T generate more effective adversarial examples that achieve higher attack success rates while preserving better perceptual quality than traditional GAN-based methods.

Title: Teaching an Old Dog New Tricks: Verifiable FHE Using Commodity Hardware

• Authors: Jules Drean, Fisher Jepsen, Edward Suh, Srini Devadas, Aamer Jaleel, Gururaj Saileshwar
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.03550
• Pdf URL: https://arxiv.org/pdf/2412.03550
• Copy Paste: [[2412.03550]] Teaching an Old Dog New Tricks: Verifiable FHE Using Commodity Hardware(https://arxiv.org/abs/2412.03550)
• Keywords: secure, security, attack
• Abstract: We present Argos, a simple approach for adding verifiability to fully homomorphic encryption (FHE) schemes using trusted hardware. Traditional approaches to verifiable FHE require expensive cryptographic proofs, which incur an overhead of up to seven orders of magnitude on top of FHE, making them impractical. With Argos, we show that trusted hardware can be securely used to provide verifiability for FHE computations, with minimal overhead relative to the baseline FHE computation. An important contribution of Argos is showing that the major security pitfall associated with trusted hardware, microarchitectural side channels, can be completely mitigated by excluding any secrets from the CPU and the memory hierarchy. This is made possible by focusing on building a platform that only enforces program and data integrity and not confidentiality (which is sufficient for verifiable FHE, since all data remain encrypted at all times). All secrets related to the attestation mechanism are kept in a separate coprocessor (e.g., a TPM) inaccessible to any software-based attacker. Relying on a discrete TPM typically incurs significant performance overhead, which is why (insecure) software-based TPMs are used in practice. As a second contribution, we show that for FHE applications, the attestation protocol can be adapted to only incur a fixed cost. Argos requires no dedicated hardware extensions and is supported on commodity processors from 2008 onward. Our prototype implementation introduces 6% overhead to the FHE evaluation, and 8% for more complex protocols. In particular, we show that Argos can be adapted for real-world applications of FHE, such as PIR and PSI. By demonstrating how to combine cryptography with trusted hardware, Argos paves the way for widespread deployment of FHE-based protocols beyond the semi-honest setting, without the overhead of cryptographic proofs.

Title: Best-of-N Jailbreaking

• Authors: John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, Mrinank Sharma
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03556
• Pdf URL: https://arxiv.org/pdf/2412.03556
• Copy Paste: [[2412.03556]] Best-of-N Jailbreaking(https://arxiv.org/abs/2412.03556)
• Keywords: defense, attack
• Abstract: We introduce Best-of-N (BoN) Jailbreaking, a simple black-box algorithm that jailbreaks frontier AI systems across modalities. BoN Jailbreaking works by repeatedly sampling variations of a prompt with a combination of augmentations - such as random shuffling or capitalization for textual prompts - until a harmful response is elicited. We find that BoN Jailbreaking achieves high attack success rates (ASRs) on closed-source language models, such as 89% on GPT-4o and 78% on Claude 3.5 Sonnet when sampling 10,000 augmented prompts. Further, it is similarly effective at circumventing state-of-the-art open-source defenses like circuit breakers. BoN also seamlessly extends to other modalities: it jailbreaks vision language models (VLMs) such as GPT-4o and audio language models (ALMs) like Gemini 1.5 Pro, using modality-specific augmentations. BoN reliably improves when we sample more augmented prompts. Across all modalities, ASR, as a function of the number of samples (N), empirically follows power-law-like behavior for many orders of magnitude. BoN Jailbreaking can also be composed with other black-box algorithms for even more effective attacks - combining BoN with an optimized prefix attack achieves up to a 35% increase in ASR. Overall, our work indicates that, despite their capability, language models are sensitive to seemingly innocuous changes to inputs, which attackers can exploit across modalities.

Title: MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation

• Authors: Zehuan Huang, Yuan-Chen Guo, Xingqiao An, Yunhan Yang, Yangguang Li, Zi-Xin Zou, Ding Liang, Xihui Liu, Yan-Pei Cao, Lu Sheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03558
• Pdf URL: https://arxiv.org/pdf/2412.03558
• Copy Paste: [[2412.03558]] MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation(https://arxiv.org/abs/2412.03558)
• Keywords: diffusion
• Abstract: This paper introduces MIDI, a novel paradigm for compositional 3D scene generation from a single image. Unlike existing methods that rely on reconstruction or retrieval techniques or recent approaches that employ multi-stage object-by-object generation, MIDI extends pre-trained image-to-3D object generation models to multi-instance diffusion models, enabling the simultaneous generation of multiple 3D instances with accurate spatial relationships and high generalizability. At its core, MIDI incorporates a novel multi-instance attention mechanism, that effectively captures inter-object interactions and spatial coherence directly within the generation process, without the need for complex multi-step processes. The method utilizes partial object images and global scene context as inputs, directly modeling object completion during 3D generation. During training, we effectively supervise the interactions between 3D instances using a limited amount of scene-level data, while incorporating single-object data for regularization, thereby maintaining the pre-trained generalization ability. MIDI demonstrates state-of-the-art performance in image-to-scene generation, validated through evaluations on synthetic data, real-world scene data, and stylized scene images generated by text-to-image diffusion models.

Title: FLAIR: VLM with Fine-grained Language-informed Image Representations

• Authors: Rui Xiao, Sanghwan Kim, Mariana-Iuliana Georgescu, Zeynep Akata, Stephan Alaniz
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03561
• Pdf URL: https://arxiv.org/pdf/2412.03561
• Copy Paste: [[2412.03561]] FLAIR: VLM with Fine-grained Language-informed Image Representations(https://arxiv.org/abs/2412.03561)
• Keywords: segmentation
• Abstract: CLIP has shown impressive results in aligning images and texts at scale. However, its ability to capture detailed visual features remains limited because CLIP matches images and texts at a global level. To address this issue, we propose FLAIR, Fine-grained Language-informed Image Representations, an approach that utilizes long and detailed image descriptions to learn localized image embeddings. By sampling diverse sub-captions that describe fine-grained details about an image, we train our vision-language model to produce not only global embeddings but also text-specific image representations. Our model introduces text-conditioned attention pooling on top of local image tokens to produce fine-grained image representations that excel at retrieving detailed image content. We achieve state-of-the-art performance on both, existing multimodal retrieval benchmarks, as well as, our newly introduced fine-grained retrieval task which evaluates vision-language models' ability to retrieve partial image content. Furthermore, our experiments demonstrate the effectiveness of FLAIR trained on 30M image-text pairs in capturing fine-grained visual information, including zero-shot semantic segmentation, outperforming models trained on billions of pairs. Code is available at this https URL .

Title: From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents

• Authors: Xinyi Mou, Xuanwen Ding, Qi He, Liang Wang, Jingcong Liang, Xinnong Zhang, Libo Sun, Jiayu Lin, Jie Zhou, Xuanjing Huang, Zhongyu Wei
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.03563
• Pdf URL: https://arxiv.org/pdf/2412.03563
• Copy Paste: [[2412.03563]] From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents(https://arxiv.org/abs/2412.03563)
• Keywords: large language model
• Abstract: Traditional sociological research often relies on human participation, which, though effective, is expensive, challenging to scale, and with ethical concerns. Recent advancements in large language models (LLMs) highlight their potential to simulate human behavior, enabling the replication of individual responses and facilitating studies on many interdisciplinary studies. In this paper, we conduct a comprehensive survey of this field, illustrating the recent progress in simulation driven by LLM-empowered agents. We categorize the simulations into three types: (1) Individual Simulation, which mimics specific individuals or demographic groups; (2) Scenario Simulation, where multiple agents collaborate to achieve goals within specific contexts; and (3) Society Simulation, which models interactions within agent societies to reflect the complexity and variety of real-world dynamics. These simulations follow a progression, ranging from detailed individual modeling to large-scale societal phenomena. We provide a detailed discussion of each simulation type, including the architecture or key components of the simulation, the classification of objectives or scenarios and the evaluation method. Afterward, we summarize commonly used datasets and benchmarks. Finally, we discuss the trends across these three types of simulation. A repository for the related sources is at {\url{this https URL}}.

Title: FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenes

• Authors: Lue Fan, Hao Zhang, Qitai Wang, Hongsheng Li, Zhaoxiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03566
• Pdf URL: https://arxiv.org/pdf/2412.03566
• Copy Paste: [[2412.03566]] FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenes(https://arxiv.org/abs/2412.03566)
• Keywords: generative
• Abstract: We propose FreeSim, a camera simulation method for autonomous driving. FreeSim emphasizes high-quality rendering from viewpoints beyond the recorded ego trajectories. In such viewpoints, previous methods have unacceptable degradation because the training data of these viewpoints is unavailable. To address such data scarcity, we first propose a generative enhancement model with a matched data construction strategy. The resulting model can generate high-quality images in a viewpoint slightly deviated from the recorded trajectories, conditioned on the degraded rendering of this viewpoint. We then propose a progressive reconstruction strategy, which progressively adds generated images of unrecorded views into the reconstruction process, starting from slightly off-trajectory viewpoints and moving progressively farther away. With this progressive generation-reconstruction pipeline, FreeSim supports high-quality off-trajectory view synthesis under large deviations of more than 3 meters.

Title: Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis

• Authors: Qitao Zhao, Shubham Tulsiani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.03570
• Pdf URL: https://arxiv.org/pdf/2412.03570
• Copy Paste: [[2412.03570]] Sparse-view Pose Estimation and Reconstruction via Analysis by Generative Synthesis(https://arxiv.org/abs/2412.03570)
• Keywords: robust, generative
• Abstract: Inferring the 3D structure underlying a set of multi-view images typically requires solving two co-dependent tasks -- accurate 3D reconstruction requires precise camera poses, and predicting camera poses relies on (implicitly or explicitly) modeling the underlying 3D. The classical framework of analysis by synthesis casts this inference as a joint optimization seeking to explain the observed pixels, and recent instantiations learn expressive 3D representations (e.g., Neural Fields) with gradient-descent-based pose refinement of initial pose estimates. However, given a sparse set of observed views, the observations may not provide sufficient direct evidence to obtain complete and accurate 3D. Moreover, large errors in pose estimation may not be easily corrected and can further degrade the inferred 3D. To allow robust 3D reconstruction and pose estimation in this challenging setup, we propose SparseAGS, a method that adapts this analysis-by-synthesis approach by: a) including novel-view-synthesis-based generative priors in conjunction with photometric objectives to improve the quality of the inferred 3D, and b) explicitly reasoning about outliers and using a discrete search with a continuous optimization-based strategy to correct them. We validate our framework across real-world and synthetic datasets in combination with several off-the-shelf pose estimation systems as initialization. We find that it significantly improves the base systems' pose accuracy while yielding high-quality 3D reconstructions that outperform the results from current multi-view reconstruction baselines.

Title: Navigation World Models

• Authors: Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, Yann LeCun
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.03572
• Pdf URL: https://arxiv.org/pdf/2412.03572
• Copy Paste: [[2412.03572]] Navigation World Models(https://arxiv.org/abs/2412.03572)
• Keywords: diffusion, transformer
• Abstract: Navigation is a fundamental skill of agents with visual-motor capabilities. We introduce a Navigation World Model (NWM), a controllable video generation model that predicts future visual observations based on past observations and navigation actions. To capture complex environment dynamics, NWM employs a Conditional Diffusion Transformer (CDiT), trained on a diverse collection of egocentric videos of both human and robotic agents, and scaled up to 1 billion parameters. In familiar environments, NWM can plan navigation trajectories by simulating them and evaluating whether they achieve the desired goal. Unlike supervised navigation policies with fixed behavior, NWM can dynamically incorporate constraints during planning. Experiments demonstrate its effectiveness in planning trajectories from scratch or by ranking trajectories sampled from an external policy. Furthermore, NWM leverages its learned visual priors to imagine trajectories in unfamiliar environments from a single input image, making it a flexible and powerful tool for next-generation navigation systems.