2024-12-25

Title: Inductive Linguistic Reasoning with Large Language Models

• Authors: Raghav Ramji, Keshav Ramji
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17819
• Pdf URL: https://arxiv.org/pdf/2412.17819
• Copy Paste: [[2412.17819]] Inductive Linguistic Reasoning with Large Language Models(https://arxiv.org/abs/2412.17819)
• Keywords: large language model
• Abstract: Evaluating large language models (LLMs) on their linguistic reasoning capabilities is an important task to understand the gaps in their skills that may surface during large-scale adoption. In this work, we investigate the abilities of such models to perform abstract multilingual reasoning through the lens of linguistic puzzles on extremely low-resource languages. As these translation tasks involve inductive and deductive reasoning from reference instances, we examine whether diverse auxiliary demonstrations can be automatically induced from seed exemplars, through analogical prompting. We employ a two-stage procedure, first generating analogical exemplars with a language model, and then applying them in-context along with provided target language exemplars. Our results on the modeLing dataset show that analogical prompting is effective in eliciting models' knowledge of language grammar similarities, boosting the performance of GPT-4o by as much as 8.1% and Llama-3.1-405B-Instruct by 5.9% over chain-of-thought approaches. These gains are attributable to the analogical demonstrations, both when self-generated as well as when produced by weaker multilingual models. Furthermore, we demonstrate that our method generalizes to other tasks present in Linguistics Olympiad competitions, achieving sizable improvements across all problem types and difficulty levels included in the LINGOLY dataset with GPT-4o. We also report several findings about interesting phenomena which drive linguistic reasoning performance, suggesting that such puzzles are a valuable benchmark for new reasoning methods.

Title: The Rosetta Paradox: Domain-Specific Performance Inversions in Large Language Models

• Authors: Basab Jha, Ujjwal Puri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17821
• Pdf URL: https://arxiv.org/pdf/2412.17821
• Copy Paste: [[2412.17821]] The Rosetta Paradox: Domain-Specific Performance Inversions in Large Language Models(https://arxiv.org/abs/2412.17821)
• Keywords: large language model
• Abstract: While large language models, such as GPT and BERT, have already demonstrated unprecedented skills in everything from natural language processing to domain-specific applications, there came an unexplored phenomenon we term the Rosetta Paradox. The Rosetta Paradox characterizes the counterintuitive performance inversions across domains of knowledge. This paradox captures how such LLMs can excel in highly specialized fields but do poorly on tasks which require general, everyday knowledge. This paper formalizes the definition of the Rosetta Paradox and introduces a panoramic analysis framework that includes both a Domain Specificity Index (DSI) and a Performance Inversion Metric (PIM) for consistent quantification of domain-specific behavior in LLMs. We adopt this paradox and conduct a series of investigations through extensive experiments across diverse models and knowledge domains, ranging from rich technical areas to common-sense reasoning. Our findings indicate that the Rosetta Paradox is likely not a mere artifact of data distribution but an intrinsic architectural and emergent property of deep neural networks. We present comparative analyses across different model architectures, sizes, and training methodologies that shed light into the peculiar ways this paradox manifests itself and challenge the standard evaluation metrics.

Title: Look Ahead Text Understanding and LLM Stitching

• Authors: Junlin Julian Jiang (Piedmont High School, Piedmont, CA, USA), Xin Li (College of Business, City University of Hong Kong, Hong Kong, China)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17836
• Pdf URL: https://arxiv.org/pdf/2412.17836
• Copy Paste: [[2412.17836]] Look Ahead Text Understanding and LLM Stitching(https://arxiv.org/abs/2412.17836)
• Keywords: transformer, generative
• Abstract: This paper proposes a look ahead text understanding problem with look ahead section identification (LASI) as an example. This problem may appear in generative AI as well as human interactions, where we want to understand the direction of a developing text or conversation. We tackle the problem using transformer-based LLMs. We show that LASI is more challenging than classic section identification (SI). We argue that both bidirectional contextual information (e.g., BERT) and unidirectional predictive ability (e.g., GPT) will benefit the task. We propose two approaches to stitch together BERT and GPT. Experiments show that our approach outperforms the established models, especially when there is noise in the text (which is often the case for developing text in generative AI). Our paper sheds light on other look ahead text understanding tasks that are important to social media, such as look ahead sentiment classification, and points out the opportunities to leverage pre-trained LLMs through stitching.

Title: Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding

• Authors: Tadesse Destaw Belay, Israel Abebe Azime, Abinew Ali Ayele, Grigori Sidorov, Dietrich Klakow, Philipp Slusallek, Olga Kolesnikova, Seid Muhie Yimam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17837
• Pdf URL: https://arxiv.org/pdf/2412.17837
• Copy Paste: [[2412.17837]] Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding(https://arxiv.org/abs/2412.17837)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) show promising learning and reasoning abilities. Compared to other NLP tasks, multilingual and multi-label emotion evaluation tasks are under-explored in LLMs. In this paper, we present EthioEmo, a multi-label emotion classification dataset for four Ethiopian languages, namely, Amharic (amh), Afan Oromo (orm), Somali (som), and Tigrinya (tir). We perform extensive experiments with an additional English multi-label emotion dataset from SemEval 2018 Task 1. Our evaluation includes encoder-only, encoder-decoder, and decoder-only language models. We compare zero and few-shot approaches of LLMs to fine-tuning smaller language models. The results show that accurate multi-label emotion classification is still insufficient even for high-resource languages such as English, and there is a large gap between the performance of high-resource and low-resource languages. The results also show varying performance levels depending on the language and model type. EthioEmo is available publicly to further improve the understanding of emotions in language models and how people convey emotions through various languages.

Title: LaMI-GO: Latent Mixture Integration for Goal-Oriented Communications Achieving High Spectrum Efficiency

• Authors: Achintha Wijesinghe, Suchinthaka Wanninayaka, Weiwei Wang, Yu-Chieh Chao, Songyang Zhang, Zhi Ding
• Subjects: cs.LG, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.17839
• Pdf URL: https://arxiv.org/pdf/2412.17839
• Copy Paste: [[2412.17839]] LaMI-GO: Latent Mixture Integration for Goal-Oriented Communications Achieving High Spectrum Efficiency(https://arxiv.org/abs/2412.17839)
• Keywords: diffusion, generative
• Abstract: The recent rise of semantic-style communications includes the development of goal-oriented communications (GOCOMs) remarkably efficient multimedia information transmissions. The concept of GO-COMS leverages advanced artificial intelligence (AI) tools to address the rising demand for bandwidth efficiency in applications, such as edge computing and Internet-of-Things (IoT). Unlike traditional communication systems focusing on source data accuracy, GO-COMs provide intelligent message delivery catering to the special needs critical to accomplishing downstream tasks at the receiver. In this work, we present a novel GO-COM framework, namely LaMI-GO that utilizes emerging generative AI for better quality-of-service (QoS) with ultra-high communication efficiency. Specifically, we design our LaMI-GO system backbone based on a latent diffusion model followed by a vector-quantized generative adversarial network (VQGAN) for efficient latent embedding and information representation. The system trains a common feature codebook the receiver side. Our experimental results demonstrate substantial improvement in perceptual quality, accuracy of downstream tasks, and bandwidth consumption over the state-of-the-art GOCOM systems and establish the power of our proposed LaMI-GO communication framework.

Title: Enhancing Knowledge Distillation for LLMs with Response-Priming Prompting

• Authors: Vijay Goyal, Mustafa Khan, Aprameya Tirupati, Harveer Saini, Michael Lam, Kevin Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17846
• Pdf URL: https://arxiv.org/pdf/2412.17846
• Copy Paste: [[2412.17846]] Enhancing Knowledge Distillation for LLMs with Response-Priming Prompting(https://arxiv.org/abs/2412.17846)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated remarkable performance across a wide range of natural language processing (NLP) tasks. However, these models are often difficult to deploy due to significant computational requirements and resource constraints. Knowledge distillation (KD) is an effective technique for transferring the performance of larger LLMs to smaller models. Traditional KD methods primarily focus on the direct output of the teacher model, with little emphasis on the role of prompting during knowledge transfer. In this paper, we propose a set of novel response-priming prompting strategies applied in the knowledge distillation pipeline to enhance the performance of student models. Our approach fine-tunes a smaller Llama 3.1 8B Instruct model by distilling knowledge from a quantized Llama 3.1 405B Instruct teacher model. We apply LoRA optimization and evaluate on the GSM8K benchmark. Experimental results demonstrate that integrating reasoning-eliciting prompting into the proposed KD pipeline significantly improves student model performance, offering an efficient way to deploy powerful models in resource-constrained environments. We find that Ground Truth prompting results in a 55\% performance increase on GSM8K for a distilled Llama 3.1 8B Instruct compared to the same model distilled without prompting. A thorough investigation into the self-attention layers of the student models indicates that the more successful prompted models tend to exhibit certain positive behaviors inside their attention heads which can be tied to their increased accuracy. Our implementation can be found at this https URL.

Title: Zero Shot Time Series Forecasting Using Kolmogorov Arnold Networks

• Authors: Abhiroop Bhattacharya, Nandinee Haq
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17853
• Pdf URL: https://arxiv.org/pdf/2412.17853
• Copy Paste: [[2412.17853]] Zero Shot Time Series Forecasting Using Kolmogorov Arnold Networks(https://arxiv.org/abs/2412.17853)
• Keywords: robust
• Abstract: Accurate energy price forecasting is crucial for participants in day-ahead energy markets, as it significantly influences their decision-making processes. While machine learning-based approaches have shown promise in enhancing these forecasts, they often remain confined to the specific markets on which they are trained, thereby limiting their adaptability to new or unseen markets. In this paper, we introduce a cross-domain adaptation model designed to forecast energy prices by learning market-invariant representations across different markets during the training phase. We propose a doubly residual N-BEATS network with Kolmogorov Arnold networks at its core for time series forecasting. These networks, grounded in the Kolmogorov-Arnold representation theorem, offer a powerful way to approximate multivariate continuous functions. The cross domain adaptation model was generated with an adversarial framework. The model's effectiveness was tested in predicting day-ahead electricity prices in a zero shot fashion. In comparison with baseline models, our proposed framework shows promising results. By leveraging the Kolmogorov-Arnold networks, our model can potentially enhance its ability to capture complex patterns in energy price data, thus improving forecast accuracy across diverse market conditions. This addition not only enriches the model's representational capacity but also contributes to a more robust and flexible forecasting tool adaptable to various energy markets.

Title: Graph Structure Refinement with Energy-based Contrastive Learning

• Authors: Xianlin Zeng, Yufeng Wang, Yuqi Sun, Guodong Guo, Wenrui Ding, Baochang Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17856
• Pdf URL: https://arxiv.org/pdf/2412.17856
• Copy Paste: [[2412.17856]] Graph Structure Refinement with Energy-based Contrastive Learning(https://arxiv.org/abs/2412.17856)
• Keywords: robust, generative
• Abstract: Graph Neural Networks (GNNs) have recently gained widespread attention as a successful tool for analyzing graph-structured data. However, imperfect graph structure with noisy links lacks enough robustness and may damage graph representations, therefore limiting the GNNs' performance in practical tasks. Moreover, existing generative architectures fail to fit discriminative graph-related tasks. To tackle these issues, we introduce an unsupervised method based on a joint of generative training and discriminative training to learn graph structure and representation, aiming to improve the discriminative performance of generative models. We propose an Energy-based Contrastive Learning (ECL) guided Graph Structure Refinement (GSR) framework, denoted as ECL-GSR. To our knowledge, this is the first work to combine energy-based models with contrastive learning for GSR. Specifically, we leverage ECL to approximate the joint distribution of sample pairs, which increases the similarity between representations of positive pairs while reducing the similarity between negative ones. Refined structure is produced by augmenting and removing edges according to the similarity metrics among node representations. Extensive experiments demonstrate that ECL-GSR outperforms \textit{the state-of-the-art on eight benchmark datasets} in node classification. ECL-GSR achieves \textit{faster training with fewer samples and memories} against the leading baseline, highlighting its simplicity and efficiency in downstream tasks.

Title: Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types

• Authors: Ziming Guo, Chao Ma, Yinggang Sun, Tiancheng Zhao, Guangyao Wang, Hai Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17867
• Pdf URL: https://arxiv.org/pdf/2412.17867
• Copy Paste: [[2412.17867]] Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types(https://arxiv.org/abs/2412.17867)
• Keywords: large language model
• Abstract: Recent advancements in large language models (LLMs) have significantly advanced text-to-SQL systems. However, most LLM-based methods often narrowly focus on SQL generation, neglecting the complexities of real-world conversational queries. This oversight can lead to unreliable responses, particularly for ambiguous questions that cannot be directly addressed with SQL. To bridge this gap, we propose MMSQL, a comprehensive test suite designed to evaluate the question classification and SQL generation capabilities of LLMs by simulating real-world scenarios with diverse question types and multi-turn Q\&A interactions. Using MMSQL, we assessed the performance of popular LLMs, including both open-source and closed-source models, and identified key factors impacting their performance in such scenarios. Moreover, we introduce an LLM-based multi-agent framework that employs specialized agents to identify question types and determine appropriate answering strategies. Our experiments demonstrate that this approach significantly enhances the model's ability to navigate the complexities of conversational dynamics, effectively handling the diverse and complex nature of user queries.

Title: Joint Knowledge Editing for Information Enrichment and Probability Promotion

• Authors: Wenhang Shi, Yiren Chen, Shuqing Bian, Xinyi Zhang, Zhe Zhao, Pengfei Hu, Wei Lu, Xiaoyong Du
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.17872
• Pdf URL: https://arxiv.org/pdf/2412.17872
• Copy Paste: [[2412.17872]] Joint Knowledge Editing for Information Enrichment and Probability Promotion(https://arxiv.org/abs/2412.17872)
• Keywords: large language model
• Abstract: Knowledge stored in large language models requires timely updates to reflect the dynamic nature of real-world information. To update the knowledge, most knowledge editing methods focus on the low layers, since recent probes into the knowledge recall process reveal that the answer information is enriched in low layers. However, these probes only and could only reveal critical recall stages for the original answers, while the goal of editing is to rectify model's prediction for the target answers. This inconsistency indicates that both the probe approaches and the associated editing methods are deficient. To mitigate the inconsistency and identify critical editing regions, we propose a contrast-based probe approach, and locate two crucial stages where the model behavior diverges between the original and target answers: Information Enrichment in low layers and Probability Promotion in high layers. Building upon the insights, we develop the Joint knowledge Editing for information Enrichment and probability Promotion (JEEP) method, which jointly edits both the low and high layers to modify the two critical recall stages. Considering the mutual interference and growing forgetting due to dual modifications, JEEP is designed to ensure that updates to distinct regions share the same objectives and are complementary. We rigorously evaluate JEEP by editing up to thousands of facts on various models, i.e., GPT-J (6B) and LLaMA (7B), and addressing diverse editing objectives, i.e., adding factual and counterfactual knowledge. In all tested scenarios, JEEP achieves best performances, validating the effectiveness of the revealings of our probe approach and the designs of our editing method. Our code and data are available at this https URL.

Title: Evaluating LLM Reasoning in the Operations Research Domain with ORQA

• Authors: Mahdi Mostajabdaveh, Timothy T. Yu, Samarendra Chandan Bindu Dash, Rindranirina Ramamonjison, Jabo Serge Byusa, Giuseppe Carenini, Zirui Zhou, Yong Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17874
• Pdf URL: https://arxiv.org/pdf/2412.17874
• Copy Paste: [[2412.17874]] Evaluating LLM Reasoning in the Operations Research Domain with ORQA(https://arxiv.org/abs/2412.17874)
• Keywords: large language model
• Abstract: In this paper, we introduce and apply Operations Research Question Answering (ORQA), a new benchmark designed to assess the generalization capabilities of Large Language Models (LLMs) in the specialized technical domain of Operations Research (OR). This benchmark evaluates whether LLMs can emulate the knowledge and reasoning skills of OR experts when confronted with diverse and complex optimization problems. The dataset, developed by OR experts, features real-world optimization problems that demand multistep reasoning to construct their mathematical models. Our evaluations of various open source LLMs, such as LLaMA 3.1, DeepSeek, and Mixtral, reveal their modest performance, highlighting a gap in their ability to generalize to specialized technical domains. This work contributes to the ongoing discourse on LLMs generalization capabilities, offering valuable insights for future research in this area. The dataset and evaluation code are publicly available.

Title: In Defence of Post-hoc Explainability

• Authors: Nick Oh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17883
• Pdf URL: https://arxiv.org/pdf/2412.17883
• Copy Paste: [[2412.17883]] In Defence of Post-hoc Explainability(https://arxiv.org/abs/2412.17883)
• Keywords: interpretability, explainability
• Abstract: The widespread adoption of machine learning in scientific research has created a fundamental tension between model opacity and scientific understanding. Whilst some advocate for intrinsically interpretable models, we introduce Computational Interpretabilism (CI) as a philosophical framework for post-hoc interpretability in scientific AI. Drawing parallels with human expertise, where post-hoc rationalisation coexists with reliable performance, CI establishes that scientific knowledge emerges through structured model interpretation when properly bounded by empirical validation. Through mediated understanding and bounded factivity, we demonstrate how post-hoc methods achieve epistemically justified insights without requiring complete mechanical transparency, resolving tensions between model complexity and scientific comprehension.

Title: The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting

• Authors: Shuzhang Cai, Twumasi Mensah-Boateng, Xander Kuksov, Jing Yuan, Shaojie Tang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17891
• Pdf URL: https://arxiv.org/pdf/2412.17891
• Copy Paste: [[2412.17891]] The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting(https://arxiv.org/abs/2412.17891)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated exceptional abilities across a broad range of language-related tasks, including generating solutions to complex reasoning problems. An effective technique to enhance LLM performance is in-context learning, which encourages a step-by-step reasoning process by including explanatory examples to guide the model's responses. However, selecting appropriate exemplars for the model poses a challenge, as each dataset demands a distinct set of exemplars to enable the LLM to learn effectively and perform well on the test set. Current studies often rely on uncertainty- or diversity-based selection strategies to select exemplars for annotation and to improve model learning. However, these studies typically employ a non-adaptive approach, selecting a set of exemplars all at once. We argue that this non-adaptive strategy may result in a set of exemplars with high redundancy in terms of the knowledge covered, ultimately reducing their overall informativeness. To address this limitation, we propose \textsc{Adaptive-Prompt}, a novel method that adaptively selects exemplars by leveraging model feedback from previously chosen exemplars. Experimental results show that \textsc{Adaptive-Prompt} significantly enhances LLM performance across a variety of reasoning tasks.

Title: Trading Devil RL: Backdoor attack via Stock market, Bayesian Optimization and Reinforcement Learning

• Authors: Orson Mengara
• Subjects: cs.LG, cs.CE, physics.comp-ph, physics.soc-ph
• Abstract URL: https://arxiv.org/abs/2412.17908
• Pdf URL: https://arxiv.org/pdf/2412.17908
• Copy Paste: [[2412.17908]] Trading Devil RL: Backdoor attack via Stock market, Bayesian Optimization and Reinforcement Learning(https://arxiv.org/abs/2412.17908)
• Keywords: attack, generative, large language model
• Abstract: With the rapid development of generative artificial intelligence, particularly large language models, a number of sub-fields of deep learning have made significant progress and are now very useful in everyday applications. For example, well-known financial institutions simulate a wide range of scenarios for various models created by their research teams using reinforcement learning, both before production and after regular operations. In this work, we propose a backdoor attack that focuses solely on data poisoning. This particular backdoor attack is classified as an attack without prior consideration or trigger, and we name it FinanceLLMsBackRL. Our aim is to examine the potential effects of large language models that use reinforcement learning systems for text production or speech recognition, finance, physics, or the ecosystem of contemporary artificial intelligence models.

Title: BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism

• Authors: Martin Fajcik, Martin Docekal, Jan Dolezal, Karel Ondrej, Karel Beneš, Jan Kapsa, Pavel Smrz, Alexander Polok, Michal Hradis, Zuzana Neverilova, Ales Horak, Radoslav Sabol, Michal Stefanik, Adam Jirkovsky, David Adamczyk, Petr Hyner, Jan Hula, Hynek Kydlicek
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.17933
• Pdf URL: https://arxiv.org/pdf/2412.17933
• Copy Paste: [[2412.17933]] BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism(https://arxiv.org/abs/2412.17933)
• Keywords: large language model
• Abstract: We present BenCzechMark (BCM), the first comprehensive Czech language benchmark designed for large language models, offering diverse tasks, multiple task formats, and multiple evaluation metrics. Its scoring system is grounded in statistical significance theory and uses aggregation across tasks inspired by social preference theory. Our benchmark encompasses 50 challenging tasks, with corresponding test datasets, primarily in native Czech, with 11 newly collected ones. These tasks span 8 categories and cover diverse domains, including historical Czech news, essays from pupils or language learners, and spoken word. Furthermore, we collect and clean BUT-Large Czech Collection, the largest publicly available clean Czech language corpus, and use it for (i) contamination analysis, (ii) continuous pretraining of the first Czech-centric 7B language model, with Czech-specific tokenization. We use our model as a baseline for comparison with publicly available multilingual models. Lastly, we release and maintain a leaderboard, with existing 44 model submissions, where new model submissions can be made at this https URL.

Title: IITR-CIOL@NLU of Devanagari Script Languages 2025: Multilingual Hate Speech Detection and Target Identification in Devanagari-Scripted Languages

• Authors: Siddhant Gupta, Siddh Singhal, Azmine Toushik Wasi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17947
• Pdf URL: https://arxiv.org/pdf/2412.17947
• Copy Paste: [[2412.17947]] IITR-CIOL@NLU of Devanagari Script Languages 2025: Multilingual Hate Speech Detection and Target Identification in Devanagari-Scripted Languages(https://arxiv.org/abs/2412.17947)
• Keywords: transformer
• Abstract: This work focuses on two subtasks related to hate speech detection and target identification in Devanagari-scripted languages, specifically Hindi, Marathi, Nepali, Bhojpuri, and Sanskrit. Subtask B involves detecting hate speech in online text, while Subtask C requires identifying the specific targets of hate speech, such as individuals, organizations, or communities. We propose the MultilingualRobertaClass model, a deep neural network built on the pretrained multilingual transformer model ia-multilingual-transliterated-roberta, optimized for classification tasks in multilingual and transliterated contexts. The model leverages contextualized embeddings to handle linguistic diversity, with a classifier head for binary classification. We received 88.40% accuracy in Subtask B and 66.11% accuracy in Subtask C, in the test set.

Title: Hyperbolic Chamfer Distance for Point Cloud Completion and Beyond

• Authors: Fangzhou Lin, Songlin Hou, Haotian Liu, Shang Gao, Kazunori D Yamada, Haichong K. Zhang, Ziming Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17951
• Pdf URL: https://arxiv.org/pdf/2412.17951
• Copy Paste: [[2412.17951]] Hyperbolic Chamfer Distance for Point Cloud Completion and Beyond(https://arxiv.org/abs/2412.17951)
• Keywords: generative
• Abstract: Chamfer Distance (CD) is widely used as a metric to quantify difference between two point clouds. In point cloud completion, Chamfer Distance (CD) is typically used as a loss function in deep learning frameworks. However, it is generally acknowledged within the field that Chamfer Distance (CD) is vulnerable to the presence of outliers, which can consequently lead to the convergence on suboptimal models. In divergence from the existing literature, which largely concentrates on resolving such concerns in the realm of Euclidean space, we put forth a notably uncomplicated yet potent metric specifically designed for point cloud completion tasks: {Hyperbolic Chamfer Distance (HyperCD)}. This metric conducts Chamfer Distance computations within the parameters of hyperbolic space. During the backpropagation process, HyperCD systematically allocates greater weight to matched point pairs exhibiting reduced Euclidean distances. This mechanism facilitates the preservation of accurate point pair matches while permitting the incremental adjustment of suboptimal matches, thereby contributing to enhanced point cloud completion outcomes. Moreover, measure the shape dissimilarity is not solely work for point cloud completion task, we further explore its applications in other generative related tasks, including single image reconstruction from point cloud, and upsampling. We demonstrate state-of-the-art performance on the point cloud completion benchmark datasets, PCN, ShapeNet-55, and ShapeNet-34, and show from visualization that HyperCD can significantly improve the surface smoothness, we also provide the provide experimental results beyond completion task.

Title: ArchComplete: Autoregressive 3D Architectural Design Generation with Hierarchical Diffusion-Based Upsampling

• Authors: S. Rasoulzadeh, M. Bank, M. Wimmer, I. Kovacic, K. Schinegger, S. Rutzinger
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.17957
• Pdf URL: https://arxiv.org/pdf/2412.17957
• Copy Paste: [[2412.17957]] ArchComplete: Autoregressive 3D Architectural Design Generation with Hierarchical Diffusion-Based Upsampling(https://arxiv.org/abs/2412.17957)
• Keywords: diffusion, transformer, generative
• Abstract: $\textit{ArchComplete}$ is a two-stage dense voxel-based 3D generative pipeline developed to tackle the high complexity in architectural geometries and topologies, assisting with ideation and geometric detailisation in the early design process. In stage 1, a $\textit{3D Voxel VQGAN}$ model is devised, whose composition is then modelled with an autoregressive transformer for generating coarse models. Subsequently, in stage 2, $\textit{Hierarchical Voxel Upsampling Networks}$ consisting of a set of 3D conditional denoising diffusion probabilistic models are defined to augment the coarse shapes with fine geometric details. The first stage is trained on a dataset of house models with fully modelled exteriors and interiors with a novel 2.5D perceptual loss to capture input complexities across multiple abstraction levels, while the second stage trains on randomly cropped local volumetric patches, requiring significantly less compute and memory. For inference, the pipeline first autoregressively generates house models at a resolution of $64^3$ and then progressively refines them to resolution of $256^3$ with voxel sizes as small as $18\text{cm}$. ArchComplete supports a range of interaction modes solving a variety of tasks, including interpolation, variation generation, unconditional synthesis, and two conditional synthesis tasks: shape completion and plan-drawing completion, as well as geometric detailisation. The results demonstrate notable improvements against state-of-the-art on established metrics.

Title: Path-of-Thoughts: Extracting and Following Paths for Robust Relational Reasoning with Large Language Models

• Authors: Ge Zhang, Mohammad Ali Alomrani, Hongjian Gu, Jiaming Zhou, Yaochen Hu, Bin Wang, Qun Liu, Mark Coates, Yingxue Zhang, Jianye Hao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.17963
• Pdf URL: https://arxiv.org/pdf/2412.17963
• Copy Paste: [[2412.17963]] Path-of-Thoughts: Extracting and Following Paths for Robust Relational Reasoning with Large Language Models(https://arxiv.org/abs/2412.17963)
• Keywords: robust, extraction, large language model
• Abstract: Large language models (LLMs) possess vast semantic knowledge but often struggle with complex reasoning tasks, particularly in relational reasoning problems such as kinship or spatial reasoning. In this paper, we present Path-of-Thoughts (PoT), a novel framework designed to tackle relation reasoning by decomposing the task into three key stages: graph extraction, path identification, and reasoning. Unlike previous approaches, PoT efficiently extracts a task-agnostic graph that identifies crucial entities, relations, and attributes within the problem context. Subsequently, PoT identifies relevant reasoning chains within the graph corresponding to the posed question, facilitating inference of potential answers. Experimental evaluations on four benchmark datasets, demanding long reasoning chains, demonstrate that PoT surpasses state-of-the-art baselines by a significant margin (maximum 21.3%) without necessitating fine-tuning or extensive LLM calls. Furthermore, as opposed to prior neuro-symbolic methods, PoT exhibits improved resilience against LLM errors by leveraging the compositional nature of graphs.

Title: CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models

• Authors: Ruibo Tu, Hedvig Kjellström, Gustav Eje Henter, Cheng Zhang
• Subjects: cs.CL, cs.LG, stat.ME
• Abstract URL: https://arxiv.org/abs/2412.17970
• Pdf URL: https://arxiv.org/pdf/2412.17970
• Copy Paste: [[2412.17970]] CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models(https://arxiv.org/abs/2412.17970)
• Keywords: large language model
• Abstract: Causal reasoning capabilities are essential for large language models (LLMs) in a wide range of applications, such as education and healthcare. But there is still a lack of benchmarks for a better understanding of such capabilities. Current LLM benchmarks are mainly based on conversational tasks, academic math tests, and coding tests. Such benchmarks evaluate LLMs in well-regularized settings, but they are limited in assessing the skills and abilities to solve real-world problems. In this work, we provide a benchmark, named by CARL-GT, which evaluates CAusal Reasoning capabilities of large Language models using Graphs and Tabular data. The benchmark has a diverse range of tasks for evaluating LLMs from causal graph reasoning, knowledge discovery, and decision-making aspects. In addition, effective zero-shot learning prompts are developed for the tasks. In our experiments, we leverage the benchmark for evaluating open-source LLMs and provide a detailed comparison of LLMs for causal reasoning abilities. We found that LLMs are still weak in casual reasoning, especially with tabular data to discover new insights. Furthermore, we investigate and discuss the relationships of different benchmark tasks by analyzing the performance of LLMs. The experimental results show that LLMs have different strength over different tasks and that their performance on tasks in different categories, i.e., causal graph reasoning, knowledge discovery, and decision-making, shows stronger correlation than tasks in the same category.

Title: Data-driven Modeling of Parameterized Nonlinear Fluid Dynamical Systems with a Dynamics-embedded Conditional Generative Adversarial Network

• Authors: Abdolvahhab Rostamijavanani, Shanwu Li, Yongchao Yang
• Subjects: cs.LG, nlin.CD
• Abstract URL: https://arxiv.org/abs/2412.17978
• Pdf URL: https://arxiv.org/pdf/2412.17978
• Copy Paste: [[2412.17978]] Data-driven Modeling of Parameterized Nonlinear Fluid Dynamical Systems with a Dynamics-embedded Conditional Generative Adversarial Network(https://arxiv.org/abs/2412.17978)
• Keywords: generative
• Abstract: This work presents a data-driven solution to accurately predict parameterized nonlinear fluid dynamical systems using a dynamics-generator conditional GAN (Dyn-cGAN) as a surrogate model. The Dyn-cGAN includes a dynamics block within a modified conditional GAN, enabling the simultaneous identification of temporal dynamics and their dependence on system parameters. The learned Dyn-cGAN model takes into account the system parameters to predict the flow fields of the system accurately. We evaluate the effectiveness and limitations of the developed Dyn-cGAN through numerical studies of various parameterized nonlinear fluid dynamical systems, including flow over a cylinder and a 2-D cavity problem, with different Reynolds numbers. Furthermore, we examine how Reynolds number affects the accuracy of the predictions for both case studies. Additionally, we investigate the impact of the number of time steps involved in the process of dynamics block training on the accuracy of predictions, and we find that an optimal value exists based on errors and mutual information relative to the ground truth.

Title: PHICOIN (PHI): The Proof of Work High-Performance Infrastructure

• Authors: Guang Yang, Peter Trinh, Sannan Iqbal, Justin Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.17979
• Pdf URL: https://arxiv.org/pdf/2412.17979
• Copy Paste: [[2412.17979]] PHICOIN (PHI): The Proof of Work High-Performance Infrastructure(https://arxiv.org/abs/2412.17979)
• Keywords: fair
• Abstract: PHICOIN (PHI) is a high-performance cryptocurrency based on the Proof-of-Work (PoW) mechanism. It aims to provide ordinary users with decentralized participation opportunities through an improved and innovative mining algorithm and fair design principles. PHI addresses the challenges of centralization in cryptocurrency mining by enhancing resistance to ASIC and FPGA devices and promoting fair participation. This paper outlines the technical specifications, mission, and roadmap for PHI, highlighting its potential to become a foundational infrastructure for PoW cryptocurrencies.

Title: Unsupervised learning of spatially varying regularization for diffeomorphic image registration

• Authors: Junyu Chen, Shuwen Wei, Yihao Liu, Zhangxing Bian, Yufan He, Aaron Carass, Harrison Bai, Yong Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.17982
• Pdf URL: https://arxiv.org/pdf/2412.17982
• Copy Paste: [[2412.17982]] Unsupervised learning of spatially varying regularization for diffeomorphic image registration(https://arxiv.org/abs/2412.17982)
• Keywords: interpretability
• Abstract: Spatially varying regularization accommodates the deformation variations that may be necessary for different anatomical regions during deformable image registration. Historically, optimization-based registration models have harnessed spatially varying regularization to address anatomical subtleties. However, most modern deep learning-based models tend to gravitate towards spatially invariant regularization, wherein a homogenous regularization strength is applied across the entire image, potentially disregarding localized variations. In this paper, we propose a hierarchical probabilistic model that integrates a prior distribution on the deformation regularization strength, enabling the end-to-end learning of a spatially varying deformation regularizer directly from the data. The proposed method is straightforward to implement and easily integrates with various registration network architectures. Additionally, automatic tuning of hyperparameters is achieved through Bayesian optimization, allowing efficient identification of optimal hyperparameters for any given registration task. Comprehensive evaluations on publicly available datasets demonstrate that the proposed method significantly improves registration performance and enhances the interpretability of deep learning-based registration, all while maintaining smooth deformations.

Title: StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs

• Authors: Hailin Chen, Fangkai Jiao, Mathieu Ravaut, Nawshad Farruque, Xuan Phi Nguyen, Chengwei Qin, Manan Dey, Bosheng Ding, Caiming Xiong, Shafiq Joty, Yingbo Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18011
• Pdf URL: https://arxiv.org/pdf/2412.18011
• Copy Paste: [[2412.18011]] StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs(https://arxiv.org/abs/2412.18011)
• Keywords: robust, large language model
• Abstract: The rapid development of large language models (LLMs) necessitates robust, unbiased, and scalable methods for evaluating their capabilities. However, human annotations are expensive to scale, model-based evaluations are prone to biases in answer style, while target-answer-based benchmarks are vulnerable to data contamination and cheating. To address these limitations, we propose StructTest, a novel benchmark that evaluates LLMs on their ability to produce compositionally specified structured outputs as an unbiased, cheap-to-run and difficult-to-cheat measure. The evaluation is done deterministically by a rule-based evaluator, which can be easily extended to new tasks. By testing structured outputs across diverse task domains -- including Summarization, Code, HTML and Math -- we demonstrate that StructTest serves as a good proxy for general reasoning abilities, as producing structured outputs often requires internal logical reasoning. We believe that StructTest offers a critical, complementary approach to objective and robust model evaluation.

Title: LayerDropBack: A Universally Applicable Approach for Accelerating Training of Deep Networks

• Authors: Evgeny Hershkovitch Neiterman, Gil Ben-Artzi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18027
• Pdf URL: https://arxiv.org/pdf/2412.18027
• Copy Paste: [[2412.18027]] LayerDropBack: A Universally Applicable Approach for Accelerating Training of Deep Networks(https://arxiv.org/abs/2412.18027)
• Keywords: transformer
• Abstract: Training very deep convolutional networks is challenging, requiring significant computational resources and time. Existing acceleration methods often depend on specific architectures or require network modifications. We introduce LayerDropBack (LDB), a simple yet effective method to accelerate training across a wide range of deep networks. LDB introduces randomness only in the backward pass, maintaining the integrity of the forward pass, guaranteeing that the same network is used during both training and inference. LDB can be seamlessly integrated into the training process of any model without altering its architecture, making it suitable for various network topologies. Our extensive experiments across multiple architectures (ViT, Swin Transformer, EfficientNet, DLA) and datasets (CIFAR-100, ImageNet) show significant training time reductions of 16.93\% to 23.97\%, while preserving or even enhancing model accuracy. Code is available at \url{this https URL}.

Title: Explainability in Neural Networks for Natural Language Processing Tasks

• Authors: Melkamu Mersha, Mingiziem Bitewa, Tsion Abay, Jugal Kalita
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18036
• Pdf URL: https://arxiv.org/pdf/2412.18036
• Copy Paste: [[2412.18036]] Explainability in Neural Networks for Natural Language Processing Tasks(https://arxiv.org/abs/2412.18036)
• Keywords: interpretability, explainability
• Abstract: Neural networks are widely regarded as black-box models, creating significant challenges in understanding their inner workings, especially in natural language processing (NLP) applications. To address this opacity, model explanation techniques like Local Interpretable Model-Agnostic Explanations (LIME) have emerged as essential tools for providing insights into the behavior of these complex systems. This study leverages LIME to interpret a multi-layer perceptron (MLP) neural network trained on a text classification task. By analyzing the contribution of individual features to model predictions, the LIME approach enhances interpretability and supports informed decision-making. Despite its effectiveness in offering localized explanations, LIME has limitations in capturing global patterns and feature interactions. This research highlights the strengths and shortcomings of LIME and proposes directions for future work to achieve more comprehensive interpretability in neural NLP models.

Title: AA-SGAN: Adversarially Augmented Social GAN with Synthetic Data

• Authors: Mirko Zaffaroni, Federico Signoretta, Marco Grangetto, Attilio Fiandrotti
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18038
• Pdf URL: https://arxiv.org/pdf/2412.18038
• Copy Paste: [[2412.18038]] AA-SGAN: Adversarially Augmented Social GAN with Synthetic Data(https://arxiv.org/abs/2412.18038)
• Keywords: generative
• Abstract: Accurately predicting pedestrian trajectories is crucial in applications such as autonomous driving or service robotics, to name a few. Deep generative models achieve top performance in this task, assuming enough labelled trajectories are available for training. To this end, large amounts of synthetically generated, labelled trajectories exist (e.g., generated by video games). However, such trajectories are not meant to represent pedestrian motion realistically and are ineffective at training a predictive model. We propose a method and an architecture to augment synthetic trajectories at training time and with an adversarial approach. We show that trajectory augmentation at training time unleashes significant gains when a state-of-the-art generative model is evaluated over real-world trajectories.

Title: Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers

• Authors: Xiaoyu Li, Yingyu Liang, Zhenmei Shi, Zhao Song, Mingda Wan
• Subjects: cs.LG, cs.AI, cs.CC, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18040
• Pdf URL: https://arxiv.org/pdf/2412.18040
• Copy Paste: [[2412.18040]] Theoretical Constraints on the Expressive Power of $\mathsf{RoPE}$-based Tensor Attention Transformers(https://arxiv.org/abs/2412.18040)
• Keywords: transformer
• Abstract: Tensor Attention extends traditional attention mechanisms by capturing high-order correlations across multiple modalities, addressing the limitations of classical matrix-based attention. Meanwhile, Rotary Position Embedding ($\mathsf{RoPE}$) has shown superior performance in encoding positional information in long-context scenarios, significantly enhancing transformer models' expressiveness. Despite these empirical successes, the theoretical limitations of these technologies remain underexplored. In this study, we analyze the circuit complexity of Tensor Attention and $\mathsf{RoPE}$-based Tensor Attention, showing that with polynomial precision, constant-depth layers, and linear or sublinear hidden dimension, they cannot solve fixed membership problems or $(A_{F,r})^*$ closure problems, under the assumption that $\mathsf{TC}^0 \neq \mathsf{NC}^1$. These findings highlight a gap between the empirical performance and theoretical constraints of Tensor Attention and $\mathsf{RoPE}$-based Tensor Attention Transformers, offering insights that could guide the development of more theoretically grounded approaches to Transformer model design and scaling.

Title: Factuality or Fiction? Benchmarking Modern LLMs on Ambiguous QA with Citations

• Authors: Maya Patel, Aditi Anand
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18051
• Pdf URL: https://arxiv.org/pdf/2412.18051
• Copy Paste: [[2412.18051]] Factuality or Fiction? Benchmarking Modern LLMs on Ambiguous QA with Citations(https://arxiv.org/abs/2412.18051)
• Keywords: large language model
• Abstract: Benchmarking modern large language models (LLMs) on complex and realistic tasks is critical to advancing their development. In this work, we evaluate the factual accuracy and citation performance of state-of-the-art LLMs on the task of Question Answering (QA) in ambiguous settings with source citations. Using three recently published datasets-DisentQA-DupliCite, DisentQA-ParaCite, and AmbigQA-Cite-featuring a range of real-world ambiguities, we analyze the performance of two leading LLMs, GPT-4o-mini and Claude-3.5. Our results show that larger, recent models consistently predict at least one correct answer in ambiguous contexts but fail to handle cases with multiple valid answers. Additionally, all models perform equally poorly in citation generation, with citation accuracy consistently at 0. However, introducing conflict-aware prompting leads to large improvements, enabling models to better address multiple valid answers and improve citation accuracy, while maintaining their ability to predict correct answers. These findings highlight the challenges and opportunities in developing LLMs that can handle ambiguity and provide reliable source citations. Our benchmarking study provides critical insights and sets a foundation for future improvements in trustworthy and interpretable QA systems.

Title: Beyond Gradient Averaging in Parallel Optimization: Improved Robustness through Gradient Agreement Filtering

• Authors: Francois Chaubard, Duncan Eddy, Mykel J. Kochenderfer
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18052
• Pdf URL: https://arxiv.org/pdf/2412.18052
• Copy Paste: [[2412.18052]] Beyond Gradient Averaging in Parallel Optimization: Improved Robustness through Gradient Agreement Filtering(https://arxiv.org/abs/2412.18052)
• Keywords: robust
• Abstract: We introduce Gradient Agreement Filtering (GAF) to improve on gradient averaging in distributed deep learning optimization. Traditional distributed data-parallel stochastic gradient descent involves averaging gradients of microbatches to calculate a macrobatch gradient that is then used to update model parameters. We find that gradients across microbatches are often orthogonal or negatively correlated, especially in late stages of training, which leads to memorization of the training set, reducing generalization. In this paper, we introduce a simple, computationally effective way to reduce gradient variance by computing the cosine distance between micro-gradients during training and filtering out conflicting updates prior to averaging. We improve validation accuracy with significantly smaller microbatch sizes. We also show this reduces memorizing noisy labels. We demonstrate the effectiveness of this technique on standard image classification benchmarks including CIFAR-100 and CIFAR-100N-Fine. We show this technique consistently outperforms validation accuracy, in some cases by up to 18.2\% compared to traditional training approaches while reducing the computation required nearly an order of magnitude because we can now rely on smaller microbatch sizes without destabilizing training.

Title: Diverse Concept Proposals for Concept Bottleneck Models

• Authors: Katrina Brown, Marton Havasi, Finale Doshi-Velez
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18059
• Pdf URL: https://arxiv.org/pdf/2412.18059
• Copy Paste: [[2412.18059]] Diverse Concept Proposals for Concept Bottleneck Models(https://arxiv.org/abs/2412.18059)
• Keywords: interpretability
• Abstract: Concept bottleneck models are interpretable predictive models that are often used in domains where model trust is a key priority, such as healthcare. They identify a small number of human-interpretable concepts in the data, which they then use to make predictions. Learning relevant concepts from data proves to be a challenging task. The most predictive concepts may not align with expert intuition, thus, failing interpretability with no recourse. Our proposed approach identifies a number of predictive concepts that explain the data. By offering multiple alternative explanations, we allow the human expert to choose the one that best aligns with their expectation. To demonstrate our method, we show that it is able discover all possible concept representations on a synthetic dataset. On EHR data, our model was able to identify 4 out of the 5 pre-defined concepts without supervision.

Title: An Ensemble Approach to Short-form Video Quality Assessment Using Multimodal LLM

• Authors: Wen Wen, Yilin Wang, Neil Birkbeck, Balu Adsumilli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18060
• Pdf URL: https://arxiv.org/pdf/2412.18060
• Copy Paste: [[2412.18060]] An Ensemble Approach to Short-form Video Quality Assessment Using Multimodal LLM(https://arxiv.org/abs/2412.18060)
• Keywords: large language model
• Abstract: The rise of short-form videos, characterized by diverse content, editing styles, and artifacts, poses substantial challenges for learning-based blind video quality assessment (BVQA) models. Multimodal large language models (MLLMs), renowned for their superior generalization capabilities, present a promising solution. This paper focuses on effectively leveraging a pretrained MLLM for short-form video quality assessment, regarding the impacts of pre-processing and response variability, and insights on combining the MLLM with BVQA models. We first investigated how frame pre-processing and sampling techniques influence the MLLM's performance. Then, we introduced a lightweight learning-based ensemble method that adaptively integrates predictions from the MLLM and state-of-the-art BVQA models. Our results demonstrated superior generalization performance with the proposed ensemble approach. Furthermore, the analysis of content-aware ensemble weights highlighted that some video characteristics are not fully represented by existing BVQA models, revealing potential directions to improve BVQA models further.

Title: BIG-MoE: Bypass Isolated Gating MoE for Generalized Multimodal Face Anti-Spoofing

• Authors: Yingjie Ma, Zitong Yu, Xun Lin, Weicheng Xie, Linlin Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18065
• Pdf URL: https://arxiv.org/pdf/2412.18065
• Copy Paste: [[2412.18065]] BIG-MoE: Bypass Isolated Gating MoE for Generalized Multimodal Face Anti-Spoofing(https://arxiv.org/abs/2412.18065)
• Keywords: security, attack
• Abstract: In the domain of facial recognition security, multimodal Face Anti-Spoofing (FAS) is essential for countering presentation attacks. However, existing technologies encounter challenges due to modality biases and imbalances, as well as domain shifts. Our research introduces a Mixture of Experts (MoE) model to address these issues effectively. We identified three limitations in traditional MoE approaches to multimodal FAS: (1) Coarse-grained experts' inability to capture nuanced spoofing indicators; (2) Gated networks' susceptibility to input noise affecting decision-making; (3) MoE's sensitivity to prompt tokens leading to overfitting with conventional learning methods. To mitigate these, we propose the Bypass Isolated Gating MoE (BIG-MoE) framework, featuring: (1) Fine-grained experts for enhanced detection of subtle spoofing cues; (2) An isolation gating mechanism to counteract input noise; (3) A novel differential convolutional prompt bypass enriching the gating network with critical local features, thereby improving perceptual capabilities. Extensive experiments on four benchmark datasets demonstrate significant generalization performance improvement in multimodal FAS task. The code is released at this https URL.

Title: Improving Factuality with Explicit Working Memory

• Authors: Mingda Chen, Yang Li, Karthik Padthe, Rulin Shao, Alicia Sun, Luke Zettlemoyer, Gargi Gosh, Wen-tau Yih
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18069
• Pdf URL: https://arxiv.org/pdf/2412.18069
• Copy Paste: [[2412.18069]] Improving Factuality with Explicit Working Memory(https://arxiv.org/abs/2412.18069)
• Keywords: large language model
• Abstract: Large language models can generate factually inaccurate content, a problem known as hallucination. Recent works have built upon retrieved-augmented generation to improve factuality through iterative prompting but these methods are limited by the traditional RAG design. To address these challenges, we introduce EWE (Explicit Working Memory), a novel approach that enhances factuality in long-form text generation by integrating a working memory that receives real-time feedback from external resources. The memory is refreshed based on online fact-checking and retrieval feedback, allowing EWE to rectify false claims during the generation process and ensure more accurate and reliable outputs. Our experiments demonstrate that Ewe outperforms strong baselines on four fact-seeking long-form generation datasets, increasing the factuality metric, VeriScore, by 2 to 10 points absolute without sacrificing the helpfulness of the responses. Further analysis reveals that the design of rules for memory updates, configurations of memory units, and the quality of the retrieval datastore are crucial factors for influencing model performance.

Title: MMFactory: A Universal Solution Search Engine for Vision-Language Tasks

• Authors: Wan-Cyuan Fan, Tanzila Rahman, Leonid Sigal
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18072
• Pdf URL: https://arxiv.org/pdf/2412.18072
• Copy Paste: [[2412.18072]] MMFactory: A Universal Solution Search Engine for Vision-Language Tasks(https://arxiv.org/abs/2412.18072)
• Keywords: robust
• Abstract: With advances in foundational and vision-language models, and effective fine-tuning techniques, a large number of both general and special-purpose models have been developed for a variety of visual tasks. Despite the flexibility and accessibility of these models, no single model is able to handle all tasks and/or applications that may be envisioned by potential users. Recent approaches, such as visual programming and multimodal LLMs with integrated tools aim to tackle complex visual tasks, by way of program synthesis. However, such approaches overlook user constraints (e.g., performance / computational needs), produce test-time sample-specific solutions that are difficult to deploy, and, sometimes, require low-level instructions that maybe beyond the abilities of a naive user. To address these limitations, we introduce MMFactory, a universal framework that includes model and metrics routing components, acting like a solution search engine across various available models. Based on a task description and few sample input-output pairs and (optionally) resource and/or performance constraints, MMFactory can suggest a diverse pool of programmatic solutions by instantiating and combining visio-lingual tools from its model repository. In addition to synthesizing these solutions, MMFactory also proposes metrics and benchmarks performance / resource characteristics, allowing users to pick a solution that meets their unique design constraints. From the technical perspective, we also introduced a committee-based solution proposer that leverages multi-agent LLM conversation to generate executable, diverse, universal, and robust solutions for the user. Experimental results show that MMFactory outperforms existing methods by delivering state-of-the-art solutions tailored to user problem specifications. Project page is available at this https URL.

Title: COMO: Cross-Mamba Interaction and Offset-Guided Fusion for Multimodal Object Detection

• Authors: Chang Liu, Xin Ma, Xiaochen Yang, Yuxiang Zhang, Yanni Dong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18076
• Pdf URL: https://arxiv.org/pdf/2412.18076
• Copy Paste: [[2412.18076]] COMO: Cross-Mamba Interaction and Offset-Guided Fusion for Multimodal Object Detection(https://arxiv.org/abs/2412.18076)
• Keywords: transformer
• Abstract: Single-modal object detection tasks often experience performance degradation when encountering diverse scenarios. In contrast, multimodal object detection tasks can offer more comprehensive information about object features by integrating data from various modalities. Current multimodal object detection methods generally use various fusion techniques, including conventional neural networks and transformer-based models, to implement feature fusion strategies and achieve complementary information. However, since multimodal images are captured by different sensors, there are often misalignments between them, making direct matching challenging. This misalignment hinders the ability to establish strong correlations for the same object across different modalities. In this paper, we propose a novel approach called the CrOss-Mamba interaction and Offset-guided fusion (COMO) framework for multimodal object detection tasks. The COMO framework employs the cross-mamba technique to formulate feature interaction equations, enabling multimodal serialized state computation. This results in interactive fusion outputs while reducing computational overhead and improving efficiency. Additionally, COMO leverages high-level features, which are less affected by misalignment, to facilitate interaction and transfer complementary information between modalities, addressing the positional offset challenges caused by variations in camera angles and capture times. Furthermore, COMO incorporates a global and local scanning mechanism in the cross-mamba module to capture features with local correlation, particularly in remote sensing images. To preserve low-level features, the offset-guided fusion mechanism ensures effective multiscale feature utilization, allowing the construction of a multiscale fusion data cube that enhances detection performance.

Title: Convolutional Prompting for Broad-Domain Retinal Vessel Segmentation

• Authors: Qijie Wei, Weihong Yu, Xirong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18089
• Pdf URL: https://arxiv.org/pdf/2412.18089
• Copy Paste: [[2412.18089]] Convolutional Prompting for Broad-Domain Retinal Vessel Segmentation(https://arxiv.org/abs/2412.18089)
• Keywords: attack, segmentation
• Abstract: Previous research on retinal vessel segmentation is targeted at a specific image domain, mostly color fundus photography (CFP). In this paper we make a brave attempt to attack a more challenging task of broad-domain retinal vessel segmentation (BD-RVS), which is to develop a unified model applicable to varied domains including CFP, SLO, UWF, OCTA and FFA. To that end, we propose Dual Convoltuional Prompting (DCP) that learns to extract domain-specific features by localized prompting along both position and channel dimensions. DCP is designed as a plug-in module that can effectively turn a R2AU-Net based vessel segmentation network to a unified model, yet without the need of modifying its network structure. For evaluation we build a broad-domain set using five public domain-specific datasets including ROSSA, FIVES, IOSTAR, PRIME-FP20 and VAMPIRE. In order to benchmark BD-RVS on the broad-domain dataset, we re-purpose a number of existing methods originally developed in other contexts, producing eight baseline methods in total. Extensive experiments show the the proposed method compares favorably against the baselines for BD-RVS.

Title: Molly: Making Large Language Model Agents Solve Python Problem More Logically

• Authors: Rui Xiao, Jiong Wang, Lu Han, Na Zong, Han Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18093
• Pdf URL: https://arxiv.org/pdf/2412.18093
• Copy Paste: [[2412.18093]] Molly: Making Large Language Model Agents Solve Python Problem More Logically(https://arxiv.org/abs/2412.18093)
• Keywords: large language model
• Abstract: Applying large language models (LLMs) as teaching assists has attracted much attention as an integral part of intelligent education, particularly in computing courses. To reduce the gap between the LLMs and the computer programming education expert, fine-tuning and retrieval augmented generation (RAG) are the two mainstream methods in existing researches. However, fine-tuning for specific tasks is resource-intensive and may diminish the model`s generalization capabilities. RAG can perform well on reducing the illusion of LLMs, but the generation of irrelevant factual content during reasoning can cause significant confusion for learners. To address these problems, we introduce the Molly agent, focusing on solving the proposed problem encountered by learners when learning Python programming language. Our agent automatically parse the learners' questioning intent through a scenario-based interaction, enabling precise retrieval of relevant documents from the constructed knowledge base. At generation stage, the agent reflect on the generated responses to ensure that they not only align with factual content but also effectively answer the user's queries. Extensive experimentation on a constructed Chinese Python QA dataset shows the effectiveness of the Molly agent, indicating an enhancement in its performance for providing useful responses to Python questions.

Title: Beyond the Known: Enhancing Open Set Domain Adaptation with Unknown Exploration

• Authors: Lucas Fernando Alvarenga e Silva, Samuel Felipe dos Santos, Nicu Sebe, Jurandy Almeida
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18105
• Pdf URL: https://arxiv.org/pdf/2412.18105
• Copy Paste: [[2412.18105]] Beyond the Known: Enhancing Open Set Domain Adaptation with Unknown Exploration(https://arxiv.org/abs/2412.18105)
• Keywords: generative
• Abstract: Convolutional neural networks (CNNs) can learn directly from raw data, resulting in exceptional performance across various research areas. However, factors present in non-controllable environments such as unlabeled datasets with varying levels of domain and category shift can reduce model accuracy. The Open Set Domain Adaptation (OSDA) is a challenging problem that arises when both of these issues occur together. Existing OSDA approaches in literature only align known classes or use supervised training to learn unknown classes as a single new category. In this work, we introduce a new approach to improve OSDA techniques by extracting a set of high-confidence unknown instances and using it as a hard constraint to tighten the classification boundaries. Specifically, we use a new loss constraint that is evaluated in three different ways: (1) using pristine negative instances directly; (2) using data augmentation techniques to create randomly transformed negatives; and (3) with generated synthetic negatives containing adversarial features. We analyze different strategies to improve the discriminator and the training of the Generative Adversarial Network (GAN) used to generate synthetic negatives. We conducted extensive experiments and analysis on OVANet using three widely-used public benchmarks, the Office-31, Office-Home, and VisDA datasets. We were able to achieve similar H-score to other state-of-the-art methods, while increasing the accuracy on unknown categories.

Title: Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach

• Authors: Jing Bi, Junjia Guo, Yunlong Tang, Lianggong Bruce Wen, Zhang Liu, Chenliang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18108
• Pdf URL: https://arxiv.org/pdf/2412.18108
• Copy Paste: [[2412.18108]] Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach(https://arxiv.org/abs/2412.18108)
• Keywords: large language model
• Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated remarkable progress in visual understanding. This impressive leap raises a compelling question: how can language models, initially trained solely on linguistic data, effectively interpret and process visual content? This paper aims to address this question with systematic investigation across 4 model families and 4 model scales, uncovering a unique class of attention heads that focus specifically on visual content. Our analysis reveals a strong correlation between the behavior of these attention heads, the distribution of attention weights, and their concentration on visual tokens within the input. These findings enhance our understanding of how LLMs adapt to multimodal tasks, demonstrating their potential to bridge the gap between textual and visual understanding. This work paves the way for the development of AI systems capable of engaging with diverse modalities.

Title: AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Models

• Authors: Yiming Wang, Jiahao Chen, Qingming Li, Xing Yang, Shouling Ji
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18123
• Pdf URL: https://arxiv.org/pdf/2412.18123
• Copy Paste: [[2412.18123]] AEIOU: A Unified Defense Framework against NSFW Prompts in Text-to-Image Models(https://arxiv.org/abs/2412.18123)
• Keywords: defense, attack, robust
• Abstract: As text-to-image (T2I) models continue to advance and gain widespread adoption, their associated safety issues are becoming increasingly prominent. Malicious users often exploit these models to generate Not-Safe-for-Work (NSFW) images using harmful or adversarial prompts, highlighting the critical need for robust safeguards to ensure the integrity and compliance of model outputs. Current internal safeguards frequently degrade image quality, while external detection methods often suffer from low accuracy and inefficiency. In this paper, we introduce AEIOU, a defense framework that is Adaptable, Efficient, Interpretable, Optimizable, and Unified against NSFW prompts in T2I models. AEIOU extracts NSFW features from the hidden states of the model's text encoder, utilizing the separable nature of these features to detect NSFW prompts. The detection process is efficient, requiring minimal inference time. AEIOU also offers real-time interpretation of results and supports optimization through data augmentation techniques. The framework is versatile, accommodating various T2I architectures. Our extensive experiments show that AEIOU significantly outperforms both commercial and open-source moderation tools, achieving over 95% accuracy across all datasets and improving efficiency by at least tenfold. It effectively counters adaptive attacks and excels in few-shot and multi-label scenarios.

Title: VisionLLM-based Multimodal Fusion Network for Glottic Carcinoma Early Detection

• Authors: Zhaohui Jin, Yi Shuai, Yongcheng Li, Lingcong Cai, Yun Li, Huifen Liu, Xiaomao Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18124
• Pdf URL: https://arxiv.org/pdf/2412.18124
• Copy Paste: [[2412.18124]] VisionLLM-based Multimodal Fusion Network for Glottic Carcinoma Early Detection(https://arxiv.org/abs/2412.18124)
• Keywords: robust, large language model
• Abstract: The early detection of glottic carcinoma is critical for improving patient outcomes, as it enables timely intervention, preserves vocal function, and significantly reduces the risk of tumor progression and metastasis. However, the similarity in morphology between glottic carcinoma and vocal cord dysplasia results in suboptimal detection accuracy. To address this issue, we propose a vision large language model-based (VisionLLM-based) multimodal fusion network for glottic carcinoma detection, known as MMGC-Net. By integrating image and text modalities, multimodal models can capture complementary information, leading to more accurate and robust predictions. In this paper, we collect a private real glottic carcinoma dataset named SYSU1H from the First Affiliated Hospital of Sun Yat-sen University, with 5,799 image-text pairs. We leverage an image encoder and additional Q-Former to extract vision embeddings and the Large Language Model Meta AI (Llama3) to obtain text embeddings. These modalities are then integrated through a laryngeal feature fusion block, enabling a comprehensive integration of image and text features, thereby improving the glottic carcinoma identification performance. Extensive experiments on the SYSU1H dataset demonstrate that MMGC-Net can achieve state-of-the-art performance, which is superior to previous multimodal models.

Title: UniPLV: Towards Label-Efficient Open-World 3D Scene Understanding by Regional Visual Language Supervision

• Authors: Yuru Wang, Songtao Wang, Zehan Zhang, Xinyan Lu, Changwei Cai, Hao Li, Fu Liu, Peng Jia, Xianpeng Lang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18131
• Pdf URL: https://arxiv.org/pdf/2412.18131
• Copy Paste: [[2412.18131]] UniPLV: Towards Label-Efficient Open-World 3D Scene Understanding by Regional Visual Language Supervision(https://arxiv.org/abs/2412.18131)
• Keywords: segmentation
• Abstract: We present UniPLV, a powerful framework that unifies point clouds, images and text in a single learning paradigm for open-world 3D scene understanding. UniPLV employs the image modal as a bridge to co-embed 3D points with pre-aligned images and text in a shared feature space without requiring carefully crafted point cloud text pairs. To accomplish multi-modal alignment, we propose two key strategies:(i) logit and feature distillation modules between images and point clouds, and (ii) a vison-point matching module is given to explicitly correct the misalignment caused by points to pixels projection. To further improve the performance of our unified framework, we adopt four task-specific losses and a two-stage training strategy. Extensive experiments show that our method outperforms the state-of-the-art methods by an average of 15.6% and 14.8% for semantic segmentation over Base-Annotated and Annotation-Free tasks, respectively. The code will be released later.

Title: LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment

• Authors: Binrui Zeng, Bin Ji, Xiaodong Liu, Jie Yu, Shasha Li, Jun Ma, Xiaopeng Li, Shangwen Wang, Xinran Hong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18135
• Pdf URL: https://arxiv.org/pdf/2412.18135
• Copy Paste: [[2412.18135]] LSAQ: Layer-Specific Adaptive Quantization for Large Language Model Deployment(https://arxiv.org/abs/2412.18135)
• Keywords: large language model
• Abstract: As large language models (LLMs) demonstrate exceptional performance across various domains, the deployment of these models on edge devices has emerged as a new trend. Quantization techniques, which reduce the size and memory footprint of LLMs, are effective for enabling deployment on resource-constrained edge devices. However, existing one-size-fits-all quantization methods often fail to dynamically adjust the memory consumption of LLMs based on specific hardware characteristics and usage scenarios. To address this limitation, we propose LSAQ (Layer-Specific Adaptive Quantization), a system for adaptive quantization and dynamic deployment of LLMs based on layer importance. LSAQ evaluates layer importance by constructing top-k token sets from the inputs and outputs of each layer and calculating their Jaccard coefficient. Using this evaluation, the system adaptively adjusts quantization strategies in real time according to the resource availability of edge devices, assigning different precision levels to layers of varying importance. This approach significantly reduces the storage requirements of LLMs while maintaining model performance, enabling efficient deployment across diverse hardware platforms and usage scenarios.

Title: ERVD: An Efficient and Robust ViT-Based Distillation Framework for Remote Sensing Image Retrieval

• Authors: Le Dong, Qixuan Cao, Lei Pu, Fangfang Wu, Weisheng Dong, Xin Li, Guangming Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18136
• Pdf URL: https://arxiv.org/pdf/2412.18136
• Copy Paste: [[2412.18136]] ERVD: An Efficient and Robust ViT-Based Distillation Framework for Remote Sensing Image Retrieval(https://arxiv.org/abs/2412.18136)
• Keywords: robust
• Abstract: ERVD: An Efficient and Robust ViT-Based Distillation Framework for Remote Sensing Image Retrieval

Title: Ensuring Consistency for In-Image Translation

• Authors: Chengpeng Fu, Xiaocheng Feng, Yichong Huang, Wenshuai Huo, Baohang Li, Zhirui Zhang, Yunfei Lu, Dandan Tu, Duyu Tang, Hui Wang, Bing Qin, Ting Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18139
• Pdf URL: https://arxiv.org/pdf/2412.18139
• Copy Paste: [[2412.18139]] Ensuring Consistency for In-Image Translation(https://arxiv.org/abs/2412.18139)
• Keywords: diffusion, large language model
• Abstract: The in-image machine translation task involves translating text embedded within images, with the translated results presented in image format. While this task has numerous applications in various scenarios such as film poster translation and everyday scene image translation, existing methods frequently neglect the aspect of consistency throughout this process. We propose the need to uphold two types of consistency in this task: translation consistency and image generation consistency. The former entails incorporating image information during translation, while the latter involves maintaining consistency between the style of the text-image and the original image, ensuring background integrity. To address these consistency requirements, we introduce a novel two-stage framework named HCIIT (High-Consistency In-Image Translation) which involves text-image translation using a multimodal multilingual large language model in the first stage and image backfilling with a diffusion model in the second stage. Chain of thought learning is utilized in the first stage to enhance the model's ability to leverage image information during translation. Subsequently, a diffusion model trained for style-consistent text-image generation ensures uniformity in text style within images and preserves background details. A dataset comprising 400,000 style-consistent pseudo text-image pairs is curated for model training. Results obtained on both curated test sets and authentic image test sets validate the effectiveness of our framework in ensuring consistency and producing high-quality translated images.

Title: Dense-Face: Personalized Face Generation Model via Dense Annotation Prediction

• Authors: Xiao Guo, Manh Tran, Jiaxin Cheng, Xiaoming Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18149
• Pdf URL: https://arxiv.org/pdf/2412.18149
• Copy Paste: [[2412.18149]] Dense-Face: Personalized Face Generation Model via Dense Annotation Prediction(https://arxiv.org/abs/2412.18149)
• Keywords: diffusion
• Abstract: The text-to-image (T2I) personalization diffusion model can generate images of the novel concept based on the user input text caption. However, existing T2I personalized methods either require test-time fine-tuning or fail to generate images that align well with the given text caption. In this work, we propose a new T2I personalization diffusion model, Dense-Face, which can generate face images with a consistent identity as the given reference subject and align well with the text caption. Specifically, we introduce a pose-controllable adapter for the high-fidelity image generation while maintaining the text-based editing ability of the pre-trained stable diffusion (SD). Additionally, we use internal features of the SD UNet to predict dense face annotations, enabling the proposed method to gain domain knowledge in face generation. Empirically, our method achieves state-of-the-art or competitive generation performance in image-text alignment, identity preservation, and pose control.

Title: EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation

• Authors: Shuhao Han, Haotian Fan, Jiachen Fu, Liang Li, Tao Li, Junhui Cui, Yunqiu Wang, Yang Tai, Jingwei Sun, Chunle Guo, Chongyi Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18150
• Pdf URL: https://arxiv.org/pdf/2412.18150
• Copy Paste: [[2412.18150]] EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation(https://arxiv.org/abs/2412.18150)
• Keywords: generative
• Abstract: Recently, Text-to-Image (T2I) generation models have achieved significant advancements. Correspondingly, many automated metrics have emerged to evaluate the image-text alignment capabilities of generative models. However, the performance comparison among these automated metrics is limited by existing small datasets. Additionally, these datasets lack the capacity to assess the performance of automated metrics at a fine-grained level. In this study, we contribute an EvalMuse-40K benchmark, gathering 40K image-text pairs with fine-grained human annotations for image-text alignment-related tasks. In the construction process, we employ various strategies such as balanced prompt sampling and data re-annotation to ensure the diversity and reliability of our benchmark. This allows us to comprehensively evaluate the effectiveness of image-text alignment metrics for T2I models. Meanwhile, we introduce two new methods to evaluate the image-text alignment capabilities of T2I models: FGA-BLIP2 which involves end-to-end fine-tuning of a vision-language model to produce fine-grained image-text alignment scores and PN-VQA which adopts a novel positive-negative VQA manner in VQA models for zero-shot fine-grained evaluation. Both methods achieve impressive performance in image-text alignment evaluations. We also use our methods to rank current AIGC models, in which the results can serve as a reference source for future study and promote the development of T2I generation. The data and code will be made publicly available.

Title: CoAM: Corpus of All-Type Multiword Expressions

• Authors: Yusuke Ide, Joshua Tanner, Adam Nohejl, Jacob Hoffman, Justin Vasselli, Hidetaka Kamigaito, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18151
• Pdf URL: https://arxiv.org/pdf/2412.18151
• Copy Paste: [[2412.18151]] CoAM: Corpus of All-Type Multiword Expressions(https://arxiv.org/abs/2412.18151)
• Keywords: large language model
• Abstract: Multiword expressions (MWEs) refer to idiomatic sequences of multiple words. MWE identification, i.e., detecting MWEs in text, can play a key role in downstream tasks such as machine translation. Existing datasets for MWE identification are inconsistently annotated, limited to a single type of MWE, or limited in size. To enable reliable and comprehensive evaluation, we created CoAM: Corpus of All-Type Multiword Expressions, a dataset of 1.3K sentences constructed through a multi-step process to enhance data quality consisting of human annotation, human review, and automated consistency checking. MWEs in CoAM are tagged with MWE types, such as Noun and Verb, to enable fine-grained error analysis. Annotations for CoAM were collected using a new interface created with our interface generator, which allows easy and flexible annotation of MWEs in any form, including discontinuous ones. Through experiments using CoAM, we find that a fine-tuned large language model outperforms the current state-of-the-art approach for MWE identification. Furthermore, analysis using our MWE type tagged data reveals that Verb MWEs are easier than Noun MWEs to identify across approaches.

Title: DepthLab: From Partial to Complete

• Authors: Zhiheng Liu, Ka Leong Cheng, Qiuyu Wang, Shuzhe Wang, Hao Ouyang, Bin Tan, Kai Zhu, Yujun Shen, Qifeng Chen, Ping Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18153
• Pdf URL: https://arxiv.org/pdf/2412.18153
• Copy Paste: [[2412.18153]] DepthLab: From Partial to Complete(https://arxiv.org/abs/2412.18153)
• Keywords: diffusion
• Abstract: Missing values remain a common challenge for depth data across its wide range of applications, stemming from various causes like incomplete data acquisition and perspective alteration. This work bridges this gap with DepthLab, a foundation depth inpainting model powered by image diffusion priors. Our model features two notable strengths: (1) it demonstrates resilience to depth-deficient regions, providing reliable completion for both continuous areas and isolated points, and (2) it faithfully preserves scale consistency with the conditioned known depth when filling in missing values. Drawing on these advantages, our approach proves its worth in various downstream tasks, including 3D scene inpainting, text-to-3D scene generation, sparse-view reconstruction with DUST3R, and LiDAR depth completion, exceeding current solutions in both numerical performance and visual quality. Our project page with source code is available at this https URL.

Title: Semantics Disentanglement and Composition for Versatile Codec toward both Human-eye Perception and Machine Vision Task

• Authors: Jinming Liu, Yuntao Wei, Junyan Lin, Shengyang Zhao, Heming Sun, Zhibo Chen, Wenjun Zeng, Xin Jin
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.18158
• Pdf URL: https://arxiv.org/pdf/2412.18158
• Copy Paste: [[2412.18158]] Semantics Disentanglement and Composition for Versatile Codec toward both Human-eye Perception and Machine Vision Task(https://arxiv.org/abs/2412.18158)
• Keywords: robust, generative
• Abstract: While learned image compression methods have achieved impressive results in either human visual perception or machine vision tasks, they are often specialized only for one domain. This drawback limits their versatility and generalizability across scenarios and also requires retraining to adapt to new applications-a process that adds significant complexity and cost in real-world scenarios. In this study, we introduce an innovative semantics DISentanglement and COmposition VERsatile codec (DISCOVER) to simultaneously enhance human-eye perception and machine vision tasks. The approach derives a set of labels per task through multimodal large models, which grounding models are then applied for precise localization, enabling a comprehensive understanding and disentanglement of image components at the encoder side. At the decoding stage, a comprehensive reconstruction of the image is achieved by leveraging these encoded components alongside priors from generative models, thereby optimizing performance for both human visual perception and machine-based analytical tasks. Extensive experimental evaluations substantiate the robustness and effectiveness of DISCOVER, demonstrating superior performance in fulfilling the dual objectives of human and machine vision requirements.

Title: Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence

• Authors: Yinbin Han, Meisam Razaviyayn, Renyuan Xu
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2412.18164
• Pdf URL: https://arxiv.org/pdf/2412.18164
• Copy Paste: [[2412.18164]] Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence(https://arxiv.org/abs/2412.18164)
• Keywords: diffusion, generative
• Abstract: Diffusion models have emerged as powerful tools for generative modeling, demonstrating exceptional capability in capturing target data distributions from large datasets. However, fine-tuning these massive models for specific downstream tasks, constraints, and human preferences remains a critical challenge. While recent advances have leveraged reinforcement learning algorithms to tackle this problem, much of the progress has been empirical, with limited theoretical understanding. To bridge this gap, we propose a stochastic control framework for fine-tuning diffusion models. Building on denoising diffusion probabilistic models as the pre-trained reference dynamics, our approach integrates linear dynamics control with Kullback-Leibler regularization. We establish the well-posedness and regularity of the stochastic control problem and develop a policy iteration algorithm (PI-FT) for numerical solution. We show that PI-FT achieves global convergence at a linear rate. Unlike existing work that assumes regularities throughout training, we prove that the control and value sequences generated by the algorithm maintain the regularity. Additionally, we explore extensions of our framework to parametric settings and continuous-time formulations.

Title: Parallel Neural Computing for Scene Understanding from LiDAR Perception in Autonomous Racing

• Authors: Suwesh Prasad Sah
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18165
• Pdf URL: https://arxiv.org/pdf/2412.18165
• Copy Paste: [[2412.18165]] Parallel Neural Computing for Scene Understanding from LiDAR Perception in Autonomous Racing(https://arxiv.org/abs/2412.18165)
• Keywords: segmentation
• Abstract: Autonomous driving in high-speed racing, as opposed to urban environments, presents significant challenges in scene understanding due to rapid changes in the track environment. Traditional sequential network approaches may struggle to meet the real-time knowledge and decision-making demands of an autonomous agent covering large displacements in a short time. This paper proposes a novel baseline architecture for developing sophisticated models capable of true hardware-enabled parallelism, achieving neural processing speeds that mirror the agent's high velocity. The proposed model (Parallel Perception Network (PPN)) consists of two independent neural networks, segmentation and reconstruction networks, running parallelly on separate accelerated hardware. The model takes raw 3D point cloud data from the LiDAR sensor as input and converts it into a 2D Bird's Eye View Map on both devices. Each network independently extracts its input features along space and time dimensions and produces outputs parallelly. The proposed method's model is trained on a system with two NVIDIA T4 GPUs, using a combination of loss functions, including edge preservation, and demonstrates a 2x speedup in model inference time compared to a sequential configuration. Implementation is available at: this https URL. Learned parameters of the trained networks are provided at: this https URL.

Title: Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models

• Authors: Xiaomeng Hu, Pin-Yu Chen, Tsung-Yi Ho
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.18171
• Pdf URL: https://arxiv.org/pdf/2412.18171
• Copy Paste: [[2412.18171]] Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models(https://arxiv.org/abs/2412.18171)
• Keywords: protect, defense, attack, large language model
• Abstract: Large Language Models (LLMs) are increasingly being integrated into services such as ChatGPT to provide responses to user queries. To mitigate potential harm and prevent misuse, there have been concerted efforts to align the LLMs with human values and legal compliance by incorporating various techniques, such as Reinforcement Learning from Human Feedback (RLHF), into the training of the LLMs. However, recent research has exposed that even aligned LLMs are susceptible to adversarial manipulations known as Jailbreak Attacks. To address this challenge, this paper proposes a method called Token Highlighter to inspect and mitigate the potential jailbreak threats in the user query. Token Highlighter introduced a concept called Affirmation Loss to measure the LLM's willingness to answer the user query. It then uses the gradient of Affirmation Loss for each token in the user query to locate the jailbreak-critical tokens. Further, Token Highlighter exploits our proposed Soft Removal technique to mitigate the jailbreak effects of critical tokens via shrinking their token embeddings. Experimental results on two aligned LLMs (LLaMA-2 and Vicuna-V1.5) demonstrate that the proposed method can effectively defend against a variety of Jailbreak Attacks while maintaining competent performance on benign questions of the AlpacaEval benchmark. In addition, Token Highlighter is a cost-effective and interpretable defense because it only needs to query the protected LLM once to compute the Affirmation Loss and can highlight the critical tokens upon refusal.

Title: VisionGRU: A Linear-Complexity RNN Model for Efficient Image Analysis

• Authors: Shicheng Yin, Kaixuan Yin, Weixing Chen, Enbo Huang, Yang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18178
• Pdf URL: https://arxiv.org/pdf/2412.18178
• Copy Paste: [[2412.18178]] VisionGRU: A Linear-Complexity RNN Model for Efficient Image Analysis(https://arxiv.org/abs/2412.18178)
• Keywords: extraction, transformer, segmentation
• Abstract: Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) are two dominant models for image analysis. While CNNs excel at extracting multi-scale features and ViTs effectively capture global dependencies, both suffer from high computational costs, particularly when processing high-resolution images. Recently, state-space models (SSMs) and recurrent neural networks (RNNs) have attracted attention due to their efficiency. However, their performance in image classification tasks remains limited. To address these challenges, this paper introduces VisionGRU, a novel RNN-based architecture designed for efficient image classification. VisionGRU leverages a simplified Gated Recurrent Unit (minGRU) to process large-scale image features with linear complexity. It divides images into smaller patches and progressively reduces the sequence length while increasing the channel depth, thus facilitating multi-scale feature extraction. A hierarchical 2DGRU module with bidirectional scanning captures both local and global contexts, improving long-range dependency modeling, particularly for tasks like semantic segmentation. Experimental results on the ImageNet and ADE20K datasets demonstrate that VisionGRU outperforms ViTs, significantly reducing memory usage and computational costs, especially for high-resolution images. These findings underscore the potential of RNN-based approaches for developing efficient and scalable computer vision solutions. Codes will be available at this https URL.

Title: Unified Stochastic Framework for Neural Network Quantization and Pruning

• Authors: Haoyu Zhang, Rayan Saab
• Subjects: cs.LG, math.NA, math.PR
• Abstract URL: https://arxiv.org/abs/2412.18184
• Pdf URL: https://arxiv.org/pdf/2412.18184
• Copy Paste: [[2412.18184]] Unified Stochastic Framework for Neural Network Quantization and Pruning(https://arxiv.org/abs/2412.18184)
• Keywords: robust
• Abstract: Quantization and pruning are two essential techniques for compressing neural networks, yet they are often treated independently, with limited theoretical analysis connecting them. This paper introduces a unified framework for post-training quantization and pruning using stochastic path-following algorithms. Our approach builds on the Stochastic Path Following Quantization (SPFQ) method, extending its applicability to pruning and low-bit quantization, including challenging 1-bit regimes. By incorporating a scaling parameter and generalizing the stochastic operator, the proposed method achieves robust error correction and yields rigorous theoretical error bounds for both quantization and pruning as well as their combination.

Title: TextMatch: Enhancing Image-Text Consistency Through Multimodal Optimization

• Authors: Yucong Luo, Mingyue Cheng, Jie Ouyang, Xiaoyu Tao, Qi Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18185
• Pdf URL: https://arxiv.org/pdf/2412.18185
• Copy Paste: [[2412.18185]] TextMatch: Enhancing Image-Text Consistency Through Multimodal Optimization(https://arxiv.org/abs/2412.18185)
• Keywords: generative, large language model
• Abstract: Text-to-image generative models excel in creating images from text but struggle with ensuring alignment and consistency between outputs and prompts. This paper introduces TextMatch, a novel framework that leverages multimodal optimization to address image-text discrepancies in text-to-image (T2I) generation and editing. TextMatch employs a scoring strategy powered by large language models (LLMs) and visual question-answering (VQA) models to evaluate semantic consistency between prompts and generated images. By integrating multimodal in-context learning and chain of thought reasoning, our method dynamically refines prompts through iterative optimization. This process ensures that the generated images better capture user intent of, resulting in higher fidelity and relevance. Extensive experiments demonstrate that TextMatch significantly improves text-image consistency across multiple benchmarks, establishing a reliable framework for advancing the capabilities of text-to-image generative models. Our code is available at this https URL.

Title: Robustness-aware Automatic Prompt Optimization

• Authors: Zeru Shi, Zhenting Wang, Yongye Su, Weidi Luo, Fan Yang, Yongfeng Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18196
• Pdf URL: https://arxiv.org/pdf/2412.18196
• Copy Paste: [[2412.18196]] Robustness-aware Automatic Prompt Optimization(https://arxiv.org/abs/2412.18196)
• Keywords: attack, robust, large language model
• Abstract: The performance of Large Language Models (LLMs) is based on the quality of the prompts and the semantic and structural integrity information of the input data. However, current prompt generation methods primarily focus on generating prompts for clean input data, often overlooking the impact of perturbed inputs on prompt performance. To address this limitation, we propose BATprompt (By Adversarial Training prompt), a novel method for prompt generation designed to withstand input perturbations (such as typos in the input). Inspired by adversarial training techniques, BATprompt demonstrates strong performance on a variety of perturbed tasks through a two-step process: adversarial perturbation and iterative optimization on unperturbed input via LLM. Unlike conventional adversarial attack methods, BATprompt avoids reliance on real gradients or model parameters. Instead, it leverages the advanced reasoning, language understanding and self reflection capabilities of LLMs to simulate gradients, guiding the generation of adversarial perturbations and optimizing prompt performance. In our experiments, we evaluate BATprompt on multiple datasets across both language understanding and generation tasks. The results indicate that BATprompt outperforms existing prompt generation methods, delivering superior robustness and performance under diverse perturbation scenarios.

Title: Leveraging Deep Learning with Multi-Head Attention for Accurate Extraction of Medicine from Handwritten Prescriptions

• Authors: Usman Ali, Sahil Ranmbail, Muhammad Nadeem, Hamid Ishfaq, Muhammad Umer Ramzan, Waqas Ali
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.18199
• Pdf URL: https://arxiv.org/pdf/2412.18199
• Copy Paste: [[2412.18199]] Leveraging Deep Learning with Multi-Head Attention for Accurate Extraction of Medicine from Handwritten Prescriptions(https://arxiv.org/abs/2412.18199)
• Keywords: robust, extraction, transformer
• Abstract: Extracting medication names from handwritten doctor prescriptions is challenging due to the wide variability in handwriting styles and prescription formats. This paper presents a robust method for extracting medicine names using a combination of Mask R-CNN and Transformer-based Optical Character Recognition (TrOCR) with Multi-Head Attention and Positional Embeddings. A novel dataset, featuring diverse handwritten prescriptions from various regions of Pakistan, was utilized to fine-tune the model on different handwriting styles. The Mask R-CNN model segments the prescription images to focus on the medicinal sections, while the TrOCR model, enhanced by Multi-Head Attention and Positional Embeddings, transcribes the isolated text. The transcribed text is then matched against a pre-existing database for accurate identification. The proposed approach achieved a character error rate (CER) of 1.4% on standard benchmarks, highlighting its potential as a reliable and efficient tool for automating medicine name extraction.

Title: Sharper Error Bounds in Late Fusion Multi-view Clustering Using Eigenvalue Proportion

• Authors: Liang Du, Henghui Jiang, Xiaodong Li, Yiqing Guo, Yan Chen, Feijiang Li, Peng Zhou, Yuhua Qian
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18207
• Pdf URL: https://arxiv.org/pdf/2412.18207
• Copy Paste: [[2412.18207]] Sharper Error Bounds in Late Fusion Multi-view Clustering Using Eigenvalue Proportion(https://arxiv.org/abs/2412.18207)
• Keywords: robust
• Abstract: Multi-view clustering (MVC) aims to integrate complementary information from multiple views to enhance clustering performance. Late Fusion Multi-View Clustering (LFMVC) has shown promise by synthesizing diverse clustering results into a unified consensus. However, current LFMVC methods struggle with noisy and redundant partitions and often fail to capture high-order correlations across views. To address these limitations, we present a novel theoretical framework for analyzing the generalization error bounds of multiple kernel $k$-means, leveraging local Rademacher complexity and principal eigenvalue proportions. Our analysis establishes a convergence rate of $\mathcal{O}(1/n)$, significantly improving upon the existing rate in the order of $\mathcal{O}(\sqrt{k/n})$. Building on this insight, we propose a low-pass graph filtering strategy within a multiple linear $k$-means framework to mitigate noise and redundancy, further refining the principal eigenvalue proportion and enhancing clustering accuracy. Experimental results on benchmark datasets confirm that our approach outperforms state-of-the-art methods in clustering performance and robustness. The related codes is available at this https URL .

Title: Accelerating AIGC Services with Latent Action Diffusion Scheduling in Edge Networks

• Authors: Changfu Xu, Jianxiong Guo, Wanyu Lin, Haodong Zou, Wentao Fan, Tian Wang, Xiaowen Chu, Jiannong Cao
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.18212
• Pdf URL: https://arxiv.org/pdf/2412.18212
• Copy Paste: [[2412.18212]] Accelerating AIGC Services with Latent Action Diffusion Scheduling in Edge Networks(https://arxiv.org/abs/2412.18212)
• Keywords: diffusion
• Abstract: Artificial Intelligence Generated Content (AIGC) has gained significant popularity for creating diverse content. Current AIGC models primarily focus on content quality within a centralized framework, resulting in a high service delay and negative user experiences. However, not only does the workload of an AIGC task depend on the AIGC model's complexity rather than the amount of data, but the large model and its multi-layer encoder structure also result in a huge demand for computational and memory resources. These unique characteristics pose new challenges in its modeling, deployment, and scheduling at edge networks. Thus, we model an offloading problem among edges for providing real AIGC services and propose LAD-TS, a novel Latent Action Diffusion-based Task Scheduling method that orchestrates multiple edge servers for expedited AIGC services. The LAD-TS generates a near-optimal offloading decision by leveraging the diffusion model's conditional generation capability and the reinforcement learning's environment interaction ability, thereby minimizing the service delays under multiple resource constraints. Meanwhile, a latent action diffusion strategy is designed to guide decision generation by utilizing historical action probability, enabling rapid achievement of near-optimal decisions. Furthermore, we develop DEdgeAI, a prototype edge system with a refined AIGC model deployment to implement and evaluate our LAD-TS method. DEdgeAI provides a real AIGC service for users, demonstrating up to 29.18% shorter service delays than the current five representative AIGC platforms. We release our open-source code at this https URL.

Title: ICM-Assistant: Instruction-tuning Multimodal Large Language Models for Rule-based Explainable Image Content Moderation

• Authors: Mengyang Wu, Yuzhi Zhao, Jialun Cao, Mingjie Xu, Zhongming Jiang, Xuehui Wang, Qinbin Li, Guangneng Hu, Shengchao Qin, Chi-Wing Fu
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18216
• Pdf URL: https://arxiv.org/pdf/2412.18216
• Copy Paste: [[2412.18216]] ICM-Assistant: Instruction-tuning Multimodal Large Language Models for Rule-based Explainable Image Content Moderation(https://arxiv.org/abs/2412.18216)
• Keywords: protect, large language model
• Abstract: Controversial contents largely inundate the Internet, infringing various cultural norms and child protection standards. Traditional Image Content Moderation (ICM) models fall short in producing precise moderation decisions for diverse standards, while recent multimodal large language models (MLLMs), when adopted to general rule-based ICM, often produce classification and explanation results that are inconsistent with human moderators. Aiming at flexible, explainable, and accurate ICM, we design a novel rule-based dataset generation pipeline, decomposing concise human-defined rules and leveraging well-designed multi-stage prompts to enrich short explicit image annotations. Our ICM-Instruct dataset includes detailed moderation explanation and moderation Q-A pairs. Built upon it, we create our ICM-Assistant model in the framework of rule-based ICM, making it readily applicable in real practice. Our ICM-Assistant model demonstrates exceptional performance and flexibility. Specifically, it significantly outperforms existing approaches on various sources, improving both the moderation classification (36.8\% on average) and moderation explanation quality (26.6\% on average) consistently over existing MLLMs. Code/Data is available at this https URL.

Title: On the Effectiveness of Adversarial Training on Malware Classifiers

• Authors: Hamid Bostani, Jacopo Cortellazzi, Daniel Arp, Fabio Pierazzi, Veelasha Moonsamy, Lorenzo Cavallaro
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.18218
• Pdf URL: https://arxiv.org/pdf/2412.18218
• Copy Paste: [[2412.18218]] On the Effectiveness of Adversarial Training on Malware Classifiers(https://arxiv.org/abs/2412.18218)
• Keywords: attack, robust
• Abstract: Adversarial Training (AT) has been widely applied to harden learning-based classifiers against adversarial evasive attacks. However, its effectiveness in identifying and strengthening vulnerable areas of the model's decision space while maintaining high performance on clean data of malware classifiers remains an under-explored area. In this context, the robustness that AT achieves has often been assessed against unrealistic or weak adversarial attacks, which negatively affect performance on clean data and are arguably no longer threats. Previous work seems to suggest robustness is a task-dependent property of AT. We instead argue it is a more complex problem that requires exploring AT and the intertwined roles played by certain factors within data, feature representations, classifiers, and robust optimization settings, as well as proper evaluation factors, such as the realism of evasion attacks, to gain a true sense of AT's effectiveness. In our paper, we address this gap by systematically exploring the role such factors have in hardening malware classifiers through AT. Contrary to recent prior work, a key observation of our research and extensive experiments confirm the hypotheses that all such factors influence the actual effectiveness of AT, as demonstrated by the varying degrees of success from our empirical analysis. We identify five evaluation pitfalls that affect state-of-the-art studies and summarize our insights in ten takeaways to draw promising research directions toward better understanding the factors' settings under which adversarial training works at best.

Title: GIMS: Image Matching System Based on Adaptive Graph Construction and Graph Neural Network

• Authors: Xianfeng Song, Yi Zou, Zheng Shi, Zheng Liu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18221
• Pdf URL: https://arxiv.org/pdf/2412.18221
• Copy Paste: [[2412.18221]] GIMS: Image Matching System Based on Adaptive Graph Construction and Graph Neural Network(https://arxiv.org/abs/2412.18221)
• Keywords: robust, transformer
• Abstract: Feature-based image matching has extensive applications in computer vision. Keypoints detected in images can be naturally represented as graph structures, and Graph Neural Networks (GNNs) have been shown to outperform traditional deep learning techniques. Consequently, the paradigm of image matching via GNNs has gained significant prominence in recent academic research. In this paper, we first introduce an innovative adaptive graph construction method that utilizes a filtering mechanism based on distance and dynamic threshold similarity. This method dynamically adjusts the criteria for incorporating new vertices based on the characteristics of existing vertices, allowing for the construction of more precise and robust graph structures while avoiding redundancy. We further combine the vertex processing capabilities of GNNs with the global awareness capabilities of Transformers to enhance the model's representation of spatial and feature information within graph structures. This hybrid model provides a deeper understanding of the interrelationships between vertices and their contributions to the matching process. Additionally, we employ the Sinkhorn algorithm to iteratively solve for optimal matching results. Finally, we validate our system using extensive image datasets and conduct comprehensive comparative experiments. Experimental results demonstrate that our system achieves an average improvement of 3.8x-40.3x in overall matching performance. Additionally, the number of vertices and edges significantly impacts training efficiency and memory usage; therefore, we employ multi-GPU technology to accelerate the training process. Our code is available at this https URL.

Title: Expand VSR Benchmark for VLLM to Expertize in Spatial Rules

• Authors: Peijin Xie, Lin Sun, Bingquan Liu, Dexin Wang, Xiangzheng Zhang, Chengjie Sun, Jiajia Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18224
• Pdf URL: https://arxiv.org/pdf/2412.18224
• Copy Paste: [[2412.18224]] Expand VSR Benchmark for VLLM to Expertize in Spatial Rules(https://arxiv.org/abs/2412.18224)
• Keywords: diffusion, large language model
• Abstract: Distinguishing spatial relations is a basic part of human cognition which requires fine-grained perception on cross-instance. Although benchmarks like MME, MMBench and SEED comprehensively have evaluated various capabilities which already include visual spatial reasoning(VSR). There is still a lack of sufficient quantity and quality evaluation and optimization datasets for Vision Large Language Models(VLLMs) specifically targeting visual positional reasoning. To handle this, we first diagnosed current VLLMs with the VSR dataset and proposed a unified test set. We found current VLLMs to exhibit a contradiction of over-sensitivity to language instructions and under-sensitivity to visual positional information. By expanding the original benchmark from two aspects of tunning data and model structure, we mitigated this phenomenon. To our knowledge, we expanded spatially positioned image data controllably using diffusion models for the first time and integrated original visual encoding(CLIP) with other 3 powerful visual encoders(SigLIP, SAM and DINO). After conducting combination experiments on scaling data and models, we obtained a VLLM VSR Expert(VSRE) that not only generalizes better to different instructions but also accurately distinguishes differences in visual positional information. VSRE achieved over a 27\% increase in accuracy on the VSR test set. It becomes a performant VLLM on the position reasoning of both the VSR dataset and relevant subsets of other evaluation benchmarks. We open-sourced the expanded model with data and Appendix at \url{this https URL} and hope it will accelerate advancements in VLLM on VSR learning.

Title: Band Prompting Aided SAR and Multi-Spectral Data Fusion Framework for Local Climate Zone Classification

• Authors: Haiyan Lan, Shujun Li, Mingjie Xie, Xuanjia Zhao, Hongning Liu, Pengming Feng, Dongli Xu, Guangjun He, Jian Guan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18235
• Pdf URL: https://arxiv.org/pdf/2412.18235
• Copy Paste: [[2412.18235]] Band Prompting Aided SAR and Multi-Spectral Data Fusion Framework for Local Climate Zone Classification(https://arxiv.org/abs/2412.18235)
• Keywords: extraction
• Abstract: Local climate zone (LCZ) classification is of great value for understanding the complex interactions between urban development and local climate. Recent studies have increasingly focused on the fusion of synthetic aperture radar (SAR) and multi-spectral data to improve LCZ classification performance. However, it remains challenging due to the distinct physical properties of these two types of data and the absence of effective fusion guidance. In this paper, a novel band prompting aided data fusion framework is proposed for LCZ classification, namely BP-LCZ, which utilizes textual prompts associated with band groups to guide the model in learning the physical attributes of different bands and semantics of various categories inherent in SAR and multi-spectral data to augment the fused feature, thus enhancing LCZ classification performance. Specifically, a band group prompting (BGP) strategy is introduced to align the visual representation effectively at the level of band groups, which also facilitates a more adequate extraction of semantic information of different bands with textual information. In addition, a multivariate supervised matrix (MSM) based training strategy is proposed to alleviate the problem of positive and negative sample confusion by completing the supervised information. The experimental results demonstrate the effectiveness and superiority of the proposed data fusion framework.

Title: Sch\"odinger Bridge Type Diffusion Models as an Extension of Variational Autoencoders

• Authors: Kentaro Kaba, Reo Shimizu, Masayuki Ohzeki, Yuki Sughiyama
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18237
• Pdf URL: https://arxiv.org/pdf/2412.18237
• Copy Paste: [[2412.18237]] Sch\"odinger Bridge Type Diffusion Models as an Extension of Variational Autoencoders(https://arxiv.org/abs/2412.18237)
• Keywords: diffusion, generative
• Abstract: Generative diffusion models use time-forward and backward stochastic differential equations to connect the data and prior distributions. While conventional diffusion models (e.g., score-based models) only learn the backward process, more flexible frameworks have been proposed to also learn the forward process by employing the Schrödinger bridge (SB). However, due to the complexity of the mathematical structure behind SB-type models, we can not easily give an intuitive understanding of their objective function. In this work, we propose a unified framework to construct diffusion models by reinterpreting the SB-type models as an extension of variational autoencoders. In this context, the data processing inequality plays a crucial role. As a result, we find that the objective function consists of the prior loss and drift matching parts.

Title: AdaCo: Overcoming Visual Foundation Model Noise in 3D Semantic Segmentation via Adaptive Label Correction

• Authors: Pufan Zou, Shijia Zhao, Weijie Huang, Qiming Xia, Chenglu Wen, Wei Li, Cheng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18255
• Pdf URL: https://arxiv.org/pdf/2412.18255
• Copy Paste: [[2412.18255]] AdaCo: Overcoming Visual Foundation Model Noise in 3D Semantic Segmentation via Adaptive Label Correction(https://arxiv.org/abs/2412.18255)
• Keywords: robust, segmentation
• Abstract: Recently, Visual Foundation Models (VFMs) have shown a remarkable generalization performance in 3D perception tasks. However, their effectiveness in large-scale outdoor datasets remains constrained by the scarcity of accurate supervision signals, the extensive noise caused by variable outdoor conditions, and the abundance of unknown objects. In this work, we propose a novel label-free learning method, Adaptive Label Correction (AdaCo), for 3D semantic segmentation. AdaCo first introduces the Cross-modal Label Generation Module (CLGM), providing cross-modal supervision with the formidable interpretive capabilities of the VFMs. Subsequently, AdaCo incorporates the Adaptive Noise Corrector (ANC), updating and adjusting the noisy samples within this supervision iteratively during training. Moreover, we develop an Adaptive Robust Loss (ARL) function to modulate each sample's sensitivity to noisy supervision, preventing potential underfitting issues associated with robust loss. Our proposed AdaCo can effectively mitigate the performance limitations of label-free learning networks in 3D semantic segmentation tasks. Extensive experiments on two outdoor benchmark datasets highlight the superior performance of our method.

Title: Robust Semi-Supervised Learning in Open Environments

• Authors: Lan-Zhe Guo, Lin-Han Jia, Jie-Jing Shao, Yu-Feng Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18256
• Pdf URL: https://arxiv.org/pdf/2412.18256
• Copy Paste: [[2412.18256]] Robust Semi-Supervised Learning in Open Environments(https://arxiv.org/abs/2412.18256)
• Keywords: robust
• Abstract: Semi-supervised learning (SSL) aims to improve performance by exploiting unlabeled data when labels are scarce. Conventional SSL studies typically assume close environments where important factors (e.g., label, feature, distribution) between labeled and unlabeled data are consistent. However, more practical tasks involve open environments where important factors between labeled and unlabeled data are inconsistent. It has been reported that exploiting inconsistent unlabeled data causes severe performance degradation, even worse than the simple supervised learning baseline. Manually verifying the quality of unlabeled data is not desirable, therefore, it is important to study robust SSL with inconsistent unlabeled data in open environments. This paper briefly introduces some advances in this line of research, focusing on techniques concerning label, feature, and data distribution inconsistency in SSL, and presents the evaluation benchmarks. Open research problems are also discussed for reference purposes.

Title: Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study

• Authors: Xuefeng Jiang, Lvhua Wu, Sheng Sun, Jia Li, Jingjing Xue, Yuwei Wang, Tingting Wu, Min Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18260
• Pdf URL: https://arxiv.org/pdf/2412.18260
• Copy Paste: [[2412.18260]] Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study(https://arxiv.org/abs/2412.18260)
• Keywords: security, large language model
• Abstract: Code vulnerability detection (CVD) is essential for addressing and preventing system security issues, playing a crucial role in ensuring software security. Previous learning-based vulnerability detection methods rely on either fine-tuning medium-size sequence models or training smaller neural networks from scratch. Recent advancements in large pre-trained language models (LLMs) have showcased remarkable capabilities in various code intelligence tasks including code understanding and generation. However, the effectiveness of LLMs in detecting code vulnerabilities is largely under-explored. This work aims to investigate the gap by fine-tuning LLMs for the CVD task, involving four widely-used open-source LLMs. We also implement other five previous graph-based or medium-size sequence models for comparison. Experiments are conducted on five commonly-used CVD datasets, including both the part of short samples and long samples. In addition, we conduct quantitative experiments to investigate the class imbalance issue and the model's performance on samples of different lengths, which are rarely studied in previous works. To better facilitate communities, we open-source all codes and resources of this study in this https URL and this https URL.

Title: Efficient Contrastive Explanations on Demand

• Authors: Yacine Izza, Joao Marques-Silva
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18262
• Pdf URL: https://arxiv.org/pdf/2412.18262
• Copy Paste: [[2412.18262]] Efficient Contrastive Explanations on Demand(https://arxiv.org/abs/2412.18262)
• Keywords: robust
• Abstract: Recent work revealed a tight connection between adversarial robustness and restricted forms of symbolic explanations, namely distance-based (formal) explanations. This connection is significant because it represents a first step towards making the computation of symbolic explanations as efficient as deciding the existence of adversarial examples, especially for highly complex machine learning (ML) models. However, a major performance bottleneck remains, because of the very large number of features that ML models may possess, in particular for deep neural networks. This paper proposes novel algorithms to compute the so-called contrastive explanations for ML models with a large number of features, by leveraging on adversarial robustness. Furthermore, the paper also proposes novel algorithms for listing explanations and finding smallest contrastive explanations. The experimental results demonstrate the performance gains achieved by the novel algorithms proposed in this paper.

Title: GenAI Content Detection Task 2: AI vs. Human -- Academic Essay Authenticity Challenge

• Authors: Shammur Absar Chowdhury, Hind Almerekhi, Mucahid Kutlu, Kaan Efe Keles, Fatema Ahmad, Tasnim Mohiuddin, George Mikros, Firoj Alam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18274
• Pdf URL: https://arxiv.org/pdf/2412.18274
• Copy Paste: [[2412.18274]] GenAI Content Detection Task 2: AI vs. Human -- Academic Essay Authenticity Challenge(https://arxiv.org/abs/2412.18274)
• Keywords: transformer, large language model
• Abstract: This paper presents a comprehensive overview of the first edition of the Academic Essay Authenticity Challenge, organized as part of the GenAI Content Detection shared tasks collocated with COLING 2025. This challenge focuses on detecting machine-generated vs. human-authored essays for academic purposes. The task is defined as follows: "Given an essay, identify whether it is generated by a machine or authored by a human.'' The challenge involves two languages: English and Arabic. During the evaluation phase, 25 teams submitted systems for English and 21 teams for Arabic, reflecting substantial interest in the task. Finally, seven teams submitted system description papers. The majority of submissions utilized fine-tuned transformer-based models, with one team employing Large Language Models (LLMs) such as Llama 2 and Llama 3. This paper outlines the task formulation, details the dataset construction process, and explains the evaluation framework. Additionally, we present a summary of the approaches adopted by participating teams. Nearly all submitted systems outperformed the n-gram-based baseline, with the top-performing systems achieving F1 scores exceeding 0.98 for both languages, indicating significant progress in the detection of machine-generated text.

Title: Towards Modality Generalization: A Benchmark and Prospective Analysis

• Authors: Xiaohao Liu, Xiaobo Xia, Zhuo Huang, Tat-Seng Chua
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18277
• Pdf URL: https://arxiv.org/pdf/2412.18277
• Copy Paste: [[2412.18277]] Towards Modality Generalization: A Benchmark and Prospective Analysis(https://arxiv.org/abs/2412.18277)
• Keywords: privacy, robust
• Abstract: Multi-modal learning has achieved remarkable success by integrating information from various modalities, achieving superior performance in tasks like recognition and retrieval compared to uni-modal approaches. However, real-world scenarios often present novel modalities that are unseen during training due to resource and privacy constraints, a challenge current methods struggle to address. This paper introduces Modality Generalization (MG), which focuses on enabling models to generalize to unseen modalities. We define two cases: weak MG, where both seen and unseen modalities can be mapped into a joint embedding space via existing perceptors, and strong MG, where no such mappings exist. To facilitate progress, we propose a comprehensive benchmark featuring multi-modal algorithms and adapt existing methods that focus on generalization. Extensive experiments highlight the complexity of MG, exposing the limitations of existing methods and identifying key directions for future research. Our work provides a foundation for advancing robust and adaptable multi-modal models, enabling them to handle unseen modalities in realistic scenarios.

Title: Improved Feature Generating Framework for Transductive Zero-shot Learning

• Authors: Zihan Ye, Xinyuan Ru, Shiming Chen, Yaochu Jin, Kaizhu Huang, Xiaobo Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18282
• Pdf URL: https://arxiv.org/pdf/2412.18282
• Copy Paste: [[2412.18282]] Improved Feature Generating Framework for Transductive Zero-shot Learning(https://arxiv.org/abs/2412.18282)
• Keywords: generative
• Abstract: Feature Generative Adversarial Networks have emerged as powerful generative models in producing high-quality representations of unseen classes within the scope of Zero-shot Learning (ZSL). This paper delves into the pivotal influence of unseen class priors within the framework of transductive ZSL (TZSL) and illuminates the finding that even a marginal prior bias can result in substantial accuracy declines. Our extensive analysis uncovers that this inefficacy fundamentally stems from the utilization of an unconditional unseen discriminator - a core component in existing TZSL. We further establish that the detrimental effects of this component are inevitable unless the generator perfectly fits class-specific distributions. Building on these insights, we introduce our Improved Feature Generation Framework, termed I-VAEGAN, which incorporates two novel components: Pseudo-conditional Feature Adversarial (PFA) learning and Variational Embedding Regression (VER). PFA circumvents the need for prior estimation by explicitly injecting the predicted semantics as pseudo conditions for unseen classes premised by precise semantic regression. Meanwhile, VER utilizes reconstructive pre-training to learn class statistics, obtaining better semantic regression. Our I-VAEGAN achieves state-of-the-art TZSL accuracy across various benchmarks and priors. Our code would be released upon acceptance.

Title: On the Local Complexity of Linear Regions in Deep ReLU Networks

• Authors: Niket Patel, Guido Montufar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18283
• Pdf URL: https://arxiv.org/pdf/2412.18283
• Copy Paste: [[2412.18283]] On the Local Complexity of Linear Regions in Deep ReLU Networks(https://arxiv.org/abs/2412.18283)
• Keywords: robust
• Abstract: We define the local complexity of a neural network with continuous piecewise linear activations as a measure of the density of linear regions over an input data distribution. We show theoretically that ReLU networks that learn low-dimensional feature representations have a lower local complexity. This allows us to connect recent empirical observations on feature learning at the level of the weight matrices with concrete properties of the learned functions. In particular, we show that the local complexity serves as an upper bound on the total variation of the function over the input data distribution and thus that feature learning can be related to adversarial robustness. Lastly, we consider how optimization drives ReLU networks towards solutions with lower local complexity. Overall, this work contributes a theoretical framework towards relating geometric properties of ReLU networks to different aspects of learning such as feature learning and representation cost.

Title: Towards understanding how attention mechanism works in deep learning

• Authors: Tianyu Ruan, Shihua Zhang
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.18288
• Pdf URL: https://arxiv.org/pdf/2412.18288
• Copy Paste: [[2412.18288]] Towards understanding how attention mechanism works in deep learning(https://arxiv.org/abs/2412.18288)
• Keywords: robust, diffusion, transformer
• Abstract: Attention mechanism has been extensively integrated within mainstream neural network architectures, such as Transformers and graph attention networks. Yet, its underlying working principles remain somewhat elusive. What is its essence? Are there any connections between it and traditional machine learning algorithms? In this study, we inspect the process of computing similarity using classic metrics and vector space properties in manifold learning, clustering, and supervised learning. We identify the key characteristics of similarity computation and information propagation in these methods and demonstrate that the self-attention mechanism in deep learning adheres to the same principles but operates more flexibly and adaptively. We decompose the self-attention mechanism into a learnable pseudo-metric function and an information propagation process based on similarity computation. We prove that the self-attention mechanism converges to a drift-diffusion process through continuous modeling provided the pseudo-metric is a transformation of a metric and certain reasonable assumptions hold. This equation could be transformed into a heat equation under a new metric. In addition, we give a first-order analysis of attention mechanism with a general pseudo-metric function. This study aids in understanding the effects and principle of attention mechanism through physical intuition. Finally, we propose a modified attention mechanism called metric-attention by leveraging the concept of metric learning to facilitate the ability to learn desired metrics more effectively. Experimental results demonstrate that it outperforms self-attention regarding training efficiency, accuracy, and robustness.

Title: Navigating Data Corruption in Machine Learning: Balancing Quality, Quantity, and Imputation Strategies

• Authors: Qi Liu, Wanjing Ma
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18296
• Pdf URL: https://arxiv.org/pdf/2412.18296
• Copy Paste: [[2412.18296]] Navigating Data Corruption in Machine Learning: Balancing Quality, Quantity, and Imputation Strategies(https://arxiv.org/abs/2412.18296)
• Keywords: robust
• Abstract: Data corruption, including missing and noisy data, poses significant challenges in real-world machine learning. This study investigates the effects of data corruption on model performance and explores strategies to mitigate these effects through two experimental setups: supervised learning with NLP tasks (NLP-SL) and deep reinforcement learning for traffic signal optimization (Signal-RL). We analyze the relationship between data corruption levels and model performance, evaluate the effectiveness of data imputation methods, and assess the utility of enlarging datasets to address data corruption. Our results show that model performance under data corruption follows a diminishing return curve, modeled by the exponential function. Missing data, while detrimental, is less harmful than noisy data, which causes severe performance degradation and training instability, particularly in sequential decision-making tasks like Signal-RL. Imputation strategies involve a trade-off: they recover missing information but may introduce noise. Their effectiveness depends on imputation accuracy and corruption ratio. We identify distinct regions in the imputation advantage heatmap, including an "imputation advantageous corner" and an "imputation disadvantageous edge" and classify tasks as "noise-sensitive" or "noise-insensitive" based on their decision boundaries. Furthermore, we find that increasing dataset size mitigates but cannot fully overcome the effects of data corruption. The marginal utility of additional data diminishes as corruption increases. An empirical rule emerges: approximately 30% of the data is critical for determining performance, while the remaining 70% has minimal impact. These findings provide actionable insights into data preprocessing, imputation strategies, and data collection practices, guiding the development of robust machine learning systems in noisy environments.

Title: Quo Vadis, Anomaly Detection? LLMs and VLMs in the Spotlight

• Authors: Xi Ding, Lei Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18298
• Pdf URL: https://arxiv.org/pdf/2412.18298
• Copy Paste: [[2412.18298]] Quo Vadis, Anomaly Detection? LLMs and VLMs in the Spotlight(https://arxiv.org/abs/2412.18298)
• Keywords: interpretability, large language model
• Abstract: Video anomaly detection (VAD) has witnessed significant advancements through the integration of large language models (LLMs) and vision-language models (VLMs), addressing critical challenges such as interpretability, temporal reasoning, and generalization in dynamic, open-world scenarios. This paper presents an in-depth review of cutting-edge LLM-/VLM-based methods in 2024, focusing on four key aspects: (i) enhancing interpretability through semantic insights and textual explanations, making visual anomalies more understandable; (ii) capturing intricate temporal relationships to detect and localize dynamic anomalies across video frames; (iii) enabling few-shot and zero-shot detection to minimize reliance on large, annotated datasets; and (iv) addressing open-world and class-agnostic anomalies by using semantic understanding and motion features for spatiotemporal coherence. We highlight their potential to redefine the landscape of VAD. Additionally, we explore the synergy between visual and textual modalities offered by LLMs and VLMs, highlighting their combined strengths and proposing future directions to fully exploit the potential in enhancing video anomaly detection.

Title: M-Ped: Multi-Prompt Ensemble Decoding for Large Language Models

• Authors: Jiaxin Guo, Daimeng Wei, Yuanchang Luo, Shimin Tao, Hengchao Shang, Zongyao Li, Shaojun Li, Jinlong Yang, Zhanglin Wu, Zhiqiang Rao, Hao Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18299
• Pdf URL: https://arxiv.org/pdf/2412.18299
• Copy Paste: [[2412.18299]] M-Ped: Multi-Prompt Ensemble Decoding for Large Language Models(https://arxiv.org/abs/2412.18299)
• Keywords: large language model
• Abstract: With the widespread application of Large Language Models (LLMs) in the field of Natural Language Processing (NLP), enhancing their performance has become a research hotspot. This paper presents a novel multi-prompt ensemble decoding approach designed to bolster the generation quality of LLMs by leveraging the aggregation of outcomes from multiple prompts. Given a unique input $X$, we submit $n$ variations of prompts with $X$ to LLMs in batch mode to decode and derive probability distributions. For each token prediction, we calculate the ensemble probability by averaging the $n$ probability distributions within the batch, utilizing this aggregated probability to generate the token. This technique is dubbed Inner-Batch Ensemble. To facilitate efficient batch inference, we implement a Left-Padding strategy to maintain uniform input lengths across the n prompts. Through extensive experimentation on diverse NLP tasks, including machine translation, code generation, and text simplification, we demonstrate the efficacy of our method in enhancing LLM performance. The results show substantial improvements in BLEU scores, pass@$k$ rates, and LENS metrics over conventional methods.

Title: FameBias: Embedding Manipulation Bias Attack in Text-to-Image Models

• Authors: Jaechul Roh, Andrew Yuan, Jinsong Mao
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18302
• Pdf URL: https://arxiv.org/pdf/2412.18302
• Copy Paste: [[2412.18302]] FameBias: Embedding Manipulation Bias Attack in Text-to-Image Models(https://arxiv.org/abs/2412.18302)
• Keywords: attack, diffusion
• Abstract: Text-to-Image (T2I) diffusion models have rapidly advanced, enabling the generation of high-quality images that align closely with textual descriptions. However, this progress has also raised concerns about their misuse for propaganda and other malicious activities. Recent studies reveal that attackers can embed biases into these models through simple fine-tuning, causing them to generate targeted imagery when triggered by specific phrases. This underscores the potential for T2I models to act as tools for disseminating propaganda, producing images aligned with an attacker's objective for end-users. Building on this concept, we introduce FameBias, a T2I biasing attack that manipulates the embeddings of input prompts to generate images featuring specific public figures. Unlike prior methods, Famebias operates solely on the input embedding vectors without requiring additional model training. We evaluate FameBias comprehensively using Stable Diffusion V2, generating a large corpus of images based on various trigger nouns and target public figures. Our experiments demonstrate that FameBias achieves a high attack success rate while preserving the semantic context of the original prompts across multiple trigger-target pairs.

Title: Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answering

• Authors: Zhongjian Hu, Peng Yang, Bing Li, Zhenqi Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18351
• Pdf URL: https://arxiv.org/pdf/2412.18351
• Copy Paste: [[2412.18351]] Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answering(https://arxiv.org/abs/2412.18351)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have achieved impressive results in knowledge-based Visual Question Answering (VQA). However existing methods still have challenges: the inability to use external tools autonomously, and the inability to work in teams. Humans tend to know whether they need to use external tools when they encounter a new question, e.g., they tend to be able to give a direct answer to a familiar question, whereas they tend to use tools such as search engines when they encounter an unfamiliar question. In addition, humans also tend to collaborate and discuss with others to get better answers. Inspired by this, we propose the multi-agent voting framework. We design three LLM-based agents that simulate different levels of staff in a team, and assign the available tools according to the levels. Each agent provides the corresponding answer, and finally all the answers provided by the agents are voted to get the final answer. Experiments on OK-VQA and A-OKVQA show that our approach outperforms other baselines by 2.2 and 1.0, respectively.

Title: Addressing Spatial-Temporal Data Heterogeneity in Federated Continual Learning via Tail Anchor

• Authors: Hao Yu, Xin Yang, Le Zhang, Hanlin Gu, Tianrui Li, Lixin Fan, Qiang Yang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18355
• Pdf URL: https://arxiv.org/pdf/2412.18355
• Copy Paste: [[2412.18355]] Addressing Spatial-Temporal Data Heterogeneity in Federated Continual Learning via Tail Anchor(https://arxiv.org/abs/2412.18355)
• Keywords: federate
• Abstract: Federated continual learning (FCL) allows each client to continually update its knowledge from task streams, enhancing the applicability of federated learning in real-world scenarios. However, FCL needs to address not only spatial data heterogeneity between clients but also temporal data heterogeneity between tasks. In this paper, empirical experiments demonstrate that such input-level heterogeneity significantly affects the model's internal parameters and outputs, leading to severe spatial-temporal catastrophic forgetting of local and previous knowledge. To this end, we propose Federated Tail Anchor (FedTA) to mix trainable Tail Anchor with the frozen output features to adjust their position in the feature space, thereby overcoming parameter-forgetting and output-forgetting. Moreover, three novel components are also included in FedTA: Input Enhancement for improving the performance of pre-trained models on downstream tasks; Selective Input Knowledge Fusion for fusion of heterogeneous local knowledge on the server side; and Best Global Prototype Selection for finding the best anchor point for each class in the feature space. Extensive experiments demonstrate that FedTA not only outperforms existing FCL methods but also effectively preserves the relative positions of features, remaining unaffected by spatial and temporal changes.

Title: Extracting triples from dialogues for conversational social agents

• Authors: Piek Vossen, Selene Báez Santamaría, Lenka Bajčetić, Thomas Belluci
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18364
• Pdf URL: https://arxiv.org/pdf/2412.18364
• Copy Paste: [[2412.18364]] Extracting triples from dialogues for conversational social agents(https://arxiv.org/abs/2412.18364)
• Keywords: extraction
• Abstract: Obtaining an explicit understanding of communication within a Hybrid Intelligence collaboration is essential to create controllable and transparent agents. In this paper, we describe a number of Natural Language Understanding models that extract explicit symbolic triples from social conversation. Triple extraction has mostly been developed and tested for Knowledge Base Completion using Wikipedia text and data for training and testing. However, social conversation is very different as a genre in which interlocutors exchange information in sequences of utterances that involve statements, questions, and answers. Phenomena such as co-reference, ellipsis, coordination, and implicit and explicit negation or confirmation are more prominent in conversation than in Wikipedia text. We therefore describe an attempt to fill this gap by releasing data sets for training and testing triple extraction from social conversation. We also created five triple extraction models and tested them in our evaluation data. The highest precision is 51.14 for complete triples and 69.32 for triple elements when tested on single utterances. However, scores for conversational triples that span multiple turns are much lower, showing that extracting knowledge from true conversational data is much more challenging.

Title: Hypergraph Attacks via Injecting Homogeneous Nodes into Elite Hyperedges

• Authors: Meixia He, Peican Zhu, Keke Tang, Yangming Guo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18365
• Pdf URL: https://arxiv.org/pdf/2412.18365
• Copy Paste: [[2412.18365]] Hypergraph Attacks via Injecting Homogeneous Nodes into Elite Hyperedges(https://arxiv.org/abs/2412.18365)
• Keywords: attack
• Abstract: Recent studies have shown that Hypergraph Neural Networks (HGNNs) are vulnerable to adversarial attacks. Existing approaches focus on hypergraph modification attacks guided by gradients, overlooking node spanning in the hypergraph and the group identity of hyperedges, thereby resulting in limited attack performance and detectable attacks. In this manuscript, we present a novel framework, i.e., Hypergraph Attacks via Injecting Homogeneous Nodes into Elite Hyperedges (IE-Attack), to tackle these challenges. Initially, utilizing the node spanning in the hypergraph, we propose the elite hyperedges sampler to identify hyperedges to be injected. Subsequently, a node generator utilizing Kernel Density Estimation (KDE) is proposed to generate the homogeneous node with the group identity of hyperedges. Finally, by injecting the homogeneous node into elite hyperedges, IE-Attack improves the attack performance and enhances the imperceptibility of attacks. Extensive experiments are conducted on five authentic datasets to validate the effectiveness of IE-Attack and the corresponding superiority to state-of-the-art methods.

Title: Towards Global AI Inclusivity: A Large-Scale Multilingual Terminology Dataset

• Authors: Jiarui Liu, Iman Ouzzani, Wenkai Li, Lechen Zhang, Tianyue Ou, Houda Bouamor, Zhijing Jin, Mona Diab
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18367
• Pdf URL: https://arxiv.org/pdf/2412.18367
• Copy Paste: [[2412.18367]] Towards Global AI Inclusivity: A Large-Scale Multilingual Terminology Dataset(https://arxiv.org/abs/2412.18367)
• Keywords: extraction
• Abstract: The field of machine translation has achieved significant advancements, yet domain-specific terminology translation, particularly in AI, remains challenging. We introduced GIST, a large-scale multilingual AI terminology dataset containing 5K terms extracted from top AI conference papers spanning 2000 to 2023. The terms were translated into Arabic, Chinese, French, Japanese, and Russian using a hybrid framework that combines LLMs for extraction with human expertise for translation. The dataset's quality was benchmarked against existing resources, demonstrating superior translation accuracy through crowdsourced evaluation. GIST was integrated into translation workflows using post-translation refinement methods that required no retraining, where LLM prompting consistently improved BLEU and COMET scores. A web demonstration on the ACL Anthology platform highlights its practical application, showcasing improved accessibility for non-English speakers. This work aims to address critical gaps in AI terminology resources and fosters global inclusivity and collaboration in AI research.

Title: Unveiling the Threat of Fraud Gangs to Graph Neural Networks: Multi-Target Graph Injection Attacks against GNN-Based Fraud Detectors

• Authors: Jinhyeok Choi, Heehyeon Kim, Joyce Jiyoung Whang
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.18370
• Pdf URL: https://arxiv.org/pdf/2412.18370
• Copy Paste: [[2412.18370]] Unveiling the Threat of Fraud Gangs to Graph Neural Networks: Multi-Target Graph Injection Attacks against GNN-Based Fraud Detectors(https://arxiv.org/abs/2412.18370)
• Keywords: attack, transformer
• Abstract: Graph neural networks (GNNs) have emerged as an effective tool for fraud detection, identifying fraudulent users, and uncovering malicious behaviors. However, attacks against GNN-based fraud detectors and their risks have rarely been studied, thereby leaving potential threats unaddressed. Recent findings suggest that frauds are increasingly organized as gangs or groups. In this work, we design attack scenarios where fraud gangs aim to make their fraud nodes misclassified as benign by camouflaging their illicit activities in collusion. Based on these scenarios, we study adversarial attacks against GNN-based fraud detectors by simulating attacks of fraud gangs in three real-world fraud cases: spam reviews, fake news, and medical insurance frauds. We define these attacks as multi-target graph injection attacks and propose MonTi, a transformer-based Multi-target one-Time graph injection attack model. MonTi simultaneously generates attributes and edges of all attack nodes with a transformer encoder, capturing interdependencies between attributes and edges more effectively than most existing graph injection attack methods that generate these elements sequentially. Additionally, MonTi adaptively allocates the degree budget for each attack node to explore diverse injection structures involving target, candidate, and attack nodes, unlike existing methods that fix the degree budget across all attack nodes. Experiments show that MonTi outperforms the state-of-the-art graph injection attack methods on five real-world graphs.

Title: Bidirectional Topic Matching: Quantifying Thematic Overlap Between Corpora Through Topic Modelling

• Authors: Raven Adam, Marie Lisa Kogler
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.18376
• Pdf URL: https://arxiv.org/pdf/2412.18376
• Copy Paste: [[2412.18376]] Bidirectional Topic Matching: Quantifying Thematic Overlap Between Corpora Through Topic Modelling(https://arxiv.org/abs/2412.18376)
• Keywords: robust
• Abstract: This study introduces Bidirectional Topic Matching (BTM), a novel method for cross-corpus topic modeling that quantifies thematic overlap and divergence between corpora. BTM is a flexible framework that can incorporate various topic modeling approaches, including BERTopic, Top2Vec, and Latent Dirichlet Allocation (LDA). BTM employs a dual-model approach, training separate topic models for each corpus and applying them reciprocally to enable comprehensive cross-corpus comparisons. This methodology facilitates the identification of shared themes and unique topics, providing nuanced insights into thematic relationships. Validation against cosine similarity-based methods demonstrates the robustness of BTM, with strong agreement metrics and distinct advantages in handling outlier topics. A case study on climate news articles showcases BTM's utility, revealing significant thematic overlaps and distinctions between corpora focused on climate change and climate action. BTM's flexibility and precision make it a valuable tool for diverse applications, from political discourse analysis to interdisciplinary studies. By integrating shared and unique topic analyses, BTM offers a comprehensive framework for exploring thematic relationships, with potential extensions to multilingual and dynamic datasets. This work highlights BTM's methodological contributions and its capacity to advance discourse analysis across various domains.

Title: ChaI-TeA: A Benchmark for Evaluating Autocompletion of Interactions with LLM-based Chatbots

• Authors: Shani Goren, Oren Kalinsky, Tomer Stav, Yuri Rapoport, Yaron Fairstein, Ram Yazdy, Nachshon Cohen, Alexander Libov, Guy Kushilevitz
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18377
• Pdf URL: https://arxiv.org/pdf/2412.18377
• Copy Paste: [[2412.18377]] ChaI-TeA: A Benchmark for Evaluating Autocompletion of Interactions with LLM-based Chatbots(https://arxiv.org/abs/2412.18377)
• Keywords: fair
• Abstract: The rise of LLMs has deflected a growing portion of human-computer interactions towards LLM-based chatbots. The remarkable abilities of these models allow users to interact using long, diverse natural language text covering a wide range of topics and styles. Phrasing these messages is a time and effort consuming task, calling for an autocomplete solution to assist users. We introduce the task of chatbot interaction autocomplete. We present ChaI-TeA: CHat InTEraction Autocomplete; An autcomplete evaluation framework for LLM-based chatbot interactions. The framework includes a formal definition of the task, coupled with suitable datasets and metrics. We use the framework to evaluate After formally defining the task along with suitable datasets and metrics, we test 9 models on the defined auto completion task, finding that while current off-the-shelf models perform fairly, there is still much room for improvement, mainly in ranking of the generated suggestions. We provide insights for practitioners working on this task and open new research directions for researchers in the field. We release our framework to serve as a foundation for future research.

Title: RDPM: Solve Diffusion Probabilistic Models via Recurrent Token Prediction

• Authors: Wu Xiaoping, Hu Jie, Wei Xiaoming
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.18390
• Pdf URL: https://arxiv.org/pdf/2412.18390
• Copy Paste: [[2412.18390]] RDPM: Solve Diffusion Probabilistic Models via Recurrent Token Prediction(https://arxiv.org/abs/2412.18390)
• Keywords: diffusion, generative, large language model
• Abstract: Diffusion Probabilistic Models (DPMs) have emerged as the de facto approach for high-fidelity image synthesis, operating diffusion processes on continuous VAE latent, which significantly differ from the text generation methods employed by Large Language Models (LLMs). In this paper, we introduce a novel generative framework, the Recurrent Diffusion Probabilistic Model (RDPM), which enhances the diffusion process through a recurrent token prediction mechanism, thereby pioneering the field of Discrete Diffusion. By progressively introducing Gaussian noise into the latent representations of images and encoding them into vector-quantized tokens in a recurrent manner, RDPM facilitates a unique diffusion process on discrete-value domains. This process iteratively predicts the token codes for subsequent timesteps, transforming the initial standard Gaussian noise into the source data distribution, aligning with GPT-style models in terms of the loss function. RDPM demonstrates superior performance while benefiting from the speed advantage of requiring only a few inference steps. This model not only leverages the diffusion process to ensure high-quality generation but also converts continuous signals into a series of high-fidelity discrete tokens, thereby maintaining a unified optimization strategy with other discrete tokens, such as text. We anticipate that this work will contribute to the development of a unified model for multimodal generation, specifically by integrating continuous signal domains such as images, videos, and audio with text. We will release the code and model weights to the open-source community.

Title: Extract Free Dense Misalignment from CLIP

• Authors: JeongYeon Nam, Jinbae Im, Wonjae Kim, Taeho Kil
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18404
• Pdf URL: https://arxiv.org/pdf/2412.18404
• Copy Paste: [[2412.18404]] Extract Free Dense Misalignment from CLIP(https://arxiv.org/abs/2412.18404)
• Keywords: interpretability
• Abstract: Recent vision-language foundation models still frequently produce outputs misaligned with their inputs, evidenced by object hallucination in captioning and prompt misalignment in the text-to-image generation model. Recent studies have explored methods for identifying misaligned elements, aiming not only to enhance interpretability but also to improve model performance. However, current approaches primarily rely on large foundation models in a zero-shot manner or fine-tuned models with human annotations, which limits scalability due to significant computational costs. This work proposes a novel approach, dubbed CLIP4DM, for detecting dense misalignments from pre-trained CLIP, specifically focusing on pinpointing misaligned words between image and text. We carefully revamp the gradient-based attribution computation method, enabling negative gradient of individual text tokens to indicate misalignment. We also propose F-CLIPScore, which aggregates misaligned attributions with a global alignment score. We evaluate our method on various dense misalignment detection benchmarks, covering various image and text domains and misalignment types. Our method demonstrates state-of-the-art performance among zero-shot models and competitive performance with fine-tuned models while maintaining superior efficiency. Our qualitative examples show that our method has a unique strength to detect entity-level objects, intangible objects, and attributes that can not be easily detected for existing works. We conduct ablation studies and analyses to highlight the strengths and limitations of our approach. Our code is publicly available at this https URL.

Title: Re-assessing ImageNet: How aligned is its single-label assumption with its multi-label nature?

• Authors: Esla Timothy Anzaku, Seyed Amir Mousavi, Arnout Van Messem, Wesley De Neve
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18409
• Pdf URL: https://arxiv.org/pdf/2412.18409
• Copy Paste: [[2412.18409]] Re-assessing ImageNet: How aligned is its single-label assumption with its multi-label nature?(https://arxiv.org/abs/2412.18409)
• Keywords: robust
• Abstract: ImageNet, an influential dataset in computer vision, is traditionally evaluated using single-label classification, which assumes that an image can be adequately described by a single concept or label. However, this approach may not fully capture the complex semantics within the images available in ImageNet, potentially hindering the development of models that effectively learn these intricacies. This study critically examines the prevalent single-label benchmarking approach and advocates for a shift to multi-label benchmarking for ImageNet. This shift would enable a more comprehensive assessment of the capabilities of deep neural network (DNN) models. We analyze the effectiveness of pre-trained state-of-the-art DNNs on ImageNet and one of its variants, ImageNetV2. Studies in the literature have reported unexpected accuracy drops of 11% to 14% on ImageNetV2. Our findings show that these reported declines are largely attributable to a characteristic of the dataset that has not received sufficient attention -- the proportion of images with multiple labels. Taking this characteristic into account, the results of our experiments provide evidence that there is no substantial degradation in effectiveness on ImageNetV2. Furthermore, we acknowledge that ImageNet pre-trained models exhibit some capability at capturing the multi-label nature of the dataset even though they were trained under the single-label assumption. Consequently, we propose a new evaluation approach to augment existing approaches that assess this capability. Our findings highlight the importance of considering the multi-label nature of the ImageNet dataset during benchmarking. Failing to do so could lead to incorrect conclusions regarding the effectiveness of DNNs and divert research efforts from addressing other substantial challenges related to the reliability and robustness of these models.

Title: Multilingual Mathematical Reasoning: Advancing Open-Source LLMs in Hindi and English

• Authors: Avinash Anand, Kritarth Prasad, Chhavi Kirtani, Ashwin R Nair, Manvendra Kumar Nema, Raj Jaiswal, Rajiv Ratn Shah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18415
• Pdf URL: https://arxiv.org/pdf/2412.18415
• Copy Paste: [[2412.18415]] Multilingual Mathematical Reasoning: Advancing Open-Source LLMs in Hindi and English(https://arxiv.org/abs/2412.18415)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) excel in linguistic tasks but struggle with mathematical reasoning, particularly in non English languages like Hindi. This research aims to enhance the mathematical reasoning skills of smaller, resource efficient open-source LLMs in both Hindi and English. We evaluate models like OpenHathi 7B, LLaMA-2 7B, WizardMath 7B, Mistral 7B, LLeMMa 7B, MAmmoTH 7B, Gemini Pro, and GPT-4 using zero-shot, few-shot chain-of-thought (CoT) methods, and supervised fine-tuning. Our approach incorporates curriculum learning, progressively training models on increasingly difficult problems, a novel Decomposition Strategy to simplify complex arithmetic operations, and a Structured Solution Design that divides solutions into phases. Our experiments result in notable performance enhancements. WizardMath 7B exceeds Gemini's accuracy on English datasets by +6% and matches Gemini's performance on Hindi datasets. Adopting a bilingual approach that combines English and Hindi samples achieves results comparable to individual language models, demonstrating the capability to learn mathematical reasoning in both languages. This research highlights the potential for improving mathematical reasoning in open-source LLMs.

Title: Fashionability-Enhancing Outfit Image Editing with Conditional Diffusion Models

• Authors: Qice Qin, Yuki Hirakawa, Ryotaro Shimizu, Takuya Furusawa, Edgar Simo-Serra
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18421
• Pdf URL: https://arxiv.org/pdf/2412.18421
• Copy Paste: [[2412.18421]] Fashionability-Enhancing Outfit Image Editing with Conditional Diffusion Models(https://arxiv.org/abs/2412.18421)
• Keywords: diffusion
• Abstract: Image generation in the fashion domain has predominantly focused on preserving body characteristics or following input prompts, but little attention has been paid to improving the inherent fashionability of the output images. This paper presents a novel diffusion model-based approach that generates fashion images with improved fashionability while maintaining control over key attributes. Key components of our method include: 1) fashionability enhancement, which ensures that the generated images are more fashionable than the input; 2) preservation of body characteristics, encouraging the generated images to maintain the original shape and proportions of the input; and 3) automatic fashion optimization, which does not rely on manual input or external prompts. We also employ two methods to collect training data for guidance while generating and evaluating the images. In particular, we rate outfit images using fashionability scores annotated by multiple fashion experts through OpenSkill-based and five critical aspect-based pairwise comparisons. These methods provide complementary perspectives for assessing and improving the fashionability of the generated images. The experimental results show that our approach outperforms the baseline Fashion++ in generating images with superior fashionability, demonstrating its effectiveness in producing more stylish and appealing fashion images.

Title: Unlocking the Potential of Multiple BERT Models for Bangla Question Answering in NCTB Textbooks

• Authors: Abdullah Khondoker, Enam Ahmed Taufik, Md Iftekhar Islam Tashik, S M Ishtiak mahmud, Antara Firoz Parsa
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18440
• Pdf URL: https://arxiv.org/pdf/2412.18440
• Copy Paste: [[2412.18440]] Unlocking the Potential of Multiple BERT Models for Bangla Question Answering in NCTB Textbooks(https://arxiv.org/abs/2412.18440)
• Keywords: robust
• Abstract: Evaluating text comprehension in educational settings is critical for understanding student performance and improving curricular effectiveness. This study investigates the capability of state-of-the-art language models-RoBERTa Base, Bangla-BERT, and BERT Base-in automatically assessing Bangla passage-based question-answering from the National Curriculum and Textbook Board (NCTB) textbooks for classes 6-10. A dataset of approximately 3,000 Bangla passage-based question-answering instances was compiled, and the models were evaluated using F1 Score and Exact Match (EM) metrics across various hyperparameter configurations. Our findings revealed that Bangla-BERT consistently outperformed the other models, achieving the highest F1 (0.75) and EM (0.53) scores, particularly with smaller batch sizes, the inclusion of stop words, and a moderate learning rate. In contrast, RoBERTa Base demonstrated the weakest performance, with the lowest F1 (0.19) and EM (0.27) scores under certain configurations. The results underscore the importance of fine-tuning hyperparameters for optimizing model performance and highlight the potential of machine learning models in evaluating text comprehension in educational contexts. However, limitations such as dataset size, spelling inconsistencies, and computational constraints emphasize the need for further research to enhance the robustness and applicability of these models. This study lays the groundwork for the future development of automated evaluation systems in educational institutions, providing critical insights into model performance in the context of Bangla text comprehension.

Title: SoK: On the Offensive Potential of AI

• Authors: Saskia Laura Schröer, Giovanni Apruzzese, Soheil Human, Pavel Laskov, Hyrum S. Anderson, Edward W. N. Bernroider, Aurore Fass, Ben Nassi, Vera Rimmer, Fabio Roli, Samer Salam, Ashley Shen, Ali Sunyaev, Tim Wadwha-Brown, Isabel Wagner, Gang Wang
• Subjects: cs.CR, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18442
• Pdf URL: https://arxiv.org/pdf/2412.18442
• Copy Paste: [[2412.18442]] SoK: On the Offensive Potential of AI(https://arxiv.org/abs/2412.18442)
• Keywords: security, privacy
• Abstract: Our society increasingly benefits from Artificial Intelligence (AI). Unfortunately, more and more evidence shows that AI is also used for offensive purposes. Prior works have revealed various examples of use cases in which the deployment of AI can lead to violation of security and privacy objectives. No extant work, however, has been able to draw a holistic picture of the offensive potential of AI. In this SoK paper we seek to lay the ground for a systematic analysis of the heterogeneous capabilities of offensive AI. In particular we (i) account for AI risks to both humans and systems while (ii) consolidating and distilling knowledge from academic literature, expert opinions, industrial venues, as well as laymen -- all of which being valuable sources of information on offensive AI. To enable alignment of such diverse sources of knowledge, we devise a common set of criteria reflecting essential technological factors related to offensive AI. With the help of such criteria, we systematically analyze: 95 research papers; 38 InfoSec briefings (from, e.g., BlackHat); the responses of a user study (N=549) entailing individuals with diverse backgrounds and expertise; and the opinion of 12 experts. Our contributions not only reveal concerning ways (some of which overlooked by prior work) in which AI can be offensively used today, but also represent a foothold to address this threat in the years to come.

Title: Is Large Language Model Good at Triple Set Prediction? An Empirical Study

• Authors: Yuan Yuan, Yajing Xu, Wen Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18443
• Pdf URL: https://arxiv.org/pdf/2412.18443
• Copy Paste: [[2412.18443]] Is Large Language Model Good at Triple Set Prediction? An Empirical Study(https://arxiv.org/abs/2412.18443)
• Keywords: large language model
• Abstract: The core of the Knowledge Graph Completion (KGC) task is to predict and complete the missing relations or nodes in a KG. Common KGC tasks are mostly about inferring unknown elements with one or two elements being known in a triple. In comparison, the Triple Set Prediction (TSP) task is a more realistic knowledge graph completion task. It aims to predict all elements of unknown triples based on the information from known triples. In recent years, large language models (LLMs) have exhibited significant advancements in language comprehension, demonstrating considerable potential for KGC tasks. However, the potential of LLM on the TSP task has not yet to be investigated. Thus in this paper we proposed a new framework to explore the strengths and limitations of LLM in the TSP task. Specifically, the framework consists of LLM-based rule mining and LLM-based triple set prediction. The relation list of KG embedded within rich semantic information is first leveraged to prompt LLM in the generation of rules. This process is both efficient and independent of statistical information, making it easier to mine effective and realistic rules. For each subgraph, the specified rule is applied in conjunction with the relevant triples within that subgraph to guide the LLM in predicting the missing triples. Subsequently, the predictions from all subgraphs are consolidated to derive the complete set of predicted triples on KG. Finally, the method is evaluated on the relatively complete CFamily dataset. The experimental results indicate that when LLMs are required to adhere to a large amount of factual knowledge to predict missing triples, significant hallucinations occurs, leading to a noticeable decline in performance. To further explore the causes of this phenomenon, this paper presents a comprehensive analysis supported by a detailed case study.

Title: LoRaWAN attack in military use case

• Authors: Georges Derache, Mounira Msahli, Aurelien Botbol, Fabien Romain, Jerome Champlon, Gauthier Canet
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.18447
• Pdf URL: https://arxiv.org/pdf/2412.18447
• Copy Paste: [[2412.18447]] LoRaWAN attack in military use case(https://arxiv.org/abs/2412.18447)
• Keywords: security, attack
• Abstract: The importance of the development of IoT and LoRaWAN in military applications has been widely established. Since security is one of its important challenges, in this paper we study two attacks scenarios: replay and sniff attacks on military LoRaWAN network. The aim is to highlight cybersecurity threats that must be taken into consideration when using such technology in critical context.

Title: 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding

• Authors: Tatiana Zemskova, Dmitry Yudin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18450
• Pdf URL: https://arxiv.org/pdf/2412.18450
• Copy Paste: [[2412.18450]] 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding(https://arxiv.org/abs/2412.18450)
• Keywords: large language model
• Abstract: A 3D scene graph represents a compact scene model, storing information about the objects and the semantic relationships between them, making its use promising for robotic tasks. When interacting with a user, an embodied intelligent agent should be capable of responding to various queries about the scene formulated in natural language. Large Language Models (LLMs) are beneficial solutions for user-robot interaction due to their natural language understanding and reasoning abilities. Recent methods for creating learnable representations of 3D scenes have demonstrated the potential to improve the quality of LLMs responses by adapting to the 3D world. However, the existing methods do not explicitly utilize information about the semantic relationships between objects, limiting themselves to information about their coordinates. In this work, we propose a method 3DGraphLLM for constructing a learnable representation of a 3D scene graph. The learnable representation is used as input for LLMs to perform 3D vision-language tasks. In our experiments on popular ScanRefer, RIORefer, Multi3DRefer, ScanQA, Sqa3D, and Scan2cap datasets, we demonstrate the advantage of this approach over baseline methods that do not use information about the semantic relationships between objects. The code is publicly available at this https URL.

Title: Underwater Image Restoration via Polymorphic Large Kernel CNNs

• Authors: Xiaojiao Guo, Yihang Dong, Xuhang Chen, Weiwen Chen, Zimeng Li, FuChen Zheng, Chi-Man Pun
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.18459
• Pdf URL: https://arxiv.org/pdf/2412.18459
• Copy Paste: [[2412.18459]] Underwater Image Restoration via Polymorphic Large Kernel CNNs(https://arxiv.org/abs/2412.18459)
• Keywords: robust, transformer
• Abstract: Underwater Image Restoration (UIR) remains a challenging task in computer vision due to the complex degradation of images in underwater environments. While recent approaches have leveraged various deep learning techniques, including Transformers and complex, parameter-heavy models to achieve significant improvements in restoration effects, we demonstrate that pure CNN architectures with lightweight parameters can achieve comparable results. In this paper, we introduce UIR-PolyKernel, a novel method for underwater image restoration that leverages Polymorphic Large Kernel CNNs. Our approach uniquely combines large kernel convolutions of diverse sizes and shapes to effectively capture long-range dependencies within underwater imagery. Additionally, we introduce a Hybrid Domain Attention module that integrates frequency and spatial domain attention mechanisms to enhance feature importance. By leveraging the frequency domain, we can capture hidden features that may not be perceptible to humans but are crucial for identifying patterns in both underwater and on-air images. This approach enhances the generalization and robustness of our UIR model. Extensive experiments on benchmark datasets demonstrate that UIR-PolyKernel achieves state-of-the-art performance in underwater image restoration tasks, both quantitatively and qualitatively. Our results show that well-designed pure CNN architectures can effectively compete with more complex models, offering a balance between performance and computational efficiency. This work provides new insights into the potential of CNN-based approaches for challenging image restoration tasks in underwater environments. The code is available at \href{this https URL}{this https URL}.

Title: GeFL: Model-Agnostic Federated Learning with Generative Models

• Authors: Honggu Kang, Seohyeon Cha, Joonhyuk Kang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.18460
• Pdf URL: https://arxiv.org/pdf/2412.18460
• Copy Paste: [[2412.18460]] GeFL: Model-Agnostic Federated Learning with Generative Models(https://arxiv.org/abs/2412.18460)
• Keywords: privacy, robust, federate, generative
• Abstract: Federated learning (FL) is a promising paradigm in distributed learning while preserving the privacy of users. However, the increasing size of recent models makes it unaffordable for a few users to encompass the model. It leads the users to adopt heterogeneous models based on their diverse computing capabilities and network bandwidth. Correspondingly, FL with heterogeneous models should be addressed, given that FL typically involves training a single global model. In this paper, we propose Generative Model-Aided Federated Learning (GeFL), incorporating a generative model that aggregates global knowledge across users of heterogeneous models. Our experiments on various classification tasks demonstrate notable performance improvements of GeFL compared to baselines, as well as limitations in terms of privacy and scalability. To tackle these concerns, we introduce a novel framework, GeFL-F. It trains target networks aided by feature-generative models. We empirically demonstrate the consistent performance gains of GeFL-F, while demonstrating better privacy preservation and robustness to a large number of clients. Codes are available at [1].

Title: Segment-Based Attention Masking for GPTs

• Authors: Shahar Katz, Liran Ringel, Yaniv Romano, Lior Wolf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18487
• Pdf URL: https://arxiv.org/pdf/2412.18487
• Copy Paste: [[2412.18487]] Segment-Based Attention Masking for GPTs(https://arxiv.org/abs/2412.18487)
• Keywords: transformer, generative
• Abstract: Modern Language Models (LMs) owe much of their success to masked causal attention, the backbone of Generative Pre-Trained Transformer (GPT) models. Although GPTs can process the entire user prompt at once, the causal masking is applied to all input tokens step-by-step, mimicking the generation process. This imposes an unnecessary constraint during the initial "prefill" phase when the model processes the input prompt and generates the internal representations before producing any output tokens. In this work, attention is masked based on the known block structure at the prefill phase, followed by the conventional token-by-token autoregressive process after that. For example, in a typical chat prompt, the system prompt is treated as one block, and the user prompt as the next one. Each of these is treated as a unit for the purpose of masking, such that the first tokens in each block can access the subsequent tokens in a non-causal manner. Then, the model answer is generated in the conventional causal manner. This Segment-by-Segment scheme entails no additional computational overhead. When integrating it into models such as Llama and Qwen, state-of-the-art performance is consistently achieved.

Title: A Survey on the Principles of Persuasion as a Social Engineering Strategy in Phishing

• Authors: Kalam Khadka, Abu Barkat Ullah, Wanli Ma, Elisa Martinez Marroquin
• Subjects: cs.CR, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.18488
• Pdf URL: https://arxiv.org/pdf/2412.18488
• Copy Paste: [[2412.18488]] A Survey on the Principles of Persuasion as a Social Engineering Strategy in Phishing(https://arxiv.org/abs/2412.18488)
• Keywords: security, attack
• Abstract: Research shows that phishing emails often utilize persuasion techniques, such as social proof, liking, consistency, authority, scarcity, and reciprocity to gain trust to obtain sensitive information or maliciously infect devices. The link between principles of persuasion and social engineering attacks, particularly in phishing email attacks, is an important topic in cyber security as they are the common and effective method used by cybercriminals to obtain sensitive information or access computer systems. This survey paper concluded that spear phishing, a targeted form of phishing, has been found to be specifically effective as attackers can tailor their messages to the specific characteristics, interests, and vulnerabilities of their targets. Understanding the uses of the principles of persuasion in spear phishing is key to the effective defence against it and eventually its elimination. This survey paper systematically summarizes and presents the current state of the art in understanding the use of principles of persuasion in phishing. Through a systematic review of the existing literature, this survey paper identifies a significant gap in the understanding of the impact of principles of persuasion as a social engineering strategy in phishing attacks and highlights the need for further research in this area.

Title: Generating event descriptions under syntactic and semantic constraints

• Authors: Angela Cao, Faye Holt, Jonas Chan, Stephanie Richter, Lelia Glass, Aaron Steven White
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18496
• Pdf URL: https://arxiv.org/pdf/2412.18496
• Copy Paste: [[2412.18496]] Generating event descriptions under syntactic and semantic constraints(https://arxiv.org/abs/2412.18496)
• Keywords: robust
• Abstract: With the goal of supporting scalable lexical semantic annotation, analysis, and theorizing, we conduct a comprehensive evaluation of different methods for generating event descriptions under both syntactic constraints -- e.g. desired clause structure -- and semantic constraints -- e.g. desired verb sense. We compare three different methods -- (i) manual generation by experts; (ii) sampling from a corpus annotated for syntactic and semantic information; and (iii) sampling from a language model (LM) conditioned on syntactic and semantic information -- along three dimensions of the generated event descriptions: (a) naturalness, (b) typicality, and (c) distinctiveness. We find that all methods reliably produce natural, typical, and distinctive event descriptions, but that manual generation continues to produce event descriptions that are more natural, typical, and distinctive than the automated generation methods. We conclude that the automated methods we consider produce event descriptions of sufficient quality for use in downstream annotation and analysis insofar as the methods used for this annotation and analysis are robust to a small amount of degradation in the resulting event descriptions.

Title: Think or Remember? Detecting and Directing LLMs Towards Memorization or Generalization

• Authors: Yi-Fu Fu, Yu-Chieh Tu, Tzu-Ling Cheng, Cheng-Yu Lin, Yi-Ting Yang, Heng-Yi Liu, Keng-Te Liao, Da-Cheng Juan, Shou-De Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18497
• Pdf URL: https://arxiv.org/pdf/2412.18497
• Copy Paste: [[2412.18497]] Think or Remember? Detecting and Directing LLMs Towards Memorization or Generalization(https://arxiv.org/abs/2412.18497)
• Keywords: large language model
• Abstract: In this paper, we explore the foundational mechanisms of memorization and generalization in Large Language Models (LLMs), inspired by the functional specialization observed in the human brain. Our investigation serves as a case study leveraging specially designed datasets and experimental-scale LLMs to lay the groundwork for understanding these behaviors. Specifically, we aim to first enable LLMs to exhibit both memorization and generalization by training with the designed dataset, then (a) examine whether LLMs exhibit neuron-level spatial differentiation for memorization and generalization, (b) predict these behaviors using model internal representations, and (c) steer the behaviors through inference-time interventions. Our findings reveal that neuron-wise differentiation of memorization and generalization is observable in LLMs, and targeted interventions can successfully direct their behavior.

Title: VORTEX: A Spatial Computing Framework for Optimized Drone Telemetry Extraction from First-Person View Flight Data

• Authors: James E. Gallagher, Edward J. Oughton
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18505
• Pdf URL: https://arxiv.org/pdf/2412.18505
• Copy Paste: [[2412.18505]] VORTEX: A Spatial Computing Framework for Optimized Drone Telemetry Extraction from First-Person View Flight Data(https://arxiv.org/abs/2412.18505)
• Keywords: robust, extraction
• Abstract: This paper presents the Visual Optical Recognition Telemetry EXtraction (VORTEX) system for extracting and analyzing drone telemetry data from First Person View (FPV) Uncrewed Aerial System (UAS) footage. VORTEX employs MMOCR, a PyTorch-based Optical Character Recognition (OCR) toolbox, to extract telemetry variables from drone Heads Up Display (HUD) recordings, utilizing advanced image preprocessing techniques, including CLAHE enhancement and adaptive thresholding. The study optimizes spatial accuracy and computational efficiency through systematic investigation of temporal sampling rates (1s, 5s, 10s, 15s, 20s) and coordinate processing methods. Results demonstrate that the 5-second sampling rate, utilizing 4.07% of available frames, provides the optimal balance with a point retention rate of 64% and mean speed accuracy within 4.2% of the 1-second baseline while reducing computational overhead by 80.5%. Comparative analysis of coordinate processing methods reveals that while UTM Zone 33N projection and Haversine calculations provide consistently similar results (within 0.1% difference), raw WGS84 coordinates underestimate distances by 15-30% and speeds by 20-35%. Altitude measurements showed unexpected resilience to sampling rate variations, with only 2.1% variation across all intervals. This research is the first of its kind, providing quantitative benchmarks for establishing a robust framework for drone telemetry extraction and analysis using open-source tools and spatial libraries.

Title: An Empirical Analysis of Federated Learning Models Subject to Label-Flipping Adversarial Attack

• Authors: Kunal Bhatnagar, Sagana Chattanathan, Angela Dang, Bhargav Eranki, Ronnit Rana, Charan Sridhar, Siddharth Vedam, Angie Yao, Mark Stamp
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18507
• Pdf URL: https://arxiv.org/pdf/2412.18507
• Copy Paste: [[2412.18507]] An Empirical Analysis of Federated Learning Models Subject to Label-Flipping Adversarial Attack(https://arxiv.org/abs/2412.18507)
• Keywords: attack, robust, federate
• Abstract: In this paper, we empirically analyze adversarial attacks on selected federated learning models. The specific learning models considered are Multinominal Logistic Regression (MLR), Support Vector Classifier (SVC), Multilayer Perceptron (MLP), Convolution Neural Network (CNN), %Recurrent Neural Network (RNN), Random Forest, XGBoost, and Long Short-Term Memory (LSTM). For each model, we simulate label-flipping attacks, experimenting extensively with 10 federated clients and 100 federated clients. We vary the percentage of adversarial clients from 10% to 100% and, simultaneously, the percentage of labels flipped by each adversarial client is also varied from 10% to 100%. Among other results, we find that models differ in their inherent robustness to the two vectors in our label-flipping attack, i.e., the percentage of adversarial clients, and the percentage of labels flipped by each adversarial client. We discuss the potential practical implications of our results.

Title: FedGIG: Graph Inversion from Gradient in Federated Learning

• Authors: Tianzhe Xiao, Yichen Li, Yining Qi, Haozhao Wang, Ruixuan Li
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.18513
• Pdf URL: https://arxiv.org/pdf/2412.18513
• Copy Paste: [[2412.18513]] FedGIG: Graph Inversion from Gradient in Federated Learning(https://arxiv.org/abs/2412.18513)
• Keywords: attack, federate
• Abstract: Recent studies have shown that Federated learning (FL) is vulnerable to Gradient Inversion Attacks (GIA), which can recover private training data from shared gradients. However, existing methods are designed for dense, continuous data such as images or vectorized texts, and cannot be directly applied to sparse and discrete graph data. This paper first explores GIA's impact on Federated Graph Learning (FGL) and introduces Graph Inversion from Gradient in Federated Learning (FedGIG), a novel GIA method specifically designed for graph-structured data. FedGIG includes the adjacency matrix constraining module, which ensures the sparsity and discreteness of the reconstructed graph data, and the subgraph reconstruction module, which is designed to complete missing common subgraph structures. Extensive experiments on molecular datasets demonstrate FedGIG's superior accuracy over existing GIA techniques.

Title: HTR-JAND: Handwritten Text Recognition with Joint Attention Network and Knowledge Distillation

• Authors: Mohammed Hamdan, Abderrahmane Rahiche, Mohamed Cheriet
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18524
• Pdf URL: https://arxiv.org/pdf/2412.18524
• Copy Paste: [[2412.18524]] HTR-JAND: Handwritten Text Recognition with Joint Attention Network and Knowledge Distillation(https://arxiv.org/abs/2412.18524)
• Keywords: robust, extraction
• Abstract: Despite significant advances in deep learning, current Handwritten Text Recognition (HTR) systems struggle with the inherent complexity of historical documents, including diverse writing styles, degraded text quality, and computational efficiency requirements across multiple languages and time periods. This paper introduces HTR-JAND (HTR-JAND: Handwritten Text Recognition with Joint Attention Network and Knowledge Distillation), an efficient HTR framework that combines advanced feature extraction with knowledge distillation. Our architecture incorporates three key components: (1) a CNN architecture integrating FullGatedConv2d layers with Squeeze-and-Excitation blocks for adaptive feature extraction, (2) a Combined Attention mechanism fusing Multi-Head Self-Attention with Proxima Attention for robust sequence modeling, and (3) a Knowledge Distillation framework enabling efficient model compression while preserving accuracy through curriculum-based training. The HTR-JAND framework implements a multi-stage training approach combining curriculum learning, synthetic data generation, and multi-task learning for cross-dataset knowledge transfer. We enhance recognition accuracy through context-aware T5 post-processing, particularly effective for historical documents. Comprehensive evaluations demonstrate HTR-JAND's effectiveness, achieving state-of-the-art Character Error Rates (CER) of 1.23\%, 1.02\%, and 2.02\% on IAM, RIMES, and Bentham datasets respectively. Our Student model achieves a 48\% parameter reduction (0.75M versus 1.5M parameters) while maintaining competitive performance through efficient knowledge transfer. Source code and pre-trained models are available at \href{this https URL}{Github}.

Title: The Key of Understanding Vision Tasks: Explanatory Instructions

• Authors: Yang Shen, Xiu-Shen Wei, Yifan Sun, Yuxin Song, Tao Yuan, Jian Jin, Heyang Xu, Yazhou Yao, Errui Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18525
• Pdf URL: https://arxiv.org/pdf/2412.18525
• Copy Paste: [[2412.18525]] The Key of Understanding Vision Tasks: Explanatory Instructions(https://arxiv.org/abs/2412.18525)
• Keywords: transformer, segmentation
• Abstract: Computer Vision (CV) has yet to fully achieve the zero-shot task generalization observed in Natural Language Processing (NLP), despite following many of the milestones established in NLP, such as large transformer models, extensive pre-training, and the auto-regression paradigm, among others. In this paper, we explore the idea that CV adopts discrete and terminological task definitions (\eg, ``image segmentation''), which may be a key barrier to zero-shot task generalization. Our hypothesis is that without truly understanding previously-seen tasks--due to these terminological definitions--deep models struggle to generalize to novel tasks. To verify this, we introduce Explanatory Instructions, which provide an intuitive way to define CV task objectives through detailed linguistic transformations from input images to outputs. We create a large-scale dataset comprising 12 million ``image input $\to$ explanatory instruction $\to$ output'' triplets, and train an auto-regressive-based vision-language model (AR-based VLM) that takes both images and explanatory instructions as input. By learning to follow these instructions, the AR-based VLM achieves instruction-level zero-shot capabilities for previously-seen tasks and demonstrates strong zero-shot generalization for unseen CV tasks. Code and dataset will be openly available on our GitHub repository.

Title: Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation

• Authors: Derong Xu Xinhang Li, Ziheng Zhang, Zhenxi Lin, Zhihong Zhu, Zhi Zheng, Xian Wu, Xiangyu Zhao, Tong Xu, Enhong Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18537
• Pdf URL: https://arxiv.org/pdf/2412.18537
• Copy Paste: [[2412.18537]] Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation(https://arxiv.org/abs/2412.18537)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) demonstrate remarkable capabilities, yet struggle with hallucination and outdated knowledge when tasked with complex knowledge reasoning, resulting in factually incorrect outputs. Previous studies have attempted to mitigate it by retrieving factual knowledge from large-scale knowledge graphs (KGs) to assist LLMs in logical reasoning and prediction of answers. However, this kind of approach often introduces noise and irrelevant data, especially in situations with extensive context from multiple knowledge aspects. In this way, LLM attention can be potentially mislead from question and relevant information. In our study, we introduce an Adaptive Multi-Aspect Retrieval-augmented over KGs (Amar) framework. This method retrieves knowledge including entities, relations, and subgraphs, and converts each piece of retrieved text into prompt embeddings. The Amar framework comprises two key sub-components: 1) a self-alignment module that aligns commonalities among entities, relations, and subgraphs to enhance retrieved text, thereby reducing noise interference; 2) a relevance gating module that employs a soft gate to learn the relevance score between question and multi-aspect retrieved data, to determine which information should be used to enhance LLMs' output, or even filtered altogether. Our method has achieved state-of-the-art performance on two common datasets, WebQSP and CWQ, showing a 1.9\% improvement in accuracy over its best competitor and a 6.6\% improvement in logical form generation over a method that directly uses retrieved text as context prompts. These results demonstrate the effectiveness of Amar in improving the reasoning of LLMs.

Title: Token-Budget-Aware LLM Reasoning

• Authors: Tingxu Han, Chunrong Fang, Shiyu Zhao, Shiqing Ma, Zhenyu Chen, Zhenting Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18547
• Pdf URL: https://arxiv.org/pdf/2412.18547
• Copy Paste: [[2412.18547]] Token-Budget-Aware LLM Reasoning(https://arxiv.org/abs/2412.18547)
• Keywords: large language model
• Abstract: Reasoning is critical for large language models (LLMs) to excel in a wide range of tasks. While methods like Chain-of-Thought (CoT) reasoning enhance LLM performance by decomposing problems into intermediate steps, they also incur significant overhead in token usage, leading to increased costs. We find that the reasoning process of current LLMs is unnecessarily lengthy and it can be compressed by including a reasonable token budget in the prompt, but the choice of token budget plays a crucial role in the actual compression effectiveness. We then propose a token-budget-aware LLM reasoning framework, which dynamically estimates token budgets for different problems based on reasoning complexity and uses the estimated token budgets to guide the reasoning process. Experiments show that our method effectively reduces token costs in CoT reasoning with only a slight performance reduction, offering a practical solution to balance efficiency and accuracy in LLM reasoning. Code: this https URL.

Title: Distilling Fine-grained Sentiment Understanding from Large Language Models

• Authors: Yice Zhang, Guangyu Xie, Hongling Xu, Kaiheng Hou, Jianzhu Bao, Qianlong Wang, Shiwei Chen, Ruifeng Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.18552
• Pdf URL: https://arxiv.org/pdf/2412.18552
• Copy Paste: [[2412.18552]] Distilling Fine-grained Sentiment Understanding from Large Language Models(https://arxiv.org/abs/2412.18552)
• Keywords: large language model
• Abstract: Fine-grained sentiment analysis (FSA) aims to extract and summarize user opinions from vast opinionated text. Recent studies demonstrate that large language models (LLMs) possess exceptional sentiment understanding capabilities. However, directly deploying LLMs for FSA applications incurs high inference costs. Therefore, this paper investigates the distillation of fine-grained sentiment understanding from LLMs into small language models (SLMs). We prompt LLMs to examine and interpret the sentiments of given reviews and then utilize the generated content to pretrain SLMs. Additionally, we develop a comprehensive FSA benchmark to evaluate both SLMs and LLMs. Extensive experiments on this benchmark reveal that: (1) distillation significantly enhances the performance of SLMs in FSA tasks, achieving a 6.00\% improvement in $F_1$-score, and the distilled model can outperform Llama-2-7b with only 220M parameters; (2) distillation equips SLMs with excellent zero-shot sentiment classification capabilities, enabling them to match or even exceed their teacher models. These results suggest that distillation from LLMs is a highly promising direction for FSA. We will release our code, data, and pretrained model weights at \url{this https URL}.

Title: FedVCK: Non-IID Robust and Communication-Efficient Federated Learning via Valuable Condensed Knowledge for Medical Image Analysis

• Authors: Guochen Yan, Luyuan Xie, Xinyi Gao, Wentao Zhang, Qingni Shen, Yuejian Fang, Zhonghai Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18557
• Pdf URL: https://arxiv.org/pdf/2412.18557
• Copy Paste: [[2412.18557]] FedVCK: Non-IID Robust and Communication-Efficient Federated Learning via Valuable Condensed Knowledge for Medical Image Analysis(https://arxiv.org/abs/2412.18557)
• Keywords: privacy, robust, federate
• Abstract: Federated learning has become a promising solution for collaboration among medical institutions. However, data owned by each institution would be highly heterogeneous and the distribution is always non-independent and identical distribution (non-IID), resulting in client drift and unsatisfactory performance. Despite existing federated learning methods attempting to solve the non-IID problems, they still show marginal advantages but rely on frequent communication which would incur high costs and privacy concerns. In this paper, we propose a novel federated learning method: \textbf{Fed}erated learning via \textbf{V}aluable \textbf{C}ondensed \textbf{K}nowledge (FedVCK). We enhance the quality of condensed knowledge and select the most necessary knowledge guided by models, to tackle the non-IID problem within limited communication budgets effectively. Specifically, on the client side, we condense the knowledge of each client into a small dataset and further enhance the condensation procedure with latent distribution constraints, facilitating the effective capture of high-quality knowledge. During each round, we specifically target and condense knowledge that has not been assimilated by the current model, thereby preventing unnecessary repetition of homogeneous knowledge and minimizing the frequency of communications required. On the server side, we propose relational supervised contrastive learning to provide more supervision signals to aid the global model updating. Comprehensive experiments across various medical tasks show that FedVCK can outperform state-of-the-art methods, demonstrating that it's non-IID robust and communication-efficient.

Title: Efficient Aircraft Design Optimization Using Multi-Fidelity Models and Multi-fidelity Physics Informed Neural Networks

• Authors: Apurba Sarker
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2412.18564
• Pdf URL: https://arxiv.org/pdf/2412.18564
• Copy Paste: [[2412.18564]] Efficient Aircraft Design Optimization Using Multi-Fidelity Models and Multi-fidelity Physics Informed Neural Networks(https://arxiv.org/abs/2412.18564)
• Keywords: generative
• Abstract: Aircraft design optimization traditionally relies on computationally expensive simulation techniques such as Finite Element Method (FEM) and Finite Volume Method (FVM), which, while accurate, can significantly slow down the design iteration process. The challenge lies in reducing the computational complexity while maintaining high accuracy for quick evaluations of multiple design alternatives. This research explores advanced methods, including surrogate models, reduced-order models (ROM), and multi-fidelity machine learning techniques, to achieve more efficient aircraft design evaluations. Specifically, the study investigates the application of Multi-fidelity Physics-Informed Neural Networks (MPINN) and autoencoders for manifold alignment, alongside the potential of Generative Adversarial Networks (GANs) for refining design geometries. Through a proof-of-concept task, the research demonstrates the ability to predict high-fidelity results from low-fidelity simulations, offering a path toward faster and more cost effective aircraft design iterations.

Title: 3DEnhancer: Consistent Multi-View Diffusion for 3D Enhancement

• Authors: Yihang Luo, Shangchen Zhou, Yushi Lan, Xingang Pan, Chen Change Loy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18565
• Pdf URL: https://arxiv.org/pdf/2412.18565
• Copy Paste: [[2412.18565]] 3DEnhancer: Consistent Multi-View Diffusion for 3D Enhancement(https://arxiv.org/abs/2412.18565)
• Keywords: diffusion
• Abstract: Despite advances in neural rendering, due to the scarcity of high-quality 3D datasets and the inherent limitations of multi-view diffusion models, view synthesis and 3D model generation are restricted to low resolutions with suboptimal multi-view consistency. In this study, we present a novel 3D enhancement pipeline, dubbed 3DEnhancer, which employs a multi-view latent diffusion model to enhance coarse 3D inputs while preserving multi-view consistency. Our method includes a pose-aware encoder and a diffusion-based denoiser to refine low-quality multi-view images, along with data augmentation and a multi-view attention module with epipolar aggregation to maintain consistent, high-quality 3D outputs across views. Unlike existing video-based approaches, our model supports seamless multi-view enhancement with improved coherence across diverse viewing angles. Extensive evaluations show that 3DEnhancer significantly outperforms existing methods, boosting both multi-view enhancement and per-instance 3D optimization tasks.

Title: Zero-resource Speech Translation and Recognition with LLMs

• Authors: Karel Mundnich, Xing Niu, Prashant Mathur, Srikanth Ronanki, Brady Houston, Veera Raghavendra Elluru, Nilaksh Das, Zejiang Hou, Goeric Huybrechts, Anshu Bhatia, Daniel Garcia-Romero, Kyu J. Han, Katrin Kirchhoff
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.18566
• Pdf URL: https://arxiv.org/pdf/2412.18566
• Copy Paste: [[2412.18566]] Zero-resource Speech Translation and Recognition with LLMs(https://arxiv.org/abs/2412.18566)
• Keywords: large language model
• Abstract: Despite recent advancements in speech processing, zero-resource speech translation (ST) and automatic speech recognition (ASR) remain challenging problems. In this work, we propose to leverage a multilingual Large Language Model (LLM) to perform ST and ASR in languages for which the model has never seen paired audio-text data. We achieve this by using a pre-trained multilingual speech encoder, a multilingual LLM, and a lightweight adaptation module that maps the audio representations to the token embedding space of the LLM. We perform several experiments both in ST and ASR to understand how to best train the model and what data has the most impact on performance in previously unseen languages. In ST, our best model is capable to achieve BLEU scores over 23 in CoVoST2 for two previously unseen languages, while in ASR, we achieve WERs of up to 28.2\%. We finally show that the performance of our system is bounded by the ability of the LLM to output text in the desired language.

Title: Exploring Embedding Priors in Prompt-Tuning for Improved Interpretability and Control

• Authors: Sergey Sedov, Sumanth Bharadwaj Hachalli Karanam, Venu Gopal Kadamba
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.18582
• Pdf URL: https://arxiv.org/pdf/2412.18582
• Copy Paste: [[2412.18582]] Exploring Embedding Priors in Prompt-Tuning for Improved Interpretability and Control(https://arxiv.org/abs/2412.18582)
• Keywords: interpretability, large language model
• Abstract: Prompt-Tuning is an efficient method for adapting pre-trained language models to new tasks with minimal computational overhead by modifying prompt embeddings. In this work, we investigate how crucial the phenomenon of embedding collapse, frequently observed in Prompt-Tuning, is for the final performance of the model. To address this question, we designed embedding priors and compared them with posteriors of the converged Soft and Deep Prompt-Tuning methods. Our findings suggest that priors strongly affect the position of the tuned embeddings, and models can effectively work with embeddings from different parts of activation spaces, including completely new regions. As the final Prompt-Tuning capabilities are limited, we hypothesize that controllable Prompt-Tuning posteriors may serve as a good starting point for tasks such as chain-of-thought (COT) distillation. Our experiments also show that generated trajectories are not localized in the activation space of the models. However, there are distinct clusters of activations for distant tasks (e.g., NLP and arithmetic), while activations between NLP tasks (e.g., Question-Answering and MLM) lie in the same cluster. These observations raise questions about the importance of a single activation cluster for the generalization abilities of large language models.

Title: Resolution-Robust 3D MRI Reconstruction with 2D Diffusion Priors: Diverse-Resolution Training Outperforms Interpolation

• Authors: Anselm Krainovic, Stefan Ruschke, Reinhard Heckel
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.18584
• Pdf URL: https://arxiv.org/pdf/2412.18584
• Copy Paste: [[2412.18584]] Resolution-Robust 3D MRI Reconstruction with 2D Diffusion Priors: Diverse-Resolution Training Outperforms Interpolation(https://arxiv.org/abs/2412.18584)
• Keywords: robust, diffusion
• Abstract: Deep learning-based 3D imaging, in particular magnetic resonance imaging (MRI), is challenging because of limited availability of 3D training data. Therefore, 2D diffusion models trained on 2D slices are starting to be leveraged for 3D MRI reconstruction. However, as we show in this paper, existing methods pertain to a fixed voxel size, and performance degrades when the voxel size is varied, as it is often the case in clinical practice. In this paper, we propose and study several approaches for resolution-robust 3D MRI reconstruction with 2D diffusion priors. As a result of this investigation, we obtain a simple resolution-robust variational 3D reconstruction approach based on diffusion-guided regularization of randomly sampled 2D slices. This method provides competitive reconstruction quality compared to posterior sampling baselines. Towards resolving the sensitivity to resolution-shifts, we investigate state-of-the-art model-based approaches including Gaussian splatting, neural representations, and infinite-dimensional diffusion models, as well as a simple data-centric approach of training the diffusion model on several resolutions. Our experiments demonstrate that the model-based approaches fail to close the performance gap in 3D MRI. In contrast, the data-centric approach of training the diffusion model on various resolutions effectively provides a resolution-robust method without compromising accuracy.

Title: ClassifyViStA:WCE Classification with Visual understanding through Segmentation and Attention

• Authors: S. Balasubramanian, Ammu Abhishek, Yedu Krishna, Darshan Gera
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18591
• Pdf URL: https://arxiv.org/pdf/2412.18591
• Copy Paste: [[2412.18591]] ClassifyViStA:WCE Classification with Visual understanding through Segmentation and Attention(https://arxiv.org/abs/2412.18591)
• Keywords: interpretability, segmentation
• Abstract: Gastrointestinal (GI) bleeding is a serious medical condition that presents significant diagnostic challenges, particularly in settings with limited access to healthcare resources. Wireless Capsule Endoscopy (WCE) has emerged as a powerful diagnostic tool for visualizing the GI tract, but it requires time-consuming manual analysis by experienced gastroenterologists, which is prone to human error and inefficient given the increasing number of this http URL address this challenge, we propose ClassifyViStA, an AI-based framework designed for the automated detection and classification of bleeding and non-bleeding frames from WCE videos. The model consists of a standard classification path, augmented by two specialized branches: an implicit attention branch and a segmentation this http URL attention branch focuses on the bleeding regions, while the segmentation branch generates accurate segmentation masks, which are used for classification and interpretability. The model is built upon an ensemble of ResNet18 and VGG16 architectures to enhance classification performance. For the bleeding region detection, we implement a Soft Non-Maximum Suppression (Soft NMS) approach with YOLOv8, which improves the handling of overlapping bounding boxes, resulting in more accurate and nuanced this http URL system's interpretability is enhanced by using the segmentation masks to explain the classification results, offering insights into the decision-making process similar to the way a gastroenterologist identifies bleeding regions. Our approach not only automates the detection of GI bleeding but also provides an interpretable solution that can ease the burden on healthcare professionals and improve diagnostic efficiency. Our code is available at ClassifyViStA.

Title: LatentCRF: Continuous CRF for Efficient Latent Diffusion

• Authors: Kanchana Ranasinghe, Sadeep Jayasumana, Andreas Veit, Ayan Chakrabarti, Daniel Glasner, Michael S Ryoo, Srikumar Ramalingam, Sanjiv Kumar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18596
• Pdf URL: https://arxiv.org/pdf/2412.18596
• Copy Paste: [[2412.18596]] LatentCRF: Continuous CRF for Efficient Latent Diffusion(https://arxiv.org/abs/2412.18596)
• Keywords: diffusion
• Abstract: Latent Diffusion Models (LDMs) produce high-quality, photo-realistic images, however, the latency incurred by multiple costly inference iterations can restrict their applicability. We introduce LatentCRF, a continuous Conditional Random Field (CRF) model, implemented as a neural network layer, that models the spatial and semantic relationships among the latent vectors in the LDM. By replacing some of the computationally-intensive LDM inference iterations with our lightweight LatentCRF, we achieve a superior balance between quality, speed and diversity. We increase inference efficiency by 33% with no loss in image quality or diversity compared to the full LDM. LatentCRF is an easy add-on, which does not require modifying the LDM.

Title: DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation

• Authors: Minghong Cai, Xiaodong Cun, Xiaoyu Li, Wenze Liu, Zhaoyang Zhang, Yong Zhang, Ying Shan, Xiangyu Yue
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.18597
• Pdf URL: https://arxiv.org/pdf/2412.18597
• Copy Paste: [[2412.18597]] DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation(https://arxiv.org/abs/2412.18597)
• Keywords: diffusion, transformer
• Abstract: Sora-like video generation models have achieved remarkable progress with a Multi-Modal Diffusion Transformer MM-DiT architecture. However, the current video generation models predominantly focus on single-prompt, struggling to generate coherent scenes with multiple sequential prompts that better reflect real-world dynamic scenarios. While some pioneering works have explored multi-prompt video generation, they face significant challenges including strict training data requirements, weak prompt following, and unnatural transitions. To address these problems, we propose DiTCtrl, a training-free multi-prompt video generation method under MM-DiT architectures for the first time. Our key idea is to take the multi-prompt video generation task as temporal video editing with smooth transitions. To achieve this goal, we first analyze MM-DiT's attention mechanism, finding that the 3D full attention behaves similarly to that of the cross/self-attention blocks in the UNet-like diffusion models, enabling mask-guided precise semantic control across different prompts with attention sharing for multi-prompt video generation. Based on our careful design, the video generated by DiTCtrl achieves smooth transitions and consistent object motion given multiple sequential prompts without additional training. Besides, we also present MPVBench, a new benchmark specially designed for multi-prompt video generation to evaluate the performance of multi-prompt generation. Extensive experiments demonstrate that our method achieves state-of-the-art performance without additional training.

Title: Decentralized Intelligence in GameFi: Embodied AI Agents and the Convergence of DeFi and Virtual Ecosystems

• Authors: Fernando Jia, Jade Zheng, Florence Li
• Subjects: cs.CR, cs.AI, cs.GT, cs.LG, cs.MA
• Abstract URL: https://arxiv.org/abs/2412.18601
• Pdf URL: https://arxiv.org/pdf/2412.18601
• Copy Paste: [[2412.18601]] Decentralized Intelligence in GameFi: Embodied AI Agents and the Convergence of DeFi and Virtual Ecosystems(https://arxiv.org/abs/2412.18601)
• Keywords: robust, large language model
• Abstract: In the rapidly evolving landscape of GameFi, a fusion of gaming and decentralized finance (DeFi), there exists a critical need to enhance player engagement and economic interaction within gaming ecosystems. Our GameFi ecosystem aims to fundamentally transform this landscape by integrating advanced embodied AI agents into GameFi platforms. These AI agents, developed using cutting-edge large language models (LLMs), such as GPT-4 and Claude AI, are capable of proactive, adaptive, and contextually rich interactions with players. By going beyond traditional scripted responses, these agents become integral participants in the game's narrative and economic systems, directly influencing player strategies and in-game economies. We address the limitations of current GameFi platforms, which often lack immersive AI interactions and mechanisms for community engagement or creator monetization. Through the deep integration of AI agents with blockchain technology, we establish a consensus-driven, decentralized GameFi ecosystem. This ecosystem empowers creators to monetize their contributions and fosters democratic collaboration among players and creators. Furthermore, by embedding DeFi mechanisms into the gaming experience, we enhance economic participation and provide new opportunities for financial interactions within the game. Our approach enhances player immersion and retention and advances the GameFi ecosystem by bridging traditional gaming with Web3 technologies. By integrating sophisticated AI and DeFi elements, we contribute to the development of more engaging, economically robust, and community-centric gaming environments. This project represents a significant advancement in the state-of-the-art in GameFi, offering insights and methodologies that can be applied throughout the gaming industry.

Title: Long-Form Speech Generation with Spoken Language Models

• Authors: Se Jin Park, Julian Salazar, Aren Jansen, Keisuke Kinoshita, Yong Man Ro, RJ Skerry-Ryan
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.18603
• Pdf URL: https://arxiv.org/pdf/2412.18603
• Copy Paste: [[2412.18603]] Long-Form Speech Generation with Spoken Language Models(https://arxiv.org/abs/2412.18603)
• Keywords: generative
• Abstract: We consider the generative modeling of speech over multiple minutes, a requirement for long-form multimedia generation and audio-native voice assistants. However, current spoken language models struggle to generate plausible speech past tens of seconds, from high temporal resolution of speech tokens causing loss of coherence, to architectural issues with long-sequence training or extrapolation, to memory costs at inference time. With these considerations we propose SpeechSSM, the first speech language model to learn from and sample long-form spoken audio (e.g., 16 minutes of read or extemporaneous speech) in a single decoding session without text intermediates, based on recent advances in linear-time sequence modeling. Furthermore, to address growing challenges in spoken language evaluation, especially in this new long-form setting, we propose: new embedding-based and LLM-judged metrics; quality measurements over length and time; and a new benchmark for long-form speech processing and generation, LibriSpeech-Long. Speech samples and the dataset are released at this https URL

Title: Explaining in Diffusion: Explaining a Classifier Through Hierarchical Semantics with Text-to-Image Diffusion Models

• Authors: Tahira Kazimi, Ritika Allada, Pinar Yanardag
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18604
• Pdf URL: https://arxiv.org/pdf/2412.18604
• Copy Paste: [[2412.18604]] Explaining in Diffusion: Explaining a Classifier Through Hierarchical Semantics with Text-to-Image Diffusion Models(https://arxiv.org/abs/2412.18604)
• Keywords: explainability, diffusion
• Abstract: Classifiers are important components in many computer vision tasks, serving as the foundational backbone of a wide variety of models employed across diverse applications. However, understanding the decision-making process of classifiers remains a significant challenge. We propose DiffEx, a novel method that leverages the capabilities of text-to-image diffusion models to explain classifier decisions. Unlike traditional GAN-based explainability models, which are limited to simple, single-concept analyses and typically require training a new model for each classifier, our approach can explain classifiers that focus on single concepts (such as faces or animals) as well as those that handle complex scenes involving multiple concepts. DiffEx employs vision-language models to create a hierarchical list of semantics, allowing users to identify not only the overarching semantic influences on classifiers (e.g., the 'beard' semantic in a facial classifier) but also their sub-types, such as 'goatee' or 'Balbo' beard. Our experiments demonstrate that DiffEx is able to cover a significantly broader spectrum of semantics compared to its GAN counterparts, providing a hierarchical tool that delivers a more detailed and fine-grained understanding of classifier decisions.

Title: Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models

• Authors: Zehan Wang, Ziang Zhang, Tianyu Pang, Chao Du, Hengshuang Zhao, Zhou Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18605
• Pdf URL: https://arxiv.org/pdf/2412.18605
• Copy Paste: [[2412.18605]] Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Models(https://arxiv.org/abs/2412.18605)
• Keywords: robust
• Abstract: Orientation is a key attribute of objects, crucial for understanding their spatial pose and arrangement in images. However, practical solutions for accurate orientation estimation from a single image remain underexplored. In this work, we introduce Orient Anything, the first expert and foundational model designed to estimate object orientation in a single- and free-view image. Due to the scarcity of labeled data, we propose extracting knowledge from the 3D world. By developing a pipeline to annotate the front face of 3D objects and render images from random views, we collect 2M images with precise orientation annotations. To fully leverage the dataset, we design a robust training objective that models the 3D orientation as probability distributions of three angles and predicts the object orientation by fitting these distributions. Besides, we employ several strategies to improve synthetic-to-real transfer. Our model achieves state-of-the-art orientation estimation accuracy in both rendered and real images and exhibits impressive zero-shot ability in various scenarios. More importantly, our model enhances many applications, such as comprehension and generation of complex spatial concepts and 3D object pose adjustment.

Title: DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers

• Authors: Yuntao Chen, Yuqi Wang, Zhaoxiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18607
• Pdf URL: https://arxiv.org/pdf/2412.18607
• Copy Paste: [[2412.18607]] DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers(https://arxiv.org/abs/2412.18607)
• Keywords: diffusion, transformer
• Abstract: World model-based searching and planning are widely recognized as a promising path toward human-level physical intelligence. However, current driving world models primarily rely on video diffusion models, which specialize in visual generation but lack the flexibility to incorporate other modalities like action. In contrast, autoregressive transformers have demonstrated exceptional capability in modeling multimodal data. Our work aims to unify both driving model simulation and trajectory planning into a single sequence modeling problem. We introduce a multimodal driving language based on interleaved image and action tokens, and develop DrivingGPT to learn joint world modeling and planning through standard next-token prediction. Our DrivingGPT demonstrates strong performance in both action-conditioned video generation and end-to-end planning, outperforming strong baselines on large-scale nuPlan and NAVSIM benchmarks.

Title: PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models

• Authors: Minghao Chen, Roman Shapovalov, Iro Laina, Tom Monnier, Jianyuan Wang, David Novotny, Andrea Vedaldi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18608
• Pdf URL: https://arxiv.org/pdf/2412.18608
• Copy Paste: [[2412.18608]] PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models(https://arxiv.org/abs/2412.18608)
• Keywords: extraction, diffusion, generative, segmentation
• Abstract: Text- or image-to-3D generators and 3D scanners can now produce 3D assets with high-quality shapes and textures. These assets typically consist of a single, fused representation, like an implicit neural field, a Gaussian mixture, or a mesh, without any useful structure. However, most applications and creative workflows require assets to be made of several meaningful parts that can be manipulated independently. To address this gap, we introduce PartGen, a novel approach that generates 3D objects composed of meaningful parts starting from text, an image, or an unstructured 3D object. First, given multiple views of a 3D object, generated or rendered, a multi-view diffusion model extracts a set of plausible and view-consistent part segmentations, dividing the object into parts. Then, a second multi-view diffusion model takes each part separately, fills in the occlusions, and uses those completed views for 3D reconstruction by feeding them to a 3D reconstruction network. This completion process considers the context of the entire object to ensure that the parts integrate cohesively. The generative completion model can make up for the information missing due to occlusions; in extreme cases, it can hallucinate entirely invisible parts based on the input 3D asset. We evaluate our method on generated and real 3D assets and show that it outperforms segmentation and part-extraction baselines by a large margin. We also showcase downstream applications such as 3D part editing.

Title: Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models

• Authors: Jinhui Yi, Syed Talal Wasim, Yanan Luo, Muzammal Naseer, Juergen Gall
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.18609
• Pdf URL: https://arxiv.org/pdf/2412.18609
• Copy Paste: [[2412.18609]] Video-Panda: Parameter-efficient Alignment for Encoder-free Video-Language Models(https://arxiv.org/abs/2412.18609)
• Keywords: extraction
• Abstract: We present an efficient encoder-free approach for video-language understanding that achieves competitive performance while significantly reducing computational overhead. Current video-language models typically rely on heavyweight image encoders (300M-1.1B parameters) or video encoders (1B-1.4B parameters), creating a substantial computational burden when processing multi-frame videos. Our method introduces a novel Spatio-Temporal Alignment Block (STAB) that directly processes video inputs without requiring pre-trained encoders while using only 45M parameters for visual processing - at least a 6.5$\times$ reduction compared to traditional approaches. The STAB architecture combines Local Spatio-Temporal Encoding for fine-grained feature extraction, efficient spatial downsampling through learned attention and separate mechanisms for modeling frame-level and video-level relationships. Our model achieves comparable or superior performance to encoder-based approaches for open-ended video question answering on standard benchmarks. The fine-grained video question-answering evaluation demonstrates our model's effectiveness, outperforming the encoder-based approaches Video-ChatGPT and Video-LLaVA in key aspects like correctness and temporal understanding. Extensive ablation studies validate our architectural choices and demonstrate the effectiveness of our spatio-temporal modeling approach while achieving 3-4$\times$ faster processing speeds than previous methods. Code is available at \url{this https URL}.