2024-12-17

Title: Reinforcement Learning Enhanced LLMs: A Survey

• Authors: Shuhe Wang, Shengyu Zhang, Jie Zhang, Runyi Hu, Xiaoya Li, Tianwei Zhang, Jiwei Li, Fei Wu, Guoyin Wang, Eduard Hovy
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10400
• Pdf URL: https://arxiv.org/pdf/2412.10400
• Copy Paste: [[2412.10400]] Reinforcement Learning Enhanced LLMs: A Survey(https://arxiv.org/abs/2412.10400)
• Keywords: large language model
• Abstract: This paper surveys research in the rapidly growing field of enhancing large language models (LLMs) with reinforcement learning (RL), a technique that enables LLMs to improve their performance by receiving feedback in the form of rewards based on the quality of their outputs, allowing them to generate more accurate, coherent, and contextually appropriate responses. In this work, we make a systematic review of the most up-to-date state of knowledge on RL-enhanced LLMs, attempting to consolidate and analyze the rapidly growing research in this field, helping researchers understand the current challenges and advancements. Specifically, we (1) detail the basics of RL; (2) introduce popular RL-enhanced LLMs; (3) review researches on two widely-used reward model-based RL techniques: Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF); and (4) explore Direct Preference Optimization (DPO), a set of methods that bypass the reward model to directly use human preference data for aligning LLM outputs with human expectations. We will also point out current challenges and deficiencies of existing methods and suggest some avenues for further improvements.

Title: Evaluating Robustness of LLMs on Crisis-Related Microblogs across Events, Information Types, and Linguistic Features

• Authors: Muhammad Imran, Abdul Wahab Ziaullah, Kai Chen, Ferda Ofli
• Subjects: cs.CL, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2412.10413
• Pdf URL: https://arxiv.org/pdf/2412.10413
• Copy Paste: [[2412.10413]] Evaluating Robustness of LLMs on Crisis-Related Microblogs across Events, Information Types, and Linguistic Features(https://arxiv.org/abs/2412.10413)
• Keywords: robust, large language model
• Abstract: The widespread use of microblogging platforms like X (formerly Twitter) during disasters provides real-time information to governments and response authorities. However, the data from these platforms is often noisy, requiring automated methods to filter relevant information. Traditionally, supervised machine learning models have been used, but they lack generalizability. In contrast, Large Language Models (LLMs) show better capabilities in understanding and processing natural language out of the box. This paper provides a detailed analysis of the performance of six well-known LLMs in processing disaster-related social media data from a large-set of real-world events. Our findings indicate that while LLMs, particularly GPT-4o and GPT-4, offer better generalizability across different disasters and information types, most LLMs face challenges in processing flood-related data, show minimal improvement despite the provision of examples (i.e., shots), and struggle to identify critical information categories like urgent requests and needs. Additionally, we examine how various linguistic features affect model performance and highlight LLMs' vulnerabilities against certain features like typos. Lastly, we provide benchmarking results for all events across both zero- and few-shot settings and observe that proprietary models outperform open-source ones in all tasks.

Title: Generative Adversarial Reviews: When LLMs Become the Critic

• Authors: Nicolas Bougie, Narimasa Watanabe
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10415
• Pdf URL: https://arxiv.org/pdf/2412.10415
• Copy Paste: [[2412.10415]] Generative Adversarial Reviews: When LLMs Become the Critic(https://arxiv.org/abs/2412.10415)
• Keywords: fair, generative, large language model
• Abstract: The peer review process is fundamental to scientific progress, determining which papers meet the quality standards for publication. Yet, the rapid growth of scholarly production and increasing specialization in knowledge areas strain traditional scientific feedback mechanisms. In light of this, we introduce Generative Agent Reviewers (GAR), leveraging LLM-empowered agents to simulate faithful peer reviewers. To enable generative reviewers, we design an architecture that extends a large language model with memory capabilities and equips agents with reviewer personas derived from historical data. Central to this approach is a graph-based representation of manuscripts, condensing content and logically organizing information - linking ideas with evidence and technical details. GAR's review process leverages external knowledge to evaluate paper novelty, followed by detailed assessment using the graph representation and multi-round assessment. Finally, a meta-reviewer aggregates individual reviews to predict the acceptance decision. Our experiments demonstrate that GAR performs comparably to human reviewers in providing detailed feedback and predicting paper outcomes. Beyond mere performance comparison, we conduct insightful experiments, such as evaluating the impact of reviewer expertise and examining fairness in reviews. By offering early expert-level feedback, typically restricted to a limited group of researchers, GAR democratizes access to transparent and in-depth evaluation.

Title: SUPERMERGE: An Approach For Gradient-Based Model Merging

• Authors: Haoyu Yang, Zheng Zhang, Saket Sathe
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10416
• Pdf URL: https://arxiv.org/pdf/2412.10416
• Copy Paste: [[2412.10416]] SUPERMERGE: An Approach For Gradient-Based Model Merging(https://arxiv.org/abs/2412.10416)
• Keywords: large language model
• Abstract: Large language models, such as ChatGPT, Claude, or LLaMA, are gigantic, monolithic, and possess the superpower to simultaneously support thousands of tasks. However, high-throughput applications often prefer smaller task-specific models because of their lower latency and cost. One challenge of using task-specific models is the incremental need for solving newer tasks after the model is already deployed for existing tasks. A straightforward solution requires fine-tuning the model again for both existing and new tasks, which is computationally expensive and time-consuming. To address this issue, we propose a model merging based approach called SUPERMERGE. SUPERMERGE is a gradient-based method to systematically merge several fine-tuned models trained on existing and new tasks. SUPERMERGE is designed to be lightweight and fast, and the merged model achieves similar performance to fully fine-tuned models on all tasks. Furthermore, we proposed a hierarchical model merging strategy to reduce the peak space requirement without sacrificing the performance of the merged model. We experimentally demonstrate that SUPERMERGE outperforms existing model merging methods on common natural language processing and computer vision tasks.

Title: Leveraging Audio and Text Modalities in Mental Health: A Study of LLMs Performance

• Authors: Abdelrahman A. Ali, Aya E. Fouda, Radwa J. Hanafy, Mohammed E. Fouda
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.10417
• Pdf URL: https://arxiv.org/pdf/2412.10417
• Copy Paste: [[2412.10417]] Leveraging Audio and Text Modalities in Mental Health: A Study of LLMs Performance(https://arxiv.org/abs/2412.10417)
• Keywords: robust, large language model
• Abstract: Mental health disorders are increasingly prevalent worldwide, creating an urgent need for innovative tools to support early diagnosis and intervention. This study explores the potential of Large Language Models (LLMs) in multimodal mental health diagnostics, specifically for detecting depression and Post Traumatic Stress Disorder through text and audio modalities. Using the E-DAIC dataset, we compare text and audio modalities to investigate whether LLMs can perform equally well or better with audio inputs. We further examine the integration of both modalities to determine if this can enhance diagnostic accuracy, which generally results in improved performance metrics. Our analysis specifically utilizes custom-formulated metrics; Modal Superiority Score and Disagreement Resolvement Score to evaluate how combined modalities influence model performance. The Gemini 1.5 Pro model achieves the highest scores in binary depression classification when using the combined modality, with an F1 score of 0.67 and a Balanced Accuracy (BA) of 77.4%, assessed across the full dataset. These results represent an increase of 3.1% over its performance with the text modality and 2.7% over the audio modality, highlighting the effectiveness of integrating modalities to enhance diagnostic accuracy. Notably, all results are obtained in zero-shot inferring, highlighting the robustness of the models without requiring task-specific fine-tuning. To explore the impact of different configurations on model performance, we conduct binary, severity, and multiclass tasks using both zero-shot and few-shot prompts, examining the effects of prompt variations on performance. The results reveal that models such as Gemini 1.5 Pro in text and audio modalities, and GPT-4o mini in the text modality, often surpass other models in balanced accuracy and F1 scores across multiple tasks.

Title: AutoPrep: Natural Language Question-Aware Data Preparation with a Multi-Agent Framework

• Authors: Meihao Fan, Ju Fan, Nan Tang, Lei Cao, Xiaoyong Du
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10422
• Pdf URL: https://arxiv.org/pdf/2412.10422
• Copy Paste: [[2412.10422]] AutoPrep: Natural Language Question-Aware Data Preparation with a Multi-Agent Framework(https://arxiv.org/abs/2412.10422)
• Keywords: large language model
• Abstract: Answering natural language (NL) questions about tables, which is referred to as Tabular Question Answering (TQA), is important because it enables users to extract meaningful insights quickly and efficiently from structured data, bridging the gap between human language and machine-readable formats. Many of these tables originate from web sources or real-world scenarios, necessitating careful data preparation (or data prep for short) to ensure accurate answers. However, unlike traditional data prep, question-aware data prep introduces new requirements, which include tasks such as column augmentation and filtering for given questions, and question-aware value normalization or conversion. Because each of the above tasks is unique, a single model (or agent) may not perform effectively across all scenarios. In this paper, we propose AUTOPREP, a large language model (LLM)-based multi-agent framework that leverages the strengths of multiple agents, each specialized in a certain type of data prep, ensuring more accurate and contextually relevant responses. Given an NL question over a table, AUTOPREP performs data prep through three key components. Planner: Determines a logical plan, outlining a sequence of high-level operations. Programmer: Translates this logical plan into a physical plan by generating the corresponding low-level code. Executor: Iteratively executes and debugs the generated code to ensure correct outcomes. To support this multi-agent framework, we design a novel chain-of-thought reasoning mechanism for high-level operation suggestion, and a tool-augmented method for low-level code generation. Extensive experiments on real-world TQA datasets demonstrate that AUTOPREP can significantly improve the SOTA TQA solutions through question-aware data prep.

Title: Look Before You Leap: Enhancing Attention and Vigilance Regarding Harmful Content with GuidelineLLM

• Authors: Shaoqing Zhang, Zhuosheng Zhang, Kehai Chen, Rongxiang Weng, Muyun Yang, Tiejun Zhao, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10423
• Pdf URL: https://arxiv.org/pdf/2412.10423
• Copy Paste: [[2412.10423]] Look Before You Leap: Enhancing Attention and Vigilance Regarding Harmful Content with GuidelineLLM(https://arxiv.org/abs/2412.10423)
• Keywords: attack, large language model
• Abstract: Despite being empowered with alignment mechanisms, large language models (LLMs) are increasingly vulnerable to emerging jailbreak attacks that can compromise their alignment mechanisms. This vulnerability poses significant risks to the real-world applications. Existing work faces challenges in both training efficiency and generalization capabilities (i.e., Reinforcement Learning from Human Feedback and Red-Teaming). Developing effective strategies to enable LLMs to resist continuously evolving jailbreak attempts represents a significant challenge. To address this challenge, we propose a novel defensive paradigm called GuidelineLLM, which assists LLMs in recognizing queries that may have harmful content. Before LLMs respond to a query, GuidelineLLM first identifies potential risks associated with the query, summarizes these risks into guideline suggestions, and then feeds these guidelines to the responding LLMs. Importantly, our approach eliminates the necessity for additional safety fine-tuning of the LLMs themselves; only the GuidelineLLM requires fine-tuning. This characteristic enhances the general applicability of GuidelineLLM across various LLMs. Experimental results demonstrate that GuidelineLLM can significantly reduce the attack success rate (ASR) against the LLMs (an average reduction of 34.17\% ASR) while maintaining the helpfulness of the LLMs in handling benign queries. Code is available at this https URL.

Title: LLM-AS-AN-INTERVIEWER: Beyond Static Testing Through Dynamic LLM Evaluation

• Authors: Eunsu Kim, Juyoung Suk, Seungone Kim, Niklas Muennighoff, Dongkwan Kim, Alice Oh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10424
• Pdf URL: https://arxiv.org/pdf/2412.10424
• Copy Paste: [[2412.10424]] LLM-AS-AN-INTERVIEWER: Beyond Static Testing Through Dynamic LLM Evaluation(https://arxiv.org/abs/2412.10424)
• Keywords: large language model
• Abstract: We introduce a novel evaluation paradigm for large language models (LLMs), LLM-as-an-Interviewer. This approach consists of a two stage process designed to assess the true capabilities of LLMs: first, modifying benchmark datasets to generate initial queries, and second, interacting with the LLM through feedback and follow up questions. Compared to existing evaluation methods such as LLM as a Judge, our framework addresses several limitations, including data contamination, verbosity bias, and self enhancement bias. Additionally, we show that our multi turn evaluation process provides valuable insights into the LLM's performance in real-world scenarios, including its adaptability to feedback and its ability to handle follow up questions, including clarification or requests for additional knowledge. Finally, we propose the Interview Report, which offers a comprehensive reflection of an LLM's strengths and weaknesses, illustrated with specific examples from the interview process. This report delivers a snapshot of the LLM's capabilities, providing a detailed picture of its practical performance.

Title: Active Inference for Self-Organizing Multi-LLM Systems: A Bayesian Thermodynamic Approach to Adaptation

• Authors: Rithvik Prakki
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10425
• Pdf URL: https://arxiv.org/pdf/2412.10425
• Copy Paste: [[2412.10425]] Active Inference for Self-Organizing Multi-LLM Systems: A Bayesian Thermodynamic Approach to Adaptation(https://arxiv.org/abs/2412.10425)
• Keywords: robust, large language model
• Abstract: This paper introduces a novel approach to creating adaptive language agents by integrating active inference with large language models (LLMs). While LLMs demonstrate remarkable capabilities, their reliance on static prompts limits adaptation to new information and changing environments. We address this by implementing an active inference framework that acts as a cognitive layer above an LLM-based agent, dynamically adjusting prompts and search strategies through principled information-seeking behavior. Our framework models the environment using three state factors (prompt, search, and information states) with seven observation modalities capturing quality metrics. By framing the agent's learning through the free energy principle, we enable systematic exploration of prompt combinations and search strategies. Experimental results demonstrate the effectiveness of this approach, with the agent developing accurate models of environment dynamics evidenced by emergent structure in observation matrices. Action selection patterns reveal sophisticated exploration-exploitation behavior, transitioning from initial information-gathering to targeted prompt testing. The integration of thermodynamic principles with language model capabilities provides a principled framework for creating robust, adaptable agents, extending active inference beyond traditional low-dimensional control problems to high-dimensional, language-driven environments.

Title: Identifying and Manipulating Personality Traits in LLMs Through Activation Engineering

• Authors: Rumi A. Allbert, James K. Wiles
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10427
• Pdf URL: https://arxiv.org/pdf/2412.10427
• Copy Paste: [[2412.10427]] Identifying and Manipulating Personality Traits in LLMs Through Activation Engineering(https://arxiv.org/abs/2412.10427)
• Keywords: interpretability, large language model
• Abstract: The field of large language models (LLMs) has grown rapidly in recent years, driven by the desire for better efficiency, interpretability, and safe use. Building on the novel approach of "activation engineering," this study explores personality modification in LLMs, drawing inspiration from research like Refusal in LLMs Is Mediated by a Single Direction (arXiv:2406.11717) and Steering Llama 2 via Contrastive Activation Addition (arXiv:2312.06681). We leverage activation engineering to develop a method for identifying and adjusting activation directions related to personality traits, which may allow for dynamic LLM personality fine-tuning. This work aims to further our understanding of LLM interpretability while examining the ethical implications of such developments.

Title: GPTDrawer: Enhancing Visual Synthesis through ChatGPT

• Authors: Kun Li, Xinwei Chen, Tianyou Song, Hansong Zhang, Wenzhe Zhang, Qing Shan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10429
• Pdf URL: https://arxiv.org/pdf/2412.10429
• Copy Paste: [[2412.10429]] GPTDrawer: Enhancing Visual Synthesis through ChatGPT(https://arxiv.org/abs/2412.10429)
• Keywords: extraction, diffusion, generative
• Abstract: In the burgeoning field of AI-driven image generation, the quest for precision and relevance in response to textual prompts remains paramount. This paper introduces GPTDrawer, an innovative pipeline that leverages the generative prowess of GPT-based models to enhance the visual synthesis process. Our methodology employs a novel algorithm that iteratively refines input prompts using keyword extraction, semantic analysis, and image-text congruence evaluation. By integrating ChatGPT for natural language processing and Stable Diffusion for image generation, GPTDrawer produces a batch of images that undergo successive refinement cycles, guided by cosine similarity metrics until a threshold of semantic alignment is attained. The results demonstrate a marked improvement in the fidelity of images generated in accordance with user-defined prompts, showcasing the system's ability to interpret and visualize complex semantic constructs. The implications of this work extend to various applications, from creative arts to design automation, setting a new benchmark for AI-assisted creative processes.

Title: Unsupervised Cross-Domain Regression for Fine-grained 3D Game Character Reconstruction

• Authors: Qi Wen, Xiang Wen, Hao Jiang, Siqi Yang, Bingfeng Han, Tianlei Hu, Gang Chen, Shuang Li
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.10430
• Pdf URL: https://arxiv.org/pdf/2412.10430
• Copy Paste: [[2412.10430]] Unsupervised Cross-Domain Regression for Fine-grained 3D Game Character Reconstruction(https://arxiv.org/abs/2412.10430)
• Keywords: robust
• Abstract: With the rise of the ``metaverse'' and the rapid development of games, it has become more and more critical to reconstruct characters in the virtual world faithfully. The immersive experience is one of the most central themes of the ``metaverse'', while the reducibility of the avatar is the crucial point. Meanwhile, the game is the carrier of the metaverse, in which players can freely edit the facial appearance of the game character. In this paper, we propose a simple but powerful cross-domain framework that can reconstruct fine-grained 3D game characters from single-view images in an end-to-end manner. Different from the previous methods, which do not resolve the cross-domain gap, we propose an effective regressor that can greatly reduce the discrepancy between the real-world domain and the game domain. To figure out the drawbacks of no ground truth, our unsupervised framework has accomplished the knowledge transfer of the target domain. Additionally, an innovative contrastive loss is proposed to solve the instance-wise disparity, which keeps the person-specific details of the reconstructed character. In contrast, an auxiliary 3D identity-aware extractor is activated to make the results of our model more impeccable. Then a large set of physically meaningful facial parameters is generated robustly and exquisitely. Experiments demonstrate that our method yields state-of-the-art performance in 3D game character reconstruction.

Title: Imitate Before Detect: Aligning Machine Stylistic Preference for Machine-Revised Text Detection

• Authors: Jiaqi Chen, Xiaoye Zhu, Tianyang Liu, Ying Chen, Xinhui Chen, Yiwen Yuan, Chak Tou Leong, Zuchao Li, Tang Long, Lei Zhang, Chenyu Yan, Guanghao Mei, Jie Zhang, Lefei Zhang
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.10432
• Pdf URL: https://arxiv.org/pdf/2412.10432
• Copy Paste: [[2412.10432]] Imitate Before Detect: Aligning Machine Stylistic Preference for Machine-Revised Text Detection(https://arxiv.org/abs/2412.10432)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have revolutionized text generation, making detecting machine-generated text increasingly challenging. Although past methods have achieved good performance on detecting pure machine-generated text, those detectors have poor performance on distinguishing machine-revised text (rewriting, expansion, and polishing), which can have only minor changes from its original human prompt. As the content of text may originate from human prompts, detecting machine-revised text often involves identifying distinctive machine styles, e.g., worded favored by LLMs. However, existing methods struggle to detect machine-style phrasing hidden within the content contributed by humans. We propose the "Imitate Before Detect" (ImBD) approach, which first imitates the machine-style token distribution, and then compares the distribution of the text to be tested with the machine-style distribution to determine whether the text has been machine-revised. To this end, we introduce style preference optimization (SPO), which aligns a scoring LLM model to the preference of text styles generated by machines. The aligned scoring model is then used to calculate the style-conditional probability curvature (Style-CPC), quantifying the log probability difference between the original and conditionally sampled texts for effective detection. We conduct extensive comparisons across various scenarios, encompassing text revisions by six LLMs, four distinct text domains, and three machine revision types. Compared to existing state-of-the-art methods, our method yields a 13% increase in AUC for detecting text revised by open-source LLMs, and improves performance by 5% and 19% for detecting GPT-3.5 and GPT-4o revised text, respectively. Notably, our method surpasses the commercially trained GPT-Zero with just $1,000$ samples and five minutes of SPO, demonstrating its efficiency and effectiveness.

Title: NAT-NL2GQL: A Novel Multi-Agent Framework for Translating Natural Language to Graph Query Language

• Authors: Yuanyuan Liang, Tingyu Xie, Gan Peng, Zihao Huang, Yunshi Lan, Weining Qian
• Subjects: cs.CL, cs.AI, cs.DB
• Abstract URL: https://arxiv.org/abs/2412.10434
• Pdf URL: https://arxiv.org/pdf/2412.10434
• Copy Paste: [[2412.10434]] NAT-NL2GQL: A Novel Multi-Agent Framework for Translating Natural Language to Graph Query Language(https://arxiv.org/abs/2412.10434)
• Keywords: extraction, large language model
• Abstract: The emergence of Large Language Models (LLMs) has revolutionized many fields, not only traditional natural language processing (NLP) tasks. Recently, research on applying LLMs to the database field has been booming, and as a typical non-relational database, the use of LLMs in graph database research has naturally gained significant attention. Recent efforts have increasingly focused on leveraging LLMs to translate natural language into graph query language (NL2GQL). Although some progress has been made, these methods have clear limitations, such as their reliance on streamlined processes that often overlook the potential of LLMs to autonomously plan and collaborate with other LLMs in tackling complex NL2GQL challenges. To address this gap, we propose NAT-NL2GQL, a novel multi-agent framework for translating natural language to graph query language. Specifically, our framework consists of three synergistic agents: the Preprocessor agent, the Generator agent, and the Refiner agent. The Preprocessor agent manages data processing as context, including tasks such as name entity recognition, query rewriting, path linking, and the extraction of query-related schemas. The Generator agent is a fine-tuned LLM trained on NL-GQL data, responsible for generating corresponding GQL statements based on queries and their related schemas. The Refiner agent is tasked with refining the GQL or context using error information obtained from the GQL execution results. Given the scarcity of high-quality open-source NL2GQL datasets based on nGQL syntax, we developed StockGQL, a dataset constructed from a financial market graph database. It is available at: this https URL. Experimental results on the StockGQL and SpCQL datasets reveal that our method significantly outperforms baseline approaches, highlighting its potential for advancing NL2GQL research.

Title: COEF-VQ: Cost-Efficient Video Quality Understanding through a Cascaded Multimodal LLM Framework

• Authors: Xin Dong, Sen Jia, Hongyu Xiong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10435
• Pdf URL: https://arxiv.org/pdf/2412.10435
• Copy Paste: [[2412.10435]] COEF-VQ: Cost-Efficient Video Quality Understanding through a Cascaded Multimodal LLM Framework(https://arxiv.org/abs/2412.10435)
• Keywords: large language model
• Abstract: Recently, with the emergence of recent Multimodal Large Language Model (MLLM) technology, it has become possible to exploit its video understanding capability on different classification tasks. In practice, we face the difficulty of huge requirements for GPU resource if we need to deploy MLLMs online. In this paper, we propose COEF-VQ, a novel cascaded MLLM framework for better video quality understanding on TikTok. To this end, we first propose a MLLM fusing all visual, textual and audio signals, and then develop a cascade framework with a lightweight model as pre-filtering stage and MLLM as fine-consideration stage, significantly reducing the need for GPU resource, while retaining the performance demonstrated solely by MLLM. To demonstrate the effectiveness of COEF-VQ, we deployed this new framework onto the video management platform (VMP) at TikTok, and performed a series of detailed experiments on two in-house tasks related to video quality understanding. We show that COEF-VQ leads to substantial performance gains with limit resource consumption in these two tasks.

Title: Benchmarking Federated Learning for Semantic Datasets: Federated Scene Graph Generation

• Authors: SeungBum Ha, Taehwan Lee, Jiyoun Lim, Sung Whan Yoon
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10436
• Pdf URL: https://arxiv.org/pdf/2412.10436
• Copy Paste: [[2412.10436]] Benchmarking Federated Learning for Semantic Datasets: Federated Scene Graph Generation(https://arxiv.org/abs/2412.10436)
• Keywords: privacy, robust, federate
• Abstract: Federated learning (FL) has recently garnered attention as a data-decentralized training framework that enables the learning of deep models from locally distributed samples while keeping data privacy. Built upon the framework, immense efforts have been made to establish FL benchmarks, which provide rigorous evaluation settings that control data heterogeneity across clients. Prior efforts have mainly focused on handling relatively simple classification tasks, where each sample is annotated with a one-hot label, such as MNIST, CIFAR, LEAF benchmark, etc. However, little attention has been paid to demonstrating an FL benchmark that handles complicated semantics, where each sample encompasses diverse semantic information from multiple labels, such as Panoptic Scene Graph Generation (PSG) with objects, subjects, and relations between them. Because the existing benchmark is designed to distribute data in a narrow view of a single semantic, e.g., a one-hot label, managing the complicated semantic heterogeneity across clients when formalizing FL benchmarks is non-trivial. In this paper, we propose a benchmark process to establish an FL benchmark with controllable semantic heterogeneity across clients: two key steps are i) data clustering with semantics and ii) data distributing via controllable semantic heterogeneity across clients. As a proof of concept, we first construct a federated PSG benchmark, demonstrating the efficacy of the existing PSG methods in an FL setting with controllable semantic heterogeneity of scene graphs. We also present the effectiveness of our benchmark by applying robust federated learning algorithms to data heterogeneity to show increased performance. Our code is available at this https URL.

Title: SVGFusion: Scalable Text-to-SVG Generation via Vector Space Diffusion

• Authors: Ximing Xing, Juncheng Hu, Jing Zhang, Dong Xu, Qian Yu
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10437
• Pdf URL: https://arxiv.org/pdf/2412.10437
• Copy Paste: [[2412.10437]] SVGFusion: Scalable Text-to-SVG Generation via Vector Space Diffusion(https://arxiv.org/abs/2412.10437)
• Keywords: diffusion, transformer
• Abstract: The generation of Scalable Vector Graphics (SVG) assets from textual data remains a significant challenge, largely due to the scarcity of high-quality vector datasets and the limitations in scalable vector representations required for modeling intricate graphic distributions. This work introduces SVGFusion, a Text-to-SVG model capable of scaling to real-world SVG data without reliance on a text-based discrete language model or prolonged SDS optimization. The essence of SVGFusion is to learn a continuous latent space for vector graphics with a popular Text-to-Image framework. Specifically, SVGFusion consists of two modules: a Vector-Pixel Fusion Variational Autoencoder (VP-VAE) and a Vector Space Diffusion Transformer (VS-DiT). VP-VAE takes both the SVGs and corresponding rasterizations as inputs and learns a continuous latent space, whereas VS-DiT learns to generate a latent code within this space based on the text prompt. Based on VP-VAE, a novel rendering sequence modeling strategy is proposed to enable the latent space to embed the knowledge of construction logics in SVGs. This empowers the model to achieve human-like design capabilities in vector graphics, while systematically preventing occlusion in complex graphic compositions. Moreover, our SVGFusion's ability can be continuously improved by leveraging the scalability of the VS-DiT by adding more VS-DiT blocks. A large-scale SVG dataset is collected to evaluate the effectiveness of our proposed method. Extensive experimentation has confirmed the superiority of our SVGFusion over existing SVG generation methods, achieving enhanced quality and generalizability, thereby establishing a novel framework for SVG content creation. Code, model, and data will be released at: \href{this https URL}{this https URL}

Title: Automatic Image Annotation for Mapped Features Detection

• Authors: Maxime Noizet (UTC, Heudiasyc), Philippe Xu (ENSTA Paris), Philippe Bonnifait (UTC, Heudiasyc)
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10438
• Pdf URL: https://arxiv.org/pdf/2412.10438
• Copy Paste: [[2412.10438]] Automatic Image Annotation for Mapped Features Detection(https://arxiv.org/abs/2412.10438)
• Keywords: segmentation
• Abstract: Detecting road features is a key enabler for autonomous driving and localization. For instance, a reliable detection of poles which are widespread in road environments can improve localization. Modern deep learning-based perception systems need a significant amount of annotated data. Automatic annotation avoids time-consuming and costly manual annotation. Because automatic methods are prone to errors, managing annotation uncertainty is crucial to ensure a proper learning process. Fusing multiple annotation sources on the same dataset can be an efficient way to reduce the errors. This not only improves the quality of annotations, but also improves the learning of perception models. In this paper, we consider the fusion of three automatic annotation methods in images: feature projection from a high accuracy vector map combined with a lidar, image segmentation and lidar segmentation. Our experimental results demonstrate the significant benefits of multi-modal automatic annotation for pole detection through a comparative evaluation on manually annotated images. Finally, the resulting multi-modal fusion is used to fine-tune an object detection model for pole base detection using unlabeled data, showing overall improvements achieved by enhancing network specialization. The dataset is publicly available.

Title: CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMs

• Authors: Yihan Cao, Jiazhao Zhang, Zhinan Yu, Shuzhen Liu, Zheng Qin, Qin Zou, Bo Du, Kai Xu
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.10439
• Pdf URL: https://arxiv.org/pdf/2412.10439
• Copy Paste: [[2412.10439]] CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMs(https://arxiv.org/abs/2412.10439)
• Keywords: large language model
• Abstract: Object goal navigation (ObjectNav) is a fundamental task of embodied AI that requires the agent to find a target object in unseen environments. This task is particularly challenging as it demands both perceptual and cognitive processes for effective perception and decision-making. While perception has gained significant progress powered by the rapidly developed visual foundation models, the progress on the cognitive side remains limited to either implicitly learning from massive navigation demonstrations or explicitly leveraging pre-defined heuristic rules. Inspired by neuroscientific evidence that humans consistently update their cognitive states while searching for objects in unseen environments, we present CogNav, which attempts to model this cognitive process with the help of large language models. Specifically, we model the cognitive process with a finite state machine composed of cognitive states ranging from exploration to identification. The transitions between the states are determined by a large language model based on an online built heterogeneous cognitive map containing spatial and semantic information of the scene being explored. Extensive experiments on both synthetic and real-world environments demonstrate that our cognitive modeling significantly improves ObjectNav efficiency, with human-like navigation behaviors. In an open-vocabulary and zero-shot setting, our method advances the SOTA of the HM3D benchmark from 69.3% to 87.2%. The code and data will be released.

Title: Multi-level Matching Network for Multimodal Entity Linking

• Authors: Zhiwei Hu, Víctor Gutiérrez-Basulto, Ru Li, Jeff Z. Pan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10440
• Pdf URL: https://arxiv.org/pdf/2412.10440
• Copy Paste: [[2412.10440]] Multi-level Matching Network for Multimodal Entity Linking(https://arxiv.org/abs/2412.10440)
• Keywords: extraction
• Abstract: Multimodal entity linking (MEL) aims to link ambiguous mentions within multimodal contexts to corresponding entities in a multimodal knowledge base. Most existing approaches to MEL are based on representation learning or vision-and-language pre-training mechanisms for exploring the complementary effect among multiple modalities. However, these methods suffer from two limitations. On the one hand, they overlook the possibility of considering negative samples from the same modality. On the other hand, they lack mechanisms to capture bidirectional cross-modal interaction. To address these issues, we propose a Multi-level Matching network for Multimodal Entity Linking (M3EL). Specifically, M3EL is composed of three different modules: (i) a Multimodal Feature Extraction module, which extracts modality-specific representations with a multimodal encoder and introduces an intra-modal contrastive learning sub-module to obtain better discriminative embeddings based on uni-modal differences; (ii) an Intra-modal Matching Network module, which contains two levels of matching granularity: Coarse-grained Global-to-Global and Fine-grained Global-to-Local, to achieve local and global level intra-modal interaction; (iii) a Cross-modal Matching Network module, which applies bidirectional strategies, Textual-to-Visual and Visual-to-Textual matching, to implement bidirectional cross-modal interaction. Extensive experiments conducted on WikiMEL, RichpediaMEL, and WikiDiverse datasets demonstrate the outstanding performance of M3EL when compared to the state-of-the-art baselines.

Title: Unlocking Visual Secrets: Inverting Features with Diffusion Priors for Image Reconstruction

• Authors: Sai Qian Zhang, Ziyun Li, Chuan Guo, Saeed Mahloujifar, Deeksha Dangwal, Edward Suh, Barbara De Salvo, Chiao Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10448
• Pdf URL: https://arxiv.org/pdf/2412.10448
• Copy Paste: [[2412.10448]] Unlocking Visual Secrets: Inverting Features with Diffusion Priors for Image Reconstruction(https://arxiv.org/abs/2412.10448)
• Keywords: security, privacy, diffusion
• Abstract: Inverting visual representations within deep neural networks (DNNs) presents a challenging and important problem in the field of security and privacy for deep learning. The main goal is to invert the features of an unidentified target image generated by a pre-trained DNN, aiming to reconstruct the original image. Feature inversion holds particular significance in understanding the privacy leakage inherent in contemporary split DNN execution techniques, as well as in various applications based on the extracted DNN features. In this paper, we explore the use of diffusion models, a promising technique for image synthesis, to enhance feature inversion quality. We also investigate the potential of incorporating alternative forms of prior knowledge, such as textual prompts and cross-frame temporal correlations, to further improve the quality of inverted features. Our findings reveal that diffusion models can effectively leverage hidden information from the DNN features, resulting in superior reconstruction performance compared to previous methods. This research offers valuable insights into how diffusion models can enhance privacy and security within applications that are reliant on DNN features.

Title: Analysis of Object Detection Models for Tiny Object in Satellite Imagery: A Dataset-Centric Approach

• Authors: Kailas PS, Selvakumaran R, Palani Murugan, Ramesh Kumar V, Malaya Kumar Biswal M
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10453
• Pdf URL: https://arxiv.org/pdf/2412.10453
• Copy Paste: [[2412.10453]] Analysis of Object Detection Models for Tiny Object in Satellite Imagery: A Dataset-Centric Approach(https://arxiv.org/abs/2412.10453)
• Keywords: segmentation
• Abstract: In recent years, significant advancements have been made in deep learning-based object detection algorithms, revolutionizing basic computer vision tasks, notably in object detection, tracking, and segmentation. This paper delves into the intricate domain of Small-Object-Detection (SOD) within satellite imagery, highlighting the unique challenges stemming from wide imaging ranges, object distribution, and their varying appearances in bird's-eye-view satellite images. Traditional object detection models face difficulties in detecting small objects due to limited contextual information and class imbalances. To address this, our research presents a meticulously curated dataset comprising 3000 images showcasing cars, ships, and airplanes in satellite imagery. Our study aims to provide valuable insights into small object detection in satellite imagery by empirically evaluating state-of-the-art models. Furthermore, we tackle the challenges of satellite video-based object tracking, employing the Byte Track algorithm on the SAT-MTB dataset. Through rigorous experimentation, we aim to offer a comprehensive understanding of the efficacy of state-of-the-art models in Small-Object-Detection for satellite applications. Our findings shed light on the effectiveness of these models and pave the way for future advancements in satellite imagery analysis.

Title: An Interoperable Machine Learning Pipeline for Pediatric Obesity Risk Estimation

• Authors: Hamed Fayyaz, Mehak Gupta, Alejandra Perez Ramirez, Claudine Jurkovitz, H. Timothy Bunnell, Thao-Ly T. Phan, Rahmatollah Beheshti
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10454
• Pdf URL: https://arxiv.org/pdf/2412.10454
• Copy Paste: [[2412.10454]] An Interoperable Machine Learning Pipeline for Pediatric Obesity Risk Estimation(https://arxiv.org/abs/2412.10454)
• Keywords: extraction
• Abstract: Reliable prediction of pediatric obesity can offer a valuable resource to providers, helping them engage in timely preventive interventions before the disease is established. Many efforts have been made to develop ML-based predictive models of obesity, and some studies have reported high predictive performances. However, no commonly used clinical decision support tool based on existing ML models currently exists. This study presents a novel end-to-end pipeline specifically designed for pediatric obesity prediction, which supports the entire process of data extraction, inference, and communication via an API or a user interface. While focusing only on routinely recorded data in pediatric electronic health records (EHRs), our pipeline uses a diverse expert-curated list of medical concepts to predict the 1-3 years risk of developing obesity. Furthermore, by using the Fast Healthcare Interoperability Resources (FHIR) standard in our design procedure, we specifically target facilitating low-effort integration of our pipeline with different EHR systems. In our experiments, we report the effectiveness of the predictive model as well as its alignment with the feedback from various stakeholders, including ML scientists, providers, health IT personnel, health administration representatives, and patient group representatives.

Title: Geo-LLaVA: A Large Multi-Modal Model for Solving Geometry Math Problems with Meta In-Context Learning

• Authors: Shihao Xu, Yiyang Luo, Wei Shi
• Subjects: cs.CV, cs.AI, cs.CG
• Abstract URL: https://arxiv.org/abs/2412.10455
• Pdf URL: https://arxiv.org/pdf/2412.10455
• Copy Paste: [[2412.10455]] Geo-LLaVA: A Large Multi-Modal Model for Solving Geometry Math Problems with Meta In-Context Learning(https://arxiv.org/abs/2412.10455)
• Keywords: large language model
• Abstract: Geometry mathematics problems pose significant challenges for large language models (LLMs) because they involve visual elements and spatial reasoning. Current methods primarily rely on symbolic character awareness to address these problems. Considering geometry problem solving is a relatively nascent field with limited suitable datasets and currently almost no work on solid geometry problem solving, we collect a geometry question-answer dataset by sourcing geometric data from Chinese high school education websites, referred to as GeoMath. It contains solid geometry questions and answers with accurate reasoning steps as compensation for existing plane geometry datasets. Additionally, we propose a Large Multi-modal Model (LMM) framework named Geo-LLaVA, which incorporates retrieval augmentation with supervised fine-tuning (SFT) in the training stage, called meta-training, and employs in-context learning (ICL) during inference to improve performance. Our fine-tuned model with ICL attains the state-of-the-art performance of 65.25% and 42.36% on selected questions of the GeoQA dataset and GeoMath dataset respectively with proper inference steps. Notably, our model initially endows the ability to solve solid geometry problems and supports the generation of reasonable solid geometry picture descriptions and problem-solving steps. Our research sets the stage for further exploration of LLMs in multi-modal math problem-solving, particularly in geometry math problems.

Title: Enriching Multimodal Sentiment Analysis through Textual Emotional Descriptions of Visual-Audio Content

• Authors: Sheng Wu, Xiaobao Wang, Longbiao Wang, Dongxiao He, Jianwu Dang
• Subjects: cs.CV, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.10460
• Pdf URL: https://arxiv.org/pdf/2412.10460
• Copy Paste: [[2412.10460]] Enriching Multimodal Sentiment Analysis through Textual Emotional Descriptions of Visual-Audio Content(https://arxiv.org/abs/2412.10460)
• Keywords: robust
• Abstract: Multimodal Sentiment Analysis (MSA) stands as a critical research frontier, seeking to comprehensively unravel human emotions by amalgamating text, audio, and visual data. Yet, discerning subtle emotional nuances within audio and video expressions poses a formidable challenge, particularly when emotional polarities across various segments appear similar. In this paper, our objective is to spotlight emotion-relevant attributes of audio and visual modalities to facilitate multimodal fusion in the context of nuanced emotional shifts in visual-audio scenarios. To this end, we introduce DEVA, a progressive fusion framework founded on textual sentiment descriptions aimed at accentuating emotional features of visual-audio content. DEVA employs an Emotional Description Generator (EDG) to transmute raw audio and visual data into textualized sentiment descriptions, thereby amplifying their emotional characteristics. These descriptions are then integrated with the source data to yield richer, enhanced features. Furthermore, DEVA incorporates the Text-guided Progressive Fusion Module (TPF), leveraging varying levels of text as a core modality guide. This module progressively fuses visual-audio minor modalities to alleviate disparities between text and visual-audio modalities. Experimental results on widely used sentiment analysis benchmark datasets, including MOSI, MOSEI, and CH-SIMS, underscore significant enhancements compared to state-of-the-art models. Moreover, fine-grained emotion experiments corroborate the robust sensitivity of DEVA to subtle emotional variations.

Title: CrossVIT-augmented Geospatial-Intelligence Visualization System for Tracking Economic Development Dynamics

• Authors: Yanbing Bai, Jinhua Su, Bin Qiao, Xiaoran Ma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10474
• Pdf URL: https://arxiv.org/pdf/2412.10474
• Copy Paste: [[2412.10474]] CrossVIT-augmented Geospatial-Intelligence Visualization System for Tracking Economic Development Dynamics(https://arxiv.org/abs/2412.10474)
• Keywords: transformer
• Abstract: Timely and accurate economic data is crucial for effective policymaking. Current challenges in data timeliness and spatial resolution can be addressed with advancements in multimodal sensing and distributed computing. We introduce Senseconomic, a scalable system for tracking economic dynamics via multimodal imagery and deep learning. Built on the Transformer framework, it integrates remote sensing and street view images using cross-attention, with nighttime light data as weak supervision. The system achieved an R-squared value of 0.8363 in county-level economic predictions and halved processing time to 23 minutes using distributed computing. Its user-friendly design includes a Vue3-based front end with Baidu maps for visualization and a Python-based back end automating tasks like image downloads and preprocessing. Senseconomic empowers policymakers and researchers with efficient tools for resource allocation and economic planning.

Title: Benchmarking large language models for materials synthesis: the case of atomic layer deposition

• Authors: Angel Yanguas-Gil, Matthew T. Dearing, Jeffrey W. Elam, Jessica C. Jones, Sungjoon Kim, Adnan Mohammad, Chi Thang Nguyen, Bratin Sengupta
• Subjects: cs.LG, cond-mat.mtrl-sci, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10477
• Pdf URL: https://arxiv.org/pdf/2412.10477
• Copy Paste: [[2412.10477]] Benchmarking large language models for materials synthesis: the case of atomic layer deposition(https://arxiv.org/abs/2412.10477)
• Keywords: large language model
• Abstract: In this work we introduce an open-ended question benchmark, ALDbench, to evaluate the performance of large language models (LLMs) in materials synthesis, and in particular in the field of atomic layer deposition, a thin film growth technique used in energy applications and microelectronics. Our benchmark comprises questions with a level of difficulty ranging from graduate level to domain expert current with the state of the art in the field. Human experts reviewed the questions along the criteria of difficulty and specificity, and the model responses along four different criteria: overall quality, specificity, relevance, and accuracy. We ran this benchmark on an instance of OpenAI's GPT-4o. The responses from the model received a composite quality score of 3.7 on a 1 to 5 scale, consistent with a passing grade. However, 36% of the questions received at least one below average score. An in-depth analysis of the responses identified at least five instances of suspected hallucination. Finally, we observed statistically significant correlations between the difficulty of the question and the quality of the response, the difficulty of the question and the relevance of the response, and the specificity of the question and the accuracy of the response as graded by the human experts. This emphasizes the need to evaluate LLMs across multiple criteria beyond difficulty or accuracy.

Title: Dynamic Entity-Masked Graph Diffusion Model for histopathological image Representation Learning

• Authors: Zhenfeng Zhuang, Min Cen, Yanfeng Li, Fangyu Zhou, Lequan Yu, Baptiste Magnier, Liansheng Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10482
• Pdf URL: https://arxiv.org/pdf/2412.10482
• Copy Paste: [[2412.10482]] Dynamic Entity-Masked Graph Diffusion Model for histopathological image Representation Learning(https://arxiv.org/abs/2412.10482)
• Keywords: robust, interpretability, diffusion
• Abstract: Significant disparities between the features of natural images and those inherent to histopathological images make it challenging to directly apply and transfer pre-trained models from natural images to histopathology tasks. Moreover, the frequent lack of annotations in histopathology patch images has driven researchers to explore self-supervised learning methods like mask reconstruction for learning representations from large amounts of unlabeled data. Crucially, previous mask-based efforts in self-supervised learning have often overlooked the spatial interactions among entities, which are essential for constructing accurate representations of pathological entities. To address these challenges, constructing graphs of entities is a promising approach. In addition, the diffusion reconstruction strategy has recently shown superior performance through its random intensity noise addition technique to enhance the robust learned representation. Therefore, we introduce H-MGDM, a novel self-supervised Histopathology image representation learning method through the Dynamic Entity-Masked Graph Diffusion Model. Specifically, we propose to use complementary subgraphs as latent diffusion conditions and self-supervised targets respectively during pre-training. We note that the graph can embed entities' topological relationships and enhance representation. Dynamic conditions and targets can improve pathological fine reconstruction. Our model has conducted pretraining experiments on three large histopathological datasets. The advanced predictive performance and interpretability of H-MGDM are clearly evaluated on comprehensive downstream tasks such as classification and survival analysis on six datasets. Our code will be publicly available at this https URL.

Title: A Hybrid Real-Time Framework for Efficient Fussell-Vesely Importance Evaluation Using Virtual Fault Trees and Graph Neural Networks

• Authors: Xingyu Xiao, Peng Chen
• Subjects: cs.LG, cs.LO
• Abstract URL: https://arxiv.org/abs/2412.10484
• Pdf URL: https://arxiv.org/pdf/2412.10484
• Copy Paste: [[2412.10484]] A Hybrid Real-Time Framework for Efficient Fussell-Vesely Importance Evaluation Using Virtual Fault Trees and Graph Neural Networks(https://arxiv.org/abs/2412.10484)
• Keywords: robust
• Abstract: The Fussell-Vesely Importance (FV) reflects the potential impact of a basic event on system failure, and is crucial for ensuring system reliability. However, traditional methods for calculating FV importance are complex and time-consuming, requiring the construction of fault trees and the calculation of minimal cut set. To address these limitations, this study proposes a hybrid real-time framework to evaluate the FV importance of basic events. Our framework combines expert knowledge with a data-driven model. First, we use Interpretive Structural Modeling (ISM) to build a virtual fault tree that captures the relationships between basic events. Unlike traditional fault trees, which include intermediate events, our virtual fault tree consists solely of basic events, reducing its complexity and space requirements. Additionally, our virtual fault tree considers the dependencies between basic events rather than assuming their independence, as is typically done in traditional fault trees. We then feed both the event relationships and relevant data into a graph neural network (GNN). This approach enables a rapid, data-driven calculation of FV importance, significantly reducing processing time and quickly identifying critical events, thus providing robust decision support for risk control. Results demonstrate that our model performs well in terms of MSE, RMSE, MAE, and R2, reducing computational energy consumption and offering real-time, risk-informed decision support for complex systems.

Title: SVGBuilder: Component-Based Colored SVG Generation with Text-Guided Autoregressive Transformers

• Authors: Zehao Chen, Rong Pan
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.10488
• Pdf URL: https://arxiv.org/pdf/2412.10488
• Copy Paste: [[2412.10488]] SVGBuilder: Component-Based Colored SVG Generation with Text-Guided Autoregressive Transformers(https://arxiv.org/abs/2412.10488)
• Keywords: transformer
• Abstract: Scalable Vector Graphics (SVG) are essential XML-based formats for versatile graphics, offering resolution independence and scalability. Unlike raster images, SVGs use geometric shapes and support interactivity, animation, and manipulation via CSS and JavaScript. Current SVG generation methods face challenges related to high computational costs and complexity. In contrast, human designers use component-based tools for efficient SVG creation. Inspired by this, SVGBuilder introduces a component-based, autoregressive model for generating high-quality colored SVGs from textual input. It significantly reduces computational overhead and improves efficiency compared to traditional methods. Our model generates SVGs up to 604 times faster than optimization-based approaches. To address the limitations of existing SVG datasets and support our research, we introduce ColorSVG-100K, the first large-scale dataset of colored SVGs, comprising 100,000 graphics. This dataset fills the gap in color information for SVG generation models and enhances diversity in model training. Evaluation against state-of-the-art models demonstrates SVGBuilder's superior performance in practical applications, highlighting its efficiency and quality in generating complex SVG graphics.

Title: CognitionCapturer: Decoding Visual Stimuli From Human EEG Signal With Multimodal Information

• Authors: Kaifan Zhang, Lihuo He, Xin Jiang, Wen Lu, Di Wang, Xinbo Gao
• Subjects: cs.CV, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2412.10489
• Pdf URL: https://arxiv.org/pdf/2412.10489
• Copy Paste: [[2412.10489]] CognitionCapturer: Decoding Visual Stimuli From Human EEG Signal With Multimodal Information(https://arxiv.org/abs/2412.10489)
• Keywords: diffusion, generative
• Abstract: Electroencephalogram (EEG) signals have attracted significant attention from researchers due to their non-invasive nature and high temporal sensitivity in decoding visual stimuli. However, most recent studies have focused solely on the relationship between EEG and image data pairs, neglecting the valuable ``beyond-image-modality" information embedded in EEG signals. This results in the loss of critical multimodal information in EEG. To address this limitation, we propose CognitionCapturer, a unified framework that fully leverages multimodal data to represent EEG signals. Specifically, CognitionCapturer trains Modality Expert Encoders for each modality to extract cross-modal information from the EEG modality. Then, it introduces a diffusion prior to map the EEG embedding space to the CLIP embedding space, followed by using a pretrained generative model, the proposed framework can reconstruct visual stimuli with high semantic and structural fidelity. Notably, the framework does not require any fine-tuning of the generative models and can be extended to incorporate more modalities. Through extensive experiments, we demonstrate that CognitionCapturer outperforms state-of-the-art methods both qualitatively and quantitatively. Code: this https URL.

Title: QSM-RimDS: A highly sensitive paramagnetic rim lesion detection and segmentation tool for multiple sclerosis lesions

• Authors: Ha Luu, Mert Sisman, Ilhami Kovanlikaya, Tam Vu, Pascal Spincemaille, Yi Wang, Francesca Bagnato, Susan Gauthier, Thanh Nguyen
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.10492
• Pdf URL: https://arxiv.org/pdf/2412.10492
• Copy Paste: [[2412.10492]] QSM-RimDS: A highly sensitive paramagnetic rim lesion detection and segmentation tool for multiple sclerosis lesions(https://arxiv.org/abs/2412.10492)
• Keywords: segmentation
• Abstract: Paramagnetic rim lesions (PRLs) are imaging biomarker of the innate immune response in MS lesions. QSM-RimNet, a state-of-the-art tool for PRLs detection on QSM, can identify PRLs but requires precise QSM lesion mask and does not provide rim segmentation. Therefore, the aims of this study are to develop QSM-RimDS algorithm to detect PRLs using the readily available FLAIR lesion mask and to provide rim segmentation for microglial quantification. QSM-RimDS, a deep-learning based tool for joint PRL rim segmentation and PRL detection has been developed. QSM-RimDS has obtained state-of-the art performance in PRL detection and therefore has the potential to be used in clinical practice as a tool to assist human readers for the time-consuming PRL detection and segmentation task. QSM-RimDS is made publicly available [this https URL]

Title: SafetyDPO: Scalable Safety Alignment for Text-to-Image Generation

• Authors: Runtao Liu, Chen I Chieh, Jindong Gu, Jipeng Zhang, Renjie Pi, Qifeng Chen, Philip Torr, Ashkan Khakzar, Fabio Pizzati
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10493
• Pdf URL: https://arxiv.org/pdf/2412.10493
• Copy Paste: [[2412.10493]] SafetyDPO: Scalable Safety Alignment for Text-to-Image Generation(https://arxiv.org/abs/2412.10493)
• Keywords: generative
• Abstract: Text-to-image (T2I) models have become widespread, but their limited safety guardrails expose end users to harmful content and potentially allow for model misuse. Current safety measures are typically limited to text-based filtering or concept removal strategies, able to remove just a few concepts from the model's generative capabilities. In this work, we introduce SafetyDPO, a method for safety alignment of T2I models through Direct Preference Optimization (DPO). We enable the application of DPO for safety purposes in T2I models by synthetically generating a dataset of harmful and safe image-text pairs, which we call CoProV2. Using a custom DPO strategy and this dataset, we train safety experts, in the form of low-rank adaptation (LoRA) matrices, able to guide the generation process away from specific safety-related concepts. Then, we merge the experts into a single LoRA using a novel merging strategy for optimal scaling performance. This expert-based approach enables scalability, allowing us to remove 7 times more harmful concepts from T2I models compared to baselines. SafetyDPO consistently outperforms the state-of-the-art on many benchmarks and establishes new practices for safety alignment in T2I networks. Code and data will be shared at this https URL.

Title: SnapGen-V: Generating a Five-Second Video within Five Seconds on a Mobile Device

• Authors: Yushu Wu, Zhixing Zhang, Yanyu Li, Yanwu Xu, Anil Kag, Yang Sui, Huseyin Coskun, Ke Ma, Aleksei Lebedev, Ju Hu, Dimitris Metaxas, Yanzhi Wang, Sergey Tulyakov, Jian Ren
• Subjects: cs.CV, cs.AI, cs.LG, cs.PF
• Abstract URL: https://arxiv.org/abs/2412.10494
• Pdf URL: https://arxiv.org/pdf/2412.10494
• Copy Paste: [[2412.10494]] SnapGen-V: Generating a Five-Second Video within Five Seconds on a Mobile Device(https://arxiv.org/abs/2412.10494)
• Keywords: diffusion
• Abstract: We have witnessed the unprecedented success of diffusion-based video generation over the past year. Recently proposed models from the community have wielded the power to generate cinematic and high-resolution videos with smooth motions from arbitrary input prompts. However, as a supertask of image generation, video generation models require more computation and are thus hosted mostly on cloud servers, limiting broader adoption among content creators. In this work, we propose a comprehensive acceleration framework to bring the power of the large-scale video diffusion model to the hands of edge users. From the network architecture scope, we initialize from a compact image backbone and search out the design and arrangement of temporal layers to maximize hardware efficiency. In addition, we propose a dedicated adversarial fine-tuning algorithm for our efficient model and reduce the denoising steps to 4. Our model, with only 0.6B parameters, can generate a 5-second video on an iPhone 16 PM within 5 seconds. Compared to server-side models that take minutes on powerful GPUs to generate a single video, we accelerate the generation by magnitudes while delivering on-par quality.

Title: DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts

• Authors: Tobias Braun, Mark Rothermel, Marcus Rohrbach, Anna Rohrbach
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10510
• Pdf URL: https://arxiv.org/pdf/2412.10510
• Copy Paste: [[2412.10510]] DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts(https://arxiv.org/abs/2412.10510)
• Keywords: robust, explainability
• Abstract: The proliferation of disinformation presents a growing threat to societal trust and democracy, necessitating robust and scalable Fact-Checking systems. In this work, we present Dynamic Evidence-based FAct-checking with Multimodal Experts (DEFAME), a modular, zero-shot MLLM pipeline for open-domain, text-image claim verification. DEFAME frames the problem of fact-checking as a six-stage process, dynamically deciding about the usage of external tools for the retrieval of textual and visual evidence. In addition to the claim's veracity, DEFAME returns a justification accompanied by a comprehensive, multimodal fact-checking report. While most alternatives either focus on sub-tasks of fact-checking, lack explainability or are limited to text-only inputs, DEFAME solves the problem of fact-checking end-to-end, including claims with images or those that require visual evidence. Evaluation on the popular benchmarks VERITE, AVeriTeC, and MOCHEG shows that DEFAME surpasses all previous methods, establishing it as the new state-of-the-art fact-checking system.

Title: Automated Image Captioning with CNNs and Transformers

• Authors: Joshua Adrian Cahyono, Jeremy Nathan Jusuf
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10511
• Pdf URL: https://arxiv.org/pdf/2412.10511
• Copy Paste: [[2412.10511]] Automated Image Captioning with CNNs and Transformers(https://arxiv.org/abs/2412.10511)
• Keywords: transformer
• Abstract: This project aims to create an automated image captioning system that generates natural language descriptions for input images by integrating techniques from computer vision and natural language processing. We employ various different techniques, ranging from CNN-RNN to the more advanced transformer-based techniques. Training is carried out on image datasets paired with descriptive captions, and model performance will be evaluated using established metrics such as BLEU, METEOR, and CIDEr. The project will also involve experimentation with advanced attention mechanisms, comparisons of different architectural choices, and hyperparameter optimization to refine captioning accuracy and overall system effectiveness.

Title: Differentially Private Multi-Sampling from Distributions

• Authors: Albert Cheu, Debanuj Nayak
• Subjects: cs.CR, cs.DS, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.10512
• Pdf URL: https://arxiv.org/pdf/2412.10512
• Copy Paste: [[2412.10512]] Differentially Private Multi-Sampling from Distributions(https://arxiv.org/abs/2412.10512)
• Keywords: privacy
• Abstract: Many algorithms have been developed to estimate probability distributions subject to differential privacy (DP): such an algorithm takes as input independent samples from a distribution and estimates the density function in a way that is insensitive to any one sample. A recent line of work, initiated by Raskhodnikova et al. (Neurips '21), explores a weaker objective: a differentially private algorithm that approximates a single sample from the distribution. Raskhodnikova et al. studied the sample complexity of DP \emph{single-sampling} i.e., the minimum number of samples needed to perform this task. They showed that the sample complexity of DP single-sampling is less than the sample complexity of DP learning for certain distribution classes. We define two variants of \emph{multi-sampling}, where the goal is to privately approximate $m>1$ samples. This better models the realistic scenario where synthetic data is needed for exploratory data analysis. A baseline solution to \emph{multi-sampling} is to invoke a single-sampling algorithm $m$ times on independently drawn datasets of samples. When the data comes from a finite domain, we improve over the baseline by a factor of $m$ in the sample complexity. When the data comes from a Gaussian, Ghazi et al. (Neurips '23) show that \emph{single-sampling} can be performed under approximate differential privacy; we show it is possible to \emph{single- and multi-sample Gaussians with known covariance subject to pure DP}. Our solution uses a variant of the Laplace mechanism that is of independent interest. We also give sample complexity lower bounds, one for strong multi-sampling of finite distributions and another for weak multi-sampling of bounded-covariance Gaussians.

Title: RowDetr: End-to-End Row Detection Using Polynomials

• Authors: Rahul Harsha Cheppally, Ajay Sharda
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.10525
• Pdf URL: https://arxiv.org/pdf/2412.10525
• Copy Paste: [[2412.10525]] RowDetr: End-to-End Row Detection Using Polynomials(https://arxiv.org/abs/2412.10525)
• Keywords: robust
• Abstract: Crop row detection has garnered significant interest due to its critical role in enabling navigation in GPS-denied environments, such as under-canopy agricultural settings. To address this challenge, we propose RowDetr, an end-to-end neural network that utilizes smooth polynomial functions to delineate crop boundaries in image space. A novel energy-based loss function, PolyOptLoss, is introduced to enhance learning robustness, even with noisy labels. The proposed model demonstrates a 3% improvement over Agronav in key performance metrics while being six times faster, making it well-suited for real-time applications. Additionally, metrics from lane detection studies were adapted to comprehensively evaluate the system, showcasing its accuracy and adaptability in various scenarios.

Title: Towards Using Machine Learning to Generatively Simulate EV Charging in Urban Areas

• Authors: Marek Miltner, Jakub Zíka, Daniel Vašata, Artem Bryksa, Magda Friedjungová, Ondřej Štogl, Ram Rajagopal, Oldřich Starý
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10531
• Pdf URL: https://arxiv.org/pdf/2412.10531
• Copy Paste: [[2412.10531]] Towards Using Machine Learning to Generatively Simulate EV Charging in Urban Areas(https://arxiv.org/abs/2412.10531)
• Keywords: generative
• Abstract: This study addresses the challenge of predicting electric vehicle (EV) charging profiles in urban locations with limited data. Utilizing a neural network architecture, we aim to uncover latent charging profiles influenced by spatio-temporal factors. Our model focuses on peak power demand and daily load shapes, providing insights into charging behavior. Our results indicate significant impacts from the type of Basic Administrative Units on predicted load curves, which contributes to the understanding and optimization of EV charging infrastructure in urban settings and allows Distribution System Operators (DSO) to more efficiently plan EV charging infrastructure expansion.

Title: On Adversarial Robustness and Out-of-Distribution Robustness of Large Language Models

• Authors: April Yang, Jordan Tab, Parth Shah, Paul Kotchavong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10535
• Pdf URL: https://arxiv.org/pdf/2412.10535
• Copy Paste: [[2412.10535]] On Adversarial Robustness and Out-of-Distribution Robustness of Large Language Models(https://arxiv.org/abs/2412.10535)
• Keywords: robust, large language model
• Abstract: The increasing reliance on large language models (LLMs) for diverse applications necessitates a thorough understanding of their robustness to adversarial perturbations and out-of-distribution (OOD) inputs. In this study, we investigate the correlation between adversarial robustness and OOD robustness in LLMs, addressing a critical gap in robustness evaluation. By applying methods originally designed to improve one robustness type across both contexts, we analyze their performance on adversarial and out-of-distribution benchmark datasets. The input of the model consists of text samples, with the output prediction evaluated in terms of accuracy, precision, recall, and F1 scores in various natural language inference tasks. Our findings highlight nuanced interactions between adversarial robustness and OOD robustness, with results indicating limited transferability between the two robustness types. Through targeted ablations, we evaluate how these correlations evolve with different model sizes and architectures, uncovering model-specific trends: smaller models like LLaMA2-7b exhibit neutral correlations, larger models like LLaMA2-13b show negative correlations, and Mixtral demonstrates positive correlations, potentially due to domain-specific alignment. These results underscore the importance of hybrid robustness frameworks that integrate adversarial and OOD strategies tailored to specific models and domains. Further research is needed to evaluate these interactions across larger models and varied architectures, offering a pathway to more reliable and generalizable LLMs.

Title: ExclaveFL: Providing Transparency to Federated Learning using Exclaves

• Authors: Jinnan Guo, Kapil Vaswani, Andrew Paverd, Peter Pietzuch
• Subjects: cs.CR, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10537
• Pdf URL: https://arxiv.org/pdf/2412.10537
• Copy Paste: [[2412.10537]] ExclaveFL: Providing Transparency to Federated Learning using Exclaves(https://arxiv.org/abs/2412.10537)
• Keywords: security, privacy, protect, attack, federate
• Abstract: In federated learning (FL), data providers jointly train a model without disclosing their training data. Despite its privacy benefits, a malicious data provider can simply deviate from the correct training protocol without being detected, thus attacking the trained model. While current solutions have explored the use of trusted execution environment (TEEs) to combat such attacks, there is a mismatch with the security needs of FL: TEEs offer confidentiality guarantees, which are unnecessary for FL and make them vulnerable to side-channel attacks, and focus on coarse-grained attestation, which does not capture the execution of FL training. We describe ExclaveFL, an FL platform that achieves end-to-end transparency and integrity for detecting attacks. ExclaveFL achieves this by employing a new hardware security abstraction, exclaves, which focus on integrity-only guarantees. ExclaveFL uses exclaves to protect the execution of FL tasks, while generating signed statements containing fine-grained, hardware-based attestation reports of task execution at runtime. ExclaveFL then enables auditing using these statements to construct an attested dataflow graph and then check that the FL training jobs satisfies claims, such as the absence of attacks. Our experiments show that ExclaveFL introduces a less than 9% overhead while detecting a wide-range of attacks.

Title: Higher Order Transformers: Enhancing Stock Movement Prediction On Multimodal Time-Series Data

• Authors: Soroush Omranpour, Guillaume Rabusseau, Reihaneh Rabbany
• Subjects: cs.LG, q-fin.ST
• Abstract URL: https://arxiv.org/abs/2412.10540
• Pdf URL: https://arxiv.org/pdf/2412.10540
• Copy Paste: [[2412.10540]] Higher Order Transformers: Enhancing Stock Movement Prediction On Multimodal Time-Series Data(https://arxiv.org/abs/2412.10540)
• Keywords: transformer
• Abstract: In this paper, we tackle the challenge of predicting stock movements in financial markets by introducing Higher Order Transformers, a novel architecture designed for processing multivariate time-series data. We extend the self-attention mechanism and the transformer architecture to a higher order, effectively capturing complex market dynamics across time and variables. To manage computational complexity, we propose a low-rank approximation of the potentially large attention tensor using tensor decomposition and employ kernel attention, reducing complexity to linear with respect to the data size. Additionally, we present an encoder-decoder model that integrates technical and fundamental analysis, utilizing multimodal signals from historical prices and related tweets. Our experiments on the Stocknet dataset demonstrate the effectiveness of our method, highlighting its potential for enhancing stock movement prediction in financial markets.

Title: RAGServe: Fast Quality-Aware RAG Systems with Configuration Adaptation

• Authors: Siddhant Ray, Rui Pan, Zhuohan Gu, Kuntai Du, Ganesh Ananthanarayanan, Ravi Netravali, Junchen Jiang
• Subjects: cs.LG, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.10543
• Pdf URL: https://arxiv.org/pdf/2412.10543
• Copy Paste: [[2412.10543]] RAGServe: Fast Quality-Aware RAG Systems with Configuration Adaptation(https://arxiv.org/abs/2412.10543)
• Keywords: large language model
• Abstract: RAG (Retrieval Augmented Generation) allows LLMs (large language models) to generate better responses with external knowledge, but using more external knowledge often improves generation quality at the expense of response delay. Prior work either reduces the response delay (through better scheduling of RAG queries) or strives to maximize quality (which involves tuning the RAG workflow), but they fall short in optimizing the tradeoff between the delay and quality of RAG responses. This paper presents RAGServe, the first RAG system that jointly schedules queries and adapts the key RAG configurations of each query, such as the number of retrieved text chunks and synthesis methods, in order to balance quality optimization and response delay reduction. Using 4 popular RAG-QA datasets, we show that compared with the state-of-the-art RAG optimization schemes, RAGServe reduces the generation latency by $1.64-2.54\times$ without sacrificing generation quality.

Title: Edge AI-based Radio Frequency Fingerprinting for IoT Networks

• Authors: Ahmed Mohamed Hussain, Nada Abughanam, Panos Papadimitratos
• Subjects: cs.LG, cs.AI, cs.CR, cs.NI, eess.SP
• Abstract URL: https://arxiv.org/abs/2412.10553
• Pdf URL: https://arxiv.org/pdf/2412.10553
• Copy Paste: [[2412.10553]] Edge AI-based Radio Frequency Fingerprinting for IoT Networks(https://arxiv.org/abs/2412.10553)
• Keywords: security, transformer
• Abstract: The deployment of the Internet of Things (IoT) in smart cities and critical infrastructure has enhanced connectivity and real-time data exchange but introduced significant security challenges. While effective, cryptography can often be resource-intensive for small-footprint resource-constrained (i.e., IoT) devices. Radio Frequency Fingerprinting (RFF) offers a promising authentication alternative by using unique RF signal characteristics for device identification at the Physical (PHY)-layer, without resorting to cryptographic solutions. The challenge is two-fold: how to deploy such RFF in a large scale and for resource-constrained environments. Edge computing, processing data closer to its source, i.e., the wireless device, enables faster decision-making, reducing reliance on centralized cloud servers. Considering a modest edge device, we introduce two truly lightweight Edge AI-based RFF schemes tailored for resource-constrained devices. We implement two Deep Learning models, namely a Convolution Neural Network and a Transformer-Encoder, to extract complex features from the IQ samples, forming device-specific RF fingerprints. We convert the models to TensorFlow Lite and evaluate them on a Raspberry Pi, demonstrating the practicality of Edge deployment. Evaluations demonstrate the Transformer-Encoder outperforms the CNN in identifying unique transmitter features, achieving high accuracy (> 0.95) and ROC-AUC scores (> 0.90) while maintaining a compact model size of 73KB, appropriate for resource-constrained devices.

Title: Too Big to Fool: Resisting Deception in Language Models

• Authors: Mohammad Reza Samsami, Mats Leon Richter, Juan Rodriguez, Megh Thakkar, Sarath Chandar, Maxime Gasse
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10558
• Pdf URL: https://arxiv.org/pdf/2412.10558
• Copy Paste: [[2412.10558]] Too Big to Fool: Resisting Deception in Language Models(https://arxiv.org/abs/2412.10558)
• Keywords: large language model
• Abstract: Large language models must balance their weight-encoded knowledge with in-context information from prompts to generate accurate responses. This paper investigates this interplay by analyzing how models of varying capacities within the same family handle intentionally misleading in-context information. Our experiments demonstrate that larger models exhibit higher resilience to deceptive prompts, showcasing an advanced ability to interpret and integrate prompt information with their internal knowledge. Furthermore, we find that larger models outperform smaller ones in following legitimate instructions, indicating that their resilience is not due to disregarding in-context information. We also show that this phenomenon is likely not a result of memorization but stems from the models' ability to better leverage implicit task-relevant information from the prompt alongside their internally stored knowledge.

Title: Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers

• Authors: Dong Hoon Lee, Seunghoon Hong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10569
• Pdf URL: https://arxiv.org/pdf/2412.10569
• Copy Paste: [[2412.10569]] Learning to Merge Tokens via Decoupled Embedding for Efficient Vision Transformers(https://arxiv.org/abs/2412.10569)
• Keywords: transformer, segmentation
• Abstract: Recent token reduction methods for Vision Transformers (ViTs) incorporate token merging, which measures the similarities between token embeddings and combines the most similar pairs. However, their merging policies are directly dependent on intermediate features in ViTs, which prevents exploiting features tailored for merging and requires end-to-end training to improve token merging. In this paper, we propose Decoupled Token Embedding for Merging (DTEM) that enhances token merging through a decoupled embedding learned via a continuously relaxed token merging process. Our method introduces a lightweight embedding module decoupled from the ViT forward pass to extract dedicated features for token merging, thereby addressing the restriction from using intermediate features. The continuously relaxed token merging, applied during training, enables us to learn the decoupled embeddings in a differentiable manner. Thanks to the decoupled structure, our method can be seamlessly integrated into existing ViT backbones and trained either modularly by learning only the decoupled embeddings or end-to-end by fine-tuning. We demonstrate the applicability of DTEM on various tasks, including classification, captioning, and segmentation, with consistent improvement in token merging. Especially in the ImageNet-1k classification, DTEM achieves a 37.2% reduction in FLOPs while maintaining a top-1 accuracy of 79.85% with DeiT-small. Code is available at \href{this https URL}{link}.

Title: Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems

• Authors: Rishiraj Saha Roy, Joel Schlotthauer, Chris Hinze, Andreas Foltyn, Luzian Hahn, Fabian Kuech
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.10571
• Pdf URL: https://arxiv.org/pdf/2412.10571
• Copy Paste: [[2412.10571]] Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems(https://arxiv.org/abs/2412.10571)
• Keywords: large language model
• Abstract: Retrieval Augmented Generation (RAG) works as a backbone for interacting with an enterprise's own data via Conversational Question Answering (ConvQA). In a RAG system, a retriever fetches passages from a collection in response to a question, which are then included in the prompt of a large language model (LLM) for generating a natural language (NL) answer. However, several RAG systems today suffer from two shortcomings: (i) retrieved passages usually contain their raw text and lack appropriate document context, negatively impacting both retrieval and answering quality; and (ii) attribution strategies that explain answer generation usually rely only on similarity between the answer and the retrieved passages, thereby only generating plausible but not causal explanations. In this work, we demonstrate RAGONITE, a RAG system that remedies the above concerns by: (i) contextualizing evidence with source metadata and surrounding text; and (ii) computing counterfactual attribution, a causal explanation approach where the contribution of an evidence to an answer is determined by the similarity of the original response to the answer obtained by removing that evidence. To evaluate our proposals, we release a new benchmark ConfQuestions, with 300 hand-created conversational questions, each in English and German, coupled with ground truth URLs, completed questions, and answers from 215 public Confluence pages, that are typical of enterprise wiki spaces with heterogeneous elements. Experiments with RAGONITE on ConfQuestions show the viability of our ideas: contextualization improves RAG performance, and counterfactual attribution is effective at explaining RAG answers.

Title: ExeChecker: Where Did I Go Wrong?

• Authors: Yiwen Gu, Mahir Patel, Margrit Betke
• Subjects: cs.CV, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10573
• Pdf URL: https://arxiv.org/pdf/2412.10573
• Copy Paste: [[2412.10573]] ExeChecker: Where Did I Go Wrong?(https://arxiv.org/abs/2412.10573)
• Keywords: transformer
• Abstract: In this paper, we present a contrastive learning based framework, ExeChecker, for the interpretation of rehabilitation exercises. Our work builds upon state-of-the-art advances in the area of human pose estimation, graph-attention neural networks, and transformer interpretablity. The downstream task is to assist rehabilitation by providing informative feedback to users while they are performing prescribed exercises. We utilize a contrastive learning strategy during training. Given a tuple of correctly and incorrectly executed exercises, our model is able to identify and highlight those joints that are involved in an incorrect movement and thus require the user's attention. We collected an in-house dataset, ExeCheck, with paired recordings of both correct and incorrect execution of exercises. In our experiments, we tested our method on this dataset as well as the UI-PRMD dataset and found ExeCheck outperformed the baseline method using pairwise sequence alignment in identifying joints of physical relevance in rehabilitation exercises.

Title: Who's the (Multi-)Fairest of Them \textsc{All}: Rethinking Interpolation-Based Data Augmentation Through the Lens of Multicalibration

• Authors: Karina Halevy, Karly Hou, Charumathi Badrinath
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.10575
• Pdf URL: https://arxiv.org/pdf/2412.10575
• Copy Paste: [[2412.10575]] Who's the (Multi-)Fairest of Them \textsc{All}: Rethinking Interpolation-Based Data Augmentation Through the Lens of Multicalibration(https://arxiv.org/abs/2412.10575)
• Keywords: fair
• Abstract: Data augmentation methods, especially SoTA interpolation-based methods such as Fair Mixup, have been widely shown to increase model fairness. However, this fairness is evaluated on metrics that do not capture model uncertainty and on datasets with only one, relatively large, minority group. As a remedy, multicalibration has been introduced to measure fairness while accommodating uncertainty and accounting for multiple minority groups. However, existing methods of improving multicalibration involve reducing initial training data to create a holdout set for post-processing, which is not ideal when minority training data is already sparse. This paper uses multicalibration to more rigorously examine data augmentation for classification fairness. We stress-test four versions of Fair Mixup on two structured data classification problems with up to 81 marginalized groups, evaluating multicalibration violations and balanced accuracy. We find that on nearly every experiment, Fair Mixup \textit{worsens} baseline performance and fairness, but the simple vanilla Mixup \textit{outperforms} both Fair Mixup and the baseline, especially when calibrating on small groups. \textit{Combining} vanilla Mixup with multicalibration post-processing, which enforces multicalibration through post-processing on a holdout set, further increases fairness.

Title: WHAT-IF: Exploring Branching Narratives by Meta-Prompting Large Language Models

• Authors: Runsheng "Anson" Huang, Lara J. Martin, Chris Callison-Burch
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10582
• Pdf URL: https://arxiv.org/pdf/2412.10582
• Copy Paste: [[2412.10582]] WHAT-IF: Exploring Branching Narratives by Meta-Prompting Large Language Models(https://arxiv.org/abs/2412.10582)
• Keywords: large language model
• Abstract: WHAT-IF -- Writing a Hero's Alternate Timeline through Interactive Fiction -- is a system that uses zero-shot meta-prompting to create branching narratives from a prewritten story. Played as an interactive fiction (IF) game, WHAT-IF lets the player choose between decisions that the large language model (LLM) GPT-4 generates as possible branches in the story. Starting with an existing linear plot as input, a branch is created at each key decision taken by the main character. By meta-prompting the LLM to consider the major plot points from the story, the system produces coherent and well-structured alternate storylines. WHAT-IF stores the branching plot tree in a graph which helps it to both keep track of the story for prompting and maintain the structure for the final IF system. A video demo of our system can be found here: this https URL.

Title: PanSR: An Object-Centric Mask Transformer for Panoptic Segmentation

• Authors: Lojze Žust, Matej Kristan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10589
• Pdf URL: https://arxiv.org/pdf/2412.10589
• Copy Paste: [[2412.10589]] PanSR: An Object-Centric Mask Transformer for Panoptic Segmentation(https://arxiv.org/abs/2412.10589)
• Keywords: transformer, segmentation
• Abstract: Panoptic segmentation is a fundamental task in computer vision and a crucial component for perception in autonomous vehicles. Recent mask-transformer-based methods achieve impressive performance on standard benchmarks but face significant challenges with small objects, crowded scenes and scenes exhibiting a wide range of object scales. We identify several fundamental shortcomings of the current approaches: (i) the query proposal generation process is biased towards larger objects, resulting in missed smaller objects, (ii) initially well-localized queries may drift to other objects, resulting in missed detections, (iii) spatially well-separated instances may be merged into a single mask causing inconsistent and false scene interpretations. To address these issues, we rethink the individual components of the network and its supervision, and propose a novel method for panoptic segmentation PanSR. PanSR effectively mitigates instance merging, enhances small-object detection and increases performance in crowded scenes, delivering a notable +3.4 PQ improvement over state-of-the-art on the challenging LaRS benchmark, while reaching state-of-the-art performance on Cityscapes. The code and models will be publicly available at this https URL.

Title: Towards Unified Benchmark and Models for Multi-Modal Perceptual Metrics

• Authors: Sara Ghazanfari, Siddharth Garg, Nicolas Flammarion, Prashanth Krishnamurthy, Farshad Khorrami, Francesco Croce
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10594
• Pdf URL: https://arxiv.org/pdf/2412.10594
• Copy Paste: [[2412.10594]] Towards Unified Benchmark and Models for Multi-Modal Perceptual Metrics(https://arxiv.org/abs/2412.10594)
• Keywords: robust, generative
• Abstract: Human perception of similarity across uni- and multimodal inputs is highly complex, making it challenging to develop automated metrics that accurately mimic it. General purpose vision-language models, such as CLIP and large multi-modal models (LMMs), can be applied as zero-shot perceptual metrics, and several recent works have developed models specialized in narrow perceptual tasks. However, the extent to which existing perceptual metrics align with human perception remains unclear. To investigate this question, we introduce UniSim-Bench, a benchmark encompassing 7 multi-modal perceptual similarity tasks, with a total of 25 datasets. Our evaluation reveals that while general-purpose models perform reasonably well on average, they often lag behind specialized models on individual tasks. Conversely, metrics fine-tuned for specific tasks fail to generalize well to unseen, though related, tasks. As a first step towards a unified multi-task perceptual similarity metric, we fine-tune both encoder-based and generative vision-language models on a subset of the UniSim-Bench tasks. This approach yields the highest average performance, and in some cases, even surpasses taskspecific models. Nevertheless, these models still struggle with generalization to unseen tasks, highlighting the ongoing challenge of learning a robust, unified perceptual similarity metric capable of capturing the human notion of similarity. The code and models are available at this https URL.

Title: Err on the Side of Texture: Texture Bias on Real Data

• Authors: Blaine Hoak, Ryan Sheatsley, Patrick McDaniel
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.10597
• Pdf URL: https://arxiv.org/pdf/2412.10597
• Copy Paste: [[2412.10597]] Err on the Side of Texture: Texture Bias on Real Data(https://arxiv.org/abs/2412.10597)
• Keywords: robust
• Abstract: Bias significantly undermines both the accuracy and trustworthiness of machine learning models. To date, one of the strongest biases observed in image classification models is texture bias-where models overly rely on texture information rather than shape information. Yet, existing approaches for measuring and mitigating texture bias have not been able to capture how textures impact model robustness in real-world settings. In this work, we introduce the Texture Association Value (TAV), a novel metric that quantifies how strongly models rely on the presence of specific textures when classifying objects. Leveraging TAV, we demonstrate that model accuracy and robustness are heavily influenced by texture. Our results show that texture bias explains the existence of natural adversarial examples, where over 90% of these samples contain textures that are misaligned with the learned texture of their true label, resulting in confident mispredictions.

Title: EvalGIM: A Library for Evaluating Generative Image Models

• Authors: Melissa Hall, Oscar Mañas, Reyhane Askari, Mark Ibrahim, Candace Ross, Pietro Astolfi, Tariq Berrada Ifriqi, Marton Havasi, Yohann Benchetrit, Karen Ullrich, Carolina Braga, Abhishek Charnalia, Maeve Ryan, Mike Rabbat, Michal Drozdzal, Jakob Verbeek, Adriana Romero Soriano
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10604
• Pdf URL: https://arxiv.org/pdf/2412.10604
• Copy Paste: [[2412.10604]] EvalGIM: A Library for Evaluating Generative Image Models(https://arxiv.org/abs/2412.10604)
• Keywords: robust, generative
• Abstract: As the use of text-to-image generative models increases, so does the adoption of automatic benchmarking methods used in their evaluation. However, while metrics and datasets abound, there are few unified benchmarking libraries that provide a framework for performing evaluations across many datasets and metrics. Furthermore, the rapid introduction of increasingly robust benchmarking methods requires that evaluation libraries remain flexible to new datasets and metrics. Finally, there remains a gap in synthesizing evaluations in order to deliver actionable takeaways about model performance. To enable unified, flexible, and actionable evaluations, we introduce EvalGIM (pronounced ''EvalGym''), a library for evaluating generative image models. EvalGIM contains broad support for datasets and metrics used to measure quality, diversity, and consistency of text-to-image generative models. In addition, EvalGIM is designed with flexibility for user customization as a top priority and contains a structure that allows plug-and-play additions of new datasets and metrics. To enable actionable evaluation insights, we introduce ''Evaluation Exercises'' that highlight takeaways for specific evaluation questions. The Evaluation Exercises contain easy-to-use and reproducible implementations of two state-of-the-art evaluation methods of text-to-image generative models: consistency-diversity-realism Pareto Fronts and disaggregated measurements of performance disparities across groups. EvalGIM also contains Evaluation Exercises that introduce two new analysis methods for text-to-image generative models: robustness analyses of model rankings and balanced evaluations across different prompt styles. We encourage text-to-image model exploration with EvalGIM and invite contributions at this https URL.

Title: Client-Side Patching against Backdoor Attacks in Federated Learning

• Authors: Borja Molina Coronado
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10605
• Pdf URL: https://arxiv.org/pdf/2412.10605
• Copy Paste: [[2412.10605]] Client-Side Patching against Backdoor Attacks in Federated Learning(https://arxiv.org/abs/2412.10605)
• Keywords: defense, attack, federate
• Abstract: Federated learning is a versatile framework for training models in decentralized environments. However, the trust placed in clients makes federated learning vulnerable to backdoor attacks launched by malicious participants. While many defenses have been proposed, they often fail short when facing heterogeneous data distributions among participating clients. In this paper, we propose a novel defense mechanism for federated learning systems designed to mitigate backdoor attacks on the clients-side. Our approach leverages adversarial learning techniques and model patching to neutralize the impact of backdoor attacks. Through extensive experiments on the MNIST and Fashion-MNIST datasets, we demonstrate that our defense effectively reduces backdoor accuracy, outperforming existing state-of-the-art defenses, such as LFighter, FLAME, and RoseAgg, in i.i.d. and non-i.i.d. scenarios, while maintaining competitive or superior accuracy on clean data.

Title: A Trust-Centric Approach To Quantifying Maturity and Security in Internet Voting Protocols

• Authors: Stanisław Barański, Ben Biedermann, Joshua Ellul
• Subjects: cs.CR, cs.CY, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.10611
• Pdf URL: https://arxiv.org/pdf/2412.10611
• Copy Paste: [[2412.10611]] A Trust-Centric Approach To Quantifying Maturity and Security in Internet Voting Protocols(https://arxiv.org/abs/2412.10611)
• Keywords: security, federate
• Abstract: Voting is a cornerstone of collective participatory decision-making in contexts ranging from political elections to decentralized autonomous organizations (DAOs). Despite the proliferation of internet voting protocols promising enhanced accessibility and efficiency, their evaluation and comparison are complicated by a lack of standardized criteria and unified definitions of security and maturity. Furthermore, socio-technical requirements by decision makers are not structurally taken into consideration when comparing internet voting systems. This paper addresses this gap by introducing a trust-centric maturity scoring framework to quantify the security and maturity of sixteen internet voting systems. A comprehensive trust model analysis is conducted for selected internet voting protocols, examining their security properties, trust assumptions, technical complexity, and practical usability. In this paper we propose the electronic voting maturity framework (EVMF) which supports nuanced assessment that reflects real-world deployment concerns and aids decision-makers in selecting appropriate systems tailored to their specific use-case requirements. The framework is general enough to be applied to other systems, where the aspects of decentralization, trust, and security are crucial, such as digital identity, Ethereum layer-two scaling solutions, and federated data infrastructures. Its objective is to provide an extendable toolkit for policy makers and technology experts alike that normalizes technical and non-technical requirements on a univariate scale.

Title: Meeting Utility Constraints in Differential Privacy: A Privacy-Boosting Approach

• Authors: Bo Jiang, Wanrong Zhang, Donghang Lu, Jian Du, Sagar Sharma, Qiang Yan
• Subjects: cs.CR, cs.DS, cs.IT
• Abstract URL: https://arxiv.org/abs/2412.10612
• Pdf URL: https://arxiv.org/pdf/2412.10612
• Copy Paste: [[2412.10612]] Meeting Utility Constraints in Differential Privacy: A Privacy-Boosting Approach(https://arxiv.org/abs/2412.10612)
• Keywords: privacy
• Abstract: Data engineering often requires accuracy (utility) constraints on results, posing significant challenges in designing differentially private (DP) mechanisms, particularly under stringent privacy parameter $\epsilon$. In this paper, we propose a privacy-boosting framework that is compatible with most noise-adding DP mechanisms. Our framework enhances the likelihood of outputs falling within a preferred subset of the support to meet utility requirements while enlarging the overall variance to reduce privacy leakage. We characterize the privacy loss distribution of our framework and present the privacy profile formulation for $(\epsilon,\delta)$-DP and Rényi DP (RDP) guarantees. We study special cases involving data-dependent and data-independent utility formulations. Through extensive experiments, we demonstrate that our framework achieves lower privacy loss than standard DP mechanisms under utility constraints. Notably, our approach is particularly effective in reducing privacy loss with large query sensitivity relative to the true answer, offering a more practical and flexible approach to designing differentially private mechanisms that meet specific utility constraints.

Title: Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration

• Authors: Avinandan Bose, Zhihan Xiong, Aadirupa Saha, Simon Shaolei Du, Maryam Fazel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10616
• Pdf URL: https://arxiv.org/pdf/2412.10616
• Copy Paste: [[2412.10616]] Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration(https://arxiv.org/abs/2412.10616)
• Keywords: large language model
• Abstract: Reinforcement Learning from Human Feedback (RLHF) is currently the leading approach for aligning large language models with human preferences. Typically, these models rely on extensive offline preference datasets for training. However, offline algorithms impose strict concentrability requirements, which are often difficult to satisfy. On the other hand, while online algorithms can avoid the concentrability issue, pure online exploration could be expensive due to the active preference query cost and real-time implementation overhead. In this paper, we propose a novel approach: Hybrid Preference Optimization (HPO) which combines online exploration with existing offline preferences by relaxing the stringent concentrability conditions for offline exploration, as well as significantly improving the sample efficiency for its online counterpart. We give the first provably optimal theoretical bound for Hybrid RLHF with preference feedback, providing sample complexity bounds for policy optimization with matching lower bounds. Our results yield improved sample efficiency of hybrid RLHF over pure offline and online exploration.

Title: BinarySelect to Improve Accessibility of Black-Box Attack Research

• Authors: Shatarupa Ghosh, Jonathan Rusert
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10617
• Pdf URL: https://arxiv.org/pdf/2412.10617
• Copy Paste: [[2412.10617]] BinarySelect to Improve Accessibility of Black-Box Attack Research(https://arxiv.org/abs/2412.10617)
• Keywords: attack, robust, transformer
• Abstract: Adversarial text attack research is useful for testing the robustness of NLP models, however, the rise of transformers has greatly increased the time required to test attacks. Especially when researchers do not have access to adequate resources (e.g. GPUs). This can hinder attack research, as modifying one example for an attack can require hundreds of queries to a model, especially for black-box attacks. Often these attacks remove one token at a time to find the ideal one to change, requiring $n$ queries (the length of the text) right away. We propose a more efficient selection method called BinarySelect which combines binary search and attack selection methods to greatly reduce the number of queries needed to find a token. We find that BinarySelect only needs $\text{log}_2(n) * 2$ queries to find the first token compared to $n$ queries. We also test BinarySelect in an attack setting against 5 classifiers across 3 datasets and find a viable tradeoff between number of queries saved and attack effectiveness. For example, on the Yelp dataset, the number of queries is reduced by 32% (72 less) with a drop in attack effectiveness of only 5 points. We believe that BinarySelect can help future researchers study adversarial attacks and black-box problems more efficiently and opens the door for researchers with access to less resources.

Title: WaveGNN: Modeling Irregular Multivariate Time Series for Accurate Predictions

• Authors: Arash Hajisafi, Maria Despoina Siampou, Bita Azarijoo, Cyrus Shahabi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10621
• Pdf URL: https://arxiv.org/pdf/2412.10621
• Copy Paste: [[2412.10621]] WaveGNN: Modeling Irregular Multivariate Time Series for Accurate Predictions(https://arxiv.org/abs/2412.10621)
• Keywords: transformer
• Abstract: Accurately modeling and analyzing time series data is crucial for downstream applications across various fields, including healthcare, finance, astronomy, and epidemiology. However, real-world time series often exhibit irregularities such as misaligned timestamps, missing entries, and variable sampling rates, complicating their analysis. Existing approaches often rely on imputation, which can introduce biases. A few approaches that directly model irregularity tend to focus exclusively on either capturing intra-series patterns or inter-series relationships, missing the benefits of integrating both. To this end, we present WaveGNN, a novel framework designed to directly (i.e., no imputation) embed irregularly sampled multivariate time series data for accurate predictions. WaveGNN utilizes a Transformer-based encoder to capture intra-series patterns by directly encoding the temporal dynamics of each time series. To capture inter-series relationships, WaveGNN uses a dynamic graph neural network model, where each node represents a sensor, and the edges capture the long- and short-term relationships between them. Our experimental results on real-world healthcare datasets demonstrate that WaveGNN consistently outperforms existing state-of-the-art methods, with an average relative improvement of 14.7% in F1-score when compared to the second-best baseline in cases with extreme sparsity. Our ablation studies reveal that both intra-series and inter-series modeling significantly contribute to this notable improvement.

Title: CATALOG: A Camera Trap Language-guided Contrastive Learning Model

• Authors: Julian D. Santamaria, Claudia Isaza, Jhony H. Giraldo
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10624
• Pdf URL: https://arxiv.org/pdf/2412.10624
• Copy Paste: [[2412.10624]] CATALOG: A Camera Trap Language-guided Contrastive Learning Model(https://arxiv.org/abs/2412.10624)
• Keywords: segmentation
• Abstract: Foundation Models (FMs) have been successful in various computer vision tasks like image classification, object detection and image segmentation. However, these tasks remain challenging when these models are tested on datasets with different distributions from the training dataset, a problem known as domain shift. This is especially problematic for recognizing animal species in camera-trap images where we have variability in factors like lighting, camouflage and occlusions. In this paper, we propose the Camera Trap Language-guided Contrastive Learning (CATALOG) model to address these issues. Our approach combines multiple FMs to extract visual and textual features from camera-trap data and uses a contrastive loss function to train the model. We evaluate CATALOG on two benchmark datasets and show that it outperforms previous state-of-the-art methods in camera-trap image recognition, especially when the training and testing data have different animal species or come from different geographical areas. Our approach demonstrates the potential of using FMs in combination with multi-modal fusion and contrastive learning for addressing domain shifts in camera-trap image recognition. The code of CATALOG is publicly available at this https URL.

Title: DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identification

• Authors: Yuhao Wang, Yang Liu, Aihua Zheng, Pingping Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10650
• Pdf URL: https://arxiv.org/pdf/2412.10650
• Copy Paste: [[2412.10650]] DeMo: Decoupled Feature-Based Mixture of Experts for Multi-Modal Object Re-Identification(https://arxiv.org/abs/2412.10650)
• Keywords: robust
• Abstract: Multi-modal object Re-IDentification (ReID) aims to retrieve specific objects by combining complementary information from multiple modalities. Existing multi-modal object ReID methods primarily focus on the fusion of heterogeneous features. However, they often overlook the dynamic quality changes in multi-modal imaging. In addition, the shared information between different modalities can weaken modality-specific information. To address these issues, we propose a novel feature learning framework called DeMo for multi-modal object ReID, which adaptively balances decoupled features using a mixture of experts. To be specific, we first deploy a Patch-Integrated Feature Extractor (PIFE) to extract multi-granularity and multi-modal features. Then, we introduce a Hierarchical Decoupling Module (HDM) to decouple multi-modal features into non-overlapping forms, preserving the modality uniqueness and increasing the feature diversity. Finally, we propose an Attention-Triggered Mixture of Experts (ATMoE), which replaces traditional gating with dynamic attention weights derived from decoupled features. With these modules, our DeMo can generate more robust multi-modal features. Extensive experiments on three multi-modal object ReID benchmarks fully verify the effectiveness of our methods. The source code is available at this https URL.

Title: Centaur: Bridging the Impossible Trinity of Privacy, Efficiency, and Performance in Privacy-Preserving Transformer Inference

• Authors: Jinglong Luo, Guanzhong Chen, Yehong Zhang, Shiyu Liu, Hui Wang, Yue Yu, Xun Zhou, Yuan Qi, Zenglin Xu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.10652
• Pdf URL: https://arxiv.org/pdf/2412.10652
• Copy Paste: [[2412.10652]] Centaur: Bridging the Impossible Trinity of Privacy, Efficiency, and Performance in Privacy-Preserving Transformer Inference(https://arxiv.org/abs/2412.10652)
• Keywords: secure, privacy, protect, attack, transformer
• Abstract: As pre-trained models, like Transformers, are increasingly deployed on cloud platforms for inference services, the privacy concerns surrounding model parameters and inference data are becoming more acute. Current Privacy-Preserving Transformer Inference (PPTI) frameworks struggle with the "impossible trinity" of privacy, efficiency, and performance. For instance, Secure Multi-Party Computation (SMPC)-based solutions offer strong privacy guarantees but come with significant inference overhead and performance trade-offs. On the other hand, PPTI frameworks that use random permutations achieve inference efficiency close to that of plaintext and maintain accurate results but require exposing some model parameters and intermediate results, thereby risking substantial privacy breaches. Addressing this "impossible trinity" with a single technique proves challenging. To overcome this challenge, we propose Centaur, a novel hybrid PPTI framework. Unlike existing methods, Centaur protects model parameters with random permutations and inference data with SMPC, leveraging the structure of Transformer models. By designing a series of efficient privacy-preserving algorithms, Centaur leverages the strengths of both techniques to achieve a better balance between privacy, efficiency, and performance in PPTI. We comprehensively evaluate the effectiveness of Centaur on various types of Transformer models and datasets. Experimental results demonstrate that the privacy protection capabilities offered by Centaur can withstand various existing model inversion attack methods. In terms of performance and efficiency, Centaur not only maintains the same performance as plaintext inference but also improves inference speed by $5.0-30.4$ times.

Title: Thinking with Knowledge Graphs: Enhancing LLM Reasoning Through Structured Data

• Authors: Xue Wu, Kostas Tsioutsiouliklis
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10654
• Pdf URL: https://arxiv.org/pdf/2412.10654
• Copy Paste: [[2412.10654]] Thinking with Knowledge Graphs: Enhancing LLM Reasoning Through Structured Data(https://arxiv.org/abs/2412.10654)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation. However, they often struggle with complex reasoning tasks and are prone to hallucination. Recent research has shown promising results in leveraging knowledge graphs (KGs) to enhance LLM performance. KGs provide a structured representation of entities and their relationships, offering a rich source of information that can enhance the reasoning capabilities of LLMs. For this work, we have developed different techniques that tightly integrate KG structures and semantics into LLM representations. Our results show that we are able to significantly improve the performance of LLMs in complex reasoning scenarios, and ground the reasoning process with KGs. We are the first to represent KGs with programming language and fine-tune pretrained LLMs with KGs. This integration facilitates more accurate and interpretable reasoning processes, paving the way for more advanced reasoning capabilities of LLMs.

Title: MEATRD: Multimodal Anomalous Tissue Region Detection Enhanced with Spatial Transcriptomics

• Authors: Kaichen Xu, Qilong Wu, Yan Lu, Yinan Zheng, Wenlin Li, Xingjie Tang, Jun Wang, Xiaobo Sun
• Subjects: cs.CV, cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2412.10659
• Pdf URL: https://arxiv.org/pdf/2412.10659
• Copy Paste: [[2412.10659]] MEATRD: Multimodal Anomalous Tissue Region Detection Enhanced with Spatial Transcriptomics(https://arxiv.org/abs/2412.10659)
• Keywords: transformer
• Abstract: The detection of anomalous tissue regions (ATRs) within affected tissues is crucial in clinical diagnosis and pathological studies. Conventional automated ATR detection methods, primarily based on histology images alone, falter in cases where ATRs and normal tissues have subtle visual differences. The recent spatial transcriptomics (ST) technology profiles gene expressions across tissue regions, offering a molecular perspective for detecting ATRs. However, there is a dearth of ATR detection methods that effectively harness complementary information from both histology images and ST. To address this gap, we propose MEATRD, a novel ATR detection method that integrates histology image and ST data. MEATRD is trained to reconstruct image patches and gene expression profiles of normal tissue spots (inliers) from their multimodal embeddings, followed by learning a one-class classification AD model based on latent multimodal reconstruction errors. This strategy harmonizes the strengths of reconstruction-based and one-class classification approaches. At the heart of MEATRD is an innovative masked graph dual-attention transformer (MGDAT) network, which not only facilitates cross-modality and cross-node information sharing but also addresses the model over-generalization issue commonly seen in reconstruction-based AD methods. Additionally, we demonstrate that modality-specific, task-relevant information is collated and condensed in multimodal bottleneck encoding generated in MGDAT, marking the first theoretical analysis of the informational properties of multimodal bottleneck encoding. Extensive evaluations across eight real ST datasets reveal MEATRD's superior performance in ATR detection, surpassing various state-of-the-art AD methods. Remarkably, MEATRD also proves adept at discerning ATRs that only show slight visual deviations from normal tissues.

Title: Structured Sampling for Robust Euclidean Distance Geometry

• Authors: Chandra Kundu, Abiy Tasissa, HanQin Cai
• Subjects: cs.LG, cs.IT, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.10664
• Pdf URL: https://arxiv.org/pdf/2412.10664
• Copy Paste: [[2412.10664]] Structured Sampling for Robust Euclidean Distance Geometry(https://arxiv.org/abs/2412.10664)
• Keywords: robust
• Abstract: This paper addresses the problem of estimating the positions of points from distance measurements corrupted by sparse outliers. Specifically, we consider a setting with two types of nodes: anchor nodes, for which exact distances to each other are known, and target nodes, for which complete but corrupted distance measurements to the anchors are available. To tackle this problem, we propose a novel algorithm powered by Nyström method and robust principal component analysis. Our method is computationally efficient as it processes only a localized subset of the distance matrix and does not require distance measurements between target nodes. Empirical evaluations on synthetic datasets, designed to mimic sensor localization, and on molecular experiments, demonstrate that our algorithm achieves accurate recovery with a modest number of anchors, even in the presence of high levels of sparse outliers.

Title: FairGP: A Scalable and Fair Graph Transformer Using Graph Partitioning

• Authors: Renqiang Luo, Huafei Huang, Ivan Lee, Chengpei Xu, Jianzhong Qi, Feng Xia
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.10669
• Pdf URL: https://arxiv.org/pdf/2412.10669
• Copy Paste: [[2412.10669]] FairGP: A Scalable and Fair Graph Transformer Using Graph Partitioning(https://arxiv.org/abs/2412.10669)
• Keywords: fair, transformer
• Abstract: Recent studies have highlighted significant fairness issues in Graph Transformer (GT) models, particularly against subgroups defined by sensitive features. Additionally, GTs are computationally intensive and memory-demanding, limiting their application to large-scale graphs. Our experiments demonstrate that graph partitioning can enhance the fairness of GT models while reducing computational complexity. To understand this improvement, we conducted a theoretical investigation into the root causes of fairness issues in GT models. We found that the sensitive features of higher-order nodes disproportionately influence lower-order nodes, resulting in sensitive feature bias. We propose Fairness-aware scalable GT based on Graph Partitioning (FairGP), which partitions the graph to minimize the negative impact of higher-order nodes. By optimizing attention mechanisms, FairGP mitigates the bias introduced by global attention, thereby enhancing fairness. Extensive empirical evaluations on six real-world datasets validate the superior performance of FairGP in achieving fairness compared to state-of-the-art methods. The codes are available at this https URL.

Title: Chasing Progress, Not Perfection: Revisiting Strategies for End-to-End LLM Plan Generation

• Authors: Sukai Huang, Trevor Cohn, Nir Lipovetzky
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10675
• Pdf URL: https://arxiv.org/pdf/2412.10675
• Copy Paste: [[2412.10675]] Chasing Progress, Not Perfection: Revisiting Strategies for End-to-End LLM Plan Generation(https://arxiv.org/abs/2412.10675)
• Keywords: robust, large language model
• Abstract: The capability of Large Language Models (LLMs) to plan remains a topic of debate. Some critics argue that strategies to boost LLMs' reasoning skills are ineffective in planning tasks, while others report strong outcomes merely from training models on a planning corpus. This study reassesses recent strategies by developing an end-to-end LLM planner and employing diverse metrics for a thorough evaluation. We find that merely fine-tuning LLMs on a corpus of planning instances does not lead to robust planning skills, as indicated by poor performance on out-of-distribution test sets. At the same time, we find that various strategies, including Chain-of-Thought, do enhance the probability of a plan being executable. This indicates progress towards better plan quality, despite not directly enhancing the final validity rate. Among the strategies we evaluated, reinforcement learning with our novel `Longest Contiguous Common Subsequence' reward emerged as the most effective, contributing to both plan validity and executability. Overall, our research addresses key misconceptions in the LLM-planning literature; we validate incremental progress in plan executability, although plan validity remains a challenge. Hence, future strategies should focus on both these aspects, drawing insights from our findings.

Title: UCDR-Adapter: Exploring Adaptation of Pre-Trained Vision-Language Models for Universal Cross-Domain Retrieval

• Authors: Haoyu Jiang, Zhi-Qi Cheng, Gabriel Moreira, Jiawen Zhu, Jingdong Sun, Bukun Ren, Jun-Yan He, Qi Dai, Xian-Sheng Hua
• Subjects: cs.CV, cs.IR, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.10680
• Pdf URL: https://arxiv.org/pdf/2412.10680
• Copy Paste: [[2412.10680]] UCDR-Adapter: Exploring Adaptation of Pre-Trained Vision-Language Models for Universal Cross-Domain Retrieval(https://arxiv.org/abs/2412.10680)
• Keywords: robust
• Abstract: Universal Cross-Domain Retrieval (UCDR) retrieves relevant images from unseen domains and classes without semantic labels, ensuring robust generalization. Existing methods commonly employ prompt tuning with pre-trained vision-language models but are inherently limited by static prompts, reducing adaptability. We propose UCDR-Adapter, which enhances pre-trained models with adapters and dynamic prompt generation through a two-phase training strategy. First, Source Adapter Learning integrates class semantics with domain-specific visual knowledge using a Learnable Textual Semantic Template and optimizes Class and Domain Prompts via momentum updates and dual loss functions for robust alignment. Second, Target Prompt Generation creates dynamic prompts by attending to masked source prompts, enabling seamless adaptation to unseen domains and classes. Unlike prior approaches, UCDR-Adapter dynamically adapts to evolving data distributions, enhancing both flexibility and generalization. During inference, only the image branch and generated prompts are used, eliminating reliance on textual inputs for highly efficient retrieval. Extensive benchmark experiments show that UCDR-Adapter consistently outperforms ProS in most cases and other state-of-the-art methods on UCDR, U(c)CDR, and U(d)CDR settings.

Title: One Pixel is All I Need

• Authors: Deng Siqin, Zhou Xiaoyi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10681
• Pdf URL: https://arxiv.org/pdf/2412.10681
• Copy Paste: [[2412.10681]] One Pixel is All I Need(https://arxiv.org/abs/2412.10681)
• Keywords: attack, robust, transformer
• Abstract: Vision Transformers (ViTs) have achieved record-breaking performance in various visual tasks. However, concerns about their robustness against backdoor attacks have grown. Backdoor attacks involve associating a specific trigger with a target label, causing the model to predict the attacker-specified label when the trigger is present, while correctly identifying clean this http URL found that ViTs exhibit higher attack success rates for quasi-triggers(patterns different from but similar to the original training triggers)compared to CNNs. Moreover, some backdoor features in clean samples can suppress the original trigger, making quasi-triggers more this http URL better understand and exploit these vulnerabilities, we developed a tool called the Perturbation Sensitivity Distribution Map (PSDM). PSDM computes and sums gradients over many inputs to show how sensitive the model is to small changes in the input. In ViTs, PSDM reveals a patch-like pattern where central pixels are more sensitive than edges. We use PSDM to guide the creation of this http URL on these findings, we designed "WorstVIT," a simple yet effective data poisoning backdoor for ViT models. This attack requires an extremely low poisoning rate, trains for just one epoch, and modifies a single pixel to successfully attack all validation images.

Title: Stochastic $k$-Submodular Bandits with Full Bandit Feedback

• Authors: Guanyu Nie, Vaneet Aggarwal, Christopher John Quinn
• Subjects: cs.LG, cs.DS, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.10682
• Pdf URL: https://arxiv.org/pdf/2412.10682
• Copy Paste: [[2412.10682]] Stochastic $k$-Submodular Bandits with Full Bandit Feedback(https://arxiv.org/abs/2412.10682)
• Keywords: robust
• Abstract: In this paper, we present the first sublinear $\alpha$-regret bounds for online $k$-submodular optimization problems with full-bandit feedback, where $\alpha$ is a corresponding offline approximation ratio. Specifically, we propose online algorithms for multiple $k$-submodular stochastic combinatorial multi-armed bandit problems, including (i) monotone functions and individual size constraints, (ii) monotone functions with matroid constraints, (iii) non-monotone functions with matroid constraints, (iv) non-monotone functions without constraints, and (v) monotone functions without constraints. We transform approximation algorithms for offline $k$-submodular maximization problems into online algorithms through the offline-to-online framework proposed by Nie et al. (2023a). A key contribution of our work is analyzing the robustness of the offline algorithms.

Title: Inference Scaling for Bridging Retrieval and Augmented Generation

• Authors: Youngwon Lee, Seung-won Hwang, Daniel Campos, Filip Graliński, Zhewei Yao, Yuxiong He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10684
• Pdf URL: https://arxiv.org/pdf/2412.10684
• Copy Paste: [[2412.10684]] Inference Scaling for Bridging Retrieval and Augmented Generation(https://arxiv.org/abs/2412.10684)
• Keywords: large language model
• Abstract: Retrieval-augmented generation (RAG) has emerged as a popular approach to steering the output of a large language model (LLM) by incorporating retrieved contexts as inputs. However, existing work observed the generator bias, such that improving the retrieval results may negatively affect the outcome. In this work, we show such bias can be mitigated, from inference scaling, aggregating inference calls from the permuted order of retrieved contexts. The proposed Mixture-of-Intervention (MOI) explicitly models the debiased utility of each passage with multiple forward passes to construct a new ranking. We also show that MOI can leverage the retriever's prior knowledge to reduce the computational cost by minimizing the number of permutations considered and lowering the cost per LLM call. We showcase the effectiveness of MOI on diverse RAG tasks, improving ROUGE-L on MS MARCO and EM on HotpotQA benchmarks by ~7 points.

Title: Linked Adapters: Linking Past and Future to Present for Effective Continual Learning

• Authors: Dupati Srikar Chandra, P. K. Srijith, Dana Rezazadegan, Chris McCarthy
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10687
• Pdf URL: https://arxiv.org/pdf/2412.10687
• Copy Paste: [[2412.10687]] Linked Adapters: Linking Past and Future to Present for Effective Continual Learning(https://arxiv.org/abs/2412.10687)
• Keywords: transformer
• Abstract: Continual learning allows the system to learn and adapt to new tasks while retaining the knowledge acquired from previous tasks. However, deep learning models suffer from catastrophic forgetting of knowledge learned from earlier tasks while learning a new task. Moreover, retraining large models like transformers from scratch for every new task is costly. An effective approach to address continual learning is to use a large pre-trained model with task-specific adapters to adapt to the new tasks. Though this approach can mitigate catastrophic forgetting, they fail to transfer knowledge across tasks as each task is learning adapters separately. To address this, we propose a novel approach Linked Adapters that allows knowledge transfer through a weighted attention mechanism to other task-specific adapters. Linked adapters use a multi-layer perceptron (MLP) to model the attention weights, which overcomes the challenge of backward knowledge transfer in continual learning in addition to modeling the forward knowledge transfer. During inference, our proposed approach effectively leverages knowledge transfer through MLP-based attention weights across all the lateral task adapters. Through numerous experiments conducted on diverse image classification datasets, we effectively demonstrated the improvement in performance on the continual learning tasks using Linked Adapters.

Title: Learning to Verify Summary Facts with Fine-Grained LLM Feedback

• Authors: Jihwan Oh, Jeonghwan Choi, Nicole Hee-Yeon Kim, Taewon Yun, Hwanjun Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10689
• Pdf URL: https://arxiv.org/pdf/2412.10689
• Copy Paste: [[2412.10689]] Learning to Verify Summary Facts with Fine-Grained LLM Feedback(https://arxiv.org/abs/2412.10689)
• Keywords: large language model
• Abstract: Training automatic summary fact verifiers often faces the challenge of a lack of human-labeled data. In this paper, we explore alternative way of leveraging Large Language Model (LLM) generated feedback to address the inherent limitation of using human-labeled data. We introduce FineSumFact, a large-scale dataset containing fine-grained factual feedback on summaries. We employ 10 distinct LLMs for diverse summary generation and Llama-3-70B-Instruct for feedback. We utilize this dataset to fine-tune the lightweight open-source model Llama-3-8B-Instruct, optimizing resource efficiency while maintaining high performance. Our experimental results reveal that the model trained on extensive LLM-generated datasets surpasses that trained on smaller human-annotated datasets when evaluated using human-generated test sets. Fine-tuning fact verification models with LLM feedback can be more effective and cost-efficient than using human feedback. The dataset is available at this https URL.

Title: On the Cyber-Physical Security of Commercial Indoor Delivery Robot Systems

• Authors: Fayzah Alshammari, Yunpeng Luo, Qi Alfred Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.10699
• Pdf URL: https://arxiv.org/pdf/2412.10699
• Copy Paste: [[2412.10699]] On the Cyber-Physical Security of Commercial Indoor Delivery Robot Systems(https://arxiv.org/abs/2412.10699)
• Keywords: security, privacy, attack
• Abstract: Indoor Delivery Robots (IDRs) play a vital role in the upcoming fourth industrial revolution, autonomously navigating and transporting items within indoor environments. In this work, we thus aim to conduct the first security analysis of the IDR systems considering both cyber- and physical-layer attack surface and domain-specific attack goals across security, safety, and privacy. As initial results, we formulated a general IDR system architecture from 40 commercial IDR models and then performed an initial cyber-physical attack entry point identification. We also performed an experimental analysis of a real commercial IDR robot-side software and identified several vulnerabilities. We then discuss future steps.

Title: Memory Efficient Matting with Adaptive Token Routing

• Authors: Yiheng Lin, Yihan Hu, Chenyi Zhang, Ting Liu, Xiaochao Qu, Luoqi Liu, Yao Zhao, Yunchao Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10702
• Pdf URL: https://arxiv.org/pdf/2412.10702
• Copy Paste: [[2412.10702]] Memory Efficient Matting with Adaptive Token Routing(https://arxiv.org/abs/2412.10702)
• Keywords: transformer
• Abstract: Transformer-based models have recently achieved outstanding performance in image matting. However, their application to high-resolution images remains challenging due to the quadratic complexity of global self-attention. To address this issue, we propose MEMatte, a memory-efficient matting framework for processing high-resolution images. MEMatte incorporates a router before each global attention block, directing informative tokens to the global attention while routing other tokens to a Lightweight Token Refinement Module (LTRM). Specifically, the router employs a local-global strategy to predict the routing probability of each token, and the LTRM utilizes efficient modules to simulate global attention. Additionally, we introduce a Batch-constrained Adaptive Token Routing (BATR) mechanism, which allows each router to dynamically route tokens based on image content and the stages of attention block in the network. Furthermore, we construct an ultra high-resolution image matting dataset, UHR-395, comprising 35,500 training images and 1,000 test images, with an average resolution of $4872\times6017$. This dataset is created by compositing 395 different alpha mattes across 11 categories onto various backgrounds, all with high-quality manual annotation. Extensive experiments demonstrate that MEMatte outperforms existing methods on both high-resolution and real-world datasets, significantly reducing memory usage by approximately 88% and latency by 50% on the Composition-1K benchmark.

Title: VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation

• Authors: Manan Suri, Puneet Mathur, Franck Dernoncourt, Kanika Goswami, Ryan A. Rossi, Dinesh Manocha
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10704
• Pdf URL: https://arxiv.org/pdf/2412.10704
• Copy Paste: [[2412.10704]] VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation(https://arxiv.org/abs/2412.10704)
• Keywords: robust, large language model
• Abstract: Understanding information from a collection of multiple documents, particularly those with visually rich elements, is important for document-grounded question answering. This paper introduces VisDoMBench, the first comprehensive benchmark designed to evaluate QA systems in multi-document settings with rich multimodal content, including tables, charts, and presentation slides. We propose VisDoMRAG, a novel multimodal Retrieval Augmented Generation (RAG) approach that simultaneously utilizes visual and textual RAG, combining robust visual retrieval capabilities with sophisticated linguistic reasoning. VisDoMRAG employs a multi-step reasoning process encompassing evidence curation and chain-of-thought reasoning for concurrent textual and visual RAG pipelines. A key novelty of VisDoMRAG is its consistency-constrained modality fusion mechanism, which aligns the reasoning processes across modalities at inference time to produce a coherent final answer. This leads to enhanced accuracy in scenarios where critical information is distributed across modalities and improved answer verifiability through implicit context attribution. Through extensive experiments involving open-source and proprietary large language models, we benchmark state-of-the-art document QA methods on VisDoMBench. Extensive results show that VisDoMRAG outperforms unimodal and long-context LLM baselines for end-to-end multimodal document QA by 12-20%.

Title: MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Prompt

• Authors: Yuhao Wang, Xuehu Liu, Tianyu Yan, Yang Liu, Aihua Zheng, Pingping Zhang, Huchuan Lu
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.10707
• Pdf URL: https://arxiv.org/pdf/2412.10707
• Copy Paste: [[2412.10707]] MambaPro: Multi-Modal Object Re-Identification with Mamba Aggregation and Synergistic Prompt(https://arxiv.org/abs/2412.10707)
• Keywords: robust
• Abstract: Multi-modal object Re-IDentification (ReID) aims to retrieve specific objects by utilizing complementary image information from different modalities. Recently, large-scale pre-trained models like CLIP have demonstrated impressive performance in traditional single-modal object ReID tasks. However, they remain unexplored for multi-modal object ReID. Furthermore, current multi-modal aggregation methods have obvious limitations in dealing with long sequences from different modalities. To address above issues, we introduce a novel framework called MambaPro for multi-modal object ReID. To be specific, we first employ a Parallel Feed-Forward Adapter (PFA) for adapting CLIP to multi-modal object ReID. Then, we propose the Synergistic Residual Prompt (SRP) to guide the joint learning of multi-modal features. Finally, leveraging Mamba's superior scalability for long sequences, we introduce Mamba Aggregation (MA) to efficiently model interactions between different modalities. As a result, MambaPro could extract more robust features with lower complexity. Extensive experiments on three multi-modal object ReID benchmarks (i.e., RGBNT201, RGBNT100 and MSVR310) validate the effectiveness of our proposed methods. The source code is available at this https URL.

Title: RAT: Adversarial Attacks on Deep Reinforcement Agents for Targeted Behaviors

• Authors: Fengshuo Bai, Runze Liu, Yali Du, Ying Wen, Yaodong Yang
• Subjects: cs.LG, cs.AI, cs.CR, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.10713
• Pdf URL: https://arxiv.org/pdf/2412.10713
• Copy Paste: [[2412.10713]] RAT: Adversarial Attacks on Deep Reinforcement Agents for Targeted Behaviors(https://arxiv.org/abs/2412.10713)
• Keywords: defense, attack, robust, transformer
• Abstract: Evaluating deep reinforcement learning (DRL) agents against targeted behavior attacks is critical for assessing their robustness. These attacks aim to manipulate the victim into specific behaviors that align with the attacker's objectives, often bypassing traditional reward-based defenses. Prior methods have primarily focused on reducing cumulative rewards; however, rewards are typically too generic to capture complex safety requirements effectively. As a result, focusing solely on reward reduction can lead to suboptimal attack strategies, particularly in safety-critical scenarios where more precise behavior manipulation is needed. To address these challenges, we propose RAT, a method designed for universal, targeted behavior attacks. RAT trains an intention policy that is explicitly aligned with human preferences, serving as a precise behavioral target for the adversary. Concurrently, an adversary manipulates the victim's policy to follow this target behavior. To enhance the effectiveness of these attacks, RAT dynamically adjusts the state occupancy measure within the replay buffer, allowing for more controlled and effective behavior manipulation. Our empirical results on robotic simulation tasks demonstrate that RAT outperforms existing adversarial attack algorithms in inducing specific behaviors. Additionally, RAT shows promise in improving agent robustness, leading to more resilient policies. We further validate RAT by guiding Decision Transformer agents to adopt behaviors aligned with human preferences in various MuJoCo tasks, demonstrating its effectiveness across diverse tasks.

Title: Control of Overfitting with Physics

• Authors: Sergei V. Kozyrev, Ilya A Lopatin, Alexander N Pechen
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.10716
• Pdf URL: https://arxiv.org/pdf/2412.10716
• Copy Paste: [[2412.10716]] Control of Overfitting with Physics(https://arxiv.org/abs/2412.10716)
• Keywords: generative
• Abstract: While there are many works on the applications of machine learning, not so many of them are trying to understand the theoretical justifications to explain their efficiency. In this work, overfitting control (or generalization property) in machine learning is explained using analogies from physics and biology. For stochastic gradient Langevin dynamics, we show that the Eyring formula of kinetic theory allows to control overfitting in the algorithmic stability approach - when wide minima of the risk function with low free energy correspond to low overfitting. For the generative adversarial network (GAN) model, we establish an analogy between GAN and the predator-prey model in biology. An application of this analogy allows us to explain the selection of wide likelihood maxima and overfitting reduction for GANs.

Title: HITgram: A Platform for Experimenting with n-gram Language Models

• Authors: Shibaranjani Dasgupta, Chandan Maity, Somdip Mukherjee, Rohan Singh, Diptendu Dutta, Debasish Jana
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10717
• Pdf URL: https://arxiv.org/pdf/2412.10717
• Copy Paste: [[2412.10717]] HITgram: A Platform for Experimenting with n-gram Language Models(https://arxiv.org/abs/2412.10717)
• Keywords: large language model
• Abstract: Large language models (LLMs) are powerful but resource intensive, limiting accessibility. HITgram addresses this gap by offering a lightweight platform for n-gram model experimentation, ideal for resource-constrained environments. It supports unigrams to 4-grams and incorporates features like context sensitive weighting, Laplace smoothing, and dynamic corpus management to e-hance prediction accuracy, even for unseen word sequences. Experiments demonstrate HITgram's efficiency, achieving 50,000 tokens/second and generating 2-grams from a 320MB corpus in 62 seconds. HITgram scales efficiently, constructing 4-grams from a 1GB file in under 298 seconds on an 8 GB RAM system. Planned enhancements include multilingual support, advanced smoothing, parallel processing, and model saving, further broadening its utility.

Title: Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm

• Authors: Jinrong Zhang, Penghui Wang, Chunxiao Liu, Wei Liu, Dian Jin, Qiong Zhang, Erli Meng, Zhengnan Hu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10719
• Pdf URL: https://arxiv.org/pdf/2412.10719
• Copy Paste: [[2412.10719]] Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm(https://arxiv.org/abs/2412.10719)
• Keywords: large language model, segmentation
• Abstract: To break through the limitations of pre-training models on fixed categories, Open-Set Object Detection (OSOD) and Open-Set Segmentation (OSS) have attracted a surge of interest from researchers. Inspired by large language models, mainstream OSOD and OSS methods generally utilize text as a prompt, achieving remarkable performance. Following SAM paradigm, some researchers use visual prompts, such as points, boxes, and masks that cover detection or segmentation targets. Despite these two prompt paradigms exhibit excellent performance, they also reveal inherent limitations. On the one hand, it is difficult to accurately describe characteristics of specialized category using textual description. On the other hand, existing visual prompt paradigms heavily rely on multi-round human interaction, which hinders them being applied to fully automated pipeline. To address the above issues, we propose a novel prompt paradigm in OSOD and OSS, that is, \textbf{Image Prompt Paradigm}. This brand new prompt paradigm enables to detect or segment specialized categories without multi-round human intervention. To achieve this goal, the proposed image prompt paradigm uses just a few image instances as prompts, and we propose a novel framework named \textbf{MI Grounding} for this new paradigm. In this framework, high-quality image prompts are automatically encoded, selected and fused, achieving the single-stage and non-interactive inference. We conduct extensive experiments on public datasets, showing that MI Grounding achieves competitive performance on OSOD and OSS benchmarks compared to text prompt paradigm methods and visual prompt paradigm methods. Moreover, MI Grounding can greatly outperform existing method on our constructed specialized ADR50K dataset.

Title: A technical solution for the rule of law, peace, security, and evolvability of global cyberspace -- solve the three genetic defects of IP network

• Authors: Hui Li, Kedan Li, Jiaqing Lv, Yuanshao Liang, Feng Han, Shuo-Yen Robert Li
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2412.10722
• Pdf URL: https://arxiv.org/pdf/2412.10722
• Copy Paste: [[2412.10722]] A technical solution for the rule of law, peace, security, and evolvability of global cyberspace -- solve the three genetic defects of IP network(https://arxiv.org/abs/2412.10722)
• Keywords: security, attack
• Abstract: Since its inception in the 1960s, the internet has profoundly transformed human life. However, its original design now struggles to meet the evolving demands of modern society. Three primary defects have emerged: First, the concentration of power among a few dominant entities has intensified international conflicts and widened the technological divide. Second, the Internet Protocol (IP)-based system lacks inherent security, leading to frequent global cybersecurity incidents. Third, the rigidity of the IP protocol has hindered the sustainable development of cyberspace, as it resists necessary adaptations and innovations. Addressing these issues is crucial for the future resilience and security of the global digital landscape. To address these challenges, we propose the Co-governed Multi-Identifier Network (CoG-MIN briefly as MIN), a novel network architecture that leverages blockchain technology to ensure equal participation of countries worldwide in cyberspace governance and the rule of law. As a next-generation network system, CoG-MIN integrates mechanisms such as user authentication, data signatures, and encryption to significantly enhance network security. In testing environments, CoG-MIN has consistently withstood extensive attacks during various international cybersecurity competitions. Additionally, CoG-MIN supports the evolution and interoperability of different identifier systems, remains IP-compatible, and facilitates a gradual transition away from IP, providing an adaptable ecosystem for diverse network architectures. This adaptability fosters the development and evolution of diverse network architectures within CoG-MIN, making it a natural progression for the internet's future development. We further introduce a trilogy of cyberspace security theorems... (Due to character limitations, the full abstract is available in the paper PDF.)

Title: HEP-NAS: Towards Efficient Few-shot Neural Architecture Search via Hierarchical Edge Partitioning

• Authors: Jianfeng Li, Jiawen Zhang, Feng Wang, Lianbo Ma
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10723
• Pdf URL: https://arxiv.org/pdf/2412.10723
• Copy Paste: [[2412.10723]] HEP-NAS: Towards Efficient Few-shot Neural Architecture Search via Hierarchical Edge Partitioning(https://arxiv.org/abs/2412.10723)
• Keywords: segmentation
• Abstract: One-shot methods have significantly advanced the field of neural architecture search (NAS) by adopting weight-sharing strategy to reduce search costs. However, the accuracy of performance estimation can be compromised by co-adaptation. Few-shot methods divide the entire supernet into individual sub-supernets by splitting edge by edge to alleviate this issue, yet neglect relationships among edges and result in performance degradation on huge search space. In this paper, we introduce HEP-NAS, a hierarchy-wise partition algorithm designed to further enhance accuracy. To begin with, HEP-NAS treats edges sharing the same end node as a hierarchy, permuting and splitting edges within the same hierarchy to directly search for the optimal operation combination for each intermediate node. This approach aligns more closely with the ultimate goal of NAS. Furthermore, HEP-NAS selects the most promising sub-supernet after each segmentation, progressively narrowing the search space in which the optimal architecture may exist. To improve performance evaluation of sub-supernets, HEP-NAS employs search space mutual distillation, stabilizing the training process and accelerating the convergence of each individual sub-supernet. Within a given budget, HEP-NAS enables the splitting of all edges and gradually searches for architectures with higher accuracy. Experimental results across various datasets and search spaces demonstrate the superiority of HEP-NAS compared to state-of-the-art methods.

Title: MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance

• Authors: Wenjun Huang, Jianguo Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10730
• Pdf URL: https://arxiv.org/pdf/2412.10730
• Copy Paste: [[2412.10730]] MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance(https://arxiv.org/abs/2412.10730)
• Keywords: robust, extraction, segmentation
• Abstract: The Long Short-Term Memory (LSTM) networks have traditionally faced challenges in scaling and effectively capturing complex dependencies in visual tasks. The xLSTM architecture has emerged to address these limitations, incorporating exponential gating and a parallel matrix memory structure to enhance performance and scalability. Despite these advancements, the potential of xLSTM in visual computing has not been fully realized, particularly in leveraging autoregressive techniques for improved feature extraction. In this paper, we introduce MAL (Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance), a novel framework that enhances xLSTM's capabilities through innovative pretraining strategies. We propose a cluster-masked masking method that significantly improves local feature capture and optimizes image scanning efficiency. Additionally, our universal encoder-decoder pretraining approach integrates multiple tasks, including image autoregression, depth estimation, and image segmentation, thereby enhancing the model's adaptability and robustness across diverse visual tasks. Our experimental results demonstrate that MAL surpasses traditional supervised models and fully leverages the scaling potential of xLSTM, setting a new benchmark in visual task performance.

Title: OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Driving

• Authors: Lianqing Zheng, Long Yang, Qunshu Lin, Wenjin Ai, Minghao Liu, Shouyi Lu, Jianan Liu, Hongze Ren, Jingyue Mo, Xiaokai Bai, Jie Bai, Zhixiong Ma, Xichan Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10734
• Pdf URL: https://arxiv.org/pdf/2412.10734
• Copy Paste: [[2412.10734]] OmniHD-Scenes: A Next-Generation Multimodal Dataset for Autonomous Driving(https://arxiv.org/abs/2412.10734)
• Keywords: robust, segmentation
• Abstract: The rapid advancement of deep learning has intensified the need for comprehensive data for use by autonomous driving algorithms. High-quality datasets are crucial for the development of effective data-driven autonomous driving solutions. Next-generation autonomous driving datasets must be multimodal, incorporating data from advanced sensors that feature extensive data coverage, detailed annotations, and diverse scene representation. To address this need, we present OmniHD-Scenes, a large-scale multimodal dataset that provides comprehensive omnidirectional high-definition data. The OmniHD-Scenes dataset combines data from 128-beam LiDAR, six cameras, and six 4D imaging radar systems to achieve full environmental perception. The dataset comprises 1501 clips, each approximately 30-s long, totaling more than 450K synchronized frames and more than 5.85 million synchronized sensor data points. We also propose a novel 4D annotation pipeline. To date, we have annotated 200 clips with more than 514K precise 3D bounding boxes. These clips also include semantic segmentation annotations for static scene elements. Additionally, we introduce a novel automated pipeline for generation of the dense occupancy ground truth, which effectively leverages information from non-key frames. Alongside the proposed dataset, we establish comprehensive evaluation metrics, baseline models, and benchmarks for 3D detection and semantic occupancy prediction. These benchmarks utilize surround-view cameras and 4D imaging radar to explore cost-effective sensor solutions for autonomous driving applications. Extensive experiments demonstrate the effectiveness of our low-cost sensor configuration and its robustness under adverse conditions. Data will be released at this https URL.

Title: Diagnosing Unknown Attacks in Smart Homes Using Abductive Reasoning

• Authors: Kushal Ramkumar, Wanling Cai, John McCarthy, Gavin Doherty, Bashar Nuseibeh, Liliana Pasquale
• Subjects: cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.10738
• Pdf URL: https://arxiv.org/pdf/2412.10738
• Copy Paste: [[2412.10738]] Diagnosing Unknown Attacks in Smart Homes Using Abductive Reasoning(https://arxiv.org/abs/2412.10738)
• Keywords: security, attack
• Abstract: Security attacks are rising, as evidenced by the number of reported vulnerabilities. Among them, unknown attacks, including new variants of existing attacks, technical blind spots or previously undiscovered attacks, challenge enduring security. This is due to the limited number of techniques that diagnose these attacks and enable the selection of adequate security controls. In this paper, we propose an automated technique that detects and diagnoses unknown attacks by identifying the class of attack and the violated security requirements, enabling the selection of adequate security controls. Our technique combines anomaly detection to detect unknown attacks with abductive reasoning to diagnose them. We first model the behaviour of the smart home and its requirements as a logic program in Answer Set Programming (ASP). We then apply Z-Score thresholding to the anomaly scores of an Isolation Forest trained using unlabeled data to simulate unknown attack scenarios. Finally, we encode the network anomaly in the logic program and perform abduction by refutation to identify the class of attack and the security requirements that this anomaly may violate. We demonstrate our technique using a smart home scenario, where we detect and diagnose anomalies in network traffic. We evaluate the precision, recall and F1-score of the anomaly detector and the diagnosis technique against 18 attacks from the ground truth labels provided by two datasets, CICIoT2023 and IoT-23. Our experiments show that the anomaly detector effectively identifies anomalies when the network traces are strong indicators of an attack. When provided with sufficient contextual data, the diagnosis logic effectively identifies true anomalies, and reduces the number of false positives reported by anomaly detectors. Finally, we discuss how our technique can support the selection of adequate security controls.

Title: DSRC: Learning Density-insensitive and Semantic-aware Collaborative Representation against Corruptions

• Authors: Jingyu Zhang, Yilei Wang, Lang Qian, Peng Sun, Zengwen Li, Sudong Jiang, Maolin Liu, Liang Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10739
• Pdf URL: https://arxiv.org/pdf/2412.10739
• Copy Paste: [[2412.10739]] DSRC: Learning Density-insensitive and Semantic-aware Collaborative Representation against Corruptions(https://arxiv.org/abs/2412.10739)
• Keywords: robust
• Abstract: As a potential application of Vehicle-to-Everything (V2X) communication, multi-agent collaborative perception has achieved significant success in 3D object detection. While these methods have demonstrated impressive results on standard benchmarks, the robustness of such approaches in the face of complex real-world environments requires additional verification. To bridge this gap, we introduce the first comprehensive benchmark designed to evaluate the robustness of collaborative perception methods in the presence of natural corruptions typical of real-world environments. Furthermore, we propose DSRC, a robustness-enhanced collaborative perception method aiming to learn Density-insensitive and Semantic-aware collaborative Representation against Corruptions. DSRC consists of two key designs: i) a semantic-guided sparse-to-dense distillation framework, which constructs multi-view dense objects painted by ground truth bounding boxes to effectively learn density-insensitive and semantic-aware collaborative representation; ii) a feature-to-point cloud reconstruction approach to better fuse critical collaborative representation across agents. To thoroughly evaluate DSRC, we conduct extensive experiments on real-world and simulated datasets. The results demonstrate that our method outperforms SOTA collaborative perception methods in both clean and corrupted conditions. Code is available at this https URL.

Title: WEPO: Web Element Preference Optimization for LLM-based Web Navigation

• Authors: Jiarun Liu, Jia Hao, Chunhong Zhang, Zheng Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10742
• Pdf URL: https://arxiv.org/pdf/2412.10742
• Copy Paste: [[2412.10742]] WEPO: Web Element Preference Optimization for LLM-based Web Navigation(https://arxiv.org/abs/2412.10742)
• Keywords: large language model
• Abstract: The rapid advancement of autonomous web navigation has significantly benefited from grounding pretrained Large Language Models (LLMs) as agents. However, current research has yet to fully leverage the redundancy of HTML elements for contrastive training. This paper introduces a novel approach to LLM-based web navigation tasks, called Web Element Preference Optimization (WEPO). WEPO utilizes unsupervised preference learning by sampling distance-based non-salient web elements as negative samples, optimizing maximum likelihood objective within Direct Preference Optimization (DPO). We evaluate WEPO on the Mind2Web benchmark and empirically demonstrate that WEPO aligns user high-level intent with output actions more effectively. The results show that our method achieved the state-of-the-art, with an improvement of 13.8% over WebAgent and 5.3% over the visual language model CogAgent baseline. Our findings underscore the potential of preference optimization to enhance web navigation and other web page based tasks, suggesting a promising direction for future research.

Title: NeuralPLexer3: Physio-Realistic Biomolecular Complex Structure Prediction with Flow Models

• Authors: Zhuoran Qiao, Feizhi Ding, Thomas Dresselhaus, Mia A. Rosenfeld, Xiaotian Han, Owen Howell, Aniketh Iyengar, Stephen Opalenski, Anders S. Christensen, Sai Krishna Sirumalla, Frederick R. Manby, Thomas F. Miller III, Matthew Welborn
• Subjects: cs.LG, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2412.10743
• Pdf URL: https://arxiv.org/pdf/2412.10743
• Copy Paste: [[2412.10743]] NeuralPLexer3: Physio-Realistic Biomolecular Complex Structure Prediction with Flow Models(https://arxiv.org/abs/2412.10743)
• Keywords: generative
• Abstract: Structure determination is essential to a mechanistic understanding of diseases and the development of novel therapeutics. Machine-learning-based structure prediction methods have made significant advancements by computationally predicting protein and bioassembly structures from sequences and molecular topology alone. Despite substantial progress in the field, challenges remain to deliver structure prediction models to real-world drug discovery. Here, we present NeuralPLexer3 -- a physics-inspired flow-based generative model that achieves state-of-the-art prediction accuracy on key biomolecular interaction types and improves training and sampling efficiency compared to its predecessors and alternative methodologies. Examined through newly developed benchmarking strategies, NeuralPLexer3 excels in vital areas that are crucial to structure-based drug design, such as physical validity and ligand-induced conformational changes.

Title: A Pioneering Neural Network Method for Efficient and Robust Fuel Sloshing Simulation in Aircraft

• Authors: Yu Chen, Shuai Zheng, Nianyi Wang, Menglong Jin, Yan Chang
• Subjects: cs.CV, cs.GR, cs.LG, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2412.10748
• Pdf URL: https://arxiv.org/pdf/2412.10748
• Copy Paste: [[2412.10748]] A Pioneering Neural Network Method for Efficient and Robust Fuel Sloshing Simulation in Aircraft(https://arxiv.org/abs/2412.10748)
• Keywords: robust
• Abstract: Simulating fuel sloshing within aircraft tanks during flight is crucial for aircraft safety research. Traditional methods based on Navier-Stokes equations are computationally expensive. In this paper, we treat fluid motion as point cloud transformation and propose the first neural network method specifically designed for simulating fuel sloshing in aircraft. This model is also the deep learning model that is the first to be capable of stably modeling fluid particle dynamics in such complex scenarios. Our triangle feature fusion design achieves an optimal balance among fluid dynamics modeling, momentum conservation constraints, and global stability control. Additionally, we constructed the Fueltank dataset, the first dataset for aircraft fuel surface sloshing. It comprises 320,000 frames across four typical tank types and covers a wide range of flight maneuvers, including multi-directional rotations. We conducted comprehensive experiments on both our dataset and the take-off scenario of the aircraft. Compared to existing neural network-based fluid simulation algorithms, we significantly enhanced accuracy while maintaining high computational speed. Compared to traditional SPH methods, our speed improved approximately 10 times. Furthermore, compared to traditional fluid simulation software such as Flow3D, our computation speed increased by more than 300 times.

Title: p-Mean Regret for Stochastic Bandits

• Authors: Anand Krishna, Philips George John, Adarsh Barik, Vincent Y. F. Tan
• Subjects: cs.LG, cs.GT
• Abstract URL: https://arxiv.org/abs/2412.10751
• Pdf URL: https://arxiv.org/pdf/2412.10751
• Copy Paste: [[2412.10751]] p-Mean Regret for Stochastic Bandits(https://arxiv.org/abs/2412.10751)
• Keywords: fair
• Abstract: In this work, we extend the concept of the $p$-mean welfare objective from social choice theory (Moulin 2004) to study $p$-mean regret in stochastic multi-armed bandit problems. The $p$-mean regret, defined as the difference between the optimal mean among the arms and the $p$-mean of the expected rewards, offers a flexible framework for evaluating bandit algorithms, enabling algorithm designers to balance fairness and efficiency by adjusting the parameter $p$. Our framework encompasses both average cumulative regret and Nash regret as special cases. We introduce a simple, unified UCB-based algorithm (Explore-Then-UCB) that achieves novel $p$-mean regret bounds. Our algorithm consists of two phases: a carefully calibrated uniform exploration phase to initialize sample means, followed by the UCB1 algorithm of Auer, Cesa-Bianchi, and Fischer (2002). Under mild assumptions, we prove that our algorithm achieves a $p$-mean regret bound of $\tilde{O}\left(\sqrt{\frac{k}{T^{\frac{1}{2|p|}}}}\right)$ for all $p \leq -1$, where $k$ represents the number of arms and $T$ the time horizon. When $-1

Title: Explainable Fuzzy Neural Network with Multi-Fidelity Reinforcement Learning for Micro-Architecture Design Space Exploration

• Authors: Hanwei Fan, Ya Wang, Sicheng Li, Tingyuan Liang, Wei Zhang
• Subjects: cs.LG, cs.AR
• Abstract URL: https://arxiv.org/abs/2412.10754
• Pdf URL: https://arxiv.org/pdf/2412.10754
• Copy Paste: [[2412.10754]] Explainable Fuzzy Neural Network with Multi-Fidelity Reinforcement Learning for Micro-Architecture Design Space Exploration(https://arxiv.org/abs/2412.10754)
• Keywords: interpretability
• Abstract: With the continuous advancement of processors, modern micro-architecture designs have become increasingly complex. The vast design space presents significant challenges for human designers, making design space exploration (DSE) algorithms a significant tool for $\mu$-arch design. In recent years, efforts have been made in the development of DSE algorithms, and promising results have been achieved. However, the existing DSE algorithms, e.g., Bayesian Optimization and ensemble learning, suffer from poor interpretability, hindering designers' understanding of the decision-making process. To address this limitation, we propose utilizing Fuzzy Neural Networks to induce and summarize knowledge and insights from the DSE process, enhancing interpretability and controllability. Furthermore, to improve efficiency, we introduce a multi-fidelity reinforcement learning approach, which primarily conducts exploration using cheap but less precise data, thereby substantially diminishing the reliance on costly data. Experimental results show that our method achieves excellent results with a very limited sample budget and successfully surpasses the current state-of-the-art. Our DSE framework is open-sourced and available at this https URL\_MFRL\_ArchDSE/\ .

Title: Damage Assessment after Natural Disasters with UAVs: Semantic Feature Extraction using Deep Learning

• Authors: Nethmi S. Hewawiththi, M. Mahesha Viduranga, Vanodhya G. Warnasooriya, Tharindu Fernando, Himal A. Suraweera, Sridha Sridharan, Clinton Fookes
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10756
• Pdf URL: https://arxiv.org/pdf/2412.10756
• Copy Paste: [[2412.10756]] Damage Assessment after Natural Disasters with UAVs: Semantic Feature Extraction using Deep Learning(https://arxiv.org/abs/2412.10756)
• Keywords: extraction
• Abstract: Unmanned aerial vehicle-assisted disaster recovery missions have been promoted recently due to their reliability and flexibility. Machine learning algorithms running onboard significantly enhance the utility of UAVs by enabling real-time data processing and efficient decision-making, despite being in a resource-constrained environment. However, the limited bandwidth and intermittent connectivity make transmitting the outputs to ground stations challenging. This paper proposes a novel semantic extractor that can be adopted into any machine learning downstream task for identifying the critical data required for decision-making. The semantic extractor can be executed onboard which results in a reduction of data that needs to be transmitted to ground stations. We test the proposed architecture together with the semantic extractor on two publicly available datasets, FloodNet and RescueNet, for two downstream tasks: visual question answering and disaster damage level classification. Our experimental results demonstrate the proposed method maintains high accuracy across different downstream tasks while significantly reducing the volume of transmitted data, highlighting the effectiveness of our semantic extractor in capturing task-specific salient information.

Title: Optimizing Vision-Language Interactions Through Decoder-Only Models

• Authors: Kaito Tanaka, Benjamin Tan, Brian Wong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10758
• Pdf URL: https://arxiv.org/pdf/2412.10758
• Copy Paste: [[2412.10758]] Optimizing Vision-Language Interactions Through Decoder-Only Models(https://arxiv.org/abs/2412.10758)
• Keywords: robust
• Abstract: Vision-Language Models (VLMs) have emerged as key enablers for multimodal tasks, but their reliance on separate visual encoders introduces challenges in efficiency, scalability, and modality alignment. To address these limitations, we propose MUDAIF (Multimodal Unified Decoder with Adaptive Input Fusion), a decoder-only vision-language model that seamlessly integrates visual and textual inputs through a novel Vision-Token Adapter (VTA) and adaptive co-attention mechanism. By eliminating the need for a visual encoder, MUDAIF achieves enhanced efficiency, flexibility, and cross-modal understanding. Trained on a large-scale dataset of 45M image-text pairs, MUDAIF consistently outperforms state-of-the-art methods across multiple benchmarks, including VQA, image captioning, and multimodal reasoning tasks. Extensive analyses and human evaluations demonstrate MUDAIF's robustness, generalization capabilities, and practical usability, establishing it as a new standard in encoder-free vision-language models.

Title: Neural Network Meta Classifier: Improving the Reliability of Anomaly Segmentation

• Authors: Jurica Runtas, Tomislav Petkovic
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10765
• Pdf URL: https://arxiv.org/pdf/2412.10765
• Copy Paste: [[2412.10765]] Neural Network Meta Classifier: Improving the Reliability of Anomaly Segmentation(https://arxiv.org/abs/2412.10765)
• Keywords: interpretability, segmentation
• Abstract: Deep neural networks (DNNs) are a contemporary solution for semantic segmentation and are usually trained to operate on a predefined closed set of classes. In open-set environments, it is possible to encounter semantically unknown objects or anomalies. Road driving is an example of such an environment in which, from a safety standpoint, it is important to ensure that a DNN indicates it is operating outside of its learned semantic domain. One possible approach to anomaly segmentation is entropy maximization, which is paired with a logistic regression based post-processing step called meta classification, which is in turn used to improve the reliability of detection of anomalous pixels. We propose to substitute the logistic regression meta classifier with a more expressive lightweight fully connected neural network. We analyze advantages and drawbacks of the proposed neural network meta classifier and demonstrate its better performance over logistic regression. We also introduce the concept of informative out-of-distribution examples which we show to improve training results when using entropy maximization in practice. Finally, we discuss the loss of interpretability and show that the behavior of logistic regression and neural network is strongly correlated.

Title: VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation

• Authors: Saksham Singh Kushwaha, Yapeng Tian
• Subjects: cs.CV, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.10768
• Pdf URL: https://arxiv.org/pdf/2412.10768
• Copy Paste: [[2412.10768]] VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generation(https://arxiv.org/abs/2412.10768)
• Keywords: transformer
• Abstract: Recent advances in audio generation have focused on text-to-audio (T2A) and video-to-audio (V2A) tasks. However, T2A or V2A methods cannot generate holistic sounds (onscreen and off-screen). This is because T2A cannot generate sounds aligning with onscreen objects, while V2A cannot generate semantically complete (offscreen sounds missing). In this work, we address the task of holistic audio generation: given a video and a text prompt, we aim to generate both onscreen and offscreen sounds that are temporally synchronized with the video and semantically aligned with text and video. Previous approaches for joint text and video-to-audio generation often suffer from modality bias, favoring one modality over the other. To overcome this limitation, we introduce VinTAGe, a flow-based transformer model that jointly considers text and video to guide audio generation. Our framework comprises two key components: a Visual-Text Encoder and a Joint VT-SiT model. To reduce modality bias and improve generation quality, we employ pretrained uni-modal text-to-audio and video-to-audio generation models for additional guidance. Due to the lack of appropriate benchmarks, we also introduce VinTAGe-Bench, a dataset of 636 video-text-audio pairs containing both onscreen and offscreen sounds. Our comprehensive experiments on VinTAGe-Bench demonstrate that joint text and visual interaction is necessary for holistic audio generation. Furthermore, VinTAGe achieves state-of-the-art results on the VGGSound benchmark. Our source code and pre-trained models will be released. Demo is available at: this https URL.

Title: Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videos

• Authors: Xin Liu, Yaran Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10778
• Pdf URL: https://arxiv.org/pdf/2412.10778
• Copy Paste: [[2412.10778]] Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videos(https://arxiv.org/abs/2412.10778)
• Keywords: robust
• Abstract: Current advanced policy learning methodologies have demonstrated the ability to develop expert-level strategies when provided enough information. However, their requirements, including task-specific rewards, expert-labeled trajectories, and huge environmental interactions, can be expensive or even unavailable in many scenarios. In contrast, humans can efficiently acquire skills within a few trials and errors by imitating easily accessible internet video, in the absence of any other supervision. In this paper, we try to let machines replicate this efficient watching-and-learning process through Unsupervised Policy from Ensemble Self-supervised labeled Videos (UPESV), a novel framework to efficiently learn policies from videos without any other expert supervision. UPESV trains a video labeling model to infer the expert actions in expert videos, through several organically combined self-supervised tasks. Each task performs its own duties, and they together enable the model to make full use of both expert videos and reward-free interactions for advanced dynamics understanding and robust prediction. Simultaneously, UPESV clones a policy from the labeled expert videos, in turn collecting environmental interactions for self-supervised tasks. After a sample-efficient and unsupervised (i.e., reward-free) training process, an advanced video-imitated policy is obtained. Extensive experiments in sixteen challenging procedurally-generated environments demonstrate that the proposed UPESV achieves state-of-the-art few-shot policy learning (outperforming five current advanced baselines on 12/16 tasks) without exposure to any other supervision except videos. Detailed analysis is also provided, verifying the necessity of each self-supervised task employed in UPESV.

Title: Video Diffusion Transformers are In-Context Learners

• Authors: Zhengcong Fei, Di Qiu, Changqian Yu, Debang Li, Mingyuan Fan, Xiang Wen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10783
• Pdf URL: https://arxiv.org/pdf/2412.10783
• Copy Paste: [[2412.10783]] Video Diffusion Transformers are In-Context Learners(https://arxiv.org/abs/2412.10783)
• Keywords: diffusion, transformer
• Abstract: This paper investigates a solution for enabling in-context capabilities of video diffusion transformers, with minimal tuning required for activation. Specifically, we propose a simple pipeline to leverage in-context generation: ($\textbf{i}$) concatenate videos along spacial or time dimension, ($\textbf{ii}$) jointly caption multi-scene video clips from one source, and ($\textbf{iii}$) apply task-specific fine-tuning using carefully curated small datasets. Through a series of diverse controllable tasks, we demonstrate qualitatively that existing advanced text-to-video models can effectively perform in-context generation. Notably, it allows for the creation of consistent multi-scene videos exceeding 30 seconds in duration, without additional computational overhead. Importantly, this method requires no modifications to the original models, results in high-fidelity video outputs that better align with prompt specifications and maintain role consistency. Our framework presents a valuable tool for the research community and offers critical insights for advancing product-level controllable video generation systems. The data, code, and model weights are publicly available at: \url{this https URL}.

Title: StyleDiT: A Unified Framework for Diverse Child and Partner Faces Synthesis with Style Latent Diffusion Transformer

• Authors: Pin-Yen Chiu, Dai-Jie Wu, Po-Hsun Chu, Chia-Hsuan Hsu, Hsiang-Chen Chiu, Chih-Yu Wang, Jun-Cheng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10785
• Pdf URL: https://arxiv.org/pdf/2412.10785
• Copy Paste: [[2412.10785]] StyleDiT: A Unified Framework for Diverse Child and Partner Faces Synthesis with Style Latent Diffusion Transformer(https://arxiv.org/abs/2412.10785)
• Keywords: diffusion, transformer
• Abstract: Kinship face synthesis is a challenging problem due to the scarcity and low quality of the available kinship data. Existing methods often struggle to generate descendants with both high diversity and fidelity while precisely controlling facial attributes such as age and gender. To address these issues, we propose the Style Latent Diffusion Transformer (StyleDiT), a novel framework that integrates the strengths of StyleGAN with the diffusion model to generate high-quality and diverse kinship faces. In this framework, the rich facial priors of StyleGAN enable fine-grained attribute control, while our conditional diffusion model is used to sample a StyleGAN latent aligned with the kinship relationship of conditioning images by utilizing the advantage of modeling complex kinship relationship distribution. StyleGAN then handles latent decoding for final face generation. Additionally, we introduce the Relational Trait Guidance (RTG) mechanism, enabling independent control of influencing conditions, such as each parent's facial image. RTG also enables a fine-grained adjustment between the diversity and fidelity in synthesized faces. Furthermore, we extend the application to an unexplored domain: predicting a partner's facial images using a child's image and one parent's image within the same framework. Extensive experiments demonstrate that our StyleDiT outperforms existing methods by striking an excellent balance between generating diverse and high-fidelity kinship faces.

Title: Optimizing Few-Step Sampler for Diffusion Probabilistic Model

• Authors: Jen-Yuan Huang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10786
• Pdf URL: https://arxiv.org/pdf/2412.10786
• Copy Paste: [[2412.10786]] Optimizing Few-Step Sampler for Diffusion Probabilistic Model(https://arxiv.org/abs/2412.10786)
• Keywords: diffusion
• Abstract: Diffusion Probabilistic Models (DPMs) have demonstrated exceptional capability of generating high-quality and diverse images, but their practical application is hindered by the intensive computational cost during inference. The DPM generation process requires solving a Probability-Flow Ordinary Differential Equation (PF-ODE), which involves discretizing the integration domain into intervals for numerical approximation. This corresponds to the sampling schedule of a diffusion ODE solver, and we notice the solution from a first-order solver can be expressed as a convex combination of model outputs at all scheduled time-steps. We derive an upper bound for the discretization error of the sampling schedule, which can be efficiently optimized with Monte-Carlo estimation. Building on these theoretical results, we purpose a two-phase alternating optimization algorithm. In Phase-1, the sampling schedule is optimized for the pre-trained DPM; in Phase-2, the DPM further tuned on the selected time-steps. Experiments on a pre-trained DPM for ImageNet64 dataset demonstrate the purposed method consistently improves the baseline across various number of sampling steps.

Title: Reliable and superior elliptic Fourier descriptor normalization and its application software ElliShape with efficient image processing

• Authors: Hui Wu (1,2,3,4), Jia-Jie Yang (1,3,4), Chao-Qun Li (5), Jin-Hua Ran (2,4,6), Ren-Hua Peng (6,7), Xiao-Quan Wang (1,2,3,4,6) ((1) Big Data and AI Biodiversity Conservation Research Center, Institute of Botany, Chinese Academy of Sciences, Beijing, China (2) State Key Laboratory of Plant Diversity and Specialty Crops and Key Laboratory of Systematic and Evolutionary Botany, Institute of Botany, Chinese Academy of Sciences, Beijing, China (3) Plant Science Data Center, Chinese Academy of Sciences, Beijing, China (4) China National Botanical Garden, Beijing, China (5) School of Life Sciences, Qilu Normal University, Jinan, China (6) University of Chinese Academy of Sciences, Beijing, China (7) Key Laboratory of Noise and Vibration Control, Institute of Acoustics, Chinese Academy of Sciences, Beijing, China)
• Subjects: cs.CV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2412.10795
• Pdf URL: https://arxiv.org/pdf/2412.10795
• Copy Paste: [[2412.10795]] Reliable and superior elliptic Fourier descriptor normalization and its application software ElliShape with efficient image processing(https://arxiv.org/abs/2412.10795)
• Keywords: robust, extraction
• Abstract: Elliptic Fourier analysis (EFA) is a powerful tool for shape analysis, which is often employed in geometric morphometrics. However, the normalization of elliptic Fourier descriptors has persistently posed challenges in obtaining unique results in basic contour transformations, requiring extensive manual alignment. Additionally, contemporary contour/outline extraction methods often struggle to handle complex digital images. Here, we reformulated the procedure of EFDs calculation to improve computational efficiency and introduced a novel approach for EFD normalization, termed true EFD normalization, which remains invariant under all basic contour transformations. These improvements are crucial for processing large sets of contour curves collected from different platforms with varying transformations. Based on these improvements, we developed ElliShape, a user-friendly software. Particularly, the improved contour/outline extraction employs an interactive approach that combines automatic contour generation for efficiency with manual correction for essential modifications and refinements. We evaluated ElliShape's stability, robustness, and ease of use by comparing it with existing software using standard datasets. ElliShape consistently produced reliable reconstructed shapes and normalized EFD values across different contours and transformations, and it demonstrated superior performance in visualization and efficient processing of various digital images for contour this http URL output annotated images and EFDs could be utilized in deep learning-based data training, thereby advancing artificial intelligence in botany and offering innovative solutions for critical challenges in biodiversity conservation, species classification, ecosystem function assessment, and related critical issues.

Title: Medical Manifestation-Aware De-Identification

• Authors: Yuan Tian, Shuo Wang, Guangtao Zhai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10804
• Pdf URL: https://arxiv.org/pdf/2412.10804
• Copy Paste: [[2412.10804]] Medical Manifestation-Aware De-Identification(https://arxiv.org/abs/2412.10804)
• Keywords: privacy
• Abstract: Face de-identification (DeID) has been widely studied for common scenes, but remains under-researched for medical scenes, mostly due to the lack of large-scale patient face datasets. In this paper, we release MeMa, consisting of over 40,000 photo-realistic patient faces. MeMa is re-generated from massive real patient photos. By carefully modulating the generation and data-filtering procedures, MeMa avoids breaching real patient privacy, while ensuring rich and plausible medical manifestations. We recruit expert clinicians to annotate MeMa with both coarse- and fine-grained labels, building the first medical-scene DeID benchmark. Additionally, we propose a baseline approach for this new medical-aware DeID task, by integrating data-driven medical semantic priors into the DeID procedure. Despite its conciseness and simplicity, our approach substantially outperforms previous ones. Dataset is available at this https URL.

Title: Are Language Models Agnostic to Linguistically Grounded Perturbations? A Case Study of Indic Languages

• Authors: Poulami Ghosh, Raj Dabre, Pushpak Bhattacharyya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10805
• Pdf URL: https://arxiv.org/pdf/2412.10805
• Copy Paste: [[2412.10805]] Are Language Models Agnostic to Linguistically Grounded Perturbations? A Case Study of Indic Languages(https://arxiv.org/abs/2412.10805)
• Keywords: attack
• Abstract: Pre-trained language models (PLMs) are known to be susceptible to perturbations to the input text, but existing works do not explicitly focus on linguistically grounded attacks, which are subtle and more prevalent in nature. In this paper, we study whether PLMs are agnostic to linguistically grounded attacks or not. To this end, we offer the first study addressing this, investigating different Indic languages and various downstream tasks. Our findings reveal that although PLMs are susceptible to linguistic perturbations, when compared to non-linguistic attacks, PLMs exhibit a slightly lower susceptibility to linguistic attacks. This highlights that even constrained attacks are effective. Moreover, we investigate the implications of these outcomes across a range of languages, encompassing diverse language families and different scripts.

Title: Towards Action Hijacking of Large Language Model-based Agent

• Authors: Yuyang Zhang, Kangjie Chen, Xudong Jiang, Yuxiang Sun, Run Wang, Lina Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.10807
• Pdf URL: https://arxiv.org/pdf/2412.10807
• Copy Paste: [[2412.10807]] Towards Action Hijacking of Large Language Model-based Agent(https://arxiv.org/abs/2412.10807)
• Keywords: attack, large language model
• Abstract: In the past few years, intelligent agents powered by large language models (LLMs) have achieved remarkable progress in performing complex tasks. These LLM-based agents receive queries as tasks and decompose them into various subtasks via the equipped LLMs to guide the action of external entities (\eg{}, tools, AI-agents) to answer the questions from users. Empowered by their exceptional capabilities of understanding and problem-solving, they are widely adopted in labor-intensive sectors including healthcare, finance, code completion, \etc{} At the same time, there are also concerns about the potential misuse of these agents, prompting the built-in safety guards from service providers. To circumvent the built-in guidelines, the prior studies proposed a multitude of attacks including memory poisoning, jailbreak, and prompt injection. These studies often fail to maintain effectiveness across safety filters employed by agents due to the restricted privileges and the harmful semantics in queries. In this paper, we introduce \Name, a novel hijacking attack to manipulate the action plans of black-box agent system. \Name first collects the action-aware memory through prompt theft from long-term memory. It then leverages the internal memory retrieval mechanism of the agent to provide an erroneous context. The huge gap between the latent spaces of the retriever and safety filters allows our method to bypass the detection easily. Extensive experimental results demonstrate the effectiveness of our apporach (\eg{}, 99.67\% ASR). Besides, our approach achieved an average bypass rate of 92.7\% for safety filters.

Title: Diffusion-based Method for Satellite Pattern-of-Life Identification

• Authors: Yongchao Ye, Xinting Zhu, Xuejin Shen, Xiaoyu Chen, Lishuai Li, S. Joe Qin
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2412.10814
• Pdf URL: https://arxiv.org/pdf/2412.10814
• Copy Paste: [[2412.10814]] Diffusion-based Method for Satellite Pattern-of-Life Identification(https://arxiv.org/abs/2412.10814)
• Keywords: robust, diffusion
• Abstract: Satellite pattern-of-life (PoL) identification is crucial for space safety and satellite monitoring, involving the analysis of typical satellite behaviors such as station-keeping, drift, etc. However, existing PoL identification methods remain underdeveloped due to the complexity of aerospace systems, variability in satellite behaviors, and fluctuating observation sampling rates. In a first attempt, we developed a domain expertise-informed machine learning method (Expert-ML) to combine satellite orbital movement knowledge and machine learning models. The Expert-ML method achieved high accuracy results in simulation data and real-world data with normal sampling rate. However, this approach lacks of generality as it requires domain expertise and its performance degraded significantly when data sampling rate varied. To achieve generality, we propose a novel diffusion-based PoL identification method. Distinct from prior approaches, the proposed method leverages a diffusion model to achieve end-to-end identification without manual refinement or domain-specific knowledge. Specifically, we employ a multivariate time-series encoder to capture hidden representations of satellite positional data. The encoded features are subsequently incorporated as conditional information in the denoising process to generate PoL labels. Through experimentation across real-world satellite settings, our proposed diffusion-based method demonstrates its high identification quality and provides a robust solution even with reduced data sampling rates, indicating its great potential in practical satellite behavior pattern identification, tracking and related mission deployment.

Title: Hyper-Fusion Network for Semi-Automatic Segmentation of Skin Lesions

• Authors: Lei Bi, Michael Fulham, Jinman Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10816
• Pdf URL: https://arxiv.org/pdf/2412.10816
• Copy Paste: [[2412.10816]] Hyper-Fusion Network for Semi-Automatic Segmentation of Skin Lesions(https://arxiv.org/abs/2412.10816)
• Keywords: segmentation
• Abstract: Automatic skin lesion segmentation methods based on fully convolutional networks (FCNs) are regarded as the state-of-the-art for accuracy. When there are, however, insufficient training data to cover all the variations in skin lesions, where lesions from different patients may have major differences in size/shape/texture, these methods failed to segment the lesions that have image characteristics, which are less common in the training datasets. FCN-based semi-automatic segmentation methods, which fuse user-inputs with high-level semantic image features derived from FCNs offer an ideal complement to overcome limitations of automatic segmentation methods. These semi-automatic methods rely on the automated state-of-the-art FCNs coupled with user-inputs for refinements, and therefore being able to tackle challenging skin lesions. However, there are a limited number of FCN-based semi-automatic segmentation methods and all these methods focused on early-fusion, where the first few convolutional layers are used to fuse image features and user-inputs and then derive fused image features for segmentation. For early-fusion based methods, because the user-input information can be lost after the first few convolutional layers, consequently, the user-input information will have limited guidance and constraint in segmenting the challenging skin lesions with inhomogeneous textures and fuzzy boundaries. Hence, in this work, we introduce a hyper-fusion network (HFN) to fuse the extracted user-inputs and image features over multiple stages. We separately extract complementary features which then allows for an iterative use of user-inputs along all the fusion stages to refine the segmentation. We evaluated our HFN on ISIC 2017, ISIC 2016 and PH2 datasets, and our results show that the HFN is more accurate and generalizable than the state-of-the-art methods.

Title: FinGPT: Enhancing Sentiment-Based Stock Movement Prediction with Dissemination-Aware and Context-Enriched LLMs

• Authors: Yixuan Liang, Yuncong Liu, Boyu Zhang, Christina Dan Wang, Hongyang Yang
• Subjects: cs.CL, cs.LG, q-fin.CP, q-fin.TR
• Abstract URL: https://arxiv.org/abs/2412.10823
• Pdf URL: https://arxiv.org/pdf/2412.10823
• Copy Paste: [[2412.10823]] FinGPT: Enhancing Sentiment-Based Stock Movement Prediction with Dissemination-Aware and Context-Enriched LLMs(https://arxiv.org/abs/2412.10823)
• Keywords: large language model
• Abstract: Financial sentiment analysis is crucial for understanding the influence of news on stock prices. Recently, large language models (LLMs) have been widely adopted for this purpose due to their advanced text analysis capabilities. However, these models often only consider the news content itself, ignoring its dissemination, which hampers accurate prediction of short-term stock movements. Additionally, current methods often lack sufficient contextual data and explicit instructions in their prompts, limiting LLMs' ability to interpret news. In this paper, we propose a data-driven approach that enhances LLM-powered sentiment-based stock movement predictions by incorporating news dissemination breadth, contextual data, and explicit instructions. We cluster recent company-related news to assess its reach and influence, enriching prompts with more specific data and precise instructions. This data is used to construct an instruction tuning dataset to fine-tune an LLM for predicting short-term stock price movements. Our experimental results show that our approach improves prediction accuracy by 8\% compared to existing methods.

Title: Diffusion Model from Scratch

• Authors: Wang Zhen, Dong Yunyun
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10824
• Pdf URL: https://arxiv.org/pdf/2412.10824
• Copy Paste: [[2412.10824]] Diffusion Model from Scratch(https://arxiv.org/abs/2412.10824)
• Keywords: diffusion, generative
• Abstract: Diffusion generative models are currently the most popular generative models. However, their underlying modeling process is quite complex, and starting directly with the seminal paper Denoising Diffusion Probability Model (DDPM) can be challenging. This paper aims to assist readers in building a foundational understanding of generative models by tracing the evolution from VAEs to DDPM through detailed mathematical derivations and a problem-oriented analytical approach. It also explores the core ideas and improvement strategies of current mainstream methodologies, providing guidance for undergraduate and graduate students interested in learning about diffusion models.

Title: Rethinking Chain-of-Thought from the Perspective of Self-Training

• Authors: Zongqian Wu, Baoduo Xu, Ruochen Cui, Mengmeng Zhan, Xiaofeng Zhu, Lei Feng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10827
• Pdf URL: https://arxiv.org/pdf/2412.10827
• Copy Paste: [[2412.10827]] Rethinking Chain-of-Thought from the Perspective of Self-Training(https://arxiv.org/abs/2412.10827)
• Keywords: large language model
• Abstract: Chain-of-thought (CoT) reasoning has emerged as an effective approach for activating latent capabilities in large language models (LLMs). We observe that CoT shares significant similarities with self-training in terms of their learning processes. Motivated by these parallels, this paper explores the underlying relationship between CoT and self-training, demonstrating how insights from self-training can enhance CoT performance. Specifically, our study first reveals that CoT, like self-training, follows the principle of semantic entropy minimization. Leveraging this insight, we propose a novel CoT framework that incorporates two key components: (i) a task-specific prompt module designed to guide LLMs in generating high-quality initial reasoning processes, and (ii) an adaptive reasoning iteration module for progressively refining the reasoning process.

Title: Unbiased General Annotated Dataset Generation

• Authors: Dengyang Jiang, Haoyu Wang, Lei Zhang, Wei Wei, Guang Dai, Mengmeng Wang, Jingdong Wang, Yanning Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10831
• Pdf URL: https://arxiv.org/pdf/2412.10831
• Copy Paste: [[2412.10831]] Unbiased General Annotated Dataset Generation(https://arxiv.org/abs/2412.10831)
• Keywords: diffusion
• Abstract: Pre-training backbone networks on a general annotated dataset (e.g., ImageNet) that comprises numerous manually collected images with category annotations has proven to be indispensable for enhancing the generalization capacity of downstream visual tasks. However, those manually collected images often exhibit bias, which is non-transferable across either categories or domains, thus causing the model's generalization capacity degeneration. To mitigate this problem, we present an unbiased general annotated dataset generation framework (ubGen). Instead of expensive manual collection, we aim at directly generating unbiased images with category annotations. To achieve this goal, we propose to leverage the advantage of a multimodal foundation model (e.g., CLIP), in terms of aligning images in an unbiased semantic space defined by language. Specifically, we develop a bi-level semantic alignment loss, which not only forces all generated images to be consistent with the semantic distribution of all categories belonging to the target dataset in an adversarial learning manner, but also requires each generated image to match the semantic description of its category name. In addition, we further cast an existing image quality scoring model into a quality assurance loss to preserve the quality of the generated image. By leveraging these two loss functions, we can obtain an unbiased image generation model by simply fine-tuning a pre-trained diffusion model using only all category names in the target dataset as input. Experimental results confirm that, compared with the manually labeled dataset or other synthetic datasets, the utilization of our generated unbiased datasets leads to stable generalization capacity enhancement of different backbone networks across various tasks, especially in tasks where the manually labeled samples are scarce.

Title: SegACIL: Solving the Stability-Plasticity Dilemma in Class-Incremental Semantic Segmentation

• Authors: Jiaxu Li, Songning Lai, Rui Li, Di Fang, Kejia Fan, Jianheng Tang, Yuhan Zhao, Rongchang Zhao, Dongzhan Zhou, Yutao Yue, Huiping Zhuang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10834
• Pdf URL: https://arxiv.org/pdf/2412.10834
• Copy Paste: [[2412.10834]] SegACIL: Solving the Stability-Plasticity Dilemma in Class-Incremental Semantic Segmentation(https://arxiv.org/abs/2412.10834)
• Keywords: robust, segmentation
• Abstract: While deep learning has made remarkable progress in recent years, models continue to struggle with catastrophic forgetting when processing continuously incoming data. This issue is particularly critical in continual learning, where the balance between retaining prior knowledge and adapting to new information-known as the stability-plasticity dilemma-remains a significant challenge. In this paper, we propose SegACIL, a novel continual learning method for semantic segmentation based on a linear closed-form solution. Unlike traditional methods that require multiple epochs for training, SegACIL only requires a single epoch, significantly reducing computational costs. Furthermore, we provide a theoretical analysis demonstrating that SegACIL achieves performance on par with joint learning, effectively retaining knowledge from previous data which makes it to keep both stability and plasticity at the same time. Extensive experiments on the Pascal VOC2012 dataset show that SegACIL achieves superior performance in the sequential, disjoint, and overlap settings, offering a robust solution to the challenges of class-incremental semantic segmentation. Code is available at this https URL.

Title: Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning

• Authors: Hai-Ming Xu, Qi Chen, Lei Wang, Lingqiao Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10840
• Pdf URL: https://arxiv.org/pdf/2412.10840
• Copy Paste: [[2412.10840]] Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning(https://arxiv.org/abs/2412.10840)
• Keywords: large language model
• Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have generated significant interest in their ability to autonomously interact with and interpret Graphical User Interfaces (GUIs). A major challenge in these systems is grounding-accurately identifying critical GUI components such as text or icons based on a GUI image and a corresponding text query. Traditionally, this task has relied on fine-tuning MLLMs with specialized training data to predict component locations directly. However, in this paper, we propose a novel Tuning-free Attention-driven Grounding (TAG) method that leverages the inherent attention patterns in pretrained MLLMs to accomplish this task without the need for additional fine-tuning. Our method involves identifying and aggregating attention maps from specific tokens within a carefully constructed query prompt. Applied to MiniCPM-Llama3-V 2.5, a state-of-the-art MLLM, our tuning-free approach achieves performance comparable to tuning-based methods, with notable success in text localization. Additionally, we demonstrate that our attention map-based grounding technique significantly outperforms direct localization predictions from MiniCPM-Llama3-V 2.5, highlighting the potential of using attention maps from pretrained MLLMs and paving the way for future innovations in this domain.

Title: Detecting Activities of Daily Living in Egocentric Video to Contextualize Hand Use at Home in Outpatient Neurorehabilitation Settings

• Authors: Adesh Kadambi, José Zariffa
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.10846
• Pdf URL: https://arxiv.org/pdf/2412.10846
• Copy Paste: [[2412.10846]] Detecting Activities of Daily Living in Egocentric Video to Contextualize Hand Use at Home in Outpatient Neurorehabilitation Settings(https://arxiv.org/abs/2412.10846)
• Keywords: robust
• Abstract: Wearable egocentric cameras and machine learning have the potential to provide clinicians with a more nuanced understanding of patient hand use at home after stroke and spinal cord injury (SCI). However, they require detailed contextual information (i.e., activities and object interactions) to effectively interpret metrics and meaningfully guide therapy planning. We demonstrate that an object-centric approach, focusing on what objects patients interact with rather than how they move, can effectively recognize Activities of Daily Living (ADL) in real-world rehabilitation settings. We evaluated our models on a complex dataset collected in the wild comprising 2261 minutes of egocentric video from 16 participants with impaired hand function. By leveraging pre-trained object detection and hand-object interaction models, our system achieves robust performance across different impairment levels and environments, with our best model achieving a mean weighted F1-score of 0.78 +/- 0.12 and maintaining an F1-score > 0.5 for all participants using leave-one-subject-out cross validation. Through qualitative analysis, we observe that this approach generates clinically interpretable information about functional object use while being robust to patient-specific movement variations, making it particularly suitable for rehabilitation contexts with prevalent upper limb impairment.

Title: Large Language Models for Medical Forecasting -- Foresight 2

• Authors: Zeljko Kraljevic, Joshua Au Yeung, Daniel Bean, James Teo, Richard J. Dobson
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10848
• Pdf URL: https://arxiv.org/pdf/2412.10848
• Copy Paste: [[2412.10848]] Large Language Models for Medical Forecasting -- Foresight 2(https://arxiv.org/abs/2412.10848)
• Keywords: large language model
• Abstract: Foresight 2 (FS2) is a large language model fine-tuned on hospital data for modelling patient timelines (GitHub 'removed for anon'). It can understand patients' clinical notes and predict SNOMED codes for a wide range of biomedical use cases, including diagnosis suggestions, risk forecasting, and procedure and medication recommendations. FS2 is trained on the free text portion of the MIMIC-III dataset, firstly through extracting biomedical concepts and then creating contextualised patient timelines, upon which the model is then fine-tuned. The results show significant improvement over the previous state-of-the-art for the next new biomedical concept prediction (P/R - 0.73/0.66 vs 0.52/0.32) and a similar improvement specifically for the next new disorder prediction (P/R - 0.69/0.62 vs 0.46/0.25). Finally, on the task of risk forecast, we compare our model to GPT-4-turbo (and a range of open-source biomedical LLMs) and show that FS2 performs significantly better on such tasks (P@5 - 0.90 vs 0.65). This highlights the need to incorporate hospital data into LLMs and shows that small models outperform much larger ones when fine-tuned on high-quality, specialised data.

Title: Improving Graph Neural Networks via Adversarial Robustness Evaluation

• Authors: Yongyu Wang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.10850
• Pdf URL: https://arxiv.org/pdf/2412.10850
• Copy Paste: [[2412.10850]] Improving Graph Neural Networks via Adversarial Robustness Evaluation(https://arxiv.org/abs/2412.10850)
• Keywords: robust
• Abstract: Graph Neural Networks (GNNs) are currently one of the most powerful types of neural network architectures. Their advantage lies in the ability to leverage both the graph topology, which represents the relationships between samples, and the features of the samples themselves. However, the given graph topology often contains noisy edges, and GNNs are vulnerable to noise in the graph structure. This issue remains unresolved. In this paper, we propose using adversarial robustness evaluation to select a small subset of robust nodes that are less affected by noise. We then only feed the features of these robust nodes, along with the KNN graph constructed from these nodes, into the GNN for classification. Additionally, we compute the centroids for each class. For the remaining non-robust nodes, we assign them to the class whose centroid is closest to them. Experimental results show that this method significantly improves the accuracy of GNNs.

Title: RWKV-edge: Deeply Compressed RWKV for Resource-Constrained Devices

• Authors: Wonkyo Choe, Yangfeng Ji, Felix Lin
• Subjects: cs.LG, cs.PF
• Abstract URL: https://arxiv.org/abs/2412.10856
• Pdf URL: https://arxiv.org/pdf/2412.10856
• Copy Paste: [[2412.10856]] RWKV-edge: Deeply Compressed RWKV for Resource-Constrained Devices(https://arxiv.org/abs/2412.10856)
• Keywords: transformer
• Abstract: To deploy LLMs on resource-contained platforms such as mobile robotics and wearables, non-transformers LLMs have achieved major breakthroughs. Recently, a novel RNN-based LLM family, Repentance Weighted Key Value (RWKV) models have shown promising results in text generation on resource-constrained devices thanks to their computational efficiency. However, these models remain too large to be deployed on embedded devices due to their high parameter count. In this paper, we propose an efficient suite of compression techniques, tailored to the RWKV architecture. These techniques include low-rank approximation, sparsity predictors, and clustering head, designed to align with the model size. Our methods compress the RWKV models by 4.95--3.8x with only 2.95pp loss in accuracy.

Title: CRENER: A Character Relation Enhanced Chinese NER Model

• Authors: Yaqiong Qiao, Shixuan Peng
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.10858
• Pdf URL: https://arxiv.org/pdf/2412.10858
• Copy Paste: [[2412.10858]] CRENER: A Character Relation Enhanced Chinese NER Model(https://arxiv.org/abs/2412.10858)
• Keywords: extraction, transformer
• Abstract: Chinese Named Entity Recognition (NER) is an important task in information extraction, which has a significant impact on downstream applications. Due to the lack of natural separators in Chinese, previous NER methods mostly relied on external dictionaries to enrich the semantic and boundary information of Chinese words. However, such methods may introduce noise that affects the accuracy of named entity recognition. To this end, we propose a character relation enhanced Chinese NER model (CRENER). This model defines four types of tags that reflect the relationships between characters, and proposes a fine-grained modeling of the relationships between characters based on three types of relationships: adjacency relations between characters, relations between characters and tags, and relations between tags, to more accurately identify entity boundaries and improve Chinese NER accuracy. Specifically, we transform the Chinese NER task into a character-character relationship classification task, ensuring the accuracy of entity boundary recognition through joint modeling of relation tags. To enhance the model's ability to understand contextual information, WRENER further constructed an adapted transformer encoder that combines unscaled direction-aware and distance-aware masked self-attention mechanisms. Moreover, a relationship representation enhancement module was constructed to model predefined relationship tags, effectively mining the relationship representations between characters and tags. Experiments conducted on four well-known Chinese NER benchmark datasets have shown that the proposed model outperforms state-of-the-art baselines. The ablation experiment also demonstrated the effectiveness of the proposed model.

Title: Heterogeneous Graph Transformer for Multiple Tiny Object Tracking in RGB-T Videos

• Authors: Qingyu Xu, Longguang Wang, Weidong Sheng, Yingqian Wang, Chao Xiao, Chao Ma, Wei An
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10861
• Pdf URL: https://arxiv.org/pdf/2412.10861
• Copy Paste: [[2412.10861]] Heterogeneous Graph Transformer for Multiple Tiny Object Tracking in RGB-T Videos(https://arxiv.org/abs/2412.10861)
• Keywords: transformer
• Abstract: Tracking multiple tiny objects is highly challenging due to their weak appearance and limited features. Existing multi-object tracking algorithms generally focus on single-modality scenes, and overlook the complementary characteristics of tiny objects captured by multiple remote sensors. To enhance tracking performance by integrating complementary information from multiple sources, we propose a novel framework called {HGT-Track (Heterogeneous Graph Transformer based Multi-Tiny-Object Tracking)}. Specifically, we first employ a Transformer-based encoder to embed images from different modalities. Subsequently, we utilize Heterogeneous Graph Transformer to aggregate spatial and temporal information from multiple modalities to generate detection and tracking features. Additionally, we introduce a target re-detection module (ReDet) to ensure tracklet continuity by maintaining consistency across different modalities. Furthermore, this paper introduces the first benchmark VT-Tiny-MOT (Visible-Thermal Tiny Multi-Object Tracking) for RGB-T fused multiple tiny object tracking. Extensive experiments are conducted on VT-Tiny-MOT, and the results have demonstrated the effectiveness of our method. Compared to other state-of-the-art methods, our method achieves better performance in terms of MOTA (Multiple-Object Tracking Accuracy) and ID-F1 score. The code and dataset will be made available at this https URL.

Title: Fully Test-time Adaptation for Tabular Data

• Authors: Zhi Zhou, Kun-Yang Yu, Lan-Zhe Guo, Yu-Feng Li
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.10871
• Pdf URL: https://arxiv.org/pdf/2412.10871
• Copy Paste: [[2412.10871]] Fully Test-time Adaptation for Tabular Data(https://arxiv.org/abs/2412.10871)
• Keywords: robust
• Abstract: Tabular data plays a vital role in various real-world scenarios and finds extensive applications. Although recent deep tabular models have shown remarkable success, they still struggle to handle data distribution shifts, leading to performance degradation when testing distributions change. To remedy this, a robust tabular model must adapt to generalize to unknown distributions during testing. In this paper, we investigate the problem of fully test-time adaptation (FTTA) for tabular data, where the model is adapted using only the testing data. We identify three key challenges: the existence of label and covariate distribution shifts, the lack of effective data augmentation, and the sensitivity of adaptation, which render existing FTTA methods ineffective for tabular data. To this end, we propose the Fully Test-time Adaptation for Tabular data, namely FTAT, which enables FTTA methods to robustly optimize the label distribution of predictions, adapt to shifted covariate distributions, and suit a variety of tasks and models effectively. We conduct comprehensive experiments on six benchmark datasets, which are evaluated using three metrics. The experimental results demonstrate that FTAT outperforms state-of-the-art methods by a margin.

Title: IntelEX: A LLM-driven Attack-level Threat Intelligence Extraction Framework

• Authors: Ming Xu, Hongtai Wang, Jiahao Liu, Yun Lin, Chenyang Xu Yingshi Liu, Hoon Wei Lim, Jin Song Dong
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.10872
• Pdf URL: https://arxiv.org/pdf/2412.10872
• Copy Paste: [[2412.10872]] IntelEX: A LLM-driven Attack-level Threat Intelligence Extraction Framework(https://arxiv.org/abs/2412.10872)
• Keywords: security, attack, extraction
• Abstract: To combat increasingly sophisticated cyberattacks, a common practice is to transform unstructured cyber threat intelligence (CTI) reports into structured intelligence, facilitating threat-focused security tasks such as summarizing detection rules or simulating attack scenarios for red team exercises.

Title: Adaptive Quantization Resolution and Power Control for Federated Learning over Cell-free Networks

• Authors: Afsaneh Mahmoudi, Emil Björnson
• Subjects: cs.LG, cs.NI, eess.SP
• Abstract URL: https://arxiv.org/abs/2412.10878
• Pdf URL: https://arxiv.org/pdf/2412.10878
• Copy Paste: [[2412.10878]] Adaptive Quantization Resolution and Power Control for Federated Learning over Cell-free Networks(https://arxiv.org/abs/2412.10878)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) is a distributed learning framework where users train a global model by exchanging local model updates with a server instead of raw datasets, preserving data privacy and reducing communication overhead. However, the latency grows with the number of users and the model size, impeding the successful FL over traditional wireless networks with orthogonal access. Cell-free massive multiple-input multipleoutput (CFmMIMO) is a promising solution to serve numerous users on the same time/frequency resource with similar rates. This architecture greatly reduces uplink latency through spatial multiplexing but does not take application characteristics into account. In this paper, we co-optimize the physical layer with the FL application to mitigate the straggler effect. We introduce a novel adaptive mixed-resolution quantization scheme of the local gradient vector updates, where only the most essential entries are given high resolution. Thereafter, we propose a dynamic uplink power control scheme to manage the varying user rates and mitigate the straggler effect. The numerical results demonstrate that the proposed method achieves test accuracy comparable to classic FL while reducing communication overhead by at least 93% on the CIFAR-10, CIFAR-100, and Fashion-MNIST datasets. We compare our methods against AQUILA, Top-q, and LAQ, using the max-sum rate and Dinkelbach power control schemes. Our approach reduces the communication overhead by 75% and achieves 10% higher test accuracy than these benchmarks within a constrained total latency budget.

Title: Zigzag Diffusion Sampling: The Path to Success Is Zigzag

• Authors: Lichen Bai, Shitong Shao, Zikai Zhou, Zipeng Qi, Zhiqiang Xu, Haoyi Xiong, Zeke Xie
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10891
• Pdf URL: https://arxiv.org/pdf/2412.10891
• Copy Paste: [[2412.10891]] Zigzag Diffusion Sampling: The Path to Success Is Zigzag(https://arxiv.org/abs/2412.10891)
• Keywords: diffusion, transformer, generative
• Abstract: Diffusion models, the most popular generative paradigm so far, can inject conditional information into the generation path to guide the latent towards desired directions. However, existing text-to-image diffusion models often fail to maintain high image quality and high prompt-image alignment for those challenging prompts. To mitigate this issue and enhance existing pretrained diffusion models, we mainly made three contributions in this paper. First, we theoretically and empirically demonstrate that the conditional guidance gap between the denoising and inversion processes captures prompt-related semantic information. Second, motivated by theoretical analysis, we derive Zigzag Diffusion Sampling (Z-Sampling), a novel sampling method that leverages the guidance gap to accumulate semantic information step-by-step throughout the entire generation process, leading to improved sampling results. Moreover, as a plug-and-play method, Z-Sampling can be generally applied to various diffusion models (e.g., accelerated ones and Transformer-based ones) with very limited coding and computational costs. Third, our extensive experiments demonstrate that Z-Sampling can generally and significantly enhance generation quality across various benchmark datasets, diffusion models, and performance evaluation metrics. For example, Z-Sampling can even make DreamShaper achieve the HPSv2 winning rate higher than 94% over the original results. Moreover, Z-Sampling can further enhance existing diffusion models combined with other orthogonal methods, including Diffusion-DPO.

Title: BgGPT 1.0: Extending English-centric LLMs to other languages

• Authors: Anton Alexandrov, Veselin Raychev, Dimitar I. Dimitrov, Ce Zhang, Martin Vechev, Kristina Toutanova
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10893
• Pdf URL: https://arxiv.org/pdf/2412.10893
• Copy Paste: [[2412.10893]] BgGPT 1.0: Extending English-centric LLMs to other languages(https://arxiv.org/abs/2412.10893)
• Keywords: robust
• Abstract: We present BgGPT-Gemma-2-27B-Instruct and BgGPT-Gemma-2-9B-Instruct: continually pretrained and fine-tuned versions of Google's Gemma-2 models, specifically optimized for Bulgarian language understanding and generation. Leveraging Gemma-2's multilingual capabilities and over 100 billion tokens of Bulgarian and English text data, our models demonstrate strong performance in Bulgarian language tasks, setting a new standard for language-specific AI models. Our approach maintains the robust capabilities of the original Gemma-2 models, ensuring that the English language performance remains intact. To preserve the base model capabilities, we incorporate continual learning strategies based on recent Branch-and-Merge techniques as well as thorough curation and selection of training data. We provide detailed insights into our methodology, including the release of model weights with a commercial-friendly license, enabling broader adoption by researchers, companies, and hobbyists. Further, we establish a comprehensive set of benchmarks based on non-public educational data sources to evaluate models on Bulgarian language tasks as well as safety and chat capabilities. Our findings demonstrate the effectiveness of fine-tuning state-of-the-art models like Gemma 2 to enhance language-specific AI applications while maintaining cross-lingual capabilities.

Title: Task Diversity in Bayesian Federated Learning: Simultaneous Processing of Classification and Regression

• Authors: Junliang Lyu, Yixuan Zhang, Xiaoling Lu, Feng Zhou
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.10897
• Pdf URL: https://arxiv.org/pdf/2412.10897
• Copy Paste: [[2412.10897]] Task Diversity in Bayesian Federated Learning: Simultaneous Processing of Classification and Regression(https://arxiv.org/abs/2412.10897)
• Keywords: federate
• Abstract: This work addresses a key limitation in current federated learning approaches, which predominantly focus on homogeneous tasks, neglecting the task diversity on local devices. We propose a principled integration of multi-task learning using multi-output Gaussian processes (MOGP) at the local level and federated learning at the global level. MOGP handles correlated classification and regression tasks, offering a Bayesian non-parametric approach that naturally quantifies uncertainty. The central server aggregates the posteriors from local devices, updating a global MOGP prior redistributed for training local models until convergence. Challenges in performing posterior inference on local devices are addressed through the Pólya-Gamma augmentation technique and mean-field variational inference, enhancing computational efficiency and convergence rate. Experimental results on both synthetic and real data demonstrate superior predictive performance, OOD detection, uncertainty calibration and convergence rate, highlighting the method's potential in diverse applications. Our code is publicly available at this https URL.

Title: Enhancing Road Crack Detection Accuracy with BsS-YOLO: Optimizing Feature Fusion and Attention Mechanisms

• Authors: Jiaze Tang, Angzehua Feng, Vladimir Korkhov, Yuxi Pu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10902
• Pdf URL: https://arxiv.org/pdf/2412.10902
• Copy Paste: [[2412.10902]] Enhancing Road Crack Detection Accuracy with BsS-YOLO: Optimizing Feature Fusion and Attention Mechanisms(https://arxiv.org/abs/2412.10902)
• Keywords: robust
• Abstract: Effective road crack detection is crucial for road safety, infrastructure preservation, and extending road lifespan, offering significant economic benefits. However, existing methods struggle with varied target scales, complex backgrounds, and low adaptability to different environments. This paper presents the BsS-YOLO model, which optimizes multi-scale feature fusion through an enhanced Path Aggregation Network (PAN) and Bidirectional Feature Pyramid Network (BiFPN). The incorporation of weighted feature fusion improves feature representation, boosting detection accuracy and robustness. Furthermore, a Simple and Effective Attention Mechanism (SimAM) within the backbone enhances precision via spatial and channel-wise attention. The detection layer integrates a Shuffle Attention mechanism, which rearranges and mixes features across channels, refining key representations and further improving accuracy. Experimental results show that BsS-YOLO achieves a 2.8% increase in mean average precision (mAP) for road crack detection, supporting its applicability in diverse scenarios, including urban road maintenance and highway inspections.

Title: CEKER: A Generalizable LLM Framework for Literature Analysis with a Case Study in Unikernel Security

• Authors: Alex Wollman, John Hastings
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10904
• Pdf URL: https://arxiv.org/pdf/2412.10904
• Copy Paste: [[2412.10904]] CEKER: A Generalizable LLM Framework for Literature Analysis with a Case Study in Unikernel Security(https://arxiv.org/abs/2412.10904)
• Keywords: security, attack, extraction, large language model
• Abstract: Literature reviews are a critical component of formulating and justifying new research, but are a manual and often time-consuming process. This research introduces a novel, generalizable approach to literature analysis called CEKER which uses a three-step process to streamline the collection of literature, the extraction of key insights, and the summarized analysis of key trends and gaps. Leveraging Large Language Models (LLMs), this methodology represents a significant shift from traditional manual literature reviews, offering a scalable, flexible, and repeatable approach that can be applied across diverse research domains. A case study on unikernel security illustrates CEKER's ability to generate novel insights validated against previous manual methods. CEKER's analysis highlighted reduced attack surface as the most prominent theme. Key security gaps included the absence of Address Space Layout Randomization, missing debugging tools, and limited entropy generation, all of which represent important challenges to unikernel security. The study also revealed a reliance on hypervisors as a potential attack vector and emphasized the need for dynamic security adjustments to address real-time threats.

Title: C3: Learning Congestion Controllers with Formal Certificates

• Authors: Chenxi Yang, Divyanshu Saxena, Rohit Dwivedula, Kshiteej Mahajan, Swarat Chaudhuri, Aditya Akella
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2412.10915
• Pdf URL: https://arxiv.org/pdf/2412.10915
• Copy Paste: [[2412.10915]] C3: Learning Congestion Controllers with Formal Certificates(https://arxiv.org/abs/2412.10915)
• Keywords: robust
• Abstract: Learning-based congestion controllers offer better adaptability compared to traditional heuristic algorithms. However, the inherent unreliability of learning techniques can cause learning-based controllers to behave poorly, creating a need for formal guarantees. While methods for formally verifying learned congestion controllers exist, these methods offer binary feedback that cannot optimize the controller toward better behavior. We improve this state-of-the-art via C3, a new learning framework for congestion control that integrates the concept of formal certification in the learning loop. C3 uses an abstract interpreter that can produce robustness and performance certificates to guide the training process, rewarding models that are robust and performant even on worst-case inputs. Our evaluation demonstrates that unlike state-of-the-art learned controllers, C3-trained controllers provide both adaptability and worst-case reliability across a range of network conditions.

Title: LLMs-in-the-Loop Part 2: Expert Small AI Models for Anonymization and De-identification of PHI Across Multiple Languages

• Authors: Murat Gunay, Bunyamin Keles, Raife Hizlan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10918
• Pdf URL: https://arxiv.org/pdf/2412.10918
• Copy Paste: [[2412.10918]] LLMs-in-the-Loop Part 2: Expert Small AI Models for Anonymization and De-identification of PHI Across Multiple Languages(https://arxiv.org/abs/2412.10918)
• Keywords: privacy, protect, extraction, large language model
• Abstract: The rise of chronic diseases and pandemics like COVID-19 has emphasized the need for effective patient data processing while ensuring privacy through anonymization and de-identification of protected health information (PHI). Anonymized data facilitates research without compromising patient confidentiality. This paper introduces expert small AI models developed using the LLM-in-the-loop methodology to meet the demand for domain-specific de-identification NER models. These models overcome the privacy risks associated with large language models (LLMs) used via APIs by eliminating the need to transmit or store sensitive data. More importantly, they consistently outperform LLMs in de-identification tasks, offering superior performance and reliability. Our de-identification NER models, developed in eight languages (English, German, Italian, French, Romanian, Turkish, Spanish, and Arabic) achieved f1-micro score averages of 0.966, 0.975, 0.976, 0.970, 0.964, 0.974, 0.978, and 0.953 respectively. These results establish them as the most accurate healthcare anonymization solutions, surpassing existing small models and even general-purpose LLMs such as GPT-4o. While Part-1 of this series introduced the LLM-in-the-loop methodology for bio-medical document translation, this second paper showcases its success in developing cost-effective expert small NER models in de-identification tasks. Our findings lay the groundwork for future healthcare AI innovations, including biomedical entity and relation extraction, demonstrating the value of specialized models for domain-specific challenges.

Title: Predicting Survival of Hemodialysis Patients using Federated Learning

• Authors: Abhiram Raju, Praneeth Vepakomma
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10919
• Pdf URL: https://arxiv.org/pdf/2412.10919
• Copy Paste: [[2412.10919]] Predicting Survival of Hemodialysis Patients using Federated Learning(https://arxiv.org/abs/2412.10919)
• Keywords: federate
• Abstract: Hemodialysis patients who are on donor lists for kidney transplant may get misidentified, delaying their wait time. Thus, predicting their survival time is crucial for optimizing waiting lists and personalizing treatment plans. Predicting survival times for patients often requires large quantities of high quality but sensitive data. This data is siloed and since individual datasets are smaller and less diverse, locally trained survival models do not perform as well as centralized ones. Hence, we propose the use of Federated Learning in the context of predicting survival for hemodialysis patients. Federated Learning or FL can have comparatively better performances than local models while not sharing data between centers. However, despite the increased use of such technologies, the application of FL in survival and even more, dialysis patients remains sparse. This paper studies the performance of FL for data of hemodialysis patients from NephroPlus, the largest private network of dialysis centers in India.

Title: Automatically Detecting Checked-In Secrets in Android Apps: How Far Are We?

• Authors: Kevin Li, Lin Ling, Jinqiu Yang, Lili Wei
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.10922
• Pdf URL: https://arxiv.org/pdf/2412.10922
• Copy Paste: [[2412.10922]] Automatically Detecting Checked-In Secrets in Android Apps: How Far Are We?(https://arxiv.org/abs/2412.10922)
• Keywords: secure
• Abstract: Mobile apps are predominantly integrated with cloud services to benefit from enhanced functionalities. Adopting authentication using secrets such as API keys is crucial to ensure secure mobile-cloud interactions. However, developers often overlook the proper storage of such secrets, opting to put them directly into their projects. These secrets are checked into the projects and can be easily extracted and exploited by malicious adversaries. While many researchers investigated the issue of checked-in secret in open-source projects, there is a notable research gap concerning checked-in secrets in Android apps deployed on platforms such as Google Play Store. Unlike open-source projects, the lack of direct access to the source code and the presence of obfuscation complicates the checked-in secret detection for Android apps. This motivates us to conduct an empirical analysis to measure and compare the performance of different checked-in secret detection tools on Android apps. We first conducted a literature review to find all the checked-in secret detection tools that can be applied to Android apps. Then, we evaluate three representative tools on 5,135 Android apps, comparing their performance and analyzing their limitations. Our experiment reveals 2,142 checked-in secrets affecting 2,115 Android apps. We also disclose that the current checked-in secret detection techniques suffer from key limitations. All of the evaluated tools can miss a significant number of checked-in secrets in Android apps. Nevertheless, we observed that the tools are complimentary, suggesting the possibility of developing a more effective checked-in secret detection tool by combining their insights. Additionally, we propose that analyzing string groups within methods containing checked-in secrets may provide a more effective strategy to overcome obfuscation challenges.

Title: Linear Programming based Approximation to Individually Fair k-Clustering with Outliers

• Authors: Binita Maity, Shrutimoy Das, Anirban Dasgupta
• Subjects: cs.LG, cs.DS, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.10923
• Pdf URL: https://arxiv.org/pdf/2412.10923
• Copy Paste: [[2412.10923]] Linear Programming based Approximation to Individually Fair k-Clustering with Outliers(https://arxiv.org/abs/2412.10923)
• Keywords: fair
• Abstract: Individual fairness guarantees are often desirable properties to have, but they become hard to formalize when the dataset contains outliers. Here, we investigate the problem of developing an individually fair $k$-means clustering algorithm for datasets that contain outliers. That is, given $n$ points and $k$ centers, we want that for each point which is not an outlier, there must be a center within the $\frac{n}{k}$ nearest neighbours of the given point. While a few of the recent works have looked into individually fair clustering, this is the first work that explores this problem in the presence of outliers for $k$-means clustering. For this purpose, we define and solve a linear program (LP) that helps us identify the outliers. We exclude these outliers from the dataset and apply a rounding algorithm that computes the $k$ centers, such that the fairness constraint of the remaining points is satisfied. We also provide theoretical guarantees that our method leads to a guaranteed approximation of the fair radius as well as the clustering cost. We also demonstrate our techniques empirically on real-world datasets.

Title: Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning

• Authors: Julia Witte Zimmerman, Denis Hudon, Kathryn Cramer, Alejandro J. Ruiz, Calla Beauregard, Ashley Fehr, Mikaela Irene Fudolig, Bradford Demarest, Yoshi Meke Bird, Milo Z. Trujillo, Christopher M. Danforth, Peter Sheridan Dodds
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10924
• Pdf URL: https://arxiv.org/pdf/2412.10924
• Copy Paste: [[2412.10924]] Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning(https://arxiv.org/abs/2412.10924)
• Keywords: transformer, generative, large language model
• Abstract: Tokenization is a necessary component within the current architecture of many language models, including the transformer-based large language models (LLMs) of Generative AI, yet its impact on the model's cognition is often overlooked. We argue that LLMs demonstrate that the Distributional Hypothesis (DM) is sufficient for reasonably human-like language performance, and that the emergence of human-meaningful linguistic units among tokens motivates linguistically-informed interventions in existing, linguistically-agnostic tokenization techniques, particularly with respect to their roles as (1) semantic primitives and as (2) vehicles for conveying salient distributional patterns from human language to the model. We explore tokenizations from a BPE tokenizer; extant model vocabularies obtained from Hugging Face and tiktoken; and the information in exemplar token vectors as they move through the layers of a RoBERTa (large) model. Besides creating sub-optimal semantic building blocks and obscuring the model's access to the necessary distributional patterns, we describe how tokenization pretraining can be a backdoor for bias and other unwanted content, which current alignment practices may not remediate. Additionally, we relay evidence that the tokenization algorithm's objective function impacts the LLM's cognition, despite being meaningfully insulated from the main system intelligence.

Title: Video Representation Learning with Joint-Embedding Predictive Architectures

• Authors: Katrina Drozdov, Ravid Shwartz-Ziv, Yann LeCun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10925
• Pdf URL: https://arxiv.org/pdf/2412.10925
• Copy Paste: [[2412.10925]] Video Representation Learning with Joint-Embedding Predictive Architectures(https://arxiv.org/abs/2412.10925)
• Keywords: generative
• Abstract: Video representation learning is an increasingly important topic in machine learning research. We present Video JEPA with Variance-Covariance Regularization (VJ-VCR): a joint-embedding predictive architecture for self-supervised video representation learning that employs variance and covariance regularization to avoid representation collapse. We show that hidden representations from our VJ-VCR contain abstract, high-level information about the input data. Specifically, they outperform representations obtained from a generative baseline on downstream tasks that require understanding of the underlying dynamics of moving objects in the videos. Additionally, we explore different ways to incorporate latent variables into the VJ-VCR framework that capture information about uncertainty in the future in non-deterministic settings.

Title: Progressive Compression with Universally Quantized Diffusion Models

• Authors: Yibo Yang, Justus C. Will, Stephan Mandt
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10935
• Pdf URL: https://arxiv.org/pdf/2412.10935
• Copy Paste: [[2412.10935]] Progressive Compression with Universally Quantized Diffusion Models(https://arxiv.org/abs/2412.10935)
• Keywords: diffusion, generative
• Abstract: Diffusion probabilistic models have achieved mainstream success in many generative modeling tasks, from image generation to inverse problem solving. A distinct feature of these models is that they correspond to deep hierarchical latent variable models optimizing a variational evidence lower bound (ELBO) on the data likelihood. Drawing on a basic connection between likelihood modeling and compression, we explore the potential of diffusion models for progressive coding, resulting in a sequence of bits that can be incrementally transmitted and decoded with progressively improving reconstruction quality. Unlike prior work based on Gaussian diffusion or conditional diffusion models, we propose a new form of diffusion model with uniform noise in the forward process, whose negative ELBO corresponds to the end-to-end compression cost using universal quantization. We obtain promising first results on image compression, achieving competitive rate-distortion and rate-realism results on a wide range of bit-rates with a single model, bringing neural codecs a step closer to practical deployment.

Title: Meta-evaluating stability measures: MAX-Senstivity & AVG-Sensitivity

• Authors: Miquel Miró-Nicolau, Antoni Jaume-i-Capó, Gabriel Moyà-Alcover
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10942
• Pdf URL: https://arxiv.org/pdf/2412.10942
• Copy Paste: [[2412.10942]] Meta-evaluating stability measures: MAX-Senstivity & AVG-Sensitivity(https://arxiv.org/abs/2412.10942)
• Keywords: robust
• Abstract: The use of eXplainable Artificial Intelligence (XAI) systems has introduced a set of challenges that need resolution. The XAI robustness, or stability, has been one of the goals of the community from its beginning. Multiple authors have proposed evaluating this feature using objective evaluation measures. Nonetheless, many questions remain. With this work, we propose a novel approach to meta-evaluate these metrics, i.e. analyze the correctness of the evaluators. We propose two new tests that allowed us to evaluate two different stability measures: AVG-Sensitiviy and MAX-Senstivity. We tested their reliability in the presence of perfect and robust explanations, generated with a Decision Tree; as well as completely random explanations and prediction. The metrics results showed their incapacity of identify as erroneous the random explanations, highlighting their overall unreliability.

Title: SegHeD+: Segmentation of Heterogeneous Data for Multiple Sclerosis Lesions with Anatomical Constraints and Lesion-aware Augmentation

• Authors: Berke Doga Basaran, Paul M. Matthews, Wenjia Bai
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10946
• Pdf URL: https://arxiv.org/pdf/2412.10946
• Copy Paste: [[2412.10946]] SegHeD+: Segmentation of Heterogeneous Data for Multiple Sclerosis Lesions with Anatomical Constraints and Lesion-aware Augmentation(https://arxiv.org/abs/2412.10946)
• Keywords: segmentation
• Abstract: Assessing lesions and tracking their progression over time in brain magnetic resonance (MR) images is essential for diagnosing and monitoring multiple sclerosis (MS). Machine learning models have shown promise in automating the segmentation of MS lesions. However, training these models typically requires large, well-annotated datasets. Unfortunately, MS imaging datasets are often limited in size, spread across multiple hospital sites, and exhibit different formats (such as cross-sectional or longitudinal) and annotation styles. This data diversity presents a significant obstacle to developing a unified model for MS lesion segmentation. To address this issue, we introduce SegHeD+, a novel segmentation model that can handle multiple datasets and tasks, accommodating heterogeneous input data and performing segmentation for all lesions, new lesions, and vanishing lesions. We integrate domain knowledge about MS lesions by incorporating longitudinal, anatomical, and volumetric constraints into the segmentation model. Additionally, we perform lesion-level data augmentation to enlarge the training set and further improve segmentation performance. SegHeD+ is evaluated on five MS datasets and demonstrates superior performance in segmenting all, new, and vanishing lesions, surpassing several state-of-the-art methods in the field.

Title: Deep Learning-Based Noninvasive Screening of Type 2 Diabetes with Chest X-ray Images and Electronic Health Records

• Authors: Sanjana Gundapaneni, Zhuo Zhi, Miguel Rodrigues
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10955
• Pdf URL: https://arxiv.org/pdf/2412.10955
• Copy Paste: [[2412.10955]] Deep Learning-Based Noninvasive Screening of Type 2 Diabetes with Chest X-ray Images and Electronic Health Records(https://arxiv.org/abs/2412.10955)
• Keywords: transformer
• Abstract: The imperative for early detection of type 2 diabetes mellitus (T2DM) is challenged by its asymptomatic onset and dependence on suboptimal clinical diagnostic tests, contributing to its widespread global prevalence. While research into noninvasive T2DM screening tools has advanced, conventional machine learning approaches remain limited to unimodal inputs due to extensive feature engineering requirements. In contrast, deep learning models can leverage multimodal data for a more holistic understanding of patients' health conditions. However, the potential of chest X-ray (CXR) imaging, one of the most commonly performed medical procedures, remains underexplored. This study evaluates the integration of CXR images with other noninvasive data sources, including electronic health records (EHRs) and electrocardiography signals, for T2DM detection. Utilising datasets meticulously compiled from the MIMIC-IV databases, we investigated two deep fusion paradigms: an early fusion-based multimodal transformer and a modular joint fusion ResNet-LSTM architecture. The end-to-end trained ResNet-LSTM model achieved an AUROC of 0.86, surpassing the CXR-only baseline by 2.3% with just 9863 training samples. These findings demonstrate the diagnostic value of CXRs within multimodal frameworks for identifying at-risk individuals early. Additionally, the dataset preprocessing pipeline has also been released to support further research in this domain.

Title: SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer

• Authors: Hao Chen, Ze Wang, Xiang Li, Ximeng Sun, Fangyi Chen, Jiang Liu, Jindong Wang, Bhiksha Raj, Zicheng Liu, Emad Barsoum
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.10958
• Pdf URL: https://arxiv.org/pdf/2412.10958
• Copy Paste: [[2412.10958]] SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer(https://arxiv.org/abs/2412.10958)
• Keywords: transformer, generative
• Abstract: Efficient image tokenization with high compression ratios remains a critical challenge for training generative models. We present SoftVQ-VAE, a continuous image tokenizer that leverages soft categorical posteriors to aggregate multiple codewords into each latent token, substantially increasing the representation capacity of the latent space. When applied to Transformer-based architectures, our approach compresses 256x256 and 512x512 images using as few as 32 or 64 1-dimensional tokens. Not only does SoftVQ-VAE show consistent and high-quality reconstruction, more importantly, it also achieves state-of-the-art and significantly faster image generation results across different denoising-based generative models. Remarkably, SoftVQ-VAE improves inference throughput by up to 18x for generating 256x256 images and 55x for 512x512 images while achieving competitive FID scores of 1.78 and 2.21 for SiT-XL. It also improves the training efficiency of the generative models by reducing the number of training iterations by 2.3x while maintaining comparable performance. With its fully-differentiable design and semantic-rich latent space, our experiment demonstrates that SoftVQ-VQE achieves efficient tokenization without compromising generation quality, paving the way for more efficient generative models. Code and model are released.

Title: Can LLMs Help Create Grammar?: Automating Grammar Creation for Endangered Languages with In-Context Learning

• Authors: Piyapath T Spencer, Nanthipat Kongborrirak
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.10960
• Pdf URL: https://arxiv.org/pdf/2412.10960
• Copy Paste: [[2412.10960]] Can LLMs Help Create Grammar?: Automating Grammar Creation for Endangered Languages with In-Context Learning(https://arxiv.org/abs/2412.10960)
• Keywords: large language model
• Abstract: Yes! In the present-day documenting and preserving endangered languages, the application of Large Language Models (LLMs) presents a promising approach. This paper explores how LLMs, particularly through in-context learning, can assist in generating grammatical information for low-resource languages with limited amount of data. We takes Moklen as a case study to evaluate the efficacy of LLMs in producing coherent grammatical rules and lexical entries using only bilingual dictionaries and parallel sentences of the unknown language without building the model from scratch. Our methodology involves organising the existing linguistic data and prompting to efficiently enable to generate formal XLE grammar. Our results demonstrate that LLMs can successfully capture key grammatical structures and lexical information, although challenges such as the potential for English grammatical biases remain. This study highlights the potential of LLMs to enhance language documentation efforts, providing a cost-effective solution for generating linguistic data and contributing to the preservation of endangered languages.

Title: FlowDock: Geometric Flow Matching for Generative Protein-Ligand Docking and Affinity Prediction

• Authors: Alex Morehead, Jianlin Cheng
• Subjects: cs.LG, cs.AI, q-bio.BM, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2412.10966
• Pdf URL: https://arxiv.org/pdf/2412.10966
• Copy Paste: [[2412.10966]] FlowDock: Geometric Flow Matching for Generative Protein-Ligand Docking and Affinity Prediction(https://arxiv.org/abs/2412.10966)
• Keywords: generative
• Abstract: Powerful generative models of protein-ligand structure have recently been proposed, but few of these methods support both flexible protein-ligand docking and affinity estimation. Of those that do, none can directly model multiple binding ligands concurrently or have been rigorously benchmarked on pharmacologically relevant drug targets, hindering their widespread adoption in drug discovery efforts. In this work, we propose FlowDock, a deep geometric generative model based on conditional flow matching that learns to directly map unbound (apo) structures to their bound (holo) counterparts for an arbitrary number of binding ligands. Furthermore, FlowDock provides predicted structural confidence scores and binding affinity values with each of its generated protein-ligand complex structures, enabling fast virtual screening of new (multi-ligand) drug targets. For the commonly-used PoseBusters Benchmark dataset, FlowDock achieves a 51% blind docking success rate using unbound (apo) protein input structures and without any information derived from multiple sequence alignments, and for the challenging new DockGen-E dataset, FlowDock matches the performance of single-sequence Chai-1 for binding pocket generalization. Additionally, in the ligand category of the 16th community-wide Critical Assessment of Techniques for Structure Prediction (CASP16), FlowDock ranked among the top-5 methods for pharmacological binding affinity estimation across 140 protein-ligand complexes, demonstrating the efficacy of its learned representations in virtual screening. Source code, data, and pre-trained models are available at this https URL.

Title: DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting

• Authors: Luis Wiedmann, Luca Wiehe, David Rozenberszki
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.10972
• Pdf URL: https://arxiv.org/pdf/2412.10972
• Copy Paste: [[2412.10972]] DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting(https://arxiv.org/abs/2412.10972)
• Keywords: segmentation
• Abstract: Open-set 3D segmentation represents a major point of interest for multiple downstream robotics and augmented/virtual reality applications. Recent advances introduce 3D Gaussian Splatting as a computationally efficient representation of the underlying scene. They enable the rendering of novel views while achieving real-time display rates and matching the quality of computationally far more expensive methods. We present a decoupled 3D segmentation pipeline to ensure modularity and adaptability to novel 3D representations and semantic segmentation foundation models. The pipeline proposes class-agnostic masks based on a 3D reconstruction of the scene. Given the resulting class-agnostic masks, we use a class-aware 2D foundation model to add class annotations to the 3D masks. We test this pipeline with 3D Gaussian Splatting and different 2D segmentation models and achieve better performance than more tailored approaches while also significantly increasing the modularity.

Title: Labeling NIDS Rules with MITRE ATT&CK Techniques: Machine Learning vs. Large Language Models

• Authors: Nir Daniel, Florian Klaus Kaiser, Shay Giladi, Sapir Sharabi, Raz Moyal, Shalev Shpolyansky, Andres Murillo, Aviad Elyashar, Rami Puzis
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.10978
• Pdf URL: https://arxiv.org/pdf/2412.10978
• Copy Paste: [[2412.10978]] Labeling NIDS Rules with MITRE ATT&CK Techniques: Machine Learning vs. Large Language Models(https://arxiv.org/abs/2412.10978)
• Keywords: security, attack, explainability, large language model
• Abstract: Analysts in Security Operations Centers (SOCs) are often occupied with time-consuming investigations of alerts from Network Intrusion Detection Systems (NIDS). Many NIDS rules lack clear explanations and associations with attack techniques, complicating the alert triage and the generation of attack hypotheses. Large Language Models (LLMs) may be a promising technology to reduce the alert explainability gap by associating rules with attack techniques. In this paper, we investigate the ability of three prominent LLMs (ChatGPT, Claude, and Gemini) to reason about NIDS rules while labeling them with MITRE ATT&CK tactics and techniques. We discuss prompt design and present experiments performed with 973 Snort rules. Our results indicate that while LLMs provide explainable, scalable, and efficient initial mappings, traditional Machine Learning (ML) models consistently outperform them in accuracy, achieving higher precision, recall, and F1-scores. These results highlight the potential for hybrid LLM-ML approaches to enhance SOC operations and better address the evolving threat landscape.

Title: Serial Scammers and Attack of the Clones: How Scammers Coordinate Multiple Rug Pulls on Decentralized Exchanges

• Authors: Phuong Duy Huynh, Son Hoang Dau, Hong Yen Tran, Nick Huppert, Hoonie Sun, Joshua Cervenjak, Xiaodong Li, Emanuele Viterbo
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.10993
• Pdf URL: https://arxiv.org/pdf/2412.10993
• Copy Paste: [[2412.10993]] Serial Scammers and Attack of the Clones: How Scammers Coordinate Multiple Rug Pulls on Decentralized Exchanges(https://arxiv.org/abs/2412.10993)
• Keywords: attack
• Abstract: We explored in this work the ubiquitous phenomenon of serial scammers, who deploy thousands of addresses to conduct a series of similar Rug Pulls on popular decentralized exchanges (DEXs). We first constructed a list of about 384,000 scammer addresses behind all 1-day Rug Pulls on the two most popular DEXs, Uniswap (Ethereum) and Pancakeswap (BSC), and identified many distinctive scam patterns including star-shaped, chain-shaped, and majority-flow scam clusters. We then proposed an algorithm to build a complete scam network from given scammer addresses, which consists of not only scammer addresses but also supporting addresses including depositors, withdrawers, transferrers, coordinators, and most importantly, wash traders. We note that profit estimations in existing works on Rug Pulls failed to capture the cost of wash trading, leading to inflated figures. Knowing who the wash traders are, we established a more accurate estimate for the true profit of individual scam pools as well as of the entire (serial) scam network by taking into account the wash-trading expenses.

Title: RapidNet: Multi-Level Dilated Convolution Based Mobile Backbone

• Authors: Mustafa Munir, Md Mostafijur Rahman, Radu Marculescu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.10995
• Pdf URL: https://arxiv.org/pdf/2412.10995
• Copy Paste: [[2412.10995]] RapidNet: Multi-Level Dilated Convolution Based Mobile Backbone(https://arxiv.org/abs/2412.10995)
• Keywords: transformer, segmentation
• Abstract: Vision transformers (ViTs) have dominated computer vision in recent years. However, ViTs are computationally expensive and not well suited for mobile devices; this led to the prevalence of convolutional neural network (CNN) and ViT-based hybrid models for mobile vision applications. Recently, Vision GNN (ViG) and CNN hybrid models have also been proposed for mobile vision tasks. However, all of these methods remain slower compared to pure CNN-based models. In this work, we propose Multi-Level Dilated Convolutions to devise a purely CNN-based mobile backbone. Using Multi-Level Dilated Convolutions allows for a larger theoretical receptive field than standard convolutions. Different levels of dilation also allow for interactions between the short-range and long-range features in an image. Experiments show that our proposed model outperforms state-of-the-art (SOTA) mobile CNN, ViT, ViG, and hybrid architectures in terms of accuracy and/or speed on image classification, object detection, instance segmentation, and semantic segmentation. Our fastest model, RapidNet-Ti, achieves 76.3\% top-1 accuracy on ImageNet-1K with 0.9 ms inference latency on an iPhone 13 mini NPU, which is faster and more accurate than MobileNetV2x1.4 (74.7\% top-1 with 1.0 ms latency). Our work shows that pure CNN architectures can beat SOTA hybrid and ViT models in terms of accuracy and speed when designed properly.

Title: Optimal Rates for Robust Stochastic Convex Optimization

• Authors: Changyu Gao, Andrew Lowy, Xingyu Zhou, Stephen J. Wright
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.11003
• Pdf URL: https://arxiv.org/pdf/2412.11003
• Copy Paste: [[2412.11003]] Optimal Rates for Robust Stochastic Convex Optimization(https://arxiv.org/abs/2412.11003)
• Keywords: robust
• Abstract: The sensitivity of machine learning algorithms to outliers, particularly in high-dimensional spaces, necessitates the development of robust methods. Within the framework of $\epsilon$-contamination model, where the adversary can inspect and replace up to $\epsilon$ fraction of the samples, a fundamental open question is determining the optimal rates for robust stochastic convex optimization (robust SCO), provided the samples under $\epsilon$-contamination. We develop novel algorithms that achieve minimax-optimal excess risk (up to logarithmic factors) under the $\epsilon$-contamination model. Our approach advances beyonds existing algorithms, which are not only suboptimal but also constrained by stringent requirements, including Lipschitzness and smoothness conditions on sample this http URL algorithms achieve optimal rates while removing these restrictive assumptions, and notably, remain effective for nonsmooth but Lipschitz population risks.

Title: Entropy-Regularized Process Reward Model

• Authors: Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, Tong Zhang
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11006
• Pdf URL: https://arxiv.org/pdf/2412.11006
• Copy Paste: [[2412.11006]] Entropy-Regularized Process Reward Model(https://arxiv.org/abs/2412.11006)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown promise in performing complex multi-step reasoning, yet they continue to struggle with mathematical reasoning, often making systematic errors. A promising solution is reinforcement learning (RL) guided by reward models, particularly those focusing on process rewards, which score each intermediate step rather than solely evaluating the final outcome. This approach is more effective at guiding policy models towards correct reasoning trajectories. In this work, we propose an entropy-regularized process reward model (ER-PRM) that integrates KL-regularized Markov Decision Processes (MDP) to balance policy optimization with the need to prevent the policy from shifting too far from its initial distribution. We derive a novel reward construction method based on the theoretical results. Our theoretical analysis shows that we could derive the optimal reward model from the initial policy sampling. Our empirical experiments on the MATH and GSM8K benchmarks demonstrate that ER-PRM consistently outperforms existing process reward models, achieving 1% improvement on GSM8K and 2-3% improvement on MATH under best-of-N evaluation, and more than 1% improvement under RLHF. These results highlight the efficacy of entropy-regularization in enhancing LLMs' reasoning capabilities.

Title: Towards Context-aware Convolutional Network for Image Restoration

• Authors: Fangwei Hao, Ji Du, Weiyun Liang, Jing Xu, Xiaoxuan Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11008
• Pdf URL: https://arxiv.org/pdf/2412.11008
• Copy Paste: [[2412.11008]] Towards Context-aware Convolutional Network for Image Restoration(https://arxiv.org/abs/2412.11008)
• Keywords: extraction, transformer
• Abstract: Image restoration (IR) is a long-standing task to recover a high-quality image from its corrupted observation. Recently, transformer-based algorithms and some attention-based convolutional neural networks (CNNs) have presented promising results on several IR tasks. However, existing convolutional residual building modules for IR encounter limited ability to map inputs into high-dimensional and non-linear feature spaces, and their local receptive fields have difficulty in capturing long-range context information like Transformer. Besides, CNN-based attention modules for IR either face static abundant parameters or have limited receptive fields. To address the first issue, we propose an efficient residual star module (ERSM) that includes context-aware "star operation" (element-wise multiplication) to contextually map features into exceedingly high-dimensional and non-linear feature spaces, which greatly enhances representation learning. To further boost the extraction of contextual information, as for the second issue, we propose a large dynamic integration module (LDIM) which possesses an extremely large receptive field. Thus, LDIM can dynamically and efficiently integrate more contextual information that helps to further significantly improve the reconstruction performance. Integrating ERSM and LDIM into an U-shaped backbone, we propose a context-aware convolutional network (CCNet) with powerful learning ability for contextual high-dimensional mapping and abundant contextual information. Extensive experiments show that our CCNet with low model complexity achieves superior performance compared to other state-of-the-art IR methods on several IR tasks, including image dehazing, image motion deblurring, and image desnowing.

Title: PromptV: Leveraging LLM-powered Multi-Agent Prompting for High-quality Verilog Generation

• Authors: Zhendong Mi, Renming Zheng, Haowen Zhong, Yue Sun, Shaoyi Huang
• Subjects: cs.LG, cs.AI, cs.AR, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.11014
• Pdf URL: https://arxiv.org/pdf/2412.11014
• Copy Paste: [[2412.11014]] PromptV: Leveraging LLM-powered Multi-Agent Prompting for High-quality Verilog Generation(https://arxiv.org/abs/2412.11014)
• Keywords: generative
• Abstract: Recent advances in agentic LLMs have demonstrated remarkable automated Verilog code generation capabilities. However, existing approaches either demand substantial computational resources or rely on LLM-assisted single-agent prompt learning techniques, which we observe for the first time has a degeneration issue - characterized by deteriorating generative performance and diminished error detection and correction capabilities. This paper proposes a novel multi-agent prompt learning framework to address these limitations and enhance code generation quality. We show for the first time that multi-agent architectures can effectively mitigate the degeneration risk while improving code error correction capabilities, resulting in higher-quality Verilog code generation. Experimental results show that the proposed method could achieve 96.4% and 96.5% pass@10 scores on VerilogEval Machine and Human benchmarks, respectively while attaining 100% Syntax and 99.9% Functionality pass@5 metrics on the RTLLM benchmark.

Title: On Distilling the Displacement Knowledge for Few-Shot Class-Incremental Learning

• Authors: Pengfei Fang, Yongchun Qin, Hui Xue
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11017
• Pdf URL: https://arxiv.org/pdf/2412.11017
• Copy Paste: [[2412.11017]] On Distilling the Displacement Knowledge for Few-Shot Class-Incremental Learning(https://arxiv.org/abs/2412.11017)
• Keywords: robust
• Abstract: Few-shot Class-Incremental Learning (FSCIL) addresses the challenges of evolving data distributions and the difficulty of data acquisition in real-world scenarios. To counteract the catastrophic forgetting typically encountered in FSCIL, knowledge distillation is employed as a way to maintain the knowledge from learned data distribution. Recognizing the limitations of generating discriminative feature representations in a few-shot context, our approach incorporates structural information between samples into knowledge distillation. This structural information serves as a remedy for the low quality of features. Diverging from traditional structured distillation methods that compute sample similarity, we introduce the Displacement Knowledge Distillation (DKD) method. DKD utilizes displacement rather than similarity between samples, incorporating both distance and angular information to significantly enhance the information density retained through knowledge distillation. Observing performance disparities in feature distribution between base and novel classes, we propose the Dual Distillation Network (DDNet). This network applies traditional knowledge distillation to base classes and DKD to novel classes, challenging the conventional integration of novel classes with base classes. Additionally, we implement an instance-aware sample selector during inference to dynamically adjust dual branch weights, thereby leveraging the complementary strengths of each approach. Extensive testing on three benchmarks demonstrates that DDNet achieves state-of-the-art results. Moreover, through rigorous experimentation and comparison, we establish the robustness and general applicability of our proposed DKD method.

Title: Exploring Enhanced Contextual Information for Video-Level Object Tracking

• Authors: Ben Kang, Xin Chen, Simiao Lai, Yang Liu, Yi Liu, Dong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11023
• Pdf URL: https://arxiv.org/pdf/2412.11023
• Copy Paste: [[2412.11023]] Exploring Enhanced Contextual Information for Video-Level Object Tracking(https://arxiv.org/abs/2412.11023)
• Keywords: robust
• Abstract: Contextual information at the video level has become increasingly crucial for visual object tracking. However, existing methods typically use only a few tokens to convey this information, which can lead to information loss and limit their ability to fully capture the context. To address this issue, we propose a new video-level visual object tracking framework called MCITrack. It leverages Mamba's hidden states to continuously record and transmit extensive contextual information throughout the video stream, resulting in more robust object tracking. The core component of MCITrack is the Contextual Information Fusion module, which consists of the mamba layer and the cross-attention layer. The mamba layer stores historical contextual information, while the cross-attention layer integrates this information into the current visual features of each backbone block. This module enhances the model's ability to capture and utilize contextual information at multiple levels through deep integration with the backbone. Experiments demonstrate that MCITrack achieves competitive performance across numerous benchmarks. For instance, it gets 76.6% AUC on LaSOT and 80.0% AO on GOT-10k, establishing a new state-of-the-art performance. Code and models are available at this https URL.

Title: Exploring Diffusion and Flow Matching Under Generator Matching

• Authors: Zeeshan Patel, James DeLoye, Lance Mathias
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11024
• Pdf URL: https://arxiv.org/pdf/2412.11024
• Copy Paste: [[2412.11024]] Exploring Diffusion and Flow Matching Under Generator Matching(https://arxiv.org/abs/2412.11024)
• Keywords: robust, diffusion, generative
• Abstract: In this paper, we present a comprehensive theoretical comparison of diffusion and flow matching under the Generator Matching framework. Despite their apparent differences, both diffusion and flow matching can be viewed under the unified framework of Generator Matching. By recasting both diffusion and flow matching under the same generative Markov framework, we provide theoretical insights into why flow matching models can be more robust empirically and how novel model classes can be constructed by mixing deterministic and stochastic components. Our analysis offers a fresh perspective on the relationships between state-of-the-art generative modeling paradigms.

Title: From Simple to Professional: A Combinatorial Controllable Image Captioning Agent

• Authors: Xinran Wang, Muxi Diao, Baoteng Li, Haiwen Zhang, Kongming Liang, Zhanyu Ma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11025
• Pdf URL: https://arxiv.org/pdf/2412.11025
• Copy Paste: [[2412.11025]] From Simple to Professional: A Combinatorial Controllable Image Captioning Agent(https://arxiv.org/abs/2412.11025)
• Keywords: large language model
• Abstract: The Controllable Image Captioning Agent (CapAgent) is an innovative system designed to bridge the gap between user simplicity and professional-level outputs in image captioning tasks. CapAgent automatically transforms user-provided simple instructions into detailed, professional instructions, enabling precise and context-aware caption generation. By leveraging multimodal large language models (MLLMs) and external tools such as object detection tool and search engines, the system ensures that captions adhere to specified guidelines, including sentiment, keywords, focus, and formatting. CapAgent transparently controls each step of the captioning process, and showcases its reasoning and tool usage at every step, fostering user trust and engagement. The project code is available at this https URL.

Title: SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generation

• Authors: Hang Zhang, Zhuoling Li, Jun Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11026
• Pdf URL: https://arxiv.org/pdf/2412.11026
• Copy Paste: [[2412.11026]] SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generation(https://arxiv.org/abs/2412.11026)
• Keywords: transformer, large language model
• Abstract: Dynamic scenes contain intricate spatio-temporal information, crucial for mobile robots, UAVs, and autonomous driving systems to make informed decisions. Parsing these scenes into semantic triplets for accurate Scene Graph Generation (SGG) is highly challenging due to the fluctuating spatio-temporal complexity. Inspired by the reasoning capabilities of Large Language Models (LLMs), we propose SceneLLM, a novel framework that leverages LLMs as powerful scene analyzers for dynamic SGG. Our framework introduces a Video-to-Language (V2L) mapping module that transforms video frames into linguistic signals (scene tokens), making the input more comprehensible for LLMs. To better encode spatial information, we devise a Spatial Information Aggregation (SIA) scheme, inspired by the structure of Chinese characters, which encodes spatial data into tokens. Using Optimal Transport (OT), we generate an implicit language signal from the frame-level token sequence that captures the video's spatio-temporal information. To further improve the LLM's ability to process this implicit linguistic input, we apply Low-Rank Adaptation (LoRA) to fine-tune the model. Finally, we use a transformer-based SGG predictor to decode the LLM's reasoning and predict semantic triplets. Our method achieves state-of-the-art results on the Action Genome (AG) benchmark, and extensive experiments show the effectiveness of SceneLLM in understanding and generating accurate dynamic scene graphs.

Title: AURORA: Automated Unleash of 3D Room Outlines for VR Applications

• Authors: Huijun Han, Yongqing Liang, Yuanlong Zhou, Wenping Wang, Edgar J. Rojas-Munoz, Xin Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11033
• Pdf URL: https://arxiv.org/pdf/2412.11033
• Copy Paste: [[2412.11033]] AURORA: Automated Unleash of 3D Room Outlines for VR Applications(https://arxiv.org/abs/2412.11033)
• Keywords: segmentation
• Abstract: Creating realistic VR experiences is challenging due to the labor-intensive process of accurately replicating real-world details into virtual scenes, highlighting the need for automated methods that maintain spatial accuracy and provide design flexibility. In this paper, we propose AURORA, a novel method that leverages RGB-D images to automatically generate both purely virtual reality (VR) scenes and VR scenes combined with real-world elements. This approach can benefit designers by streamlining the process of converting real-world details into virtual scenes. AURORA integrates advanced techniques in image processing, segmentation, and 3D reconstruction to efficiently create realistic and detailed interior designs from real-world environments. The design of this integration ensures optimal performance and precision, addressing key challenges in automated indoor design generation by uniquely combining and leveraging the strengths of foundation models. We demonstrate the effectiveness of our approach through experiments, both on self-captured data and public datasets, showcasing its potential to enhance virtual reality (VR) applications by providing interior designs that conform to real-world positioning.

Title: SAM-IF: Leveraging SAM for Incremental Few-Shot Instance Segmentation

• Authors: Xudong Zhou, Wenhao He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11034
• Pdf URL: https://arxiv.org/pdf/2412.11034
• Copy Paste: [[2412.11034]] SAM-IF: Leveraging SAM for Incremental Few-Shot Instance Segmentation(https://arxiv.org/abs/2412.11034)
• Keywords: segmentation
• Abstract: We propose SAM-IF, a novel method for incremental few-shot instance segmentation leveraging the Segment Anything Model (SAM). SAM-IF addresses the challenges of class-agnostic instance segmentation by introducing a multi-class classifier and fine-tuning SAM to focus on specific target objects. To enhance few-shot learning capabilities, SAM-IF employs a cosine-similarity-based classifier, enabling efficient adaptation to novel classes with minimal data. Additionally, SAM-IF supports incremental learning by updating classifier weights without retraining the decoder. Our method achieves competitive but more reasonable results compared to existing approaches, particularly in scenarios requiring specific object segmentation with limited labeled data.

Title: Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models

• Authors: Di Wu, Xin Lu, Yanyan Zhao, Bing Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11041
• Pdf URL: https://arxiv.org/pdf/2412.11041
• Copy Paste: [[2412.11041]] Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models(https://arxiv.org/abs/2412.11041)
• Keywords: attack, large language model
• Abstract: Although large language models (LLMs) achieve effective safety alignment at the time of release, they still face various safety challenges. A key issue is that fine-tuning often compromises the safety alignment of LLMs. To address this issue, we propose a method named \textbf{IRR} (\textbf{I}dentify, \textbf{R}emove, and \textbf{R}ecalibrate for Safety Realignment) that performs safety realignment for LLMs. The core of IRR is to identify and remove unsafe delta parameters from the fine-tuned models, while recalibrating the retained ones. We evaluate the effectiveness of IRR across various datasets, including both full fine-tuning and LoRA methods. Our results demonstrate that IRR significantly enhances the safety performance of fine-tuned models on safety benchmarks, such as harmful queries and jailbreak attacks, while maintaining their performance on downstream tasks. The source code is available at: \url{this https URL}.

Title: Semantic Steganography: A Framework for Robust and High-Capacity Information Hiding using Large Language Models

• Authors: Minhao Bai, Jinshuai Yang, Kaiyi Pang, Yongfeng Huang, Yue Gao
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.11043
• Pdf URL: https://arxiv.org/pdf/2412.11043
• Copy Paste: [[2412.11043]] Semantic Steganography: A Framework for Robust and High-Capacity Information Hiding using Large Language Models(https://arxiv.org/abs/2412.11043)
• Keywords: robust, generative, large language model
• Abstract: In the era of Large Language Models (LLMs), generative linguistic steganography has become a prevalent technique for hiding information within model-generated texts. However, traditional steganography methods struggle to effectively align steganographic texts with original model-generated texts due to the lower entropy of the predicted probability distribution of LLMs. This results in a decrease in embedding capacity and poses challenges for decoding stegos in real-world communication channels. To address these challenges, we propose a semantic steganography framework based on LLMs, which construct a semantic space and map secret messages onto this space using ontology-entity trees. This framework offers robustness and reliability for transmission in complex channels, as well as resistance to text rendering and word blocking. Additionally, the stegos generated by our framework are indistinguishable from the covers and achieve a higher embedding capacity compared to state-of-the-art steganography methods, while producing higher quality stegos.

Title: Understanding and Mitigating Memorization in Diffusion Models for Tabular Data

• Authors: Zhengyu Fang, Zhimeng Jiang, Huiyuan Chen, Xiao Li, Jing Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11044
• Pdf URL: https://arxiv.org/pdf/2412.11044
• Copy Paste: [[2412.11044]] Understanding and Mitigating Memorization in Diffusion Models for Tabular Data(https://arxiv.org/abs/2412.11044)
• Keywords: diffusion
• Abstract: Tabular data generation has attracted significant research interest in recent years, with the tabular diffusion models greatly improving the quality of synthetic data. However, while memorization, where models inadvertently replicate exact or near-identical training data, has been thoroughly investigated in image and text generation, its effects on tabular data remain largely unexplored. In this paper, we conduct the first comprehensive investigation of memorization phenomena in diffusion models for tabular data. Our empirical analysis reveals that memorization appears in tabular diffusion models and increases with larger training epochs. We further examine the influence of factors such as dataset sizes, feature dimensions, and different diffusion models on memorization. Additionally, we provide a theoretical explanation for why memorization occurs in tabular diffusion models. To address this issue, we propose TabCutMix, a simple yet effective data augmentation technique that exchanges randomly selected feature segments between random same-class training sample pairs. Building upon this, we introduce TabCutMixPlus, an enhanced method that clusters features based on feature correlations and ensures that features within the same cluster are exchanged together during augmentation. This clustering mechanism mitigates out-of-distribution (OOD) generation issues by maintaining feature coherence. Experimental results across various datasets and diffusion models demonstrate that TabCutMix effectively mitigates memorization while maintaining high-quality data generation.

Title: RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Models

• Authors: Yujin Wang, Quanfeng Liu, Jiaqi Fan, Jinlong Hong, Hongqing Chu, Mengjian Tian, Bingzhao Gao, Hong Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11050
• Pdf URL: https://arxiv.org/pdf/2412.11050
• Copy Paste: [[2412.11050]] RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving with Vision-Language Models(https://arxiv.org/abs/2412.11050)
• Keywords: robust
• Abstract: Understanding and addressing corner cases is essential for ensuring the safety and reliability of autonomous driving systems. Vision-Language Models (VLMs) play a crucial role in enhancing scenario comprehension, yet they face significant challenges, such as hallucination and insufficient real-world grounding, which compromise their performance in critical driving scenarios. In this work, we propose RAC3, a novel framework designed to improve VLMs' ability to handle corner cases effectively. The framework integrates Retrieval-Augmented Generation (RAG) to mitigate hallucination by dynamically incorporating context-specific external knowledge. A cornerstone of RAC3 is its cross-modal alignment fine-tuning, which utilizes contrastive learning to embed image-text pairs into a unified semantic space, enabling robust retrieval of similar scenarios. We evaluate RAC3 through extensive experiments using a curated dataset of corner case scenarios, demonstrating its ability to enhance semantic alignment, improve hallucination mitigation, and achieve superior performance metrics, such as Cosine Similarity and ROUGE-L scores. For example, for the LLaVA-v1.6-34B VLM, the cosine similarity between the generated text and the reference text has increased by 5.22\%. The F1-score in ROUGE-L has increased by 39.91\%, the Precision has increased by 55.80\%, and the Recall has increased by 13.74\%. This work underscores the potential of retrieval-augmented VLMs to advance the robustness and safety of autonomous driving in complex environments.

Title: DisCo-DSO: Coupling Discrete and Continuous Optimization for Efficient Generative Design in Hybrid Spaces

• Authors: Jacob F. Pettit, Chak Shing Lee, Jiachen Yang, Alex Ho, Daniel Faissol, Brenden Petersen, Mikel Landajuela
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2412.11051
• Pdf URL: https://arxiv.org/pdf/2412.11051
• Copy Paste: [[2412.11051]] DisCo-DSO: Coupling Discrete and Continuous Optimization for Efficient Generative Design in Hybrid Spaces(https://arxiv.org/abs/2412.11051)
• Keywords: robust, generative
• Abstract: We consider the challenge of black-box optimization within hybrid discrete-continuous and variable-length spaces, a problem that arises in various applications, such as decision tree learning and symbolic regression. We propose DisCo-DSO (Discrete-Continuous Deep Symbolic Optimization), a novel approach that uses a generative model to learn a joint distribution over discrete and continuous design variables to sample new hybrid designs. In contrast to standard decoupled approaches, in which the discrete and continuous variables are optimized separately, our joint optimization approach uses fewer objective function evaluations, is robust against non-differentiable objectives, and learns from prior samples to guide the search, leading to significant improvement in performance and sample efficiency. Our experiments on a diverse set of optimization tasks demonstrate that the advantages of DisCo-DSO become increasingly evident as the complexity of the problem increases. In particular, we illustrate DisCo-DSO's superiority over the state-of-the-art methods for interpretable reinforcement learning with decision trees.

Title: NITRO: LLM Inference on Intel Laptop NPUs

• Authors: Anthony Fei, Mohamed S. Abdelfattah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11053
• Pdf URL: https://arxiv.org/pdf/2412.11053
• Copy Paste: [[2412.11053]] NITRO: LLM Inference on Intel Laptop NPUs(https://arxiv.org/abs/2412.11053)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs) have become essential tools in natural language processing, finding large usage in chatbots such as ChatGPT and Gemini, and are a central area of research. A particular area of interest includes designing hardware specialized for these AI applications, with one such example being the neural processing unit (NPU). In 2023, Intel released the Intel Core Ultra processor with codename Meteor Lake, featuring a CPU, GPU, and NPU system-on-chip. However, official software support for the NPU through Intel's OpenVINO framework is limited to static model inference. The dynamic nature of autoregressive token generation in LLMs is therefore not supported out of the box. To address this shortcoming, we present NITRO (NPU Inference for Transformers Optimization), a Python-based framework built on top of OpenVINO to support text and chat generation on NPUs. In this paper, we discuss in detail the key modifications made to the transformer architecture to enable inference, some performance benchmarks, and future steps towards improving the package. The code repository for NITRO can be found here: this https URL.

Title: Set-Valued Sensitivity Analysis of Deep Neural Networks

• Authors: Xin Wang, Feiling wang, Xuegang Ban (Jeff)
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11057
• Pdf URL: https://arxiv.org/pdf/2412.11057
• Copy Paste: [[2412.11057]] Set-Valued Sensitivity Analysis of Deep Neural Networks(https://arxiv.org/abs/2412.11057)
• Keywords: robust
• Abstract: This paper proposes a sensitivity analysis framework based on set valued mapping for deep neural networks (DNN) to understand and compute how the solutions (model weights) of DNN respond to perturbations in the training data. As a DNN may not exhibit a unique solution (minima) and the algorithm of solving a DNN may lead to different solutions with minor perturbations to input data, we focus on the sensitivity of the solution set of DNN, instead of studying a single solution. In particular, we are interested in the expansion and contraction of the set in response to data perturbations. If the change of solution set can be bounded by the extent of the data perturbation, the model is said to exhibit the Lipschitz like property. This "set-to-set" analysis approach provides a deeper understanding of the robustness and reliability of DNNs during training. Our framework incorporates both isolated and non-isolated minima, and critically, does not require the assumption that the Hessian of loss function is non-singular. By developing set-level metrics such as distance between sets, convergence of sets, derivatives of set-valued mapping, and stability across the solution set, we prove that the solution set of the Fully Connected Neural Network holds Lipschitz-like properties. For general neural networks (e.g., Resnet), we introduce a graphical-derivative-based method to estimate the new solution set following data perturbation without retraining.

Title: SHMT: Self-supervised Hierarchical Makeup Transfer via Latent Diffusion Models

• Authors: Zhaoyang Sun, Shengwu Xiong, Yaxiong Chen, Fei Du, Weihua Chen, Fan Wang, Yi Rong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11058
• Pdf URL: https://arxiv.org/pdf/2412.11058
• Copy Paste: [[2412.11058]] SHMT: Self-supervised Hierarchical Makeup Transfer via Latent Diffusion Models(https://arxiv.org/abs/2412.11058)
• Keywords: diffusion
• Abstract: This paper studies the challenging task of makeup transfer, which aims to apply diverse makeup styles precisely and naturally to a given facial image. Due to the absence of paired data, current methods typically synthesize sub-optimal pseudo ground truths to guide the model training, resulting in low makeup fidelity. Additionally, different makeup styles generally have varying effects on the person face, but existing methods struggle to deal with this diversity. To address these issues, we propose a novel Self-supervised Hierarchical Makeup Transfer (SHMT) method via latent diffusion models. Following a "decoupling-and-reconstruction" paradigm, SHMT works in a self-supervised manner, freeing itself from the misguidance of imprecise pseudo-paired data. Furthermore, to accommodate a variety of makeup styles, hierarchical texture details are decomposed via a Laplacian pyramid and selectively introduced to the content representation. Finally, we design a novel Iterative Dual Alignment (IDA) module that dynamically adjusts the injection condition of the diffusion model, allowing the alignment errors caused by the domain gap between content and makeup representations to be corrected. Extensive quantitative and qualitative analyses demonstrate the effectiveness of our method. Our code is available at \url{this https URL}.

Title: Making Bias Amplification in Balanced Datasets Directional and Interpretable

• Authors: Bhanu Tokas, Rahul Nair, Hannah Kerner
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11060
• Pdf URL: https://arxiv.org/pdf/2412.11060
• Copy Paste: [[2412.11060]] Making Bias Amplification in Balanced Datasets Directional and Interpretable(https://arxiv.org/abs/2412.11060)
• Keywords: protect, attack
• Abstract: Most of the ML datasets we use today are biased. When we train models on these biased datasets, they often not only learn dataset biases but can also amplify them -- a phenomenon known as bias amplification. Several co-occurrence-based metrics have been proposed to measure bias amplification between a protected attribute A (e.g., gender) and a task T (e.g., cooking). However, these metrics fail to measure biases when A is balanced with T. To measure bias amplification in balanced datasets, recent work proposed a predictability-based metric called leakage amplification. However, leakage amplification cannot identify the direction in which biases are amplified. In this work, we propose a new predictability-based metric called directional predictability amplification (DPA). DPA measures directional bias amplification, even for balanced datasets. Unlike leakage amplification, DPA is easier to interpret and less sensitive to attacker models (a hyperparameter in predictability-based metrics). Our experiments on tabular and image datasets show that DPA is an effective metric for measuring directional bias amplification. The code will be available soon.

Title: Classification Drives Geographic Bias in Street Scene Segmentation

• Authors: Rahul Nair, Gabriel Tseng, Esther Rolf, Bhanu Tokas, Hannah Kerner
• Subjects: cs.CV, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11061
• Pdf URL: https://arxiv.org/pdf/2412.11061
• Copy Paste: [[2412.11061]] Classification Drives Geographic Bias in Street Scene Segmentation(https://arxiv.org/abs/2412.11061)
• Keywords: robust, segmentation
• Abstract: Previous studies showed that image datasets lacking geographic diversity can lead to biased performance in models trained on them. While earlier work studied general-purpose image datasets (e.g., ImageNet) and simple tasks like image recognition, we investigated geo-biases in real-world driving datasets on a more complex task: instance segmentation. We examined if instance segmentation models trained on European driving scenes (Eurocentric models) are geo-biased. Consistent with previous work, we found that Eurocentric models were geo-biased. Interestingly, we found that geo-biases came from classification errors rather than localization errors, with classification errors alone contributing 10-90% of the geo-biases in segmentation and 19-88% of the geo-biases in detection. This showed that while classification is geo-biased, localization (including detection and segmentation) is geographically robust. Our findings show that in region-specific models (e.g., Eurocentric models), geo-biases from classification errors can be significantly mitigated by using coarser classes (e.g., grouping car, bus, and truck as 4-wheeler).

Title: Learning Robust and Privacy-Preserving Representations via Information Theory

• Authors: Binghui Zhang, Sayedeh Leila Noorbakhsh, Yun Dong, Yuan Hong, Binghui Wang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.11066
• Pdf URL: https://arxiv.org/pdf/2412.11066
• Copy Paste: [[2412.11066]] Learning Robust and Privacy-Preserving Representations via Information Theory(https://arxiv.org/abs/2412.11066)
• Keywords: security, privacy, attack, robust
• Abstract: Machine learning models are vulnerable to both security attacks (e.g., adversarial examples) and privacy attacks (e.g., private attribute inference). We take the first step to mitigate both the security and privacy attacks, and maintain task utility as well. Particularly, we propose an information-theoretic framework to achieve the goals through the lens of representation learning, i.e., learning representations that are robust to both adversarial examples and attribute inference adversaries. We also derive novel theoretical results under our framework, e.g., the inherent trade-off between adversarial robustness/utility and attribute privacy, and guaranteed attribute privacy leakage against attribute inference adversaries.

Title: CFSynthesis: Controllable and Free-view 3D Human Video Synthesis

• Authors: Cui Liyuan, Xu Xiaogang, Dong Wenqi, Yang Zesong, Bao Hujun, Cui Zhaopeng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11067
• Pdf URL: https://arxiv.org/pdf/2412.11067
• Copy Paste: [[2412.11067]] CFSynthesis: Controllable and Free-view 3D Human Video Synthesis(https://arxiv.org/abs/2412.11067)
• Keywords: diffusion
• Abstract: Human video synthesis aims to create lifelike characters in various environments, with wide applications in VR, storytelling, and content creation. While 2D diffusion-based methods have made significant progress, they struggle to generalize to complex 3D poses and varying scene backgrounds. To address these limitations, we introduce CFSynthesis, a novel framework for generating high-quality human videos with customizable attributes, including identity, motion, and scene configurations. Our method leverages a texture-SMPL-based representation to ensure consistent and stable character appearances across free viewpoints. Additionally, we introduce a novel foreground-background separation strategy that effectively decomposes the scene as foreground and background, enabling seamless integration of user-defined backgrounds. Experimental results on multiple datasets show that CFSynthesis not only achieves state-of-the-art performance in complex human animations but also adapts effectively to 3D motions in free-view and user-specified scenarios.

Title: HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generation

• Authors: Tengfei Liu, Jiapu Wang, Yongli Hu, Mingjie Li, Junfei Yi, Xiaojun Chang, Junbin Gao, Baocai Yin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11070
• Pdf URL: https://arxiv.org/pdf/2412.11070
• Copy Paste: [[2412.11070]] HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generation(https://arxiv.org/abs/2412.11070)
• Keywords: large language model
• Abstract: Radiology report generation (RRG) models typically focus on individual exams, often overlooking the integration of historical visual or textual data, which is crucial for patient follow-ups. Traditional methods usually struggle with long sequence dependencies when incorporating historical information, but large language models (LLMs) excel at in-context learning, making them well-suited for analyzing longitudinal medical data. In light of this, we propose a novel Historical-Constrained Large Language Models (HC-LLM) framework for RRG, empowering LLMs with longitudinal report generation capabilities by constraining the consistency and differences between longitudinal images and their corresponding reports. Specifically, our approach extracts both time-shared and time-specific features from longitudinal chest X-rays and diagnostic reports to capture disease progression. Then, we ensure consistent representation by applying intra-modality similarity constraints and aligning various features across modalities with multimodal contrastive and structural constraints. These combined constraints effectively guide the LLMs in generating diagnostic reports that accurately reflect the progression of the disease, achieving state-of-the-art results on the Longitudinal-MIMIC dataset. Notably, our approach performs well even without historical data during testing and can be easily adapted to other multimodal large models, enhancing its versatility.

Title: Navigating Towards Fairness with Data Selection

• Authors: Yixuan Zhang, Zhidong Li, Yang Wang, Fang Chen, Xuhui Fan, Feng Zhou
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.11072
• Pdf URL: https://arxiv.org/pdf/2412.11072
• Copy Paste: [[2412.11072]] Navigating Towards Fairness with Data Selection(https://arxiv.org/abs/2412.11072)
• Keywords: fair
• Abstract: Machine learning algorithms often struggle to eliminate inherent data biases, particularly those arising from unreliable labels, which poses a significant challenge in ensuring fairness. Existing fairness techniques that address label bias typically involve modifying models and intervening in the training process, but these lack flexibility for large-scale datasets. To address this limitation, we introduce a data selection method designed to efficiently and flexibly mitigate label bias, tailored to more practical needs. Our approach utilizes a zero-shot predictor as a proxy model that simulates training on a clean holdout set. This strategy, supported by peer predictions, ensures the fairness of the proxy model and eliminates the need for an additional holdout set, which is a common requirement in previous methods. Without altering the classifier's architecture, our modality-agnostic method effectively selects appropriate training data and has proven efficient and effective in handling label bias and improving fairness across diverse datasets in experimental evaluations.

Title: MoRe: Class Patch Attention Needs Regularization for Weakly Supervised Semantic Segmentation

• Authors: Zhiwei Yang, Yucong Meng, Kexue Fu, Shuo Wang, Zhijian Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11076
• Pdf URL: https://arxiv.org/pdf/2412.11076
• Copy Paste: [[2412.11076]] MoRe: Class Patch Attention Needs Regularization for Weakly Supervised Semantic Segmentation(https://arxiv.org/abs/2412.11076)
• Keywords: transformer, segmentation
• Abstract: Weakly Supervised Semantic Segmentation (WSSS) with image-level labels typically uses Class Activation Maps (CAM) to achieve dense predictions. Recently, Vision Transformer (ViT) has provided an alternative to generate localization maps from class-patch attention. However, due to insufficient constraints on modeling such attention, we observe that the Localization Attention Maps (LAM) often struggle with the artifact issue, i.e., patch regions with minimal semantic relevance are falsely activated by class tokens. In this work, we propose MoRe to address this issue and further explore the potential of LAM. Our findings suggest that imposing additional regularization on class-patch attention is necessary. To this end, we first view the attention as a novel directed graph and propose the Graph Category Representation module to implicitly regularize the interaction among class-patch entities. It ensures that class tokens dynamically condense the related patch information and suppress unrelated artifacts at a graph level. Second, motivated by the observation that CAM from classification weights maintains smooth localization of objects, we devise the Localization-informed Regularization module to explicitly regularize the class-patch attention. It directly mines the token relations from CAM and further supervises the consistency between class and patch tokens in a learnable manner. Extensive experiments are conducted on PASCAL VOC and MS COCO, validating that MoRe effectively addresses the artifact issue and achieves state-of-the-art performance, surpassing recent single-stage and even multi-stage methods. Code is available at this https URL.

Title: Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval

• Authors: Yuanmin Tang, Xiaoting Qin, Jue Zhang, Jing Yu, Gaopeng Gou, Gang Xiong, Qingwei Ling, Saravan Rajmohan, Dongmei Zhang, Qi Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11077
• Pdf URL: https://arxiv.org/pdf/2412.11077
• Copy Paste: [[2412.11077]] Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrieval(https://arxiv.org/abs/2412.11077)
• Keywords: large language model
• Abstract: Composed Image Retrieval (CIR) aims to retrieve target images that closely resemble a reference image while integrating user-specified textual modifications, thereby capturing user intent more precisely. Existing training-free zero-shot CIR (ZS-CIR) methods often employ a two-stage process: they first generate a caption for the reference image and then use Large Language Models for reasoning to obtain a target description. However, these methods suffer from missing critical visual details and limited reasoning capabilities, leading to suboptimal retrieval performance. To address these challenges, we propose a novel, training-free one-stage method, One-Stage Reflective Chain-of-Thought Reasoning for ZS-CIR (OSrCIR), which employs Multimodal Large Language Models to retain essential visual information in a single-stage reasoning process, eliminating the information loss seen in two-stage methods. Our Reflective Chain-of-Thought framework further improves interpretative accuracy by aligning manipulation intent with contextual cues from reference images. OSrCIR achieves performance gains of 1.80% to 6.44% over existing training-free methods across multiple tasks, setting new state-of-the-art results in ZS-CIR and enhancing its utility in vision-language applications. Our code will be available at this https URL.

Title: EquiFlow: Equivariant Conditional Flow Matching with Optimal Transport for 3D Molecular Conformation Prediction

• Authors: Qingwen Tian, Yuxin Xu, Yixuan Yang, Zhen Wang, Ziqi Liu, Pengju Yan, Xiaolin Li
• Subjects: cs.LG, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2412.11082
• Pdf URL: https://arxiv.org/pdf/2412.11082
• Copy Paste: [[2412.11082]] EquiFlow: Equivariant Conditional Flow Matching with Optimal Transport for 3D Molecular Conformation Prediction(https://arxiv.org/abs/2412.11082)
• Keywords: diffusion
• Abstract: Molecular 3D conformations play a key role in determining how molecules interact with other molecules or protein surfaces. Recent deep learning advancements have improved conformation prediction, but slow training speeds and difficulties in utilizing high-degree features limit performance. We propose EquiFlow, an equivariant conditional flow matching model with optimal transport. EquiFlow uniquely applies conditional flow matching in molecular 3D conformation prediction, leveraging simulation-free training to address slow training speeds. It uses a modified Equiformer model to encode Cartesian molecular conformations along with their atomic and bond properties into higher-degree embeddings. Additionally, EquiFlow employs an ODE solver, providing faster inference speeds compared to diffusion models with SDEs. Experiments on the QM9 dataset show that EquiFlow predicts small molecule conformations more accurately than current state-of-the-art models.

Title: BarcodeMamba: State Space Models for Biodiversity Analysis

• Authors: Tiancheng Gao, Graham W. Taylor
• Subjects: cs.LG, q-bio.GN, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2412.11084
• Pdf URL: https://arxiv.org/pdf/2412.11084
• Copy Paste: [[2412.11084]] BarcodeMamba: State Space Models for Biodiversity Analysis(https://arxiv.org/abs/2412.11084)
• Keywords: transformer
• Abstract: DNA barcodes are crucial in biodiversity analysis for building automatic identification systems that recognize known species and discover unseen species. Unlike human genome modeling, barcode-based invertebrate identification poses challenges in the vast diversity of species and taxonomic complexity. Among Transformer-based foundation models, BarcodeBERT excelled in species-level identification of invertebrates, highlighting the effectiveness of self-supervised pretraining on barcode-specific datasets. Recently, structured state space models (SSMs) have emerged, with a time complexity that scales sub-quadratically with the context length. SSMs provide an efficient parameterization of sequence modeling relative to attention-based architectures. Given the success of Mamba and Mamba-2 in natural language, we designed BarcodeMamba, a performant and efficient foundation model for DNA barcodes in biodiversity analysis. We conducted a comprehensive ablation study on the impacts of self-supervised training and tokenization methods, and compared both versions of Mamba layers in terms of expressiveness and their capacity to identify "unseen" species held back from training. Our study shows that BarcodeMamba has better performance than BarcodeBERT even when using only 8.3% as many parameters, and improves accuracy to 99.2% on species-level accuracy in linear probing without fine-tuning for "seen" species. In our scaling study, BarcodeMamba with 63.6% of BarcodeBERT's parameters achieved 70.2% genus-level accuracy in 1-nearest neighbor (1-NN) probing for unseen species. The code repository to reproduce our experiments is available at this https URL.

Title: GraphMoRE: Mitigating Topological Heterogeneity via Mixture of Riemannian Experts

• Authors: Zihao Guo, Qingyun Sun, Haonan Yuan, Xingcheng Fu, Min Zhou, Yisen Gao, Jianxin Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11085
• Pdf URL: https://arxiv.org/pdf/2412.11085
• Copy Paste: [[2412.11085]] GraphMoRE: Mitigating Topological Heterogeneity via Mixture of Riemannian Experts(https://arxiv.org/abs/2412.11085)
• Keywords: fair
• Abstract: Real-world graphs have inherently complex and diverse topological patterns, known as topological heterogeneity. Most existing works learn graph representation in a single constant curvature space that is insufficient to match the complex geometric shapes, resulting in low-quality embeddings with high distortion. This also constitutes a critical challenge for graph foundation models, which are expected to uniformly handle a wide variety of diverse graph data. Recent studies have indicated that product manifold gains the possibility to address topological heterogeneity. However, the product manifold is still homogeneous, which is inadequate and inflexible for representing the mixed heterogeneous topology. In this paper, we propose a novel Graph Mixture of Riemannian Experts (GraphMoRE) framework to effectively tackle topological heterogeneity by personalized fine-grained topology geometry pattern preservation. Specifically, to minimize the embedding distortion, we propose a topology-aware gating mechanism to select the optimal embedding space for each node. By fusing the outputs of diverse Riemannian experts with learned gating weights, we construct personalized mixed curvature spaces for nodes, effectively embedding the graph into a heterogeneous manifold with varying curvatures at different points. Furthermore, to fairly measure pairwise distances between different embedding spaces, we present a concise and effective alignment strategy. Extensive experiments on real-world and synthetic datasets demonstrate that our method achieves superior performance with lower distortion, highlighting its potential for modeling complex graphs with topological heterogeneity, and providing a novel architectural perspective for graph foundation models.

Title: DynamicScaler: Seamless and Scalable Video Generation for Panoramic Scenes

• Authors: Jinxiu Liu, Shaoheng Lin, Yinxiao Li, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11100
• Pdf URL: https://arxiv.org/pdf/2412.11100
• Copy Paste: [[2412.11100]] DynamicScaler: Seamless and Scalable Video Generation for Panoramic Scenes(https://arxiv.org/abs/2412.11100)
• Keywords: diffusion
• Abstract: The increasing demand for immersive AR/VR applications and spatial intelligence has heightened the need to generate high-quality scene-level and 360° panoramic video. However, most video diffusion models are constrained by limited resolution and aspect ratio, which restricts their applicability to scene-level dynamic content synthesis. In this work, we propose the DynamicScaler, addressing these challenges by enabling spatially scalable and panoramic dynamic scene synthesis that preserves coherence across panoramic scenes of arbitrary size. Specifically, we introduce a Offset Shifting Denoiser, facilitating efficient, synchronous, and coherent denoising panoramic dynamic scenes via a diffusion model with fixed resolution through a seamless rotating Window, which ensures seamless boundary transitions and consistency across the entire panoramic space, accommodating varying resolutions and aspect ratios. Additionally, we employ a Global Motion Guidance mechanism to ensure both local detail fidelity and global motion continuity. Extensive experiments demonstrate our method achieves superior content and motion quality in panoramic scene-level video generation, offering a training-free, efficient, and scalable solution for immersive dynamic scene creation with constant VRAM consumption regardless of the output video resolution. Our project page is available at \url{this https URL}.

Title: Empowering LLMs to Understand and Generate Complex Vector Graphics

• Authors: Ximing Xing, Juncheng Hu, Guotao Liang, Jing Zhang, Dong Xu, Qian Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11102
• Pdf URL: https://arxiv.org/pdf/2412.11102
• Copy Paste: [[2412.11102]] Empowering LLMs to Understand and Generate Complex Vector Graphics(https://arxiv.org/abs/2412.11102)
• Keywords: large language model
• Abstract: The unprecedented advancements in Large Language Models (LLMs) have profoundly impacted natural language processing but have yet to fully embrace the realm of scalable vector graphics (SVG) generation. While LLMs encode partial knowledge of SVG data from web pages during training, recent findings suggest that semantically ambiguous and tokenized representations within LLMs may result in hallucinations in vector primitive predictions. Additionally, LLM training typically lacks modeling and understanding of the rendering sequence of vector paths, which can lead to occlusion between output vector primitives. In this paper, we present LLM4SVG, an initial yet substantial step toward bridging this gap by enabling LLMs to better understand and generate vector graphics. LLM4SVG facilitates a deeper understanding of SVG components through learnable semantic tokens, which precisely encode these tokens and their corresponding properties to generate semantically aligned SVG outputs. Using a series of learnable semantic tokens, a structured dataset for instruction following is developed to support comprehension and generation across two primary tasks. Our method introduces a modular architecture to existing large language models, integrating semantic tags, vector instruction encoders, fine-tuned commands, and powerful LLMs to tightly combine geometric, appearance, and language information. To overcome the scarcity of SVG-text instruction data, we developed an automated data generation pipeline that collected a massive dataset of more than 250k SVG data and 580k SVG-text instructions, which facilitated the adoption of the two-stage training strategy popular in LLM development. By exploring various training strategies, we developed LLM4SVG, which significantly moves beyond optimized rendering-based approaches and language-model-based baselines to achieve remarkable results in human evaluation tasks.

Title: SpearBot: Leveraging Large Language Models in a Generative-Critique Framework for Spear-Phishing Email Generation

• Authors: Qinglin Qi, Yun Luo, Yijia Xu, Wenbo Guo, Yong Fang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.11109
• Pdf URL: https://arxiv.org/pdf/2412.11109
• Copy Paste: [[2412.11109]] SpearBot: Leveraging Large Language Models in a Generative-Critique Framework for Spear-Phishing Email Generation(https://arxiv.org/abs/2412.11109)
• Keywords: security, generative, large language model
• Abstract: Large Language Models (LLMs) are increasingly capable, aiding in tasks such as content generation, yet they also pose risks, particularly in generating harmful spear-phishing emails. These emails, crafted to entice clicks on malicious URLs, threaten personal information security. This paper proposes an adversarial framework, SpearBot, which utilizes LLMs to generate spear-phishing emails with various phishing strategies. Through specifically crafted jailbreak prompts, SpearBot circumvents security policies and introduces other LLM instances as critics. When a phishing email is identified by the critic, SpearBot refines the generated email based on the critique feedback until it can no longer be recognized as phishing, thereby enhancing its deceptive quality. To evaluate the effectiveness of SpearBot, we implement various machine-based defenders and assess how well the phishing emails generated could deceive them. Results show these emails often evade detection to a large extent, underscoring their deceptive quality. Additionally, human evaluations of the emails' readability and deception are conducted through questionnaires, confirming their convincing nature and the significant potential harm of the generated phishing emails.

Title: Impact of Adversarial Attacks on Deep Learning Model Explainability

• Authors: Gazi Nazia Nur, Mohammad Ahnaf Sadat
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11119
• Pdf URL: https://arxiv.org/pdf/2412.11119
• Copy Paste: [[2412.11119]] Impact of Adversarial Attacks on Deep Learning Model Explainability(https://arxiv.org/abs/2412.11119)
• Keywords: attack, robust, extraction, explainability
• Abstract: In this paper, we investigate the impact of adversarial attacks on the explainability of deep learning models, which are commonly criticized for their black-box nature despite their capacity for autonomous feature extraction. This black-box nature can affect the perceived trustworthiness of these models. To address this, explainability techniques such as GradCAM, SmoothGrad, and LIME have been developed to clarify model decision-making processes. Our research focuses on the robustness of these explanations when models are subjected to adversarial attacks, specifically those involving subtle image perturbations that are imperceptible to humans but can significantly mislead models. For this, we utilize attack methods like the Fast Gradient Sign Method (FGSM) and the Basic Iterative Method (BIM) and observe their effects on model accuracy and explanations. The results reveal a substantial decline in model accuracy, with accuracies dropping from 89.94% to 58.73% and 45.50% under FGSM and BIM attacks, respectively. Despite these declines in accuracy, the explanation of the models measured by metrics such as Intersection over Union (IoU) and Root Mean Square Error (RMSE) shows negligible changes, suggesting that these metrics may not be sensitive enough to detect the presence of adversarial perturbations.

Title: Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning

• Authors: Yun Qu, Yuhang Jiang, Boyuan Wang, Yixiu Mao, Cheems Wang, Chang Liu, Xiangyang Ji
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11120
• Pdf URL: https://arxiv.org/pdf/2412.11120
• Copy Paste: [[2412.11120]] Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning(https://arxiv.org/abs/2412.11120)
• Keywords: large language model
• Abstract: Reinforcement learning (RL) often encounters delayed and sparse feedback in real-world applications, even with only episodic rewards. Previous approaches have made some progress in reward redistribution for credit assignment but still face challenges, including training difficulties due to redundancy and ambiguous attributions stemming from overlooking the multifaceted nature of mission performance evaluation. Hopefully, Large Language Model (LLM) encompasses fruitful decision-making knowledge and provides a plausible tool for reward redistribution. Even so, deploying LLM in this case is non-trivial due to the misalignment between linguistic knowledge and the symbolic form requirement, together with inherent randomness and hallucinations in inference. To tackle these issues, we introduce LaRe, a novel LLM-empowered symbolic-based decision-making framework, to improve credit assignment. Key to LaRe is the concept of the Latent Reward, which works as a multi-dimensional performance evaluation, enabling more interpretable goal attainment from various perspectives and facilitating more effective reward redistribution. We examine that semantically generated code from LLM can bridge linguistic knowledge and symbolic latent rewards, as it is executable for symbolic objects. Meanwhile, we design latent reward self-verification to increase the stability and reliability of LLM inference. Theoretically, reward-irrelevant redundancy elimination in the latent reward benefits RL performance from more accurate reward estimation. Extensive experimental results witness that LaRe (i) achieves superior temporal credit assignment to SOTA methods, (ii) excels in allocating contributions among multiple agents, and (iii) outperforms policies trained with ground truth rewards for certain tasks.

Title: Combating Multimodal LLM Hallucination via Bottom-up Holistic Reasoning

• Authors: Shengqiong Wu, Hao Fei, Liangming Pan, William Yang Wang, Shuicheng Yan, Tat-Seng Chua
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11124
• Pdf URL: https://arxiv.org/pdf/2412.11124
• Copy Paste: [[2412.11124]] Combating Multimodal LLM Hallucination via Bottom-up Holistic Reasoning(https://arxiv.org/abs/2412.11124)
• Keywords: large language model
• Abstract: Recent advancements in multimodal large language models (MLLMs) have shown unprecedented capabilities in advancing various vision-language tasks. However, MLLMs face significant challenges with hallucinations, and misleading outputs that do not align with the input data. While existing efforts are paid to combat MLLM hallucinations, several pivotal challenges are still unsolved. First, while current approaches aggressively focus on addressing errors at the perception level, another important type at the cognition level requiring factual commonsense can be overlooked. In addition, existing methods might fall short in finding a more effective way to represent visual input, which is yet a key bottleneck that triggers visual hallucinations. Moreover, MLLMs can frequently be misled by faulty textual inputs and cause hallucinations, while unfortunately, this type of issue has long been overlooked by existing studies. Inspired by human intuition in handling hallucinations, this paper introduces a novel bottom-up reasoning framework. Our framework systematically addresses potential issues in both visual and textual inputs by verifying and integrating perception-level information with cognition-level commonsense knowledge, ensuring more reliable outputs. Extensive experiments demonstrate significant improvements in multiple hallucination benchmarks after integrating MLLMs with the proposed framework. In-depth analyses reveal the great potential of our methods in addressing perception- and cognition-level hallucinations.

Title: Feature engineering vs. deep learning for paper section identification: Toward applications in Chinese medical literature

• Authors: Sijia Zhou, Xin Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11125
• Pdf URL: https://arxiv.org/pdf/2412.11125
• Copy Paste: [[2412.11125]] Feature engineering vs. deep learning for paper section identification: Toward applications in Chinese medical literature(https://arxiv.org/abs/2412.11125)
• Keywords: extraction
• Abstract: Section identification is an important task for library science, especially knowledge management. Identifying the sections of a paper would help filter noise in entity and relation extraction. In this research, we studied the paper section identification problem in the context of Chinese medical literature analysis, where the subjects, methods, and results are more valuable from a physician's perspective. Based on previous studies on English literature section identification, we experiment with the effective features to use with classic machine learning algorithms to tackle the problem. It is found that Conditional Random Fields, which consider sentence interdependency, is more effective in combining different feature sets, such as bag-of-words, part-of-speech, and headings, for Chinese literature section identification. Moreover, we find that classic machine learning algorithms are more effective than generic deep learning models for this problem. Based on these observations, we design a novel deep learning model, the Structural Bidirectional Long Short-Term Memory (SLSTM) model, which models word and sentence interdependency together with the contextual information. Experiments on a human-curated asthma literature dataset show that our approach outperforms the traditional machine learning methods and other deep learning methods and achieves close to 90% precision and recall in the task. The model shows good potential for use in other text mining tasks. The research has significant methodological and practical implications.

Title: ViSymRe: Vision-guided Multimodal Symbolic Regression

• Authors: Da Li, Junping Yin, Jin Xu, Xinxin Li, Juan Zhang
• Subjects: cs.LG, cs.AI, cs.SC
• Abstract URL: https://arxiv.org/abs/2412.11139
• Pdf URL: https://arxiv.org/pdf/2412.11139
• Copy Paste: [[2412.11139]] ViSymRe: Vision-guided Multimodal Symbolic Regression(https://arxiv.org/abs/2412.11139)
• Keywords: interpretability
• Abstract: Symbolic regression automatically searches for mathematical equations to reveal underlying mechanisms within datasets, offering enhanced interpretability compared to black box models. Traditionally, symbolic regression has been considered to be purely numeric-driven, with insufficient attention given to the potential contributions of visual information in augmenting this process. When dealing with high-dimensional and complex datasets, existing symbolic regression models are often inefficient and tend to generate overly complex equations, making subsequent mechanism analysis complicated. In this paper, we propose the vision-guided multimodal symbolic regression model, called ViSymRe, that systematically explores how visual information can improve various metrics of symbolic regression. Compared to traditional models, our proposed model has the following innovations: (1) It integrates three modalities: vision, symbol and numeric to enhance symbolic regression, enabling the model to benefit from the strengths of each modality; (2) It establishes a meta-learning framework that can learn from historical experiences to efficiently solve new symbolic regression problems; (3) It emphasizes the simplicity and structural rationality of the equations rather than merely numerical fitting. Extensive experiments show that our proposed model exhibits strong generalization capability and noise resistance. The equations it generates outperform state-of-the-art numeric-only baselines in terms of fitting effect, simplicity and structural accuracy, thus being able to facilitate accurate mechanism analysis and the development of theoretical models.

Title: AD-LLM: Benchmarking Large Language Models for Anomaly Detection

• Authors: Tiankai Yang, Yi Nian, Shawn Li, Ruiyao Xu, Yuangang Li, Jiaqi Li, Zhuo Xiao, Xiyang Hu, Ryan Rossi, Kaize Ding, Xia Hu, Yue Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11142
• Pdf URL: https://arxiv.org/pdf/2412.11142
• Copy Paste: [[2412.11142]] AD-LLM: Benchmarking Large Language Models for Anomaly Detection(https://arxiv.org/abs/2412.11142)
• Keywords: large language model
• Abstract: Anomaly detection (AD) is an important machine learning task with many real-world uses, including fraud detection, medical diagnosis, and industrial monitoring. Within natural language processing (NLP), AD helps detect issues like spam, misinformation, and unusual user activity. Although large language models (LLMs) have had a strong impact on tasks such as text generation and summarization, their potential in AD has not been studied enough. This paper introduces AD-LLM, the first benchmark that evaluates how LLMs can help with NLP anomaly detection. We examine three key tasks: (i) zero-shot detection, using LLMs' pre-trained knowledge to perform AD without tasks-specific training; (ii) data augmentation, generating synthetic data and category descriptions to improve AD models; and (iii) model selection, using LLMs to suggest unsupervised AD models. Through experiments with different datasets, we find that LLMs can work well in zero-shot AD, that carefully designed augmentation methods are useful, and that explaining model selection for specific datasets remains challenging. Based on these results, we outline six future research directions on LLMs for AD.

Title: The Superalignment of Superhuman Intelligence with Large Language Models

• Authors: Minlie Huang, Yingkang Wang, Shiyao Cui, Pei Ke, Jie Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11145
• Pdf URL: https://arxiv.org/pdf/2412.11145
• Copy Paste: [[2412.11145]] The Superalignment of Superhuman Intelligence with Large Language Models(https://arxiv.org/abs/2412.11145)
• Keywords: attack, large language model
• Abstract: We have witnessed superhuman intelligence thanks to the fast development of large language models and multimodal language models. As the application of such superhuman models becomes more and more common, a critical question rises here: how can we ensure superhuman models are still safe, reliable and aligned well to human values? In this position paper, we discuss the concept of superalignment from the learning perspective to answer this question by outlining the learning paradigm shift from large-scale pretraining, supervised fine-tuning, to alignment training. We define superalignment as designing effective and efficient alignment algorithms to learn from noisy-labeled data (point-wise samples or pair-wise preference data) in a scalable way when the task becomes very complex for human experts to annotate and the model is stronger than human experts. We highlight some key research problems in superalignment, namely, weak-to-strong generalization, scalable oversight, and evaluation. We then present a conceptual framework for superalignment, which consists of three modules: an attacker which generates adversary queries trying to expose the weaknesses of a learner model; a learner which will refine itself by learning from scalable feedbacks generated by a critic model along with minimal human experts; and a critic which generates critics or explanations for a given query-response pair, with a target of improving the learner by criticizing. We discuss some important research problems in each component of this framework and highlight some interesting research ideas that are closely related to our proposed framework, for instance, self-alignment, self-play, self-refinement, and more. Last, we highlight some future research directions for superalignment, including identification of new emergent risks and multi-dimensional alignment.

Title: A Comprehensive Survey of Action Quality Assessment: Method and Benchmark

• Authors: Kanglei Zhou, Ruizhi Cai, Liyuan Wang, Hubert P. H. Shum, Xiaohui Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11149
• Pdf URL: https://arxiv.org/pdf/2412.11149
• Copy Paste: [[2412.11149]] A Comprehensive Survey of Action Quality Assessment: Method and Benchmark(https://arxiv.org/abs/2412.11149)
• Keywords: fair
• Abstract: Action Quality Assessment (AQA) quantitatively evaluates the quality of human actions, providing automated assessments that reduce biases in human judgment. Its applications span domains such as sports analysis, skill assessment, and medical care. Recent advances in AQA have introduced innovative methodologies, but similar methods often intertwine across different domains, highlighting the fragmented nature that hinders systematic reviews. In addition, the lack of a unified benchmark and limited computational comparisons hinder consistent evaluation and fair assessment of AQA approaches. In this work, we address these gaps by systematically analyzing over 150 AQA-related papers to develop a hierarchical taxonomy, construct a unified benchmark, and provide an in-depth analysis of current trends, challenges, and future directions. Our hierarchical taxonomy categorizes AQA methods based on input modalities (video, skeleton, multi-modal) and their specific characteristics, highlighting the evolution and interrelations across various approaches. To promote standardization, we present a unified benchmark, integrating diverse datasets to evaluate the assessment precision and computational efficiency. Finally, we review emerging task-specific applications and identify under-explored challenges in AQA, providing actionable insights into future research directions. This survey aims to deepen understanding of AQA progress, facilitate method comparison, and guide future innovations. The project web page can be found at this https URL.

Title: Dual-Schedule Inversion: Training- and Tuning-Free Inversion for Real Image Editing

• Authors: Jiancheng Huang, Yi Huang, Jianzhuang Liu, Donghao Zhou, Yifan Liu, Shifeng Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11152
• Pdf URL: https://arxiv.org/pdf/2412.11152
• Copy Paste: [[2412.11152]] Dual-Schedule Inversion: Training- and Tuning-Free Inversion for Real Image Editing(https://arxiv.org/abs/2412.11152)
• Keywords: diffusion
• Abstract: Text-conditional image editing is a practical AIGC task that has recently emerged with great commercial and academic value. For real image editing, most diffusion model-based methods use DDIM Inversion as the first stage before editing. However, DDIM Inversion often results in reconstruction failure, leading to unsatisfactory performance for downstream editing. To address this problem, we first analyze why the reconstruction via DDIM Inversion fails. We then propose a new inversion and sampling method named Dual-Schedule Inversion. We also design a classifier to adaptively combine Dual-Schedule Inversion with different editing methods for user-friendly image editing. Our work can achieve superior reconstruction and editing performance with the following advantages: 1) It can reconstruct real images perfectly without fine-tuning, and its reversibility is guaranteed mathematically. 2) The edited object/scene conforms to the semantics of the text prompt. 3) The unedited parts of the object/scene retain the original identity.

Title: Early Concept Drift Detection via Prediction Uncertainty

• Authors: Pengqian Lu, Jie Lu, Anjin Liu, Guangquan Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11158
• Pdf URL: https://arxiv.org/pdf/2412.11158
• Copy Paste: [[2412.11158]] Early Concept Drift Detection via Prediction Uncertainty(https://arxiv.org/abs/2412.11158)
• Keywords: robust
• Abstract: Concept drift, characterized by unpredictable changes in data distribution over time, poses significant challenges to machine learning models in streaming data scenarios. Although error rate-based concept drift detectors are widely used, they often fail to identify drift in the early stages when the data distribution changes but error rates remain constant. This paper introduces the Prediction Uncertainty Index (PU-index), derived from the prediction uncertainty of the classifier, as a superior alternative to the error rate for drift detection. Our theoretical analysis demonstrates that: (1) The PU-index can detect drift even when error rates remain stable. (2) Any change in the error rate will lead to a corresponding change in the PU-index. These properties make the PU-index a more sensitive and robust indicator for drift detection compared to existing methods. We also propose a PU-index-based Drift Detector (PUDD) that employs a novel Adaptive PU-index Bucketing algorithm for detecting drift. Empirical evaluations on both synthetic and real-world datasets demonstrate PUDD's efficacy in detecting drift in structured and image data.

Title: Why and How: Knowledge-Guided Learning for Cross-Spectral Image Patch Matching

• Authors: Chuang Yu, Yunpeng Liu, Jinmiao Zhao, Xiangyu Yue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11161
• Pdf URL: https://arxiv.org/pdf/2412.11161
• Copy Paste: [[2412.11161]] Why and How: Knowledge-Guided Learning for Cross-Spectral Image Patch Matching(https://arxiv.org/abs/2412.11161)
• Keywords: extraction
• Abstract: Recently, cross-spectral image patch matching based on feature relation learning has attracted extensive attention. However, performance bottleneck problems have gradually emerged in existing methods. To address this challenge, we make the first attempt to explore a stable and efficient bridge between descriptor learning and metric learning, and construct a knowledge-guided learning network (KGL-Net), which achieves amazing performance improvements while abandoning complex network structures. Specifically, we find that there is feature extraction consistency between metric learning based on feature difference learning and descriptor learning based on Euclidean distance. This provides the foundation for bridge building. To ensure the stability and efficiency of the constructed bridge, on the one hand, we conduct an in-depth exploration of 20 combined network architectures. On the other hand, a feature-guided loss is constructed to achieve mutual guidance of features. In addition, unlike existing methods, we consider that the feature mapping ability of the metric branch should receive more attention. Therefore, a hard negative sample mining for metric learning (HNSM-M) strategy is constructed. To the best of our knowledge, this is the first time that hard negative sample mining for metric networks has been implemented and brings significant performance gains. Extensive experimental results show that our KGL-Net achieves SOTA performance in three different cross-spectral image patch matching scenarios. Our code are available at this https URL.

Title: Missing data imputation for noisy time-series data and applications in healthcare

• Authors: Lien P. Le, Xuan-Hien Nguyen Thi, Thu Nguyen, Michael A. Riegler, Pål Halvorsen, Binh T. Nguyen
• Subjects: cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2412.11164
• Pdf URL: https://arxiv.org/pdf/2412.11164
• Copy Paste: [[2412.11164]] Missing data imputation for noisy time-series data and applications in healthcare(https://arxiv.org/abs/2412.11164)
• Keywords: transformer
• Abstract: Healthcare time series data is vital for monitoring patient activity but often contains noise and missing values due to various reasons such as sensor errors or data interruptions. Imputation, i.e., filling in the missing values, is a common way to deal with this issue. In this study, we compare imputation methods, including Multiple Imputation with Random Forest (MICE-RF) and advanced deep learning approaches (SAITS, BRITS, Transformer) for noisy, missing time series data in terms of MAE, F1-score, AUC, and MCC, across missing data rates (10 % - 80 %). Our results show that MICE-RF can effectively impute missing data compared to deep learning methods and the improvement in classification of data imputed indicates that imputation can have denoising effects. Therefore, using an imputation algorithm on time series with missing data can, at the same time, offer denoising effects.

Title: OTLRM: Orthogonal Learning-based Low-Rank Metric for Multi-Dimensional Inverse Problems

• Authors: Xiangming Wang, Haijin Zeng, Jiaoyang Chen, Sheng Liu, Yongyong Chen, Guoqing Chao
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11165
• Pdf URL: https://arxiv.org/pdf/2412.11165
• Copy Paste: [[2412.11165]] OTLRM: Orthogonal Learning-based Low-Rank Metric for Multi-Dimensional Inverse Problems(https://arxiv.org/abs/2412.11165)
• Keywords: robust, generative
• Abstract: In real-world scenarios, complex data such as multispectral images and multi-frame videos inherently exhibit robust low-rank property. This property is vital for multi-dimensional inverse problems, such as tensor completion, spectral imaging reconstruction, and multispectral image denoising. Existing tensor singular value decomposition (t-SVD) definitions rely on hand-designed or pre-given transforms, which lack flexibility for defining tensor nuclear norm (TNN). The TNN-regularized optimization problem is solved by the singular value thresholding (SVT) operator, which leverages the t-SVD framework to obtain the low-rank tensor. However, it is quite complicated to introduce SVT into deep neural networks due to the numerical instability problem in solving the derivatives of the eigenvectors. In this paper, we introduce a novel data-driven generative low-rank t-SVD model based on the learnable orthogonal transform, which can be naturally solved under its representation. Prompted by the linear algebra theorem of the Householder transformation, our learnable orthogonal transform is achieved by constructing an endogenously orthogonal matrix adaptable to neural networks, optimizing it as arbitrary orthogonal matrices. Additionally, we propose a low-rank solver as a generalization of SVT, which utilizes an efficient representation of generative networks to obtain low-rank structures. Extensive experiments highlight its significant restoration enhancements.

Title: Cultural Palette: Pluralising Culture Alignment via Multi-agent Palette

• Authors: Jiahao Yuan, Zixiang Di, Shangzixin Zhao, Usman Naseem
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11167
• Pdf URL: https://arxiv.org/pdf/2412.11167
• Copy Paste: [[2412.11167]] Cultural Palette: Pluralising Culture Alignment via Multi-agent Palette(https://arxiv.org/abs/2412.11167)
• Keywords: large language model
• Abstract: Large language models (LLMs) face challenges in aligning with diverse cultural values despite their remarkable performance in generation, which stems from inherent monocultural biases and difficulties in capturing nuanced cultural semantics. Existing methods lack adaptability to unkown culture after finetuning. Inspired by cultural geography across five continents, we propose Cultural Palette, a multi-agent framework for cultural alignment. We first introduce the Pentachromatic Cultural Palette Dataset synthesized using LLMs to capture diverse cultural values from social dialogues across five continents. Building on this, Cultural Palette integrates five continent-level alignment agents with a meta-agent using our superior Cultural MoErges alignment technique by dynamically activating relevant cultural expertise based on user prompts to adapting new culture, which outperforms other joint and merging alignment strategies in overall cultural value alignment. Each continent agent generates a cultural draft, which is then refined and self-regulated by the meta-agent to produce the final culturally aligned response. Experiments across various countries demonstrate that Cultural Palette surpasses existing baselines in cultural alignment.

Title: PGD-Imp: Rethinking and Unleashing Potential of Classic PGD with Dual Strategies for Imperceptible Adversarial Attacks

• Authors: Jin Li, Zitong Yu, Ziqiang He, Z. Jane Wang, Xiangui Kang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.11168
• Pdf URL: https://arxiv.org/pdf/2412.11168
• Copy Paste: [[2412.11168]] PGD-Imp: Rethinking and Unleashing Potential of Classic PGD with Dual Strategies for Imperceptible Adversarial Attacks(https://arxiv.org/abs/2412.11168)
• Keywords: attack
• Abstract: Imperceptible adversarial attacks have recently attracted increasing research interests. Existing methods typically incorporate external modules or loss terms other than a simple $l_p$-norm into the attack process to achieve imperceptibility, while we argue that such additional designs may not be necessary. In this paper, we rethink the essence of imperceptible attacks and propose two simple yet effective strategies to unleash the potential of PGD, the common and classical attack, for imperceptibility from an optimization perspective. Specifically, the Dynamic Step Size is introduced to find the optimal solution with minimal attack cost towards the decision boundary of the attacked model, and the Adaptive Early Stop strategy is adopted to reduce the redundant strength of adversarial perturbations to the minimum level. The proposed PGD-Imperceptible (PGD-Imp) attack achieves state-of-the-art results in imperceptible adversarial attacks for both untargeted and targeted scenarios. When performing untargeted attacks against ResNet-50, PGD-Imp attains 100$\%$ (+0.3$\%$) ASR, 0.89 (-1.76) $l_2$ distance, and 52.93 (+9.2) PSNR with 57s (-371s) running time, significantly outperforming existing methods.

Title: Unpacking the Resilience of SNLI Contradiction Examples to Attacks

• Authors: Chetan Verma, Archit Agarwal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11172
• Pdf URL: https://arxiv.org/pdf/2412.11172
• Copy Paste: [[2412.11172]] Unpacking the Resilience of SNLI Contradiction Examples to Attacks(https://arxiv.org/abs/2412.11172)
• Keywords: attack, robust
• Abstract: Pre-trained models excel on NLI benchmarks like SNLI and MultiNLI, but their true language understanding remains uncertain. Models trained only on hypotheses and labels achieve high accuracy, indicating reliance on dataset biases and spurious correlations. To explore this issue, we applied the Universal Adversarial Attack to examine the model's vulnerabilities. Our analysis revealed substantial drops in accuracy for the entailment and neutral classes, whereas the contradiction class exhibited a smaller decline. Fine-tuning the model on an augmented dataset with adversarial examples restored its performance to near-baseline levels for both the standard and challenge sets. Our findings highlight the value of adversarial triggers in identifying spurious correlations and improving robustness while providing insights into the resilience of the contradiction class to adversarial attacks.

Title: Knowledge Migration Framework for Smart Contract Vulnerability Detection

• Authors: Luqi Wang, Wenbao Jiang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11175
• Pdf URL: https://arxiv.org/pdf/2412.11175
• Copy Paste: [[2412.11175]] Knowledge Migration Framework for Smart Contract Vulnerability Detection(https://arxiv.org/abs/2412.11175)
• Keywords: extraction, data-free
• Abstract: As a cornerstone of blockchain technology in the 3.0 era, smart contracts play a pivotal role in the evolution of blockchain systems. In order to address the limitations of existing smart contract vulnerability detection models with regard to their generalisation capability, an AF-STip smart contract vulnerability detection framework incorporating efficient knowledge migration is proposed. AF-STip employs the teacher network as the main model and migrates the knowledge processed by the smart contract to the student model using a data-free knowledge distillation method. The student model utilises this knowledge to enhance its vulnerability detection capabilities. The approach markedly enhances the model's capacity for feature extraction and cross-class adaptation, while concurrently reducing computational this http URL order to further enhance the extraction of vulnerability features, an adaptive fusion module is proposed in this paper, which aims to strengthen the interaction and fusion of feature this http URL experimental results demonstrate that the STip model attains an average F1 value detection score of 91.16% for the four vulnerabilities without disclosing the original smart contract data. To validate the viability of the proposed lightweight migration approach, the student model is deployed in a migration learning task targeting a novel vulnerability type, resulting in an accuracy of 91.02% and an F1 score of 90.46%. To the best of our knowledge, AF-STip is the inaugural model to apply data-free knowledge migration to smart contract vulnerability detection. While markedly reducing the computational overhead, the method still demonstrates exceptional performance in detecting novel vulnerabilities.

Title: OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation

• Authors: Bohan Li, Xin Jin, Jianan Wang, Yukai Shi, Yasheng Sun, Xiaofeng Wang, Zhuang Ma, Baao Xie, Chao Ma, Xiaokang Yang, Wenjun Zeng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11183
• Pdf URL: https://arxiv.org/pdf/2412.11183
• Copy Paste: [[2412.11183]] OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation(https://arxiv.org/abs/2412.11183)
• Keywords: diffusion
• Abstract: Recent diffusion models have demonstrated remarkable performance in both 3D scene generation and perception tasks. Nevertheless, existing methods typically separate these two processes, acting as a data augmenter to generate synthetic data for downstream perception tasks. In this work, we propose OccScene, a novel mutual learning paradigm that integrates fine-grained 3D perception and high-quality generation in a unified framework, achieving a cross-task win-win effect. OccScene generates new and consistent 3D realistic scenes only depending on text prompts, guided with semantic occupancy in a joint-training diffusion framework. To align the occupancy with the diffusion latent, a Mamba-based Dual Alignment module is introduced to incorporate fine-grained semantics and geometry as perception priors. Within OccScene, the perception module can be effectively improved with customized and diverse generated scenes, while the perception priors in return enhance the generation performance for mutual benefits. Extensive experiments show that OccScene achieves realistic 3D scene generation in broad indoor and outdoor scenarios, while concurrently boosting the perception models to achieve substantial performance improvements in the 3D perception task of semantic occupancy prediction.

Title: Efficient Quantization-Aware Training on Segment Anything Model in Medical Images and Its Deployment

• Authors: Haisheng Lu, Yujie Fu, Fan Zhang, Le Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11186
• Pdf URL: https://arxiv.org/pdf/2412.11186
• Copy Paste: [[2412.11186]] Efficient Quantization-Aware Training on Segment Anything Model in Medical Images and Its Deployment(https://arxiv.org/abs/2412.11186)
• Keywords: segmentation
• Abstract: Medical image segmentation is a critical component of clinical practice, and the state-of-the-art MedSAM model has significantly advanced this field. Nevertheless, critiques highlight that MedSAM demands substantial computational resources during inference. To address this issue, the CVPR 2024 MedSAM on Laptop Challenge was established to find an optimal balance between accuracy and processing speed. In this paper, we introduce a quantization-aware training pipeline designed to efficiently quantize the Segment Anything Model for medical images and deploy it using the OpenVINO inference engine. This pipeline optimizes both training time and disk storage. Our experimental results confirm that this approach considerably enhances processing speed over the baseline, while still achieving an acceptable accuracy level. The training script, inference script, and quantized model are publicly accessible at this https URL.

Title: Drawing the Line: Enhancing Trustworthiness of MLLMs Through the Power of Refusal

• Authors: Yuhao Wang, Zhiyuan Zhu, Heyang Liu, Yusheng Liao, Hongcheng Liu, Yanfeng Wang, Yu Wang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11196
• Pdf URL: https://arxiv.org/pdf/2412.11196
• Copy Paste: [[2412.11196]] Drawing the Line: Enhancing Trustworthiness of MLLMs Through the Power of Refusal(https://arxiv.org/abs/2412.11196)
• Keywords: large language model
• Abstract: Multimodal large language models (MLLMs) excel at multimodal perception and understanding, yet their tendency to generate hallucinated or inaccurate responses undermines their trustworthiness. Existing methods have largely overlooked the importance of refusal responses as a means of enhancing MLLMs reliability. To bridge this gap, we present the Information Boundary-aware Learning Framework (InBoL), a novel approach that empowers MLLMs to refuse to answer user queries when encountering insufficient information. To the best of our knowledge, InBoL is the first framework that systematically defines the conditions under which refusal is appropriate for MLLMs using the concept of information boundaries proposed in our paper. This framework introduces a comprehensive data generation pipeline and tailored training strategies to improve the model's ability to deliver appropriate refusal responses. To evaluate the trustworthiness of MLLMs, we further propose a user-centric alignment goal along with corresponding metrics. Experimental results demonstrate a significant improvement in refusal accuracy without noticeably compromising the model's helpfulness, establishing InBoL as a pivotal advancement in building more trustworthy MLLMs.

Title: Task-Oriented Dialog Systems for the Senegalese Wolof Language

• Authors: Derguene Mbaye, Moussa Diallo
• Subjects: cs.CL, cs.AI, cs.HC, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.11203
• Pdf URL: https://arxiv.org/pdf/2412.11203
• Copy Paste: [[2412.11203]] Task-Oriented Dialog Systems for the Senegalese Wolof Language(https://arxiv.org/abs/2412.11203)
• Keywords: robust, large language model
• Abstract: In recent years, we are seeing considerable interest in conversational agents with the rise of large language models (LLMs). Although they offer considerable advantages, LLMs also present significant risks, such as hallucination, which hinder their widespread deployment in industry. Moreover, low-resource languages such as African ones are still underrepresented in these systems limiting their performance in these languages. In this paper, we illustrate a more classical approach based on modular architectures of Task-oriented Dialog Systems (ToDS) offering better control over outputs. We propose a chatbot generation engine based on the Rasa framework and a robust methodology for projecting annotations onto the Wolof language using an in-house machine translation system. After evaluating a generated chatbot trained on the Amazon Massive dataset, our Wolof Intent Classifier performs similarly to the one obtained for French, which is a resource-rich language. We also show that this approach is extensible to other low-resource languages, thanks to the intent classifier's language-agnostic pipeline, simplifying the design of chatbots in these languages.

Title: ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes

• Authors: Pedro Miguel Sánchez Sánchez, Enrique Tomás Martínez Beltrán, Miguel Fernández Llamas, Gérôme Bovet, Gregorio Martínez Pérez, Alberto Huertas Celdrán
• Subjects: cs.LG, cs.AI, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2412.11207
• Pdf URL: https://arxiv.org/pdf/2412.11207
• Copy Paste: [[2412.11207]] ProFe: Communication-Efficient Decentralized Federated Learning via Distillation and Prototypes(https://arxiv.org/abs/2412.11207)
• Keywords: privacy, federate
• Abstract: Decentralized Federated Learning (DFL) trains models in a collaborative and privacy-preserving manner while removing model centralization risks and improving communication bottlenecks. However, DFL faces challenges in efficient communication management and model aggregation within decentralized environments, especially with heterogeneous data distributions. Thus, this paper introduces ProFe, a novel communication optimization algorithm for DFL that combines knowledge distillation, prototype learning, and quantization techniques. ProFe utilizes knowledge from large local models to train smaller ones for aggregation, incorporates prototypes to better learn unseen classes, and applies quantization to reduce data transmitted during communication rounds. The performance of ProFe has been validated and compared to the literature by using benchmark datasets like MNIST, CIFAR10, and CIFAR100. Results showed that the proposed algorithm reduces communication costs by up to ~40-50% while maintaining or improving model performance. In addition, it adds ~20% training time due to increased complexity, generating a trade-off.

Title: Image Forgery Localization with State Space Models

• Authors: Zijie Lou, Gang Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11214
• Pdf URL: https://arxiv.org/pdf/2412.11214
• Copy Paste: [[2412.11214]] Image Forgery Localization with State Space Models(https://arxiv.org/abs/2412.11214)
• Keywords: extraction, transformer
• Abstract: Pixel dependency modeling from tampered images is pivotal for image forgery localization. Current approaches predominantly rely on convolutional neural network (CNN) or Transformer-based models, which often either lack sufficient receptive fields or entail significant computational overheads. In this paper, we propose LoMa, a novel image forgery localization method that leverages the Selective State Space (S6) model for global pixel dependency modeling and inverted residual CNN for local pixel dependency modeling. Our method introduces the Mixed-SSM Block, which initially employs atrous selective scan to traverse the spatial domain and convert the tampered image into order patch sequences, and subsequently applies multidirectional S6 modeling. In addition, an auxiliary convolutional branch is introduced to enhance local feature extraction. This design facilitates the efficient extraction of global dependencies while upholding linear complexity. Upon modeling the pixel dependency with the SSM and CNN blocks, the pixel-wise forgery localization results are obtained by a simple MLP decoder. Extensive experimental results validate the superiority of LoMa over CNN-based and Transformer-based state-of-the-arts.

Title: GenLit: Reformulating Single-Image Relighting as Video Generation

• Authors: Shrisha Bharadwaj, Haiwen Feng, Victoria Abrevaya, Michael J. Black
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.11224
• Pdf URL: https://arxiv.org/pdf/2412.11224
• Copy Paste: [[2412.11224]] GenLit: Reformulating Single-Image Relighting as Video Generation(https://arxiv.org/abs/2412.11224)
• Keywords: diffusion
• Abstract: Manipulating the illumination within a single image represents a fundamental challenge in computer vision and graphics. This problem has been traditionally addressed using inverse rendering techniques, which require explicit 3D asset reconstruction and costly ray tracing simulations. Meanwhile, recent advancements in visual foundation models suggest that a new paradigm could soon be practical and possible -- one that replaces explicit physical models with networks that are trained on massive amounts of image and video data. In this paper, we explore the potential of exploiting video diffusion models, and in particular Stable Video Diffusion (SVD), in understanding the physical world to perform relighting tasks given a single image. Specifically, we introduce GenLit, a framework that distills the ability of a graphics engine to perform light manipulation into a video generation model, enabling users to directly insert and manipulate a point light in the 3D world within a given image and generate the results directly as a video sequence. We find that a model fine-tuned on only a small synthetic dataset (270 objects) is able to generalize to real images, enabling single-image relighting with realistic ray tracing effects and cast shadows. These results reveal the ability of video foundation models to capture rich information about lighting, material, and shape. Our findings suggest that such models, with minimal training, can be used for physically-based rendering without explicit physically asset reconstruction and complex ray tracing. This further suggests the potential of such models for controllable and physically accurate image synthesis tasks.

Title: Smaller Language Models Are Better Instruction Evolvers

• Authors: Tingfeng Hui, Lulu Zhao, Guanting Dong, Yaqi Zhang, Hua Zhou, Sen Su
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11231
• Pdf URL: https://arxiv.org/pdf/2412.11231
• Copy Paste: [[2412.11231]] Smaller Language Models Are Better Instruction Evolvers(https://arxiv.org/abs/2412.11231)
• Keywords: large language model
• Abstract: Instruction tuning has been widely used to unleash the complete potential of large language models. Notably, complex and diverse instructions are of significant importance as they can effectively align models with various downstream tasks. However, current approaches to constructing large-scale instructions predominantly favour powerful models such as GPT-4 or those with over 70 billion parameters, under the empirical presumption that such larger language models (LLMs) inherently possess enhanced capabilities. In this study, we question this prevalent assumption and conduct an in-depth exploration into the potential of smaller language models (SLMs) in the context of instruction evolution. Extensive experiments across three scenarios of instruction evolution reveal that smaller language models (SLMs) can synthesize more effective instructions than LLMs. Further analysis demonstrates that SLMs possess a broader output space during instruction evolution, resulting in more complex and diverse variants. We also observe that the existing metrics fail to focus on the impact of the instructions. Thus, we propose Instruction Complex-Aware IFD (IC-IFD), which introduces instruction complexity in the original IFD score to evaluate the effectiveness of instruction data more accurately. Our source code is available at: \href{this https URL}{this https URL}

Title: On the Generalizability of Iterative Patch Selection for Memory-Efficient High-Resolution Image Classification

• Authors: Max Riffi-Aslett, Christina Fell
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11237
• Pdf URL: https://arxiv.org/pdf/2412.11237
• Copy Paste: [[2412.11237]] On the Generalizability of Iterative Patch Selection for Memory-Efficient High-Resolution Image Classification(https://arxiv.org/abs/2412.11237)
• Keywords: transformer
• Abstract: Classifying large images with small or tiny regions of interest (ROI) is challenging due to computational and memory constraints. Weakly supervised memory-efficient patch selectors have achieved results comparable with strongly supervised methods. However, low signal-to-noise ratios and low entropy attention still cause overfitting. We explore these issues using a novel testbed on a memory-efficient cross-attention transformer with Iterative Patch Selection (IPS) as the patch selection module. Our testbed extends the megapixel MNIST benchmark to four smaller O2I (object-to-image) ratios ranging from 0.01% to 0.14% while keeping the canvas size fixed and introducing a noise generation component based on Bézier curves. Experimental results generalize the observations made on CNNs to IPS whereby the O2I threshold below which the classifier fails to generalize is affected by the training dataset size. We further observe that the magnitude of this interaction differs for each task of the Megapixel MNIST. For tasks "Maj" and "Top", the rate is at its highest, followed by tasks "Max" and "Multi" where in the latter, this rate is almost at 0. Moreover, results show that in a low data setting, tuning the patch size to be smaller relative to the ROI improves generalization, resulting in an improvement of + 15% for the megapixel MNIST and + 5% for the Swedish traffic signs dataset compared to the original object-to-patch ratios in IPS. Further outcomes indicate that the similarity between the thickness of the noise component and the digits in the megapixel MNIST gradually causes IPS to fail to generalize, contributing to previous suspicions.

Title: TrimLLM: Progressive Layer Dropping for Domain-Specific LLMs

• Authors: Lanxiang Hu, Tajana Rosing, Hao Zhang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11242
• Pdf URL: https://arxiv.org/pdf/2412.11242
• Copy Paste: [[2412.11242]] TrimLLM: Progressive Layer Dropping for Domain-Specific LLMs(https://arxiv.org/abs/2412.11242)
• Keywords: privacy, large language model
• Abstract: Specializing large language models (LLMs) for local deployment in domain-specific use cases is necessary for strong performance while meeting latency and privacy constraints. However, conventional task-specific adaptation approaches do not show simultaneous memory saving and inference speedup at deployment time. Practical compression techniques like quantization and pruning require dedicated hardware or kernel support to achieve measured inference speedup. We develop TrimLLM based on the layer-wise specialization phenomenon we empirically observed and verified on contemporary LLMs. TrimLLM reduces the depth of LLMs via progressive layer dropping. We show it retains LLMs' capacity in specific domains and achieves inference speedup irrespective of hardware and deep learning frameworks. We evaluated TrimLLM on LLMs of various sizes for inference; models adapted on medical, legal, and financial datasets all demonstrate $2.1-5.7\times$ inference speedup on consumer GPUs and up to $3.1\times$ speedup on A100 when compared to state-of-the-art model compression algorithms, with no loss in accuracy at 50$\sim$60\% model compression ratio.

Title: Transformer-Based Bearing Fault Detection using Temporal Decomposition Attention Mechanism

• Authors: Marzieh Mirzaeibonehkhater, Mohammad Ali Labbaf-Khaniki, Mohammad Manthouri
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2412.11245
• Pdf URL: https://arxiv.org/pdf/2412.11245
• Copy Paste: [[2412.11245]] Transformer-Based Bearing Fault Detection using Temporal Decomposition Attention Mechanism(https://arxiv.org/abs/2412.11245)
• Keywords: extraction, interpretability, transformer
• Abstract: Bearing fault detection is a critical task in predictive maintenance, where accurate and timely fault identification can prevent costly downtime and equipment damage. Traditional attention mechanisms in Transformer neural networks often struggle to capture the complex temporal patterns in bearing vibration data, leading to suboptimal performance. To address this limitation, we propose a novel attention mechanism, Temporal Decomposition Attention (TDA), which combines temporal bias encoding with seasonal-trend decomposition to capture both long-term dependencies and periodic fluctuations in time series data. Additionally, we incorporate the Hull Exponential Moving Average (HEMA) for feature extraction, enabling the model to effectively capture meaningful characteristics from the data while reducing noise. Our approach integrates TDA into the Transformer architecture, allowing the model to focus separately on the trend and seasonal components of the data. Experimental results on the Case Western Reserve University (CWRU) bearing fault detection dataset demonstrate that our approach outperforms traditional attention mechanisms and achieves state-of-the-art performance in terms of accuracy and interpretability. The HEMA-Transformer-TDA model achieves an accuracy of 98.1%, with exceptional precision, recall, and F1-scores, demonstrating its effectiveness in bearing fault detection and its potential for application in other time series tasks with seasonal patterns or trends.

Title: Beyond Discrete Personas: Personality Modeling Through Journal Intensive Conversations

• Authors: Sayantan Pal, Souvik Das, Rohini K. Srihari
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11250
• Pdf URL: https://arxiv.org/pdf/2412.11250
• Copy Paste: [[2412.11250]] Beyond Discrete Personas: Personality Modeling Through Journal Intensive Conversations(https://arxiv.org/abs/2412.11250)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have significantly improved personalized conversational capabilities. However, existing datasets like Persona Chat, Synthetic Persona Chat, and Blended Skill Talk rely on static, predefined personas. This approach often results in dialogues that fail to capture human personalities' fluid and evolving nature. To overcome these limitations, we introduce a novel dataset with around 400,000 dialogues and a framework for generating personalized conversations using long-form journal entries from Reddit. Our approach clusters journal entries for each author and filters them by selecting the most representative cluster, ensuring that the retained entries best reflect the author's personality. We further refine the data by capturing the Big Five personality traits --openness, conscientiousness, extraversion, agreeableness, and neuroticism --ensuring that dialogues authentically reflect an individual's personality. Using Llama 3 70B, we generate high-quality, personality-rich dialogues grounded in these journal entries. Fine-tuning models on this dataset leads to an 11% improvement in capturing personality traits on average, outperforming existing approaches in generating more coherent and personality-driven dialogues.

Title: Wasserstein Bounds for generative diffusion models with Gaussian tail targets

• Authors: Xixian Wang, Zhongjian Wang
• Subjects: cs.LG, math.AP, math.NA
• Abstract URL: https://arxiv.org/abs/2412.11251
• Pdf URL: https://arxiv.org/pdf/2412.11251
• Copy Paste: [[2412.11251]] Wasserstein Bounds for generative diffusion models with Gaussian tail targets(https://arxiv.org/abs/2412.11251)
• Keywords: diffusion, generative
• Abstract: We present an estimate of the Wasserstein distance between the data distribution and the generation of score-based generative models, assuming an $\epsilon$-accurate approximation of the score and a Gaussian-type tail behavior of the data distribution. The complexity bound in dimension is $O(\sqrt{d})$, with a logarithmic constant. Such Gaussian tail assumption applies to the distribution of a compact support target with early stopping technique and the Bayesian posterior with a bounded observation operator. Corresponding convergence and complexity bounds are derived. The crux of the analysis lies in the Lipchitz bound of the score, which is related to the Hessian estimate of a viscous Hamilton-Jacobi equation (vHJ). This latter is demonstrated by employing a dimension independent kernel estimate. Consequently, our complexity bound scales linearly (up to a logarithmic constant) with the square root of the trace of the covariance operator, which relates to the invariant distribution of forward process. Our analysis also extends to the probabilistic flow ODE, as the sampling process.

Title: CATER: Leveraging LLM to Pioneer a Multidimensional, Reference-Independent Paradigm in Translation Quality Evaluation

• Authors: Kurando IIDA, Kenjiro MIMURA
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11261
• Pdf URL: https://arxiv.org/pdf/2412.11261
• Copy Paste: [[2412.11261]] CATER: Leveraging LLM to Pioneer a Multidimensional, Reference-Independent Paradigm in Translation Quality Evaluation(https://arxiv.org/abs/2412.11261)
• Keywords: large language model
• Abstract: This paper introduces the Comprehensive AI-assisted Translation Edit Ratio (CATER), a novel and fully prompt-driven framework for evaluating machine translation (MT) quality. Leveraging large language models (LLMs) via a carefully designed prompt-based protocol, CATER expands beyond traditional reference-bound metrics, offering a multidimensional, reference-independent evaluation that addresses linguistic accuracy, semantic fidelity, contextual coherence, stylistic appropriateness, and information completeness. CATER's unique advantage lies in its immediate implementability: by providing the source and target texts along with a standardized prompt, an LLM can rapidly identify errors, quantify edit effort, and produce category-level and overall scores. This approach eliminates the need for pre-computed references or domain-specific resources, enabling instant adaptation to diverse languages, genres, and user priorities through adjustable weights and prompt modifications. CATER's LLM-enabled strategy supports more nuanced assessments, capturing phenomena such as subtle omissions, hallucinations, and discourse-level shifts that increasingly challenge contemporary MT systems. By uniting the conceptual rigor of frameworks like MQM and DQF with the scalability and flexibility of LLM-based evaluation, CATER emerges as a valuable tool for researchers, developers, and professional translators worldwide. The framework and example prompts are openly available, encouraging community-driven refinement and further empirical validation.

Title: VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video Face Swapping

• Authors: Hao Shao, Shulun Wang, Yang Zhou, Guanglu Song, Dailan He, Shuo Qin, Zhuofan Zong, Bingqi Ma, Yu Liu, Hongsheng Li
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.11279
• Pdf URL: https://arxiv.org/pdf/2412.11279
• Copy Paste: [[2412.11279]] VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video Face Swapping(https://arxiv.org/abs/2412.11279)
• Keywords: robust, diffusion
• Abstract: Video face swapping is becoming increasingly popular across various applications, yet existing methods primarily focus on static images and struggle with video face swapping because of temporal consistency and complex scenarios. In this paper, we present the first diffusion-based framework specifically designed for video face swapping. Our approach introduces a novel image-video hybrid training framework that leverages both abundant static image data and temporal video sequences, addressing the inherent limitations of video-only training. The framework incorporates a specially designed diffusion model coupled with a VidFaceVAE that effectively processes both types of data to better maintain temporal coherence of the generated videos. To further disentangle identity and pose features, we construct the Attribute-Identity Disentanglement Triplet (AIDT) Dataset, where each triplet has three face images, with two images sharing the same pose and two sharing the same identity. Enhanced with a comprehensive occlusion augmentation, this dataset also improves robustness against occlusions. Additionally, we integrate 3D reconstruction techniques as input conditioning to our network for handling large pose variations. Extensive experiments demonstrate that our framework achieves superior performance in identity preservation, temporal consistency, and visual quality compared to existing methods, while requiring fewer inference steps. Our approach effectively mitigates key challenges in video face swapping, including temporal flickering, identity preservation, and robustness to occlusions and pose variations.

Title: Learning Normal Flow Directly From Event Neighborhoods

• Authors: Dehao Yuan, Levi Burner, Jiayi Wu, Minghui Liu, Jingxi Chen, Yiannis Aloimonos, Cornelia Fermüller
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11284
• Pdf URL: https://arxiv.org/pdf/2412.11284
• Copy Paste: [[2412.11284]] Learning Normal Flow Directly From Event Neighborhoods(https://arxiv.org/abs/2412.11284)
• Keywords: robust
• Abstract: Event-based motion field estimation is an important task. However, current optical flow methods face challenges: learning-based approaches, often frame-based and relying on CNNs, lack cross-domain transferability, while model-based methods, though more robust, are less accurate. To address the limitations of optical flow estimation, recent works have focused on normal flow, which can be more reliably measured in regions with limited texture or strong edges. However, existing normal flow estimators are predominantly model-based and suffer from high errors. In this paper, we propose a novel supervised point-based method for normal flow estimation that overcomes the limitations of existing event learning-based approaches. Using a local point cloud encoder, our method directly estimates per-event normal flow from raw events, offering multiple unique advantages: 1) It produces temporally and spatially sharp predictions. 2) It supports more diverse data augmentation, such as random rotation, to improve robustness across various domains. 3) It naturally supports uncertainty quantification via ensemble inference, which benefits downstream tasks. 4) It enables training and inference on undistorted data in normalized camera coordinates, improving transferability across cameras. Extensive experiments demonstrate our method achieves better and more consistent performance than state-of-the-art methods when transferred across different datasets. Leveraging this transferability, we train our model on the union of datasets and release it for public use. Finally, we introduce an egomotion solver based on a maximum-margin problem that uses normal flow and IMU to achieve strong performance in challenging scenarios.

Title: Detecting Daily Living Gait Amid Huntington's Disease Chorea using a Foundation Deep Learning Model

• Authors: Dafna Schwartz, Lori Quinn, Nora E. Fritz, Lisa M. Muratori, Jeffery M. Hausdorff, Ran Gilad Bachrach
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11286
• Pdf URL: https://arxiv.org/pdf/2412.11286
• Copy Paste: [[2412.11286]] Detecting Daily Living Gait Amid Huntington's Disease Chorea using a Foundation Deep Learning Model(https://arxiv.org/abs/2412.11286)
• Keywords: robust, generative, segmentation
• Abstract: Wearable sensors offer a non-invasive way to collect physical activity (PA) data, with walking as a key component. Existing models often struggle to detect gait bouts in individuals with neurodegenerative diseases (NDDs) involving involuntary movements. We developed J-Net, a deep learning model inspired by U-Net, which uses a pre-trained self-supervised foundation model fine-tuned with Huntington`s disease (HD) in-lab data and paired with a segmentation head for gait detection. J-Net processes wrist-worn accelerometer data to detect gait during daily living. We evaluated J-Net on in-lab and daily-living data from HD, Parkinson`s disease (PD), and controls. J-Net achieved a 10-percentage point improvement in ROC-AUC for HD over existing methods, reaching 0.97 for in-lab data. In daily-living environments, J-Net estimates showed no significant differences in median daily walking time between HD and controls (p = 0.23), in contrast to other models, which indicated counterintuitive results (p < 0.005). Walking time measured by J-Net correlated with the UHDRS-TMS clinical severity score (r=-0.52; p=0.02), confirming its clinical relevance. Fine-tuning J-Net on PD data also improved gait detection over current methods. J-Net`s architecture effectively addresses the challenges of gait detection in severe chorea and offers robust performance in daily living. The dataset and J-Net model are publicly available, providing a resource for further research into NDD-related gait impairments.

Title: Grassmannian Geometry Meets Dynamic Mode Decomposition in DMD-GEN: A New Metric for Mode Collapse in Time Series Generative Models

• Authors: Amime Mohamed Aboussalah, Yassine Abbahaddou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11292
• Pdf URL: https://arxiv.org/pdf/2412.11292
• Copy Paste: [[2412.11292]] Grassmannian Geometry Meets Dynamic Mode Decomposition in DMD-GEN: A New Metric for Mode Collapse in Time Series Generative Models(https://arxiv.org/abs/2412.11292)
• Keywords: interpretability, diffusion, generative
• Abstract: Generative models like Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) often fail to capture the full diversity of their training data, leading to mode collapse. While this issue is well-explored in image generation, it remains underinvestigated for time series data. We introduce a new definition of mode collapse specific to time series and propose a novel metric, DMD-GEN, to quantify its severity. Our metric utilizes Dynamic Mode Decomposition (DMD), a data-driven technique for identifying coherent spatiotemporal patterns, and employs Optimal Transport between DMD eigenvectors to assess discrepancies between the underlying dynamics of the original and generated data. This approach not only quantifies the preservation of essential dynamic characteristics but also provides interpretability by pinpointing which modes have collapsed. We validate DMD-GEN on both synthetic and real-world datasets using various generative models, including TimeGAN, TimeVAE, and DiffusionTS. The results demonstrate that DMD-GEN correlates well with traditional evaluation metrics for static data while offering the advantage of applicability to dynamic data. This work offers for the first time a definition of mode collapse for time series, improving understanding, and forming the basis of our tool for assessing and improving generative models in the time series domain.

Title: A Comparative Study on Dynamic Graph Embedding based on Mamba and Transformers

• Authors: Ashish Parmanand Pandey, Alan John Varghese, Sarang Patil, Mengjia Xu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11293
• Pdf URL: https://arxiv.org/pdf/2412.11293
• Copy Paste: [[2412.11293]] A Comparative Study on Dynamic Graph Embedding based on Mamba and Transformers(https://arxiv.org/abs/2412.11293)
• Keywords: transformer
• Abstract: Dynamic graph embedding has emerged as an important technique for modeling complex time-evolving networks across diverse domains. While transformer-based models have shown promise in capturing long-range dependencies in temporal graph data, they face scalability challenges due to quadratic computational complexity. This study presents a comparative analysis of dynamic graph embedding approaches using transformers and the recently proposed Mamba architecture, a state-space model with linear complexity. We introduce three novel models: TransformerG2G augment with graph convolutional networks, DG-Mamba, and GDG-Mamba with graph isomorphism network edge convolutions. Our experiments on multiple benchmark datasets demonstrate that Mamba-based models achieve comparable or superior performance to transformer-based approaches in link prediction tasks while offering significant computational efficiency gains on longer sequences. Notably, DG-Mamba variants consistently outperform transformer-based models on datasets with high temporal variability, such as UCI, Bitcoin, and Reality Mining, while maintaining competitive performance on more stable graphs like SBM. We provide insights into the learned temporal dependencies through analysis of attention weights and state matrices, revealing the models' ability to capture complex temporal patterns. By effectively combining state-space models with graph neural networks, our work addresses key limitations of previous approaches and contributes to the growing body of research on efficient temporal graph representation learning. These findings offer promising directions for scaling dynamic graph embedding to larger, more complex real-world networks, potentially enabling new applications in areas such as social network analysis, financial modeling, and biological system dynamics.

Title: Semi-Implicit Neural Ordinary Differential Equations

• Authors: Hong Zhang, Ying Liu, Romit Maulik
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11301
• Pdf URL: https://arxiv.org/pdf/2412.11301
• Copy Paste: [[2412.11301]] Semi-Implicit Neural Ordinary Differential Equations(https://arxiv.org/abs/2412.11301)
• Keywords: robust
• Abstract: Classical neural ODEs trained with explicit methods are intrinsically limited by stability, crippling their efficiency and robustness for stiff learning problems that are common in graph learning and scientific machine learning. We present a semi-implicit neural ODE approach that exploits the partitionable structure of the underlying dynamics. Our technique leads to an implicit neural network with significant computational advantages over existing approaches because of enhanced stability and efficient linear solves during time integration. We show that our approach outperforms existing approaches on a variety of applications including graph classification and learning complex dynamical systems. We also demonstrate that our approach can train challenging neural ODEs where both explicit methods and fully implicit methods are intractable.

Title: Sequence-Level Analysis of Leakage Risk of Training Data in Large Language Models

• Authors: Trishita Tiwari, G. Edward Suh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11302
• Pdf URL: https://arxiv.org/pdf/2412.11302
• Copy Paste: [[2412.11302]] Sequence-Level Analysis of Leakage Risk of Training Data in Large Language Models(https://arxiv.org/abs/2412.11302)
• Keywords: extraction, large language model
• Abstract: This work advocates for the use of sequence level probabilities for quantifying the risk of extraction training data from Large Language Models (LLMs) as they provide much finer-grained information than has been previously obtained. We re-analyze the effects of decoding schemes, model-size, prefix length, partial sequence leakages, and token positions to uncover new insights that have were not possible in prior work due to their choice of metrics. We perform this study on two pre-trained models, LLaMa and OPT, trained on the Common Crawl and Pile respectively. We discover that 1) Extraction rate, the predominant metric used in prior quantification work, underestimates the threat of leakage of training data in randomized LLMs by as much as 2.14x. 2) Though, on average, larger models and longer prefixes can extract more data, this is not true with a substantial portion of individual sequences. 30.4-41.5% of our sequences are easier to extract with either shorter prefixes or smaller models. 3) Contrary to prior belief, partial leakage in the commonly used decoding schemes like top-k and top-p are not easier than leaking verbatim training data. 4) Extracting later tokens in a sequence is as much as 912% easier than extracting earlier tokens. The insights gained from our analysis show that it is important to look at leakage of training data on a per-sequence basis.

Title: Reliable, Reproducible, and Really Fast Leaderboards with Evalica

• Authors: Dmitry Ustalov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11314
• Pdf URL: https://arxiv.org/pdf/2412.11314
• Copy Paste: [[2412.11314]] Reliable, Reproducible, and Really Fast Leaderboards with Evalica(https://arxiv.org/abs/2412.11314)
• Keywords: large language model
• Abstract: The rapid advancement of natural language processing (NLP) technologies, such as instruction-tuned large language models (LLMs), urges the development of modern evaluation protocols with human and machine feedback. We introduce Evalica, an open-source toolkit that facilitates the creation of reliable and reproducible model leaderboards. This paper presents its design, evaluates its performance, and demonstrates its usability through its Web interface, command-line interface, and Python API.

Title: RoLargeSum: A Large Dialect-Aware Romanian News Dataset for Summary, Headline, and Keyword Generation

• Authors: Andrei-Marius Avram, Mircea Timpuriu, Andreea Iuga, Vlad-Cristian Matei, Iulian-Marius Tăiatu, Tudor Găină, Dumitru-Clementin Cercel, Florin Pop, Mihaela-Claudia Cercel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11317
• Pdf URL: https://arxiv.org/pdf/2412.11317
• Copy Paste: [[2412.11317]] RoLargeSum: A Large Dialect-Aware Romanian News Dataset for Summary, Headline, and Keyword Generation(https://arxiv.org/abs/2412.11317)
• Keywords: large language model
• Abstract: Using supervised automatic summarisation methods requires sufficient corpora that include pairs of documents and their summaries. Similarly to many tasks in natural language processing, most of the datasets available for summarization are in English, posing challenges for developing summarization models in other languages. Thus, in this work, we introduce RoLargeSum, a novel large-scale summarization dataset for the Romanian language crawled from various publicly available news websites from Romania and the Republic of Moldova that were thoroughly cleaned to ensure a high-quality standard. RoLargeSum contains more than 615K news articles, together with their summaries, as well as their headlines, keywords, dialect, and other metadata that we found on the targeted websites. We further evaluated the performance of several BART variants and open-source large language models on RoLargeSum for benchmarking purposes. We manually evaluated the results of the best-performing system to gain insight into the potential pitfalls of this data set and future development.

Title: Sonicmesh: Enhancing 3D Human Mesh Reconstruction in Vision-Impaired Environments With Acoustic Signals

• Authors: Xiaoxuan Liang, Wuyang Zhang, Hong Zhou, Zhaolong Wei, Sicheng Zhu, Yansong Li, Rui Yin, Jiantao Yuan, Jeremy Gummeson
• Subjects: cs.CV, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.11325
• Pdf URL: https://arxiv.org/pdf/2412.11325
• Copy Paste: [[2412.11325]] Sonicmesh: Enhancing 3D Human Mesh Reconstruction in Vision-Impaired Environments With Acoustic Signals(https://arxiv.org/abs/2412.11325)
• Keywords: privacy, robust, extraction
• Abstract: 3D Human Mesh Reconstruction (HMR) from 2D RGB images faces challenges in environments with poor lighting, privacy concerns, or occlusions. These weaknesses of RGB imaging can be complemented by acoustic signals, which are widely available, easy to deploy, and capable of penetrating obstacles. However, no existing methods effectively combine acoustic signals with RGB data for robust 3D HMR. The primary challenges include the low-resolution images generated by acoustic signals and the lack of dedicated processing backbones. We introduce SonicMesh, a novel approach combining acoustic signals with RGB images to reconstruct 3D human mesh. To address the challenges of low resolution and the absence of dedicated processing backbones in images generated by acoustic signals, we modify an existing method, HRNet, for effective feature extraction. We also integrate a universal feature embedding technique to enhance the precision of cross-dimensional feature alignment, enabling SonicMesh to achieve high accuracy. Experimental results demonstrate that SonicMesh accurately reconstructs 3D human mesh in challenging environments such as occlusions, non-line-of-sight scenarios, and poor lighting.

Title: Segment-Level Diffusion: A Framework for Controllable Long-Form Generation with Diffusion Language Models

• Authors: Xiaochen Zhu, Georgi Karadzhov, Chenxi Whitehouse, Andreas Vlachos
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11333
• Pdf URL: https://arxiv.org/pdf/2412.11333
• Copy Paste: [[2412.11333]] Segment-Level Diffusion: A Framework for Controllable Long-Form Generation with Diffusion Language Models(https://arxiv.org/abs/2412.11333)
• Keywords: robust, diffusion, segmentation
• Abstract: Diffusion models have shown promise in text generation but often struggle with generating long, coherent, and contextually accurate text. Token-level diffusion overlooks word-order dependencies and enforces short output windows, while passage-level diffusion struggles with learning robust representation for long-form text. To address these challenges, we propose Segment-Level Diffusion (SLD), a framework that enhances diffusion-based text generation through text segmentation, robust representation training with adversarial and contrastive learning, and improved latent-space guidance. By segmenting long-form outputs into separate latent representations and decoding them with an autoregressive decoder, SLD simplifies diffusion predictions and improves scalability. Experiments on XSum, ROCStories, DialogSum, and DeliData demonstrate that SLD achieves competitive or superior performance in fluency, coherence, and contextual compatibility across automatic and human evaluation metrics comparing with other diffusion and autoregressive baselines. Ablation studies further validate the effectiveness of our segmentation and representation learning strategies.

Title: Coupling-based Convergence Diagnostic and Stepsize Scheme for Stochastic Gradient Descent

• Authors: Xiang Li, Qiaomin Xie
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.11341
• Pdf URL: https://arxiv.org/pdf/2412.11341
• Copy Paste: [[2412.11341]] Coupling-based Convergence Diagnostic and Stepsize Scheme for Stochastic Gradient Descent(https://arxiv.org/abs/2412.11341)
• Keywords: robust
• Abstract: The convergence behavior of Stochastic Gradient Descent (SGD) crucially depends on the stepsize configuration. When using a constant stepsize, the SGD iterates form a Markov chain, enjoying fast convergence during the initial transient phase. However, when reaching stationarity, the iterates oscillate around the optimum without making further progress. In this paper, we study the convergence diagnostics for SGD with constant stepsize, aiming to develop an effective dynamic stepsize scheme. We propose a novel coupling-based convergence diagnostic procedure, which monitors the distance of two coupled SGD iterates for stationarity detection. Our diagnostic statistic is simple and is shown to track the transition from transience stationarity theoretically. We conduct extensive numerical experiments and compare our method against various existing approaches. Our proposed coupling-based stepsize scheme is observed to achieve superior performance across a diverse set of convex and non-convex problems. Moreover, our results demonstrate the robustness of our approach to a wide range of hyperparameters.

Title: One-Shot Multilingual Font Generation Via ViT

• Authors: Zhiheng Wang, Jiarui Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11342
• Pdf URL: https://arxiv.org/pdf/2412.11342
• Copy Paste: [[2412.11342]] One-Shot Multilingual Font Generation Via ViT(https://arxiv.org/abs/2412.11342)
• Keywords: transformer
• Abstract: Font design poses unique challenges for logographic languages like Chinese, Japanese, and Korean (CJK), where thousands of unique characters must be individually crafted. This paper introduces a novel Vision Transformer (ViT)-based model for multi-language font generation, effectively addressing the complexities of both logographic and alphabetic scripts. By leveraging ViT and pretraining with a strong visual pretext task (Masked Autoencoding, MAE), our model eliminates the need for complex design components in prior frameworks while achieving comprehensive results with enhanced generalizability. Remarkably, it can generate high-quality fonts across multiple languages for unseen, unknown, and even user-crafted characters. Additionally, we integrate a Retrieval-Augmented Guidance (RAG) module to dynamically retrieve and adapt style references, improving scalability and real-world applicability. We evaluated our approach in various font generation tasks, demonstrating its effectiveness, adaptability, and scalability.

Title: Can AI Extract Antecedent Factors of Human Trust in AI? An Application of Information Extraction for Scientific Literature in Behavioural and Computer Sciences

• Authors: Melanie McGrath, Harrison Bailey, Necva Bölücü, Xiang Dai, Sarvnaz Karimi, Cecile Paris
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11344
• Pdf URL: https://arxiv.org/pdf/2412.11344
• Copy Paste: [[2412.11344]] Can AI Extract Antecedent Factors of Human Trust in AI? An Application of Information Extraction for Scientific Literature in Behavioural and Computer Sciences(https://arxiv.org/abs/2412.11344)
• Keywords: extraction, large language model
• Abstract: Information extraction from the scientific literature is one of the main techniques to transform unstructured knowledge hidden in the text into structured data which can then be used for decision-making in down-stream tasks. One such area is Trust in AI, where factors contributing to human trust in artificial intelligence applications are studied. The relationships of these factors with human trust in such applications are complex. We hence explore this space from the lens of information extraction where, with the input of domain experts, we carefully design annotation guidelines, create the first annotated English dataset in this domain, investigate an LLM-guided annotation, and benchmark it with state-of-the-art methods using large language models in named entity and relation extraction. Our results indicate that this problem requires supervised learning which may not be currently feasible with prompt-based LLMs.

Title: PSGraph: Differentially Private Streaming Graph Synthesis by Considering Temporal Dynamics

• Authors: Quan Yuan, Zhikun Zhang, Linkang Du, Min Chen, Mingyang Sun, Yunjun Gao, Michael Backes, Shibo He, Jiming Chen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.11369
• Pdf URL: https://arxiv.org/pdf/2412.11369
• Copy Paste: [[2412.11369]] PSGraph: Differentially Private Streaming Graph Synthesis by Considering Temporal Dynamics(https://arxiv.org/abs/2412.11369)
• Keywords: privacy
• Abstract: Streaming graphs are ubiquitous in daily life, such as evolving social networks and dynamic communication systems. Due to the sensitive information contained in the graph, directly sharing the streaming graphs poses significant privacy risks. Differential privacy, offering strict theoretical guarantees, has emerged as a standard approach for private graph data synthesis. However, existing methods predominantly focus on static graph publishing, neglecting the intrinsic relationship between adjacent graphs, thereby resulting in limited performance in streaming data publishing scenarios. To address this gap, we propose PSGraph, the first differentially private streaming graph synthesis framework that integrates temporal dynamics. PSGraph adaptively adjusts the privacy budget allocation mechanism by analyzing the variations in the current graph compared to the previous one for conserving the privacy budget. Moreover, PSGraph aggregates information across various timestamps and adopts crucial post-processing techniques to enhance the synthetic streaming graphs. We conduct extensive experiments on four real-world datasets under five commonly used metrics. The experimental results demonstrate the superiority of PSGraph.

Title: ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data

• Authors: Chengsen Wang, Qi Qi, Jingyu Wang, Haifeng Sun, Zirui Zhuang, Jinming Wu, Lei Zhang, Jianxin Liao
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11376
• Pdf URL: https://arxiv.org/pdf/2412.11376
• Copy Paste: [[2412.11376]] ChatTime: A Unified Multimodal Time Series Foundation Model Bridging Numerical and Textual Data(https://arxiv.org/abs/2412.11376)
• Keywords: large language model
• Abstract: Human experts typically integrate numerical and textual multimodal information to analyze time series. However, most traditional deep learning predictors rely solely on unimodal numerical data, using a fixed-length window for training and prediction on a single dataset, and cannot adapt to different scenarios. The powered pre-trained large language model has introduced new opportunities for time series analysis. Yet, existing methods are either inefficient in training, incapable of handling textual information, or lack zero-shot forecasting capability. In this paper, we innovatively model time series as a foreign language and construct ChatTime, a unified framework for time series and text processing. As an out-of-the-box multimodal time series foundation model, ChatTime provides zero-shot forecasting capability and supports bimodal input/output for both time series and text. We design a series of experiments to verify the superior performance of ChatTime across multiple tasks and scenarios, and create four multimodal datasets to address data gaps. The experimental results demonstrate the potential and utility of ChatTime.

Title: FinLoRA: Finetuning Quantized Financial Large Language Models Using Low-Rank Adaptation

• Authors: Dannong Wang, Daniel Kim, Bo Jin, Xingjian Zhao, Tianfan Fu, Steve Yang, Xiao-Yang Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11378
• Pdf URL: https://arxiv.org/pdf/2412.11378
• Copy Paste: [[2412.11378]] FinLoRA: Finetuning Quantized Financial Large Language Models Using Low-Rank Adaptation(https://arxiv.org/abs/2412.11378)
• Keywords: privacy, large language model
• Abstract: Finetuned large language models (LLMs) have shown remarkable performance in financial tasks, such as sentiment analysis and information retrieval. Due to privacy concerns, finetuning and deploying Financial LLMs (FinLLMs) locally are crucial for institutions. However, finetuning FinLLMs poses challenges including GPU memory constraints and long input sequences. In this paper, we employ quantized low-rank adaptation (QLoRA) to finetune FinLLMs, which leverage low-rank matrix decomposition and quantization techniques to significantly reduce computational requirements while maintaining high model performance. We also employ data and pipeline parallelism to enable local finetuning using cost-effective, widely accessible GPUs. Experiments on financial datasets demonstrate that our method achieves substantial improvements in accuracy, GPU memory usage, and time efficiency, underscoring the potential of lowrank methods for scalable and resource-efficient LLM finetuning.

Title: Relation-Guided Adversarial Learning for Data-free Knowledge Transfer

• Authors: Yingping Liang, Ying Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11380
• Pdf URL: https://arxiv.org/pdf/2412.11380
• Copy Paste: [[2412.11380]] Relation-Guided Adversarial Learning for Data-free Knowledge Transfer(https://arxiv.org/abs/2412.11380)
• Keywords: data-free
• Abstract: Data-free knowledge distillation transfers knowledge by recovering training data from a pre-trained model. Despite the recent success of seeking global data diversity, the diversity within each class and the similarity among different classes are largely overlooked, resulting in data homogeneity and limited performance. In this paper, we introduce a novel Relation-Guided Adversarial Learning method with triplet losses, which solves the homogeneity problem from two aspects. To be specific, our method aims to promote both intra-class diversity and inter-class confusion of the generated samples. To this end, we design two phases, an image synthesis phase and a student training phase. In the image synthesis phase, we construct an optimization process to push away samples with the same labels and pull close samples with different labels, leading to intra-class diversity and inter-class confusion, respectively. Then, in the student training phase, we perform an opposite optimization, which adversarially attempts to reduce the distance of samples of the same classes and enlarge the distance of samples of different classes. To mitigate the conflict of seeking high global diversity and keeping inter-class confusing, we propose a focal weighted sampling strategy by selecting the negative in the triplets unevenly within a finite range of distance. RGAL shows significant improvement over previous state-of-the-art methods in accuracy and data efficiency. Besides, RGAL can be inserted into state-of-the-art methods on various data-free knowledge transfer applications. Experiments on various benchmarks demonstrate the effectiveness and generalizability of our proposed method on various tasks, specially data-free knowledge distillation, data-free quantization, and non-exemplar incremental learning. Our code is available at this https URL.

Title: Adapting Segment Anything Model (SAM) to Experimental Datasets via Fine-Tuning on GAN-based Simulation: A Case Study in Additive Manufacturing

• Authors: Anika Tabassum, Amirkoushyar Ziabari
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.11381
• Pdf URL: https://arxiv.org/pdf/2412.11381
• Copy Paste: [[2412.11381]] Adapting Segment Anything Model (SAM) to Experimental Datasets via Fine-Tuning on GAN-based Simulation: A Case Study in Additive Manufacturing(https://arxiv.org/abs/2412.11381)
• Keywords: robust, generative, segmentation
• Abstract: Industrial X-ray computed tomography (XCT) is a powerful tool for non-destructive characterization of materials and manufactured components. XCT commonly accompanied by advanced image analysis and computer vision algorithms to extract relevant information from the images. Traditional computer vision models often struggle due to noise, resolution variability, and complex internal structures, particularly in scientific imaging applications. State-of-the-art foundational models, like the Segment Anything Model (SAM)-designed for general-purpose image segmentation-have revolutionized image segmentation across various domains, yet their application in specialized fields like materials science remains under-explored. In this work, we explore the application and limitations of SAM for industrial X-ray CT inspection of additive manufacturing components. We demonstrate that while SAM shows promise, it struggles with out-of-distribution data, multiclass segmentation, and computational efficiency during fine-tuning. To address these issues, we propose a fine-tuning strategy utilizing parameter-efficient techniques, specifically Conv-LoRa, to adapt SAM for material-specific datasets. Additionally, we leverage generative adversarial network (GAN)-generated data to enhance the training process and improve the model's segmentation performance on complex X-ray CT data. Our experimental results highlight the importance of tailored segmentation models for accurate inspection, showing that fine-tuning SAM on domain-specific scientific imaging data significantly improves performance. However, despite improvements, the model's ability to generalize across diverse datasets remains limited, highlighting the need for further research into robust, scalable solutions for domain-specific segmentation tasks.

Title: A Comprehensive Review of Adversarial Attacks on Machine Learning

• Authors: Syed Quiser Ahmed, Bharathi Vokkaliga Ganesh, Sathyanarayana Sampath Kumar, Prakhar Mishra, Ravi Anand, Bhanuteja Akurathi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.11384
• Pdf URL: https://arxiv.org/pdf/2412.11384
• Copy Paste: [[2412.11384]] A Comprehensive Review of Adversarial Attacks on Machine Learning(https://arxiv.org/abs/2412.11384)
• Keywords: secure, attack, robust
• Abstract: This research provides a comprehensive overview of adversarial attacks on AI and ML models, exploring various attack types, techniques, and their potential harms. We also delve into the business implications, mitigation strategies, and future research directions. To gain practical insights, we employ the Adversarial Robustness Toolbox (ART) [1] library to simulate these attacks on real-world use cases, such as self-driving cars. Our goal is to inform practitioners and researchers about the challenges and opportunities in defending AI systems against adversarial threats. By providing a comprehensive comparison of different attack methods, we aim to contribute to the development of more robust and secure AI systems.

Title: Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models

• Authors: Tom S. Juzek, Zina B. Ward
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11385
• Pdf URL: https://arxiv.org/pdf/2412.11385
• Copy Paste: [[2412.11385]] Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models(https://arxiv.org/abs/2412.11385)
• Keywords: large language model
• Abstract: Scientific English is currently undergoing rapid change, with words like "delve," "intricate," and "underscore" appearing far more frequently than just a few years ago. It is widely assumed that scientists' use of large language models (LLMs) is responsible for such trends. We develop a formal, transferable method to characterize these linguistic changes. Application of our method yields 21 focal words whose increased occurrence in scientific abstracts is likely the result of LLM usage. We then pose "the puzzle of lexical overrepresentation": WHY are such words overused by LLMs? We fail to find evidence that lexical overrepresentation is caused by model architecture, algorithm choices, or training data. To assess whether reinforcement learning from human feedback (RLHF) contributes to the overuse of focal words, we undertake comparative model testing and conduct an exploratory online study. While the model testing is consistent with RLHF playing a role, our experimental results suggest that participants may be reacting differently to "delve" than to other focal words. With LLMs quickly becoming a driver of global language change, investigating these potential sources of lexical overrepresentation is important. We note that while insights into the workings of LLMs are within reach, a lack of transparency surrounding model development remains an obstacle to such research.

Title: INTERACT: Enabling Interactive, Question-Driven Learning in Large Language Models

• Authors: Aum Kendapadi, Kerem Zaman, Rakesh R. Menon, Shashank Srivastava
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11388
• Pdf URL: https://arxiv.org/pdf/2412.11388
• Copy Paste: [[2412.11388]] INTERACT: Enabling Interactive, Question-Driven Learning in Large Language Models(https://arxiv.org/abs/2412.11388)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) excel at answering questions but remain passive learners--absorbing static data without the ability to question and refine knowledge. This paper explores how LLMs can transition to interactive, question-driven learning through student-teacher dialogues. We introduce INTERACT (INTEReractive Learning for Adaptive Concept Transfer), a framework in which a "student" LLM engages a "teacher" LLM through iterative inquiries to acquire knowledge across 1,347 contexts, including song lyrics, news articles, movie plots, academic papers, and images. Our experiments show that across a wide range of scenarios and LLM architectures, interactive learning consistently enhances performance, achieving up to a 25% improvement, with 'cold-start' student models matching static learning baselines in as few as five dialogue turns. Interactive setups can also mitigate the disadvantages of weaker teachers, showcasing the robustness of question-driven learning.

Title: Temporal Contrastive Learning for Video Temporal Reasoning in Large Vision-Language Models

• Authors: Rafael Souza, Jia-Hao Lim, Alexander Davis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11391
• Pdf URL: https://arxiv.org/pdf/2412.11391
• Copy Paste: [[2412.11391]] Temporal Contrastive Learning for Video Temporal Reasoning in Large Vision-Language Models(https://arxiv.org/abs/2412.11391)
• Keywords: robust, large language model
• Abstract: Temporal reasoning is a critical challenge in video-language understanding, as it requires models to align semantic concepts consistently across time. While existing large vision-language models (LVLMs) and large language models (LLMs) excel at static tasks, they struggle to capture dynamic interactions and temporal dependencies in video sequences. In this work, we propose Temporal Semantic Alignment via Dynamic Prompting (TSADP), a novel framework that enhances temporal reasoning capabilities through dynamic task-specific prompts and temporal contrastive learning. TSADP leverages a Dynamic Prompt Generator (DPG) to encode fine-grained temporal relationships and a Temporal Contrastive Loss (TCL) to align visual and textual embeddings across time. We evaluate our method on the VidSitu dataset, augmented with enriched temporal annotations, and demonstrate significant improvements over state-of-the-art models in tasks such as Intra-Video Entity Association, Temporal Relationship Understanding, and Chronology Prediction. Human evaluations further confirm TSADP's ability to generate coherent and semantically accurate descriptions. Our analysis highlights the robustness, efficiency, and practical utility of TSADP, making it a step forward in the field of video-language understanding.

Title: Leveraging Retrieval-Augmented Tags for Large Vision-Language Understanding in Complex Scenes

• Authors: Antonio Carlos Rivera, Anthony Moore, Steven Robinson
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11396
• Pdf URL: https://arxiv.org/pdf/2412.11396
• Copy Paste: [[2412.11396]] Leveraging Retrieval-Augmented Tags for Large Vision-Language Understanding in Complex Scenes(https://arxiv.org/abs/2412.11396)
• Keywords: robust, generative
• Abstract: Object-aware reasoning in vision-language tasks poses significant challenges for current models, particularly in handling unseen objects, reducing hallucinations, and capturing fine-grained relationships in complex visual scenes. To address these limitations, we propose the Vision-Aware Retrieval-Augmented Prompting (VRAP) framework, a generative approach that enhances Large Vision-Language Models (LVLMs) by integrating retrieval-augmented object tags into their prompts. VRAP introduces a novel pipeline where structured tags, including objects, attributes, and relationships, are extracted using pretrained visual encoders and scene graph parsers. These tags are enriched with external knowledge and incorporated into the LLM's input, enabling detailed and accurate reasoning. We evaluate VRAP across multiple vision-language benchmarks, including VQAv2, GQA, VizWiz, and COCO, achieving state-of-the-art performance in fine-grained reasoning and multimodal understanding. Additionally, our ablation studies highlight the importance of retrieval-augmented tags and contrastive learning, while human evaluations confirm VRAP's ability to generate accurate, detailed, and contextually relevant responses. Notably, VRAP achieves a 40% reduction in inference latency by eliminating runtime retrieval. These results demonstrate that VRAP is a robust and efficient framework for advancing object-aware multimodal reasoning.

Title: Quantization of Climate Change Impacts on Renewable Energy Generation Capacity: A Super-Resolution Recurrent Diffusion Model

• Authors: Xiaochong Dong, Jun Dan, Yingyun Sun, Yang Liu, Xuemin Zhang, Shengwei Mei
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2412.11399
• Pdf URL: https://arxiv.org/pdf/2412.11399
• Copy Paste: [[2412.11399]] Quantization of Climate Change Impacts on Renewable Energy Generation Capacity: A Super-Resolution Recurrent Diffusion Model(https://arxiv.org/abs/2412.11399)
• Keywords: diffusion, generative
• Abstract: Driven by global climate change and the ongoing energy transition, the coupling between power supply capabilities and meteorological factors has become increasingly significant. Over the long term, accurately quantifying the power generation capacity of renewable energy under the influence of climate change is essential for the development of sustainable power systems. However, due to interdisciplinary differences in data requirements, climate data often lacks the necessary hourly resolution to capture the short-term variability and uncertainties of renewable energy resources. To address this limitation, a super-resolution recurrent diffusion model (SRDM) has been developed to enhance the temporal resolution of climate data and model the short-term uncertainty. The SRDM incorporates a pre-trained decoder and a denoising network, that generates long-term, high-resolution climate data through a recurrent coupling mechanism. The high-resolution climate data is then converted into power value using the mechanism model, enabling the simulation of wind and photovoltaic (PV) power generation capacity on future long-term scales. Case studies were conducted in the Ejina region of Inner Mongolia, China, using fifth-generation reanalysis (ERA5) and coupled model intercomparison project (CMIP6) data under two climate pathways: SSP126 and SSP585. The results demonstrate that the SRDM outperforms existing generative models in generating super-resolution climate data. For the Ejina region, under a high-emission pathway, the annual utilization hours of wind power are projected to decrease by 2.82 hours/year, while those for PV power are projected to decrease by 0.26 hours/year. Furthermore, the research highlights the estimation biases introduced when low-resolution climate data is used for power conversion.

Title: Scaled Conjugate Gradient Method for Nonconvex Optimization in Deep Neural Networks

• Authors: Naoki Sato, Koshiro Izumi, Hideaki Iiduka
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.11400
• Pdf URL: https://arxiv.org/pdf/2412.11400
• Copy Paste: [[2412.11400]] Scaled Conjugate Gradient Method for Nonconvex Optimization in Deep Neural Networks(https://arxiv.org/abs/2412.11400)
• Keywords: generative
• Abstract: A scaled conjugate gradient method that accelerates existing adaptive methods utilizing stochastic gradients is proposed for solving nonconvex optimization problems with deep neural networks. It is shown theoretically that, whether with constant or diminishing learning rates, the proposed method can obtain a stationary point of the problem. Additionally, its rate of convergence with diminishing learning rates is verified to be superior to that of the conjugate gradient method. The proposed method is shown to minimize training loss functions faster than the existing adaptive methods in practical applications of image and text classification. Furthermore, in the training of generative adversarial networks, one version of the proposed method achieved the lowest Frechet inception distance score among those of the adaptive methods.

Title: Modeling Inter-Intra Heterogeneity for Graph Federated Learning

• Authors: Wentao Yu, Shuo Chen, Yongxin Tong, Tianlong Gu, Chen Gong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11402
• Pdf URL: https://arxiv.org/pdf/2412.11402
• Copy Paste: [[2412.11402]] Modeling Inter-Intra Heterogeneity for Graph Federated Learning(https://arxiv.org/abs/2412.11402)
• Keywords: robust, federate
• Abstract: Heterogeneity is a fundamental and challenging issue in federated learning, especially for the graph data due to the complex relationships among the graph nodes. To deal with the heterogeneity, lots of existing methods perform the weighted federation based on their calculated similarities between pairwise clients (i.e., subgraphs). However, their inter-subgraph similarities estimated with the outputs of local models are less reliable, because the final outputs of local models may not comprehensively represent the real distribution of subgraph data. In addition, they ignore the critical intra-heterogeneity which usually exists within each subgraph itself. To address these issues, we propose a novel Federated learning method by integrally modeling the Inter-Intra Heterogeneity (FedIIH). For the inter-subgraph relationship, we propose a novel hierarchical variational model to infer the whole distribution of subgraph data in a multi-level form, so that we can accurately characterize the inter-subgraph similarities with the global perspective. For the intra-heterogeneity, we disentangle the subgraph into multiple latent factors and partition the model parameters into multiple parts, where each part corresponds to a single latent factor. Our FedIIH not only properly computes the distribution similarities between subgraphs, but also learns disentangled representations that are robust to irrelevant factors within subgraphs, so that it successfully considers the inter- and intra- heterogeneity simultaneously. Extensive experiments on six homophilic and five heterophilic graph datasets in both non-overlapping and overlapping settings demonstrate the effectiveness of our method when compared with nine state-of-the-art methods. Specifically, FedIIH averagely outperforms the second-best method by a large margin of 5.79% on all heterophilic datasets.

Title: Formulations and scalability of neural network surrogates in nonlinear optimization problems

• Authors: Robert B. Parker, Oscar Dowson, Nicole LoGiudice, Manuel Garcia, Russell Bent
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2412.11403
• Pdf URL: https://arxiv.org/pdf/2412.11403
• Copy Paste: [[2412.11403]] Formulations and scalability of neural network surrogates in nonlinear optimization problems(https://arxiv.org/abs/2412.11403)
• Keywords: security
• Abstract: We compare full-space, reduced-space, and gray-box formulations for representing trained neural networks in nonlinear constrained optimization problems. We test these formulations on a transient stability-constrained, security-constrained alternating current optimal power flow (SCOPF) problem where the transient stability criteria are represented by a trained neural network surrogate. Optimization problems are implemented in JuMP and trained neural networks are embedded using a new Julia package: this http URL. To study the bottlenecks of the three formulations, we use neural networks with up to 590 million trained parameters. The full-space formulation is bottlenecked by the linear solver used by the optimization algorithm, while the reduced-space formulation is bottlenecked by the algebraic modeling environment and derivative computations. The gray-box formulation is the most scalable and is capable of solving with the largest neural networks tested. It is bottlenecked by evaluation of the neural network's outputs and their derivatives, which may be accelerated with a graphics processing unit (GPU). Leveraging the gray-box formulation and GPU acceleration, we solve our test problem with our largest neural network surrogate in 2.5$\times$ the time required for a simpler SCOPF problem without the stability constraint.

Title: Attention with Dependency Parsing Augmentation for Fine-Grained Attribution

• Authors: Qiang Ding, Lvzhou Luo, Yixuan Cao, Ping Luo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11404
• Pdf URL: https://arxiv.org/pdf/2412.11404
• Copy Paste: [[2412.11404]] Attention with Dependency Parsing Augmentation for Fine-Grained Attribution(https://arxiv.org/abs/2412.11404)
• Keywords: transformer
• Abstract: To assist humans in efficiently validating RAG-generated content, developing a fine-grained attribution mechanism that provides supporting evidence from retrieved documents for every answer span is essential. Existing fine-grained attribution methods rely on model-internal similarity metrics between responses and documents, such as saliency scores and hidden state similarity. However, these approaches suffer from either high computational complexity or coarse-grained representations. Additionally, a common problem shared by the previous works is their reliance on decoder-only Transformers, limiting their ability to incorporate contextual information after the target span. To address the above problems, we propose two techniques applicable to all model-internals-based methods. First, we aggregate token-wise evidence through set union operations, preserving the granularity of representations. Second, we enhance the attributor by integrating dependency parsing to enrich the semantic completeness of target spans. For practical implementation, our approach employs attention weights as the similarity metric. Experimental results demonstrate that the proposed method consistently outperforms all prior works.

Title: Federated Domain Generalization with Label Smoothing and Balanced Decentralized Training

• Authors: Milad Soltany, Farhad Pourpanah, Mahdiyar Molahasani, Michael Greenspan, Ali Etemad
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11408
• Pdf URL: https://arxiv.org/pdf/2412.11408
• Copy Paste: [[2412.11408]] Federated Domain Generalization with Label Smoothing and Balanced Decentralized Training(https://arxiv.org/abs/2412.11408)
• Keywords: federate
• Abstract: In this paper, we propose a novel approach, Federated Domain Generalization with Label Smoothing and Balanced Decentralized Training (FedSB), to address the challenges of data heterogeneity within a federated learning framework. FedSB utilizes label smoothing at the client level to prevent overfitting to domain-specific features, thereby enhancing generalization capabilities across diverse domains when aggregating local models into a global model. Additionally, FedSB incorporates a decentralized budgeting mechanism which balances training among clients, which is shown to improve the performance of the aggregated global model. Extensive experiments on four commonly used multi-domain datasets, PACS, VLCS, OfficeHome, and TerraInc, demonstrate that FedSB outperforms competing methods, achieving state-of-the-art results on three out of four datasets, indicating the effectiveness of FedSB in addressing data heterogeneity.

Title: Biased or Flawed? Mitigating Stereotypes in Generative Language Models by Addressing Task-Specific Flaws

• Authors: Akshita Jha, Sanchit Kabra, Chandan K. Reddy
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11414
• Pdf URL: https://arxiv.org/pdf/2412.11414
• Copy Paste: [[2412.11414]] Biased or Flawed? Mitigating Stereotypes in Generative Language Models by Addressing Task-Specific Flaws(https://arxiv.org/abs/2412.11414)
• Keywords: generative
• Abstract: Recent studies have shown that generative language models often reflect and amplify societal biases in their outputs. However, these studies frequently conflate observed biases with other task-specific shortcomings, such as comprehension failure. For example, when a model misinterprets a text and produces a response that reinforces a stereotype, it becomes difficult to determine whether the issue arises from inherent bias or from a misunderstanding of the given content. In this paper, we conduct a multi-faceted evaluation that distinctly disentangles bias from flaws within the reading comprehension task. We propose a targeted stereotype mitigation framework that implicitly mitigates observed stereotypes in generative models through instruction-tuning on general-purpose datasets. We reduce stereotypical outputs by over 60% across multiple dimensions -- including nationality, age, gender, disability, and physical appearance -- by addressing comprehension-based failures, and without relying on explicit debiasing techniques. We evaluate several state-of-the-art generative models to demonstrate the effectiveness of our approach while maintaining the overall utility. Our findings highlight the need to critically disentangle the concept of `bias' from other types of errors to build more targeted and effective mitigation strategies. CONTENT WARNING: Some examples contain offensive stereotypes.

Title: ConceptEdit: Conceptualization-Augmented Knowledge Editing in Large Language Models for Commonsense Reasoning

• Authors: Liyu Zhang, Weiqi Wang, Tianqing Fang, Yangqiu Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11418
• Pdf URL: https://arxiv.org/pdf/2412.11418
• Copy Paste: [[2412.11418]] ConceptEdit: Conceptualization-Augmented Knowledge Editing in Large Language Models for Commonsense Reasoning(https://arxiv.org/abs/2412.11418)
• Keywords: large language model
• Abstract: Knowledge Editing (KE) aims to adjust a Large Language Model's (LLM) internal representations and parameters to correct inaccuracies and improve output consistency without incurring the computational expense of re-training the entire model. However, editing commonsense knowledge still faces difficulties, including limited knowledge coverage in existing resources, the infeasibility of annotating labels for an overabundance of commonsense knowledge, and the strict knowledge formats of current editing methods. In this paper, we address these challenges by presenting ConceptEdit, a framework that integrates conceptualization and instantiation into the KE pipeline for LLMs to enhance their commonsense reasoning capabilities. ConceptEdit dynamically diagnoses implausible commonsense knowledge within an LLM using another verifier LLM and augments the source knowledge to be edited with conceptualization for stronger generalizability. Experimental results demonstrate that LLMs enhanced with ConceptEdit successfully generate commonsense knowledge with improved plausibility compared to other baselines and achieve stronger performance across multiple question answering benchmarks.

Title: Category Level 6D Object Pose Estimation from a Single RGB Image using Diffusion

• Authors: Adam Bethell, Ravi Garg, Ian Reid
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11420
• Pdf URL: https://arxiv.org/pdf/2412.11420
• Copy Paste: [[2412.11420]] Category Level 6D Object Pose Estimation from a Single RGB Image using Diffusion(https://arxiv.org/abs/2412.11420)
• Keywords: diffusion
• Abstract: Estimating the 6D pose and 3D size of an object from an image is a fundamental task in computer vision. Most current approaches are restricted to specific instances with known models or require ground truth depth information or point cloud captures from LIDAR. We tackle the harder problem of pose estimation for category-level objects from a single RGB image. We propose a novel solution that eliminates the need for specific object models or depth information. Our method utilises score-based diffusion models to generate object pose hypotheses to model the distribution of possible poses for the object. Unlike previous methods that rely on costly trained likelihood estimators to remove outliers before pose aggregation using mean pooling, we introduce a simpler approach using Mean Shift to estimate the mode of the distribution as the final pose estimate. Our approach outperforms the current state-of-the-art on the REAL275 dataset by a significant margin.

Title: Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models

• Authors: Namhyuk Ahn, KiYoon Yoo, Wonhyuk Ahn, Daesik Kim, Seung-Hun Nam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11423
• Pdf URL: https://arxiv.org/pdf/2412.11423
• Copy Paste: [[2412.11423]] Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models(https://arxiv.org/abs/2412.11423)
• Keywords: protect, robust, diffusion
• Abstract: Recent advancements in diffusion models revolutionize image generation but pose risks of misuse, such as replicating artworks or generating deepfakes. Existing image protection methods, though effective, struggle to balance protection efficacy, invisibility, and latency, thus limiting practical use. We introduce perturbation pre-training to reduce latency and propose a mixture-of-perturbations approach that dynamically adapts to input images to minimize performance degradation. Our novel training strategy computes protection loss across multiple VAE feature spaces, while adaptive targeted protection at inference enhances robustness and invisibility. Experiments show comparable protection performance with improved invisibility and drastically reduced inference time. The code and demo are available at \url{this https URL}

Title: Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges

• Authors: Chandan K Reddy, Parshin Shojaee
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11427
• Pdf URL: https://arxiv.org/pdf/2412.11427
• Copy Paste: [[2412.11427]] Towards Scientific Discovery with Generative AI: Progress, Opportunities, and Challenges(https://arxiv.org/abs/2412.11427)
• Keywords: generative, large language model
• Abstract: Scientific discovery is a complex cognitive process that has driven human knowledge and technological progress for centuries. While artificial intelligence (AI) has made significant advances in automating aspects of scientific reasoning, simulation, and experimentation, we still lack integrated AI systems capable of performing autonomous long-term scientific research and discovery. This paper examines the current state of AI for scientific discovery, highlighting recent progress in large language models and other AI techniques applied to scientific tasks. We then outline key challenges and promising research directions toward developing more comprehensive AI systems for scientific discovery, including the need for science-focused AI agents, improved benchmarks and evaluation metrics, multimodal scientific representations, and unified frameworks combining reasoning, theorem proving, and data-driven modeling. Addressing these challenges could lead to transformative AI tools to accelerate progress across disciplines towards scientific discovery.

Title: View Transformation Robustness for Multi-View 3D Object Reconstruction with Reconstruction Error-Guided View Selection

• Authors: Qi Zhang, Zhouhang Luo, Tao Yu, Hui Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11428
• Pdf URL: https://arxiv.org/pdf/2412.11428
• Copy Paste: [[2412.11428]] View Transformation Robustness for Multi-View 3D Object Reconstruction with Reconstruction Error-Guided View Selection(https://arxiv.org/abs/2412.11428)
• Keywords: robust, diffusion
• Abstract: View transformation robustness (VTR) is critical for deep-learning-based multi-view 3D object reconstruction models, which indicates the methods' stability under inputs with various view transformations. However, existing research seldom focused on view transformation robustness in multi-view 3D object reconstruction. One direct way to improve the models' VTR is to produce data with more view transformations and add them to model training. Recent progress on large vision models, particularly Stable Diffusion models, has provided great potential for generating 3D models or synthesizing novel view images with only a single image input. Directly deploying these models at inference consumes heavy computation resources and their robustness to view transformations is not guaranteed either. To fully utilize the power of Stable Diffusion models without extra inference computation burdens, we propose to generate novel views with Stable Diffusion models for better view transformation robustness. Instead of synthesizing random views, we propose a reconstruction error-guided view selection method, which considers the reconstruction errors' spatial distribution of the 3D predictions and chooses the views that could cover the reconstruction errors as much as possible. The methods are trained and tested on sets with large view transformations to validate the 3D reconstruction models' robustness to view transformations. Extensive experiments demonstrate that the proposed method can outperform state-of-the-art 3D reconstruction methods and other view transformation robustness comparison methods.

Title: Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models

• Authors: Mohamed Basem, Islam Oshallah, Baraa Hikal, Ali Hamdi, Ammar Mohamed
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.11431
• Pdf URL: https://arxiv.org/pdf/2412.11431
• Copy Paste: [[2412.11431]] Optimized Quran Passage Retrieval Using an Expanded QA Dataset and Fine-Tuned Language Models(https://arxiv.org/abs/2412.11431)
• Keywords: transformer
• Abstract: Understanding the deep meanings of the Qur'an and bridging the language gap between modern standard Arabic and classical Arabic is essential to improve the question-and-answer system for the Holy Qur'an. The Qur'an QA 2023 shared task dataset had a limited number of questions with weak model retrieval. To address this challenge, this work updated the original dataset and improved the model accuracy. The original dataset, which contains 251 questions, was reviewed and expanded to 629 questions with question diversification and reformulation, leading to a comprehensive set of 1895 categorized into single-answer, multi-answer, and zero-answer types. Extensive experiments fine-tuned transformer models, including AraBERT, RoBERTa, CAMeLBERT, AraELECTRA, and BERT. The best model, AraBERT-base, achieved a MAP@10 of 0.36 and MRR of 0.59, representing improvements of 63% and 59%, respectively, compared to the baseline scores (MAP@10: 0.22, MRR: 0.37). Additionally, the dataset expansion led to improvements in handling "no answer" cases, with the proposed approach achieving a 75% success rate for such instances, compared to the baseline's 25%. These results demonstrate the effect of dataset improvement and model architecture optimization in increasing the performance of QA systems for the Holy Qur'an, with higher accuracy, recall, and precision.

Title: Auto-bidding in real-time auctions via Oracle Imitation Learning

• Authors: Alberto Chiappa, Briti Gangopadhyay, Zhao Wang, Shingo Takamatsu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11434
• Pdf URL: https://arxiv.org/pdf/2412.11434
• Copy Paste: [[2412.11434]] Auto-bidding in real-time auctions via Oracle Imitation Learning(https://arxiv.org/abs/2412.11434)
• Keywords: secure
• Abstract: Online advertising has become one of the most successful business models of the internet era. Impression opportunities are typically allocated through real-time auctions, where advertisers bid to secure advertisement slots. Deciding the best bid for an impression opportunity is challenging, due to the stochastic nature of user behavior and the variability of advertisement traffic over time. In this work, we propose a framework for training auto-bidding agents in multi-slot second-price auctions to maximize acquisitions (e.g., clicks, conversions) while adhering to budget and cost-per-acquisition (CPA) constraints. We exploit the insight that, after an advertisement campaign concludes, determining the optimal bids for each impression opportunity can be framed as a multiple-choice knapsack problem (MCKP) with a nonlinear objective. We propose an "oracle" algorithm that identifies a near-optimal combination of impression opportunities and advertisement slots, considering both past and future advertisement traffic data. This oracle solution serves as a training target for a student network which bids having access only to real-time information, a method we term Oracle Imitation Learning (OIL). Through numerical experiments, we demonstrate that OIL achieves superior performance compared to both online and offline reinforcement learning algorithms, offering improved sample efficiency. Notably, OIL shifts the complexity of training auto-bidding agents from crafting sophisticated learning algorithms to solving a nonlinear constrained optimization problem efficiently.

Title: Learning Implicit Features with Flow Infused Attention for Realistic Virtual Try-On

• Authors: Delong Zhang, Qiwei Huang, Yuanliu Liu, Yang Sun, Wei-Shi Zheng, Pengfei Xiong, Wei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11435
• Pdf URL: https://arxiv.org/pdf/2412.11435
• Copy Paste: [[2412.11435]] Learning Implicit Features with Flow Infused Attention for Realistic Virtual Try-On(https://arxiv.org/abs/2412.11435)
• Keywords: robust
• Abstract: Image-based virtual try-on is challenging since the generated image should fit the garment to model images in various poses and keep the characteristics and details of the garment simultaneously. A popular research stream warps the garment image firstly to reduce the burden of the generation stage, which relies highly on the performance of the warping module. Other methods without explicit warping often lack sufficient guidance to fit the garment to the model images. In this paper, we propose FIA-VTON, which leverages the implicit warp feature by adopting a Flow Infused Attention module on virtual try-on. The dense warp flow map is projected as indirect guidance attention to enhance the feature map warping in the generation process implicitly, which is less sensitive to the warping estimation accuracy than an explicit warp of the garment image. To further enhance implicit warp guidance, we incorporate high-level spatial attention to complement the dense warp. Experimental results on the VTON-HD and DressCode dataset significantly outperform state-of-the-art methods, demonstrating that FIA-VTON is effective and robust for virtual try-on.

Title: Bayesian Flow Is All You Need to Sample Out-of-Distribution Chemical Spaces

• Authors: Nianze Tao
• Subjects: cs.LG, cs.AI, physics.chem-ph
• Abstract URL: https://arxiv.org/abs/2412.11439
• Pdf URL: https://arxiv.org/pdf/2412.11439
• Copy Paste: [[2412.11439]] Bayesian Flow Is All You Need to Sample Out-of-Distribution Chemical Spaces(https://arxiv.org/abs/2412.11439)
• Keywords: diffusion
• Abstract: Generating novel molecules with higher properties than the training space, namely the out-of-distribution generation, is important for ${de~novo}$ drug design. However, it is not easy for distribution learning-based models, for example diffusion models, to solve this challenge as these methods are designed to fit the distribution of training data as close as possible. In this paper, we show that Bayesian flow network is capable of effortlessly generating high quality out-of-distribution samples that meet several scenarios. We introduce a semi-autoregressive training/sampling method that helps to enhance the model performance and surpass the state-of-the-art models.

Title: UIBDiffusion: Universal Imperceptible Backdoor Attack for Diffusion Models

• Authors: Yuning Han, Bingyin Zhao, Rui Chu, Feng Luo, Biplab Sikdar, Yingjie Lao
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11441
• Pdf URL: https://arxiv.org/pdf/2412.11441
• Copy Paste: [[2412.11441]] UIBDiffusion: Universal Imperceptible Backdoor Attack for Diffusion Models(https://arxiv.org/abs/2412.11441)
• Keywords: defense, attack, steal, diffusion
• Abstract: Recent studies show that diffusion models (DMs) are vulnerable to backdoor attacks. Existing backdoor attacks impose unconcealed triggers (e.g., a gray box and eyeglasses) that contain evident patterns, rendering remarkable attack effects yet easy detection upon human inspection and defensive algorithms. While it is possible to improve stealthiness by reducing the strength of the backdoor, doing so can significantly compromise its generality and effectiveness. In this paper, we propose UIBDiffusion, the universal imperceptible backdoor attack for diffusion models, which allows us to achieve superior attack and generation performance while evading state-of-the-art defenses. We propose a novel trigger generation approach based on universal adversarial perturbations (UAPs) and reveal that such perturbations, which are initially devised for fooling pre-trained discriminative models, can be adapted as potent imperceptible backdoor triggers for DMs. We evaluate UIBDiffusion on multiple types of DMs with different kinds of samplers across various datasets and targets. Experimental results demonstrate that UIBDiffusion brings three advantages: 1) Universality, the imperceptible trigger is universal (i.e., image and model agnostic) where a single trigger is effective to any images and all diffusion models with different samplers; 2) Utility, it achieves comparable generation quality (e.g., FID) and even better attack success rate (i.e., ASR) at low poison rates compared to the prior works; and 3) Undetectability, UIBDiffusion is plausible to human perception and can bypass Elijah and TERD, the SOTA defenses against backdoors for DMs. We will release our backdoor triggers and code.

Title: TRAIL: Trust-Aware Client Scheduling for Semi-Decentralized Federated Learning

• Authors: Gangqiang Hu, Jianfeng Lu, Jianmin Han, Shuqin Cao, Jing Liu, Hao Fu
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.11448
• Pdf URL: https://arxiv.org/pdf/2412.11448
• Copy Paste: [[2412.11448]] TRAIL: Trust-Aware Client Scheduling for Semi-Decentralized Federated Learning(https://arxiv.org/abs/2412.11448)
• Keywords: privacy, federate
• Abstract: Due to the sensitivity of data, federated learning (FL) is employed to enable distributed machine learning while safeguarding data privacy and accommodating the requirements of various devices. However, in the context of semi-decentralized federated learning (SD-FL), clients' communication and training states are dynamic. This variability arises from local training fluctuations, heterogeneous data distributions, and intermittent client participation. Most existing studies primarily focus on stable client states, neglecting the dynamic challenges present in real-world scenarios. To tackle this issue, we propose a trust-aware client scheduling mechanism (TRAIL) that assesses client states and contributions, enhancing model training efficiency through selective client participation. Our focus is on a semi-decentralized federated learning framework where edge servers and clients train a shared global model using unreliable intra-cluster model aggregation and inter-cluster model consensus. First, we develop an adaptive hidden semi-Markov model (AHSMM) to estimate clients' communication states and contributions. Next, we address a client-server association optimization problem to minimize global training loss. Using convergence analysis, we propose a greedy client scheduling algorithm. Finally, our experiments conducted on real-world datasets demonstrate that TRAIL outperforms state-of-the-art baselines, achieving an improvement of 8.7\% in test accuracy and a reduction of 15.3\% in training loss.

Title: Data-Dependent Generalization Bounds for Parameterized Quantum Models Under Noise

• Authors: Bikram Khanal, Pablo Rivas
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11451
• Pdf URL: https://arxiv.org/pdf/2412.11451
• Copy Paste: [[2412.11451]] Data-Dependent Generalization Bounds for Parameterized Quantum Models Under Noise(https://arxiv.org/abs/2412.11451)
• Keywords: robust
• Abstract: Quantum machine learning offers a transformative approach to solving complex problems, but the inherent noise hinders its practical implementation in near-term quantum devices. This obstacle makes it challenging to understand the generalization capabilities of quantum circuit models. Designing robust quantum machine learning models under noise requires a principled understanding of complexity and generalization, extending beyond classical capacity measures. This study investigates the generalization properties of parameterized quantum machine learning models under the influence of noise. We present a data-dependent generalization bound grounded in the quantum Fisher information matrix. We leverage statistical learning theory to relate the parameter space volumes and training sizes to estimate the generalization capability of the trained model. By integrating local parameter neighborhoods and effective dimensions defined through quantum Fisher information matrix eigenvalues, we provide a structured characterization of complexity in quantum models. We analyze the tightness of the bound and discuss the trade-off between model expressiveness and generalization performance.

Title: Multilabel Classification for Lung Disease Detection: Integrating Deep Learning and Natural Language Processing

• Authors: Maria Efimovich, Jayden Lim, Vedant Mehta, Ethan Poon
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.11452
• Pdf URL: https://arxiv.org/pdf/2412.11452
• Copy Paste: [[2412.11452]] Multilabel Classification for Lung Disease Detection: Integrating Deep Learning and Natural Language Processing(https://arxiv.org/abs/2412.11452)
• Keywords: extraction
• Abstract: Classifying chest radiographs is a time-consuming and challenging task, even for experienced radiologists. This provides an area for improvement due to the difficulty in precisely distinguishing between conditions such as pleural effusion, pneumothorax, and pneumonia. We propose a novel transfer learning model for multi-label lung disease classification, utilizing the CheXpert dataset with over 12,617 images of frontal radiographs being analyzed. By integrating RadGraph parsing for efficient annotation extraction, we enhance the model's ability to accurately classify multiple lung diseases from complex medical images. The proposed model achieved an F1 score of 0.69 and an AUROC of 0.86, demonstrating its potential for clinical applications. Also explored was the use of Natural Language Processing (NLP) to parse report metadata and address uncertainties in disease classification. By comparing uncertain reports with more certain cases, the NLP-enhanced model improves its ability to conclusively classify conditions. This research highlights the connection between deep learning and NLP, underscoring their potential to enhance radiological diagnostics and aid in the efficient analysis of chest radiographs.

Title: ACE-$M^3$: Automatic Capability Evaluator for Multimodal Medical Models

• Authors: Xiechi Zhang, Shunfan Zheng, Linlin Wang, Gerard de Melo, Zhu Cao, Xiaoling Wang, Liang He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11453
• Pdf URL: https://arxiv.org/pdf/2412.11453
• Copy Paste: [[2412.11453]] ACE-$M^3$: Automatic Capability Evaluator for Multimodal Medical Models(https://arxiv.org/abs/2412.11453)
• Keywords: large language model
• Abstract: As multimodal large language models (MLLMs) gain prominence in the medical field, the need for precise evaluation methods to assess their effectiveness has become critical. While benchmarks provide a reliable means to evaluate the capabilities of MLLMs, traditional metrics like ROUGE and BLEU employed for open domain evaluation only focus on token overlap and may not align with human judgment. Although human evaluation is more reliable, it is labor-intensive, costly, and not scalable. LLM-based evaluation methods have proven promising, but to date, there is still an urgent need for open-source multimodal LLM-based evaluators in the medical field. To address this issue, we introduce ACE-$M^3$, an open-sourced \textbf{A}utomatic \textbf{C}apability \textbf{E}valuator for \textbf{M}ultimodal \textbf{M}edical \textbf{M}odels specifically designed to assess the question answering abilities of medical MLLMs. It first utilizes a branch-merge architecture to provide both detailed analysis and a concise final score based on standard medical evaluation criteria. Subsequently, a reward token-based direct preference optimization (RTDPO) strategy is incorporated to save training time without compromising performance of our model. Extensive experiments have demonstrated the effectiveness of our ACE-$M^3$ model\footnote{\url{this https URL}} in evaluating the capabilities of medical MLLMs.

Title: Towards Better Multi-task Learning: A Framework for Optimizing Dataset Combinations in Large Language Models

• Authors: Zaifu Zhan, Rui Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11455
• Pdf URL: https://arxiv.org/pdf/2412.11455
• Copy Paste: [[2412.11455]] Towards Better Multi-task Learning: A Framework for Optimizing Dataset Combinations in Large Language Models(https://arxiv.org/abs/2412.11455)
• Keywords: extraction, large language model
• Abstract: To efficiently select optimal dataset combinations for enhancing multi-task learning (MTL) performance in large language models, we proposed a novel framework that leverages a neural network to predict the best dataset combinations. The framework iteratively refines the selection, greatly improving efficiency, while being model-, dataset-, and domain-independent. Through experiments on 12 biomedical datasets across four tasks - named entity recognition, relation extraction, event extraction, and text classification-we demonstrate that our approach effectively identifies better combinations, even for tasks that may seem unpromising from a human perspective. This verifies that our framework provides a promising solution for maximizing MTL potential.

Title: MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes

• Authors: Ruijie Lu, Yixin Chen, Junfeng Ni, Baoxiong Jia, Yu Liu, Diwen Wan, Gang Zeng, Siyuan Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11457
• Pdf URL: https://arxiv.org/pdf/2412.11457
• Copy Paste: [[2412.11457]] MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes(https://arxiv.org/abs/2412.11457)
• Keywords: diffusion
• Abstract: Repurposing pre-trained diffusion models has been proven to be effective for NVS. However, these methods are mostly limited to a single object; directly applying such methods to compositional multi-object scenarios yields inferior results, especially incorrect object placement and inconsistent shape and appearance under novel views. How to enhance and systematically evaluate the cross-view consistency of such models remains under-explored. To address this issue, we propose MOVIS to enhance the structural awareness of the view-conditioned diffusion model for multi-object NVS in terms of model inputs, auxiliary tasks, and training strategy. First, we inject structure-aware features, including depth and object mask, into the denoising U-Net to enhance the model's comprehension of object instances and their spatial relationships. Second, we introduce an auxiliary task requiring the model to simultaneously predict novel view object masks, further improving the model's capability in differentiating and placing objects. Finally, we conduct an in-depth analysis of the diffusion sampling process and carefully devise a structure-guided timestep sampling scheduler during training, which balances the learning of global object placement and fine-grained detail recovery. To systematically evaluate the plausibility of synthesized images, we propose to assess cross-view consistency and novel view object placement alongside existing image-level NVS metrics. Extensive experiments on challenging synthetic and realistic datasets demonstrate that our method exhibits strong generalization capabilities and produces consistent novel view synthesis, highlighting its potential to guide future 3D-aware multi-object NVS tasks.

Title: HResFormer: Hybrid Residual Transformer for Volumetric Medical Image Segmentation

• Authors: Sucheng Ren, Xiaomeng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11458
• Pdf URL: https://arxiv.org/pdf/2412.11458
• Copy Paste: [[2412.11458]] HResFormer: Hybrid Residual Transformer for Volumetric Medical Image Segmentation(https://arxiv.org/abs/2412.11458)
• Keywords: transformer, segmentation
• Abstract: Vision Transformer shows great superiority in medical image segmentation due to the ability in learning long-range dependency. For medical image segmentation from 3D data, such as computed tomography (CT), existing methods can be broadly classified into 2D-based and 3D-based methods. One key limitation in 2D-based methods is that the intra-slice information is ignored, while the limitation in 3D-based methods is the high computation cost and memory consumption, resulting in a limited feature representation for inner-slice information. During the clinical examination, radiologists primarily use the axial plane and then routinely review both axial and coronal planes to form a 3D understanding of anatomy. Motivated by this fact, our key insight is to design a hybrid model which can first learn fine-grained inner-slice information and then generate a 3D understanding of anatomy by incorporating 3D information. We present a novel \textbf{H}ybrid \textbf{Res}idual trans\textbf{Former} \textbf{(HResFormer)} for 3D medical image segmentation. Building upon standard 2D and 3D Transformer backbones, HResFormer involves two novel key designs: \textbf{(1)} a \textbf{H}ybrid \textbf{L}ocal-\textbf{G}lobal fusion \textbf{M}odule \textbf{(HLGM)} to effectively and adaptively fuse inner-slice information from 2D Transformer and intra-slice information from 3D volumes for 3D Transformer with local fine-grained and global long-range representation. \textbf{(2)} a residual learning of the hybrid model, which can effectively leverage the inner-slice and intra-slice information for better 3D understanding of anatomy. Experiments show that our HResFormer outperforms prior art on widely-used medical image segmentation benchmarks. This paper sheds light on an important but neglected way to design Transformers for 3D medical image segmentation.

Title: Understanding Knowledge Hijack Mechanism in In-context Learning through Associative Memory

• Authors: Shuo Wang, Issei Sato
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11459
• Pdf URL: https://arxiv.org/pdf/2412.11459
• Copy Paste: [[2412.11459]] Understanding Knowledge Hijack Mechanism in In-context Learning through Associative Memory(https://arxiv.org/abs/2412.11459)
• Keywords: transformer, large language model
• Abstract: In-context learning (ICL) enables large language models (LLMs) to adapt to new tasks without fine-tuning by leveraging contextual information provided within a prompt. However, ICL relies not only on contextual clues but also on the global knowledge acquired during pretraining for the next token prediction. Analyzing this process has been challenging due to the complex computational circuitry of LLMs. This paper investigates the balance between in-context information and pretrained bigram knowledge in token prediction, focusing on the induction head mechanism, a key component in ICL. Leveraging the fact that a two-layer transformer can implement the induction head mechanism with associative memories, we theoretically analyze the logits when a two-layer transformer is given prompts generated by a bigram model. In the experiments, we design specific prompts to evaluate whether the outputs of a two-layer transformer align with the theoretical results.

Title: FedCAR: Cross-client Adaptive Re-weighting for Generative Models in Federated Learning

• Authors: Minjun Kim, Minjee Kim, Jinhoon Jeong
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11463
• Pdf URL: https://arxiv.org/pdf/2412.11463
• Copy Paste: [[2412.11463]] FedCAR: Cross-client Adaptive Re-weighting for Generative Models in Federated Learning(https://arxiv.org/abs/2412.11463)
• Keywords: privacy, federate, generative
• Abstract: Generative models trained on multi-institutional datasets can provide an enriched understanding through diverse data distributions. However, training the models on medical images is often challenging due to hospitals' reluctance to share data for privacy reasons. Federated learning(FL) has emerged as a privacy-preserving solution for training distributed datasets across data centers by aggregating model weights from multiple clients instead of sharing raw data. Previous research has explored the adaptation of FL to generative models, yet effective aggregation algorithms specifically tailored for generative models remain unexplored. We hereby propose a novel algorithm aimed at improving the performance of generative models within FL. Our approach adaptively re-weights the contribution of each client, resulting in well-trained shared parameters. In each round, the server side measures the distribution distance between fake images generated by clients instead of directly comparing the Fréchet Inception Distance per client, thereby enhancing efficiency of the learning. Experimental results on three public chest X-ray datasets show superior performance in medical image generation, outperforming both centralized learning and conventional FL algorithms. Our code is available at this https URL.

Title: MaskCLIP++: A Mask-Based CLIP Fine-tuning Framework for Open-Vocabulary Image Segmentation

• Authors: Quan-Sheng Zeng, Yunheng Li, Daquan Zhou, Guanbin Li, Qibin Hou, Ming-Ming Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11464
• Pdf URL: https://arxiv.org/pdf/2412.11464
• Copy Paste: [[2412.11464]] MaskCLIP++: A Mask-Based CLIP Fine-tuning Framework for Open-Vocabulary Image Segmentation(https://arxiv.org/abs/2412.11464)
• Keywords: segmentation
• Abstract: Open-vocabulary image segmentation has been advanced through the synergy between mask generators and vision-language models like Contrastive Language-Image Pre-training (CLIP). Previous approaches focus on generating masks while aligning mask features with text embeddings during training. In this paper, we observe that relying on generated low-quality masks can weaken the alignment of vision and language in regional representations. This motivates us to present a new fine-tuning framework, named MaskCLIP++, which uses ground-truth masks instead of generated masks to enhance the mask classification capability of CLIP. Due to the limited diversity of image segmentation datasets with mask annotations, we propose incorporating a consistency alignment constraint during fine-tuning, which alleviates categorical bias toward the fine-tuning dataset. After low-cost fine-tuning, combining with the mask generator in previous state-of-the-art mask-based open vocabulary segmentation methods, we achieve performance improvements of +1.7, +2.3, +2.1, +3.1, and +0.3 mIoU on the A-847, PC-459, A-150, PC-59, and PAS-20 datasets, respectively.

Title: Exploring Temporal Event Cues for Dense Video Captioning in Cyclic Co-learning

• Authors: Zhuyang Xie, Yan Yang, Yankai Yu, Jie Wang, Yongquan Jiang, Xiao Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11467
• Pdf URL: https://arxiv.org/pdf/2412.11467
• Copy Paste: [[2412.11467]] Exploring Temporal Event Cues for Dense Video Captioning in Cyclic Co-learning(https://arxiv.org/abs/2412.11467)
• Keywords: interpretability
• Abstract: Dense video captioning aims to detect and describe all events in untrimmed videos. This paper presents a dense video captioning network called Multi-Concept Cyclic Learning (MCCL), which aims to: (1) detect multiple concepts at the frame level, using these concepts to enhance video features and provide temporal event cues; and (2) design cyclic co-learning between the generator and the localizer within the captioning network to promote semantic perception and event localization. Specifically, we perform weakly supervised concept detection for each frame, and the detected concept embeddings are integrated into the video features to provide event cues. Additionally, video-level concept contrastive learning is introduced to obtain more discriminative concept embeddings. In the captioning network, we establish a cyclic co-learning strategy where the generator guides the localizer for event localization through semantic matching, while the localizer enhances the generator's event semantic perception through location matching, making semantic perception and event localization mutually beneficial. MCCL achieves state-of-the-art performance on the ActivityNet Captions and YouCook2 datasets. Extensive experiments demonstrate its effectiveness and interpretability.

Title: Red Pill and Blue Pill: Controllable Website Fingerprinting Defense via Dynamic Backdoor Learning

• Authors: Siyuan Liang, Jiajun Gong, Tianmeng Fang, Aishan Liu, Tao Wang, Xianglong Liu, Xiaochun Cao, Dacheng Tao, Chang Ee-Chien
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11471
• Pdf URL: https://arxiv.org/pdf/2412.11471
• Copy Paste: [[2412.11471]] Red Pill and Blue Pill: Controllable Website Fingerprinting Defense via Dynamic Backdoor Learning(https://arxiv.org/abs/2412.11471)
• Keywords: privacy, defense, attack
• Abstract: Website fingerprint (WF) attacks, which covertly monitor user communications to identify the web pages they visit, pose a serious threat to user privacy. Existing WF defenses attempt to reduce the attacker's accuracy by disrupting unique traffic patterns; however, they often suffer from the trade-off between overhead and effectiveness, resulting in less usefulness in practice. To overcome this limitation, we introduce Controllable Website Fingerprint Defense (CWFD), a novel defense perspective based on backdoor learning. CWFD exploits backdoor vulnerabilities in neural networks to directly control the attacker's model by designing trigger patterns based on network traffic. Specifically, CWFD injects only incoming packets on the server side into the target web page's traffic, keeping overhead low while effectively poisoning the attacker's model during training. During inference, the defender can influence the attacker's model through a 'red pill, blue pill' choice: traces with the trigger (red pill) lead to misclassification as the target web page, while normal traces (blue pill) are classified correctly, achieving directed control over the defense outcome. We use the Fast Levenshtein-like distance as the optimization objective to compute trigger patterns that can be effectively associated with our target page. Experiments show that CWFD significantly reduces RF's accuracy from 99% to 6% with 74% data overhead. In comparison, FRONT reduces accuracy to only 97% at similar overhead, while Palette achieves 32% accuracy with 48% more overhead. We further validate the practicality of our method in a real Tor network environment.

Title: Leveraging Foundation Language Models (FLMs) for Automated Cohort Extraction from Large EHR Databases

• Authors: Purity Mugambi, Alexandra Meliou, Madalina Fiterau
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11472
• Pdf URL: https://arxiv.org/pdf/2412.11472
• Copy Paste: [[2412.11472]] Leveraging Foundation Language Models (FLMs) for Automated Cohort Extraction from Large EHR Databases(https://arxiv.org/abs/2412.11472)
• Keywords: extraction
• Abstract: A crucial step in cohort studies is to extract the required cohort from one or more study datasets. This step is time-consuming, especially when a researcher is presented with a dataset that they have not previously worked with. When the cohort has to be extracted from multiple datasets, cohort extraction can be extremely laborious. In this study, we present an approach for partially automating cohort extraction from multiple electronic health record (EHR) databases. We formulate the guided multi-dataset cohort extraction problem in which selection criteria are first converted into queries, translating them from natural language text to language that maps to database entities. Then, using FLMs, columns of interest identified from the queries are automatically matched between the study databases. Finally, the generated queries are run across all databases to extract the study cohort. We propose and evaluate an algorithm for automating column matching on two large, popular and publicly-accessible EHR databases -- MIMIC-III and eICU. Our approach achieves a high top-three accuracy of $92\%$, correctly matching $12$ out of the $13$ columns of interest, when using a small, pre-trained general purpose language model. Furthermore, this accuracy is maintained even as the search space (i.e., size of the database) increases.

Title: Noise-Resilient Homomorphic Encryption: A Framework for Secure Data Processing in Health care Domain

• Authors: B. Shuriya, S. Vimal Kumar, K. Bagyalakshmi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.11474
• Pdf URL: https://arxiv.org/pdf/2412.11474
• Copy Paste: [[2412.11474]] Noise-Resilient Homomorphic Encryption: A Framework for Secure Data Processing in Health care Domain(https://arxiv.org/abs/2412.11474)
• Keywords: secure, security, privacy, robust
• Abstract: In this paper, we introduce the Fully Homomorphic Integrity Model (HIM), a novel approach designed to enhance security, efficiency, and reliability in encrypted data processing, primarily within the health care industry. HIM addresses the key challenges that noise accumulation, computational overheads, and data integrity pose during homomorphic operations. Our contribution of HIM: advances in noise management through the rational number adjustment; key generation based on personalized prime numbers; and time complexity analysis details for key operations. In HIM, some additional mechanisms were introduced, including robust mechanisms of decryption. Indeed, the decryption mechanism ensures that the data recovered upon doing complex homomorphic computation will be valid and reliable. The healthcare id model is tested, and it supports real-time processing of data with privacy maintained concerning patients. It supports analytics and decision-making processes without any compromise on the integrity of information concerning patients. Output HIM promotes the efficiency of encryption to a greater extent as it reduces the encryption time up to 35ms and decryption time up to 140ms, which is better when compared to other models in the existence. Ciphertext size also becomes the smallest one, which is 4KB. Our experiments confirm that HIM is indeed a very efficient and secure privacy-preserving solution for healthcare applications

Title: Vertical Federated Unlearning via Backdoor Certification

• Authors: Mengde Han, Tianqing Zhu, Lefeng Zhang, Huan Huo, Wanlei Zhou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11476
• Pdf URL: https://arxiv.org/pdf/2412.11476
• Copy Paste: [[2412.11476]] Vertical Federated Unlearning via Backdoor Certification(https://arxiv.org/abs/2412.11476)
• Keywords: privacy, federate
• Abstract: Vertical Federated Learning (VFL) offers a novel paradigm in machine learning, enabling distinct entities to train models cooperatively while maintaining data privacy. This method is particularly pertinent when entities possess datasets with identical sample identifiers but diverse attributes. Recent privacy regulations emphasize an individual's \emph{right to be forgotten}, which necessitates the ability for models to unlearn specific training data. The primary challenge is to develop a mechanism to eliminate the influence of a specific client from a model without erasing all relevant data from other clients. Our research investigates the removal of a single client's contribution within the VFL framework. We introduce an innovative modification to traditional VFL by employing a mechanism that inverts the typical learning trajectory with the objective of extracting specific data contributions. This approach seeks to optimize model performance using gradient ascent, guided by a pre-defined constrained model. We also introduce a backdoor mechanism to verify the effectiveness of the unlearning procedure. Our method avoids fully accessing the initial training data and avoids storing parameter updates. Empirical evidence shows that the results align closely with those achieved by retraining from scratch. Utilizing gradient ascent, our unlearning approach addresses key challenges in VFL, laying the groundwork for future advancements in this domain. All the code and implementations related to this paper are publicly available at this https URL.

Title: NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text

• Authors: Prajwal Kailas, Max Homilius, Rahul C. Deo, Calum A. MacRae
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11477
• Pdf URL: https://arxiv.org/pdf/2412.11477
• Copy Paste: [[2412.11477]] NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text(https://arxiv.org/abs/2412.11477)
• Keywords: transformer, large language model
• Abstract: Accurate diagnostic coding of medical notes is crucial for enhancing patient care, medical research, and error-free billing in healthcare organizations. Manual coding is a time-consuming task for providers, and diagnostic codes often exhibit low sensitivity and specificity, whereas the free text in medical notes can be a more precise description of a patients status. Thus, accurate automated diagnostic coding of medical notes has become critical for a learning healthcare system. Recent developments in long-document transformer architectures have enabled attention-based deep-learning models to adjudicate medical notes. In addition, contrastive loss functions have been used to jointly pre-train large language and image models with noisy labels. To further improve the automated adjudication of medical notes, we developed an approach based on i) models for ICD-10 diagnostic code sequences using a large real-world data set, ii) large language models for medical notes, and iii) contrastive pre-training to build an integrated model of both ICD-10 diagnostic codes and corresponding medical text. We demonstrate that a contrastive approach for pre-training improves performance over prior state-of-the-art models for the MIMIC-III-50, MIMIC-III-rare50, and MIMIC-III-full diagnostic coding tasks.

Title: WFCAT: Augmenting Website Fingerprinting with Channel-wise Attention on Timing Features

• Authors: Jiajun Gong, Wei Cai, Siyuan Liang, Zhong Guan, Tao Wang, Ee-Chien Chang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.11487
• Pdf URL: https://arxiv.org/pdf/2412.11487
• Copy Paste: [[2412.11487]] WFCAT: Augmenting Website Fingerprinting with Channel-wise Attention on Timing Features(https://arxiv.org/abs/2412.11487)
• Keywords: defense, attack, robust
• Abstract: Website Fingerprinting (WF) aims to deanonymize users on the Tor network by analyzing encrypted network traffic. Recent deep-learning-based attacks show high accuracy on undefended traces. However, they struggle against modern defenses that use tactics like injecting dummy packets and delaying real packets, which significantly degrade classification performance. Our analysis reveals that current attacks inadequately leverage the timing information inherent in traffic traces, which persists as a source of leakage even under robust defenses. Addressing this shortfall, we introduce a novel feature representation named the Inter-Arrival Time (IAT) histogram, which quantifies the frequencies of packet inter-arrival times across predetermined time slots. Complementing this feature, we propose a new CNN-based attack, WFCAT, enhanced with two innovative architectural blocks designed to optimally extract and utilize timing information. Our approach uses kernels of varying sizes to capture multi-scale features, which are then integrated using a weighted sum across all feature channels to enhance the model's efficacy in identifying temporal patterns. Our experiments validate that WFCAT substantially outperforms existing methods on defended traces in both closed- and open-world scenarios. Notably, WFCAT achieves over 59% accuracy against Surakav, a recently developed robust defense, marking an improvement of over 28% and 48% against the state-of-the-art attacks RF and Tik-Tok, respectively, in the closed-world scenario.

Title: FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing

• Authors: Zekai Li, Jintu Zheng, Ji Liu, Han Liu, Haowei Zhu, Zeping Li, Fuwei Yang, Haiduo Huang, Jinzhang Peng, Dong Li, Lu Tian, Emad Barsoum
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11494
• Pdf URL: https://arxiv.org/pdf/2412.11494
• Copy Paste: [[2412.11494]] FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing(https://arxiv.org/abs/2412.11494)
• Keywords: large language model
• Abstract: Recently, large language models (LLMs) have demonstrated superior performance across various tasks by adhering to scaling laws, which significantly increase model size. However, the huge computation overhead during inference hinders the deployment in industrial applications. Many works leverage traditional compression approaches to boost model inference, but these always introduce additional training costs to restore the performance and the pruning results typically show noticeable performance drops compared to the original model when aiming for a specific level of acceleration. To address these issues, we propose a fine-grained token-wise pruning approach for the LLMs, which presents a learnable router to adaptively identify the less important tokens and skip them across model blocks to reduce computational cost during inference. To construct the router efficiently, we present a search-based sparsity scheduler for pruning sparsity allocation, a trainable router combined with our proposed four low-dimensional factors as input and three proposed losses. We conduct extensive experiments across different benchmarks on different LLMs to demonstrate the superiority of our method. Our approach achieves state-of-the-art (SOTA) pruning results, surpassing other existing pruning methods. For instance, our method outperforms BlockPruner and ShortGPT by approximately 10 points on both LLaMA2-7B and Qwen1.5-7B in accuracy retention at comparable token sparsity levels.

Title: Exploring More from Multiple Gait Modalities for Human Identification

• Authors: Dongyang Jin, Chao Fan, Weihua Chen, Shiqi Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11495
• Pdf URL: https://arxiv.org/pdf/2412.11495
• Copy Paste: [[2412.11495]] Exploring More from Multiple Gait Modalities for Human Identification(https://arxiv.org/abs/2412.11495)
• Keywords: biometric, fair
• Abstract: The gait, as a kind of soft biometric characteristic, can reflect the distinct walking patterns of individuals at a distance, exhibiting a promising technique for unrestrained human identification. With largely excluding gait-unrelated cues hidden in RGB videos, the silhouette and skeleton, though visually compact, have acted as two of the most prevailing gait modalities for a long time. Recently, several attempts have been made to introduce more informative data forms like human parsing and optical flow images to capture gait characteristics, along with multi-branch architectures. However, due to the inconsistency within model designs and experiment settings, we argue that a comprehensive and fair comparative study among these popular gait modalities, involving the representational capacity and fusion strategy exploration, is still lacking. From the perspectives of fine vs. coarse-grained shape and whole vs. pixel-wise motion modeling, this work presents an in-depth investigation of three popular gait representations, i.e., silhouette, human parsing, and optical flow, with various fusion evaluations, and experimentally exposes their similarities and differences. Based on the obtained insights, we further develop a C$^2$Fusion strategy, consequently building our new framework MultiGait++. C$^2$Fusion preserves commonalities while highlighting differences to enrich the learning of gait features. To verify our findings and conclusions, extensive experiments on Gait3D, GREW, CCPG, and SUSTech1K are conducted. The code is available at this https URL.

Title: Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection

• Authors: Guangsheng Bao, Yanbin Zhao, Juncai He, Yue Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11506
• Pdf URL: https://arxiv.org/pdf/2412.11506
• Copy Paste: [[2412.11506]] Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection(https://arxiv.org/abs/2412.11506)
• Keywords: large language model
• Abstract: Advanced large language models (LLMs) can generate text almost indistinguishable from human-written text, highlighting the importance of LLM-generated text detection. However, current zero-shot techniques face challenges as white-box methods are restricted to use weaker open-source LLMs, and black-box methods are limited by partial observation from stronger proprietary LLMs. It seems impossible to enable white-box methods to use proprietary models because API-level access to the models neither provides full predictive distributions nor inner embeddings. To traverse the divide, we propose Glimpse, a probability distribution estimation approach, predicting the full distributions from partial observations. Despite the simplicity of Glimpse, we successfully extend white-box methods like Entropy, Rank, Log-Rank, and Fast-DetectGPT to latest proprietary models. Experiments show that Glimpse with Fast-DetectGPT and GPT-3.5 achieves an average AUROC of about 0.95 in five latest source models, improving the score by 51% relative to the remaining space of the open source baseline (Table 1). It demonstrates that the latest LLMs can effectively detect their own outputs, suggesting that advanced LLMs may be the best shield against themselves.

Title: IGR: Improving Diffusion Model for Garment Restoration from Person Image

• Authors: Le Shen, Rong Huang, Zhijie Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11513
• Pdf URL: https://arxiv.org/pdf/2412.11513
• Copy Paste: [[2412.11513]] IGR: Improving Diffusion Model for Garment Restoration from Person Image(https://arxiv.org/abs/2412.11513)
• Keywords: diffusion
• Abstract: Garment restoration, the inverse of virtual try-on task, focuses on restoring standard garment from a person image, requiring accurate capture of garment details. However, existing methods often fail to preserve the identity of the garment or rely on complex processes. To address these limitations, we propose an improved diffusion model for restoring authentic garments. Our approach employs two garment extractors to independently capture low-level features and high-level semantics from the person image. Leveraging a pretrained latent diffusion model, these features are integrated into the denoising process through garment fusion blocks, which combine self-attention and cross-attention layers to align the restored garment with the person image. Furthermore, a coarse-to-fine training strategy is introduced to enhance the fidelity and authenticity of the generated garments. Experimental results demonstrate that our model effectively preserves garment identity and generates high-quality restorations, even in challenging scenarios such as complex garments or those with occlusions.

Title: DART: An AIGT Detector using AMR of Rephrased Text

• Authors: Hyeonchu Park, Byungjun Kim, Bugeun Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11517
• Pdf URL: https://arxiv.org/pdf/2412.11517
• Copy Paste: [[2412.11517]] DART: An AIGT Detector using AMR of Rephrased Text(https://arxiv.org/abs/2412.11517)
• Keywords: large language model
• Abstract: As large language models (LLMs) generate more human-like texts, concerns about the side effects of AI-generated texts (AIGT) have grown. So, researchers have developed methods for detecting AIGT. However, two challenges remain. First, the performance on detecting black-box LLMs is low, because existing models have focused on syntactic features. Second, most AIGT detectors have been tested on a single-candidate setting, which assumes that we know the origin of an AIGT and may deviate from the real-world scenario. To resolve these challenges, we propose DART, which consists of four steps: rephrasing, semantic parsing, scoring, and multiclass classification. We conducted several experiments to test the performance of DART by following previous work. The experimental result shows that DART can discriminate multiple black-box LLMs without using syntactic features and knowing the origin of AIGT.

Title: LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model

• Authors: Xi Wang, Hongzhen Li, Heng Fang, Yichen Peng, Haoran Xie, Xi Yang, Chuntao Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11519
• Pdf URL: https://arxiv.org/pdf/2412.11519
• Copy Paste: [[2412.11519]] LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model(https://arxiv.org/abs/2412.11519)
• Keywords: diffusion
• Abstract: Image rendering from line drawings is vital in design and image generation technologies reduce costs, yet professional line drawings demand preserving complex details. Text prompts struggle with accuracy, and image translation struggles with consistency and fine-grained control. We present LineArt, a framework that transfers complex appearance onto detailed design drawings, facilitating design and artistic creation. It generates high-fidelity appearance while preserving structural accuracy by simulating hierarchical visual cognition and integrating human artistic experience to guide the diffusion process. LineArt overcomes the limitations of current methods in terms of difficulty in fine-grained control and style degradation in design drawings. It requires no precise 3D modeling, physical property specs, or network training, making it more convenient for design tasks. LineArt consists of two stages: a multi-frequency lines fusion module to supplement the input design drawing with detailed structural information and a two-part painting process for Base Layer Shaping and Surface Layer Coloring. We also present a new design drawing dataset ProLines for evaluation. The experiments show that LineArt performs better in accuracy, realism, and material precision compared to SOTAs.

Title: EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splatting

• Authors: Dong In Lee, Hyeongcheol Park, Jiyoung Seo, Eunbyung Park, Hyunje Park, Ha Dam Baek, Shin Sangheon, Sangmin kim, Sangpil Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11520
• Pdf URL: https://arxiv.org/pdf/2412.11520
• Copy Paste: [[2412.11520]] EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splatting(https://arxiv.org/abs/2412.11520)
• Keywords: diffusion
• Abstract: Recent advancements in 3D editing have highlighted the potential of text-driven methods in real-time, user-friendly AR/VR applications. However, current methods rely on 2D diffusion models without adequately considering multi-view information, resulting in multi-view inconsistency. While 3D Gaussian Splatting (3DGS) significantly improves rendering quality and speed, its 3D editing process encounters difficulties with inefficient optimization, as pre-trained Gaussians retain excessive source information, hindering optimization. To address these limitations, we propose \textbf{EditSplat}, a novel 3D editing framework that integrates Multi-view Fusion Guidance (MFG) and Attention-Guided Trimming (AGT). Our MFG ensures multi-view consistency by incorporating essential multi-view information into the diffusion process, leveraging classifier-free guidance from the text-to-image diffusion model and the geometric properties of 3DGS. Additionally, our AGT leverages the explicit representation of 3DGS to selectively prune and optimize 3D Gaussians, enhancing optimization efficiency and enabling precise, semantically rich local edits. Through extensive qualitative and quantitative evaluations, EditSplat achieves superior multi-view consistency and editing quality over existing methods, significantly enhancing overall efficiency.

Title: RoMeO: Robust Metric Visual Odometry

• Authors: Junda Cheng, Zhipeng Cai, Zhaoxing Zhang, Wei Yin, Matthias Muller, Michael Paulitsch, Xin Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11530
• Pdf URL: https://arxiv.org/pdf/2412.11530
• Copy Paste: [[2412.11530]] RoMeO: Robust Metric Visual Odometry(https://arxiv.org/abs/2412.11530)
• Keywords: robust
• Abstract: Visual odometry (VO) aims to estimate camera poses from visual inputs -- a fundamental building block for many applications such as VR/AR and robotics. This work focuses on monocular RGB VO where the input is a monocular RGB video without IMU or 3D sensors. Existing approaches lack robustness under this challenging scenario and fail to generalize to unseen data (especially outdoors); they also cannot recover metric-scale poses. We propose Robust Metric Visual Odometry (RoMeO), a novel method that resolves these issues leveraging priors from pre-trained depth models. RoMeO incorporates both monocular metric depth and multi-view stereo (MVS) models to recover metric-scale, simplify correspondence search, provide better initialization and regularize optimization. Effective strategies are proposed to inject noise during training and adaptively filter noisy depth priors, which ensure the robustness of RoMeO on in-the-wild data. As shown in Fig.1, RoMeO advances the state-of-the-art (SOTA) by a large margin across 6 diverse datasets covering both indoor and outdoor scenes. Compared to the current SOTA DPVO, RoMeO reduces the relative (align the trajectory scale with GT) and absolute trajectory errors both by >50%. The performance gain also transfers to the full SLAM pipeline (with global BA & loop closure). Code will be released upon acceptance.

Title: Near Large Far Small: Relative Distance Based Partition Learning for UAV-view Geo-Localization

• Authors: Quan Chen, Tingyu Wang, Rongfeng Lu, Bolun Zheng, Zhedong Zheng, Chenggang Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11535
• Pdf URL: https://arxiv.org/pdf/2412.11535
• Copy Paste: [[2412.11535]] Near Large Far Small: Relative Distance Based Partition Learning for UAV-view Geo-Localization(https://arxiv.org/abs/2412.11535)
• Keywords: robust
• Abstract: UAV-view Geo-Localization (UVGL) presents substantial challenges, primarily due to appearance differences between drone-view and satellite-view. Existing methods develop partition learning strategies aimed at mining more comprehensive information by constructing diverse part-level feature representations, which rely on consistent cross-view scales. However, variations of UAV flight state leads to the scale mismatch of cross-views, resulting in serious performance degradation of partition-based methods. To overcome this issue, we propose a partition learning framework based on relative distance, which alleviates the dependence on scale consistency while mining fine-grained features. Specifically, we propose a distance guided dynamic partition learning strategy (DGDPL), consisting of a square partition strategy and a dynamic-guided adjustment strategy. The former is utilized to extract fine-grained features and global features in a simple manner. The latter calculates the relative distance ratio between drone- and satellite-view to adjust the partition size, thereby aligning the semantic information between partition pairs. Furthermore, we propose a saliency-guided refinement strategy to refine part-level features, so as to further improve the retrieval accuracy. Extensive experiments show that our approach achieves superior geo-localization accuracy across various scale-inconsistent scenarios, and exhibits remarkable robustness against scale variations. The code will be released.

Title: Let your LLM generate a few tokens and you will reduce the need for retrieval

• Authors: Hervé Déjean
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11536
• Pdf URL: https://arxiv.org/pdf/2412.11536
• Copy Paste: [[2412.11536]] Let your LLM generate a few tokens and you will reduce the need for retrieval(https://arxiv.org/abs/2412.11536)
• Keywords: robust, large language model
• Abstract: In this paper, we investigate how efficiently large language models (LLM) can be trained to check whether an answer is already stored in their parametric memory. We distill an LLM-as-a-judge to compute the IK (I Know) score. We found that this method is particularly beneficial in the context of retrieval-assisted augmented generation (RAG), with a respectable accuracy of 80%. It enables a significant reduction (more than 50%) in the number of search and reranking steps required for certain data sets. We have also introduced the IK score, which serves as a useful tool for characterising datasets by facilitating the classification task. Interestingly, through the inclusion of response tokens as input, our results suggest that only about 20,000 training samples are required to achieve good performance. The central element of this work is the use of a teacher model - the LLM as a judge - to generate training data. We also assess the robustness of the IK classifier by evaluating it with various types of teachers, including both string-based methods and LLMs, with the latter providing better results.

Title: Towards a Speech Foundation Model for Singapore and Beyond

• Authors: Muhammad Huzaifah, Tianchi Liu, Hardik B. Sailor, Kye Min Tan, Tarun K. Vangani, Qiongqiong Wang, Jeremy H. M. Wong, Nancy F. Chen, Ai Ti Aw
• Subjects: cs.CL, cs.AI, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.11538
• Pdf URL: https://arxiv.org/pdf/2412.11538
• Copy Paste: [[2412.11538]] Towards a Speech Foundation Model for Singapore and Beyond(https://arxiv.org/abs/2412.11538)
• Keywords: large language model
• Abstract: This technical report describes the MERaLiON Speech Encoder, a foundation model designed to support a wide range of downstream speech applications. Developed as part of Singapore's National Multimodal Large Language Model Programme, the MERaLiON Speech Encoder is tailored to address the speech processing needs in Singapore and the surrounding Southeast Asian region. The model currently supports mainly English, including the variety spoken in Singapore. We are actively expanding our datasets to gradually cover other languages in subsequent releases. The MERaLiON Speech Encoder was pre-trained from scratch on 200K hours of unlabelled speech data using a self-supervised learning approach based on masked language modelling. We describe our training procedure and hyperparameter tuning experiments in detail below. Our evaluation demonstrates improvements to spontaneous and Singapore speech benchmarks for speech recognition, while remaining competitive to other state-of-the-art speech encoders across ten other speech tasks. We commit to releasing our model, supporting broader research endeavours, both in Singapore and beyond.

Title: Android App Feature Extraction: A review of approaches for malware and app similarity detection

• Authors: Simon Torka, Sahin Albayrak
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.11539
• Pdf URL: https://arxiv.org/pdf/2412.11539
• Copy Paste: [[2412.11539]] Android App Feature Extraction: A review of approaches for malware and app similarity detection(https://arxiv.org/abs/2412.11539)
• Keywords: extraction
• Abstract: This paper reviews work published between 2002 and 2022 in the fields of Android malware, clone, and similarity detection. It examines the data sources, tools, and features used in existing research and identifies the need for a comprehensive, cross-domain dataset to facilitate interdisciplinary collaboration and the exploitation of synergies between different research areas. Furthermore, it shows that many research papers do not publish the dataset or a description of how it was created, making it difficult to reproduce or compare the results. The paper highlights the necessity for a dataset that is accessible, well-documented, and suitable for a range of applications. Guidelines are provided for this purpose, along with a schematic method for creating the dataset.

Title: SP$^2$T: Sparse Proxy Attention for Dual-stream Point Transformer

• Authors: Jiaxu Wan, Hong Zhang, Ziqi He, Qishu Wang, Ding Yuan, Yifan Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11540
• Pdf URL: https://arxiv.org/pdf/2412.11540
• Copy Paste: [[2412.11540]] SP$^2$T: Sparse Proxy Attention for Dual-stream Point Transformer(https://arxiv.org/abs/2412.11540)
• Keywords: robust, extraction, transformer, segmentation
• Abstract: In 3D understanding, point transformers have yielded significant advances in broadening the receptive field. However, further enhancement of the receptive field is hindered by the constraints of grouping attention. The proxy-based model, as a hot topic in image and language feature extraction, uses global or local proxies to expand the model's receptive field. But global proxy-based methods fail to precisely determine proxy positions and are not suited for tasks like segmentation and detection in the point cloud, and exist local proxy-based methods for image face difficulties in global-local balance, proxy sampling in various point clouds, and parallel cross-attention computation for sparse association. In this paper, we present SP$^2$T, a local proxy-based dual stream point transformer, which promotes global receptive field while maintaining a balance between local and global information. To tackle robust 3D proxy sampling, we propose a spatial-wise proxy sampling with vertex-based point proxy associations, ensuring robust point-cloud sampling in many scales of point cloud. To resolve economical association computation, we introduce sparse proxy attention combined with table-based relative bias, which enables low-cost and precise interactions between proxy and point features. Comprehensive experiments across multiple datasets reveal that our model achieves SOTA performance in downstream tasks. The code has been released in this https URL .

Title: Error Diversity Matters: An Error-Resistant Ensemble Method for Unsupervised Dependency Parsing

• Authors: Behzad Shayegh, Hobie H.-B. Lee, Xiaodan Zhu, Jackie Chi Kit Cheung, Lili Mou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11543
• Pdf URL: https://arxiv.org/pdf/2412.11543
• Copy Paste: [[2412.11543]] Error Diversity Matters: An Error-Resistant Ensemble Method for Unsupervised Dependency Parsing(https://arxiv.org/abs/2412.11543)
• Keywords: robust
• Abstract: We address unsupervised dependency parsing by building an ensemble of diverse existing models through post hoc aggregation of their output dependency parse structures. We observe that these ensembles often suffer from low robustness against weak ensemble components due to error accumulation. To tackle this problem, we propose an efficient ensemble-selection approach that avoids error accumulation. Results demonstrate that our approach outperforms each individual model as well as previous ensemble techniques. Additionally, our experiments show that the proposed ensemble-selection method significantly enhances the performance and robustness of our ensemble, surpassing previously proposed strategies, which have not accounted for error diversity.

Title: MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models

• Authors: Weilun Feng, Haotong Qin, Chuanguang Yang, Zhulin An, Libo Huang, Boyu Diao, Fei Wang, Renshuai Tao, Yongjun Xu, Michele Magno
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11549
• Pdf URL: https://arxiv.org/pdf/2412.11549
• Copy Paste: [[2412.11549]] MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Models(https://arxiv.org/abs/2412.11549)
• Keywords: robust, diffusion
• Abstract: Diffusion models have received wide attention in generation tasks. However, the expensive computation cost prevents the application of diffusion models in resource-constrained scenarios. Quantization emerges as a practical solution that significantly saves storage and computation by reducing the bit-width of parameters. However, the existing quantization methods for diffusion models still cause severe degradation in performance, especially under extremely low bit-widths (2-4 bit). The primary decrease in performance comes from the significant discretization of activation values at low bit quantization. Too few activation candidates are unfriendly for outlier significant weight channel quantization, and the discretized features prevent stable learning over different time steps of the diffusion model. This paper presents MPQ-DM, a Mixed-Precision Quantization method for Diffusion Models. The proposed MPQ-DM mainly relies on two techniques:(1) To mitigate the quantization error caused by outlier severe weight channels, we propose an Outlier-Driven Mixed Quantization (OMQ) technique that uses $Kurtosis$ to quantify outlier salient channels and apply optimized intra-layer mixed-precision bit-width allocation to recover accuracy performance within target efficiency.(2) To robustly learn representations crossing time steps, we construct a Time-Smoothed Relation Distillation (TRD) scheme between the quantized diffusion model and its full-precision counterpart, transferring discrete and continuous latent to a unified relation space to reduce the representation inconsistency. Comprehensive experiments demonstrate that MPQ-DM achieves significant accuracy gains under extremely low bit-widths compared with SOTA quantization methods. MPQ-DM achieves a 58\% FID decrease under W2A4 setting compared with baseline, while all other methods even collapse.

Title: Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs

• Authors: Yuchen Fu, Zifeng Cheng, Zhiwei Jiang, Zhonghui Wang, Yafeng Yin, Zhengliang Li, Qing Gu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11556
• Pdf URL: https://arxiv.org/pdf/2412.11556
• Copy Paste: [[2412.11556]] Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMs(https://arxiv.org/abs/2412.11556)
• Keywords: large language model
• Abstract: Extracting sentence embeddings from large language models (LLMs) is a promising direction, as LLMs have demonstrated stronger semantic understanding capabilities. Previous studies typically focus on prompt engineering to elicit sentence embeddings from LLMs by prompting the model to encode sentence information into the embedding of the last token. However, LLMs are mostly decoder-only models with causal attention and the earlier tokens in the sentence cannot attend to the latter tokens, resulting in biased encoding of sentence information and cascading effects on the final decoded token. To this end, we propose a novel Token Prepending (TP) technique that prepends each layer's decoded sentence embedding to the beginning of the sentence in the next layer's input, allowing earlier tokens to attend to the complete sentence information under the causal attention mechanism. The proposed TP technique is a plug-and-play and training-free technique, which means it can be seamlessly integrated with various prompt-based sentence embedding methods and autoregressive LLMs. Extensive experiments on various Semantic Textual Similarity (STS) tasks and downstream classification tasks demonstrate that our proposed TP technique can significantly improve the performance of existing prompt-based sentence embedding methods across different LLMs, while incurring negligible additional inference cost.

Title: The Role of Natural Language Processing Tasks in Automatic Literary Character Network Construction

• Authors: Arthur Amalvy (LIA), Vincent Labatut (LIA), Richard Dufour (LS2N - équipe TALN)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11560
• Pdf URL: https://arxiv.org/pdf/2412.11560
• Copy Paste: [[2412.11560]] The Role of Natural Language Processing Tasks in Automatic Literary Character Network Construction(https://arxiv.org/abs/2412.11560)
• Keywords: extraction, large language model
• Abstract: The automatic extraction of character networks from literary texts is generally carried out using natural language processing (NLP) cascading pipelines. While this approach is widespread, no study exists on the impact of low-level NLP tasks on their performance. In this article, we conduct such a study on a literary dataset, focusing on the role of named entity recognition (NER) and coreference resolution when extracting co-occurrence networks. To highlight the impact of these tasks' performance, we start with gold-standard annotations, progressively add uniformly distributed errors, and observe their impact in terms of character network quality. We demonstrate that NER performance depends on the tested novel and strongly affects character detection. We also show that NER-detected mentions alone miss a lot of character co-occurrences, and that coreference resolution is needed to prevent this. Finally, we present comparison points with 2 methods based on large language models (LLMs), including a fully end-to-end one, and show that these models are outperformed by traditional NLP pipelines in terms of recall.

Title: RADARSAT Constellation Mission Compact Polarisation SAR Data for Burned Area Mapping with Deep Learning

• Authors: Yu Zhao, Yifang Ban
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11561
• Pdf URL: https://arxiv.org/pdf/2412.11561
• Copy Paste: [[2412.11561]] RADARSAT Constellation Mission Compact Polarisation SAR Data for Burned Area Mapping with Deep Learning(https://arxiv.org/abs/2412.11561)
• Keywords: transformer
• Abstract: Monitoring wildfires has become increasingly critical due to the sharp rise in wildfire incidents in recent years. Optical satellites like Sentinel-2 and Landsat are extensively utilized for mapping burned areas. However, the effectiveness of optical sensors is compromised by clouds and smoke, which obstruct the detection of burned areas. Thus, satellites equipped with Synthetic Aperture Radar (SAR), such as dual-polarization Sentinel-1 and quad-polarization RADARSAT-1/-2 C-band SAR, which can penetrate clouds and smoke, are investigated for mapping burned areas. However, there is limited research on using compact polarisation (compact-pol) C-band RADARSAT Constellation Mission (RCM) SAR data for this purpose. This study aims to investigate the capacity of compact polarisation RCM data for burned area mapping through deep learning. Compact-pol m-chi decomposition and Compact-pol Radar Vegetation Index (CpRVI) are derived from the RCM Multi-look Complex product. A deep-learning-based processing pipeline incorporating ConvNet-based and Transformer-based models is applied for burned area mapping, with three different input settings: using only log-ratio dual-polarization intensity images images, using only compact-pol decomposition plus CpRVI, and using all three data sources. The results demonstrate that compact-pol m-chi decomposition and CpRVI images significantly complement log-ratio images for burned area mapping. The best-performing Transformer-based model, UNETR, trained with log-ratio, m-chi decomposition, and CpRVI data, achieved an F1 Score of 0.718 and an IoU Score of 0.565, showing a notable improvement compared to the same model trained using only log-ratio images.

Title: OTA-Key: Over the Air Key Management for Flexible and Reliable IoT Device Provision

• Authors: Qian Zhang, Yi He, Yue Xiao, Xiaoli Zhang, Chunhua Song
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.11564
• Pdf URL: https://arxiv.org/pdf/2412.11564
• Copy Paste: [[2412.11564]] OTA-Key: Over the Air Key Management for Flexible and Reliable IoT Device Provision(https://arxiv.org/abs/2412.11564)
• Keywords: secure, security, attack
• Abstract: As the Internet of Things (IoT) industry advances, the imperative to secure IoT devices has become increasingly critical. Current practices in both industry and academia advocate for the enhancement of device security through key installation. However, it has been observed that, in practice, IoT vendors frequently assign shared keys to batches of devices. This practice can expose devices to risks, such as data theft by attackers or large-scale Distributed Denial of Service (DDoS) attacks. To address this issue, our intuition is to assign a unique key to each device. Unfortunately, this strategy proves to be highly complex within the IoT context, as existing keys are typically hardcoded into the firmware, necessitating the creation of bespoke firmware for each device. Furthermore, correct pairing of device keys with their respective devices is crucial. Errors in this pairing process would incur substantial human and temporal resources to rectify and require extensive communication between IoT vendors, device manufacturers, and cloud platforms, leading to significant communication overhead. To overcome these challenges, we propose the OTA-Key scheme. This approach fundamentally decouples device keys from the firmware features stored in flash memory, utilizing an intermediary server to allocate unique device keys in two distinct stages and update keys. We conducted a formal security verification of our scheme using ProVerif and assessed its performance through a series of evaluations. The results demonstrate that our scheme is secure and effectively manages the large-scale distribution and updating of unique device keys. Additionally, it achieves significantly lower update times and data transfer volumes compared to other schemes.

Title: DB-PAISA: Discovery-Based Privacy-Agile IoT Sensing+Actuation

• Authors: Isita Bagayatkar, Youngil Kim, Gene Tsudik
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.11572
• Pdf URL: https://arxiv.org/pdf/2412.11572
• Copy Paste: [[2412.11572]] DB-PAISA: Discovery-Based Privacy-Agile IoT Sensing+Actuation(https://arxiv.org/abs/2412.11572)
• Keywords: secure, security, privacy
• Abstract: Internet of Things (IoT) devices are becoming increasingly commonplace in numerous public and semi-private settings. Currently, most such devices lack mechanisms to facilitate their discovery by casual (nearby) users who are not owners or operators. However, these users are potentially being sensed, and/or actuated upon, by these devices, without their knowledge or consent. This naturally triggers privacy, security, and safety issues. To address this problem, some recent work explored device transparency in the IoT ecosystem. The intuitive approach is for each device to periodically and securely broadcast (announce) its presence and capabilities to all nearby users. While effective, when no new users are present, this push-based approach generates a substantial amount of unnecessary network traffic and needlessly interferes with normal device operation. In this work, we construct DB-PAISA which addresses these issues via a pull-based method, whereby devices reveal their presence and capabilities only upon explicit user request. Each device guarantees a secure timely response (even if fully compromised by malware) based on a small active Root-of-Trust (RoT). DB-PAISA requires no hardware modifications and is suitable for a range of current IoT devices. To demonstrate its feasibility and practicality, we built a fully functional and publicly available prototype. It is implemented atop a commodity MCU (NXP LCP55S69) and operates in tandem with a smartphone-based app. Using this prototype, we evaluate energy consumption and other performance factors.

Title: PyPotteryLens: An Open-Source Deep Learning Framework for Automated Digitisation of Archaeological Pottery Documentation

• Authors: Lorenzo Cardarelli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11574
• Pdf URL: https://arxiv.org/pdf/2412.11574
• Copy Paste: [[2412.11574]] PyPotteryLens: An Open-Source Deep Learning Framework for Automated Digitisation of Archaeological Pottery Documentation(https://arxiv.org/abs/2412.11574)
• Keywords: robust, segmentation
• Abstract: Archaeological pottery documentation and study represents a crucial but time-consuming aspect of archaeology. While recent years have seen advances in digital documentation methods, vast amounts of legacy data remain locked in traditional publications. This paper introduces PyPotteryLens, an open-source framework that leverages deep learning to automate the digitisation and processing of archaeological pottery drawings from published sources. The system combines state-of-the-art computer vision models (YOLO for instance segmentation and EfficientNetV2 for classification) with an intuitive user interface, making advanced digital methods accessible to archaeologists regardless of technical expertise. The framework achieves over 97\% precision and recall in pottery detection and classification tasks, while reducing processing time by up to 5x to 20x compared to manual methods. Testing across diverse archaeological contexts demonstrates robust generalisation capabilities. Also, the system's modular architecture facilitates extension to other archaeological materials, while its standardised output format ensures long-term preservation and reusability of digitised data as well as solid basis for training machine learning algorithms. The software, documentation, and examples are available on GitHub (this https URL).

Title: Aligning Visual and Semantic Interpretability through Visually Grounded Concept Bottleneck Models

• Authors: Patrick Knab, Katharina Prasse, Sascha Marton, Christian Bartelt, Margret Keuper
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11576
• Pdf URL: https://arxiv.org/pdf/2412.11576
• Copy Paste: [[2412.11576]] Aligning Visual and Semantic Interpretability through Visually Grounded Concept Bottleneck Models(https://arxiv.org/abs/2412.11576)
• Keywords: interpretability, large language model, segmentation
• Abstract: The performance of neural networks increases steadily, but our understanding of their decision-making lags behind. Concept Bottleneck Models (CBMs) address this issue by incorporating human-understandable concepts into the prediction process, thereby enhancing transparency and interpretability. Since existing approaches often rely on large language models (LLMs) to infer concepts, their results may contain inaccurate or incomplete mappings, especially in complex visual domains. We introduce visually Grounded Concept Bottleneck Models (GCBM), which derive concepts on the image level using segmentation and detection foundation models. Our method generates inherently interpretable concepts, which can be grounded in the input image using attribution methods, allowing interpretations to be traced back to the image plane. We show that GCBM concepts are meaningful interpretability vehicles, which aid our understanding of model embedding spaces. GCBMs allow users to control the granularity, number, and naming of concepts, providing flexibility and are easily adaptable to new datasets without pre-training or additional data needed. Prediction accuracy is within 0.3-6% of the linear probe and GCBMs perform especially well for fine-grained classification interpretability on CUB, due to their dataset specificity. Our code is available on this https URL.

Title: DVP-MVS: Synergize Depth-Edge and Visibility Prior for Multi-View Stereo

• Authors: Zhenlong Yuan, Jinguo Luo, Fei Shen, Zhaoxin Li, Cong Liu, Tianlu Mao, Zhaoqi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11578
• Pdf URL: https://arxiv.org/pdf/2412.11578
• Copy Paste: [[2412.11578]] DVP-MVS: Synergize Depth-Edge and Visibility Prior for Multi-View Stereo(https://arxiv.org/abs/2412.11578)
• Keywords: robust
• Abstract: Patch deformation-based methods have recently exhibited substantial effectiveness in multi-view stereo, due to the incorporation of deformable and expandable perception to reconstruct textureless areas. However, such approaches typically focus on exploring correlative reliable pixels to alleviate match ambiguity during patch deformation, but ignore the deformation instability caused by mistaken edge-skipping and visibility occlusion, leading to potential estimation deviation. To remedy the above issues, we propose DVP-MVS, which innovatively synergizes depth-edge aligned and cross-view prior for robust and visibility-aware patch deformation. Specifically, to avoid unexpected edge-skipping, we first utilize Depth Anything V2 followed by the Roberts operator to initialize coarse depth and edge maps respectively, both of which are further aligned through an erosion-dilation strategy to generate fine-grained homogeneous boundaries for guiding patch deformation. In addition, we reform view selection weights as visibility maps and restore visible areas by cross-view depth reprojection, then regard them as cross-view prior to facilitate visibility-aware patch deformation. Finally, we improve propagation and refinement with multi-view geometry consistency by introducing aggregated visible hemispherical normals based on view selection and local projection depth differences based on epipolar lines, respectively. Extensive evaluations on ETH3D and Tanks & Temples benchmarks demonstrate that our method can achieve state-of-the-art performance with excellent robustness and generalization.

Title: SweepEvGS: Event-Based 3D Gaussian Splatting for Macro and Micro Radiance Field Rendering from a Single Sweep

• Authors: Jingqian Wu, Shuo Zhu, Chutian Wang, Boxin Shi, Edmund Y. Lam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11579
• Pdf URL: https://arxiv.org/pdf/2412.11579
• Copy Paste: [[2412.11579]] SweepEvGS: Event-Based 3D Gaussian Splatting for Macro and Micro Radiance Field Rendering from a Single Sweep(https://arxiv.org/abs/2412.11579)
• Keywords: robust
• Abstract: Recent advancements in 3D Gaussian Splatting (3D-GS) have demonstrated the potential of using 3D Gaussian primitives for high-speed, high-fidelity, and cost-efficient novel view synthesis from continuously calibrated input views. However, conventional methods require high-frame-rate dense and high-quality sharp images, which are time-consuming and inefficient to capture, especially in dynamic environments. Event cameras, with their high temporal resolution and ability to capture asynchronous brightness changes, offer a promising alternative for more reliable scene reconstruction without motion blur. In this paper, we propose SweepEvGS, a novel hardware-integrated method that leverages event cameras for robust and accurate novel view synthesis across various imaging settings from a single sweep. SweepEvGS utilizes the initial static frame with dense event streams captured during a single camera sweep to effectively reconstruct detailed scene views. We also introduce different real-world hardware imaging systems for real-world data collection and evaluation for future research. We validate the robustness and efficiency of SweepEvGS through experiments in three different imaging settings: synthetic objects, real-world macro-level, and real-world micro-level view synthesis. Our results demonstrate that SweepEvGS surpasses existing methods in visual rendering quality, rendering speed, and computational efficiency, highlighting its potential for dynamic practical applications.

Title: StrandHead: Text to Strand-Disentangled 3D Head Avatars Using Hair Geometric Priors

• Authors: Xiaokun Sun, Zeyu Cai, Zhenyu Zhang, Ying Tai, Jian Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11586
• Pdf URL: https://arxiv.org/pdf/2412.11586
• Copy Paste: [[2412.11586]] StrandHead: Text to Strand-Disentangled 3D Head Avatars Using Hair Geometric Priors(https://arxiv.org/abs/2412.11586)
• Keywords: diffusion, generative
• Abstract: While haircut indicates distinct personality, existing avatar generation methods fail to model practical hair due to the general or entangled representation. We propose StrandHead, a novel text to 3D head avatar generation method capable of generating disentangled 3D hair with strand representation. Without using 3D data for supervision, we demonstrate that realistic hair strands can be generated from prompts by distilling 2D generative diffusion models. To this end, we propose a series of reliable priors on shape initialization, geometric primitives, and statistical haircut features, leading to a stable optimization and text-aligned performance. Extensive experiments show that StrandHead achieves the state-of-the-art reality and diversity of generated 3D head and hair. The generated 3D hair can also be easily implemented in the Unreal Engine for physical simulation and other applications. The code will be available at this https URL.

Title: VersaGen: Unleashing Versatile Visual Control for Text-to-Image Synthesis

• Authors: Zhipeng Chen, Lan Yang, Yonggang Qi, Honggang Zhang, Kaiyue Pang, Ke Li, Yi-Zhe Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11594
• Pdf URL: https://arxiv.org/pdf/2412.11594
• Copy Paste: [[2412.11594]] VersaGen: Unleashing Versatile Visual Control for Text-to-Image Synthesis(https://arxiv.org/abs/2412.11594)
• Keywords: diffusion, generative
• Abstract: Despite the rapid advancements in text-to-image (T2I) synthesis, enabling precise visual control remains a significant challenge. Existing works attempted to incorporate multi-facet controls (text and sketch), aiming to enhance the creative control over generated images. However, our pilot study reveals that the expressive power of humans far surpasses the capabilities of current methods. Users desire a more versatile approach that can accommodate their diverse creative intents, ranging from controlling individual subjects to manipulating the entire scene composition. We present VersaGen, a generative AI agent that enables versatile visual control in T2I synthesis. VersaGen admits four types of visual controls: i) single visual subject; ii) multiple visual subjects; iii) scene background; iv) any combination of the three above or merely no control at all. We train an adaptor upon a frozen T2I model to accommodate the visual information into the text-dominated diffusion process. We introduce three optimization strategies during the inference phase of VersaGen to improve generation results and enhance user experience. Comprehensive experiments on COCO and Sketchy validate the effectiveness and flexibility of VersaGen, as evidenced by both qualitative and quantitative results.

Title: MeshArt: Generating Articulated Meshes with Structure-guided Transformers

• Authors: Daoyi Gao, Yawar Siddiqui, Lei Li, Angela Dai
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.11596
• Pdf URL: https://arxiv.org/pdf/2412.11596
• Copy Paste: [[2412.11596]] MeshArt: Generating Articulated Meshes with Structure-guided Transformers(https://arxiv.org/abs/2412.11596)
• Keywords: transformer
• Abstract: Articulated 3D object generation is fundamental for creating realistic, functional, and interactable virtual assets which are not simply static. We introduce MeshArt, a hierarchical transformer-based approach to generate articulated 3D meshes with clean, compact geometry, reminiscent of human-crafted 3D models. We approach articulated mesh generation in a part-by-part fashion across two stages. First, we generate a high-level articulation-aware object structure; then, based on this structural information, we synthesize each part's mesh faces. Key to our approach is modeling both articulation structures and part meshes as sequences of quantized triangle embeddings, leading to a unified hierarchical framework with transformers for autoregressive generation. Object part structures are first generated as their bounding primitives and articulation modes; a second transformer, guided by these articulation structures, then generates each part's mesh triangles. To ensure coherency among generated parts, we introduce structure-guided conditioning that also incorporates local part mesh connectivity. MeshArt shows significant improvements over state of the art, with 57.1% improvement in structure coverage and a 209-point improvement in mesh generation FID.

Title: 3D$^2$-Actor: Learning Pose-Conditioned 3D-Aware Denoiser for Realistic Gaussian Avatar Modeling

• Authors: Zichen Tang, Hongyu Yang, Hanchen Zhang, Jiaxin Chen, Di Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11599
• Pdf URL: https://arxiv.org/pdf/2412.11599
• Copy Paste: [[2412.11599]] 3D$^2$-Actor: Learning Pose-Conditioned 3D-Aware Denoiser for Realistic Gaussian Avatar Modeling(https://arxiv.org/abs/2412.11599)
• Keywords: robust, diffusion
• Abstract: Advancements in neural implicit representations and differentiable rendering have markedly improved the ability to learn animatable 3D avatars from sparse multi-view RGB videos. However, current methods that map observation space to canonical space often face challenges in capturing pose-dependent details and generalizing to novel poses. While diffusion models have demonstrated remarkable zero-shot capabilities in 2D image generation, their potential for creating animatable 3D avatars from 2D inputs remains underexplored. In this work, we introduce 3D$^2$-Actor, a novel approach featuring a pose-conditioned 3D-aware human modeling pipeline that integrates iterative 2D denoising and 3D rectifying steps. The 2D denoiser, guided by pose cues, generates detailed multi-view images that provide the rich feature set necessary for high-fidelity 3D reconstruction and pose rendering. Complementing this, our Gaussian-based 3D rectifier renders images with enhanced 3D consistency through a two-stage projection strategy and a novel local coordinate representation. Additionally, we propose an innovative sampling strategy to ensure smooth temporal continuity across frames in video synthesis. Our method effectively addresses the limitations of traditional numerical solutions in handling ill-posed mappings, producing realistic and animatable 3D human avatars. Experimental results demonstrate that 3D$^2$-Actor excels in high-fidelity avatar modeling and robustly generalizes to novel poses. Code is available at: this https URL.

Title: SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models

• Authors: Jiale Cheng, Xiao Liu, Cunxiang Wang, Xiaotao Gu, Yida Lu, Dan Zhang, Yuxiao Dong, Jie Tang, Hongning Wang, Minlie Huang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11605
• Pdf URL: https://arxiv.org/pdf/2412.11605
• Copy Paste: [[2412.11605]] SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models(https://arxiv.org/abs/2412.11605)
• Keywords: large language model
• Abstract: Instruction-following is a fundamental capability of language models, requiring the model to recognize even the most subtle requirements in the instructions and accurately reflect them in its output. Such an ability is well-suited for and often optimized by preference learning. However, existing methods often directly sample multiple independent responses from the model when creating preference pairs. Such practice can introduce content variations irrelevant to whether the instruction is precisely followed (e.g., different expressions about the same semantic), interfering with the goal of teaching models to recognize the key differences that lead to improved instruction following. In light of this, we introduce SPaR, a self-play framework integrating tree-search self-refinement to yield valid and comparable preference pairs free from distractions. By playing against itself, an LLM employs a tree-search strategy to refine its previous responses with respect to the instruction while minimizing unnecessary variations. Our experiments show that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses GPT-4-Turbo on the IFEval benchmark without losing general capabilities. Furthermore, SPaR demonstrates promising scalability and transferability, greatly enhancing models like GLM-4-9B and LLaMA3-70B. We also identify how inference scaling in tree search would impact model performance. Our code and data are publicly available at this https URL.

Title: Towards Adversarial Robustness of Model-Level Mixture-of-Experts Architectures for Semantic Segmentation

• Authors: Svetlana Pavlitska, Enrico Eisen, J. Marius Zöllner
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11608
• Pdf URL: https://arxiv.org/pdf/2412.11608
• Copy Paste: [[2412.11608]] Towards Adversarial Robustness of Model-Level Mixture-of-Experts Architectures for Semantic Segmentation(https://arxiv.org/abs/2412.11608)
• Keywords: attack, robust, segmentation
• Abstract: Vulnerability to adversarial attacks is a well-known deficiency of deep neural networks. Larger networks are generally more robust, and ensembling is one method to increase adversarial robustness: each model's weaknesses are compensated by the strengths of others. While an ensemble uses a deterministic rule to combine model outputs, a mixture of experts (MoE) includes an additional learnable gating component that predicts weights for the outputs of the expert models, thus determining their contributions to the final prediction. MoEs have been shown to outperform ensembles on specific tasks, yet their susceptibility to adversarial attacks has not been studied yet. In this work, we evaluate the adversarial vulnerability of MoEs for semantic segmentation of urban and highway traffic scenes. We show that MoEs are, in most cases, more robust to per-instance and universal white-box adversarial attacks and can better withstand transfer attacks. Our code is available at \url{this https URL}.

Title: MT-LENS: An all-in-one Toolkit for Better Machine Translation Evaluation

• Authors: Javier García Gilabert, Carlos Escolano, Audrey Mash, Xixian Liao, Maite Melero
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11615
• Pdf URL: https://arxiv.org/pdf/2412.11615
• Copy Paste: [[2412.11615]] MT-LENS: An all-in-one Toolkit for Better Machine Translation Evaluation(https://arxiv.org/abs/2412.11615)
• Keywords: robust, large language model
• Abstract: We introduce MT-LENS, a framework designed to evaluate Machine Translation (MT) systems across a variety of tasks, including translation quality, gender bias detection, added toxicity, and robustness to misspellings. While several toolkits have become very popular for benchmarking the capabilities of Large Language Models (LLMs), existing evaluation tools often lack the ability to thoroughly assess the diverse aspects of MT performance. MT-LENS addresses these limitations by extending the capabilities of LM-eval-harness for MT, supporting state-of-the-art datasets and a wide range of evaluation metrics. It also offers a user-friendly platform to compare systems and analyze translations with interactive visualizations. MT-LENS aims to broaden access to evaluation strategies that go beyond traditional translation quality evaluation, enabling researchers and engineers to better understand the performance of a NMT model and also easily measure system's biases.

Title: EvoLlama: Enhancing LLMs' Understanding of Proteins via Multimodal Structure and Sequence Representations

• Authors: Nuowei Liu, Changzhi Sun, Tao Ji, Junfeng Tian, Jianxin Tang, Yuanbin Wu, Man Lan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11618
• Pdf URL: https://arxiv.org/pdf/2412.11618
• Copy Paste: [[2412.11618]] EvoLlama: Enhancing LLMs' Understanding of Proteins via Multimodal Structure and Sequence Representations(https://arxiv.org/abs/2412.11618)
• Keywords: large language model
• Abstract: Current Large Language Models (LLMs) for understanding proteins primarily treats amino acid sequences as a text modality. Meanwhile, Protein Language Models (PLMs), such as ESM-2, have learned massive sequential evolutionary knowledge from the universe of natural protein sequences. Furthermore, structure-based encoders like ProteinMPNN learn the structural information of proteins through Graph Neural Networks. However, whether the incorporation of protein encoders can enhance the protein understanding of LLMs has not been explored. To bridge this gap, we propose EvoLlama, a multimodal framework that connects a structure-based encoder, a sequence-based protein encoder and an LLM for protein understanding. EvoLlama consists of a ProteinMPNN structure encoder, an ESM-2 protein sequence encoder, a multimodal projector to align protein and text representations and a Llama-3 text decoder. To train EvoLlama, we fine-tune it on protein-oriented instructions and protein property prediction datasets verbalized via natural language instruction templates. Our experiments show that EvoLlama's protein understanding capabilities have been significantly enhanced, outperforming other fine-tuned protein-oriented LLMs in zero-shot settings by an average of 1%-8% and surpassing the state-of-the-art baseline with supervised fine-tuning by an average of 6%. On protein property prediction datasets, our approach achieves promising results that are competitive with state-of-the-art task-specific baselines. We will release our code in a future version.

Title: Combating Semantic Contamination in Learning with Label Noise

• Authors: Wenxiao Fan, Kan Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11620
• Pdf URL: https://arxiv.org/pdf/2412.11620
• Copy Paste: [[2412.11620]] Combating Semantic Contamination in Learning with Label Noise(https://arxiv.org/abs/2412.11620)
• Keywords: robust
• Abstract: Noisy labels can negatively impact the performance of deep neural networks. One common solution is label refurbishment, which involves reconstructing noisy labels through predictions and distributions. However, these methods may introduce problematic semantic associations, a phenomenon that we identify as Semantic Contamination. Through an analysis of Robust LR, a representative label refurbishment method, we found that utilizing the logits of views for refurbishment does not adequately balance the semantic information of individual classes. Conversely, using the logits of models fails to maintain consistent semantic relationships across models, which explains why label refurbishment methods frequently encounter issues related to Semantic Contamination. To address this issue, we propose a novel method called Collaborative Cross Learning, which utilizes semi-supervised learning on refurbished labels to extract appropriate semantic associations from embeddings across views and models. Experimental results show that our method outperforms existing approaches on both synthetic and real-world noisy datasets, effectively mitigating the impact of label noise and Semantic Contamination.

Title: VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting

• Authors: Muhammet Furkan Ilaslan, Ali Koksal, Kevin Qinhong Lin, Burak Satar, Mike Zheng Shou, Qianli Xu
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.11621
• Pdf URL: https://arxiv.org/pdf/2412.11621
• Copy Paste: [[2412.11621]] VG-TVP: Multimodal Procedural Planning via Visually Grounded Text-Video Prompting(https://arxiv.org/abs/2412.11621)
• Keywords: diffusion, large language model
• Abstract: Large Language Model (LLM)-based agents have shown promise in procedural tasks, but the potential of multimodal instructions augmented by texts and videos to assist users remains under-explored. To address this gap, we propose the Visually Grounded Text-Video Prompting (VG-TVP) method which is a novel LLM-empowered Multimodal Procedural Planning (MPP) framework. It generates cohesive text and video procedural plans given a specified high-level objective. The main challenges are achieving textual and visual informativeness, temporal coherence, and accuracy in procedural plans. VG-TVP leverages the zero-shot reasoning capability of LLMs, the video-to-text generation ability of the video captioning models, and the text-to-video generation ability of diffusion models. VG-TVP improves the interaction between modalities by proposing a novel Fusion of Captioning (FoC) method and using Text-to-Video Bridge (T2V-B) and Video-to-Text Bridge (V2T-B). They allow LLMs to guide the generation of visually-grounded text plans and textual-grounded video plans. To address the scarcity of datasets suitable for MPP, we have curated a new dataset called Daily-Life Task Procedural Plans (Daily-PP). We conduct comprehensive experiments and benchmarks to evaluate human preferences (regarding textual and visual informativeness, temporal coherence, and plan accuracy). Our VG-TVP method outperforms unimodal baselines on the Daily-PP dataset.

Title: Fool Me, Fool Me: User Attitudes Toward LLM Falsehoods

• Authors: Diana Bar-Or Nirman, Ariel Weizman, Amos Azaria
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11625
• Pdf URL: https://arxiv.org/pdf/2412.11625
• Copy Paste: [[2412.11625]] Fool Me, Fool Me: User Attitudes Toward LLM Falsehoods(https://arxiv.org/abs/2412.11625)
• Keywords: large language model
• Abstract: While Large Language Models (LLMs) have become central tools in various fields, they often provide inaccurate or false information. This study examines user preferences regarding falsehood responses from LLMs. Specifically, we evaluate preferences for LLM responses where false statements are explicitly marked versus unmarked responses and preferences for confident falsehoods compared to LLM disclaimers acknowledging a lack of knowledge. Additionally, we investigate how requiring users to assess the truthfulness of statements influences these preferences. Surprisingly, 61\% of users prefer unmarked falsehood responses over marked ones, and 69\% prefer confident falsehoods over LLMs admitting lack of knowledge. In all our experiments, a total of 300 users participated, contributing valuable data to our analysis and conclusions. When users are required to evaluate the truthfulness of statements, preferences for unmarked and falsehood responses decrease slightly but remain high. These findings suggest that user preferences, which influence LLM training via feedback mechanisms, may inadvertently encourage the generation of falsehoods. Future research should address the ethical and practical implications of aligning LLM behavior with such preferences.

Title: QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models

• Authors: Changhai Zhou, Yuhua Zhou, Shijie Han, Qian Qiao, Hongguang Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11629
• Pdf URL: https://arxiv.org/pdf/2412.11629
• Copy Paste: [[2412.11629]] QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models(https://arxiv.org/abs/2412.11629)
• Keywords: large language model
• Abstract: The rise of large language models (LLMs) has significantly advanced various natural language processing (NLP) tasks. However, the resource demands of these models pose substantial challenges. Structured pruning is an effective approach to reducing model size, but it often results in significant accuracy degradation, necessitating parameter updates to adapt. Unfortunately, such fine-tuning requires substantial memory, which limits its applicability. To address these challenges, we introduce quantization into the structured pruning framework to reduce memory consumption during both fine-tuning and inference. However, the combined errors from pruning and quantization increase the difficulty of fine-tuning, requiring a more refined quantization scheme. To this end, we propose QPruner, a novel framework that employs structured pruning to reduce model size, followed by a layer-wise mixed-precision quantization scheme. Quantization precisions are assigned to each layer based on their importance to the target task, and Bayesian optimization is employed to refine precision allocation strategies, ensuring a balance between model accuracy and memory efficiency. Extensive experiments on benchmark datasets demonstrate that QPruner significantly outperforms existing methods in memory savings while maintaining or improving model performance.

Title: A Mapper Algorithm with implicit intervals and its optimization

• Authors: Yuyang Tao, Shufei Ge
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.11631
• Pdf URL: https://arxiv.org/pdf/2412.11631
• Copy Paste: [[2412.11631]] A Mapper Algorithm with implicit intervals and its optimization(https://arxiv.org/abs/2412.11631)
• Keywords: robust
• Abstract: The Mapper algorithm is an essential tool for visualizing complex, high dimensional data in topology data analysis (TDA) and has been widely used in biomedical research. It outputs a combinatorial graph whose structure implies the shape of the data. However,the need for manual parameter tuning and fixed intervals, along with fixed overlapping ratios may impede the performance of the standard Mapper algorithm. Variants of the standard Mapper algorithms have been developed to address these limitations, yet most of them still require manual tuning of parameters. Additionally, many of these variants, including the standard version found in the literature, were built within a deterministic framework and overlooked the uncertainty inherent in the data. To relax these limitations, in this work, we introduce a novel framework that implicitly represents intervals through a hidden assignment matrix, enabling automatic parameter optimization via stochastic gradient descent. In this work, we develop a soft Mapper framework based on a Gaussian mixture model(GMM) for flexible and implicit interval construction. We further illustrate the robustness of the soft Mapper algorithm by introducing the Mapper graph mode as a point estimation for the output graph. Moreover, a stochastic gradient descent algorithm with a specific topological loss function is proposed for optimizing parameters in the model. Both simulation and application studies demonstrate its effectiveness in capturing the underlying topological structures. In addition, the application to an RNA expression dataset obtained from the Mount Sinai/JJ Peters VA Medical Center Brain Bank (MSBB) successfully identifies a distinct subgroup of Alzheimer's Disease.

Title: Predicting the Original Appearance of Damaged Historical Documents

• Authors: Zhenhua Yang, Dezhi Peng, Yongxin Shi, Yuyi Zhang, Chongyu Liu, Lianwen Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11634
• Pdf URL: https://arxiv.org/pdf/2412.11634
• Copy Paste: [[2412.11634]] Predicting the Original Appearance of Damaged Historical Documents(https://arxiv.org/abs/2412.11634)
• Keywords: diffusion
• Abstract: Historical documents encompass a wealth of cultural treasures but suffer from severe damages including character missing, paper damage, and ink erosion over time. However, existing document processing methods primarily focus on binarization, enhancement, etc., neglecting the repair of these damages. To this end, we present a new task, termed Historical Document Repair (HDR), which aims to predict the original appearance of damaged historical documents. To fill the gap in this field, we propose a large-scale dataset HDR28K and a diffusion-based network DiffHDR for historical document repair. Specifically, HDR28K contains 28,552 damaged-repaired image pairs with character-level annotations and multi-style degradations. Moreover, DiffHDR augments the vanilla diffusion framework with semantic and spatial information and a meticulously designed character perceptual loss for contextual and visual coherence. Experimental results demonstrate that the proposed DiffHDR trained using HDR28K significantly surpasses existing approaches and exhibits remarkable performance in handling real damaged documents. Notably, DiffHDR can also be extended to document editing and text block generation, showcasing its high flexibility and generalization capacity. We believe this study could pioneer a new direction of document processing and contribute to the inheritance of invaluable cultures and civilizations. The dataset and code is available at this https URL.

Title: IDProtector: An Adversarial Noise Encoder to Protect Against ID-Preserving Image Generation

• Authors: Yiren Song, Pei Yang, Hai Ci, Mike Zheng Shou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11638
• Pdf URL: https://arxiv.org/pdf/2412.11638
• Copy Paste: [[2412.11638]] IDProtector: An Adversarial Noise Encoder to Protect Against ID-Preserving Image Generation(https://arxiv.org/abs/2412.11638)
• Keywords: protect, robust, generative
• Abstract: Recently, zero-shot methods like InstantID have revolutionized identity-preserving generation. Unlike multi-image finetuning approaches such as DreamBooth, these zero-shot methods leverage powerful facial encoders to extract identity information from a single portrait photo, enabling efficient identity-preserving generation through a single inference pass. However, this convenience introduces new threats to the facial identity protection. This paper aims to safeguard portrait photos from unauthorized encoder-based customization. We introduce IDProtector, an adversarial noise encoder that applies imperceptible adversarial noise to portrait photos in a single forward pass. Our approach offers universal protection for portraits against multiple state-of-the-art encoder-based methods, including InstantID, IP-Adapter, and PhotoMaker, while ensuring robustness to common image transformations such as JPEG compression, resizing, and affine transformations. Experiments across diverse portrait datasets and generative models reveal that IDProtector generalizes effectively to unseen data and even closed-source proprietary models.

Title: SeSeMI: Secure Serverless Model Inference on Sensitive Data

• Authors: Guoyu Hu, Yuncheng Wu, Gang Chen, Tien Tuan Anh Dinh, Beng Chin Ooi
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.11640
• Pdf URL: https://arxiv.org/pdf/2412.11640
• Copy Paste: [[2412.11640]] SeSeMI: Secure Serverless Model Inference on Sensitive Data(https://arxiv.org/abs/2412.11640)
• Keywords: secure, protect
• Abstract: Model inference systems are essential for implementing end-to-end data analytics pipelines that deliver the benefits of machine learning models to users. Existing cloud-based model inference systems are costly, not easy to scale, and must be trusted in handling the models and user request data. Serverless computing presents a new opportunity, as it provides elasticity and fine-grained pricing. Our goal is to design a serverless model inference system that protects models and user request data from untrusted cloud providers. It offers high performance and low cost, while requiring no intrusive changes to the current serverless platforms. To realize our goal, we leverage trusted hardware. We identify and address three challenges in using trusted hardware for serverless model inference. These challenges arise from the high-level abstraction of serverless computing, the performance overhead of trusted hardware, and the characteristics of model inference workloads. We present SeSeMI, a secure, efficient, and cost-effective serverless model inference system. It adds three novel features non-intrusively to the existing serverless infrastructure and nothing this http URL first feature is a key service that establishes secure channels between the user and the serverless instances, which also provides access control to models and users' data. The second is an enclave runtime that allows one enclave to process multiple concurrent requests. The final feature is a model packer that allows multiple models to be executed by one serverless instance. We build SeSeMI on top of Apache OpenWhisk, and conduct extensive experiments with three popular machine learning models. The results show that SeSeMI achieves low latency and low cost at scale for realistic workloads.

Title: BA-BFL: Barycentric Aggregation for Bayesian Federated Learning

• Authors: Nour Jamoussi, Giuseppe Serra, Photios A. Stavrou, Marios Kountouris
• Subjects: cs.LG, cs.IT, cs.NI
• Abstract URL: https://arxiv.org/abs/2412.11646
• Pdf URL: https://arxiv.org/pdf/2412.11646
• Copy Paste: [[2412.11646]] BA-BFL: Barycentric Aggregation for Bayesian Federated Learning(https://arxiv.org/abs/2412.11646)
• Keywords: federate, fair
• Abstract: In this work, we study the problem of aggregation in the context of Bayesian Federated Learning (BFL). Using an information geometric perspective, we interpret the BFL aggregation step as finding the barycenter of the trained posteriors for a pre-specified divergence metric. We study the barycenter problem for the parametric family of $\alpha$-divergences and, focusing on the standard case of independent and Gaussian distributed parameters, we recover the closed-form solution of the reverse Kullback-Leibler barycenter and develop the analytical form of the squared Wasserstein-2 barycenter. Considering a non-IID setup, where clients possess heterogeneous data, we analyze the performance of the developed algorithms against state-of-the-art (SOTA) Bayesian aggregation methods in terms of accuracy, uncertainty quantification (UQ), model calibration (MC), and fairness. Finally, we extend our analysis to the framework of Hybrid Bayesian Deep Learning (HBDL), where we study how the number of Bayesian layers in the architecture impacts the considered performance metrics. Our experimental results show that the proposed methodology presents comparable performance with the SOTA while offering a geometric interpretation of the aggregation phase.

Title: Self-Adaptive Paraphrasing and Preference Learning for Improved Claim Verifiability

• Authors: Amelie Wührl, Roman Klinger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11653
• Pdf URL: https://arxiv.org/pdf/2412.11653
• Copy Paste: [[2412.11653]] Self-Adaptive Paraphrasing and Preference Learning for Improved Claim Verifiability(https://arxiv.org/abs/2412.11653)
• Keywords: extraction, generative
• Abstract: In fact-checking, structure and phrasing of claims critically influence a model's ability to predict verdicts accurately. Social media content in particular rarely serves as optimal input for verification systems, which necessitates pre-processing to extract the claim from noisy context before fact checking. Prior work suggests extracting a claim representation that humans find to be checkworthy and verifiable. This has two limitations: (1) the format may not be optimal for a fact-checking model, and (2), it requires annotated data to learn the extraction task from. We address both issues and propose a method to extract claims that is not reliant on labeled training data. Instead, our self-adaptive approach only requires a black-box fact checking model and a generative language model (LM). Given a tweet, we iteratively optimize the LM to generate a claim paraphrase that increases the performance of a fact checking model. By learning from preference pairs, we align the LM to the fact checker using direct preference optimization. We show that this novel setup extracts a claim paraphrase that is more verifiable than their original social media formulations, and is on par with competitive baselines. For refuted claims, our method consistently outperforms all baselines.

Title: CNNtention: Can CNNs do better with Attention?

• Authors: Julian Glattki, Nikhil Kapila, Tejas Rathi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11657
• Pdf URL: https://arxiv.org/pdf/2412.11657
• Copy Paste: [[2412.11657]] CNNtention: Can CNNs do better with Attention?(https://arxiv.org/abs/2412.11657)
• Keywords: extraction
• Abstract: Convolutional Neural Networks (CNNs) have been the standard for image classification tasks for a long time, but more recently attention-based mechanisms have gained traction. This project aims to compare traditional CNNs with attention-augmented CNNs across an image classification task. By evaluating and comparing their performance, accuracy and computational efficiency, the project will highlight benefits and trade-off of the localized feature extraction of traditional CNNs and the global context capture in attention-augmented CNNs. By doing this, we can reveal further insights into their respective strengths and weaknesses, guide the selection of models based on specific application needs and ultimately, enhance understanding of these architectures in the deep learning community. This was our final project for CS7643 Deep Learning course at Georgia Tech.

Title: Non-Convex Optimization in Federated Learning via Variance Reduction and Adaptive Learning

• Authors: Dipanwita Thakur, Antonella Guzzo, Giancarlo Fortino, Sajal K. Das
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11660
• Pdf URL: https://arxiv.org/pdf/2412.11660
• Copy Paste: [[2412.11660]] Non-Convex Optimization in Federated Learning via Variance Reduction and Adaptive Learning(https://arxiv.org/abs/2412.11660)
• Keywords: federate
• Abstract: This paper proposes a novel federated algorithm that leverages momentum-based variance reduction with adaptive learning to address non-convex settings across heterogeneous data. We intend to minimize communication and computation overhead, thereby fostering a sustainable federated learning system. We aim to overcome challenges related to gradient variance, which hinders the model's efficiency, and the slow convergence resulting from learning rate adjustments with heterogeneous data. The experimental results on the image classification tasks with heterogeneous data reveal the effectiveness of our suggested algorithms in non-convex settings with an improved communication complexity of $\mathcal{O}(\epsilon^{-1})$ to converge to an $\epsilon$-stationary point - compared to the existing communication complexity $\mathcal{O}(\epsilon^{-2})$ of most prior works. The proposed federated version maintains the trade-off between the convergence rate, number of communication rounds, and test accuracy while mitigating the client drift in heterogeneous settings. The experimental results demonstrate the efficiency of our algorithms in image classification tasks (MNIST, CIFAR-10) with heterogeneous data.

Title: C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness

• Authors: Yu Kang, Xianghui Sun, Liangyu Chen, Wei Zou
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11664
• Pdf URL: https://arxiv.org/pdf/2412.11664
• Copy Paste: [[2412.11664]] C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness(https://arxiv.org/abs/2412.11664)
• Keywords: interpretability, large language model
• Abstract: Generating Chain-of-Thought (CoT) before deriving the answer can effectively improve the reasoning capabilities of large language models (LLMs) and significantly improve the accuracy of the generated answer. However, in most cases, the length of the generated CoT is much longer than the desired final answer, which results in additional decoding costs. Furthermore, existing research has discovered that shortening the reasoning steps in CoT, even while preserving the key information, diminishes LLMs' abilities. These phenomena make it difficult to use LLMs and CoT in many real-world applications that only require the final answer and are sensitive to latency, such as search and recommendation. To reduce the costs of model decoding and shorten the length of the generated CoT, this paper presents $\textbf{C}$onditioned $\textbf{C}$ompressed $\textbf{C}$hain-of-$\textbf{T}$hought (C3oT), a CoT compression framework that involves a compressor to compress an original longer CoT into a shorter CoT while maintaining key information and interpretability, a conditioned training method to train LLMs with both longer CoT and shorter CoT simultaneously to learn the corresponding relationships between them, and a conditioned inference method to gain the reasoning ability learned from longer CoT by generating shorter CoT. We conduct experiments over four datasets from arithmetic and commonsense scenarios, showing that the proposed method is capable of compressing the length of generated CoT by up to more than 50% without compromising its effectiveness.

Title: $\texttt{DINO-Foresight}$: Looking into the Future with DINO

• Authors: Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, Nikos Komodakis
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11673
• Pdf URL: https://arxiv.org/pdf/2412.11673
• Copy Paste: [[2412.11673]] $\texttt{DINO-Foresight}$: Looking into the Future with DINO(https://arxiv.org/abs/2412.11673)
• Keywords: robust, transformer
• Abstract: Predicting future dynamics is crucial for applications like autonomous driving and robotics, where understanding the environment is key. Existing pixel-level methods are computationally expensive and often focus on irrelevant details. To address these challenges, we introduce $\texttt{DINO-Foresight}$, a novel framework that operates in the semantic feature space of pretrained Vision Foundation Models (VFMs). Our approach trains a masked feature transformer in a self-supervised manner to predict the evolution of VFM features over time. By forecasting these features, we can apply off-the-shelf, task-specific heads for various scene understanding tasks. In this framework, VFM features are treated as a latent space, to which different heads attach to perform specific tasks for future-frame analysis. Extensive experiments show that our framework outperforms existing methods, demonstrating its robustness and scalability. Additionally, we highlight how intermediate transformer representations in $\texttt{DINO-Foresight}$ improve downstream task performance, offering a promising path for the self-supervised enhancement of VFM features. We provide the implementation code at this https URL .

Title: UA-PDFL: A Personalized Approach for Decentralized Federated Learning

• Authors: Hangyu Zhu, Yuxiang Fan, Zhenping Xie
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11674
• Pdf URL: https://arxiv.org/pdf/2412.11674
• Copy Paste: [[2412.11674]] UA-PDFL: A Personalized Approach for Decentralized Federated Learning(https://arxiv.org/abs/2412.11674)
• Keywords: security, privacy, federate
• Abstract: Federated learning (FL) is a privacy preserving machine learning paradigm designed to collaboratively learn a global model without data leakage. Specifically, in a typical FL system, the central server solely functions as an coordinator to iteratively aggregate the collected local models trained by each client, potentially introducing single-point transmission bottleneck and security threats. To mitigate this issue, decentralized federated learning (DFL) has been proposed, where all participating clients engage in peer-to-peer communication without a central server. Nonetheless, DFL still suffers from training degradation as FL does due to the non-independent and identically distributed (non-IID) nature of client data. And incorporating personalization layers into DFL may be the most effective solutions to alleviate the side effects caused by non-IID data. Therefore, in this paper, we propose a novel unit representation aided personalized decentralized federated learning framework, named UA-PDFL, to deal with the non-IID challenge in DFL. By adaptively adjusting the level of personalization layers through the guidance of the unit representation, UA-PDFL is able to address the varying degrees of data skew. Based on this scheme, client-wise dropout and layer-wise personalization are proposed to further enhance the learning performance of DFL. Extensive experiments empirically prove the effectiveness of our proposed method.

Title: Multimodal LLM for Intelligent Transportation Systems

• Authors: Dexter Le, Aybars Yunusoglu, Karn Tiwari, Murat Isik, I. Can Dikmen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11683
• Pdf URL: https://arxiv.org/pdf/2412.11683
• Copy Paste: [[2412.11683]] Multimodal LLM for Intelligent Transportation Systems(https://arxiv.org/abs/2412.11683)
• Keywords: large language model
• Abstract: In the evolving landscape of transportation systems, integrating Large Language Models (LLMs) offers a promising frontier for advancing intelligent decision-making across various applications. This paper introduces a novel 3-dimensional framework that encapsulates the intersection of applications, machine learning methodologies, and hardware devices, particularly emphasizing the role of LLMs. Instead of using multiple machine learning algorithms, our framework uses a single, data-centric LLM architecture that can analyze time series, images, and videos. We explore how LLMs can enhance data interpretation and decision-making in transportation. We apply this LLM framework to different sensor datasets, including time-series data and visual data from sources like Oxford Radar RobotCar, D-Behavior (D-Set), nuScenes by Motional, and Comma2k19. The goal is to streamline data processing workflows, reduce the complexity of deploying multiple models, and make intelligent transportation systems more efficient and accurate. The study was conducted using state-of-the-art hardware, leveraging the computational power of AMD RTX 3060 GPUs and Intel i9-12900 processors. The experimental results demonstrate that our framework achieves an average accuracy of 91.33\% across these datasets, with the highest accuracy observed in time-series data (92.7\%), showcasing the model's proficiency in handling sequential information essential for tasks such as motion planning and predictive maintenance. Through our exploration, we demonstrate the versatility and efficacy of LLMs in handling multimodal data within the transportation sector, ultimately providing insights into their application in real-world scenarios. Our findings align with the broader conference themes, highlighting the transformative potential of LLMs in advancing transportation technologies.

Title: Ultra-High-Definition Dynamic Multi-Exposure Image Fusion via Infinite Pixel Learning

• Authors: Xingchi Chen, Zhuoran Zheng, Xuerui Li, Yuying Chen, Shu Wang, Wenqi Ren
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11685
• Pdf URL: https://arxiv.org/pdf/2412.11685
• Copy Paste: [[2412.11685]] Ultra-High-Definition Dynamic Multi-Exposure Image Fusion via Infinite Pixel Learning(https://arxiv.org/abs/2412.11685)
• Keywords: large language model
• Abstract: With the continuous improvement of device imaging resolution, the popularity of Ultra-High-Definition (UHD) images is increasing. Unfortunately, existing methods for fusing multi-exposure images in dynamic scenes are designed for low-resolution images, which makes them inefficient for generating high-quality UHD images on a resource-constrained device. To alleviate the limitations of extremely long-sequence inputs, inspired by the Large Language Model (LLM) for processing infinitely long texts, we propose a novel learning paradigm to achieve UHD multi-exposure dynamic scene image fusion on a single consumer-grade GPU, named Infinite Pixel Learning (IPL). The design of our approach comes from three key components: The first step is to slice the input sequences to relieve the pressure generated by the model processing the data stream; Second, we develop an attention cache technique, which is similar to KV cache for infinite data stream processing; Finally, we design a method for attention cache compression to alleviate the storage burden of the cache on the device. In addition, we provide a new UHD benchmark to evaluate the effectiveness of our method. Extensive experimental results show that our method maintains high-quality visual performance while fusing UHD dynamic multi-exposure images in real-time (>40fps) on a single consumer-grade GPU.

Title: Just a Simple Transformation is Enough for Data Protection in Vertical Federated Learning

• Authors: Andrei Semenov, Philip Zmushko, Alexander Pichugin, Aleksandr Beznosikov
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.11689
• Pdf URL: https://arxiv.org/pdf/2412.11689
• Copy Paste: [[2412.11689]] Just a Simple Transformation is Enough for Data Protection in Vertical Federated Learning(https://arxiv.org/abs/2412.11689)
• Keywords: privacy, protect, attack, federate
• Abstract: Vertical Federated Learning (VFL) aims to enable collaborative training of deep learning models while maintaining privacy protection. However, the VFL procedure still has components that are vulnerable to attacks by malicious parties. In our work, we consider feature reconstruction attacks, a common risk targeting input data compromise. We theoretically claim that feature reconstruction attacks cannot succeed without knowledge of the prior distribution on data. Consequently, we demonstrate that even simple model architecture transformations can significantly impact the protection of input data during VFL. Confirming these findings with experimental results, we show that MLP-based models are resistant to state-of-the-art feature reconstruction attacks.

Title: CiTrus: Squeezing Extra Performance out of Low-data Bio-signal Transfer Learning

• Authors: Eloy Geenjaar, Lie Lu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11695
• Pdf URL: https://arxiv.org/pdf/2412.11695
• Copy Paste: [[2412.11695]] CiTrus: Squeezing Extra Performance out of Low-data Bio-signal Transfer Learning(https://arxiv.org/abs/2412.11695)
• Keywords: transformer
• Abstract: Transfer learning for bio-signals has recently become an important technique to improve prediction performance on downstream tasks with small bio-signal datasets. Recent works have shown that pre-training a neural network model on a large dataset (e.g. EEG) with a self-supervised task, replacing the self-supervised head with a linear classification head, and fine-tuning the model on different downstream bio-signal datasets (e.g., EMG or ECG) can dramatically improve the performance on those datasets. In this paper, we propose a new convolution-transformer hybrid model architecture with masked auto-encoding for low-data bio-signal transfer learning, introduce a frequency-based masked auto-encoding task, employ a more comprehensive evaluation framework, and evaluate how much and when (multimodal) pre-training improves fine-tuning performance. We also introduce a dramatically more performant method of aligning a downstream dataset with a different temporal length and sampling rate to the original pre-training dataset. Our findings indicate that the convolution-only part of our hybrid model can achieve state-of-the-art performance on some low-data downstream tasks. The performance is often improved even further with our full model. In the case of transformer-based models we find that pre-training especially improves performance on downstream datasets, multimodal pre-training often increases those gains further, and our frequency-based pre-training performs the best on average for the lowest and highest data regimes.

Title: On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?

• Authors: Matteo Esposito, Francesco Palagiano, Valentina Lenarduzzi, Davide Taibi
• Subjects: cs.CR, cs.AI, cs.ET, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.11698
• Pdf URL: https://arxiv.org/pdf/2412.11698
• Copy Paste: [[2412.11698]] On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?(https://arxiv.org/abs/2412.11698)
• Keywords: secure, security, protect, generative, large language model
• Abstract: Context. The security of critical infrastructure has been a fundamental concern since the advent of computers, and this concern has only intensified in today's cyber warfare landscape. Protecting mission-critical systems (MCSs), including essential assets like healthcare, telecommunications, and military coordination, is vital for national security. These systems require prompt and comprehensive governance to ensure their resilience, yet recent events have shown that meeting these demands is increasingly challenging. Aim. Building on prior research that demonstrated the potential of GAI, particularly Large Language Models (LLMs), in improving risk analysis tasks, we aim to explore practitioners' perspectives, specifically developers and security personnel, on using generative AI (GAI) in the governance of IT MCSs seeking to provide insights and recommendations for various stakeholders, including researchers, practitioners, and policymakers. Method. We designed a survey to collect practical experiences, concerns, and expectations of practitioners who develop and implement security solutions in the context of MCSs. Analyzing this data will help identify key trends, challenges, and opportunities for introducing GAIs in this niche domain. Conclusions and Future Works. Our findings highlight that the safe use of LLMs in MCS governance requires interdisciplinary collaboration. Researchers should focus on designing regulation-oriented models and focus on accountability; practitioners emphasize data protection and transparency, while policymakers must establish a unified AI framework with global benchmarks to ensure ethical and secure LLMs-based MCS governance.

Title: CoinMath: Harnessing the Power of Coding Instruction for Math LLMs

• Authors: Chengwei Wei, Bin Wang, Jung-jae Kim, Guimei Liu, Nancy F. Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11699
• Pdf URL: https://arxiv.org/pdf/2412.11699
• Copy Paste: [[2412.11699]] CoinMath: Harnessing the Power of Coding Instruction for Math LLMs(https://arxiv.org/abs/2412.11699)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown strong performance in solving mathematical problems, with code-based solutions proving particularly effective. However, the best practice to leverage coding instruction data to enhance mathematical reasoning remains underexplored. This study investigates three key questions: (1) How do different coding styles of mathematical code-based rationales impact LLMs' learning performance? (2) Can general-domain coding instructions improve performance? (3) How does integrating textual rationales with code-based ones during training enhance mathematical reasoning abilities? Our findings reveal that code-based rationales with concise comments, descriptive naming, and hardcoded solutions are beneficial, while improvements from general-domain coding instructions and textual rationales are relatively minor. Based on these insights, we propose CoinMath, a learning strategy designed to enhance mathematical reasoning by diversifying the coding styles of code-based rationales. CoinMath generates a variety of code-based rationales incorporating concise comments, descriptive naming conventions, and hardcoded solutions. Experimental results demonstrate that CoinMath significantly outperforms its baseline model, MAmmoTH, one of the SOTA math LLMs.

Title: AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration

• Authors: Wenhao Sun, Rong-Cheng Tu, Jingyi Liao, Zhao Jin, Dacheng Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11706
• Pdf URL: https://arxiv.org/pdf/2412.11706
• Copy Paste: [[2412.11706]] AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restoration(https://arxiv.org/abs/2412.11706)
• Keywords: diffusion, transformer
• Abstract: Video Diffusion Transformers (DiTs) have demonstrated significant potential for generating high-fidelity videos but are computationally intensive. Existing acceleration methods include distillation, which requires costly retraining, and feature caching, which is highly sensitive to network architecture. Recent token reduction methods are training-free and architecture-agnostic, offering greater flexibility and wider applicability. However, they enforce the same sequence length across different components, constraining their acceleration potential. We observe that intra-sequence redundancy in video DiTs varies across features, blocks, and denoising timesteps. Building on this observation, we propose Asymmetric Reduction and Restoration (AsymRnR), a training-free approach to accelerate video DiTs. It offers a flexible and adaptive strategy that reduces the number of tokens based on their redundancy to enhance both acceleration and generation quality. We further propose matching cache to facilitate faster processing. Integrated into state-of-the-art video DiTs, AsymRnR achieves a superior speedup without compromising the quality.

Title: Re-Attentional Controllable Video Diffusion Editing

• Authors: Yuanzhi Wang, Yong Li, Mengyi Liu, Xiaoya Zhang, Xin Liu, Zhen Cui, Antoni B. Chan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11710
• Pdf URL: https://arxiv.org/pdf/2412.11710
• Copy Paste: [[2412.11710]] Re-Attentional Controllable Video Diffusion Editing(https://arxiv.org/abs/2412.11710)
• Keywords: diffusion
• Abstract: Editing videos with textual guidance has garnered popularity due to its streamlined process which mandates users to solely edit the text prompt corresponding to the source video. Recent studies have explored and exploited large-scale text-to-image diffusion models for text-guided video editing, resulting in remarkable video editing capabilities. However, they may still suffer from some limitations such as mislocated objects, incorrect number of objects. Therefore, the controllability of video editing remains a formidable challenge. In this paper, we aim to challenge the above limitations by proposing a Re-Attentional Controllable Video Diffusion Editing (ReAtCo) method. Specially, to align the spatial placement of the target objects with the edited text prompt in a training-free manner, we propose a Re-Attentional Diffusion (RAD) to refocus the cross-attention activation responses between the edited text prompt and the target video during the denoising stage, resulting in a spatially location-aligned and semantically high-fidelity manipulated video. In particular, to faithfully preserve the invariant region content with less border artifacts, we propose an Invariant Region-guided Joint Sampling (IRJS) strategy to mitigate the intrinsic sampling errors w.r.t the invariant regions at each denoising timestep and constrain the generated content to be harmonized with the invariant region content. Experimental results verify that ReAtCo consistently improves the controllability of video diffusion editing and achieves superior video editing performance.

Title: MiMoTable: A Multi-scale Spreadsheet Benchmark with Meta Operations for Table Reasoning

• Authors: Zheng Li, Yang Du, Mao Zheng, Mingyang Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11711
• Pdf URL: https://arxiv.org/pdf/2412.11711
• Copy Paste: [[2412.11711]] MiMoTable: A Multi-scale Spreadsheet Benchmark with Meta Operations for Table Reasoning(https://arxiv.org/abs/2412.11711)
• Keywords: large language model
• Abstract: Extensive research has been conducted to explore the capability of Large Language Models (LLMs) for table reasoning and has significantly improved the performance on existing benchmarks. However, tables and user questions in real-world applications are more complex and diverse, presenting an unignorable gap compared to the existing benchmarks. To fill the gap, we propose a \textbf{M}ult\textbf{i}-scale spreadsheet benchmark with \textbf{M}eta \textbf{o}perations for \textbf{Table} reasoning, named as MiMoTable. Specifically, MiMoTable incorporates two key features. First, the tables in MiMoTable are all spreadsheets used in real-world scenarios, which cover seven domains and contain different types. Second, we define a new criterion with six categories of meta operations for measuring the difficulty of each question in MiMoTable, simultaneously as a new perspective for measuring the difficulty of the existing benchmarks. Experimental results show that Claude-3.5-Sonnet achieves the best performance with 77.4\% accuracy, indicating that there is still significant room to improve for LLMs on MiMoTable. Furthermore, we grade the difficulty of existing benchmarks according to our new criteria. Experiments have shown that the performance of LLMs decreases as the difficulty of benchmarks increases, thereby proving the effectiveness of our proposed new criterion.

Title: Seeker: Towards Exception Safety Code Generation with Intermediate Language Agents Framework

• Authors: Xuanming Zhang, Yuxuan Chen, Yiming Zheng, Zhexin Zhang, Yuan Yuan, Minlie Huang
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.11713
• Pdf URL: https://arxiv.org/pdf/2412.11713
• Copy Paste: [[2412.11713]] Seeker: Towards Exception Safety Code Generation with Intermediate Language Agents Framework(https://arxiv.org/abs/2412.11713)
• Keywords: robust, large language model
• Abstract: In real world software development, improper or missing exception handling can severely impact the robustness and reliability of code. Exception handling mechanisms require developers to detect, capture, and manage exceptions according to high standards, but many developers struggle with these tasks, leading to fragile code. This problem is particularly evident in open-source projects and impacts the overall quality of the software ecosystem. To address this challenge, we explore the use of large language models (LLMs) to improve exception handling in code. Through extensive analysis, we identify three key issues: Insensitive Detection of Fragile Code, Inaccurate Capture of Exception Block, and Distorted Handling Solution. These problems are widespread across real world repositories, suggesting that robust exception handling practices are often overlooked or mishandled. In response, we propose Seeker, a multi-agent framework inspired by expert developer strategies for exception handling. Seeker uses agents: Scanner, Detector, Predator, Ranker, and Handler to assist LLMs in detecting, capturing, and resolving exceptions more effectively. Our work is the first systematic study on leveraging LLMs to enhance exception handling practices in real development scenarios, providing valuable insights for future improvements in code reliability.

Title: LLMs Can Simulate Standardized Patients via Agent Coevolution

• Authors: Zhuoyun Du, Lujie Zheng, Renjun Hu, Yuyang Xu, Xiawei Li, Ying Sun, Wei Chen, Jian Wu, Haolei Cai, Haohao Ying
• Subjects: cs.CL, cs.AI, cs.HC, cs.MA
• Abstract URL: https://arxiv.org/abs/2412.11716
• Pdf URL: https://arxiv.org/pdf/2412.11716
• Copy Paste: [[2412.11716]] LLMs Can Simulate Standardized Patients via Agent Coevolution(https://arxiv.org/abs/2412.11716)
• Keywords: large language model
• Abstract: Training medical personnel using standardized patients (SPs) remains a complex challenge, requiring extensive domain expertise and role-specific practice. Most research on Large Language Model (LLM)-based simulated patients focuses on improving data retrieval accuracy or adjusting prompts through human feedback. However, this focus has overlooked the critical need for patient agents to learn a standardized presentation pattern that transforms data into human-like patient responses through unsupervised simulations. To address this gap, we propose EvoPatient, a novel simulated patient framework in which a patient agent and doctor agents simulate the diagnostic process through multi-turn dialogues, simultaneously gathering experience to improve the quality of both questions and answers, ultimately enabling human doctor training. Extensive experiments on various cases demonstrate that, by providing only overall SP requirements, our framework improves over existing reasoning methods by more than 10% in requirement alignment and better human preference, while achieving an optimal balance of resource consumption after evolving over 200 cases for 10 hours, with excellent generalizability. The code will be available at this https URL.

Title: Transferable Adversarial Face Attack with Text Controlled Attribute

• Authors: Wenyun Li, Zheng Zhang, Xiangyuan Lan, Dongmei Jiang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11735
• Pdf URL: https://arxiv.org/pdf/2412.11735
• Copy Paste: [[2412.11735]] Transferable Adversarial Face Attack with Text Controlled Attribute(https://arxiv.org/abs/2412.11735)
• Keywords: attack, generative
• Abstract: Traditional adversarial attacks typically produce adversarial examples under norm-constrained conditions, whereas unrestricted adversarial examples are free-form with semantically meaningful perturbations. Current unrestricted adversarial impersonation attacks exhibit limited control over adversarial face attributes and often suffer from low transferability. In this paper, we propose a novel Text Controlled Attribute Attack (TCA$^2$) to generate photorealistic adversarial impersonation faces guided by natural language. Specifically, the category-level personal softmax vector is employed to precisely guide the impersonation attacks. Additionally, we propose both data and model augmentation strategies to achieve transferable attacks on unknown target models. Finally, a generative model, \textit{i.e}, Style-GAN, is utilized to synthesize impersonated faces with desired attributes. Extensive experiments on two high-resolution face recognition datasets validate that our TCA$^2$ method can generate natural text-guided adversarial impersonation faces with high transferability. We also evaluate our method on real-world face recognition systems, \textit{i.e}, Face++ and Aliyun, further demonstrating the practical potential of our approach.

Title: Personalized LLM for Generating Customized Responses to the Same Query from Different Users

• Authors: Hang Zeng, Chaoyue Niu, Fan Wu, Chengfei Lv, Guihai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11736
• Pdf URL: https://arxiv.org/pdf/2412.11736
• Copy Paste: [[2412.11736]] Personalized LLM for Generating Customized Responses to the Same Query from Different Users(https://arxiv.org/abs/2412.11736)
• Keywords: large language model
• Abstract: Existing work on large language model (LLM) personalization assigned different responding roles to LLM, but overlooked the diversity of questioners. In this work, we propose a new form of questioner-aware LLM personalization, generating different responses even for the same query from different questioners. We design a dual-tower model architecture with a cross-questioner general encoder and a questioner-specific encoder. We further apply contrastive learning with multi-view augmentation, pulling close the dialogue representations of the same questioner, while pulling apart those of different questioners. To mitigate the impact of question diversity on questioner-contrastive learning, we cluster the dialogues based on question similarity and restrict the scope of contrastive learning within each cluster. We also build a multi-questioner dataset from English and Chinese scripts and WeChat records, called MQDialog, containing 173 questioners and 12 responders. Extensive evaluation with different metrics shows a significant improvement in the quality of personalized response generation.

Title: Efficiently Achieving Secure Model Training and Secure Aggregation to Ensure Bidirectional Privacy-Preservation in Federated Learning

• Authors: Xue Yang, Depan Peng, Yan Feng, Xiaohu Tang, Weijun Fang, Jun Shao
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.11737
• Pdf URL: https://arxiv.org/pdf/2412.11737
• Copy Paste: [[2412.11737]] Efficiently Achieving Secure Model Training and Secure Aggregation to Ensure Bidirectional Privacy-Preservation in Federated Learning(https://arxiv.org/abs/2412.11737)
• Keywords: secure, privacy, defense, attack, federate
• Abstract: Bidirectional privacy-preservation federated learning is crucial as both local gradients and the global model may leak privacy. However, only a few works attempt to achieve it, and they often face challenges such as excessive communication and computational overheads, or significant degradation of model accuracy, which hinders their practical applications. In this paper, we design an efficient and high-accuracy bidirectional privacy-preserving scheme for federated learning to complete secure model training and secure aggregation. To efficiently achieve bidirectional privacy, we design an efficient and accuracy-lossless model perturbation method on the server side (called $\mathbf{MP\_Server}$) that can be combined with local differential privacy (LDP) to prevent clients from accessing the model, while ensuring that the local gradients obtained on the server side satisfy LDP. Furthermore, to ensure model accuracy, we customize a distributed differential privacy mechanism on the client side (called $\mathbf{DDP\_Client}$). When combined with $\mathbf{MP\_Server}$, it ensures LDP of the local gradients, while ensuring that the aggregated result matches the accuracy of central differential privacy (CDP). Extensive experiments demonstrate that our scheme significantly outperforms state-of-the-art bidirectional privacy-preservation baselines (SOTAs) in terms of computational cost, model accuracy, and defense ability against privacy attacks. Particularly, given target accuracy, the training time of SOTAs is approximately $200$ times, or even over $1000$ times, longer than that of our scheme. When the privacy budget is set relatively small, our scheme incurs less than $6\%$ accuracy loss compared to the privacy-ignoring method, while SOTAs suffer up to $20\%$ accuracy loss. Experimental results also show that the defense capability of our scheme outperforms than SOTAs.

Title: CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation

• Authors: Hongxuan Zhang, Yao Zhao, Jiaqi Zheng, Chenyi Zhuang, Jinjie Gu, Guihai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11741
• Pdf URL: https://arxiv.org/pdf/2412.11741
• Copy Paste: [[2412.11741]] CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation(https://arxiv.org/abs/2412.11741)
• Keywords: robust, large language model
• Abstract: The emergence of long-context text applications utilizing large language models (LLMs) has presented significant scalability challenges, particularly in memory footprint. The linear growth of the Key-Value (KV) cache responsible for storing attention keys and values to minimize redundant computations can lead to substantial increases in memory consumption, potentially causing models to fail to serve with limited memory resources. To address this issue, we propose a novel approach called Cache Sparse Representation (CSR), which converts the KV cache by transforming the dense Key-Value cache tensor into sparse indexes and weights, offering a more memory-efficient representation during LLM inference. Furthermore, we introduce NeuralDict, a novel neural network-based method for automatically generating the dictionary used in our sparse representation. Our extensive experiments demonstrate that CSR achieves performance comparable to state-of-the-art KV cache quantization algorithms while maintaining robust functionality in memory-constrained environments.

Title: Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection

• Authors: Arij Riabi, Virginie Mouilleron, Menel Mahamdi, Wissam Antoun, Djamé Seddah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11745
• Pdf URL: https://arxiv.org/pdf/2412.11745
• Copy Paste: [[2412.11745]] Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection(https://arxiv.org/abs/2412.11745)
• Keywords: privacy, protect, robust, fair
• Abstract: The proliferation of radical content on online platforms poses significant risks, including inciting violence and spreading extremist ideologies. Despite ongoing research, existing datasets and models often fail to address the complexities of multilingual and diverse data. To bridge this gap, we introduce a publicly available multilingual dataset annotated with radicalization levels, calls for action, and named entities in English, French, and Arabic. This dataset is pseudonymized to protect individual privacy while preserving contextual information. Beyond presenting our \href{this https URL}{freely available dataset}, we analyze the annotation process, highlighting biases and disagreements among annotators and their implications for model performance. Additionally, we use synthetic data to investigate the influence of socio-demographic traits on annotation patterns and model predictions. Our work offers a comprehensive examination of the challenges and opportunities in building robust datasets for radical content detection, emphasizing the importance of fairness and transparency in model development.

Title: Common Ground, Diverse Roots: The Difficulty of Classifying Common Examples in Spanish Varieties

• Authors: Javier A. Lopetegui, Arij Riabi, Djamé Seddah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11750
• Pdf URL: https://arxiv.org/pdf/2412.11750
• Copy Paste: [[2412.11750]] Common Ground, Diverse Roots: The Difficulty of Classifying Common Examples in Spanish Varieties(https://arxiv.org/abs/2412.11750)
• Keywords: robust, fair
• Abstract: Variations in languages across geographic regions or cultures are crucial to address to avoid biases in NLP systems designed for culturally sensitive tasks, such as hate speech detection or dialog with conversational agents. In languages such as Spanish, where varieties can significantly overlap, many examples can be valid across them, which we refer to as common examples. Ignoring these examples may cause misclassifications, reducing model accuracy and fairness. Therefore, accounting for these common examples is essential to improve the robustness and representativeness of NLP systems trained on such data. In this work, we address this problem in the context of Spanish varieties. We use training dynamics to automatically detect common examples or errors in existing Spanish datasets. We demonstrate the efficacy of using predicted label confidence for our Datamaps \cite{swayamdipta-etal-2020-dataset} implementation for the identification of hard-to-classify examples, especially common examples, enhancing model performance in variety identification tasks. Additionally, we introduce a Cuban Spanish Variety Identification dataset with common examples annotations developed to facilitate more accurate detection of Cuban and Caribbean Spanish varieties. To our knowledge, this is the first dataset focused on identifying the Cuban, or any other Caribbean, Spanish variety.

Title: Deformable Radial Kernel Splatting

• Authors: Yi-Hua Huang, Ming-Xian Lin, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, Xiaojuan Qi
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.11752
• Pdf URL: https://arxiv.org/pdf/2412.11752
• Copy Paste: [[2412.11752]] Deformable Radial Kernel Splatting(https://arxiv.org/abs/2412.11752)
• Keywords: robust
• Abstract: Recently, Gaussian splatting has emerged as a robust technique for representing 3D scenes, enabling real-time rasterization and high-fidelity rendering. However, Gaussians' inherent radial symmetry and smoothness constraints limit their ability to represent complex shapes, often requiring thousands of primitives to approximate detailed geometry. We introduce Deformable Radial Kernel (DRK), which extends Gaussian splatting into a more general and flexible framework. Through learnable radial bases with adjustable angles and scales, DRK efficiently models diverse shape primitives while enabling precise control over edge sharpness and boundary curvature. iven DRK's planar nature, we further develop accurate ray-primitive intersection computation for depth sorting and introduce efficient kernel culling strategies for improved rasterization efficiency. Extensive experiments demonstrate that DRK outperforms existing methods in both representation efficiency and rendering quality, achieving state-of-the-art performance while dramatically reducing primitive count.

Title: DriveGazen: Event-Based Driving Status Recognition using Conventional Camera

• Authors: Xiaoyin Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11753
• Pdf URL: https://arxiv.org/pdf/2412.11753
• Copy Paste: [[2412.11753]] DriveGazen: Event-Based Driving Status Recognition using Conventional Camera(https://arxiv.org/abs/2412.11753)
• Keywords: robust
• Abstract: We introduce a wearable driving status recognition device and our open-source dataset, along with a new real-time method robust to changes in lighting conditions for identifying driving status from eye observations of drivers. The core of our method is generating event frames from conventional intensity frames, and the other is a newly designed Attention Driving State Network (ADSN). Compared to event cameras, conventional cameras offer complete information and lower hardware costs, enabling captured frames to encode rich spatial information. However, these textures lack temporal information, posing challenges in effectively identifying driving status. DriveGazen addresses this issue from three perspectives. First, we utilize video frames to generate realistic synthetic dynamic vision sensor (DVS) events. Second, we adopt a spiking neural network to decode pertinent temporal information. Lastly, ADSN extracts crucial spatial cues from corresponding intensity frames and conveys spatial attention to convolutional spiking layers during both training and inference through a novel guide attention module to guide the feature learning and feature enhancement of the event frame. We specifically collected the Driving Status (DriveGaze) dataset to demonstrate the effectiveness of our approach. Additionally, we validate the superiority of the DriveGazen on the Single-eye Event-based Emotion (SEE) dataset. To the best of our knowledge, our method is the first to utilize guide attention spiking neural networks and eye-based event frames generated from conventional cameras for driving status recognition. Please refer to our project page for more details: this https URL.

Title: Generative Inbetweening through Frame-wise Conditions-Driven Video Generation

• Authors: Tianyi Zhu, Dongwei Ren, Qilong Wang, Xiaohe Wu, Wangmeng Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11755
• Pdf URL: https://arxiv.org/pdf/2412.11755
• Copy Paste: [[2412.11755]] Generative Inbetweening through Frame-wise Conditions-Driven Video Generation(https://arxiv.org/abs/2412.11755)
• Keywords: generative
• Abstract: Generative inbetweening aims to generate intermediate frame sequences by utilizing two key frames as input. Although remarkable progress has been made in video generation models, generative inbetweening still faces challenges in maintaining temporal stability due to the ambiguous interpolation path between two key frames. This issue becomes particularly severe when there is a large motion gap between input frames. In this paper, we propose a straightforward yet highly effective Frame-wise Conditions-driven Video Generation (FCVG) method that significantly enhances the temporal stability of interpolated video frames. Specifically, our FCVG provides an explicit condition for each frame, making it much easier to identify the interpolation path between two input frames and thus ensuring temporally stable production of visually plausible video frames. To achieve this, we suggest extracting matched lines from two input frames that can then be easily interpolated frame by frame, serving as frame-wise conditions seamlessly integrated into existing video generation models. In extensive evaluations covering diverse scenarios such as natural landscapes, complex human poses, camera movements and animations, existing methods often exhibit incoherent transitions across frames. In contrast, our FCVG demonstrates the capability to generate temporally stable videos using both linear and non-linear interpolation curves. Our project page and code are available at \url{this https URL}.

Title: QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMs

• Authors: Mohammad Aflah Khan, Neemesh Yadav, Sarah Masud, Md. Shad Akhtar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11763
• Pdf URL: https://arxiv.org/pdf/2412.11763
• Copy Paste: [[2412.11763]] QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMs(https://arxiv.org/abs/2412.11763)
• Keywords: large language model
• Abstract: The rise of large language models (LLMs) has created a need for advanced benchmarking systems beyond traditional setups. To this end, we introduce QUENCH, a novel text-based English Quizzing Benchmark manually curated and transcribed from YouTube quiz videos. QUENCH possesses masked entities and rationales for the LLMs to predict via generation. At the intersection of geographical context and common sense reasoning, QUENCH helps assess world knowledge and deduction capabilities of LLMs via a zero-shot, open-domain quizzing setup. We perform an extensive evaluation on 7 LLMs and 4 metrics, investigating the influence of model size, prompting style, geographical context, and gold-labeled rationale generation. The benchmarking concludes with an error analysis to which the LLMs are prone.

Title: IDEA-Bench: How Far are Generative Models from Professional Designing?

• Authors: Chen Liang, Lianghua Huang, Jingwu Fang, Huanzhang Dou, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Junge Zhang, Xin Zhao, Yu Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11767
• Pdf URL: https://arxiv.org/pdf/2412.11767
• Copy Paste: [[2412.11767]] IDEA-Bench: How Far are Generative Models from Professional Designing?(https://arxiv.org/abs/2412.11767)
• Keywords: extraction, generative, large language model
• Abstract: Real-world design tasks - such as picture book creation, film storyboard development using character sets, photo retouching, visual effects, and font transfer - are highly diverse and complex, requiring deep interpretation and extraction of various elements from instructions, descriptions, and reference images. The resulting images often implicitly capture key features from references or user inputs, making it challenging to develop models that can effectively address such varied tasks. While existing visual generative models can produce high-quality images based on prompts, they face significant limitations in professional design scenarios that involve varied forms and multiple inputs and outputs, even when enhanced with adapters like ControlNets and LoRAs. To address this, we introduce IDEA-Bench, a comprehensive benchmark encompassing 100 real-world design tasks, including rendering, visual effects, storyboarding, picture books, fonts, style-based, and identity-preserving generation, with 275 test cases to thoroughly evaluate a model's general-purpose generation capabilities. Notably, even the best-performing model only achieves 22.48 on IDEA-Bench, while the best general-purpose model only achieves 6.81. We provide a detailed analysis of these results, highlighting the inherent challenges and providing actionable directions for improvement. Additionally, we provide a subset of 18 representative tasks equipped with multimodal large language model (MLLM)-based auto-evaluation techniques to facilitate rapid model development and comparison. We releases the benchmark data, evaluation toolkits, and an online leaderboard at this https URL, aiming to drive the advancement of generative models toward more versatile and applicable intelligent design systems.

Title: No More Adam: Learning Rate Scaling at Initialization is All You Need

• Authors: Minghao Xu, Lichuan Xiang, Xu Cai, Hongkai Wen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11768
• Pdf URL: https://arxiv.org/pdf/2412.11768
• Copy Paste: [[2412.11768]] No More Adam: Learning Rate Scaling at Initialization is All You Need(https://arxiv.org/abs/2412.11768)
• Keywords: robust, diffusion, transformer, large language model
• Abstract: In this work, we question the necessity of adaptive gradient methods for training deep neural networks. SGD-SaI is a simple yet effective enhancement to stochastic gradient descent with momentum (SGDM). SGD-SaI performs learning rate Scaling at Initialization (SaI) to distinct parameter groups, guided by their respective gradient signal-to-noise ratios (g-SNR). By adjusting learning rates without relying on adaptive second-order momentum, SGD-SaI helps prevent training imbalances from the very first iteration and cuts the optimizer's memory usage by half compared to AdamW. Despite its simplicity and efficiency, SGD-SaI consistently matches or outperforms AdamW in training a variety of Transformer-based tasks, effectively overcoming a long-standing challenge of using SGD for training Transformers. SGD-SaI excels in ImageNet-1K classification with Vision Transformers(ViT) and GPT-2 pretraining for large language models (LLMs, transformer decoder-only), demonstrating robustness to hyperparameter variations and practicality for diverse applications. We further tested its robustness on tasks like LoRA fine-tuning for LLMs and diffusion models, where it consistently outperforms state-of-the-art optimizers. From a memory efficiency perspective, SGD-SaI achieves substantial memory savings for optimizer states, reducing memory usage by 5.93 GB for GPT-2 (1.5B parameters) and 25.15 GB for Llama2-7B compared to AdamW in full-precision training settings.

Title: Impact of Face Alignment on Face Image Quality

• Authors: Eren Onaran, Erdi Sarıtaş, Hazım Kemal Ekenel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11779
• Pdf URL: https://arxiv.org/pdf/2412.11779
• Copy Paste: [[2412.11779]] Impact of Face Alignment on Face Image Quality(https://arxiv.org/abs/2412.11779)
• Keywords: extraction
• Abstract: Face alignment is a crucial step in preparing face images for feature extraction in facial analysis tasks. For applications such as face recognition, facial expression recognition, and facial attribute classification, alignment is widely utilized during both training and inference to standardize the positions of key landmarks in the face. It is well known that the application and method of face alignment significantly affect the performance of facial analysis models. However, the impact of alignment on face image quality has not been thoroughly investigated. Current FIQA studies often assume alignment as a prerequisite but do not explicitly evaluate how alignment affects quality metrics, especially with the advent of modern deep learning-based detectors that integrate detection and landmark localization. To address this need, our study examines the impact of face alignment on face image quality scores. We conducted experiments on the LFW, IJB-B, and SCFace datasets, employing MTCNN and RetinaFace models for face detection and alignment. To evaluate face image quality, we utilized several assessment methods, including SER-FIQ, FaceQAN, DifFIQA, and SDD-FIQA. Our analysis included examining quality score distributions for the LFW and IJB-B datasets and analyzing average quality scores at varying distances in the SCFace dataset. Our findings reveal that face image quality assessment methods are sensitive to alignment. Moreover, this sensitivity increases under challenging real-life conditions, highlighting the importance of evaluating alignment's role in quality assessment.

Title: InterDyn: Controllable Interactive Dynamics with Video Diffusion Models

• Authors: Rick Akkerman, Haiwen Feng, Michael J. Black, Dimitrios Tzionas, Victoria Fernández Abrevaya
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11785
• Pdf URL: https://arxiv.org/pdf/2412.11785
• Copy Paste: [[2412.11785]] InterDyn: Controllable Interactive Dynamics with Video Diffusion Models(https://arxiv.org/abs/2412.11785)
• Keywords: diffusion, generative
• Abstract: Predicting the dynamics of interacting objects is essential for both humans and intelligent systems. However, existing approaches are limited to simplified, toy settings and lack generalizability to complex, real-world environments. Recent advances in generative models have enabled the prediction of state transitions based on interventions, but focus on generating a single future state which neglects the continuous motion and subsequent dynamics resulting from the interaction. To address this gap, we propose InterDyn, a novel framework that generates videos of interactive dynamics given an initial frame and a control signal encoding the motion of a driving object or actor. Our key insight is that large video foundation models can act as both neural renderers and implicit physics simulators by learning interactive dynamics from large-scale video data. To effectively harness this capability, we introduce an interactive control mechanism that conditions the video generation process on the motion of the driving entity. Qualitative results demonstrate that InterDyn generates plausible, temporally consistent videos of complex object interactions while generalizing to unseen objects. Quantitative evaluations show that InterDyn outperforms baselines that focus on static state transitions. This work highlights the potential of leveraging video generative models as implicit physics engines.

Title: ProsodyFM: Unsupervised Phrasing and Intonation Control for Intelligible Speech Synthesis

• Authors: Xiangheng He, Junjie Chen, Zixing Zhang, Björn W. Schuller
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.11795
• Pdf URL: https://arxiv.org/pdf/2412.11795
• Copy Paste: [[2412.11795]] ProsodyFM: Unsupervised Phrasing and Intonation Control for Intelligible Speech Synthesis(https://arxiv.org/abs/2412.11795)
• Keywords: robust
• Abstract: Prosody contains rich information beyond the literal meaning of words, which is crucial for the intelligibility of speech. Current models still fall short in phrasing and intonation; they not only miss or misplace breaks when synthesizing long sentences with complex structures but also produce unnatural intonation. We propose ProsodyFM, a prosody-aware text-to-speech synthesis (TTS) model with a flow-matching (FM) backbone that aims to enhance the phrasing and intonation aspects of prosody. ProsodyFM introduces two key components: a Phrase Break Encoder to capture initial phrase break locations, followed by a Duration Predictor for the flexible adjustment of break durations; and a Terminal Intonation Encoder which integrates a set of intonation shape tokens combined with a novel Pitch Processor for more robust modeling of human-perceived intonation change. ProsodyFM is trained with no explicit prosodic labels and yet can uncover a broad spectrum of break durations and intonation patterns. Experimental results demonstrate that ProsodyFM can effectively improve the phrasing and intonation aspects of prosody, thereby enhancing the overall intelligibility compared to four state-of-the-art (SOTA) models. Out-of-distribution experiments show that this prosody improvement can further bring ProsodyFM superior generalizability for unseen complex sentences and speakers. Our case study intuitively illustrates the powerful and fine-grained controllability of ProsodyFM over phrasing and intonation.

Title: UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models

• Authors: Boyang Xue, Fei Mi, Qi Zhu, Hongru Wang, Rui Wang, Sheng Wang, Erxin Yu, Xuming Hu, Kam-Fai Wong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11803
• Pdf URL: https://arxiv.org/pdf/2412.11803
• Copy Paste: [[2412.11803]] UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models(https://arxiv.org/abs/2412.11803)
• Keywords: large language model
• Abstract: Despite demonstrating impressive capabilities, Large Language Models (LLMs) still often struggle to accurately express the factual knowledge they possess, especially in cases where the LLMs' knowledge boundaries are ambiguous. To improve LLMs' factual expressions, we propose the UAlign framework, which leverages Uncertainty estimations to represent knowledge boundaries, and then explicitly incorporates these representations as input features into prompts for LLMs to Align with factual knowledge. First, we prepare the dataset on knowledge question-answering (QA) samples by calculating two uncertainty estimations, including confidence score and semantic entropy, to represent the knowledge boundaries for LLMs. Subsequently, using the prepared dataset, we train a reward model that incorporates uncertainty estimations and then employ the Proximal Policy Optimization (PPO) algorithm for factuality alignment on LLMs. Experimental results indicate that, by integrating uncertainty representations in LLM alignment, the proposed UAlign can significantly enhance the LLMs' capacities to confidently answer known questions and refuse unknown questions on both in-domain and out-of-domain tasks, showing reliability improvements and good generalizability over various prompt- and training-based baselines.

Title: PhysAug: A Physical-guided and Frequency-based Data Augmentation for Single-Domain Generalized Object Detection

• Authors: Xiaoran Xu, Jiangang Yang, Wenhui Shi, Siyuan Ding, Luqing Luo, Jian Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11807
• Pdf URL: https://arxiv.org/pdf/2412.11807
• Copy Paste: [[2412.11807]] PhysAug: A Physical-guided and Frequency-based Data Augmentation for Single-Domain Generalized Object Detection(https://arxiv.org/abs/2412.11807)
• Keywords: robust
• Abstract: Single-Domain Generalized Object Detection~(S-DGOD) aims to train on a single source domain for robust performance across a variety of unseen target domains by taking advantage of an object detector. Existing S-DGOD approaches often rely on data augmentation strategies, including a composition of visual transformations, to enhance the detector's generalization ability. However, the absence of real-world prior knowledge hinders data augmentation from contributing to the diversity of training data distributions. To address this issue, we propose PhysAug, a novel physical model-based non-ideal imaging condition data augmentation method, to enhance the adaptability of the S-DGOD tasks. Drawing upon the principles of atmospheric optics, we develop a universal perturbation model that serves as the foundation for our proposed PhysAug. Given that visual perturbations typically arise from the interaction of light with atmospheric particles, the image frequency spectrum is harnessed to simulate real-world variations during training. This approach fosters the detector to learn domain-invariant representations, thereby enhancing its ability to generalize across various settings. Without altering the network architecture or loss function, our approach significantly outperforms the state-of-the-art across various S-DGOD datasets. In particular, it achieves a substantial improvement of $7.3\%$ and $7.2\%$ over the baseline on DWD and Cityscape-C, highlighting its enhanced generalizability in real-world settings.

Title: CLDA-YOLO: Visual Contrastive Learning Based Domain Adaptive YOLO Detector

• Authors: Tianheng Qiu, Ka Lung Law, Guanghua Pan, Jufei Wang, Xin Gao, Xuan Huang, Hu Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11812
• Pdf URL: https://arxiv.org/pdf/2412.11812
• Copy Paste: [[2412.11812]] CLDA-YOLO: Visual Contrastive Learning Based Domain Adaptive YOLO Detector(https://arxiv.org/abs/2412.11812)
• Keywords: robust
• Abstract: Unsupervised domain adaptive (UDA) algorithms can markedly enhance the performance of object detectors under conditions of domain shifts, thereby reducing the necessity for extensive labeling and retraining. Current domain adaptive object detection algorithms primarily cater to two-stage detectors, which tend to offer minimal improvements when directly applied to single-stage detectors such as YOLO. Intending to benefit the YOLO detector from UDA, we build a comprehensive domain adaptive architecture using a teacher-student cooperative system for the YOLO detector. In this process, we propose uncertainty learning to cope with pseudo-labeling generated by the teacher model with extreme uncertainty and leverage dynamic data augmentation to asymptotically adapt the teacher-student system to the environment. To address the inability of single-stage object detectors to align at multiple stages, we utilize a unified visual contrastive learning paradigm that aligns instance at backbone and head respectively, which steadily improves the robustness of the detectors in cross-domain tasks. In summary, we present an unsupervised domain adaptive YOLO detector based on visual contrastive learning (CLDA-YOLO), which achieves highly competitive results across multiple domain adaptive datasets without any reduction in inference speed.

Title: EventSum: A Large-Scale Event-Centric Summarization Dataset for Chinese Multi-News Documents

• Authors: Mengna Zhu, Kaisheng Zeng, Mao Wang, Kaiming Xiao, Lei Hou, Hongbin Huang, Juanzi Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11814
• Pdf URL: https://arxiv.org/pdf/2412.11814
• Copy Paste: [[2412.11814]] EventSum: A Large-Scale Event-Centric Summarization Dataset for Chinese Multi-News Documents(https://arxiv.org/abs/2412.11814)
• Keywords: large language model
• Abstract: In real life, many dynamic events, such as major disasters and large-scale sports events, evolve continuously over time. Obtaining an overview of these events can help people quickly understand the situation and respond more effectively. This is challenging because the key information of the event is often scattered across multiple documents, involving complex event knowledge understanding and reasoning, which is under-explored in previous work. Therefore, we proposed the Event-Centric Multi-Document Summarization (ECS) task, which aims to generate concise and comprehensive summaries of a given event based on multiple related news documents. Based on this, we constructed the EventSum dataset, which was constructed using Baidu Baike entries and underwent extensive human annotation, to facilitate relevant research. It is the first large scale Chinese multi-document summarization dataset, containing 5,100 events and a total of 57,984 news documents, with an average of 11.4 input news documents and 13,471 characters per event. To ensure data quality and mitigate potential data leakage, we adopted a multi-stage annotation approach for manually labeling the test set. Given the complexity of event-related information, existing metrics struggle to comprehensively assess the quality of generated summaries. We designed specific metrics including Event Recall, Argument Recall, Causal Recall, and Temporal Recall along with corresponding calculation methods for evaluation. We conducted comprehensive experiments on EventSum to evaluate the performance of advanced long-context Large Language Models (LLMs) on this task. Our experimental results indicate that: 1) The event-centric multi-document summarization task remains challenging for existing long-context LLMs; 2) The recall metrics we designed are crucial for evaluating the comprehensiveness of the summary information.

Title: ColorFlow: Retrieval-Augmented Image Sequence Colorization

• Authors: Junhao Zhuang, Xuan Ju, Zhaoyang Zhang, Yong Liu, Shiyi Zhang, Chun Yuan, Ying Shan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11815
• Pdf URL: https://arxiv.org/pdf/2412.11815
• Copy Paste: [[2412.11815]] ColorFlow: Retrieval-Augmented Image Sequence Colorization(https://arxiv.org/abs/2412.11815)
• Keywords: robust, extraction, diffusion, generative
• Abstract: Automatic black-and-white image sequence colorization while preserving character and object identity (ID) is a complex task with significant market demand, such as in cartoon or comic series colorization. Despite advancements in visual colorization using large-scale generative models like diffusion models, challenges with controllability and identity consistency persist, making current solutions unsuitable for industrial this http URL address this, we propose ColorFlow, a three-stage diffusion-based framework tailored for image sequence colorization in industrial applications. Unlike existing methods that require per-ID finetuning or explicit ID embedding extraction, we propose a novel robust and generalizable Retrieval Augmented Colorization pipeline for colorizing images with relevant color references. Our pipeline also features a dual-branch design: one branch for color identity extraction and the other for colorization, leveraging the strengths of diffusion models. We utilize the self-attention mechanism in diffusion models for strong in-context learning and color identity matching. To evaluate our model, we introduce ColorFlow-Bench, a comprehensive benchmark for reference-based colorization. Results show that ColorFlow outperforms existing models across multiple metrics, setting a new standard in sequential image colorization and potentially benefiting the art industry. We release our codes and models on our project page: this https URL.

Title: Are You Doubtful? Oh, It Might Be Difficult Then! Exploring the Use of Model Uncertainty for Question Difficulty Estimation

• Authors: Leonidas Zotos, Hedderik van Rijn, Malvina Nissim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11831
• Pdf URL: https://arxiv.org/pdf/2412.11831
• Copy Paste: [[2412.11831]] Are You Doubtful? Oh, It Might Be Difficult Then! Exploring the Use of Model Uncertainty for Question Difficulty Estimation(https://arxiv.org/abs/2412.11831)
• Keywords: large language model
• Abstract: In an educational setting, an estimate of the difficulty of multiple-choice questions (MCQs), a commonly used strategy to assess learning progress, constitutes very useful information for both teachers and students. Since human assessment is costly from multiple points of view, automatic approaches to MCQ item difficulty estimation are investigated, yielding however mixed success until now. Our approach to this problem takes a different angle from previous work: asking various Large Language Models to tackle the questions included in two different MCQ datasets, we leverage model uncertainty to estimate item difficulty. By using both model uncertainty features as well as textual features in a Random Forest regressor, we show that uncertainty features contribute substantially to difficulty prediction, where difficulty is inversely proportional to the number of students who can correctly answer a question. In addition to showing the value of our approach, we also observe that our model achieves state-of-the-art results on the BEA publicly available dataset.

Title: Improved Models for Media Bias Detection and Subcategorization

• Authors: Tim Menzner, Jochen L. Leidner
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11835
• Pdf URL: https://arxiv.org/pdf/2412.11835
• Copy Paste: [[2412.11835]] Improved Models for Media Bias Detection and Subcategorization(https://arxiv.org/abs/2412.11835)
• Keywords: transformer
• Abstract: We present improved models for the granular detection and sub-classification news media bias in English news articles. We compare the performance of zero-shot versus fine-tuned large pre-trained neural transformer language models, explore how the level of detail of the classes affects performance on a novel taxonomy of 27 news bias-types, and demonstrate how using synthetically generated example data can be used to improve quality

Title: UnMA-CapSumT: Unified and Multi-Head Attention-driven Caption Summarization Transformer

• Authors: Dhruv Sharma, Chhavi Dhiman, Dinesh Kumar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11836
• Pdf URL: https://arxiv.org/pdf/2412.11836
• Copy Paste: [[2412.11836]] UnMA-CapSumT: Unified and Multi-Head Attention-driven Caption Summarization Transformer(https://arxiv.org/abs/2412.11836)
• Keywords: transformer
• Abstract: Image captioning is the generation of natural language descriptions of images which have increased immense popularity in the recent past. With this different deep-learning techniques are devised for the development of factual and stylized image captioning models. Previous models focused more on the generation of factual and stylized captions separately providing more than one caption for a single image. The descriptions generated from these suffer from out-of-vocabulary and repetition issues. To the best of our knowledge, no such work exists that provided a description that integrates different captioning methods to describe the contents of an image with factual and stylized (romantic and humorous) elements. To overcome these limitations, this paper presents a novel Unified Attention and Multi-Head Attention-driven Caption Summarization Transformer (UnMA-CapSumT) based Captioning Framework. It utilizes both factual captions and stylized captions generated by the Modified Adaptive Attention-based factual image captioning model (MAA-FIC) and Style Factored Bi-LSTM with attention (SF-Bi-ALSTM) driven stylized image captioning model respectively. SF-Bi-ALSTM-based stylized IC model generates two prominent styles of expression- {romance, and humor}. The proposed summarizer UnMHA-ST combines both factual and stylized descriptions of an input image to generate styled rich coherent summarized captions. The proposed UnMHA-ST transformer learns and summarizes different linguistic styles efficiently by incorporating proposed word embedding fastText with Attention Word Embedding (fTA-WE) and pointer-generator network with coverage mechanism concept to solve the out-of-vocabulary issues and repetition problem. Extensive experiments are conducted on Flickr8K and a subset of FlickrStyle10K with supporting ablation studies to prove the efficiency and efficacy of the proposed framework.

Title: A Benchmark and Robustness Study of In-Context-Learning with Large Language Models in Music Entity Detection

• Authors: Simon Hachmeier, Robert Jäschke
• Subjects: cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.11851
• Pdf URL: https://arxiv.org/pdf/2412.11851
• Copy Paste: [[2412.11851]] A Benchmark and Robustness Study of In-Context-Learning with Large Language Models in Music Entity Detection(https://arxiv.org/abs/2412.11851)
• Keywords: robust, large language model
• Abstract: Detecting music entities such as song titles or artist names is a useful application to help use cases like processing music search queries or analyzing music consumption on the web. Recent approaches incorporate smaller language models (SLMs) like BERT and achieve high results. However, further research indicates a high influence of entity exposure during pre-training on the performance of the models. With the advent of large language models (LLMs), these outperform SLMs in a variety of downstream tasks. However, researchers are still divided if this is applicable to tasks like entity detection in texts due to issues like hallucination. In this paper, we provide a novel dataset of user-generated metadata and conduct a benchmark and a robustness study using recent LLMs with in-context-learning (ICL). Our results indicate that LLMs in the ICL setting yield higher performance than SLMs. We further uncover the large impact of entity exposure on the best performing LLM in our study.

Title: GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training

• Authors: Renqiu Xia, Mingsheng Li, Hancheng Ye, Wenjie Wu, Hongbin Zhou, Jiakang Yuan, Tianshuo Peng, Xinyu Cai, Xiangchao Yan, Bin Wang, Conghui He, Botian Shi, Tao Chen, Junchi Yan, Bo Zhang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11863
• Pdf URL: https://arxiv.org/pdf/2412.11863
• Copy Paste: [[2412.11863]] GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training(https://arxiv.org/abs/2412.11863)
• Keywords: transformer, large language model
• Abstract: Despite their proficiency in general tasks, Multi-modal Large Language Models (MLLMs) struggle with automatic Geometry Problem Solving (GPS), which demands understanding diagrams, interpreting symbols, and performing complex reasoning. This limitation arises from their pre-training on natural images and texts, along with the lack of automated verification in the problem-solving process. Besides, current geometric specialists are limited by their task-specific designs, making them less effective for broader geometric problems. To this end, we present GeoX, a multi-modal large model focusing on geometric understanding and reasoning tasks. Given the significant differences between geometric diagram-symbol and natural image-text, we introduce unimodal pre-training to develop a diagram encoder and symbol decoder, enhancing the understanding of geometric images and corpora. Furthermore, we introduce geometry-language alignment, an effective pre-training paradigm that bridges the modality gap between unimodal geometric experts. We propose a Generator-And-Sampler Transformer (GS-Former) to generate discriminative queries and eliminate uninformative representations from unevenly distributed geometric signals. Finally, GeoX benefits from visual instruction tuning, empowering it to take geometric images and questions as input and generate verifiable solutions. Experiments show that GeoX outperforms both generalists and geometric specialists on publicly recognized benchmarks, such as GeoQA, UniGeo, Geometry3K, and PGPS9k.

Title: Event-based Motion Deblurring via Multi-Temporal Granularity Fusion

• Authors: Xiaopeng Lin, Hongwei Ren, Yulong Huang, Zunchang Liu, Yue Zhou, Haotian Fu, Biao Pan, Bojun Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11866
• Pdf URL: https://arxiv.org/pdf/2412.11866
• Copy Paste: [[2412.11866]] Event-based Motion Deblurring via Multi-Temporal Granularity Fusion(https://arxiv.org/abs/2412.11866)
• Keywords: diffusion
• Abstract: Conventional frame-based cameras inevitably produce blurry effects due to motion occurring during the exposure time. Event camera, a bio-inspired sensor offering continuous visual information could enhance the deblurring performance. Effectively utilizing the high-temporal-resolution event data is crucial for extracting precise motion information and enhancing deblurring performance. However, existing event-based image deblurring methods usually utilize voxel-based event representations, losing the fine-grained temporal details that are mathematically essential for fast motion deblurring. In this paper, we first introduce point cloud-based event representation into the image deblurring task and propose a Multi-Temporal Granularity Network (MTGNet). It combines the spatially dense but temporally coarse-grained voxel-based event representation and the temporally fine-grained but spatially sparse point cloud-based event. To seamlessly integrate such complementary representations, we design a Fine-grained Point Branch. An Aggregation and Mapping Module (AMM) is proposed to align the low-level point-based features with frame-based features and an Adaptive Feature Diffusion Module (AFDM) is designed to manage the resolution discrepancies between event data and image data by enriching the sparse point feature. Extensive subjective and objective evaluations demonstrate that our method outperforms current state-of-the-art approaches on both synthetic and real-world datasets.

Title: Transformers Use Causal World Models in Maze-Solving Tasks

• Authors: Alex F. Spies, William Edwards, Michael I. Ivanitskiy, Adrians Skapars, Tilman Räuker, Katsumi Inoue, Alessandra Russo, Murray Shanahan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11867
• Pdf URL: https://arxiv.org/pdf/2412.11867
• Copy Paste: [[2412.11867]] Transformers Use Causal World Models in Maze-Solving Tasks(https://arxiv.org/abs/2412.11867)
• Keywords: interpretability, transformer
• Abstract: Recent studies in interpretability have explored the inner workings of transformer models trained on tasks across various domains, often discovering that these networks naturally develop surprisingly structured representations. When such representations comprehensively reflect the task domain's structure, they are commonly referred to as ``World Models'' (WMs). In this work, we discover such WMs in transformers trained on maze tasks. In particular, by employing Sparse Autoencoders (SAEs) and analysing attention patterns, we examine the construction of WMs and demonstrate consistency between the circuit analysis and the SAE feature-based analysis. We intervene upon the isolated features to confirm their causal role and, in doing so, find asymmetries between certain types of interventions. Surprisingly, we find that models are able to reason with respect to a greater number of active features than they see during training, even if attempting to specify these in the input token sequence would lead the model to fail. Futhermore, we observe that varying positional encodings can alter how WMs are encoded in a model's residual stream. By analyzing the causal role of these WMs in a toy domain we hope to make progress toward an understanding of emergent structure in the representations acquired by Transformers, leading to the development of more interpretable and controllable AI systems.

Title: Using Instruction-Tuned Large Language Models to Identify Indicators of Vulnerability in Police Incident Narratives

• Authors: Sam Relins, Daniel Birks, Charlie Lloyd
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11878
• Pdf URL: https://arxiv.org/pdf/2412.11878
• Copy Paste: [[2412.11878]] Using Instruction-Tuned Large Language Models to Identify Indicators of Vulnerability in Police Incident Narratives(https://arxiv.org/abs/2412.11878)
• Keywords: protect, large language model
• Abstract: Objectives: Compare qualitative coding of instruction tuned large language models (IT-LLMs) against human coders in classifying the presence or absence of vulnerability in routinely collected unstructured text that describes police-public interactions. Evaluate potential bias in IT-LLM codings. Methods: Analyzing publicly available text narratives of police-public interactions recorded by Boston Police Department, we provide humans and IT-LLMs with qualitative labelling codebooks and compare labels generated by both, seeking to identify situations associated with (i) mental ill health; (ii) substance misuse; (iii) alcohol dependence; and (iv) homelessness. We explore multiple prompting strategies and model sizes, and the variability of labels generated by repeated prompts. Additionally, to explore model bias, we utilize counterfactual methods to assess the impact of two protected characteristics - race and gender - on IT-LLM classification. Results: Results demonstrate that IT-LLMs can effectively support human qualitative coding of police incident narratives. While there is some disagreement between LLM and human generated labels, IT-LLMs are highly effective at screening narratives where no vulnerabilities are present, potentially vastly reducing the requirement for human coding. Counterfactual analyses demonstrate that manipulations to both gender and race of individuals described in narratives have very limited effects on IT-LLM classifications beyond those expected by chance. Conclusions: IT-LLMs offer effective means to augment human qualitative coding in a way that requires much lower levels of resource to analyze large unstructured datasets. Moreover, they encourage specificity in qualitative coding, promote transparency, and provide the opportunity for more standardized, replicable approaches to analyzing large free-text police data sources.

Title: SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentation

• Authors: Yunxiang Fu, Meng Lou, Yizhou Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11890
• Pdf URL: https://arxiv.org/pdf/2412.11890
• Copy Paste: [[2412.11890]] SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentation(https://arxiv.org/abs/2412.11890)
• Keywords: extraction, segmentation
• Abstract: High-quality semantic segmentation relies on three key capabilities: global context modeling, local detail encoding, and multi-scale feature extraction. However, recent methods struggle to possess all these capabilities simultaneously. Hence, we aim to empower segmentation networks to simultaneously carry out efficient global context modeling, high-quality local detail encoding, and rich multi-scale feature representation for varying input resolutions. In this paper, we introduce SegMAN, a novel linear-time model comprising a hybrid feature encoder dubbed SegMAN Encoder, and a decoder based on state space models. Specifically, the SegMAN Encoder synergistically integrates sliding local attention with dynamic state space models, enabling highly efficient global context modeling while preserving fine-grained local details. Meanwhile, the MMSCopE module in our decoder enhances multi-scale context feature extraction and adaptively scales with the input resolution. We comprehensively evaluate SegMAN on three challenging datasets: ADE20K, Cityscapes, and COCO-Stuff. For instance, SegMAN-B achieves 52.6% mIoU on ADE20K, outperforming SegNeXt-L by 1.6% mIoU while reducing computational complexity by over 15% GFLOPs. On Cityscapes, SegMAN-B attains 83.8% mIoU, surpassing SegFormer-B3 by 2.1% mIoU with approximately half the GFLOPs. Similarly, SegMAN-B improves upon VWFormer-B3 by 1.6% mIoU with lower GFLOPs on the COCO-Stuff dataset. Our code is available at this https URL.

Title: Classification of Spontaneous and Scripted Speech for Multilingual Audio

• Authors: Shahar Elisha, Andrew McDowell, Mariano Beguerisse-Díaz, Emmanouil Benetos
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.11896
• Pdf URL: https://arxiv.org/pdf/2412.11896
• Copy Paste: [[2412.11896]] Classification of Spontaneous and Scripted Speech for Multilingual Audio(https://arxiv.org/abs/2412.11896)
• Keywords: transformer, segmentation
• Abstract: Distinguishing scripted from spontaneous speech is an essential tool for better understanding how speech styles influence speech processing research. It can also improve recommendation systems and discovery experiences for media users through better segmentation of large recorded speech catalogues. This paper addresses the challenge of building a classifier that generalises well across different formats and languages. We systematically evaluate models ranging from traditional, handcrafted acoustic and prosodic features to advanced audio transformers, utilising a large, multilingual proprietary podcast dataset for training and validation. We break down the performance of each model across 11 language groups to evaluate cross-lingual biases. Our experimental analysis extends to publicly available datasets to assess the models' generalisability to non-podcast domains. Our results indicate that transformer-based models consistently outperform traditional feature-based techniques, achieving state-of-the-art performance in distinguishing between scripted and spontaneous speech across various languages.

Title: PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension

• Authors: Kun Ouyang, Yuanxin Liu, Shicheng Li, Yi Liu, Hao Zhou, Fandong Meng, Jie Zhou, Xu Sun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11906
• Pdf URL: https://arxiv.org/pdf/2412.11906
• Copy Paste: [[2412.11906]] PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension(https://arxiv.org/abs/2412.11906)
• Keywords: large language model
• Abstract: Multimodal punchlines, which involve humor or sarcasm conveyed in image-caption pairs, are a popular way of communication on online multimedia platforms. With the rapid development of multimodal large language models (MLLMs), it is essential to assess their ability to effectively comprehend these punchlines. However, existing benchmarks on punchline comprehension suffer from three major limitations: 1) language shortcuts that allow models to solely rely on text, 2) lack of question diversity, and 3) narrow focus on a specific domain of multimodal content (e.g., cartoon). To address these limitations, we introduce a multimodal \textbf{Punch}line comprehension \textbf{Bench}mark, named \textbf{PunchBench}, which is tailored for accurate and comprehensive evaluation of punchline comprehension. To enhance the evaluation accuracy, we generate synonymous and antonymous captions by modifying original captions, which mitigates the impact of shortcuts in the captions. To provide a comprehensive evaluation, PunchBench incorporates diverse question formats and image-captions from various domains. On this basis, we conduct extensive evaluations and reveal a significant gap between state-of-the-art MLLMs and humans in punchline comprehension. To improve punchline comprehension, we propose Simple-to-Complex Chain-of-Question (SC-CoQ) strategy, enabling the models to incrementally address complicated questions by first mastering simple ones. SC-CoQ effectively enhances the performance of various MLLMs on PunchBench, surpassing in-context learning and chain-of-thought.

Title: Can Language Models Rival Mathematics Students? Evaluating Mathematical Reasoning through Textual Manipulation and Human Experiments

• Authors: Andrii Nikolaiev, Yiannos Stathopoulos, Simone Teufel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11908
• Pdf URL: https://arxiv.org/pdf/2412.11908
• Copy Paste: [[2412.11908]] Can Language Models Rival Mathematics Students? Evaluating Mathematical Reasoning through Textual Manipulation and Human Experiments(https://arxiv.org/abs/2412.11908)
• Keywords: large language model
• Abstract: In this paper we look at the ability of recent large language models (LLMs) at solving mathematical problems in combinatorics. We compare models LLaMA-2, LLaMA-3.1, GPT-4, and Mixtral against each other and against human pupils and undergraduates with prior experience in mathematical olympiads. To facilitate these comparisons we introduce the Combi-Puzzles dataset, which contains 125 problem variants based on 25 combinatorial reasoning problems. Each problem is presented in one of five distinct forms, created by systematically manipulating the problem statements through adversarial additions, numeric parameter changes, and linguistic obfuscation. Our variations preserve the mathematical core and are designed to measure the generalisability of LLM problem-solving abilities, while also increasing confidence that problems are submitted to LLMs in forms that have not been seen as training instances. We found that a model based on GPT-4 outperformed all other models in producing correct responses, and performed significantly better in the mathematical variation of the problems than humans. We also found that modifications to problem statements significantly impact the LLM's performance, while human performance remains unaffected.

Title: CharacterBench: Benchmarking Character Customization of Large Language Models

• Authors: Jinfeng Zhou, Yongkang Huang, Bosi Wen, Guanqun Bi, Yuxuan Chen, Pei Ke, Zhuang Chen, Xiyao Xiao, Libiao Peng, Kuntian Tang, Rongsheng Zhang, Le Zhang, Tangjie Lv, Zhipeng Hu, Hongning Wang, Minlie Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11912
• Pdf URL: https://arxiv.org/pdf/2412.11912
• Copy Paste: [[2412.11912]] CharacterBench: Benchmarking Character Customization of Large Language Models(https://arxiv.org/abs/2412.11912)
• Keywords: robust, generative, large language model
• Abstract: Character-based dialogue (aka role-playing) enables users to freely customize characters for interaction, which often relies on LLMs, raising the need to evaluate LLMs' character customization capability. However, existing benchmarks fail to ensure a robust evaluation as they often only involve a single character category or evaluate limited dimensions. Moreover, the sparsity of character features in responses makes feature-focused generative evaluation both ineffective and inefficient. To address these issues, we propose CharacterBench, the largest bilingual generative benchmark, with 22,859 human-annotated samples covering 3,956 characters from 25 detailed character categories. We define 11 dimensions of 6 aspects, classified as sparse and dense dimensions based on whether character features evaluated by specific dimensions manifest in each response. We enable effective and efficient evaluation by crafting tailored queries for each dimension to induce characters' responses related to specific dimensions. Further, we develop CharacterJudge model for cost-effective and stable evaluations. Experiments show its superiority over SOTA automatic judges (e.g., GPT-4) and our benchmark's potential to optimize LLMs' character customization. Our repository is at this https URL.

Title: Does VLM Classification Benefit from LLM Description Semantics?

• Authors: Pingchuan Ma, Lennart Rietdorf, Dmytro Kotovenko, Vincent Tao Hu, Björn Ommer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11917
• Pdf URL: https://arxiv.org/pdf/2412.11917
• Copy Paste: [[2412.11917]] Does VLM Classification Benefit from LLM Description Semantics?(https://arxiv.org/abs/2412.11917)
• Keywords: explainability, large language model
• Abstract: Accurately describing images via text is a foundation of explainable AI. Vision-Language Models (VLMs) like CLIP have recently addressed this by aligning images and texts in a shared embedding space, expressing semantic similarities between vision and language embeddings. VLM classification can be improved with descriptions generated by Large Language Models (LLMs). However, it is difficult to determine the contribution of actual description semantics, as the performance gain may also stem from a semantic-agnostic ensembling effect. Considering this, we ask how to distinguish the actual discriminative power of descriptions from performance boosts that potentially rely on an ensembling effect. To study this, we propose an alternative evaluation scenario that shows a characteristic behavior if the used descriptions have discriminative power. Furthermore, we propose a training-free method to select discriminative descriptions that work independently of classname ensembling effects. The training-free method works in the following way: A test image has a local CLIP label neighborhood, i.e., its top-$k$ label predictions. Then, w.r.t. to a small selection set, we extract descriptions that distinguish each class well in the local neighborhood. Using the selected descriptions, we demonstrate improved classification accuracy across seven datasets and provide in-depth analysis and insights into the explainability of description-based image classification by VLMs.

Title: RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation

• Authors: Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yongkang Wu, Zhonghua Li, Qi Ye, Zhicheng Dou
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.11919
• Pdf URL: https://arxiv.org/pdf/2412.11919
• Copy Paste: [[2412.11919]] RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation(https://arxiv.org/abs/2412.11919)
• Keywords: generative, large language model
• Abstract: Large language models (LLMs) exhibit remarkable generative capabilities but often suffer from hallucinations. Retrieval-augmented generation (RAG) offers an effective solution by incorporating external knowledge, but existing methods still face several limitations: additional deployment costs of separate retrievers, redundant input tokens from retrieved text chunks, and the lack of joint optimization of retrieval and generation. To address these issues, we propose \textbf{RetroLLM}, a unified framework that integrates retrieval and generation into a single, cohesive process, enabling LLMs to directly generate fine-grained evidence from the corpus with constrained decoding. Moreover, to mitigate false pruning in the process of constrained evidence generation, we introduce (1) hierarchical FM-Index constraints, which generate corpus-constrained clues to identify a subset of relevant documents before evidence generation, reducing irrelevant decoding space; and (2) a forward-looking constrained decoding strategy, which considers the relevance of future sequences to improve evidence accuracy. Extensive experiments on five open-domain QA datasets demonstrate RetroLLM's superior performance across both in-domain and out-of-domain tasks. The code is available at \url{this https URL}.

Title: PICLe: Pseudo-Annotations for In-Context Learning in Low-Resource Named Entity Detection

• Authors: Sepideh Mamooler, Syrielle Montariol, Alexander Mathis, Antoine Bosselut
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11923
• Pdf URL: https://arxiv.org/pdf/2412.11923
• Copy Paste: [[2412.11923]] PICLe: Pseudo-Annotations for In-Context Learning in Low-Resource Named Entity Detection(https://arxiv.org/abs/2412.11923)
• Keywords: large language model
• Abstract: In-context learning (ICL) enables Large Language Models (LLMs) to perform tasks using few demonstrations, facilitating task adaptation when labeled examples are hard to obtain. However, ICL is sensitive to the choice of demonstrations, and it remains unclear which demonstration attributes enable in-context generalization. In this work, we conduct a perturbation study of in-context demonstrations for low-resource Named Entity Detection (NED). Our surprising finding is that in-context demonstrations with partially correct annotated entity mentions can be as effective for task transfer as fully correct demonstrations. Based off our findings, we propose Pseudo-annotated In-Context Learning (PICLe), a framework for in-context learning with noisy, pseudo-annotated demonstrations. PICLe leverages LLMs to annotate many demonstrations in a zero-shot first pass. We then cluster these synthetic demonstrations, sample specific sets of in-context demonstrations from each cluster, and predict entity mentions using each set independently. Finally, we use self-verification to select the final set of entity mentions. We evaluate PICLe on five biomedical NED datasets and show that, with zero human annotation, PICLe outperforms ICL in low-resource settings where limited gold examples can be used as in-context demonstrations.

Title: A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges

• Authors: Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, Xuming Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11936
• Pdf URL: https://arxiv.org/pdf/2412.11936
• Copy Paste: [[2412.11936]] A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges(https://arxiv.org/abs/2412.11936)
• Keywords: large language model
• Abstract: Mathematical reasoning, a core aspect of human cognition, is vital across many domains, from educational problem-solving to scientific advancements. As artificial general intelligence (AGI) progresses, integrating large language models (LLMs) with mathematical reasoning tasks is becoming increasingly significant. This survey provides the first comprehensive analysis of mathematical reasoning in the era of multimodal large language models (MLLMs). We review over 200 studies published since 2021, and examine the state-of-the-art developments in Math-LLMs, with a focus on multimodal settings. We categorize the field into three dimensions: benchmarks, methodologies, and challenges. In particular, we explore multimodal mathematical reasoning pipeline, as well as the role of (M)LLMs and the associated methodologies. Finally, we identify five major challenges hindering the realization of AGI in this domain, offering insights into the future direction for enhancing multimodal reasoning capabilities. This survey serves as a critical resource for the research community in advancing the capabilities of LLMs to tackle complex multimodal reasoning tasks.

Title: Precise Length Control in Large Language Models

• Authors: Bradley Butcher, Michael O'Keefe, James Titchener
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11937
• Pdf URL: https://arxiv.org/pdf/2412.11937
• Copy Paste: [[2412.11937]] Precise Length Control in Large Language Models(https://arxiv.org/abs/2412.11937)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are increasingly used in production systems, powering applications such as chatbots, summarization, and question answering. Despite their success, controlling the length of their response remains a significant challenge, particularly for tasks requiring structured outputs or specific levels of detail. In this work, we propose a method to adapt pre-trained decoder-only LLMs for precise control of response length. Our approach incorporates a secondary length-difference positional encoding (LDPE) into the input embeddings, which counts down to a user-set response termination length. Fine-tuning with LDPE allows the model to learn to terminate responses coherently at the desired length, achieving mean token errors of less than 3 tokens. We also introduce Max New Tokens++, an extension that enables flexible upper-bound length control, rather than an exact target. Experimental results on tasks such as question answering and document summarization demonstrate that our method enables precise length control without compromising response quality.

Title: The Impact of Generalization Techniques on the Interplay Among Privacy, Utility, and Fairness in Image Classification

• Authors: Ahmad Hassanpour, Amir Zarei, Khawla Mallat, Anderson Santana de Oliveira, Bian Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11951
• Pdf URL: https://arxiv.org/pdf/2412.11951
• Copy Paste: [[2412.11951]] The Impact of Generalization Techniques on the Interplay Among Privacy, Utility, and Fairness in Image Classification(https://arxiv.org/abs/2412.11951)
• Keywords: privacy, attack, membership infer, fair
• Abstract: This study investigates the trade-offs between fairness, privacy, and utility in image classification using machine learning (ML). Recent research suggests that generalization techniques can improve the balance between privacy and utility. One focus of this work is sharpness-aware training (SAT) and its integration with differential privacy (DP-SAT) to further improve this balance. Additionally, we examine fairness in both private and non-private learning models trained on datasets with synthetic and real-world biases. We also measure the privacy risks involved in these scenarios by performing membership inference attacks (MIAs) and explore the consequences of eliminating high-privacy risk samples, termed outliers. Moreover, we introduce a new metric, named \emph{harmonic score}, which combines accuracy, privacy, and fairness into a single measure. Through empirical analysis using generalization techniques, we achieve an accuracy of 81.11\% under $(8, 10^{-5})$-DP on CIFAR-10, surpassing the 79.5\% reported by De et al. (2022). Moreover, our experiments show that memorization of training samples can begin before the overfitting point, and generalization techniques do not guarantee the prevention of this memorization. Our analysis of synthetic biases shows that generalization techniques can amplify model bias in both private and non-private models. Additionally, our results indicate that increased bias in training data leads to reduced accuracy, greater vulnerability to privacy attacks, and higher model bias. We validate these findings with the CelebA dataset, demonstrating that similar trends persist with real-world attribute imbalances. Finally, our experiments show that removing outlier data decreases accuracy and further amplifies model bias.

Title: Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning

• Authors: Yuti Liu, Shice Liu, Junyuan Gao, Pengtao Jiang, Hao Zhang, Jinwei Chen, Bo Li
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11952
• Pdf URL: https://arxiv.org/pdf/2412.11952
• Copy Paste: [[2412.11952]] Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning(https://arxiv.org/abs/2412.11952)
• Keywords: large language model
• Abstract: Image Aesthetic Assessment (IAA) is a vital and intricate task that entails analyzing and assessing an image's aesthetic values, and identifying its highlights and areas for improvement. Traditional methods of IAA often concentrate on a single aesthetic task and suffer from inadequate labeled datasets, thus impairing in-depth aesthetic comprehension. Despite efforts to overcome this challenge through the application of Multi-modal Large Language Models (MLLMs), such models remain underdeveloped for IAA purposes. To address this, we propose a comprehensive aesthetic MLLM capable of nuanced aesthetic insight. Central to our approach is an innovative multi-scale text-guided self-supervised learning technique. This technique features a multi-scale feature alignment module and capitalizes on a wealth of unlabeled data in a self-supervised manner to structurally and functionally enhance aesthetic ability. The empirical evidence indicates that accompanied with extensive instruct-tuning, our model sets new state-of-the-art benchmarks across multiple tasks, including aesthetic scoring, aesthetic commenting, and personalized image aesthetic assessment. Remarkably, it also demonstrates zero-shot learning capabilities in the emerging task of aesthetic suggesting. Furthermore, for personalized image aesthetic assessment, we harness the potential of in-context learning and showcase its inherent advantages.

Title: Inferring Functionality of Attention Heads from their Parameters

• Authors: Amit Elhelo, Mor Geva
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11965
• Pdf URL: https://arxiv.org/pdf/2412.11965
• Copy Paste: [[2412.11965]] Inferring Functionality of Attention Heads from their Parameters(https://arxiv.org/abs/2412.11965)
• Keywords: large language model
• Abstract: Attention heads are one of the building blocks of large language models (LLMs). Prior work on investigating their operation mostly focused on analyzing their behavior during inference for specific circuits or tasks. In this work, we seek a comprehensive mapping of the operations they implement in a model. We propose MAPS (Mapping Attention head ParameterS), an efficient framework that infers the functionality of attention heads from their parameters, without any model training or inference. We showcase the utility of MAPS for answering two types of questions: (a) given a predefined operation, mapping how strongly heads across the model implement it, and (b) given an attention head, inferring its salient functionality. Evaluating MAPS on 20 operations across 6 popular LLMs shows its estimations correlate with the head's outputs during inference and are causally linked to the model's predictions. Moreover, its mappings reveal attention heads of certain operations that were overlooked in previous studies, and valuable insights on function universality and architecture biases in LLMs. Next, we present an automatic pipeline and analysis that leverage MAPS to characterize the salient operations of a given head. Our pipeline produces plausible operation descriptions for most heads, as assessed by human judgment, while revealing diverse operations.

Title: DARWIN 1.5: Large Language Models as Materials Science Adapted Learners

• Authors: Tong Xie, Yuwei Wan, Yixuan Liu, Yuchen Zeng, Wenjie Zhang, Chunyu Kit, Dongzhan Zhou, Bram Hoex
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11970
• Pdf URL: https://arxiv.org/pdf/2412.11970
• Copy Paste: [[2412.11970]] DARWIN 1.5: Large Language Models as Materials Science Adapted Learners(https://arxiv.org/abs/2412.11970)
• Keywords: large language model
• Abstract: Materials discovery and design aim to find components and structures with desirable properties over highly complex and diverse search spaces. Traditional solutions, such as high-throughput simulations and machine learning (ML), often rely on complex descriptors, which hinder generalizability and transferability across tasks. Moreover, these descriptors may deviate from experimental data due to inevitable defects and purity issues in the real world, which may reduce their effectiveness in practical applications. To address these challenges, we propose Darwin 1.5, an open-source large language model (LLM) tailored for materials science. By leveraging natural language as input, Darwin eliminates the need for task-specific descriptors and enables a flexible, unified approach to material property prediction and discovery. We employ a two-stage training strategy combining question-answering (QA) fine-tuning with multi-task learning (MTL) to inject domain-specific knowledge in various modalities and facilitate cross-task knowledge transfer. Through our strategic approach, we achieved a significant enhancement in the prediction accuracy of LLMs, with a maximum improvement of 60\% compared to LLaMA-7B base models. It further outperforms traditional machine learning models on various tasks in material science, showcasing the potential of LLMs to provide a more versatile and scalable foundation model for materials discovery and design.

Title: Controllable Shadow Generation with Single-Step Diffusion Models from Synthetic Data

• Authors: Onur Tasar, Clément Chadebec, Benjamin Aubin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11972
• Pdf URL: https://arxiv.org/pdf/2412.11972
• Copy Paste: [[2412.11972]] Controllable Shadow Generation with Single-Step Diffusion Models from Synthetic Data(https://arxiv.org/abs/2412.11972)
• Keywords: diffusion
• Abstract: Realistic shadow generation is a critical component for high-quality image compositing and visual effects, yet existing methods suffer from certain limitations: Physics-based approaches require a 3D scene geometry, which is often unavailable, while learning-based techniques struggle with control and visual artifacts. We introduce a novel method for fast, controllable, and background-free shadow generation for 2D object images. We create a large synthetic dataset using a 3D rendering engine to train a diffusion model for controllable shadow generation, generating shadow maps for diverse light source parameters. Through extensive ablation studies, we find that rectified flow objective achieves high-quality results with just a single sampling step enabling real-time applications. Furthermore, our experiments demonstrate that the model generalizes well to real-world images. To facilitate further research in evaluating quality and controllability in shadow generation, we release a new public benchmark containing a diverse set of object images and shadow maps in various settings. The project page is available at this https URL

Title: Industrial-scale Prediction of Cement Clinker Phases using Machine Learning

• Authors: Sheikh Junaid Fayaz, Nestor Montiel-Bohorquez, Shashank Bishnoi, Matteo Romano, Manuele Gatti, N. M. Anoop Krishnan
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2412.11981
• Pdf URL: https://arxiv.org/pdf/2412.11981
• Copy Paste: [[2412.11981]] Industrial-scale Prediction of Cement Clinker Phases using Machine Learning(https://arxiv.org/abs/2412.11981)
• Keywords: robust
• Abstract: Cement production, exceeding 4.1 billion tonnes and contributing 2.4 tonnes of CO2 annually, faces critical challenges in quality control and process optimization. While traditional process models for cement manufacturing are confined to steady-state conditions with limited predictive capability for mineralogical phases, modern plants operate under dynamic conditions that demand real-time quality assessment. Here, exploiting a comprehensive two-year operational dataset from an industrial cement plant, we present a machine learning framework that accurately predicts clinker mineralogy from process data. Our model achieves unprecedented prediction accuracy for major clinker phases while requiring minimal input parameters, demonstrating robust performance under varying operating conditions. Through post-hoc explainable algorithms, we interpret the hierarchical relationships between clinker oxides and phase formation, providing insights into the functioning of an otherwise black-box model. This digital twin framework can potentially enable real-time optimization of cement production, thereby providing a route toward reducing material waste and ensuring quality while reducing the associated emissions under real plant conditions. Our approach represents a significant advancement in industrial process control, offering a scalable solution for sustainable cement manufacturing.

Title: Cost-Effective Label-free Node Classification with LLMs

• Authors: Taiyan Zhang, Renchi Yang, Mingyu Yan, Xiaochun Ye, Dongrui Fan, Yurui Lai
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.11983
• Pdf URL: https://arxiv.org/pdf/2412.11983
• Copy Paste: [[2412.11983]] Cost-Effective Label-free Node Classification with LLMs(https://arxiv.org/abs/2412.11983)
• Keywords: large language model
• Abstract: Graph neural networks (GNNs) have emerged as go-to models for node classification in graph data due to their powerful abilities in fusing graph structures and attributes. However, such models strongly rely on adequate high-quality labeled data for training, which are expensive to acquire in practice. With the advent of large language models (LLMs), a promising way is to leverage their superb zero-shot capabilities and massive knowledge for node labeling. Despite promising results reported, this methodology either demands considerable queries to LLMs, or suffers from compromised performance caused by noisy labels produced by LLMs. To remedy these issues, this work presents Cella, an active self-training framework that integrates LLMs into GNNs in a cost-effective manner. The design recipe of Cella is to iteratively identify small sets of "critical" samples using GNNs and extract informative pseudo-labels for them with both LLMs and GNNs as additional supervision signals to enhance model training. Particularly, Cella includes three major components: (i) an effective active node selection strategy for initial annotations; (ii) a judicious sample selection scheme to sift out the "critical" nodes based on label disharmonicity and entropy; and (iii) a label refinement module combining LLMs and GNNs with rewired topology. Our extensive experiments over five benchmark text-attributed graph datasets demonstrate that Cella significantly outperforms the state of the arts under the same query budget to LLMs in terms of label-free node classification. In particular, on the DBLP dataset with 14.3k nodes, Cella is able to achieve an 8.08% conspicuous improvement in accuracy over the state-of-the-art at a cost of less than one cent.

Title: SciFaultyQA: Benchmarking LLMs on Faulty Science Question Detection with a GAN-Inspired Approach to Synthetic Dataset Generation

• Authors: Debarshi Kundu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.11988
• Pdf URL: https://arxiv.org/pdf/2412.11988
• Copy Paste: [[2412.11988]] SciFaultyQA: Benchmarking LLMs on Faulty Science Question Detection with a GAN-Inspired Approach to Synthetic Dataset Generation(https://arxiv.org/abs/2412.11988)
• Keywords: large language model
• Abstract: Consider the problem: ``If one man and one woman can produce one child in one year, how many children will be produced by one woman and three men in 0.5 years?" Current large language models (LLMs) such as GPT-4o, GPT-o1-preview, and Gemini Flash frequently answer "0.5," which does not make sense. While these models sometimes acknowledge the unrealistic nature of the question, in many cases (8 out of 10 trials), they provide the nonsensical answer of "0.5 child." Additionally, temporal variation has been observed: if an LLM answers correctly once (by recognizing the faulty nature of the question), subsequent responses are more likely to also reflect this understanding. However, this is inconsistent. These types of questions have motivated us to develop a dataset of science questions, SciFaultyQA, where the questions themselves are intentionally faulty. We observed that LLMs often proceed to answer these flawed questions without recognizing their inherent issues, producing results that are logically or scientifically invalid. By analyzing such patterns, we developed a novel method for generating synthetic datasets to evaluate and benchmark the performance of various LLMs in identifying these flawed questions. We have also developed novel approaches to reduce the errors.

Title: ExecRepoBench: Multi-level Executable Code Completion Evaluation

• Authors: Jian Yang, Jiajun Zhang, Jiaxi Yang, Ke Jin, Lei Zhang, Qiyao Peng, Ken Deng, Yibo Miao, Tianyu Liu, Zeyu Cui, Binyuan Hui, Junyang Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.11990
• Pdf URL: https://arxiv.org/pdf/2412.11990
• Copy Paste: [[2412.11990]] ExecRepoBench: Multi-level Executable Code Completion Evaluation(https://arxiv.org/abs/2412.11990)
• Keywords: large language model
• Abstract: Code completion has become an essential tool for daily software development. Existing evaluation benchmarks often employ static methods that do not fully capture the dynamic nature of real-world coding environments and face significant challenges, including limited context length, reliance on superficial evaluation metrics, and potential overfitting to training datasets. In this work, we introduce a novel framework for enhancing code completion in software development through the creation of a repository-level benchmark ExecRepoBench and the instruction corpora Repo-Instruct, aim at improving the functionality of open-source large language models (LLMs) in real-world coding scenarios that involve complex interdependencies across multiple files. ExecRepoBench includes 1.2K samples from active Python repositories. Plus, we present a multi-level grammar-based completion methodology conditioned on the abstract syntax tree to mask code fragments at various logical units (e.g. statements, expressions, and functions). Then, we fine-tune the open-source LLM with 7B parameters on Repo-Instruct to produce a strong code completion baseline model Qwen2.5-Coder-Instruct-C based on the open-source model. Qwen2.5-Coder-Instruct-C is rigorously evaluated against existing benchmarks, including MultiPL-E and ExecRepoBench, which consistently outperforms prior baselines across all programming languages. The deployment of \ourmethod{} can be used as a high-performance, local service for programming development\footnote{\url{this https URL}}.

Title: Efficient Layered New Bit-Flipping QC-MDPC Decoder for BIKE Post-Quantum Cryptography

• Authors: Jiaxuan Cai, Xinmiao Zhang
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2412.11997
• Pdf URL: https://arxiv.org/pdf/2412.11997
• Copy Paste: [[2412.11997]] Efficient Layered New Bit-Flipping QC-MDPC Decoder for BIKE Post-Quantum Cryptography(https://arxiv.org/abs/2412.11997)
• Keywords: security
• Abstract: The medium-density parity-check (MDPC) code-based Bit Flipping Key Encapsulation (BIKE) mechanism remains a candidate of post-quantum cryptography standardization. The latest version utilizes a new bit-flipping (BF) decoding algorithm, which decides the BF threshold by an affine function with high-precision coefficients. Previous BF decoder implementations can be extended to the new algorithm. However, they suffer from large memories that dominate the overall complexity. This paper proposes a column-layered decoder for the new BIKE BF decoding algorithm to substantially reduce the memory requirement, and optimizes the affine BF threshold function coefficients to reduce the code length needed for the same security level. For the first time, our work also investigates the impact of finite precision representation of the threshold coefficients on the decoding performance. For an example MDPC code considered for the standard, the proposed layered BF decoder achieves 20% complexity reduction compared to the best prior effort with a very small latency overhead.

Title: SAMIC: Segment Anything with In-Context Spatial Prompt Engineering

• Authors: Savinay Nagendra, Kashif Rashid, Chaopeng Shen, Daniel Kifer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.11998
• Pdf URL: https://arxiv.org/pdf/2412.11998
• Copy Paste: [[2412.11998]] SAMIC: Segment Anything with In-Context Spatial Prompt Engineering(https://arxiv.org/abs/2412.11998)
• Keywords: segmentation
• Abstract: Few-shot segmentation is the problem of learning to identify specific types of objects (e.g., airplanes) in images from a small set of labeled reference images. The current state of the art is driven by resource-intensive construction of models for every new domain-specific application. Such models must be trained on enormous labeled datasets of unrelated objects (e.g., cars, trains, animals) so that their ``knowledge'' can be transferred to new types of objects. In this paper, we show how to leverage existing vision foundation models (VFMs) to reduce the incremental cost of creating few-shot segmentation models for new domains. Specifically, we introduce SAMIC, a small network that learns how to prompt VFMs in order to segment new types of objects in domain-specific applications. SAMIC enables any task to be approached as a few-shot learning problem. At 2.6 million parameters, it is 94% smaller than the leading models (e.g., having ResNet 101 backbone with 45+ million parameters). Even using 1/5th of the training data provided by one-shot benchmarks, SAMIC is competitive with, or sets the state of the art, on a variety of few-shot and semantic segmentation datasets including COCO-$20^i$, Pascal-$5^i$, PerSeg, FSS-1000, and NWPU VHR-10.

Title: LLM-RG4: Flexible and Factual Radiology Report Generation across Diverse Input Contexts

• Authors: Zhuhao Wang, Yihua Sun, Zihan Li, Xuan Yang, Fang Chen, Hongen Liao
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12001
• Pdf URL: https://arxiv.org/pdf/2412.12001
• Copy Paste: [[2412.12001]] LLM-RG4: Flexible and Factual Radiology Report Generation across Diverse Input Contexts(https://arxiv.org/abs/2412.12001)
• Keywords: large language model
• Abstract: Drafting radiology reports is a complex task requiring flexibility, where radiologists tail content to available information and particular clinical demands. However, most current radiology report generation (RRG) models are constrained to a fixed task paradigm, such as predicting the full ``finding'' section from a single image, inherently involving a mismatch between inputs and outputs. The trained models lack the flexibility for diverse inputs and could generate harmful, input-agnostic hallucinations. To bridge the gap between current RRG models and the clinical demands in practice, we first develop a data generation pipeline to create a new MIMIC-RG4 dataset, which considers four common radiology report drafting scenarios and has perfectly corresponded input and output. Secondly, we propose a novel large language model (LLM) based RRG framework, namely LLM-RG4, which utilizes LLM's flexible instruction-following capabilities and extensive general knowledge. We further develop an adaptive token fusion module that offers flexibility to handle diverse scenarios with different input combinations, while minimizing the additional computational burden associated with increased input volumes. Besides, we propose a token-level loss weighting strategy to direct the model's attention towards positive and uncertain descriptions. Experimental results demonstrate that LLM-RG4 achieves state-of-the-art performance in both clinical efficiency and natural language generation on the MIMIC-RG4 and MIMIC-CXR datasets. We quantitatively demonstrate that our model has minimal input-agnostic hallucinations, whereas current open-source models commonly suffer from this problem.

Title: The Open Source Advantage in Large Language Models (LLMs)

• Authors: Jiya Manchanda, Laura Boettcher, Matheus Westphalen, Jasser Jasser
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12004
• Pdf URL: https://arxiv.org/pdf/2412.12004
• Copy Paste: [[2412.12004]] The Open Source Advantage in Large Language Models (LLMs)(https://arxiv.org/abs/2412.12004)
• Keywords: transformer, large language model
• Abstract: Large language models (LLMs) mark a key shift in natural language processing (NLP), having advanced text generation, translation, and domain-specific reasoning. Closed-source models like GPT-4, powered by proprietary datasets and extensive computational resources, lead with state-of-the-art performance today. However, they face criticism for their "black box" nature and for limiting accessibility in a manner that hinders reproducibility and equitable AI development. By contrast, open-source initiatives like LLaMA and BLOOM prioritize democratization through community-driven development and computational efficiency. These models have significantly reduced performance gaps, particularly in linguistic diversity and domain-specific applications, while providing accessible tools for global researchers and developers. Notably, both paradigms rely on foundational architectural innovations, such as the Transformer framework by Vaswani et al. (2017). Closed-source models excel by scaling effectively, while open-source models adapt to real-world applications in underrepresented languages and domains. Techniques like Low-Rank Adaptation (LoRA) and instruction-tuning datasets enable open-source models to achieve competitive results despite limited resources. To be sure, the tension between closed-source and open-source approaches underscores a broader debate on transparency versus proprietary control in AI. Ethical considerations further highlight this divide. Closed-source systems restrict external scrutiny, while open-source models promote reproducibility and collaboration but lack standardized auditing documentation frameworks to mitigate biases. Hybrid approaches that leverage the strengths of both paradigms are likely to shape the future of LLM innovation, ensuring accessibility, competitive technical performance, and ethical deployment.

Title: Learning to Navigate in Mazes with Novel Layouts using Abstract Top-down Maps

• Authors: Linfeng Zhao, Lawson L.S. Wong
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.12024
• Pdf URL: https://arxiv.org/pdf/2412.12024
• Copy Paste: [[2412.12024]] Learning to Navigate in Mazes with Novel Layouts using Abstract Top-down Maps(https://arxiv.org/abs/2412.12024)
• Keywords: robust
• Abstract: Learning navigation capabilities in different environments has long been one of the major challenges in decision-making. In this work, we focus on zero-shot navigation ability using given abstract $2$-D top-down maps. Like human navigation by reading a paper map, the agent reads the map as an image when navigating in a novel layout, after learning to navigate on a set of training maps. We propose a model-based reinforcement learning approach for this multi-task learning problem, where it jointly learns a hypermodel that takes top-down maps as input and predicts the weights of the transition network. We use the DeepMind Lab environment and customize layouts using generated maps. Our method can adapt better to novel environments in zero-shot and is more robust to noise.

Title: RepFace: Refining Closed-Set Noise with Progressive Label Correction for Face Recognition

• Authors: Jie Zhang, Xun Gong, Zhonglin Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12031
• Pdf URL: https://arxiv.org/pdf/2412.12031
• Copy Paste: [[2412.12031]] RepFace: Refining Closed-Set Noise with Progressive Label Correction for Face Recognition(https://arxiv.org/abs/2412.12031)
• Keywords: robust
• Abstract: Face recognition has made remarkable strides, driven by the expanding scale of datasets, advancements in various backbone and discriminative losses. However, face recognition performance is heavily affected by the label noise, especially closed-set noise. While numerous studies have focused on handling label noise, addressing closed-set noise still poses challenges. This paper identifies this challenge as training isn't robust to noise at the early-stage training, and necessitating an appropriate learning strategy for samples with low confidence, which are often misclassified as closed-set noise in later training phases. To address these issues, we propose a new framework to stabilize the training at early stages and split the samples into clean, ambiguous and noisy groups which are devised with separate training strategies. Initially, we employ generated auxiliary closed-set noisy samples to enable the model to identify noisy data at the early stages of training. Subsequently, we introduce how samples are split into clean, ambiguous and noisy groups by their similarity to the positive and nearest negative centers. Then we perform label fusion for ambiguous samples by incorporating accumulated model predictions. Finally, we apply label smoothing within the closed set, adjusting the label to a point between the nearest negative class and the initially assigned label. Extensive experiments validate the effectiveness of our method on mainstream face datasets, achieving state-of-the-art results. The code will be released upon acceptance.

Title: FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning

• Authors: Gaojian Wang, Feng Lin, Tong Wu, Zhenguang Liu, Zhongjie Ba, Kui Ren
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12032
• Pdf URL: https://arxiv.org/pdf/2412.12032
• Copy Paste: [[2412.12032]] FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning(https://arxiv.org/abs/2412.12032)
• Keywords: security, robust, diffusion
• Abstract: This work asks: with abundant, unlabeled real faces, how to learn a robust and transferable facial representation that boosts various face security tasks with respect to generalization performance? We make the first attempt and propose a self-supervised pretraining framework to learn fundamental representations of real face images, FSFM, that leverages the synergy between masked image modeling (MIM) and instance discrimination (ID). We explore various facial masking strategies for MIM and present a simple yet powerful CRFR-P masking, which explicitly forces the model to capture meaningful intra-region consistency and challenging inter-region coherency. Furthermore, we devise the ID network that naturally couples with MIM to establish underlying local-to-global correspondence via tailored self-distillation. These three learning objectives, namely 3C, empower encoding both local features and global semantics of real faces. After pretraining, a vanilla ViT serves as a universal vision foundation model for downstream face security tasks: cross-dataset deepfake detection, cross-domain face anti-spoofing, and unseen diffusion facial forgery detection. Extensive experiments on 10 public datasets demonstrate that our model transfers better than supervised pretraining, visual and facial self-supervised learning arts, and even outperforms task-specialized SOTA methods.

Title: Thermodynamics-informed graph neural networks for real-time simulation of digital human twins

• Authors: Lucas Tesán, David González, Pedro Martins, Elías Cueto
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12034
• Pdf URL: https://arxiv.org/pdf/2412.12034
• Copy Paste: [[2412.12034]] Thermodynamics-informed graph neural networks for real-time simulation of digital human twins(https://arxiv.org/abs/2412.12034)
• Keywords: robust
• Abstract: The growing importance of real-time simulation in the medical field has exposed the limitations and bottlenecks inherent in the digital representation of complex biological systems. This paper presents a novel methodology aimed at advancing current lines of research in soft tissue simulation. The proposed approach introduces a hybrid model that integrates the geometric bias of graph neural networks with the physical bias derived from the imposition of a metriplectic structure as soft and hard constrains in the architecture, being able to simulate hepatic tissue with dissipative properties. This approach provides an efficient solution capable of generating predictions at high feedback rate while maintaining a remarkable generalization ability for previously unseen anatomies. This makes these features particularly relevant in the context of precision medicine and haptic rendering. Based on the adopted methodologies, we propose a model that predicts human liver responses to traction and compression loads in as little as 7.3 milliseconds for optimized configurations and as fast as 1.65 milliseconds in the most efficient cases, all in the forward pass. The model achieves relative position errors below 0.15\%, with stress tensor and velocity estimations maintaining relative errors under 7\%. This demonstrates the robustness of the approach developed, which is capable of handling diverse load states and anatomies effectively. This work highlights the feasibility of integrating real-time simulation with patient-specific geometries through deep learning, paving the way for more robust digital human twins in medical applications.

Title: LeARN: Learnable and Adaptive Representations for Nonlinear Dynamics in System Identification

• Authors: Arunabh Singh, Joyjit Mukherjee
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.12036
• Pdf URL: https://arxiv.org/pdf/2412.12036
• Copy Paste: [[2412.12036]] LeARN: Learnable and Adaptive Representations for Nonlinear Dynamics in System Identification(https://arxiv.org/abs/2412.12036)
• Keywords: robust
• Abstract: System identification, the process of deriving mathematical models of dynamical systems from observed input-output data, has undergone a paradigm shift with the advent of learning-based methods. Addressing the intricate challenges of data-driven discovery in nonlinear dynamical systems, these methods have garnered significant attention. Among them, Sparse Identification of Nonlinear Dynamics (SINDy) has emerged as a transformative approach, distilling complex dynamical behaviors into interpretable linear combinations of basis functions. However, SINDy relies on domain-specific expertise to construct its foundational "library" of basis functions, which limits its adaptability and universality. In this work, we introduce a nonlinear system identification framework called LeARN that transcends the need for prior domain knowledge by learning the library of basis functions directly from data. To enhance adaptability to evolving system dynamics under varying noise conditions, we employ a novel meta-learning-based system identification approach that uses a lightweight deep neural network (DNN) to dynamically refine these basis functions. This not only captures intricate system behaviors but also adapts seamlessly to new dynamical regimes. We validate our framework on the Neural Fly dataset, showcasing its robust adaptation and generalization capabilities. Despite its simplicity, our LeARN achieves competitive dynamical error performance compared to SINDy. This work presents a step toward the autonomous discovery of dynamical systems, paving the way for a future where machine learning uncovers the governing principles of complex systems without requiring extensive domain-specific interventions.