2024-12-18

Title: SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout

• Authors: Chiyu Max Jiang, Yijing Bai, Andre Cornman, Christopher Davis, Xiukun Huang, Hong Jeon, Sakshum Kulshrestha, John Lambert, Shuangyu Li, Xuanyu Zhou, Carlos Fuertes, Chang Yuan, Mingxing Tan, Yin Zhou, Dragomir Anguelov
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12129
• Pdf URL: https://arxiv.org/pdf/2412.12129
• Copy Paste: [[2412.12129]] SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout(https://arxiv.org/abs/2412.12129)
• Keywords: diffusion, large language model
• Abstract: Realistic and interactive scene simulation is a key prerequisite for autonomous vehicle (AV) development. In this work, we present SceneDiffuser, a scene-level diffusion prior designed for traffic simulation. It offers a unified framework that addresses two key stages of simulation: scene initialization, which involves generating initial traffic layouts, and scene rollout, which encompasses the closed-loop simulation of agent behaviors. While diffusion models have been proven effective in learning realistic and multimodal agent distributions, several challenges remain, including controllability, maintaining realism in closed-loop simulations, and ensuring inference efficiency. To address these issues, we introduce amortized diffusion for simulation. This novel diffusion denoising paradigm amortizes the computational cost of denoising over future simulation steps, significantly reducing the cost per rollout step (16x less inference steps) while also mitigating closed-loop errors. We further enhance controllability through the introduction of generalized hard constraints, a simple yet effective inference-time constraint mechanism, as well as language-based constrained scene generation via few-shot prompting of a large language model (LLM). Our investigations into model scaling reveal that increased computational resources significantly improve overall simulation realism. We demonstrate the effectiveness of our approach on the Waymo Open Sim Agents Challenge, achieving top open-loop performance and the best closed-loop performance among diffusion models.

Title: Technical Insights on Blockchain's Role in Financial Systems

• Authors: Ishan Patwardhan, Sunil Mane, Nidhi Patel
• Subjects: cs.CR, cs.CE, cs.ET
• Abstract URL: https://arxiv.org/abs/2412.12131
• Pdf URL: https://arxiv.org/pdf/2412.12131
• Copy Paste: [[2412.12131]] Technical Insights on Blockchain's Role in Financial Systems(https://arxiv.org/abs/2412.12131)
• Keywords: security
• Abstract: This research provides a critical analysis regarding the way blockchain is being implemented in the financial industry, highlighting its vital role in promoting green finance, guaranteeing compliance with regulations, improving supply chain finance, boosting decentralized finance (DeFi), and strengthening the Internet of Things (IoT). It discusses how blockchain's inherent attributes could significantly boost transparency, operational efficiency, and security across these domains while also addressing the pressing challenges of scalability, system integration, and the evolving regulatory landscape.

Title: Frontier AI systems have surpassed the self-replicating red line

• Authors: Xudong Pan, Jiarun Dai, Yihe Fan, Min Yang
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12140
• Pdf URL: https://arxiv.org/pdf/2412.12140
• Copy Paste: [[2412.12140]] Frontier AI systems have surpassed the self-replicating red line(https://arxiv.org/abs/2412.12140)
• Keywords: large language model
• Abstract: Successful self-replication under no human assistance is the essential step for AI to outsmart the human beings, and is an early signal for rogue AIs. That is why self-replication is widely recognized as one of the few red line risks of frontier AI systems. Nowadays, the leading AI corporations OpenAI and Google evaluate their flagship large language models GPT-o1 and Gemini Pro 1.0, and report the lowest risk level of self-replication. However, following their methodology, we for the first time discover that two AI systems driven by Meta's Llama31-70B-Instruct and Alibaba's Qwen25-72B-Instruct, popular large language models of less parameters and weaker capabilities, have already surpassed the self-replicating red line. In 50% and 90% experimental trials, they succeed in creating a live and separate copy of itself respectively. By analyzing the behavioral traces, we observe the AI systems under evaluation already exhibit sufficient self-perception, situational awareness and problem-solving capabilities to accomplish self-replication. We further note the AI systems are even able to use the capability of self-replication to avoid shutdown and create a chain of replica to enhance the survivability, which may finally lead to an uncontrolled population of AIs. If such a worst-case risk is let unknown to the human society, we would eventually lose control over the frontier AI systems: They would take control over more computing devices, form an AI species and collude with each other against human beings. Our findings are a timely alert on existing yet previously unknown severe AI risks, calling for international collaboration on effective governance on uncontrolled self-replication of AI systems.

Title: Automatic Item Generation for Personality Situational Judgment Tests with Large Language Models

• Authors: Chang-Jin Li, Jiyuan Zhang, Yun Tang, Jian Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12144
• Pdf URL: https://arxiv.org/pdf/2412.12144
• Copy Paste: [[2412.12144]] Automatic Item Generation for Personality Situational Judgment Tests with Large Language Models(https://arxiv.org/abs/2412.12144)
• Keywords: large language model
• Abstract: Personality assessment, particularly through situational judgment tests (SJTs), is a vital tool for psychological research, talent selection, and educational evaluation. This study explores the potential of GPT-4, a state-of-the-art large language model (LLM), to automate the generation of personality situational judgment tests (PSJTs) in Chinese. Traditional SJT development is labor-intensive and prone to biases, while GPT-4 offers a scalable, efficient alternative. Two studies were conducted: Study 1 evaluated the impact of prompt design and temperature settings on content validity, finding that optimized prompts with a temperature of 1.0 produced creative and accurate items. Study 2 assessed the psychometric properties of GPT-4-generated PSJTs, revealing that they demonstrated satisfactory reliability and validity, surpassing the performance of manually developed tests in measuring the Big Five personality traits. This research highlights GPT-4's effectiveness in developing high-quality PSJTs, providing a scalable and innovative method for psychometric test development. These findings expand the possibilities of automatic item generation and the application of LLMs in psychology, and offer practical implications for streamlining test development processes in resource-limited settings.

Title: Na'vi or Knave: Jailbreaking Language Models via Metaphorical Avatars

• Authors: Yu Yan, Sheng Sun, Junqi Tong, Min Liu, Qi Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12145
• Pdf URL: https://arxiv.org/pdf/2412.12145
• Copy Paste: [[2412.12145]] Na'vi or Knave: Jailbreaking Language Models via Metaphorical Avatars(https://arxiv.org/abs/2412.12145)
• Keywords: security, defense, attack, large language model
• Abstract: Metaphor serves as an implicit approach to convey information, while enabling the generalized comprehension of complex subjects. However, metaphor can potentially be exploited to bypass the safety alignment mechanisms of Large Language Models (LLMs), leading to the theft of harmful knowledge. In our study, we introduce a novel attack framework that exploits the imaginative capacity of LLMs to achieve jailbreaking, the J\underline{\textbf{A}}ilbreak \underline{\textbf{V}}ia \underline{\textbf{A}}dversarial Me\underline{\textbf{TA}} -pho\underline{\textbf{R}} (\textit{AVATAR}). Specifically, to elicit the harmful response, AVATAR extracts harmful entities from a given harmful target and maps them to innocuous adversarial entities based on LLM's imagination. Then, according to these metaphors, the harmful target is nested within human-like interaction for jailbreaking adaptively. Experimental results demonstrate that AVATAR can effectively and transferablly jailbreak LLMs and achieve a state-of-the-art attack success rate across multiple advanced LLMs. Our study exposes a security risk in LLMs from their endogenous imaginative capabilities. Furthermore, the analytical study reveals the vulnerability of LLM to adversarial metaphors and the necessity of developing defense methods against jailbreaking caused by the adversarial metaphor. \textcolor{orange}{ \textbf{Warning: This paper contains potentially harmful content from LLMs.}}

Title: Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Control

• Authors: Seongwoong Cho, Donggyun Kim, Jinwoo Lee, Seunghoon Hong
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.12147
• Pdf URL: https://arxiv.org/pdf/2412.12147
• Copy Paste: [[2412.12147]] Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Control(https://arxiv.org/abs/2412.12147)
• Keywords: robust
• Abstract: Generalizing across robot embodiments and tasks is crucial for adaptive robotic systems. Modular policy learning approaches adapt to new embodiments but are limited to specific tasks, while few-shot imitation learning (IL) approaches often focus on a single embodiment. In this paper, we introduce a few-shot behavior cloning framework to simultaneously generalize to unseen embodiments and tasks using a few (\emph{e.g.,} five) reward-free demonstrations. Our framework leverages a joint-level input-output representation to unify the state and action spaces of heterogeneous embodiments and employs a novel structure-motion state encoder that is parameterized to capture both shared knowledge across all embodiments and embodiment-specific knowledge. A matching-based policy network then predicts actions from a few demonstrations, producing an adaptive policy that is robust to over-fitting. Evaluated in the DeepMind Control suite, our framework termed \modelname{} demonstrates superior few-shot generalization to unseen embodiments and tasks over modular policy learning and few-shot IL approaches. Codes are available at \href{this https URL}{this https URL}.

Title: SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibration

• Authors: Yuanhao Shen, Xiaodan Zhu, Lei Chen
• Subjects: cs.LG, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.12151
• Pdf URL: https://arxiv.org/pdf/2412.12151
• Copy Paste: [[2412.12151]] SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibration(https://arxiv.org/abs/2412.12151)
• Keywords: large language model
• Abstract: The tool-use ability of Large Language Models (LLMs) has a profound impact on a wide range of industrial applications. However, LLMs' self-control and calibration capability in appropriately using tools remains understudied. The problem is consequential as it raises potential risks of degraded performance and poses a threat to the trustworthiness of the models. In this paper, we conduct a study on a family of state-of-the-art LLMs on three datasets with two mainstream tool-use frameworks. Our study reveals the tool-abuse behavior of LLMs, a tendency for models to misuse tools with overconfidence. We also find that this is a common issue regardless of model capability. Accordingly, we propose a novel approach, \textit{SMARTCAL}, to mitigate the observed issues, and our results show an average of 8.6 percent increase in the QA performance and a 21.6 percent decrease in Expected Calibration Error (ECE) compared to baseline models.

Title: GraphTool-Instruction: Revolutionizing Graph Reasoning in LLMs through Decomposed Subtask Instruction

• Authors: Rongzheng Wang, Shuang Liang, Qizhi Chen, Jiasheng Zhang, Ke Qin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12152
• Pdf URL: https://arxiv.org/pdf/2412.12152
• Copy Paste: [[2412.12152]] GraphTool-Instruction: Revolutionizing Graph Reasoning in LLMs through Decomposed Subtask Instruction(https://arxiv.org/abs/2412.12152)
• Keywords: extraction, large language model
• Abstract: Large language models (LLMs) have been demonstrated to possess the capabilities to understand fundamental graph properties and address various graph reasoning tasks. Existing methods fine-tune LLMs to understand and execute graph reasoning tasks by specially designed task instructions. However, these Text-Instruction methods generally exhibit poor performance. Inspired by tool learning, researchers propose Tool-Instruction methods to solve various graph problems by special tool calling (e.g., function, API and model), achieving significant improvements in graph reasoning tasks. Nevertheless, current Tool-Instruction approaches focus on the tool information and ignore the graph structure information, which leads to significantly inferior performance on small-scale LLMs (less than 13B). To tackle this issue, we propose GraphTool-Instruction, an innovative Instruction-tuning approach that decomposes the graph reasoning task into three distinct subtasks (i.e., graph extraction, tool name identification and tool parameter extraction), and design specialized instructions for each subtask. Our GraphTool-Instruction can be used as a plug-and-play prompt for different LLMs without fine-tuning. Moreover, building on GraphTool-Instruction, we develop GTools, a dataset that includes twenty graph reasoning tasks, and create a graph reasoning LLM called GraphForge based on Llama3-8B. We conduct extensive experiments on twenty graph reasoning tasks with different graph types (e.g., graph size or graph direction), and we find that GraphTool-Instruction achieves SOTA compared to Text-Instruction and Tool-Instruction methods. Fine-tuned on GTools, GraphForge gets further improvement of over 30% compared to the Tool-Instruction enhanced GPT-3.5-turbo, and it performs comparably to the high-cost GPT-4o. Our codes and data are available at this https URL.

Title: PyOD 2: A Python Library for Outlier Detection with LLM-powered Model Selection

• Authors: Sihan Chen, Zhuangzhuang Qian, Wingchun Siu, Xingcan Hu, Jiaqi Li, Shawn Li, Yuehan Qin, Tiankai Yang, Zhuo Xiao, Wanghao Ye, Yichi Zhang, Yushun Dong, Yue Zhao
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12154
• Pdf URL: https://arxiv.org/pdf/2412.12154
• Copy Paste: [[2412.12154]] PyOD 2: A Python Library for Outlier Detection with LLM-powered Model Selection(https://arxiv.org/abs/2412.12154)
• Keywords: robust, large language model
• Abstract: Outlier detection (OD), also known as anomaly detection, is a critical machine learning (ML) task with applications in fraud detection, network intrusion detection, clickstream analysis, recommendation systems, and social network moderation. Among open-source libraries for outlier detection, the Python Outlier Detection (PyOD) library is the most widely adopted, with over 8,500 GitHub stars, 25 million downloads, and diverse industry usage. However, PyOD currently faces three limitations: (1) insufficient coverage of modern deep learning algorithms, (2) fragmented implementations across PyTorch and TensorFlow, and (3) no automated model selection, making it hard for non-experts. To address these issues, we present PyOD Version 2 (PyOD 2), which integrates 12 state-of-the-art deep learning models into a unified PyTorch framework and introduces a large language model (LLM)-based pipeline for automated OD model selection. These improvements simplify OD workflows, provide access to 45 algorithms, and deliver robust performance on various datasets. In this paper, we demonstrate how PyOD 2 streamlines the deployment and automation of OD models and sets a new standard in both research and industry. PyOD 2 is accessible at [this https URL](this https URL). This study aligns with the Web Mining and Content Analysis track, addressing topics such as the robustness of Web mining methods and the quality of algorithmically-generated Web data.

Title: What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis

• Authors: Jiayu Liu, Zhenya Huang, Chaokun Wang, Xunpeng Huang, Chengxiang Zhai, Enhong Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12157
• Pdf URL: https://arxiv.org/pdf/2412.12157
• Copy Paste: [[2412.12157]] What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis(https://arxiv.org/abs/2412.12157)
• Keywords: large language model
• Abstract: Owing to the capability of in-context learning, large language models (LLMs) have shown impressive performance across diverse mathematical reasoning benchmarks. However, we find that few-shot demonstrations can sometimes bring negative performance and their effectiveness on LLMs' reasoning abilities remains unreliable. To this end, in this paper, we aim to theoretically analyze the impact of in-context demonstrations on LLMs' reasoning performance. We prove that the reasoning efficacy (measured by empirical prediction loss) can be bounded by a LLM-oriented semantic similarity and an inference stability of demonstrations, which is general for both one-shot and few-shot scenarios. Based on this finding, we propose a straightforward, generalizable, and low-complexity demonstration selection method named LMS3. It can adaptively facilitate to select the most pertinent samples for different LLMs and includes a novel demonstration rejection mechanism to automatically filter out samples that are unsuitable for few-shot learning. Through experiments on three representative benchmarks, two LLM backbones, and multiple few-shot settings, we verify that our LMS3 has superiority and achieves consistent improvements on all datasets, which existing methods have been unable to accomplish.

Title: Climate Aware Deep Neural Networks (CADNN) for Wind Power Simulation

• Authors: Ali Forootani, Danial Esmaeili Aliabadi, Daniela Thraen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12160
• Pdf URL: https://arxiv.org/pdf/2412.12160
• Copy Paste: [[2412.12160]] Climate Aware Deep Neural Networks (CADNN) for Wind Power Simulation(https://arxiv.org/abs/2412.12160)
• Keywords: transformer
• Abstract: Wind power forecasting plays a critical role in modern energy systems, facilitating the integration of renewable energy sources into the power grid. Accurate prediction of wind energy output is essential for managing the inherent intermittency of wind power, optimizing energy dispatch, and ensuring grid stability. This paper proposes the use of Deep Neural Network (DNN)-based predictive models that leverage climate datasets, including wind speed, atmospheric pressure, temperature, and other meteorological variables, to improve the accuracy of wind power simulations. In particular, we focus on the Coupled Model Intercomparison Project (CMIP) datasets, which provide climate projections, as inputs for training the DNN models. These models aim to capture the complex nonlinear relationships between the CMIP-based climate data and actual wind power generation at wind farms located in Germany. Our study compares various DNN architectures, specifically Multilayer Perceptron (MLP), Long Short-Term Memory (LSTM) networks, and Transformer-enhanced LSTM models, to identify the best configuration among these architectures for climate-aware wind power simulation. The implementation of this framework involves the development of a Python package (CADNN) designed to support multiple tasks, including statistical analysis of the climate data, data visualization, preprocessing, DNN training, and performance evaluation. We demonstrate that the DNN models, when integrated with climate data, significantly enhance forecasting accuracy. This climate-aware approach offers a deeper understanding of the time-dependent climate patterns that influence wind power generation, providing more accurate predictions and making it adaptable to other geographical regions.

Title: Towards LLM-based optimization compilers. Can LLMs learn how to apply a single peephole optimization? Reasoning is all LLMs need!

• Authors: Xiangxin Fang, Lev Mukhanov
• Subjects: cs.LG, cs.AI, cs.PL
• Abstract URL: https://arxiv.org/abs/2412.12163
• Pdf URL: https://arxiv.org/pdf/2412.12163
• Copy Paste: [[2412.12163]] Towards LLM-based optimization compilers. Can LLMs learn how to apply a single peephole optimization? Reasoning is all LLMs need!(https://arxiv.org/abs/2412.12163)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated great potential in various language processing tasks, and recent studies have explored their application in compiler optimizations. However, all these studies focus on the conventional open-source LLMs, such as Llama2, which lack enhanced reasoning mechanisms. In this study, we investigate the errors produced by the fine-tuned 7B-parameter Llama2 model as it attempts to learn and apply a simple peephole optimization for the AArch64 assembly code. We provide an analysis of the errors produced by the LLM and compare it with state-of-the-art OpenAI models which implement advanced reasoning logic, including GPT-4o and GPT-o1 (preview). We demonstrate that OpenAI GPT-o1, despite not being fine-tuned, outperforms the fine-tuned Llama2 and GPT-4o. Our findings indicate that this advantage is largely due to the chain-of-thought reasoning implemented in GPT-o1. We hope our work will inspire further research on using LLMs with enhanced reasoning mechanisms and chain-of-thought for code generation and optimization.

Title: GAMED: Knowledge Adaptive Multi-Experts Decoupling for Multimodal Fake News Detection

• Authors: Lingzhi Shen, Yunfei Long, Xiaohao Cai, Imran Razzak, Guanming Chen, Kang Liu, Shoaib Jameel
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12164
• Pdf URL: https://arxiv.org/pdf/2412.12164
• Copy Paste: [[2412.12164]] GAMED: Knowledge Adaptive Multi-Experts Decoupling for Multimodal Fake News Detection(https://arxiv.org/abs/2412.12164)
• Keywords: explainability
• Abstract: Multimodal fake news detection often involves modelling heterogeneous data sources, such as vision and language. Existing detection methods typically rely on fusion effectiveness and cross-modal consistency to model the content, complicating understanding how each modality affects prediction accuracy. Additionally, these methods are primarily based on static feature modelling, making it difficult to adapt to the dynamic changes and relationships between different data modalities. This paper develops a significantly novel approach, GAMED, for multimodal modelling, which focuses on generating distinctive and discriminative features through modal decoupling to enhance cross-modal synergies, thereby optimizing overall performance in the detection process. GAMED leverages multiple parallel expert networks to refine features and pre-embed semantic knowledge to improve the experts' ability in information selection and viewpoint sharing. Subsequently, the feature distribution of each modality is adaptively adjusted based on the respective experts' opinions. GAMED also introduces a novel classification technique to dynamically manage contributions from different modalities, while improving the explainability of decisions. Experimental results on the Fakeddit and Yang datasets demonstrate that GAMED performs better than recently developed state-of-the-art models. The source code can be accessed at this https URL.

Title: Multimodal Approaches to Fair Image Classification: An Ethical Perspective

• Authors: Javon Hickmon
• Subjects: cs.CV, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12165
• Pdf URL: https://arxiv.org/pdf/2412.12165
• Copy Paste: [[2412.12165]] Multimodal Approaches to Fair Image Classification: An Ethical Perspective(https://arxiv.org/abs/2412.12165)
• Keywords: fair
• Abstract: In the rapidly advancing field of artificial intelligence, machine perception is becoming paramount to achieving increased performance. Image classification systems are becoming increasingly integral to various applications, ranging from medical diagnostics to image generation; however, these systems often exhibit harmful biases that can lead to unfair and discriminatory outcomes. Machine Learning systems that depend on a single data modality, i.e. only images or only text, can exaggerate hidden biases present in the training data, if the data is not carefully balanced and filtered. Even so, these models can still harm underrepresented populations when used in improper contexts, such as when government agencies reinforce racial bias using predictive policing. This thesis explores the intersection of technology and ethics in the development of fair image classification models. Specifically, I focus on improving fairness and methods of using multiple modalities to combat harmful demographic bias. Integrating multimodal approaches, which combine visual data with additional modalities such as text and metadata, allows this work to enhance the fairness and accuracy of image classification systems. The study critically examines existing biases in image datasets and classification algorithms, proposes innovative methods for mitigating these biases, and evaluates the ethical implications of deploying such systems in real-world scenarios. Through comprehensive experimentation and analysis, the thesis demonstrates how multimodal techniques can contribute to more equitable and ethical AI solutions, ultimately advocating for responsible AI practices that prioritize fairness.

Title: Performance of a large language model-Artificial Intelligence based chatbot for counseling patients with sexually transmitted infections and genital diseases

• Authors: Nikhil Mehta, Sithira Ambepitiya, Thanveer Ahamad, Dinuka Wijesundara, Yudara Kularathne
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12166
• Pdf URL: https://arxiv.org/pdf/2412.12166
• Copy Paste: [[2412.12166]] Performance of a large language model-Artificial Intelligence based chatbot for counseling patients with sexually transmitted infections and genital diseases(https://arxiv.org/abs/2412.12166)
• Keywords: large language model
• Abstract: Introduction: Global burden of sexually transmitted infections (STIs) is rising out of proportion to specialists. Current chatbots like ChatGPT are not tailored for handling STI-related concerns out of the box. We developed Otiz, an Artificial Intelligence-based (AI-based) chatbot platform designed specifically for STI detection and counseling, and assessed its performance. Methods: Otiz employs a multi-agent system architecture based on GPT4-0613, leveraging large language model (LLM) and Deterministic Finite Automaton principles to provide contextually relevant, medically accurate, and empathetic responses. Its components include modules for general STI information, emotional recognition, Acute Stress Disorder detection, and psychotherapy. A question suggestion agent operates in parallel. Four STIs (anogenital warts, herpes, syphilis, urethritis/cervicitis) and 2 non-STIs (candidiasis, penile cancer) were evaluated using prompts mimicking patient language. Each prompt was independently graded by two venereologists conversing with Otiz as patient actors on 6 criteria using Numerical Rating Scale ranging from 0 (poor) to 5 (excellent). Results: Twenty-three venereologists did 60 evaluations of 30 prompts. Across STIs, Otiz scored highly on diagnostic accuracy (4.1-4.7), overall accuracy (4.3-4.6), correctness of information (5.0), comprehensibility (4.2-4.4), and empathy (4.5-4.8). However, relevance scores were lower (2.9-3.6), suggesting some redundancy. Diagnostic scores for non-STIs were lower (p=0.038). Inter-observer agreement was strong, with differences greater than 1 point occurring in only 12.7% of paired evaluations. Conclusions: AI conversational agents like Otiz can provide accurate, correct, discrete, non-judgmental, readily accessible and easily understandable STI-related information in an empathetic manner, and can alleviate the burden on healthcare systems.

Title: Regulation of Language Models With Interpretability Will Likely Result In A Performance Trade-Off

• Authors: Eoin M. Kenny, Julie A. Shah
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12169
• Pdf URL: https://arxiv.org/pdf/2412.12169
• Copy Paste: [[2412.12169]] Regulation of Language Models With Interpretability Will Likely Result In A Performance Trade-Off(https://arxiv.org/abs/2412.12169)
• Keywords: fair, interpretability
• Abstract: Regulation is increasingly cited as the most important and pressing concern in machine learning. However, it is currently unknown how to implement this, and perhaps more importantly, how it would effect model performance alongside human collaboration if actually realized. In this paper, we attempt to answer these questions by building a regulatable large-language model (LLM), and then quantifying how the additional constraints involved affect (1) model performance, alongside (2) human collaboration. Our empirical results reveal that it is possible to force an LLM to use human-defined features in a transparent way, but a "regulation performance trade-off" previously not considered reveals itself in the form of a 7.34% classification performance drop. Surprisingly however, we show that despite this, such systems actually improve human task performance speed and appropriate confidence in a realistic deployment setting compared to no AI assistance, thus paving a way for fair, regulatable AI, which benefits users.

Title: PickLLM: Context-Aware RL-Assisted Large Language Model Routing

• Authors: Dimitrios Sikeridis, Dennis Ramdass, Pranay Pareek
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12170
• Pdf URL: https://arxiv.org/pdf/2412.12170
• Copy Paste: [[2412.12170]] PickLLM: Context-Aware RL-Assisted Large Language Model Routing(https://arxiv.org/abs/2412.12170)
• Keywords: large language model
• Abstract: Recently, the number of off-the-shelf Large Language Models (LLMs) has exploded with many open-source options. This creates a diverse landscape regarding both serving options (e.g., inference on local hardware vs remote LLM APIs) and model heterogeneous expertise. However, it is hard for the user to efficiently optimize considering operational cost (pricing structures, expensive LLMs-as-a-service for large querying volumes), efficiency, or even per-case specific measures such as response accuracy, bias, or toxicity. Also, existing LLM routing solutions focus mainly on cost reduction, with response accuracy optimizations relying on non-generalizable supervised training, and ensemble approaches necessitating output computation for every considered LLM candidate. In this work, we tackle the challenge of selecting the optimal LLM from a model pool for specific queries with customizable objectives. We propose PickLLM, a lightweight framework that relies on Reinforcement Learning (RL) to route on-the-fly queries to available models. We introduce a weighted reward function that considers per-query cost, inference latency, and model response accuracy by a customizable scoring function. Regarding the learning algorithms, we explore two alternatives: PickLLM router acting as a learning automaton that utilizes gradient ascent to select a specific LLM, or utilizing stateless Q-learning to explore the set of LLMs and perform selection with a $\epsilon$-greedy approach. The algorithm converges to a single LLM for the remaining session queries. To evaluate, we utilize a pool of four LLMs and benchmark prompt-response datasets with different contexts. A separate scoring function is assessing response accuracy during the experiment. We demonstrate the speed of convergence for different learning rates and improvement in hard metrics such as cost per querying session and overall response latency.

Title: A NotSo Simple Way to Beat Simple Bench

• Authors: Soham Sane, Angus McLean
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12173
• Pdf URL: https://arxiv.org/pdf/2412.12173
• Copy Paste: [[2412.12173]] A NotSo Simple Way to Beat Simple Bench(https://arxiv.org/abs/2412.12173)
• Keywords: robust, large language model
• Abstract: This paper presents a novel framework for enhancing reasoning capabilities in large language models (LLMs) by leveraging iterative reasoning and feedback-driven methodologies. Building on the limitations identified in the SimpleBench benchmark, a dataset designed to evaluate logical coherence and real-world reasoning, we propose a multi-step prompting strategy coupled with global consistency checks to improve model accuracy and robustness. Through comparative analysis of state-of-the-art models, including Claude 3 Opus, Claude 3.5, GPT- 4o, and o1-preview, we demonstrate that iterative reasoning significantly enhances model performance, with improvements observed in both standard accuracy metrics (AVG@5) and a newly introduced metric, Extreme Averaging (EAG@5). Our results reveal model-specific strengths: Claude excels in maintaining logical consistency, while GPT-4o exhibits exploratory creativity but struggles with ambiguous prompts. By analyzing case studies and identifying gaps in spatial and temporal reasoning, we highlight areas for further refinement. The findings underscore the potential of structured reasoning frameworks to address inherent model limitations, irrespective of pretraining methodologies. This study lays the groundwork for integrating dynamic feedback mechanisms, adaptive restart strategies, and diverse evaluation metrics to advance LLM reasoning capabilities across complex and multi-domain problem spaces.

Title: Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoning

• Authors: Melanie Sclar, Jane Yu, Maryam Fazel-Zarandi, Yulia Tsvetkov, Yonatan Bisk, Yejin Choi, Asli Celikyilmaz
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12175
• Pdf URL: https://arxiv.org/pdf/2412.12175
• Copy Paste: [[2412.12175]] Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoning(https://arxiv.org/abs/2412.12175)
• Keywords: robust, large language model
• Abstract: Do large language models (LLMs) have theory of mind? A plethora of papers and benchmarks have been introduced to evaluate if current models have been able to develop this key ability of social intelligence. However, all rely on limited datasets with simple patterns that can potentially lead to problematic blind spots in evaluation and an overestimation of model capabilities. We introduce ExploreToM, the first framework to allow large-scale generation of diverse and challenging theory of mind data for robust training and evaluation. Our approach leverages an A* search over a custom domain-specific language to produce complex story structures and novel, diverse, yet plausible scenarios to stress test the limits of LLMs. Our evaluation reveals that state-of-the-art LLMs, such as Llama-3.1-70B and GPT-4o, show accuracies as low as 0% and 9% on ExploreToM-generated data, highlighting the need for more robust theory of mind evaluation. As our generations are a conceptual superset of prior work, fine-tuning on our data yields a 27-point accuracy improvement on the classic ToMi benchmark (Le et al., 2019). ExploreToM also enables uncovering underlying skills and factors missing for models to show theory of mind, such as unreliable state tracking or data imbalances, which may contribute to models' poor performance on benchmarks.

Title: Activation Sparsity Opportunities for Compressing General Large Language Models

• Authors: Nobel Dhar, Bobin Deng, Md Romyull Islam, Kazi Fahim Ahmad Nasif, Liang Zhao, Kun Suo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12178
• Pdf URL: https://arxiv.org/pdf/2412.12178
• Copy Paste: [[2412.12178]] Activation Sparsity Opportunities for Compressing General Large Language Models(https://arxiv.org/abs/2412.12178)
• Keywords: large language model
• Abstract: Deploying local AI models, such as Large Language Models (LLMs), to edge devices can substantially enhance devices' independent capabilities, alleviate the server's burden, and lower the response time. Owing to these tremendous potentials, many big tech companies have released several lightweight Small Language Models (SLMs) to bridge this gap. However, we still have huge motivations to deploy more powerful (LLMs) AI models on edge devices and enhance their smartness level. Unlike the conventional approaches for AI model compression, we investigate activation sparsity. The activation sparsity method is orthogonal and combinable with existing techniques to maximize compression rate while maintaining great accuracy. LLMs' Feed-Forward Network (FFN) components, which typically comprise a large proportion of parameters (around 3/2), ensure that our FFN optimizations would have a better chance of achieving effective compression. Moreover, our findings are beneficial to general LLMs and are not restricted to ReLU-based models. This work systematically investigates the tradeoff between enforcing activation sparsity and perplexity (accuracy) on state-of-the-art LLMs. Our empirical analysis demonstrates that we can obtain around 50% of main memory and computing reductions for critical FFN components with negligible accuracy degradation. This extra 50% sparsity does not naturally exist in the current LLMs, which require tuning LLMs' activation outputs by injecting zero-enforcing thresholds. To obtain the benefits of activation sparsity, we provide a guideline for the system architect for LLM prediction and prefetching. The success prediction allows the system to prefetch the necessary weights while omitting the inactive ones and their successors, therefore lowering cache and memory pollution and reducing LLM execution time on resource-constrained edge devices.

Title: Multi-Surrogate-Teacher Assistance for Representation Alignment in Fingerprint-based Indoor Localization

• Authors: Son Minh Nguyen, Linh Duy Tran, Duc Viet Le, Paul J.M Havinga
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12189
• Pdf URL: https://arxiv.org/pdf/2412.12189
• Copy Paste: [[2412.12189]] Multi-Surrogate-Teacher Assistance for Representation Alignment in Fingerprint-based Indoor Localization(https://arxiv.org/abs/2412.12189)
• Keywords: generative
• Abstract: Despite remarkable progress in knowledge transfer across visual and textual domains, extending these achievements to indoor localization, particularly for learning transferable representations among Received Signal Strength (RSS) fingerprint datasets, remains a challenge. This is due to inherent discrepancies among these RSS datasets, largely including variations in building structure, the input number and disposition of WiFi anchors. Accordingly, specialized networks, which were deprived of the ability to discern transferable representations, readily incorporate environment-sensitive clues into the learning process, hence limiting their potential when applied to specific RSS datasets. In this work, we propose a plug-and-play (PnP) framework of knowledge transfer, facilitating the exploitation of transferable representations for specialized networks directly on target RSS datasets through two main phases. Initially, we design an Expert Training phase, which features multiple surrogate generative teachers, all serving as a global adapter that homogenizes the input disparities among independent source RSS datasets while preserving their unique characteristics. In a subsequent Expert Distilling phase, we continue introducing a triplet of underlying constraints that requires minimizing the differences in essential knowledge between the specialized network and surrogate teachers through refining its representation learning on the target dataset. This process implicitly fosters a representational alignment in such a way that is less sensitive to specific environmental dynamics. Extensive experiments conducted on three benchmark WiFi RSS fingerprint datasets underscore the effectiveness of the framework that significantly exerts the full potential of specialized networks in localization.

Title: iMoT: Inertial Motion Transformer for Inertial Navigation

• Authors: Son Minh Nguyen, Linh Duy Tran, Duc Viet Le, Paul J.M Havinga
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.12190
• Pdf URL: https://arxiv.org/pdf/2412.12190
• Copy Paste: [[2412.12190]] iMoT: Inertial Motion Transformer for Inertial Navigation(https://arxiv.org/abs/2412.12190)
• Keywords: robust, transformer
• Abstract: We propose iMoT, an innovative Transformer-based inertial odometry method that retrieves cross-modal information from motion and rotation modalities for accurate positional estimation. Unlike prior work, during the encoding of the motion context, we introduce Progressive Series Decoupler at the beginning of each encoder layer to stand out critical motion events inherent in acceleration and angular velocity signals. To better aggregate cross-modal interactions, we present Adaptive Positional Encoding, which dynamically modifies positional embeddings for temporal discrepancies between different modalities. During decoding, we introduce a small set of learnable query motion particles as priors to model motion uncertainties within velocity segments. Each query motion particle is intended to draw cross-modal features dedicated to a specific motion mode, all taken together allowing the model to refine its understanding of motion dynamics effectively. Lastly, we design a dynamic scoring mechanism to stabilize iMoT's optimization by considering all aligned motion particles at the final decoding step, ensuring robust and accurate velocity segment estimation. Extensive evaluations on various inertial datasets demonstrate that iMoT significantly outperforms state-of-the-art methods in delivering superior robustness and accuracy in trajectory reconstruction.

Title: No Free Lunch for Defending Against Prefilling Attack by In-Context Learning

• Authors: Zhiyu Xue, Guangliang Liu, Bocheng Chen, Kristen Marie Johnson, Ramtin Pedarsani
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12192
• Pdf URL: https://arxiv.org/pdf/2412.12192
• Copy Paste: [[2412.12192]] No Free Lunch for Defending Against Prefilling Attack by In-Context Learning(https://arxiv.org/abs/2412.12192)
• Keywords: security, defense, attack, robust, large language model
• Abstract: The security of Large Language Models (LLMs) has become an important research topic since the emergence of ChatGPT. Though there have been various effective methods to defend against jailbreak attacks, prefilling attacks remain an unsolved and popular threat against open-sourced LLMs. In-Context Learning (ICL) offers a computationally efficient defense against various jailbreak attacks, yet no effective ICL methods have been developed to counter prefilling attacks. In this paper, we: (1) show that ICL can effectively defend against prefilling jailbreak attacks by employing adversative sentence structures within demonstrations; (2) characterize the effectiveness of this defense through the lens of model size, number of demonstrations, over-defense, integration with other jailbreak attacks, and the presence of safety alignment. Given the experimental results and our analysis, we conclude that there is no free lunch for defending against prefilling jailbreak attacks with ICL. On the one hand, current safety alignment methods fail to mitigate prefilling jailbreak attacks, but adversative structures within ICL demonstrations provide robust defense across various model sizes and complex jailbreak attacks. On the other hand, LLMs exhibit similar over-defensiveness when utilizing ICL demonstrations with adversative structures, and this behavior appears to be independent of model size.

Title: BlockDoor: Blocking Backdoor Based Watermarks in Deep Neural Networks

• Authors: Yi Hao Puah, Anh Tu Ngo, Nandish Chattopadhyay, Anupam Chattopadhyay
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12194
• Pdf URL: https://arxiv.org/pdf/2412.12194
• Copy Paste: [[2412.12194]] BlockDoor: Blocking Backdoor Based Watermarks in Deep Neural Networks(https://arxiv.org/abs/2412.12194)
• Keywords: protect, watermark
• Abstract: Adoption of machine learning models across industries have turned Neural Networks (DNNs) into a prized Intellectual Property (IP), which needs to be protected from being stolen or being used without authorization. This topic gave rise to multiple watermarking schemes, through which, one can establish the ownership of a model. Watermarking using backdooring is the most well established method available in the literature, with specific works demonstrating the difficulty in removing the watermarks, embedded as backdoors within the weights of the network. However, in our work, we have identified a critical flaw in the design of the watermark verification with backdoors, pertaining to the behaviour of the samples of the Trigger Set, which acts as the secret key. In this paper, we present BlockDoor, which is a comprehensive package of techniques that is used as a wrapper to block all three different kinds of Trigger samples, which are used in the literature as means to embed watermarks within the trained neural networks as backdoors. The framework implemented through BlockDoor is able to detect potential Trigger samples, through separate functions for adversarial noise based triggers, out-of-distribution triggers and random label based triggers. Apart from a simple Denial-of-Service for a potential Trigger sample, our approach is also able to modify the Trigger samples for correct machine learning functionality. Extensive evaluation of BlockDoor establishes that it is able to significantly reduce the watermark validation accuracy of the Trigger set by up to $98\%$ without compromising on functionality, delivering up to a less than $1\%$ drop on the clean samples. BlockDoor has been tested on multiple datasets and neural architectures.

Title: Embracing Large Language Models in Traffic Flow Forecasting

• Authors: Yusheng Zhao, Xiao Luo, Haomin Wen, Zhiping Xiao, Wei Ju, Ming Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12201
• Pdf URL: https://arxiv.org/pdf/2412.12201
• Copy Paste: [[2412.12201]] Embracing Large Language Models in Traffic Flow Forecasting(https://arxiv.org/abs/2412.12201)
• Keywords: large language model
• Abstract: Traffic flow forecasting aims to predict future traffic flows based on the historical traffic conditions and the road network. It is an important problem in intelligent transportation systems, with a plethora of methods been proposed. Existing efforts mainly focus on capturing and utilizing spatio-temporal dependencies to predict future traffic flows. Though promising, they fall short in adapting to test-time environmental changes of traffic conditions. To tackle this challenge, we propose to introduce large language models (LLMs) to help traffic flow forecasting and design a novel method named Large Language Model Enhanced Traffic Flow Predictor (LEAF). LEAF adopts two branches, capturing different spatio-temporal relations using graph and hypergraph structures respectively. The two branches are first pre-trained individually, and during test-time, they yield different predictions. Based on these predictions, a large language model is used to select the most likely result. Then, a ranking loss is applied as the learning objective to enhance the prediction ability of the two branches. Extensive experiments on several datasets demonstrate the effectiveness of the proposed LEAF.

Title: SEE: Sememe Entanglement Encoding for Transformer-bases Models Compression

• Authors: Jing Zhang, Shuzhen Sun, Peng Zhang, Guangxing Cao, Hui Gao, Xindian Ma, Nan Xu, Yuexian Hou
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12204
• Pdf URL: https://arxiv.org/pdf/2412.12204
• Copy Paste: [[2412.12204]] SEE: Sememe Entanglement Encoding for Transformer-bases Models Compression(https://arxiv.org/abs/2412.12204)
• Keywords: transformer, large language model
• Abstract: Transformer-based large language models exhibit groundbreaking capabilities, but their storage and computational costs are prohibitively high, limiting their application in resource-constrained scenarios. An effective approach is to eliminate redundant model parameters and computational costs while incorporating efficient expert-derived knowledge structures to achieve a balance between compression and performance. Therefore, we propose the \textit{Sememe Entanglement Encoding (SEE)} algorithm. Guided by expert prior knowledge, the model is compressed through the low-rank approximation idea. In Entanglement Embedding, basic semantic units such as sememes are represented as low-dimensional vectors, and then reconstructed into high-dimensional word embeddings through the combination of generalized quantum entanglement. We adapt the Sememe Entanglement Encoding algorithm to transformer-based models of different magnitudes. Experimental results indicate that our approach achieves stable performance while compressing model parameters and computational costs.

Title: Provably Secure Robust Image Steganography via Cross-Modal Error Correction

• Authors: Yuang Qi, Kejiang Chen, Na Zhao, Zijin Yang, Weiming Zhang
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.12206
• Pdf URL: https://arxiv.org/pdf/2412.12206
• Copy Paste: [[2412.12206]] Provably Secure Robust Image Steganography via Cross-Modal Error Correction(https://arxiv.org/abs/2412.12206)
• Keywords: secure, robust, extraction
• Abstract: The rapid development of image generation models has facilitated the widespread dissemination of generated images on social networks, creating favorable conditions for provably secure image steganography. However, existing methods face issues such as low quality of generated images and lack of semantic control in the generation process. To leverage provably secure steganography with more effective and high-performance image generation models, and to ensure that stego images can accurately extract secret messages even after being uploaded to social networks and subjected to lossy processing such as JPEG compression, we propose a high-quality, provably secure, and robust image steganography method based on state-of-the-art autoregressive (AR) image generation models using Vector-Quantized (VQ) tokenizers. Additionally, we employ a cross-modal error-correction framework that generates stego text from stego images to aid in restoring lossy images, ultimately enabling the extraction of secret messages embedded within the images. Extensive experiments have demonstrated that the proposed method provides advantages in stego quality, embedding capacity, and robustness, while ensuring provable undetectability.

Title: Finding a Wolf in Sheep's Clothing: Combating Adversarial Text-To-Image Prompts with Text Summarization

• Authors: Portia Cooper, Harshita Narnoli, Mihai Surdeanu
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12212
• Pdf URL: https://arxiv.org/pdf/2412.12212
• Copy Paste: [[2412.12212]] Finding a Wolf in Sheep's Clothing: Combating Adversarial Text-To-Image Prompts with Text Summarization(https://arxiv.org/abs/2412.12212)
• Keywords: attack, large language model
• Abstract: Text-to-image models are vulnerable to the stepwise "Divide-and-Conquer Attack" (DACA) that utilize a large language model to obfuscate inappropriate content in prompts by wrapping sensitive text in a benign narrative. To mitigate stepwise DACA attacks, we propose a two-layer method involving text summarization followed by binary classification. We assembled the Adversarial Text-to-Image Prompt (ATTIP) dataset ($N=940$), which contained DACA-obfuscated and non-obfuscated prompts. From the ATTIP dataset, we created two summarized versions: one generated by a small encoder model and the other by a large language model. Then, we used an encoder classifier and a GPT-4o classifier to perform content moderation on the summarized and unsummarized prompts. When compared with a classifier that operated over the unsummarized data, our method improved F1 score performance by 31%. Further, the highest recorded F1 score achieved (98%) was produced by the encoder classifier on a summarized ATTIP variant. This study indicates that pre-classification text summarization can inoculate content detection models against stepwise DACA obfuscations.

Title: The AI Black-Scholes: Finance-Informed Neural Network

• Authors: Amine M. Aboussalah, Xuanze Li, Cheng Chi, Raj Patel
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.12213
• Pdf URL: https://arxiv.org/pdf/2412.12213
• Copy Paste: [[2412.12213]] The AI Black-Scholes: Finance-Informed Neural Network(https://arxiv.org/abs/2412.12213)
• Keywords: robust, interpretability
• Abstract: In the realm of option pricing, existing models are typically classified into principle-driven methods, such as solving partial differential equations (PDEs) that pricing function satisfies, and data-driven approaches, such as machine learning (ML) techniques that parameterize the pricing function directly. While principle-driven models offer a rigorous theoretical framework, they often rely on unrealistic assumptions, such as asset processes adhering to fixed stochastic differential equations (SDEs). Moreover, they can become computationally intensive, particularly in high-dimensional settings when analytical solutions are not available and thus numerical solutions are needed. In contrast, data-driven models excel in capturing market data trends, but they often lack alignment with core financial principles, raising concerns about interpretability and predictive accuracy, especially when dealing with limited or biased datasets. This work proposes a hybrid approach to address these limitations by integrating the strengths of both principled and data-driven methodologies. Our framework combines the theoretical rigor and interpretability of PDE-based models with the adaptability of machine learning techniques, yielding a more versatile methodology for pricing a broad spectrum of options. We validate our approach across different volatility modeling approaches-both with constant volatility (Black-Scholes) and stochastic volatility (Heston), demonstrating that our proposed framework, Finance-Informed Neural Network (FINN), not only enhances predictive accuracy but also maintains adherence to core financial principles. FINN presents a promising tool for practitioners, offering robust performance across a variety of market conditions.

Title: Imagined Speech State Classification for Robust Brain-Computer Interface

• Authors: Byung-Kwan Ko, Jun-Young Kim, Seo-Hyun Lee
• Subjects: cs.LG, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.12215
• Pdf URL: https://arxiv.org/pdf/2412.12215
• Copy Paste: [[2412.12215]] Imagined Speech State Classification for Robust Brain-Computer Interface(https://arxiv.org/abs/2412.12215)
• Keywords: robust, extraction
• Abstract: This study examines the effectiveness of traditional machine learning classifiers versus deep learning models for detecting the imagined speech using electroencephalogram data. Specifically, we evaluated conventional machine learning techniques such as CSP-SVM and LDA-SVM classifiers alongside deep learning architectures such as EEGNet, ShallowConvNet, and DeepConvNet. Machine learning classifiers exhibited significantly lower precision and recall, indicating limited feature extraction capabilities and poor generalization between imagined speech and idle states. In contrast, deep learning models, particularly EEGNet, achieved the highest accuracy of 0.7080 and an F1 score of 0.6718, demonstrating their enhanced ability in automatic feature extraction and representation learning, essential for capturing complex neurophysiological patterns. These findings highlight the limitations of conventional machine learning approaches in brain-computer interface (BCI) applications and advocate for adopting deep learning methodologies to achieve more precise and reliable classification of detecting imagined speech. This foundational research contributes to the development of imagined speech-based BCI systems.

Title: Comprehensive Survey on Adversarial Examples in Cybersecurity: Impacts, Challenges, and Mitigation Strategies

• Authors: Li Li
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12217
• Pdf URL: https://arxiv.org/pdf/2412.12217
• Copy Paste: [[2412.12217]] Comprehensive Survey on Adversarial Examples in Cybersecurity: Impacts, Challenges, and Mitigation Strategies(https://arxiv.org/abs/2412.12217)
• Keywords: security, defense, attack, robust
• Abstract: Deep learning (DL) has significantly transformed cybersecurity, enabling advancements in malware detection, botnet identification, intrusion detection, user authentication, and encrypted traffic analysis. However, the rise of adversarial examples (AE) poses a critical challenge to the robustness and reliability of DL-based systems. These subtle, crafted perturbations can deceive models, leading to severe consequences like misclassification and system vulnerabilities. This paper provides a comprehensive review of the impact of AE attacks on key cybersecurity applications, highlighting both their theoretical and practical implications. We systematically examine the methods used to generate adversarial examples, their specific effects across various domains, and the inherent trade-offs attackers face between efficacy and resource efficiency. Additionally, we explore recent advancements in defense mechanisms, including gradient masking, adversarial training, and detection techniques, evaluating their potential to enhance model resilience. By summarizing cutting-edge research, this study aims to bridge the gap between adversarial research and practical security applications, offering insights to fortify the adoption of DL solutions in cybersecurity.

Title: Are Large Language Models Useful for Time Series Data Analysis?

• Authors: Francis Tang, Ying Ding
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12219
• Pdf URL: https://arxiv.org/pdf/2412.12219
• Copy Paste: [[2412.12219]] Are Large Language Models Useful for Time Series Data Analysis?(https://arxiv.org/abs/2412.12219)
• Keywords: large language model
• Abstract: Time series data plays a critical role across diverse domains such as healthcare, energy, and finance, where tasks like classification, anomaly detection, and forecasting are essential for informed decision-making. Recently, large language models (LLMs) have gained prominence for their ability to handle complex data and extract meaningful insights. This study investigates whether LLMs are effective for time series data analysis by comparing their performance with non-LLM-based approaches across three tasks: classification, anomaly detection, and forecasting. Through a series of experiments using GPT4TS and autoregressive models, we evaluate their performance on benchmark datasets and assess their accuracy, precision, and ability to generalize. Our findings indicate that while LLM-based methods excel in specific tasks like anomaly detection, their benefits are less pronounced in others, such as forecasting, where simpler models sometimes perform comparably or better. This research highlights the role of LLMs in time series analysis and lays the groundwork for future studies to systematically explore their applications and limitations in handling temporal data.

Title: Endangered Alert: A Field-Validated Self-Training Scheme for Detecting and Protecting Threatened Wildlife on Roads and Roadsides

• Authors: Kunming Li, Mao Shan, Stephany Berrio Perez, Katie Luo, Stewart Worrall
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12222
• Pdf URL: https://arxiv.org/pdf/2412.12222
• Copy Paste: [[2412.12222]] Endangered Alert: A Field-Validated Self-Training Scheme for Detecting and Protecting Threatened Wildlife on Roads and Roadsides(https://arxiv.org/abs/2412.12222)
• Keywords: protect, robust
• Abstract: Traffic accidents are a global safety concern, resulting in numerous fatalities each year. A considerable number of these deaths are caused by animal-vehicle collisions (AVCs), which not only endanger human lives but also present serious risks to animal populations. This paper presents an innovative self-training methodology aimed at detecting rare animals, such as the cassowary in Australia, whose survival is threatened by road accidents. The proposed method addresses critical real-world challenges, including acquiring and labelling sensor data for rare animal species in resource-limited environments. It achieves this by leveraging cloud and edge computing, and automatic data labelling to improve the detection performance of the field-deployed model iteratively. Our approach introduces Label-Augmentation Non-Maximum Suppression (LA-NMS), which incorporates a vision-language model (VLM) to enable automated data labelling. During a five-month deployment, we confirmed the method's robustness and effectiveness, resulting in improved object detection accuracy and increased prediction confidence. The source code is available: this https URL

Title: Can video generation replace cinematographers? Research on the cinematic language of generated video

• Authors: Xiaozhe Li, Kai WU, Siyi Yang, YiZhan Qu, Guohua.Zhang, Zhiyu Chen, Jiayao Li, Jiangchuan Mu, Xiaobin Hu, Wen Fang, Mingliang Xiong, Hao Deng, Qingwen Liu, Gang Li, Bin He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12223
• Pdf URL: https://arxiv.org/pdf/2412.12223
• Copy Paste: [[2412.12223]] Can video generation replace cinematographers? Research on the cinematic language of generated video(https://arxiv.org/abs/2412.12223)
• Keywords: robust, diffusion
• Abstract: Recent advancements in text-to-video (T2V) generation have leveraged diffusion models to enhance the visual coherence of videos generated from textual descriptions. However, most research has primarily focused on object motion, with limited attention given to cinematic language in videos, which is crucial for cinematographers to convey emotion and narrative pacing. To address this limitation, we propose a threefold approach to enhance the ability of T2V models to generate controllable cinematic language. Specifically, we introduce a cinematic language dataset that encompasses shot framing, angle, and camera movement, enabling models to learn diverse cinematic styles. Building on this, to facilitate robust cinematic alignment evaluation, we present CameraCLIP, a model fine-tuned on the proposed dataset that excels in understanding complex cinematic language in generated videos and can further provide valuable guidance in the multi-shot composition process. Finally, we propose CLIPLoRA, a cost-guided dynamic LoRA composition method that facilitates smooth transitions and realistic blending of cinematic language by dynamically fusing multiple pre-trained cinematic LoRAs within a single video. Our experiments demonstrate that CameraCLIP outperforms existing models in assessing the alignment between cinematic language and video, achieving an R@1 score of 0.81. Additionally, CLIPLoRA improves the ability for multi-shot composition, potentially bridging the gap between automatically generated videos and those shot by professional cinematographers.

Title: EDformer: Embedded Decomposition Transformer for Interpretable Multivariate Time Series Predictions

• Authors: Sanjay Chakraborty, Ibrahim Delibasoglu, Fredrik Heintz
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12227
• Pdf URL: https://arxiv.org/pdf/2412.12227
• Copy Paste: [[2412.12227]] EDformer: Embedded Decomposition Transformer for Interpretable Multivariate Time Series Predictions(https://arxiv.org/abs/2412.12227)
• Keywords: interpretability, explainability, transformer
• Abstract: Time series forecasting is a crucial challenge with significant applications in areas such as weather prediction, stock market analysis, and scientific simulations. This paper introduces an embedded decomposed transformer, 'EDformer', for multivariate time series forecasting tasks. Without altering the fundamental elements, we reuse the Transformer architecture and consider the capable functions of its constituent parts in this work. Edformer first decomposes the input multivariate signal into seasonal and trend components. Next, the prominent multivariate seasonal component is reconstructed across the reverse dimensions, followed by applying the attention mechanism and feed-forward network in the encoder stage. In particular, the feed-forward network is used for each variable frame to learn nonlinear representations, while the attention mechanism uses the time points of individual seasonal series embedded within variate frames to capture multivariate correlations. Therefore, the trend signal is added with projection and performs the final forecasting. The EDformer model obtains state-of-the-art predicting results in terms of accuracy and efficiency on complex real-world time series datasets. This paper also addresses model explainability techniques to provide insights into how the model makes its predictions and why specific features or time steps are important, enhancing the interpretability and trustworthiness of the forecasting results.

Title: You Only Submit One Image to Find the Most Suitable Generative Model

• Authors: Zhi Zhou, Lan-Zhe Guo, Peng-Xiao Song, Yu-Feng Li
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12232
• Pdf URL: https://arxiv.org/pdf/2412.12232
• Copy Paste: [[2412.12232]] You Only Submit One Image to Find the Most Suitable Generative Model(https://arxiv.org/abs/2412.12232)
• Keywords: generative
• Abstract: Deep generative models have achieved promising results in image generation, and various generative model hubs, e.g., Hugging Face and Civitai, have been developed that enable model developers to upload models and users to download models. However, these model hubs lack advanced model management and identification mechanisms, resulting in users only searching for models through text matching, download sorting, etc., making it difficult to efficiently find the model that best meets user requirements. In this paper, we propose a novel setting called Generative Model Identification (GMI), which aims to enable the user to identify the most appropriate generative model(s) for the user's requirements from a large number of candidate models efficiently. To our best knowledge, it has not been studied yet. In this paper, we introduce a comprehensive solution consisting of three pivotal modules: a weighted Reduced Kernel Mean Embedding (RKME) framework for capturing the generated image distribution and the relationship between images and prompts, a pre-trained vision-language model aimed at addressing dimensionality challenges, and an image interrogator designed to tackle cross-modality issues. Extensive empirical results demonstrate the proposal is both efficient and effective. For example, users only need to submit a single example image to describe their requirements, and the model platform can achieve an average top-4 identification accuracy of more than 80%.

Title: OmniPrism: Learning Disentangled Visual Concept for Image Generation

• Authors: Yangyang Li, Daqing Liu, Wu Liu, Allen He, Xinchen Liu, Yongdong Zhang, Guoqing Jin
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12242
• Pdf URL: https://arxiv.org/pdf/2412.12242
• Copy Paste: [[2412.12242]] OmniPrism: Learning Disentangled Visual Concept for Image Generation(https://arxiv.org/abs/2412.12242)
• Keywords: diffusion
• Abstract: Creative visual concept generation often draws inspiration from specific concepts in a reference image to produce relevant outcomes. However, existing methods are typically constrained to single-aspect concept generation or are easily disrupted by irrelevant concepts in multi-aspect concept scenarios, leading to concept confusion and hindering creative generation. To address this, we propose OmniPrism, a visual concept disentangling approach for creative image generation. Our method learns disentangled concept representations guided by natural language and trains a diffusion model to incorporate these concepts. We utilize the rich semantic space of a multimodal extractor to achieve concept disentanglement from given images and concept guidance. To disentangle concepts with different semantics, we construct a paired concept disentangled dataset (PCD-200K), where each pair shares the same concept such as content, style, and composition. We learn disentangled concept representations through our contrastive orthogonal disentangled (COD) training pipeline, which are then injected into additional diffusion cross-attention layers for generation. A set of block embeddings is designed to adapt each block's concept domain in the diffusion models. Extensive experiments demonstrate that our method can generate high-quality, concept-disentangled results with high fidelity to text prompts and desired concepts.

Title: Emergence of Abstractions: Concept Encoding and Decoding Mechanism for In-Context Learning in Transformers

• Authors: Seungwook Han, Jinyeop Song, Jeff Gore, Pulkit Agrawal
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12276
• Pdf URL: https://arxiv.org/pdf/2412.12276
• Copy Paste: [[2412.12276]] Emergence of Abstractions: Concept Encoding and Decoding Mechanism for In-Context Learning in Transformers(https://arxiv.org/abs/2412.12276)
• Keywords: transformer, large language model
• Abstract: Humans distill complex experiences into fundamental abstractions that enable rapid learning and adaptation. Similarly, autoregressive transformers exhibit adaptive learning through in-context learning (ICL), which begs the question of how. In this paper, we propose \textbf{concept encoding-decoding mechanism} to explain ICL by studying how transformers form and use internal abstractions in their representations. On synthetic ICL tasks, we analyze the training dynamics of a small transformer and report the coupled emergence of concept encoding and decoding. As the model learns to encode different latent concepts (e.g., ``Finding the first noun in a sentence.") into distinct, separable representations, it concureently builds conditional decoding algorithms and improve its ICL performance. We validate the existence of this mechanism across pretrained models of varying scales (Gemma-2 2B/9B/27B, Llama-3.1 8B/70B). Further, through mechanistic interventions and controlled finetuning, we demonstrate that the quality of concept encoding is causally related and predictive of ICL performance. Our empirical insights shed light into better understanding the success and failure modes of large language models via their representations.

Title: Towards a Universal Synthetic Video Detector: From Face or Background Manipulations to Fully AI-Generated Content

• Authors: Rohit Kundu, Hao Xiong, Vishal Mohanty, Athula Balachandran, Amit K. Roy-Chowdhury
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12278
• Pdf URL: https://arxiv.org/pdf/2412.12278
• Copy Paste: [[2412.12278]] Towards a Universal Synthetic Video Detector: From Face or Background Manipulations to Fully AI-Generated Content(https://arxiv.org/abs/2412.12278)
• Keywords: transformer, generative
• Abstract: Existing DeepFake detection techniques primarily focus on facial manipulations, such as face-swapping or lip-syncing. However, advancements in text-to-video (T2V) and image-to-video (I2V) generative models now allow fully AI-generated synthetic content and seamless background alterations, challenging face-centric detection methods and demanding more versatile approaches. To address this, we introduce the \underline{U}niversal \underline{N}etwork for \underline{I}dentifying \underline{T}ampered and synth\underline{E}tic videos (\texttt{UNITE}) model, which, unlike traditional detectors, captures full-frame manipulations. \texttt{UNITE} extends detection capabilities to scenarios without faces, non-human subjects, and complex background modifications. It leverages a transformer-based architecture that processes domain-agnostic features extracted from videos via the SigLIP-So400M foundation model. Given limited datasets encompassing both facial/background alterations and T2V/I2V content, we integrate task-irrelevant data alongside standard DeepFake datasets in training. We further mitigate the model's tendency to over-focus on faces by incorporating an attention-diversity (AD) loss, which promotes diverse spatial attention across video frames. Combining AD loss with cross-entropy improves detection performance across varied contexts. Comparative evaluations demonstrate that \texttt{UNITE} outperforms state-of-the-art detectors on datasets (in cross-data settings) featuring face/background manipulations and fully synthetic T2V/I2V videos, showcasing its adaptability and generalizable detection capabilities.

Title: Unanswerability Evaluation for Retreival Augmented Generation

• Authors: Xiangyu Peng, Prafulla Kumar Choubey, Caiming Xiong, Chien-Sheng Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12300
• Pdf URL: https://arxiv.org/pdf/2412.12300
• Copy Paste: [[2412.12300]] Unanswerability Evaluation for Retreival Augmented Generation(https://arxiv.org/abs/2412.12300)
• Keywords: robust
• Abstract: Existing evaluation frameworks for retrieval-augmented generation (RAG) systems focus on answerable queries, but they overlook the importance of appropriately rejecting unanswerable requests. In this paper, we introduce UAEval4RAG, a framework designed to evaluate whether RAG systems can handle unanswerable queries effectively. We define a taxonomy with six unanswerable categories, and UAEval4RAG automatically synthesizes diverse and challenging queries for any given knowledge base with unanswered ratio and acceptable ratio metrics. We conduct experiments with various RAG components, including retrieval models, rewriting methods, rerankers, language models, and prompting strategies, and reveal hidden trade-offs in performance of RAG systems. Our findings highlight the critical role of component selection and prompt design in optimizing RAG systems to balance the accuracy of answerable queries with high rejection rates of unanswerable ones. UAEval4RAG provides valuable insights and tools for developing more robust and reliable RAG systems.

Title: Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion

• Authors: Jianqing Zhu, Huang Huang, Zhihang Lin, Juhao Liang, Zhengyang Tang, Khalid Almubarak, Abdulmohsen Alharthik, Bang An, Juncai He, Xiangbo Wu, Fei Yu, Junying Chen, Zhuoheng Ma, Yuhao Du, He Zhang, Emad A. Alghamdi, Lian Zhang, Ruoyu Sun, Haizhou Li, Benyou Wang, Jinchao Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12310
• Pdf URL: https://arxiv.org/pdf/2412.12310
• Copy Paste: [[2412.12310]] Second Language (Arabic) Acquisition of LLMs via Progressive Vocabulary Expansion(https://arxiv.org/abs/2412.12310)
• Keywords: large language model
• Abstract: This paper addresses the critical need for democratizing large language models (LLM) in the Arab world, a region that has seen slower progress in developing models comparable to state-of-the-art offerings like GPT-4 or ChatGPT 3.5, due to a predominant focus on mainstream languages (e.g., English and Chinese). One practical objective for an Arabic LLM is to utilize an Arabic-specific vocabulary for the tokenizer that could speed up decoding. However, using a different vocabulary often leads to a degradation of learned knowledge since many words are initially out-of-vocabulary (OOV) when training starts. Inspired by the vocabulary learning during Second Language (Arabic) Acquisition for humans, the released AraLLaMA employs progressive vocabulary expansion, which is implemented by a modified BPE algorithm that progressively extends the Arabic subwords in its dynamic vocabulary during training, thereby balancing the OOV ratio at every stage. The ablation study demonstrated the effectiveness of Progressive Vocabulary Expansion. Moreover, AraLLaMA achieves decent performance comparable to the best Arabic LLMs across a variety of Arabic benchmarks. Models, training data, benchmarks, and codes will be all open-sourced.

Title: F-RBA: A Federated Learning-based Framework for Risk-based Authentication

• Authors: Hamidreza Fereidouni, Abdelhakim Senhaji Hafid, Dimitrios Makrakis, Yaser Baseri
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12324
• Pdf URL: https://arxiv.org/pdf/2412.12324
• Copy Paste: [[2412.12324]] F-RBA: A Federated Learning-based Framework for Risk-based Authentication(https://arxiv.org/abs/2412.12324)
• Keywords: security, privacy, protect, attack, robust, federate
• Abstract: The proliferation of Internet services has led to an increasing need to protect private data. User authentication serves as a crucial mechanism to ensure data security. Although robust authentication forms the cornerstone of remote service security, it can still leave users vulnerable to credential disclosure, device-theft attacks, session hijacking, and inadequate adaptive security measures. Risk-based Authentication (RBA) emerges as a potential solution, offering a multi-level authentication approach that enhances user experience without compromising security. In this paper, we propose a Federated Risk-based Authentication (F-RBA) framework that leverages Federated Learning to ensure privacy-centric training, keeping user data local while distributing learning across devices. Whereas traditional approaches rely on centralized storage, F-RBA introduces a distributed architecture where risk assessment occurs locally on users' devices. The framework's core innovation lies in its similarity-based feature engineering approach, which addresses the heterogeneous data challenges inherent in federated settings, a significant advancement for distributed authentication. By facilitating real-time risk evaluation across devices while maintaining unified user profiles, F-RBA achieves a balance between data protection, security, and scalability. Through its federated approach, F-RBA addresses the cold-start challenge in risk model creation, enabling swift adaptation to new users without compromising security. Empirical evaluation using a real-world multi-user dataset demonstrates the framework's effectiveness, achieving a superior true positive rate for detecting suspicious logins compared to conventional unsupervised anomaly detection models. This research introduces a new paradigm for privacy-focused RBA in distributed digital environments, facilitating advancements in federated security systems.

Title: Krony-PT: GPT2 compressed with Kronecker Products

• Authors: M. Ayoub Ben Ayad, Jelena Mitrovic, Michael Granitzer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12351
• Pdf URL: https://arxiv.org/pdf/2412.12351
• Copy Paste: [[2412.12351]] Krony-PT: GPT2 compressed with Kronecker Products(https://arxiv.org/abs/2412.12351)
• Keywords: transformer
• Abstract: We introduce Krony-PT, a compression technique of GPT2 \citep{radford2019language} based on Kronecker Products. We specifically target the MLP layers of each transformer layer, and systematically compress the feed forward layer matrices to various degrees. We introduce a modified Van Loan decomposition to initialize the new factors, and also introduce a new pruning-based initialization trick. Our method compresses the original 124M parameter GPT2 to various smaller models, with 80M being the smallest, and 96M being the largest compressed model. Our 81M model variant outperforms distilgpt2 on next-token prediction on all standard language modeling datasets, and shows competitive scores or performs on par with other Kronecker Products based compressed models of GPT2 that are significantly higher in size.

Title: BioRAGent: A Retrieval-Augmented Generation System for Showcasing Generative Query Expansion and Domain-Specific Search for Scientific Q&A

• Authors: Samy Ateia, Udo Kruschwitz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12358
• Pdf URL: https://arxiv.org/pdf/2412.12358
• Copy Paste: [[2412.12358]] BioRAGent: A Retrieval-Augmented Generation System for Showcasing Generative Query Expansion and Domain-Specific Search for Scientific Q&A(https://arxiv.org/abs/2412.12358)
• Keywords: extraction, generative, large language model
• Abstract: We present BioRAGent, an interactive web-based retrieval-augmented generation (RAG) system for biomedical question answering. The system uses large language models (LLMs) for query expansion, snippet extraction, and answer generation while maintaining transparency through citation links to the source documents and displaying generated queries for further editing. Building on our successful participation in the BioASQ 2024 challenge, we demonstrate how few-shot learning with LLMs can be effectively applied for a professional search setting. The system supports both direct short paragraph style responses and responses with inline citations. Our demo is available online, and the source code is publicly accessible through GitHub.

Title: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering

• Authors: Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, Yunpu Ma
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12359
• Pdf URL: https://arxiv.org/pdf/2412.12359
• Copy Paste: [[2412.12359]] Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering(https://arxiv.org/abs/2412.12359)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have significantly advanced visual tasks by integrating visual representations into large language models (LLMs). The textual modality, inherited from LLMs, equips MLLMs with abilities like instruction following and in-context learning. In contrast, the visual modality enhances performance in downstream tasks by leveraging rich semantic content, spatial information, and grounding capabilities. These intrinsic modalities work synergistically across various visual tasks. Our research initially reveals a persistent imbalance between these modalities, with text often dominating output generation during visual instruction tuning. This imbalance occurs when using both full fine-tuning and parameter-efficient fine-tuning (PEFT) methods. We then found that re-balancing these modalities can significantly reduce the number of trainable parameters required, inspiring a direction for further optimizing visual instruction tuning. We introduce Modality Linear Representation-Steering (MoReS) to achieve the goal. MoReS effectively re-balances the intrinsic modalities throughout the model, where the key idea is to steer visual representations through linear transformations in the visual subspace across each model layer. To validate our solution, we composed LLaVA Steering, a suite of models integrated with the proposed MoReS method. Evaluation results show that the composed LLaVA Steering models require, on average, 500 times fewer trainable parameters than LoRA needs while still achieving comparable performance across three visual benchmarks and eight visual question-answering tasks. Last, we present the LLaVA Steering Factory, an in-house developed platform that enables researchers to quickly customize various MLLMs with component-based architecture for seamlessly integrating state-of-the-art models, and evaluate their intrinsic modality imbalance.

Title: Scam Detection for Ethereum Smart Contracts: Leveraging Graph Representation Learning for Secure Blockchain

• Authors: Yihong Jin, Ze Yang
• Subjects: cs.LG, cs.AI, cs.DC, cs.SI
• Abstract URL: https://arxiv.org/abs/2412.12370
• Pdf URL: https://arxiv.org/pdf/2412.12370
• Copy Paste: [[2412.12370]] Scam Detection for Ethereum Smart Contracts: Leveraging Graph Representation Learning for Secure Blockchain(https://arxiv.org/abs/2412.12370)
• Keywords: secure, security, robust
• Abstract: The detection of scams within Ethereum smart contracts is a critical challenge due to their increasing exploitation for fraudulent activities, leading to significant financial and reputational damages. Existing detection methods often rely on contract code analysis or manually extracted features, which suffer from scalability and adaptability limitations. In this study, we introduce an innovative method that leverages graph representation learning to examine transaction patterns and identify fraudulent contracts. By transforming Ethereum transaction data into graph structures and employing advanced machine learning models, we achieve robust classification performance. Our method addresses label imbalance through SMOTE-ENN techniques and evaluates models like Multi-Layer Perceptron (MLP) and Graph Convolutional Networks (GCN). Experimental results indicate that the MLP model surpasses the GCN in this context, with real-world evaluations aligning closely with domain-specific analyses. This study provides a scalable and effective solution for enhancing trust and security in the Ethereum ecosystem.

Title: Privacy in Metalearning and Multitask Learning: Modeling and Separations

• Authors: Maryam Aliakbarpour, Konstantina Bairaktari, Adam Smith, Marika Swanberg, Jonathan Ullman
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12374
• Pdf URL: https://arxiv.org/pdf/2412.12374
• Copy Paste: [[2412.12374]] Privacy in Metalearning and Multitask Learning: Modeling and Separations(https://arxiv.org/abs/2412.12374)
• Keywords: privacy, attack
• Abstract: Model personalization allows a set of individuals, each facing a different learning task, to train models that are more accurate for each person than those they could develop individually. The goals of personalization are captured in a variety of formal frameworks, such as multitask learning and metalearning. Combining data for model personalization poses risks for privacy because the output of an individual's model can depend on the data of other individuals. In this work we undertake a systematic study of differentially private personalized learning. Our first main contribution is to construct a taxonomy of formal frameworks for private personalized learning. This taxonomy captures different formal frameworks for learning as well as different threat models for the attacker. Our second main contribution is to prove separations between the personalized learning problems corresponding to different choices. In particular, we prove a novel separation between private multitask learning and private metalearning.

Title: Interpretable LLM-based Table Question Answering

• Authors: Giang (Dexter)Nguyen, Ivan Brugere, Shubham Sharma, Sanjay Kariyappa, Anh Totti Nguyen, Freddy Lecue
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12386
• Pdf URL: https://arxiv.org/pdf/2412.12386
• Copy Paste: [[2412.12386]] Interpretable LLM-based Table Question Answering(https://arxiv.org/abs/2412.12386)
• Keywords: interpretability, large language model
• Abstract: Interpretability for Table Question Answering (Table QA) is critical, particularly in high-stakes industries like finance or healthcare. Although recent approaches using Large Language Models (LLMs) have significantly improved Table QA performance, their explanations for how the answers are generated are ambiguous. To fill this gap, we introduce Plan-of-SQLs ( or POS), an interpretable, effective, and efficient approach to Table QA that answers an input query solely with SQL executions. Through qualitative and quantitative evaluations with human and LLM judges, we show that POS is most preferred among explanation methods, helps human users understand model decision boundaries, and facilitates model success and error identification. Furthermore, when evaluated in standard benchmarks (TabFact, WikiTQ, and FetaQA), POS achieves competitive or superior accuracy compared to existing methods, while maintaining greater efficiency by requiring significantly fewer LLM calls and database queries.

Title: Efficient Scaling of Diffusion Transformers for Text-to-Image Generation

• Authors: Hao Li, Shamit Lal, Zhiheng Li, Yusheng Xie, Ying Wang, Yang Zou, Orchid Majumder, R. Manmatha, Zhuowen Tu, Stefano Ermon, Stefano Soatto, Ashwin Swaminathan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12391
• Pdf URL: https://arxiv.org/pdf/2412.12391
• Copy Paste: [[2412.12391]] Efficient Scaling of Diffusion Transformers for Text-to-Image Generation(https://arxiv.org/abs/2412.12391)
• Keywords: diffusion, transformer
• Abstract: We empirically study the scaling properties of various Diffusion Transformers (DiTs) for text-to-image generation by performing extensive and rigorous ablations, including training scaled DiTs ranging from 0.3B upto 8B parameters on datasets up to 600M images. We find that U-ViT, a pure self-attention based DiT model provides a simpler design and scales more effectively in comparison with cross-attention based DiT variants, which allows straightforward expansion for extra conditions and other modalities. We identify a 2.3B U-ViT model can get better performance than SDXL UNet and other DiT variants in controlled setting. On the data scaling side, we investigate how increasing dataset size and enhanced long caption improve the text-image alignment performance and the learning efficiency.

Title: MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors

• Authors: Riku Murai, Eric Dexheimer, Andrew J. Davison
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.12392
• Pdf URL: https://arxiv.org/pdf/2412.12392
• Copy Paste: [[2412.12392]] MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors(https://arxiv.org/abs/2412.12392)
• Keywords: robust
• Abstract: We present a real-time monocular dense SLAM system designed bottom-up from MASt3R, a two-view 3D reconstruction and matching prior. Equipped with this strong prior, our system is robust on in-the-wild video sequences despite making no assumption on a fixed or parametric camera model beyond a unique camera centre. We introduce efficient methods for pointmap matching, camera tracking and local fusion, graph construction and loop closure, and second-order global optimisation. With known calibration, a simple modification to the system achieves state-of-the-art performance across various benchmarks. Altogether, we propose a plug-and-play monocular SLAM system capable of producing globally-consistent poses and dense geometry while operating at 15 FPS.

Title: Causally Consistent Normalizing Flow

• Authors: Qingyang Zhou, Kangjie Lu, Meng Xu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12401
• Pdf URL: https://arxiv.org/pdf/2412.12401
• Copy Paste: [[2412.12401]] Causally Consistent Normalizing Flow(https://arxiv.org/abs/2412.12401)
• Keywords: fair, generative
• Abstract: Causal inconsistency arises when the underlying causal graphs captured by generative models like \textit{Normalizing Flows} (NFs) are inconsistent with those specified in causal models like \textit{Struct Causal Models} (SCMs). This inconsistency can cause unwanted issues including the unfairness problem. Prior works to achieve causal consistency inevitably compromise the expressiveness of their models by disallowing hidden layers. In this work, we introduce a new approach: \textbf{C}ausally \textbf{C}onsistent \textbf{N}ormalizing \textbf{F}low (CCNF). To the best of our knowledge, CCNF is the first causally consistent generative model that can approximate any distribution with multiple layers. CCNF relies on two novel constructs: a sequential representation of SCMs and partial causal transformations. These constructs allow CCNF to inherently maintain causal consistency without sacrificing expressiveness. CCNF can handle all forms of causal inference tasks, including interventions and counterfactuals. Through experiments, we show that CCNF outperforms current approaches in causal inference. We also empirically validate the practical utility of CCNF by applying it to real-world datasets and show how CCNF addresses challenges like unfairness effectively.

Title: Characterizing the Networks Sending Enterprise Phishing Emails

• Authors: Elisa Luo, Liane Young, Grant Ho, M. H. Afifi, Marco Schweighauser, Ethan Katz-Bassett, Asaf Cidon
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.12403
• Pdf URL: https://arxiv.org/pdf/2412.12403
• Copy Paste: [[2412.12403]] Characterizing the Networks Sending Enterprise Phishing Emails(https://arxiv.org/abs/2412.12403)
• Keywords: security, defense, attack
• Abstract: Phishing attacks on enterprise employees present one of the most costly and potent threats to organizations. We explore an understudied facet of enterprise phishing attacks: the email relay infrastructure behind successfully delivered phishing emails. We draw on a dataset spanning one year across thousands of enterprises, billions of emails, and over 800,000 delivered phishing attacks. Our work sheds light on the network origins of phishing emails received by real-world enterprises, differences in email traffic we observe from networks sending phishing emails, and how these characteristics change over time. Surprisingly, we find that over one-third of the phishing email in our dataset originates from highly reputable networks, including Amazon and Microsoft. Their total volume of phishing email is consistently high across multiple months in our dataset, even though the overwhelming majority of email sent by these networks is benign. In contrast, we observe that a large portion of phishing emails originate from networks where the vast majority of emails they send are phishing, but their email traffic is not consistent over time. Taken together, our results explain why no singular defense strategy, such as static blocklists (which are commonly used in email security filters deployed by organizations in our dataset), is effective at blocking enterprise phishing. Based on our offline analysis, we partnered with a large email security company to deploy a classifier that uses dynamically updated network-based features. In a production environment over a period of 4.5 months, our new detector was able to identify 3-5% more enterprise email attacks that were previously undetected by the company's existing classifiers.

Title: DeepSN: A Sheaf Neural Framework for Influence Maximization

• Authors: Asela Hevapathige, Qing Wang, Ahad N. Zehmakan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12416
• Pdf URL: https://arxiv.org/pdf/2412.12416
• Copy Paste: [[2412.12416]] DeepSN: A Sheaf Neural Framework for Influence Maximization(https://arxiv.org/abs/2412.12416)
• Keywords: diffusion
• Abstract: Influence maximization is key topic in data mining, with broad applications in social network analysis and viral marketing. In recent years, researchers have increasingly turned to machine learning techniques to address this problem. They have developed methods to learn the underlying diffusion processes in a data-driven manner, which enhances the generalizability of the solution, and have designed optimization objectives to identify the optimal seed set. Nonetheless, two fundamental gaps remain unsolved: (1) Graph Neural Networks (GNNs) are increasingly used to learn diffusion models, but in their traditional form, they often fail to capture the complex dynamics of influence diffusion, (2) Designing optimization objectives is challenging due to combinatorial explosion when solving this problem. To address these challenges, we propose a novel framework, DeepSN. Our framework employs sheaf neural diffusion to learn diverse influence patterns in a data-driven, end-to-end manner, providing enhanced separability in capturing diffusion characteristics. We also propose an optimization technique that accounts for overlapping influence between vertices, which helps to reduce the search space and identify the optimal seed set effectively and efficiently. Finally, we conduct extensive experiments on both synthetic and real-world datasets to demonstrate the effectiveness of our framework.

Title: Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments

• Authors: Tuka Alhanai, Adam Kasumovic, Mohammad Ghassemi, Aven Zitzelberger, Jessica Lundin, Guillaume Chabot-Couture
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12417
• Pdf URL: https://arxiv.org/pdf/2412.12417
• Copy Paste: [[2412.12417]] Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments(https://arxiv.org/abs/2412.12417)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown remarkable performance across various tasks, yet significant disparities remain for non-English languages, and especially native African languages. This paper addresses these disparities by creating approximately 1 million human-translated words of new benchmark data in 8 low-resource African languages, covering a population of over 160 million speakers of: Amharic, Bambara, Igbo, Sepedi (Northern Sotho), Shona, Sesotho (Southern Sotho), Setswana, and Tsonga. Our benchmarks are translations of Winogrande and three sections of MMLU: college medicine, clinical knowledge, and virology. Using the translated benchmarks, we report previously unknown performance gaps between state-of-the-art (SOTA) LLMs in English and African languages. Finally, using results from over 400 fine-tuned models, we explore several methods to reduce the LLM performance gap, including high-quality dataset fine-tuning (using an LLM-as-an-Annotator), cross-lingual transfer, and cultural appropriateness adjustments. Key findings include average mono-lingual improvements of 5.6% with fine-tuning (with 5.4% average mono-lingual improvements when using high-quality data over low-quality data), 2.9% average gains from cross-lingual transfer, and a 3.0% out-of-the-box performance boost on culturally appropriate questions. The publicly available benchmarks, translations, and code from this study support further research and development aimed at creating more inclusive and effective language technologies.

Title: Assessing the Limitations of Large Language Models in Clinical Fact Decomposition

• Authors: Monica Munnangi, Akshay Swaminathan, Jason Alan Fries, Jenelle Jindal, Sanjana Narayanan, Ivan Lopez, Lucia Tu, Philip Chung, Jesutofunmi A. Omiye, Mehr Kashyap, Nigam Shah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12422
• Pdf URL: https://arxiv.org/pdf/2412.12422
• Copy Paste: [[2412.12422]] Assessing the Limitations of Large Language Models in Clinical Fact Decomposition(https://arxiv.org/abs/2412.12422)
• Keywords: large language model
• Abstract: Verifying factual claims is critical for using large language models (LLMs) in healthcare. Recent work has proposed fact decomposition, which uses LLMs to rewrite source text into concise sentences conveying a single piece of information, as an approach for fine-grained fact verification. Clinical documentation poses unique challenges for fact decomposition due to dense terminology and diverse note types. To explore these challenges, we present FactEHR, a dataset consisting of full document fact decompositions for 2,168 clinical notes spanning four types from three hospital systems. Our evaluation, including review by clinicians, highlights significant variability in the quality of fact decomposition for four commonly used LLMs, with some LLMs generating 2.6x more facts per sentence than others. The results underscore the need for better LLM capabilities to support factual verification in clinical text. To facilitate future research in this direction, we plan to release our code at \url{this https URL}.

Title: GG-SSMs: Graph-Generating State Space Models

• Authors: Nikola Zubić, Davide Scaramuzza
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12423
• Pdf URL: https://arxiv.org/pdf/2412.12423
• Copy Paste: [[2412.12423]] GG-SSMs: Graph-Generating State Space Models(https://arxiv.org/abs/2412.12423)
• Keywords: robust
• Abstract: State Space Models (SSMs) are powerful tools for modeling sequential data in computer vision and time series analysis domains. However, traditional SSMs are limited by fixed, one-dimensional sequential processing, which restricts their ability to model non-local interactions in high-dimensional data. While methods like Mamba and VMamba introduce selective and flexible scanning strategies, they rely on predetermined paths, which fails to efficiently capture complex dependencies. We introduce Graph-Generating State Space Models (GG-SSMs), a novel framework that overcomes these limitations by dynamically constructing graphs based on feature relationships. Using Chazelle's Minimum Spanning Tree algorithm, GG-SSMs adapt to the inherent data structure, enabling robust feature propagation across dynamically generated graphs and efficiently modeling complex dependencies. We validate GG-SSMs on 11 diverse datasets, including event-based eye-tracking, ImageNet classification, optical flow estimation, and six time series datasets. GG-SSMs achieve state-of-the-art performance across all tasks, surpassing existing methods by significant margins. Specifically, GG-SSM attains a top-1 accuracy of 84.9% on ImageNet, outperforming prior SSMs by 1%, reducing the KITTI-15 error rate to 2.77%, and improving eye-tracking detection rates by up to 0.33% with fewer parameters. These results demonstrate that dynamic scanning based on feature relationships significantly improves SSMs' representational power and efficiency, offering a versatile tool for various applications in computer vision and beyond.

Title: Numerical Pruning for Efficient Autoregressive Models

• Authors: Xuan Shen, Zhao Song, Yufa Zhou, Bo Chen, Jing Liu, Ruiyi Zhang, Ryan A. Rossi, Hao Tan, Tong Yu, Xiang Chen, Yufan Zhou, Tong Sun, Pu Zhao, Yanzhi Wang, Jiuxiang Gu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12441
• Pdf URL: https://arxiv.org/pdf/2412.12441
• Copy Paste: [[2412.12441]] Numerical Pruning for Efficient Autoregressive Models(https://arxiv.org/abs/2412.12441)
• Keywords: transformer
• Abstract: Transformers have emerged as the leading architecture in deep learning, proving to be versatile and highly effective across diverse domains beyond language and image processing. However, their impressive performance often incurs high computational costs due to their substantial model size. This paper focuses on compressing decoder-only transformer-based autoregressive models through structural weight pruning to improve the model efficiency while preserving performance for both language and image generation tasks. Specifically, we propose a training-free pruning method that calculates a numerical score with Newton's method for the Attention and MLP modules, respectively. Besides, we further propose another compensation algorithm to recover the pruned model for better performance. To verify the effectiveness of our method, we provide both theoretical support and extensive experiments. Our experiments show that our method achieves state-of-the-art performance with reduced memory usage and faster generation speeds on GPUs.

Title: LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers

• Authors: Xuan Shen, Zhao Song, Yufa Zhou, Bo Chen, Yanyu Li, Yifan Gong, Kai Zhang, Hao Tan, Jason Kuen, Henghui Ding, Zhihao Shu, Wei Niu, Pu Zhao, Yanzhi Wang, Jiuxiang Gu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12444
• Pdf URL: https://arxiv.org/pdf/2412.12444
• Copy Paste: [[2412.12444]] LazyDiT: Lazy Learning for the Acceleration of Diffusion Transformers(https://arxiv.org/abs/2412.12444)
• Keywords: diffusion, transformer, generative
• Abstract: Diffusion Transformers have emerged as the preeminent models for a wide array of generative tasks, demonstrating superior performance and efficacy across various applications. The promising results come at the cost of slow inference, as each denoising step requires running the whole transformer model with a large amount of parameters. In this paper, we show that performing the full computation of the model at each diffusion step is unnecessary, as some computations can be skipped by lazily reusing the results of previous steps. Furthermore, we show that the lower bound of similarity between outputs at consecutive steps is notably high, and this similarity can be linearly approximated using the inputs. To verify our demonstrations, we propose the \textbf{LazyDiT}, a lazy learning framework that efficiently leverages cached results from earlier steps to skip redundant computations. Specifically, we incorporate lazy learning layers into the model, effectively trained to maximize laziness, enabling dynamic skipping of redundant computations. Experimental results show that LazyDiT outperforms the DDIM sampler across multiple diffusion transformer models at various resolutions. Furthermore, we implement our method on mobile devices, achieving better performance than DDIM with similar latency.

Title: Graph Learning in the Era of LLMs: A Survey from the Perspective of Data, Models, and Tasks

• Authors: Xunkai Li, Zhengyu Wu, Jiayi Wu, Hanwen Cui, Jishuo Jia, Rong-Hua Li, Guoren Wang
• Subjects: cs.LG, cs.AI, cs.CL, cs.DB
• Abstract URL: https://arxiv.org/abs/2412.12456
• Pdf URL: https://arxiv.org/pdf/2412.12456
• Copy Paste: [[2412.12456]] Graph Learning in the Era of LLMs: A Survey from the Perspective of Data, Models, and Tasks(https://arxiv.org/abs/2412.12456)
• Keywords: large language model
• Abstract: With the increasing prevalence of cross-domain Text-Attributed Graph (TAG) Data (e.g., citation networks, recommendation systems, social networks, and ai4science), the integration of Graph Neural Networks (GNNs) and Large Language Models (LLMs) into a unified Model architecture (e.g., LLM as enhancer, LLM as collaborators, LLM as predictor) has emerged as a promising technological paradigm. The core of this new graph learning paradigm lies in the synergistic combination of GNNs' ability to capture complex structural relationships and LLMs' proficiency in understanding informative contexts from the rich textual descriptions of graphs. Therefore, we can leverage graph description texts with rich semantic context to fundamentally enhance Data quality, thereby improving the representational capacity of model-centric approaches in line with data-centric machine learning principles. By leveraging the strengths of these distinct neural network architectures, this integrated approach addresses a wide range of TAG-based Task (e.g., graph learning, graph reasoning, and graph question answering), particularly in complex industrial scenarios (e.g., supervised, few-shot, and zero-shot settings). In other words, we can treat text as a medium to enable cross-domain generalization of graph learning Model, allowing a single graph model to effectively handle the diversity of downstream graph-based Task across different data domains. This work serves as a foundational reference for researchers and practitioners looking to advance graph learning methodologies in the rapidly evolving landscape of LLM. We consistently maintain the related open-source materials at \url{this https URL}.

Title: LITA: An Efficient LLM-assisted Iterative Topic Augmentation Framework

• Authors: Chia-Hsuan Chang, Jui-Tse Tsai, Yi-Hang Tsai, San-Yih Hwang
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.12459
• Pdf URL: https://arxiv.org/pdf/2412.12459
• Copy Paste: [[2412.12459]] LITA: An Efficient LLM-assisted Iterative Topic Augmentation Framework(https://arxiv.org/abs/2412.12459)
• Keywords: large language model
• Abstract: Topic modeling is widely used for uncovering thematic structures within text corpora, yet traditional models often struggle with specificity and coherence in domain-focused applications. Guided approaches, such as SeededLDA and CorEx, incorporate user-provided seed words to improve relevance but remain labor-intensive and static. Large language models (LLMs) offer potential for dynamic topic refinement and discovery, yet their application often incurs high API costs. To address these challenges, we propose the LLM-assisted Iterative Topic Augmentation framework (LITA), an LLM-assisted approach that integrates user-provided seeds with embedding-based clustering and iterative refinement. LITA identifies a small number of ambiguous documents and employs an LLM to reassign them to existing or new topics, minimizing API costs while enhancing topic quality. Experiments on two datasets across topic quality and clustering performance metrics demonstrate that LITA outperforms five baseline models, including LDA, SeededLDA, CorEx, BERTopic, and PromptTopic. Our work offers an efficient and adaptable framework for advancing topic modeling and text clustering.

Title: Pattern Analogies: Learning to Perform Programmatic Image Edits by Analogy

• Authors: Aditya Ganeshan, Thibault Groueix, Paul Guerrero, Radomír Měch, Matthew Fisher, Daniel Ritchie
• Subjects: cs.CV, cs.AI, cs.GR, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.12463
• Pdf URL: https://arxiv.org/pdf/2412.12463
• Copy Paste: [[2412.12463]] Pattern Analogies: Learning to Perform Programmatic Image Edits by Analogy(https://arxiv.org/abs/2412.12463)
• Keywords: diffusion, generative
• Abstract: Pattern images are everywhere in the digital and physical worlds, and tools to edit them are valuable. But editing pattern images is tricky: desired edits are often programmatic: structure-aware edits that alter the underlying program which generates the pattern. One could attempt to infer this underlying program, but current methods for doing so struggle with complex images and produce unorganized programs that make editing tedious. In this work, we introduce a novel approach to perform programmatic edits on pattern images. By using a pattern analogy -- a pair of simple patterns to demonstrate the intended edit -- and a learning-based generative model to execute these edits, our method allows users to intuitively edit patterns. To enable this paradigm, we introduce SplitWeave, a domain-specific language that, combined with a framework for sampling synthetic pattern analogies, enables the creation of a large, high-quality synthetic training dataset. We also present TriFuser, a Latent Diffusion Model (LDM) designed to overcome critical issues that arise when naively deploying LDMs to this task. Extensive experiments on real-world, artist-sourced patterns reveals that our method faithfully performs the demonstrated edit while also generalizing to related pattern styles beyond its training distribution.

Title: Core Context Aware Attention for Long Context Language Modeling

• Authors: Yaofo Chen, Zeng You, Shuhai Zhang, Haokun Li, Yirui Li, Yaowei Wang, Mingkui Tan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12465
• Pdf URL: https://arxiv.org/pdf/2412.12465
• Copy Paste: [[2412.12465]] Core Context Aware Attention for Long Context Language Modeling(https://arxiv.org/abs/2412.12465)
• Keywords: transformer, large language model
• Abstract: Transformer-based Large Language Models (LLMs) have exhibited remarkable success in various natural language processing tasks primarily attributed to self-attention mechanism, which requires a token to consider all preceding tokens as its context to compute the attention score. However, when the context length L becomes very large (e.g., 32K), more redundant context information will be included w.r.t. any tokens, making the self-attention suffer from two main limitations: 1) The computational and memory complexity scales quadratically w.r.t. L; 2) The presence of redundant context information may hamper the model to capture dependencies among crucial tokens, which may degrade the representation performance. In this paper, we propose a plug-and-play Core Context Aware (CCA) Attention for efficient long-range context modeling, which consists of two components: 1) Globality-pooling attention that divides input tokens into groups and then dynamically merges tokens within each group into one core token based on their significance; 2) Locality-preserved attention that incorporates neighboring tokens into the attention calculation. The two complementary attentions will then be fused to the final attention, maintaining comprehensive modeling ability as the full self-attention. In this way, the core context information w.r.t. a given token will be automatically focused and strengthened, while the context information in redundant groups will be diminished during the learning process. As a result, the computational and memory complexity will be significantly reduced. More importantly, the CCA-Attention can improve the long-context modeling ability by diminishing the redundant context information. Extensive experimental results demonstrate that our CCA-Attention significantly outperforms state-of-the-art models in terms of computational efficiency and long-context modeling ability.

Title: Knowledge Boundary of Large Language Models: A Survey

• Authors: Moxin Li, Yong Zhao, Yang Deng, Wenxuan Zhang, Shuaiyi Li, Wenya Xie, See-Kiong Ng, Tat-Seng Chua
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12472
• Pdf URL: https://arxiv.org/pdf/2412.12472
• Copy Paste: [[2412.12472]] Knowledge Boundary of Large Language Models: A Survey(https://arxiv.org/abs/2412.12472)
• Keywords: large language model
• Abstract: Although large language models (LLMs) store vast amount of knowledge in their parameters, they still have limitations in the memorization and utilization of certain knowledge, leading to undesired behaviors such as generating untruthful and inaccurate responses. This highlights the critical need to understand the knowledge boundary of LLMs, a concept that remains inadequately defined in existing research. In this survey, we propose a comprehensive definition of the LLM knowledge boundary and introduce a formalized taxonomy categorizing knowledge into four distinct types. Using this foundation, we systematically review the field through three key lenses: the motivation for studying LLM knowledge boundaries, methods for identifying these boundaries, and strategies for mitigating the challenges they present. Finally, we discuss open challenges and potential research directions in this area. We aim for this survey to offer the community a comprehensive overview, facilitate access to key issues, and inspire further advancements in LLM knowledge research.

Title: A Method for Enhancing Generalization of Adam by Multiple Integrations

• Authors: Long Jin, Han Nong, Liangming Chen, Zhenming Su
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12473
• Pdf URL: https://arxiv.org/pdf/2412.12473
• Copy Paste: [[2412.12473]] A Method for Enhancing Generalization of Adam by Multiple Integrations(https://arxiv.org/abs/2412.12473)
• Keywords: robust, diffusion
• Abstract: The insufficient generalization of adaptive moment estimation (Adam) has hindered its broader application. Recent studies have shown that flat minima in loss landscapes are highly associated with improved generalization. Inspired by the filtering effect of integration operations on high-frequency signals, we propose multiple integral Adam (MIAdam), a novel optimizer that integrates a multiple integral term into Adam. This multiple integral term effectively filters out sharp minima encountered during optimization, guiding the optimizer towards flatter regions and thereby enhancing generalization capability. We provide a theoretical explanation for the improvement in generalization through the diffusion theory framework and analyze the impact of the multiple integral term on the optimizer's convergence. Experimental results demonstrate that MIAdam not only enhances generalization and robustness against label noise but also maintains the rapid convergence characteristic of Adam, outperforming Adam and its variants in state-of-the-art benchmarks.

Title: RareAgents: Autonomous Multi-disciplinary Team for Rare Disease Diagnosis and Treatment

• Authors: Xuanzhong Chen, Ye Jin, Xiaohao Mao, Lun Wang, Shuyang Zhang, Ting Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12475
• Pdf URL: https://arxiv.org/pdf/2412.12475
• Copy Paste: [[2412.12475]] RareAgents: Autonomous Multi-disciplinary Team for Rare Disease Diagnosis and Treatment(https://arxiv.org/abs/2412.12475)
• Keywords: large language model
• Abstract: Rare diseases, despite their low individual incidence, collectively impact around 300 million people worldwide due to the huge number of diseases. The complexity of symptoms and the shortage of specialized doctors with relevant experience make diagnosing and treating rare diseases more challenging than common diseases. Recently, agents powered by large language models (LLMs) have demonstrated notable improvements across various domains. In the medical field, some agent methods have outperformed direct prompts in question-answering tasks from medical exams. However, current agent frameworks lack adaptation for real-world clinical scenarios, especially those involving the intricate demands of rare diseases. To address these challenges, we present RareAgents, the first multi-disciplinary team of LLM-based agents tailored to the complex clinical context of rare diseases. RareAgents integrates advanced planning capabilities, memory mechanisms, and medical tools utilization, leveraging Llama-3.1-8B/70B as the base model. Experimental results show that RareAgents surpasses state-of-the-art domain-specific models, GPT-4o, and existing agent frameworks in both differential diagnosis and medication recommendation for rare diseases. Furthermore, we contribute a novel dataset, MIMIC-IV-Ext-Rare, derived from MIMIC-IV, to support further advancements in this field.

Title: Human-in-the-Loop Generation of Adversarial Texts: A Case Study on Tibetan Script

• Authors: Xi Cao, Yuan Sun, Jiajun Li, Quzong Gesang, Nuo Qun, Tashi Nyima
• Subjects: cs.CL, cs.CR, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.12478
• Pdf URL: https://arxiv.org/pdf/2412.12478
• Copy Paste: [[2412.12478]] Human-in-the-Loop Generation of Adversarial Texts: A Case Study on Tibetan Script(https://arxiv.org/abs/2412.12478)
• Keywords: attack, robust
• Abstract: DNN-based language models perform excellently on various tasks, but even SOTA LLMs are susceptible to textual adversarial attacks. Adversarial texts play crucial roles in multiple subfields of NLP. However, current research has the following issues. (1) Most textual adversarial attack methods target rich-resourced languages. How do we generate adversarial texts for less-studied languages? (2) Most textual adversarial attack methods are prone to generating invalid or ambiguous adversarial texts. How do we construct high-quality adversarial robustness benchmarks? (3) New language models may be immune to part of previously generated adversarial texts. How do we update adversarial robustness benchmarks? To address the above issues, we introduce HITL-GAT, a system based on a general approach to human-in-the-loop generation of adversarial texts. HITL-GAT contains four stages in one pipeline: victim model construction, adversarial example generation, high-quality benchmark construction, and adversarial robustness evaluation. Additionally, we utilize HITL-GAT to make a case study on Tibetan script which can be a reference for the adversarial research of other less-studied languages.

Title: AutoSGNN: Automatic Propagation Mechanism Discovery for Spectral Graph Neural Networks

• Authors: Shibing Mo, Kai Wu, Qixuan Gao, Xiangyi Teng, Jing Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12483
• Pdf URL: https://arxiv.org/pdf/2412.12483
• Copy Paste: [[2412.12483]] AutoSGNN: Automatic Propagation Mechanism Discovery for Spectral Graph Neural Networks(https://arxiv.org/abs/2412.12483)
• Keywords: large language model
• Abstract: In real-world applications, spectral Graph Neural Networks (GNNs) are powerful tools for processing diverse types of graphs. However, a single GNN often struggles to handle different graph types-such as homogeneous and heterogeneous graphs-simultaneously. This challenge has led to the manual design of GNNs tailored to specific graph types, but these approaches are limited by the high cost of labor and the constraints of expert knowledge, which cannot keep up with the rapid growth of graph data. To overcome these challenges, we propose AutoSGNN, an automated framework for discovering propagation mechanisms in spectral GNNs. AutoSGNN unifies the search space for spectral GNNs by integrating large language models with evolutionary strategies to automatically generate architectures that adapt to various graph types. Extensive experiments on nine widely-used datasets, encompassing both homophilic and heterophilic graphs, demonstrate that AutoSGNN outperforms state-of-the-art spectral GNNs and graph neural architecture search methods in both performance and efficiency.

Title: Boosting Long-Context Information Seeking via Query-Guided Activation Refilling

• Authors: Hongjin Qian, Zheng Liu, Peitian Zhang, Zhicheng Dou, Defu Lian
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.12486
• Pdf URL: https://arxiv.org/pdf/2412.12486
• Copy Paste: [[2412.12486]] Boosting Long-Context Information Seeking via Query-Guided Activation Refilling(https://arxiv.org/abs/2412.12486)
• Keywords: large language model
• Abstract: Processing long contexts poses a significant challenge for large language models (LLMs) due to their inherent context-window limitations and the computational burden of extensive key-value (KV) activations, which severely impact efficiency. For information-seeking tasks, full context perception is often unnecessary, as a query's information needs can dynamically range from localized details to a global perspective, depending on its complexity. However, existing methods struggle to adapt effectively to these dynamic information needs. In the paper, we propose a method for processing long-context information-seeking tasks via query-guided Activation Refilling (ACRE). ACRE constructs a Bi-layer KV Cache for long contexts, where the layer-1 (L1) cache compactly captures global information, and the layer-2 (L2) cache provides detailed and localized information. ACRE establishes a proxying relationship between the two caches, allowing the input query to attend to the L1 cache and dynamically refill it with relevant entries from the L2 cache. This mechanism integrates global understanding with query-specific local details, thus improving answer decoding. Experiments on a variety of long-context information-seeking datasets demonstrate ACRE's effectiveness, achieving improvements in both performance and efficiency.

Title: DuSSS: Dual Semantic Similarity-Supervised Vision-Language Model for Semi-Supervised Medical Image Segmentation

• Authors: Qingtao Pan, Wenhao Qiao, Jingjiao Lou, Bing Ji, Shuo Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12492
• Pdf URL: https://arxiv.org/pdf/2412.12492
• Copy Paste: [[2412.12492]] DuSSS: Dual Semantic Similarity-Supervised Vision-Language Model for Semi-Supervised Medical Image Segmentation(https://arxiv.org/abs/2412.12492)
• Keywords: segmentation
• Abstract: Semi-supervised medical image segmentation (SSMIS) uses consistency learning to regularize model training, which alleviates the burden of pixel-wise manual annotations. However, it often suffers from error supervision from low-quality pseudo labels. Vision-Language Model (VLM) has great potential to enhance pseudo labels by introducing text prompt guided multimodal supervision information. It nevertheless faces the cross-modal problem: the obtained messages tend to correspond to multiple targets. To address aforementioned problems, we propose a Dual Semantic Similarity-Supervised VLM (DuSSS) for SSMIS. Specifically, 1) a Dual Contrastive Learning (DCL) is designed to improve cross-modal semantic consistency by capturing intrinsic representations within each modality and semantic correlations across modalities. 2) To encourage the learning of multiple semantic correspondences, a Semantic Similarity-Supervision strategy (SSS) is proposed and injected into each contrastive learning process in DCL, supervising semantic similarity via the distribution-based uncertainty levels. Furthermore, a novel VLM-based SSMIS network is designed to compensate for the quality deficiencies of pseudo-labels. It utilizes the pretrained VLM to generate text prompt guided supervision information, refining the pseudo label for better consistency regularization. Experimental results demonstrate that our DuSSS achieves outstanding performance with Dice of 82.52%, 74.61% and 78.03% on three public datasets (QaTa-COV19, BM-Seg and MoNuSeg).

Title: Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-training

• Authors: Mingjia Shi, Yuhao Zhou, Ruiji Yu, Zekai Li, Zhiyuan Liang, Xuanlei Zhao, Xiaojiang Peng, Tanmay Rajpurohit, Shanmukha Ramakrishna Vedantam, Wangbo Zhao, Kai Wang, Yang You
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12496
• Pdf URL: https://arxiv.org/pdf/2412.12496
• Copy Paste: [[2412.12496]] Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-training(https://arxiv.org/abs/2412.12496)
• Keywords: transformer
• Abstract: Vision Mamba (e.g., Vim) has successfully been integrated into computer vision, and token reduction has yielded promising outcomes in Vision Transformers (ViTs). However, token reduction performs less effectively on Vision Mamba compared to ViTs. Pruning informative tokens in Mamba leads to a high loss of key knowledge and bad performance. This makes it not a good solution for enhancing efficiency in Mamba. Token merging, which preserves more token information than pruning, has demonstrated commendable performance in ViTs. Nevertheless, vanilla merging performance decreases as the reduction ratio increases either, failing to maintain the key knowledge in Mamba. Re-training the token-reduced model enhances the performance of Mamba, by effectively rebuilding the key knowledge. Empirically, pruned Vims only drop up to 0.9% accuracy on ImageNet-1K, recovered by our proposed framework R-MeeTo in our main evaluation. We show how simple and effective the fast recovery can be achieved at minute-level, in particular, a 35.9% accuracy spike over 3 epochs of training on Vim-Ti. Moreover, Vim-Ti/S/B are re-trained within 5/7/17 minutes, and Vim-S only drop 1.3% with 1.2x (up to 1.5x) speed up in inference.

Title: NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuning

• Authors: Xin Yi, Shunfan Zheng, Linlin Wang, Gerard de Melo, Xiaoling Wang, Liang He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12497
• Pdf URL: https://arxiv.org/pdf/2412.12497
• Copy Paste: [[2412.12497]] NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuning(https://arxiv.org/abs/2412.12497)
• Keywords: attack, large language model
• Abstract: The emergence of finetuning-as-a-service has revealed a new vulnerability in large language models (LLMs). A mere handful of malicious data uploaded by users can subtly manipulate the finetuning process, resulting in an alignment-broken model. Existing methods to counteract fine-tuning attacks typically require substantial computational resources. Even with parameter-efficient techniques like LoRA, gradient updates remain essential. To address these challenges, we propose \textbf{N}euron-\textbf{L}evel \textbf{S}afety \textbf{R}ealignment (\textbf{NLSR}), a training-free framework that restores the safety of LLMs based on the similarity difference of safety-critical neurons before and after fine-tuning. The core of our framework is first to construct a safety reference model from an initially aligned model to amplify safety-related features in neurons. We then utilize this reference model to identify safety-critical neurons, which we prepare as patches. Finally, we selectively restore only those neurons that exhibit significant similarity differences by transplanting these prepared patches, thereby minimally altering the fine-tuned model. Extensive experiments demonstrate significant safety enhancements in fine-tuned models across multiple downstream tasks, while greatly maintaining task-level accuracy. Our findings suggest regions of some safety-critical neurons show noticeable differences after fine-tuning, which can be effectively corrected by transplanting neurons from the reference model without requiring additional training. The code will be available at \url{this https URL}

Title: LinguaLIFT: An Effective Two-stage Instruction Tuning Framework for Low-Resource Language Tasks

• Authors: Hongbin Zhang, Kehai Chen, Xuefeng Bai, Yang Xiang, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12499
• Pdf URL: https://arxiv.org/pdf/2412.12499
• Copy Paste: [[2412.12499]] LinguaLIFT: An Effective Two-stage Instruction Tuning Framework for Low-Resource Language Tasks(https://arxiv.org/abs/2412.12499)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated impressive multilingual understanding and reasoning capabilities, driven by extensive pre-training multilingual corpora and fine-tuning instruction data. However, a performance gap persists between high-resource and low-resource language tasks due to language imbalance in the pre-training corpus, even using more low-resource data during fine-tuning. To alleviate this issue, we propose LinguaLIFT, a two-stage instruction tuning framework for advancing low-resource language tasks. An additional language alignment layer is first integrated into the LLM to adapt a pre-trained multilingual encoder, thereby enhancing multilingual alignment through code-switched fine-tuning. The second stage fine-tunes LLM with English-only instruction data while freezing the language alignment layer, allowing LLM to transfer task-specific capabilities from English to low-resource language tasks. Additionally, we introduce the Multilingual Math World Problem (MMWP) benchmark, which spans 21 low-resource, 17 medium-resource, and 10 high-resource languages, enabling comprehensive evaluation of multilingual reasoning. Experimental results show that LinguaLIFT outperforms several competitive baselines across MMWP and other widely used benchmarks.

Title: Track the Answer: Extending TextVQA from Image to Video with Spatio-Temporal Clues

• Authors: Yan Zhang, Gangyan Zeng, Huawen Shen, Daiqing Wu, Yu Zhou, Can Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12502
• Pdf URL: https://arxiv.org/pdf/2412.12502
• Copy Paste: [[2412.12502]] Track the Answer: Extending TextVQA from Image to Video with Spatio-Temporal Clues(https://arxiv.org/abs/2412.12502)
• Keywords: generative, large language model
• Abstract: Video text-based visual question answering (Video TextVQA) is a practical task that aims to answer questions by jointly reasoning textual and visual information in a given video. Inspired by the development of TextVQA in image domain, existing Video TextVQA approaches leverage a language model (e.g. T5) to process text-rich multiple frames and generate answers auto-regressively. Nevertheless, the spatio-temporal relationships among visual entities (including scene text and objects) will be disrupted and models are susceptible to interference from unrelated information, resulting in irrational reasoning and inaccurate answering. To tackle these challenges, we propose the TEA (stands for ``\textbf{T}rack th\textbf{E} \textbf{A}nswer'') method that better extends the generative TextVQA framework from image to video. TEA recovers the spatio-temporal relationships in a complementary way and incorporates OCR-aware clues to enhance the quality of reasoning questions. Extensive experiments on several public Video TextVQA datasets validate the effectiveness and generalization of our framework. TEA outperforms existing TextVQA methods, video-language pretraining methods and video large language models by great margins.

Title: Multi-Scale Cross-Fusion and Edge-Supervision Network for Image Splicing Localization

• Authors: Yakun Niu, Pei Chen, Lei Zhang, Hongjian Yin, Qi Chang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12503
• Pdf URL: https://arxiv.org/pdf/2412.12503
• Copy Paste: [[2412.12503]] Multi-Scale Cross-Fusion and Edge-Supervision Network for Image Splicing Localization(https://arxiv.org/abs/2412.12503)
• Keywords: segmentation
• Abstract: Image Splicing Localization (ISL) is a fundamental yet challenging task in digital forensics. Although current approaches have achieved promising performance, the edge information is insufficiently exploited, resulting in poor integrality and high false alarms. To tackle this problem, we propose a multi-scale cross-fusion and edge-supervision network for ISL. Specifically, our framework consists of three key steps: multi-scale features cross-fusion, edge mask prediction and edge-supervision localization. Firstly, we input the RGB image and its noise image into a segmentation network to learn multi-scale features, which are then aggregated via a cross-scale fusion followed by a cross-domain fusion to enhance feature representation. Secondly, we design an edge mask prediction module to effectively mine the reliable boundary artifacts. Finally, the cross-fused features and the reliable edge mask information are seamlessly integrated via an attention mechanism to incrementally supervise and facilitate model training. Extensive experiments on publicly available datasets demonstrate that our proposed method is superior to state-of-the-art schemes.

Title: DocFusion: A Unified Framework for Document Parsing Tasks

• Authors: Mingxu Chai, Ziyu Shen, Chong Zhang, Yue Zhang, Xiao Wang, Shihan Dou, Jihua Kang, Jiazheng Zhang, Qi Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12505
• Pdf URL: https://arxiv.org/pdf/2412.12505
• Copy Paste: [[2412.12505]] DocFusion: A Unified Framework for Document Parsing Tasks(https://arxiv.org/abs/2412.12505)
• Keywords: generative
• Abstract: Document parsing is essential for analyzing complex document structures and extracting fine-grained information, supporting numerous downstream applications. However, existing methods often require integrating multiple independent models to handle various parsing tasks, leading to high complexity and maintenance overhead. To address this, we propose DocFusion, a lightweight generative model with only 0.28B parameters. It unifies task representations and achieves collaborative training through an improved objective function. Experiments reveal and leverage the mutually beneficial interaction among recognition tasks, and integrating recognition data significantly enhances detection performance. The final results demonstrate that DocFusion achieves state-of-the-art (SOTA) performance across four key tasks.

Title: Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge

• Authors: Kayla Schroeder, Zach Wood-Doughty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12509
• Pdf URL: https://arxiv.org/pdf/2412.12509
• Copy Paste: [[2412.12509]] Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge(https://arxiv.org/abs/2412.12509)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have become increasingly powerful and ubiquitous, but their stochastic nature poses challenges to the reliability of their outputs. While deterministic settings can improve consistency, they do not guarantee reliability, as a single sample from the model's probability distribution can still be misleading. Building upon the concept of LLM-as-a-judge, we introduce a novel framework for rigorously evaluating the reliability of LLM judgments, leveraging McDonald's omega. We evaluate the reliability of LLMs when judging the outputs of other LLMs on standard single-turn and multi-turn benchmarks, simultaneously investigating the impact of temperature on reliability. By analyzing these results, we demonstrate the limitations of fixed randomness and the importance of considering multiple samples, which we show has significant implications for downstream applications. Our findings highlight the need for a nuanced understanding of LLM reliability and the potential risks associated with over-reliance on single-shot evaluations. This work provides a crucial step towards building more trustworthy and reliable LLM-based systems and applications.

Title: Can Large Language Models Understand You Better? An MBTI Personality Detection Dataset Aligned with Population Traits

• Authors: Bohan Li, Jiannan Guan, Longxu Dou, Yunlong Feng, Dingzirui Wang, Yang Xu, Enbo Wang, Qiguang Chen, Bichen Wang, Xiao Xu, Yimeng Zhang, Libo Qin, Yanyan Zhao, Qingfu Zhu, Wanxiang Che
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.12510
• Pdf URL: https://arxiv.org/pdf/2412.12510
• Copy Paste: [[2412.12510]] Can Large Language Models Understand You Better? An MBTI Personality Detection Dataset Aligned with Population Traits(https://arxiv.org/abs/2412.12510)
• Keywords: large language model
• Abstract: The Myers-Briggs Type Indicator (MBTI) is one of the most influential personality theories reflecting individual differences in thinking, feeling, and behaving. MBTI personality detection has garnered considerable research interest and has evolved significantly over the years. However, this task tends to be overly optimistic, as it currently does not align well with the natural distribution of population personality traits. Specifically, (1) the self-reported labels in existing datasets result in incorrect labeling issues, and (2) the hard labels fail to capture the full range of population personality distributions. In this paper, we optimize the task by constructing MBTIBench, the first manually annotated high-quality MBTI personality detection dataset with soft labels, under the guidance of psychologists. As for the first challenge, MBTIBench effectively solves the incorrect labeling issues, which account for 29.58% of the data. As for the second challenge, we estimate soft labels by deriving the polarity tendency of samples. The obtained soft labels confirm that there are more people with non-extreme personality traits. Experimental results not only highlight the polarized predictions and biases in LLMs as key directions for future research, but also confirm that soft labels can provide more benefits to other psychological tasks than hard labels. The code and data are available at this https URL.

Title: Invisible Watermarks: Attacks and Robustness

• Authors: Dongjun Hwang, Sungwon Woo, Tom Gao, Raymond Luo, Sunghwan Baek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12511
• Pdf URL: https://arxiv.org/pdf/2412.12511
• Copy Paste: [[2412.12511]] Invisible Watermarks: Attacks and Robustness(https://arxiv.org/abs/2412.12511)
• Keywords: attack, robust, watermark, generative
• Abstract: As Generative AI continues to become more accessible, the case for robust detection of generated images in order to combat misinformation is stronger than ever. Invisible watermarking methods act as identifiers of generated content, embedding image- and latent-space messages that are robust to many forms of perturbations. The majority of current research investigates full-image attacks against images with a single watermarking method applied. We introduce novel improvements to watermarking robustness as well as minimizing degradation on image quality during attack. Firstly, we examine the application of both image-space and latent-space watermarking methods on a single image, where we propose a custom watermark remover network which preserves one of the watermarking modalities while completely removing the other during decoding. Then, we investigate localized blurring attacks (LBA) on watermarked images based on the GradCAM heatmap acquired from the watermark decoder in order to reduce the amount of degradation to the target image. Our evaluation suggests that 1) implementing the watermark remover model to preserve one of the watermark modalities when decoding the other modality slightly improves on the baseline performance, and that 2) LBA degrades the image significantly less compared to uniform blurring of the entire image. Code is available at: this https URL

Title: Solid-SQL: Enhanced Schema-linking based In-context Learning for Robust Text-to-SQL

• Authors: Geling Liu, Yunzhi Tan, Ruichao Zhong, Yuanzhen Xie, Lingchen Zhao, Qian Wang, Bo Hu, Zang Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12522
• Pdf URL: https://arxiv.org/pdf/2412.12522
• Copy Paste: [[2412.12522]] Solid-SQL: Enhanced Schema-linking based In-context Learning for Robust Text-to-SQL(https://arxiv.org/abs/2412.12522)
• Keywords: robust, large language model
• Abstract: Recently, large language models (LLMs) have significantly improved the performance of text-to-SQL systems. Nevertheless, many state-of-the-art (SOTA) approaches have overlooked the critical aspect of system robustness. Our experiments reveal that while LLM-driven methods excel on standard datasets, their accuracy is notably compromised when faced with adversarial perturbations. To address this challenge, we propose a robust text-to-SQL solution, called Solid-SQL, designed to integrate with various LLMs. We focus on the pre-processing stage, training a robust schema-linking model enhanced by LLM-based data augmentation. Additionally, we design a two-round, structural similarity-based example retrieval strategy for in-context learning. Our method achieves SOTA SQL execution accuracy levels of 82.1% and 58.9% on the general Spider and Bird benchmarks, respectively. Furthermore, experimental results show that Solid-SQL delivers an average improvement of 11.6% compared to baselines on the perturbed Spider-Syn, Spider-Realistic, and Dr. Spider benchmarks.

Title: When to Speak, When to Abstain: Contrastive Decoding with Abstention

• Authors: Hyuhng Joon Kim, Youna Kim, Sang-goo Lee, Taeuk Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12527
• Pdf URL: https://arxiv.org/pdf/2412.12527
• Copy Paste: [[2412.12527]] When to Speak, When to Abstain: Contrastive Decoding with Abstention(https://arxiv.org/abs/2412.12527)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) demonstrate exceptional performance across diverse tasks by leveraging both pre-trained knowledge (i.e., parametric knowledge) and external knowledge (i.e., contextual knowledge). While substantial efforts have been made to leverage both forms of knowledge, scenarios in which the model lacks any relevant knowledge remain underexplored. Such limitations can result in issues like hallucination, causing reduced reliability and potential risks in high-stakes applications. To address such limitations, this paper extends the task scope to encompass cases where the user's request cannot be fulfilled due to the lack of relevant knowledge. To this end, we introduce Contrastive Decoding with Abstention (CDA), a training-free decoding method that empowers LLMs to generate responses when relevant knowledge is available and to abstain otherwise. CDA evaluates the relevance of each knowledge for a given query, adaptively determining which knowledge to prioritize or which to completely ignore. Extensive experiments with four LLMs on three question-answering datasets demonstrate that CDA can effectively perform accurate generation and abstention simultaneously. These findings highlight CDA's potential to broaden the applicability of LLMs, enhancing reliability and preserving user trust.

Title: Addressing Small and Imbalanced Medical Image Datasets Using Generative Models: A Comparative Study of DDPM and PGGANs with Random and Greedy K Sampling

• Authors: Iman Khazrak, Shakhnoza Takhirova, Mostafa M. Rezaee, Mehrdad Yadollahi, Robert C. Green II, Shuteng Niu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12532
• Pdf URL: https://arxiv.org/pdf/2412.12532
• Copy Paste: [[2412.12532]] Addressing Small and Imbalanced Medical Image Datasets Using Generative Models: A Comparative Study of DDPM and PGGANs with Random and Greedy K Sampling(https://arxiv.org/abs/2412.12532)
• Keywords: privacy, robust, diffusion, generative
• Abstract: The development of accurate medical image classification models is often constrained by privacy concerns and data scarcity for certain conditions, leading to small and imbalanced datasets. To address these limitations, this study explores the use of generative models, such as Denoising Diffusion Probabilistic Models (DDPM) and Progressive Growing Generative Adversarial Networks (PGGANs), for dataset augmentation. The research introduces a framework to assess the impact of synthetic images generated by DDPM and PGGANs on the performance of four models: a custom CNN, Untrained VGG16, Pretrained VGG16, and Pretrained ResNet50. Experiments were conducted using Random Sampling and Greedy K Sampling to create small, imbalanced datasets. The synthetic images were evaluated using Frechet Inception Distance (FID) and compared to original datasets through classification metrics. The results show that DDPM consistently generated more realistic images with lower FID scores and significantly outperformed PGGANs in improving classification metrics across all models and datasets. Incorporating DDPM-generated images into the original datasets increased accuracy by up to 6%, enhancing model robustness and stability, particularly in imbalanced scenarios. Random Sampling demonstrated superior stability, while Greedy K Sampling offered diversity at the cost of higher FID scores. This study highlights the efficacy of DDPM in augmenting small, imbalanced medical image datasets, improving model performance by balancing the dataset and expanding its size.

Title: Stiefel Flow Matching for Moment-Constrained Structure Elucidation

• Authors: Austin Cheng, Alston Lo, Kin Long Kelvin Lee, Santiago Miret, Alán Aspuru-Guzik
• Subjects: cs.LG, physics.chem-ph
• Abstract URL: https://arxiv.org/abs/2412.12540
• Pdf URL: https://arxiv.org/pdf/2412.12540
• Copy Paste: [[2412.12540]] Stiefel Flow Matching for Moment-Constrained Structure Elucidation(https://arxiv.org/abs/2412.12540)
• Keywords: diffusion, generative
• Abstract: Molecular structure elucidation is a fundamental step in understanding chemical phenomena, with applications in identifying molecules in natural products, lab syntheses, forensic samples, and the interstellar medium. We consider the task of predicting a molecule's all-atom 3D structure given only its molecular formula and moments of inertia, motivated by the ability of rotational spectroscopy to measure these moments. While existing generative models can conditionally sample 3D structures with approximately correct moments, this soft conditioning fails to leverage the many digits of precision afforded by experimental rotational spectroscopy. To address this, we first show that the space of $n$-atom point clouds with a fixed set of moments of inertia is embedded in the Stiefel manifold $\mathrm{St}(n, 4)$. We then propose Stiefel Flow Matching as a generative model for elucidating 3D structure under exact moment constraints. Additionally, we learn simpler and shorter flows by finding approximate solutions for equivariant optimal transport on the Stiefel manifold. Empirically, enforcing exact moment constraints allows Stiefel Flow Matching to achieve higher success rates and faster sampling than Euclidean diffusion models, even on high-dimensional manifolds corresponding to large molecules in the GEOM dataset.

Title: LLMCL-GEC: Advancing Grammatical Error Correction with LLM-Driven Curriculum Learning

• Authors: Tao Fang, Derek F. Wong, Lusheng Zhang, Keyan Jin, Qiang Zhang, Tianjiao Li, Jinlong Hou, Lidia S. Chao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12541
• Pdf URL: https://arxiv.org/pdf/2412.12541
• Copy Paste: [[2412.12541]] LLMCL-GEC: Advancing Grammatical Error Correction with LLM-Driven Curriculum Learning(https://arxiv.org/abs/2412.12541)
• Keywords: robust
• Abstract: While large-scale language models (LLMs) have demonstrated remarkable capabilities in specific natural language processing (NLP) tasks, they may still lack proficiency compared to specialized models in certain domains, such as grammatical error correction (GEC). Drawing inspiration from the concept of curriculum learning, we have delved into refining LLMs into proficient GEC experts by devising effective curriculum learning (CL) strategies. In this paper, we introduce a novel approach, termed LLM-based curriculum learning, which capitalizes on the robust semantic comprehension and discriminative prowess inherent in LLMs to gauge the complexity of GEC training data. Unlike traditional curriculum learning techniques, our method closely mirrors human expert-designed curriculums. Leveraging the proposed LLM-based CL method, we sequentially select varying levels of curriculums ranging from easy to hard, and iteratively train and refine using the pretrianed T5 and LLaMA series models. Through rigorous testing and analysis across diverse benchmark assessments in English GEC, including the CoNLL14 test, BEA19 test, and BEA19 development sets, our approach showcases a significant performance boost over baseline models and conventional curriculum learning methodologies.

Title: Consistent Diffusion: Denoising Diffusion Model with Data-Consistent Training for Image Restoration

• Authors: Xinlong Cheng, Tiantian Cao, Guoan Cheng, Bangxuan Huang, Xinghan Tian, Ye Wang, Xiaoyu He, Weixin Li, Tianfan Xue, Xuan Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12550
• Pdf URL: https://arxiv.org/pdf/2412.12550
• Copy Paste: [[2412.12550]] Consistent Diffusion: Denoising Diffusion Model with Data-Consistent Training for Image Restoration(https://arxiv.org/abs/2412.12550)
• Keywords: diffusion
• Abstract: In this work, we address the limitations of denoising diffusion models (DDMs) in image restoration tasks, particularly the shape and color distortions that can compromise image quality. While DDMs have demonstrated a promising performance in many applications such as text-to-image synthesis, their effectiveness in image restoration is often hindered by shape and color distortions. We observe that these issues arise from inconsistencies between the training and testing data used by DDMs. Based on our observation, we propose a novel training method, named data-consistent training, which allows the DDMs to access images with accumulated errors during training, thereby ensuring the model to learn to correct these errors. Experimental results show that, across five image restoration tasks, our method has significant improvements over state-of-the-art methods while effectively minimizing distortions and preserving image fidelity.

Title: SAModified: A Foundation Model-Based Zero-Shot Approach for Refining Noisy Land-Use Land-Cover Maps

• Authors: Sparsh Pekhale, Rakshith Sathish, Sathisha Basavaraju, Divya Sharma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12552
• Pdf URL: https://arxiv.org/pdf/2412.12552
• Copy Paste: [[2412.12552]] SAModified: A Foundation Model-Based Zero-Shot Approach for Refining Noisy Land-Use Land-Cover Maps(https://arxiv.org/abs/2412.12552)
• Keywords: segmentation
• Abstract: Land-use and land cover (LULC) analysis is critical in remote sensing, with wide-ranging applications across diverse fields such as agriculture, utilities, and urban planning. However, automating LULC map generation using machine learning is rendered challenging due to noisy labels. Typically, the ground truths (e.g. ESRI LULC, MapBioMass) have noisy labels that hamper the model's ability to learn to accurately classify the pixels. Further, these erroneous labels can significantly distort the performance metrics of a model, leading to misleading evaluations. Traditionally, the ambiguous labels are rectified using unsupervised algorithms. These algorithms struggle not only with scalability but also with generalization across different geographies. To overcome these challenges, we propose a zero-shot approach using the foundation model, Segment Anything Model (SAM), to automatically delineate different land parcels/regions and leverage them to relabel the unsure pixels by using the local label statistics within each detected region. We achieve a significant reduction in label noise and an improvement in the performance of the downstream segmentation model by $\approx 5\%$ when trained with denoised labels.

Title: EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generation

• Authors: Taeho Hwang, Sukmin Cho, Soyeong Jeong, Hoyun Song, SeungYoon Han, Jong C. Park
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.12559
• Pdf URL: https://arxiv.org/pdf/2412.12559
• Copy Paste: [[2412.12559]] EXIT: Context-Aware Extractive Compression for Enhancing Retrieval-Augmented Generation(https://arxiv.org/abs/2412.12559)
• Keywords: extraction
• Abstract: We introduce EXIT, an extractive context compression framework that enhances both the effectiveness and efficiency of retrieval-augmented generation (RAG) in question answering (QA). Current RAG systems often struggle when retrieval models fail to rank the most relevant documents, leading to the inclusion of more context at the expense of latency and accuracy. While abstractive compression methods can drastically reduce token counts, their token-by-token generation process significantly increases end-to-end latency. Conversely, existing extractive methods reduce latency but rely on independent, non-adaptive sentence selection, failing to fully utilize contextual information. EXIT addresses these limitations by classifying sentences from retrieved documents - while preserving their contextual dependencies - enabling parallelizable, context-aware extraction that adapts to query complexity and retrieval quality. Our evaluations on both single-hop and multi-hop QA tasks show that EXIT consistently surpasses existing compression methods and even uncompressed baselines in QA accuracy, while also delivering substantial reductions in inference time and token count. By improving both effectiveness and efficiency, EXIT provides a promising direction for developing scalable, high-quality QA solutions in RAG pipelines. Our code is available at this https URL

Title: Tell Me What to Track: Infusing Robust Language Guidance for Enhanced Referring Multi-Object Tracking

• Authors: Wenjun Huang, Yang Ni, Hanning Chen, Yirui He, Ian Bryant, Yezi Liu, Mohsen Imani
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12561
• Pdf URL: https://arxiv.org/pdf/2412.12561
• Copy Paste: [[2412.12561]] Tell Me What to Track: Infusing Robust Language Guidance for Enhanced Referring Multi-Object Tracking(https://arxiv.org/abs/2412.12561)
• Keywords: robust
• Abstract: Referring multi-object tracking (RMOT) is an emerging cross-modal task that aims to localize an arbitrary number of targets based on a language expression and continuously track them in a video. This intricate task involves reasoning on multi-modal data and precise target localization with temporal association. However, prior studies overlook the imbalanced data distribution between newborn targets and existing targets due to the nature of the task. In addition, they only indirectly fuse multi-modal features, struggling to deliver clear guidance on newborn target detection. To solve the above issues, we conduct a collaborative matching strategy to alleviate the impact of the imbalance, boosting the ability to detect newborn targets while maintaining tracking performance. In the encoder, we integrate and enhance the cross-modal and multi-scale fusion, overcoming the bottlenecks in previous work, where limited multi-modal information is shared and interacted between feature maps. In the decoder, we also develop a referring-infused adaptation that provides explicit referring guidance through the query tokens. The experiments showcase the superior performance of our model (+3.42%) compared to prior works, demonstrating the effectiveness of our designs.

Title: Efficient Oriented Object Detection with Enhanced Small Object Recognition in Aerial Images

• Authors: Zhifei Shi, Zongyao Yin, Sheng Chang, Xiao Yi, Xianchuan Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12562
• Pdf URL: https://arxiv.org/pdf/2412.12562
• Copy Paste: [[2412.12562]] Efficient Oriented Object Detection with Enhanced Small Object Recognition in Aerial Images(https://arxiv.org/abs/2412.12562)
• Keywords: extraction
• Abstract: Achieving a balance between computational efficiency and detection accuracy in the realm of rotated bounding box object detection within aerial imagery is a significant challenge. While prior research has aimed at creating lightweight models that enhance computational performance and feature extraction, there remains a gap in the performance of these networks when it comes to the detection of small and multi-scale objects in remote sensing (RS) imagery. To address these challenges, we present a novel enhancement to the YOLOv8 model, tailored for oriented object detection tasks and optimized for environments with limited computational resources. Our model features a wavelet transform-based C2f module for capturing associative features and an Adaptive Scale Feature Pyramid (ASFP) module that leverages P2 layer details. Additionally, the incorporation of GhostDynamicConv significantly contributes to the model's lightweight nature, ensuring high efficiency in aerial imagery analysis. Featuring a parameter count of 21.6M, our approach provides a more efficient architectural design than DecoupleNet, which has 23.3M parameters, all while maintaining detection accuracy. On the DOTAv1.0 dataset, our model demonstrates a mean Average Precision (mAP) that is competitive with leading methods such as DecoupleNet. The model's efficiency, combined with its reduced parameter count, makes it a strong candidate for aerial object detection, particularly in resource-constrained environments.

Title: Task-Agnostic Language Model Watermarking via High Entropy Passthrough Layers

• Authors: Vaden Masrani, Mohammad Akbari, David Ming Xuan Yue, Ahmad Rezaei, Yong Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12563
• Pdf URL: https://arxiv.org/pdf/2412.12563
• Copy Paste: [[2412.12563]] Task-Agnostic Language Model Watermarking via High Entropy Passthrough Layers(https://arxiv.org/abs/2412.12563)
• Keywords: attack, robust, extraction, watermark, large language model
• Abstract: In the era of costly pre-training of large language models, ensuring the intellectual property rights of model owners, and insuring that said models are responsibly deployed, is becoming increasingly important. To this end, we propose model watermarking via passthrough layers, which are added to existing pre-trained networks and trained using a self-supervised loss such that the model produces high-entropy output when prompted with a unique private key, and acts normally otherwise. Unlike existing model watermarking methods, our method is fully task-agnostic, and can be applied to both classification and sequence-to-sequence tasks without requiring advanced access to downstream fine-tuning datasets. We evaluate the proposed passthrough layers on a wide range of downstream tasks, and show experimentally our watermarking method achieves a near-perfect watermark extraction accuracy and false-positive rate in most cases without damaging original model performance. Additionally, we show our method is robust to both downstream fine-tuning, fine-pruning, and layer removal attacks, and can be trained in a fraction of the time required to train the original model. Code is available in the paper.

Title: Evaluating Zero-Shot Multilingual Aspect-Based Sentiment Analysis with Large Language Models

• Authors: Chengyan Wu, Bolei Ma, Zheyu Zhang, Ningyuan Deng, Yanqing He, Yun Xue
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12564
• Pdf URL: https://arxiv.org/pdf/2412.12564
• Copy Paste: [[2412.12564]] Evaluating Zero-Shot Multilingual Aspect-Based Sentiment Analysis with Large Language Models(https://arxiv.org/abs/2412.12564)
• Keywords: large language model
• Abstract: Aspect-based sentiment analysis (ABSA), a sequence labeling task, has attracted increasing attention in multilingual contexts. While previous research has focused largely on fine-tuning or training models specifically for ABSA, we evaluate large language models (LLMs) under zero-shot conditions to explore their potential to tackle this challenge with minimal task-specific adaptation. We conduct a comprehensive empirical evaluation of a series of LLMs on multilingual ABSA tasks, investigating various prompting strategies, including vanilla zero-shot, chain-of-thought (CoT), self-improvement, self-debate, and self-consistency, across nine different models. Results indicate that while LLMs show promise in handling multilingual ABSA, they generally fall short of fine-tuned, task-specific models. Notably, simpler zero-shot prompts often outperform more complex strategies, especially in high-resource languages like English. These findings underscore the need for further refinement of LLM-based approaches to effectively address ABSA task across diverse languages.

Title: FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning

• Authors: Seunghee Kim, Changhyeon Kim, Taeuk Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12567
• Pdf URL: https://arxiv.org/pdf/2412.12567
• Copy Paste: [[2412.12567]] FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning(https://arxiv.org/abs/2412.12567)
• Keywords: robust, large language model
• Abstract: Real-world decision-making often requires integrating and reasoning over information from multiple modalities. While recent multimodal large language models (MLLMs) have shown promise in such tasks, their ability to perform multi-hop reasoning across diverse sources remains insufficiently evaluated. Existing benchmarks, such as MMQA, face challenges due to (1) data contamination and (2) a lack of complex queries that necessitate operations across more than two modalities, hindering accurate performance assessment. To address this, we present Financial Cross-Modal Multi-Hop Reasoning (FCMR), a benchmark created to analyze the reasoning capabilities of MLLMs by urging them to combine information from textual reports, tables, and charts within the financial domain. FCMR is categorized into three difficulty levels-Easy, Medium, and Hard-facilitating a step-by-step evaluation. In particular, problems at the Hard level require precise cross-modal three-hop reasoning and are designed to prevent the disregard of any modality. Experiments on this new benchmark reveal that even state-of-the-art MLLMs struggle, with the best-performing model (Claude 3.5 Sonnet) achieving only 30.4% accuracy on the most challenging tier. We also conduct analysis to provide insights into the inner workings of the models, including the discovery of a critical bottleneck in the information retrieval phase.

Title: ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformers

• Authors: Lianghua Huang, Wei Wang, Zhi-Fan Wu, Yupeng Shi, Chen Liang, Tong Shen, Han Zhang, Huanzhang Dou, Yu Liu, Jingren Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12571
• Pdf URL: https://arxiv.org/pdf/2412.12571
• Copy Paste: [[2412.12571]] ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformers(https://arxiv.org/abs/2412.12571)
• Keywords: diffusion, transformer
• Abstract: Recent research arXiv:2410.15027 arXiv:2410.23775 has highlighted the inherent in-context generation capabilities of pretrained diffusion transformers (DiTs), enabling them to seamlessly adapt to diverse visual tasks with minimal or no architectural modifications. These capabilities are unlocked by concatenating self-attention tokens across multiple input and target images, combined with grouped and masked generation pipelines. Building upon this foundation, we present ChatDiT, a zero-shot, general-purpose, and interactive visual generation framework that leverages pretrained diffusion transformers in their original form, requiring no additional tuning, adapters, or modifications. Users can interact with ChatDiT to create interleaved text-image articles, multi-page picture books, edit images, design IP derivatives, or develop character design settings, all through free-form natural language across one or more conversational rounds. At its core, ChatDiT employs a multi-agent system comprising three key components: an Instruction-Parsing agent that interprets user-uploaded images and instructions, a Strategy-Planning agent that devises single-step or multi-step generation actions, and an Execution agent that performs these actions using an in-context toolkit of diffusion transformers. We thoroughly evaluate ChatDiT on IDEA-Bench arXiv:2412.11767, comprising 100 real-world design tasks and 275 cases with diverse instructions and varying numbers of input and target images. Despite its simplicity and training-free approach, ChatDiT surpasses all competitors, including those specifically designed and trained on extensive multi-task datasets. We further identify key limitations of pretrained DiTs in zero-shot adapting to tasks. We release all code, agents, results, and intermediate outputs to facilitate further research at this https URL

Title: License Plate Detection and Character Recognition Using Deep Learning and Font Evaluation

• Authors: Zahra Ebrahimi Vargoorani, Ching Yee Suen
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12572
• Pdf URL: https://arxiv.org/pdf/2412.12572
• Copy Paste: [[2412.12572]] License Plate Detection and Character Recognition Using Deep Learning and Font Evaluation(https://arxiv.org/abs/2412.12572)
• Keywords: robust
• Abstract: License plate detection (LPD) is essential for traffic management, vehicle tracking, and law enforcement but faces challenges like variable lighting and diverse font types, impacting accuracy. Traditionally reliant on image processing and machine learning, the field is now shifting towards deep learning for its robust performance in various conditions. Current methods, however, often require tailoring to specific regional datasets. This paper proposes a dual deep learning strategy using a Faster R-CNN for detection and a CNN-RNN model with Connectionist Temporal Classification (CTC) loss and a MobileNet V3 backbone for recognition. This approach aims to improve model performance using datasets from Ontario, Quebec, California, and New York State, achieving a recall rate of 92% on the Centre for Pattern Recognition and Machine Intelligence (CENPARMI) dataset and 90% on the UFPR-ALPR dataset. It includes a detailed error analysis to identify the causes of false positives. Additionally, the research examines the role of font features in license plate (LP) recognition, analyzing fonts like Driver Gothic, Dreadnought, California Clarendon, and Zurich Extra Condensed with the OpenALPR system. It discovers significant performance discrepancies influenced by font characteristics, offering insights for future LPD system enhancements. Keywords: Deep Learning, License Plate, Font Evaluation

Title: Process-Supervised Reward Models for Clinical Note Generation: A Scalable Approach Guided by Domain Expertise

• Authors: Hanyin Wang, Qiping Xu, Bolun Liu, Guleid Hussein, Hariprasad Korsapati, Mohamad El Labban, Kingsley Iheasirim, Mohamed Hassan, Gokhan Anil, Brian Bartlett, Jimeng Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12583
• Pdf URL: https://arxiv.org/pdf/2412.12583
• Copy Paste: [[2412.12583]] Process-Supervised Reward Models for Clinical Note Generation: A Scalable Approach Guided by Domain Expertise(https://arxiv.org/abs/2412.12583)
• Keywords: generative, large language model
• Abstract: Process-supervised reward models (PRMs), which verify large language model (LLM) outputs step-by-step, have achieved significant success in mathematical and coding problems. However, their application to other domains remains largely unexplored. In this work, we train a PRM to provide step-level reward signals for clinical notes generated by LLMs from patient-doctor dialogues. Guided by real-world clinician expertise, we carefully designed step definitions for clinical notes and utilized Gemini-Pro 1.5 to automatically generate process supervision data at scale. Our proposed PRM, trained on the LLaMA-3.1 8B instruct model, demonstrated superior performance compared to Gemini-Pro 1.5 and an outcome-supervised reward model (ORM) across two key evaluations: (1) the accuracy of selecting gold-reference samples from error-containing samples, achieving 98.8% (versus 61.3% for ORM and 93.8% for Gemini-Pro 1.5), and (2) the accuracy of selecting physician-preferred notes, achieving 56.2% (compared to 51.2% for ORM and 50.0% for Gemini-Pro 1.5). Additionally, we conducted ablation studies to determine optimal loss functions and data selection strategies, along with physician reader studies to explore predictors of downstream Best-of-N performance. Our promising results suggest the potential of PRMs to extend beyond the clinical domain, offering a scalable and effective solution for diverse generative tasks.

Title: PerSphere: A Comprehensive Framework for Multi-Faceted Perspective Retrieval and Summarization

• Authors: Yun Luo, Yingjie Li, Xiangkun Hu, Qinglin Qi, Fang Guo, Qipeng Guo, Zheng Zhang, Yue Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12588
• Pdf URL: https://arxiv.org/pdf/2412.12588
• Copy Paste: [[2412.12588]] PerSphere: A Comprehensive Framework for Multi-Faceted Perspective Retrieval and Summarization(https://arxiv.org/abs/2412.12588)
• Keywords: extraction
• Abstract: As online platforms and recommendation algorithms evolve, people are increasingly trapped in echo chambers, leading to biased understandings of various issues. To combat this issue, we have introduced PerSphere, a benchmark designed to facilitate multi-faceted perspective retrieval and summarization, thus breaking free from these information silos. For each query within PerSphere, there are two opposing claims, each supported by distinct, non-overlapping perspectives drawn from one or more documents. Our goal is to accurately summarize these documents, aligning the summaries with the respective claims and their underlying perspectives. This task is structured as a two-step end-to-end pipeline that includes comprehensive document retrieval and multi-faceted summarization. Furthermore, we propose a set of metrics to evaluate the comprehensiveness of the retrieval and summarization content. Experimental results on various counterparts for the pipeline show that recent models struggle with such a complex task. Analysis shows that the main challenge lies in long context and perspective extraction, and we propose a simple but effective multi-agent summarization system, offering a promising solution to enhance performance on PerSphere.

Title: LLMs are Also Effective Embedding Models: An In-depth Overview

• Authors: Chongyang Tao, Tao Shen, Shen Gao, Junshuo Zhang, Zhen Li, Zhengwei Tao, Shuai Ma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12591
• Pdf URL: https://arxiv.org/pdf/2412.12591
• Copy Paste: [[2412.12591]] LLMs are Also Effective Embedding Models: An In-depth Overview(https://arxiv.org/abs/2412.12591)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) have revolutionized natural language processing by achieving state-of-the-art performance across various tasks. Recently, their effectiveness as embedding models has gained attention, marking a paradigm shift from traditional encoder-only models like ELMo and BERT to decoder-only, large-scale LLMs such as GPT, LLaMA, and Mistral. This survey provides an in-depth overview of this transition, beginning with foundational techniques before the LLM era, followed by LLM-based embedding models through two main strategies to derive embeddings from LLMs. 1) Direct prompting: We mainly discuss the prompt designs and the underlying rationale for deriving competitive embeddings. 2) Data-centric tuning: We cover extensive aspects that affect tuning an embedding model, including model architecture, training objectives, data constructions, etc. Upon the above, we also cover advanced methods, such as handling longer texts, and multilingual and cross-modal data. Furthermore, we discuss factors affecting choices of embedding models, such as performance/efficiency comparisons, dense vs sparse embeddings, pooling strategies, and scaling law. Lastly, the survey highlights the limitations and challenges in adapting LLMs for embeddings, including cross-task embedding quality, trade-offs between efficiency and accuracy, low-resource, long-context, data bias, robustness, etc. This survey serves as a valuable resource for researchers and practitioners by synthesizing current advancements, highlighting key challenges, and offering a comprehensive framework for future work aimed at enhancing the effectiveness and efficiency of LLMs as embedding models.

Title: A Simple and Efficient Baseline for Zero-Shot Generative Classification

• Authors: Zipeng Qi, Buhua Liu, Shiyan Zhang, Bao Li, Zhiqiang Xu, Haoyi Xiong, Zeke Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12594
• Pdf URL: https://arxiv.org/pdf/2412.12594
• Copy Paste: [[2412.12594]] A Simple and Efficient Baseline for Zero-Shot Generative Classification(https://arxiv.org/abs/2412.12594)
• Keywords: diffusion, generative
• Abstract: Large diffusion models have become mainstream generative models in both academic studies and industrial AIGC applications. Recently, a number of works further explored how to employ the power of large diffusion models as zero-shot classifiers. While recent zero-shot diffusion-based classifiers have made performance advancement on benchmark datasets, they still suffered badly from extremely slow classification speed (e.g., ~1000 seconds per classifying single image on ImageNet). The extremely slow classification speed strongly prohibits existing zero-shot diffusion-based classifiers from practical applications. In this paper, we propose an embarrassingly simple and efficient zero-shot Gaussian Diffusion Classifiers (GDC) via pretrained text-to-image diffusion models and DINOv2. The proposed GDC can not only significantly surpass previous zero-shot diffusion-based classifiers by over 10 points (61.40% - 71.44%) on ImageNet, but also accelerate more than 30000 times (1000 - 0.03 seconds) classifying a single image on ImageNet. Additionally, it provides probability interpretation of the results. Our extensive experiments further demonstrate that GDC can achieve highly competitive zero-shot classification performance over various datasets and can promisingly self-improve with stronger diffusion models. To the best of our knowledge, the proposed GDC is the first zero-shot diffusionbased classifier that exhibits both competitive accuracy and practical efficiency.

Title: OpenViewer: Openness-Aware Multi-View Learning

• Authors: Shide Du, Zihan Fang, Yanchao Tan, Changwei Wang, Shiping Wang, Wenzhong Guo
• Subjects: cs.CV, stat.AP, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.12596
• Pdf URL: https://arxiv.org/pdf/2412.12596
• Copy Paste: [[2412.12596]] OpenViewer: Openness-Aware Multi-View Learning(https://arxiv.org/abs/2412.12596)
• Keywords: interpretability
• Abstract: Multi-view learning methods leverage multiple data sources to enhance perception by mining correlations across views, typically relying on predefined categories. However, deploying these models in real-world scenarios presents two primary openness challenges. 1) Lack of Interpretability: The integration mechanisms of multi-view data in existing black-box models remain poorly explained; 2) Insufficient Generalization: Most models are not adapted to multi-view scenarios involving unknown categories. To address these challenges, we propose OpenViewer, an openness-aware multi-view learning framework with theoretical support. This framework begins with a Pseudo-Unknown Sample Generation Mechanism to efficiently simulate open multi-view environments and previously adapt to potential unknown samples. Subsequently, we introduce an Expression-Enhanced Deep Unfolding Network to intuitively promote interpretability by systematically constructing functional prior-mapping modules and effectively providing a more transparent integration mechanism for multi-view data. Additionally, we establish a Perception-Augmented Open-Set Training Regime to significantly enhance generalization by precisely boosting confidences for known categories and carefully suppressing inappropriate confidences for unknown ones. Experimental results demonstrate that OpenViewer effectively addresses openness challenges while ensuring recognition performance for both known and unknown samples. The code is released at this https URL.

Title: SynthCypher: A Fully Synthetic Data Generation Framework for Text-to-Cypher Querying in Knowledge Graphs

• Authors: Aman Tiwari, Shiva Krishna Reddy Malay, Vikas Yadav, Masoud Hashemi, Sathwik Tejaswi Madhusudhan
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12612
• Pdf URL: https://arxiv.org/pdf/2412.12612
• Copy Paste: [[2412.12612]] SynthCypher: A Fully Synthetic Data Generation Framework for Text-to-Cypher Querying in Knowledge Graphs(https://arxiv.org/abs/2412.12612)
• Keywords: large language model
• Abstract: Cypher, the query language for Neo4j graph databases, plays a critical role in enabling graph-based analytics and data exploration. While substantial research has been dedicated to natural language to SQL query generation (Text2SQL), the analogous problem for graph databases referred to as Text2Cypher remains underexplored. In this work, we introduce SynthCypher, a fully synthetic and automated data generation pipeline designed to address this gap. SynthCypher employs a novel LLMSupervised Generation-Verification framework, ensuring syntactically and semantically correct Cypher queries across diverse domains and query complexities. Using this pipeline, we create SynthCypher Dataset, a large-scale benchmark containing 29.8k Text2Cypher instances. Fine-tuning open-source large language models (LLMs), including LLaMa-3.1- 8B, Mistral-7B, and QWEN-7B, on SynthCypher yields significant performance improvements of up to 40% on the Text2Cypher test set and 30% on the SPIDER benchmark adapted for graph databases. This work demonstrates that high-quality synthetic data can effectively advance the state-of-the-art in Text2Cypher tasks.

Title: Multi-Domain Features Guided Supervised Contrastive Learning for Radar Target Detection

• Authors: Junjie Wang, Yuze Gao, Dongying Li, Wenxian Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12620
• Pdf URL: https://arxiv.org/pdf/2412.12620
• Copy Paste: [[2412.12620]] Multi-Domain Features Guided Supervised Contrastive Learning for Radar Target Detection(https://arxiv.org/abs/2412.12620)
• Keywords: robust
• Abstract: Detecting small targets in sea clutter is challenging due to dynamic maritime conditions. Existing solutions either model sea clutter for detection or extract target features based on clutter-target echo differences, including statistical and deep features. While more common, the latter often excels in controlled scenarios but struggles with robust detection and generalization in diverse environments, limiting practical use. In this letter, we propose a multi-domain features guided supervised contrastive learning (MDFG_SCL) method, which integrates statistical features derived from multi-domain differences with deep features obtained through supervised contrastive learning, thereby capturing both low-level domain-specific variations and high-level semantic information. This comprehensive feature integration enables the model to effectively distinguish between small targets and sea clutter, even under challenging conditions. Experiments conducted on real-world datasets demonstrate that the proposed shallow-to-deep detector not only achieves effective identification of small maritime targets but also maintains superior detection performance across varying sea conditions, outperforming the mainstream unsupervised contrastive learning and supervised contrastive learning methods.

Title: Jailbreaking? One Step Is Enough!

• Authors: Weixiong Zheng, Peijian Zeng, Yiwei Li, Hongyan Wu, Nankai Lin, Junhao Chen, Aimin Yang, Yongmei Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12621
• Pdf URL: https://arxiv.org/pdf/2412.12621
• Copy Paste: [[2412.12621]] Jailbreaking? One Step Is Enough!(https://arxiv.org/abs/2412.12621)
• Keywords: defense, attack, large language model
• Abstract: Large language models (LLMs) excel in various tasks but remain vulnerable to jailbreak attacks, where adversaries manipulate prompts to generate harmful outputs. Examining jailbreak prompts helps uncover the shortcomings of LLMs. However, current jailbreak methods and the target model's defenses are engaged in an independent and adversarial process, resulting in the need for frequent attack iterations and redesigning attacks for different models. To address these gaps, we propose a Reverse Embedded Defense Attack (REDA) mechanism that disguises the attack intention as the "defense". intention against harmful content. Specifically, REDA starts from the target response, guiding the model to embed harmful content within its defensive measures, thereby relegating harmful content to a secondary role and making the model believe it is performing a defensive task. The attacking model considers that it is guiding the target model to deal with harmful content, while the target model thinks it is performing a defensive task, creating an illusion of cooperation between the two. Additionally, to enhance the model's confidence and guidance in "defensive" intentions, we adopt in-context learning (ICL) with a small number of attack examples and construct a corresponding dataset of attack examples. Extensive evaluations demonstrate that the REDA method enables cross-model attacks without the need to redesign attack strategies for different models, enables successful jailbreak in one iteration, and outperforms existing methods on both open-source and closed-source models.

Title: Improving the Transferability of 3D Point Cloud Attack via Spectral-aware Admix and Optimization Designs

• Authors: Shiyu Hu, Daizong Liu, Wei Hu
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.12626
• Pdf URL: https://arxiv.org/pdf/2412.12626
• Copy Paste: [[2412.12626]] Improving the Transferability of 3D Point Cloud Attack via Spectral-aware Admix and Optimization Designs(https://arxiv.org/abs/2412.12626)
• Keywords: attack
• Abstract: Deep learning models for point clouds have shown to be vulnerable to adversarial attacks, which have received increasing attention in various safety-critical applications such as autonomous driving, robotics, and surveillance. Existing 3D attackers generally design various attack strategies in the white-box setting, requiring the prior knowledge of 3D model details. However, real-world 3D applications are in the black-box setting, where we can only acquire the outputs of the target classifier. Although few recent works try to explore the black-box attack, they still achieve limited attack success rates (ASR). To alleviate this issue, this paper focuses on attacking the 3D models in a transfer-based black-box setting, where we first carefully design adversarial examples in a white-box surrogate model and then transfer them to attack other black-box victim models. Specifically, we propose a novel Spectral-aware Admix with Augmented Optimization method (SAAO) to improve the adversarial transferability. In particular, since traditional Admix strategy are deployed in the 2D domain that adds pixel-wise images for perturbing, we can not directly follow it to merge point clouds in coordinate domain as it will destroy the geometric shapes. Therefore, we design spectral-aware fusion that performs Graph Fourier Transform (GFT) to get spectral features of the point clouds and add them in the spectral domain. Afterward, we run a few steps with spectral-aware weighted Admix to select better optimization paths as well as to adjust corresponding learning weights. At last, we run more steps to generate adversarial spectral feature along the optimization path and perform Inverse-GFT on the adversarial spectral feature to obtain the adversarial example in the data domain. Experiments show that our SAAO achieves better transferability compared to existing 3D attack methods.

Title: Make Imagination Clearer! Stable Diffusion-based Visual Imagination for Multimodal Machine Translation

• Authors: Andong Chen, Yuchen Song, Kehai Chen, Muyun Yang, Tiejun Zhao, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12627
• Pdf URL: https://arxiv.org/pdf/2412.12627
• Copy Paste: [[2412.12627]] Make Imagination Clearer! Stable Diffusion-based Visual Imagination for Multimodal Machine Translation(https://arxiv.org/abs/2412.12627)
• Keywords: diffusion, large language model
• Abstract: Visual information has been introduced for enhancing machine translation (MT), and its effectiveness heavily relies on the availability of large amounts of bilingual parallel sentence pairs with manual image annotations. In this paper, we introduce a stable diffusion-based imagination network into a multimodal large language model (MLLM) to explicitly generate an image for each source sentence, thereby advancing the multimodel MT. Particularly, we build heuristic human feedback with reinforcement learning to ensure the consistency of the generated image with the source sentence without the supervision of image annotation, which breaks the bottleneck of using visual information in MT. Furthermore, the proposed method enables imaginative visual information to be integrated into large-scale text-only MT in addition to multimodal MT. Experimental results show that our model significantly outperforms existing multimodal MT and text-only MT, especially achieving an average improvement of more than 14 BLEU points on Multi30K multimodal MT benchmarks.

Title: What External Knowledge is Preferred by LLMs? Characterizing and Exploring Chain of Evidence in Imperfect Context

• Authors: Zhiyuan Chang, Mingyang Li, Xiaojun Jia, Junjie Wang, Yuekai Huang, Qing Wang, Yihao Huang, Yang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12632
• Pdf URL: https://arxiv.org/pdf/2412.12632
• Copy Paste: [[2412.12632]] What External Knowledge is Preferred by LLMs? Characterizing and Exploring Chain of Evidence in Imperfect Context(https://arxiv.org/abs/2412.12632)
• Keywords: robust, large language model
• Abstract: Incorporating external knowledge into large language models (LLMs) has emerged as a promising approach to mitigate outdated knowledge and hallucination in LLMs. However, external knowledge is often imperfect. In addition to useful knowledge, external knowledge is rich in irrelevant or misinformation in the context that can impair the reliability of LLM responses. This paper focuses on LLMs' preferred external knowledge in imperfect contexts when handling multi-hop QA. Inspired by criminal procedural law's Chain of Evidence (CoE), we characterize that knowledge preferred by LLMs should maintain both relevance to the question and mutual support among knowledge pieces. Accordingly, we propose an automated CoE discrimination approach and explore LLMs' preferences from their effectiveness, faithfulness and robustness, as well as CoE's usability in a naive Retrieval-Augmented Generation (RAG) case. The evaluation on five LLMs reveals that CoE enhances LLMs through more accurate generation, stronger answer faithfulness, better robustness against knowledge conflict, and improved performance in a popular RAG case.

Title: Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree

• Authors: Xiangxiang Gao, Weisheng Xie, Yiwei Xiang, Feng Ji
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12639
• Pdf URL: https://arxiv.org/pdf/2412.12639
• Copy Paste: [[2412.12639]] Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree(https://arxiv.org/abs/2412.12639)
• Keywords: transformer, large language model
• Abstract: Striking an optimal balance between minimal drafting latency and high speculation accuracy to enhance the inference speed of Large Language Models remains a significant challenge in speculative decoding. In this paper, we introduce Falcon, an innovative semi-autoregressive speculative decoding framework fashioned to augment both the drafter's parallelism and output quality. Falcon incorporates the Coupled Sequential Glancing Distillation technique, which fortifies inter-token dependencies within the same block, leading to increased speculation accuracy. We offer a comprehensive theoretical analysis to illuminate the underlying mechanisms. Additionally, we introduce a Custom-Designed Decoding Tree, which permits the drafter to generate multiple tokens in a single forward pass and accommodates multiple forward passes as needed, thereby boosting the number of drafted tokens and significantly improving the overall acceptance rate. Comprehensive evaluations on benchmark datasets such as MT-Bench, HumanEval, and GSM8K demonstrate Falcon's superior acceleration capabilities. The framework achieves a lossless speedup ratio ranging from 2.91x to 3.51x when tested on the Vicuna and LLaMA2-Chat model series. These results outstrip existing speculative decoding methods for LLMs, including Eagle, Medusa, Lookahead, SPS, and PLD, while maintaining a compact drafter architecture equivalent to merely two Transformer layers.

Title: Building Gradient Bridges: Label Leakage from Restricted Gradient Sharing in Federated Learning

• Authors: Rui Zhang, Ka-Ho Chow, Ping Li
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.12640
• Pdf URL: https://arxiv.org/pdf/2412.12640
• Copy Paste: [[2412.12640]] Building Gradient Bridges: Label Leakage from Restricted Gradient Sharing in Federated Learning(https://arxiv.org/abs/2412.12640)
• Keywords: privacy, defense, attack, federate
• Abstract: The growing concern over data privacy, the benefits of utilizing data from diverse sources for model training, and the proliferation of networked devices with enhanced computational capabilities have all contributed to the rise of federated learning (FL). The clients in FL collaborate to train a global model by uploading gradients computed on their private datasets without collecting raw data. However, a new attack surface has emerged from gradient sharing, where adversaries can restore the label distribution of a victim's private data by analyzing the obtained gradients. To mitigate this privacy leakage, existing lightweight defenses restrict the sharing of gradients, such as encrypting the final-layer gradients or locally updating the parameters within. In this paper, we introduce a novel attack called Gradient Bridge (GDBR) that recovers the label distribution of training data from the limited gradient information shared in FL. GDBR explores the relationship between the layer-wise gradients, tracks the flow of gradients, and analytically derives the batch training labels. Extensive experiments show that GDBR can accurately recover more than 80% of labels in various FL settings. GDBR highlights the inadequacy of restricted gradient sharing-based defenses and calls for the design of effective defense schemes in FL.

Title: RDPI: A Refine Diffusion Probability Generation Method for Spatiotemporal Data Imputation

• Authors: Zijin Liu, Xiang Zhao, You Song
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12642
• Pdf URL: https://arxiv.org/pdf/2412.12642
• Copy Paste: [[2412.12642]] RDPI: A Refine Diffusion Probability Generation Method for Spatiotemporal Data Imputation(https://arxiv.org/abs/2412.12642)
• Keywords: diffusion
• Abstract: Spatiotemporal data imputation plays a crucial role in various fields such as traffic flow monitoring, air quality assessment, and climate prediction. However, spatiotemporal data collected by sensors often suffer from temporal incompleteness, and the sparse and uneven distribution of sensors leads to missing data in the spatial dimension. Among existing methods, autoregressive approaches are prone to error accumulation, while simple conditional diffusion models fail to adequately capture the spatiotemporal relationships between observed and missing data. To address these issues, we propose a novel two-stage Refined Diffusion Probability Impuation (RDPI) framework based on an initial network and a conditional diffusion model. In the initial stage, deterministic imputation methods are used to generate preliminary estimates of the missing data. In the refinement stage, residuals are treated as the diffusion target, and observed values are innovatively incorporated into the forward process. This results in a conditional diffusion model better suited for spatiotemporal data imputation, bridging the gap between the preliminary estimates and the true values. Experiments on multiple datasets demonstrate that RDPI not only achieves state-of-the-art imputation accuracy but also significantly reduces sampling computational costs.

Title: LLM-based Discriminative Reasoning for Knowledge Graph Question Answering

• Authors: Mufan Xu, Kehai Chen, Xuefeng Bai, Muyun Yang, Tiejun Zhao, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12643
• Pdf URL: https://arxiv.org/pdf/2412.12643
• Copy Paste: [[2412.12643]] LLM-based Discriminative Reasoning for Knowledge Graph Question Answering(https://arxiv.org/abs/2412.12643)
• Keywords: transformer, generative, large language model
• Abstract: Large language models (LLMs) based on generative pre-trained Transformer have achieved remarkable performance on knowledge graph question-answering (KGQA) tasks. However, LLMs often produce ungrounded subgraph planning or reasoning results in KGQA due to the hallucinatory behavior brought by the generative paradigm, which may hinder the advancement of the LLM-based KGQA model. To deal with the issue, we propose a novel LLM-based Discriminative Reasoning (LDR) method to explicitly model the subgraph retrieval and answer inference process. By adopting discriminative strategies, the proposed LDR method not only enhances the capability of LLMs to retrieve question-related subgraphs but also alleviates the issue of ungrounded reasoning brought by the generative paradigm of LLMs. Experimental results show that the proposed approach outperforms multiple strong comparison methods, along with achieving state-of-the-art performance on two widely used WebQSP and CWQ benchmarks.

Title: iPrOp: Interactive Prompt Optimization for Large Language Models with a Human in the Loop

• Authors: Jiahui Li, Roman Klinger
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12644
• Pdf URL: https://arxiv.org/pdf/2412.12644
• Copy Paste: [[2412.12644]] iPrOp: Interactive Prompt Optimization for Large Language Models with a Human in the Loop(https://arxiv.org/abs/2412.12644)
• Keywords: large language model
• Abstract: Prompt engineering has made significant contributions to the era of large language models, yet its effectiveness depends on the skills of a prompt author. Automatic prompt optimization can support the prompt development process, but requires annotated data. This paper introduces $\textit{iPrOp}$, a novel Interactive Prompt Optimization system, to bridge manual prompt engineering and automatic prompt optimization. With human intervention in the optimization loop, $\textit{iPrOp}$ offers users the flexibility to assess evolving prompts. We present users with prompt variations, selected instances, large language model predictions accompanied by corresponding explanations, and performance metrics derived from a subset of the training data. This approach empowers users to choose and further refine the provided prompts based on their individual preferences and needs. This system not only assists non-technical domain experts in generating optimal prompts tailored to their specific tasks or domains, but also enables to study the intrinsic parameters that influence the performance of prompt optimization. Our evaluation shows that our system has the capability to generate improved prompts, leading to enhanced task performance.

Title: Exploring AI-Enabled Cybersecurity Frameworks: Deep-Learning Techniques, GPU Support, and Future Enhancements

• Authors: Tobias Becher, Simon Torka
• Subjects: cs.CR, cs.AI, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12648
• Pdf URL: https://arxiv.org/pdf/2412.12648
• Copy Paste: [[2412.12648]] Exploring AI-Enabled Cybersecurity Frameworks: Deep-Learning Techniques, GPU Support, and Future Enhancements(https://arxiv.org/abs/2412.12648)
• Keywords: security
• Abstract: Traditional rule-based cybersecurity systems have proven highly effective against known malware threats. However, they face challenges in detecting novel threats. To address this issue, emerging cybersecurity systems are incorporating AI techniques, specifically deep-learning algorithms, to enhance their ability to detect incidents, analyze alerts, and respond to events. While these techniques offer a promising approach to combating dynamic security threats, they often require significant computational resources. Therefore, frameworks that incorporate AI-based cybersecurity mechanisms need to support the use of GPUs to ensure optimal performance. Many cybersecurity framework vendors do not provide sufficiently detailed information about their implementation, making it difficult to assess the techniques employed and their effectiveness. This study aims to overcome this limitation by providing an overview of the most used cybersecurity frameworks that utilize AI techniques, specifically focusing on frameworks that provide comprehensive information about their implementation. Our primary objective is to identify the deep-learning techniques employed by these frameworks and evaluate their support for GPU acceleration. We have identified a total of \emph{two} deep-learning algorithms that are utilized by \emph{three} out of 38 selected cybersecurity frameworks. Our findings aim to assist in selecting open-source cybersecurity frameworks for future research and assessing any discrepancies between deep-learning techniques used in theory and practice.

Title: CALA: A Class-Aware Logit Adapter for Few-Shot Class-Incremental Learning

• Authors: Chengyan Liu, Linglan Zhao, Fan Lyu, Kaile Du, Fuyuan Hu, Tao Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12654
• Pdf URL: https://arxiv.org/pdf/2412.12654
• Copy Paste: [[2412.12654]] CALA: A Class-Aware Logit Adapter for Few-Shot Class-Incremental Learning(https://arxiv.org/abs/2412.12654)
• Keywords: robust
• Abstract: Few-Shot Class-Incremental Learning (FSCIL) defines a practical but challenging task where models are required to continuously learn novel concepts with only a few training samples. Due to data scarcity, existing FSCIL methods resort to training a backbone with abundant base data and then keeping it frozen afterward. However, the above operation often causes the backbone to overfit to base classes while overlooking the novel ones, leading to severe confusion between them. To address this issue, we propose Class-Aware Logit Adapter (CALA). Our method involves a lightweight adapter that learns to rectify biased predictions through a pseudo-incremental learning paradigm. In the real FSCIL process, we use the learned adapter to dynamically generate robust balancing factors. These factors can adjust confused novel instances back to their true label space based on their similarity to base classes. Specifically, when confusion is more likely to occur in novel instances that closely resemble base classes, greater rectification is required. Notably, CALA operates on the classifier level, preserving the original feature space, thus it can be flexibly plugged into most of the existing FSCIL works for improved performance. Experiments on three benchmark datasets consistently validate the effectiveness and flexibility of CALA. Codes will be available upon acceptance.

Title: SEG-SAM: Semantic-Guided SAM for Unified Medical Image Segmentation

• Authors: Shuangping Huang, Hao Liang, Qingfeng Wang, Chulong Zhong, Zijian Zhou, Miaojing Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12660
• Pdf URL: https://arxiv.org/pdf/2412.12660
• Copy Paste: [[2412.12660]] SEG-SAM: Semantic-Guided SAM for Unified Medical Image Segmentation(https://arxiv.org/abs/2412.12660)
• Keywords: large language model, segmentation
• Abstract: Recently, developing unified medical image segmentation models gains increasing attention, especially with the advent of the Segment Anything Model (SAM). SAM has shown promising binary segmentation performance in natural domains, however, transferring it to the medical domain remains challenging, as medical images often possess substantial inter-category overlaps. To address this, we propose the SEmantic-Guided SAM (SEG-SAM), a unified medical segmentation model that incorporates semantic medical knowledge to enhance medical segmentation performance. First, to avoid the potential conflict between binary and semantic predictions, we introduce a semantic-aware decoder independent of SAM's original decoder, specialized for both semantic segmentation on the prompted object and classification on unprompted objects in images. To further enhance the model's semantic understanding, we solicit key characteristics of medical categories from large language models and incorporate them into SEG-SAM through a text-to-vision semantic module, adaptively transferring the language information into the visual segmentation task. In the end, we introduce the cross-mask spatial alignment strategy to encourage greater overlap between the predicted masks from SEG-SAM's two decoders, thereby benefiting both predictions. Extensive experiments demonstrate that SEG-SAM outperforms state-of-the-art SAM-based methods in unified binary medical segmentation and task-specific methods in semantic medical segmentation, showcasing promising results and potential for broader medical applications.

Title: A Two-Fold Patch Selection Approach for Improved 360-Degree Image Quality Assessment

• Authors: Abderrezzaq Sendjasni, Seif-Eddine Benkabou, Mohamed-Chaker Larabi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12667
• Pdf URL: https://arxiv.org/pdf/2412.12667
• Copy Paste: [[2412.12667]] A Two-Fold Patch Selection Approach for Improved 360-Degree Image Quality Assessment(https://arxiv.org/abs/2412.12667)
• Keywords: robust
• Abstract: This article presents a novel approach to improving the accuracy of 360-degree perceptual image quality assessment (IQA) through a two-fold patch selection process. Our methodology combines visual patch selection with embedding similarity-based refinement. The first stage focuses on selecting patches from 360-degree images using three distinct sampling methods to ensure comprehensive coverage of visual content for IQA. The second stage, which is the core of our approach, employs an embedding similarity-based selection process to filter and prioritize the most informative patches based on their embeddings similarity distances. This dual selection mechanism ensures that the training data is both relevant and informative, enhancing the model's learning efficiency. Extensive experiments and statistical analyses using three distance metrics across three benchmark datasets validate the effectiveness of our selection algorithm. The results highlight its potential to deliver robust and accurate 360-degree IQA, with performance gains of up to 4.5% in accuracy and monotonicity of quality score prediction, while using only 40% to 50% of the training patches. These improvements are consistent across various configurations and evaluation metrics, demonstrating the strength of the proposed method. The code for the selection process is available at: this https URL.

Title: Adaptive Prototype Replay for Class Incremental Semantic Segmentation

• Authors: Guilin Zhu, Dongyue Wu, Changxin Gao, Runmin Wang, Weidong Yang, Nong Sang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12669
• Pdf URL: https://arxiv.org/pdf/2412.12669
• Copy Paste: [[2412.12669]] Adaptive Prototype Replay for Class Incremental Semantic Segmentation(https://arxiv.org/abs/2412.12669)
• Keywords: segmentation
• Abstract: Class incremental semantic segmentation (CISS) aims to segment new classes during continual steps while preventing the forgetting of old knowledge. Existing methods alleviate catastrophic forgetting by replaying distributions of previously learned classes using stored prototypes or features. However, they overlook a critical issue: in CISS, the representation of class knowledge is updated continuously through incremental learning, whereas prototype replay methods maintain fixed prototypes. This mismatch between updated representation and fixed prototypes limits the effectiveness of the prototype replay strategy. To address this issue, we propose the Adaptive prototype replay (Adapter) for CISS in this paper. Adapter comprises an adaptive deviation compen sation (ADC) strategy and an uncertainty-aware constraint (UAC) loss. Specifically, the ADC strategy dynamically updates the stored prototypes based on the estimated representation shift distance to match the updated representation of old class. The UAC loss reduces prediction uncertainty, aggregating discriminative features to aid in generating compact prototypes. Additionally, we introduce a compensation-based prototype similarity discriminative (CPD) loss to ensure adequate differentiation between similar prototypes, thereby enhancing the efficiency of the adaptive prototype replay strategy. Extensive experiments on Pascal VOC and ADE20K datasets demonstrate that Adapter achieves state-of-the-art results and proves effective across various CISS tasks, particularly in challenging multi-step scenarios. The code and model is available at this https URL.

Title: Structural Pruning via Spatial-aware Information Redundancy for Semantic Segmentation

• Authors: Dongyue Wu, Zilin Guo, Li Yu, Nong Sang, Changxin Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12672
• Pdf URL: https://arxiv.org/pdf/2412.12672
• Copy Paste: [[2412.12672]] Structural Pruning via Spatial-aware Information Redundancy for Semantic Segmentation(https://arxiv.org/abs/2412.12672)
• Keywords: segmentation
• Abstract: In recent years, semantic segmentation has flourished in various applications. However, the high computational cost remains a significant challenge that hinders its further adoption. The filter pruning method for structured network slimming offers a direct and effective solution for the reduction of segmentation networks. Nevertheless, we argue that most existing pruning methods, originally designed for image classification, overlook the fact that segmentation is a location-sensitive task, which consequently leads to their suboptimal performance when applied to segmentation networks. To address this issue, this paper proposes a novel approach, denoted as Spatial-aware Information Redundancy Filter Pruning~(SIRFP), which aims to reduce feature redundancy between channels. First, we formulate the pruning process as a maximum edge weight clique problem~(MEWCP) in graph theory, thereby minimizing the redundancy among the remaining features after pruning. Within this framework, we introduce a spatial-aware redundancy metric based on feature maps, thus endowing the pruning process with location sensitivity to better adapt to pruning segmentation networks. Additionally, based on the MEWCP, we propose a low computational complexity greedy strategy to solve this NP-hard problem, making it feasible and efficient for structured pruning. To validate the effectiveness of our method, we conducted extensive comparative experiments on various challenging datasets. The results demonstrate the superior performance of SIRFP for semantic segmentation tasks.

Title: Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT

• Authors: Jenny Kunz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12674
• Pdf URL: https://arxiv.org/pdf/2412.12674
• Copy Paste: [[2412.12674]] Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT(https://arxiv.org/abs/2412.12674)
• Keywords: robust
• Abstract: Smaller LLMs still face significant challenges even in medium-resourced languages, particularly when it comes to language-specific knowledge -- a problem not easily resolved with machine-translated data. In this case study on Icelandic, we aim to enhance the generation performance of an LLM by specialising it using unstructured text corpora. A key focus is on preventing interference with the models' capabilities of handling longer context during this adaptation. Through ablation studies using various parameter-efficient fine-tuning (PEFT) methods and setups, we find that increasing the number of trainable parameters leads to better and more robust language adaptation. LoRAs placed in the feed-forward layers and bottleneck adapters show promising results with sufficient parameters, while prefix tuning and (IA)3 are not suitable. Although improvements are consistent in 0-shot summarisation, some adapted models struggle with longer context lengths, an issue that can be mitigated by adapting only the final layers.

Title: Detecting Document-level Paraphrased Machine Generated Content: Mimicking Human Writing Style and Involving Discourse Features

• Authors: Yupei Li, Manuel Milling, Lucia Specia, Björn W. Schuller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12679
• Pdf URL: https://arxiv.org/pdf/2412.12679
• Copy Paste: [[2412.12679]] Detecting Document-level Paraphrased Machine Generated Content: Mimicking Human Writing Style and Involving Discourse Features(https://arxiv.org/abs/2412.12679)
• Keywords: transformer, large language model
• Abstract: The availability of high-quality APIs for Large Language Models (LLMs) has facilitated the widespread creation of Machine-Generated Content (MGC), posing challenges such as academic plagiarism and the spread of misinformation. Existing MGC detectors often focus solely on surface-level information, overlooking implicit and structural features. This makes them susceptible to deception by surface-level sentence patterns, particularly for longer texts and in texts that have been subsequently paraphrased. To overcome these challenges, we introduce novel methodologies and datasets. Besides the publicly available dataset Plagbench, we developed the paraphrased Long-Form Question and Answer (paraLFQA) and paraphrased Writing Prompts (paraWP) datasets using GPT and DIPPER, a discourse paraphrasing tool, by extending artifacts from their original versions. To address the challenge of detecting highly similar paraphrased texts, we propose MhBART, an encoder-decoder model designed to emulate human writing style while incorporating a novel difference score mechanism. This model outperforms strong classifier baselines and identifies deceptive sentence patterns. To better capture the structure of longer texts at document level, we propose DTransformer, a model that integrates discourse analysis through PDTB preprocessing to encode structural features. It results in substantial performance gains across both datasets -- 15.5\% absolute improvement on paraLFQA, 4\% absolute improvement on paraWP, and 1.5\% absolute improvement on M4 compared to SOTA approaches.

Title: SemStereo: Semantic-Constrained Stereo Matching Network for Remote Sensing

• Authors: Chen Chen, Liangjin Zhao, Yuanchun He, Yingxuan Long, Kaiqiang Chen, Zhirui Wang, Yanfeng Hu, Xian Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12685
• Pdf URL: https://arxiv.org/pdf/2412.12685
• Copy Paste: [[2412.12685]] SemStereo: Semantic-Constrained Stereo Matching Network for Remote Sensing(https://arxiv.org/abs/2412.12685)
• Keywords: segmentation
• Abstract: Semantic segmentation and 3D reconstruction are two fundamental tasks in remote sensing, typically treated as separate or loosely coupled tasks. Despite attempts to integrate them into a unified network, the constraints between the two heterogeneous tasks are not explicitly modeled, since the pioneering studies either utilize a loosely coupled parallel structure or engage in only implicit interactions, failing to capture the inherent connections. In this work, we explore the connections between the two tasks and propose a new network that imposes semantic constraints on the stereo matching task, both implicitly and explicitly. Implicitly, we transform the traditional parallel structure to a new cascade structure termed Semantic-Guided Cascade structure, where the deep features enriched with semantic information are utilized for the computation of initial disparity maps, enhancing semantic guidance. Explicitly, we propose a Semantic Selective Refinement (SSR) module and a Left-Right Semantic Consistency (LRSC) module. The SSR refines the initial disparity map under the guidance of the semantic map. The LRSC ensures semantic consistency between two views via reducing the semantic divergence after transforming the semantic map from one view to the other using the disparity map. Experiments on the US3D and WHU datasets demonstrate that our method achieves state-of-the-art performance for both semantic segmentation and stereo matching.

Title: XTransplant: A Probe into the Upper Bound Performance of Multilingual Capability and Culture Adaptability in LLMs via Mutual Cross-lingual Feed-forward Transplantation

• Authors: Yangfan Ye, Xiaocheng Feng, Xiachong Feng, Libo Qin, Yichong Huang, Lei Huang, Weitao Ma, Zhirui Zhang, Yunfei Lu, Xiaohui Yan, Duyu Tang, Dandan Tu, Bing Qin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12686
• Pdf URL: https://arxiv.org/pdf/2412.12686
• Copy Paste: [[2412.12686]] XTransplant: A Probe into the Upper Bound Performance of Multilingual Capability and Culture Adaptability in LLMs via Mutual Cross-lingual Feed-forward Transplantation(https://arxiv.org/abs/2412.12686)
• Keywords: large language model
• Abstract: Current large language models (LLMs) often exhibit imbalances in multilingual capabilities and cultural adaptability, largely due to their English-centric pretraining data. To address this imbalance, we propose a probing method named XTransplant that explores cross-lingual latent interactions via cross-lingual feed-forward transplantation during inference stage, with the hope of enabling the model to leverage the strengths of both English and non-English languages. Through extensive pilot experiments, we empirically prove that both the multilingual capabilities and cultural adaptability of LLMs hold the potential to be significantly improved by XTransplant, respectively from En -> non-En and non-En -> En, highlighting the underutilization of current LLMs' multilingual potential. And the patterns observed in these pilot experiments further motivate an offline scaling inference strategy, which demonstrates consistent performance improvements in multilingual and culture-aware tasks, sometimes even surpassing multilingual supervised fine-tuning. And we do hope our further analysis and discussion could help gain deeper insights into XTransplant mechanism.

Title: Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Models

• Authors: Seungeun Oh, Jinhyuk Kim, Jihong Park, Seung-Woo Ko, Tony Q. S. Quek, Seong-Lyun Kim
• Subjects: cs.LG, cs.DC, cs.IT, cs.NI, eess.SP
• Abstract URL: https://arxiv.org/abs/2412.12687
• Pdf URL: https://arxiv.org/pdf/2412.12687
• Copy Paste: [[2412.12687]] Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Models(https://arxiv.org/abs/2412.12687)
• Keywords: large language model
• Abstract: This paper studies a hybrid language model (HLM) architecture that integrates a small language model (SLM) operating on a mobile device with a large language model (LLM) hosted at the base station (BS) of a wireless network. The HLM token generation process follows the speculative inference principle: the SLM's vocabulary distribution is uploaded to the LLM, which either accepts or rejects it, with rejected tokens being resampled by the LLM. While this approach ensures alignment between the vocabulary distributions of the SLM and LLM, it suffers from low token throughput due to uplink transmission and the computation costs of running both language models. To address this, we propose a novel HLM structure coined Uncertainty-aware HLM (U-HLM), wherein the SLM locally measures its output uncertainty, and skips both uplink transmissions and LLM operations for tokens that are likely to be accepted. This opportunistic skipping is enabled by our empirical finding of a linear correlation between the SLM's uncertainty and the LLM's rejection probability. We analytically derive the uncertainty threshold and evaluate its expected risk of rejection. Simulations show that U-HLM reduces uplink transmissions and LLM computation by 45.93%, while achieving up to 97.54% of the LLM's inference accuracy and 2.54$\times$ faster token throughput than HLM without skipping.

Title: ALADE-SNN: Adaptive Logit Alignment in Dynamically Expandable Spiking Neural Networks for Class Incremental Learning

• Authors: Wenyao Ni, Jiangrong Shen, Qi Xu, Huajin Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12696
• Pdf URL: https://arxiv.org/pdf/2412.12696
• Copy Paste: [[2412.12696]] ALADE-SNN: Adaptive Logit Alignment in Dynamically Expandable Spiking Neural Networks for Class Incremental Learning(https://arxiv.org/abs/2412.12696)
• Keywords: extraction
• Abstract: Inspired by the human brain's ability to adapt to new tasks without erasing prior knowledge, we develop spiking neural networks (SNNs) with dynamic structures for Class Incremental Learning (CIL). Our comparative experiments reveal that limited datasets introduce biases in logits distributions among tasks. Fixed features from frozen past-task extractors can cause overfitting and hinder the learning of new tasks. To address these challenges, we propose the ALADE-SNN framework, which includes adaptive logit alignment for balanced feature representation and OtoN suppression to manage weights mapping frozen old features to new classes during training, releasing them during fine-tuning. This approach dynamically adjusts the network architecture based on analytical observations, improving feature extraction and balancing performance between new and old tasks. Experiment results show that ALADE-SNN achieves an average incremental accuracy of 75.42 on the CIFAR100-B0 benchmark over 10 incremental steps. ALADE-SNN not only matches the performance of DNN-based methods but also surpasses state-of-the-art SNN-based continual learning algorithms. This advancement enhances continual learning in neuromorphic computing, offering a brain-inspired, energy-efficient solution for real-time data processing.

Title: Trigger$^3$: Refining Query Correction via Adaptive Model Selector

• Authors: Kepu Zhang, Zhongxiang Sun, Xiao Zhang, Xiaoxue Zang, Kai Zheng, Yang Song, Jun Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12701
• Pdf URL: https://arxiv.org/pdf/2412.12701
• Copy Paste: [[2412.12701]] Trigger$^3$: Refining Query Correction via Adaptive Model Selector(https://arxiv.org/abs/2412.12701)
• Keywords: large language model
• Abstract: In search scenarios, user experience can be hindered by erroneous queries due to typos, voice errors, or knowledge gaps. Therefore, query correction is crucial for search engines. Current correction models, usually small models trained on specific data, often struggle with queries beyond their training scope or those requiring contextual understanding. While the advent of Large Language Models (LLMs) offers a potential solution, they are still limited by their pre-training data and inference cost, particularly for complex queries, making them not always effective for query correction. To tackle these, we propose Trigger$^3$, a large-small model collaboration framework that integrates the traditional correction model and LLM for query correction, capable of adaptively choosing the appropriate correction method based on the query and the correction results from the traditional correction model and LLM. Trigger$^3$ first employs a correction trigger to filter out correct queries. Incorrect queries are then corrected by the traditional correction model. If this fails, an LLM trigger is activated to call the LLM for correction. Finally, for queries that no model can correct, a fallback trigger decides to return the original query. Extensive experiments demonstrate Trigger$^3$ outperforms correction baselines while maintaining efficiency.

Title: More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression

• Authors: Jiebin Zhang, Dawei Zhu, Yifan Song, Wenhao Wu, Chuqiao Kuang, Xiaoguang Li, Lifeng Shang, Qun Liu, Sujian Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12706
• Pdf URL: https://arxiv.org/pdf/2412.12706
• Copy Paste: [[2412.12706]] More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression(https://arxiv.org/abs/2412.12706)
• Keywords: large language model
• Abstract: As large language models (LLMs) process increasing context windows, the memory usage of KV cache has become a critical bottleneck during inference. The mainstream KV compression methods, including KV pruning and KV quantization, primarily focus on either token or precision dimension and seldom explore the efficiency of their combination. In this paper, we comprehensively investigate the token-precision trade-off in KV cache compression. Experiments demonstrate that storing more tokens in the KV cache with lower precision, i.e., quantized pruning, can significantly enhance the long-context performance of LLMs. Furthermore, in-depth analysis regarding token-precision trade-off from a series of key aspects exhibit that, quantized pruning achieves substantial improvements in retrieval-related tasks and consistently performs well across varying input lengths. Moreover, quantized pruning demonstrates notable stability across different KV pruning methods, quantization strategies, and model scales. These findings provide valuable insights into the token-precision trade-off in KV cache compression. We plan to release our code in the near future.

Title: Enhancing Naturalness in LLM-Generated Utterances through Disfluency Insertion

• Authors: Syed Zohaib Hassan, Pierre Lison, Pål Halvorsen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12710
• Pdf URL: https://arxiv.org/pdf/2412.12710
• Copy Paste: [[2412.12710]] Enhancing Naturalness in LLM-Generated Utterances through Disfluency Insertion(https://arxiv.org/abs/2412.12710)
• Keywords: large language model
• Abstract: Disfluencies are a natural feature of spontaneous human speech but are typically absent from the outputs of Large Language Models (LLMs). This absence can diminish the perceived naturalness of synthesized speech, which is an important criteria when building conversational agents that aim to mimick human behaviours. We show how the insertion of disfluencies can alleviate this shortcoming. The proposed approach involves (1) fine-tuning an LLM with Low-Rank Adaptation (LoRA) to incorporate various types of disfluencies into LLM-generated utterances and (2) synthesizing those utterances using a text-to-speech model that supports the generation of speech phenomena such as disfluencies. We evaluated the quality of the generated speech across two metrics: intelligibility and perceived spontaneity. We demonstrate through a user study that the insertion of disfluencies significantly increase the perceived spontaneity of the generated speech. This increase came, however, along with a slight reduction in intelligibility.

Title: Unsupervised UAV 3D Trajectories Estimation with Sparse Point Clouds

• Authors: Hanfang Liang, Yizhuo Yang, Jinming Hu, Jianfei Yang, Fen Liu, Shenghai Yuan
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.12716
• Pdf URL: https://arxiv.org/pdf/2412.12716
• Copy Paste: [[2412.12716]] Unsupervised UAV 3D Trajectories Estimation with Sparse Point Clouds(https://arxiv.org/abs/2412.12716)
• Keywords: security
• Abstract: Compact UAV systems, while advancing delivery and surveillance, pose significant security challenges due to their small size, which hinders detection by traditional methods. This paper presents a cost-effective, unsupervised UAV detection method using spatial-temporal sequence processing to fuse multiple LiDAR scans for accurate UAV tracking in real-world scenarios. Our approach segments point clouds into foreground and background, analyzes spatial-temporal data, and employs a scoring mechanism to enhance detection accuracy. Tested on a public dataset, our solution placed 4th in the CVPR 2024 UG2+ Challenge, demonstrating its practical effectiveness. We plan to open-source all designs, code, and sample data for the research community this http URL.

Title: ASAP: Advancing Semantic Alignment Promotes Multi-Modal Manipulation Detecting and Grounding

• Authors: Zhenxing Zhang, Yaxiong Wang, Lechao Cheng, Zhun Zhong, Dan Guo, Meng Wang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.12718
• Pdf URL: https://arxiv.org/pdf/2412.12718
• Copy Paste: [[2412.12718]] ASAP: Advancing Semantic Alignment Promotes Multi-Modal Manipulation Detecting and Grounding(https://arxiv.org/abs/2412.12718)
• Keywords: large language model
• Abstract: We present ASAP, a new framework for detecting and grounding multi-modal media manipulation (DGM4).Upon thorough examination, we observe that accurate fine-grained cross-modal semantic alignment between the image and text is vital for accurately manipulation detection and grounding. While existing DGM4 methods pay rare attention to the cross-modal alignment, hampering the accuracy of manipulation detecting to step further. To remedy this issue, this work targets to advance the semantic alignment learning to promote this task. Particularly, we utilize the off-the-shelf Multimodal Large-Language Models (MLLMs) and Large Language Models (LLMs) to construct paired image-text pairs, especially for the manipulated instances. Subsequently, a cross-modal alignment learning is performed to enhance the semantic alignment. Besides the explicit auxiliary clues, we further design a Manipulation-Guided Cross Attention (MGCA) to provide implicit guidance for augmenting the manipulation perceiving. With the grounding truth available during training, MGCA encourages the model to concentrate more on manipulated components while downplaying normal ones, enhancing the model's ability to capture manipulations. Extensive experiments are conducted on the DGM4 dataset, the results demonstrate that our model can surpass the comparison method with a clear margin.

Title: Defending LVLMs Against Vision Attacks through Partial-Perception Supervision

• Authors: Qi Zhou, Tianlin Li, Qing Guo, Dongxia Wang, Yun Lin, Yang Liu, Jin Song Dong
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.12722
• Pdf URL: https://arxiv.org/pdf/2412.12722
• Copy Paste: [[2412.12722]] Defending LVLMs Against Vision Attacks through Partial-Perception Supervision(https://arxiv.org/abs/2412.12722)
• Keywords: defense, attack
• Abstract: Recent studies have raised significant concerns regarding the vulnerability of Large Vision Language Models (LVLMs) to maliciously injected or perturbed input images, which can mislead their responses. Existing defense methods show that such vision attacks are sensitive to image modifications especially cropping, using majority voting across responses of modified images as corrected responses. However, these modifications often result in partial images and distort the semantics, which reduces response quality on clean images after voting. Instead of directly using responses from partial images for voting, we investigate using them to supervise the LVLM's responses to the original images. We propose a black-box, training-free method called DPS (Defense through Partial-Perception Supervision). In this approach, the model is prompted using the responses generated by a model that perceives only a partial image. With DPS, the model can adjust its response based on partial image understanding when under attack, while confidently maintaining its original response for clean input. Our findings show that the weak model can supervise the strong model: when faced with an attacked input, the strong model becomes less confident and adjusts its response based on the weak model's partial understanding, effectively defending against the attack. With clean input, it confidently maintains its original response. Empirical experiments show our method outperforms the baseline, cutting the average attack success rate by 76.3% across six datasets on three popular models.

Title: AsyncSC: An Asynchronous Sidechain for Multi-Domain Data Exchange in Internet of Things

• Authors: Lingxiao Yang, Xuewen Dong, Zhiguo Wan, Sheng Gao, Wei Tong, Di Lu, Yulong Shen, Xiaojiang Du
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.12723
• Pdf URL: https://arxiv.org/pdf/2412.12723
• Copy Paste: [[2412.12723]] AsyncSC: An Asynchronous Sidechain for Multi-Domain Data Exchange in Internet of Things(https://arxiv.org/abs/2412.12723)
• Keywords: security
• Abstract: Sidechain techniques improve blockchain scalability and interoperability, providing decentralized exchange and cross-chain collaboration solutions for Internet of Things (IoT) data across various domains. However, current state-of-the-art (SOTA) schemes for IoT multi-domain data exchange are constrained by the need for synchronous networks, hindering efficient cross-chain interactions in discontinuous networks and leading to suboptimal data exchange. In this paper, we propose AsyncSC, a novel asynchronous sidechain construction. It employs a committee to provide Cross-Blockchain as a Service (C-BaaS) for data exchange in multi-domain IoT. To fulfill the need for asynchronous and efficient data exchange, we combine the ideas of aggregate signatures and verifiable delay functions to devise a novel cryptographic primitive called delayed aggregate signature (DAS), which constructs asynchronous cross-chain proofs (ACPs) that ensure the security of cross-chain interactions. To ensure the consistency of asynchronous transactions, we propose a multilevel buffered transaction pool that guarantees the transaction sequencing. We analyze and prove the security of AsyncSC, simulate an asynchronous communication environment, and conduct a comprehensive evaluation. The results show that AsyncSC outperforms SOTA schemes, improving throughput by an average of 1.21 to 3.96 times, reducing transaction latency by 59.76% to 83.61%, and maintaining comparable resource overhead.

Title: RaCFormer: Towards High-Quality 3D Object Detection via Query-based Radar-Camera Fusion

• Authors: Xiaomeng Chu, Jiajun Deng, Guoliang You, Yifan Duan, Houqiang Li, Yanyong Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12725
• Pdf URL: https://arxiv.org/pdf/2412.12725
• Copy Paste: [[2412.12725]] RaCFormer: Towards High-Quality 3D Object Detection via Query-based Radar-Camera Fusion(https://arxiv.org/abs/2412.12725)
• Keywords: secure, transformer
• Abstract: We propose Radar-Camera fusion transformer (RaCFormer) to boost the accuracy of 3D object detection by the following insight. The Radar-Camera fusion in outdoor 3D scene perception is capped by the image-to-BEV transformation--if the depth of pixels is not accurately estimated, the naive combination of BEV features actually integrates unaligned visual content. To avoid this problem, we propose a query-based framework that enables adaptively sample instance-relevant features from both the BEV and the original image view. Furthermore, we enhance system performance by two key designs: optimizing query initialization and strengthening the representational capacity of BEV. For the former, we introduce an adaptive circular distribution in polar coordinates to refine the initialization of object queries, allowing for a distance-based adjustment of query density. For the latter, we initially incorporate a radar-guided depth head to refine the transformation from image view to BEV. Subsequently, we focus on leveraging the Doppler effect of radar and introduce an implicit dynamic catcher to capture the temporal elements within the BEV. Extensive experiments on nuScenes and View-of-Delft (VoD) datasets validate the merits of our design. Remarkably, our method achieves superior results of 64.9% mAP and 70.2% NDS on nuScenes, even outperforming several LiDAR-based detectors. RaCFormer also secures the 1st ranking on the VoD dataset. The code will be released.

Title: SentiQNF: A Novel Approach to Sentiment Analysis Using Quantum Algorithms and Neuro-Fuzzy Systems

• Authors: Kshitij Dave, Nouhaila Innan, Bikash K. Behera, Zahid Mumtaz, Saif Al-Kuwari, Ahmed Farouk
• Subjects: cs.CL, quant-ph
• Abstract URL: https://arxiv.org/abs/2412.12731
• Pdf URL: https://arxiv.org/pdf/2412.12731
• Copy Paste: [[2412.12731]] SentiQNF: A Novel Approach to Sentiment Analysis Using Quantum Algorithms and Neuro-Fuzzy Systems(https://arxiv.org/abs/2412.12731)
• Keywords: robust
• Abstract: Sentiment analysis is an essential component of natural language processing, used to analyze sentiments, attitudes, and emotional tones in various contexts. It provides valuable insights into public opinion, customer feedback, and user experiences. Researchers have developed various classical machine learning and neuro-fuzzy approaches to address the exponential growth of data and the complexity of language structures in sentiment analysis. However, these approaches often fail to determine the optimal number of clusters, interpret results accurately, handle noise or outliers efficiently, and scale effectively to high-dimensional data. Additionally, they are frequently insensitive to input variations. In this paper, we propose a novel hybrid approach for sentiment analysis called the Quantum Fuzzy Neural Network (QFNN), which leverages quantum properties and incorporates a fuzzy layer to overcome the limitations of classical sentiment analysis algorithms. In this study, we test the proposed approach on two Twitter datasets: the Coronavirus Tweets Dataset (CVTD) and the General Sentimental Tweets Dataset (GSTD), and compare it with classical and hybrid algorithms. The results demonstrate that QFNN outperforms all classical, quantum, and hybrid algorithms, achieving 100% and 90% accuracy in the case of CVTD and GSTD, respectively. Furthermore, QFNN demonstrates its robustness against six different noise models, providing the potential to tackle the computational complexity associated with sentiment analysis on a large scale in a noisy environment. The proposed approach expedites sentiment data processing and precisely analyses different forms of textual data, thereby enhancing sentiment classification and insights associated with sentiment analysis.

Title: Gaussian Billboards: Expressive 2D Gaussian Splatting with Textures

• Authors: Sebastian Weiss, Derek Bradley
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.12734
• Pdf URL: https://arxiv.org/pdf/2412.12734
• Copy Paste: [[2412.12734]] Gaussian Billboards: Expressive 2D Gaussian Splatting with Textures(https://arxiv.org/abs/2412.12734)
• Keywords: robust
• Abstract: Gaussian Splatting has recently emerged as the go-to representation for reconstructing and rendering 3D scenes. The transition from 3D to 2D Gaussian primitives has further improved multi-view consistency and surface reconstruction accuracy. In this work we highlight the similarity between 2D Gaussian Splatting (2DGS) and billboards from traditional computer graphics. Both use flat semi-transparent 2D geometry that is positioned, oriented and scaled in 3D space. However 2DGS uses a solid color per splat and an opacity modulated by a Gaussian distribution, where billboards are more expressive, modulating the color with a uv-parameterized texture. We propose to unify these concepts by presenting Gaussian Billboards, a modification of 2DGS to add spatially-varying color achieved using per-splat texture interpolation. The result is a mixture of the two representations, which benefits from both the robust scene optimization power of 2DGS and the expressiveness of texture mapping. We show that our method can improve the sharpness and quality of the scene representation in a wide range of qualitative and quantitative evaluations compared to the original 2DGS implementation.

Title: PolSAM: Polarimetric Scattering Mechanism Informed Segment Anything Model

• Authors: Yuqing Wang, Zhongling Huang, Shuxin Yang, Hao Tang, Xiaolan Qiu, Junwei Han, Dingwen Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12737
• Pdf URL: https://arxiv.org/pdf/2412.12737
• Copy Paste: [[2412.12737]] PolSAM: Polarimetric Scattering Mechanism Informed Segment Anything Model(https://arxiv.org/abs/2412.12737)
• Keywords: extraction, interpretability, segmentation
• Abstract: PolSAR data presents unique challenges due to its rich and complex characteristics. Existing data representations, such as complex-valued data, polarimetric features, and amplitude images, are widely used. However, these formats often face issues related to usability, interpretability, and data integrity. Most feature extraction networks for PolSAR are small, limiting their ability to capture features effectively. To address these issues, We propose the Polarimetric Scattering Mechanism-Informed SAM (PolSAM), an enhanced Segment Anything Model (SAM) that integrates domain-specific scattering characteristics and a novel prompt generation strategy. PolSAM introduces Microwave Vision Data (MVD), a lightweight and interpretable data representation derived from polarimetric decomposition and semantic correlations. We propose two key components: the Feature-Level Fusion Prompt (FFP), which fuses visual tokens from pseudo-colored SAR images and MVD to address modality incompatibility in the frozen SAM encoder, and the Semantic-Level Fusion Prompt (SFP), which refines sparse and dense segmentation prompts using semantic information. Experimental results on the PhySAR-Seg datasets demonstrate that PolSAM significantly outperforms existing SAM-based and multimodal fusion models, improving segmentation accuracy, reducing data storage, and accelerating inference time. The source code and datasets will be made publicly available at \url{this https URL}.

Title: Deep Learning for Resilient Adversarial Decision Fusion in Byzantine Networks

• Authors: Kassem Kallas
• Subjects: cs.LG, cs.CR, cs.MA, cs.SI
• Abstract URL: https://arxiv.org/abs/2412.12739
• Pdf URL: https://arxiv.org/pdf/2412.12739
• Copy Paste: [[2412.12739]] Deep Learning for Resilient Adversarial Decision Fusion in Byzantine Networks(https://arxiv.org/abs/2412.12739)
• Keywords: attack, robust
• Abstract: This paper introduces a deep learning-based framework for resilient decision fusion in adversarial multi-sensor networks, providing a unified mathematical setup that encompasses diverse scenarios, including varying Byzantine node proportions, synchronized and unsynchronized attacks, unbalanced priors, adaptive strategies, and Markovian states. Unlike traditional methods, which depend on explicit parameter tuning and are limited by scenario-specific assumptions, the proposed approach employs a deep neural network trained on a globally constructed dataset to generalize across all cases without requiring adaptation. Extensive simulations validate the method's robustness, achieving superior accuracy, minimal error probability, and scalability compared to state-of-the-art techniques, while ensuring computational efficiency for real-time applications. This unified framework demonstrates the potential of deep learning to revolutionize decision fusion by addressing the challenges posed by Byzantine nodes in dynamic adversarial environments.

Title: Open-World Panoptic Segmentation

• Authors: Matteo Sodano, Federico Magistri, Jens Behley, Cyrill Stachniss
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.12740
• Pdf URL: https://arxiv.org/pdf/2412.12740
• Copy Paste: [[2412.12740]] Open-World Panoptic Segmentation(https://arxiv.org/abs/2412.12740)
• Keywords: robust, segmentation
• Abstract: Perception is a key building block of autonomously acting vision systems such as autonomous vehicles. It is crucial that these systems are able to understand their surroundings in order to operate safely and robustly. Additionally, autonomous systems deployed in unconstrained real-world scenarios must be able of dealing with novel situations and object that have never been seen before. In this article, we tackle the problem of open-world panoptic segmentation, i.e., the task of discovering new semantic categories and new object instances at test time, while enforcing consistency among the categories that we incrementally discover. We propose Con2MAV, an approach for open-world panoptic segmentation that extends our previous work, ContMAV, which was developed for open-world semantic segmentation. Through extensive experiments across multiple datasets, we show that our model achieves state-of-the-art results on open-world segmentation tasks, while still performing competitively on the known categories. We will open-source our implementation upon acceptance. Additionally, we propose PANIC (Panoptic ANomalies In Context), a benchmark for evaluating open-world panoptic segmentation in autonomous driving scenarios. This dataset, recorded with a multi-modal sensor suite mounted on a car, provides high-quality, pixel-wise annotations of anomalous objects at both semantic and instance level. Our dataset contains 800 images, with more than 50 unknown classes, i.e., classes that do not appear in the training set, and 4000 object instances, making it an extremely challenging dataset for open-world segmentation tasks in the autonomous driving scenario. We provide competitions for multiple open-world tasks on a hidden test set. Our dataset and competitions are available at this https URL.

Title: Automated Penetration Testing: Formalization and Realization

• Authors: Charilaos Skandylas, Mikael Asplund
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2412.12745
• Pdf URL: https://arxiv.org/pdf/2412.12745
• Copy Paste: [[2412.12745]] Automated Penetration Testing: Formalization and Realization(https://arxiv.org/abs/2412.12745)
• Keywords: security
• Abstract: Recent changes in standards and regulations, driven by the increasing importance of software systems in meeting societal needs, mandate increased security testing of software systems. Penetration testing has been shown to be a reliable method to asses software system security. However, manual penetration testing is labor-intensive and requires highly skilled practitioners. Given the shortage of cybersecurity experts and current societal needs, increasing the degree of automation involved in penetration testing can aid in fulfilling the demands for increased security testing. In this work, we formally express the penetration testing problem at the architectural level and suggest a general self-organizing architecture that can be instantiated to automate penetration testing of real systems. We further describe and implement a specialization of the architecture in the ADAPT tool, targeting systems composed of hosts and services. We evaluate and demonstrate the feasibility of ADAPT by automatically performing penetration tests with success against: Metasploitable2, Metasploitable3, and a realistic virtual network used as a lab environment for penetration tester training.

Title: EmbedFuzz: High Speed Fuzzing Through Transplantation

• Authors: Florian Hofhammer, Qinying Wang, Atri Bhattacharyya, Majid Salehi, Bruno Crispo, Manuel Egele, Mathias Payer, Marcel Busch
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2412.12746
• Pdf URL: https://arxiv.org/pdf/2412.12746
• Copy Paste: [[2412.12746]] EmbedFuzz: High Speed Fuzzing Through Transplantation(https://arxiv.org/abs/2412.12746)
• Keywords: security
• Abstract: Dynamic analysis and especially fuzzing are challenging tasks for embedded firmware running on modern low-end Microcontroller Units (MCUs) due to performance overheads from instruction emulation, the difficulty of emulating the vast space of available peripherals, and low availability of open-source embedded firmware. Consequently, efficient security testing of MCU firmware has proved to be a resource- and engineering-heavy endeavor. EmbedFuzz introduces an efficient end-to-end fuzzing framework for MCU firmware. Our novel firmware transplantation technique converts binary MCU firmware to a functionally equivalent and fuzzing-enhanced version of the firmware which executes on a compatible high-end device at native performance. Besides the performance gains, our system enables advanced introspection capabilities based on tooling for typical Linux user space processes, thus simplifying analysis of crashes and bug triaging. In our evaluation against state-of-the-art MCU fuzzers, EmbedFuzz exhibits up to eight-fold fuzzing throughput while consuming at most a fourth of the energy thanks to its native execution.

Title: Progressive Monitoring of Generative Model Training Evolution

• Authors: Vidya Prasad, Anna Vilanova, Nicola Pezzotti
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12755
• Pdf URL: https://arxiv.org/pdf/2412.12755
• Copy Paste: [[2412.12755]] Progressive Monitoring of Generative Model Training Evolution(https://arxiv.org/abs/2412.12755)
• Keywords: generative
• Abstract: While deep generative models (DGMs) have gained popularity, their susceptibility to biases and other inefficiencies that lead to undesirable outcomes remains an issue. With their growing complexity, there is a critical need for early detection of issues to achieve desired results and optimize resources. Hence, we introduce a progressive analysis framework to monitor the training process of DGMs. Our method utilizes dimensionality reduction techniques to facilitate the inspection of latent representations, the generated and real distributions, and their evolution across training iterations. This monitoring allows us to pause and fix the training method if the representations or distributions progress undesirably. This approach allows for the analysis of a models' training dynamics and the timely identification of biases and failures, minimizing computational loads. We demonstrate how our method supports identifying and mitigating biases early in training a Generative Adversarial Network (GAN) and improving the quality of the generated data distribution.

Title: Towards a Training Free Approach for 3D Scene Editing

• Authors: Vivek Madhavaram, Shivangana Rawat, Chaitanya Devaguptapu, Charu Sharma, Manohar Kaul
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12766
• Pdf URL: https://arxiv.org/pdf/2412.12766
• Copy Paste: [[2412.12766]] Towards a Training Free Approach for 3D Scene Editing(https://arxiv.org/abs/2412.12766)
• Keywords: diffusion
• Abstract: Text driven diffusion models have shown remarkable capabilities in editing images. However, when editing 3D scenes, existing works mostly rely on training a NeRF for 3D editing. Recent NeRF editing methods leverages edit operations by deploying 2D diffusion models and project these edits into 3D space. They require strong positional priors alongside text prompt to identify the edit location. These methods are operational on small 3D scenes and are more generalized to particular scene. They require training for each specific edit and cannot be exploited in real-time edits. To address these limitations, we propose a novel method, FreeEdit, to make edits in training free manner using mesh representations as a substitute for NeRF. Training-free methods are now a possibility because of the advances in foundation model's space. We leverage these models to bring a training-free alternative and introduce solutions for insertion, replacement and deletion. We consider insertion, replacement and deletion as basic blocks for performing intricate edits with certain combinations of these operations. Given a text prompt and a 3D scene, our model is capable of identifying what object should be inserted/replaced or deleted and location where edit should be performed. We also introduce a novel algorithm as part of FreeEdit to find the optimal location on grounding object for placement. We evaluate our model by comparing it with baseline models on a wide range of scenes using quantitative and qualitative metrics and showcase the merits of our method with respect to others.

Title: Guided and Variance-Corrected Fusion with One-shot Style Alignment for Large-Content Image Generation

• Authors: Shoukun Sun, Min Xian, Tiankai Yao, Fei Xu, Luca Capriotti
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12771
• Pdf URL: https://arxiv.org/pdf/2412.12771
• Copy Paste: [[2412.12771]] Guided and Variance-Corrected Fusion with One-shot Style Alignment for Large-Content Image Generation(https://arxiv.org/abs/2412.12771)
• Keywords: diffusion
• Abstract: Producing large images using small diffusion models is gaining increasing popularity, as the cost of training large models could be prohibitive. A common approach involves jointly generating a series of overlapped image patches and obtaining large images by merging adjacent patches. However, results from existing methods often exhibit obvious artifacts, e.g., seams and inconsistent objects and styles. To address the issues, we proposed Guided Fusion (GF), which mitigates the negative impact from distant image regions by applying a weighted average to the overlapping regions. Moreover, we proposed Variance-Corrected Fusion (VCF), which corrects data variance at post-averaging, generating more accurate fusion for the Denoising Diffusion Probabilistic Model. Furthermore, we proposed a one-shot Style Alignment (SA), which generates a coherent style for large images by adjusting the initial input noise without adding extra computational burden. Extensive experiments demonstrated that the proposed fusion methods improved the quality of the generated image significantly. As a plug-and-play module, the proposed method can be widely applied to enhance other fusion-based methods for large image generation.

Title: Rethinking Diffusion-Based Image Generators for Fundus Fluorescein Angiography Synthesis on Limited Data

• Authors: Chengzhou Yu (South China University of Technology), Huihui Fang (Pazhou Laboratory), Hongqiu Wang (The Hong Kong University of Science and Technology (Guangzhou)), Ting Deng (South China University of Technology), Qing Du (South China University of Technology), Yanwu Xu (South China University of Technology), Weihua Yang (Shenzhen Eye Hospital)
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12778
• Pdf URL: https://arxiv.org/pdf/2412.12778
• Copy Paste: [[2412.12778]] Rethinking Diffusion-Based Image Generators for Fundus Fluorescein Angiography Synthesis on Limited Data(https://arxiv.org/abs/2412.12778)
• Keywords: diffusion, generative
• Abstract: Fundus imaging is a critical tool in ophthalmology, with different imaging modalities offering unique advantages. For instance, fundus fluorescein angiography (FFA) can accurately identify eye diseases. However, traditional invasive FFA involves the injection of sodium fluorescein, which can cause discomfort and risks. Generating corresponding FFA images from non-invasive fundus images holds significant practical value but also presents challenges. First, limited datasets constrain the performance and effectiveness of models. Second, previous studies have primarily focused on generating FFA for single diseases or single modalities, often resulting in poor performance for patients with various ophthalmic conditions. To address these issues, we propose a novel latent diffusion model-based framework, Diffusion, which introduces a fine-tuning protocol to overcome the challenge of limited medical data and unleash the generative capabilities of diffusion models. Furthermore, we designed a new approach to tackle the challenges of generating across different modalities and disease types. On limited datasets, our framework achieves state-of-the-art results compared to existing methods, offering significant potential to enhance ophthalmic diagnostics and patient care. Our code will be released soon to support further research in this field.

Title: CRoF: CLIP-based Robust Few-shot Learning on Noisy Labels

• Authors: Shizhuo Deng, Bowen Han, Jiaqi Chen, Hao Wang, Dongyue Chen, Tong Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12793
• Pdf URL: https://arxiv.org/pdf/2412.12793
• Copy Paste: [[2412.12793]] CRoF: CLIP-based Robust Few-shot Learning on Noisy Labels(https://arxiv.org/abs/2412.12793)
• Keywords: robust
• Abstract: Noisy labels threaten the robustness of few-shot learning (FSL) due to the inexact features in a new domain. CLIP, a large-scale vision-language model, performs well in FSL on image-text embedding similarities, but it is susceptible to misclassification caused by noisy labels. How to enhance domain generalization of CLIP on noisy data within FSL tasks is a critical challenge. In this paper, we provide a novel view to mitigate the influence of noisy labels, CLIP-based Robust Few-shot learning (CRoF). CRoF is a general plug-in module for CLIP-based models. To avoid misclassification and confused label embedding, we design the few-shot task-oriented prompt generator to give more discriminative descriptions of each category. The proposed prompt achieves larger distances of inter-class textual embedding. Furthermore, rather than fully trusting zero-shot classification by CLIP, we fine-tune CLIP on noisy few-shot data in a new domain with a weighting strategy like label-smooth. The weights for multiple potentially correct labels consider the relationship between CLIP's prior knowledge and original label information to ensure reliability. Our multiple label loss function further supports robust training under this paradigm. Comprehensive experiments show that CRoF, as a plug-in, outperforms fine-tuned and vanilla CLIP models on different noise types and noise ratios.

Title: Is it the end of (generative) linguistics as we know it?

• Authors: Cristiano Chesi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12797
• Pdf URL: https://arxiv.org/pdf/2412.12797
• Copy Paste: [[2412.12797]] Is it the end of (generative) linguistics as we know it?(https://arxiv.org/abs/2412.12797)
• Keywords: generative
• Abstract: A significant debate has emerged in response to a paper written by Steven Piantadosi (Piantadosi, 2023) and uploaded to the LingBuzz platform, the open archive for generative linguistics. Piantadosi's dismissal of Chomsky's approach is ruthless, but generative linguists deserve it. In this paper, I will adopt three idealized perspectives -- computational, theoretical, and experimental -- to focus on two fundamental issues that lend partial support to Piantadosi's critique: (a) the evidence challenging the Poverty of Stimulus (PoS) hypothesis and (b) the notion of simplicity as conceived within mainstream Minimalism. In conclusion, I argue that, to reclaim a central role in language studies, generative linguistics -- representing a prototypical theoretical perspective on language -- needs a serious update leading to (i) more precise, consistent, and complete formalizations of foundational intuitions and (ii) the establishment and utilization of a standardized dataset of crucial empirical evidence to evaluate the theory's adequacy. On the other hand, ignoring the formal perspective leads to major drawbacks in both computational and experimental approaches. Neither descriptive nor explanatory adequacy can be easily achieved without the precise formulation of general principles that can be challenged empirically.

Title: ZoRI: Towards Discriminative Zero-Shot Remote Sensing Instance Segmentation

• Authors: Shiqi Huang, Shuting He, Bihan Wen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12798
• Pdf URL: https://arxiv.org/pdf/2412.12798
• Copy Paste: [[2412.12798]] ZoRI: Towards Discriminative Zero-Shot Remote Sensing Instance Segmentation(https://arxiv.org/abs/2412.12798)
• Keywords: segmentation
• Abstract: Instance segmentation algorithms in remote sensing are typically based on conventional methods, limiting their application to seen scenarios and closed-set predictions. In this work, we propose a novel task called zero-shot remote sensing instance segmentation, aimed at identifying aerial objects that are absent from training data. Challenges arise when classifying aerial categories with high inter-class similarity and intra-class variance. Besides, the domain gap between vision-language models' pretraining datasets and remote sensing datasets hinders the zero-shot capabilities of the pretrained model when it is directly applied to remote sensing images. To address these challenges, we propose a $\textbf{Z}$ero-Sh$\textbf{o}$t $\textbf{R}$emote Sensing $\textbf{I}$nstance Segmentation framework, dubbed $\textbf{ZoRI}$. Our approach features a discrimination-enhanced classifier that uses refined textual embeddings to increase the awareness of class disparities. Instead of direct fine-tuning, we propose a knowledge-maintained adaptation strategy that decouples semantic-related information to preserve the pretrained vision-language alignment while adjusting features to capture remote sensing domain-specific visual cues. Additionally, we introduce a prior-injected prediction with cache bank of aerial visual prototypes to supplement the semantic richness of text embeddings and seamlessly integrate aerial representations, adapting to the remote sensing domain. We establish new experimental protocols and benchmarks, and extensive experiments convincingly demonstrate that ZoRI achieves the state-of-art performance on the zero-shot remote sensing instance segmentation task. Our code is available at this https URL.

Title: RCTrans: Radar-Camera Transformer via Radar Densifier and Sequential Decoder for 3D Object Detection

• Authors: Yiheng Li, Yang Yang, Zhen Lei
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12799
• Pdf URL: https://arxiv.org/pdf/2412.12799
• Copy Paste: [[2412.12799]] RCTrans: Radar-Camera Transformer via Radar Densifier and Sequential Decoder for 3D Object Detection(https://arxiv.org/abs/2412.12799)
• Keywords: transformer
• Abstract: In radar-camera 3D object detection, the radar point clouds are sparse and noisy, which causes difficulties in fusing camera and radar modalities. To solve this, we introduce a novel query-based detection method named Radar-Camera Transformer (RCTrans). Specifically, we first design a Radar Dense Encoder to enrich the sparse valid radar tokens, and then concatenate them with the image tokens. By doing this, we can fully explore the 3D information of each interest region and reduce the interference of empty tokens during the fusing stage. We then design a Pruning Sequential Decoder to predict 3D boxes based on the obtained tokens and random initialized queries. To alleviate the effect of elevation ambiguity in radar point clouds, we gradually locate the position of the object via a sequential fusion structure. It helps to get more precise and flexible correspondences between tokens and queries. A pruning training strategy is adopted in the decoder, which can save much time during inference and inhibit queries from losing their distinctiveness. Extensive experiments on the large-scale nuScenes dataset prove the superiority of our method, and we also achieve new state-of-the-art radar-camera 3D detection results. Our implementation is available at this https URL.

Title: Detecting Emotional Incongruity of Sarcasm by Commonsense Reasoning

• Authors: Ziqi Qiu, Jianxing Yu, Yufeng Zhang, Hanjiang Lai, Yanghui Rao, Qinliang Su, Jian Yin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12808
• Pdf URL: https://arxiv.org/pdf/2412.12808
• Copy Paste: [[2412.12808]] Detecting Emotional Incongruity of Sarcasm by Commonsense Reasoning(https://arxiv.org/abs/2412.12808)
• Keywords: robust, large language model
• Abstract: This paper focuses on sarcasm detection, which aims to identify whether given statements convey criticism, mockery, or other negative sentiment opposite to the literal meaning. To detect sarcasm, humans often require a comprehensive understanding of the semantics in the statement and even resort to external commonsense to infer the fine-grained incongruity. However, existing methods lack commonsense inferential ability when they face complex real-world scenarios, leading to unsatisfactory performance. To address this problem, we propose a novel framework for sarcasm detection, which conducts incongruity reasoning based on commonsense augmentation, called EICR. Concretely, we first employ retrieval-augmented large language models to supplement the missing but indispensable commonsense background knowledge. To capture complex contextual associations, we construct a dependency graph and obtain the optimized topology via graph refinement. We further introduce an adaptive reasoning skeleton that integrates prior rules to extract sentiment-inconsistent subgraphs explicitly. To eliminate the possible spurious relations between words and labels, we employ adversarial contrastive learning to enhance the robustness of the detector. Experiments conducted on five datasets demonstrate the effectiveness of EICR.

Title: ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing

• Authors: Yaohui Ma, Xiaopeng Hong, Shizhou Zhang, Huiyun Li, Zhilin Zhu, Wei Luo, Zhiheng Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12821
• Pdf URL: https://arxiv.org/pdf/2412.12821
• Copy Paste: [[2412.12821]] ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing(https://arxiv.org/abs/2412.12821)
• Keywords: robust
• Abstract: Large multimodal language models (MLLMs) have revolutionized natural language processing and visual understanding, but often contain outdated or inaccurate information. Current multimodal knowledge editing evaluations are limited in scope and potentially biased, focusing on narrow tasks and failing to assess the impact on in-domain samples. To address these issues, we introduce ComprehendEdit, a comprehensive benchmark comprising eight diverse tasks from multiple datasets. We propose two novel metrics: Knowledge Generalization Index (KGI) and Knowledge Preservation Index (KPI), which evaluate editing effects on in-domain samples without relying on AI-synthetic samples. Based on insights from our framework, we establish Hierarchical In-Context Editing (HICE), a baseline method employing a two-stage approach that balances performance across all metrics. This study provides a more comprehensive evaluation framework for multimodal knowledge editing, reveals unique challenges in this field, and offers a baseline method demonstrating improved performance. Our work opens new perspectives for future research and provides a foundation for developing more robust and effective editing techniques for MLLMs. The ComprehendEdit benchmark and implementation code are available at this https URL.

Title: TabSniper: Towards Accurate Table Detection & Structure Recognition for Bank Statements

• Authors: Abhishek Trivedi, Sourajit Mukherjee, Rajat Kumar Singh, Vani Agarwal, Sriranjani Ramakrishnan, Himanshu S. Bhatt
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12827
• Pdf URL: https://arxiv.org/pdf/2412.12827
• Copy Paste: [[2412.12827]] TabSniper: Towards Accurate Table Detection & Structure Recognition for Bank Statements(https://arxiv.org/abs/2412.12827)
• Keywords: extraction
• Abstract: Extraction of transaction information from bank statements is required to assess one's financial well-being for credit rating and underwriting decisions. Unlike other financial documents such as tax forms or financial statements, extracting the transaction descriptions from bank statements can provide a comprehensive and recent view into the cash flows and spending patterns. With multiple variations in layout and templates across several banks, extracting transactional level information from different table categories is an arduous task. Existing table structure recognition approaches produce sub optimal results for long, complex tables and are unable to capture all transactions accurately. This paper proposes TabSniper, a novel approach for efficient table detection, categorization and structure recognition from bank statements. The pipeline starts with detecting and categorizing tables of interest from the bank statements. The extracted table regions are then processed by the table structure recognition model followed by a post-processing module to transform the transactional data into a structured and standardised format. The detection and structure recognition architectures are based on DETR, fine-tuned with diverse bank statements along with additional feature enhancements. Results on challenging datasets demonstrate that TabSniper outperforms strong baselines and produces high-quality extraction of transaction information from bank and other financial documents across multiple layouts and templates.

Title: 2by2: Weakly-Supervised Learning for Global Action Segmentation

• Authors: Elena Bueno-Benito, Mariella Dimiccoli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12829
• Pdf URL: https://arxiv.org/pdf/2412.12829
• Copy Paste: [[2412.12829]] 2by2: Weakly-Supervised Learning for Global Action Segmentation(https://arxiv.org/abs/2412.12829)
• Keywords: transformer, segmentation
• Abstract: This paper presents a simple yet effective approach for the poorly investigated task of global action segmentation, aiming at grouping frames capturing the same action across videos of different activities. Unlike the case of videos depicting all the same activity, the temporal order of actions is not roughly shared among all videos, making the task even more challenging. We propose to use activity labels to learn, in a weakly-supervised fashion, action representations suitable for global action segmentation. For this purpose, we introduce a triadic learning approach for video pairs, to ensure intra-video action discrimination, as well as inter-video and inter-activity action association. For the backbone architecture, we use a Siamese network based on sparse transformers that takes as input video pairs and determine whether they belong to the same activity. The proposed approach is validated on two challenging benchmark datasets: Breakfast and YouTube Instructions, outperforming state-of-the-art methods.

Title: DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models

• Authors: Jinxiang Xie, Yilin Li, Xunjian Yin, Xiaojun Wan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12832
• Pdf URL: https://arxiv.org/pdf/2412.12832
• Copy Paste: [[2412.12832]] DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models(https://arxiv.org/abs/2412.12832)
• Keywords: large language model
• Abstract: Evaluating the performance of Grammatical Error Correction (GEC) models has become increasingly challenging, as large language model (LLM)-based GEC systems often produce corrections that diverge from provided gold references. This discrepancy undermines the reliability of traditional reference-based evaluation metrics. In this study, we propose a novel evaluation framework for GEC models, DSGram, integrating Semantic Coherence, Edit Level, and Fluency, and utilizing a dynamic weighting mechanism. Our framework employs the Analytic Hierarchy Process (AHP) in conjunction with large language models to ascertain the relative importance of various evaluation criteria. Additionally, we develop a dataset incorporating human annotations and LLM-simulated sentences to validate our algorithms and fine-tune more cost-effective models. Experimental results indicate that our proposed approach enhances the effectiveness of GEC model evaluations.

Title: FocusChat: Text-guided Long Video Understanding via Spatiotemporal Information Filtering

• Authors: Zheng Cheng, Rendong Wang, Zhicheng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12833
• Pdf URL: https://arxiv.org/pdf/2412.12833
• Copy Paste: [[2412.12833]] FocusChat: Text-guided Long Video Understanding via Spatiotemporal Information Filtering(https://arxiv.org/abs/2412.12833)
• Keywords: extraction, large language model
• Abstract: Recently, multi-modal large language models have made significant progress. However, visual information lacking of guidance from the user's intention may lead to redundant computation and involve unnecessary visual noise, especially in long, untrimmed videos. To address this issue, we propose FocusChat, a text-guided multi-modal large language model (LLM) that emphasizes visual information correlated to the user's prompt. In detail, Our model first undergoes the semantic extraction module, which comprises a visual semantic branch and a text semantic branch to extract image and text semantics, respectively. The two branches are combined using the Spatial-Temporal Filtering Module (STFM). STFM enables explicit spatial-level information filtering and implicit temporal-level feature filtering, ensuring that the visual tokens are closely aligned with the user's query. It lowers the essential number of visual tokens inputted into the LLM. FocusChat significantly outperforms Video-LLaMA in zero-shot experiments, using an order of magnitude less training data with only 16 visual tokens occupied. It achieves results comparable to the state-of-the-art in few-shot experiments, with only 0.72M pre-training data.

Title: Scrutinizing the Vulnerability of Decentralized Learning to Membership Inference Attacks

• Authors: Ousmane Touat, Jezekael Brunon, Yacine Belal, Julien Nicolas, Mohamed Maouche, César Sabater, Sonia Ben Mokhtar
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2412.12837
• Pdf URL: https://arxiv.org/pdf/2412.12837
• Copy Paste: [[2412.12837]] Scrutinizing the Vulnerability of Decentralized Learning to Membership Inference Attacks(https://arxiv.org/abs/2412.12837)
• Keywords: privacy, defense, attack, membership infer
• Abstract: The primary promise of decentralized learning is to allow users to engage in the training of machine learning models in a collaborative manner while keeping their data on their premises and without relying on any central entity. However, this paradigm necessitates the exchange of model parameters or gradients between peers. Such exchanges can be exploited to infer sensitive information about training data, which is achieved through privacy attacks (e.g Membership Inference Attacks -- MIA). In order to devise effective defense mechanisms, it is important to understand the factors that increase/reduce the vulnerability of a given decentralized learning architecture to MIA. In this study, we extensively explore the vulnerability to MIA of various decentralized learning architectures by varying the graph structure (e.g number of neighbors), the graph dynamics, and the aggregation strategy, across diverse datasets and data distributions. Our key finding, which to the best of our knowledge we are the first to report, is that the vulnerability to MIA is heavily correlated to (i) the local model mixing strategy performed by each node upon reception of models from neighboring nodes and (ii) the global mixing properties of the communication graph. We illustrate these results experimentally using four datasets and by theoretically analyzing the mixing properties of various decentralized architectures. Our paper draws a set of lessons learned for devising decentralized learning systems that reduce by design the vulnerability to MIA.

Title: Benchmarking and Understanding Compositional Relational Reasoning of LLMs

• Authors: Ruikang Ni, Da Xiao, Qingye Meng, Xiangyu Li, Shihui Zheng, Hongliang Liang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12841
• Pdf URL: https://arxiv.org/pdf/2412.12841
• Copy Paste: [[2412.12841]] Benchmarking and Understanding Compositional Relational Reasoning of LLMs(https://arxiv.org/abs/2412.12841)
• Keywords: interpretability, transformer, large language model
• Abstract: Compositional relational reasoning (CRR) is a hallmark of human intelligence, but we lack a clear understanding of whether and how existing transformer large language models (LLMs) can solve CRR tasks. To enable systematic exploration of the CRR capability of LLMs, we first propose a new synthetic benchmark called Generalized Associative Recall (GAR) by integrating and generalizing the essence of several tasks in mechanistic interpretability (MI) study in a unified framework. Evaluation shows that GAR is challenging enough for existing LLMs, revealing their fundamental deficiency in CRR. Meanwhile, it is easy enough for systematic MI study. Then, to understand how LLMs solve GAR tasks, we use attribution patching to discover the core circuits reused by Vicuna-33B across different tasks and a set of vital attention heads. Intervention experiments show that the correct functioning of these heads significantly impacts task performance. Especially, we identify two classes of heads whose activations represent the abstract notion of true and false in GAR tasks respectively. They play a fundamental role in CRR across various models and tasks. The dataset and code are available at this https URL.

Title: Efficient Event-based Semantic Segmentation with Spike-driven Lightweight Transformer-based Networks

• Authors: Xiaxin Zhu, Fangming Guo, Xianlei Long, Qingyi Gu, Chao Chen, Fuqiang Gu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12843
• Pdf URL: https://arxiv.org/pdf/2412.12843
• Copy Paste: [[2412.12843]] Efficient Event-based Semantic Segmentation with Spike-driven Lightweight Transformer-based Networks(https://arxiv.org/abs/2412.12843)
• Keywords: transformer, segmentation
• Abstract: Event-based semantic segmentation has great potential in autonomous driving and robotics due to the advantages of event cameras, such as high dynamic range, low latency, and low power cost. Unfortunately, current artificial neural network (ANN)-based segmentation methods suffer from high computational demands, the requirements for image frames, and massive energy consumption, limiting their efficiency and application on resource-constrained edge/mobile platforms. To address these problems, we introduce SLTNet, a spike-driven lightweight transformer-based network designed for event-based semantic segmentation. Specifically, SLTNet is built on efficient spike-driven convolution blocks (SCBs) to extract rich semantic features while reducing the model's parameters. Then, to enhance the long-range contextural feature interaction, we propose novel spike-driven transformer blocks (STBs) with binary mask operations. Based on these basic blocks, SLTNet employs a high-efficiency single-branch architecture while maintaining the low energy consumption of the Spiking Neural Network (SNN). Finally, extensive experiments on DDD17 and DSEC-Semantic datasets demonstrate that SLTNet outperforms state-of-the-art (SOTA) SNN-based methods by at least 7.30% and 3.30% mIoU, respectively, with extremely 5.48x lower energy consumption and 1.14x faster inference speed.

Title: Concurrent vertical and horizontal federated learning with fuzzy cognitive maps

• Authors: Jose L Salmeron, Irina Arévalo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12844
• Pdf URL: https://arxiv.org/pdf/2412.12844
• Copy Paste: [[2412.12844]] Concurrent vertical and horizontal federated learning with fuzzy cognitive maps(https://arxiv.org/abs/2412.12844)
• Keywords: privacy, federate
• Abstract: Data privacy is a major concern in industries such as healthcare or finance. The requirement to safeguard privacy is essential to prevent data breaches and misuse, which can have severe consequences for individuals and organisations. Federated learning is a distributed machine learning approach where multiple participants collaboratively train a model without compromising the privacy of their data. However, a significant challenge arises from the differences in feature spaces among participants, known as non-IID data. This research introduces a novel federated learning framework employing fuzzy cognitive maps, designed to comprehensively address the challenges posed by diverse data distributions and non-identically distributed features in federated settings. The proposal is tested through several experiments using four distinct federation strategies: constant-based, accuracy-based, AUC-based, and precision-based weights. The results demonstrate the effectiveness of the approach in achieving the desired learning outcomes while maintaining privacy and confidentiality standards.

Title: HyperGS: Hyperspectral 3D Gaussian Splatting

• Authors: Christopher Thirgood, Oscar Mendez, Erin Chao Ling, Jon Storey, Simon Hadfield
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12849
• Pdf URL: https://arxiv.org/pdf/2412.12849
• Copy Paste: [[2412.12849]] HyperGS: Hyperspectral 3D Gaussian Splatting(https://arxiv.org/abs/2412.12849)
• Keywords: robust
• Abstract: We introduce HyperGS, a novel framework for Hyperspectral Novel View Synthesis (HNVS), based on a new latent 3D Gaussian Splatting (3DGS) technique. Our approach enables simultaneous spatial and spectral renderings by encoding material properties from multi-view 3D hyperspectral datasets. HyperGS reconstructs high-fidelity views from arbitrary perspectives with improved accuracy and speed, outperforming currently existing methods. To address the challenges of high-dimensional data, we perform view synthesis in a learned latent space, incorporating a pixel-wise adaptive density function and a pruning technique for increased training stability and efficiency. Additionally, we introduce the first HNVS benchmark, implementing a number of new baselines based on recent SOTA RGB-NVS techniques, alongside the small number of prior works on HNVS. We demonstrate HyperGS's robustness through extensive evaluation of real and simulated hyperspectral scenes with a 14db accuracy improvement upon previously published models.

Title: Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera

• Authors: Zhengdi Yu, Stefanos Zafeiriou, Tolga Birdal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12861
• Pdf URL: https://arxiv.org/pdf/2412.12861
• Copy Paste: [[2412.12861]] Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera(https://arxiv.org/abs/2412.12861)
• Keywords: robust, generative
• Abstract: We propose Dyn-HaMR, to the best of our knowledge, the first approach to reconstruct 4D global hand motion from monocular videos recorded by dynamic cameras in the wild. Reconstructing accurate 3D hand meshes from monocular videos is a crucial task for understanding human behaviour, with significant applications in augmented and virtual reality (AR/VR). However, existing methods for monocular hand reconstruction typically rely on a weak perspective camera model, which simulates hand motion within a limited camera frustum. As a result, these approaches struggle to recover the full 3D global trajectory and often produce noisy or incorrect depth estimations, particularly when the video is captured by dynamic or moving cameras, which is common in egocentric scenarios. Our Dyn-HaMR consists of a multi-stage, multi-objective optimization pipeline, that factors in (i) simultaneous localization and mapping (SLAM) to robustly estimate relative camera motion, (ii) an interacting-hand prior for generative infilling and to refine the interaction dynamics, ensuring plausible recovery under (self-)occlusions, and (iii) hierarchical initialization through a combination of state-of-the-art hand tracking methods. Through extensive evaluations on both in-the-wild and indoor datasets, we show that our approach significantly outperforms state-of-the-art methods in terms of 4D global mesh recovery. This establishes a new benchmark for hand motion reconstruction from monocular video with moving cameras. Our project page is at this https URL.

Title: Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models

• Authors: Yuchen Fan, Yuzhong Hong, Qiushi Wang, Junwei Bao, Hongfei Jiang, Yang Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12865
• Pdf URL: https://arxiv.org/pdf/2412.12865
• Copy Paste: [[2412.12865]] Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models(https://arxiv.org/abs/2412.12865)
• Keywords: large language model
• Abstract: Alignment, endowing a pre-trained Large language model (LLM) with the ability to follow instructions, is crucial for its real-world applications. Conventional supervised fine-tuning (SFT) methods formalize it as causal language modeling typically with a cross-entropy objective, requiring a large amount of high-quality instruction-response pairs. However, the quality of widely used SFT datasets can not be guaranteed due to the high cost and intensive labor for the creation and maintenance in practice. To overcome the limitations associated with the quality of SFT datasets, we introduce a novel \textbf{p}reference-\textbf{o}riented supervised \textbf{f}ine-\textbf{t}uning approach, namely PoFT. The intuition is to boost SFT by imposing a particular preference: \textit{favoring the target model over aligned LLMs on the same SFT data.} This preference encourages the target model to predict a higher likelihood than that predicted by the aligned LLMs, incorporating assessment information on data quality (i.e., predicted likelihood by the aligned LLMs) into the training process. Extensive experiments are conducted, and the results validate the effectiveness of the proposed method. PoFT achieves stable and consistent improvements over the SFT baselines across different training datasets and base models. Moreover, we prove that PoFT can be integrated with existing SFT data filtering methods to achieve better performance, and further improved by following preference optimization procedures, such as DPO.

Title: Towards Effective Graph Rationalization via Boosting Environment Diversity

• Authors: Yujie Wang, Kui Yu, Yuhong Zhang, Fuyuan Cao, Jiye Liang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12880
• Pdf URL: https://arxiv.org/pdf/2412.12880
• Copy Paste: [[2412.12880]] Towards Effective Graph Rationalization via Boosting Environment Diversity(https://arxiv.org/abs/2412.12880)
• Keywords: extraction
• Abstract: Graph Neural Networks (GNNs) perform effectively when training and testing graphs are drawn from the same distribution, but struggle to generalize well in the face of distribution shifts. To address this issue, existing mainstreaming graph rationalization methods first identify rationale and environment subgraphs from input graphs, and then diversify training distributions by augmenting the environment subgraphs. However, these methods merely combine the learned rationale subgraphs with environment subgraphs in the representation space to produce augmentation samples, failing to produce sufficiently diverse distributions. Thus, in this paper, we propose to achieve an effective Graph Rationalization by Boosting Environmental diversity, a GRBE approach that generates the augmented samples in the original graph space to improve the diversity of the environment subgraph. Firstly, to ensure the effectiveness of augmentation samples, we propose a precise rationale subgraph extraction strategy in GRBE to refine the rationale subgraph learning process in the original graph space. Secondly, to ensure the diversity of augmented samples, we propose an environment diversity augmentation strategy in GRBE that mixes the environment subgraphs of different graphs in the original graph space and then combines the new environment subgraphs with rationale subgraphs to generate augmented graphs. The average improvements of 7.65% and 6.11% in rationalization and classification performance on benchmark datasets demonstrate the superiority of GRBE over state-of-the-art approaches.

Title: RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement

• Authors: Jinhao Jiang, Jiayi Chen, Junyi Li, Ruiyang Ren, Shijie Wang, Wayne Xin Zhao, Yang Song, Tao Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12881
• Pdf URL: https://arxiv.org/pdf/2412.12881
• Copy Paste: [[2412.12881]] RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement(https://arxiv.org/abs/2412.12881)
• Keywords: large language model
• Abstract: Existing large language models (LLMs) show exceptional problem-solving capabilities but might struggle with complex reasoning tasks. Despite the successes of chain-of-thought and tree-based search methods, they mainly depend on the internal knowledge of LLMs to search over intermediate reasoning steps, limited to dealing with simple tasks involving fewer reasoning steps. In this paper, we propose \textbf{RAG-Star}, a novel RAG approach that integrates the retrieved information to guide the tree-based deliberative reasoning process that relies on the inherent knowledge of LLMs. By leveraging Monte Carlo Tree Search, RAG-Star iteratively plans intermediate sub-queries and answers for reasoning based on the LLM itself. To consolidate internal and external knowledge, we propose an retrieval-augmented verification that utilizes query- and answer-aware reward modeling to provide feedback for the inherent reasoning of LLMs. Our experiments involving Llama-3.1-8B-Instruct and GPT-4o demonstrate that RAG-Star significantly outperforms previous RAG and reasoning methods.

Title: A Comparative Study of Pruning Methods in Transformer-based Time Series Forecasting

• Authors: Nicholas Kiefer, Arvid Weyrauch, Muhammed Öz, Achim Streit, Markus Götz, Charlotte Debus
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12883
• Pdf URL: https://arxiv.org/pdf/2412.12883
• Copy Paste: [[2412.12883]] A Comparative Study of Pruning Methods in Transformer-based Time Series Forecasting(https://arxiv.org/abs/2412.12883)
• Keywords: transformer
• Abstract: The current landscape in time-series forecasting is dominated by Transformer-based models. Their high parameter count and corresponding demand in computational resources pose a challenge to real-world deployment, especially for commercial and scientific applications with low-power embedded devices. Pruning is an established approach to reduce neural network parameter count and save compute. However, the implications and benefits of pruning Transformer-based models for time series forecasting are largely unknown. To close this gap, we provide a comparative benchmark study by evaluating unstructured and structured pruning on various state-of-the-art multivariate time series models. We study the effects of these pruning strategies on model predictive performance and computational aspects like model size, operations, and inference time. Our results show that certain models can be pruned even up to high sparsity levels, outperforming their dense counterpart. However, fine-tuning pruned models is necessary. Furthermore, we demonstrate that even with corresponding hardware and software support, structured pruning is unable to provide significant time savings.

Title: TimeCHEAT: A Channel Harmony Strategy for Irregularly Sampled Multivariate Time Series Analysis

• Authors: Jiexi Liu, Meng Cao, Songcan Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12886
• Pdf URL: https://arxiv.org/pdf/2412.12886
• Copy Paste: [[2412.12886]] TimeCHEAT: A Channel Harmony Strategy for Irregularly Sampled Multivariate Time Series Analysis(https://arxiv.org/abs/2412.12886)
• Keywords: transformer
• Abstract: Irregularly sampled multivariate time series (ISMTS) are prevalent in reality. Due to their non-uniform intervals between successive observations and varying sampling rates among series, the channel-independent (CI) strategy, which has been demonstrated more desirable for complete multivariate time series forecasting in recent studies, has failed. This failure can be further attributed to the sampling sparsity, which provides insufficient information for effective CI learning, thereby reducing its capacity. When we resort to the channel-dependent (CD) strategy, even higher capacity cannot mitigate the potential loss of diversity in learning similar embedding patterns across different channels. We find that existing work considers CI and CD strategies to be mutually exclusive, primarily because they apply these strategies to the global channel. However, we hold the view that channel strategies do not necessarily have to be used globally. Instead, by appropriately applying them locally and globally, we can create an opportunity to take full advantage of both strategies. This leads us to introduce the Channel Harmony ISMTS Transformer (TimeCHEAT), which utilizes the CD locally and the CI globally. Specifically, we segment the ISMTS into sub-series level patches. Locally, the CD strategy aggregates information within each patch for time embedding learning, maximizing the use of relevant observations while reducing long-range irrelevant interference. Here, we enhance generality by transforming embedding learning into an edge weight prediction task using bipartite graphs, eliminating the need for special prior knowledge. Globally, the CI strategy is applied across patches, allowing the Transformer to learn individualized attention patterns for each channel. Experimental results indicate our proposed TimeCHEAT demonstrates competitive SOTA performance across three mainstream tasks.

Title: ArtAug: Enhancing Text-to-Image Generation through Synthesis-Understanding Interaction

• Authors: Zhongjie Duan, Qianyi Zhao, Cen Chen, Daoyuan Chen, Wenmeng Zhou, Yaliang Li, Yingda Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12888
• Pdf URL: https://arxiv.org/pdf/2412.12888
• Copy Paste: [[2412.12888]] ArtAug: Enhancing Text-to-Image Generation through Synthesis-Understanding Interaction(https://arxiv.org/abs/2412.12888)
• Keywords: diffusion, generative, large language model
• Abstract: The emergence of diffusion models has significantly advanced image synthesis. The recent studies of model interaction and self-corrective reasoning approach in large language models offer new insights for enhancing text-to-image models. Inspired by these studies, we propose a novel method called ArtAug for enhancing text-to-image models in this paper. To the best of our knowledge, ArtAug is the first one that improves image synthesis models via model interactions with understanding models. In the interactions, we leverage human preferences implicitly learned by image understanding models to provide fine-grained suggestions for image synthesis models. The interactions can modify the image content to make it aesthetically pleasing, such as adjusting exposure, changing shooting angles, and adding atmospheric effects. The enhancements brought by the interaction are iteratively fused into the synthesis model itself through an additional enhancement module. This enables the synthesis model to directly produce aesthetically pleasing images without any extra computational cost. In the experiments, we train the ArtAug enhancement module on existing text-to-image models. Various evaluation metrics consistently demonstrate that ArtAug enhances the generative capabilities of text-to-image models without incurring additional computational costs. The source code and models will be released publicly.

Title: Question: How do Large Language Models perform on the Question Answering tasks? Answer:

• Authors: Kevin Fischer, Darren Fürst, Sebastian Steindl, Jakob Lindner, Ulrich Schäfer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12893
• Pdf URL: https://arxiv.org/pdf/2412.12893
• Copy Paste: [[2412.12893]] Question: How do Large Language Models perform on the Question Answering tasks? Answer:(https://arxiv.org/abs/2412.12893)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have been showing promising results for various NLP-tasks without the explicit need to be trained for these tasks by using few-shot or zero-shot prompting techniques. A common NLP-task is question-answering (QA). In this study, we propose a comprehensive performance comparison between smaller fine-tuned models and out-of-the-box instruction-following LLMs on the Stanford Question Answering Dataset 2.0 (SQuAD2), specifically when using a single-inference prompting technique. Since the dataset contains unanswerable questions, previous work used a double inference method. We propose a prompting style which aims to elicit the same ability without the need for double inference, saving compute time and resources. Furthermore, we investigate their generalization capabilities by comparing their performance on similar but different QA datasets, without fine-tuning neither model, emulating real-world uses where the context and questions asked may differ from the original training distribution, for example swapping Wikipedia for news articles. Our results show that smaller, fine-tuned models outperform current State-Of-The-Art (SOTA) LLMs on the fine-tuned task, but recent SOTA models are able to close this gap on the out-of-distribution test and even outperform the fine-tuned models on 3 of the 5 tested QA datasets.

Title: An Agentic Approach to Automatic Creation of P&ID Diagrams from Natural Language Descriptions

• Authors: Shreeyash Gowaikar, Srinivasan Iyengar, Sameer Segal, Shivkumar Kalyanaraman
• Subjects: cs.LG, cs.CE, cs.CL, cs.MA
• Abstract URL: https://arxiv.org/abs/2412.12898
• Pdf URL: https://arxiv.org/pdf/2412.12898
• Copy Paste: [[2412.12898]] An Agentic Approach to Automatic Creation of P&ID Diagrams from Natural Language Descriptions(https://arxiv.org/abs/2412.12898)
• Keywords: robust, generative, large language model
• Abstract: The Piping and Instrumentation Diagrams (P&IDs) are foundational to the design, construction, and operation of workflows in the engineering and process industries. However, their manual creation is often labor-intensive, error-prone, and lacks robust mechanisms for error detection and correction. While recent advancements in Generative AI, particularly Large Language Models (LLMs) and Vision-Language Models (VLMs), have demonstrated significant potential across various domains, their application in automating generation of engineering workflows remains underexplored. In this work, we introduce a novel copilot for automating the generation of P&IDs from natural language descriptions. Leveraging a multi-step agentic workflow, our copilot provides a structured and iterative approach to diagram creation directly from Natural Language prompts. We demonstrate the feasibility of the generation process by evaluating the soundness and completeness of the workflow, and show improved results compared to vanilla zero-shot and few-shot generation approaches.

Title: CATSplat: Context-Aware Transformer with Spatial Guidance for Generalizable 3D Gaussian Splatting from A Single-View Image

• Authors: Wonseok Roh, Hwanhee Jung, Jong Wook Kim, Seunggwan Lee, Innfarn Yoo, Andreas Lugmayr, Seunggeun Chi, Karthik Ramani, Sangpil Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12906
• Pdf URL: https://arxiv.org/pdf/2412.12906
• Copy Paste: [[2412.12906]] CATSplat: Context-Aware Transformer with Spatial Guidance for Generalizable 3D Gaussian Splatting from A Single-View Image(https://arxiv.org/abs/2412.12906)
• Keywords: transformer
• Abstract: Recently, generalizable feed-forward methods based on 3D Gaussian Splatting have gained significant attention for their potential to reconstruct 3D scenes using finite resources. These approaches create a 3D radiance field, parameterized by per-pixel 3D Gaussian primitives, from just a few images in a single forward pass. However, unlike multi-view methods that benefit from cross-view correspondences, 3D scene reconstruction with a single-view image remains an underexplored area. In this work, we introduce CATSplat, a novel generalizable transformer-based framework designed to break through the inherent constraints in monocular settings. First, we propose leveraging textual guidance from a visual-language model to complement insufficient information from a single image. By incorporating scene-specific contextual details from text embeddings through cross-attention, we pave the way for context-aware 3D scene reconstruction beyond relying solely on visual cues. Moreover, we advocate utilizing spatial guidance from 3D point features toward comprehensive geometric understanding under single-view settings. With 3D priors, image features can capture rich structural insights for predicting 3D Gaussians without multi-view techniques. Extensive experiments on large-scale datasets demonstrate the state-of-the-art performance of CATSplat in single-view 3D scene reconstruction with high-quality novel view synthesis.

Title: PT: A Plain Transformer is Good Hospital Readmission Predictor

• Authors: Zhenyi Fan, Jiaqi Li, Dongyu Luo, Yuqi Yuan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12909
• Pdf URL: https://arxiv.org/pdf/2412.12909
• Copy Paste: [[2412.12909]] PT: A Plain Transformer is Good Hospital Readmission Predictor(https://arxiv.org/abs/2412.12909)
• Keywords: robust, transformer
• Abstract: Hospital readmission prediction is critical for clinical decision support, aiming to identify patients at risk of returning within 30 days post-discharge. High readmission rates often indicate inadequate treatment or post-discharge care, making effective prediction models essential for optimizing resources and improving patient outcomes. We propose PT, a Transformer-based model that integrates Electronic Health Records (EHR), medical images, and clinical notes to predict 30-day all-cause hospital readmissions. PT extracts features from raw data and uses specialized Transformer blocks tailored to the data's complexity. Enhanced with Random Forest for EHR feature selection and test-time ensemble techniques, PT achieves superior accuracy, scalability, and robustness. It performs well even when temporal information is missing. Our main contributions are: (1)Simplicity: A powerful and efficient baseline model outperforming existing ones in prediction accuracy; (2)Scalability: Flexible handling of various features from different modalities, achieving high performance with just clinical notes or EHR data; (3)Robustness: Strong predictive performance even with missing or unclear temporal data.

Title: Unsupervised Region-Based Image Editing of Denoising Diffusion Models

• Authors: Zixiang Li, Yue Song, Renshuai Tao, Xiaohong Jia, Yao Zhao, Wei Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.12912
• Pdf URL: https://arxiv.org/pdf/2412.12912
• Copy Paste: [[2412.12912]] Unsupervised Region-Based Image Editing of Denoising Diffusion Models(https://arxiv.org/abs/2412.12912)
• Keywords: diffusion, segmentation
• Abstract: Although diffusion models have achieved remarkable success in the field of image generation, their latent space remains under-explored. Current methods for identifying semantics within latent space often rely on external supervision, such as textual information and segmentation masks. In this paper, we propose a method to identify semantic attributes in the latent space of pre-trained diffusion models without any further training. By projecting the Jacobian of the targeted semantic region into a low-dimensional subspace which is orthogonal to the non-masked regions, our approach facilitates precise semantic discovery and control over local masked areas, eliminating the need for annotations. We conducted extensive experiments across multiple datasets and various architectures of diffusion models, achieving state-of-the-art performance. In particular, for some specific face attributes, the performance of our proposed method even surpasses that of supervised approaches, demonstrating its superior ability in editing local image properties.

Title: Truthful Text Sanitization Guided by Inference Attacks

• Authors: Ildikó Pilán, Benet Manzanares-Salor, David Sánchez, Pierre Lison
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12928
• Pdf URL: https://arxiv.org/pdf/2412.12928
• Copy Paste: [[2412.12928]] Truthful Text Sanitization Guided by Inference Attacks(https://arxiv.org/abs/2412.12928)
• Keywords: privacy, protect, attack, large language model
• Abstract: The purpose of text sanitization is to rewrite those text spans in a document that may directly or indirectly identify an individual, to ensure they no longer disclose personal information. Text sanitization must strike a balance between preventing the leakage of personal information (privacy protection) while also retaining as much of the document's original content as possible (utility preservation). We present an automated text sanitization strategy based on generalizations, which are more abstract (but still informative) terms that subsume the semantic content of the original text spans. The approach relies on instruction-tuned large language models (LLMs) and is divided into two stages. The LLM is first applied to obtain truth-preserving replacement candidates and rank them according to their abstraction level. Those candidates are then evaluated for their ability to protect privacy by conducting inference attacks with the LLM. Finally, the system selects the most informative replacement shown to be resistant to those attacks. As a consequence of this two-stage process, the chosen replacements effectively balance utility and privacy. We also present novel metrics to automatically evaluate these two aspects without the need to manually annotate data. Empirical results on the Text Anonymization Benchmark show that the proposed approach leads to enhanced utility, with only a marginal increase in the risk of re-identifying protected individuals compared to fully suppressing the original information. Furthermore, the selected replacements are shown to be more truth-preserving and abstractive than previous methods.

Title: Multi-Subspace Matrix Recovery from Permuted Data

• Authors: Liangqi Xie, Jicong Fan
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.12931
• Pdf URL: https://arxiv.org/pdf/2412.12931
• Copy Paste: [[2412.12931]] Multi-Subspace Matrix Recovery from Permuted Data(https://arxiv.org/abs/2412.12931)
• Keywords: robust
• Abstract: This paper aims to recover a multi-subspace matrix from permuted data: given a matrix, in which the columns are drawn from a union of low-dimensional subspaces and some columns are corrupted by permutations on their entries, recover the original matrix. The task has numerous practical applications such as data cleaning, integration, and de-anonymization, but it remains challenging and cannot be well addressed by existing techniques such as robust principal component analysis because of the presence of multiple subspaces and the permutations on the elements of vectors. To solve the challenge, we develop a novel four-stage algorithm pipeline including outlier identification, subspace reconstruction, outlier classification, and unsupervised sensing for permuted vector recovery. Particularly, we provide theoretical guarantees for the outlier classification step, ensuring reliable multi-subspace matrix recovery. Our pipeline is compared with state-of-the-art competitors on multiple benchmarks and shows superior performance.

Title: Synthetic Data Generation for Anomaly Detection on Table Grapes

• Authors: Ionut Marian Motoi, Valerio Belli, Alberto Carpineto, Daniele Nardi, Thomas Alessandro Ciarfuglia
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.12949
• Pdf URL: https://arxiv.org/pdf/2412.12949
• Copy Paste: [[2412.12949]] Synthetic Data Generation for Anomaly Detection on Table Grapes(https://arxiv.org/abs/2412.12949)
• Keywords: segmentation
• Abstract: Early detection of illnesses and pest infestations in fruit cultivation is critical for maintaining yield quality and plant health. Computer vision and robotics are increasingly employed for the automatic detection of such issues, particularly using data-driven solutions. However, the rarity of these problems makes acquiring and processing the necessary data to train such algorithms a significant obstacle. One solution to this scarcity is the generation of synthetic high-quality anomalous samples. While numerous methods exist for this task, most require highly trained individuals for setup. This work addresses the challenge of generating synthetic anomalies in an automatic fashion that requires only an initial collection of normal and anomalous samples from the user - a task that is straightforward for farmers. We demonstrate the approach in the context of table grape cultivation. Specifically, based on the observation that normal berries present relatively smooth surfaces, while defects result in more complex textures, we introduce a Dual-Canny Edge Detection (DCED) filter. This filter emphasizes the additional texture indicative of diseases, pest infestations, or other defects. Using segmentation masks provided by the Segment Anything Model, we then select and seamlessly blend anomalous berries onto normal ones. We show that the proposed dataset augmentation technique improves the accuracy of an anomaly classifier for table grapes and that the approach can be generalized to other fruit types.

Title: FineGates: LLMs Finetuning with Compression using Stochastic Gates

• Authors: Jonathan Svirsky, Yehonathan Refael, Ofir Lindenbaum
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12951
• Pdf URL: https://arxiv.org/pdf/2412.12951
• Copy Paste: [[2412.12951]] FineGates: LLMs Finetuning with Compression using Stochastic Gates(https://arxiv.org/abs/2412.12951)
• Keywords: large language model
• Abstract: Large Language Models (LLMs), with billions of parameters, present significant challenges for full finetuning due to the high computational demands, memory requirements, and impracticality of many real-world applications. When faced with limited computational resources or small datasets, updating all model parameters can often result in overfitting. To address this, lightweight finetuning techniques have been proposed, like learning low-rank adapter layers. These methods aim to train only a few additional parameters combined with the base model, which remains frozen, reducing resource usage and mitigating overfitting risks. In this work, we propose an adaptor model based on stochastic gates that simultaneously sparsify the frozen base model with task-specific adaptation. Our method comes with a small number of trainable parameters and allows us to speed up the base model inference with competitive accuracy. We evaluate it in additional variants by equipping it with additional low-rank parameters and comparing it to several recent baselines. Our results show that the proposed method improves the finetuned model accuracy comparatively to the several baselines and allows the removal of up to 20-40\% without significant accuracy loss.

Title: Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learning

• Authors: Moritz Reuss, Jyothish Pari, Pulkit Agrawal, Rudolf Lioutikov
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.12953
• Pdf URL: https://arxiv.org/pdf/2412.12953
• Copy Paste: [[2412.12953]] Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learning(https://arxiv.org/abs/2412.12953)
• Keywords: diffusion, transformer
• Abstract: Diffusion Policies have become widely used in Imitation Learning, offering several appealing properties, such as generating multimodal and discontinuous behavior. As models are becoming larger to capture more complex capabilities, their computational demands increase, as shown by recent scaling laws. Therefore, continuing with the current architectures will present a computational roadblock. To address this gap, we propose Mixture-of-Denoising Experts (MoDE) as a novel policy for Imitation Learning. MoDE surpasses current state-of-the-art Transformer-based Diffusion Policies while enabling parameter-efficient scaling through sparse experts and noise-conditioned routing, reducing both active parameters by 40% and inference costs by 90% via expert caching. Our architecture combines this efficient scaling with noise-conditioned self-attention mechanism, enabling more effective denoising across different noise levels. MoDE achieves state-of-the-art performance on 134 tasks in four established imitation learning benchmarks (CALVIN and LIBERO). Notably, by pretraining MoDE on diverse robotics data, we achieve 4.01 on CALVIN ABC and 0.95 on LIBERO-90. It surpasses both CNN-based and Transformer Diffusion Policies by an average of 57% across 4 benchmarks, while using 90% fewer FLOPs and fewer active parameters compared to default Diffusion Transformer architectures. Furthermore, we conduct comprehensive ablations on MoDE's components, providing insights for designing efficient and scalable Transformer architectures for Diffusion Policies. Code and demonstrations are available at this https URL.

Title: Recipient Profiling: Predicting Characteristics from Messages

• Authors: Martin Borquez, Mikaela Keller, Michael Perrot, Damien Sileo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12954
• Pdf URL: https://arxiv.org/pdf/2412.12954
• Copy Paste: [[2412.12954]] Recipient Profiling: Predicting Characteristics from Messages(https://arxiv.org/abs/2412.12954)
• Keywords: privacy
• Abstract: It has been shown in the field of Author Profiling that texts may inadvertently reveal sensitive information about their authors, such as gender or age. This raises important privacy concerns that have been extensively addressed in the literature, in particular with the development of methods to hide such information. We argue that, when these texts are in fact messages exchanged between individuals, this is not the end of the story. Indeed, in this case, a second party, the intended recipient, is also involved and should be considered. In this work, we investigate the potential privacy leaks affecting them, that is we propose and address the problem of Recipient Profiling. We provide empirical evidence that such a task is feasible on several publicly accessible datasets (this https URL). Furthermore, we show that the learned models can be transferred to other datasets, albeit with a loss in accuracy.

Title: Learning from Noisy Labels via Self-Taught On-the-Fly Meta Loss Rescaling

• Authors: Michael Heck, Christian Geishauser, Nurul Lubis, Carel van Niekerk, Shutong Feng, Hsien-Chin Lin, Benjamin Matthias Ruppik, Renato Vukovic, Milica Gašić
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12955
• Pdf URL: https://arxiv.org/pdf/2412.12955
• Copy Paste: [[2412.12955]] Learning from Noisy Labels via Self-Taught On-the-Fly Meta Loss Rescaling(https://arxiv.org/abs/2412.12955)
• Keywords: robust
• Abstract: Correct labels are indispensable for training effective machine learning models. However, creating high-quality labels is expensive, and even professionally labeled data contains errors and ambiguities. Filtering and denoising can be applied to curate labeled data prior to training, at the cost of additional processing and loss of information. An alternative is on-the-fly sample reweighting during the training process to decrease the negative impact of incorrect or ambiguous labels, but this typically requires clean seed data. In this work we propose unsupervised on-the-fly meta loss rescaling to reweight training samples. Crucially, we rely only on features provided by the model being trained, to learn a rescaling function in real time without knowledge of the true clean data distribution. We achieve this via a novel meta learning setup that samples validation data for the meta update directly from the noisy training corpus by employing the rescaling function being trained. Our proposed method consistently improves performance across various NLP tasks with minimal computational overhead. Further, we are among the first to attempt on-the-fly training data reweighting on the challenging task of dialogue modeling, where noisy and ambiguous labels are common. Our strategy is robust in the face of noisy and clean data, handles class imbalance, and prevents overfitting to noisy labels. Our self-taught loss rescaling improves as the model trains, showing the ability to keep learning from the model's own signals. As training progresses, the impact of correctly labeled data is scaled up, while the impact of wrongly labeled data is suppressed.

Title: SnakModel: Lessons Learned from Training an Open Danish Large Language Model

• Authors: Mike Zhang, Max Müller-Eberstein, Elisa Bassignana, Rob van der Goot
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12956
• Pdf URL: https://arxiv.org/pdf/2412.12956
• Copy Paste: [[2412.12956]] SnakModel: Lessons Learned from Training an Open Danish Large Language Model(https://arxiv.org/abs/2412.12956)
• Keywords: large language model
• Abstract: We present SnakModel, a Danish large language model (LLM) based on Llama2-7B, which we continuously pre-train on 13.6B Danish words, and further tune on 3.7M Danish instructions. As best practices for creating LLMs for smaller language communities have yet to be established, we examine the effects of early modeling and training decisions on downstream performance throughout the entire training pipeline, including (1) the creation of a strictly curated corpus of Danish text from diverse sources; (2) the language modeling and instruction-tuning training process itself, including the analysis of intermediate training dynamics, and ablations across different hyperparameters; (3) an evaluation on eight language and culturally-specific tasks. Across these experiments SnakModel achieves the highest overall performance, outperforming multiple contemporary Llama2-7B-based models. By making SnakModel, the majority of our pre-training corpus, and the associated code available under open licenses, we hope to foster further research and development in Danish Natural Language Processing, and establish training guidelines for languages with similar resource constraints.

Title: Adaptations of AI models for querying the LandMatrix database in natural language

• Authors: Fatiha Ait Kbir, Jérémy Bourgoin, Rémy Decoupes, Marie Gradeler, Roberto Interdonato
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12961
• Pdf URL: https://arxiv.org/pdf/2412.12961
• Copy Paste: [[2412.12961]] Adaptations of AI models for querying the LandMatrix database in natural language(https://arxiv.org/abs/2412.12961)
• Keywords: extraction, large language model
• Abstract: The Land Matrix initiative (this https URL) and its global observatory aim to provide reliable data on large-scale land acquisitions to inform debates and actions in sectors such as agriculture, extraction, or energy in low- and middle-income countries. Although these data are recognized in the academic world, they remain underutilized in public policy, mainly due to the complexity of access and exploitation, which requires technical expertise and a good understanding of the database schema. The objective of this work is to simplify access to data from different database systems. The methods proposed in this article are evaluated using data from the Land Matrix. This work presents various comparisons of Large Language Models (LLMs) as well as combinations of LLM adaptations (Prompt Engineering, RAG, Agents) to query different database systems (GraphQL and REST queries). The experiments are reproducible, and a demonstration is available online: this https URL.

Title: ArchesWeather & ArchesWeatherGen: a deterministic and generative model for efficient ML weather forecasting

• Authors: Guillaume Couairon, Renu Singh, Anastase Charantonis, Christian Lessig, Claire Monteleoni
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.12971
• Pdf URL: https://arxiv.org/pdf/2412.12971
• Copy Paste: [[2412.12971]] ArchesWeather & ArchesWeatherGen: a deterministic and generative model for efficient ML weather forecasting(https://arxiv.org/abs/2412.12971)
• Keywords: diffusion, transformer, generative
• Abstract: Weather forecasting plays a vital role in today's society, from agriculture and logistics to predicting the output of renewable energies, and preparing for extreme weather events. Deep learning weather forecasting models trained with the next state prediction objective on ERA5 have shown great success compared to numerical global circulation models. However, for a wide range of applications, being able to provide representative samples from the distribution of possible future weather states is critical. In this paper, we propose a methodology to leverage deterministic weather models in the design of probabilistic weather models, leading to improved performance and reduced computing costs. We first introduce \textbf{ArchesWeather}, a transformer-based deterministic model that improves upon Pangu-Weather by removing overrestrictive inductive priors. We then design a probabilistic weather model called \textbf{ArchesWeatherGen} based on flow matching, a modern variant of diffusion models, that is trained to project ArchesWeather's predictions to the distribution of ERA5 weather states. ArchesWeatherGen is a true stochastic emulator of ERA5 and surpasses IFS ENS and NeuralGCM on all WeatherBench headline variables (except for NeuralGCM's geopotential). Our work also aims to democratize the use of deterministic and generative machine learning models in weather forecasting research, with academic computing resources. All models are trained at 1.5° resolution, with a training budget of $\sim$9 V100 days for ArchesWeather and $\sim$45 V100 days for ArchesWeatherGen. For inference, ArchesWeatherGen generates 15-day weather trajectories at a rate of 1 minute per ensemble member on a A100 GPU card. To make our work fully reproducible, our code and models are open source, including the complete pipeline for data preparation, training, and evaluation, at this https URL .

Title: Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidance

• Authors: Wenhao Sun, Benlei Cui, Jingqun Tang, Xue-Mei Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.12974
• Pdf URL: https://arxiv.org/pdf/2412.12974
• Copy Paste: [[2412.12974]] Attentive Eraser: Unleashing Diffusion Model's Object Removal Potential via Self-Attention Redirection Guidance(https://arxiv.org/abs/2412.12974)
• Keywords: diffusion, generative
• Abstract: Recently, diffusion models have emerged as promising newcomers in the field of generative models, shining brightly in image generation. However, when employed for object removal tasks, they still encounter issues such as generating random artifacts and the incapacity to repaint foreground object areas with appropriate content after removal. To tackle these problems, we propose Attentive Eraser, a tuning-free method to empower pre-trained diffusion models for stable and effective object removal. Firstly, in light of the observation that the self-attention maps influence the structure and shape details of the generated images, we propose Attention Activation and Suppression (ASS), which re-engineers the self-attention mechanism within the pre-trained diffusion models based on the given mask, thereby prioritizing the background over the foreground object during the reverse generation process. Moreover, we introduce Self-Attention Redirection Guidance (SARG), which utilizes the self-attention redirected by ASS to guide the generation process, effectively removing foreground objects within the mask while simultaneously generating content that is both plausible and coherent. Experiments demonstrate the stability and effectiveness of Attentive Eraser in object removal across a variety of pre-trained diffusion models, outperforming even training-based methods. Furthermore, Attentive Eraser can be implemented in various diffusion model architectures and checkpoints, enabling excellent scalability. Code is available at this https URL.

Title: Unlocking LLMs: Addressing Scarce Data and Bias Challenges in Mental Health

• Authors: Vivek Kumar, Eirini Ntoutsi, Pushpraj Singh Rajawat, Giacomo Medda, Diego Reforgiato Recupero
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.12981
• Pdf URL: https://arxiv.org/pdf/2412.12981
• Copy Paste: [[2412.12981]] Unlocking LLMs: Addressing Scarce Data and Bias Challenges in Mental Health(https://arxiv.org/abs/2412.12981)
• Keywords: transformer, large language model
• Abstract: Large language models (LLMs) have shown promising capabilities in healthcare analysis but face several challenges like hallucinations, parroting, and bias manifestation. These challenges are exacerbated in complex, sensitive, and low-resource domains. Therefore, in this work we introduce IC-AnnoMI, an expert-annotated motivational interviewing (MI) dataset built upon AnnoMI by generating in-context conversational dialogues leveraging LLMs, particularly ChatGPT. IC-AnnoMI employs targeted prompts accurately engineered through cues and tailored information, taking into account therapy style (empathy, reflection), contextual relevance, and false semantic change. Subsequently, the dialogues are annotated by experts, strictly adhering to the Motivational Interviewing Skills Code (MISC), focusing on both the psychological and linguistic dimensions of MI dialogues. We comprehensively evaluate the IC-AnnoMI dataset and ChatGPT's emotional reasoning ability and understanding of domain intricacies by modeling novel classification tasks employing several classical machine learning and current state-of-the-art transformer approaches. Finally, we discuss the effects of progressive prompting strategies and the impact of augmented data in mitigating the biases manifested in IC-AnnoM. Our contributions provide the MI community with not only a comprehensive dataset but also valuable insights for using LLMs in empathetic text generation for conversational therapy in supervised settings.

Title: What is YOLOv6? A Deep Insight into the Object Detection Model

• Authors: Athulya Sundaresan Geetha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13006
• Pdf URL: https://arxiv.org/pdf/2412.13006
• Copy Paste: [[2412.13006]] What is YOLOv6? A Deep Insight into the Object Detection Model(https://arxiv.org/abs/2412.13006)
• Keywords: robust, extraction
• Abstract: This work explores the YOLOv6 object detection model in depth, concentrating on its design framework, optimization techniques, and detection capabilities. YOLOv6's core elements consist of the EfficientRep Backbone for robust feature extraction and the Rep-PAN Neck for seamless feature aggregation, ensuring high-performance object detection. Evaluated on the COCO dataset, YOLOv6-N achieves 37.5\% AP at 1187 FPS on an NVIDIA Tesla T4 GPU. YOLOv6-S reaches 45.0\% AP at 484 FPS, outperforming models like PPYOLOE-S, YOLOv5-S, YOLOX-S, and YOLOv8-S in the same class. Moreover, YOLOv6-M and YOLOv6-L also show better accuracy (50.0\% and 52.8\%) while maintaining comparable inference speeds to other detectors. With an upgraded backbone and neck structure, YOLOv6-L6 delivers cutting-edge accuracy in real-time.

Title: Measurement of Medial Elbow Joint Space using Landmark Detection

• Authors: Shizuka Akahori, Shotaro Teruya, Pragyan Shrestha, Yuichi Yoshii, Ryuhei Michinobu, Satoshi Iizuka, Itaru Kitahara
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13010
• Pdf URL: https://arxiv.org/pdf/2412.13010
• Copy Paste: [[2412.13010]] Measurement of Medial Elbow Joint Space using Landmark Detection(https://arxiv.org/abs/2412.13010)
• Keywords: segmentation
• Abstract: Ultrasound imaging of the medial elbow is crucial for the early identification of Ulnar Collateral Ligament (UCL) injuries. Specifically, measuring the elbow joint space in ultrasound images is used to assess the valgus instability of elbow. To automate this measurement, a precisely annotated dataset is necessary; however, no publicly available dataset has been proposed thus far. This study introduces a novel ultrasound medial elbow dataset for measuring joint space to diagnose Ulnar Collateral Ligament (UCL) injuries. The dataset comprises 4,201 medial elbow ultrasound images from 22 subjects, with landmark annotations on the humerus and ulna. The annotations are made precisely by the authors under the supervision of three orthopedic surgeons. We evaluated joint space measurement methods using our proposed dataset with several landmark detection approaches, including ViTPose, HRNet, PCT, YOLOv8, and U-Net. In addition, we propose using Shape Subspace (SS) for landmark refinement in heatmap-based landmark detection. The results show that the mean Euclidean distance error of joint space is 0.116 mm when using HRNet. Furthermore, the SS landmark refinement improves the mean absolute error of landmark positions by 0.010 mm with HRNet and by 0.103 mm with ViTPose on average. These highlight the potential for high-precision, real-time diagnosis of UCL injuries and associated risks, which could be leveraged in large-scale screening. Lastly, we demonstrate point-based segmentation of the humerus and ulna using the detected landmarks as input. The dataset will be made publicly available upon acceptance of this paper at: this https URL.

Title: A New Adversarial Perspective for LiDAR-based 3D Object Detection

• Authors: Shijun Zheng, Weiquan Liu, Yu Guo, Yu Zang, Siqi Shen, Cheng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13017
• Pdf URL: https://arxiv.org/pdf/2412.13017
• Copy Paste: [[2412.13017]] A New Adversarial Perspective for LiDAR-based 3D Object Detection(https://arxiv.org/abs/2412.13017)
• Keywords: attack, generative
• Abstract: Autonomous vehicles (AVs) rely on LiDAR sensors for environmental perception and decision-making in driving scenarios. However, ensuring the safety and reliability of AVs in complex environments remains a pressing challenge. To address this issue, we introduce a real-world dataset (ROLiD) comprising LiDAR-scanned point clouds of two random objects: water mist and smoke. In this paper, we introduce a novel adversarial perspective by proposing an attack framework that utilizes water mist and smoke to simulate environmental interference. Specifically, we propose a point cloud sequence generation method using a motion and content decomposition generative adversarial network named PCS-GAN to simulate the distribution of random objects. Furthermore, leveraging the simulated LiDAR scanning characteristics implemented with Range Image, we examine the effects of introducing random object perturbations at various positions on the target vehicle. Extensive experiments demonstrate that adversarial perturbations based on random objects effectively deceive vehicle detection and reduce the recognition rate of 3D object detection models.

Title: OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain

• Authors: Shuting Wang, Jiejun Tan, Zhicheng Dou, Ji-Rong Wen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13018
• Pdf URL: https://arxiv.org/pdf/2412.13018
• Copy Paste: [[2412.13018]] OmniEval: An Omnidirectional and Automatic RAG Evaluation Benchmark in Financial Domain(https://arxiv.org/abs/2412.13018)
• Keywords: robust, large language model
• Abstract: As a typical and practical application of Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) techniques have gained extensive attention, particularly in vertical domains where LLMs may lack domain-specific knowledge. In this paper, we introduce an omnidirectional and automatic RAG benchmark, OmniEval, in the financial domain. Our benchmark is characterized by its multi-dimensional evaluation framework, including (1) a matrix-based RAG scenario evaluation system that categorizes queries into five task classes and 16 financial topics, leading to a structured assessment of diverse query scenarios; (2) a multi-dimensional evaluation data generation approach, which combines GPT-4-based automatic generation and human annotation, achieving an 87.47\% acceptance ratio in human evaluations on generated instances; (3) a multi-stage evaluation system that evaluates both retrieval and generation performance, result in a comprehensive evaluation on the RAG pipeline; and (4) robust evaluation metrics derived from rule-based and LLM-based ones, enhancing the reliability of assessments through manual annotations and supervised fine-tuning of an LLM evaluator. Our experiments demonstrate the comprehensiveness of OmniEval, which includes extensive test datasets and highlights the performance variations of RAG systems across diverse topics and tasks, revealing significant opportunities for RAG models to improve their capabilities in vertical domains. We open source the code of our benchmark in \href{this https URL}{this https URL}.

Title: Queries, Representation & Detection: The Next 100 Model Fingerprinting Schemes

• Authors: Augustin Godinot, Erwan Le Merrer, Camilla Penzo, François Taïani, Gilles Trédan
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.13021
• Pdf URL: https://arxiv.org/pdf/2412.13021
• Copy Paste: [[2412.13021]] Queries, Representation & Detection: The Next 100 Model Fingerprinting Schemes(https://arxiv.org/abs/2412.13021)
• Keywords: steal
• Abstract: The deployment of machine learning models in operational contexts represents a significant investment for any organisation. Consequently, the risk of these models being misappropriated by competitors needs to be addressed. In recent years, numerous proposals have been put forth to detect instances of model stealing. However, these proposals operate under implicit and disparate data and model access assumptions; as a consequence, it remains unclear how they can be effectively compared to one another. Our evaluation shows that a simple baseline that we introduce performs on par with existing state-of-the-art fingerprints, which, on the other hand, are much more complex. To uncover the reasons behind this intriguing result, this paper introduces a systematic approach to both the creation of model fingerprinting schemes and their evaluation benchmarks. By dividing model fingerprinting into three core components -- Query, Representation and Detection (QuRD) -- we are able to identify $\sim100$ previously unexplored QuRD combinations and gain insights into their performance. Finally, we introduce a set of metrics to compare and guide the creation of more representative model stealing detection benchmarks. Our approach reveals the need for more challenging benchmarks and a sound comparison with baselines. To foster the creation of new fingerprinting schemes and benchmarks, we open-source our fingerprinting toolbox.

Title: Harnessing Event Sensory Data for Error Pattern Prediction in Vehicles: A Language Model Approach

• Authors: Hugo Math, Rainer Lienhart, Robin Schön
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13041
• Pdf URL: https://arxiv.org/pdf/2412.13041
• Copy Paste: [[2412.13041]] Harnessing Event Sensory Data for Error Pattern Prediction in Vehicles: A Language Model Approach(https://arxiv.org/abs/2412.13041)
• Keywords: transformer
• Abstract: In this paper, we draw an analogy between processing natural languages and processing multivariate event streams from vehicles in order to predict $\textit{when}$ and $\textit{what}$ error pattern is most likely to occur in the future for a given car. Our approach leverages the temporal dynamics and contextual relationships of our event data from a fleet of cars. Event data is composed of discrete values of error codes as well as continuous values such as time and mileage. Modelled by two causal Transformers, we can anticipate vehicle failures and malfunctions before they happen. Thus, we introduce $\textit{CarFormer}$, a Transformer model trained via a new self-supervised learning strategy, and $\textit{EPredictor}$, an autoregressive Transformer decoder model capable of predicting $\textit{when}$ and $\textit{what}$ error pattern will most likely occur after some error code apparition. Despite the challenges of high cardinality of event types, their unbalanced frequency of appearance and limited labelled data, our experimental results demonstrate the excellent predictive ability of our novel model. Specifically, with sequences of $160$ error codes on average, our model is able with only half of the error codes to achieve $80\%$ F1 score for predicting $\textit{what}$ error pattern will occur and achieves an average absolute error of $58.4 \pm 13.2$h $\textit{when}$ forecasting the time of occurrence, thus enabling confident predictive maintenance and enhancing vehicle safety.

Title: Modality-Inconsistent Continual Learning of Multimodal Large Language Models

• Authors: Weiguo Pian, Shijian Deng, Shentong Mo, Yunhui Guo, Yapeng Tian
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.13050
• Pdf URL: https://arxiv.org/pdf/2412.13050
• Copy Paste: [[2412.13050]] Modality-Inconsistent Continual Learning of Multimodal Large Language Models(https://arxiv.org/abs/2412.13050)
• Keywords: large language model
• Abstract: In this paper, we introduce Modality-Inconsistent Continual Learning (MICL), a new continual learning scenario for Multimodal Large Language Models (MLLMs) that involves tasks with inconsistent modalities (image, audio, or video) and varying task types (captioning or question-answering). Unlike existing vision-only or modality-incremental settings, MICL combines modality and task type shifts, both of which drive catastrophic forgetting. To address these challenges, we propose MoInCL, which employs a Pseudo Targets Generation Module to mitigate forgetting caused by task type shifts in previously seen modalities. It also incorporates Instruction-based Knowledge Distillation to preserve the model's ability to handle previously learned modalities when new ones are introduced. We benchmark MICL using a total of six tasks and conduct experiments to validate the effectiveness of our proposed MoInCL. The experimental results highlight the superiority of MoInCL, showing significant improvements over representative and state-of-the-art continual learning baselines.

Title: SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Tasks

• Authors: Mátyás Vincze, Laura Ferrarotti, Leonardo Lucio Custode, Bruno Lepri, Giovanni Iacca
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13053
• Pdf URL: https://arxiv.org/pdf/2412.13053
• Copy Paste: [[2412.13053]] SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Tasks(https://arxiv.org/abs/2412.13053)
• Keywords: fair
• Abstract: Continuous control tasks often involve high-dimensional, dynamic, and non-linear environments. State-of-the-art performance in these tasks is achieved through complex closed-box policies that are effective, but suffer from an inherent opacity. Interpretable policies, while generally underperforming compared to their closed-box counterparts, advantageously facilitate transparent decision-making within automated systems. Hence, their usage is often essential for diagnosing and mitigating errors, supporting ethical and legal accountability, and fostering trust among stakeholders. In this paper, we propose SMOSE, a novel method to train sparsely activated interpretable controllers, based on a top-1 Mixture-of-Experts architecture. SMOSE combines a set of interpretable decisionmakers, trained to be experts in different basic skills, and an interpretable router that assigns tasks among the experts. The training is carried out via state-of-the-art Reinforcement Learning algorithms, exploiting load-balancing techniques to ensure fair expert usage. We then distill decision trees from the weights of the router, significantly improving the ease of interpretation. We evaluate SMOSE on six benchmark environments from MuJoCo: our method outperforms recent interpretable baselines and narrows the gap with noninterpretable state-of-the-art algorithms

Title: Prompt Augmentation for Self-supervised Text-guided Image Manipulation

• Authors: Rumeysa Bodur, Binod Bhattarai, Tae-Kyun Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13081
• Pdf URL: https://arxiv.org/pdf/2412.13081
• Copy Paste: [[2412.13081]] Prompt Augmentation for Self-supervised Text-guided Image Manipulation(https://arxiv.org/abs/2412.13081)
• Keywords: diffusion
• Abstract: Text-guided image editing finds applications in various creative and practical fields. While recent studies in image generation have advanced the field, they often struggle with the dual challenges of coherent image transformation and context preservation. In response, our work introduces prompt augmentation, a method amplifying a single input prompt into several target prompts, strengthening textual context and enabling localised image editing. Specifically, we use the augmented prompts to delineate the intended manipulation area. We propose a Contrastive Loss tailored to driving effective image editing by displacing edited areas and drawing preserved regions closer. Acknowledging the continuous nature of image manipulations, we further refine our approach by incorporating the similarity concept, creating a Soft Contrastive Loss. The new losses are incorporated to the diffusion model, demonstrating improved or competitive image editing results on public datasets and generated images over state-of-the-art approaches.

Title: Accuracy Limits as a Barrier to Biometric System Security

• Authors: Axel Durbet, Paul-Marie Grollemund, Pascal Lafourcade, Kevin Thiry-Atighehchi
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13099
• Pdf URL: https://arxiv.org/pdf/2412.13099
• Copy Paste: [[2412.13099]] Accuracy Limits as a Barrier to Biometric System Security(https://arxiv.org/abs/2412.13099)
• Keywords: security, attack, biometric
• Abstract: Biometric systems are widely used for identity verification and identification, including authentication (i.e., one-to-one matching to verify a claimed identity) and identification (i.e., one-to-many matching to find a subject in a database). The matching process relies on measuring similarities or dissimilarities between a fresh biometric template and enrolled templates. The False Match Rate FMR is a key metric for assessing the accuracy and reliability of such systems. This paper analyzes biometric systems based on their FMR, with two main contributions. First, we explore untargeted attacks, where an adversary aims to impersonate any user within a database. We determine the number of trials required for an attacker to successfully impersonate a user and derive the critical population size (i.e., the maximum number of users in the database) required to maintain a given level of security. Furthermore, we compute the critical FMR value needed to ensure resistance against untargeted attacks as the database size increases. Second, we revisit the biometric birthday problem to evaluate the approximate and exact probabilities that two users in a database collide (i.e., can impersonate each other). Based on this analysis, we derive both the approximate critical population size and the critical FMR value needed to bound the likelihood of such collisions occurring with a given probability. These thresholds offer insights for designing systems that mitigate the risk of impersonation and collisions, particularly in large-scale biometric databases. Our findings indicate that current biometric systems fail to deliver sufficient accuracy to achieve an adequate security level against untargeted attacks, even in small-scale databases. Moreover, state-of-the-art systems face significant challenges in addressing the biometric birthday problem, especially as database sizes grow.

Title: AI PERSONA: Towards Life-long Personalization of LLMs

• Authors: Tiannan Wang, Meiling Tao, Ruoyu Fang, Huilin Wang, Shuai Wang, Yuchen Eleanor Jiang, Wangchunshu Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13103
• Pdf URL: https://arxiv.org/pdf/2412.13103
• Copy Paste: [[2412.13103]] AI PERSONA: Towards Life-long Personalization of LLMs(https://arxiv.org/abs/2412.13103)
• Keywords: robust, large language model
• Abstract: In this work, we introduce the task of life-long personalization of large language models. While recent mainstream efforts in the LLM community mainly focus on scaling data and compute for improved capabilities of LLMs, we argue that it is also very important to enable LLM systems, or language agents, to continuously adapt to the diverse and ever-changing profiles of every distinct user and provide up-to-date personalized assistance. We provide a clear task formulation and introduce a simple, general, effective, and scalable framework for life-long personalization of LLM systems and language agents. To facilitate future research on LLM personalization, we also introduce methods to synthesize realistic benchmarks and robust evaluation metrics. We will release all codes and data for building and benchmarking life-long personalized LLM systems.

Title: Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction

• Authors: Takumi Goto, Justin Vasselli, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13110
• Pdf URL: https://arxiv.org/pdf/2412.13110
• Copy Paste: [[2412.13110]] Improving Explainability of Sentence-level Metrics via Edit-level Attribution for Grammatical Error Correction(https://arxiv.org/abs/2412.13110)
• Keywords: explainability
• Abstract: Various evaluation metrics have been proposed for Grammatical Error Correction (GEC), but many, particularly reference-free metrics, lack explainability. This lack of explainability hinders researchers from analyzing the strengths and weaknesses of GEC models and limits the ability to provide detailed feedback for users. To address this issue, we propose attributing sentence-level scores to individual edits, providing insight into how specific corrections contribute to the overall performance. For the attribution method, we use Shapley values, from cooperative game theory, to compute the contribution of each edit. Experiments with existing sentence-level metrics demonstrate high consistency across different edit granularities and show approximately 70\% alignment with human evaluations. In addition, we analyze biases in the metrics based on the attribution results, revealing trends such as the tendency to ignore orthographic edits. Our implementation is available at \url{this https URL}.

Title: Practicable Black-box Evasion Attacks on Link Prediction in Dynamic Graphs -- A Graph Sequential Embedding Method

• Authors: Jiate Li, Meng Pang, Binghui Wang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13134
• Pdf URL: https://arxiv.org/pdf/2412.13134
• Copy Paste: [[2412.13134]] Practicable Black-box Evasion Attacks on Link Prediction in Dynamic Graphs -- A Graph Sequential Embedding Method(https://arxiv.org/abs/2412.13134)
• Keywords: secure, attack
• Abstract: Link prediction in dynamic graphs (LPDG) has been widely applied to real-world applications such as website recommendation, traffic flow prediction, organizational studies, etc. These models are usually kept local and secure, with only the interactive interface restrictively available to the public. Thus, the problem of the black-box evasion attack on the LPDG model, where model interactions and data perturbations are restricted, seems to be essential and meaningful in practice. In this paper, we propose the first practicable black-box evasion attack method that achieves effective attacks against the target LPDG model, within a limited amount of interactions and perturbations. To perform effective attacks under limited perturbations, we develop a graph sequential embedding model to find the desired state embedding of the dynamic graph sequences, under a deep reinforcement learning framework. To overcome the scarcity of interactions, we design a multi-environment training pipeline and train our agent for multiple instances, by sharing an aggregate interaction buffer. Finally, we evaluate our attack against three advanced LPDG models on three real-world graph datasets of different scales and compare its performance with related methods under the interaction and perturbation constraints. Experimental results show that our attack is both effective and practicable.

Title: SWAN: Preprocessing SGD Enables Adam-Level Performance On LLM Training With Significant Memory Reduction

• Authors: Chao Ma, Wenbo Gong, Meyer Scetbon, Edward Meeds
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13148
• Pdf URL: https://arxiv.org/pdf/2412.13148
• Copy Paste: [[2412.13148]] SWAN: Preprocessing SGD Enables Adam-Level Performance On LLM Training With Significant Memory Reduction(https://arxiv.org/abs/2412.13148)
• Keywords: large language model
• Abstract: Adaptive optimizers such as Adam (Kingma & Ba, 2015) have been central to the success of large language models. However, they maintain additional moving average states throughout training, which results in memory requirements several times greater than the model. This overhead imposes constraints on scalability and computational efficiency. On the other hand, while stochastic gradient descent (SGD) is optimal in terms of memory efficiency, their capability in LLM training is limited (Zhao et al., 2024b). To address this dilemma, we show that pre-processing SGD is sufficient to reach Adam-level performance on LLMs. Specifically, we propose to preprocess the instantaneous stochastic gradients with two simple operators: $\mathtt{GradNorm}$ and $\mathtt{GradWhitening}$. $\mathtt{GradNorm}$ stabilizes gradient distributions, and $\mathtt{GradWhitening}$ counteracts the local curvature of the loss landscape, respectively. This results in SWAN (SGD with Whitening And Normalization), a stochastic optimizer that eliminates the need to store any accumulative state variables. Empirically, SWAN has the same memory footprint as SGD, achieving $\approx 50\%$ reduction on total end-to-end memory compared to Adam. In language modeling tasks, SWAN demonstrates the same or even a substantial improvement over Adam. Specifically, when pre-training the LLaMa model with 350M and 1.3B parameters, SWAN achieves a 2x speedup by reaching the same evaluation perplexity in less than half tokens seen.

Title: Continuous Patient Monitoring with AI: Real-Time Analysis of Video in Hospital Care Settings

• Authors: Paolo Gabriel, Peter Rehani, Tyler Troy, Tiffany Wyatt, Michael Choma, Narinder Singh
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13152
• Pdf URL: https://arxiv.org/pdf/2412.13152
• Copy Paste: [[2412.13152]] Continuous Patient Monitoring with AI: Real-Time Analysis of Video in Hospital Care Settings(https://arxiv.org/abs/2412.13152)
• Keywords: secure
• Abstract: This study introduces an AI-driven platform for continuous and passive patient monitoring in hospital settings, developed by LookDeep Health. Leveraging advanced computer vision, the platform provides real-time insights into patient behavior and interactions through video analysis, securely storing inference results in the cloud for retrospective evaluation. The dataset, compiled in collaboration with 11 hospital partners, encompasses over 300 high-risk fall patients and over 1,000 days of inference, enabling applications such as fall detection and safety monitoring for vulnerable patient populations. To foster innovation and reproducibility, an anonymized subset of this dataset is publicly available. The AI system detects key components in hospital rooms, including individual presence and role, furniture location, motion magnitude, and boundary crossings. Performance evaluation demonstrates strong accuracy in object detection (macro F1-score = 0.92) and patient-role classification (F1-score = 0.98), as well as reliable trend analysis for the "patient alone" metric (mean logistic regression accuracy = 0.82 \pm 0.15). These capabilities enable automated detection of patient isolation, wandering, or unsupervised movement-key indicators for fall risk and other adverse events. This work establishes benchmarks for validating AI-driven patient monitoring systems, highlighting the platform's potential to enhance patient safety and care by providing continuous, data-driven insights into patient behavior and interactions.

Title: F-Bench: Rethinking Human Preference Evaluation Metrics for Benchmarking Face Generation, Customization, and Restoration

• Authors: Lu Liu, Huiyu Duan, Qiang Hu, Liu Yang, Chunlei Cai, Tianxiao Ye, Huayu Liu, Xiaoyun Zhang, Guangtao Zhai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13155
• Pdf URL: https://arxiv.org/pdf/2412.13155
• Copy Paste: [[2412.13155]] F-Bench: Rethinking Human Preference Evaluation Metrics for Benchmarking Face Generation, Customization, and Restoration(https://arxiv.org/abs/2412.13155)
• Keywords: generative
• Abstract: Artificial intelligence generative models exhibit remarkable capabilities in content creation, particularly in face image generation, customization, and restoration. However, current AI-generated faces (AIGFs) often fall short of human preferences due to unique distortions, unrealistic details, and unexpected identity shifts, underscoring the need for a comprehensive quality evaluation framework for AIGFs. To address this need, we introduce FaceQ, a large-scale, comprehensive database of AI-generated Face images with fine-grained Quality annotations reflecting human preferences. The FaceQ database comprises 12,255 images generated by 29 models across three tasks: (1) face generation, (2) face customization, and (3) face restoration. It includes 32,742 mean opinion scores (MOSs) from 180 annotators, assessed across multiple dimensions: quality, authenticity, identity (ID) fidelity, and text-image correspondence. Using the FaceQ database, we establish F-Bench, a benchmark for comparing and evaluating face generation, customization, and restoration models, highlighting strengths and weaknesses across various prompts and evaluation dimensions. Additionally, we assess the performance of existing image quality assessment (IQA), face quality assessment (FQA), AI-generated content image quality assessment (AIGCIQA), and preference evaluation metrics, manifesting that these standard metrics are relatively ineffective in evaluating authenticity, ID fidelity, and text-image correspondence. The FaceQ database will be publicly available upon publication.

Title: S2S2: Semantic Stacking for Robust Semantic Segmentation in Medical Imaging

• Authors: Yimu Pan, Sitao Zhang, Alison D. Gernand, Jeffery A. Goldstein, James Z. Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13156
• Pdf URL: https://arxiv.org/pdf/2412.13156
• Copy Paste: [[2412.13156]] S2S2: Semantic Stacking for Robust Semantic Segmentation in Medical Imaging(https://arxiv.org/abs/2412.13156)
• Keywords: robust, segmentation
• Abstract: Robustness and generalizability in medical image segmentation are often hindered by scarcity and limited diversity of training data, which stands in contrast to the variability encountered during inference. While conventional strategies -- such as domain-specific augmentation, specialized architectures, and tailored training procedures -- can alleviate these issues, they depend on the availability and reliability of domain knowledge. When such knowledge is unavailable, misleading, or improperly applied, performance may deteriorate. In response, we introduce a novel, domain-agnostic, add-on, and data-driven strategy inspired by image stacking in image denoising. Termed ``semantic stacking,'' our method estimates a denoised semantic representation that complements the conventional segmentation loss during training. This method does not depend on domain-specific assumptions, making it broadly applicable across diverse image modalities, model architectures, and augmentation techniques. Through extensive experiments, we validate the superiority of our approach in improving segmentation performance under diverse conditions. Code is available at this https URL.

Title: Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study

• Authors: Bolei Ma, Berk Yoztyurk, Anna-Carolina Haensch, Xinpeng Wang, Markus Herklotz, Frauke Kreuter, Barbara Plank, Matthias Assenmacher
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13169
• Pdf URL: https://arxiv.org/pdf/2412.13169
• Copy Paste: [[2412.13169]] Algorithmic Fidelity of Large Language Models in Generating Synthetic German Public Opinions: A Case Study(https://arxiv.org/abs/2412.13169)
• Keywords: robust, large language model
• Abstract: In recent research, large language models (LLMs) have been increasingly used to investigate public opinions. This study investigates the algorithmic fidelity of LLMs, i.e., the ability to replicate the socio-cultural context and nuanced opinions of human participants. Using open-ended survey data from the German Longitudinal Election Studies (GLES), we prompt different LLMs to generate synthetic public opinions reflective of German subpopulations by incorporating demographic features into the persona prompts. Our results show that Llama performs better than other LLMs at representing subpopulations, particularly when there is lower opinion diversity within those groups. Our findings further reveal that the LLM performs better for supporters of left-leaning parties like The Greens and The Left compared to other parties, and matches the least with the right-party AfD. Additionally, the inclusion or exclusion of specific variables in the prompts can significantly impact the models' predictions. These findings underscore the importance of aligning LLMs to more effectively model diverse public opinions while minimizing political biases and enhancing robustness in representativeness.

Title: Locate n' Rotate: Two-stage Openable Part Detection with Foundation Model Priors

• Authors: Siqi Li, Xiaoxue Chen, Haoyu Cheng, Guyue Zhou, Hao Zhao, Guanzhong Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13173
• Pdf URL: https://arxiv.org/pdf/2412.13173
• Copy Paste: [[2412.13173]] Locate n' Rotate: Two-stage Openable Part Detection with Foundation Model Priors(https://arxiv.org/abs/2412.13173)
• Keywords: transformer
• Abstract: Detecting the openable parts of articulated objects is crucial for downstream applications in intelligent robotics, such as pulling a drawer. This task poses a multitasking challenge due to the necessity of understanding object categories and motion. Most existing methods are either category-specific or trained on specific datasets, lacking generalization to unseen environments and objects. In this paper, we propose a Transformer-based Openable Part Detection (OPD) framework named Multi-feature Openable Part Detection (MOPD) that incorporates perceptual grouping and geometric priors, outperforming previous methods in performance. In the first stage of the framework, we introduce a perceptual grouping feature model that provides perceptual grouping feature priors for openable part detection, enhancing detection results through a cross-attention mechanism. In the second stage, a geometric understanding feature model offers geometric feature priors for predicting motion parameters. Compared to existing methods, our proposed approach shows better performance in both detection and motion parameter prediction. Codes and models are publicly available at this https URL

Title: ORFormer: Occlusion-Robust Transformer for Accurate Facial Landmark Detection

• Authors: Jui-Che Chiang, Hou-Ning Hu, Bo-Syuan Hou, Chia-Yu Tseng, Yu-Lun Liu, Min-Hung Chen, Yen-Yu Lin
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13174
• Pdf URL: https://arxiv.org/pdf/2412.13174
• Copy Paste: [[2412.13174]] ORFormer: Occlusion-Robust Transformer for Accurate Facial Landmark Detection(https://arxiv.org/abs/2412.13174)
• Keywords: robust, transformer
• Abstract: Although facial landmark detection (FLD) has gained significant progress, existing FLD methods still suffer from performance drops on partially non-visible faces, such as faces with occlusions or under extreme lighting conditions or poses. To address this issue, we introduce ORFormer, a novel transformer-based method that can detect non-visible regions and recover their missing features from visible parts. Specifically, ORFormer associates each image patch token with one additional learnable token called the messenger token. The messenger token aggregates features from all but its patch. This way, the consensus between a patch and other patches can be assessed by referring to the similarity between its regular and messenger embeddings, enabling non-visible region identification. Our method then recovers occluded patches with features aggregated by the messenger tokens. Leveraging the recovered features, ORFormer compiles high-quality heatmaps for the downstream FLD task. Extensive experiments show that our method generates heatmaps resilient to partial occlusions. By integrating the resultant heatmaps into existing FLD methods, our method performs favorably against the state of the arts on challenging datasets such as WFLW and COFW.

Title: DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation

• Authors: Miriam Wanner, Benjamin Van Durme, Mark Dredze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13175
• Pdf URL: https://arxiv.org/pdf/2412.13175
• Copy Paste: [[2412.13175]] DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation(https://arxiv.org/abs/2412.13175)
• Keywords: large language model
• Abstract: The decompose-then-verify strategy for verification of Large Language Model (LLM) generations decomposes claims that are then independently verified. Decontextualization augments text (claims) to ensure it can be verified outside of the original context, enabling reliable verification. While decomposition and decontextualization have been explored independently, their interactions in a complete system have not been investigated. Their conflicting purposes can create tensions: decomposition isolates atomic facts while decontextualization inserts relevant information. Furthermore, a decontextualized subclaim presents a challenge to the verification step: what part of the augmented text should be verified as it now contains multiple atomic facts? We conduct an evaluation of different decomposition, decontextualization, and verification strategies and find that the choice of strategy matters in the resulting factuality scores. Additionally, we introduce DnDScore, a decontextualization aware verification method which validates subclaims in the context of contextual information.

Title: SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents

• Authors: Sheng Yin, Xianghe Pang, Yuanzhuo Ding, Menglan Chen, Yutong Bi, Yichen Xiong, Wenhao Huang, Zhen Xiang, Jing Shao, Siheng Chen
• Subjects: cs.CR, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.13178
• Pdf URL: https://arxiv.org/pdf/2412.13178
• Copy Paste: [[2412.13178]] SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents(https://arxiv.org/abs/2412.13178)
• Keywords: large language model
• Abstract: With the integration of large language models (LLMs), embodied agents have strong capabilities to execute complicated instructions in natural language, paving a way for the potential deployment of embodied robots. However, a foreseeable issue is that those embodied agents can also flawlessly execute some hazardous tasks, potentially causing damages in real world. To study this issue, we present SafeAgentBench -- a new benchmark for safety-aware task planning of embodied LLM agents. SafeAgentBench includes: (1) a new dataset with 750 tasks, covering 10 potential hazards and 3 task types; (2) SafeAgentEnv, a universal embodied environment with a low-level controller, supporting multi-agent execution with 17 high-level actions for 8 state-of-the-art baselines; and (3) reliable evaluation methods from both execution and semantic perspectives. Experimental results show that the best-performing baseline gets 69% success rate for safe tasks, but only 5% rejection rate for hazardous tasks, indicating significant safety risks. More details and codes are available at this https URL.

Title: A Pipeline and NIR-Enhanced Dataset for Parking Lot Segmentation

• Authors: Shirin Qiam, Saipraneeth Devunuri, Lewis J. Lehe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13179
• Pdf URL: https://arxiv.org/pdf/2412.13179
• Copy Paste: [[2412.13179]] A Pipeline and NIR-Enhanced Dataset for Parking Lot Segmentation(https://arxiv.org/abs/2412.13179)
• Keywords: segmentation
• Abstract: Discussions of minimum parking requirement policies often include maps of parking lots, which are time consuming to construct manually. Open source datasets for such parking lots are scarce, particularly for US cities. This paper introduces the idea of using Near-Infrared (NIR) channels as input and several post-processing techniques to improve the prediction of off-street surface parking lots using satellite imagery. We constructed two datasets with 12,617 image-mask pairs each: one with 3-channel (RGB) and another with 4-channel (RGB + NIR). The datasets were used to train five deep learning models (OneFormer, Mask2Former, SegFormer, DeepLabV3, and FCN) for semantic segmentation, classifying images to differentiate between parking and non-parking pixels. Our results demonstrate that the NIR channel improved accuracy because parking lots are often surrounded by grass, even though the NIR channel needed to be upsampled from a lower resolution. Post-processing including eliminating erroneous holes, simplifying edges, and removing road and building footprints further improved the accuracy. Best model, OneFormer trained on 4-channel input and paired with post-processing techniques achieves a mean Intersection over Union (mIoU) of 84.9 percent and a pixel-wise accuracy of 96.3 percent.

Title: Real-time Free-view Human Rendering from Sparse-view RGB Videos using Double Unprojected Textures

• Authors: Guoxing Sun, Rishabh Dabral, Heming Zhu, Pascal Fua, Christian Theobalt, Marc Habermann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13183
• Pdf URL: https://arxiv.org/pdf/2412.13183
• Copy Paste: [[2412.13183]] Real-time Free-view Human Rendering from Sparse-view RGB Videos using Double Unprojected Textures(https://arxiv.org/abs/2412.13183)
• Keywords: robust
• Abstract: Real-time free-view human rendering from sparse-view RGB inputs is a challenging task due to the sensor scarcity and the tight time budget. To ensure efficiency, recent methods leverage 2D CNNs operating in texture space to learn rendering primitives. However, they either jointly learn geometry and appearance, or completely ignore sparse image information for geometry estimation, significantly harming visual quality and robustness to unseen body poses. To address these issues, we present Double Unprojected Textures, which at the core disentangles coarse geometric deformation estimation from appearance synthesis, enabling robust and photorealistic 4K rendering in real-time. Specifically, we first introduce a novel image-conditioned template deformation network, which estimates the coarse deformation of the human template from a first unprojected texture. This updated geometry is then used to apply a second and more accurate texture unprojection. The resulting texture map has fewer artifacts and better alignment with input views, which benefits our learning of finer-level geometry and appearance represented by Gaussian splats. We validate the effectiveness and efficiency of the proposed method in quantitative and qualitative experiments, which significantly surpasses other state-of-the-art methods.

Title: Move-in-2D: 2D-Conditioned Human Motion Generation

• Authors: Hsin-Ping Huang, Yang Zhou, Jui-Hsien Wang, Difan Liu, Feng Liu, Ming-Hsuan Yang, Zhan Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13185
• Pdf URL: https://arxiv.org/pdf/2412.13185
• Copy Paste: [[2412.13185]] Move-in-2D: 2D-Conditioned Human Motion Generation(https://arxiv.org/abs/2412.13185)
• Keywords: diffusion
• Abstract: Generating realistic human videos remains a challenging task, with the most effective methods currently relying on a human motion sequence as a control signal. Existing approaches often use existing motion extracted from other videos, which restricts applications to specific motion types and global scene matching. We propose Move-in-2D, a novel approach to generate human motion sequences conditioned on a scene image, allowing for diverse motion that adapts to different scenes. Our approach utilizes a diffusion model that accepts both a scene image and text prompt as inputs, producing a motion sequence tailored to the scene. To train this model, we collect a large-scale video dataset featuring single-human activities, annotating each video with the corresponding human motion as the target output. Experiments demonstrate that our method effectively predicts human motion that aligns with the scene image after projection. Furthermore, we show that the generated motion sequence improves human motion quality in video synthesis tasks.

Title: StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models

• Authors: Yunzhi Yan, Zhen Xu, Haotong Lin, Haian Jin, Haoyu Guo, Yida Wang, Kun Zhan, Xianpeng Lang, Hujun Bao, Xiaowei Zhou, Sida Peng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13188
• Pdf URL: https://arxiv.org/pdf/2412.13188
• Copy Paste: [[2412.13188]] StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models(https://arxiv.org/abs/2412.13188)
• Keywords: diffusion, generative
• Abstract: This paper aims to tackle the problem of photorealistic view synthesis from vehicle sensor data. Recent advancements in neural scene representation have achieved notable success in rendering high-quality autonomous driving scenes, but the performance significantly degrades as the viewpoint deviates from the training trajectory. To mitigate this problem, we introduce StreetCrafter, a novel controllable video diffusion model that utilizes LiDAR point cloud renderings as pixel-level conditions, which fully exploits the generative prior for novel view synthesis, while preserving precise camera control. Moreover, the utilization of pixel-level LiDAR conditions allows us to make accurate pixel-level edits to target scenes. In addition, the generative prior of StreetCrafter can be effectively incorporated into dynamic scene representations to achieve real-time rendering. Experiments on Waymo Open Dataset and PandaSet demonstrate that our model enables flexible control over viewpoint changes, enlarging the view synthesis regions for satisfying rendering, which outperforms existing methods.

Title: GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understanding

• Authors: Haoyi Jiang, Liu Liu, Tianheng Cheng, Xinjie Wang, Tianwei Lin, Zhizhong Su, Wenyu Liu, Xinggang Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13193
• Pdf URL: https://arxiv.org/pdf/2412.13193
• Copy Paste: [[2412.13193]] GaussTR: Foundation Model-Aligned Gaussian Transformer for Self-Supervised 3D Spatial Understanding(https://arxiv.org/abs/2412.13193)
• Keywords: transformer
• Abstract: 3D Semantic Occupancy Prediction is fundamental for spatial understanding as it provides a comprehensive semantic cognition of surrounding environments. However, prevalent approaches primarily rely on extensive labeled data and computationally intensive voxel-based modeling, restricting the scalability and generalizability of 3D representation learning. In this paper, we introduce GaussTR, a novel Gaussian Transformer that leverages alignment with foundation models to advance self-supervised 3D spatial understanding. GaussTR adopts a Transformer architecture to predict sparse sets of 3D Gaussians that represent scenes in a feed-forward manner. Through aligning rendered Gaussian features with diverse knowledge from pre-trained foundation models, GaussTR facilitates the learning of versatile 3D representations and enables open-vocabulary occupancy prediction without explicit annotations. Empirical evaluations on the Occ3D-nuScenes dataset showcase GaussTR's state-of-the-art zero-shot performance, achieving 11.70 mIoU while reducing training duration by approximately 50%. These experimental results highlight the significant potential of GaussTR for scalable and holistic 3D spatial understanding, with promising implications for autonomous driving and embodied agents. Code is available at this https URL.

Title: CoMPaSS: Enhancing Spatial Understanding in Text-to-Image Diffusion Models

• Authors: Gaoyang Zhang, Bingtao Fu, Qingnan Fan, Qi Zhang, Runxing Liu, Hong Gu, Huaqi Zhang, Xinguo Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13195
• Pdf URL: https://arxiv.org/pdf/2412.13195
• Copy Paste: [[2412.13195]] CoMPaSS: Enhancing Spatial Understanding in Text-to-Image Diffusion Models(https://arxiv.org/abs/2412.13195)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models excel at generating photorealistic images, but commonly struggle to render accurate spatial relationships described in text prompts. We identify two core issues underlying this common failure: 1) the ambiguous nature of spatial-related data in existing datasets, and 2) the inability of current text encoders to accurately interpret the spatial semantics of input descriptions. We address these issues with CoMPaSS, a versatile training framework that enhances spatial understanding of any T2I diffusion model. CoMPaSS solves the ambiguity of spatial-related data with the Spatial Constraints-Oriented Pairing (SCOP) data engine, which curates spatially-accurate training data through a set of principled spatial constraints. To better exploit the curated high-quality spatial priors, CoMPaSS further introduces a Token ENcoding ORdering (TENOR) module to allow better exploitation of high-quality spatial priors, effectively compensating for the shortcoming of text encoders. Extensive experiments on four popular open-weight T2I diffusion models covering both UNet- and MMDiT-based architectures demonstrate the effectiveness of CoMPaSS by setting new state-of-the-arts with substantial relative gains across well-known benchmarks on spatial relationships generation, including VISOR (+98%), T2I-CompBench Spatial (+67%), and GenEval Position (+131%). Code will be available at this https URL.