2024-11-25

Title: GeMID: Generalizable Models for IoT Device Identification

• Authors: Kahraman Kostas, Rabia Yasa Kostas, Mike Just, Michael A. Lones
• Subjects: cs.CR, cs.AI, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.14441
• Pdf URL: https://arxiv.org/pdf/2411.14441
• Copy Paste: [[2411.14441]] GeMID: Generalizable Models for IoT Device Identification(https://arxiv.org/abs/2411.14441)
• Keywords: security, robust
• Abstract: With the proliferation of Internet of Things (IoT) devices, ensuring their security has become paramount. Device identification (DI), which distinguishes IoT devices based on their traffic patterns, plays a crucial role in both differentiating devices and identifying vulnerable ones, closing a serious security gap. However, existing approaches to DI that build machine learning models often overlook the challenge of model generalizability across diverse network environments. In this study, we propose a novel framework to address this limitation and evaluate the generalizability of DI models across datasets collected within different network environments. Our approach involves a two-step process: first, we develop a feature and model selection method that is more robust to generalization issues by using a genetic algorithm with external feedback and datasets from distinct environments to refine the selections. Second, the resulting DI models are then tested on further independent datasets in order to robustly assess their generalizability. We demonstrate the effectiveness of our method by empirically comparing it to alternatives, highlighting how fundamental limitations of commonly employed techniques such as sliding window and flow statistics limit their generalizability. Our findings advance research in IoT security and device identification, offering insights into improving model effectiveness and mitigating risks in IoT networks.

Title: Unlocking the Future: A Cloud-Based Artificial Intelligence Access Control System

• Authors: Hamidreza Yaghoubi, Navtaj Randhawa, Igor Ivkić
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14444
• Pdf URL: https://arxiv.org/pdf/2411.14444
• Copy Paste: [[2411.14444]] Unlocking the Future: A Cloud-Based Artificial Intelligence Access Control System(https://arxiv.org/abs/2411.14444)
• Keywords: secure, security
• Abstract: Traditional access control systems, such as key cards, PIN pads, and physical keys, face challenges in scalability, security, and user experience in today's digital world. We present a cloud-based entry system using Raspberry Pi hardware and Amazon Web Services (AWS) technologies like Lambda, Simple Storage Service (S3), and Rekognition. This solution (AWSecure Entry System) enhances security, streamlines authentication, and increases operational efficiency.

Title: Deferred Backdoor Functionality Attacks on Deep Learning Models

• Authors: Jeongjin Shin, Sangdon Park
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14449
• Pdf URL: https://arxiv.org/pdf/2411.14449
• Copy Paste: [[2411.14449]] Deferred Backdoor Functionality Attacks on Deep Learning Models(https://arxiv.org/abs/2411.14449)
• Keywords: defense, attack, steal
• Abstract: Deep learning models are vulnerable to backdoor attacks, where adversaries inject malicious functionality during training that activates on trigger inputs at inference time. Extensive research has focused on developing stealthy backdoor attacks to evade detection and defense mechanisms. However, these approaches still have limitations that leave the door open for detection and mitigation due to their inherent design to cause malicious behavior in the presence of a trigger. To address this limitation, we introduce Deferred Backdoor Functionality Activation (DBFA), a new paradigm in backdoor attacks. Unlike conventional attacks, DBFA initially conceals its backdoor, producing benign outputs even when triggered. This stealthy behavior allows DBFA to bypass multiple detection and defense methods, remaining undetected during initial inspections. The backdoor functionality is strategically activated only after the model undergoes subsequent updates, such as retraining on benign data. DBFA attacks exploit the common practice in the life cycle of machine learning models to perform model updates and fine-tuning after initial deployment. To implement DBFA attacks, we approach the problem by making the unlearning of the backdoor fragile, allowing it to be easily cancelled and subsequently reactivate the backdoor functionality. To achieve this, we propose a novel two-stage training scheme, called DeferBad. Our extensive experiments across various fine-tuning scenarios, backdoor attack types, datasets, and model architectures demonstrate the effectiveness and stealthiness of DeferBad.

Title: Development of a threat modelling framework and a web-based threat modelling tool for micro businesses

• Authors: Etkin Getir
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14450
• Pdf URL: https://arxiv.org/pdf/2411.14450
• Copy Paste: [[2411.14450]] Development of a threat modelling framework and a web-based threat modelling tool for micro businesses(https://arxiv.org/abs/2411.14450)
• Keywords: security, attack
• Abstract: While there is a plethora of cybersecurity and risk management frameworks for different target audiences and use cases, micro-businesses (MBs) are often overlooked. As the smallest business entities, MBs represent a special case with regard to cybersecurity for two reasons: (1) Having fewer than 10 employees, they tend to lack cybersecurity expertise. (2) Because of their low turnover, they usually have a limited budget for cybersecurity. As a result, MBs are often the victims of security breaches and cyber-attacks every year, as demonstrated by various studies. This calls for a non-technical, simple solution tailored specifically for MBs. To address this pressing need, the SEANCE Cybersecurity Framework was developed through a 7-step methodology: (1) A literature review was conducted to explore the current state of research and available frameworks and methodologies, (2) followed by a qualitative survey to identify the cybersecurity challenges faced by MBs. (3) After analyzing the results of the literature review and the survey, (4) the relevant aspects of existing frameworks and tools for MBs were identified and (5) a non-technical framework was developed. (6) A web-based tool was developed to facilitate the implementation of the framework and (7) another qualitative survey was conducted to gather feedback. The SEANCE Framework suggests considering possible vulnerabilities and cyber threats in six hierarchical layers: (1) Self, (2) Employees, (3) Assets, (4) Network, (5) Customers and (6) Environment, with the underlying idea of a vulnerability in an inner layer propagates to the outer layers and therefore needs to be prioritized.

Title: The Evolution of Cryptography through Number Theory

• Authors: Fernando Peralta Castro
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14451
• Pdf URL: https://arxiv.org/pdf/2411.14451
• Copy Paste: [[2411.14451]] The Evolution of Cryptography through Number Theory(https://arxiv.org/abs/2411.14451)
• Keywords: secure, protect
• Abstract: Cryptography, derived from Greek meaning hidden writing, uses mathematical techniques to secure information by converting it into an unreadable format. While cryptography as a science began around 100 years ago, its roots trace back to ancient civilizations like Mesopotamia and Egypt. Over time, cryptography evolved from basic methods to complex systems involving number theory, such as modular arithmetic, the Euclidean algorithm, and Eulers totient function. This paper explores the link between early information hiding techniques and modern cryptographic algorithms like RSA, which use advanced number theory to secure data for billions of people. By analyzing historical methods, this study shows how the development of number theory enabled the transition from simple letter shifting ciphers, like the Caesar and Vigenere ciphers, to more sophisticated encryption methods. This evolution reflects a profound impact on daily life and the importance of number theory in protecting information.

Title: Guiding Reinforcement Learning Using Uncertainty-Aware Large Language Models

• Authors: Maryam Shoaeinaeini, Brent Harrison
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14457
• Pdf URL: https://arxiv.org/pdf/2411.14457
• Copy Paste: [[2411.14457]] Guiding Reinforcement Learning Using Uncertainty-Aware Large Language Models(https://arxiv.org/abs/2411.14457)
• Keywords: robust, large language model
• Abstract: Human guidance in reinforcement learning (RL) is often impractical for large-scale applications due to high costs and time constraints. Large Language Models (LLMs) offer a promising alternative to mitigate RL sample inefficiency and potentially replace human trainers. However, applying LLMs as RL trainers is challenging due to their overconfidence and less reliable solutions in sequential tasks. We address this limitation by introducing a calibrated guidance system that uses Monte Carlo Dropout to enhance LLM advice reliability by assessing prediction variances from multiple forward passes. Additionally, we develop a novel RL policy shaping method based on dynamic model average entropy to adjust the LLM's influence on RL policies according to guidance uncertainty. This approach ensures robust RL training by relying on reliable LLM guidance. To validate our contributions, we conduct extensive experiments in a Minigrid environment with three goals in varying environment sizes. The results showcase superior model performance compared to uncalibrated LLMs, unguided RL, and calibrated LLMs with different shaping policies. Moreover, we analyze various uncertainty estimation methods, demonstrating the effectiveness of average entropy in reflecting higher uncertainty in incorrect guidance. These findings highlight the persistent overconfidence in fine-tuned LLMs and underscore the importance of effective calibration in sequential decision-making problems.

Title: Unveiling User Preferences: A Knowledge Graph and LLM-Driven Approach for Conversational Recommendation

• Authors: Zhangchi Qiu, Linhao Luo, Shirui Pan, Alan Wee-Chung Liew
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.14459
• Pdf URL: https://arxiv.org/pdf/2411.14459
• Copy Paste: [[2411.14459]] Unveiling User Preferences: A Knowledge Graph and LLM-Driven Approach for Conversational Recommendation(https://arxiv.org/abs/2411.14459)
• Keywords: interpretability, explainability, large language model
• Abstract: Conversational Recommender Systems (CRSs) aim to provide personalized recommendations through dynamically capturing user preferences in interactive conversations. Conventional CRSs often extract user preferences as hidden representations, which are criticized for their lack of interpretability. This diminishes the transparency and trustworthiness of the recommendation process. Recent works have explored combining the impressive capabilities of Large Language Models (LLMs) with the domain-specific knowledge of Knowledge Graphs (KGs) to generate human-understandable recommendation explanations. Despite these efforts, the integration of LLMs and KGs for CRSs remains challenging due to the modality gap between unstructured dialogues and structured KGs. Moreover, LLMs pre-trained on large-scale corpora may not be well-suited for analyzing user preferences, which require domain-specific knowledge. In this paper, we propose COMPASS, a plug-and-play framework that synergizes LLMs and KGs to unveil user preferences, enhancing the performance and explainability of existing CRSs. To address integration challenges, COMPASS employs a two-stage training approach: first, it bridges the gap between the structured KG and natural language through an innovative graph entity captioning pre-training mechanism. This enables the LLM to transform KG entities into concise natural language descriptions, allowing them to comprehend domain-specific knowledge. Following, COMPASS optimizes user preference modeling via knowledge-aware instruction fine-tuning, where the LLM learns to reason and summarize user preferences from both dialogue histories and KG-augmented context. This enables COMPASS to perform knowledge-aware reasoning and generate comprehensive and interpretable user preferences that can seamlessly integrate with existing CRS models for improving recommendation performance and explainability.

Title: LLaSA: Large Language and Structured Data Assistant

• Authors: Yao Xu, Shizhu He, Zeng Xiangrong, Jiabei Chen, Guang Liu, Bingning Wang, Jun Zhao, Kang Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14460
• Pdf URL: https://arxiv.org/pdf/2411.14460
• Copy Paste: [[2411.14460]] LLaSA: Large Language and Structured Data Assistant(https://arxiv.org/abs/2411.14460)
• Keywords: large language model
• Abstract: Structured data, such as tables, graphs, and databases, play a critical role in plentiful NLP tasks such as question answering and dialogue system. Recently, inspired by Vision-Language Models, Graph Neutral Networks (GNNs) have been introduced as an additional modality into the input of Large Language Models (LLMs) to improve their performance on Structured Knowledge Grounding (SKG) tasks. However, those GNN-enhanced LLMs have the following limitations: (1) They employ diverse GNNs to model varying types of structured data, rendering them unable to uniformly process various forms of structured data. (2) The pretraining of GNNs is coupled with specific LLMs, which prevents GNNs from fully aligning with the textual space and limits their adaptability to other LLMs. To address these issues, we propose \textbf{L}arge \textbf{L}anguage and \textbf{S}tructured Data \textbf{A}ssistant (LLaSA), a general framework for enhancing LLMs' ability to handle structured data. Specifically, we represent various types of structured data in a unified hypergraph format, and use self-supervised learning to pretrain a hypergraph encoder, and a G-Former compressing encoded hypergraph representations with cross-attention. The compressed hypergraph representations are appended to the serialized inputs during training and inference stages of LLMs. Experimental results on multiple SKG tasks show that our pretrained hypergraph encoder can adapt to various LLMs and enhance their ability to process different types of structured data. Besides, LLaSA, with LoRA fine-tuning, outperforms previous SOTA method using full parameters tuning.

Title: Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarios

• Authors: Shaochen Xu, Yifan Zhou, Zhengliang Liu, Zihao Wu, Tianyang Zhong, Huaqin Zhao, Yiwei Li, Hanqi Jiang, Yi Pan, Junhao Chen, Jin Lu, Wei Zhang, Tuo Zhang, Lu Zhang, Dajiang Zhu, Xiang Li, Wei Liu, Quanzheng Li, Andrea Sikora, Xiaoming Zhai, Zhen Xiang, Tianming Liu
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.14461
• Pdf URL: https://arxiv.org/pdf/2411.14461
• Copy Paste: [[2411.14461]] Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarios(https://arxiv.org/abs/2411.14461)
• Keywords: large language model
• Abstract: Artificial Intelligence (AI) has become essential in modern healthcare, with large language models (LLMs) offering promising advances in clinical decision-making. Traditional model-based approaches, including those leveraging in-context demonstrations and those with specialized medical fine-tuning, have demonstrated strong performance in medical language processing but struggle with real-time adaptability, multi-step reasoning, and handling complex medical tasks. Agent-based AI systems address these limitations by incorporating reasoning traces, tool selection based on context, knowledge retrieval, and both short- and long-term memory. These additional features enable the medical AI agent to handle complex medical scenarios where decision-making should be built on real-time interaction with the environment. Therefore, unlike conventional model-based approaches that treat medical queries as isolated questions, medical AI agents approach them as complex tasks and behave more like human doctors. In this paper, we study the choice of the backbone LLM for medical AI agents, which is the foundation for the agent's overall reasoning and action generation. In particular, we consider the emergent o1 model and examine its impact on agents' reasoning, tool-use adaptability, and real-time information retrieval across diverse clinical scenarios, including high-stakes settings such as intensive care units (ICUs). Our findings demonstrate o1's ability to enhance diagnostic accuracy and consistency, paving the way for smarter, more responsive AI tools that support better patient outcomes and decision-making efficacy in clinical practice.

Title: Leveraging AI and NLP for Bank Marketing: A Systematic Review and Gap Analysis

• Authors: Christopher Gerling, Stefan Lessmann
• Subjects: cs.CL, cs.AI, econ.GN
• Abstract URL: https://arxiv.org/abs/2411.14463
• Pdf URL: https://arxiv.org/pdf/2411.14463
• Copy Paste: [[2411.14463]] Leveraging AI and NLP for Bank Marketing: A Systematic Review and Gap Analysis(https://arxiv.org/abs/2411.14463)
• Keywords: transformer
• Abstract: This paper explores the growing impact of AI and NLP in bank marketing, highlighting their evolving roles in enhancing marketing strategies, improving customer engagement, and creating value within this sector. While AI and NLP have been widely studied in general marketing, there is a notable gap in understanding their specific applications and potential within the banking sector. This research addresses this specific gap by providing a systematic review and strategic analysis of AI and NLP applications in bank marketing, focusing on their integration across the customer journey and operational excellence. Employing the PRISMA methodology, this study systematically reviews existing literature to assess the current landscape of AI and NLP in bank marketing. Additionally, it incorporates semantic mapping using Sentence Transformers and UMAP for strategic gap analysis to identify underexplored areas and opportunities for future research. The systematic review reveals limited research specifically focused on NLP applications in bank marketing. The strategic gap analysis identifies key areas where NLP can further enhance marketing strategies, including customer-centric applications like acquisition, retention, and personalized engagement, offering valuable insights for both academic research and practical implementation. This research contributes to the field of bank marketing by mapping the current state of AI and NLP applications and identifying strategic gaps. The findings provide actionable insights for developing NLP-driven growth and innovation frameworks and highlight the role of NLP in improving operational efficiency and regulatory compliance. This work has broader implications for enhancing customer experience, profitability, and innovation in the banking industry.

Title: Testing Uncertainty of Large Language Models for Physics Knowledge and Reasoning

• Authors: Elizaveta Reganova, Peter Steinbach
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14465
• Pdf URL: https://arxiv.org/pdf/2411.14465
• Copy Paste: [[2411.14465]] Testing Uncertainty of Large Language Models for Physics Knowledge and Reasoning(https://arxiv.org/abs/2411.14465)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have gained significant popularity in recent years for their ability to answer questions in various fields. However, these models have a tendency to "hallucinate" their responses, making it challenging to evaluate their performance. A major challenge is determining how to assess the certainty of a model's predictions and how it correlates with accuracy. In this work, we introduce an analysis for evaluating the performance of popular open-source LLMs, as well as gpt-3.5 Turbo, on multiple choice physics questionnaires. We focus on the relationship between answer accuracy and variability in topics related to physics. Our findings suggest that most models provide accurate replies in cases where they are certain, but this is by far not a general behavior. The relationship between accuracy and uncertainty exposes a broad horizontal bell-shaped distribution. We report how the asymmetry between accuracy and uncertainty intensifies as the questions demand more logical reasoning of the LLM agent, while the same relationship remains sharp for knowledge retrieval tasks.

Title: Learning to Ask: Conversational Product Search via Representation Learning

• Authors: Jie Zou, Jimmy Xiangji Huang, Zhaochun Ren, Evangelos Kanoulas
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.14466
• Pdf URL: https://arxiv.org/pdf/2411.14466
• Copy Paste: [[2411.14466]] Learning to Ask: Conversational Product Search via Representation Learning(https://arxiv.org/abs/2411.14466)
• Keywords: robust, generative
• Abstract: Online shopping platforms, such as Amazon and AliExpress, are increasingly prevalent in society, helping customers purchase products conveniently. With recent progress in natural language processing, researchers and practitioners shift their focus from traditional product search to conversational product search. Conversational product search enables user-machine conversations and through them collects explicit user feedback that allows to actively clarify the users' product preferences. Therefore, prospective research on an intelligent shopping assistant via conversations is indispensable. Existing publications on conversational product search either model conversations independently from users, queries, and products or lead to a vocabulary mismatch. In this work, we propose a new conversational product search model, ConvPS, to assist users in locating desirable items. The model is first trained to jointly learn the semantic representations of user, query, item, and conversation via a unified generative framework. After learning these representations, they are integrated to retrieve the target items in the latent semantic space. Meanwhile, we propose a set of greedy and explore-exploit strategies to learn to ask the user a sequence of high-performance questions for conversations. Our proposed ConvPS model can naturally integrate the representation learning of the user, query, item, and conversation into a unified generative framework, which provides a promising avenue for constructing accurate and robust conversational product search systems that are flexible and adaptive. Experimental results demonstrate that our ConvPS model significantly outperforms state-of-the-art baselines.

Title: A Neural Network Training Method Based on Distributed PID Control

• Authors: Jiang Kun
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2411.14468
• Pdf URL: https://arxiv.org/pdf/2411.14468
• Copy Paste: [[2411.14468]] A Neural Network Training Method Based on Distributed PID Control(https://arxiv.org/abs/2411.14468)
• Keywords: interpretability
• Abstract: In the previous article, we introduced a neural network framework based on symmetric differential equations. This novel framework exhibits complete symmetry, endowing it with perfect mathematical properties. While we have examined some of the system's mathematical characteristics, a detailed discussion of the network training methodology has not yet been presented. Drawing on the principles of the traditional backpropagation algorithm, this study proposes an alternative training approach that utilizes differential equation signal propagation instead of chain rule derivation. This approach not only preserves the effectiveness of training but also offers enhanced biological interpretability. The foundation of this methodology lies in the system's reversibility, which stems from its inherent symmetry,a key aspect of our research. However, this method alone is insufficient for effective neural network training. To address this, we further introduce a distributed Proportional-Integral-Derivative (PID) control approach, emphasizing its implementation within a closed system. By incorporating this method, we achieved both faster training speeds and improved accuracy. This approach not only offers novel insights into neural network training but also extends the scope of research into control methodologies. To validate its effectiveness, we apply this method to the MNIST dataset, demonstrating its practical utility.

Title: Popular LLMs Amplify Race and Gender Disparities in Human Mobility

• Authors: Xinhua Wu, Qi R. Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14469
• Pdf URL: https://arxiv.org/pdf/2411.14469
• Copy Paste: [[2411.14469]] Popular LLMs Amplify Race and Gender Disparities in Human Mobility(https://arxiv.org/abs/2411.14469)
• Keywords: large language model
• Abstract: As large language models (LLMs) are increasingly applied in areas influencing societal outcomes, it is critical to understand their tendency to perpetuate and amplify biases. This study investigates whether LLMs exhibit biases in predicting human mobility -- a fundamental human behavior -- based on race and gender. Using three prominent LLMs -- GPT-4, Gemini, and Claude -- we analyzed their predictions of visitations to points of interest (POIs) for individuals, relying on prompts that included names with and without explicit demographic details. We find that LLMs frequently reflect and amplify existing societal biases. Specifically, predictions for minority groups were disproportionately skewed, with these individuals being significantly less likely to be associated with wealth-related points of interest (POIs). Gender biases were also evident, as female individuals were consistently linked to fewer career-related POIs compared to their male counterparts. These biased associations suggest that LLMs not only mirror but also exacerbate societal stereotypes, particularly in contexts involving race and gender.

Title: Exploring the Potential Role of Generative AI in the TRAPD Procedure for Survey Translation

• Authors: Erica Ann Metheney, Lauren Yehle
• Subjects: cs.CL, stat.AP, stat.ME
• Abstract URL: https://arxiv.org/abs/2411.14472
• Pdf URL: https://arxiv.org/pdf/2411.14472
• Copy Paste: [[2411.14472]] Exploring the Potential Role of Generative AI in the TRAPD Procedure for Survey Translation(https://arxiv.org/abs/2411.14472)
• Keywords: generative
• Abstract: This paper explores and assesses in what ways generative AI can assist in translating survey instruments. Writing effective survey questions is a challenging and complex task, made even more difficult for surveys that will be translated and deployed in multiple linguistic and cultural settings. Translation errors can be detrimental, with known errors rendering data unusable for its intended purpose and undetected errors leading to incorrect conclusions. A growing number of institutions face this problem as surveys deployed by private and academic organizations globalize, and the success of their current efforts depends heavily on researchers' and translators' expertise and the amount of time each party has to contribute to the task. Thus, multilinguistic and multicultural surveys produced by teams with limited expertise, budgets, or time are at significant risk for translation-based errors in their data. We implement a zero-shot prompt experiment using ChatGPT to explore generative AI's ability to identify features of questions that might be difficult to translate to a linguistic audience other than the source language. We find that ChatGPT can provide meaningful feedback on translation issues, including common source survey language, inconsistent conceptualization, sensitivity and formality issues, and nonexistent concepts. In addition, we provide detailed information on the practicality of the approach, including accessing the necessary software, associated costs, and computational run times. Lastly, based on our findings, we propose avenues for future research that integrate AI into survey translation practices.

Title: Large Language Model for Qualitative Research -- A Systematic Mapping Study

• Authors: Cauã Ferreira Barros, Bruna Borges Azevedo, Valdemar Vicente Graciano Neto, Mohamad Kassab, Marcos Kalinowski, Hugo Alexandre D. do Nascimento, Michelle C.G.S.P. Bandeira
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14473
• Pdf URL: https://arxiv.org/pdf/2411.14473
• Copy Paste: [[2411.14473]] Large Language Model for Qualitative Research -- A Systematic Mapping Study(https://arxiv.org/abs/2411.14473)
• Keywords: robust, generative, large language model
• Abstract: The exponential growth of text-based data in domains such as healthcare, education, and social sciences has outpaced the capacity of traditional qualitative analysis methods, which are time-intensive and prone to subjectivity. Large Language Models (LLMs), powered by advanced generative AI, have emerged as transformative tools capable of automating and enhancing qualitative analysis. This study systematically maps the literature on the use of LLMs for qualitative research, exploring their application contexts, configurations, methodologies, and evaluation metrics. Findings reveal that LLMs are utilized across diverse fields, demonstrating the potential to automate processes traditionally requiring extensive human input. However, challenges such as reliance on prompt engineering, occasional inaccuracies, and contextual limitations remain significant barriers. This research highlights opportunities for integrating LLMs with human expertise, improving model robustness, and refining evaluation methodologies. By synthesizing trends and identifying research gaps, this study aims to guide future innovations in the application of LLMs for qualitative analysis.

Title: StreetviewLLM: Extracting Geographic Information Using a Chain-of-Thought Multimodal Large Language Model

• Authors: Zongrong Li, Junhao Xu, Siqin Wang, Yifan Wu, Haiyang Li
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14476
• Pdf URL: https://arxiv.org/pdf/2411.14476
• Copy Paste: [[2411.14476]] StreetviewLLM: Extracting Geographic Information Using a Chain-of-Thought Multimodal Large Language Model(https://arxiv.org/abs/2411.14476)
• Keywords: extraction, large language model
• Abstract: Geospatial predictions are crucial for diverse fields such as disaster management, urban planning, and public health. Traditional machine learning methods often face limitations when handling unstructured or multi-modal data like street view imagery. To address these challenges, we propose StreetViewLLM, a novel framework that integrates a large language model with the chain-of-thought reasoning and multimodal data sources. By combining street view imagery with geographic coordinates and textual data, StreetViewLLM improves the precision and granularity of geospatial predictions. Using retrieval-augmented generation techniques, our approach enhances geographic information extraction, enabling a detailed analysis of urban environments. The model has been applied to seven global cities, including Hong Kong, Tokyo, Singapore, Los Angeles, New York, London, and Paris, demonstrating superior performance in predicting urban indicators, including population density, accessibility to healthcare, normalized difference vegetation index, building height, and impervious surface. The results show that StreetViewLLM consistently outperforms baseline models, offering improved predictive accuracy and deeper insights into the built environment. This research opens new opportunities for integrating the large language model into urban analytics, decision-making in urban planning, infrastructure management, and environmental monitoring.

Title: GRL-Prompt: Towards Knowledge Graph based Prompt Optimization via Reinforcement Learning

• Authors: Yuze Liu, Tingjie Liu, Tiehua Zhang, Youhua Xia, Jinze Wang, Zhishu Shen, Jiong Jin, Fei Richard Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14479
• Pdf URL: https://arxiv.org/pdf/2411.14479
• Copy Paste: [[2411.14479]] GRL-Prompt: Towards Knowledge Graph based Prompt Optimization via Reinforcement Learning(https://arxiv.org/abs/2411.14479)
• Keywords: large language model
• Abstract: Large language models (LLMs) have demonstrated impressive success in a wide range of natural language processing (NLP) tasks due to their extensive general knowledge of the world. Recent works discovered that the performance of LLMs is heavily dependent on the input prompt. However, prompt engineering is usually done manually in a trial-and-error fashion, which can be labor-intensive and challenging in order to find the optimal prompts. To address these problems and unleash the utmost potential of LLMs, we propose a novel LLMs-agnostic framework for prompt optimization, namely GRL-Prompt, which aims to automatically construct optimal prompts via reinforcement learning (RL) in an end-to-end manner. To provide structured action/state representation for optimizing prompts, we construct a knowledge graph (KG) that better encodes the correlation between the user query and candidate in-context examples. Furthermore, a policy network is formulated to generate the optimal action by selecting a set of in-context examples in a rewardable order to construct the prompt. Additionally, the embedding-based reward shaping is utilized to stabilize the RL training process. The experimental results show that GRL-Prompt outperforms recent state-of-the-art methods, achieving an average increase of 0.10 in ROUGE-1, 0.07 in ROUGE-2, 0.07 in ROUGE-L, and 0.05 in BLEU.

Title: Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat

• Authors: Roland Daynauth, Christopher Clarke, Krisztian Flautner, Lingjia Tang, Jason Mars
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14483
• Pdf URL: https://arxiv.org/pdf/2411.14483
• Copy Paste: [[2411.14483]] Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat(https://arxiv.org/abs/2411.14483)
• Keywords: robust, large language model
• Abstract: Deciding which large language model (LLM) to use is a complex challenge. Pairwise ranking has emerged as a new method for evaluating human preferences for LLMs. This approach entails humans evaluating pairs of model outputs based on a predefined criterion. By collecting these comparisons, a ranking can be constructed using methods such as Elo. However, applying these algorithms as constructed in the context of LLM evaluation introduces several challenges. In this paper, we explore the effectiveness of ranking systems for head-to-head comparisons of LLMs. We formally define a set of fundamental principles for effective ranking and conduct a series of extensive evaluations on the robustness of several ranking algorithms in the context of LLMs. Our analysis uncovers key insights into the factors that affect ranking accuracy and efficiency, offering guidelines for selecting the most appropriate methods based on specific evaluation contexts and resource constraints.

Title: Robust Planning with Compound LLM Architectures: An LLM-Modulo Approach

• Authors: Atharva Gundawar, Karthik Valmeekam, Mudit Verma, Subbarao Kambhampati
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14484
• Pdf URL: https://arxiv.org/pdf/2411.14484
• Copy Paste: [[2411.14484]] Robust Planning with Compound LLM Architectures: An LLM-Modulo Approach(https://arxiv.org/abs/2411.14484)
• Keywords: robust, large language model
• Abstract: Previous work has attempted to boost Large Language Model (LLM) performance on planning and scheduling tasks through a variety of prompt engineering techniques. While these methods can work within the distributions tested, they are neither robust nor predictable. This limitation can be addressed through compound LLM architectures where LLMs work in conjunction with other components to ensure reliability. In this paper, we present a technical evaluation of a compound LLM architecture--the LLM-Modulo framework. In this framework, an LLM is paired with a complete set of sound verifiers that validate its output, re-prompting it if it fails. This approach ensures that the system can never output any fallacious output, and therefore that every output generated is guaranteed correct--something previous techniques have not been able to claim. Our results, evaluated across four scheduling domains, demonstrate significant performance gains with the LLM-Modulo framework using various models. Additionally, we explore modifications to the base configuration of the framework and assess their impact on overall system performance.

Title: Mediating Modes of Thought: LLM's for design scripting

• Authors: Moritz Rietschel, Fang Guo, Kyle Steinfeld
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.14485
• Pdf URL: https://arxiv.org/pdf/2411.14485
• Copy Paste: [[2411.14485]] Mediating Modes of Thought: LLM's for design scripting(https://arxiv.org/abs/2411.14485)
• Keywords: large language model
• Abstract: Here is an updated version of your abstract, cleaned for submission to arXiv with potential "bad characters" corrected to conform to ASCII standards: Architects adopt visual scripting and parametric design tools to explore more expansive design spaces (Coates, 2010), refine their thinking about the geometric logic of their design (Woodbury, 2010), and overcome conventional software limitations (Burry, 2011). Despite two decades of effort to make design scripting more accessible, a disconnect between a designer's free ways of thinking and the rigidity of algorithms remains (Burry, 2011). Recent developments in Large Language Models (LLMs) suggest this might soon change, as LLMs encode a general understanding of human context and exhibit the capacity to produce geometric logic. This project speculates that if LLMs can effectively mediate between user intent and algorithms, they become a powerful tool to make scripting in design more widespread and fun. We explore if such systems can interpret natural language prompts to assemble geometric operations relevant to computational design scripting. In the system, multiple layers of LLM agents are configured with specific context to infer the user intent and construct a sequential logic. Given a user's high-level text prompt, a geometric description is created, distilled into a sequence of logic operations, and mapped to software-specific commands. The completed script is constructed in the user's visual programming interface. The system succeeds in generating complete visual scripts up to a certain complexity but fails beyond this complexity threshold. It shows how LLMs can make design scripting much more aligned with human creativity and thought. Future research should explore conversational interactions, expand to multimodal inputs and outputs, and assess the performance of these tools.

Title: The Impossible Test: A 2024 Unsolvable Dataset and A Chance for an AGI Quiz

• Authors: David Noever, Forrest McKee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14486
• Pdf URL: https://arxiv.org/pdf/2411.14486
• Copy Paste: [[2411.14486]] The Impossible Test: A 2024 Unsolvable Dataset and A Chance for an AGI Quiz(https://arxiv.org/abs/2411.14486)
• Keywords: large language model
• Abstract: This research introduces a novel evaluation framework designed to assess large language models' (LLMs) ability to acknowledge uncertainty on 675 fundamentally unsolvable problems. Using a curated dataset of graduate-level grand challenge questions with intentionally unknowable answers, we evaluated twelve state-of-the-art LLMs, including both open and closed-source models, on their propensity to admit ignorance rather than generate plausible but incorrect responses. The best models scored in 62-68% accuracy ranges for admitting the problem solution was unknown in fields ranging from biology to philosophy and mathematics. We observed an inverse relationship between problem difficulty and model accuracy, with GPT-4 demonstrating higher rates of uncertainty acknowledgment on more challenging problems (35.8%) compared to simpler ones (20.0%). This pattern indicates that models may be more prone to generate speculative answers when problems appear more tractable. The study also revealed significant variations across problem categories, with models showing difficulty in acknowledging uncertainty in invention and NP-hard problems while performing relatively better on philosophical and psychological challenges. These results contribute to the growing body of research on artificial general intelligence (AGI) assessment by highlighting the importance of uncertainty recognition as a critical component of future machine intelligence evaluation. This impossibility test thus extends previous theoretical frameworks for universal intelligence testing by providing empirical evidence of current limitations in LLMs' ability to recognize their own knowledge boundaries, suggesting new directions for improving model training architectures and evaluation approaches.

Title: Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine

• Authors: Yifan Yang, Qiao Jin, Robert Leaman, Xiaoyu Liu, Guangzhi Xiong, Maame Sarfo-Gyamfi, Changlin Gong, Santiago Ferrière-Steinert, W. John Wilbur, Xiaojun Li, Jiaxin Yuan, Bang An, Kelvin S. Castro, Francisco Erramuspe Álvarez, Matías Stockle, Aidong Zhang, Furong Huang, Zhiyong Lu
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.14487
• Pdf URL: https://arxiv.org/pdf/2411.14487
• Copy Paste: [[2411.14487]] Ensuring Safety and Trust: Analyzing the Risks of Large Language Models in Medicine(https://arxiv.org/abs/2411.14487)
• Keywords: privacy, robust, fair, large language model
• Abstract: The remarkable capabilities of Large Language Models (LLMs) make them increasingly compelling for adoption in real-world healthcare applications. However, the risks associated with using LLMs in medical applications have not been systematically characterized. We propose using five key principles for safe and trustworthy medical AI: Truthfulness, Resilience, Fairness, Robustness, and Privacy, along with ten specific aspects. Under this comprehensive framework, we introduce a novel MedGuard benchmark with 1,000 expert-verified questions. Our evaluation of 11 commonly used LLMs shows that the current language models, regardless of their safety alignment mechanisms, generally perform poorly on most of our benchmarks, particularly when compared to the high performance of human physicians. Despite recent reports indicate that advanced LLMs like ChatGPT can match or even exceed human performance in various medical tasks, this study underscores a significant safety gap, highlighting the crucial need for human oversight and the implementation of AI safety guardrails.

Title: A Survey on Human-Centric LLMs

• Authors: Jing Yi Wang, Nicholas Sukiennik, Tong Li, Weikang Su, Qianyue Hao, Jingbo Xu, Zihan Huang, Fengli Xu, Yong Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14491
• Pdf URL: https://arxiv.org/pdf/2411.14491
• Copy Paste: [[2411.14491]] A Survey on Human-Centric LLMs(https://arxiv.org/abs/2411.14491)
• Keywords: large language model
• Abstract: The rapid evolution of large language models (LLMs) and their capacity to simulate human cognition and behavior has given rise to LLM-based frameworks and tools that are evaluated and applied based on their ability to perform tasks traditionally performed by humans, namely those involving cognition, decision-making, and social interaction. This survey provides a comprehensive examination of such human-centric LLM capabilities, focusing on their performance in both individual tasks (where an LLM acts as a stand-in for a single human) and collective tasks (where multiple LLMs coordinate to mimic group dynamics). We first evaluate LLM competencies across key areas including reasoning, perception, and social cognition, comparing their abilities to human-like skills. Then, we explore real-world applications of LLMs in human-centric domains such as behavioral science, political science, and sociology, assessing their effectiveness in replicating human behaviors and interactions. Finally, we identify challenges and future research directions, such as improving LLM adaptability, emotional intelligence, and cultural sensitivity, while addressing inherent biases and enhancing frameworks for human-AI collaboration. This survey aims to provide a foundational understanding of LLMs from a human-centric perspective, offering insights into their current capabilities and potential for future development.

Title: dc-GAN: Dual-Conditioned GAN for Face Demorphing From a Single Morph

• Authors: Nitish Shukla, Arun Ross
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14494
• Pdf URL: https://arxiv.org/pdf/2411.14494
• Copy Paste: [[2411.14494]] dc-GAN: Dual-Conditioned GAN for Face Demorphing From a Single Morph(https://arxiv.org/abs/2411.14494)
• Keywords: attack
• Abstract: A facial morph is an image created by combining two face images pertaining to two distinct identities. Face demorphing inverts the process and tries to recover the original images constituting a facial morph. While morph attack detection (MAD) techniques can be used to flag morph images, they do not divulge any visual information about the faces used to create them. Demorphing helps address this problem. Existing demorphing techniques are either very restrictive (assume identities during testing) or produce feeble outputs (both outputs look very similar). In this paper, we overcome these issues by proposing dc-GAN, a novel GAN-based demorphing method conditioned on the morph images. Our method overcomes morph-replication and produces high quality reconstructions of the bonafide images used to create the morphs. Moreover, our method is highly generalizable across demorphing paradigms (differential/reference-free). We conduct experiments on AMSL, FRLL-Morphs and MorDiff datasets to showcase the efficacy of our method.

Title: Test-Time Adaptation of 3D Point Clouds via Denoising Diffusion Models

• Authors: Hamidreza Dastmalchi, Aijun An, Ali Cheraghian, Shafin Rahman, Sameera Ramasinghe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14495
• Pdf URL: https://arxiv.org/pdf/2411.14495
• Copy Paste: [[2411.14495]] Test-Time Adaptation of 3D Point Clouds via Denoising Diffusion Models(https://arxiv.org/abs/2411.14495)
• Keywords: diffusion
• Abstract: Test-time adaptation (TTA) of 3D point clouds is crucial for mitigating discrepancies between training and testing samples in real-world scenarios, particularly when handling corrupted point clouds. LiDAR data, for instance, can be affected by sensor failures or environmental factors, causing domain gaps. Adapting models to these distribution shifts online is crucial, as training for every possible variation is impractical. Existing methods often focus on fine-tuning pre-trained models based on self-supervised learning or pseudo-labeling, which can lead to forgetting valuable source domain knowledge over time and reduce generalization on future tests. In this paper, we introduce a novel 3D test-time adaptation method, termed 3DD-TTA, which stands for 3D Denoising Diffusion Test-Time Adaptation. This method uses a diffusion strategy that adapts input point cloud samples to the source domain while keeping the source model parameters intact. The approach uses a Variational Autoencoder (VAE) to encode the corrupted point cloud into a shape latent and latent points. These latent points are corrupted with Gaussian noise and subjected to a denoising diffusion process. During this process, both the shape latent and latent points are updated to preserve fidelity, guiding the denoising toward generating consistent samples that align more closely with the source domain. We conduct extensive experiments on the ShapeNet dataset and investigate its generalizability on ModelNet40 and ScanObjectNN, achieving state-of-the-art results. The code has been released at \url{this https URL}.

Title: Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning

• Authors: Hang Zhou, Yehui Tang, Haochen Qin, Yujie Yang, Renren Jin, Deyi Xiong, Kai Han, Yunhe Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14497
• Pdf URL: https://arxiv.org/pdf/2411.14497
• Copy Paste: [[2411.14497]] Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning(https://arxiv.org/abs/2411.14497)
• Keywords: large language model
• Abstract: The efficacy of large language models (LLMs) on downstream tasks usually hinges on instruction tuning, which relies critically on the quality of training data. Unfortunately, collecting high-quality and diverse data is both expensive and time-consuming. To mitigate this issue, we propose a novel Star-Agents framework, which automates the enhancement of data quality across datasets through multi-agent collaboration and assessment. The framework adopts a three-pronged strategy. It initially generates diverse instruction data with multiple LLM agents through a bespoke sampling method. Subsequently, the generated data undergo a rigorous evaluation using a dual-model method that assesses both difficulty and quality. Finaly, the above process evolves in a dynamic refinement phase, where more effective LLMs are prioritized, enhancing the overall data quality. Our empirical studies, including instruction tuning experiments with models such as Pythia and LLaMA, demonstrate the effectiveness of the proposed framework. Optimized datasets have achieved substantial improvements, with an average increase of 12% and notable gains in specific metrics, such as a 40% improvement in Fermi, as evidenced by benchmarks like MT-bench, Vicuna bench, and WizardLM testset.

Title: Understanding World or Predicting Future? A Comprehensive Survey of World Models

• Authors: Jingtao Ding, Yunke Zhang, Yu Shang, Yuheng Zhang, Zefang Zong, Jie Feng, Yuan Yuan, Hongyuan Su, Nian Li, Nicholas Sukiennik, Fengli Xu, Yong Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14499
• Pdf URL: https://arxiv.org/pdf/2411.14499
• Copy Paste: [[2411.14499]] Understanding World or Predicting Future? A Comprehensive Survey of World Models(https://arxiv.org/abs/2411.14499)
• Keywords: large language model
• Abstract: The concept of world models has garnered significant attention due to advancements in multimodal large language models such as GPT-4 and video generation models such as Sora, which are central to the pursuit of artificial general intelligence. This survey offers a comprehensive review of the literature on world models. Generally, world models are regarded as tools for either understanding the present state of the world or predicting its future dynamics. This review presents a systematic categorization of world models, emphasizing two primary functions: (1) constructing internal representations to understand the mechanisms of the world, and (2) predicting future states to simulate and guide decision-making. Initially, we examine the current progress in these two categories. We then explore the application of world models in key domains, including autonomous driving, robotics, and social simulacra, with a focus on how each domain utilizes these aspects. Finally, we outline key challenges and provide insights into potential future research directions.

Title: Exploring Accuracy-Fairness Trade-off in Large Language Models

• Authors: Qingquan Zhang, Qiqi Duan, Bo Yuan, Yuhui Shi, Jialin Liu
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14500
• Pdf URL: https://arxiv.org/pdf/2411.14500
• Copy Paste: [[2411.14500]] Exploring Accuracy-Fairness Trade-off in Large Language Models(https://arxiv.org/abs/2411.14500)
• Keywords: fair, large language model
• Abstract: Large Language Models (LLMs) have made significant strides in the field of artificial intelligence, showcasing their ability to interact with humans and influence human cognition through information dissemination. However, recent studies have brought to light instances of bias inherent within these LLMs, presenting a critical issue that demands attention. In our research, we delve deeper into the intricate challenge of harmonising accuracy and fairness in the enhancement of LLMs. While improving accuracy can indeed enhance overall LLM performance, it often occurs at the expense of fairness. Overemphasising optimisation of one metric invariably leads to a significant degradation of the other. This underscores the necessity of taking into account multiple considerations during the design and optimisation phases of LLMs. Therefore, we advocate for reformulating the LLM training process as a multi-objective learning task. Our investigation reveals that multi-objective evolutionary learning (MOEL) methodologies offer promising avenues for tackling this challenge. Our MOEL framework enables the simultaneous optimisation of both accuracy and fairness metrics, resulting in a Pareto-optimal set of LLMs. In summary, our study sheds valuable lights on the delicate equilibrium between accuracy and fairness within LLMs, which is increasingly significant for their real-world applications. By harnessing MOEL, we present a promising pathway towards fairer and more efficacious AI technologies.

Title: Global Challenge for Safe and Secure LLMs Track 1

• Authors: Xiaojun Jia, Yihao Huang, Yang Liu, Peng Yan Tan, Weng Kuan Yau, Mun-Thye Mak, Xin Ming Sim, Wee Siong Ng, See Kiong Ng, Hanqing Liu, Lifeng Zhou, Huanqian Yan, Xiaobing Sun, Wei Liu, Long Wang, Yiming Qian, Yong Liu, Junxiao Yang, Zhexin Zhang, Leqi Lei, Renmiao Chen, Yida Lu, Shiyao Cui, Zizhou Wang, Shaohua Li, Yan Wang, Rick Siow Mong Goh, Liangli Zhen, Yingjie Zhang, Zhe Zhao
• Subjects: cs.CR, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.14502
• Pdf URL: https://arxiv.org/pdf/2411.14502
• Copy Paste: [[2411.14502]] Global Challenge for Safe and Secure LLMs Track 1(https://arxiv.org/abs/2411.14502)
• Keywords: secure, security, defense, attack, robust, large language model
• Abstract: This paper introduces the Global Challenge for Safe and Secure Large Language Models (LLMs), a pioneering initiative organized by AI Singapore (AISG) and the CyberSG R&D Programme Office (CRPO) to foster the development of advanced defense mechanisms against automated jailbreaking attacks. With the increasing integration of LLMs in critical sectors such as healthcare, finance, and public administration, ensuring these models are resilient to adversarial attacks is vital for preventing misuse and upholding ethical standards. This competition focused on two distinct tracks designed to evaluate and enhance the robustness of LLM security frameworks. Track 1 tasked participants with developing automated methods to probe LLM vulnerabilities by eliciting undesirable responses, effectively testing the limits of existing safety protocols within LLMs. Participants were challenged to devise techniques that could bypass content safeguards across a diverse array of scenarios, from offensive language to misinformation and illegal activities. Through this process, Track 1 aimed to deepen the understanding of LLM vulnerabilities and provide insights for creating more resilient models.

Title: LLaVA-MR: Large Language-and-Vision Assistant for Video Moment Retrieval

• Authors: Weiheng Lu, Jian Li, An Yu, Ming-Ching Chang, Shengpeng Ji, Min Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14505
• Pdf URL: https://arxiv.org/pdf/2411.14505
• Copy Paste: [[2411.14505]] LLaVA-MR: Large Language-and-Vision Assistant for Video Moment Retrieval(https://arxiv.org/abs/2411.14505)
• Keywords: extraction, large language model
• Abstract: Multimodal Large Language Models (MLLMs) are widely used for visual perception, understanding, and reasoning. However, long video processing and precise moment retrieval remain challenging due to LLMs' limited context size and coarse frame extraction. We propose the Large Language-and-Vision Assistant for Moment Retrieval (LLaVA-MR), which enables accurate moment retrieval and contextual grounding in videos using MLLMs. LLaVA-MR combines Dense Frame and Time Encoding (DFTE) for spatial-temporal feature extraction, Informative Frame Selection (IFS) for capturing brief visual and motion patterns, and Dynamic Token Compression (DTC) to manage LLM context limitations. Evaluations on benchmarks like Charades-STA and QVHighlights demonstrate that LLaVA-MR outperforms 11 state-of-the-art methods, achieving an improvement of 1.82% in R1@0.5 and 1.29% in mAP@0.5 on the QVHighlights dataset. Our implementation will be open-sourced upon acceptance.

Title: FuseGPT: Learnable Layers Fusion of Generative Pre-trained Transformers

• Authors: Zehua Pei, Hui-Ling Zhen, Xianzhi Yu, Sinno Jialin Pan, Mingxuan Yuan, Bei Yu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14507
• Pdf URL: https://arxiv.org/pdf/2411.14507
• Copy Paste: [[2411.14507]] FuseGPT: Learnable Layers Fusion of Generative Pre-trained Transformers(https://arxiv.org/abs/2411.14507)
• Keywords: transformer, generative, large language model
• Abstract: Generative Pre-trained Transformers (GPTs) have demonstrated remarkable performance across diverse domains through the extensive scaling of model parameters. Recent works observe the redundancy across the transformer blocks and develop compression methods by structured pruning of the unimportant blocks. However, such straightforward elimination will always provide irreversible performance degradation. In this paper, we propose FuseGPT, a novel methodology to recycle the pruned transformer blocks to further recover the model performance. Firstly we introduce a new importance detection metric, Macro Influence (MI), to detect the long-term influence of each transformer block by calculating their loss of information after removal. Then we propose group-level layers fusion, which adopts the parameters in layers of the unimportant blocks and injects them into the corresponding layers inside the neighboring blocks. The fusion is not one-off but through iterative parameter updates by lightweight group-level fine-tuning. Specifically, these injected parameters are frozen but weighted with learnable rank decomposition matrices to reduce the overhead during fine-tuning. Our approach not only works well on large language models but also on large multimodal models. The experiments have shown that, by using modest amounts of data, FuseGPT can outperform previous works in both perplexity and zero-shot task performance.

Title: Variational Autoencoders for Efficient Simulation-Based Inference

• Authors: Mayank Nautiyal, Andrey Shternshis, Andreas Hellander, Prashant Singh
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14511
• Pdf URL: https://arxiv.org/pdf/2411.14511
• Copy Paste: [[2411.14511]] Variational Autoencoders for Efficient Simulation-Based Inference(https://arxiv.org/abs/2411.14511)
• Keywords: generative
• Abstract: We present a generative modeling approach based on the variational inference framework for likelihood-free simulation-based inference. The method leverages latent variables within variational autoencoders to efficiently estimate complex posterior distributions arising from stochastic simulations. We explore two variations of this approach distinguished by their treatment of the prior distribution. The first model adapts the prior based on observed data using a multivariate prior network, enhancing generalization across various posterior queries. In contrast, the second model utilizes a standard Gaussian prior, offering simplicity while still effectively capturing complex posterior distributions. We demonstrate the efficacy of these models on well-established benchmark problems, achieving results comparable to flow-based approaches while maintaining computational efficiency and scalability.

Title: Detecting Distributed Denial of Service Attacks Using Logistic Regression and SVM Methods

• Authors: Mohammad Arafat Ullah, Arthy Anjum, Rashedul Amin Tuhin, Shamim Akhter
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14512
• Pdf URL: https://arxiv.org/pdf/2411.14512
• Copy Paste: [[2411.14512]] Detecting Distributed Denial of Service Attacks Using Logistic Regression and SVM Methods(https://arxiv.org/abs/2411.14512)
• Keywords: security, attack
• Abstract: A distributed denial-of-service (DDoS) attack is an attempt to produce humongous traffic within a network by overwhelming a targeted server or its neighboring infrastructure with a flood of service requests ceaselessly coming from multiple remotely controlled malware-infected computers or network-connected devices. Thus, exploring DDoS attacks by recognizing their functionalities and differentiating them from normal traffic services are the primary concerns of network security issues particularly for online businesses. In modern networks, most DDoS attacks occur in the network and application layer including HTTP flood, UDP flood, SIDDOS, SMURF, SNMP flood, IP NULL, etc. The goal of this paper is to detect DDoS attacks from all service requests and classify them according to DDoS classes. In this regard, a standard dataset is collected from the internet which contains several network-related attributes and their corresponding DDoS attack class name. Two(2) different machine learning approaches, SVM and Logistic Regression, are implemented in the dataset for detecting and classifying DDoS attacks, and a comparative study is accomplished among them in terms of accuracy, precision, and recall rates. Logistic Regression and SVM both achieve 98.65% classification accuracy which is the highest achieved accuracy among other previous experiments with the same dataset.

Title: Are Anomaly Scores Telling the Whole Story? A Benchmark for Multilevel Anomaly Detection

• Authors: Tri Cao, Minh-Huy Trinh, Ailin Deng, Quoc-Nam Nguyen, Khoa Duong, Ngai-Man Cheung, Bryan Hooi
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14515
• Pdf URL: https://arxiv.org/pdf/2411.14515
• Copy Paste: [[2411.14515]] Are Anomaly Scores Telling the Whole Story? A Benchmark for Multilevel Anomaly Detection(https://arxiv.org/abs/2411.14515)
• Keywords: robust
• Abstract: Anomaly detection (AD) is a machine learning task that identifies anomalies by learning patterns from normal training data. In many real-world scenarios, anomalies vary in severity, from minor anomalies with little risk to severe abnormalities requiring immediate attention. However, existing models primarily operate in a binary setting, and the anomaly scores they produce are usually based on the deviation of data points from normal data, which may not accurately reflect practical severity. In this paper, we address this gap by making three key contributions. First, we propose a novel setting, Multilevel AD (MAD), in which the anomaly score represents the severity of anomalies in real-world applications, and we highlight its diverse applications across various domains. Second, we introduce a novel benchmark, MAD-Bench, that evaluates models not only on their ability to detect anomalies, but also on how effectively their anomaly scores reflect severity. This benchmark incorporates multiple types of baselines and real-world applications involving severity. Finally, we conduct a comprehensive performance analysis on MAD-Bench. We evaluate models on their ability to assign severity-aligned scores, investigate the correspondence between their performance on binary and multilevel detection, and study their robustness. This analysis offers key insights into improving AD models for practical severity alignment. The code framework and datasets used for the benchmark will be made publicly available.

Title: Memory Backdoor Attacks on Neural Networks

• Authors: Eden Luzon, Guy Amit, Roy Weiss, Yisroel Mirsky
• Subjects: cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14516
• Pdf URL: https://arxiv.org/pdf/2411.14516
• Copy Paste: [[2411.14516]] Memory Backdoor Attacks on Neural Networks(https://arxiv.org/abs/2411.14516)
• Keywords: secure, attack, extraction, federate, large language model, segmentation
• Abstract: Neural networks, such as image classifiers, are frequently trained on proprietary and confidential datasets. It is generally assumed that once deployed, the training data remains secure, as adversaries are limited to query response interactions with the model, where at best, fragments of arbitrary data can be inferred without any guarantees on their authenticity. In this paper, we propose the memory backdoor attack, where a model is covertly trained to memorize specific training samples and later selectively output them when triggered with an index pattern. What makes this attack unique is that it (1) works even when the tasks conflict (making a classifier output images), (2) enables the systematic extraction of training samples from deployed models and (3) offers guarantees on the extracted authenticity of the data. We demonstrate the attack on image classifiers, segmentation models, and a large language model (LLM). We demonstrate the attack on image classifiers, segmentation models, and a large language model (LLM). With this attack, it is possible to hide thousands of images and texts in modern vision architectures and LLMs respectively, all while maintaining model performance. The memory back door attack poses a significant threat not only to conventional model deployments but also to federated learning paradigms and other modern frameworks. Therefore, we suggest an efficient and effective countermeasure that can be immediately applied and advocate for further work on the topic.

Title: The importance of the clustering model to detect new types of intrusion in data traffic

• Authors: Noor Saud Abd, Kamel Karoui
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14550
• Pdf URL: https://arxiv.org/pdf/2411.14550
• Copy Paste: [[2411.14550]] The importance of the clustering model to detect new types of intrusion in data traffic(https://arxiv.org/abs/2411.14550)
• Keywords: security, attack
• Abstract: In the current digital age, the volume of data generated by various cyber activities has become enormous and is constantly increasing. The data may contain valuable insights that can be harnessed to improve cyber security measures. However, much of this data is unclassified and qualitative, which poses significant challenges to traditional analysis methods. Clustering facilitates the identification of hidden patterns and structures in data through grouping similar data points, which makes it simpler to identify and address threats. Clustering can be defined as a data mining (DM) approach, which uses similarity calculations for dividing a data set into several categories. Hierarchical, density-based, along with partitioning clustering algorithms are typical. The presented work use K-means algorithm, which is a popular clustering technique. Utilizing K-means algorithm, we worked with two different types of data: first, we gathered data with the use of XG-boost algorithm following completing the aggregation with K-means algorithm. Data was gathered utilizing Kali Linux environment, cicflowmeter traffic, and Putty Software tools with the use of diverse and simple attacks. The concept could assist in identifying new attack types, which are distinct from the known attacks, and labeling them based on the characteristics they will exhibit, as the dynamic nature regarding cyber threats means that new attack types often emerge, for which labeled data might not yet exist. The model counted the attacks and assigned numbers to each one of them. Secondly, We tried the same work on the ready data inside the Kaggle repository called (Intrusion Detection in Internet of Things Network), and the clustering model worked well and detected the number of attacks correctly as shown in the results section.

Title: Privacy-Preserving Power Flow Analysis via Secure Multi-Party Computation

• Authors: Jonas von der Heyden, Nils Schlüter, Philipp Binfet, Martin Asman, Markus Zdrallek, Tibor Jager, Moritz Schulze Darup
• Subjects: cs.CR, eess.SY
• Abstract URL: https://arxiv.org/abs/2411.14557
• Pdf URL: https://arxiv.org/pdf/2411.14557
• Copy Paste: [[2411.14557]] Privacy-Preserving Power Flow Analysis via Secure Multi-Party Computation(https://arxiv.org/abs/2411.14557)
• Keywords: secure, security, privacy
• Abstract: Smart grids feature a bidirectional flow of electricity and data, enhancing flexibility, efficiency, and reliability in increasingly volatile energy grids. However, data from smart meters can reveal sensitive private information. Consequently, the adoption of smart meters is often restricted via legal means and hampered by limited user acceptance. Since metering data is beneficial for fault-free grid operation, power management, and resource allocation, applying privacy-preserving techniques to smart metering data is an important research problem. This work addresses this by using secure multi-party computation (SMPC), allowing multiple parties to jointly evaluate functions of their private inputs without revealing the latter. Concretely, we show how to perform power flow analysis on cryptographically hidden prosumer data. More precisely, we present a tailored solution to the power flow problem building on an SMPC implementation of Newtons method. We analyze the security of our approach in the universal composability framework and provide benchmarks for various grid types, threat models, and solvers. Our results indicate that secure multi-party computation can be able to alleviate privacy issues in smart grids in certain applications.

Title: Constructing Trustworthy Smart Contracts

• Authors: Devora Chait-Roth, Kedar S. Namjoshi
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2411.14563
• Pdf URL: https://arxiv.org/pdf/2411.14563
• Copy Paste: [[2411.14563]] Constructing Trustworthy Smart Contracts(https://arxiv.org/abs/2411.14563)
• Keywords: secure
• Abstract: Smart contracts form the core of Web3 applications. Contracts mediate the transfer of cryptocurrency, making them irresistible targets for hackers. We introduce ASP, a system aimed at easing the construction of provably secure contracts. The Asp system consists of three closely-linked components: a programming language, a defensive compiler, and a proof checker. The language semantics guarantee that Asp contracts are free of commonly exploited vulnerabilities such as arithmetic overflow and reentrancy. The defensive compiler enforces the semantics and translates Asp to Solidity, the most popular contract language. Deductive proofs establish functional correctness and freedom from critical vulnerabilities such as unauthorized access.

Title: Privacy-Preserving Video Anomaly Detection: A Survey

• Authors: Jing Liu, Yang Liu, Xiaoguang Zhu
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14565
• Pdf URL: https://arxiv.org/pdf/2411.14565
• Copy Paste: [[2411.14565]] Privacy-Preserving Video Anomaly Detection: A Survey(https://arxiv.org/abs/2411.14565)
• Keywords: privacy, interpretability
• Abstract: Video Anomaly Detection (VAD) aims to automatically analyze spatiotemporal patterns in surveillance videos collected from open spaces to detect anomalous events that may cause harm without physical contact. However, vision-based surveillance systems such as closed-circuit television often capture personally identifiable information. The lack of transparency and interpretability in video transmission and usage raises public concerns about privacy and ethics, limiting the real-world application of VAD. Recently, researchers have focused on privacy concerns in VAD by conducting systematic studies from various perspectives including data, features, and systems, making Privacy-Preserving Video Anomaly Detection (P2VAD) a hotspot in the AI community. However, current research in P2VAD is fragmented, and prior reviews have mostly focused on methods using RGB sequences, overlooking privacy leakage and appearance bias considerations. To address this gap, this article systematically reviews the progress of P2VAD for the first time, defining its scope and providing an intuitive taxonomy. We outline the basic assumptions, learning frameworks, and optimization objectives of various approaches, analyzing their strengths, weaknesses, and potential correlations. Additionally, we provide open access to research resources such as benchmark datasets and available code. Finally, we discuss key challenges and future opportunities from the perspectives of AI development and P2VAD deployment, aiming to guide future work in the field.

Title: Assessment of LLM Responses to End-user Security Questions

• Authors: Vijay Prakash, Kevin Lee, Arkaprabha Bhattacharya, Danny Yuxing Huang, Jessica Staddon
• Subjects: cs.CR, cs.AI, cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.14571
• Pdf URL: https://arxiv.org/pdf/2411.14571
• Copy Paste: [[2411.14571]] Assessment of LLM Responses to End-user Security Questions(https://arxiv.org/abs/2411.14571)
• Keywords: security, large language model
• Abstract: Answering end user security questions is challenging. While large language models (LLMs) like GPT, LLAMA, and Gemini are far from error-free, they have shown promise in answering a variety of questions outside of security. We studied LLM performance in the area of end user security by qualitatively evaluating 3 popular LLMs on 900 systematically collected end user security questions. While LLMs demonstrate broad generalist ``knowledge'' of end user security information, there are patterns of errors and limitations across LLMs consisting of stale and inaccurate answers, and indirect or unresponsive communication styles, all of which impacts the quality of information received. Based on these patterns, we suggest directions for model improvement and recommend user strategies for interacting with LLMs when seeking assistance with security.

Title: Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective

• Authors: Shenglai Zeng, Jiankun Zhang, Bingheng Li, Yuping Lin, Tianqi Zheng, Dante Everaert, Hanqing Lu, Hui Liu, Hui Liu, Yue Xing, Monica Xiao Cheng, Jiliang Tang
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14572
• Pdf URL: https://arxiv.org/pdf/2411.14572
• Copy Paste: [[2411.14572]] Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective(https://arxiv.org/abs/2411.14572)
• Keywords: large language model
• Abstract: Retrieval-Augmented Generation (RAG) systems have shown promise in enhancing the performance of Large Language Models (LLMs). However, these systems face challenges in effectively integrating external knowledge with the LLM's internal knowledge, often leading to issues with misleading or unhelpful information. This work aims to provide a systematic study on knowledge checking in RAG systems. We conduct a comprehensive analysis of LLM representation behaviors and demonstrate the significance of using representations in knowledge checking. Motivated by the findings, we further develop representation-based classifiers for knowledge filtering. We show substantial improvements in RAG performance, even when dealing with noisy knowledge databases. Our study provides new insights into leveraging LLM representations for enhancing the reliability and effectiveness of RAG systems.

Title: Efficient Spatio-Temporal Signal Recognition on Edge Devices Using PointLCA-Net

• Authors: Sanaz Mahmoodi Takaghaj, Jack Sampson
• Subjects: cs.LG, cs.ET
• Abstract URL: https://arxiv.org/abs/2411.14585
• Pdf URL: https://arxiv.org/pdf/2411.14585
• Copy Paste: [[2411.14585]] Efficient Spatio-Temporal Signal Recognition on Edge Devices Using PointLCA-Net(https://arxiv.org/abs/2411.14585)
• Keywords: extraction, segmentation
• Abstract: Recent advancements in machine learning, particularly through deep learning architectures like PointNet, have transformed the processing of three-dimensional (3D) point clouds, significantly improving 3D object classification and segmentation tasks. While 3D point clouds provide detailed spatial information, spatio-temporal signals introduce a dynamic element that accounts for changes over time. However, applying deep learning techniques to spatio-temporal signals and deploying them on edge devices presents challenges, including real-time processing, memory capacity, and power consumption. To address these issues, this paper presents a novel approach that combines PointNet's feature extraction with the in-memory computing capabilities and energy efficiency of neuromorphic systems for spatio-temporal signal recognition. The proposed method consists of a two-stage process: in the first stage, PointNet extracts features from the spatio-temporal signals, which are then stored in non-volatile memristor crossbar arrays. In the second stage, these features are processed by a single-layer spiking neural encoder-decoder that employs the Locally Competitive Algorithm (LCA) for efficient encoding and classification. This work integrates the strengths of both PointNet and LCA, enhancing computational efficiency and energy performance on edge devices. PointLCA-Net achieves high recognition accuracy for spatio-temporal data with substantially lower energy burden during both inference and training than comparable approaches, thus advancing the deployment of advanced neural architectures in energy-constrained environments.

Title: Solving Zero-Shot 3D Visual Grounding as Constraint Satisfaction Problems

• Authors: Qihao Yuan, Jiaming Zhang, Kailai Li, Rainer Stiefelhagen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14594
• Pdf URL: https://arxiv.org/pdf/2411.14594
• Copy Paste: [[2411.14594]] Solving Zero-Shot 3D Visual Grounding as Constraint Satisfaction Problems(https://arxiv.org/abs/2411.14594)
• Keywords: large language model
• Abstract: 3D visual grounding (3DVG) aims to locate objects in a 3D scene with natural language descriptions. Supervised methods have achieved decent accuracy, but have a closed vocabulary and limited language understanding ability. Zero-shot methods mostly utilize large language models (LLMs) to handle natural language descriptions, yet suffer from slow inference speed. To address these problems, in this work, we propose a zero-shot method that reformulates the 3DVG task as a Constraint Satisfaction Problem (CSP), where the variables and constraints represent objects and their spatial relations, respectively. This allows a global reasoning of all relevant objects, producing grounding results of both the target and anchor objects. Moreover, we demonstrate the flexibility of our framework by handling negation- and counting-based queries with only minor extra coding efforts. Our system, Constraint Satisfaction Visual Grounding (CSVG), has been extensively evaluated on the public datasets ScanRefer and Nr3D datasets using only open-source LLMs. Results show the effectiveness of CSVG and superior grounding accuracy over current state-of-the-art zero-shot 3DVG methods with improvements of $+7.0\%$ (Acc@0.5 score) and $+11.2\%$ on the ScanRefer and Nr3D datasets, respectively. The code of our system is publicly available at this https URL.

Title: Exploiting Boosting in Hyperdimensional Computing for Enhanced Reliability in Healthcare

• Authors: SungHeon Jeong, Hamza Errahmouni Barkam, Sanggeon Yun, Yeseong Kim, Shaahin Angizi, Mohsen Imani
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14612
• Pdf URL: https://arxiv.org/pdf/2411.14612
• Copy Paste: [[2411.14612]] Exploiting Boosting in Hyperdimensional Computing for Enhanced Reliability in Healthcare(https://arxiv.org/abs/2411.14612)
• Keywords: robust
• Abstract: Hyperdimensional computing (HDC) enables efficient data encoding and processing in high-dimensional space, benefiting machine learning and data analysis. However, underutilization of these spaces can lead to overfitting and reduced model reliability, especially in data-limited systems a critical issue in sectors like healthcare that demand robustness and consistent performance. We introduce BoostHD, an approach that applies boosting algorithms to partition the hyperdimensional space into subspaces, creating an ensemble of weak learners. By integrating boosting with HDC, BoostHD enhances performance and reliability beyond existing HDC methods. Our analysis highlights the importance of efficient utilization of hyperdimensional spaces for improved model performance. Experiments on healthcare datasets show that BoostHD outperforms state-of-the-art methods. On the WESAD dataset, it achieved an accuracy of 98.37%, surpassing Random Forest, XGBoost, and OnlineHD. BoostHD also demonstrated superior inference efficiency and stability, maintaining high accuracy under data imbalance and noise. In person-specific evaluations, it achieved an average accuracy of 96.19%, outperforming other models. By addressing the limitations of both boosting and HDC, BoostHD expands the applicability of HDC in critical domains where reliability and precision are paramount.

Title: Initial Evidence of Elevated Reconnaissance Attacks Against Nodes in P2P Overlay Networks

• Authors: Scott Seidenberger, Anindya Maiti
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.14623
• Pdf URL: https://arxiv.org/pdf/2411.14623
• Copy Paste: [[2411.14623]] Initial Evidence of Elevated Reconnaissance Attacks Against Nodes in P2P Overlay Networks(https://arxiv.org/abs/2411.14623)
• Keywords: security, attack
• Abstract: We hypothesize that peer-to-peer (P2P) overlay network nodes can be attractive to attackers due to their visibility, sustained uptime, and resource potential. Towards validating this hypothesis, we investigate the state of active reconnaissance attacks on Ethereum P2P network nodes by deploying a series of honeypots alongside actual Ethereum nodes across globally distributed vantage points. We find that Ethereum nodes experience not only increased attacks, but also specific types of attacks targeting particular ports and services. Furthermore, we find evidence that the threat assessment on our nodes is applicable to the wider P2P network by having performed port scans on other reachable peers. Our findings provide insights into potential mitigation strategies to improve the security of the P2P networking layer.

Title: Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings

• Authors: Pura Peetathawatchai, Wei-Ning Chen, Berivan Isik, Sanmi Koyejo, Albert No
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14639
• Pdf URL: https://arxiv.org/pdf/2411.14639
• Copy Paste: [[2411.14639]] Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings(https://arxiv.org/abs/2411.14639)
• Keywords: privacy, protect, diffusion, generative
• Abstract: We introduce novel methods for adapting diffusion models under differential privacy (DP) constraints, enabling privacy-preserving style and content transfer without fine-tuning. Traditional approaches to private adaptation, such as DP-SGD, incur significant computational overhead and degrade model performance when applied to large, complex models. Our approach instead leverages embedding-based techniques: Universal Guidance and Textual Inversion (TI), adapted with differentially private mechanisms. We apply these methods to Stable Diffusion for style adaptation using two private datasets: a collection of artworks by a single artist and pictograms from the Paris 2024 Olympics. Experimental results show that the TI-based adaptation achieves superior fidelity in style transfer, even under strong privacy guarantees, while both methods maintain high privacy resilience by employing calibrated noise and subsampling strategies. Our findings demonstrate a feasible and efficient pathway for privacy-preserving diffusion model adaptation, balancing data protection with the fidelity of generated images, and offer insights into embedding-driven methods for DP in generative AI applications.

Title: VQalAttent: a Transparent Speech Generation Pipeline based on Transformer-learned VQ-VAE Latent Space

• Authors: Armani Rodriguez, Silvija Kokalj-Filipovic
• Subjects: cs.LG, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.14642
• Pdf URL: https://arxiv.org/pdf/2411.14642
• Copy Paste: [[2411.14642]] VQalAttent: a Transparent Speech Generation Pipeline based on Transformer-learned VQ-VAE Latent Space(https://arxiv.org/abs/2411.14642)
• Keywords: interpretability, transformer, generative
• Abstract: Generating high-quality speech efficiently remains a key challenge for generative models in speech synthesis. This paper introduces VQalAttent, a lightweight model designed to generate fake speech with tunable performance and interpretability. Leveraging the AudioMNIST dataset, consisting of human utterances of decimal digits (0-9), our method employs a two-step architecture: first, a scalable vector quantized autoencoder (VQ-VAE) that compresses audio spectrograms into discrete latent representations, and second, a decoder-only transformer that learns the probability model of these latents. Trained transformer generates similar latent sequences, convertible to audio spectrograms by the VQ-VAE decoder, from which we generate fake utterances. Interpreting statistical and perceptual quality of the fakes, depending on the dimension and the extrinsic information of the latent space, enables guided improvements in larger, commercial generative models. As a valuable tool for understanding and refining audio synthesis, our results demonstrate VQalAttent's capacity to generate intelligible speech samples with limited computational resources, while the modularity and transparency of the training pipeline helps easily correlate the analytics with modular modifications, hence providing insights for the more complex models.

Title: Comparative Analysis of Pooling Mechanisms in LLMs: A Sentiment Analysis Perspective

• Authors: Jinming Xing, Ruilin Xing, Yan Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14654
• Pdf URL: https://arxiv.org/pdf/2411.14654
• Copy Paste: [[2411.14654]] Comparative Analysis of Pooling Mechanisms in LLMs: A Sentiment Analysis Perspective(https://arxiv.org/abs/2411.14654)
• Keywords: transformer, large language model
• Abstract: Large Language Models (LLMs) have revolutionized natural language processing (NLP) by delivering state-of-the-art performance across a variety of tasks. Among these, Transformer-based models like BERT and GPT rely on pooling layers to aggregate token-level embeddings into sentence-level representations. Common pooling mechanisms such as Mean, Max, and Weighted Sum play a pivotal role in this aggregation process. Despite their widespread use, the comparative performance of these strategies on different LLM architectures remains underexplored. To address this gap, this paper investigates the effects of these pooling mechanisms on two prominent LLM families -- BERT and GPT, in the context of sentence-level sentiment analysis. Comprehensive experiments reveal that each pooling mechanism exhibits unique strengths and weaknesses depending on the task's specific requirements. Our findings underline the importance of selecting pooling methods tailored to the demands of particular applications, prompting a re-evaluation of common assumptions regarding pooling operations. By offering actionable insights, this study contributes to the optimization of LLM-based models for downstream tasks.

Title: Multiset Transformer: Advancing Representation Learning in Persistence Diagrams

• Authors: Minghua Wang, Ziyun Huang, Jinhui Xu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14662
• Pdf URL: https://arxiv.org/pdf/2411.14662
• Copy Paste: [[2411.14662]] Multiset Transformer: Advancing Representation Learning in Persistence Diagrams(https://arxiv.org/abs/2411.14662)
• Keywords: transformer
• Abstract: To improve persistence diagram representation learning, we propose Multiset Transformer. This is the first neural network that utilizes attention mechanisms specifically designed for multisets as inputs and offers rigorous theoretical guarantees of permutation invariance. The architecture integrates multiset-enhanced attentions with a pool-decomposition scheme, allowing multiplicities to be preserved across equivariant layers. This capability enables full leverage of multiplicities while significantly reducing both computational and spatial complexity compared to the Set Transformer. Additionally, our method can greatly benefit from clustering as a preprocessing step to further minimize complexity, an advantage not possessed by the Set Transformer. Experimental results demonstrate that the Multiset Transformer outperforms existing neural network methods in the realm of persistence diagram representation learning.

Title: Multiverse of Greatness: Generating Story Branches with LLMs

• Authors: Pittawat Taveekitworachai, Chollakorn Nimpattanavong, Mustafa Can Gursesli, Antonio Lanata, Andrea Guazzini, Ruck Thawonmas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14672
• Pdf URL: https://arxiv.org/pdf/2411.14672
• Copy Paste: [[2411.14672]] Multiverse of Greatness: Generating Story Branches with LLMs(https://arxiv.org/abs/2411.14672)
• Keywords: extraction
• Abstract: This paper presents Dynamic Context Prompting/Programming (DCP/P), a novel framework for interacting with LLMs to generate graph-based content with a dynamic context window history. While there is an existing study utilizing LLMs to generate a visual novel game, the previous study involved a manual process of output extraction and did not provide flexibility in generating a longer, coherent story. We evaluate DCP/P against our baseline, which does not provide context history to an LLM and only relies on the initial story data. Through objective evaluation, we show that simply providing the LLM with a summary leads to a subpar story compared to additionally providing the LLM with the proper context of the story. We also provide an extensive qualitative analysis and discussion. We qualitatively examine the quality of the objectively best-performing generated game from each approach. In addition, we examine biases in word choices and word sentiment of the generated content. We find a consistent observation with previous studies that LLMs are biased towards certain words, even with a different LLM family. Finally, we provide a comprehensive discussion on opportunities for future studies.

Title: Recursive Gaussian Process State Space Model

• Authors: Tengjie Zheng, Lin Cheng, Shengping Gong, Xu Huang
• Subjects: cs.LG, eess.SY, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.14679
• Pdf URL: https://arxiv.org/pdf/2411.14679
• Copy Paste: [[2411.14679]] Recursive Gaussian Process State Space Model(https://arxiv.org/abs/2411.14679)
• Keywords: interpretability
• Abstract: Learning dynamical models from data is not only fundamental but also holds great promise for advancing principle discovery, time-series prediction, and controller design. Among various approaches, Gaussian Process State-Space Models (GPSSMs) have recently gained significant attention due to their combination of flexibility and interpretability. However, for online learning, the field lacks an efficient method suitable for scenarios where prior information regarding data distribution and model function is limited. To address this issue, this paper proposes a recursive GPSSM method with adaptive capabilities for both operating domains and Gaussian process (GP) hyperparameters. Specifically, we first utilize first-order linearization to derive a Bayesian update equation for the joint distribution between the system state and the GP model, enabling closed-form and domain-independent learning. Second, an online selection algorithm for inducing points is developed based on informative criteria to achieve lightweight learning. Third, to support online hyperparameter optimization, we recover historical measurement information from the current filtering distribution. Comprehensive evaluations on both synthetic and real-world datasets demonstrate the superior accuracy, computational efficiency, and adaptability of our method compared to state-of-the-art online GPSSM techniques.

Title: Self-Supervised Learning for Ordered Three-Dimensional Structures

• Authors: Matthew Spellings, Maya Martirossyan, Julia Dshemuchadse
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14680
• Pdf URL: https://arxiv.org/pdf/2411.14680
• Copy Paste: [[2411.14680]] Self-Supervised Learning for Ordered Three-Dimensional Structures(https://arxiv.org/abs/2411.14680)
• Keywords: large language model
• Abstract: Recent work has proven that training large language models with self-supervised tasks and fine-tuning these models to complete new tasks in a transfer learning setting is a powerful idea, enabling the creation of models with many parameters, even with little labeled data; however, the number of domains that have harnessed these advancements has been limited. In this work, we formulate a set of geometric tasks suitable for the large-scale study of ordered three-dimensional structures, without requiring any human intervention in data labeling. We build deep rotation- and permutation-equivariant neural networks based on geometric algebra and use them to solve these tasks on both idealized and simulated three-dimensional structures. Quantifying order in complex-structured assemblies remains a long-standing challenge in materials physics; these models can elucidate the behavior of real self-assembling systems in a variety of ways, from distilling insights from learned tasks without further modification to solving new tasks with smaller amounts of labeled data via transfer learning.

Title: TrojanEdit: Backdooring Text-Based Image Editing Models

• Authors: Ji Guo, Peihong Chen, Wenbo Jiang, Guoming Lu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14681
• Pdf URL: https://arxiv.org/pdf/2411.14681
• Copy Paste: [[2411.14681]] TrojanEdit: Backdooring Text-Based Image Editing Models(https://arxiv.org/abs/2411.14681)
• Keywords: attack, diffusion
• Abstract: As diffusion models have achieved success in image generation tasks, many studies have extended them to other related fields like image editing. Unlike image generation, image editing aims to modify an image based on user requests while keeping other parts of the image unchanged. Among these, text-based image editing is the most representative this http URL studies have shown that diffusion models are vulnerable to backdoor attacks, where attackers may poison the training data to inject the backdoor into models. However, previous backdoor attacks on diffusion models primarily focus on image generation models without considering image editing models. Given that image editing models accept multimodal inputs, it raises a new question regarding the effectiveness of different modalities triggers in backdoor attacks on these models. To address this question, we propose a backdoor attack framework for image editing models, named TrojanEdit, which can handle different modalities triggers. We explore five types of visual triggers, three types of textual triggers, and combine them together as fifteen types of multimodal triggers, conducting extensive experiments for three types of backdoor attack goals. Our experimental results show that the image editing model has a backdoor bias for texture triggers. Compared to visual triggers, textual triggers have stronger attack effectiveness but also cause more damage to the model's normal functionality. Furthermore, we found that multimodal triggers can achieve a good balance between the attack effectiveness and model's normal functionality.

Title: EV-PINN: A Physics-Informed Neural Network for Predicting Electric Vehicle Dynamics

• Authors: Hansol Lim, Jee Won Lee, Jonathan Boyack, Jongseong Brad Choi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14691
• Pdf URL: https://arxiv.org/pdf/2411.14691
• Copy Paste: [[2411.14691]] EV-PINN: A Physics-Informed Neural Network for Predicting Electric Vehicle Dynamics(https://arxiv.org/abs/2411.14691)
• Keywords: generative
• Abstract: An onboard prediction of dynamic parameters (e.g. Aerodynamic drag, rolling resistance) enables accurate path planning for EVs. This paper presents EV-PINN, a Physics-Informed Neural Network approach in predicting instantaneous battery power and cumulative energy consumption during cruising while generalizing to the nonlinear dynamics of an EV. Our method learns real-world parameters such as motor efficiency, regenerative braking efficiency, vehicle mass, coefficient of aerodynamic drag, and coefficient of rolling resistance using automatic differentiation based on dynamics and ensures consistency with ground truth vehicle data. EV-PINN was validated using 15 and 35 minutes of in-situ battery log data from the Tesla Model 3 Long Range and Tesla Model S, respectively. With only vehicle speed and time as inputs, our model achieves high accuracy and generalization to dynamics, with validation losses of 0.002195 and 0.002292, respectively. This demonstrates EV-PINN's effectiveness in estimating parameters and predicting battery usage under actual driving conditions without the need for additional sensors.

Title: Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation

• Authors: Xunyu Zhu, Jian Li, Can Ma, Weiping Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14698
• Pdf URL: https://arxiv.org/pdf/2411.14698
• Copy Paste: [[2411.14698]] Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation(https://arxiv.org/abs/2411.14698)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) demonstrate exceptional reasoning capabilities, often achieving state-of-the-art performance in various tasks. However, their substantial computational and memory demands, due to billions of parameters, hinder deployment in resource-constrained environments. A promising solution is knowledge distillation, where LLMs transfer reasoning capabilities to Small Language Models (SLMs, $\le$ 1B parameters), enabling wider deployment on low-resource devices. Existing methods primarily focus on generating high-quality reasoning rationales for distillation datasets but often neglect the critical role of data quantity and quality. To address these challenges, we propose a Feedback-Driven Distillation (FDD) framework to enhance SLMs' mathematical reasoning capabilities. In the initialization stage, a distillation dataset is constructed by prompting LLMs to pair mathematical problems with corresponding reasoning rationales. We classify problems into easy and hard categories based on SLM performance. For easy problems, LLMs generate more complex variations, while for hard problems, new questions of similar complexity are synthesized. In addition, we propose a multi-round distillation paradigm to iteratively enrich the distillation datasets, thereby progressively improving the mathematical reasoning abilities of SLMs. Experimental results demonstrate that our method can make SLMs achieve SOTA mathematical reasoning performance.

Title: Cross-Modal Pre-Aligned Method with Global and Local Information for Remote-Sensing Image and Text Retrieval

• Authors: Zengbao Sun, Ming Zhao, Gaorui Liu, André Kaup
• Subjects: cs.CV, cs.IR
• Abstract URL: https://arxiv.org/abs/2411.14704
• Pdf URL: https://arxiv.org/pdf/2411.14704
• Copy Paste: [[2411.14704]] Cross-Modal Pre-Aligned Method with Global and Local Information for Remote-Sensing Image and Text Retrieval(https://arxiv.org/abs/2411.14704)
• Keywords: transformer
• Abstract: Remote sensing cross-modal text-image retrieval (RSCTIR) has gained attention for its utility in information mining. However, challenges remain in effectively integrating global and local information due to variations in remote sensing imagery and ensuring proper feature pre-alignment before modal fusion, which affects retrieval accuracy and efficiency. To address these issues, we propose CMPAGL, a cross-modal pre-aligned method leveraging global and local information. Our Gswin transformer block combines local window self-attention and global-local window cross-attention to capture multi-scale features. A pre-alignment mechanism simplifies modal fusion training, improving retrieval performance. Additionally, we introduce a similarity matrix reweighting (SMR) algorithm for reranking, and enhance the triplet loss function with an intra-class distance term to optimize feature learning. Experiments on four datasets, including RSICD and RSITMD, validate CMPAGL's effectiveness, achieving up to 4.65% improvement in R@1 and 2.28% in mean Recall (mR) over state-of-the-art methods.

Title: Understanding LLM Embeddings for Regression

• Authors: Eric Tang, Bangding Yang, Xingyou Song
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14708
• Pdf URL: https://arxiv.org/pdf/2411.14708
• Copy Paste: [[2411.14708]] Understanding LLM Embeddings for Regression(https://arxiv.org/abs/2411.14708)
• Keywords: large language model
• Abstract: With the rise of large language models (LLMs) for flexibly processing information as strings, a natural application is regression, specifically by preprocessing string representations into LLM embeddings as downstream features for metric prediction. In this paper, we provide one of the first comprehensive investigations into embedding-based regression and demonstrate that LLM embeddings as features can be better for high-dimensional regression tasks than using traditional feature engineering. This regression performance can be explained in part due to LLM embeddings over numeric data inherently preserving Lipschitz continuity over the feature space. Furthermore, we quantify the contribution of different model effects, most notably model size and language understanding, which we find surprisingly do not always improve regression performance.

Title: Any-to-3D Generation via Hybrid Diffusion Supervision

• Authors: Yijun Fan, Yiwei Ma, Jiayi Ji, Xiaoshuai Sun, Rongrong Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14715
• Pdf URL: https://arxiv.org/pdf/2411.14715
• Copy Paste: [[2411.14715]] Any-to-3D Generation via Hybrid Diffusion Supervision(https://arxiv.org/abs/2411.14715)
• Keywords: diffusion
• Abstract: Recent progress in 3D object generation has been fueled by the strong priors offered by diffusion models. However, existing models are tailored to specific tasks, accommodating only one modality at a time and necessitating retraining to change modalities. Given an image-to-3D model and a text prompt, a naive approach is to convert text prompts to images and then use the image-to-3D model for generation. This approach is both time-consuming and labor-intensive, resulting in unavoidable information loss during modality conversion. To address this, we introduce XBind, a unified framework for any-to-3D generation using cross-modal pre-alignment techniques. XBind integrates an multimodal-aligned encoder with pre-trained diffusion models to generate 3D objects from any modalities, including text, images, and audio. We subsequently present a novel loss function, termed Modality Similarity (MS) Loss, which aligns the embeddings of the modality prompts and the rendered images, facilitating improved alignment of the 3D objects with multiple modalities. Additionally, Hybrid Diffusion Supervision combined with a Three-Phase Optimization process improves the quality of the generated 3D objects. Extensive experiments showcase XBind's broad generation capabilities in any-to-3D scenarios. To our knowledge, this is the first method to generate 3D objects from any modality prompts. Project page: this https URL.

Title: VisionPAD: A Vision-Centric Pre-training Paradigm for Autonomous Driving

• Authors: Haiming Zhang, Wending Zhou, Yiyao Zhu, Xu Yan, Jiantao Gao, Dongfeng Bai, Yingjie Cai, Bingbing Liu, Shuguang Cui, Zhen Li
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.14716
• Pdf URL: https://arxiv.org/pdf/2411.14716
• Copy Paste: [[2411.14716]] VisionPAD: A Vision-Centric Pre-training Paradigm for Autonomous Driving(https://arxiv.org/abs/2411.14716)
• Keywords: segmentation
• Abstract: This paper introduces VisionPAD, a novel self-supervised pre-training paradigm designed for vision-centric algorithms in autonomous driving. In contrast to previous approaches that employ neural rendering with explicit depth supervision, VisionPAD utilizes more efficient 3D Gaussian Splatting to reconstruct multi-view representations using only images as supervision. Specifically, we introduce a self-supervised method for voxel velocity estimation. By warping voxels to adjacent frames and supervising the rendered outputs, the model effectively learns motion cues in the sequential data. Furthermore, we adopt a multi-frame photometric consistency approach to enhance geometric perception. It projects adjacent frames to the current frame based on rendered depths and relative poses, boosting the 3D geometric representation through pure image supervision. Extensive experiments on autonomous driving datasets demonstrate that VisionPAD significantly improves performance in 3D object detection, occupancy prediction and map segmentation, surpassing state-of-the-art pre-training strategies by a considerable margin.

Title: FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data

• Authors: Binqian Xu, Xiangbo Shu, Haiyang Mei, Guosen Xie, Basura Fernando, Mike Zheng Shou, Jinhui Tang
• Subjects: cs.LG, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14717
• Pdf URL: https://arxiv.org/pdf/2411.14717
• Copy Paste: [[2411.14717]] FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data(https://arxiv.org/abs/2411.14717)
• Keywords: privacy, federate, large language model
• Abstract: Multimodal Large Language Models (MLLMs) have made significant advancements, demonstrating powerful capabilities in processing and understanding multimodal data. Fine-tuning MLLMs with Federated Learning (FL) allows for expanding the training data scope by including private data sources, thereby enhancing their practical applicability in privacy-sensitive domains. However, current research remains in the early stage, particularly in addressing the \textbf{multimodal heterogeneities} in real-world applications. In this paper, we introduce a benchmark for evaluating various downstream tasks in the federated fine-tuning of MLLMs within multimodal heterogeneous scenarios, laying the groundwork for the research in the field. Our benchmark encompasses two datasets, five comparison baselines, and four multimodal scenarios, incorporating over ten types of modal heterogeneities. To address the challenges posed by modal heterogeneity, we develop a general FedMLLM framework that integrates four representative FL methods alongside two modality-agnostic strategies. Extensive experimental results show that our proposed FL paradigm improves the performance of MLLMs by broadening the range of training data and mitigating multimodal heterogeneity. Code is available at this https URL

Title: GraphTheft: Quantifying Privacy Risks in Graph Prompt Learning

• Authors: Jiani Zhu, Xi Lin, Yuxin Qi, Qinghua Mao
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14718
• Pdf URL: https://arxiv.org/pdf/2411.14718
• Copy Paste: [[2411.14718]] GraphTheft: Quantifying Privacy Risks in Graph Prompt Learning(https://arxiv.org/abs/2411.14718)
• Keywords: privacy, protect, defense, attack
• Abstract: Graph Prompt Learning (GPL) represents an innovative approach in graph representation learning, enabling task-specific adaptations by fine-tuning prompts without altering the underlying pre-trained model. Despite its growing prominence, the privacy risks inherent in GPL remain unexplored. In this study, we provide the first evaluation of privacy leakage in GPL across three attacker capabilities: black-box attacks when GPL as a service, and scenarios where node embeddings and prompt representations are accessible to third parties. We assess GPL's privacy vulnerabilities through Attribute Inference Attacks (AIAs) and Link Inference Attacks (LIAs), finding that under any capability, attackers can effectively infer the properties and relationships of sensitive nodes, and the success rate of inference on some data sets is as high as 98%. Importantly, while targeted inference attacks on specific prompts (e.g., GPF-plus) maintain high success rates, our analysis suggests that the prompt-tuning in GPL does not significantly elevate privacy risks compared to traditional GNNs. To mitigate these risks, we explored defense mechanisms, identifying that Laplacian noise perturbation can substantially reduce inference success, though balancing privacy protection with model performance remains challenging. This work highlights critical privacy risks in GPL, offering new insights and foundational directions for future privacy-preserving strategies in graph learning.

Title: Optimizing Social Media Annotation of HPV Vaccine Skepticism and Misinformation Using Large Language Models: An Experimental Evaluation of In-Context Learning and Fine-Tuning Stance Detection Across Multiple Models

• Authors: Luhang Sun, Varsha Pendyala, Yun-Shiuan Chuang, Shanglin Yang, Jonathan Feldman, Andrew Zhao, Munmun De Choudhury, Sijia Yang, Dhavan Shah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14720
• Pdf URL: https://arxiv.org/pdf/2411.14720
• Copy Paste: [[2411.14720]] Optimizing Social Media Annotation of HPV Vaccine Skepticism and Misinformation Using Large Language Models: An Experimental Evaluation of In-Context Learning and Fine-Tuning Stance Detection Across Multiple Models(https://arxiv.org/abs/2411.14720)
• Keywords: large language model
• Abstract: This paper leverages large-language models (LLMs) to experimentally determine optimal strategies for scaling up social media content annotation for stance detection on HPV vaccine-related tweets. We examine both conventional fine-tuning and emergent in-context learning methods, systematically varying strategies of prompt engineering across widely used LLMs and their variants (e.g., GPT4, Mistral, and Llama3, etc.). Specifically, we varied prompt template design, shot sampling methods, and shot quantity to detect stance on HPV vaccination. Our findings reveal that 1) in general, in-context learning outperforms fine-tuning in stance detection for HPV vaccine social media content; 2) increasing shot quantity does not necessarily enhance performance across models; and 3) different LLMs and their variants present differing sensitivity to in-context learning conditions. We uncovered that the optimal in-context learning configuration for stance detection on HPV vaccine tweets involves six stratified shots paired with detailed contextual prompts. This study highlights the potential and provides an applicable approach for applying LLMs to research on social media stance and skepticism detection.

Title: MolReFlect: Towards In-Context Fine-grained Alignments between Molecules and Texts

• Authors: Jiatong Li, Yunqing Liu, Wei Liu, Jingdi Le, Di Zhang, Wenqi Fan, Dongzhan Zhou, Yuqiang Li, Qing Li
• Subjects: cs.CL, cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2411.14721
• Pdf URL: https://arxiv.org/pdf/2411.14721
• Copy Paste: [[2411.14721]] MolReFlect: Towards In-Context Fine-grained Alignments between Molecules and Texts(https://arxiv.org/abs/2411.14721)
• Keywords: extraction, generative, large language model
• Abstract: Molecule discovery is a pivotal research field, impacting everything from the medicines we take to the materials we use. Recently, Large Language Models (LLMs) have been widely adopted in molecule understanding and generation, yet the alignments between molecules and their corresponding captions remain a significant challenge. Previous endeavours often treat the molecule as a general SMILES string or molecular graph, neglecting the fine-grained alignments between the molecular sub-structures and the descriptive textual phrases, which are crucial for accurate and explainable predictions. In this case, we introduce MolReFlect, a novel teacher-student framework designed to contextually perform the molecule-caption alignments in a fine-grained way. Our approach initially leverages a larger teacher LLM to label the detailed alignments by directly extracting critical phrases from molecule captions or SMILES strings and implying them to corresponding sub-structures or characteristics. To refine these alignments, we propose In-Context Selective Reflection, which retrieves previous extraction results as context examples for teacher LLM to reflect and lets a smaller student LLM select from in-context reflection and previous extraction results. Finally, we enhance the learning process of the student LLM through Chain-of-Thought In-Context Molecule Tuning, integrating the fine-grained alignments and the reasoning processes within the Chain-of-Thought format. Our experimental results demonstrate that MolReFlect enables LLMs like Mistral-7B to significantly outperform the previous baselines, achieving SOTA performance on the ChEBI-20 dataset. This advancement not only enhances the generative capabilities of LLMs in the molecule-caption translation task, but also contributes to a more explainable framework.

Title: Effective SAM Combination for Open-Vocabulary Semantic Segmentation

• Authors: Minhyeok Lee, Suhwan Cho, Jungho Lee, Sunghun Yang, Heeseung Choi, Ig-Jae Kim, Sangyoun Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14723
• Pdf URL: https://arxiv.org/pdf/2411.14723
• Copy Paste: [[2411.14723]] Effective SAM Combination for Open-Vocabulary Semantic Segmentation(https://arxiv.org/abs/2411.14723)
• Keywords: robust, segmentation
• Abstract: Open-vocabulary semantic segmentation aims to assign pixel-level labels to images across an unlimited range of classes. Traditional methods address this by sequentially connecting a powerful mask proposal generator, such as the Segment Anything Model (SAM), with a pre-trained vision-language model like CLIP. But these two-stage approaches often suffer from high computational costs, memory inefficiencies. In this paper, we propose ESC-Net, a novel one-stage open-vocabulary segmentation model that leverages the SAM decoder blocks for class-agnostic segmentation within an efficient inference framework. By embedding pseudo prompts generated from image-text correlations into SAM's promptable segmentation framework, ESC-Net achieves refined spatial aggregation for accurate mask predictions. ESC-Net achieves superior performance on standard benchmarks, including ADE20K, PASCAL-VOC, and PASCAL-Context, outperforming prior methods in both efficiency and accuracy. Comprehensive ablation studies further demonstrate its robustness across challenging conditions.

Title: Evaluating and Advancing Multimodal Large Language Models in Ability Lens

• Authors: Feng Chen, Chenhui Gou, Jing Liu, Yang Yang, Zhaoyang Li, Jiyuan Zhang, Zhenbang Sun, Bohan Zhuang, Qi Wu
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14725
• Pdf URL: https://arxiv.org/pdf/2411.14725
• Copy Paste: [[2411.14725]] Evaluating and Advancing Multimodal Large Language Models in Ability Lens(https://arxiv.org/abs/2411.14725)
• Keywords: robust, large language model
• Abstract: As multimodal large language models (MLLMs) advance rapidly, rigorous evaluation has become essential, providing further guidance for their development. In this work, we focus on a unified and robust evaluation of \textbf{vision perception} abilities, the foundational skill of MLLMs. We find that existing perception benchmarks, each focusing on different question types, domains, and evaluation metrics, introduce significant evaluation variance, complicating comprehensive assessments of perception abilities when relying on any single benchmark. To address this, we introduce \textbf{AbilityLens}, a unified benchmark designed to evaluate MLLMs across six key perception abilities, focusing on both accuracy and stability, with each ability encompassing diverse question types, domains, and metrics. With the assistance of AbilityLens, we: (1) identify the strengths and weaknesses of current models, highlighting stability patterns and revealing a notable performance gap between open-source and closed-source models; (2) introduce an online evaluation mode, which uncovers interesting ability conflict and early convergence phenomena during MLLM training; and (3) design a simple ability-specific model merging method that combines the best ability checkpoint from early training stages, effectively mitigating performance decline due to ability conflict. The benchmark and online leaderboard will be released soon.

Title: A Lightweight Edge-CNN-Transformer Model for Detecting Coordinated Cyber and Digital Twin Attacks in Cooperative Smart Farming

• Authors: Lopamudra Praharaj, Deepti Gupta, Maanak Gupta
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14729
• Pdf URL: https://arxiv.org/pdf/2411.14729
• Copy Paste: [[2411.14729]] A Lightweight Edge-CNN-Transformer Model for Detecting Coordinated Cyber and Digital Twin Attacks in Cooperative Smart Farming(https://arxiv.org/abs/2411.14729)
• Keywords: secure, security, privacy, attack, transformer
• Abstract: The agriculture sector is increasingly adopting innovative technologies to meet the growing food demands of the global population. To optimize resource utilization and minimize crop losses, farmers are joining cooperatives to share their data and resources among member farms. However, while farmers benefit from this data sharing and interconnection, it exposes them to cybersecurity threats and privacy concerns. A cyberattack on one farm can have widespread consequences, affecting the targeted farm as well as all member farms within a cooperative. In this research, we address existing gaps by proposing a novel and secure architecture for Cooperative Smart Farming (CSF). First, we highlight the role of edge-based DTs in enhancing the efficiency and resilience of agricultural operations. To validate this, we develop a test environment for CSF, implementing various cyberattacks on both the DTs and their physical counterparts using different attack vectors. We collect two smart farming network datasets to identify potential threats. After identifying these threats, we focus on preventing the transmission of malicious data from compromised farms to the central cloud server. To achieve this, we propose a CNN-Transformer-based network anomaly detection model, specifically designed for deployment at the edge. As a proof of concept, we implement this model and evaluate its performance by varying the number of encoder layers. Additionally, we apply Post-Quantization to compress the model and demonstrate the impact of compression on its performance in edge environments. Finally, we compare the model's performance with traditional machine learning approaches to assess its overall effectiveness.

Title: FLARE: FP-Less PTQ and Low-ENOB ADC Based AMS-PiM for Error-Resilient, Fast, and Efficient Transformer Acceleration

• Authors: Donghyeon Yi, Seoyoung Lee, Jongho Kim, Junyoung Kim, Sohmyung Ha, Ik Joon Chang, Minkyu Je
• Subjects: cs.LG, eess.IV, eess.SY
• Abstract URL: https://arxiv.org/abs/2411.14733
• Pdf URL: https://arxiv.org/pdf/2411.14733
• Copy Paste: [[2411.14733]] FLARE: FP-Less PTQ and Low-ENOB ADC Based AMS-PiM for Error-Resilient, Fast, and Efficient Transformer Acceleration(https://arxiv.org/abs/2411.14733)
• Keywords: transformer
• Abstract: Encoder-based transformers, powered by self-attention layers, have revolutionized machine learning with their context-aware representations. However, their quadratic growth in computational and memory demands presents significant bottlenecks. Analog-Mixed-Signal Process-in-Memory (AMS-PiM) architectures address these challenges by enabling efficient on-chip processing. Traditionally, AMS-PiM relies on Quantization-Aware Training (QAT), which is hardware-efficient but requires extensive retraining to adapt models to AMS-PiMs, making it increasingly impractical for transformer models. Post-Training Quantization (PTQ) mitigates this training overhead but introduces significant hardware inefficiencies. PTQ relies on dequantization-quantization (DQ-Q) processes, floating-point units (FPUs), and high-ENOB (Effective Number of Bits) analog-to-digital converters (ADCs). Particularly, High-ENOB ADCs scale exponentially in area and energy ($2^{ENOB}$), reduce sensing margins, and increase susceptibility to process, voltage, and temperature (PVT) variations, further compounding PTQ's challenges in AMS-PiM systems. To overcome these limitations, we propose RAP, an AMS-PiM architecture that eliminates DQ-Q processes, introduces FPU- and division-free nonlinear processing, and employs a low-ENOB-ADC-based sparse Matrix Vector multiplication technique. Using the proposed techniques, RAP improves error resiliency, area/energy efficiency, and computational speed while preserving numerical stability. Experimental results demonstrate that RAP outperforms state-of-the-art GPUs and conventional PiM architectures in energy efficiency, latency, and accuracy, making it a scalable solution for the efficient deployment of transformers.

Title: AI Tailoring: Evaluating Influence of Image Features on Fashion Product Popularity

• Authors: Xiaomin Li, Junyi Sha
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14737
• Pdf URL: https://arxiv.org/pdf/2411.14737
• Copy Paste: [[2411.14737]] AI Tailoring: Evaluating Influence of Image Features on Fashion Product Popularity(https://arxiv.org/abs/2411.14737)
• Keywords: robust, diffusion, transformer
• Abstract: Identifying key product features that influence consumer preferences is essential in the fashion industry. In this study, we introduce a robust methodology to ascertain the most impactful features in fashion product images, utilizing past market sales data. First, we propose the metric called "influence score" to quantitatively assess the importance of product features. Then we develop a forecasting model, the Fashion Demand Predictor (FDP), which integrates Transformer-based models and Random Forest to predict market popularity based on product images. We employ image-editing diffusion models to modify these images and perform an ablation study, which validates the impact of the highest and lowest-scoring features on the model's popularity predictions. Additionally, we further validate these results through surveys that gather human rankings of preferences, confirming the accuracy of the FDP model's predictions and the efficacy of our method in identifying influential features. Notably, products enhanced with "good" features show marked improvements in predicted popularity over their modified counterparts. Our approach develops a fully automated and systematic framework for fashion image analysis that provides valuable guidance for downstream tasks such as fashion product design and marketing strategy development.

Title: Universal and Context-Independent Triggers for Precise Control of LLM Outputs

• Authors: Jiashuo Liang, Guancheng Li, Yang Yu
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14738
• Pdf URL: https://arxiv.org/pdf/2411.14738
• Copy Paste: [[2411.14738]] Universal and Context-Independent Triggers for Precise Control of LLM Outputs(https://arxiv.org/abs/2411.14738)
• Keywords: attack, robust, large language model
• Abstract: Large language models (LLMs) have been widely adopted in applications such as automated content generation and even critical decision-making systems. However, the risk of prompt injection allows for potential manipulation of LLM outputs. While numerous attack methods have been documented, achieving full control over these outputs remains challenging, often requiring experienced attackers to make multiple attempts and depending heavily on the prompt context. Recent advancements in gradient-based white-box attack techniques have shown promise in tasks like jailbreaks and system prompt leaks. Our research generalizes gradient-based attacks to find a trigger that is (1) Universal: effective irrespective of the target output; (2) Context-Independent: robust across diverse prompt contexts; and (3) Precise Output: capable of manipulating LLM inputs to yield any specified output with high accuracy. We propose a novel method to efficiently discover such triggers and assess the effectiveness of the proposed attack. Furthermore, we discuss the substantial threats posed by such attacks to LLM-based applications, highlighting the potential for adversaries to taking over the decisions and actions made by AI agents.

Title: TEXGen: a Generative Diffusion Model for Mesh Textures

• Authors: Xin Yu, Ze Yuan, Yuan-Chen Guo, Ying-Tian Liu, JianHui Liu, Yangguang Li, Yan-Pei Cao, Ding Liang, Xiaojuan Qi
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.14740
• Pdf URL: https://arxiv.org/pdf/2411.14740
• Copy Paste: [[2411.14740]] TEXGen: a Generative Diffusion Model for Mesh Textures(https://arxiv.org/abs/2411.14740)
• Keywords: diffusion, generative
• Abstract: While high-quality texture maps are essential for realistic 3D asset rendering, few studies have explored learning directly in the texture space, especially on large-scale datasets. In this work, we depart from the conventional approach of relying on pre-trained 2D diffusion models for test-time optimization of 3D textures. Instead, we focus on the fundamental problem of learning in the UV texture space itself. For the first time, we train a large diffusion model capable of directly generating high-resolution texture maps in a feed-forward manner. To facilitate efficient learning in high-resolution UV spaces, we propose a scalable network architecture that interleaves convolutions on UV maps with attention layers on point clouds. Leveraging this architectural design, we train a 700 million parameter diffusion model that can generate UV texture maps guided by text prompts and single-view images. Once trained, our model naturally supports various extended applications, including text-guided texture inpainting, sparse-view texture completion, and text-driven texture synthesis. Project page is at this http URL.

Title: FOCUS: Knowledge-enhanced Adaptive Visual Compression for Few-shot Whole Slide Image Classification

• Authors: Zhengrui Guo, Conghao Xiong, Jiabo Ma, Qichen Sun, Lishuang Feng, Jinzhuo Wang, Hao Chen
• Subjects: cs.CV, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2411.14743
• Pdf URL: https://arxiv.org/pdf/2411.14743
• Copy Paste: [[2411.14743]] FOCUS: Knowledge-enhanced Adaptive Visual Compression for Few-shot Whole Slide Image Classification(https://arxiv.org/abs/2411.14743)
• Keywords: privacy, extraction
• Abstract: Few-shot learning presents a critical solution for cancer diagnosis in computational pathology (CPath), addressing fundamental limitations in data availability, particularly the scarcity of expert annotations and patient privacy constraints. A key challenge in this paradigm stems from the inherent disparity between the limited training set of whole slide images (WSIs) and the enormous number of contained patches, where a significant portion of these patches lacks diagnostically relevant information, potentially diluting the model's ability to learn and focus on critical diagnostic features. While recent works attempt to address this by incorporating additional knowledge, several crucial gaps hinder further progress: (1) despite the emergence of powerful pathology foundation models (FMs), their potential remains largely untapped, with most approaches limiting their use to basic feature extraction; (2) current language guidance mechanisms attempt to align text prompts with vast numbers of WSI patches all at once, struggling to leverage rich pathological semantic information. To this end, we introduce the knowledge-enhanced adaptive visual compression framework, dubbed FOCUS, which uniquely combines pathology FMs with language prior knowledge to enable a focused analysis of diagnostically relevant regions by prioritizing discriminative WSI patches. Our approach implements a progressive three-stage compression strategy: we first leverage FMs for global visual redundancy elimination, and integrate compressed features with language prompts for semantic relevance assessment, then perform neighbor-aware visual token filtering while preserving spatial coherence. Extensive experiments on pathological datasets spanning breast, lung, and ovarian cancers demonstrate its superior performance in few-shot pathology diagnosis. Code will be made available at this https URL.

Title: Point Cloud Understanding via Attention-Driven Contrastive Learning

• Authors: Yi Wang, Jiaze Wang, Ziyu Guo, Renrui Zhang, Donghao Zhou, Guangyong Chen, Anfeng Liu, Pheng-Ann Heng
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14744
• Pdf URL: https://arxiv.org/pdf/2411.14744
• Copy Paste: [[2411.14744]] Point Cloud Understanding via Attention-Driven Contrastive Learning(https://arxiv.org/abs/2411.14744)
• Keywords: robust, transformer, segmentation
• Abstract: Recently Transformer-based models have advanced point cloud understanding by leveraging self-attention mechanisms, however, these methods often overlook latent information in less prominent regions, leading to increased sensitivity to perturbations and limited global comprehension. To solve this issue, we introduce PointACL, an attention-driven contrastive learning framework designed to address these limitations. Our method employs an attention-driven dynamic masking strategy that guides the model to focus on under-attended regions, enhancing the understanding of global structures within the point cloud. Then we combine the original pre-training loss with a contrastive learning loss, improving feature discrimination and generalization. Extensive experiments validate the effectiveness of PointACL, as it achieves state-of-the-art performance across a variety of 3D understanding tasks, including object classification, part segmentation, and few-shot learning. Specifically, when integrated with different Transformer backbones like Point-MAE and PointGPT, PointACL demonstrates improved performance on datasets such as ScanObjectNN, ModelNet40, and ShapeNetPart. This highlights its superior capability in capturing both global and local features, as well as its enhanced robustness against perturbations and incomplete data.

Title: Ordinal Multiple-instance Learning for Ulcerative Colitis Severity Estimation with Selective Aggregated Transformer

• Authors: Kaito Shiku, Kazuya Nishimura, Daiki Suehiro, Kiyohito Tanaka, Ryoma Bise
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14750
• Pdf URL: https://arxiv.org/pdf/2411.14750
• Copy Paste: [[2411.14750]] Ordinal Multiple-instance Learning for Ulcerative Colitis Severity Estimation with Selective Aggregated Transformer(https://arxiv.org/abs/2411.14750)
• Keywords: transformer
• Abstract: Patient-level diagnosis of severity in ulcerative colitis (UC) is common in real clinical settings, where the most severe score in a patient is recorded. However, previous UC classification methods (i.e., image-level estimation) mainly assumed the input was a single image. Thus, these methods can not utilize severity labels recorded in real clinical settings. In this paper, we propose a patient-level severity estimation method by a transformer with selective aggregator tokens, where a severity label is estimated from multiple images taken from a patient, similar to a clinical setting. Our method can effectively aggregate features of severe parts from a set of images captured in each patient, and it facilitates improving the discriminative ability between adjacent severity classes. Experiments demonstrate the effectiveness of the proposed method on two datasets compared with the state-of-the-art MIL methods. Moreover, we evaluated our method in real clinical settings and confirmed that our method outperformed the previous image-level methods. The code is publicly available at this https URL.

Title: TopoSD: Topology-Enhanced Lane Segment Perception with SDMap Prior

• Authors: Sen Yang, Minyue Jiang, Ziwei Fan, Xiaolu Xie, Xiao Tan, Yingying Li, Errui Ding, Liang Wang, Jingdong Wang
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.14751
• Pdf URL: https://arxiv.org/pdf/2411.14751
• Copy Paste: [[2411.14751]] TopoSD: Topology-Enhanced Lane Segment Perception with SDMap Prior(https://arxiv.org/abs/2411.14751)
• Keywords: robust
• Abstract: Recent advances in autonomous driving systems have shifted towards reducing reliance on high-definition maps (HDMaps) due to the huge costs of annotation and maintenance. Instead, researchers are focusing on online vectorized HDMap construction using on-board sensors. However, sensor-only approaches still face challenges in long-range perception due to the restricted views imposed by the mounting angles of onboard cameras, just as human drivers also rely on bird's-eye-view navigation maps for a comprehensive understanding of road structures. To address these issues, we propose to train the perception model to "see" standard definition maps (SDMaps). We encode SDMap elements into neural spatial map representations and instance tokens, and then incorporate such complementary features as prior information to improve the bird's eye view (BEV) feature for lane geometry and topology decoding. Based on the lane segment representation framework, the model simultaneously predicts lanes, centrelines and their topology. To further enhance the ability of geometry prediction and topology reasoning, we also use a topology-guided decoder to refine the predictions by exploiting the mutual relationships between topological and geometric features. We perform extensive experiments on OpenLane-V2 datasets to validate the proposed method. The results show that our model outperforms state-of-the-art methods by a large margin, with gains of +6.7 and +9.1 on the mAP and topology metrics. Our analysis also reveals that models trained with SDMap noise augmentation exhibit enhanced robustness.

Title: FairAdapter: Detecting AI-generated Images with Improved Fairness

• Authors: Feng Ding, Jun Zhang, Xinan He, Jianfeng Xu
• Subjects: cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.14755
• Pdf URL: https://arxiv.org/pdf/2411.14755
• Copy Paste: [[2411.14755]] FairAdapter: Detecting AI-generated Images with Improved Fairness(https://arxiv.org/abs/2411.14755)
• Keywords: fair, generative
• Abstract: The high-quality, realistic images generated by generative models pose significant challenges for exposing this http URL far, data-driven deep neural networks have been justified as the most efficient forensics tools for the challenges. However, they may be over-fitted to certain semantics, resulting in considerable inconsistency in detection performance across different contents of generated samples. It could be regarded as an issue of detection fairness. In this paper, we propose a novel framework named Fairadapter to tackle the issue. In comparison with existing state-of-the-art methods, our model achieves improved fairness performance. Our project: this https URL

Title: Efficient Long Video Tokenization via Coordinated-based Patch Reconstruction

• Authors: Huiwon Jang, Sihyun Yu, Jinwoo Shin, Pieter Abbeel, Younggyo Seo
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14762
• Pdf URL: https://arxiv.org/pdf/2411.14762
• Copy Paste: [[2411.14762]] Efficient Long Video Tokenization via Coordinated-based Patch Reconstruction(https://arxiv.org/abs/2411.14762)
• Keywords: diffusion, transformer, generative
• Abstract: Efficient tokenization of videos remains a challenge in training vision models that can process long videos. One promising direction is to develop a tokenizer that can encode long video clips, as it would enable the tokenizer to leverage the temporal coherence of videos better for tokenization. However, training existing tokenizers on long videos often incurs a huge training cost as they are trained to reconstruct all the frames at once. In this paper, we introduce CoordTok, a video tokenizer that learns a mapping from coordinate-based representations to the corresponding patches of input videos, inspired by recent advances in 3D generative models. In particular, CoordTok encodes a video into factorized triplane representations and reconstructs patches that correspond to randomly sampled $(x,y,t)$ coordinates. This allows for training large tokenizer models directly on long videos without requiring excessive training resources. Our experiments show that CoordTok can drastically reduce the number of tokens for encoding long video clips. For instance, CoordTok can encode a 128-frame video with 128$\times$128 resolution into 1280 tokens, while baselines need 6144 or 8192 tokens to achieve similar reconstruction quality. We further show that this efficient video tokenization enables memory-efficient training of a diffusion transformer that can generate 128 frames at once.

Title: An Attention-based Framework for Fair Contrastive Learning

• Authors: Stefan K. Nielsen, Tan M. Nguyen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14765
• Pdf URL: https://arxiv.org/pdf/2411.14765
• Copy Paste: [[2411.14765]] An Attention-based Framework for Fair Contrastive Learning(https://arxiv.org/abs/2411.14765)
• Keywords: fair
• Abstract: Contrastive learning has proven instrumental in learning unbiased representations of data, especially in complex environments characterized by high-cardinality and high-dimensional sensitive information. However, existing approaches within this setting require predefined modelling assumptions of bias-causing interactions that limit the model's ability to learn debiased representations. In this work, we propose a new method for fair contrastive learning that employs an attention mechanism to model bias-causing interactions, enabling the learning of a fairer and semantically richer embedding space. In particular, our attention mechanism avoids bias-causing samples that confound the model and focuses on bias-reducing samples that help learn semantically meaningful representations. We verify the advantages of our method against existing baselines in fair contrastive learning and show that our approach can significantly boost bias removal from learned representations without compromising downstream accuracy.

Title: Resolution-Agnostic Transformer-based Climate Downscaling

• Authors: Declan Curran, Hira Saleem, Flora Salim, Sanaa Hobeichi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14774
• Pdf URL: https://arxiv.org/pdf/2411.14774
• Copy Paste: [[2411.14774]] Resolution-Agnostic Transformer-based Climate Downscaling(https://arxiv.org/abs/2411.14774)
• Keywords: transformer
• Abstract: Understanding future weather changes at regional and local scales is crucial for planning and decision-making, particularly in the context of extreme weather events, as well as for broader applications in agriculture, insurance, and infrastructure development. However, the computational cost of downscaling Global Climate Models (GCMs) to the fine resolutions needed for such applications presents a significant barrier. Drawing on advancements in weather forecasting models, this study introduces a cost-efficient downscaling method using a pretrained Earth Vision Transformer (Earth ViT) model. Initially trained on ERA5 data to downscale from 50 km to 25 km resolution, the model is then tested on the higher resolution BARRA-SY dataset at a 3 km resolution. Remarkably, it performs well without additional training, demonstrating its ability to generalize across different resolutions. This approach holds promise for generating large ensembles of regional climate simulations by downscaling GCMs with varying input resolutions without incurring additional training costs. Ultimately, this method could provide more comprehensive estimates of potential future changes in key climate variables, aiding in effective planning for extreme weather events and climate change adaptation strategies.

Title: Reconciling Semantic Controllability and Diversity for Remote Sensing Image Synthesis with Hybrid Semantic Embedding

• Authors: Junde Liu, Danpei Zhao, Bo Yuan, Wentao Li, Tian Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14781
• Pdf URL: https://arxiv.org/pdf/2411.14781
• Copy Paste: [[2411.14781]] Reconciling Semantic Controllability and Diversity for Remote Sensing Image Synthesis with Hybrid Semantic Embedding(https://arxiv.org/abs/2411.14781)
• Keywords: generative
• Abstract: Significant advancements have been made in semantic image synthesis in remote sensing. However, existing methods still face formidable challenges in balancing semantic controllability and diversity. In this paper, we present a Hybrid Semantic Embedding Guided Generative Adversarial Network (HySEGGAN) for controllable and efficient remote sensing image synthesis. Specifically, HySEGGAN leverages hierarchical information from a single source. Motivated by feature description, we propose a hybrid semantic Embedding method, that coordinates fine-grained local semantic layouts to characterize the geometric structure of remote sensing objects without extra information. Besides, a Semantic Refinement Network (SRN) is introduced, incorporating a novel loss function to ensure fine-grained semantic feedback. The proposed approach mitigates semantic confusion and prevents geometric pattern collapse. Experimental results indicate that the method strikes an excellent balance between semantic controllability and diversity. Furthermore, HySEGGAN significantly improves the quality of synthesized images and achieves state-of-the-art performance as a data augmentation technique across multiple datasets for downstream tasks.

Title: Simplifying CLIP: Unleashing the Power of Large-Scale Models on Consumer-level Computers

• Authors: Hongbo Liu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14789
• Pdf URL: https://arxiv.org/pdf/2411.14789
• Copy Paste: [[2411.14789]] Simplifying CLIP: Unleashing the Power of Large-Scale Models on Consumer-level Computers(https://arxiv.org/abs/2411.14789)
• Keywords: transformer
• Abstract: Contrastive Language-Image Pre-training (CLIP) has attracted a surge of attention for its superior zero-shot performance and excellent transferability to downstream tasks. However, training such large-scale models usually requires substantial computation and storage, which poses barriers for general users with consumer-level computers. Motivated by this observation, in this paper we investigate how to achieve competitive performance on only one Nvidia RTX3090 GPU and with one terabyte for storing dataset. On one hand, we simplify the transformer block structure and combine Weight Inheritance with multi-stage Knowledge Distillation (WIKD), thereby reducing the parameters and improving the inference speed during training along with deployment. On the other hand, confronted with the convergence challenge posed by small dataset, we generate synthetic captions for each sample as data augmentation, and devise a novel Pair Matching (PM) loss to fully exploit the distinguishment among positive and negative image-text pairs. Extensive experiments demonstrate that our model can achieve a new state-of-the-art datascale-parameter-accuracy tradeoff, which could further popularize the CLIP model in the related research community.

Title: KBAda: Efficient Self Adaptation on Specific Knowledge Bases

• Authors: Zheni Zeng, Yuxuan Chen, Shi Yu, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14790
• Pdf URL: https://arxiv.org/pdf/2411.14790
• Copy Paste: [[2411.14790]] KBAda: Efficient Self Adaptation on Specific Knowledge Bases(https://arxiv.org/abs/2411.14790)
• Keywords: large language model
• Abstract: Humans can utilize techniques to quickly acquire knowledge from specific materials in advance, such as creating self-assessment questions, enabling us to achieving related tasks more efficiently. In contrast, large language models (LLMs) usually relies on retrieval-augmented generation to exploit knowledge materials in an instant manner, or requires external signals such as human preference data and stronger LLM annotations to conduct knowledge adaptation. To unleash the self-learning potential of LLMs, we propose KBAda, an approach designed for efficient adaptation to downstream tasks involving knowledge bases. Our method utilizes iterative training with self-annotated data such as Q&A pairs and revision suggestions, enabling the model to grasp the knowledge content efficiently. Experimental results on multiple datasets demonstrate the effectiveness of our approach, significantly boosting model performance in downstream tasks that require specific knowledge at a low cost. Notably, our approach achieves over 90% of the performance improvement that can be obtained by using GPT-4-turbo annotation, while relying entirely on self-supervision. We release our experimental data, models, and process analyses to the community for further exploration (this https URL).

Title: Style-Friendly SNR Sampler for Style-Driven Generation

• Authors: Jooyoung Choi, Chaehun Shin, Yeongtak Oh, Heeseung Kim, Sungroh Yoon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14793
• Pdf URL: https://arxiv.org/pdf/2411.14793
• Copy Paste: [[2411.14793]] Style-Friendly SNR Sampler for Style-Driven Generation(https://arxiv.org/abs/2411.14793)
• Keywords: diffusion
• Abstract: Recent large-scale diffusion models generate high-quality images but struggle to learn new, personalized artistic styles, which limits the creation of unique style templates. Fine-tuning with reference images is the most promising approach, but it often blindly utilizes objectives and noise level distributions used for pre-training, leading to suboptimal style alignment. We propose the Style-friendly SNR sampler, which aggressively shifts the signal-to-noise ratio (SNR) distribution toward higher noise levels during fine-tuning to focus on noise levels where stylistic features emerge. This enables models to better capture unique styles and generate images with higher style alignment. Our method allows diffusion models to learn and share new "style templates", enhancing personalized content creation. We demonstrate the ability to generate styles such as personal watercolor paintings, minimal flat cartoons, 3D renderings, multi-panel images, and memes with text, thereby broadening the scope of style-driven generation.

Title: De-biased Multimodal Electrocardiogram Analysis

• Authors: Haitao Li, Ziyu Li, Yiheng Mao, Ziyi Liu, Zhoujian Sun, Zhengxing Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14795
• Pdf URL: https://arxiv.org/pdf/2411.14795
• Copy Paste: [[2411.14795]] De-biased Multimodal Electrocardiogram Analysis(https://arxiv.org/abs/2411.14795)
• Keywords: large language model
• Abstract: Multimodal large language models (MLLMs) are increasingly being applied in the medical field, particularly in medical imaging. However, developing MLLMs for ECG signals, which are crucial in clinical settings, has been a significant challenge beyond medical imaging. Previous studies have attempted to address this by converting ECGs into several text tags using an external classifier in a training-free manner. However, this approach significantly compresses the information in ECGs and underutilizes the reasoning capabilities of LLMs. In this work, we directly feed the embeddings of ECGs into the LLM through a projection layer, retaining more information about ECGs and better leveraging the reasoning abilities of LLMs. Our method can also effectively handle a common situation in clinical practice where it is necessary to compare two ECGs taken at different times. Recent studies found that MLLMs may rely solely on text input to provide answers, ignoring inputs from other modalities. We analyzed this phenomenon from a causal perspective in the context of ECG MLLMs and discovered that the confounder, severity of illness, introduces a spurious correlation between the question and answer, leading the model to rely on this spurious correlation and ignore the ECG input. Such models do not comprehend the ECG input and perform poorly in adversarial tests where different expressions of the same question are used in the training and testing sets. We designed a de-biased pre-training method to eliminate the confounder's effect according to the theory of backdoor adjustment. Our model performed well on the ECG-QA task under adversarial testing and demonstrated zero-shot capabilities. An interesting random ECG test further validated that our model effectively understands and utilizes the input ECG signal.

Title: Facial Features Matter: a Dynamic Watermark based Proactive Deepfake Detection Approach

• Authors: Shulin Lan, Kanlin Liu, Yazhou Zhao, Chen Yang, Yingchao Wang, Xingshan Yao, Liehuang Zhu
• Subjects: cs.CV, cs.CR, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.14798
• Pdf URL: https://arxiv.org/pdf/2411.14798
• Copy Paste: [[2411.14798]] Facial Features Matter: a Dynamic Watermark based Proactive Deepfake Detection Approach(https://arxiv.org/abs/2411.14798)
• Keywords: security, protect, attack, watermark
• Abstract: Current passive deepfake face-swapping detection methods encounter significance bottlenecks in model generalization capabilities. Meanwhile, proactive detection methods often use fixed watermarks which lack a close relationship with the content they protect and are vulnerable to security risks. Dynamic watermarks based on facial features offer a promising solution, as these features provide unique identifiers. Therefore, this paper proposes a Facial Feature-based Proactive deepfake detection method (FaceProtect), which utilizes changes in facial characteristics during deepfake manipulation as a novel detection mechanism. We introduce a GAN-based One-way Dynamic Watermark Generating Mechanism (GODWGM) that uses 128-dimensional facial feature vectors as inputs. This method creates irreversible mappings from facial features to watermarks, enhancing protection against various reverse inference attacks. Additionally, we propose a Watermark-based Verification Strategy (WVS) that combines steganography with GODWGM, allowing simultaneous transmission of the benchmark watermark representing facial features within the image. Experimental results demonstrate that our proposed method maintains exceptional detection performance and exhibits high practicality on images altered by various deepfake techniques.

Title: High-Resolution Image Synthesis via Next-Token Prediction

• Authors: Dengsheng Chen, Jie Hu, Tiezhu Yue, Xiaoming Wei
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14808
• Pdf URL: https://arxiv.org/pdf/2411.14808
• Copy Paste: [[2411.14808]] High-Resolution Image Synthesis via Next-Token Prediction(https://arxiv.org/abs/2411.14808)
• Keywords: transformer
• Abstract: Denoising with a Joint-Embedding Predictive Architecture (D-JEPA), an autoregressive model, has demonstrated outstanding performance in class-conditional image generation. However, the application of next-token prediction in high-resolution text-to-image generation remains underexplored. In this paper, we introduce D-JEPA$\cdot$T2I, an extension of D-JEPA incorporating flow matching loss, designed to enable data-efficient continuous resolution learning. D-JEPA$\cdot$T2I leverages a multimodal visual transformer to effectively integrate textual and visual features and adopts Visual Rotary Positional Embedding (VoPE) to facilitate continuous resolution learning. Furthermore, we devise a data feedback mechanism that significantly enhances data utilization efficiency. For the first time, we achieve state-of-the-art \textbf{high-resolution} image synthesis via next-token prediction. The experimental code and pretrained models will be open-sourced at \url{this https URL}.

Title: Unsupervised Multi-view UAV Image Geo-localization via Iterative Rendering

• Authors: Haoyuan Li, Chang Xu, Wen Yang, Li Mi, Huai Yu, Haijian Zhang
• Subjects: cs.CV, cs.RO, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.14816
• Pdf URL: https://arxiv.org/pdf/2411.14816
• Copy Paste: [[2411.14816]] Unsupervised Multi-view UAV Image Geo-localization via Iterative Rendering(https://arxiv.org/abs/2411.14816)
• Keywords: robust
• Abstract: Unmanned Aerial Vehicle (UAV) Cross-View Geo-Localization (CVGL) presents significant challenges due to the view discrepancy between oblique UAV images and overhead satellite images. Existing methods heavily rely on the supervision of labeled datasets to extract viewpoint-invariant features for cross-view retrieval. However, these methods have expensive training costs and tend to overfit the region-specific cues, showing limited generalizability to new regions. To overcome this issue, we propose an unsupervised solution that lifts the scene representation to 3d space from UAV observations for satellite image generation, providing robust representation against view distortion. By generating orthogonal images that closely resemble satellite views, our method reduces view discrepancies in feature representation and mitigates shortcuts in region-specific image pairing. To further align the rendered image's perspective with the real one, we design an iterative camera pose updating mechanism that progressively modulates the rendered query image with potential satellite targets, eliminating spatial offsets relative to the reference images. Additionally, this iterative refinement strategy enhances cross-view feature invariance through view-consistent fusion across iterations. As such, our unsupervised paradigm naturally avoids the problem of region-specific overfitting, enabling generic CVGL for UAV images without feature fine-tuning or data-driven training. Experiments on the University-1652 and SUES-200 datasets demonstrate that our approach significantly improves geo-localization accuracy while maintaining robustness across diverse regions. Notably, without model fine-tuning or paired training, our method achieves competitive performance with recent supervised methods.

Title: Omni-IML: Towards Unified Image Manipulation Localization

• Authors: Chenfan Qu, Yiwu Zhong, Fengjun Guo, Lianwen Jin
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14823
• Pdf URL: https://arxiv.org/pdf/2411.14823
• Copy Paste: [[2411.14823]] Omni-IML: Towards Unified Image Manipulation Localization(https://arxiv.org/abs/2411.14823)
• Keywords: security
• Abstract: Image manipulation can lead to misinterpretation of visual content, posing significant risks to information security. Image Manipulation Localization (IML) has thus received increasing attention. However, existing IML methods rely heavily on task-specific designs, making them perform well only on one target image type but are mostly random guessing on other image types, and even joint training on multiple image types causes significant performance degradation. This hinders the deployment for real applications as it notably increases maintenance costs and the misclassification of image types leads to serious error accumulation. To this end, we propose Omni-IML, the first generalist model to unify diverse IML tasks. Specifically, Omni-IML achieves generalism by adopting the Modal Gate Encoder and the Dynamic Weight Decoder to adaptively determine the optimal encoding modality and the optimal decoder filters for each sample. We additionally propose an Anomaly Enhancement module that enhances the features of tampered regions with box supervision and helps the generalist model to extract common features across different IML tasks. We validate our approach on IML tasks across three major scenarios: natural images, document images, and face images. Without bells and whistles, our Omni-IML achieves state-of-the-art performance on all three tasks with a single unified model, providing valuable strategies and insights for real-world application and future research in generalist image forensics. Our code will be publicly available.

Title: OSPtrack: A Labeled Dataset Targeting Simulated Open-Source Package Execution

• Authors: Zhuoran Tan, Christos Anagnosstopoulos, Jeremy Singer
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14829
• Pdf URL: https://arxiv.org/pdf/2411.14829
• Copy Paste: [[2411.14829]] OSPtrack: A Labeled Dataset Targeting Simulated Open-Source Package Execution(https://arxiv.org/abs/2411.14829)
• Keywords: attack
• Abstract: Open-source software is a fundamental part of the internet and the cyber supply chain, but its exploitation has become more frequent. While vulnerability detection in OSS has advanced, previous work mainly focuses on static code analysis, neglecting runtime indicators. To address this, we created a dataset spanning multiple ecosystems, capturing features generated during the execution of packages and libraries in isolated environments. The dataset includes 9,461 package reports (1,962 malicious), with static and dynamic features such as files, sockets, commands, and DNS records. Labeled with verified information and detailed sub-labels for attack types, this dataset helps identify malicious indicators, especially when source code access is limited, and supports efficient detection methods during runtime.

Title: VisGraphVar: A Benchmark Generator for Assessing Variability in Graph Analysis Using Large Vision-Language Models

• Authors: Camilo Chacón Sartori, Christian Blum, Filippo Bistaffa
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14832
• Pdf URL: https://arxiv.org/pdf/2411.14832
• Copy Paste: [[2411.14832]] VisGraphVar: A Benchmark Generator for Assessing Variability in Graph Analysis Using Large Vision-Language Models(https://arxiv.org/abs/2411.14832)
• Keywords: robust, segmentation
• Abstract: The fast advancement of Large Vision-Language Models (LVLMs) has shown immense potential. These models are increasingly capable of tackling abstract visual tasks. Geometric structures, particularly graphs with their inherent flexibility and complexity, serve as an excellent benchmark for evaluating these models' predictive capabilities. While human observers can readily identify subtle visual details and perform accurate analyses, our investigation reveals that state-of-the-art LVLMs exhibit consistent limitations in specific visual graph scenarios, especially when confronted with stylistic variations. In response to these challenges, we introduce VisGraphVar (Visual Graph Variability), a customizable benchmark generator able to produce graph images for seven distinct task categories (detection, classification, segmentation, pattern recognition, link prediction, reasoning, matching), designed to systematically evaluate the strengths and limitations of individual LVLMs. We use VisGraphVar to produce 990 graph images and evaluate six LVLMs, employing two distinct prompting strategies, namely zero-shot and chain-of-thought. The findings demonstrate that variations in visual attributes of images (e.g., node labeling and layout) and the deliberate inclusion of visual imperfections, such as overlapping nodes, significantly affect model performance. This research emphasizes the importance of a comprehensive evaluation across graph-related tasks, extending beyond reasoning alone. VisGraphVar offers valuable insights to guide the development of more reliable and robust systems capable of performing advanced visual graph analysis.

Title: Gradient Masking All-at-Once: Ensemble Everything Everywhere Is Not Robust

• Authors: Jie Zhang, Kristina Nikolić, Nicholas Carlini, Florian Tramèr
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14834
• Pdf URL: https://arxiv.org/pdf/2411.14834
• Copy Paste: [[2411.14834]] Gradient Masking All-at-Once: Ensemble Everything Everywhere Is Not Robust(https://arxiv.org/abs/2411.14834)
• Keywords: defense, attack, robust
• Abstract: Ensemble everything everywhere is a defense to adversarial examples that was recently proposed to make image classifiers robust. This defense works by ensembling a model's intermediate representations at multiple noisy image resolutions, producing a single robust classification. This defense was shown to be effective against multiple state-of-the-art attacks. Perhaps even more convincingly, it was shown that the model's gradients are perceptually aligned: attacks against the model produce noise that perceptually resembles the targeted class. In this short note, we show that this defense is not robust to adversarial attack. We first show that the defense's randomness and ensembling method cause severe gradient masking. We then use standard adaptive attack techniques to reduce the defense's robust accuracy from 48% to 1% on CIFAR-100 and from 62% to 4% on CIFAR-10, under the $\ell_\infty$-norm threat model with $\varepsilon=8/255$.

Title: Latent Schrodinger Bridge: Prompting Latent Diffusion for Fast Unpaired Image-to-Image Translation

• Authors: Jeongsol Kim, Beomsu Kim, Jong Chul Ye
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14863
• Pdf URL: https://arxiv.org/pdf/2411.14863
• Copy Paste: [[2411.14863]] Latent Schrodinger Bridge: Prompting Latent Diffusion for Fast Unpaired Image-to-Image Translation(https://arxiv.org/abs/2411.14863)
• Keywords: diffusion
• Abstract: Diffusion models (DMs), which enable both image generation from noise and inversion from data, have inspired powerful unpaired image-to-image (I2I) translation algorithms. However, they often require a larger number of neural function evaluations (NFEs), limiting their practical applicability. In this paper, we tackle this problem with Schrodinger Bridges (SBs), which are stochastic differential equations (SDEs) between distributions with minimal transport cost. We analyze the probability flow ordinary differential equation (ODE) formulation of SBs, and observe that we can decompose its vector field into a linear combination of source predictor, target predictor, and noise predictor. Inspired by this observation, we propose Latent Schrodinger Bridges (LSBs) that approximate the SB ODE via pre-trained Stable Diffusion, and develop appropriate prompt optimization and change of variables formula to match the training and inference between distributions. We demonstrate that our algorithm successfully conduct competitive I2I translation in unsupervised setting with only a fraction of computation cost required by previous DM-based I2I methods.

Title: Prioritize Denoising Steps on Diffusion Model Preference Alignment via Explicit Denoised Distribution Estimation

• Authors: Dingyuan Shi, Yong Wang, Hangyu Li, Xiangxiang Chu
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14871
• Pdf URL: https://arxiv.org/pdf/2411.14871
• Copy Paste: [[2411.14871]] Prioritize Denoising Steps on Diffusion Model Preference Alignment via Explicit Denoised Distribution Estimation(https://arxiv.org/abs/2411.14871)
• Keywords: diffusion
• Abstract: Diffusion models have shown remarkable success in text-to-image generation, making alignment methods for these models increasingly important. A key challenge is the sparsity of preference labels, which are typically available only at the terminal of denoising trajectories. This raises the issue of how to assign credit across denoising steps based on these sparse labels. In this paper, we propose Denoised Distribution Estimation (DDE), a novel method for credit assignment. Unlike previous approaches that rely on auxiliary models or hand-crafted schemes, DDE derives its strategy more explicitly. The proposed DDE directly estimates the terminal denoised distribution from the perspective of each step. It is equipped with two estimation strategies and capable of representing the entire denoising trajectory with a single model inference. Theoretically and empirically, we show that DDE prioritizes optimizing the middle part of the denoising trajectory, resulting in a novel and effective credit assignment scheme. Extensive experiments demonstrate that our approach achieves superior performance, both quantitatively and qualitatively.

Title: Astro-HEP-BERT: A bidirectional language model for studying the meanings of concepts in astrophysics and high energy physics

• Authors: Arno Simons
• Subjects: cs.CL, physics.hist-ph
• Abstract URL: https://arxiv.org/abs/2411.14877
• Pdf URL: https://arxiv.org/pdf/2411.14877
• Copy Paste: [[2411.14877]] Astro-HEP-BERT: A bidirectional language model for studying the meanings of concepts in astrophysics and high energy physics(https://arxiv.org/abs/2411.14877)
• Keywords: transformer
• Abstract: I present Astro-HEP-BERT, a transformer-based language model specifically designed for generating contextualized word embeddings (CWEs) to study the meanings of concepts in astrophysics and high-energy physics. Built on a general pretrained BERT model, Astro-HEP-BERT underwent further training over three epochs using the Astro-HEP Corpus, a dataset I curated from 21.84 million paragraphs extracted from more than 600,000 scholarly articles on arXiv, all belonging to at least one of these two scientific domains. The project demonstrates both the effectiveness and feasibility of adapting a bidirectional transformer for applications in the history, philosophy, and sociology of science (HPSS). The entire training process was conducted using freely available code, pretrained weights, and text inputs, completed on a single MacBook Pro Laptop (M2/96GB). Preliminary evaluations indicate that Astro-HEP-BERT's CWEs perform comparably to domain-adapted BERT models trained from scratch on larger datasets for domain-specific word sense disambiguation and induction and related semantic change analyses. This suggests that retraining general language models for specific scientific domains can be a cost-effective and efficient strategy for HPSS researchers, enabling high performance without the need for extensive training from scratch.

Title: Physical and Software Based Fault Injection Attacks Against TEEs in Mobile Devices: A Systemisation of Knowledge

• Authors: Aaron Joy, Ben Soh, Zhi Zhang, Sri Parameswaran, Darshana Jayasinghe
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14878
• Pdf URL: https://arxiv.org/pdf/2411.14878
• Copy Paste: [[2411.14878]] Physical and Software Based Fault Injection Attacks Against TEEs in Mobile Devices: A Systemisation of Knowledge(https://arxiv.org/abs/2411.14878)
• Keywords: secure, security, protect, attack, robust
• Abstract: Trusted Execution Environments (TEEs) are critical components of modern secure computing, providing isolated zones in processors to safeguard sensitive data and execute secure operations. Despite their importance, TEEs are increasingly vulnerable to fault injection (FI) attacks, including both physical methods, such as Electromagnetic Fault Injection (EMFI), and software-based techniques. This survey examines these FI methodologies, exploring their ability to disrupt TEE operations and expose vulnerabilities in devices ranging from smartphones and IoT systems to cloud platforms. The study highlights the evolution and effectiveness of non-invasive techniques, such as EMFI, which induce faults through electromagnetic disturbances without physical modifications to hardware, making them harder to detect and mitigate. Real-world case studies illustrate the significant risks posed by these attacks, including unauthorised access, privilege escalation, and data corruption. In addition, the survey identifies gaps in existing TEE security architectures and emphasises the need for enhanced countermeasures, such as dynamic anomaly detection and updated threat models. The findings underline the importance of interdisciplinary collaboration to address these vulnerabilities, involving researchers, manufacturers, and policymakers. This survey provides actionable insights and recommendations to guide the development of more robust TEE architectures in mobile devices, fortify FI resilience, and shape global security standards. By advancing TEE security, this research aims to protect critical digital infrastructure and maintain trust in secure computing systems worldwide.

Title: Boundless Across Domains: A New Paradigm of Adaptive Feature and Cross-Attention for Domain Generalization in Medical Image Segmentation

• Authors: Yuheng Xu, Taiping Zhang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14883
• Pdf URL: https://arxiv.org/pdf/2411.14883
• Copy Paste: [[2411.14883]] Boundless Across Domains: A New Paradigm of Adaptive Feature and Cross-Attention for Domain Generalization in Medical Image Segmentation(https://arxiv.org/abs/2411.14883)
• Keywords: robust, segmentation
• Abstract: Domain-invariant representation learning is a powerful method for domain generalization. Previous approaches face challenges such as high computational demands, training instability, and limited effectiveness with high-dimensional data, potentially leading to the loss of valuable features. To address these issues, we hypothesize that an ideal generalized representation should exhibit similar pattern responses within the same channel across cross-domain images. Based on this hypothesis, we use deep features from the source domain as queries, and deep features from the generated domain as keys and values. Through a cross-channel attention mechanism, the original deep features are reconstructed into robust regularization representations, forming an explicit constraint that guides the model to learn domain-invariant representations. Additionally, style augmentation is another common method. However, existing methods typically generate new styles through convex combinations of source domains, which limits the diversity of training samples by confining the generated styles to the original distribution. To overcome this limitation, we propose an Adaptive Feature Blending (AFB) method that generates out-of-distribution samples while exploring the in-distribution space, significantly expanding the domain range. Extensive experimental results demonstrate that our proposed methods achieve superior performance on two standard domain generalization benchmarks for medical image segmentation.

Title: Evaluating LLM Prompts for Data Augmentation in Multi-label Classification of Ecological Texts

• Authors: Anna Glazkova, Olga Zakharova
• Subjects: cs.CL, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2411.14896
• Pdf URL: https://arxiv.org/pdf/2411.14896
• Copy Paste: [[2411.14896]] Evaluating LLM Prompts for Data Augmentation in Multi-label Classification of Ecological Texts(https://arxiv.org/abs/2411.14896)
• Keywords: large language model
• Abstract: Large language models (LLMs) play a crucial role in natural language processing (NLP) tasks, improving the understanding, generation, and manipulation of human language across domains such as translating, summarizing, and classifying text. Previous studies have demonstrated that instruction-based LLMs can be effectively utilized for data augmentation to generate diverse and realistic text samples. This study applied prompt-based data augmentation to detect mentions of green practices in Russian social media. Detecting green practices in social media aids in understanding their prevalence and helps formulate recommendations for scaling eco-friendly actions to mitigate environmental issues. We evaluated several prompts for augmenting texts in a multi-label classification task, either by rewriting existing datasets using LLMs, generating new data, or combining both approaches. Our results revealed that all strategies improved classification performance compared to the models fine-tuned only on the original dataset, outperforming baselines in most cases. The best results were obtained with the prompt that paraphrased the original text while clearly indicating the relevant categories.

Title: ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos

• Authors: Tanveer Hannan, Md Mohaiminul Islam, Jindong Gu, Thomas Seidl, Gedas Bertasius
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14901
• Pdf URL: https://arxiv.org/pdf/2411.14901
• Copy Paste: [[2411.14901]] ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos(https://arxiv.org/abs/2411.14901)
• Keywords: large language model
• Abstract: Large language models (LLMs) excel at retrieving information from lengthy text, but their vision-language counterparts (VLMs) face difficulties with hour-long videos, especially for temporal grounding. Specifically, these VLMs are constrained by frame limitations, often losing essential temporal details needed for accurate event localization in extended video content. We propose ReVisionLLM, a recursive vision-language model designed to locate events in hour-long videos. Inspired by human search strategies, our model initially targets broad segments of interest, progressively revising its focus to pinpoint exact temporal boundaries. Our model can seamlessly handle videos of vastly different lengths, from minutes to hours. We also introduce a hierarchical training strategy that starts with short clips to capture distinct events and progressively extends to longer videos. To our knowledge, ReVisionLLM is the first VLM capable of temporal grounding in hour-long videos, outperforming previous state-of-the-art methods across multiple datasets by a significant margin (+2.6% R1@0.1 on MAD). The code is available at this https URL.

Title: Exploring Kolmogorov-Arnold Networks for Interpretable Time Series Classification

• Authors: Irina Barašin, Blaž Bertalanič, Miha Mohorčič, Carolina Fortuna
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14904
• Pdf URL: https://arxiv.org/pdf/2411.14904
• Copy Paste: [[2411.14904]] Exploring Kolmogorov-Arnold Networks for Interpretable Time Series Classification(https://arxiv.org/abs/2411.14904)
• Keywords: robust, interpretability
• Abstract: Time series classification is a relevant step supporting decision-making processes in various domains, and deep neural models have shown promising performance. Despite significant advancements in deep learning, the theoretical understanding of how and why complex architectures function remains limited, prompting the need for more interpretable models. Recently, the Kolmogorov-Arnold Networks (KANs) have been proposed as a more interpretable alternative. While KAN-related research is significantly rising, to date, the study of KAN architectures for time series classification has been limited. In this paper, we aim to conduct a comprehensive and robust exploration of the KAN architecture for time series classification on the UCR benchmark. More specifically, we look at a) how reference architectures for forecasting transfer to classification, at the b) hyperparameter and implementation influence on the classification performance in view of finding the one that performs best on the selected benchmark, the c) complexity trade-offs and d) interpretability advantages. Our results show that (1) Efficient KAN outperforms MLP in performance and computational efficiency, showcasing its suitability for tasks classification tasks. (2) Efficient KAN is more stable than KAN across grid sizes, depths, and layer configurations, particularly with lower learning rates. (3) KAN maintains competitive accuracy compared to state-of-the-art models like HIVE-COTE2, with smaller architectures and faster training times, supporting its balance of performance and transparency. (4) The interpretability of the KAN model aligns with findings from SHAP analysis, reinforcing its capacity for transparent decision-making.

Title: Feasibility Study for Supporting Static Malware Analysis Using LLM

• Authors: Shota Fujii, Rei Yamagishi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14905
• Pdf URL: https://arxiv.org/pdf/2411.14905
• Copy Paste: [[2411.14905]] Feasibility Study for Supporting Static Malware Analysis Using LLM(https://arxiv.org/abs/2411.14905)
• Keywords: security, large language model
• Abstract: Large language models (LLMs) are becoming more advanced and widespread and have shown their applicability to various domains, including cybersecurity. Static malware analysis is one of the most important tasks in cybersecurity; however, it is time-consuming and requires a high level of expertise. Therefore, we conducted a demonstration experiment focusing on whether an LLM can be used to support static analysis. First, we evaluated the ability of the LLM to explain malware functionality. The results showed that the LLM can generate descriptions that cover functions with an accuracy of up to 90.9\%. In addition, we asked six static analysts to perform a pseudo static analysis task using LLM explanations to verify that the LLM can be used in practice. Through subsequent questionnaires and interviews with the participants, we also demonstrated the practical applicability of LLMs. Lastly, we summarized the problems and required functions when using an LLM as static analysis support, as well as recommendations for future research opportunities.

Title: LiDAR-based End-to-end Temporal Perception for Vehicle-Infrastructure Cooperation

• Authors: Zhenwei Yang, Jilei Mao, Wenxian Yang, Yibo Ai, Yu Kong, Haibao Yu, Weidong Zhang
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.14927
• Pdf URL: https://arxiv.org/pdf/2411.14927
• Copy Paste: [[2411.14927]] LiDAR-based End-to-end Temporal Perception for Vehicle-Infrastructure Cooperation(https://arxiv.org/abs/2411.14927)
• Keywords: robust
• Abstract: Temporal perception, the ability to detect and track objects over time, is critical in autonomous driving for maintaining a comprehensive understanding of dynamic environments. However, this task is hindered by significant challenges, including incomplete perception caused by occluded objects and observational blind spots, which are common in single-vehicle perception systems. To address these issues, we introduce LET-VIC, a LiDAR-based End-to-End Tracking framework for Vehicle-Infrastructure Cooperation (VIC). LET-VIC leverages Vehicle-to-Everything (V2X) communication to enhance temporal perception by fusing spatial and temporal data from both vehicle and infrastructure sensors. First, it spatially integrates Bird's Eye View (BEV) features from vehicle-side and infrastructure-side LiDAR data, creating a comprehensive view that mitigates occlusions and compensates for blind spots. Second, LET-VIC incorporates temporal context across frames, allowing the model to leverage historical data for enhanced tracking stability and accuracy. To further improve robustness, LET-VIC includes a Calibration Error Compensation (CEC) module to address sensor misalignments and ensure precise feature alignment. Experiments on the V2X-Seq-SPD dataset demonstrate that LET-VIC significantly outperforms baseline models, achieving at least a 13.7% improvement in mAP and a 13.1% improvement in AMOTA without considering communication delays. This work offers a practical solution and a new research direction for advancing temporal perception in autonomous driving through vehicle-infrastructure cooperation.

Title: Geminio: Language-Guided Gradient Inversion Attacks in Federated Learning

• Authors: Junjie Shan, Ziqi Zhao, Jialin Lu, Rui Zhang, Siu Ming Yiu, Ka-Ho Chow
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14937
• Pdf URL: https://arxiv.org/pdf/2411.14937
• Copy Paste: [[2411.14937]] Geminio: Language-Guided Gradient Inversion Attacks in Federated Learning(https://arxiv.org/abs/2411.14937)
• Keywords: privacy, defense, attack, federate
• Abstract: Foundation models that bridge vision and language have made significant progress, inspiring numerous life-enriching applications. However, their potential for misuse to introduce new threats remains largely unexplored. This paper reveals that vision-language models (VLMs) can be exploited to overcome longstanding limitations in gradient inversion attacks (GIAs) within federated learning (FL), where an FL server reconstructs private data samples from gradients shared by victim clients. Current GIAs face challenges in reconstructing high-resolution images, especially when the victim has a large local data batch. While focusing reconstruction on valuable samples rather than the entire batch is promising, existing methods lack the flexibility to allow attackers to specify their target data. In this paper, we introduce Geminio, the first approach to transform GIAs into semantically meaningful, targeted attacks. Geminio enables a brand new privacy attack experience: attackers can describe, in natural language, the types of data they consider valuable, and Geminio will prioritize reconstruction to focus on those high-value samples. This is achieved by leveraging a pretrained VLM to guide the optimization of a malicious global model that, when shared with and optimized by a victim, retains only gradients of samples that match the attacker-specified query. Extensive experiments demonstrate Geminio's effectiveness in pinpointing and reconstructing targeted samples, with high success rates across complex datasets under FL and large batch sizes and showing resilience against existing defenses.

Title: Reliable Evaluation of Attribution Maps in CNNs: A Perturbation-Based Approach

• Authors: Lars Nieradzik, Henrike Stephani, Janis Keuper
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14946
• Pdf URL: https://arxiv.org/pdf/2411.14946
• Copy Paste: [[2411.14946]] Reliable Evaluation of Attribution Maps in CNNs: A Perturbation-Based Approach(https://arxiv.org/abs/2411.14946)
• Keywords: robust
• Abstract: In this paper, we present an approach for evaluating attribution maps, which play a central role in interpreting the predictions of convolutional neural networks (CNNs). We show that the widely used insertion/deletion metrics are susceptible to distribution shifts that affect the reliability of the ranking. Our method proposes to replace pixel modifications with adversarial perturbations, which provides a more robust evaluation framework. By using smoothness and monotonicity measures, we illustrate the effectiveness of our approach in correcting distribution shifts. In addition, we conduct the most comprehensive quantitative and qualitative assessment of attribution maps to date. Introducing baseline attribution maps as sanity checks, we find that our metric is the only contender to pass all checks. Using Kendall's $\tau$ rank correlation coefficient, we show the increased consistency of our metric across 15 dataset-architecture combinations. Of the 16 attribution maps tested, our results clearly show SmoothGrad to be the best map currently available. This research makes an important contribution to the development of attribution maps by providing a reliable and consistent evaluation framework. To ensure reproducibility, we will provide the code along with our results.

Title: Evaluating Vision Transformer Models for Visual Quality Control in Industrial Manufacturing

• Authors: Miriam Alber, Christoph Hönes, Patrick Baier
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14953
• Pdf URL: https://arxiv.org/pdf/2411.14953
• Copy Paste: [[2411.14953]] Evaluating Vision Transformer Models for Visual Quality Control in Industrial Manufacturing(https://arxiv.org/abs/2411.14953)
• Keywords: transformer
• Abstract: One of the most promising use-cases for machine learning in industrial manufacturing is the early detection of defective products using a quality control system. Such a system can save costs and reduces human errors due to the monotonous nature of visual inspections. Today, a rich body of research exists which employs machine learning methods to identify rare defective products in unbalanced visual quality control datasets. These methods typically rely on two components: A visual backbone to capture the features of the input image and an anomaly detection algorithm that decides if these features are within an expected distribution. With the rise of transformer architecture as visual backbones of choice, there exists now a great variety of different combinations of these two components, ranging all along the trade-off between detection quality and inference time. Facing this variety, practitioners in the field often have to spend a considerable amount of time on researching the right combination for their use-case at hand. Our contribution is to help practitioners with this choice by reviewing and evaluating current vision transformer models together with anomaly detection methods. For this, we chose SotA models of both disciplines, combined them and evaluated them towards the goal of having small, fast and efficient anomaly detection models suitable for industrial manufacturing. We evaluated the results of our experiments on the well-known MVTecAD and BTAD datasets. Moreover, we give guidelines for choosing a suitable model architecture for a quality control system in practice, considering given use-case and hardware constraints.

Title: Information Extraction from Heterogenous Documents without Ground Truth Labels using Synthetic Label Generation and Knowledge Distillation

• Authors: Aniket Bhattacharyya, Anurag Tripathi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14957
• Pdf URL: https://arxiv.org/pdf/2411.14957
• Copy Paste: [[2411.14957]] Information Extraction from Heterogenous Documents without Ground Truth Labels using Synthetic Label Generation and Knowledge Distillation(https://arxiv.org/abs/2411.14957)
• Keywords: protect, extraction
• Abstract: Invoices and receipts submitted by employees are visually rich documents (VRDs) with textual, visual and layout information. To protect against the risk of fraud and abuse, it is crucial for organizations to efficiently extract desired information from submitted receipts. This helps in the assessment of key factors such as appropriateness of the expense claim, adherence to spending and transaction policies, the validity of the receipt, as well as downstream anomaly detection at various levels. These documents are heterogenous, with multiple formats and languages, uploaded with different image qualities, and often do not contain ground truth labels for the efficient training of models. In this paper we propose Task Aware Instruction-based Labelling (TAIL), a method for synthetic label generation in VRD corpuses without labels, and fine-tune a multimodal Visually Rich Document Understanding Model (VRDU) on TAIL labels using response-based knowledge distillation without using the teacher model's weights or training dataset to conditionally generate annotations in the appropriate format. Using a benchmark external dataset where ground truth labels are available, we demonstrate conditions under which our approach performs at par with Claude 3 Sonnet through empirical studies. We then show that the resulting model performs at par or better on the internal expense documents of a large multinational organization than state-of-the-art LMM (large multimodal model) Claude 3 Sonnet while being 85% less costly and ~5X faster, and outperforms layout-aware baselines by more than 10% in Average Normalized Levenshtein Similarity (ANLS) scores due to its ability to reason and extract information from rare formats. Finally, we illustrate the usage of our approach in overpayment prevention.

Title: LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement

• Authors: Jieming Bian, Lei Wang, Letian Zhang, Jie Xu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14961
• Pdf URL: https://arxiv.org/pdf/2411.14961
• Copy Paste: [[2411.14961]] LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement(https://arxiv.org/abs/2411.14961)
• Keywords: privacy, federate, fair
• Abstract: Foundation models (FMs) achieve strong performance across diverse tasks with task-specific fine-tuning, yet full parameter fine-tuning is often computationally prohibitive for large models. Parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) reduce this cost by introducing low-rank matrices for tuning fewer parameters. While LoRA allows for efficient fine-tuning, it requires significant data for adaptation, making Federated Learning (FL) an appealing solution due to its privacy-preserving collaborative framework. However, combining LoRA with FL introduces two key challenges: the \textbf{Server-Side LoRA Aggregation Bias}, where server-side averaging of LoRA matrices diverges from the ideal global update, and the \textbf{Client-Side LoRA Initialization Drift}, emphasizing the need for consistent initialization across rounds. Existing approaches address these challenges individually, limiting their effectiveness. We propose LoRA-FAIR, a novel method that tackles both issues by introducing a correction term on the server while keeping the original LoRA modules, enhancing aggregation efficiency and accuracy. LoRA-FAIR maintains computational and communication efficiency, yielding superior performance over state-of-the-art methods. Experimental results on ViT and MLP-Mixer models across large-scale datasets demonstrate that LoRA-FAIR consistently achieves performance improvements in FL settings.

Title: LLM for Barcodes: Generating Diverse Synthetic Data for Identity Documents

• Authors: Hitesh Laxmichand Patel, Amit Agarwal, Bhargava Kumar, Karan Gupta, Priyaranjan Pattnayak
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14962
• Pdf URL: https://arxiv.org/pdf/2411.14962
• Copy Paste: [[2411.14962]] LLM for Barcodes: Generating Diverse Synthetic Data for Identity Documents(https://arxiv.org/abs/2411.14962)
• Keywords: security, privacy, robust, extraction
• Abstract: Accurate barcode detection and decoding in Identity documents is crucial for applications like security, healthcare, and education, where reliable data extraction and verification are essential. However, building robust detection models is challenging due to the lack of diverse, realistic datasets an issue often tied to privacy concerns and the wide variety of document formats. Traditional tools like Faker rely on predefined templates, making them less effective for capturing the complexity of real-world identity documents. In this paper, we introduce a new approach to synthetic data generation that uses LLMs to create contextually rich and realistic data without relying on predefined field. Using the vast knowledge LLMs have about different documents and content, our method creates data that reflects the variety found in real identity documents. This data is then encoded into barcode and overlayed on templates for documents such as Driver's licenses, Insurance cards, Student IDs. Our approach simplifies the process of dataset creation, eliminating the need for extensive domain knowledge or predefined fields. Compared to traditional methods like Faker, data generated by LLM demonstrates greater diversity and contextual relevance, leading to improved performance in barcode detection models. This scalable, privacy-first solution is a big step forward in advancing machine learning for automated document processing and identity verification.

Title: SwissADT: An Audio Description Translation System for Swiss Languages

• Authors: Lukas Fischer, Yingqiang Gao, Alexa Lintner, Sarah Ebling
• Subjects: cs.CL, cs.AI, cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.14967
• Pdf URL: https://arxiv.org/pdf/2411.14967
• Copy Paste: [[2411.14967]] SwissADT: An Audio Description Translation System for Swiss Languages(https://arxiv.org/abs/2411.14967)
• Keywords: large language model
• Abstract: Audio description (AD) is a crucial accessibility service provided to blind persons and persons with visual impairment, designed to convey visual information in acoustic form. Despite recent advancements in multilingual machine translation research, the lack of well-crafted and time-synchronized AD data impedes the development of audio description translation (ADT) systems that address the needs of multilingual countries such as Switzerland. Furthermore, since the majority of ADT systems rely solely on text, uncertainty exists as to whether incorporating visual information from the corresponding video clips can enhance the quality of ADT outputs. In this work, we present SwissADT, the first ADT system implemented for three main Swiss languages and English. By collecting well-crafted AD data augmented with video clips in German, French, Italian, and English, and leveraging the power of Large Language Models (LLMs), we aim to enhance information accessibility for diverse language populations in Switzerland by automatically translating AD scripts to the desired Swiss language. Our extensive experimental ADT results, composed of both automatic and human evaluations of ADT quality, demonstrate the promising capability of SwissADT for the ADT task. We believe that combining human expertise with the generation power of LLMs can further enhance the performance of ADT systems, ultimately benefiting a larger multilingual target population.

Title: Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation

• Authors: Colin Diggs, Michael Doyle, Amit Madan, Siggy Scott, Emily Escamilla, Jacob Zimmer, Naveed Nekoo, Paul Ursino, Michael Bartholf, Zachary Robin, Anand Patel, Chris Glasz, William Macke, Paul Kirk, Jasper Phillips, Arun Sridharan, Doug Wendt, Scott Rosen, Nitin Naik, Justin F. Brunelle, Samruddhi Thaker
• Subjects: cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2411.14971
• Pdf URL: https://arxiv.org/pdf/2411.14971
• Copy Paste: [[2411.14971]] Leveraging LLMs for Legacy Code Modernization: Challenges and Opportunities for LLM-Generated Documentation(https://arxiv.org/abs/2411.14971)
• Keywords: security
• Abstract: Legacy software systems, written in outdated languages like MUMPS and mainframe assembly, pose challenges in efficiency, maintenance, staffing, and security. While LLMs offer promise for modernizing these systems, their ability to understand legacy languages is largely unknown. This paper investigates the utilization of LLMs to generate documentation for legacy code using two datasets: an electronic health records (EHR) system in MUMPS and open-source applications in IBM mainframe Assembly Language Code (ALC). We propose a prompting strategy for generating line-wise code comments and a rubric to evaluate their completeness, readability, usefulness, and hallucination. Our study assesses the correlation between human evaluations and automated metrics, such as code complexity and reference-based metrics. We find that LLM-generated comments for MUMPS and ALC are generally hallucination-free, complete, readable, and useful compared to ground-truth comments, though ALC poses challenges. However, no automated metrics strongly correlate with comment quality to predict or measure LLM performance. Our findings highlight the limitations of current automated measures and the need for better evaluation metrics for LLM-generated documentation in legacy systems.

Title: Adaptive Group Robust Ensemble Knowledge Distillation

• Authors: Patrik Kenfack, Ulrich Aïvodji, Samira Ebrahimi Kahou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14984
• Pdf URL: https://arxiv.org/pdf/2411.14984
• Copy Paste: [[2411.14984]] Adaptive Group Robust Ensemble Knowledge Distillation(https://arxiv.org/abs/2411.14984)
• Keywords: robust
• Abstract: Neural networks can learn spurious correlations in the data, often leading to performance disparity for underrepresented subgroups. Studies have demonstrated that the disparity is amplified when knowledge is distilled from a complex teacher model to a relatively "simple" student model. Prior work has shown that ensemble deep learning methods can improve the performance of the worst-case subgroups; however, it is unclear if this advantage carries over when distilling knowledge from an ensemble of teachers, especially when the teacher models are debiased. This study demonstrates that traditional ensemble knowledge distillation can significantly drop the performance of the worst-case subgroups in the distilled student model even when the teacher models are debiased. To overcome this, we propose Adaptive Group Robust Ensemble Knowledge Distillation (AGRE-KD), a simple ensembling strategy to ensure that the student model receives knowledge beneficial for unknown underrepresented subgroups. Leveraging an additional biased model, our method selectively chooses teachers whose knowledge would better improve the worst-performing subgroups by upweighting the teachers with gradient directions deviating from the biased model. Our experiments on several datasets demonstrate the superiority of the proposed ensemble distillation technique and show that it can even outperform classic model ensembles based on majority voting.

Title: ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data

• Authors: Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, Ameet Talwalkar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.15004
• Pdf URL: https://arxiv.org/pdf/2411.15004
• Copy Paste: [[2411.15004]] ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data(https://arxiv.org/abs/2411.15004)
• Keywords: large language model
• Abstract: Large Language Model (LLM) agents are rapidly improving to handle increasingly complex web-based tasks. Most of these agents rely on general-purpose, proprietary models like GPT-4 and focus on designing better prompts to improve their planning abilities. However, general-purpose LLMs are not specifically trained to understand specialized web contexts such as HTML, and they often struggle with long-horizon planning. We explore an alternative approach that fine-tunes open-source LLMs using production-scale workflow data collected from over 250 domains corresponding to 6 billion tokens. This simple yet effective approach shows substantial gains over prompting-based agents on existing benchmarks -- ScribeAgent achieves state-of-the-art direct generation performance on Mind2Web and improves the task success rate by 14.1% over the previous best text-only web agents on WebArena. We further perform detailed ablation studies on various fine-tuning design choices and provide insights into LLM selection, training recipes, context window optimization, and effect of dataset sizes.

Title: On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations

• Authors: Guojun Xiong, Shufan Wang, Daniel Jiang, Jian Li
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.15014
• Pdf URL: https://arxiv.org/pdf/2411.15014
• Copy Paste: [[2411.15014]] On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representations(https://arxiv.org/abs/2411.15014)
• Keywords: federate
• Abstract: Federated reinforcement learning (FedRL) enables multiple agents to collaboratively learn a policy without sharing their local trajectories collected during agent-environment interactions. However, in practice, the environments faced by different agents are often heterogeneous, leading to poor performance by the single policy learned by existing FedRL algorithms on individual agents. In this paper, we take a further step and introduce a \emph{personalized} FedRL framework (PFedRL) by taking advantage of possibly shared common structure among agents in heterogeneous environments. Specifically, we develop a class of PFedRL algorithms named PFedRL-Rep that learns (1) a shared feature representation collaboratively among all agents, and (2) an agent-specific weight vector personalized to its local environment. We analyze the convergence of PFedTD-Rep, a particular instance of the framework with temporal difference (TD) learning and linear representations. To the best of our knowledge, we are the first to prove a linear convergence speedup with respect to the number of agents in the PFedRL setting. To achieve this, we show that PFedTD-Rep is an example of the federated two-timescale stochastic approximation with Markovian noise. Experimental results demonstrate that PFedTD-Rep, along with an extension to the control setting based on deep Q-networks (DQN), not only improve learning in heterogeneous settings, but also provide better generalization to new environments.

Title: MSSF: A 4D Radar and Camera Fusion Framework With Multi-Stage Sampling for 3D Object Detection in Autonomous Driving

• Authors: Hongsi Liu, Jun Liu, Guangfeng Jiang, Xin Jin
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.15016
• Pdf URL: https://arxiv.org/pdf/2411.15016
• Copy Paste: [[2411.15016]] MSSF: A 4D Radar and Camera Fusion Framework With Multi-Stage Sampling for 3D Object Detection in Autonomous Driving(https://arxiv.org/abs/2411.15016)
• Keywords: robust, segmentation
• Abstract: As one of the automotive sensors that have emerged in recent years, 4D millimeter-wave radar has a higher resolution than conventional 3D radar and provides precise elevation measurements. But its point clouds are still sparse and noisy, making it challenging to meet the requirements of autonomous driving. Camera, as another commonly used sensor, can capture rich semantic information. As a result, the fusion of 4D radar and camera can provide an affordable and robust perception solution for autonomous driving systems. However, previous radar-camera fusion methods have not yet been thoroughly investigated, resulting in a large performance gap compared to LiDAR-based methods. Specifically, they ignore the feature-blurring problem and do not deeply interact with image semantic information. To this end, we present a simple but effective multi-stage sampling fusion (MSSF) network based on 4D radar and camera. On the one hand, we design a fusion block that can deeply interact point cloud features with image features, and can be applied to commonly used single-modal backbones in a plug-and-play manner. The fusion block encompasses two types, namely, simple feature fusion (SFF) and multiscale deformable feature fusion (MSDFF). The SFF is easy to implement, while the MSDFF has stronger fusion abilities. On the other hand, we propose a semantic-guided head to perform foreground-background segmentation on voxels with voxel feature re-weighting, further alleviating the problem of feature blurring. Extensive experiments on the View-of-Delft (VoD) and TJ4DRadset datasets demonstrate the effectiveness of our MSSF. Notably, compared to state-of-the-art methods, MSSF achieves a 7.0% and 4.0% improvement in 3D mean average precision on the VoD and TJ4DRadSet datasets, respectively. It even surpasses classical LiDAR-based methods on the VoD dataset.

Title: ZT-SDN: An ML-powered Zero-Trust Architecture for Software-Defined Networks

• Authors: Charalampos Katsis, Elisa Bertino
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.15020
• Pdf URL: https://arxiv.org/pdf/2411.15020
• Copy Paste: [[2411.15020]] ZT-SDN: An ML-powered Zero-Trust Architecture for Software-Defined Networks(https://arxiv.org/abs/2411.15020)
• Keywords: security, attack
• Abstract: Zero Trust (ZT) is a security paradigm aiming to curtail an attacker's lateral movements within a network by implementing least-privilege and per-request access control policies. However, its widespread adoption is hindered by the difficulty of generating proper rules due to the lack of detailed knowledge of communication requirements and the characteristic behaviors of communicating entities under benign conditions. Consequently, manual rule generation becomes cumbersome and error-prone. To address these problems, we propose ZT-SDN, an automated framework for learning and enforcing network access control in Software-Defined Networks. ZT-SDN collects data from the underlying network and models the network "transactions" performed by communicating entities as graphs. The nodes represent entities, while the directed edges represent transactions identified by different protocol stacks observed. It uses novel unsupervised learning approaches to extract transaction patterns directly from the network data, such as the allowed protocol stacks and port numbers and data transmission behavior. Finally, ZT-SDN uses an innovative approach to generate correct access control rules and infer strong associations between them, allowing proactive rule deployment in forwarding devices. We show the framework's efficacy in detecting abnormal network accesses and abuses of permitted flows in changing network conditions with real network datasets. Additionally, we showcase ZT-SDN's scalability and the network's performance when applied in an SDN environment.

Title: DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models

• Authors: Keda Tao, Can Qin, Haoxuan You, Yang Sui, Huan Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.15024
• Pdf URL: https://arxiv.org/pdf/2411.15024
• Copy Paste: [[2411.15024]] DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models(https://arxiv.org/abs/2411.15024)
• Keywords: large language model
• Abstract: Video large language models (VLLMs) have significantly advanced recently in processing complex video content, yet their inference efficiency remains constrained because of the high computational cost stemming from the thousands of visual tokens generated from the video inputs. We empirically observe that, unlike single image inputs, VLLMs typically attend visual tokens from different frames at different decoding iterations, making a one-shot pruning strategy prone to removing important tokens by mistake. Motivated by this, we present DyCoke, a training-free token compression method to optimize token representation and accelerate VLLMs. DyCoke incorporates a plug-and-play temporal compression module to minimize temporal redundancy by merging redundant tokens across frames, and applies dynamic KV cache reduction to prune spatially redundant tokens selectively. It ensures high-quality inference by dynamically retaining the critical tokens at each decoding step. Extensive experimental results demonstrate that DyCoke can outperform the prior SoTA counterparts, achieving 1.5X inference speedup, 1.4X memory reduction against the baseline VLLM, while still improving the performance, with no training.

Title: FloAt: Flow Warping of Self-Attention for Clothing Animation Generation

• Authors: Swasti Shreya Mishra, Kuldeep Kulkarni, Duygu Ceylan, Balaji Vasan Srinivasan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.15028
• Pdf URL: https://arxiv.org/pdf/2411.15028
• Copy Paste: [[2411.15028]] FloAt: Flow Warping of Self-Attention for Clothing Animation Generation(https://arxiv.org/abs/2411.15028)
• Keywords: diffusion
• Abstract: We propose a diffusion model-based approach, FloAtControlNet to generate cinemagraphs composed of animations of human clothing. We focus on human clothing like dresses, skirts and pants. The input to our model is a text prompt depicting the type of clothing and the texture of clothing like leopard, striped, or plain, and a sequence of normal maps that capture the underlying animation that we desire in the output. The backbone of our method is a normal-map conditioned ControlNet which is operated in a training-free regime. The key observation is that the underlying animation is embedded in the flow of the normal maps. We utilize the flow thus obtained to manipulate the self-attention maps of appropriate layers. Specifically, the self-attention maps of a particular layer and frame are recomputed as a linear combination of itself and the self-attention maps of the same layer and the previous frame, warped by the flow on the normal maps of the two frames. We show that manipulating the self-attention maps greatly enhances the quality of the clothing animation, making it look more natural as well as suppressing the background artifacts. Through extensive experiments, we show that the method proposed beats all baselines both qualitatively in terms of visual results and user study. Specifically, our method is able to alleviate the background flickering that exists in other diffusion model-based baselines that we consider. In addition, we show that our method beats all baselines in terms of RMSE and PSNR computed using the input normal map sequences and the normal map sequences obtained from the output RGB frames. Further, we show that well-established evaluation metrics like LPIPS, SSIM, and CLIP scores that are generally for visual quality are not necessarily suitable for capturing the subtle motions in human clothing animations.

Title: HeadRouter: A Training-free Image Editing Framework for MM-DiTs by Adaptively Routing Attention Heads

• Authors: Yu Xu, Fan Tang, Juan Cao, Yuxin Zhang, Xiaoyu Kong, Jintao Li, Oliver Deussen, Tong-Yee Lee
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.15034
• Pdf URL: https://arxiv.org/pdf/2411.15034
• Copy Paste: [[2411.15034]] HeadRouter: A Training-free Image Editing Framework for MM-DiTs by Adaptively Routing Attention Heads(https://arxiv.org/abs/2411.15034)
• Keywords: robust, diffusion, transformer
• Abstract: Diffusion Transformers (DiTs) have exhibited robust capabilities in image generation tasks. However, accurate text-guided image editing for multimodal DiTs (MM-DiTs) still poses a significant challenge. Unlike UNet-based structures that could utilize self/cross-attention maps for semantic editing, MM-DiTs inherently lack support for explicit and consistent incorporated text guidance, resulting in semantic misalignment between the edited results and texts. In this study, we disclose the sensitivity of different attention heads to different image semantics within MM-DiTs and introduce HeadRouter, a training-free image editing framework that edits the source image by adaptively routing the text guidance to different attention heads in MM-DiTs. Furthermore, we present a dual-token refinement module to refine text/image token representations for precise semantic guidance and accurate region expression. Experimental results on multiple benchmarks demonstrate HeadRouter's performance in terms of editing fidelity and image quality.

Title: OVO-SLAM: Open-Vocabulary Online Simultaneous Localization and Mapping

• Authors: Tomas Berriel Martins, Martin R. Oswald, Javier Civera
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.15043
• Pdf URL: https://arxiv.org/pdf/2411.15043
• Copy Paste: [[2411.15043]] OVO-SLAM: Open-Vocabulary Online Simultaneous Localization and Mapping(https://arxiv.org/abs/2411.15043)
• Keywords: segmentation
• Abstract: This paper presents the first Open-Vocabulary Online 3D semantic SLAM pipeline, that we denote as OVO-SLAM. Our primary contribution is in the pipeline itself, particularly in the mapping thread. Given a set of posed RGB-D frames, we detect and track 3D segments, which we describe using CLIP vectors, calculated through a novel aggregation from the viewpoints where these 3D segments are observed. Notably, our OVO-SLAM pipeline is not only faster but also achieves better segmentation metrics compared to offline approaches in the literature. Along with superior segmentation performance, we show experimental results of our contributions integrated with Gaussian-SLAM, being the first ones demonstrating end-to-end open-vocabulary online 3D reconstructions without relying on ground-truth camera poses or scene geometry.

Title: Fantastic Biases (What are They) and Where to Find Them

• Authors: Valentin Barriere
• Subjects: cs.CL, cs.CV, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.15051
• Pdf URL: https://arxiv.org/pdf/2411.15051
• Copy Paste: [[2411.15051]] Fantastic Biases (What are They) and Where to Find Them(https://arxiv.org/abs/2411.15051)
• Keywords: fair
• Abstract: Deep Learning models tend to learn correlations of patterns on huge datasets. The bigger these systems are, the more complex are the phenomena they can detect, and the more data they need for this. The use of Artificial Intelligence (AI) is becoming increasingly ubiquitous in our society, and its impact is growing everyday. The promises it holds strongly depend on their fair and universal use, such as access to information or education for all. In a world of inequalities, they can help to reach the most disadvantaged areas. However, such a universal systems must be able to represent society, without benefiting some at the expense of others. We must not reproduce the inequalities observed throughout the world, but educate these IAs to go beyond them. We have seen cases where these systems use gender, race, or even class information in ways that are not appropriate for resolving their tasks. Instead of real causal reasoning, they rely on spurious correlations, which is what we usually call a bias. In this paper, we first attempt to define what is a bias in general terms. It helps us to demystify the concept of bias, to understand why we can find them everywhere and why they are sometimes useful. Second, we focus over the notion of what is generally seen as negative bias, the one we want to avoid in machine learning, before presenting a general zoology containing the most common of these biases. We finally conclude by looking at classical methods to detect them, by means of specially crafted datasets of templates and specific algorithms, and also classical methods to mitigate them.

Title: Instance-Aware Generalized Referring Expression Segmentation

• Authors: E-Ro Nguyen, Hieu Le, Dimitris Samaras, Michael Ryoo
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.15087
• Pdf URL: https://arxiv.org/pdf/2411.15087
• Copy Paste: [[2411.15087]] Instance-Aware Generalized Referring Expression Segmentation(https://arxiv.org/abs/2411.15087)
• Keywords: segmentation
• Abstract: Recent works on Generalized Referring Expression Segmentation (GRES) struggle with handling complex expressions referring to multiple distinct objects. This is because these methods typically employ an end-to-end foreground-background segmentation and lack a mechanism to explicitly differentiate and associate different object instances to the text query. To this end, we propose InstAlign, a method that incorporates object-level reasoning into the segmentation process. Our model leverages both text and image inputs to extract a set of object-level tokens that capture both the semantic information in the input prompt and the objects within the image. By modeling the text-object alignment via instance-level supervision, each token uniquely represents an object segment in the image, while also aligning with relevant semantic information from the text. Extensive experiments on the gRefCOCO and Ref-ZOM benchmarks demonstrate that our method significantly advances state-of-the-art performance, setting a new standard for precise and flexible GRES.

Title: RED: Effective Trajectory Representation Learning with Comprehensive Information

• Authors: Silin Zhou, Shuo Shang, Lisi Chen, Christian S. Jensen, Panos Kalnis
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.15096
• Pdf URL: https://arxiv.org/pdf/2411.15096
• Copy Paste: [[2411.15096]] RED: Effective Trajectory Representation Learning with Comprehensive Information(https://arxiv.org/abs/2411.15096)
• Keywords: transformer
• Abstract: Trajectory representation learning (TRL) maps trajectories to vectors that can then be used for various downstream tasks, including trajectory similarity computation, trajectory classification, and travel-time estimation. However, existing TRL methods often produce vectors that, when used in downstream tasks, yield insufficiently accurate results. A key reason is that they fail to utilize the comprehensive information encompassed by trajectories. We propose a self-supervised TRL framework, called RED, which effectively exploits multiple types of trajectory information. Overall, RED adopts the Transformer as the backbone model and masks the constituting paths in trajectories to train a masked autoencoder (MAE). In particular, RED considers the moving patterns of trajectories by employing a Road-aware masking strategy} that retains key paths of trajectories during masking, thereby preserving crucial information of the trajectories. RED also adopts a spatial-temporal-user joint Embedding scheme to encode comprehensive information when preparing the trajectories as model inputs. To conduct training, RED adopts Dual-objective task learning}: the Transformer encoder predicts the next segment in a trajectory, and the Transformer decoder reconstructs the entire trajectory. RED also considers the spatial-temporal correlations of trajectories by modifying the attention mechanism of the Transformer. We compare RED with 9 state-of-the-art TRL methods for 4 downstream tasks on 3 real-world datasets, finding that RED can usually improve the accuracy of the best-performing baseline by over 5%.

Title: OminiControl: Minimal and Universal Control for Diffusion Transformer

• Authors: Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, Xinchao Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.15098
• Pdf URL: https://arxiv.org/pdf/2411.15098
• Copy Paste: [[2411.15098]] OminiControl: Minimal and Universal Control for Diffusion Transformer(https://arxiv.org/abs/2411.15098)
• Keywords: diffusion, transformer
• Abstract: In this paper, we introduce OminiControl, a highly versatile and parameter-efficient framework that integrates image conditions into pre-trained Diffusion Transformer (DiT) models. At its core, OminiControl leverages a parameter reuse mechanism, enabling the DiT to encode image conditions using itself as a powerful backbone and process them with its flexible multi-modal attention processors. Unlike existing methods, which rely heavily on additional encoder modules with complex architectures, OminiControl (1) effectively and efficiently incorporates injected image conditions with only ~0.1% additional parameters, and (2) addresses a wide range of image conditioning tasks in a unified manner, including subject-driven generation and spatially-aligned conditions such as edges, depth, and more. Remarkably, these capabilities are achieved by training on images generated by the DiT itself, which is particularly beneficial for subject-driven generation. Extensive evaluations demonstrate that OminiControl outperforms existing UNet-based and DiT-adapted models in both subject-driven and spatially-aligned conditional generation. Additionally, we release our training dataset, Subjects200K, a diverse collection of over 200,000 identity-consistent images, along with an efficient data synthesis pipeline to advance research in subject-consistent generation.

Title: XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models

• Authors: Yixin Dong, Charlie F. Ruan, Yaxing Cai, Ruihang Lai, Ziyi Xu, Yilong Zhao, Tianqi Chen
• Subjects: cs.CL, cs.AI, cs.PL
• Abstract URL: https://arxiv.org/abs/2411.15100
• Pdf URL: https://arxiv.org/pdf/2411.15100
• Copy Paste: [[2411.15100]] XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models(https://arxiv.org/abs/2411.15100)
• Keywords: large language model
• Abstract: The applications of LLM Agents are becoming increasingly complex and diverse, leading to a high demand for structured outputs that can be parsed into code, structured function calls, and embodied agent commands. These developments bring significant demands for structured generation in LLM inference. Context-free grammar is a flexible approach to enable structured generation via constrained decoding. However, executing context-free grammar requires going through several stack states over all tokens in vocabulary during runtime, bringing non-negligible overhead for structured generation. In this paper, we propose XGrammar, a flexible and efficient structure generation engine for large language models. XGrammar accelerates context-free grammar execution by dividing the vocabulary into context-independent tokens that can be prechecked and context-dependent tokens that need to be interpreted during runtime. We further build transformations to expand the grammar context and reduce the number of context-independent tokens. Additionally, we build an efficient persistent stack to accelerate the context-dependent token checks. Finally, we co-design the grammar engine with LLM inference engine to overlap grammar computation with GPU executions. Evaluation results show that XGrammar can achieve up to 100x speedup over existing solutions. Combined with an LLM inference engine, it can generate near-zero overhead structure generation in end-to-end low-LLM serving.

Title: AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution

• Authors: Fengyuan Liu, Nikhil Kandpal, Colin Raffel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.15102
• Pdf URL: https://arxiv.org/pdf/2411.15102
• Copy Paste: [[2411.15102]] AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution(https://arxiv.org/abs/2411.15102)
• Keywords: interpretability, large language model
• Abstract: The influence of contextual input on the behavior of large language models (LLMs) has prompted the development of context attribution methods that aim to quantify each context span's effect on an LLM's generations. The leave-one-out (LOO) error, which measures the change in the likelihood of the LLM's response when a given span of the context is removed, provides a principled way to perform context attribution, but can be prohibitively expensive to compute for large models. In this work, we introduce AttriBoT, a series of novel techniques for efficiently computing an approximation of the LOO error for context attribution. Specifically, AttriBoT uses cached activations to avoid redundant operations, performs hierarchical attribution to reduce computation, and emulates the behavior of large target models with smaller proxy models. Taken together, AttriBoT can provide a >300x speedup while remaining more faithful to a target model's LOO error than prior context attribution methods. This stark increase in performance makes computing context attributions for a given response 30x faster than generating the response itself, empowering real-world applications that require computing attributions at scale. We release a user-friendly and efficient implementation of AttriBoT to enable efficient LLM interpretability as well as encourage future development of efficient context attribution methods.

Title: A Real-Time DETR Approach to Bangladesh Road Object Detection for Autonomous Vehicles

• Authors: Irfan Nafiz Shahan, Arban Hossain, Saadman Sakib, Al-Mubin Nabil
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.15110
• Pdf URL: https://arxiv.org/pdf/2411.15110
• Copy Paste: [[2411.15110]] A Real-Time DETR Approach to Bangladesh Road Object Detection for Autonomous Vehicles(https://arxiv.org/abs/2411.15110)
• Keywords: transformer
• Abstract: In the recent years, we have witnessed a paradigm shift in the field of Computer Vision, with the forthcoming of the transformer architecture. Detection Transformers has become a state of the art solution to object detection and is a potential candidate for Road Object Detection in Autonomous Vehicles. Despite the abundance of object detection schemes, real-time DETR models are shown to perform significantly better on inference times, with minimal loss of accuracy and performance. In our work, we used Real-Time DETR (RTDETR) object detection on the BadODD Road Object Detection dataset based in Bangladesh, and performed necessary experimentation and testing. Our results gave a mAP50 score of 0.41518 in the public 60% test set, and 0.28194 in the private 40% test set.

Title: Efficient Pruning of Text-to-Image Models: Insights from Pruning Stable Diffusion

• Authors: Samarth N Ramesh, Zhixue Zhao
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.15113
• Pdf URL: https://arxiv.org/pdf/2411.15113
• Copy Paste: [[2411.15113]] Efficient Pruning of Text-to-Image Models: Insights from Pruning Stable Diffusion(https://arxiv.org/abs/2411.15113)
• Keywords: diffusion
• Abstract: As text-to-image models grow increasingly powerful and complex, their burgeoning size presents a significant obstacle to widespread adoption, especially on resource-constrained devices. This paper presents a pioneering study on post-training pruning of Stable Diffusion 2, addressing the critical need for model compression in text-to-image domain. Our study tackles the pruning techniques for the previously unexplored multi-modal generation models, and particularly examines the pruning impact on the textual component and the image generation component separately. We conduct a comprehensive comparison on pruning the model or the single component of the model in various sparsities. Our results yield previously undocumented findings. For example, contrary to established trends in language model pruning, we discover that simple magnitude pruning outperforms more advanced techniques in text-to-image context. Furthermore, our results show that Stable Diffusion 2 can be pruned to 38.5% sparsity with minimal quality loss, achieving a significant reduction in model size. We propose an optimal pruning configuration that prunes the text encoder to 47.5% and the diffusion generator to 35%. This configuration maintains image generation quality while substantially reducing computational requirements. In addition, our work uncovers intriguing questions about information encoding in text-to-image models: we observe that pruning beyond certain thresholds leads to sudden performance drops (unreadable images), suggesting that specific weights encode critical semantics information. This finding opens new avenues for future research in model compression, interoperability, and bias identification in text-to-image models. By providing crucial insights into the pruning behavior of text-to-image models, our study lays the groundwork for developing more efficient and accessible AI-driven image generation systems

Title: VideoRepair: Improving Text-to-Video Generation via Misalignment Evaluation and Localized Refinement

• Authors: Daeun Lee, Jaehong Yoon, Jaemin Cho, Mohit Bansal
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.15115
• Pdf URL: https://arxiv.org/pdf/2411.15115
• Copy Paste: [[2411.15115]] VideoRepair: Improving Text-to-Video Generation via Misalignment Evaluation and Localized Refinement(https://arxiv.org/abs/2411.15115)
• Keywords: diffusion
• Abstract: Recent text-to-video (T2V) diffusion models have demonstrated impressive generation capabilities across various domains. However, these models often generate videos that have misalignments with text prompts, especially when the prompts describe complex scenes with multiple objects and attributes. To address this, we introduce VideoRepair, a novel model-agnostic, training-free video refinement framework that automatically identifies fine-grained text-video misalignments and generates explicit spatial and textual feedback, enabling a T2V diffusion model to perform targeted, localized refinements. VideoRepair consists of four stages: In (1) video evaluation, we detect misalignments by generating fine-grained evaluation questions and answering those questions with MLLM. In (2) refinement planning, we identify accurately generated objects and then create localized prompts to refine other areas in the video. Next, in (3) region decomposition, we segment the correctly generated area using a combined grounding module. We regenerate the video by adjusting the misaligned regions while preserving the correct regions in (4) localized refinement. On two popular video generation benchmarks (EvalCrafter and T2V-CompBench), VideoRepair substantially outperforms recent baselines across various text-video alignment metrics. We provide a comprehensive analysis of VideoRepair components and qualitative examples.

Title: ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation

• Authors: Xiaoman Zhang, Hong-Yu Zhou, Xiaoli Yang, Oishi Banerjee, Julián N. Acosta, Josh Miller, Ouwen Huang, Pranav Rajpurkar
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.15122
• Pdf URL: https://arxiv.org/pdf/2411.15122
• Copy Paste: [[2411.15122]] ReXrank: A Public Leaderboard for AI-Powered Radiology Report Generation(https://arxiv.org/abs/2411.15122)
• Keywords: robust
• Abstract: AI-driven models have demonstrated significant potential in automating radiology report generation for chest X-rays. However, there is no standardized benchmark for objectively evaluating their performance. To address this, we present ReXrank, this https URL, a public leaderboard and challenge for assessing AI-powered radiology report generation. Our framework incorporates ReXGradient, the largest test dataset consisting of 10,000 studies, and three public datasets (MIMIC-CXR, IU-Xray, CheXpert Plus) for report generation assessment. ReXrank employs 8 evaluation metrics and separately assesses models capable of generating only findings sections and those providing both findings and impressions sections. By providing this standardized evaluation framework, ReXrank enables meaningful comparisons of model performance and offers crucial insights into their robustness across diverse clinical settings. Beyond its current focus on chest X-rays, ReXrank's framework sets the stage for comprehensive evaluation of automated reporting across the full spectrum of medical imaging.

Title: T\"ULU 3: Pushing Frontiers in Open Language Model Post-Training

• Authors: Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, Hannaneh Hajishirzi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.15124
• Pdf URL: https://arxiv.org/pdf/2411.15124
• Copy Paste: [[2411.15124]] T\"ULU 3: Pushing Frontiers in Open Language Model Post-Training(https://arxiv.org/abs/2411.15124)
• Keywords: robust
• Abstract: Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques lag behind proprietary ones. The underlying training data and recipes for post-training are simultaneously the most important pieces of the puzzle and the portion with the least transparency. To bridge this gap, we introduce TÜLU 3, a family of fully-open state-of-the-art post-trained models, alongside its data, code, and training recipes, serving as a comprehensive guide for modern post-training techniques. TÜLU 3, which builds on Llama 3.1 base models, achieves results surpassing the instruct versions of Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and Claude 3.5-Haiku. The training algorithms for our models include supervised finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR). With TÜLU 3, we introduce a multi-task evaluation scheme for post-training recipes with development and unseen evaluations, standard benchmark implementations, and substantial decontamination of existing open datasets on said benchmarks. We conclude with analysis and discussion of training methods that did not reliably improve performance. In addition to the TÜLU 3 model weights and demo, we release the complete recipe -- including datasets for diverse core skills, a robust toolkit for data curation and evaluation, the training code and infrastructure, and, most importantly, a detailed report for reproducing and further adapting the TÜLU 3 approach to more domains.

Title: Health AI Developer Foundations

• Authors: Atilla P. Kiraly, Sebastien Baur, Kenneth Philbrick, Fereshteh Mahvar, Liron Yatziv, Tiffany Chen, Bram Sterling, Nick George, Fayaz Jamil, Jing Tang, Kai Bailey, Faruk Ahmed, Akshay Goel, Abbi Ward, Lin Yang, Andrew Sellergren, Yossi Matias, Avinatan Hassidim, Shravya Shetty, Daniel Golden, Shekoofeh Azizi, David F. Steiner, Yun Liu, Tim Thelin, Rory Pilgrim, Can Kirmizibayrak
• Subjects: cs.LG, cs.AI, cs.CV, cs.MM, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.15128
• Pdf URL: https://arxiv.org/pdf/2411.15128
• Copy Paste: [[2411.15128]] Health AI Developer Foundations(https://arxiv.org/abs/2411.15128)
• Keywords: robust, fair
• Abstract: Robust medical Machine Learning (ML) models have the potential to revolutionize healthcare by accelerating clinical research, improving workflows and outcomes, and producing novel insights or capabilities. Developing such ML models from scratch is cost prohibitive and requires substantial compute, data, and time (e.g., expert labeling). To address these challenges, we introduce Health AI Developer Foundations (HAI-DEF), a suite of pre-trained, domain-specific foundation models, tools, and recipes to accelerate building ML for health applications. The models cover various modalities and domains, including radiology (X-rays and computed tomography), histopathology, dermatological imaging, and audio. These models provide domain specific embeddings that facilitate AI development with less labeled data, shorter training times, and reduced computational costs compared to traditional approaches. In addition, we utilize a common interface and style across these models, and prioritize usability to enable developers to integrate HAI-DEF efficiently. We present model evaluations across various tasks and conclude with a discussion of their application and evaluation, covering the importance of ensuring efficacy, fairness, and equity. Finally, while HAI-DEF and specifically the foundation models lower the barrier to entry for ML in healthcare, we emphasize the importance of validation with problem- and population-specific data for each desired usage setting. This technical report will be updated over time as more modalities and features are added.

Title: Measuring Bullshit in the Language Games played by ChatGPT

• Authors: Alessandro Trevisan, Harry Giddens, Sarah Dillon, Alan F. Blackwell
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.15129
• Pdf URL: https://arxiv.org/pdf/2411.15129
• Copy Paste: [[2411.15129]] Measuring Bullshit in the Language Games played by ChatGPT(https://arxiv.org/abs/2411.15129)
• Keywords: generative, large language model
• Abstract: Generative large language models (LLMs), which create text without direct correspondence to truth value, are widely understood to resemble the uses of language described in Frankfurt's popular monograph On Bullshit. In this paper, we offer a rigorous investigation of this topic, identifying how the phenomenon has arisen, and how it might be analysed. In this paper, we elaborate on this argument to propose that LLM-based chatbots play the 'language game of bullshit'. We use statistical text analysis to investigate the features of this Wittgensteinian language game, based on a dataset constructed to contrast the language of 1,000 scientific publications with typical pseudo-scientific text generated by ChatGPT. We then explore whether the same language features can be detected in two well-known contexts of social dysfunction: George Orwell's critique of politics and language, and David Graeber's characterisation of bullshit jobs. Using simple hypothesis-testing methods, we demonstrate that a statistical model of the language of bullshit can reliably relate the Frankfurtian artificial bullshit of ChatGPT to the political and workplace functions of bullshit as observed in natural human language.

Title: Material Anything: Generating Materials for Any 3D Object via Diffusion

• Authors: Xin Huang, Tengfei Wang, Ziwei Liu, Qing Wang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.15138
• Pdf URL: https://arxiv.org/pdf/2411.15138
• Copy Paste: [[2411.15138]] Material Anything: Generating Materials for Any 3D Object via Diffusion(https://arxiv.org/abs/2411.15138)
• Keywords: robust, diffusion
• Abstract: We present Material Anything, a fully-automated, unified diffusion framework designed to generate physically-based materials for 3D objects. Unlike existing methods that rely on complex pipelines or case-specific optimizations, Material Anything offers a robust, end-to-end solution adaptable to objects under diverse lighting conditions. Our approach leverages a pre-trained image diffusion model, enhanced with a triple-head architecture and rendering loss to improve stability and material quality. Additionally, we introduce confidence masks as a dynamic switcher within the diffusion model, enabling it to effectively handle both textured and texture-less objects across varying lighting conditions. By employing a progressive material generation strategy guided by these confidence masks, along with a UV-space material refiner, our method ensures consistent, UV-ready material outputs. Extensive experiments demonstrate our approach outperforms existing methods across a wide range of object categories and lighting conditions.

Title: DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving

• Authors: Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, Xinggang Wang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.15139
• Pdf URL: https://arxiv.org/pdf/2411.15139
• Copy Paste: [[2411.15139]] DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving(https://arxiv.org/abs/2411.15139)
• Keywords: robust, diffusion, generative
• Abstract: Recently, the diffusion model has emerged as a powerful generative technique for robotic policy learning, capable of modeling multi-mode action distributions. Leveraging its capability for end-to-end autonomous driving is a promising direction. However, the numerous denoising steps in the robotic diffusion policy and the more dynamic, open-world nature of traffic scenes pose substantial challenges for generating diverse driving actions at a real-time speed. To address these challenges, we propose a novel truncated diffusion policy that incorporates prior multi-mode anchors and truncates the diffusion schedule, enabling the model to learn denoising from anchored Gaussian distribution to the multi-mode driving action distribution. Additionally, we design an efficient cascade diffusion decoder for enhanced interaction with conditional scene context. The proposed model, DiffusionDrive, demonstrates 10$\times$ reduction in denoising steps compared to vanilla diffusion policy, delivering superior diversity and quality in just 2 steps. On the planning-oriented NAVSIM dataset, with the aligned ResNet-34 backbone, DiffusionDrive achieves 88.1 PDMS without bells and whistles, setting a new record, while running at a real-time speed of 45 FPS on an NVIDIA 4090. Qualitative results on challenging scenarios further confirm that DiffusionDrive can robustly generate diverse plausible driving actions. Code and model will be available at this https URL.