2024-09-13

Title: Multi-Modal Instruction-Tuning Small-Scale Language-and-Vision Assistant for Semiconductor Electron Micrograph Analysis

• Authors: Sakhinana Sagar Srinivas, Geethan Sannidhi, Venkataramana Runkana
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07463
• Pdf URL: https://arxiv.org/pdf/2409.07463
• Copy Paste: [[2409.07463]] Multi-Modal Instruction-Tuning Small-Scale Language-and-Vision Assistant for Semiconductor Electron Micrograph Analysis(https://arxiv.org/abs/2409.07463)
• Keywords: secure, large language model
• Abstract: We present a novel framework for analyzing and interpreting electron microscopy images in semiconductor manufacturing using vision-language instruction tuning. The framework employs a unique teacher-student approach, leveraging pre-trained multimodal large language models such as GPT-4 to generate instruction-following data for zero-shot visual question answering (VQA) and classification tasks, customizing smaller multimodal models (SMMs) for microscopy image analysis, resulting in an instruction-tuned language-and-vision assistant. Our framework merges knowledge engineering with machine learning to integrate domain-specific expertise from larger to smaller multimodal models within this specialized field, greatly reducing the need for extensive human labeling. Our study presents a secure, cost-effective, and customizable approach for analyzing microscopy images, addressing the challenges of adopting proprietary models in semiconductor manufacturing.

Title: Polynomial Methods for Ensuring Data Integrity in Financial Systems

• Authors: Ignacio Brasca
• Subjects: cs.CR, cs.IT, math.NA
• Abstract URL: https://arxiv.org/abs/2409.07490
• Pdf URL: https://arxiv.org/pdf/2409.07490
• Copy Paste: [[2409.07490]] Polynomial Methods for Ensuring Data Integrity in Financial Systems(https://arxiv.org/abs/2409.07490)
• Keywords: robust
• Abstract: Ensuring data integrity is a critical requirement in complex systems, especially in financial platforms where vast amounts of data must be consistently accurate and reliable. This paper presents a robust approach using polynomial interpolation methods to maintain data integrity across multiple indicators and dimensions.

Title: DV-FSR: A Dual-View Target Attack Framework for Federated Sequential Recommendation

• Authors: Qitao Qin, Yucong Luo, Mingyue Cheng, Qingyang Mao, Chenyi Lei
• Subjects: cs.CR, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.07500
• Pdf URL: https://arxiv.org/pdf/2409.07500
• Copy Paste: [[2409.07500]] DV-FSR: A Dual-View Target Attack Framework for Federated Sequential Recommendation(https://arxiv.org/abs/2409.07500)
• Keywords: privacy, defense, attack, robust, federate
• Abstract: Federated recommendation (FedRec) preserves user privacy by enabling decentralized training of personalized models, but this architecture is inherently vulnerable to adversarial attacks. Significant research has been conducted on targeted attacks in FedRec systems, motivated by commercial and social influence considerations. However, much of this work has largely overlooked the differential robustness of recommendation models. Moreover, our empirical findings indicate that existing targeted attack methods achieve only limited effectiveness in Federated Sequential Recommendation (FSR) tasks. Driven by these observations, we focus on investigating targeted attacks in FSR and propose a novel dualview attack framework, named DV-FSR. This attack method uniquely combines a sampling-based explicit strategy with a contrastive learning-based implicit gradient strategy to orchestrate a coordinated attack. Additionally, we introduce a specific defense mechanism tailored for targeted attacks in FSR, aiming to evaluate the mitigation effects of the attack method we proposed. Extensive experiments validate the effectiveness of our proposed approach on representative sequential models.

Title: AdaPPA: Adaptive Position Pre-Fill Jailbreak Attack Approach Targeting LLMs

• Authors: Lijia Lv, Weigang Zhang, Xuehai Tang, Jie Wen, Feng Liu, Jizhong Han, Songlin Hu
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07503
• Pdf URL: https://arxiv.org/pdf/2409.07503
• Copy Paste: [[2409.07503]] AdaPPA: Adaptive Position Pre-Fill Jailbreak Attack Approach Targeting LLMs(https://arxiv.org/abs/2409.07503)
• Keywords: secure, protect, attack, large language model
• Abstract: Jailbreak vulnerabilities in Large Language Models (LLMs) refer to methods that extract malicious content from the model by carefully crafting prompts or suffixes, which has garnered significant attention from the research community. However, traditional attack methods, which primarily focus on the semantic level, are easily detected by the model. These methods overlook the difference in the model's alignment protection capabilities at different output stages. To address this issue, we propose an adaptive position pre-fill jailbreak attack approach for executing jailbreak attacks on LLMs. Our method leverages the model's instruction-following capabilities to first output pre-filled safe content, then exploits its narrative-shifting abilities to generate harmful content. Extensive black-box experiments demonstrate our method can improve the attack success rate by 47% on the widely recognized secure model (Llama2) compared to existing approaches. Our code can be found at: this https URL.

Title: A Survey of Anomaly Detection in In-Vehicle Networks

• Authors: Övgü Özdemir, M. Tuğberk İşyapar, Pınar Karagöz, Klaus Werner Schmidt, Demet Demir, N. Alpay Karagöz
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.07505
• Pdf URL: https://arxiv.org/pdf/2409.07505
• Copy Paste: [[2409.07505]] A Survey of Anomaly Detection in In-Vehicle Networks(https://arxiv.org/abs/2409.07505)
• Keywords: attack
• Abstract: Modern vehicles are equipped with Electronic Control Units (ECU) that are used for controlling important vehicle functions including safety-critical operations. ECUs exchange information via in-vehicle communication buses, of which the Controller Area Network (CAN bus) is by far the most widespread representative. Problems that may occur in the vehicle's physical parts or malicious attacks may cause anomalies in the CAN traffic, impairing the correct vehicle operation. Therefore, the detection of such anomalies is vital for vehicle safety. This paper reviews the research on anomaly detection for in-vehicle networks, more specifically for the CAN bus. Our main focus is the evaluation of methods used for CAN bus anomaly detection together with the datasets used in such analysis. To provide the reader with a more comprehensive understanding of the subject, we first give a brief review of related studies on time series-based anomaly detection. Then, we conduct an extensive survey of recent deep learning-based techniques as well as conventional techniques for CAN bus anomaly detection. Our comprehensive analysis delves into anomaly detection algorithms employed in in-vehicle networks, specifically focusing on their learning paradigms, inherent strengths, and weaknesses, as well as their efficacy when applied to CAN bus datasets. Lastly, we highlight challenges and open research problems in CAN bus anomaly detection.

Title: SafeBPF: Hardware-assisted Defense-in-depth for eBPF Kernel Extensions

• Authors: Soo Yee Lim, Tanya Prasad, Xueyuan Han, Thomas Pasquier
• Subjects: cs.CR, cs.OS
• Abstract URL: https://arxiv.org/abs/2409.07508
• Pdf URL: https://arxiv.org/pdf/2409.07508
• Copy Paste: [[2409.07508]] SafeBPF: Hardware-assisted Defense-in-depth for eBPF Kernel Extensions(https://arxiv.org/abs/2409.07508)
• Keywords: security, defense
• Abstract: The eBPF framework enables execution of user-provided code in the Linux kernel. In the last few years, a large ecosystem of cloud services has leveraged eBPF to enhance container security, system observability, and network management. Meanwhile, incessant discoveries of memory safety vulnerabilities have left the systems community with no choice but to disallow unprivileged eBPF programs, which unfortunately limits eBPF use to only privileged users. To improve run-time safety of the framework, we introduce SafeBPF, a general design that isolates eBPF programs from the rest of the kernel to prevent memory safety vulnerabilities from being exploited. We present a pure software implementation using a Software-based Fault Isolation (SFI) approach and a hardware-assisted implementation that leverages ARM's Memory Tagging Extension (MTE). We show that SafeBPF incurs up to 4% overhead on macrobenchmarks while achieving desired security properties.

Title: ENACT: Entropy-based Clustering of Attention Input for Improving the Computational Performance of Object Detection Transformers

• Authors: Giorgos Savathrakis, Antonis Argyros
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07541
• Pdf URL: https://arxiv.org/pdf/2409.07541
• Copy Paste: [[2409.07541]] ENACT: Entropy-based Clustering of Attention Input for Improving the Computational Performance of Object Detection Transformers(https://arxiv.org/abs/2409.07541)
• Keywords: transformer
• Abstract: Transformers demonstrate competitive performance in terms of precision on the problem of vision-based object detection. However, they require considerable computational resources due to the quadratic size of the attention weights. In this work, we propose to cluster the transformer input on the basis of its entropy. The reason for this is that the self-information of each pixel (whose sum is the entropy), is likely to be similar among pixels corresponding to the same objects. Clustering reduces the size of data given as input to the transformer and therefore reduces training time and GPU memory usage, while at the same time preserves meaningful information to be passed through the remaining parts of the network. The proposed process is organized in a module called ENACT, that can be plugged-in any transformer architecture that consists of a multi-head self-attention computation in its encoder. We ran extensive experiments using the COCO object detection dataset, and three detection transformers. The obtained results demonstrate that in all tested cases, there is consistent reduction in the required computational resources, while the precision of the detection task is only slightly reduced. The code of the ENACT module will become available at this https URL

Title: Unsupervised Point Cloud Registration with Self-Distillation

• Authors: Christian Löwens, Thorben Funke, André Wagner, Alexandru Paul Condurache
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.07558
• Pdf URL: https://arxiv.org/pdf/2409.07558
• Copy Paste: [[2409.07558]] Unsupervised Point Cloud Registration with Self-Distillation(https://arxiv.org/abs/2409.07558)
• Keywords: robust
• Abstract: Rigid point cloud registration is a fundamental problem and highly relevant in robotics and autonomous driving. Nowadays deep learning methods can be trained to match a pair of point clouds, given the transformation between them. However, this training is often not scalable due to the high cost of collecting ground truth poses. Therefore, we present a self-distillation approach to learn point cloud registration in an unsupervised fashion. Here, each sample is passed to a teacher network and an augmented view is passed to a student network. The teacher includes a trainable feature extractor and a learning-free robust solver such as RANSAC. The solver forces consistency among correspondences and optimizes for the unsupervised inlier ratio, eliminating the need for ground truth labels. Our approach simplifies the training procedure by removing the need for initial hand-crafted features or consecutive point cloud frames as seen in related methods. We show that our method not only surpasses them on the RGB-D benchmark 3DMatch but also generalizes well to automotive radar, where classical features adopted by others fail. The code is available at this https URL .

Title: EchoDFKD: Data-Free Knowledge Distillation for Cardiac Ultrasound Segmentation using Synthetic Data

• Authors: Grégoire Petit, Nathan Palluau, Axel Bauer, Clemens Dlaska
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07566
• Pdf URL: https://arxiv.org/pdf/2409.07566
• Copy Paste: [[2409.07566]] EchoDFKD: Data-Free Knowledge Distillation for Cardiac Ultrasound Segmentation using Synthetic Data(https://arxiv.org/abs/2409.07566)
• Keywords: data-free, generative, segmentation
• Abstract: The application of machine learning to medical ultrasound videos of the heart, i.e., echocardiography, has recently gained traction with the availability of large public datasets. Traditional supervised tasks, such as ejection fraction regression, are now making way for approaches focusing more on the latent structure of data distributions, as well as generative methods. We propose a model trained exclusively by knowledge distillation, either on real or synthetical data, involving retrieving masks suggested by a teacher model. We achieve state-of-the-art (SOTA) values on the task of identifying end-diastolic and end-systolic frames. By training the model only on synthetic data, it reaches segmentation capabilities close to the performance when trained on real data with a significantly reduced number of weights. A comparison with the 5 main existing methods shows that our method outperforms the others in most cases. We also present a new evaluation method that does not require human annotation and instead relies on a large auxiliary model. We show that this method produces scores consistent with those obtained from human annotations. Relying on the integrated knowledge from a vast amount of records, this method overcomes certain inherent limitations of human annotator labeling. Code: this https URL

Title: Cybersecurity Challenge Analysis of Work-from-Anywhere (WFA) and Recommendations guided by a User Study

• Authors: Mohammed Mahyoub, Ashraf Matrawy, Kamal Isleem, Olakunle Ibitoye
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07567
• Pdf URL: https://arxiv.org/pdf/2409.07567
• Copy Paste: [[2409.07567]] Cybersecurity Challenge Analysis of Work-from-Anywhere (WFA) and Recommendations guided by a User Study(https://arxiv.org/abs/2409.07567)
• Keywords: security, robust
• Abstract: Many organizations were forced to quickly transition to the work-from-anywhere (WFA) model as a necessity to continue with their operations and remain in business despite the restrictions imposed during the COVID-19 pandemic. Many decisions were made in a rush, and cybersecurity decency tools were not in place to support this transition. In this paper, we first attempt to uncover some challenges and implications related to the cybersecurity of the WFA model. Secondly, we conducted an online user study to investigate the readiness and cybersecurity awareness of employers and their employees who shifted to work remotely from anywhere. The user study questionnaire addressed different resilience perspectives of individuals and organizations. The collected data includes 45 responses from remotely working employees of different organizational types: universities, government, private, and non-profit organizations. Despite the importance of security training and guidelines, it was surprising that many participants had not received them. A robust communication strategy is necessary to ensure that employees are informed and updated on security incidents that the organization encounters. Additionally, there is an increased need to pay attention to the security-related attributes of employees, such as their behavior, awareness, and compliance. Finally, we outlined best practice recommendations and mitigation tips guided by the study results to help individuals and organizations resist cybercrime and fraud and mitigate WFA-related cybersecurity risks.

Title: FaVoR: Features via Voxel Rendering for Camera Relocalization

• Authors: Vincenzo Polizzi, Marco Cannici, Davide Scaramuzza, Jonathan Kelly
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.07571
• Pdf URL: https://arxiv.org/pdf/2409.07571
• Copy Paste: [[2409.07571]] FaVoR: Features via Voxel Rendering for Camera Relocalization(https://arxiv.org/abs/2409.07571)
• Keywords: robust
• Abstract: Camera relocalization methods range from dense image alignment to direct camera pose regression from a query image. Among these, sparse feature matching stands out as an efficient, versatile, and generally lightweight approach with numerous applications. However, feature-based methods often struggle with significant viewpoint and appearance changes, leading to matching failures and inaccurate pose estimates. To overcome this limitation, we propose a novel approach that leverages a globally sparse yet locally dense 3D representation of 2D features. By tracking and triangulating landmarks over a sequence of frames, we construct a sparse voxel map optimized to render image patch descriptors observed during tracking. Given an initial pose estimate, we first synthesize descriptors from the voxels using volumetric rendering and then perform feature matching to estimate the camera pose. This methodology enables the generation of descriptors for unseen views, enhancing robustness to view changes. We extensively evaluate our method on the 7-Scenes and Cambridge Landmarks datasets. Our results show that our method significantly outperforms existing state-of-the-art feature representation techniques in indoor environments, achieving up to a 39% improvement in median translation error. Additionally, our approach yields comparable results to other methods for outdoor scenarios while maintaining lower memory and computational costs.

Title: fence.t.s: Closing Timing Channels in High-Performance Out-of-Order Cores through ISA-Supported Temporal Partitioning

• Authors: Nils Wistoff, Gernot Heiser, Luca Benini
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07576
• Pdf URL: https://arxiv.org/pdf/2409.07576
• Copy Paste: [[2409.07576]] fence.t.s: Closing Timing Channels in High-Performance Out-of-Order Cores through ISA-Supported Temporal Partitioning(https://arxiv.org/abs/2409.07576)
• Keywords: security, protect
• Abstract: Microarchitectural timing channels exploit information leakage between security domains that should be isolated, bypassing the operating system's security boundaries. These channels result from contention for shared microarchitectural state. In the RISC-V instruction set, the temporal fence instruction (fence.t) was proposed to close timing channels by providing an operating system with the means to temporally partition microarchitectural state inexpensively in simple in-order cores. This work explores challenges with fence.t in superscalar out-of-order cores featuring large and pervasive microarchitectural state. To overcome these challenges, we propose a novel SW-supported temporal fence (fence.t.s), which reuses existing mechanisms and supports advanced microarchitectural features, enabling full timing channel protection of an exemplary out-of-order core (OpenC910) at negligible hardware costs and a minimal performance impact of 1.0 %.

Title: New constructions of pseudorandom codes

• Authors: Surendra Ghentiyala, Venkatesan Guruswami
• Subjects: cs.CR, cs.CC
• Abstract URL: https://arxiv.org/abs/2409.07580
• Pdf URL: https://arxiv.org/pdf/2409.07580
• Copy Paste: [[2409.07580]] New constructions of pseudorandom codes(https://arxiv.org/abs/2409.07580)
• Keywords: secure, security, robust, watermark, generative
• Abstract: Introduced in [CG24], pseudorandom error-correcting codes (PRCs) are a new cryptographic primitive with applications in watermarking generative AI models. These are codes where a collection of polynomially many codewords is computationally indistinguishable from random, except to individuals with the decoding key. In this work, we examine the assumptions under which PRCs with robustness to a constant error rate exist. 1. We show that if both the planted hyperloop assumption introduced in [BKR23] and security of a version of Goldreich's PRG hold, then there exist public-key PRCs for which no efficient adversary can distinguish a polynomial number of codewords from random with better than $o(1)$ advantage. 2. We revisit the construction of [CG24] and show that it can be based on a wider range of assumptions than presented in [CG24]. To do this, we introduce a weakened version of the planted XOR assumption which we call the weak planted XOR assumption and which may be of independent interest. 3. We initiate the study of PRCs which are secure against space-bounded adversaries. We show how to construct secret-key PRCs of length $O(n)$ which are $\textit{unconditionally}$ indistinguishable from random by $\text{poly}(n)$ time, $O(n^{1.5-\varepsilon})$ space adversaries.

Title: Minimizing Embedding Distortion for Robust Out-of-Distribution Performance

• Authors: Tom Shaked, Yuval Goldman, Oran Shayer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07582
• Pdf URL: https://arxiv.org/pdf/2409.07582
• Copy Paste: [[2409.07582]] Minimizing Embedding Distortion for Robust Out-of-Distribution Performance(https://arxiv.org/abs/2409.07582)
• Keywords: robust
• Abstract: Foundational models, trained on vast and diverse datasets, have demonstrated remarkable capabilities in generalizing across different domains and distributions for various zero-shot tasks. Our work addresses the challenge of retaining these powerful generalization capabilities when adapting foundational models to specific downstream tasks through fine-tuning. To this end, we introduce a novel approach we call "similarity loss", which can be incorporated into the fine-tuning process of any task. By minimizing the distortion of fine-tuned embeddings from the pre-trained embeddings, our method strikes a balance between task-specific adaptation and preserving broad generalization abilities. We evaluate our approach on two diverse tasks: image classification on satellite imagery and face recognition, focusing on open-class and domain shift scenarios to assess out-of-distribution (OOD) performance. We demonstrate that this approach significantly improves OOD performance while maintaining strong in-distribution (ID) performance.

Title: Analyzing the Impact of Copying-and-Pasting Vulnerable Solidity Code Snippets from Question-and-Answer Websites

• Authors: Konrad Weiss, Christof Ferreira Torres, Florian Wendland
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07586
• Pdf URL: https://arxiv.org/pdf/2409.07586
• Copy Paste: [[2409.07586]] Analyzing the Impact of Copying-and-Pasting Vulnerable Solidity Code Snippets from Question-and-Answer Websites(https://arxiv.org/abs/2409.07586)
• Keywords: attack
• Abstract: Ethereum smart contracts are executable programs deployed on a blockchain. Once deployed, they cannot be updated due to their inherent immutability. Moreover, they often manage valuable assets that are worth millions of dollars, making them attractive targets for attackers. The introduction of vulnerabilities in programs due to the reuse of vulnerable code posted on Q&A websites such as Stack Overflow is not a new issue. However, little effort has been made to analyze the extent of this issue on deployed smart contracts. In this paper, we conduct a study on the impact of vulnerable code reuse from Q&A websites during the development of smart contracts and provide tools uniquely fit to detect vulnerable code patterns in complete and incomplete Smart Contract code. This paper proposes a pattern-based vulnerability detection tool that is able to analyze code snippets (i.e., incomplete code) as well as full smart contracts based on the concept of code property graphs. We also propose a methodology that leverages fuzzy hashing to quickly detect code clones of vulnerable snippets among deployed smart contracts. Our results show that our vulnerability search, as well as our code clone detection, are comparable to state-of-the-art while being applicable to code snippets. Our large-scale study on 18,660 code snippets reveals that 4,596 of them are vulnerable, out of which 616 can be found in 7,852 deployed smart contracts. These results highlight that the reuse of vulnerable code snippets is indeed an issue in currently deployed smart contracts.

Title: Exploring LLMs for Malware Detection: Review, Framework Design, and Countermeasure Approaches

• Authors: Jamal Al-Karaki, Muhammad Al-Zafar Khan, Marwan Omar
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07587
• Pdf URL: https://arxiv.org/pdf/2409.07587
• Copy Paste: [[2409.07587]] Exploring LLMs for Malware Detection: Review, Framework Design, and Countermeasure Approaches(https://arxiv.org/abs/2409.07587)
• Keywords: secure, security, attack, large language model
• Abstract: The rising use of Large Language Models (LLMs) to create and disseminate malware poses a significant cybersecurity challenge due to their ability to generate and distribute attacks with ease. A single prompt can initiate a wide array of malicious activities. This paper addresses this critical issue through a multifaceted approach. First, we provide a comprehensive overview of LLMs and their role in malware detection from diverse sources. We examine five specific applications of LLMs: Malware honeypots, identification of text-based threats, code analysis for detecting malicious intent, trend analysis of malware, and detection of non-standard disguised malware. Our review includes a detailed analysis of the existing literature and establishes guiding principles for the secure use of LLMs. We also introduce a classification scheme to categorize the relevant literature. Second, we propose performance metrics to assess the effectiveness of LLMs in these contexts. Third, we present a risk mitigation framework designed to prevent malware by leveraging LLMs. Finally, we evaluate the performance of our proposed risk mitigation strategies against various factors and demonstrate their effectiveness in countering LLM-enabled malware. The paper concludes by suggesting future advancements and areas requiring deeper exploration in this fascinating field of artificial intelligence.

Title: The Role of Deep Learning Regularizations on Actors in Offline RL

• Authors: Denis Tarasov, Anja Surina, Caglar Gulcehre
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07606
• Pdf URL: https://arxiv.org/pdf/2409.07606
• Copy Paste: [[2409.07606]] The Role of Deep Learning Regularizations on Actors in Offline RL(https://arxiv.org/abs/2409.07606)
• Keywords: robust
• Abstract: Deep learning regularization techniques, such as \emph{dropout}, \emph{layer normalization}, or \emph{weight decay}, are widely adopted in the construction of modern artificial neural networks, often resulting in more robust training processes and improved generalization capabilities. However, in the domain of \emph{Reinforcement Learning} (RL), the application of these techniques has been limited, usually applied to value function estimators \citep{hiraoka2021dropout, smith2022walk}, and may result in detrimental effects. This issue is even more pronounced in offline RL settings, which bear greater similarity to supervised learning but have received less attention. Recent work in continuous offline RL has demonstrated that while we can build sufficiently powerful critic networks, the generalization of actor networks remains a bottleneck. In this study, we empirically show that applying standard regularization techniques to actor networks in offline RL actor-critic algorithms yields improvements of 6\% on average across two algorithms and three different continuous D4RL domains.

Title: A Cost-Aware Approach to Adversarial Robustness in Neural Networks

• Authors: Charles Meyers, Mohammad Reza Saleh Sedghpour, Tommy Löfstedt, Erik Elmroth
• Subjects: cs.CR, cs.CV, cs.LG, stat.AP
• Abstract URL: https://arxiv.org/abs/2409.07609
• Pdf URL: https://arxiv.org/pdf/2409.07609
• Copy Paste: [[2409.07609]] A Cost-Aware Approach to Adversarial Robustness in Neural Networks(https://arxiv.org/abs/2409.07609)
• Keywords: attack, robust
• Abstract: Considering the growing prominence of production-level AI and the threat of adversarial attacks that can evade a model at run-time, evaluating the robustness of models to these evasion attacks is of critical importance. Additionally, testing model changes likely means deploying the models to (e.g. a car or a medical imaging device), or a drone to see how it affects performance, making un-tested changes a public problem that reduces development speed, increases cost of development, and makes it difficult (if not impossible) to parse cause from effect. In this work, we used survival analysis as a cloud-native, time-efficient and precise method for predicting model performance in the presence of adversarial noise. For neural networks in particular, the relationships between the learning rate, batch size, training time, convergence time, and deployment cost are highly complex, so researchers generally rely on benchmark datasets to assess the ability of a model to generalize beyond the training data. To address this, we propose using accelerated failure time models to measure the effect of hardware choice, batch size, number of epochs, and test-set accuracy by using adversarial attacks to induce failures on a reference model architecture before deploying the model to the real world. We evaluate several GPU types and use the Tree Parzen Estimator to maximize model robustness and minimize model run-time simultaneously. This provides a way to evaluate the model and optimise it in a single step, while simultaneously allowing us to model the effect of model parameters on training time, prediction time, and accuracy. Using this technique, we demonstrate that newer, more-powerful hardware does decrease the training time, but with a monetary and power cost that far outpaces the marginal gains in accuracy.

Title: Token Turing Machines are Efficient Vision Models

• Authors: Purvish Jajal, Nick John Eliopoulos, Benjamin Shiue-Hal Chou, George K. Thiravathukal, James C. Davis, Yung-Hsiang Lu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07613
• Pdf URL: https://arxiv.org/pdf/2409.07613
• Copy Paste: [[2409.07613]] Token Turing Machines are Efficient Vision Models(https://arxiv.org/abs/2409.07613)
• Keywords: transformer, segmentation
• Abstract: We propose Vision Token Turing Machines (ViTTM), an efficient, low-latency, memory-augmented Vision Transformer (ViT). Our approach builds on Neural Turing Machines and Token Turing Machines, which were applied to NLP and sequential visual understanding tasks. ViTTMs are designed for non-sequential computer vision tasks such as image classification and segmentation. Our model creates two sets of tokens: process tokens and memory tokens; process tokens pass through encoder blocks and read-write from memory tokens at each encoder block in the network, allowing them to store and retrieve information from memory. By ensuring that there are fewer process tokens than memory tokens, we are able to reduce the inference time of the network while maintaining its accuracy. On ImageNet-1K, the state-of-the-art ViT-B has median latency of 529.5ms and 81.0% accuracy, while our ViTTM-B is 56% faster (234.1ms), with 2.4 times fewer FLOPs, with an accuracy of 82.9%. On ADE20K semantic segmentation, ViT-B achieves 45.65mIoU at 13.8 frame-per-second (FPS) whereas our ViTTM-B model acheives a 45.17 mIoU with 26.8 FPS (+94%).

Title: Zero-Shot Machine-Generated Text Detection Using Mixture of Large Language Models

• Authors: Matthieu Dubois, François Yvon, Pablo Piantanida
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07615
• Pdf URL: https://arxiv.org/pdf/2409.07615
• Copy Paste: [[2409.07615]] Zero-Shot Machine-Generated Text Detection Using Mixture of Large Language Models(https://arxiv.org/abs/2409.07615)
• Keywords: robust, generative, large language model
• Abstract: The dissemination of Large Language Models (LLMs), trained at scale, and endowed with powerful text-generating abilities has vastly increased the threats posed by generative AI technologies by reducing the cost of producing harmful, toxic, faked or forged content. In response, various proposals have been made to automatically discriminate artificially generated from human-written texts, typically framing the problem as a classification problem. Most approaches evaluate an input document by a well-chosen detector LLM, assuming that low-perplexity scores reliably signal machine-made content. As using one single detector can induce brittleness of performance, we instead consider several and derive a new, theoretically grounded approach to combine their respective strengths. Our experiments, using a variety of generator LLMs, suggest that our method effectively increases the robustness of detection.

Title: Public-key encryption from a trapdoor one-way embedding of $SL_2(\mathbb{N}$)

• Authors: Robert Hines
• Subjects: cs.CR, math.NT
• Abstract URL: https://arxiv.org/abs/2409.07616
• Pdf URL: https://arxiv.org/pdf/2409.07616
• Copy Paste: [[2409.07616]] Public-key encryption from a trapdoor one-way embedding of $SL_2(\mathbb{N}$)(https://arxiv.org/abs/2409.07616)
• Keywords: security
• Abstract: We obfuscate words of a given length in a free monoid on two generators with a simple factorization algorithm (namely $SL_2(\mathbb{N})$) to create a public-key encryption scheme. We provide a reference implementation in Python and suggested parameters. The security analysis is between weak and non-existent, left to future work.

Title: Ensemble Methods for Sequence Classification with Hidden Markov Models

• Authors: Maxime Kawawa-Beaudan, Srijan Sood, Soham Palande, Ganapathy Mani, Tucker Balch, Manuela Veloso
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07619
• Pdf URL: https://arxiv.org/pdf/2409.07619
• Copy Paste: [[2409.07619]] Ensemble Methods for Sequence Classification with Hidden Markov Models(https://arxiv.org/abs/2409.07619)
• Keywords: robust, interpretability
• Abstract: We present a lightweight approach to sequence classification using Ensemble Methods for Hidden Markov Models (HMMs). HMMs offer significant advantages in scenarios with imbalanced or smaller datasets due to their simplicity, interpretability, and efficiency. These models are particularly effective in domains such as finance and biology, where traditional methods struggle with high feature dimensionality and varied sequence lengths. Our ensemble-based scoring method enables the comparison of sequences of any length and improves performance on imbalanced datasets. This study focuses on the binary classification problem, particularly in scenarios with data imbalance, where the negative class is the majority (e.g., normal data) and the positive class is the minority (e.g., anomalous data), often with extreme distribution skews. We propose a novel training approach for HMM Ensembles that generalizes to multi-class problems and supports classification and anomaly detection. Our method fits class-specific groups of diverse models using random data subsets, and compares likelihoods across classes to produce composite scores, achieving high average precisions and AUCs. In addition, we compare our approach with neural network-based methods such as Convolutional Neural Networks (CNNs) and Long Short-Term Memory networks (LSTMs), highlighting the efficiency and robustness of HMMs in data-scarce environments. Motivated by real-world use cases, our method demonstrates robust performance across various benchmarks, offering a flexible framework for diverse applications.

Title: HERL: Tiered Federated Learning with Adaptive Homomorphic Encryption using Reinforcement Learning

• Authors: Jiaxang Tang, Zeshan Fayyaz, Mohammad A. Salahuddin, Raouf Boutaba, Zhi-Li Zhang, Ali Anwar
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.07631
• Pdf URL: https://arxiv.org/pdf/2409.07631
• Copy Paste: [[2409.07631]] HERL: Tiered Federated Learning with Adaptive Homomorphic Encryption using Reinforcement Learning(https://arxiv.org/abs/2409.07631)
• Keywords: security, privacy, federate
• Abstract: Federated Learning is a well-researched approach for collaboratively training machine learning models across decentralized data while preserving privacy. However, integrating Homomorphic Encryption to ensure data confidentiality introduces significant computational and communication overheads, particularly in heterogeneous environments where clients have varying computational capacities and security needs. In this paper, we propose HERL, a Reinforcement Learning-based approach that uses Q-Learning to dynamically optimize encryption parameters, specifically the polynomial modulus degree, $N$, and the coefficient modulus, $q$, across different client tiers. Our proposed method involves first profiling and tiering clients according to the chosen clustering approach, followed by dynamically selecting the most suitable encryption parameters using an RL-agent. Experimental results demonstrate that our approach significantly reduces the computational overhead while maintaining utility and a high level of security. Empirical results show that HERL improves utility by 17%, reduces the convergence time by up to 24%, and increases convergence efficiency by up to 30%, with minimal security loss.

Title: SimulBench: Evaluating Language Models with Creative Simulation Tasks

• Authors: Qi Jia, Xiang Yue, Tianyu Zheng, Jie Huang, Bill Yuchen Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07641
• Pdf URL: https://arxiv.org/pdf/2409.07641
• Copy Paste: [[2409.07641]] SimulBench: Evaluating Language Models with Creative Simulation Tasks(https://arxiv.org/abs/2409.07641)
• Keywords: fair, large language model
• Abstract: We introduce SimulBench, a benchmark designed to evaluate large language models (LLMs) across a diverse collection of creative simulation scenarios, such as acting as a Linux terminal or playing text games with users. While these simulation tasks serve as effective measures of an LLM's general intelligence, they are seldom incorporated into existing benchmarks. A major challenge is to develop an evaluation framework for testing different LLMs fairly while preserving the multi-round interactive nature of simulation tasks between users and AI. To tackle this issue, we suggest using a fixed LLM as a user agent to engage with an LLM to collect dialogues first under different tasks. Then, challenging dialogue scripts are extracted for evaluating different target LLMs. To facilitate automatic assessment on \DataName{}, GPT-4 is employed as the evaluator, tasked with reviewing the quality of the final response generated by the target LLMs given multi-turn dialogue scripts. Our comprehensive experiments indicate that these simulation tasks continue to pose a significant challenge with their unique natures and show the gap between proprietary models and the most advanced open LLMs. For example, GPT-4-turbo outperforms LLaMA-3-70b-Chat on 18.55\% more cases.

Title: Feature Importance in Pedestrian Intention Prediction: A Context-Aware Review

• Authors: Mohsen Azarmi, Mahdi Rezaei, He Wang, Ali Arabian
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.07645
• Pdf URL: https://arxiv.org/pdf/2409.07645
• Copy Paste: [[2409.07645]] Feature Importance in Pedestrian Intention Prediction: A Context-Aware Review(https://arxiv.org/abs/2409.07645)
• Keywords: robust, interpretability
• Abstract: Recent advancements in predicting pedestrian crossing intentions for Autonomous Vehicles using Computer Vision and Deep Neural Networks are promising. However, the black-box nature of DNNs poses challenges in understanding how the model works and how input features contribute to final predictions. This lack of interpretability delimits the trust in model performance and hinders informed decisions on feature selection, representation, and model optimisation; thereby affecting the efficacy of future research in the field. To address this, we introduce Context-aware Permutation Feature Importance (CAPFI), a novel approach tailored for pedestrian intention prediction. CAPFI enables more interpretability and reliable assessments of feature importance by leveraging subdivided scenario contexts, mitigating the randomness of feature values through targeted shuffling. This aims to reduce variance and prevent biased estimations in importance scores during permutations. We divide the Pedestrian Intention Estimation (PIE) dataset into 16 comparable context sets, measure the baseline performance of five distinct neural network architectures for intention prediction in each context, and assess input feature importance using CAPFI. We observed nuanced differences among models across various contextual characteristics. The research reveals the critical role of pedestrian bounding boxes and ego-vehicle speed in predicting pedestrian intentions, and potential prediction biases due to the speed feature through cross-context permutation evaluation. We propose an alternative feature representation by considering proximity change rate for rendering dynamic pedestrian-vehicle locomotion, thereby enhancing the contributions of input features to intention prediction. These findings underscore the importance of contextual features and their diversity to develop accurate and robust intent-predictive models.

Title: DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gestures

• Authors: Steven Hogue, Chenxu Zhang, Hamza Daruger, Yapeng Tian, Xiaohu Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07649
• Pdf URL: https://arxiv.org/pdf/2409.07649
• Copy Paste: [[2409.07649]] DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gestures(https://arxiv.org/abs/2409.07649)
• Keywords: diffusion, generative
• Abstract: Audio-driven talking video generation has advanced significantly, but existing methods often depend on video-to-video translation techniques and traditional generative networks like GANs and they typically generate taking heads and co-speech gestures separately, leading to less coherent outputs. Furthermore, the gestures produced by these methods often appear overly smooth or subdued, lacking in diversity, and many gesture-centric approaches do not integrate talking head generation. To address these limitations, we introduce DiffTED, a new approach for one-shot audio-driven TED-style talking video generation from a single image. Specifically, we leverage a diffusion model to generate sequences of keypoints for a Thin-Plate Spline motion model, precisely controlling the avatar's animation while ensuring temporally coherent and diverse gestures. This innovative approach utilizes classifier-free guidance, empowering the gestures to flow naturally with the audio input without relying on pre-trained classifiers. Experiments demonstrate that DiffTED generates temporally coherent talking videos with diverse co-speech gestures.

Title: Foundation Models Boost Low-Level Perceptual Similarity Metrics

• Authors: Abhijay Ghildyal, Nabajeet Barman, Saman Zadtootaghaj
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07650
• Pdf URL: https://arxiv.org/pdf/2409.07650
• Copy Paste: [[2409.07650]] Foundation Models Boost Low-Level Perceptual Similarity Metrics(https://arxiv.org/abs/2409.07650)
• Keywords: transformer
• Abstract: For full-reference image quality assessment (FR-IQA) using deep-learning approaches, the perceptual similarity score between a distorted image and a reference image is typically computed as a distance measure between features extracted from a pretrained CNN or more recently, a Transformer network. Often, these intermediate features require further fine-tuning or processing with additional neural network layers to align the final similarity scores with human judgments. So far, most IQA models based on foundation models have primarily relied on the final layer or the embedding for the quality score estimation. In contrast, this work explores the potential of utilizing the intermediate features of these foundation models, which have largely been unexplored so far in the design of low-level perceptual similarity metrics. We demonstrate that the intermediate features are comparatively more effective. Moreover, without requiring any training, these metrics can outperform both traditional and state-of-the-art learned metrics by utilizing distance measures between the features.

Title: An Unsupervised Dialogue Topic Segmentation Model Based on Utterance Rewriting

• Authors: Xia Hou, Qifeng Li, Tongliang Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07672
• Pdf URL: https://arxiv.org/pdf/2409.07672
• Copy Paste: [[2409.07672]] An Unsupervised Dialogue Topic Segmentation Model Based on Utterance Rewriting(https://arxiv.org/abs/2409.07672)
• Keywords: segmentation
• Abstract: Dialogue topic segmentation plays a crucial role in various types of dialogue modeling tasks. The state-of-the-art unsupervised DTS methods learn topic-aware discourse representations from conversation data through adjacent discourse matching and pseudo segmentation to further mine useful clues in unlabeled conversational relations. However, in multi-round dialogs, discourses often have co-references or omissions, leading to the fact that direct use of these discourses for representation learning may negatively affect the semantic similarity computation in the neighboring discourse matching task. In order to fully utilize the useful cues in conversational relations, this study proposes a novel unsupervised dialog topic segmentation method that combines the Utterance Rewriting (UR) technique with an unsupervised learning algorithm to efficiently utilize the useful cues in unlabeled dialogs by rewriting the dialogs in order to recover the co-referents and omitted words. Compared with existing unsupervised models, the proposed Discourse Rewriting Topic Segmentation Model (UR-DTS) significantly improves the accuracy of topic segmentation. The main finding is that the performance on DialSeg711 improves by about 6% in terms of absolute error score and WD, achieving 11.42% in terms of absolute error score and 12.97% in terms of WD. on Doc2Dial the absolute error score and WD improves by about 3% and 2%, respectively, resulting in SOTA reaching 35.17% in terms of absolute error score and 38.49% in terms of WD. This shows that the model is very effective in capturing the nuances of conversational topics, as well as the usefulness and challenges of utilizing unlabeled conversations.

Title: Open-Vocabulary Remote Sensing Image Semantic Segmentation

• Authors: Qinglong Cao, Yuntian Chen, Chao Ma, Xiaokang Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07683
• Pdf URL: https://arxiv.org/pdf/2409.07683
• Copy Paste: [[2409.07683]] Open-Vocabulary Remote Sensing Image Semantic Segmentation(https://arxiv.org/abs/2409.07683)
• Keywords: segmentation
• Abstract: Open-vocabulary image semantic segmentation (OVS) seeks to segment images into semantic regions across an open set of categories. Existing OVS methods commonly depend on foundational vision-language models and utilize similarity computation to tackle OVS tasks. However, these approaches are predominantly tailored to natural images and struggle with the unique characteristics of remote sensing images, such as rapidly changing orientations and significant scale variations. These challenges complicate OVS tasks in earth vision, requiring specialized approaches. To tackle this dilemma, we propose the first OVS framework specifically designed for remote sensing imagery, drawing inspiration from the distinct remote sensing traits. Particularly, to address the varying orientations, we introduce a rotation-aggregative similarity computation module that generates orientation-adaptive similarity maps as initial semantic maps. These maps are subsequently refined at both spatial and categorical levels to produce more accurate semantic maps. Additionally, to manage significant scale changes, we integrate multi-scale image features into the upsampling process, resulting in the final scale-aware semantic masks. To advance OVS in earth vision and encourage reproducible research, we establish the first open-sourced OVS benchmark for remote sensing imagery, including four public remote sensing datasets. Extensive experiments on this benchmark demonstrate our proposed method achieves state-of-the-art performance. All codes and datasets are available at this https URL.

Title: Learn from Balance: Rectifying Knowledge Transfer for Long-Tailed Scenarios

• Authors: Xinlei Huang, Jialiang Tang, Xubin Zheng, Jinjia Zhou, Wenxin Yu, Ning Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07694
• Pdf URL: https://arxiv.org/pdf/2409.07694
• Copy Paste: [[2409.07694]] Learn from Balance: Rectifying Knowledge Transfer for Long-Tailed Scenarios(https://arxiv.org/abs/2409.07694)
• Keywords: robust
• Abstract: Knowledge Distillation (KD) transfers knowledge from a large pre-trained teacher network to a compact and efficient student network, making it suitable for deployment on resource-limited media terminals. However, traditional KD methods require balanced data to ensure robust training, which is often unavailable in practical applications. In such scenarios, a few head categories occupy a substantial proportion of examples. This imbalance biases the trained teacher network towards the head categories, resulting in severe performance degradation on the less represented tail categories for both the teacher and student networks. In this paper, we propose a novel framework called Knowledge Rectification Distillation (KRDistill) to address the imbalanced knowledge inherited in the teacher network through the incorporation of the balanced category priors. Furthermore, we rectify the biased predictions produced by the teacher network, particularly focusing on the tail categories. Consequently, the teacher network can provide balanced and accurate knowledge to train a reliable student network. Intensive experiments conducted on various long-tailed datasets demonstrate that our KRDistill can effectively train reliable student networks in realistic scenarios of data imbalance.

Title: TMFNet: Two-Stream Multi-Channels Fusion Networks for Color Image Operation Chain Detection

• Authors: Yakun Niu, Lei Tan, Lei Zhang, Xianyu Zuo
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.07701
• Pdf URL: https://arxiv.org/pdf/2409.07701
• Copy Paste: [[2409.07701]] TMFNet: Two-Stream Multi-Channels Fusion Networks for Color Image Operation Chain Detection(https://arxiv.org/abs/2409.07701)
• Keywords: robust
• Abstract: Image operation chain detection techniques have gained increasing attention recently in the field of multimedia forensics. However, existing detection methods suffer from the generalization problem. Moreover, the channel correlation of color images that provides additional forensic evidence is often ignored. To solve these issues, in this article, we propose a novel two-stream multi-channels fusion networks for color image operation chain detection in which the spatial artifact stream and the noise residual stream are explored in a complementary manner. Specifically, we first propose a novel deep residual architecture without pooling in the spatial artifact stream for learning the global features representation of multi-channel correlation. Then, a set of filters is designed to aggregate the correlation information of multi-channels while capturing the low-level features in the noise residual stream. Subsequently, the high-level features are extracted by the deep residual model. Finally, features from the two streams are fed into a fusion module, to effectively learn richer discriminative representations of the operation chain. Extensive experiments show that the proposed method achieves state-of-the-art generalization ability while maintaining robustness to JPEG compression. The source code used in these experiments will be released at this https URL.

Title: Attack End-to-End Autonomous Driving through Module-Wise Noise

• Authors: Lu Wang, Tianyuan Zhang, Yikai Han, Muyang Fang, Ting Jin, Jiaqi Kang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07706
• Pdf URL: https://arxiv.org/pdf/2409.07706
• Copy Paste: [[2409.07706]] Attack End-to-End Autonomous Driving through Module-Wise Noise(https://arxiv.org/abs/2409.07706)
• Keywords: security, attack
• Abstract: With recent breakthroughs in deep neural networks, numerous tasks within autonomous driving have exhibited remarkable performance. However, deep learning models are susceptible to adversarial attacks, presenting significant security risks to autonomous driving systems. Presently, end-to-end architectures have emerged as the predominant solution for autonomous driving, owing to their collaborative nature across different tasks. Yet, the implications of adversarial attacks on such models remain relatively unexplored. In this paper, we conduct comprehensive adversarial security research on the modular end-to-end autonomous driving model for the first time. We thoroughly consider the potential vulnerabilities in the model inference process and design a universal attack scheme through module-wise noise injection. We conduct large-scale experiments on the full-stack autonomous driving model and demonstrate that our attack method outperforms previous attack methods. We trust that our research will offer fresh insights into ensuring the safety and reliability of autonomous driving systems.

Title: Harnessing TI Feeds for Exploitation Detection

• Authors: Kajal Patel, Zubair Shafiq, Mateus Nogueira, Daniel Sadoc Menasché, Enrico Lovat, Taimur Kashif, Ashton Woiwood, Matheus Martins
• Subjects: cs.CR, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.07709
• Pdf URL: https://arxiv.org/pdf/2409.07709
• Copy Paste: [[2409.07709]] Harnessing TI Feeds for Exploitation Detection(https://arxiv.org/abs/2409.07709)
• Keywords: security
• Abstract: Many organizations rely on Threat Intelligence (TI) feeds to assess the risk associated with security threats. Due to the volume and heterogeneity of data, it is prohibitive to manually analyze the threat information available in different loosely structured TI feeds. Thus, there is a need to develop automated methods to vet and extract actionable information from TI feeds. To this end, we present a machine learning pipeline to automatically detect vulnerability exploitation from TI feeds. We first model threat vocabulary in loosely structured TI feeds using state-of-the-art embedding techniques (Doc2Vec and BERT) and then use it to train a supervised machine learning classifier to detect exploitation of security vulnerabilities. We use our approach to identify exploitation events in 191 different TI feeds. Our longitudinal evaluation shows that it is able to accurately identify exploitation events from TI feeds only using past data for training and even on TI feeds withheld from training. Our proposed approach is useful for a variety of downstream tasks such as data-driven vulnerability risk assessment.

Title: Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice

• Authors: Jonathan Li, Rohan Bhambhoria, Samuel Dahan, Xiaodan Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07713
• Pdf URL: https://arxiv.org/pdf/2409.07713
• Copy Paste: [[2409.07713]] Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice(https://arxiv.org/abs/2409.07713)
• Keywords: generative
• Abstract: Generative AI models, such as the GPT and Llama series, have significant potential to assist laypeople in answering legal questions. However, little prior work focuses on the data sourcing, inference, and evaluation of these models in the context of laypersons. To this end, we propose a human-centric legal NLP pipeline, covering data sourcing, inference, and evaluation. We introduce and release a dataset, LegalQA, with real and specific legal questions spanning from employment law to criminal law, corresponding answers written by legal experts, and citations for each answer. We develop an automatic evaluation protocol for this dataset, then show that retrieval-augmented generation from only 850 citations in the train set can match or outperform internet-wide retrieval, despite containing 9 orders of magnitude less data. Finally, we propose future directions for open-sourced efforts, which fall behind closed-sourced models.

Title: CollaMamba: Efficient Collaborative Perception with Cross-Agent Spatial-Temporal State Space Model

• Authors: Yang Li, Quan Yuan, Guiyang Luo, Xiaoyuan Fu, Xuanhan Zhu, Yujia Yang, Rui Pan, Jinglin Li
• Subjects: cs.CV, cs.MA
• Abstract URL: https://arxiv.org/abs/2409.07714
• Pdf URL: https://arxiv.org/pdf/2409.07714
• Copy Paste: [[2409.07714]] CollaMamba: Efficient Collaborative Perception with Cross-Agent Spatial-Temporal State Space Model(https://arxiv.org/abs/2409.07714)
• Keywords: transformer
• Abstract: By sharing complementary perceptual information, multi-agent collaborative perception fosters a deeper understanding of the environment. Recent studies on collaborative perception mostly utilize CNNs or Transformers to learn feature representation and fusion in the spatial dimension, which struggle to handle long-range spatial-temporal features under limited computing and communication resources. Holistically modeling the dependencies over extensive spatial areas and extended temporal frames is crucial to enhancing feature quality. To this end, we propose a resource efficient cross-agent spatial-temporal collaborative state space model (SSM), named CollaMamba. Initially, we construct a foundational backbone network based on spatial SSM. This backbone adeptly captures positional causal dependencies from both single-agent and cross-agent views, yielding compact and comprehensive intermediate features while maintaining linear complexity. Furthermore, we devise a history-aware feature boosting module based on temporal SSM, extracting contextual cues from extended historical frames to refine vague features while preserving low overhead. Extensive experiments across several datasets demonstrate that CollaMamba outperforms state-of-the-art methods, achieving higher model accuracy while reducing computational and communication overhead by up to 71.9% and 1/64, respectively. This work pioneers the exploration of the Mamba's potential in collaborative perception. The source code will be made available.

Title: FIReStereo: Forest InfraRed Stereo Dataset for UAS Depth Perception in Visually Degraded Environments

• Authors: Devansh Dhrafani, Yifei Liu, Andrew Jong, Ukcheol Shin, Yao He, Tyler Harp, Yaoyu Hu, Jean Oh, Sebastian Scherer
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.07715
• Pdf URL: https://arxiv.org/pdf/2409.07715
• Copy Paste: [[2409.07715]] FIReStereo: Forest InfraRed Stereo Dataset for UAS Depth Perception in Visually Degraded Environments(https://arxiv.org/abs/2409.07715)
• Keywords: robust
• Abstract: Robust depth perception in visually-degraded environments is crucial for autonomous aerial systems. Thermal imaging cameras, which capture infrared radiation, are robust to visual degradation. However, due to lack of a large-scale dataset, the use of thermal cameras for unmanned aerial system (UAS) depth perception has remained largely unexplored. This paper presents a stereo thermal depth perception dataset for autonomous aerial perception applications. The dataset consists of stereo thermal images, LiDAR, IMU and ground truth depth maps captured in urban and forest settings under diverse conditions like day, night, rain, and smoke. We benchmark representative stereo depth estimation algorithms, offering insights into their performance in degraded conditions. Models trained on our dataset generalize well to unseen smoky conditions, highlighting the robustness of stereo thermal imaging for depth perception. We aim for this work to enhance robotic perception in disaster scenarios, allowing for exploration and operations in previously unreachable areas. The dataset and source code are available at this https URL.

Title: Advancing Depth Anything Model for Unsupervised Monocular Depth Estimation in Endoscopy

• Authors: Bojian Li, Bo Liu, Jinghua Yue, Fugen Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07723
• Pdf URL: https://arxiv.org/pdf/2409.07723
• Copy Paste: [[2409.07723]] Advancing Depth Anything Model for Unsupervised Monocular Depth Estimation in Endoscopy(https://arxiv.org/abs/2409.07723)
• Keywords: transformer
• Abstract: Depth estimation is a cornerstone of 3D reconstruction and plays a vital role in minimally invasive endoscopic surgeries. However, most current depth estimation networks rely on traditional convolutional neural networks, which are limited in their ability to capture global information. Foundation models offer a promising avenue for enhancing depth estimation, but those currently available are primarily trained on natural images, leading to suboptimal performance when applied to endoscopic images. In this work, we introduce a novel fine-tuning strategy for the Depth Anything Model and integrate it with an intrinsic-based unsupervised monocular depth estimation framework. Our approach includes a low-rank adaptation technique based on random vectors, which improves the model's adaptability to different scales. Additionally, we propose a residual block built on depthwise separable convolution to compensate for the transformer's limited ability to capture high-frequency details, such as edges and textures. Our experimental results on the SCARED dataset show that our method achieves state-of-the-art performance while minimizing the number of trainable parameters. Applying this method in minimally invasive endoscopic surgery could significantly enhance both the precision and safety of these procedures.

Title: Large Language Models are Pattern Matchers: Editing Semi-Structured and Structured Documents with ChatGPT

• Authors: Irene Weber
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07732
• Pdf URL: https://arxiv.org/pdf/2409.07732
• Copy Paste: [[2409.07732]] Large Language Models are Pattern Matchers: Editing Semi-Structured and Structured Documents with ChatGPT(https://arxiv.org/abs/2409.07732)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) offer numerous applications, the full extent of which is not yet understood. This paper investigates if LLMs can be applied for editing structured and semi-structured documents with minimal effort. Using a qualitative research approach, we conduct two case studies with ChatGPT and thoroughly analyze the results. Our experiments indicate that LLMs can effectively edit structured and semi-structured documents when provided with basic, straightforward prompts. ChatGPT demonstrates a strong ability to recognize and process the structure of annotated documents. This suggests that explicitly structuring tasks and data in prompts might enhance an LLM's ability to understand and solve tasks. Furthermore, the experiments also reveal impressive pattern matching skills in ChatGPT. This observation deserves further investigation, as it may contribute to understanding the processes leading to hallucinations in LLMs.

Title: LOCKEY: A Novel Approach to Model Authentication and Deepfake Tracking

• Authors: Mayank Kumar Singh, Naoya Takahashi, Wei-Hsiang Liao, Yuki Mitsufuji
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07743
• Pdf URL: https://arxiv.org/pdf/2409.07743
• Copy Paste: [[2409.07743]] LOCKEY: A Novel Approach to Model Authentication and Deepfake Tracking(https://arxiv.org/abs/2409.07743)
• Keywords: robust, watermark, generative
• Abstract: This paper presents a novel approach to deter unauthorized deepfakes and enable user tracking in generative models, even when the user has full access to the model parameters, by integrating key-based model authentication with watermarking techniques. Our method involves providing users with model parameters accompanied by a unique, user-specific key. During inference, the model is conditioned upon the key along with the standard input. A valid key results in the expected output, while an invalid key triggers a degraded output, thereby enforcing key-based model authentication. For user tracking, the model embeds the user's unique key as a watermark within the generated content, facilitating the identification of the user's ID. We demonstrate the effectiveness of our approach on two types of models, audio codecs and vocoders, utilizing the SilentCipher watermarking method. Additionally, we assess the robustness of the embedded watermarks against various distortions, validating their reliability in various scenarios.

Title: Learning Brain Tumor Representation in 3D High-Resolution MR Images via Interpretable State Space Models

• Authors: Qingqiao Hu, Daoan Zhang, Jiebo Luo, Zhenyu Gong, Benedikt Wiestler, Jianguo Zhang, Hongwei Bran Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07746
• Pdf URL: https://arxiv.org/pdf/2409.07746
• Copy Paste: [[2409.07746]] Learning Brain Tumor Representation in 3D High-Resolution MR Images via Interpretable State Space Models(https://arxiv.org/abs/2409.07746)
• Keywords: interpretability, transformer
• Abstract: Learning meaningful and interpretable representations from high-dimensional volumetric magnetic resonance (MR) images is essential for advancing personalized medicine. While Vision Transformers (ViTs) have shown promise in handling image data, their application to 3D multi-contrast MR images faces challenges due to computational complexity and interpretability. To address this, we propose a novel state-space-model (SSM)-based masked autoencoder which scales ViT-like models to handle high-resolution data effectively while also enhancing the interpretability of learned representations. We propose a latent-to-spatial mapping technique that enables direct visualization of how latent features correspond to specific regions in the input volumes in the context of SSM. We validate our method on two key neuro-oncology tasks: identification of isocitrate dehydrogenase mutation status and 1p/19q co-deletion classification, achieving state-of-the-art accuracy. Our results highlight the potential of SSM-based self-supervised learning to transform radiomics analysis by combining efficiency and interpretability.

Title: Multi-object event graph representation learning for Video Question Answering

• Authors: Yanan Wang, Shuichiro Haruta, Donghuo Zeng, Julio Vizcarra, Mori Kurokawa
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07747
• Pdf URL: https://arxiv.org/pdf/2409.07747
• Copy Paste: [[2409.07747]] Multi-object event graph representation learning for Video Question Answering(https://arxiv.org/abs/2409.07747)
• Keywords: transformer
• Abstract: Video question answering (VideoQA) is a task to predict the correct answer to questions posed about a given video. The system must comprehend spatial and temporal relationships among objects extracted from videos to perform causal and temporal reasoning. While prior works have focused on modeling individual object movements using transformer-based methods, they falter when capturing complex scenarios involving multiple objects (e.g., "a boy is throwing a ball in a hoop"). We propose a contrastive language event graph representation learning method called CLanG to address this limitation. Aiming to capture event representations associated with multiple objects, our method employs a multi-layer GNN-cluster module for adversarial graph representation learning, enabling contrastive learning between the question text and its relevant multi-object event graph. Our method outperforms a strong baseline, achieving up to 2.2% higher accuracy on two challenging VideoQA datasets, NExT-QA and TGIF-QA-R. In particular, it is 2.8% better than baselines in handling causal and temporal questions, highlighting its strength in reasoning multiple object-based events.

Title: Efficient Privacy-Preserving KAN Inference Using Homomorphic Encryption

• Authors: Zhizheng Lai, Yufei Zhou, Peijia Zheng, Lin Chen
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07751
• Pdf URL: https://arxiv.org/pdf/2409.07751
• Copy Paste: [[2409.07751]] Efficient Privacy-Preserving KAN Inference Using Homomorphic Encryption(https://arxiv.org/abs/2409.07751)
• Keywords: security, privacy, interpretability
• Abstract: The recently proposed Kolmogorov-Arnold Networks (KANs) offer enhanced interpretability and greater model expressiveness. However, KANs also present challenges related to privacy leakage during inference. Homomorphic encryption (HE) facilitates privacy-preserving inference for deep learning models, enabling resource-limited users to benefit from deep learning services while ensuring data security. Yet, the complex structure of KANs, incorporating nonlinear elements like the SiLU activation function and B-spline functions, renders existing privacy-preserving inference techniques inadequate. To address this issue, we propose an accurate and efficient privacy-preserving inference scheme tailored for KANs. Our approach introduces a task-specific polynomial approximation for the SiLU activation function, dynamically adjusting the approximation range to ensure high accuracy on real-world datasets. Additionally, we develop an efficient method for computing B-spline functions within the HE domain, leveraging techniques such as repeat packing, lazy combination, and comparison functions. We evaluate the effectiveness of our privacy-preserving KAN inference scheme on both symbolic formula evaluation and image classification. The experimental results show that our model achieves accuracy comparable to plaintext KANs across various datasets and outperforms plaintext MLPs. Additionally, on the CIFAR-10 dataset, our inference latency achieves over 7 times speedup compared to the naive method.

Title: DiTAS: Quantizing Diffusion Transformers via Enhanced Activation Smoothing

• Authors: Zhenyuan Dong, Sai Qian Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07756
• Pdf URL: https://arxiv.org/pdf/2409.07756
• Copy Paste: [[2409.07756]] DiTAS: Quantizing Diffusion Transformers via Enhanced Activation Smoothing(https://arxiv.org/abs/2409.07756)
• Keywords: diffusion, data-free, transformer
• Abstract: Diffusion Transformers (DiTs) have recently attracted significant interest from both industry and academia due to their enhanced capabilities in visual generation, surpassing the performance of traditional diffusion models that employ U-Net. However, the improved performance of DiTs comes at the expense of higher parameter counts and implementation costs, which significantly limits their deployment on resource-constrained devices like mobile phones. We propose DiTAS, a data-free post-training quantization (PTQ) method for efficient DiT inference. DiTAS relies on the proposed temporal-aggregated smoothing techniques to mitigate the impact of the channel-wise outliers within the input activations, leading to much lower quantization error under extremely low bitwidth. To further enhance the performance of the quantized DiT, we adopt the layer-wise grid search strategy to optimize the smoothing factor. Experimental results demonstrate that our approach enables 4-bit weight, 8-bit activation (W4A8) quantization for DiTs while maintaining comparable performance as the full-precision model.

Title: Alignment with Preference Optimization Is All You Need for LLM Safety

• Authors: Reda Alami, Ali Khalifa Almansoori, Ahmed Alzubaidi, Mohamed El Amine Seddik, Mugariya Farooq, Hakim Hacid
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07772
• Pdf URL: https://arxiv.org/pdf/2409.07772
• Copy Paste: [[2409.07772]] Alignment with Preference Optimization Is All You Need for LLM Safety(https://arxiv.org/abs/2409.07772)
• Keywords: robust
• Abstract: We demonstrate that preference optimization methods can effectively enhance LLM safety. Applying various alignment techniques to the Falcon 11B model using safety datasets, we achieve a significant boost in global safety score (from $57.64\%$ to $99.90\%$) as measured by LlamaGuard 3 8B, competing with state-of-the-art models. On toxicity benchmarks, average scores in adversarial settings dropped from over $0.6$ to less than $0.07$. However, this safety improvement comes at the cost of reduced general capabilities, particularly in math, suggesting a trade-off. We identify noise contrastive alignment (Safe-NCA) as an optimal method for balancing safety and performance. Our study ultimately shows that alignment techniques can be sufficient for building safe and robust models.

Title: ASSNet: Adaptive Semantic Segmentation Network for Microtumors and Multi-Organ Segmentation

• Authors: Fuchen Zheng, Xinyi Chen, Xuhang Chen, Haolun Li, Xiaojiao Guo, Guoheng Huang, Chi-Man Pun, Shoujun Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07779
• Pdf URL: https://arxiv.org/pdf/2409.07779
• Copy Paste: [[2409.07779]] ASSNet: Adaptive Semantic Segmentation Network for Microtumors and Multi-Organ Segmentation(https://arxiv.org/abs/2409.07779)
• Keywords: extraction, transformer, segmentation
• Abstract: Medical image segmentation, a crucial task in computer vision, facilitates the automated delineation of anatomical structures and pathologies, supporting clinicians in diagnosis, treatment planning, and disease monitoring. Notably, transformers employing shifted window-based self-attention have demonstrated exceptional performance. However, their reliance on local window attention limits the fusion of local and global contextual information, crucial for segmenting microtumors and miniature organs. To address this limitation, we propose the Adaptive Semantic Segmentation Network (ASSNet), a transformer architecture that effectively integrates local and global features for precise medical image segmentation. ASSNet comprises a transformer-based U-shaped encoder-decoder network. The encoder utilizes shifted window self-attention across five resolutions to extract multi-scale features, which are then propagated to the decoder through skip connections. We introduce an augmented multi-layer perceptron within the encoder to explicitly model long-range dependencies during feature extraction. Recognizing the constraints of conventional symmetrical encoder-decoder designs, we propose an Adaptive Feature Fusion (AFF) decoder to complement our encoder. This decoder incorporates three key components: the Long Range Dependencies (LRD) block, the Multi-Scale Feature Fusion (MFF) block, and the Adaptive Semantic Center (ASC) block. These components synergistically facilitate the effective fusion of multi-scale features extracted by the decoder while capturing long-range dependencies and refining object boundaries. Comprehensive experiments on diverse medical image segmentation tasks, including multi-organ, liver tumor, and bladder tumor segmentation, demonstrate that ASSNet achieves state-of-the-art results. Code and models are available at: \url{this https URL}.

Title: XMOL: Explainable Multi-property Optimization of Molecules

• Authors: Aye Phyu Phyu Aung, Jay Chaudhary, Ji Wei Yoon, Senthilnath Jayavelu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07786
• Pdf URL: https://arxiv.org/pdf/2409.07786
• Copy Paste: [[2409.07786]] XMOL: Explainable Multi-property Optimization of Molecules(https://arxiv.org/abs/2409.07786)
• Keywords: explainability, diffusion
• Abstract: Molecular optimization is a key challenge in drug discovery and material science domain, involving the design of molecules with desired properties. Existing methods focus predominantly on single-property optimization, necessitating repetitive runs to target multiple properties, which is inefficient and computationally expensive. Moreover, these methods often lack transparency, making it difficult for researchers to understand and control the optimization process. To address these issues, we propose a novel framework, Explainable Multi-property Optimization of Molecules (XMOL), to optimize multiple molecular properties simultaneously while incorporating explainability. Our approach builds on state-of-the-art geometric diffusion models, extending them to multi-property optimization through the introduction of spectral normalization and enhanced molecular constraints for stabilized training. Additionally, we integrate interpretive and explainable techniques throughout the optimization process. We evaluated XMOL on the real-world molecular datasets i.e., QM9, demonstrating its effectiveness in both single property and multiple properties optimization while offering interpretable results, paving the way for more efficient and reliable molecular design.

Title: Full-text Error Correction for Chinese Speech Recognition with Large Language Model

• Authors: Zhiyuan Tang, Dong Wang, Shen Huang, Shidong Shang
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.07790
• Pdf URL: https://arxiv.org/pdf/2409.07790
• Copy Paste: [[2409.07790]] Full-text Error Correction for Chinese Speech Recognition with Large Language Model(https://arxiv.org/abs/2409.07790)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated substantial potential for error correction in Automatic Speech Recognition (ASR). However, most research focuses on utterances from short-duration speech recordings, which are the predominant form of speech data for supervised ASR training. This paper investigates the effectiveness of LLMs for error correction in full-text generated by ASR systems from longer speech recordings, such as transcripts from podcasts, news broadcasts, and meetings. First, we develop a Chinese dataset for full-text error correction, named ChFT, utilizing a pipeline that involves text-to-speech synthesis, ASR, and error-correction pair extractor. This dataset enables us to correct errors across contexts, including both full-text and segment, and to address a broader range of error types, such as punctuation restoration and inverse text normalization, thus making the correction process comprehensive. Second, we fine-tune a pre-trained LLM on the constructed dataset using a diverse set of prompts and target formats, and evaluate its performance on full-text error correction. Specifically, we design prompts based on full-text and segment, considering various output formats, such as directly corrected text and JSON-based error-correction pairs. Through various test settings, including homogeneous, up-to-date, and hard test sets, we find that the fine-tuned LLMs perform well in the full-text setting with different prompts, each presenting its own strengths and weaknesses. This establishes a promising baseline for further research. The dataset is available on the website.

Title: Lagrange Duality and Compound Multi-Attention Transformer for Semi-Supervised Medical Image Segmentation

• Authors: Fuchen Zheng, Quanjun Li, Weixuan Li, Xuhang Chen, Yihang Dong, Guoheng Huang, Chi-Man Pun, Shoujun Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07793
• Pdf URL: https://arxiv.org/pdf/2409.07793
• Copy Paste: [[2409.07793]] Lagrange Duality and Compound Multi-Attention Transformer for Semi-Supervised Medical Image Segmentation(https://arxiv.org/abs/2409.07793)
• Keywords: transformer, segmentation
• Abstract: Medical image segmentation, a critical application of semantic segmentation in healthcare, has seen significant advancements through specialized computer vision techniques. While deep learning-based medical image segmentation is essential for assisting in medical diagnosis, the lack of diverse training data causes the long-tail problem. Moreover, most previous hybrid CNN-ViT architectures have limited ability to combine various attentions in different layers of the Convolutional Neural Network. To address these issues, we propose a Lagrange Duality Consistency (LDC) Loss, integrated with Boundary-Aware Contrastive Loss, as the overall training objective for semi-supervised learning to mitigate the long-tail problem. Additionally, we introduce CMAformer, a novel network that synergizes the strengths of ResUNet and Transformer. The cross-attention block in CMAformer effectively integrates spatial attention and channel attention for multi-scale feature fusion. Overall, our results indicate that CMAformer, combined with the feature fusion framework and the new consistency loss, demonstrates strong complementarity in semi-supervised learning ensembles. We achieve state-of-the-art results on multiple public medical image datasets. Example code are available at: \url{this https URL}.

Title: In-Situ Fine-Tuning of Wildlife Models in IoT-Enabled Camera Traps for Efficient Adaptation

• Authors: Mohammad Mehdi Rastikerdar, Jin Huang, Hui Guan, Deepak Ganesan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07796
• Pdf URL: https://arxiv.org/pdf/2409.07796
• Copy Paste: [[2409.07796]] In-Situ Fine-Tuning of Wildlife Models in IoT-Enabled Camera Traps for Efficient Adaptation(https://arxiv.org/abs/2409.07796)
• Keywords: robust
• Abstract: Wildlife monitoring via camera traps has become an essential tool in ecology, but the deployment of machine learning models for on-device animal classification faces significant challenges due to domain shifts and resource constraints. This paper introduces WildFit, a novel approach that reconciles the conflicting goals of achieving high domain generalization performance and ensuring efficient inference for camera trap applications. WildFit leverages continuous background-aware model fine-tuning to deploy ML models tailored to the current location and time window, allowing it to maintain robust classification accuracy in the new environment without requiring significant computational resources. This is achieved by background-aware data synthesis, which generates training images representing the new domain by blending background images with animal images from the source domain. We further enhance fine-tuning effectiveness through background drift detection and class distribution drift detection, which optimize the quality of synthesized data and improve generalization performance. Our extensive evaluation across multiple camera trap datasets demonstrates that WildFit achieves significant improvements in classification accuracy and computational efficiency compared to traditional approaches.

Title: GateAttentionPose: Enhancing Pose Estimation with Agent Attention and Improved Gated Convolutions

• Authors: Liang Feng, Zhixuan Shen, Lihua Wen, Shiyao Li, Ming Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07798
• Pdf URL: https://arxiv.org/pdf/2409.07798
• Copy Paste: [[2409.07798]] GateAttentionPose: Enhancing Pose Estimation with Agent Attention and Improved Gated Convolutions(https://arxiv.org/abs/2409.07798)
• Keywords: robust, extraction
• Abstract: This paper introduces GateAttentionPose, an innovative approach that enhances the UniRepLKNet architecture for pose estimation tasks. We present two key contributions: the Agent Attention module and the Gate-Enhanced Feedforward Block (GEFB). The Agent Attention module replaces large kernel convolutions, significantly improving computational efficiency while preserving global context modeling. The GEFB augments feature extraction and processing capabilities, particularly in complex scenes. Extensive evaluations on COCO and MPII datasets demonstrate that GateAttentionPose outperforms existing state-of-the-art methods, including the original UniRepLKNet, achieving superior or comparable results with improved efficiency. Our approach offers a robust solution for pose estimation across diverse applications, including autonomous driving, human motion capture, and virtual reality.

Title: SURGIVID: Annotation-Efficient Surgical Video Object Discovery

• Authors: Çağhan Köksal, Ghazal Ghazaei, Nassir Navab
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07801
• Pdf URL: https://arxiv.org/pdf/2409.07801
• Copy Paste: [[2409.07801]] SURGIVID: Annotation-Efficient Surgical Video Object Discovery(https://arxiv.org/abs/2409.07801)
• Keywords: segmentation
• Abstract: Surgical scenes convey crucial information about the quality of surgery. Pixel-wise localization of tools and anatomical structures is the first task towards deeper surgical analysis for microscopic or endoscopic surgical views. This is typically done via fully-supervised methods which are annotation greedy and in several cases, demanding medical expertise. Considering the profusion of surgical videos obtained through standardized surgical workflows, we propose an annotation-efficient framework for the semantic segmentation of surgical scenes. We employ image-based self-supervised object discovery to identify the most salient tools and anatomical structures in surgical videos. These proposals are further refined within a minimally supervised fine-tuning step. Our unsupervised setup reinforced with only 36 annotation labels indicates comparable localization performance with fully-supervised segmentation models. Further, leveraging surgical phase labels as weak labels can better guide model attention towards surgical tools, leading to $\sim 2\%$ improvement in tool localization. Extensive ablation studies on the CaDIS dataset validate the effectiveness of our proposed solution in discovering relevant surgical objects with minimal or no supervision.

Title: FedHide: Federated Learning by Hiding in the Neighbors

• Authors: Hyunsin Park, Sungrack Yun
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07808
• Pdf URL: https://arxiv.org/pdf/2409.07808
• Copy Paste: [[2409.07808]] FedHide: Federated Learning by Hiding in the Neighbors(https://arxiv.org/abs/2409.07808)
• Keywords: privacy, attack, robust, federate
• Abstract: We propose a prototype-based federated learning method designed for embedding networks in classification or verification tasks. Our focus is on scenarios where each client has data from a single class. The main challenge is to develop an embedding network that can distinguish between different classes while adhering to privacy constraints. Sharing true class prototypes with the server or other clients could potentially compromise sensitive information. To tackle this issue, we propose a proxy class prototype that will be shared among clients instead of the true class prototype. Our approach generates proxy class prototypes by linearly combining them with their nearest neighbors. This technique conceals the true class prototype while enabling clients to learn discriminative embedding networks. We compare our method to alternative techniques, such as adding random Gaussian noise and using random selection with cosine similarity constraints. Furthermore, we evaluate the robustness of our approach against gradient inversion attacks and introduce a measure for prototype leakage. This measure quantifies the extent of private information revealed when sharing the proposed proxy class prototype. Moreover, we provide a theoretical analysis of the convergence properties of our approach. Our proposed method for federated learning from scratch demonstrates its effectiveness through empirical results on three benchmark datasets: CIFAR-100, VoxCeleb1, and VGGFace2.

Title: Controllable Synthetic Clinical Note Generation with Privacy Guarantees

• Authors: Tal Baumel, Andre Manoel, Daniel Jones, Shize Su, Huseyin Inan, Aaron (Ari)Bornstein, Robert Sim
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07809
• Pdf URL: https://arxiv.org/pdf/2409.07809
• Copy Paste: [[2409.07809]] Controllable Synthetic Clinical Note Generation with Privacy Guarantees(https://arxiv.org/abs/2409.07809)
• Keywords: privacy, robust
• Abstract: In the field of machine learning, domain-specific annotated data is an invaluable resource for training effective models. However, in the medical domain, this data often includes Personal Health Information (PHI), raising significant privacy concerns. The stringent regulations surrounding PHI limit the availability and sharing of medical datasets, which poses a substantial challenge for researchers and practitioners aiming to develop advanced machine learning models. In this paper, we introduce a novel method to "clone" datasets containing PHI. Our approach ensures that the cloned datasets retain the essential characteristics and utility of the original data without compromising patient privacy. By leveraging differential-privacy techniques and a novel fine-tuning task, our method produces datasets that are free from identifiable information while preserving the statistical properties necessary for model training. We conduct utility testing to evaluate the performance of machine learning models trained on the cloned datasets. The results demonstrate that our cloned datasets not only uphold privacy standards but also enhance model performance compared to those trained on traditional anonymized datasets. This work offers a viable solution for the ethical and effective utilization of sensitive medical data in machine learning, facilitating progress in medical research and the development of robust predictive models.

Title: What is YOLOv9: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector

• Authors: Muhammad Yaseen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07813
• Pdf URL: https://arxiv.org/pdf/2409.07813
• Copy Paste: [[2409.07813]] What is YOLOv9: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector(https://arxiv.org/abs/2409.07813)
• Keywords: extraction
• Abstract: This study provides a comprehensive analysis of the YOLOv9 object detection model, focusing on its architectural innovations, training methodologies, and performance improvements over its predecessors. Key advancements, such as the Generalized Efficient Layer Aggregation Network GELAN and Programmable Gradient Information PGI, significantly enhance feature extraction and gradient flow, leading to improved accuracy and efficiency. By incorporating Depthwise Convolutions and the lightweight C3Ghost architecture, YOLOv9 reduces computational complexity while maintaining high precision. Benchmark tests on Microsoft COCO demonstrate its superior mean Average Precision mAP and faster inference times, outperforming YOLOv8 across multiple metrics. The model versatility is highlighted by its seamless deployment across various hardware platforms, from edge devices to high performance GPUs, with built in support for PyTorch and TensorRT integration. This paper provides the first in depth exploration of YOLOv9s internal features and their real world applicability, establishing it as a state of the art solution for real time object detection across industries, from IoT devices to large scale industrial applications.

Title: A Comprehensive Survey on Deep Multimodal Learning with Missing Modality

• Authors: Renjie Wu, Hu Wang, Hsiang-Ting Chen
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07825
• Pdf URL: https://arxiv.org/pdf/2409.07825
• Copy Paste: [[2409.07825]] A Comprehensive Survey on Deep Multimodal Learning with Missing Modality(https://arxiv.org/abs/2409.07825)
• Keywords: privacy
• Abstract: During multimodal model training and reasoning, data samples may miss certain modalities and lead to compromised model performance due to sensor limitations, cost constraints, privacy concerns, data loss, and temporal and spatial factors. This survey provides an overview of recent progress in Multimodal Learning with Missing Modality (MLMM), focusing on deep learning techniques. It is the first comprehensive survey that covers the historical background and the distinction between MLMM and standard multimodal learning setups, followed by a detailed analysis of current MLMM methods, applications, and datasets, concluding with a discussion about challenges and potential future directions in the field.

Title: ReGentS: Real-World Safety-Critical Driving Scenario Generation Made Stable

• Authors: Yuan Yin, Pegah Khayatan, Éloi Zablocki, Alexandre Boulch, Matthieu Cord
• Subjects: cs.LG, cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.07830
• Pdf URL: https://arxiv.org/pdf/2409.07830
• Copy Paste: [[2409.07830]] ReGentS: Real-World Safety-Critical Driving Scenario Generation Made Stable(https://arxiv.org/abs/2409.07830)
• Keywords: robust
• Abstract: Machine learning based autonomous driving systems often face challenges with safety-critical scenarios that are rare in real-world data, hindering their large-scale deployment. While increasing real-world training data coverage could address this issue, it is costly and dangerous. This work explores generating safety-critical driving scenarios by modifying complex real-world regular scenarios through trajectory optimization. We propose ReGentS, which stabilizes generated trajectories and introduces heuristics to avoid obvious collisions and optimization problems. Our approach addresses unrealistic diverging trajectories and unavoidable collision scenarios that are not useful for training robust planner. We also extend the scenario generation framework to handle real-world data with up to 32 agents. Additionally, by using a differentiable simulator, our approach simplifies gradient descent-based optimization involving a simulator, paving the way for future advancements. The code is available at this https URL.

Title: Structured Pruning for Efficient Visual Place Recognition

• Authors: Oliver Grainge, Michael Milford, Indu Bodala, Sarvapali D. Ramchurn, Shoaib Ehsan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07834
• Pdf URL: https://arxiv.org/pdf/2409.07834
• Copy Paste: [[2409.07834]] Structured Pruning for Efficient Visual Place Recognition(https://arxiv.org/abs/2409.07834)
• Keywords: extraction
• Abstract: Visual Place Recognition (VPR) is fundamental for the global re-localization of robots and devices, enabling them to recognize previously visited locations based on visual inputs. This capability is crucial for maintaining accurate mapping and localization over large areas. Given that VPR methods need to operate in real-time on embedded systems, it is critical to optimize these systems for minimal resource consumption. While the most efficient VPR approaches employ standard convolutional backbones with fixed descriptor dimensions, these often lead to redundancy in the embedding space as well as in the network architecture. Our work introduces a novel structured pruning method, to not only streamline common VPR architectures but also to strategically remove redundancies within the feature embedding space. This dual focus significantly enhances the efficiency of the system, reducing both map and model memory requirements and decreasing feature extraction and retrieval latencies. Our approach has reduced memory usage and latency by 21% and 16%, respectively, across models, while minimally impacting recall@1 accuracy by less than 1%. This significant improvement enhances real-time applications on edge devices with negligible accuracy loss.

Title: FPMT: Enhanced Semi-Supervised Model for Traffic Incident Detection

• Authors: Xinying Lu, Jianli Xiao
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07839
• Pdf URL: https://arxiv.org/pdf/2409.07839
• Copy Paste: [[2409.07839]] FPMT: Enhanced Semi-Supervised Model for Traffic Incident Detection(https://arxiv.org/abs/2409.07839)
• Keywords: robust, generative
• Abstract: For traffic incident detection, the acquisition of data and labels is notably resource-intensive, rendering semi-supervised traffic incident detection both a formidable and consequential challenge. Thus, this paper focuses on traffic incident detection with a semi-supervised learning way. It proposes a semi-supervised learning model named FPMT within the framework of MixText. The data augmentation module introduces Generative Adversarial Networks to balance and expand the dataset. During the mix-up process in the hidden space, it employs a probabilistic pseudo-mixing mechanism to enhance regularization and elevate model precision. In terms of training strategy, it initiates with unsupervised training on all data, followed by supervised fine-tuning on a subset of labeled data, and ultimately completing the goal of semi-supervised training. Through empirical validation on four authentic datasets, our FPMT model exhibits outstanding performance across various metrics. Particularly noteworthy is its robust performance even in scenarios with low label rates.

Title: Real-time Multi-view Omnidirectional Depth Estimation System for Robots and Autonomous Driving on Real Scenes

• Authors: Ming Li, Xiong Yang, Chaofan Wu, Jiaheng Li, Pinzhi Wang, Xuejiao Hu, Sidan Du, Yang Li
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.07843
• Pdf URL: https://arxiv.org/pdf/2409.07843
• Copy Paste: [[2409.07843]] Real-time Multi-view Omnidirectional Depth Estimation System for Robots and Autonomous Driving on Real Scenes(https://arxiv.org/abs/2409.07843)
• Keywords: robust
• Abstract: Omnidirectional Depth Estimation has broad application prospects in fields such as robotic navigation and autonomous driving. In this paper, we propose a robotic prototype system and corresponding algorithm designed to validate omnidirectional depth estimation for navigation and obstacle avoidance in real-world scenarios for both robots and vehicles. The proposed HexaMODE system captures 360$^\circ$ depth maps using six surrounding arranged fisheye cameras. We introduce a combined spherical sweeping method and optimize the model architecture for proposed RtHexa-OmniMVS algorithm to achieve real-time omnidirectional depth estimation. To ensure high accuracy, robustness, and generalization in real-world environments, we employ a teacher-student self-training strategy, utilizing large-scale unlabeled real-world data for model training. The proposed algorithm demonstrates high accuracy in various complex real-world scenarios, both indoors and outdoors, achieving an inference speed of 15 fps on edge computing platforms.

Title: A Toolchain for Assisting Migration of Software Executables Towards Post-Quantum Crytography

• Authors: Norrathep Rattanavipanon, Jakapan Suaboot, Warodom Werapun
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07852
• Pdf URL: https://arxiv.org/pdf/2409.07852
• Copy Paste: [[2409.07852]] A Toolchain for Assisting Migration of Software Executables Towards Post-Quantum Crytography(https://arxiv.org/abs/2409.07852)
• Keywords: security
• Abstract: Quantum computing poses a significant global threat to today's security mechanisms. As a result, security experts and public sectors have issued guidelines to help organizations migrate their software to post-quantum cryptography (PQC). Despite these efforts, there is a lack of (semi-)automatic tools to support this transition especially when software is used and deployed as binary executables. To address this gap, in this work, we first propose a set of requirements necessary for a tool to detect quantum-vulnerable software executables. Following these requirements, we introduce QED: a toolchain for Quantum-vulnerable Executable Detection. QED uses a three-phase approach to identify quantum-vulnerable dependencies in a given set of executables, from file-level to API-level, and finally, precise identification of a static trace that triggers a quantum-vulnerable API. We evaluate QED on both a synthetic dataset with four cryptography libraries and a real-world dataset with over 200 software executables. The results demonstrate that: (1) QED discerns quantum-vulnerable from quantum-safe executables with 100% accuracy in the synthetic dataset; (2) QED is practical and scalable, completing analyses on average in less than 4 seconds per real-world executable; and (3) QED reduces the manual workload required by analysts to identify quantum-vulnerable executables in the real-world dataset by more than 90%. We hope that QED can become a crucial tool to facilitate the transition to PQC, particularly for small and medium-sized businesses with limited resources.

Title: Learning Rules from KGs Guided by Language Models

• Authors: Zihang Peng, Daria Stepanova, Vinh Thinh Ho, Heike Adel, Alessandra Russo, Simon Ott
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07869
• Pdf URL: https://arxiv.org/pdf/2409.07869
• Copy Paste: [[2409.07869]] Learning Rules from KGs Guided by Language Models(https://arxiv.org/abs/2409.07869)
• Keywords: extraction
• Abstract: Advances in information extraction have enabled the automatic construction of large knowledge graphs (e.g., Yago, Wikidata or Google KG), which are widely used in many applications like semantic search or data analytics. However, due to their semi-automatic construction, KGs are often incomplete. Rule learning methods, concerned with the extraction of frequent patterns from KGs and casting them into rules, can be applied to predict potentially missing facts. A crucial step in this process is rule ranking. Ranking of rules is especially challenging over highly incomplete or biased KGs (e.g., KGs predominantly storing facts about famous people), as in this case biased rules might fit the data best and be ranked at the top based on standard statistical metrics like rule confidence. To address this issue, prior works proposed to rank rules not only relying on the original KG but also facts predicted by a KG embedding model. At the same time, with the recent rise of Language Models (LMs), several works have claimed that LMs can be used as alternative means for KG completion. In this work, our goal is to verify to which extent the exploitation of LMs is helpful for improving the quality of rule learning systems.

Title: UNIT: Unsupervised Online Instance Segmentation through Time

• Authors: Corentin Sautier, Gilles Puy, Alexandre Boulch, Renaud Marlet, Vincent Lepetit
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07887
• Pdf URL: https://arxiv.org/pdf/2409.07887
• Copy Paste: [[2409.07887]] UNIT: Unsupervised Online Instance Segmentation through Time(https://arxiv.org/abs/2409.07887)
• Keywords: segmentation
• Abstract: Online object segmentation and tracking in Lidar point clouds enables autonomous agents to understand their surroundings and make safe decisions. Unfortunately, manual annotations for these tasks are prohibitively costly. We tackle this problem with the task of class-agnostic unsupervised online instance segmentation and tracking. To that end, we leverage an instance segmentation backbone and propose a new training recipe that enables the online tracking of objects. Our network is trained on pseudo-labels, eliminating the need for manual annotations. We conduct an evaluation using metrics adapted for temporal instance segmentation. Computing these metrics requires temporally-consistent instance labels. When unavailable, we construct these labels using the available 3D bounding boxes and semantic labels in the dataset. We compare our method against strong baselines and demonstrate its superiority across two different outdoor Lidar datasets.

Title: BLens: Contrastive Captioning of Binary Functions using Ensemble Embedding

• Authors: Tristan Benoit, Yunru Wang, Moritz Dannehl, Johannes Kinder
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07889
• Pdf URL: https://arxiv.org/pdf/2409.07889
• Copy Paste: [[2409.07889]] BLens: Contrastive Captioning of Binary Functions using Ensemble Embedding(https://arxiv.org/abs/2409.07889)
• Keywords: transformer
• Abstract: Function names can greatly aid human reverse engineers, which has spurred development of machine learning-based approaches to predicting function names in stripped binaries. Much current work in this area now uses transformers, applying a metaphor of machine translation from code to function names. Still, function naming models face challenges in generalizing to projects completely unrelated to the training set. In this paper, we take a completely new approach by transferring advances in automated image captioning to the domain of binary reverse engineering, such that different parts of a binary function can be associated with parts of its name. We propose BLens, which combines multiple binary function embeddings into a new ensemble representation, aligns it with the name representation latent space via a contrastive learning approach, and generates function names with a transformer architecture tailored for function names. In our experiments, we demonstrate that BLens significantly outperforms the state of the art. In the usual setting of splitting per binary, we achieve an $F_1$ score of 0.77 compared to 0.67. Moreover, in the cross-project setting, which emphasizes generalizability, we achieve an $F_1$ score of 0.46 compared to 0.29.

Title: Microscopic-Mamba: Revealing the Secrets of Microscopic Images with Just 4M Parameters

• Authors: Shun Zou, Zhuo Zhang, Yi Zou, Guangwei Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07896
• Pdf URL: https://arxiv.org/pdf/2409.07896
• Copy Paste: [[2409.07896]] Microscopic-Mamba: Revealing the Secrets of Microscopic Images with Just 4M Parameters(https://arxiv.org/abs/2409.07896)
• Keywords: extraction, transformer
• Abstract: In the field of medical microscopic image classification (MIC), CNN-based and Transformer-based models have been extensively studied. However, CNNs struggle with modeling long-range dependencies, limiting their ability to fully utilize semantic information in images. Conversely, Transformers are hampered by the complexity of quadratic computations. To address these challenges, we propose a model based on the Mamba architecture: Microscopic-Mamba. Specifically, we designed the Partially Selected Feed-Forward Network (PSFFN) to replace the last linear layer of the Visual State Space Module (VSSM), enhancing Mamba's local feature extraction capabilities. Additionally, we introduced the Modulation Interaction Feature Aggregation (MIFA) module to effectively modulate and dynamically aggregate global and local features. We also incorporated a parallel VSSM mechanism to improve inter-channel information interaction while reducing the number of parameters. Extensive experiments have demonstrated that our method achieves state-of-the-art performance on five public datasets. Code is available at this https URL

Title: Building a Cybersecurity Risk Metamodel for Improved Method and Tool Integration

• Authors: Christophe Ponsard
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.07906
• Pdf URL: https://arxiv.org/pdf/2409.07906
• Copy Paste: [[2409.07906]] Building a Cybersecurity Risk Metamodel for Improved Method and Tool Integration(https://arxiv.org/abs/2409.07906)
• Keywords: security, protect
• Abstract: Nowadays, companies are highly exposed to cyber security threats. In many industrial domains, protective measures are being deployed and actively supported by standards. However the global process remains largely dependent on document driven approach or partial modelling which impacts both the efficiency and effectiveness of the cybersecurity process from the risk analysis step. In this paper, we report on our experience in applying a model-driven approach on the initial risk analysis step in connection with a later security testing. Our work rely on a common metamodel which is used to map, synchronise and ensure information traceability across different tools. We validate our approach using different scenarios relying domain modelling, system modelling, risk assessment and security testing tools.

Title: UGAD: Universal Generative AI Detector utilizing Frequency Fingerprints

• Authors: Inzamamul Alam, Muhammad Shahid Muneer, Simon S. Woo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07913
• Pdf URL: https://arxiv.org/pdf/2409.07913
• Copy Paste: [[2409.07913]] UGAD: Universal Generative AI Detector utilizing Frequency Fingerprints(https://arxiv.org/abs/2409.07913)
• Keywords: extraction, diffusion, generative
• Abstract: In the wake of a fabricated explosion image at the Pentagon, an ability to discern real images from fake counterparts has never been more critical. Our study introduces a novel multi-modal approach to detect AI-generated images amidst the proliferation of new generation methods such as Diffusion models. Our method, UGAD, encompasses three key detection steps: First, we transform the RGB images into YCbCr channels and apply an Integral Radial Operation to emphasize salient radial features. Secondly, the Spatial Fourier Extraction operation is used for a spatial shift, utilizing a pre-trained deep learning network for optimal feature extraction. Finally, the deep neural network classification stage processes the data through dense layers using softmax for classification. Our approach significantly enhances the accuracy of differentiating between real and AI-generated images, as evidenced by a 12.64% increase in accuracy and 28.43% increase in AUC compared to existing state-of-the-art methods.

Title: Mobile App Security Trends and Topics: An Examination of Questions From Stack Overflow

• Authors: Timothy Huo, Ana Catarina Araújo, Jake Imanaka, Anthony Peruma, Rick Kazman
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.07926
• Pdf URL: https://arxiv.org/pdf/2409.07926
• Copy Paste: [[2409.07926]] Mobile App Security Trends and Topics: An Examination of Questions From Stack Overflow(https://arxiv.org/abs/2409.07926)
• Keywords: secure, security
• Abstract: The widespread use of smartphones and tablets has made society heavily reliant on mobile applications (apps) for accessing various resources and services. These apps often handle sensitive personal, financial, and health data, making app security a critical concern for developers. While there is extensive research on software security topics like malware and vulnerabilities, less is known about the practical security challenges mobile app developers face and the guidance they seek. \rev{In this study, we mine Stack Overflow for questions on mobile app security, which we analyze using quantitative and qualitative techniques.} The findings reveal that Stack Overflow is a major resource for developers seeking help with mobile app security, especially for Android apps, and identifies seven main categories of security questions: Secured Communications, Database, App Distribution Service, Encryption, Permissions, File-Specific, and General Security. Insights from this research can inform the development of tools, techniques, and resources by the research and vendor community to better support developers in securing their mobile apps.

Title: Reinforcement Learning Discovers Efficient Decentralized Graph Path Search Strategies

• Authors: Alexei Pisacane, Victor-Alexandru Darvariu, Mirco Musolesi
• Subjects: cs.LG, cs.AI, cs.MA, cs.SI
• Abstract URL: https://arxiv.org/abs/2409.07932
• Pdf URL: https://arxiv.org/pdf/2409.07932
• Copy Paste: [[2409.07932]] Reinforcement Learning Discovers Efficient Decentralized Graph Path Search Strategies(https://arxiv.org/abs/2409.07932)
• Keywords: privacy
• Abstract: Graph path search is a classic computer science problem that has been recently approached with Reinforcement Learning (RL) due to its potential to outperform prior methods. Existing RL techniques typically assume a global view of the network, which is not suitable for large-scale, dynamic, and privacy-sensitive settings. An area of particular interest is search in social networks due to its numerous applications. Inspired by seminal work in experimental sociology, which showed that decentralized yet efficient search is possible in social networks, we frame the problem as a collaborative task between multiple agents equipped with a limited local view of the network. We propose a multi-agent approach for graph path search that successfully leverages both homophily and structural heterogeneity. Our experiments, carried out over synthetic and real-world social networks, demonstrate that our model significantly outperforms learned and heuristic baselines. Furthermore, our results show that meaningful embeddings for graph navigation can be constructed using reward-driven learning.

Title: Control+Shift: Generating Controllable Distribution Shifts

• Authors: Roy Friedman, Rhea Chowers
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07940
• Pdf URL: https://arxiv.org/pdf/2409.07940
• Copy Paste: [[2409.07940]] Control+Shift: Generating Controllable Distribution Shifts(https://arxiv.org/abs/2409.07940)
• Keywords: robust, generative
• Abstract: We propose a new method for generating realistic datasets with distribution shifts using any decoder-based generative model. Our approach systematically creates datasets with varying intensities of distribution shifts, facilitating a comprehensive analysis of model performance degradation. We then use these generated datasets to evaluate the performance of various commonly used networks and observe a consistent decline in performance with increasing shift intensity, even when the effect is almost perceptually unnoticeable to the human eye. We see this degradation even when using data augmentations. We also find that enlarging the training dataset beyond a certain point has no effect on the robustness and that stronger inductive biases increase robustness.

Title: Enhanced Online Grooming Detection Employing Context Determination and Message-Level Analysis

• Authors: Jake Street, Isibor Ihianle, Funminiyi Olajide, Ahmad Lotfi
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07958
• Pdf URL: https://arxiv.org/pdf/2409.07958
• Copy Paste: [[2409.07958]] Enhanced Online Grooming Detection Employing Context Determination and Message-Level Analysis(https://arxiv.org/abs/2409.07958)
• Keywords: attack, robust
• Abstract: Online Grooming (OG) is a prevalent threat facing predominately children online, with groomers using deceptive methods to prey on the vulnerability of children on social media/messaging platforms. These attacks can have severe psychological and physical impacts, including a tendency towards revictimization. Current technical measures are inadequate, especially with the advent of end-to-end encryption which hampers message monitoring. Existing solutions focus on the signature analysis of child abuse media, which does not effectively address real-time OG detection. This paper proposes that OG attacks are complex, requiring the identification of specific communication patterns between adults and children. It introduces a novel approach leveraging advanced models such as BERT and RoBERTa for Message-Level Analysis and a Context Determination approach for classifying actor interactions, including the introduction of Actor Significance Thresholds and Message Significance Thresholds. The proposed method aims to enhance accuracy and robustness in detecting OG by considering the dynamic and multi-faceted nature of these attacks. Cross-dataset experiments evaluate the robustness and versatility of our approach. This paper's contributions include improved detection methodologies and the potential for application in various scenarios, addressing gaps in current literature and practices.

Title: Do Vision Foundation Models Enhance Domain Generalization in Medical Image Segmentation?

• Authors: Kerem Cekmeceli, Meva Himmetoglu, Guney I. Tombak, Anna Susmelj, Ertunc Erdil, Ender Konukoglu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07960
• Pdf URL: https://arxiv.org/pdf/2409.07960
• Copy Paste: [[2409.07960]] Do Vision Foundation Models Enhance Domain Generalization in Medical Image Segmentation?(https://arxiv.org/abs/2409.07960)
• Keywords: segmentation
• Abstract: Neural networks achieve state-of-the-art performance in many supervised learning tasks when the training data distribution matches the test data distribution. However, their performance drops significantly under domain (covariate) shift, a prevalent issue in medical image segmentation due to varying acquisition settings across different scanner models and protocols. Recently, foundational models (FMs) trained on large datasets have gained attention for their ability to be adapted for downstream tasks and achieve state-of-the-art performance with excellent generalization capabilities on natural images. However, their effectiveness in medical image segmentation remains underexplored. In this paper, we investigate the domain generalization performance of various FMs, including DinoV2, SAM, MedSAM, and MAE, when fine-tuned using various parameter-efficient fine-tuning (PEFT) techniques such as Ladder and Rein (+LoRA) and decoder heads. We introduce a novel decode head architecture, HQHSAM, which simply integrates elements from two state-of-the-art decoder heads, HSAM and HQSAM, to enhance segmentation performance. Our extensive experiments on multiple datasets, encompassing various anatomies and modalities, reveal that FMs, particularly with the HQHSAM decode head, improve domain generalization for medical image segmentation. Moreover, we found that the effectiveness of PEFT techniques varies across different FMs. These findings underscore the potential of FMs to enhance the domain generalization performance of neural networks in medical image segmentation across diverse clinical settings, providing a solid foundation for future research. Code and models are available for research purposes at \url{this https URL}.

Title: Estimating atmospheric variables from Digital Typhoon Satellite Images via Conditional Denoising Diffusion Models

• Authors: Zhangyue Ling, Pritthijit Nath, César Quilodrán-Casas
• Subjects: cs.CV, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2409.07961
• Pdf URL: https://arxiv.org/pdf/2409.07961
• Copy Paste: [[2409.07961]] Estimating atmospheric variables from Digital Typhoon Satellite Images via Conditional Denoising Diffusion Models(https://arxiv.org/abs/2409.07961)
• Keywords: robust, diffusion
• Abstract: This study explores the application of diffusion models in the field of typhoons, predicting multiple ERA5 meteorological variables simultaneously from Digital Typhoon satellite images. The focus of this study is taken to be Taiwan, an area very vulnerable to typhoons. By comparing the performance of Conditional Denoising Diffusion Probability Model (CDDPM) with Convolutional Neural Networks (CNN) and Squeeze-and-Excitation Networks (SENet), results suggest that the CDDPM performs best in generating accurate and realistic meteorological data. Specifically, CDDPM achieved a PSNR of 32.807, which is approximately 7.9% higher than CNN and 5.5% higher than SENet. Furthermore, CDDPM recorded an RMSE of 0.032, showing a 11.1% improvement over CNN and 8.6% improvement over SENet. A key application of this research can be for imputation purposes in missing meteorological datasets and generate additional high-quality meteorological data using satellite images. It is hoped that the results of this analysis will enable more robust and detailed forecasting, reducing the impact of severe weather events on vulnerable regions. Code accessible at this https URL.

Title: ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAE

• Authors: Sichun Wu, Kazi Injamamul Haque, Zerrin Yumak
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07966
• Pdf URL: https://arxiv.org/pdf/2409.07966
• Copy Paste: [[2409.07966]] ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAE(https://arxiv.org/abs/2409.07966)
• Keywords: generative
• Abstract: Audio-driven 3D facial animation synthesis has been an active field of research with attention from both academia and industry. While there are promising results in this area, recent approaches largely focus on lip-sync and identity control, neglecting the role of emotions and emotion control in the generative process. That is mainly due to the lack of emotionally rich facial animation data and algorithms that can synthesize speech animations with emotional expressions at the same time. In addition, majority of the models are deterministic, meaning given the same audio input, they produce the same output motion. We argue that emotions and non-determinism are crucial to generate diverse and emotionally-rich facial animations. In this paper, we propose ProbTalk3D a non-deterministic neural network approach for emotion controllable speech-driven 3D facial animation synthesis using a two-stage VQ-VAE model and an emotionally rich facial animation dataset 3DMEAD. We provide an extensive comparative analysis of our model against the recent 3D facial animation synthesis approaches, by evaluating the results objectively, qualitatively, and with a perceptual user study. We highlight several objective metrics that are more suitable for evaluating stochastic outputs and use both in-the-wild and ground truth data for subjective evaluation. To our knowledge, that is the first non-deterministic 3D facial animation synthesis method incorporating a rich emotion dataset and emotion control with emotion labels and intensity levels. Our evaluation demonstrates that the proposed model achieves superior performance compared to state-of-the-art emotion-controlled, deterministic and non-deterministic models. We recommend watching the supplementary video for quality judgement. The entire codebase is publicly available (this https URL).

Title: Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localization

• Authors: Ling Xing, Hongyu Qu, Rui Yan, Xiangbo Shu, Jinhui Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07967
• Pdf URL: https://arxiv.org/pdf/2409.07967
• Copy Paste: [[2409.07967]] Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localization(https://arxiv.org/abs/2409.07967)
• Keywords: extraction
• Abstract: Dense-localization Audio-Visual Events (DAVE) aims to identify time boundaries and corresponding categories for events that can be heard and seen concurrently in an untrimmed video. Existing methods typically encode audio and visual representation separately without any explicit cross-modal alignment constraint. Then they adopt dense cross-modal attention to integrate multimodal information for DAVE. Thus these methods inevitably aggregate irrelevant noise and events, especially in complex and long videos, leading to imprecise detection. In this paper, we present LOCO, a Locality-aware cross-modal Correspondence learning framework for DAVE. The core idea is to explore local temporal continuity nature of audio-visual events, which serves as informative yet free supervision signals to guide the filtering of irrelevant information and inspire the extraction of complementary multimodal information during both unimodal and cross-modal learning stages. i) Specifically, LOCO applies Locality-aware Correspondence Correction (LCC) to uni-modal features via leveraging cross-modal local-correlated properties without any extra annotations. This enforces uni-modal encoders to highlight similar semantics shared by audio and visual features. ii) To better aggregate such audio and visual features, we further customize Cross-modal Dynamic Perception layer (CDP) in cross-modal feature pyramid to understand local temporal patterns of audio-visual events by imposing local consistency within multimodal features in a data-driven manner. By incorporating LCC and CDP, LOCO provides solid performance gains and outperforms existing methods for DAVE. The source code will be released.

Title: Enhancing Few-Shot Image Classification through Learnable Multi-Scale Embedding and Attention Mechanisms

• Authors: Fatemeh Askari, Amirreza Fateh, Mohammad Reza Mohammadi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07989
• Pdf URL: https://arxiv.org/pdf/2409.07989
• Copy Paste: [[2409.07989]] Enhancing Few-Shot Image Classification through Learnable Multi-Scale Embedding and Attention Mechanisms(https://arxiv.org/abs/2409.07989)
• Keywords: robust
• Abstract: In the context of few-shot classification, the goal is to train a classifier using a limited number of samples while maintaining satisfactory performance. However, traditional metric-based methods exhibit certain limitations in achieving this objective. These methods typically rely on a single distance value between the query feature and support feature, thereby overlooking the contribution of shallow features. To overcome this challenge, we propose a novel approach in this paper. Our approach involves utilizing multi-output embedding network that maps samples into distinct feature spaces. The proposed method extract feature vectors at different stages, enabling the model to capture both global and abstract features. By utilizing these diverse feature spaces, our model enhances its performance. Moreover, employing a self-attention mechanism improves the refinement of features at each stage, leading to even more robust representations and improved overall performance. Furthermore, assigning learnable weights to each stage significantly improved performance and results. We conducted comprehensive evaluations on the MiniImageNet and FC100 datasets, specifically in the 5-way 1-shot and 5-way 5-shot scenarios. Additionally, we performed a cross-domain task from MiniImageNet to the CUB dataset, achieving high accuracy in the testing domain. These evaluations demonstrate the efficacy of our proposed method in comparison to state-of-the-art approaches. this https URL

Title: Depth Matters: Exploring Deep Interactions of RGB-D for Semantic Segmentation in Traffic Scenes

• Authors: Siyu Chen, Ting Han, Changshe Zhang, Weiquan Liu, Jinhe Su, Zongyue Wang, Guorong Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07995
• Pdf URL: https://arxiv.org/pdf/2409.07995
• Copy Paste: [[2409.07995]] Depth Matters: Exploring Deep Interactions of RGB-D for Semantic Segmentation in Traffic Scenes(https://arxiv.org/abs/2409.07995)
• Keywords: transformer, segmentation
• Abstract: RGB-D has gradually become a crucial data source for understanding complex scenes in assisted driving. However, existing studies have paid insufficient attention to the intrinsic spatial properties of depth maps. This oversight significantly impacts the attention representation, leading to prediction errors caused by attention shift issues. To this end, we propose a novel learnable Depth interaction Pyramid Transformer (DiPFormer) to explore the effectiveness of depth. Firstly, we introduce Depth Spatial-Aware Optimization (Depth SAO) as offset to represent real-world spatial relationships. Secondly, the similarity in the feature space of RGB-D is learned by Depth Linear Cross-Attention (Depth LCA) to clarify spatial differences at the pixel level. Finally, an MLP Decoder is utilized to effectively fuse multi-scale features for meeting real-time requirements. Comprehensive experiments demonstrate that the proposed DiPFormer significantly addresses the issue of attention misalignment in both road detection (+7.5%) and semantic segmentation (+4.9% / +1.5%) tasks. DiPFormer achieves state-of-the-art performance on the KITTI (97.57% F-score on KITTI road and 68.74% mIoU on KITTI-360) and Cityscapes (83.4% mIoU) datasets.

Title: Privacy-preserving federated prediction of pain intensity change based on multi-center survey data

• Authors: Supratim Das, Mahdie Rafie, Paula Kammer, Søren T. Skou, Dorte T. Grønne, Ewa M. Roos, André Hajek, Hans-Helmut König, Md Shihab Ullaha, Niklas Probul, Jan Baumbacha, Linda Baumbach
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07997
• Pdf URL: https://arxiv.org/pdf/2409.07997
• Copy Paste: [[2409.07997]] Privacy-preserving federated prediction of pain intensity change based on multi-center survey data(https://arxiv.org/abs/2409.07997)
• Keywords: privacy, robust, federate
• Abstract: Background: Patient-reported survey data are used to train prognostic models aimed at improving healthcare. However, such data are typically available multi-centric and, for privacy reasons, cannot easily be centralized in one data repository. Models trained locally are less accurate, robust, and generalizable. We present and apply privacy-preserving federated machine learning techniques for prognostic model building, where local survey data never leaves the legally safe harbors of the medical centers. Methods: We used centralized, local, and federated learning techniques on two healthcare datasets (GLA:D data from the five health regions of Denmark and international SHARE data of 27 countries) to predict two different health outcomes. We compared linear regression, random forest regression, and random forest classification models trained on local data with those trained on the entire data in a centralized and in a federated fashion. Results: In GLA:D data, federated linear regression (R2 0.34, RMSE 18.2) and federated random forest regression (R2 0.34, RMSE 18.3) models outperform their local counterparts (i.e., R2 0.32, RMSE 18.6, R2 0.30, RMSE 18.8) with statistical significance. We also found that centralized models (R2 0.34, RMSE 18.2, R2 0.32, RMSE 18.5, respectively) did not perform significantly better than the federated models. In SHARE, the federated model (AC 0.78, AUROC: 0.71) and centralized model (AC 0.84, AUROC: 0.66) perform significantly better than the local models (AC: 0.74, AUROC: 0.69). Conclusion: Federated learning enables the training of prognostic models from multi-center surveys without compromising privacy and with only minimal or no compromise regarding model performance.

Title: Network Anomaly Traffic Detection via Multi-view Feature Fusion

• Authors: Song Hao, Wentao Fu, Xuanze Chen, Chengxiang Jin, Jiajun Zhou, Shanqing Yu, Qi Xuan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08020
• Pdf URL: https://arxiv.org/pdf/2409.08020
• Copy Paste: [[2409.08020]] Network Anomaly Traffic Detection via Multi-view Feature Fusion(https://arxiv.org/abs/2409.08020)
• Keywords: attack
• Abstract: Traditional anomalous traffic detection methods are based on single-view analysis, which has obvious limitations in dealing with complex attacks and encrypted communications. In this regard, we propose a Multi-view Feature Fusion (MuFF) method for network anomaly traffic detection. MuFF models the temporal and interactive relationships of packets in network traffic based on the temporal and interactive viewpoints respectively. It learns temporal and interactive features. These features are then fused from different perspectives for anomaly traffic detection. Extensive experiments on six real traffic datasets show that MuFF has excellent performance in network anomalous traffic detection, which makes up for the shortcomings of detection under a single perspective.

Title: Scribble-Guided Diffusion for Training-free Text-to-Image Generation

• Authors: Seonho Lee, Jiho Choi, Seohyun Lim, Jiwook Kim, Hyunjung Shim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08026
• Pdf URL: https://arxiv.org/pdf/2409.08026
• Copy Paste: [[2409.08026]] Scribble-Guided Diffusion for Training-free Text-to-Image Generation(https://arxiv.org/abs/2409.08026)
• Keywords: diffusion
• Abstract: Recent advancements in text-to-image diffusion models have demonstrated remarkable success, yet they often struggle to fully capture the user's intent. Existing approaches using textual inputs combined with bounding boxes or region masks fall short in providing precise spatial guidance, often leading to misaligned or unintended object orientation. To address these limitations, we propose Scribble-Guided Diffusion (ScribbleDiff), a training-free approach that utilizes simple user-provided scribbles as visual prompts to guide image generation. However, incorporating scribbles into diffusion models presents challenges due to their sparse and thin nature, making it difficult to ensure accurate orientation alignment. To overcome these challenges, we introduce moment alignment and scribble propagation, which allow for more effective and flexible alignment between generated images and scribble inputs. Experimental results on the PASCAL-Scribble dataset demonstrate significant improvements in spatial control and consistency, showcasing the effectiveness of scribble-based guidance in diffusion models. Our code is available at this https URL.

Title: Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreaking

• Authors: Stav Cohen, Ron Bitton, Ben Nassi
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08045
• Pdf URL: https://arxiv.org/pdf/2409.08045
• Copy Paste: [[2409.08045]] Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreaking(https://arxiv.org/abs/2409.08045)
• Keywords: protect, attack, extraction, membership infer
• Abstract: In this paper, we show that with the ability to jailbreak a GenAI model, attackers can escalate the outcome of attacks against RAG-based GenAI-powered applications in severity and scale. In the first part of the paper, we show that attackers can escalate RAG membership inference attacks and RAG entity extraction attacks to RAG documents extraction attacks, forcing a more severe outcome compared to existing attacks. We evaluate the results obtained from three extraction methods, the influence of the type and the size of five embeddings algorithms employed, the size of the provided context, and the GenAI engine. We show that attackers can extract 80%-99.8% of the data stored in the database used by the RAG of a Q&A chatbot. In the second part of the paper, we show that attackers can escalate the scale of RAG data poisoning attacks from compromising a single GenAI-powered application to compromising the entire GenAI ecosystem, forcing a greater scale of damage. This is done by crafting an adversarial self-replicating prompt that triggers a chain reaction of a computer worm within the ecosystem and forces each affected application to perform a malicious activity and compromise the RAG of additional applications. We evaluate the performance of the worm in creating a chain of confidential data extraction about users within a GenAI ecosystem of GenAI-powered email assistants and analyze how the performance of the worm is affected by the size of the context, the adversarial self-replicating prompt used, the type and size of the embeddings algorithm employed, and the number of hops in the propagation. Finally, we review and analyze guardrails to protect RAG-based inference and discuss the tradeoffs.

Title: Spatial Adaptation Layer: Interpretable Domain Adaptation For Biosignal Sensor Array Applications

• Authors: Joao Pereira, Michael Alummoottil, Dimitrios Halatsis, Dario Farina
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.08058
• Pdf URL: https://arxiv.org/pdf/2409.08058
• Copy Paste: [[2409.08058]] Spatial Adaptation Layer: Interpretable Domain Adaptation For Biosignal Sensor Array Applications(https://arxiv.org/abs/2409.08058)
• Keywords: robust, interpretability
• Abstract: Biosignal acquisition is key for healthcare applications and wearable devices, with machine learning offering promising methods for processing signals like surface electromyography (sEMG) and electroencephalography (EEG). Despite high within-session performance, intersession performance is hindered by electrode shift, a known issue across modalities. Existing solutions often require large and expensive datasets and/or lack robustness and interpretability. Thus, we propose the Spatial Adaptation Layer (SAL), which can be prepended to any biosignal array model and learns a parametrized affine transformation at the input between two recording sessions. We also introduce learnable baseline normalization (LBN) to reduce baseline fluctuations. Tested on two HD-sEMG gesture recognition datasets, SAL and LBN outperform standard fine-tuning on regular arrays, achieving competitive performance even with a logistic regressor, with orders of magnitude less, physically interpretable parameters. Our ablation study shows that forearm circumferential translations account for the majority of performance improvements, in line with sEMG physiological expectations.

Title: Q-value Regularized Decision ConvFormer for Offline Reinforcement Learning

• Authors: Teng Yan, Zhendong Ruan, Yaobang Cai, Yu Han, Wenxian Li, Yang Zhang
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.08062
• Pdf URL: https://arxiv.org/pdf/2409.08062
• Copy Paste: [[2409.08062]] Q-value Regularized Decision ConvFormer for Offline Reinforcement Learning(https://arxiv.org/abs/2409.08062)
• Keywords: transformer
• Abstract: As a data-driven paradigm, offline reinforcement learning (Offline RL) has been formulated as sequence modeling, where the Decision Transformer (DT) has demonstrated exceptional capabilities. Unlike previous reinforcement learning methods that fit value functions or compute policy gradients, DT adjusts the autoregressive model based on the expected returns, past states, and actions, using a causally masked Transformer to output the optimal action. However, due to the inconsistency between the sampled returns within a single trajectory and the optimal returns across multiple trajectories, it is challenging to set an expected return to output the optimal action and stitch together suboptimal trajectories. Decision ConvFormer (DC) is easier to understand in the context of modeling RL trajectories within a Markov Decision Process compared to DT. We propose the Q-value Regularized Decision ConvFormer (QDC), which combines the understanding of RL trajectories by DC and incorporates a term that maximizes action values using dynamic programming methods during training. This ensures that the expected returns of the sampled actions are consistent with the optimal returns. QDC achieves excellent performance on the D4RL benchmark, outperforming or approaching the optimal level in all tested environments. It particularly demonstrates outstanding competitiveness in trajectory stitching capability.

Title: Diffusion-Based Image-to-Image Translation by Noise Correction via Prompt Interpolation

• Authors: Junsung Lee, Minsoo Kang, Bohyung Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08077
• Pdf URL: https://arxiv.org/pdf/2409.08077
• Copy Paste: [[2409.08077]] Diffusion-Based Image-to-Image Translation by Noise Correction via Prompt Interpolation(https://arxiv.org/abs/2409.08077)
• Keywords: diffusion
• Abstract: We propose a simple but effective training-free approach tailored to diffusion-based image-to-image translation. Our approach revises the original noise prediction network of a pretrained diffusion model by introducing a noise correction term. We formulate the noise correction term as the difference between two noise predictions; one is computed from the denoising network with a progressive interpolation of the source and target prompt embeddings, while the other is the noise prediction with the source prompt embedding. The final noise prediction network is given by a linear combination of the standard denoising term and the noise correction term, where the former is designed to reconstruct must-be-preserved regions while the latter aims to effectively edit regions of interest relevant to the target prompt. Our approach can be easily incorporated into existing image-to-image translation methods based on diffusion models. Extensive experiments verify that the proposed technique achieves outstanding performance with low latency and consistently improves existing frameworks when combined with them.

Title: SimMAT: Exploring Transferability from Vision Foundation Models to Any Image Modality

• Authors: Chenyang Lei, Liyi Chen, Jun Cen, Xiao Chen, Zhen Lei, Felix Heide, Ziwei Liu, Qifeng Chen, Zhaoxiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08083
• Pdf URL: https://arxiv.org/pdf/2409.08083
• Copy Paste: [[2409.08083]] SimMAT: Exploring Transferability from Vision Foundation Models to Any Image Modality(https://arxiv.org/abs/2409.08083)
• Keywords: segmentation
• Abstract: Foundation models like ChatGPT and Sora that are trained on a huge scale of data have made a revolutionary social impact. However, it is extremely challenging for sensors in many different fields to collect similar scales of natural images to train strong foundation models. To this end, this work presents a simple and effective framework SimMAT to study an open problem: the transferability from vision foundation models trained on natural RGB images to other image modalities of different physical properties (e.g., polarization). SimMAT consists of a modality-agnostic transfer layer (MAT) and a pretrained foundation model. We apply SimMAT to a representative vision foundation model Segment Anything Model (SAM) to support any evaluated new image modality. Given the absence of relevant benchmarks, we construct a new benchmark to evaluate the transfer learning performance. Our experiments confirm the intriguing potential of transferring vision foundation models in enhancing other sensors' performance. Specifically, SimMAT can improve the segmentation performance (mIoU) from 22.15% to 53.88% on average for evaluated modalities and consistently outperforms other baselines. We hope that SimMAT can raise awareness of cross-modal transfer learning and benefit various fields for better results with vision foundation models.

Title: Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks

• Authors: Benji Peng, Keyu Chen, Ming Li, Pohsun Feng, Ziqian Bi, Junyu Liu, Qian Niu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.08087
• Pdf URL: https://arxiv.org/pdf/2409.08087
• Copy Paste: [[2409.08087]] Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks(https://arxiv.org/abs/2409.08087)
• Keywords: security, defense, attack, watermark, large language model
• Abstract: Large Language Models (LLMs) demonstrate impressive capabilities across various fields, yet their increasing use raises critical security concerns. This article reviews recent literature addressing key issues in LLM security, with a focus on accuracy, bias, content detection, and vulnerability to attacks. Issues related to inaccurate or misleading outputs from LLMs is discussed, with emphasis on the implementation from fact-checking methodologies to enhance response reliability. Inherent biases within LLMs are critically examined through diverse evaluation techniques, including controlled input studies and red teaming exercises. A comprehensive analysis of bias mitigation strategies is presented, including approaches from pre-processing interventions to in-training adjustments and post-processing refinements. The article also probes the complexity of distinguishing LLM-generated content from human-produced text, introducing detection mechanisms like DetectGPT and watermarking techniques while noting the limitations of machine learning enabled classifiers under intricate circumstances. Moreover, LLM vulnerabilities, including jailbreak attacks and prompt injection exploits, are analyzed by looking into different case studies and large-scale competitions like HackAPrompt. This review is concluded by retrospecting defense mechanisms to safeguard LLMs, accentuating the need for more extensive research into the LLM security field.

Title: EZIGen: Enhancing zero-shot subject-driven image generation with precise subject encoding and decoupled guidance

• Authors: Zicheng Duan, Yuxuan Ding, Chenhui Gou, Ziqin Zhou, Ethan Smith, Lingqiao Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08091
• Pdf URL: https://arxiv.org/pdf/2409.08091
• Copy Paste: [[2409.08091]] EZIGen: Enhancing zero-shot subject-driven image generation with precise subject encoding and decoupled guidance(https://arxiv.org/abs/2409.08091)
• Keywords: diffusion
• Abstract: Zero-shot subject-driven image generation aims to produce images that incorporate a subject from a given example image. The challenge lies in preserving the subject's identity while aligning with the text prompt, which often requires modifying certain aspects of the subject's appearance. Despite advancements in diffusion model based methods, existing approaches still struggle to balance identity preservation with text prompt alignment. In this study, we conducted an in-depth investigation into this issue and uncovered key insights for achieving effective identity preservation while maintaining a strong balance. Our key findings include: (1) the design of the subject image encoder significantly impacts identity preservation quality, and (2) generating an initial layout is crucial for both text alignment and identity preservation. Building on these insights, we introduce a new approach called EZIGen, which employs two main strategies: a carefully crafted subject image Encoder based on the UNet architecture of the pretrained Stable Diffusion model to ensure high-quality identity transfer, following a process that decouples the guidance stages and iteratively refines the initial image layout. Through these strategies, EZIGen achieves state-of-the-art results on multiple subject-driven benchmarks with a unified model and 100 times less training data.

Title: The CLC-UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunal

• Authors: Huiyuan Xie, Felix Steffek, Joana Ribeiro de Faria, Christine Carter, Jonathan Rutherford
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08098
• Pdf URL: https://arxiv.org/pdf/2409.08098
• Copy Paste: [[2409.08098]] The CLC-UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunal(https://arxiv.org/abs/2409.08098)
• Keywords: transformer, large language model
• Abstract: This paper explores the intersection of technological innovation and access to justice by developing a benchmark for predicting case outcomes in the UK Employment Tribunal (UKET). To address the challenge of extensive manual annotation, the study employs a large language model (LLM) for automatic annotation, resulting in the creation of the CLC-UKET dataset. The dataset consists of approximately 19,000 UKET cases and their metadata. Comprehensive legal annotations cover facts, claims, precedent references, statutory references, case outcomes, reasons and jurisdiction codes. Facilitated by the CLC-UKET data, we examine a multi-class case outcome prediction task in the UKET. Human predictions are collected to establish a performance reference for model comparison. Empirical results from baseline models indicate that finetuned transformer models outperform zero-shot and few-shot LLMs on the UKET prediction task. The performance of zero-shot LLMs can be enhanced by integrating task-related information into few-shot examples. We hope that the CLC-UKET dataset, along with human annotations and empirical findings, can serve as a valuable benchmark for employment-related dispute resolution.

Title: Bayesian Self-Training for Semi-Supervised 3D Segmentation

• Authors: Ozan Unal, Christos Sakaridis, Luc Van Gool
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08102
• Pdf URL: https://arxiv.org/pdf/2409.08102
• Copy Paste: [[2409.08102]] Bayesian Self-Training for Semi-Supervised 3D Segmentation(https://arxiv.org/abs/2409.08102)
• Keywords: segmentation
• Abstract: 3D segmentation is a core problem in computer vision and, similarly to many other dense prediction tasks, it requires large amounts of annotated data for adequate training. However, densely labeling 3D point clouds to employ fully-supervised training remains too labor intensive and expensive. Semi-supervised training provides a more practical alternative, where only a small set of labeled data is given, accompanied by a larger unlabeled set. This area thus studies the effective use of unlabeled data to reduce the performance gap that arises due to the lack of annotations. In this work, inspired by Bayesian deep learning, we first propose a Bayesian self-training framework for semi-supervised 3D semantic segmentation. Employing stochastic inference, we generate an initial set of pseudo-labels and then filter these based on estimated point-wise uncertainty. By constructing a heuristic $n$-partite matching algorithm, we extend the method to semi-supervised 3D instance segmentation, and finally, with the same building blocks, to dense 3D visual grounding. We demonstrate state-of-the-art results for our semi-supervised method on SemanticKITTI and ScribbleKITTI for 3D semantic segmentation and on ScanNet and S3DIS for 3D instance segmentation. We further achieve substantial improvements in dense 3D visual grounding over supervised-only baselines on ScanRefer. Our project page is available at this http URL.

Title: Towards a graph-based foundation model for network traffic analysis

• Authors: Louis Van Langendonck, Ismael Castell-Uroz, Pere Barlet-Ros
• Subjects: cs.LG, cs.AI, cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.08111
• Pdf URL: https://arxiv.org/pdf/2409.08111
• Copy Paste: [[2409.08111]] Towards a graph-based foundation model for network traffic analysis(https://arxiv.org/abs/2409.08111)
• Keywords: transformer
• Abstract: Foundation models have shown great promise in various fields of study. A potential application of such models is in computer network traffic analysis, where these models can grasp the complexities of network traffic dynamics and adapt to any specific task or network environment with minimal fine-tuning. Previous approaches have used tokenized hex-level packet data and the model architecture of large language transformer models. We propose a new, efficient graph-based alternative at the flow-level. Our approach represents network traffic as a dynamic spatio-temporal graph, employing a self-supervised link prediction pretraining task to capture the spatial and temporal dynamics in this network graph framework. To evaluate the effectiveness of our approach, we conduct a few-shot learning experiment for three distinct downstream network tasks: intrusion detection, traffic classification, and botnet classification. Models finetuned from our pretrained base achieve an average performance increase of 6.87\% over training from scratch, demonstrating their ability to effectively learn general network traffic dynamics during pretraining. This success suggests the potential for a large-scale version to serve as an operational foundational model.

Title: LLM-POTUS Score: A Framework of Analyzing Presidential Debates with Large Language Models

• Authors: Zhengliang Liu, Yiwei Li, Oleksandra Zolotarevych, Rongwei Yang, Tianming Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.08147
• Pdf URL: https://arxiv.org/pdf/2409.08147
• Copy Paste: [[2409.08147]] LLM-POTUS Score: A Framework of Analyzing Presidential Debates with Large Language Models(https://arxiv.org/abs/2409.08147)
• Keywords: large language model
• Abstract: Large language models have demonstrated remarkable capabilities in natural language processing, yet their application to political discourse analysis remains underexplored. This paper introduces a novel approach to evaluating presidential debate performances using LLMs, addressing the longstanding challenge of objectively assessing debate outcomes. We propose a framework that analyzes candidates' "Policies, Persona, and Perspective" (3P) and how they resonate with the "Interests, Ideologies, and Identity" (3I) of four key audience groups: voters, businesses, donors, and politicians. Our method employs large language models to generate the LLM-POTUS Score, a quantitative measure of debate performance based on the alignment between 3P and 3I. We apply this framework to analyze transcripts from recent U.S. presidential debates, demonstrating its ability to provide nuanced, multi-dimensional assessments of candidate performances. Our results reveal insights into the effectiveness of different debating strategies and their impact on various audience segments. This study not only offers a new tool for political analysis but also explores the potential and limitations of using LLMs as impartial judges in complex social contexts. In addition, this framework provides individual citizens with an independent tool to evaluate presidential debate performances, which enhances democratic engagement and reduces reliance on potentially biased media interpretations and institutional influence, thereby strengthening the foundation of informed civic participation.

Title: MagicStyle: Portrait Stylization Based on Reference Image

• Authors: Zhaoli Deng, Kaibin Zhou, Fanyi Wang, Zhenpeng Mi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08156
• Pdf URL: https://arxiv.org/pdf/2409.08156
• Copy Paste: [[2409.08156]] MagicStyle: Portrait Stylization Based on Reference Image(https://arxiv.org/abs/2409.08156)
• Keywords: diffusion
• Abstract: The development of diffusion models has significantly advanced the research on image stylization, particularly in the area of stylizing a content image based on a given style image, which has attracted many scholars. The main challenge in this reference image stylization task lies in how to maintain the details of the content image while incorporating the color and texture features of the style image. This challenge becomes even more pronounced when the content image is a portrait which has complex textural details. To address this challenge, we propose a diffusion model-based reference image stylization method specifically for portraits, called MagicStyle. MagicStyle consists of two phases: Content and Style DDIM Inversion (CSDI) and Feature Fusion Forward (FFF). The CSDI phase involves a reverse denoising process, where DDIM Inversion is performed separately on the content image and the style image, storing the self-attention query, key and value features of both images during the inversion process. The FFF phase executes forward denoising, harmoniously integrating the texture and color information from the pre-stored feature queries, keys and values into the diffusion generation process based on our Well-designed Feature Fusion Attention (FFA). We conducted comprehensive comparative and ablation experiments to validate the effectiveness of our proposed MagicStyle and FFA.

Title: SDformer: Efficient End-to-End Transformer for Depth Completion

• Authors: Jian Qian, Miao Sun, Ashley Lee, Jie Li, Shenglong Zhuo, Patrick Yin Chiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08159
• Pdf URL: https://arxiv.org/pdf/2409.08159
• Copy Paste: [[2409.08159]] SDformer: Efficient End-to-End Transformer for Depth Completion(https://arxiv.org/abs/2409.08159)
• Keywords: extraction, transformer
• Abstract: Depth completion aims to predict dense depth maps with sparse depth measurements from a depth sensor. Currently, Convolutional Neural Network (CNN) based models are the most popular methods applied to depth completion tasks. However, despite the excellent high-end performance, they suffer from a limited representation area. To overcome the drawbacks of CNNs, a more effective and powerful method has been presented: the Transformer, which is an adaptive self-attention setting sequence-to-sequence model. While the standard Transformer quadratically increases the computational cost from the key-query dot-product of input resolution which improperly employs depth completion tasks. In this work, we propose a different window-based Transformer architecture for depth completion tasks named Sparse-to-Dense Transformer (SDformer). The network consists of an input module for the depth map and RGB image features extraction and concatenation, a U-shaped encoder-decoder Transformer for extracting deep features, and a refinement module. Specifically, we first concatenate the depth map features with the RGB image features through the input model. Then, instead of calculating self-attention with the whole feature maps, we apply different window sizes to extract the long-range depth dependencies. Finally, we refine the predicted features from the input module and the U-shaped encoder-decoder Transformer module to get the enriching depth features and employ a convolution layer to obtain the dense depth map. In practice, the SDformer obtains state-of-the-art results against the CNN-based depth completion models with lower computing loads and parameters on the NYU Depth V2 and KITTI DC datasets.

Title: On the Role of Context in Reading Time Prediction

• Authors: Andreas Opedal, Eleanor Chodroff, Ryan Cotterell, Ethan Gotlieb Wilcox
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08160
• Pdf URL: https://arxiv.org/pdf/2409.08160
• Copy Paste: [[2409.08160]] On the Role of Context in Reading Time Prediction(https://arxiv.org/abs/2409.08160)
• Keywords: interpretability
• Abstract: We present a new perspective on how readers integrate context during real-time language comprehension. Our proposals build on surprisal theory, which posits that the processing effort of a linguistic unit (e.g., a word) is an affine function of its in-context information content. We first observe that surprisal is only one out of many potential ways that a contextual predictor can be derived from a language model. Another one is the pointwise mutual information (PMI) between a unit and its context, which turns out to yield the same predictive power as surprisal when controlling for unigram frequency. Moreover, both PMI and surprisal are correlated with frequency. This means that neither PMI nor surprisal contains information about context alone. In response to this, we propose a technique where we project surprisal onto the orthogonal complement of frequency, yielding a new contextual predictor that is uncorrelated with frequency. Our experiments show that the proportion of variance in reading times explained by context is a lot smaller when context is represented by the orthogonalized predictor. From an interpretability standpoint, this indicates that previous studies may have overstated the role that context has in predicting reading times.

Title: Open Source Infrastructure for Automatic Cell Segmentation

• Authors: Aaron Rock Menezes, Bharath Ramsundar
• Subjects: cs.LG, cs.CV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2409.08163
• Pdf URL: https://arxiv.org/pdf/2409.08163
• Copy Paste: [[2409.08163]] Open Source Infrastructure for Automatic Cell Segmentation(https://arxiv.org/abs/2409.08163)
• Keywords: robust, segmentation
• Abstract: Automated cell segmentation is crucial for various biological and medical applications, facilitating tasks like cell counting, morphology analysis, and drug discovery. However, manual segmentation is time-consuming and prone to subjectivity, necessitating robust automated methods. This paper presents open-source infrastructure, utilizing the UNet model, a deep-learning architecture noted for its effectiveness in image segmentation tasks. This implementation is integrated into the open-source DeepChem package, enhancing accessibility and usability for researchers and practitioners. The resulting tool offers a convenient and user-friendly interface, reducing the barrier to entry for cell segmentation while maintaining high accuracy. Additionally, we benchmark this model against various datasets, demonstrating its robustness and versatility across different imaging conditions and cell types.

Title: High-Frequency Anti-DreamBooth: Robust Defense Against Image Synthesis

• Authors: Takuto Onikubo, Yusuke Matsui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08167
• Pdf URL: https://arxiv.org/pdf/2409.08167
• Copy Paste: [[2409.08167]] High-Frequency Anti-DreamBooth: Robust Defense Against Image Synthesis(https://arxiv.org/abs/2409.08167)
• Keywords: protect, defense, attack, robust, generative
• Abstract: Recently, text-to-image generative models have been misused to create unauthorized malicious images of individuals, posing a growing social problem. Previous solutions, such as Anti-DreamBooth, add adversarial noise to images to protect them from being used as training data for malicious generation. However, we found that the adversarial noise can be removed by adversarial purification methods such as DiffPure. Therefore, we propose a new adversarial attack method that adds strong perturbation on the high-frequency areas of images to make it more robust to adversarial purification. Our experiment showed that the adversarial images retained noise even after adversarial purification, hindering malicious image generation.

Title: Learning to Match 2D Keypoints Across Preoperative MR and Intraoperative Ultrasound

• Authors: Hassan Rasheed, Reuben Dorent, Maximilian Fehrentz, Tina Kapur, William M. Wells III, Alexandra Golby, Sarah Frisken, Julia A. Schnabel, Nazim Haouchine
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08169
• Pdf URL: https://arxiv.org/pdf/2409.08169
• Copy Paste: [[2409.08169]] Learning to Match 2D Keypoints Across Preoperative MR and Intraoperative Ultrasound(https://arxiv.org/abs/2409.08169)
• Keywords: robust
• Abstract: We propose in this paper a texture-invariant 2D keypoints descriptor specifically designed for matching preoperative Magnetic Resonance (MR) images with intraoperative Ultrasound (US) images. We introduce a matching-by-synthesis strategy, where intraoperative US images are synthesized from MR images accounting for multiple MR modalities and intraoperative US variability. We build our training set by enforcing keypoints localization over all images then train a patient-specific descriptor network that learns texture-invariant discriminant features in a supervised contrastive manner, leading to robust keypoints descriptors. Our experiments on real cases with ground truth show the effectiveness of the proposed approach, outperforming the state-of-the-art methods and achieving 80.35% matching precision on average.

Title: Low-Cost Tree Crown Dieback Estimation Using Deep Learning-Based Segmentation

• Authors: M. J. Allen, D. Moreno-Fernández, P. Ruiz-Benito, S. W. D. Grieve, E. R. Lines
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08171
• Pdf URL: https://arxiv.org/pdf/2409.08171
• Copy Paste: [[2409.08171]] Low-Cost Tree Crown Dieback Estimation Using Deep Learning-Based Segmentation(https://arxiv.org/abs/2409.08171)
• Keywords: robust, segmentation
• Abstract: The global increase in observed forest dieback, characterised by the death of tree foliage, heralds widespread decline in forest ecosystems. This degradation causes significant changes to ecosystem services and functions, including habitat provision and carbon sequestration, which can be difficult to detect using traditional monitoring techniques, highlighting the need for large-scale and high-frequency monitoring. Contemporary developments in the instruments and methods to gather and process data at large-scales mean this monitoring is now possible. In particular, the advancement of low-cost drone technology and deep learning on consumer-level hardware provide new opportunities. Here, we use an approach based on deep learning and vegetation indices to assess crown dieback from RGB aerial data without the need for expensive instrumentation such as LiDAR. We use an iterative approach to match crown footprints predicted by deep learning with field-based inventory data from a Mediterranean ecosystem exhibiting drought-induced dieback, and compare expert field-based crown dieback estimation with vegetation index-based estimates. We obtain high overall segmentation accuracy (mAP: 0.519) without the need for additional technical development of the underlying Mask R-CNN model, underscoring the potential of these approaches for non-expert use and proving their applicability to real-world conservation. We also find colour-coordinate based estimates of dieback correlate well with expert field-based estimation. Substituting ground truth for Mask R-CNN model predictions showed negligible impact on dieback estimates, indicating robustness. Our findings demonstrate the potential of automated data collection and processing, including the application of deep learning, to improve the coverage, speed and cost of forest dieback monitoring.

Title: Fine-tuning Large Language Models for Entity Matching

• Authors: Aaron Steiner, Ralph Peeters, Christian Bizer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08185
• Pdf URL: https://arxiv.org/pdf/2409.08185
• Copy Paste: [[2409.08185]] Fine-tuning Large Language Models for Entity Matching(https://arxiv.org/abs/2409.08185)
• Keywords: generative, large language model
• Abstract: Generative large language models (LLMs) are a promising alternative to pre-trained language models for entity matching due to their high zero-shot performance and their ability to generalize to unseen entities. Existing research on using LLMs for entity matching has focused on prompt engineering and in-context learning. This paper explores the potential of fine-tuning LLMs for entity matching. We analyze fine-tuning along two dimensions: 1) The representation of training examples, where we experiment with adding different types of LLM-generated explanations to the training set, and 2) the selection and generation of training examples using LLMs. In addition to the matching performance on the source dataset, we investigate how fine-tuning affects the model's ability to generalize to other in-domain datasets as well as across topical domains. Our experiments show that fine-tuning significantly improves the performance of the smaller models while the results for the larger models are mixed. Fine-tuning also improves the generalization to in-domain datasets while hurting cross-domain transfer. We show that adding structured explanations to the training set has a positive impact on the performance of three out of four LLMs, while the proposed example selection and generation methods only improve the performance of Llama 3.1 8B while decreasing the performance of GPT-4o Mini.

Title: A Secure Standard for NFT Fractionalization

• Authors: Wejdene Haouari, Marios Fokaefs
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.08190
• Pdf URL: https://arxiv.org/pdf/2409.08190
• Copy Paste: [[2409.08190]] A Secure Standard for NFT Fractionalization(https://arxiv.org/abs/2409.08190)
• Keywords: secure, security
• Abstract: Non-fungible tokens (NFTs) offer a unique method for representing digital and physical assets on the blockchain. However, the NFT market has recently experienced a downturn in interest, mainly due to challenges related to high entry barriers and limited market liquidity. Fractionalization emerges as a promising solution, allowing multiple parties to hold a stake in a single NFT. By breaking down ownership into fractional shares, this approach lowers the entry barrier for investors, enhances market liquidity, and democratizes access to valuable digital assets. Despite these benefits, the current landscape of NFT fractionalization is fragmented, with no standardized framework to guide the secure and interoperable implementation of fractionalization mechanisms. This paper contributions are twofold: first, we provide a detailed analysis of the current NFT fractionalization landscape focusing on security challenges; second, we introduce a standardized approach that addresses these challenges, paving the way for more secure, interoperable, and accessible NFT fractionalization platforms.

Title: What Makes a Maze Look Like a Maze?

• Authors: Joy Hsu, Jiayuan Mao, Joshua B. Tenenbaum, Noah D. Goodman, Jiajun Wu
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08202
• Pdf URL: https://arxiv.org/pdf/2409.08202
• Copy Paste: [[2409.08202]] What Makes a Maze Look Like a Maze?(https://arxiv.org/abs/2409.08202)
• Keywords: large language model
• Abstract: A unique aspect of human visual understanding is the ability to flexibly interpret abstract concepts: acquiring lifted rules explaining what they symbolize, grounding them across familiar and unfamiliar contexts, and making predictions or reasoning about them. While off-the-shelf vision-language models excel at making literal interpretations of images (e.g., recognizing object categories such as tree branches), they still struggle to make sense of such visual abstractions (e.g., how an arrangement of tree branches may form the walls of a maze). To address this challenge, we introduce Deep Schema Grounding (DSG), a framework that leverages explicit structured representations of visual abstractions for grounding and reasoning. At the core of DSG are schemas--dependency graph descriptions of abstract concepts that decompose them into more primitive-level symbols. DSG uses large language models to extract schemas, then hierarchically grounds concrete to abstract components of the schema onto images with vision-language models. The grounded schema is used to augment visual abstraction understanding. We systematically evaluate DSG and different methods in reasoning on our new Visual Abstractions Dataset, which consists of diverse, real-world images of abstract concepts and corresponding question-answer pairs labeled by humans. We show that DSG significantly improves the abstract visual reasoning performance of vision-language models, and is a step toward human-aligned understanding of visual abstractions.

Title: VI3DRM:Towards meticulous 3D Reconstruction from Sparse Views via Photo-Realistic Novel View Synthesis

• Authors: Hao Chen, Jiafu Wu, Ying Jin, Jinlong Peng, Xiaofeng Mao, Mingmin Chi, Mufeng Yao, Bo Peng, Jian Li, Yun Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08207
• Pdf URL: https://arxiv.org/pdf/2409.08207
• Copy Paste: [[2409.08207]] VI3DRM:Towards meticulous 3D Reconstruction from Sparse Views via Photo-Realistic Novel View Synthesis(https://arxiv.org/abs/2409.08207)
• Keywords: diffusion
• Abstract: Recently, methods like Zero-1-2-3 have focused on single-view based 3D reconstruction and have achieved remarkable success. However, their predictions for unseen areas heavily rely on the inductive bias of large-scale pretrained diffusion models. Although subsequent work, such as DreamComposer, attempts to make predictions more controllable by incorporating additional views, the results remain unrealistic due to feature entanglement in the vanilla latent space, including factors such as lighting, material, and structure. To address these issues, we introduce the Visual Isotropy 3D Reconstruction Model (VI3DRM), a diffusion-based sparse views 3D reconstruction model that operates within an ID consistent and perspective-disentangled 3D latent space. By facilitating the disentanglement of semantic information, color, material properties and lighting, VI3DRM is capable of generating highly realistic images that are indistinguishable from real photographs. By leveraging both real and synthesized images, our approach enables the accurate construction of pointmaps, ultimately producing finely textured meshes or point clouds. On the NVS task, tested on the GSO dataset, VI3DRM significantly outperforms state-of-the-art method DreamComposer, achieving a PSNR of 38.61, an SSIM of 0.929, and an LPIPS of 0.027. Code will be made available upon publication.

Title: LT3SD: Latent Trees for 3D Scene Diffusion

• Authors: Quan Meng, Lei Li, Matthias Nießner, Angela Dai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08215
• Pdf URL: https://arxiv.org/pdf/2409.08215
• Copy Paste: [[2409.08215]] LT3SD: Latent Trees for 3D Scene Diffusion(https://arxiv.org/abs/2409.08215)
• Keywords: diffusion, generative
• Abstract: We present LT3SD, a novel latent diffusion model for large-scale 3D scene generation. Recent advances in diffusion models have shown impressive results in 3D object generation, but are limited in spatial extent and quality when extended to 3D scenes. To generate complex and diverse 3D scene structures, we introduce a latent tree representation to effectively encode both lower-frequency geometry and higher-frequency detail in a coarse-to-fine hierarchy. We can then learn a generative diffusion process in this latent 3D scene space, modeling the latent components of a scene at each resolution level. To synthesize large-scale scenes with varying sizes, we train our diffusion model on scene patches and synthesize arbitrary-sized output 3D scenes through shared diffusion generation across multiple scene patches. Through extensive experiments, we demonstrate the efficacy and benefits of LT3SD for large-scale, high-quality unconditional 3D scene generation and for probabilistic completion for partial scene observations.

Title: Tweezers: A Framework for Security Event Detection via Event Attribution-centric Tweet Embedding

• Authors: Jian Cui, Hanna Kim, Eugene Jang, Dayeon Yim, Kicheol Kim, Yongjae Lee, Jin-Woo Chung, Seungwon Shin, Xiaojing Liao
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.08221
• Pdf URL: https://arxiv.org/pdf/2409.08221
• Copy Paste: [[2409.08221]] Tweezers: A Framework for Security Event Detection via Event Attribution-centric Tweet Embedding(https://arxiv.org/abs/2409.08221)
• Keywords: security
• Abstract: Twitter is recognized as a crucial platform for the dissemination and gathering of Cyber Threat Intelligence (CTI). Its capability to provide real-time, actionable intelligence makes it an indispensable tool for detecting security events, helping security professionals cope with ever-growing threats. However, the large volume of tweets and inherent noises of human-crafted tweets pose significant challenges in accurately identifying security events. While many studies tried to filter out event-related tweets based on keywords, they are not effective due to their limitation in understanding the semantics of tweets. Another challenge in security event detection from Twitter is the comprehensive coverage of security events. Previous studies emphasized the importance of early detection of security events, but they overlooked the importance of event coverage. To cope with these challenges, in our study, we introduce a novel event attribution-centric tweet embedding method to enable the high precision and coverage of events. Our experiment result shows that the proposed method outperforms existing text and graph-based tweet embedding methods in identifying security events. Leveraging this novel embedding approach, we have developed and implemented a framework, Tweezers, that is applicable to security event detection from Twitter for CTI gathering. This framework has demonstrated its effectiveness, detecting twice as many events compared to established baselines. Additionally, we have showcased two applications, built on Tweezers for the integration and inspection of security events, i.e., security event trend analysis and informative security user identification.

Title: LLM Honeypot: Leveraging Large Language Models as Advanced Interactive Honeypot Systems

• Authors: Hakan T. Otal, M. Abdullah Canbaz
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.08234
• Pdf URL: https://arxiv.org/pdf/2409.08234
• Copy Paste: [[2409.08234]] LLM Honeypot: Leveraging Large Language Models as Advanced Interactive Honeypot Systems(https://arxiv.org/abs/2409.08234)
• Keywords: security, attack, large language model
• Abstract: The rapid evolution of cyber threats necessitates innovative solutions for detecting and analyzing malicious activity. Honeypots, which are decoy systems designed to lure and interact with attackers, have emerged as a critical component in cybersecurity. In this paper, we present a novel approach to creating realistic and interactive honeypot systems using Large Language Models (LLMs). By fine-tuning a pre-trained open-source language model on a diverse dataset of attacker-generated commands and responses, we developed a honeypot capable of sophisticated engagement with attackers. Our methodology involved several key steps: data collection and processing, prompt engineering, model selection, and supervised fine-tuning to optimize the model's performance. Evaluation through similarity metrics and live deployment demonstrated that our approach effectively generates accurate and informative responses. The results highlight the potential of LLMs to revolutionize honeypot technology, providing cybersecurity professionals with a powerful tool to detect and analyze malicious activity, thereby enhancing overall security infrastructure.

Title: Multi-Model based Federated Learning Against Model Poisoning Attack: A Deep Learning Based Model Selection for MEC Systems

• Authors: Somayeh Kianpisheh, Chafika Benzaid, Tarik Taleb
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.08237
• Pdf URL: https://arxiv.org/pdf/2409.08237
• Copy Paste: [[2409.08237]] Multi-Model based Federated Learning Against Model Poisoning Attack: A Deep Learning Based Model Selection for MEC Systems(https://arxiv.org/abs/2409.08237)
• Keywords: privacy, attack, robust, federate
• Abstract: Federated Learning (FL) enables training of a global model from distributed data, while preserving data privacy. However, the singular-model based operation of FL is open with uploading poisoned models compatible with the global model structure and can be exploited as a vulnerability to conduct model poisoning attacks. This paper proposes a multi-model based FL as a proactive mechanism to enhance the opportunity of model poisoning attack mitigation. A master model is trained by a set of slave models. To enhance the opportunity of attack mitigation, the structure of client models dynamically change within learning epochs, and the supporter FL protocol is provided. For a MEC system, the model selection problem is modeled as an optimization to minimize loss and recognition time, while meeting a robustness confidence. In adaption with dynamic network condition, a deep reinforcement learning based model selection is proposed. For a DDoS attack detection scenario, results illustrate a competitive accuracy gain under poisoning attack with the scenario that the system is without attack, and also a potential of recognition time improvement.

Title: Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources

• Authors: Alisia Lupidi, Carlos Gemmell, Nicola Cancedda, Jane Dwivedi-Yu, Jason Weston, Jakob Foerster, Roberta Raileanu, Maria Lomeli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08239
• Pdf URL: https://arxiv.org/pdf/2409.08239
• Copy Paste: [[2409.08239]] Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources(https://arxiv.org/abs/2409.08239)
• Keywords: large language model
• Abstract: Large Language Models still struggle in challenging scenarios that leverage structured data, complex reasoning, or tool usage. In this paper, we propose Source2Synth: a new method that can be used for teaching LLMs new skills without relying on costly human annotations. Source2Synth takes as input a custom data source and produces synthetic data points with intermediate reasoning steps grounded in real-world sources. Source2Synth improves the dataset quality by discarding low-quality generations based on their answerability. We demonstrate the generality of this approach by applying it to two challenging domains: we test reasoning abilities in multi-hop question answering (MHQA), and tool usage in tabular question answering (TQA). Our method improves performance by 25.51% for TQA on WikiSQL and 22.57% for MHQA on HotPotQA compared to the fine-tuned baselines.

Title: IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation

• Authors: Yinwei Wu, Xianpan Zhou, Bing Ma, Xuefeng Su, Kai Ma, Xinchao Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08240
• Pdf URL: https://arxiv.org/pdf/2409.08240
• Copy Paste: [[2409.08240]] IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation(https://arxiv.org/abs/2409.08240)
• Keywords: diffusion
• Abstract: While Text-to-Image (T2I) diffusion models excel at generating visually appealing images of individual instances, they struggle to accurately position and control the features generation of multiple instances. The Layout-to-Image (L2I) task was introduced to address the positioning challenges by incorporating bounding boxes as spatial control signals, but it still falls short in generating precise instance features. In response, we propose the Instance Feature Generation (IFG) task, which aims to ensure both positional accuracy and feature fidelity in generated instances. To address the IFG task, we introduce the Instance Feature Adapter (IFAdapter). The IFAdapter enhances feature depiction by incorporating additional appearance tokens and utilizing an Instance Semantic Map to align instance-level features with spatial locations. The IFAdapter guides the diffusion process as a plug-and-play module, making it adaptable to various community models. For evaluation, we contribute an IFG benchmark and develop a verification pipeline to objectively compare models' abilities to generate instances with accurate positioning and features. Experimental results demonstrate that IFAdapter outperforms other models in both quantitative and qualitative evaluations.

Title: Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding

• Authors: Hongyu Li, Tianrui Hui, Zihan Ding, Jing Zhang, Bin Ma, Xiaoming Wei, Jizhong Han, Si Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08251
• Pdf URL: https://arxiv.org/pdf/2409.08251
• Copy Paste: [[2409.08251]] Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding(https://arxiv.org/abs/2409.08251)
• Keywords: diffusion, segmentation
• Abstract: Panoptic narrative grounding (PNG), whose core target is fine-grained image-text alignment, requires a panoptic segmentation of referred objects given a narrative caption. Previous discriminative methods achieve only weak or coarse-grained alignment by panoptic segmentation pretraining or CLIP model adaptation. Given the recent progress of text-to-image Diffusion models, several works have shown their capability to achieve fine-grained image-text alignment through cross-attention maps and improved general segmentation performance. However, the direct use of phrase features as static prompts to apply frozen Diffusion models to the PNG task still suffers from a large task gap and insufficient vision-language interaction, yielding inferior performance. Therefore, we propose an Extractive-Injective Phrase Adapter (EIPA) bypass within the Diffusion UNet to dynamically update phrase prompts with image features and inject the multimodal cues back, which leverages the fine-grained image-text alignment capability of Diffusion models more sufficiently. In addition, we also design a Multi-Level Mutual Aggregation (MLMA) module to reciprocally fuse multi-level image and phrase features for segmentation refinement. Extensive experiments on the PNG benchmark show that our method achieves new state-of-the-art performance.

Title: LoRID: Low-Rank Iterative Diffusion for Adversarial Purification

• Authors: Geigh Zollicoffer, Minh Vu, Ben Nebgen, Juan Castorena, Boian Alexandrov, Manish Bhattarai
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.08255
• Pdf URL: https://arxiv.org/pdf/2409.08255
• Copy Paste: [[2409.08255]] LoRID: Low-Rank Iterative Diffusion for Adversarial Purification(https://arxiv.org/abs/2409.08255)
• Keywords: defense, attack, robust, diffusion
• Abstract: This work presents an information-theoretic examination of diffusion-based purification methods, the state-of-the-art adversarial defenses that utilize diffusion models to remove malicious perturbations in adversarial examples. By theoretically characterizing the inherent purification errors associated with the Markov-based diffusion purifications, we introduce LoRID, a novel Low-Rank Iterative Diffusion purification method designed to remove adversarial perturbation with low intrinsic purification errors. LoRID centers around a multi-stage purification process that leverages multiple rounds of diffusion-denoising loops at the early time-steps of the diffusion models, and the integration of Tucker decomposition, an extension of matrix factorization, to remove adversarial noise at high-noise regimes. Consequently, LoRID increases the effective diffusion time-steps and overcomes strong adversarial attacks, achieving superior robustness performance in CIFAR-10/100, CelebA-HQ, and ImageNet datasets under both white-box and black-box settings.

Title: Improving Virtual Try-On with Garment-focused Diffusion Models

• Authors: Siqi Wan, Yehao Li, Jingwen Chen, Yingwei Pan, Ting Yao, Yang Cao, Tao Mei
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.08258
• Pdf URL: https://arxiv.org/pdf/2409.08258
• Copy Paste: [[2409.08258]] Improving Virtual Try-On with Garment-focused Diffusion Models(https://arxiv.org/abs/2409.08258)
• Keywords: diffusion, generative
• Abstract: Diffusion models have led to the revolutionizing of generative modeling in numerous image synthesis tasks. Nevertheless, it is not trivial to directly apply diffusion models for synthesizing an image of a target person wearing a given in-shop garment, i.e., image-based virtual try-on (VTON) task. The difficulty originates from the aspect that the diffusion process should not only produce holistically high-fidelity photorealistic image of the target person, but also locally preserve every appearance and texture detail of the given garment. To address this, we shape a new Diffusion model, namely GarDiff, which triggers the garment-focused diffusion process with amplified guidance of both basic visual appearance and detailed textures (i.e., high-frequency details) derived from the given garment. GarDiff first remoulds a pre-trained latent diffusion model with additional appearance priors derived from the CLIP and VAE encodings of the reference garment. Meanwhile, a novel garment-focused adapter is integrated into the UNet of diffusion model, pursuing local fine-grained alignment with the visual appearance of reference garment and human pose. We specifically design an appearance loss over the synthesized garment to enhance the crucial, high-frequency details. Extensive experiments on VITON-HD and DressCode datasets demonstrate the superiority of our GarDiff when compared to state-of-the-art VTON approaches. Code is publicly available at: \href{this https URL}{this https URL}.

Title: Improving Text-guided Object Inpainting with Semantic Pre-inpainting

• Authors: Yifu Chen, Jingwen Chen, Yingwei Pan, Yehao Li, Ting Yao, Zhineng Chen, Tao Mei
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.08260
• Pdf URL: https://arxiv.org/pdf/2409.08260
• Copy Paste: [[2409.08260]] Improving Text-guided Object Inpainting with Semantic Pre-inpainting(https://arxiv.org/abs/2409.08260)
• Keywords: diffusion, transformer
• Abstract: Recent years have witnessed the success of large text-to-image diffusion models and their remarkable potential to generate high-quality images. The further pursuit of enhancing the editability of images has sparked significant interest in the downstream task of inpainting a novel object described by a text prompt within a designated region in the image. Nevertheless, the problem is not trivial from two aspects: 1) Solely relying on one single U-Net to align text prompt and visual object across all the denoising timesteps is insufficient to generate desired objects; 2) The controllability of object generation is not guaranteed in the intricate sampling space of diffusion model. In this paper, we propose to decompose the typical single-stage object inpainting into two cascaded processes: 1) semantic pre-inpainting that infers the semantic features of desired objects in a multi-modal feature space; 2) high-fieldity object generation in diffusion latent space that pivots on such inpainted semantic features. To achieve this, we cascade a Transformer-based semantic inpainter and an object inpainting diffusion model, leading to a novel CAscaded Transformer-Diffusion (CAT-Diffusion) framework for text-guided object inpainting. Technically, the semantic inpainter is trained to predict the semantic features of the target object conditioning on unmasked context and text prompt. The outputs of the semantic inpainter then act as the informative visual prompts to guide high-fieldity object generation through a reference adapter layer, leading to controllable object inpainting. Extensive evaluations on OpenImages-V6 and MSCOCO validate the superiority of CAT-Diffusion against the state-of-the-art methods. Code is available at \url{this https URL}.

Title: FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimally

• Authors: Qiuhong Shen, Xingyi Yang, Xinchao Wang
• Subjects: cs.CV, cs.AI, cs.GR, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.08270
• Pdf URL: https://arxiv.org/pdf/2409.08270
• Copy Paste: [[2409.08270]] FlashSplat: 2D to 3D Gaussian Splatting Segmentation Solved Optimally(https://arxiv.org/abs/2409.08270)
• Keywords: robust, segmentation
• Abstract: This study addresses the challenge of accurately segmenting 3D Gaussian Splatting from 2D masks. Conventional methods often rely on iterative gradient descent to assign each Gaussian a unique label, leading to lengthy optimization and sub-optimal solutions. Instead, we propose a straightforward yet globally optimal solver for 3D-GS segmentation. The core insight of our method is that, with a reconstructed 3D-GS scene, the rendering of the 2D masks is essentially a linear function with respect to the labels of each Gaussian. As such, the optimal label assignment can be solved via linear programming in closed form. This solution capitalizes on the alpha blending characteristic of the splatting process for single step optimization. By incorporating the background bias in our objective function, our method shows superior robustness in 3D segmentation against noises. Remarkably, our optimization completes within 30 seconds, about 50$\times$ faster than the best existing methods. Extensive experiments demonstrate the efficiency and robustness of our method in segmenting various scenes, and its superior performance in downstream tasks such as object removal and inpainting. Demos and code will be available at this https URL.

Title: DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfer

• Authors: Runjia Li, Junlin Han, Luke Melas-Kyriazi, Chunyi Sun, Zhaochong An, Zhongrui Gui, Shuyang Sun, Philip Torr, Tomas Jakab
• Subjects: cs.CV, cs.GR, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.08271
• Pdf URL: https://arxiv.org/pdf/2409.08271
• Copy Paste: [[2409.08271]] DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfer(https://arxiv.org/abs/2409.08271)
• Keywords: diffusion
• Abstract: We present DreamBeast, a novel method based on score distillation sampling (SDS) for generating fantastical 3D animal assets composed of distinct parts. Existing SDS methods often struggle with this generation task due to a limited understanding of part-level semantics in text-to-image diffusion models. While recent diffusion models, such as Stable Diffusion 3, demonstrate a better part-level understanding, they are prohibitively slow and exhibit other common problems associated with single-view diffusion models. DreamBeast overcomes this limitation through a novel part-aware knowledge transfer mechanism. For each generated asset, we efficiently extract part-level knowledge from the Stable Diffusion 3 model into a 3D Part-Affinity implicit representation. This enables us to instantly generate Part-Affinity maps from arbitrary camera views, which we then use to modulate the guidance of a multi-view diffusion model during SDS to create 3D assets of fantastical animals. DreamBeast significantly enhances the quality of generated 3D creatures with user-specified part compositions while reducing computational overhead, as demonstrated by extensive quantitative and qualitative evaluations.

Title: Click2Mask: Local Editing with Dynamic Mask Generation

• Authors: Omer Regev, Omri Avrahami, Dani Lischinski
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08272
• Pdf URL: https://arxiv.org/pdf/2409.08272
• Copy Paste: [[2409.08272]] Click2Mask: Local Editing with Dynamic Mask Generation(https://arxiv.org/abs/2409.08272)
• Keywords: diffusion, generative, segmentation
• Abstract: Recent advancements in generative models have revolutionized image generation and editing, making these tasks accessible to non-experts. This paper focuses on local image editing, particularly the task of adding new content to a loosely specified area. Existing methods often require a precise mask or a detailed description of the location, which can be cumbersome and prone to errors. We propose Click2Mask, a novel approach that simplifies the local editing process by requiring only a single point of reference (in addition to the content description). A mask is dynamically grown around this point during a Blended Latent Diffusion (BLD) process, guided by a masked CLIP-based semantic loss. Click2Mask surpasses the limitations of segmentation-based and fine-tuning dependent methods, offering a more user-friendly and contextually accurate solution. Our experiments demonstrate that Click2Mask not only minimizes user effort but also delivers competitive or superior local image manipulation results compared to SoTA methods, according to both human judgement and automatic metrics. Key contributions include the simplification of user input, the ability to freely add objects unconstrained by existing segments, and the integration potential of our dynamic mask approach within other editing methods.

Title: DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priors

• Authors: Thomas Hanwen Zhu, Ruining Li, Tomas Jakab
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08278
• Pdf URL: https://arxiv.org/pdf/2409.08278
• Copy Paste: [[2409.08278]] DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priors(https://arxiv.org/abs/2409.08278)
• Keywords: diffusion
• Abstract: We present DreamHOI, a novel method for zero-shot synthesis of human-object interactions (HOIs), enabling a 3D human model to realistically interact with any given object based on a textual description. This task is complicated by the varying categories and geometries of real-world objects and the scarcity of datasets encompassing diverse HOIs. To circumvent the need for extensive data, we leverage text-to-image diffusion models trained on billions of image-caption pairs. We optimize the articulation of a skinned human mesh using Score Distillation Sampling (SDS) gradients obtained from these models, which predict image-space edits. However, directly backpropagating image-space gradients into complex articulation parameters is ineffective due to the local nature of such gradients. To overcome this, we introduce a dual implicit-explicit representation of a skinned mesh, combining (implicit) neural radiance fields (NeRFs) with (explicit) skeleton-driven mesh articulation. During optimization, we transition between implicit and explicit forms, grounding the NeRF generation while refining the mesh articulation. We validate our approach through extensive experiments, demonstrating its effectiveness in generating realistic HOIs.