2024-08-15

Title: MathBridge: A Large-Scale Dataset for Translating Mathematical Expressions into Formula Images

• Authors: Kyudan Jung, Sieun Hyeon, Kwon Jeong Youn, Nam-Joon Kim, Hyun Gon Ryu, Hyuk-Jae Lee, Jaeyoung Do
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07081
• Pdf URL: https://arxiv.org/pdf/2408.07081
• Copy Paste: [[2408.07081]] MathBridge: A Large-Scale Dataset for Translating Mathematical Expressions into Formula Images(https://arxiv.org/abs/2408.07081)
• Keywords: robust
• Abstract: Understanding sentences that contain mathematical expressions in text form poses significant challenges. To address this, the importance of converting these expressions into formula images has been highlighted. For instance, the expression ``x equals minus b plus or minus the square root of b squared minus four a c, all over two a'' is more readily comprehensible when displayed as an image $x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}$. To develop a text-to-image conversion system, we can break down the process into text-to-LaTeX and LaTeX-to-image conversions, with the latter being managed with by existing various LaTeX engines. However, the former approach has been notably hindered by the severe scarcity of text-to-LaTeX paired data, presenting a significant challenge in the this http URL this context, we introduce MathBridge, the first extensive dataset for translating mathematical spoken English into LaTeX, which aims to establish a robust baseline for future research in text-to-LaTeX translation. MathBridge comprises approximately 23 million LaTeX formulas paired with corresponding spoken English expressions. Through comprehensive evaluations, including fine-tuning and testing with data, we discovered that MathBridge significantly enhances pre-trained language models' capabilities for text-to-LaTeX translation. Specifically, for the T5-large model, the sacreBLEU score increased from 4.77 to 46.8, demonstrating substantial enhancement. Our findings indicate the necessity for a new metric specifically for text-to-LaTeX conversion evaluation.

Title: Dynamic Hypergraph-Enhanced Prediction of Sequential Medical Visits

• Authors: Wangying Yang, Zhizhong Wu, Zitao Zheng, Bo Zhang, Shi Bo, Yuanfang Yang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07084
• Pdf URL: https://arxiv.org/pdf/2408.07084
• Copy Paste: [[2408.07084]] Dynamic Hypergraph-Enhanced Prediction of Sequential Medical Visits(https://arxiv.org/abs/2408.07084)
• Keywords: robust
• Abstract: This study introduces a pioneering Dynamic Hypergraph Networks (DHCE) model designed to predict future medical diagnoses from electronic health records with enhanced accuracy. The DHCE model innovates by identifying and differentiating acute and chronic diseases within a patient's visit history, constructing dynamic hypergraphs that capture the complex, high-order interactions between diseases. It surpasses traditional recurrent neural networks and graph neural networks by effectively integrating clinical event data, reflected through medical language model-assisted encoding, into a robust patient representation. Through extensive experiments on two benchmark datasets, MIMIC-III and MIMIC-IV, the DHCE model exhibits superior performance, significantly outpacing established baseline models in the precision of sequential diagnosis prediction.

Title: InfinityMATH: A Scalable Instruction Tuning Dataset in Programmatic Mathematical Reasoning

• Authors: Bo-Wen Zhang, Yan Yan, Lin Li, Guang Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07089
• Pdf URL: https://arxiv.org/pdf/2408.07089
• Copy Paste: [[2408.07089]] InfinityMATH: A Scalable Instruction Tuning Dataset in Programmatic Mathematical Reasoning(https://arxiv.org/abs/2408.07089)
• Keywords: robust
• Abstract: Recent advancements in Chain-of-Thoughts (CoT) and Program-of-Thoughts (PoT) methods have greatly enhanced language models' mathematical reasoning capabilities, facilitating their integration into instruction tuning datasets with LLMs. However, existing methods for large-scale dataset creation require substantial seed data and high computational costs for data synthesis, posing significant challenges for scalability. We introduce InfinityMATH, a scalable instruction tuning dataset for programmatic mathematical reasoning. The construction pipeline emphasizes decoupling numbers from mathematical problems to synthesize number-independent programs, enabling efficient and flexible scaling while minimizing dependency on specific numerical values. Fine-tuning experiments with open-source language and code models, such as Llama2 and CodeLlama, demonstrate the practical benefits of InfinityMATH. These fine-tuned models, showed significant relative improvements on both in-domain and out-of-domain benchmarks, ranging from 184.7% to 514.3% on average. Additionally, these models exhibited high robustness on the GSM8K+ and MATH+ benchmarks, which are enhanced version of test sets with simply the number variations. InfinityMATH ensures that models are more versatile and effective across a broader range of mathematical problems. The data is available at this https URL.

Title: Post-Training Sparse Attention with Double Sparsity

• Authors: Shuo Yang, Ying Sheng, Joseph E. Gonzalez, Ion Stoica, Lianmin Zheng
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07092
• Pdf URL: https://arxiv.org/pdf/2408.07092
• Copy Paste: [[2408.07092]] Post-Training Sparse Attention with Double Sparsity(https://arxiv.org/abs/2408.07092)
• Keywords: large language model
• Abstract: The inference process for large language models is slow and memory-intensive, with one of the most critical bottlenecks being excessive Key-Value (KV) cache accesses. This paper introduces "Double Sparsity," a novel post-training sparse attention technique designed to alleviate this bottleneck by reducing KV cache access. Double Sparsity combines token sparsity, which focuses on utilizing only the important tokens for computing self-attention, with channel sparsity, an approach that uses important feature channels for identifying important tokens. Our key insight is that the pattern of channel sparsity is relatively static, allowing us to use offline calibration to make it efficient at runtime, thereby enabling accurate and efficient identification of important tokens. Moreover, this method can be combined with offloading to achieve significant memory usage reduction. Experimental results demonstrate that Double Sparsity can achieve \(\frac{1}{16}\) token and channel sparsity with minimal impact on accuracy across various tasks, including wiki-2 perplexity, key-value retrieval, and long context benchmarks with models including Llama-2-7B, Llama-2-70B, and Mixtral-8x7B. It brings up to a 14.1$\times$ acceleration in attention operations and a 1.9$\times$ improvement in end-to-end inference on GPUs. With offloading, it achieves a decoding speed acceleration of 16.3$\times$ compared to state-of-the-art solutions at a sequence length of 256K. Our code is publicly available at \url{this https URL}.

Title: Overcoming Imbalanced Safety Data Using Extended Accident Triangle

• Authors: Kailai Sun, Tianxiang Lan, Yang Miang Goh, Yueng-Hsiang Huang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.07094
• Pdf URL: https://arxiv.org/pdf/2408.07094
• Copy Paste: [[2408.07094]] Overcoming Imbalanced Safety Data Using Extended Accident Triangle(https://arxiv.org/abs/2408.07094)
• Keywords: robust
• Abstract: There is growing interest in using safety analytics and machine learning to support the prevention of workplace incidents, especially in high-risk industries like construction and trucking. Although existing safety analytics studies have made remarkable progress, they suffer from imbalanced datasets, a common problem in safety analytics, resulting in prediction inaccuracies. This can lead to management problems, e.g., incorrect resource allocation and improper interventions. To overcome the imbalanced data problem, we extend the theory of accident triangle to claim that the importance of data samples should be based on characteristics such as injury severity, accident frequency, and accident type. Thus, three oversampling methods are proposed based on assigning different weights to samples in the minority class. We find robust improvements among different machine learning algorithms. For the lack of open-source safety datasets, we are sharing three imbalanced datasets, e.g., a 9-year nationwide construction accident record dataset, and their corresponding codes.

Title: Attention Please: What Transformer Models Really Learn for Process Prediction

• Authors: Martin Käppel, Lars Ackermann, Stefan Jablonski, Simon Härtl
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07097
• Pdf URL: https://arxiv.org/pdf/2408.07097
• Copy Paste: [[2408.07097]] Attention Please: What Transformer Models Really Learn for Process Prediction(https://arxiv.org/abs/2408.07097)
• Keywords: transformer
• Abstract: Predictive process monitoring aims to support the execution of a process during runtime with various predictions about the further evolution of a process instance. In the last years a plethora of deep learning architectures have been established as state-of-the-art for different prediction targets, among others the transformer architecture. The transformer architecture is equipped with a powerful attention mechanism, assigning attention scores to each input part that allows to prioritize most relevant information leading to more accurate and contextual output. However, deep learning models largely represent a black box, i.e., their reasoning or decision-making process cannot be understood in detail. This paper examines whether the attention scores of a transformer based next-activity prediction model can serve as an explanation for its decision-making. We find that attention scores in next-activity prediction models can serve as explainers and exploit this fact in two proposed graph-based explanation approaches. The gained insights could inspire future work on the improvement of predictive business process models as well as enabling a neural network based mining of process models from event logs.

Title: Generative Photomontage

• Authors: Sean J. Liu, Nupur Kumari, Ariel Shamir, Jun-Yan Zhu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.07116
• Pdf URL: https://arxiv.org/pdf/2408.07116
• Copy Paste: [[2408.07116]] Generative Photomontage(https://arxiv.org/abs/2408.07116)
• Keywords: diffusion, generative
• Abstract: Text-to-image models are powerful tools for image creation. However, the generation process is akin to a dice roll and makes it difficult to achieve a single image that captures everything a user wants. In this paper, we propose a framework for creating the desired image by compositing it from various parts of generated images, in essence forming a Generative Photomontage. Given a stack of images generated by ControlNet using the same input condition and different seeds, we let users select desired parts from the generated results using a brush stroke interface. We introduce a novel technique that takes in the user's brush strokes, segments the generated images using a graph-based optimization in diffusion feature space, and then composites the segmented regions via a new feature-space blending method. Our method faithfully preserves the user-selected regions while compositing them harmoniously. We demonstrate that our flexible framework can be used for many applications, including generating new appearance combinations, fixing incorrect shapes and artifacts, and improving prompt alignment. We show compelling results for each application and demonstrate that our method outperforms existing image blending methods and various baselines.

Title: ELLA: Empowering LLMs for Interpretable, Accurate and Informative Legal Advice

• Authors: Yutong Hu, Kangcheng Luo, Yansong Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07137
• Pdf URL: https://arxiv.org/pdf/2408.07137
• Copy Paste: [[2408.07137]] ELLA: Empowering LLMs for Interpretable, Accurate and Informative Legal Advice(https://arxiv.org/abs/2408.07137)
• Keywords: large language model
• Abstract: Despite remarkable performance in legal consultation exhibited by legal Large Language Models(LLMs) combined with legal article retrieval components, there are still cases when the advice given is incorrect or baseless. To alleviate these problems, we propose {\bf ELLA}, a tool for {\bf E}mpowering {\bf L}LMs for interpretable, accurate, and informative {\bf L}egal {\bf A}dvice. ELLA visually presents the correlation between legal articles and LLM's response by calculating their similarities, providing users with an intuitive legal basis for the responses. Besides, based on the users' queries, ELLA retrieves relevant legal articles and displays them to users. Users can interactively select legal articles for LLM to generate more accurate responses. ELLA also retrieves relevant legal cases for user reference. Our user study shows that presenting the legal basis for the response helps users understand better. The accuracy of LLM's responses also improves when users intervene in selecting legal articles for LLM. Providing relevant legal cases also aids individuals in obtaining comprehensive information.

Title: Vision Language Model for Interpretable and Fine-grained Detection of Safety Compliance in Diverse Workplaces

• Authors: Zhiling Chen, Hanning Chen, Mohsen Imani, Ruimin Chen, Farhad Imani
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07146
• Pdf URL: https://arxiv.org/pdf/2408.07146
• Copy Paste: [[2408.07146]] Vision Language Model for Interpretable and Fine-grained Detection of Safety Compliance in Diverse Workplaces(https://arxiv.org/abs/2408.07146)
• Keywords: protect
• Abstract: Workplace accidents due to personal protective equipment (PPE) non-compliance raise serious safety concerns and lead to legal liabilities, financial penalties, and reputational damage. While object detection models have shown the capability to address this issue by identifying safety items, most existing models, such as YOLO, Faster R-CNN, and SSD, are limited in verifying the fine-grained attributes of PPE across diverse workplace scenarios. Vision language models (VLMs) are gaining traction for detection tasks by leveraging the synergy between visual and textual information, offering a promising solution to traditional object detection limitations in PPE recognition. Nonetheless, VLMs face challenges in consistently verifying PPE attributes due to the complexity and variability of workplace environments, requiring them to interpret context-specific language and visual cues simultaneously. We introduce Clip2Safety, an interpretable detection framework for diverse workplace safety compliance, which comprises four main modules: scene recognition, the visual prompt, safety items detection, and fine-grained verification. The scene recognition identifies the current scenario to determine the necessary safety gear. The visual prompt formulates the specific visual prompts needed for the detection process. The safety items detection identifies whether the required safety gear is being worn according to the specified scenario. Lastly, the fine-grained verification assesses whether the worn safety equipment meets the fine-grained attribute requirements. We conduct real-world case studies across six different scenarios. The results show that Clip2Safety not only demonstrates an accuracy improvement over state-of-the-art question-answering based VLMs but also achieves inference times two hundred times faster.

Title: Controlling the World by Sleight of Hand

• Authors: Sruthi Sudhakar, Ruoshi Liu, Basile Van Hoorick, Carl Vondrick, Richard Zemel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07147
• Pdf URL: https://arxiv.org/pdf/2408.07147
• Copy Paste: [[2408.07147]] Controlling the World by Sleight of Hand(https://arxiv.org/abs/2408.07147)
• Keywords: generative
• Abstract: Humans naturally build mental models of object interactions and dynamics, allowing them to imagine how their surroundings will change if they take a certain action. While generative models today have shown impressive results on generating/editing images unconditionally or conditioned on text, current methods do not provide the ability to perform object manipulation conditioned on actions, an important tool for world modeling and action planning. Therefore, we propose to learn an action-conditional generative models by learning from unlabeled videos of human hands interacting with objects. The vast quantity of such data on the internet allows for efficient scaling which can enable high-performing action-conditional models. Given an image, and the shape/location of a desired hand interaction, CosHand, synthesizes an image of a future after the interaction has occurred. Experiments show that the resulting model can predict the effects of hand-object interactions well, with strong generalization particularly to translation, stretching, and squeezing interactions of unseen objects in unseen environments. Further, CosHand can be sampled many times to predict multiple possible effects, modeling the uncertainty of forces in the interaction/environment. Finally, method generalizes to different embodiments, including non-human hands, i.e. robot hands, suggesting that generative video models can be powerful models for robotics.

Title: FedMADE: Robust Federated Learning for Intrusion Detection in IoT Networks Using a Dynamic Aggregation Method

• Authors: Shihua Sun, Pragya Sharma, Kenechukwu Nwodo, Angelos Stavrou, Haining Wang
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2408.07152
• Pdf URL: https://arxiv.org/pdf/2408.07152
• Copy Paste: [[2408.07152]] FedMADE: Robust Federated Learning for Intrusion Detection in IoT Networks Using a Dynamic Aggregation Method(https://arxiv.org/abs/2408.07152)
• Keywords: security, privacy, attack, robust, federate
• Abstract: The rapid proliferation of Internet of Things (IoT) devices across multiple sectors has escalated serious network security concerns. This has prompted ongoing research in Machine Learning (ML)-based Intrusion Detection Systems (IDSs) for cyber-attack classification. Traditional ML models require data transmission from IoT devices to a centralized server for traffic analysis, raising severe privacy concerns. To address this issue, researchers have studied Federated Learning (FL)-based IDSs that train models across IoT devices while keeping their data localized. However, the heterogeneity of data, stemming from distinct vulnerabilities of devices and complexity of attack vectors, poses a significant challenge to the effectiveness of FL models. While current research focuses on adapting various ML models within the FL framework, they fail to effectively address the issue of attack class imbalance among devices, which significantly degrades the classification accuracy of minority attacks. To overcome this challenge, we introduce FedMADE, a novel dynamic aggregation method, which clusters devices by their traffic patterns and aggregates local models based on their contributions towards overall performance. We evaluate FedMADE against other FL algorithms designed for non-IID data and observe up to 71.07% improvement in minority attack classification accuracy. We further show that FedMADE is robust to poisoning attacks and incurs only a 4.7% (5.03 seconds) latency overhead in each communication round compared to FedAvg, without increasing the computational load of IoT devices.

Title: Flexible 3D Lane Detection by Hierarchical Shape MatchingFlexible 3D Lane Detection by Hierarchical Shape Matching

• Authors: Zhihao Guan, Ruixin Liu, Zejian Yuan, Ao Liu, Kun Tang, Tong Zhou, Erlong Li, Chao Zheng, Shuqi Mei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07163
• Pdf URL: https://arxiv.org/pdf/2408.07163
• Copy Paste: [[2408.07163]] Flexible 3D Lane Detection by Hierarchical Shape MatchingFlexible 3D Lane Detection by Hierarchical Shape Matching(https://arxiv.org/abs/2408.07163)
• Keywords: robust
• Abstract: As one of the basic while vital technologies for HD map construction, 3D lane detection is still an open problem due to varying visual conditions, complex typologies, and strict demands for precision. In this paper, an end-to-end flexible and hierarchical lane detector is proposed to precisely predict 3D lane lines from point clouds. Specifically, we design a hierarchical network predicting flexible representations of lane shapes at different levels, simultaneously collecting global instance semantics and avoiding local errors. In the global scope, we propose to regress parametric curves w.r.t adaptive axes that help to make more robust predictions towards complex scenes, while in the local vision the structure of lane segment is detected in each of the dynamic anchor cells sampled along the global predicted curves. Moreover, corresponding global and local shape matching losses and anchor cell generation strategies are designed. Experiments on two datasets show that we overwhelm current top methods under high precision standards, and full ablation studies also verify each part of our method. Our codes will be released at this https URL.

Title: Unlocking Efficiency: Adaptive Masking for Gene Transformer Models

• Authors: Soumyadeep Roy, Shamik Sural, Niloy Ganguly
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07180
• Pdf URL: https://arxiv.org/pdf/2408.07180
• Copy Paste: [[2408.07180]] Unlocking Efficiency: Adaptive Masking for Gene Transformer Models(https://arxiv.org/abs/2408.07180)
• Keywords: transformer
• Abstract: Gene transformer models such as Nucleotide Transformer, DNABert, and LOGO are trained to learn optimal gene sequence representations by using the Masked Language Modeling (MLM) training objective over the complete Human Reference Genome. However, the typical tokenization methods employ a basic sliding window of tokens, such as k-mers, that fail to utilize gene-centric semantics. This could result in the (trivial) masking of easily predictable sequences, leading to inefficient MLM training. Time-variant training strategies are known to improve pretraining efficiency in both language and vision tasks. In this work, we focus on using curriculum masking where we systematically increase the difficulty of masked token prediction task by using a Pointwise Mutual Information-based difficulty criterion, as gene sequences lack well-defined semantic units similar to words or sentences of NLP domain. Our proposed Curriculum Masking-based Gene Masking Strategy (CM-GEMS) demonstrates superior representation learning capabilities compared to baseline masking approaches when evaluated on downstream gene sequence classification tasks. We perform extensive evaluation in both few-shot (five datasets) and full dataset settings (Genomic Understanding Evaluation benchmark consisting of 27 tasks). Our findings reveal that CM-GEMS outperforms state-of-the-art models (DNABert-2, Nucleotide transformer, DNABert) trained at 120K steps, achieving similar results in just 10K and 1K steps. We also demonstrate that Curriculum-Learned LOGO (a 2-layer DNABert-like model) can achieve nearly 90% of the state-of-the-art model performance of 120K steps. We will make the models and codes publicly available at this https URL.

Title: VulCatch: Enhancing Binary Vulnerability Detection through CodeT5 Decompilation and KAN Advanced Feature Extraction

• Authors: Abdulrahman Hamman Adama Chukkol, Senlin Luo, Kashif Sharif, Yunusa Haruna, Muhammad Muhammad Abdullahi
• Subjects: cs.CR, cs.AI, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.07181
• Pdf URL: https://arxiv.org/pdf/2408.07181
• Copy Paste: [[2408.07181]] VulCatch: Enhancing Binary Vulnerability Detection through CodeT5 Decompilation and KAN Advanced Feature Extraction(https://arxiv.org/abs/2408.07181)
• Keywords: security, extraction
• Abstract: Binary program vulnerability detection is critical for software security, yet existing deep learning approaches often rely on source code analysis, limiting their ability to detect unknown vulnerabilities. To address this, we propose VulCatch, a binary-level vulnerability detection framework. VulCatch introduces a Synergy Decompilation Module (SDM) and Kolmogorov-Arnold Networks (KAN) to transform raw binary code into pseudocode using CodeT5, preserving high-level semantics for deep analysis with tools like Ghidra and IDA. KAN further enhances feature transformation, enabling the detection of complex vulnerabilities. VulCatch employs word2vec, Inception Blocks, BiLSTM Attention, and Residual connections to achieve high detection accuracy (98.88%) and precision (97.92%), while minimizing false positives (1.56%) and false negatives (2.71%) across seven CVE datasets.

Title: SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesis

• Authors: Yuchen Mao, Hongwei Li, Wei Pang, Giorgos Papanastasiou, Guang Yang, Chengjia Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07196
• Pdf URL: https://arxiv.org/pdf/2408.07196
• Copy Paste: [[2408.07196]] SeLoRA: Self-Expanding Low-Rank Adaptation of Latent Diffusion Model for Medical Image Synthesis(https://arxiv.org/abs/2408.07196)
• Keywords: diffusion, large language model
• Abstract: The persistent challenge of medical image synthesis posed by the scarcity of annotated data and the need to synthesize `missing modalities' for multi-modal analysis, underscored the imperative development of effective synthesis methods. Recently, the combination of Low-Rank Adaptation (LoRA) with latent diffusion models (LDMs) has emerged as a viable approach for efficiently adapting pre-trained large language models, in the medical field. However, the direct application of LoRA assumes uniform ranking across all linear layers, overlooking the significance of different weight matrices, and leading to sub-optimal outcomes. Prior works on LoRA prioritize the reduction of trainable parameters, and there exists an opportunity to further tailor this adaptation process to the intricate demands of medical image synthesis. In response, we present SeLoRA, a Self-Expanding Low-Rank Adaptation Module, that dynamically expands its ranking across layers during training, strategically placing additional ranks on crucial layers, to allow the model to elevate synthesis quality where it matters most. The proposed method not only enables LDMs to fine-tune on medical data efficiently but also empowers the model to achieve improved image quality with minimal ranking. The code of our SeLoRA method is publicly available on https://anonymous.4open.science/r/SeLoRA-980D .

Title: Quantification of total uncertainty in the physics-informed reconstruction of CVSim-6 physiology

• Authors: Mario De Florio, Zongren Zou, Daniele E. Schiavazzi, George Em Karniadakis
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2408.07201
• Pdf URL: https://arxiv.org/pdf/2408.07201
• Copy Paste: [[2408.07201]] Quantification of total uncertainty in the physics-informed reconstruction of CVSim-6 physiology(https://arxiv.org/abs/2408.07201)
• Keywords: robust
• Abstract: When predicting physical phenomena through simulation, quantification of the total uncertainty due to multiple sources is as crucial as making sure the underlying numerical model is accurate. Possible sources include irreducible aleatoric uncertainty due to noise in the data, epistemic uncertainty induced by insufficient data or inadequate parameterization, and model-form uncertainty related to the use of misspecified model equations. Physics-based regularization interacts in nontrivial ways with aleatoric, epistemic and model-form uncertainty and their combination, and a better understanding of this interaction is needed to improve the predictive performance of physics-informed digital twins that operate under real conditions. With a specific focus on biological and physiological models, this study investigates the decomposition of total uncertainty in the estimation of states and parameters of a differential system simulated with MC X-TFC, a new physics-informed approach for uncertainty quantification based on random projections and Monte-Carlo sampling. MC X-TFC is applied to a six-compartment stiff ODE system, the CVSim-6 model, developed in the context of human physiology. The system is analyzed by progressively removing data while estimating an increasing number of parameters and by investigating total uncertainty under model-form misspecification of non-linear resistance in the pulmonary compartment. In particular, we focus on the interaction between the formulation of the discrepancy term and quantification of model-form uncertainty, and show how additional physics can help in the estimation process. The method demonstrates robustness and efficiency in estimating unknown states and parameters, even with limited, sparse, and noisy data. It also offers great flexibility in integrating data with physics for improved estimation, even in cases of model misspecification.

Title: Neural embedding of beliefs reveals the role of relative dissonance in human decision-making

• Authors: Byunghwee Lee, Rachith Aiyappa, Yong-Yeol Ahn, Haewoon Kwak, Jisun An
• Subjects: cs.CL, cs.CY, physics.soc-ph
• Abstract URL: https://arxiv.org/abs/2408.07237
• Pdf URL: https://arxiv.org/pdf/2408.07237
• Copy Paste: [[2408.07237]] Neural embedding of beliefs reveals the role of relative dissonance in human decision-making(https://arxiv.org/abs/2408.07237)
• Keywords: large language model
• Abstract: Beliefs serve as the foundation for human cognition and decision-making. They guide individuals in deriving meaning from their lives, shaping their behaviors, and forming social connections. Therefore, a model that encapsulates beliefs and their interrelationships is crucial for quantitatively studying the influence of beliefs on our actions. Despite its importance, research on the interplay between human beliefs has often been limited to a small set of beliefs pertaining to specific issues, with a heavy reliance on surveys or experiments. Here, we propose a method for extracting nuanced relations between thousands of beliefs by leveraging large-scale user participation data from an online debate platform and mapping these beliefs to an embedding space using a fine-tuned large language model (LLM). This belief embedding space effectively encapsulates the interconnectedness of diverse beliefs as well as polarization across various social issues. We discover that the positions within this belief space predict new beliefs of individuals. Furthermore, we find that the relative distance between one's existing beliefs and new beliefs can serve as a quantitative estimate of cognitive dissonance, allowing us to predict new beliefs. Our study highlights how modern LLMs, when combined with collective online records of human beliefs, can offer insights into the fundamental principles that govern human belief formation and decision-making processes.

Title: Using Advanced LLMs to Enhance Smaller LLMs: An Interpretable Knowledge Distillation Approach

• Authors: Tong Wang, K. Sudhir, Dat Hong
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07238
• Pdf URL: https://arxiv.org/pdf/2408.07238
• Copy Paste: [[2408.07238]] Using Advanced LLMs to Enhance Smaller LLMs: An Interpretable Knowledge Distillation Approach(https://arxiv.org/abs/2408.07238)
• Keywords: security, privacy, large language model
• Abstract: Advanced Large language models (LLMs) like GPT-4 or LlaMa 3 provide superior performance in complex human-like interactions. But they are costly, or too large for edge devices such as smartphones and harder to self-host, leading to security and privacy concerns. This paper introduces a novel interpretable knowledge distillation approach to enhance the performance of smaller, more economical LLMs that firms can self-host. We study this problem in the context of building a customer service agent aimed at achieving high customer satisfaction through goal-oriented dialogues. Unlike traditional knowledge distillation, where the "student" model learns directly from the "teacher" model's responses via fine-tuning, our interpretable "strategy" teaching approach involves the teacher providing strategies to improve the student's performance in various scenarios. This method alternates between a "scenario generation" step and a "strategies for improvement" step, creating a customized library of scenarios and optimized strategies for automated prompting. The method requires only black-box access to both student and teacher models; hence it can be used without manipulating model parameters. In our customer service application, the method improves performance, and the learned strategies are transferable to other LLMs and scenarios beyond the training set. The method's interpretabilty helps safeguard against potential harms through human audit.

Title: Enhancing Autonomous Vehicle Perception in Adverse Weather through Image Augmentation during Semantic Segmentation Training

• Authors: Ethan Kou, Noah Curran
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07239
• Pdf URL: https://arxiv.org/pdf/2408.07239
• Copy Paste: [[2408.07239]] Enhancing Autonomous Vehicle Perception in Adverse Weather through Image Augmentation during Semantic Segmentation Training(https://arxiv.org/abs/2408.07239)
• Keywords: robust, segmentation
• Abstract: Robust perception is crucial in autonomous vehicle navigation and localization. Visual processing tasks, like semantic segmentation, should work in varying weather conditions and during different times of day. Semantic segmentation is where each pixel is assigned a class, which is useful for locating overall features (1). Training a segmentation model requires large amounts of data, and the labeling process for segmentation data is especially tedious. Additionally, many large datasets include only images taken in clear weather. This is a problem because training a model exclusively on clear weather data hinders performance in adverse weather conditions like fog or rain. We hypothesize that given a dataset of only clear days images, applying image augmentation (such as random rain, fog, and brightness) during training allows for domain adaptation to diverse weather conditions. We used CARLA, a 3D realistic autonomous vehicle simulator, to collect 1200 images in clear weather composed of 29 classes from 10 different towns (2). We also collected 1200 images of random weather effects. We trained encoder-decoder UNet models to perform semantic segmentation. Applying augmentations significantly improved segmentation under weathered night conditions (p < 0.001). However, models trained on weather data have significantly lower losses than those trained on augmented data in all conditions except for clear days. This shows there is room for improvement in the domain adaptation approach. Future work should test more types of augmentations and also use real-life images instead of CARLA. Ideally, the augmented model meets or exceeds the performance of the weather model.

Title: Leveraging Perceptual Scores for Dataset Pruning in Computer Vision Tasks

• Authors: Raghavendra Singh
• Subjects: cs.CV, cs.IT
• Abstract URL: https://arxiv.org/abs/2408.07243
• Pdf URL: https://arxiv.org/pdf/2408.07243
• Copy Paste: [[2408.07243]] Leveraging Perceptual Scores for Dataset Pruning in Computer Vision Tasks(https://arxiv.org/abs/2408.07243)
• Keywords: segmentation
• Abstract: In this paper we propose a score of an image to use for coreset selection in image classification and semantic segmentation tasks. The score is the entropy of an image as approximated by the bits-per-pixel of its compressed version. Thus the score is intrinsic to an image and does not require supervision or training. It is very simple to compute and readily available as all images are stored in a compressed format. The motivation behind our choice of score is that most other scores proposed in literature are expensive to compute. More importantly, we want a score that captures the perceptual complexity of an image. Entropy is one such measure, images with clutter tend to have a higher entropy. However sampling only low entropy iconic images, for example, leads to biased learning and an overall decrease in test performance with current deep learning models. To mitigate the bias we use a graph based method that increases the spatial diversity of the selected samples. We show that this simple score yields good results, particularly for semantic segmentation tasks.

Title: Seeing and Understanding: Bridging Vision with Chemical Knowledge Via ChemVLM

• Authors: Junxian Li, Di Zhang, Xunzhi Wang, Zeying Hao, Jingdi Lei, Qian Tan, Cai Zhou, Wei Liu, Weiyun Wang, Zhe Chen, Wenhai Wang, Wei Li, Shufei Zhang, Mao Su, Wanli Ouyang, Yuqiang Li, Dongzhan Zhou
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07246
• Pdf URL: https://arxiv.org/pdf/2408.07246
• Copy Paste: [[2408.07246]] Seeing and Understanding: Bridging Vision with Chemical Knowledge Via ChemVLM(https://arxiv.org/abs/2408.07246)
• Keywords: robust, large language model
• Abstract: In this technical report, we propose ChemVLM, the first open-source multimodal large language model dedicated to the fields of chemistry, designed to address the incompatibility between chemical image understanding and text analysis. Built upon the VIT-MLP-LLM architecture, we leverage ChemLLM-20B as the foundational large model, endowing our model with robust capabilities in understanding and utilizing chemical text knowledge. Additionally, we employ InternVIT-6B as a powerful image encoder. We have curated high-quality data from the chemical domain, including molecules, reaction formulas, and chemistry examination data, and compiled these into a bilingual multimodal question-answering dataset. We test the performance of our model on multiple open-source benchmarks and three custom evaluation sets. Experimental results demonstrate that our model achieves excellent performance, securing state-of-the-art results in five out of six involved tasks. Our model can be found at this https URL.

Title: BiLSTM and Attention-Based Modulation Classification of Realistic Wireless Signals

• Authors: Rohit Udaiwal, Nayan Baishya, Yash Gupta, B. R. Manoj
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2408.07247
• Pdf URL: https://arxiv.org/pdf/2408.07247
• Copy Paste: [[2408.07247]] BiLSTM and Attention-Based Modulation Classification of Realistic Wireless Signals(https://arxiv.org/abs/2408.07247)
• Keywords: robust, extraction
• Abstract: This work proposes a novel and efficient quadstream BiLSTM-Attention network, abbreviated as QSLA network, for robust automatic modulation classification (AMC) of wireless signals. The proposed model exploits multiple representations of the wireless signal as inputs to the network and the feature extraction process combines convolutional and BiLSTM layers for processing the spatial and temporal features of the signal, respectively. An attention layer is used after the BiLSTM layer to emphasize the important temporal features. The experimental results on the recent and realistic RML22 dataset demonstrate the superior performance of the proposed model with an accuracy up to around 99%. The model is compared with other benchmark models in the literature in terms of classification accuracy, computational complexity, memory usage, and training time to show the effectiveness of our proposed approach.

Title: GQE: Generalized Query Expansion for Enhanced Text-Video Retrieval

• Authors: Zechen Bai, Tianjun Xiao, Tong He, Pichao Wang, Zheng Zhang, Thomas Brox, Mike Zheng Shou
• Subjects: cs.CV, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.07249
• Pdf URL: https://arxiv.org/pdf/2408.07249
• Copy Paste: [[2408.07249]] GQE: Generalized Query Expansion for Enhanced Text-Video Retrieval(https://arxiv.org/abs/2408.07249)
• Keywords: large language model
• Abstract: In the rapidly expanding domain of web video content, the task of text-video retrieval has become increasingly critical, bridging the semantic gap between textual queries and video data. This paper introduces a novel data-centric approach, Generalized Query Expansion (GQE), to address the inherent information imbalance between text and video, enhancing the effectiveness of text-video retrieval systems. Unlike traditional model-centric methods that focus on designing intricate cross-modal interaction mechanisms, GQE aims to expand the text queries associated with videos both during training and testing phases. By adaptively segmenting videos into short clips and employing zero-shot captioning, GQE enriches the training dataset with comprehensive scene descriptions, effectively bridging the data imbalance gap. Furthermore, during retrieval, GQE utilizes Large Language Models (LLM) to generate a diverse set of queries and a query selection module to filter these queries based on relevance and diversity, thus optimizing retrieval performance while reducing computational overhead. Our contributions include a detailed examination of the information imbalance challenge, a novel approach to query expansion in video-text datasets, and the introduction of a query selection strategy that enhances retrieval accuracy without increasing computational costs. GQE achieves state-of-the-art performance on several benchmarks, including MSR-VTT, MSVD, LSMDC, and VATEX, demonstrating the effectiveness of addressing text-video retrieval from a data-centric perspective.

Title: GRIF-DM: Generation of Rich Impression Fonts using Diffusion Models

• Authors: Lei Kang, Fei Yang, Kai Wang, Mohamed Ali Souibgui, Lluis Gomez, Alicia Fornés, Ernest Valveny, Dimosthenis Karatzas
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07259
• Pdf URL: https://arxiv.org/pdf/2408.07259
• Copy Paste: [[2408.07259]] GRIF-DM: Generation of Rich Impression Fonts using Diffusion Models(https://arxiv.org/abs/2408.07259)
• Keywords: diffusion, generative
• Abstract: Fonts are integral to creative endeavors, design processes, and artistic productions. The appropriate selection of a font can significantly enhance artwork and endow advertisements with a higher level of expressivity. Despite the availability of numerous diverse font designs online, traditional retrieval-based methods for font selection are increasingly being supplanted by generation-based approaches. These newer methods offer enhanced flexibility, catering to specific user preferences and capturing unique stylistic impressions. However, current impression font techniques based on Generative Adversarial Networks (GANs) necessitate the utilization of multiple auxiliary losses to provide guidance during generation. Furthermore, these methods commonly employ weighted summation for the fusion of impression-related keywords. This leads to generic vectors with the addition of more impression keywords, ultimately lacking in detail generation capacity. In this paper, we introduce a diffusion-based method, termed \ourmethod, to generate fonts that vividly embody specific impressions, utilizing an input consisting of a single letter and a set of descriptive impression keywords. The core innovation of \ourmethod lies in the development of dual cross-attention modules, which process the characteristics of the letters and impression keywords independently but synergistically, ensuring effective integration of both types of information. Our experimental results, conducted on the MyFonts dataset, affirm that this method is capable of producing realistic, vibrant, and high-fidelity fonts that are closely aligned with user specifications. This confirms the potential of our approach to revolutionize font generation by accommodating a broad spectrum of user-driven design requirements. Our code is publicly available at \url{this https URL}.

Title: Ensemble architecture in polyp segmentation

• Authors: Hao-Yun Hsu, Yi-Ching Cheng, Guan-Hua Huang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07262
• Pdf URL: https://arxiv.org/pdf/2408.07262
• Copy Paste: [[2408.07262]] Ensemble architecture in polyp segmentation(https://arxiv.org/abs/2408.07262)
• Keywords: transformer, segmentation
• Abstract: In this research, we revisit the architecture of semantic segmentation and evaluate the models excelling in polyp segmentation. We introduce an integrated framework that harnesses the advantages of different models to attain an optimal outcome. More specifically, we fuse the learned features from convolutional and transformer models for prediction, and we view this approach as an ensemble technique to enhance model performance. Our experiments on polyp segmentation reveal that the proposed architecture surpasses other top models, exhibiting improved learning capacity and resilience. The code is available at this https URL.

Title: Eavesdropping Mobile Apps and Actions through Wireless Traffic in the Open World

• Authors: Xiaoguang Yang, Yong Huang, Junli Guo, Dalong Zhang, Qingxian Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.07263
• Pdf URL: https://arxiv.org/pdf/2408.07263
• Copy Paste: [[2408.07263]] Eavesdropping Mobile Apps and Actions through Wireless Traffic in the Open World(https://arxiv.org/abs/2408.07263)
• Keywords: attack
• Abstract: While smartphones and WiFi networks are bringing many positive changes to people's lives, they are susceptible to traffic analysis attacks, which infer user's private information from encrypted traffic. Existing traffic analysis attacks mainly target TCP/IP layers or are limited to the closed-world assumption, where all possible apps and actions have been involved in the model training. To overcome these limitations, we propose MACPrint, a novel system that infers mobile apps and in-app actions based on WiFi MAC layer traffic in the open-world setting. MACPrint first extracts rich statistical and contextual features of encrypted wireless traffic. Then, we develop Label Recorder, an automatic traffic labeling app, to improve labeling accuracy in the training phase. Finally, TCN models with OpenMax functions are used to recognize mobile apps and actions in the open world accurately. To evaluate our system, we collect MAC layer traffic data over 125 hours from more than 40 apps. The experimental results show that MAC-Print can achieve an accuracy of over 96% for recognizing apps and actions in the closed-world setting, and obtains an accuracy of over 86% in the open-world setting.

Title: Image-Based Leopard Seal Recognition: Approaches and Challenges in Current Automated Systems

• Authors: Jorge Yero Salazar, Pablo Rivas, Renato Borras-Chavez, Sarah Kienle
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07269
• Pdf URL: https://arxiv.org/pdf/2408.07269
• Copy Paste: [[2408.07269]] Image-Based Leopard Seal Recognition: Approaches and Challenges in Current Automated Systems(https://arxiv.org/abs/2408.07269)
• Keywords: transformer, segmentation
• Abstract: This paper examines the challenges and advancements in recognizing seals within their natural habitats using conventional photography, underscored by the emergence of machine learning technologies. We used the leopard seal, \emph{Hydrurga leptonyx}, a key species within Antarctic ecosystems, to review the different available methods found. As apex predators, Leopard seals are characterized by their significant ecological role and elusive nature so studying them is crucial to understand the health of their ecosystem. Traditional methods of monitoring seal species are often constrained by the labor-intensive and time-consuming processes required for collecting data, compounded by the limited insights these methods provide. The advent of machine learning, particularly through the application of vision transformers, heralds a new era of efficiency and precision in species monitoring. By leveraging state-of-the-art approaches in detection, segmentation, and recognition within digital imaging, this paper presents a synthesis of the current landscape, highlighting both the cutting-edge methodologies and the predominant challenges faced in accurately identifying seals through photographic data.

Title: DDIM Redux: Mathematical Foundation and Some Extension

• Authors: Manhyung Han
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07285
• Pdf URL: https://arxiv.org/pdf/2408.07285
• Copy Paste: [[2408.07285]] DDIM Redux: Mathematical Foundation and Some Extension(https://arxiv.org/abs/2408.07285)
• Keywords: diffusion
• Abstract: This note provides a critical review of the mathematical concepts underlying the generalized diffusion denoising implicit model (gDDIM) and the exponential integrator (EI) scheme. We present enhanced mathematical results, including an exact expression for the reverse trajectory in the probability flow ODE and an exact expression for the covariance matrix in the gDDIM scheme. Furthermore, we offer an improved understanding of the EI scheme's efficiency in terms of the change of variables. The noising process in DDIM is analyzed from the perspective of non-equilibrium statistical physics. Additionally, we propose a new scheme for DDIM, called the principal-axis DDIM (paDDIM).

Title: Evaluating Large Language Model based Personal Information Extraction and Countermeasures

• Authors: Yupei Liu, Yuqi Jia, Jinyuan Jia, Neil Zhenqiang Gong
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.07291
• Pdf URL: https://arxiv.org/pdf/2408.07291
• Copy Paste: [[2408.07291]] Evaluating Large Language Model based Personal Information Extraction and Countermeasures(https://arxiv.org/abs/2408.07291)
• Keywords: security, attack, extraction, large language model
• Abstract: Automatically extracting personal information--such as name, phone number, and email address--from publicly available profiles at a large scale is a stepstone to many other security attacks including spear phishing. Traditional methods--such as regular expression, keyword search, and entity detection--achieve limited success at such personal information extraction. In this work, we perform a systematic measurement study to benchmark large language model (LLM) based personal information extraction and countermeasures. Towards this goal, we present a framework for LLM-based extraction attacks; collect three datasets including a synthetic dataset generated by GPT-4 and two real-world datasets with manually labeled 8 categories of personal information; introduce a novel mitigation strategy based on \emph{prompt injection}; and systematically benchmark LLM-based attacks and countermeasures using 10 LLMs and our 3 datasets. Our key findings include: LLM can be misused by attackers to accurately extract various personal information from personal profiles; LLM outperforms conventional methods at such extraction; and prompt injection can mitigate such risk to a large extent and outperforms conventional countermeasures. Our code and data are available at: \url{this https URL}.

Title: Enhancing Visual Question Answering through Ranking-Based Hybrid Training and Multimodal Fusion

• Authors: Peiyuan Chen, Zecheng Zhang, Yiping Dong, Li Zhou, Han Wang
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07303
• Pdf URL: https://arxiv.org/pdf/2408.07303
• Copy Paste: [[2408.07303]] Enhancing Visual Question Answering through Ranking-Based Hybrid Training and Multimodal Fusion(https://arxiv.org/abs/2408.07303)
• Keywords: robust
• Abstract: Visual Question Answering (VQA) is a challenging task that requires systems to provide accurate answers to questions based on image content. Current VQA models struggle with complex questions due to limitations in capturing and integrating multimodal information effectively. To address these challenges, we propose the Rank VQA model, which leverages a ranking-inspired hybrid training strategy to enhance VQA performance. The Rank VQA model integrates high-quality visual features extracted using the Faster R-CNN model and rich semantic text features obtained from a pre-trained BERT model. These features are fused through a sophisticated multimodal fusion technique employing multi-head self-attention mechanisms. Additionally, a ranking learning module is incorporated to optimize the relative ranking of answers, thus improving answer accuracy. The hybrid training strategy combines classification and ranking losses, enhancing the model's generalization ability and robustness across diverse datasets. Experimental results demonstrate the effectiveness of the Rank VQA model. Our model significantly outperforms existing state-of-the-art models on standard VQA datasets, including VQA v2.0 and COCO-QA, in terms of both accuracy and Mean Reciprocal Rank (MRR). The superior performance of Rank VQA is evident in its ability to handle complex questions that require understanding nuanced details and making sophisticated inferences from the image and text. This work highlights the effectiveness of a ranking-based hybrid training strategy in improving VQA performance and lays the groundwork for further research in multimodal learning methods.

Title: At Least Factor-of-Two Optimization for RWLE-Based Homomorphic Encryption

• Authors: Jonathan Ly
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.07304
• Pdf URL: https://arxiv.org/pdf/2408.07304
• Copy Paste: [[2408.07304]] At Least Factor-of-Two Optimization for RWLE-Based Homomorphic Encryption(https://arxiv.org/abs/2408.07304)
• Keywords: security, privacy
• Abstract: Many modern applications that deal with sensitive data, such as healthcare and government services, outsource computation to cloud platforms. In such untrusted environments, privacy is of vital importance. One solution to this problem is homomorphic encryption (HE), a family of cryptographic schemes that support certain algebraic operations on encrypted data without the need for decryption. However, despite major advancements, encryption in modern HE schemes still comes with a non-trivial computational overhead that can hamper data-intensive workloads. To resolve this, recent research has shown that leveraging caching techniques, such as Rache, can significantly enhance the performance of HE schemes while maintaining security. Rache unfortunately displays a key limitation in the time complexity of its caching procedure, which scales with the size of the plaintext space. Smuche is another caching scheme that simultaneously improves the scalability of the caching procedure and turns the encryption process into a constant-time operation, utilizing only a single scalar multiplication. Even still, more can be done. In this paper, we present an encryption method we call ``Zinc" which entirely forgoes the multiple caching process, replacing it with a single scalar addition, and then injecting randomness that takes constant time with respect to the plaintext space. This injection of randomness is similar to Smuche, and a great improvement from Rache, allowing Zinc to achieve efficiency without compromising security. We implement the scheme using Microsoft SEAL and compare its performance to vanilla CKKS.

Title: Kolmogorov-Arnold Networks (KAN) for Time Series Classification and Robust Analysis

• Authors: Chang Dong, Liangwei Zheng, Weitong Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07314
• Pdf URL: https://arxiv.org/pdf/2408.07314
• Copy Paste: [[2408.07314]] Kolmogorov-Arnold Networks (KAN) for Time Series Classification and Robust Analysis(https://arxiv.org/abs/2408.07314)
• Keywords: robust, fair
• Abstract: Kolmogorov-Arnold Networks (KAN) has recently attracted significant attention as a promising alternative to traditional Multi-Layer Perceptrons (MLP). Despite their theoretical appeal, KAN require validation on large-scale benchmark datasets. Time series data, which has become increasingly prevalent in recent years, especially univariate time series are naturally suited for validating KAN. Therefore, we conducted a fair comparison among KAN, MLP, and mixed structures. The results indicate that KAN can achieve performance comparable to, or even slightly better than, MLP across 128 time series datasets. We also performed an ablation study on KAN, revealing that the output is primarily determined by the base component instead of b-spline function. Furthermore, we assessed the robustness of these models and found that KAN and the hybrid structure MLP\_KAN exhibit significant robustness advantages, attributed to their lower Lipschitz constants. This suggests that KAN and KAN layers hold strong potential to be robust models or to improve the adversarial robustness of other models.

Title: KIND: Knowledge Integration and Diversion in Diffusion Models

• Authors: Yucheng Xie, Fu Feng, Jing Wang, Xin Geng, Yong Rui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07337
• Pdf URL: https://arxiv.org/pdf/2408.07337
• Copy Paste: [[2408.07337]] KIND: Knowledge Integration and Diversion in Diffusion Models(https://arxiv.org/abs/2408.07337)
• Keywords: diffusion
• Abstract: Pre-trained models have become the preferred backbone due to the expansion of model parameters, with techniques like Parameter-Efficient Fine-Tuning (PEFTs) typically fixing the parameters of these models. However, pre-trained models may not always be optimal, especially when there are discrepancies between training tasks and target tasks, potentially resulting in negative transfer. To address this, we introduce \textbf{KIND}, which performs \textbf{K}nowledge \textbf{IN}tegration and \textbf{D}iversion in diffusion models. KIND first integrates knowledge by decomposing parameter matrices of models using $U$, $\Sigma$, and $V$ matrices, formally inspired by singular value decomposition (SVD). Then it explicitly partitions the components of these matrices into \textbf{learngenes} and \textbf{tailors} to condense common and class-specific knowledge, respectively, through a class gate. In this way, KIND redefines traditional pre-training methods by adjusting training objectives from maximizing model performance on current tasks to condensing transferable common knowledge, leveraging the \textit{Learngene} framework. We conduct experiments on ImageNet-1K and compare KIND with PEFT and other learngene methods. Results indicate that KIND achieves state-of-the-art performance compared to other PEFT and learngene methods. Specifically, the images generated by KIND achieves more than 6.54 and 1.07 decrease in FID and sFID on DiT-L/2, utilizing only 45.4M trainable parameters and saving at least 35.4G FLOPs in computational cost.

Title: Towards Few-shot Self-explaining Graph Neural Networks

• Authors: Jingyu Peng, Qi Liu, Linan Yue, Zaixi Zhang, Kai Zhang, Yunhao Sha
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07340
• Pdf URL: https://arxiv.org/pdf/2408.07340
• Copy Paste: [[2408.07340]] Towards Few-shot Self-explaining Graph Neural Networks(https://arxiv.org/abs/2408.07340)
• Keywords: explainability
• Abstract: Recent advancements in Graph Neural Networks (GNNs) have spurred an upsurge of research dedicated to enhancing the explainability of GNNs, particularly in critical domains such as medicine. A promising approach is the self-explaining method, which outputs explanations along with predictions. However, existing self-explaining models require a large amount of training data, rendering them unavailable in few-shot scenarios. To address this challenge, in this paper, we propose a Meta-learned Self-Explaining GNN (MSE-GNN), a novel framework that generates explanations to support predictions in few-shot settings. MSE-GNN adopts a two-stage self-explaining structure, consisting of an explainer and a predictor. Specifically, the explainer first imitates the attention mechanism of humans to select the explanation subgraph, whereby attention is naturally paid to regions containing important characteristics. Subsequently, the predictor mimics the decision-making process, which makes predictions based on the generated explanation. Moreover, with a novel meta-training process and a designed mechanism that exploits task information, MSE-GNN can achieve remarkable performance on new few-shot tasks. Extensive experimental results on four datasets demonstrate that MSE-GNN can achieve superior performance on prediction tasks while generating high-quality explanations compared with existing methods. The code is publicly available at this https URL.

Title: Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration

• Authors: Xiaogen Zhon, Yiyou Sun, Min Deng, Winnie Chiu Wing Chu, Qi Dou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07341
• Pdf URL: https://arxiv.org/pdf/2408.07341
• Copy Paste: [[2408.07341]] Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration(https://arxiv.org/abs/2408.07341)
• Keywords: robust, segmentation
• Abstract: Multimodal learning leverages complementary information derived from different modalities, thereby enhancing performance in medical image segmentation. However, prevailing multimodal learning methods heavily rely on extensive well-annotated data from various modalities to achieve accurate segmentation performance. This dependence often poses a challenge in clinical settings due to limited availability of such data. Moreover, the inherent anatomical misalignment between different imaging modalities further complicates the endeavor to enhance segmentation performance. To address this problem, we propose a novel semi-supervised multimodal segmentation framework that is robust to scarce labeled data and misaligned modalities. Our framework employs a novel cross modality collaboration strategy to distill modality-independent knowledge, which is inherently associated with each modality, and integrates this information into a unified fusion layer for feature amalgamation. With a channel-wise semantic consistency loss, our framework ensures alignment of modality-independent information from a feature-wise perspective across modalities, thereby fortifying it against misalignments in multimodal scenarios. Furthermore, our framework effectively integrates contrastive consistent learning to regulate anatomical structures, facilitating anatomical-wise prediction alignment on unlabeled data in semi-supervised segmentation tasks. Our method achieves competitive performance compared to other multimodal methods across three tasks: cardiac, abdominal multi-organ, and thyroid-associated orbitopathy segmentations. It also demonstrates outstanding robustness in scenarios involving scarce labeled data and misaligned modalities.

Title: Gradient Alignment Improves Test-Time Adaptation for Medical Image Segmentation

• Authors: Ziyang Chen, Yiwen Ye, Yongsheng Pan, Yong Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07343
• Pdf URL: https://arxiv.org/pdf/2408.07343
• Copy Paste: [[2408.07343]] Gradient Alignment Improves Test-Time Adaptation for Medical Image Segmentation(https://arxiv.org/abs/2408.07343)
• Keywords: segmentation
• Abstract: Although recent years have witnessed significant advancements in medical image segmentation, the pervasive issue of domain shift among medical images from diverse centres hinders the effective deployment of pre-trained models. Many Test-time Adaptation (TTA) methods have been proposed to address this issue by fine-tuning pre-trained models with test data during inference. These methods, however, often suffer from less-satisfactory optimization due to suboptimal optimization direction (dictated by the gradient) and fixed step-size (predicated on the learning rate). In this paper, we propose the Gradient alignment-based Test-time adaptation (GraTa) method to improve both the gradient direction and learning rate in the optimization procedure. Unlike conventional TTA methods, which primarily optimize the pseudo gradient derived from a self-supervised objective, our method incorporates an auxiliary gradient with the pseudo one to facilitate gradient alignment. Such gradient alignment enables the model to excavate the similarities between different gradients and correct the gradient direction to approximate the empirical gradient related to the current segmentation task. Additionally, we design a dynamic learning rate based on the cosine similarity between the pseudo and auxiliary gradients, thereby empowering the adaptive fine-tuning of pre-trained models on diverse test data. Extensive experiments establish the effectiveness of the proposed gradient alignment and dynamic learning rate and substantiate the superiority of our GraTa method over other state-of-the-art TTA methods on a benchmark medical image segmentation task. The code and weights of pre-trained source models will be available.

Title: RTAT: A Robust Two-stage Association Tracker for Multi-Object Tracking

• Authors: Song Guo, Rujie Liu, Narishige Abe
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07344
• Pdf URL: https://arxiv.org/pdf/2408.07344
• Copy Paste: [[2408.07344]] RTAT: A Robust Two-stage Association Tracker for Multi-Object Tracking(https://arxiv.org/abs/2408.07344)
• Keywords: robust
• Abstract: Data association is an essential part in the tracking-by-detection based Multi-Object Tracking (MOT). Most trackers focus on how to design a better data association strategy to improve the tracking performance. The rule-based handcrafted association methods are simple and highly efficient but lack generalization capability to deal with complex scenes. While the learnt association methods can learn high-order contextual information to deal with various complex scenes, but they have the limitations of higher complexity and cost. To address these limitations, we propose a Robust Two-stage Association Tracker, named RTAT. The first-stage association is performed between tracklets and detections to generate tracklets with high purity, and the second-stage association is performed between tracklets to form complete trajectories. For the first-stage association, we use a simple data association strategy to generate tracklets with high purity by setting a low threshold for the matching cost in the assignment process. We conduct the tracklet association in the second-stage based on the framework of message-passing GNN. Our method models the tracklet association as a series of edge classification problem in hierarchical graphs, which can recursively merge short tracklets into longer ones. Our tracker RTAT ranks first on the test set of MOT17 and MOT20 benchmarks in most of the main MOT metrics: HOTA, IDF1, and AssA. We achieve 67.2 HOTA, 84.7 IDF1, and 69.7 AssA on MOT17, and 66.2 HOTA, 82.5 IDF1, and 68.1 AssA on MOT20.

Title: Only One Relation Possible? Modeling the Ambiguity in Event Temporal Relation Extraction

• Authors: Yutong Hu, Quzhe Huang, Yansong Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07353
• Pdf URL: https://arxiv.org/pdf/2408.07353
• Copy Paste: [[2408.07353]] Only One Relation Possible? Modeling the Ambiguity in Event Temporal Relation Extraction(https://arxiv.org/abs/2408.07353)
• Keywords: extraction
• Abstract: Event Temporal Relation Extraction (ETRE) aims to identify the temporal relationship between two events, which plays an important role in natural language understanding. Most previous works follow a single-label classification style, classifying an event pair into either a specific temporal relation (e.g., \textit{Before}, \textit{After}), or a special label \textit{Vague} when there may be multiple possible temporal relations between the pair. In our work, instead of directly making predictions on \textit{Vague}, we propose a multi-label classification solution for ETRE (METRE) to infer the possibility of each temporal relation independently, where we treat \textit{Vague} as the cases when there is more than one possible relation between two events. We design a speculation mechanism to explore the possible relations hidden behind \textit{Vague}, which enables the latent information to be used efficiently. Experiments on TB-Dense, MATRES and UDS-T show that our method can effectively utilize the \textit{Vague} instances to improve the recognition for specific temporal relations and outperforms most state-of-the-art methods.

Title: BadMerging: Backdoor Attacks Against Model Merging

• Authors: Jinghuai Zhang, Jianfeng Chi, Zheng Li, Kunlin Cai, Yang Zhang, Yuan Tian
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07362
• Pdf URL: https://arxiv.org/pdf/2408.07362
• Copy Paste: [[2408.07362]] BadMerging: Backdoor Attacks Against Model Merging(https://arxiv.org/abs/2408.07362)
• Keywords: security, defense, attack, robust
• Abstract: Fine-tuning pre-trained models for downstream tasks has led to a proliferation of open-sourced task-specific models. Recently, Model Merging (MM) has emerged as an effective approach to facilitate knowledge transfer among these independently fine-tuned models. MM directly combines multiple fine-tuned task-specific models into a merged model without additional training, and the resulting model shows enhanced capabilities in multiple tasks. Although MM provides great utility, it may come with security risks because an adversary can exploit MM to affect multiple downstream tasks. However, the security risks of MM have barely been studied. In this paper, we first find that MM, as a new learning paradigm, introduces unique challenges for existing backdoor attacks due to the merging process. To address these challenges, we introduce BadMerging, the first backdoor attack specifically designed for MM. Notably, BadMerging allows an adversary to compromise the entire merged model by contributing as few as one backdoored task-specific model. BadMerging comprises a two-stage attack mechanism and a novel feature-interpolation-based loss to enhance the robustness of embedded backdoors against the changes of different merging parameters. Considering that a merged model may incorporate tasks from different domains, BadMerging can jointly compromise the tasks provided by the adversary (on-task attack) and other contributors (off-task attack) and solve the corresponding unique challenges with novel attack designs. Extensive experiments show that BadMerging achieves remarkable attacks against various MM algorithms. Our ablation study demonstrates that the proposed attack designs can progressively contribute to the attack performance. Finally, we show that prior defense mechanisms fail to defend against our attacks, highlighting the need for more advanced defense.

Title: Robust Active Learning (RoAL): Countering Dynamic Adversaries in Active Learning with Elastic Weight Consolidation

• Authors: Ricky Maulana Fajri, Yulong Pei, Lu Yin, Mykola Pechenizkiy
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07364
• Pdf URL: https://arxiv.org/pdf/2408.07364
• Copy Paste: [[2408.07364]] Robust Active Learning (RoAL): Countering Dynamic Adversaries in Active Learning with Elastic Weight Consolidation(https://arxiv.org/abs/2408.07364)
• Keywords: attack, robust
• Abstract: Despite significant advancements in active learning and adversarial attacks, the intersection of these two fields remains underexplored, particularly in developing robust active learning frameworks against dynamic adversarial threats. The challenge of developing robust active learning frameworks under dynamic adversarial attacks is critical, as these attacks can lead to catastrophic forgetting within the active learning cycle. This paper introduces Robust Active Learning (RoAL), a novel approach designed to address this issue by integrating Elastic Weight Consolidation (EWC) into the active learning process. Our contributions are threefold: First, we propose a new dynamic adversarial attack that poses significant threats to active learning frameworks. Second, we introduce a novel method that combines EWC with active learning to mitigate catastrophic forgetting caused by dynamic adversarial attacks. Finally, we conduct extensive experimental evaluations to demonstrate the efficacy of our approach. The results show that RoAL not only effectively counters dynamic adversarial threats but also significantly reduces the impact of catastrophic forgetting, thereby enhancing the robustness and performance of active learning systems in adversarial environments.

Title: Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics

• Authors: Peter Romero, Stephen Fitz, Teruo Nakatsuma
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.07377
• Pdf URL: https://arxiv.org/pdf/2408.07377
• Copy Paste: [[2408.07377]] Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics(https://arxiv.org/abs/2408.07377)
• Keywords: large language model
• Abstract: Previous research on emergence in large language models shows these display apparent human-like abilities and psychological latent traits. However, results are partly contradicting in expression and magnitude of these latent traits, yet agree on the worrisome tendencies to score high on the Dark Triad of narcissism, psychopathy, and Machiavellianism, which, together with a track record of derailments, demands more rigorous research on safety of these models. We provided a state of the art language model with the same personality questionnaire in nine languages, and performed Bayesian analysis of Gaussian Mixture Model, finding evidence for a deeper-rooted issue. Our results suggest both interlingual and intralingual instabilities, which indicate that current language models do not develop a consistent core personality. This can lead to unsafe behaviour of artificial intelligence systems that are based on these foundation models, and are increasingly integrated in human life. We subsequently discuss the shortcomings of modern psychometrics, abstract it, and provide a framework for its species-neutral, substrate-free formulation.

Title: Segment Using Just One Example

• Authors: Pratik Vora, Sudipan Saha
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.07393
• Pdf URL: https://arxiv.org/pdf/2408.07393
• Copy Paste: [[2408.07393]] Segment Using Just One Example(https://arxiv.org/abs/2408.07393)
• Keywords: segmentation
• Abstract: Semantic segmentation is an important topic in computer vision with many relevant application in Earth observation. While supervised methods exist, the constraints of limited annotated data has encouraged development of unsupervised approaches. However, existing unsupervised methods resemble clustering and cannot be directly mapped to explicit target classes. In this paper, we deal with single shot semantic segmentation, where one example for the target class is provided, which is used to segment the target class from query/test images. Our approach exploits recently popular Segment Anything (SAM), a promptable foundation model. We specifically design several techniques to automatically generate prompts from the only example/key image in such a way that the segmentation is successfully achieved on a stitch or concatenation of the example/key and query/test images. Proposed technique does not involve any training phase and just requires one example image to grasp the concept. Furthermore, no text-based prompt is required for the proposed method. We evaluated the proposed techniques on building and car classes.

Title: Sum-Product-Set Networks

• Authors: Milan Papež, Martin Rektoris, Tomáš Pevný, Václav Šmídl
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07394
• Pdf URL: https://arxiv.org/pdf/2408.07394
• Copy Paste: [[2408.07394]] Sum-Product-Set Networks(https://arxiv.org/abs/2408.07394)
• Keywords: generative
• Abstract: Daily internet communication relies heavily on tree-structured graphs, embodied by popular data formats such as XML and JSON. However, many recent generative (probabilistic) models utilize neural networks to learn a probability distribution over undirected cyclic graphs. This assumption of a generic graph structure brings various computational challenges, and, more importantly, the presence of non-linearities in neural networks does not permit tractable probabilistic inference. We address these problems by proposing sum-product-set networks, an extension of probabilistic circuits from unstructured tensor data to tree-structured graph data. To this end, we use random finite sets to reflect a variable number of nodes and edges in the graph and to allow for exact and efficient inference. We demonstrate that our tractable model performs comparably to various intractable models based on neural networks.

Title: A Quantum-Inspired Analysis of Human Disambiguation Processes

• Authors: Daphne Wang
• Subjects: cs.CL, cs.AI, cs.LO, quant-ph
• Abstract URL: https://arxiv.org/abs/2408.07402
• Pdf URL: https://arxiv.org/pdf/2408.07402
• Copy Paste: [[2408.07402]] A Quantum-Inspired Analysis of Human Disambiguation Processes(https://arxiv.org/abs/2408.07402)
• Keywords: large language model
• Abstract: Formal languages are essential for computer programming and are constructed to be easily processed by computers. In contrast, natural languages are much more challenging and instigated the field of Natural Language Processing (NLP). One major obstacle is the ubiquity of ambiguities. Recent advances in NLP have led to the development of large language models, which can resolve ambiguities with high accuracy. At the same time, quantum computers have gained much attention in recent years as they can solve some computational problems faster than classical computers. This new computing paradigm has reached the fields of machine learning and NLP, where hybrid classical-quantum learning algorithms have emerged. However, more research is needed to identify which NLP tasks could benefit from a genuine quantum advantage. In this thesis, we applied formalisms arising from foundational quantum mechanics, such as contextuality and causality, to study ambiguities arising from linguistics. By doing so, we also reproduced psycholinguistic results relating to the human disambiguation process. These results were subsequently used to predict human behaviour and outperformed current NLP methods.

Title: Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models

• Authors: Chenhui Hu, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07413
• Pdf URL: https://arxiv.org/pdf/2408.07413
• Copy Paste: [[2408.07413]] Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models(https://arxiv.org/abs/2408.07413)
• Keywords: large language model
• Abstract: Knowledge editing aims to update outdated or incorrect knowledge in large language models (LLMs). However, current knowledge editing methods have limited scalability for lifelong editing. This study explores the fundamental reason why knowledge editing fails in lifelong editing. We begin with the closed-form solution derived from linear associative memory, which underpins state-of-the-art knowledge editing methods. We extend the solution from single editing to lifelong editing, and through rigorous mathematical derivation, identify an interference term in the final solution, suggesting that editing knowledge may impact irrelevant knowledge. Further analysis of the interference term reveals a close relationship with superposition between knowledge representations. When knowledge superposition does not exist in language models, the interference term vanishes, allowing for lossless knowledge editing. Experiments across numerous language models reveal that knowledge superposition is universal, exhibiting high kurtosis, zero mean, and heavy-tailed distributions with clear scaling laws. Ultimately, by combining theory and experiments, we demonstrate that knowledge superposition is the fundamental reason for the failure of lifelong editing. Moreover, this is the first study to investigate knowledge editing from the perspective of superposition and provides a comprehensive observation of superposition across numerous real-world language models. Code available at this https URL.

Title: Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space

• Authors: Hyunjee Lee, Youngsik Yun, Jeongmin Bae, Seoha Kim, Youngjung Uh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07416
• Pdf URL: https://arxiv.org/pdf/2408.07416
• Copy Paste: [[2408.07416]] Rethinking Open-Vocabulary Segmentation of Radiance Fields in 3D Space(https://arxiv.org/abs/2408.07416)
• Keywords: segmentation
• Abstract: Understanding the 3D semantics of a scene is a fundamental problem for various scenarios such as embodied agents. While NeRFs and 3DGS excel at novel-view synthesis, previous methods for understanding their semantics have been limited to incomplete 3D understanding: their segmentation results are 2D masks and their supervision is anchored at 2D pixels. This paper revisits the problem set to pursue a better 3D understanding of a scene modeled by NeRFs and 3DGS as follows. 1) We directly supervise the 3D points to train the language embedding field. It achieves state-of-the-art accuracy without relying on multi-scale language embeddings. 2) We transfer the pre-trained language field to 3DGS, achieving the first real-time rendering speed without sacrificing training time or accuracy. 3) We introduce a 3D querying and evaluation protocol for assessing the reconstructed geometry and semantics together. Code, checkpoints, and annotations will be available online. Project page: this https URL

Title: Unsupervised Stereo Matching Network For VHR Remote Sensing Images Based On Error Prediction

• Authors: Liting Jiang, Yuming Xiang, Feng Wang, Hongjian You
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07419
• Pdf URL: https://arxiv.org/pdf/2408.07419
• Copy Paste: [[2408.07419]] Unsupervised Stereo Matching Network For VHR Remote Sensing Images Based On Error Prediction(https://arxiv.org/abs/2408.07419)
• Keywords: robust
• Abstract: Stereo matching in remote sensing has recently garnered increased attention, primarily focusing on supervised learning. However, datasets with ground truth generated by expensive airbone Lidar exhibit limited quantity and diversity, constraining the effectiveness of supervised networks. In contrast, unsupervised learning methods can leverage the increasing availability of very-high-resolution (VHR) remote sensing images, offering considerable potential in the realm of stereo matching. Motivated by this intuition, we propose a novel unsupervised stereo matching network for VHR remote sensing images. A light-weight module to bridge confidence with predicted error is introduced to refine the core model. Robust unsupervised losses are formulated to enhance network convergence. The experimental results on US3D and WHU-Stereo datasets demonstrate that the proposed network achieves superior accuracy compared to other unsupervised networks and exhibits better generalization capabilities than supervised models. Our code will be available at this https URL.

Title: LLMI3D: Empowering LLM with 3D Perception from a Single 2D Image

• Authors: Fan Yang, Sicheng Zhao, Yanhao Zhang, Haoxiang Chen, Hui Chen, Wenbo Tang, Haonan Lu, Pengfei Xu, Zhenyu Yang, Jungong Han, Guiguang Ding
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07422
• Pdf URL: https://arxiv.org/pdf/2408.07422
• Copy Paste: [[2408.07422]] LLMI3D: Empowering LLM with 3D Perception from a Single 2D Image(https://arxiv.org/abs/2408.07422)
• Keywords: extraction, generative, large language model
• Abstract: Recent advancements in autonomous driving, augmented reality, robotics, and embodied intelligence have necessitated 3D perception algorithms. However, current 3D perception methods, particularly small models, struggle with processing logical reasoning, question-answering, and handling open scenario categories. On the other hand, generative multimodal large language models (MLLMs) excel in general capacity but underperform in 3D tasks, due to weak spatial and local object perception, poor text-based geometric numerical output, and inability to handle camera focal variations. To address these challenges, we propose the following solutions: Spatial-Enhanced Local Feature Mining for better spatial feature extraction, 3D Query Token-Derived Info Decoding for precise geometric regression, and Geometry Projection-Based 3D Reasoning for handling camera focal length variations. We employ parameter-efficient fine-tuning for a pre-trained MLLM and develop LLMI3D, a powerful 3D perception MLLM. Additionally, we have constructed the IG3D dataset, which provides fine-grained descriptions and question-answer annotations. Extensive experiments demonstrate that our LLMI3D achieves state-of-the-art performance, significantly outperforming existing methods.

Title: UAHOI: Uncertainty-aware Robust Interaction Learning for HOI Detection

• Authors: Mu Chen, Minghan Chen, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07430
• Pdf URL: https://arxiv.org/pdf/2408.07430
• Copy Paste: [[2408.07430]] UAHOI: Uncertainty-aware Robust Interaction Learning for HOI Detection(https://arxiv.org/abs/2408.07430)
• Keywords: robust, transformer
• Abstract: This paper focuses on Human-Object Interaction (HOI) detection, addressing the challenge of identifying and understanding the interactions between humans and objects within a given image or video frame. Spearheaded by Detection Transformer (DETR), recent developments lead to significant improvements by replacing traditional region proposals by a set of learnable queries. However, despite the powerful representation capabilities provided by Transformers, existing Human-Object Interaction (HOI) detection methods still yield low confidence levels when dealing with complex interactions and are prone to overlooking interactive actions. To address these issues, we propose a novel approach \textsc{UAHOI}, Uncertainty-aware Robust Human-Object Interaction Learning that explicitly estimates prediction uncertainty during the training process to refine both detection and interaction predictions. Our model not only predicts the HOI triplets but also quantifies the uncertainty of these predictions. Specifically, we model this uncertainty through the variance of predictions and incorporate it into the optimization objective, allowing the model to adaptively adjust its confidence threshold based on prediction variance. This integration helps in mitigating the adverse effects of incorrect or ambiguous predictions that are common in traditional methods without any hand-designed components, serving as an automatic confidence threshold. Our method is flexible to existing HOI detection methods and demonstrates improved accuracy. We evaluate \textsc{UAHOI} on two standard benchmarks in the field: V-COCO and HICO-DET, which represent challenging scenarios for HOI detection. Through extensive experiments, we demonstrate that \textsc{UAHOI} achieves significant improvements over existing state-of-the-art methods, enhancing both the accuracy and robustness of HOI detection.

Title: MagicFace: Training-free Universal-Style Human Image Customized Synthesis

• Authors: Yibin Wang, Weizhong Zhang, Cheng Jin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07433
• Pdf URL: https://arxiv.org/pdf/2408.07433
• Copy Paste: [[2408.07433]] MagicFace: Training-free Universal-Style Human Image Customized Synthesis(https://arxiv.org/abs/2408.07433)
• Keywords: segmentation
• Abstract: Existing human image personalized generation methods often require tedious training: either fine-tuning with a few images or retraining on large-scale datasets. In such cases, these methods are prone to overfitting and encounter difficulties when personalizing individuals of diverse styles. Moreover, these training-based approaches also struggle with multi-concept human image customizing. To this end, we propose MagicFace, the first method for universal-style human image personalized synthesis that enables single/multi-concept customization for humans of any style in a training-free manner. MagicFace introduces a coarse-to-fine generation pipeline, involving two sequential stages: semantic scene construction and concept feature injection. This is achieved by our Reference-aware Self-Attention (RSA) and Region-grouped Blend Attention (RBA) mechanisms. Specifically, in the first stage, RSA enables the latent image to query features from reference concepts simultaneously, extracting the coarse-grained overall semantic understanding to facilitate the initial semantic layout establishment. In the second stage, we employ an attention-based semantic segmentation method to pinpoint the generated regions of all concepts in the latent image at each step. Following this, RBA divides the pixels of the latent image into semantic groups, with each group querying fine-grained features from its reference concept, which ensures precise attribute alignment and feature injection. Throughout the two-stage process, a weight mask strategy is employed to ensure the model focuses more on the reference concepts. Extensive experiments demonstrate our superiority in both human-centric subject-to-image synthesis and multi-concept human image customization. Our approach also can be applied to texture transformation, further enhancing its versatility and applicability.

Title: Achieving Data Efficient Neural Networks with Hybrid Concept-based Models

• Authors: Tobias A. Opsahl, Vegard Antun
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07438
• Pdf URL: https://arxiv.org/pdf/2408.07438
• Copy Paste: [[2408.07438]] Achieving Data Efficient Neural Networks with Hybrid Concept-based Models(https://arxiv.org/abs/2408.07438)
• Keywords: attack
• Abstract: Most datasets used for supervised machine learning consist of a single label per data point. However, in cases where more information than just the class label is available, would it be possible to train models more efficiently? We introduce two novel model architectures, which we call hybrid concept-based models, that train using both class labels and additional information in the dataset referred to as concepts. In order to thoroughly assess their performance, we introduce ConceptShapes, an open and flexible class of datasets with concept labels. We show that the hybrid concept-based models outperform standard computer vision models and previously proposed concept-based models with respect to accuracy, especially in sparse data settings. We also introduce an algorithm for performing adversarial concept attacks, where an image is perturbed in a way that does not change a concept-based model's concept predictions, but changes the class prediction. The existence of such adversarial examples raises questions about the interpretable qualities promised by concept-based models.

Title: BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learning

• Authors: Asif Hanif, Fahad Shamshad, Muhammad Awais, Muzammal Naseer, Fahad Shahbaz Khan, Karthik Nandakumar, Salman Khan, Rao Muhammad Anwer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07440
• Pdf URL: https://arxiv.org/pdf/2408.07440
• Copy Paste: [[2408.07440]] BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learning(https://arxiv.org/abs/2408.07440)
• Keywords: attack
• Abstract: Medical foundation models are gaining prominence in the medical community for their ability to derive general representations from extensive collections of medical image-text pairs. Recent research indicates that these models are susceptible to backdoor attacks, which allow them to classify clean images accurately but fail when specific triggers are introduced. However, traditional backdoor attacks necessitate a considerable amount of additional data to maliciously pre-train a model. This requirement is often impractical in medical imaging applications due to the usual scarcity of data. Inspired by the latest developments in learnable prompts, this work introduces a method to embed a backdoor into the medical foundation model during the prompt learning phase. By incorporating learnable prompts within the text encoder and introducing imperceptible learnable noise trigger to the input images, we exploit the full capabilities of the medical foundation models (Med-FM). Our method, BAPLe, requires only a minimal subset of data to adjust the noise trigger and the text prompts for downstream tasks, enabling the creation of an effective backdoor attack. Through extensive experiments with four medical foundation models, each pre-trained on different modalities and evaluated across six downstream datasets, we demonstrate the efficacy of our approach. BAPLe achieves a high backdoor success rate across all models and datasets, outperforming the baseline backdoor attack methods. Our work highlights the vulnerability of Med-FMs towards backdoor attacks and strives to promote the safe adoption of Med-FMs before their deployment in real-world applications. Code is available at this https URL.

Title: Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach

• Authors: Muhammad Saad Saeed, Shah Nawaz, Muhammad Zaigham Zaheer, Muhammad Haris Khan, Karthik Nandakumar, Muhammad Haroon Yousaf, Hassan Sajjad, Tom De Schepper, Markus Schedl
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07445
• Pdf URL: https://arxiv.org/pdf/2408.07445
• Copy Paste: [[2408.07445]] Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach(https://arxiv.org/abs/2408.07445)
• Keywords: robust
• Abstract: Multimodal networks have demonstrated remarkable performance improvements over their unimodal counterparts. Existing multimodal networks are designed in a multi-branch fashion that, due to the reliance on fusion strategies, exhibit deteriorated performance if one or more modalities are missing. In this work, we propose a modality invariant multimodal learning method, which is less susceptible to the impact of missing modalities. It consists of a single-branch network sharing weights across multiple modalities to learn inter-modality representations to maximize performance as well as robustness to missing modalities. Extensive experiments are performed on four challenging datasets including textual-visual (UPMC Food-101, Hateful Memes, Ferramenta) and audio-visual modalities (VoxCeleb1). Our proposed method achieves superior performance when all modalities are present as well as in the case of missing modalities during training or testing compared to the existing state-of-the-art methods.

Title: Infra-YOLO: Efficient Neural Network Structure with Model Compression for Real-Time Infrared Small Object Detection

• Authors: Zhonglin Chen, Anyu Geng, Jianan Jiang, Jiwu Lu, Di Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07455
• Pdf URL: https://arxiv.org/pdf/2408.07455
• Copy Paste: [[2408.07455]] Infra-YOLO: Efficient Neural Network Structure with Model Compression for Real-Time Infrared Small Object Detection(https://arxiv.org/abs/2408.07455)
• Keywords: extraction
• Abstract: Although convolutional neural networks have made outstanding achievements in visible light target detection, there are still many challenges in infrared small object detection because of the low signal-to-noise ratio, incomplete object structure, and a lack of reliable infrared small object dataset. To resolve limitations of the infrared small object dataset, a new dataset named InfraTiny was constructed, and more than 85% bounding box is less than 32x32 pixels (3218 images and a total of 20,893 bounding boxes). A multi-scale attention mechanism module (MSAM) and a Feature Fusion Augmentation Pyramid Module (FFAFPM) were proposed and deployed onto embedded devices. The MSAM enables the network to obtain scale perception information by acquiring different receptive fields, while the background noise information is suppressed to enhance feature extraction ability. The proposed FFAFPM can enrich semantic information, and enhance the fusion of shallow feature and deep feature, thus false positive results have been significantly reduced. By integrating the proposed methods into the YOLO model, which is named Infra-YOLO, infrared small object detection performance has been improved. Compared to yolov3, mAP@0.5 has been improved by 2.7%; and compared to yolov4, that by 2.5% on the InfraTiny dataset. The proposed Infra-YOLO was also transferred onto the embedded device in the unmanned aerial vehicle (UAV) for real application scenarios, where the channel pruning method is adopted to reduce FLOPs and to achieve a tradeoff between speed and accuracy. Even if the parameters of Infra-YOLO are reduced by 88% with the pruning method, a gain of 0.7% is still achieved on mAP@0.5 compared to yolov3, and a gain of 0.5% compared to yolov4. Experimental results show that the proposed MSAM and FFAFPM method can improve infrared small object detection performance compared with the previous benchmark method.

Title: A Survey on Immersive Cyber Situational Awareness Systems

• Authors: Hussain Ahmad, Faheem Ullah, Rehan Jafri
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.07456
• Pdf URL: https://arxiv.org/pdf/2408.07456
• Copy Paste: [[2408.07456]] A Survey on Immersive Cyber Situational Awareness Systems(https://arxiv.org/abs/2408.07456)
• Keywords: security
• Abstract: Cyber situational awareness systems are increasingly used for creating cyber common operating pictures for cybersecurity analysis and education. However, these systems face data occlusion and convolution issues due to the burgeoning complexity, dimensionality, and heterogeneity of cybersecurity data, which damages cyber Situational Awareness (SA) of end-users. Moreover, conventional ways of human-computer interactions, such as mouse and keyboard, increase the mental effort and cognitive load of cybersecurity practitioners, when analyzing cyber situations of large-scale infrastructures. Therefore, immersive technologies, such as virtual reality, augmented reality, and mixed reality, are employed in the cybersecurity realm to create intuitive, engaging, and interactive cyber common operating pictures. The Immersive Cyber Situational Awareness (ICSA) systems provide several unique visualization techniques and interaction features for the perception, comprehension, and projection of cyber SA. However, there has been no attempt to comprehensively investigate and classify the existing state of the art in the use of immersive technologies for cyber SA. Therefore, in this paper, we have gathered, analyzed, and synthesized the existing body of knowledge on ICSA systems. In particular, our survey has identified visualization and interaction techniques, evaluation mechanisms, and different levels of cyber SA (i.e., perception, comprehension, and projection) for ICSA systems. Consequently, our survey has enabled us to propose: (i) a reference framework for designing and analyzing ICSA systems by mapping immersive visualization and interaction techniques to the different levels of ICSA; (ii) future research directions for advancing the state-of-the-art on ICSA systems; and (iii) an in-depth analysis of the industrial implications of ICSA systems to enhance cybersecurity operations.

Title: From Brazilian Portuguese to European Portuguese

• Authors: João Sanches, Rui Ribeiro, Luísa Coheur
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07457
• Pdf URL: https://arxiv.org/pdf/2408.07457
• Copy Paste: [[2408.07457]] From Brazilian Portuguese to European Portuguese(https://arxiv.org/abs/2408.07457)
• Keywords: large language model
• Abstract: Brazilian Portuguese and European Portuguese are two varieties of the same language and, despite their close similarities, they exhibit several differences. However, there is a significant disproportion in the availability of resources between the two variants, with Brazilian Portuguese having more abundant resources. This inequity can impact the quality of translation services accessible to European Portuguese speakers. To address this issue, we propose the development of a Brazilian Portuguese to European Portuguese translation system, leveraging recent advancements in neural architectures and models. To evaluate the performance of such systems, we manually curated a gold test set comprising 500 sentences across five different topics. Each sentence in the gold test set has two distinct references, facilitating a straightforward evaluation of future translation models. We experimented with various models by fine-tuning existing Large Language Models using parallel data extracted from movie subtitles and TED Talks transcripts in both Brazilian and European Portuguese. Our evaluation involved the use of conventional automatic metrics as well as a human evaluation. In addition, all models were compared against ChatGPT 3.5 Turbo, which currently yields the best results.

Title: Large Language Models Prompting With Episodic Memory

• Authors: Dai Do, Quan Tran, Svetha Venkatesh, Hung Le
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07465
• Pdf URL: https://arxiv.org/pdf/2408.07465
• Copy Paste: [[2408.07465]] Large Language Models Prompting With Episodic Memory(https://arxiv.org/abs/2408.07465)
• Keywords: large language model
• Abstract: Prompt optimization is essential for enhancing the performance of Large Language Models (LLMs) in a range of Natural Language Processing (NLP) tasks, particularly in scenarios of few-shot learning where training examples are incorporated directly into the prompt. Despite the growing interest in optimizing prompts with few-shot examples, existing methods for prompt optimization are often resource-intensive or perform inadequately. In this work, we propose PrOmpting with Episodic Memory (POEM), a novel prompt optimization technique that is simple, efficient, and demonstrates strong generalization capabilities. We approach prompt optimization as a Reinforcement Learning (RL) challenge, using episodic memory to archive combinations of input data, permutations of few-shot examples, and the rewards observed during training. In the testing phase, we optimize the sequence of examples for each test query by selecting the sequence that yields the highest total rewards from the top-k most similar training examples in the episodic memory. Our results show that POEM outperforms recent techniques like TEMPERA and RLPrompt by over 5.3% in various text classification tasks. Furthermore, our approach adapts well to broader language understanding tasks, consistently outperforming conventional heuristic methods for ordering examples.

Title: Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization

• Authors: Yuxin Jiang, Bo Huang, Yufei Wang, Xingshan Zeng, Liangyou Li, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Wei Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07471
• Pdf URL: https://arxiv.org/pdf/2408.07471
• Copy Paste: [[2408.07471]] Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization(https://arxiv.org/abs/2408.07471)
• Keywords: large language model
• Abstract: Direct preference optimization (DPO), a widely adopted offline preference optimization algorithm, aims to align large language models (LLMs) with human-desired behaviors using pairwise preference data. However, the winning response and the losing response within pairwise data are generated isolatedly, leading to weak correlations between them as well as suboptimal alignment performance. To address this issue, we propose an effective framework named BMC, for bridging and modeling correlations in pairwise data. Firstly, we increase the consistency and informativeness of the pairwise preference signals by targeted modifications, synthesizing a pseudo winning response through improving the losing response based on the winning response. Secondly, we identify that DPO alone is insufficient to model these correlations and capture nuanced variations. Therefore, we propose learning token-level correlations by dynamically leveraging the policy model's confidence during training. Comprehensive experiments on QA, math, and instruction-following tasks demonstrate the effectiveness of our approach, significantly surpassing competitive baselines, including DPO. Additionally, our in-depth quantitative analysis reveals the reasons behind our method's superior performance over DPO and showcases its versatility to other DPO variants.

Title: One Step Diffusion-based Super-Resolution with Time-Aware Distillation

• Authors: Xiao He, Huaao Tang, Zhijun Tu, Junchao Zhang, Kun Cheng, Hanting Chen, Yong Guo, Mingrui Zhu, Nannan Wang, Xinbo Gao, Jie Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07476
• Pdf URL: https://arxiv.org/pdf/2408.07476
• Copy Paste: [[2408.07476]] One Step Diffusion-based Super-Resolution with Time-Aware Distillation(https://arxiv.org/abs/2408.07476)
• Keywords: diffusion
• Abstract: Diffusion-based image super-resolution (SR) methods have shown promise in reconstructing high-resolution images with fine details from low-resolution counterparts. However, these approaches typically require tens or even hundreds of iterative samplings, resulting in significant latency. Recently, techniques have been devised to enhance the sampling efficiency of diffusion-based SR models via knowledge distillation. Nonetheless, when aligning the knowledge of student and teacher models, these solutions either solely rely on pixel-level loss constraints or neglect the fact that diffusion models prioritize varying levels of information at different time steps. To accomplish effective and efficient image super-resolution, we propose a time-aware diffusion distillation method, named TAD-SR. Specifically, we introduce a novel score distillation strategy to align the data distribution between the outputs of the student and teacher models after minor noise perturbation. This distillation strategy enables the student network to concentrate more on the high-frequency details. Furthermore, to mitigate performance limitations stemming from distillation, we integrate a latent adversarial loss and devise a time-aware discriminator that leverages diffusion priors to effectively distinguish between real images and generated images. Extensive experiments conducted on synthetic and real-world datasets demonstrate that the proposed method achieves comparable or even superior performance compared to both previous state-of-the-art (SOTA) methods and the teacher model in just one sampling step. Codes are available at this https URL.

Title: DeCo: Decoupled Human-Centered Diffusion Video Editing with Motion Consistency

• Authors: Xiaojing Zhong, Xinyi Huang, Xiaofeng Yang, Guosheng Lin, Qingyao Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07481
• Pdf URL: https://arxiv.org/pdf/2408.07481
• Copy Paste: [[2408.07481]] DeCo: Decoupled Human-Centered Diffusion Video Editing with Motion Consistency(https://arxiv.org/abs/2408.07481)
• Keywords: diffusion
• Abstract: Diffusion models usher a new era of video editing, flexibly manipulating the video contents with text prompts. Despite the widespread application demand in editing human-centered videos, these models face significant challenges in handling complex objects like humans. In this paper, we introduce DeCo, a novel video editing framework specifically designed to treat humans and the background as separate editable targets, ensuring global spatial-temporal consistency by maintaining the coherence of each individual component. Specifically, we propose a decoupled dynamic human representation that utilizes a parametric human body prior to generate tailored humans while preserving the consistent motions as the original video. In addition, we consider the background as a layered atlas to apply text-guided image editing approaches on it. To further enhance the geometry and texture of humans during the optimization, we extend the calculation of score distillation sampling into normal space and image space. Moreover, we tackle inconsistent lighting between the edited targets by leveraging a lighting-aware video harmonizer, a problem previously overlooked in decompose-edit-combine approaches. Extensive qualitative and numerical experiments demonstrate that DeCo outperforms prior video editing methods in human-centered videos, especially in longer videos.

Title: GRFormer: Grouped Residual Self-Attention for Lightweight Single Image Super-Resolution

• Authors: Yuzhen Li, Zehang Deng, Yuxin Cao, Lihua Liu
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.07484
• Pdf URL: https://arxiv.org/pdf/2408.07484
• Copy Paste: [[2408.07484]] GRFormer: Grouped Residual Self-Attention for Lightweight Single Image Super-Resolution(https://arxiv.org/abs/2408.07484)
• Keywords: transformer
• Abstract: Previous works have shown that reducing parameter overhead and computations for transformer-based single image super-resolution (SISR) models (e.g., SwinIR) usually leads to a reduction of performance. In this paper, we present GRFormer, an efficient and lightweight method, which not only reduces the parameter overhead and computations, but also greatly improves performance. The core of GRFormer is Grouped Residual Self-Attention (GRSA), which is specifically oriented towards two fundamental components. Firstly, it introduces a novel grouped residual layer (GRL) to replace the Query, Key, Value (QKV) linear layer in self-attention, aimed at efficiently reducing parameter overhead, computations, and performance loss at the same time. Secondly, it integrates a compact Exponential-Space Relative Position Bias (ES-RPB) as a substitute for the original relative position bias to improve the ability to represent position information while further minimizing the parameter count. Extensive experimental results demonstrate that GRFormer outperforms state-of-the-art transformer-based methods for $\times$2, $\times$3 and $\times$4 SISR tasks, notably outperforming SOTA by a maximum PSNR of 0.23dB when trained on the DIV2K dataset, while reducing the number of parameter and MACs by about \textbf{60\%} and \textbf{49\% } in only self-attention module respectively. We hope that our simple and effective method that can easily applied to SR models based on window-division self-attention can serve as a useful tool for further research in image super-resolution. The code is available at \url{this https URL}.

Title: Large Language Models Know What Makes Exemplary Contexts

• Authors: Quanyu Long, Jianda Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07505
• Pdf URL: https://arxiv.org/pdf/2408.07505
• Copy Paste: [[2408.07505]] Large Language Models Know What Makes Exemplary Contexts(https://arxiv.org/abs/2408.07505)
• Keywords: large language model
• Abstract: In-context learning (ICL) has proven to be a significant capability with the advancement of Large Language models (LLMs). By instructing LLMs using few-shot demonstrative examples, ICL enables them to perform a wide range of tasks without needing to update millions of parameters. This paper presents a unified framework for LLMs that allows them to self-select influential in-context examples to compose their contexts; self-rank candidates with different demonstration compositions; self-optimize the demonstration selection and ordering through reinforcement learning. Specifically, our method designs a parameter-efficient retrieval head that generates the optimized demonstration after training with rewards from LLM's own preference. Experimental results validate the proposed method's effectiveness in enhancing ICL performance. Additionally, our approach effectively identifies and selects the most representative examples for the current task, and includes more diversity in retrieval.

Title: Protected Test-Time Adaptation via Online Entropy Matching: A Betting Approach

• Authors: Yarin Bar, Shalev Shaer, Yaniv Romano
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.07511
• Pdf URL: https://arxiv.org/pdf/2408.07511
• Copy Paste: [[2408.07511]] Protected Test-Time Adaptation via Online Entropy Matching: A Betting Approach(https://arxiv.org/abs/2408.07511)
• Keywords: protect
• Abstract: We present a novel approach for test-time adaptation via online self-training, consisting of two components. First, we introduce a statistical framework that detects distribution shifts in the classifier's entropy values obtained on a stream of unlabeled samples. Second, we devise an online adaptation mechanism that utilizes the evidence of distribution shifts captured by the detection tool to dynamically update the classifier's parameters. The resulting adaptation process drives the distribution of test entropy values obtained from the self-trained classifier to match those of the source domain, building invariance to distribution shifts. This approach departs from the conventional self-training method, which focuses on minimizing the classifier's entropy. Our approach combines concepts in betting martingales and online learning to form a detection tool capable of quickly reacting to distribution shifts. We then reveal a tight relation between our adaptation scheme and optimal transport, which forms the basis of our novel self-supervised loss. Experimental results demonstrate that our approach improves test-time accuracy under distribution shifts while maintaining accuracy and calibration in their absence, outperforming leading entropy minimization methods across various scenarios.

Title: CNN-JEPA: Self-Supervised Pretraining Convolutional Neural Networks Using Joint Embedding Predictive Architecture

• Authors: András Kalapos, Bálint Gyires-Tóth
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07514
• Pdf URL: https://arxiv.org/pdf/2408.07514
• Copy Paste: [[2408.07514]] CNN-JEPA: Self-Supervised Pretraining Convolutional Neural Networks Using Joint Embedding Predictive Architecture(https://arxiv.org/abs/2408.07514)
• Keywords: transformer
• Abstract: Self-supervised learning (SSL) has become an important approach in pretraining large neural networks, enabling unprecedented scaling of model and dataset sizes. While recent advances like I-JEPA have shown promising results for Vision Transformers, adapting such methods to Convolutional Neural Networks (CNNs) presents unique challenges. In this paper, we introduce CNN-JEPA, a novel SSL method that successfully applies the joint embedding predictive architecture approach to CNNs. Our method incorporates a sparse CNN encoder to handle masked inputs, a fully convolutional predictor using depthwise separable convolutions, and an improved masking strategy. We demonstrate that CNN-JEPA outperforms I-JEPA with ViT architectures on ImageNet-100, achieving 73.3% linear top-1 accuracy with a standard ResNet-50 encoder. Compared to other CNN-based SSL methods, CNN-JEPA requires 17-35% less training time for the same number of epochs and approaches the linear and k-NN top-1 accuracies of BYOL, SimCLR, and VICReg. Our approach offers a simpler, more efficient alternative to existing SSL methods for CNNs, requiring minimal augmentations and no separate projector network.

Title: DIffSteISR: Harnessing Diffusion Prior for Superior Real-world Stereo Image Super-Resolution

• Authors: Yuanbo Zhou, Xinlin Zhang, Wei Deng, Tao Wang, Tao Tan, Qinquan Gao, Tong Tong
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.07516
• Pdf URL: https://arxiv.org/pdf/2408.07516
• Copy Paste: [[2408.07516]] DIffSteISR: Harnessing Diffusion Prior for Superior Real-world Stereo Image Super-Resolution(https://arxiv.org/abs/2408.07516)
• Keywords: diffusion
• Abstract: We introduce DiffSteISR, a pioneering framework for reconstructing real-world stereo images. DiffSteISR utilizes the powerful prior knowledge embedded in pre-trained text-to-image model to efficiently recover the lost texture details in low-resolution stereo images. Specifically, DiffSteISR implements a time-aware stereo cross attention with temperature adapter (TASCATA) to guide the diffusion process, ensuring that the generated left and right views exhibit high texture consistency thereby reducing disparity error between the super-resolved images and the ground truth (GT) images. Additionally, a stereo omni attention control network (SOA ControlNet) is proposed to enhance the consistency of super-resolved images with GT images in the pixel, perceptual, and distribution space. Finally, DiffSteISR incorporates a stereo semantic extractor (SSE) to capture unique viewpoint soft semantic information and shared hard tag semantic information, thereby effectively improving the semantic accuracy and consistency of the generated left and right images. Extensive experimental results demonstrate that DiffSteISR accurately reconstructs natural and precise textures from low-resolution stereo images while maintaining a high consistency of semantic and texture between the left and right views.

Title: Cross-aware Early Fusion with Stage-divided Vision and Language Transformer Encoders for Referring Image Segmentation

• Authors: Yubin Cho, Hyunwoo Yu, Suk-ju Kang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07539
• Pdf URL: https://arxiv.org/pdf/2408.07539
• Copy Paste: [[2408.07539]] Cross-aware Early Fusion with Stage-divided Vision and Language Transformer Encoders for Referring Image Segmentation(https://arxiv.org/abs/2408.07539)
• Keywords: robust, transformer, segmentation
• Abstract: Referring segmentation aims to segment a target object related to a natural language expression. Key challenges of this task are understanding the meaning of complex and ambiguous language expressions and determining the relevant regions in the image with multiple objects by referring to the expression. Recent models have focused on the early fusion with the language features at the intermediate stage of the vision encoder, but these approaches have a limitation that the language features cannot refer to the visual information. To address this issue, this paper proposes a novel architecture, Cross-aware early fusion with stage-divided Vision and Language Transformer encoders (CrossVLT), which allows both language and vision encoders to perform the early fusion for improving the ability of the cross-modal context modeling. Unlike previous methods, our method enables the vision and language features to refer to each other's information at each stage to mutually enhance the robustness of both encoders. Furthermore, unlike the conventional scheme that relies solely on the high-level features for the cross-modal alignment, we introduce a feature-based alignment scheme that enables the low-level to high-level features of the vision and language encoders to engage in the cross-modal alignment. By aligning the intermediate cross-modal features in all encoder stages, this scheme leads to effective cross-modal fusion. In this way, the proposed approach is simple but effective for referring image segmentation, and it outperforms the previous state-of-the-art methods on three public benchmarks.

Title: DifuzCam: Replacing Camera Lens with a Mask and a Diffusion Model

• Authors: Erez Yosef, Raja Giryes
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.07541
• Pdf URL: https://arxiv.org/pdf/2408.07541
• Copy Paste: [[2408.07541]] DifuzCam: Replacing Camera Lens with a Mask and a Diffusion Model(https://arxiv.org/abs/2408.07541)
• Keywords: diffusion
• Abstract: The flat lensless camera design reduces the camera size and weight significantly. In this design, the camera lens is replaced by another optical element that interferes with the incoming light. The image is recovered from the raw sensor measurements using a reconstruction algorithm. Yet, the quality of the reconstructed images is not satisfactory. To mitigate this, we propose utilizing a pre-trained diffusion model with a control network and a learned separable transformation for reconstruction. This allows us to build a prototype flat camera with high-quality imaging, presenting state-of-the-art results in both terms of quality and perceptuality. We demonstrate its ability to leverage also textual descriptions of the captured scene to further enhance reconstruction. Our reconstruction method which leverages the strong capabilities of a pre-trained diffusion model can be used in other imaging systems for improved reconstruction results.

Title: MathScape: Evaluating MLLMs in multimodal Math Scenarios through a Hierarchical Benchmark

• Authors: Minxuan Zhou, Hao Liang, Tianpeng Li, Zhiyu Wu, Mingan Lin, Linzhuang Sun, Yaqi Zhou, Yan Zhang, Xiaoqin Huang, Yicong Chen, Yujing Qiao, Weipeng Chen, Bin Cui, Wentao Zhang, Zenan Zhou
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07543
• Pdf URL: https://arxiv.org/pdf/2408.07543
• Copy Paste: [[2408.07543]] MathScape: Evaluating MLLMs in multimodal Math Scenarios through a Hierarchical Benchmark(https://arxiv.org/abs/2408.07543)
• Keywords: large language model
• Abstract: With the development of Multimodal Large Language Models (MLLMs), the evaluation of multimodal models in the context of mathematical problems has become a valuable research field. Multimodal visual-textual mathematical reasoning serves as a critical indicator for evaluating the comprehension and complex multi-step quantitative reasoning abilities of MLLMs. However, previous multimodal math benchmarks have not sufficiently integrated visual and textual information. To address this gap, we proposed MathScape, a new benchmark that emphasizes the understanding and application of combined visual and textual information. MathScape is designed to evaluate photo-based math problem scenarios, assessing the theoretical understanding and application ability of MLLMs through a categorical hierarchical approach. We conduct a multi-dimensional evaluation on 11 advanced MLLMs, revealing that our benchmark is challenging even for the most sophisticated models. By analyzing the evaluation results, we identify the limitations of MLLMs, offering valuable insights for enhancing model performance.

Title: Sonic: Fast and Transferable Data Poisoning on Clustering Algorithms

• Authors: Francesco Villani, Dario Lazzaro, Antonio Emanuele Cinà, Matteo Dell'Amico, Battista Biggio, Fabio Roli
• Subjects: cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07558
• Pdf URL: https://arxiv.org/pdf/2408.07558
• Copy Paste: [[2408.07558]] Sonic: Fast and Transferable Data Poisoning on Clustering Algorithms(https://arxiv.org/abs/2408.07558)
• Keywords: attack, robust
• Abstract: Data poisoning attacks on clustering algorithms have received limited attention, with existing methods struggling to scale efficiently as dataset sizes and feature counts increase. These attacks typically require re-clustering the entire dataset multiple times to generate predictions and assess the attacker's objectives, significantly hindering their scalability. This paper addresses these limitations by proposing Sonic, a novel genetic data poisoning attack that leverages incremental and scalable clustering algorithms, e.g., FISHDBC, as surrogates to accelerate poisoning attacks against graph-based and density-based clustering methods, such as HDBSCAN. We empirically demonstrate the effectiveness and efficiency of Sonic in poisoning the target clustering algorithms. We then conduct a comprehensive analysis of the factors affecting the scalability and transferability of poisoning attacks against clustering algorithms, and we conclude by examining the robustness of hyperparameters in our attack strategy Sonic.

Title: Multi-task Heterogeneous Graph Learning on Electronic Health Records

• Authors: Tsai Hor Chan, Guosheng Yin, Kyongtae Bae, Lequan Yu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07569
• Pdf URL: https://arxiv.org/pdf/2408.07569
• Copy Paste: [[2408.07569]] Multi-task Heterogeneous Graph Learning on Electronic Health Records(https://arxiv.org/abs/2408.07569)
• Keywords: robust
• Abstract: Learning electronic health records (EHRs) has received emerging attention because of its capability to facilitate accurate medical diagnosis. Since the EHRs contain enriched information specifying complex interactions between entities, modeling EHRs with graphs is shown to be effective in practice. The EHRs, however, present a great degree of heterogeneity, sparsity, and complexity, which hamper the performance of most of the models applied to them. Moreover, existing approaches modeling EHRs often focus on learning the representations for a single task, overlooking the multi-task nature of EHR analysis problems and resulting in limited generalizability across different tasks. In view of these limitations, we propose a novel framework for EHR modeling, namely MulT-EHR (Multi-Task EHR), which leverages a heterogeneous graph to mine the complex relations and model the heterogeneity in the EHRs. To mitigate the large degree of noise, we introduce a denoising module based on the causal inference framework to adjust for severe confounding effects and reduce noise in the EHR data. Additionally, since our model adopts a single graph neural network for simultaneous multi-task prediction, we design a multi-task learning module to leverage the inter-task knowledge to regularize the training process. Extensive empirical studies on MIMIC-III and MIMIC-IV datasets validate that the proposed method consistently outperforms the state-of-the-art designs in four popular EHR analysis tasks -- drug recommendation, and predictions of the length of stay, mortality, and readmission. Thorough ablation studies demonstrate the robustness of our method upon variations to key components and hyperparameters.

Title: MetaSeg: MetaFormer-based Global Contexts-aware Network for Efficient Semantic Segmentation

• Authors: Beoungwoo Kang, Seunghun Moon, Yubin Cho, Hyunwoo Yu, Suk-Ju Kang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07576
• Pdf URL: https://arxiv.org/pdf/2408.07576
• Copy Paste: [[2408.07576]] MetaSeg: MetaFormer-based Global Contexts-aware Network for Efficient Semantic Segmentation(https://arxiv.org/abs/2408.07576)
• Keywords: extraction, transformer, segmentation
• Abstract: Beyond the Transformer, it is important to explore how to exploit the capacity of the MetaFormer, an architecture that is fundamental to the performance improvements of the Transformer. Previous studies have exploited it only for the backbone network. Unlike previous studies, we explore the capacity of the Metaformer architecture more extensively in the semantic segmentation task. We propose a powerful semantic segmentation network, MetaSeg, which leverages the Metaformer architecture from the backbone to the decoder. Our MetaSeg shows that the MetaFormer architecture plays a significant role in capturing the useful contexts for the decoder as well as for the backbone. In addition, recent segmentation methods have shown that using a CNN-based backbone for extracting the spatial information and a decoder for extracting the global information is more effective than using a transformer-based backbone with a CNN-based decoder. This motivates us to adopt the CNN-based backbone using the MetaFormer block and design our MetaFormer-based decoder, which consists of a novel self-attention module to capture the global contexts. To consider both the global contexts extraction and the computational efficiency of the self-attention for semantic segmentation, we propose a Channel Reduction Attention (CRA) module that reduces the channel dimension of the query and key into the one dimension. In this way, our proposed MetaSeg outperforms the previous state-of-the-art methods with more efficient computational costs on popular semantic segmentation and a medical image segmentation benchmark, including ADE20K, Cityscapes, COCO-stuff, and Synapse. The code is available at \url{this https URL}.

Title: TabularBench: Benchmarking Adversarial Robustness for Tabular Deep Learning in Real-world Use-cases

• Authors: Thibault Simonetto, Salah Ghamizi, Maxime Cordy
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07579
• Pdf URL: https://arxiv.org/pdf/2408.07579
• Copy Paste: [[2408.07579]] TabularBench: Benchmarking Adversarial Robustness for Tabular Deep Learning in Real-world Use-cases(https://arxiv.org/abs/2408.07579)
• Keywords: security, defense, attack, robust
• Abstract: While adversarial robustness in computer vision is a mature research field, fewer researchers have tackled the evasion attacks against tabular deep learning, and even fewer investigated robustification mechanisms and reliable defenses. We hypothesize that this lag in the research on tabular adversarial attacks is in part due to the lack of standardized benchmarks. To fill this gap, we propose TabularBench, the first comprehensive benchmark of robustness of tabular deep learning classification models. We evaluated adversarial robustness with CAA, an ensemble of gradient and search attacks which was recently demonstrated as the most effective attack against a tabular model. In addition to our open benchmark (this https URL) where we welcome submissions of new models and defenses, we implement 7 robustification mechanisms inspired by state-of-the-art defenses in computer vision and propose the largest benchmark of robust tabular deep learning over 200 models across five critical scenarios in finance, healthcare and security. We curated real datasets for each use case, augmented with hundreds of thousands of realistic synthetic inputs, and trained and assessed our models with and without data augmentations. We open-source our library that provides API access to all our pre-trained robust tabular models, and the largest datasets of real and synthetic tabular inputs. Finally, we analyze the impact of various defenses on the robustness and provide actionable insights to design new defenses and robustification mechanisms.

Title: Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey

• Authors: Hamza Kheddar
• Subjects: cs.CR, cs.AI, cs.CL, cs.CV, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.07583
• Pdf URL: https://arxiv.org/pdf/2408.07583
• Copy Paste: [[2408.07583]] Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Survey(https://arxiv.org/abs/2408.07583)
• Keywords: security, protect, attack, interpretability, transformer, large language model
• Abstract: With significant advancements in Transformers LLMs, NLP has extended its reach into many research fields due to its enhanced capabilities in text generation and user interaction. One field benefiting greatly from these advancements is cybersecurity. In cybersecurity, many parameters that need to be protected and exchanged between senders and receivers are in the form of text and tabular data, making NLP a valuable tool in enhancing the security measures of communication protocols. This survey paper provides a comprehensive analysis of the utilization of Transformers and LLMs in cyber-threat detection systems. The methodology of paper selection and bibliometric analysis is outlined to establish a rigorous framework for evaluating existing research. The fundamentals of Transformers are discussed, including background information on various cyber-attacks and datasets commonly used in this field. The survey explores the application of Transformers in IDSs, focusing on different architectures such as Attention-based models, LLMs like BERT and GPT, CNN/LSTM-Transformer hybrids, emerging approaches like ViTs, among others. Furthermore, it explores the diverse environments and applications where Transformers and LLMs-based IDS have been implemented, including computer networks, IoT devices, critical infrastructure protection, cloud computing, SDN, as well as in autonomous vehicles. The paper also addresses research challenges and future directions in this area, identifying key issues such as interpretability, scalability, and adaptability to evolving threats, and more. Finally, the conclusion summarizes the findings and highlights the significance of Transformers and LLMs in enhancing cyber-threat detection capabilities, while also outlining potential avenues for further research and development.

Title: FedQUIT: On-Device Federated Unlearning via a Quasi-Competent Virtual Teacher

• Authors: Alessio Mora, Lorenzo Valerio, Paolo Bellavista, Andrea Passarella
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.07587
• Pdf URL: https://arxiv.org/pdf/2408.07587
• Copy Paste: [[2408.07587]] FedQUIT: On-Device Federated Unlearning via a Quasi-Competent Virtual Teacher(https://arxiv.org/abs/2408.07587)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) promises better privacy guarantees for individuals' data when machine learning models are collaboratively trained. When an FL participant exercises its right to be forgotten, i.e., to detach from the FL framework it has participated and to remove its past contributions to the global model, the FL solution should perform all the necessary steps to make it possible without sacrificing the overall performance of the global model, which is not supported in state-of-the-art related solutions nowadays. In this paper, we propose FedQUIT, a novel algorithm that uses knowledge distillation to scrub the contribution of the forgetting data from an FL global model while preserving its generalization ability. FedQUIT directly works on clients' devices and does not require sharing additional information if compared with a regular FL process, nor does it assume the availability of publicly available proxy data. Our solution is efficient, effective, and applicable in both centralized and federated settings. Our experimental results show that, on average, FedQUIT requires less than 2.5% additional communication rounds to recover generalization performances after unlearning, obtaining a sanitized global model whose predictions are comparable to those of a global model that has never seen the data to be forgotten.

Title: WeKnow-RAG: An Adaptive Approach for Retrieval-Augmented Generation Integrating Web Search and Knowledge Graphs

• Authors: Weijian Xie, Xuefeng Liang, Yuhui Liu, Kaihua Ni, Hong Cheng, Zetian Hu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.07611
• Pdf URL: https://arxiv.org/pdf/2408.07611
• Copy Paste: [[2408.07611]] WeKnow-RAG: An Adaptive Approach for Retrieval-Augmented Generation Integrating Web Search and Knowledge Graphs(https://arxiv.org/abs/2408.07611)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have greatly contributed to the development of adaptive intelligent agents and are positioned as an important way to achieve Artificial General Intelligence (AGI). However, LLMs are prone to produce factually incorrect information and often produce "phantom" content that undermines their reliability, which poses a serious challenge for their deployment in real-world scenarios. Enhancing LLMs by combining external databases and information retrieval mechanisms is an effective path. To address the above challenges, we propose a new approach called WeKnow-RAG, which integrates Web search and Knowledge Graphs into a "Retrieval-Augmented Generation (RAG)" system. First, the accuracy and reliability of LLM responses are improved by combining the structured representation of Knowledge Graphs with the flexibility of dense vector retrieval. WeKnow-RAG then utilizes domain-specific knowledge graphs to satisfy a variety of queries and domains, thereby improving performance on factual information and complex reasoning tasks by employing multi-stage web page retrieval techniques using both sparse and dense retrieval methods. Our approach effectively balances the efficiency and accuracy of information retrieval, thus improving the overall retrieval process. Finally, we also integrate a self-assessment mechanism for the LLM to evaluate the trustworthiness of the answers it generates. Our approach proves its outstanding effectiveness in a wide range of offline experiments and online submissions.

Title: Practical Considerations for Differential Privacy

• Authors: Kareem Amin, Alex Kulesza, Sergei Vassilvitskii
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.07614
• Pdf URL: https://arxiv.org/pdf/2408.07614
• Copy Paste: [[2408.07614]] Practical Considerations for Differential Privacy(https://arxiv.org/abs/2408.07614)
• Keywords: privacy, protect, attack, robust
• Abstract: Differential privacy is the gold standard for statistical data release. Used by governments, companies, and academics, its mathematically rigorous guarantees and worst-case assumptions on the strength and knowledge of attackers make it a robust and compelling framework for reasoning about privacy. However, even with landmark successes, differential privacy has not achieved widespread adoption in everyday data use and data protection. In this work we examine some of the practical obstacles that stand in the way.

Title: Latent Anomaly Detection Through Density Matrices

• Authors: Joseph Gallego-Mejia, Oscar Bustos-Brinez, Fabio A. González
• Subjects: cs.LG, quant-ph, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.07623
• Pdf URL: https://arxiv.org/pdf/2408.07623
• Copy Paste: [[2408.07623]] Latent Anomaly Detection Through Density Matrices(https://arxiv.org/abs/2408.07623)
• Keywords: robust
• Abstract: This paper introduces a novel anomaly detection framework that combines the robust statistical principles of density-estimation-based anomaly detection methods with the representation-learning capabilities of deep learning models. The method originated from this framework is presented in two different versions: a shallow approach employing a density-estimation model based on adaptive Fourier features and density matrices, and a deep approach that integrates an autoencoder to learn a low-dimensional representation of the data. By estimating the density of new samples, both methods are able to find normality scores. The methods can be seamlessly integrated into an end-to-end architecture and optimized using gradient-based optimization techniques. To evaluate their performance, extensive experiments were conducted on various benchmark datasets. The results demonstrate that both versions of the method can achieve comparable or superior performance when compared to other state-of-the-art methods. Notably, the shallow approach performs better on datasets with fewer dimensions, while the autoencoder-based approach shows improved performance on datasets with higher dimensions.

Title: See It All: Contextualized Late Aggregation for 3D Dense Captioning

• Authors: Minjung Kim, Hyung Suk Lim, Seung Hwan Kim, Soonyoung Lee, Bumsoo Kim, Gunhee Kim
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07648
• Pdf URL: https://arxiv.org/pdf/2408.07648
• Copy Paste: [[2408.07648]] See It All: Contextualized Late Aggregation for 3D Dense Captioning(https://arxiv.org/abs/2408.07648)
• Keywords: transformer
• Abstract: 3D dense captioning is a task to localize objects in a 3D scene and generate descriptive sentences for each object. Recent approaches in 3D dense captioning have adopted transformer encoder-decoder frameworks from object detection to build an end-to-end pipeline without hand-crafted components. However, these approaches struggle with contradicting objectives where a single query attention has to simultaneously view both the tightly localized object regions and contextual environment. To overcome this challenge, we introduce SIA (See-It-All), a transformer pipeline that engages in 3D dense captioning with a novel paradigm called late aggregation. SIA simultaneously decodes two sets of queries-context query and instance query. The instance query focuses on localization and object attribute descriptions, while the context query versatilely captures the region-of-interest of relationships between multiple objects or with the global scene, then aggregated afterwards (i.e., late aggregation) via simple distance-based measures. To further enhance the quality of contextualized caption generation, we design a novel aggregator to generate a fully informed caption based on the surrounding context, the global environment, and object instances. Extensive experiments on two of the most widely-used 3D dense captioning datasets demonstrate that our proposed method achieves a significant improvement over prior methods.

Title: Graph Triple Attention Network: A Decoupled Perspective

• Authors: Xiaotang Wang, Yun Zhu, Haizhou Shi, Yongchao Liu, Chuntao Hong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07654
• Pdf URL: https://arxiv.org/pdf/2408.07654
• Copy Paste: [[2408.07654]] Graph Triple Attention Network: A Decoupled Perspective(https://arxiv.org/abs/2408.07654)
• Keywords: interpretability, transformer
• Abstract: Graph Transformers (GTs) have recently achieved significant success in the graph domain by effectively capturing both long-range dependencies and graph inductive biases. However, these methods face two primary challenges: (1) multi-view chaos, which results from coupling multi-view information (positional, structural, attribute), thereby impeding flexible usage and the interpretability of the propagation process. (2) local-global chaos, which arises from coupling local message passing with global attention, leading to issues of overfitting and over-globalizing. To address these challenges, we propose a high-level decoupled perspective of GTs, breaking them down into three components and two interaction levels: positional attention, structural attention, and attribute attention, alongside local and global interaction. Based on this decoupled perspective, we design a decoupled graph triple attention network named DeGTA, which separately computes multi-view attentions and adaptively integrates multi-view local and global information. This approach offers three key advantages: enhanced interpretability, flexible design, and adaptive integration of local and global information. Through extensive experiments, DeGTA achieves state-of-the-art performance across various datasets and tasks, including node classification and graph classification. Comprehensive ablation studies demonstrate that decoupling is essential for improving performance and enhancing interpretability. Our code is available at: this https URL

Title: Alignment-Enhanced Decoding:Defending via Token-Level Adaptive Refining of Probability Distributions

• Authors: Quan Liu, Zhenhong Zhou, Longzhu He, Yi Liu, Wei Zhang, Sen Su
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07663
• Pdf URL: https://arxiv.org/pdf/2408.07663
• Copy Paste: [[2408.07663]] Alignment-Enhanced Decoding:Defending via Token-Level Adaptive Refining of Probability Distributions(https://arxiv.org/abs/2408.07663)
• Keywords: defense, attack, large language model
• Abstract: Large language models are susceptible to jailbreak attacks, which can result in the generation of harmful content. While prior defenses mitigate these risks by perturbing or inspecting inputs, they ignore competing objectives, the underlying cause of alignment failures. In this paper, we propose Alignment-Enhanced Decoding (AED), a novel defense that employs adaptive decoding to address the root causes of jailbreak issues. We first define the Competitive Index to quantify alignment failures and utilize feedback from self-evaluation to compute post-alignment logits. Then, AED adaptively combines AED and post-alignment logits with the original logits to obtain harmless and helpful distributions. Consequently, our method enhances safety alignment while maintaining helpfulness. We conduct experiments across five models and four common jailbreaks, with the results validating the effectiveness of our approach. Code is available at this https URL.

Title: Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models

• Authors: Yi-Cheng Lin, Wei-Chih Chen, Hung-yi Lee
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.07665
• Pdf URL: https://arxiv.org/pdf/2408.07665
• Copy Paste: [[2408.07665]] Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models(https://arxiv.org/abs/2408.07665)
• Keywords: large language model
• Abstract: Warning: This paper may contain texts with uncomfortable content. Large Language Models (LLMs) have achieved remarkable performance in various tasks, including those involving multimodal data like speech. However, these models often exhibit biases due to the nature of their training data. Recently, more Speech Large Language Models (SLLMs) have emerged, underscoring the urgent need to address these biases. This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in SLLMs. By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. Our experiments reveal significant insights into their performance and bias levels. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies.

Title: Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities

• Authors: Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, Dacheng Tao
• Subjects: cs.LG, cs.AI, cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07666
• Pdf URL: https://arxiv.org/pdf/2408.07666
• Copy Paste: [[2408.07666]] Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities(https://arxiv.org/abs/2408.07666)
• Keywords: large language model
• Abstract: Model merging is an efficient empowerment technique in the machine learning community that does not require the collection of raw training data and does not require expensive computation. As model merging becomes increasingly prevalent across various fields, it is crucial to understand the available model merging techniques comprehensively. However, there is a significant gap in the literature regarding a systematic and thorough review of these techniques. This survey provides a comprehensive overview of model merging methods and theories, their applications in various domains and settings, and future research directions. Specifically, we first propose a new taxonomic approach that exhaustively discusses existing model merging methods. Secondly, we discuss the application of model merging techniques in large language models, multimodal large language models, and 10+ machine learning subfields, including continual learning, multi-task learning, few-shot learning, etc. Finally, we highlight the remaining challenges of model merging and discuss future research directions. A comprehensive list of papers about model merging is available at \url{this https URL}.

Title: G$^2$V$^2$former: Graph Guided Video Vision Transformer for Face Anti-Spoofing

• Authors: Jingyi Yang, Zitong Yu, Xiuming Ni, Jia He, Hui Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07675
• Pdf URL: https://arxiv.org/pdf/2408.07675
• Copy Paste: [[2408.07675]] G$^2$V$^2$former: Graph Guided Video Vision Transformer for Face Anti-Spoofing(https://arxiv.org/abs/2408.07675)
• Keywords: transformer
• Abstract: In videos containing spoofed faces, we may uncover the spoofing evidence based on either photometric or dynamic abnormality, even a combination of both. Prevailing face anti-spoofing (FAS) approaches generally concentrate on the single-frame scenario, however, purely photometric-driven methods overlook the dynamic spoofing clues that may be exposed over time. This may lead FAS systems to conclude incorrect judgments, especially in cases where it is easily distinguishable in terms of dynamics but challenging to discern in terms of photometrics. To this end, we propose the Graph Guided Video Vision Transformer (G$^2$V$^2$former), which combines faces with facial landmarks for photometric and dynamic feature fusion. We factorize the attention into space and time, and fuse them via a spatiotemporal block. Specifically, we design a novel temporal attention called Kronecker temporal attention, which has a wider receptive field, and is beneficial for capturing dynamic information. Moreover, we leverage the low-semantic motion of facial landmarks to guide the high-semantic change of facial expressions based on the motivation that regions containing landmarks may reveal more dynamic clues. Extensive experiments on nine benchmark datasets demonstrate that our method achieves superior performance under various scenarios. The codes will be released soon.

Title: A Spitting Image: Modular Superpixel Tokenization in Vision Transformers

• Authors: Marius Aasan, Odd Kolbjørnsen, Anne Schistad Solberg, Adín Ramirez Rivera
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07680
• Pdf URL: https://arxiv.org/pdf/2408.07680
• Copy Paste: [[2408.07680]] A Spitting Image: Modular Superpixel Tokenization in Vision Transformers(https://arxiv.org/abs/2408.07680)
• Keywords: extraction, transformer
• Abstract: Vision Transformer (ViT) architectures traditionally employ a grid-based approach to tokenization independent of the semantic content of an image. We propose a modular superpixel tokenization strategy which decouples tokenization and feature extraction; a shift from contemporary approaches where these are treated as an undifferentiated whole. Using on-line content-aware tokenization and scale- and shape-invariant positional embeddings, we perform experiments and ablations that contrast our approach with patch-based tokenization and randomized partitions as baselines. We show that our method significantly improves the faithfulness of attributions, gives pixel-level granularity on zero-shot unsupervised dense prediction tasks, while maintaining predictive performance in classification tasks. Our approach provides a modular tokenization framework commensurable with standard architectures, extending the space of ViTs to a larger class of semantically-rich models.

Title: Detecting Near-Duplicate Face Images

• Authors: Sudipta Banerjee, Arun Ross
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07689
• Pdf URL: https://arxiv.org/pdf/2408.07689
• Copy Paste: [[2408.07689]] Detecting Near-Duplicate Face Images(https://arxiv.org/abs/2408.07689)
• Keywords: robust, biometric, generative
• Abstract: Near-duplicate images are often generated when applying repeated photometric and geometric transformations that produce imperceptible variants of the original image. Consequently, a deluge of near-duplicates can be circulated online posing copyright infringement concerns. The concerns are more severe when biometric data is altered through such nuanced transformations. In this work, we address the challenge of near-duplicate detection in face images by, firstly, identifying the original image from a set of near-duplicates and, secondly, deducing the relationship between the original image and the near-duplicates. We construct a tree-like structure, called an Image Phylogeny Tree (IPT) using a graph-theoretic approach to estimate the relationship, i.e., determine the sequence in which they have been generated. We further extend our method to create an ensemble of IPTs known as Image Phylogeny Forests (IPFs). We rigorously evaluate our method to demonstrate robustness across other modalities, unseen transformations by latest generative models and IPT configurations, thereby significantly advancing the state-of-the-art performance by 42% on IPF reconstruction accuracy.

Title: End-to-end Semantic-centric Video-based Multimodal Affective Computing

• Authors: Ronghao Lin, Ying Zeng, Sijie Mai, Haifeng Hu
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.07694
• Pdf URL: https://arxiv.org/pdf/2408.07694
• Copy Paste: [[2408.07694]] End-to-end Semantic-centric Video-based Multimodal Affective Computing(https://arxiv.org/abs/2408.07694)
• Keywords: transformer
• Abstract: In the pathway toward Artificial General Intelligence (AGI), understanding human's affection is essential to enhance machine's cognition abilities. For achieving more sensual human-AI interaction, Multimodal Affective Computing (MAC) in human-spoken videos has attracted increasing attention. However, previous methods are mainly devoted to designing multimodal fusion algorithms, suffering from two issues: semantic imbalance caused by diverse pre-processing operations and semantic mismatch raised by inconsistent affection content contained in different modalities comparing with the multimodal ground truth. Besides, the usage of manual features extractors make they fail in building end-to-end pipeline for multiple MAC downstream tasks. To address above challenges, we propose a novel end-to-end framework named SemanticMAC to compute multimodal semantic-centric affection for human-spoken videos. We firstly employ pre-trained Transformer model in multimodal data pre-processing and design Affective Perceiver module to capture unimodal affective information. Moreover, we present a semantic-centric approach to unify multimodal representation learning in three ways, including gated feature interaction, multi-task pseudo label generation, and intra-/inter-sample contrastive learning. Finally, SemanticMAC effectively learn specific- and shared-semantic representations in the guidance of semantic-centric labels. Extensive experimental results demonstrate that our approach surpass the state-of-the-art methods on 7 public datasets in four MAC downstream tasks.

Title: The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models

• Authors: Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, Amine Mhedhbi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07702
• Pdf URL: https://arxiv.org/pdf/2408.07702
• Copy Paste: [[2408.07702]] The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models(https://arxiv.org/abs/2408.07702)
• Keywords: large language model
• Abstract: Schema linking is a crucial step in Text-to-SQL pipelines, which translate natural language queries into SQL. The goal of schema linking is to retrieve relevant tables and columns (signal) while disregarding irrelevant ones (noise). However, imperfect schema linking can often exclude essential columns needed for accurate query generation. In this work, we revisit the need for schema linking when using the latest generation of large language models (LLMs). We find empirically that newer models are adept at identifying relevant schema elements during generation, without the need for explicit schema linking. This allows Text-to-SQL pipelines to bypass schema linking entirely and instead pass the full database schema to the LLM, eliminating the risk of excluding necessary information. Furthermore, as alternatives to schema linking, we propose techniques that improve Text-to-SQL accuracy without compromising on essential schema information. Our approach achieves 71.83\% execution accuracy on the BIRD benchmark, ranking first at the time of submission.