2024-07-25

Title: Towards an Improved Taxonomy of Attacks related to Digital Identities and Identity Management Systems

• Authors: Daniela Pöhn, Wolfgang Hommel
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.16718
• Pdf URL: https://arxiv.org/pdf/2407.16718
• Copy Paste: [[2407.16718]] Towards an Improved Taxonomy of Attacks related to Digital Identities and Identity Management Systems(https://arxiv.org/abs/2407.16718)
• Keywords: secure, security, attack
• Abstract: Digital transformation with the adoption of cloud technologies, outsourcing, and working-from-home possibilities permits flexibility for organizations and persons. At the same time, it makes it more difficult to secure the IT infrastructure as the IT team needs to keep track of who is accessing what data from where and when on which device. With these changes, identity management as a key element of security becomes more important. Identity management relates to the technologies and policies for the identification, authentication, and authorization of users (humans, devices) in computer networks. Due to the diversity of identity management (i.e., models, protocols, and implementations), different requirements, problems, and attack vectors need to be taken into account. In order to secure identity management systems with their identities, a systematic approach is required. In this article, we propose the improved framework Taxonomy for Identity Management related to Attacks (TaxIdMA). The purpose of TaxIdMA is to classify existing attacks, attack vectors, and vulnerabilities associated with system identities, identity management systems, and end-user identities. In addition, the background of these attacks can be described in a structured and systematic way. The taxonomy is applied to the Internet of Things and self-sovereign identities. It is enhanced by a description language for threat intelligence sharing. Last but not least, TaxIdMA is evaluated and improved based on expert interviews, statistics, and discussions. This step enables broader applicability and level of detail at the same time. The combination of TaxIdMA, which allows a structured way to outline attacks and is applicable to different scenarios, and a description language for threat intelligence help to improve the security identity management systems and processes.

Title: Educating LLMs like Human Students: Structure-aware Injection of Domain Knowledge

• Authors: Kai Liu, Ze Chen, Zhihang Fu, Rongxin Jiang, Fan Zhou, Yaowu Chen, Yue Wu, Jieping Ye
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16724
• Pdf URL: https://arxiv.org/pdf/2407.16724
• Copy Paste: [[2407.16724]] Educating LLMs like Human Students: Structure-aware Injection of Domain Knowledge(https://arxiv.org/abs/2407.16724)
• Keywords: large language model
• Abstract: This paper presents a pioneering methodology, termed StructTuning, to efficiently transform foundation Large Language Models (LLMs) into domain specialists. It significantly minimizes the training corpus requirement to a mere 0.3% while achieving an impressive 50% of traditional knowledge injection performance. Our method is inspired by the educational processes for human students, particularly how structured domain knowledge from textbooks is absorbed and then applied to tackle real-world challenges through specific exercises. Based on this, we propose a novel two-stage knowledge injection strategy: Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT). In the SCPT phase, we organize the training data into an auto-generated taxonomy of domain knowledge, enabling LLMs to effectively memorize textual segments linked to specific expertise within the taxonomy's architecture. Subsequently, in the SSFT phase, we explicitly prompt models to reveal the underlying knowledge structure in their outputs, leveraging this structured domain insight to address practical problems adeptly. Our ultimate method has undergone extensive evaluations across model architectures and scales, using closed-book question-answering tasks on LongBench and MMedBench datasets. Remarkably, our method matches 50% of the improvement displayed by the state-of-the-art MMedLM2 on MMedBench, but with only 0.3% quantity of the training corpus. This breakthrough showcases the potential to scale up our StructTuning for stronger domain-specific LLMs. Code will be made public soon.

Title: Category-Extensible Out-of-Distribution Detection via Hierarchical Context Descriptions

• Authors: Kai Liu, Zhihang Fu, Chao Chen, Sheng Jin, Ze Chen, Mingyuan Tao, Rongxin Jiang, Jieping Ye
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16725
• Pdf URL: https://arxiv.org/pdf/2407.16725
• Copy Paste: [[2407.16725]] Category-Extensible Out-of-Distribution Detection via Hierarchical Context Descriptions(https://arxiv.org/abs/2407.16725)
• Keywords: robust, large language model
• Abstract: The key to OOD detection has two aspects: generalized feature representation and precise category description. Recently, vision-language models such as CLIP provide significant advances in both two issues, but constructing precise category descriptions is still in its infancy due to the absence of unseen categories. This work introduces two hierarchical contexts, namely perceptual context and spurious context, to carefully describe the precise category boundary through automatic prompt tuning. Specifically, perceptual contexts perceive the inter-category difference (e.g., cats vs apples) for current classification tasks, while spurious contexts further identify spurious (similar but exactly not) OOD samples for every single category (e.g., cats vs panthers, apples vs peaches). The two contexts hierarchically construct the precise description for a certain category, which is, first roughly classifying a sample to the predicted category and then delicately identifying whether it is truly an ID sample or actually OOD. Moreover, the precise descriptions for those categories within the vision-language framework present a novel application: CATegory-EXtensible OOD detection (CATEX). One can efficiently extend the set of recognizable categories by simply merging the hierarchical contexts learned under different sub-task settings. And extensive experiments are conducted to demonstrate CATEX's effectiveness, robustness, and category-extensibility. For instance, CATEX consistently surpasses the rivals by a large margin with several protocols on the challenging ImageNet-1K dataset. In addition, we offer new insights on how to efficiently scale up the prompt engineering in vision-language models to recognize thousands of object categories, as well as how to incorporate large language models (like GPT-3) to boost zero-shot applications. Code will be made public soon.

Title: A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms

• Authors: Ari Blau, Evan S Schaffer, Neeli Mishra, Nathaniel J Miska, The International Brain Laboratory, Liam Paninski, Matthew R Whiteway
• Subjects: cs.CV, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2407.16727
• Pdf URL: https://arxiv.org/pdf/2407.16727
• Copy Paste: [[2407.16727]] A study of animal action segmentation algorithms across supervised, unsupervised, and semi-supervised learning paradigms(https://arxiv.org/abs/2407.16727)
• Keywords: segmentation
• Abstract: Action segmentation of behavioral videos is the process of labeling each frame as belonging to one or more discrete classes, and is a crucial component of many studies that investigate animal behavior. A wide range of algorithms exist to automatically parse discrete animal behavior, encompassing supervised, unsupervised, and semi-supervised learning paradigms. These algorithms -- which include tree-based models, deep neural networks, and graphical models -- differ widely in their structure and assumptions on the data. Using four datasets spanning multiple species -- fly, mouse, and human -- we systematically study how the outputs of these various algorithms align with manually annotated behaviors of interest. Along the way, we introduce a semi-supervised action segmentation model that bridges the gap between supervised deep neural networks and unsupervised graphical models. We find that fully supervised temporal convolutional networks with the addition of temporal information in the observations perform the best on our supervised metrics across all datasets.

Title: PateGail: A Privacy-Preserving Mobility Trajectory Generator with Imitation Learning

• Authors: Huandong Wang, Changzheng Gao, Yuchen Wu, Depeng Jin, Lina Yao, Yong Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16729
• Pdf URL: https://arxiv.org/pdf/2407.16729
• Copy Paste: [[2407.16729]] PateGail: A Privacy-Preserving Mobility Trajectory Generator with Imitation Learning(https://arxiv.org/abs/2407.16729)
• Keywords: privacy, protect, generative
• Abstract: Generating human mobility trajectories is of great importance to solve the lack of large-scale trajectory data in numerous applications, which is caused by privacy concerns. However, existing mobility trajectory generation methods still require real-world human trajectories centrally collected as the training data, where there exists an inescapable risk of privacy leakage. To overcome this limitation, in this paper, we propose PateGail, a privacy-preserving imitation learning model to generate mobility trajectories, which utilizes the powerful generative adversary imitation learning model to simulate the decision-making process of humans. Further, in order to protect user privacy, we train this model collectively based on decentralized mobility data stored in user devices, where personal discriminators are trained locally to distinguish and reward the real and generated human trajectories. In the training process, only the generated trajectories and their rewards obtained based on personal discriminators are shared between the server and devices, whose privacy is further preserved by our proposed perturbation mechanisms with theoretical proof to satisfy differential privacy. Further, to better model the human decision-making process, we propose a novel aggregation mechanism of the rewards obtained from personal discriminators. We theoretically prove that under the reward obtained based on the aggregation mechanism, our proposed model maximizes the lower bound of the discounted total rewards of users. Extensive experiments show that the trajectories generated by our model are able to resemble real-world trajectories in terms of five key statistical metrics, outperforming state-of-the-art algorithms by over 48.03%. Furthermore, we demonstrate that the synthetic trajectories are able to efficiently support practical applications, including mobility prediction and location recommendation.

Title: Theoretical Analysis of Privacy Leakage in Trustworthy Federated Learning: A Perspective from Linear Algebra and Optimization Theory

• Authors: Xiaojin Zhang, Wei Chen
• Subjects: cs.CR, cs.AI, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.16735
• Pdf URL: https://arxiv.org/pdf/2407.16735
• Copy Paste: [[2407.16735]] Theoretical Analysis of Privacy Leakage in Trustworthy Federated Learning: A Perspective from Linear Algebra and Optimization Theory(https://arxiv.org/abs/2407.16735)
• Keywords: privacy, attack, federate
• Abstract: Federated learning has emerged as a promising paradigm for collaborative model training while preserving data privacy. However, recent studies have shown that it is vulnerable to various privacy attacks, such as data reconstruction attacks. In this paper, we provide a theoretical analysis of privacy leakage in federated learning from two perspectives: linear algebra and optimization theory. From the linear algebra perspective, we prove that when the Jacobian matrix of the batch data is not full rank, there exist different batches of data that produce the same model update, thereby ensuring a level of privacy. We derive a sufficient condition on the batch size to prevent data reconstruction attacks. From the optimization theory perspective, we establish an upper bound on the privacy leakage in terms of the batch size, the distortion extent, and several other factors. Our analysis provides insights into the relationship between privacy leakage and various aspects of federated learning, offering a theoretical foundation for designing privacy-preserving federated learning algorithms.

Title: VisMin: Visual Minimal-Change Understanding

• Authors: Rabiul Awal, Saba Ahmadi, Le Zhang, Aishwarya Agrawal
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16772
• Pdf URL: https://arxiv.org/pdf/2407.16772
• Copy Paste: [[2407.16772]] VisMin: Visual Minimal-Change Understanding(https://arxiv.org/abs/2407.16772)
• Keywords: diffusion, large language model
• Abstract: Fine-grained understanding of objects, attributes, and relationships between objects is crucial for visual-language models (VLMs). Existing benchmarks primarily focus on evaluating VLMs' capability to distinguish between two very similar \textit{captions} given an image. In this paper, we introduce a new, challenging benchmark termed \textbf{Vis}ual \textbf{Min}imal-Change Understanding (VisMin), which requires models to predict the correct image-caption match given two images and two captions. The image pair and caption pair contain minimal changes, i.e., only one aspect changes at a time from among the following: \textit{object}, \textit{attribute}, \textit{count}, and \textit{spatial relation}. These changes test the models' understanding of objects, attributes (such as color, material, shape), counts, and spatial relationships between objects. We built an automatic framework using large language models and diffusion models, followed by a rigorous 4-step verification process by human annotators. Empirical experiments reveal that current VLMs exhibit notable deficiencies in understanding spatial relationships and counting abilities. We also generate a large-scale training dataset to finetune CLIP and Idefics2, showing significant improvements in fine-grained understanding across benchmarks and in CLIP's general image-text alignment. We release all resources, including the benchmark, training data, and finetuned model checkpoints, at \url{this https URL}.

Title: Occlusion-Aware 3D Motion Interpretation for Abnormal Behavior Detection

• Authors: Su Li, Wang Liang, Jianye Wang, Ziheng Zhang, Lei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16788
• Pdf URL: https://arxiv.org/pdf/2407.16788
• Copy Paste: [[2407.16788]] Occlusion-Aware 3D Motion Interpretation for Abnormal Behavior Detection(https://arxiv.org/abs/2407.16788)
• Keywords: robust, extraction
• Abstract: Estimating abnormal posture based on 3D pose is vital in human pose analysis, yet it presents challenges, especially when reconstructing 3D human poses from monocular datasets with occlusions. Accurate reconstructions enable the restoration of 3D movements, which assist in the extraction of semantic details necessary for analyzing abnormal behaviors. However, most existing methods depend on predefined key points as a basis for estimating the coordinates of occluded joints, where variations in data quality have adversely affected the performance of these models. In this paper, we present OAD2D, which discriminates against motion abnormalities based on reconstructing 3D coordinates of mesh vertices and human joints from monocular videos. The OAD2D employs optical flow to capture motion prior information in video streams, enriching the information on occluded human movements and ensuring temporal-spatial alignment of poses. Moreover, we reformulate the abnormal posture estimation by coupling it with Motion to Text (M2T) model in which, the VQVAE is employed to quantize motion features. This approach maps motion tokens to text tokens, allowing for a semantically interpretable analysis of motion, and enhancing the generalization of abnormal posture detection boosted by Language model. Our approach demonstrates the robustness of abnormal behavior detection against severe and self-occlusions, as it reconstructs human motion trajectories in global coordinates to effectively mitigate occlusion issues. Our method, validated using the Human3.6M, 3DPW, and NTU RGB+D datasets, achieves a high $F_1-$Score of 0.94 on the NTU RGB+D dataset for medical condition detection. And we will release all of our code and data.

Title: Wasserstein Distributionally Robust Shallow Convex Neural Networks

• Authors: Julien Pallage, Antoine Lesage-Landry
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2407.16800
• Pdf URL: https://arxiv.org/pdf/2407.16800
• Copy Paste: [[2407.16800]] Wasserstein Distributionally Robust Shallow Convex Neural Networks(https://arxiv.org/abs/2407.16800)
• Keywords: robust
• Abstract: In this work, we propose Wasserstein distributionally robust shallow convex neural networks (WaDiRo-SCNNs) to provide reliable nonlinear predictions when subject to adverse and corrupted datasets. Our approach is based on a new convex training program for ReLU shallow neural networks which allows us to cast the problem as an exact, tractable reformulation of its order-1 Wasserstein distributionally robust equivalent. Our training procedure is conservative by design, has low stochasticity, is solvable with open-source solvers, and is scalable to large industrial deployments. We provide out-of-sample performance guarantees and show that hard convex physical constraints can be enforced in the training program. WaDiRo-SCNN aims to make neural networks safer for critical applications, such as in the energy sector. Finally, we numerically demonstrate the performance of our model on a synthetic experiment and a real-world power system application, i.e., the prediction of non-residential buildings' hourly energy consumption. The experimental results are convincing and showcase the strengths of the proposed model.

Title: Distribution-Aware Robust Learning from Long-Tailed Data with Noisy Labels

• Authors: Jae Soon Baik, In Young Yoon, Kun Hoon Kim, Jun Won Choi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16802
• Pdf URL: https://arxiv.org/pdf/2407.16802
• Copy Paste: [[2407.16802]] Distribution-Aware Robust Learning from Long-Tailed Data with Noisy Labels(https://arxiv.org/abs/2407.16802)
• Keywords: robust
• Abstract: Deep neural networks have demonstrated remarkable advancements in various fields using large, well-annotated datasets. However, real-world data often exhibit long-tailed distributions and label noise, significantly degrading generalization performance. Recent studies addressing these issues have focused on noisy sample selection methods that estimate the centroid of each class based on high-confidence samples within each target class. The performance of these methods is limited because they use only the training samples within each class for class centroid estimation, making the quality of centroids susceptible to long-tailed distributions and noisy labels. In this study, we present a robust training framework called Distribution-aware Sample Selection and Contrastive Learning (DaSC). Specifically, DaSC introduces a Distribution-aware Class Centroid Estimation (DaCC) to generate enhanced class centroids. DaCC performs weighted averaging of the features from all samples, with weights determined based on model predictions. Additionally, we propose a confidence-aware contrastive learning strategy to obtain balanced and robust representations. The training samples are categorized into high-confidence and low-confidence samples. Our method then applies Semi-supervised Balanced Contrastive Loss (SBCL) using high-confidence samples, leveraging reliable label information to mitigate class bias. For the low-confidence samples, our method computes Mixup-enhanced Instance Discrimination Loss (MIDL) to improve their representations in a self-supervised manner. Our experimental results on CIFAR and real-world noisy-label datasets demonstrate the superior performance of the proposed DaSC compared to previous approaches.

Title: Fusion and Cross-Modal Transfer for Zero-Shot Human Action Recognition

• Authors: Abhi Kamboj, Anh Duy Nguyen, Minh Do
• Subjects: cs.CV, cs.AI, cs.HC, cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2407.16803
• Pdf URL: https://arxiv.org/pdf/2407.16803
• Copy Paste: [[2407.16803]] Fusion and Cross-Modal Transfer for Zero-Shot Human Action Recognition(https://arxiv.org/abs/2407.16803)
• Keywords: interpretability
• Abstract: Despite living in a multi-sensory world, most AI models are limited to textual and visual interpretations of human motion and behavior. Inertial measurement units (IMUs) provide a salient signal to understand human motion; however, they are challenging to use due to their uninterpretability and scarcity of their data. We investigate a method to transfer knowledge between visual and inertial modalities using the structure of an informative joint representation space designed for human action recognition (HAR). We apply the resulting Fusion and Cross-modal Transfer (FACT) method to a novel setup, where the model does not have access to labeled IMU data during training and is able to perform HAR with only IMU data during testing. Extensive experiments on a wide range of RGB-IMU datasets demonstrate that FACT significantly outperforms existing methods in zero-shot cross-modal transfer.

Title: In Search for Architectures and Loss Functions in Multi-Objective Reinforcement Learning

• Authors: Mikhail Terekhov, Caglar Gulcehre
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16807
• Pdf URL: https://arxiv.org/pdf/2407.16807
• Copy Paste: [[2407.16807]] In Search for Architectures and Loss Functions in Multi-Objective Reinforcement Learning(https://arxiv.org/abs/2407.16807)
• Keywords: robust
• Abstract: Multi-objective reinforcement learning (MORL) is essential for addressing the intricacies of real-world RL problems, which often require trade-offs between multiple utility functions. However, MORL is challenging due to unstable learning dynamics with deep learning-based function approximators. The research path most taken has been to explore different value-based loss functions for MORL to overcome this issue. Our work empirically explores model-free policy learning loss functions and the impact of different architectural choices. We introduce two different approaches: Multi-objective Proximal Policy Optimization (MOPPO), which extends PPO to MORL, and Multi-objective Advantage Actor Critic (MOA2C), which acts as a simple baseline in our ablations. Our proposed approach is straightforward to implement, requiring only small modifications at the level of function approximator. We conduct comprehensive evaluations on the MORL Deep Sea Treasure, Minecart, and Reacher environments and show that MOPPO effectively captures the Pareto front. Our extensive ablation studies and empirical analyses reveal the impact of different architectural choices, underscoring the robustness and versatility of MOPPO compared to popular MORL approaches like Pareto Conditioned Networks (PCN) and Envelope Q-learning in terms of MORL metrics, including hypervolume and expected utility.

Title: SINDER: Repairing the Singular Defects of DINOv2

• Authors: Haoqi Wang, Tong Zhang, Mathieu Salzmann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16826
• Pdf URL: https://arxiv.org/pdf/2407.16826
• Copy Paste: [[2407.16826]] SINDER: Repairing the Singular Defects of DINOv2(https://arxiv.org/abs/2407.16826)
• Keywords: transformer, segmentation
• Abstract: Vision Transformer models trained on large-scale datasets, although effective, often exhibit artifacts in the patch token they extract. While such defects can be alleviated by re-training the entire model with additional classification tokens, the underlying reasons for the presence of these tokens remain unclear. In this paper, we conduct a thorough investigation of this phenomenon, combining theoretical analysis with empirical observations. Our findings reveal that these artifacts originate from the pre-trained network itself, specifically stemming from the leading left singular vector of the network's weights. Furthermore, to mitigate these defects, we propose a novel fine-tuning smooth regularization that rectifies structural deficiencies using only a small dataset, thereby avoiding the need for complete re-training. We validate our method on various downstream tasks, including unsupervised segmentation, classification, supervised segmentation, and depth estimation, demonstrating its effectiveness in improving model performance. Codes and checkpoints are available at this https URL.

Title: Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

• Authors: Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16833
• Pdf URL: https://arxiv.org/pdf/2407.16833
• Copy Paste: [[2407.16833]] Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach(https://arxiv.org/abs/2407.16833)
• Keywords: large language model
• Abstract: Retrieval Augmented Generation (RAG) has been a powerful tool for Large Language Models (LLMs) to efficiently process overly lengthy contexts. However, recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. We conduct a comprehensive comparison between RAG and long-context (LC) LLMs, aiming to leverage the strengths of both. We benchmark RAG and LC across various public datasets using three latest LLMs. Results reveal that when resourced sufficiently, LC consistently outperforms RAG in terms of average performance. However, RAG's significantly lower cost remains a distinct advantage. Based on this observation, we propose Self-Route, a simple yet effective method that routes queries to RAG or LC based on model self-reflection. Self-Route significantly reduces the computation cost while maintaining a comparable performance to LC. Our findings provide a guideline for long-context applications of LLMs using RAG and LC.

Title: A Multi-Level Hierarchical Framework for the Classification of Weather Conditions and Hazard Prediction

• Authors: Harish Neelam
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16834
• Pdf URL: https://arxiv.org/pdf/2407.16834
• Copy Paste: [[2407.16834]] A Multi-Level Hierarchical Framework for the Classification of Weather Conditions and Hazard Prediction(https://arxiv.org/abs/2407.16834)
• Keywords: robust
• Abstract: This paper presents a multilevel hierarchical framework for the classification of weather conditions and hazard prediction. In recent years, the importance of data has grown significantly, with various types like text, numbers, images, audio, and videos playing a key role. Among these, images make up a large portion of the data available. This application shows promise for various purposes, especially when combined with decision support systems for traffic management, afforestation, and weather forecasting. It's particularly useful in situations where traditional weather predictions are not very accurate, such as ensuring the safe operation of self driving cars in dangerous weather. While previous studies have looked at this topic with fewer categories, this paper focuses on eleven specific types of weather images. The goal is to create a model that can accurately predict weather conditions after being trained on a large dataset of images. Accuracy is crucial in real-life situations to prevent accidents, making it the top priority for this paper. This work lays the groundwork for future applications in weather prediction, especially in situations where human expertise is not available or may be biased. The framework, capable of classifying images into eleven weather categories: dew, frost, glaze, rime, snow, hail, rain, lightning, rainbow, and sandstorm, provides real-time weather information with an accuracy of 0.9329. The proposed framework addresses the growing need for accurate weather classification and hazard prediction, offering a robust solution for various applications in the field.

Title: CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs

• Authors: Jihyung Kil, Zheda Mai, Justin Lee, Zihe Wang, Kerrie Cheng, Lemeng Wang, Ye Liu, Arpita Chowdhury, Wei-Lun Chao
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16837
• Pdf URL: https://arxiv.org/pdf/2407.16837
• Copy Paste: [[2407.16837]] CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs(https://arxiv.org/abs/2407.16837)
• Keywords: large language model
• Abstract: The ability to compare objects, scenes, or situations is crucial for effective decision-making and problem-solving in everyday life. For instance, comparing the freshness of apples enables better choices during grocery shopping, while comparing sofa designs helps optimize the aesthetics of our living space. Despite its significance, the comparative capability is largely unexplored in artificial general intelligence (AGI). In this paper, we introduce CompBench, a benchmark designed to evaluate the comparative reasoning capability of multimodal large language models (MLLMs). CompBench mines and pairs images through visually oriented questions covering eight dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. We curate a collection of around 40K image pairs using metadata from diverse vision datasets and CLIP similarity scores. These image pairs span a broad array of visual domains, including animals, fashion, sports, and both outdoor and indoor scenes. The questions are carefully crafted to discern relative characteristics between two images and are labeled by human annotators for accuracy and relevance. We use CompBench to evaluate recent MLLMs, including GPT-4V(ision), Gemini-Pro, and LLaVA-1.6. Our results reveal notable shortcomings in their comparative abilities. We believe CompBench not only sheds light on these limitations but also establishes a solid foundation for future enhancements in the comparative capability of MLLMs.

Title: $\textit{BenchIE}^{FL}$ : A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark

• Authors: Fabrice Lamarche, Philippe Langlais
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16860
• Pdf URL: https://arxiv.org/pdf/2407.16860
• Copy Paste: [[2407.16860]] $\textit{BenchIE}^{FL}$ : A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark(https://arxiv.org/abs/2407.16860)
• Keywords: extraction
• Abstract: Open Information Extraction (OIE) is a field of natural language processing that aims to present textual information in a format that allows it to be organized, analyzed and reflected upon. Numerous OIE systems are developed, claiming ever-increasing performance, marking the need for objective benchmarks. BenchIE is the latest reference we know of. Despite being very well thought out, we noticed a number of issues we believe are limiting. Therefore, we propose $\textit{BenchIE}^{FL}$, a new OIE benchmark which fully enforces the principles of BenchIE while containing fewer errors, omissions and shortcomings when candidate facts are matched towards reference ones. $\textit{BenchIE}^{FL}$ allows insightful conclusions to be drawn on the actual performance of OIE extractors.

Title: Blockchain security for ransomware detection

• Authors: Elodie Ngoie Mutombo, Mike Wa Nkongolo
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.16862
• Pdf URL: https://arxiv.org/pdf/2407.16862
• Copy Paste: [[2407.16862]] Blockchain security for ransomware detection(https://arxiv.org/abs/2407.16862)
• Keywords: security, attack
• Abstract: Blockchain networks are critical for safeguarding digital transactions and assets, but they are increasingly targeted by ransomware attacks exploiting zero-day vulnerabilities. Traditional detection techniques struggle due to the complexity of these exploits and the lack of comprehensive datasets. The UGRansome dataset addresses this gap by offering detailed features for analysing ransomware and zero-day attacks, including timestamps, attack types, protocols, network flows, and financial impacts in bitcoins (BTC). This study uses the Lazy Predict library to automate machine learning (ML) on the UGRansome dataset. The study aims to enhance blockchain security through ransomware detection based on zero-day exploit recognition using the UGRansome dataset. Lazy Predict streamlines different ML model comparisons and identifies effective algorithms for threat detection. Key features such as timestamps, protocols, and financial data are used to predict anomalies as zero-day threats and to classify known signatures as ransomware. Results demonstrate that ML can significantly improve cybersecurity in blockchain environments. The DecisionTreeClassifier and ExtraTreeClassifier, with their high performance and low training times, are ideal candidates for deployment in real-time threat detection systems.

Title: PathwayBench: Assessing Routability of Pedestrian Pathway Networks Inferred from Multi-City Imagery

• Authors: Yuxiang Zhang, Bill Howe, Sachin Mehta, Nicholas-J Bolten, Anat Caspi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16875
• Pdf URL: https://arxiv.org/pdf/2407.16875
• Copy Paste: [[2407.16875]] PathwayBench: Assessing Routability of Pedestrian Pathway Networks Inferred from Multi-City Imagery(https://arxiv.org/abs/2407.16875)
• Keywords: extraction
• Abstract: Applications to support pedestrian mobility in urban areas require a complete, and routable graph representation of the built environment. Globally available information, including aerial imagery provides a scalable source for constructing these path networks, but the associated learning problem is challenging: Relative to road network pathways, pedestrian network pathways are narrower, more frequently disconnected, often visually and materially variable in smaller areas, and their boundaries are broken up by driveway incursions, alleyways, marked or unmarked crossings through roadways. Existing algorithms to extract pedestrian pathway network graphs are inconsistently evaluated and tend to ignore routability, making it difficult to assess utility for mobility applications: Even if all path segments are available, discontinuities could dramatically and arbitrarily shift the overall path taken by a pedestrian. In this paper, we describe a first standard benchmark for the pedestrian pathway graph extraction problem, comprising the largest available dataset equipped with manually vetted ground truth annotations (covering $3,000 km^2$ land area in regions from 8 cities), and a family of evaluation metrics centering routability and downstream utility. By partitioning the data into polygons at the scale of individual intersections, we compute local routability as an efficient proxy for global routability. We consider multiple measures of polygon-level routability and compare predicted measures with ground truth to construct evaluation metrics. Using these metrics, we show that this benchmark can surface strengths and weaknesses of existing methods that are hidden by simple edge-counting metrics over single-region datasets used in prior work, representing a challenging, high-impact problem in computer vision and machine learning.

Title: Generation Constraint Scaling Can Mitigate Hallucination

• Authors: Georgios Kollias, Payel Das, Subhajit Chaudhury
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16908
• Pdf URL: https://arxiv.org/pdf/2407.16908
• Copy Paste: [[2407.16908]] Generation Constraint Scaling Can Mitigate Hallucination(https://arxiv.org/abs/2407.16908)
• Keywords: large language model
• Abstract: Addressing the issue of hallucinations in large language models (LLMs) is a critical challenge. As the cognitive mechanisms of hallucination have been related to memory, here we explore hallucination for LLM that is enabled with explicit memory mechanisms. We empirically demonstrate that by simply scaling the readout vector that constrains generation in a memory-augmented LLM decoder, hallucination mitigation can be achieved in a training-free manner. Our method is geometry-inspired and outperforms a state-of-the-art LLM editing method on the task of generation of Wikipedia-like biography entries both in terms of generation quality and runtime complexity.

Title: Train-Attention: Meta-Learning Where to Focus in Continual Knowledge Learning

• Authors: Yeongbin Seo, Dongha Lee, Jinyoung Yeo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16920
• Pdf URL: https://arxiv.org/pdf/2407.16920
• Copy Paste: [[2407.16920]] Train-Attention: Meta-Learning Where to Focus in Continual Knowledge Learning(https://arxiv.org/abs/2407.16920)
• Keywords: large language model
• Abstract: Previous studies on continual knowledge learning (CKL) in large language models (LLMs) have predominantly focused on approaches such as regularization, architectural modifications, and rehearsal techniques to mitigate catastrophic forgetting. However, these methods naively inherit the inefficiencies of standard training procedures, indiscriminately applying uniform weight across all tokens, which can lead to unnecessary parameter updates and increased forgetting. To address these shortcomings, we propose a novel CKL approach termed Train-Attention-Augmented Language Model (TAALM), which enhances learning efficiency by dynamically predicting and applying weights to tokens based on their usefulness. This method employs a meta-learning framework that optimizes token importance predictions, facilitating targeted knowledge updates and minimizing forgetting. Also, we observe that existing benchmarks do not clearly exhibit the trade-off between learning and retaining, therefore we propose a new benchmark, \textsc{LAMA-ckl}, to address this issue. Through experiments conducted on both newly introduced and established CKL benchmarks, TAALM proves the state-of-the-art performance upon the baselines, and also shows synergistic compatibility when integrated with previous CKL approaches.

Title: SAR to Optical Image Translation with Color Supervised Diffusion Model

• Authors: Xinyu Bai, Feng Xu
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.16921
• Pdf URL: https://arxiv.org/pdf/2407.16921
• Copy Paste: [[2407.16921]] SAR to Optical Image Translation with Color Supervised Diffusion Model(https://arxiv.org/abs/2407.16921)
• Keywords: interpretability, diffusion, generative
• Abstract: Synthetic Aperture Radar (SAR) offers all-weather, high-resolution imaging capabilities, but its complex imaging mechanism often poses challenges for interpretation. In response to these limitations, this paper introduces an innovative generative model designed to transform SAR images into more intelligible optical images, thereby enhancing the interpretability of SAR images. Specifically, our model backbone is based on the recent diffusion models, which have powerful generative capabilities. We employ SAR images as conditional guides in the sampling process and integrate color supervision to counteract color shift issues effectively. We conducted experiments on the SEN12 dataset and employed quantitative evaluations using peak signal-to-noise ratio, structural similarity, and fréchet inception distance. The results demonstrate that our model not only surpasses previous methods in quantitative assessments but also significantly enhances the visual quality of the generated images.

Title: From Sands to Mansions: Enabling Automatic Full-Life-Cycle Cyberattack Construction with LLM

• Authors: Lingzhi Wang, Jiahui Wang, Kyle Jung, Kedar Thiagarajan, Emily Wei, Xiangmin Shen, Yan Chen, Zhenyuan Li
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.16928
• Pdf URL: https://arxiv.org/pdf/2407.16928
• Copy Paste: [[2407.16928]] From Sands to Mansions: Enabling Automatic Full-Life-Cycle Cyberattack Construction with LLM(https://arxiv.org/abs/2407.16928)
• Keywords: security, defense, attack, large language model
• Abstract: The escalating battles between attackers and defenders in cybersecurity make it imperative to test and evaluate defense capabilities from the attackers' perspective. However, constructing full-life-cycle cyberattacks and performing red team emulations requires significant time and domain knowledge from security experts. Existing cyberattack simulation frameworks face challenges such as limited technical coverage, inability to conduct full-life-cycle attacks, and the need for manual infrastructure building. These limitations hinder the quality and diversity of the constructed attacks. In this paper, we leveraged the capabilities of Large Language Models (LLMs) in summarizing knowledge from existing attack intelligence and generating executable machine code based on human knowledge. we proposed AURORA, an automatic end-to-end cyberattack construction and emulation framework. AURORA can autonomously build multi-stage cyberattack plans based on Cyber Threat Intelligence (CTI) reports, construct the emulation infrastructures, and execute the attack procedures. We also developed an attack procedure knowledge graph to integrate knowledge about attack techniques throughout the full life cycle of advanced cyberattacks from various sources. We constructed and evaluated more than 20 full-life-cycle cyberattacks based on existing CTI reports. Compared to previous attack simulation frameworks, AURORA can construct multi-step attacks and the infrastructures in several minutes without human intervention. Furthermore, AURORA incorporates a wider range (40% more) of attack techniques into the constructed attacks in a more efficient way than the professional red teams. To benefit further research, we open-sourced the dataset containing the execution files and infrastructures of 20 emulated cyberattacks.

Title: Synthetic Data, Similarity-based Privacy Metrics, and Regulatory (Non-)Compliance

• Authors: Georgi Ganev
• Subjects: cs.CR, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.16929
• Pdf URL: https://arxiv.org/pdf/2407.16929
• Copy Paste: [[2407.16929]] Synthetic Data, Similarity-based Privacy Metrics, and Regulatory (Non-)Compliance(https://arxiv.org/abs/2407.16929)
• Keywords: privacy, protect
• Abstract: In this paper, we argue that similarity-based privacy metrics cannot ensure regulatory compliance of synthetic data. Our analysis and counter-examples show that they do not protect against singling out and linkability and, among other fundamental issues, completely ignore the motivated intruder test.

Title: ScholarChemQA: Unveiling the Power of Language Models in Chemical Research Question Answering

• Authors: Xiuying Chen, Tairan Wang, Taicheng Guo, Kehan Guo, Juexiao Zhou, Haoyang Li, Mingchen Zhuge, Jürgen Schmidhuber, Xin Gao, Xiangliang Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16931
• Pdf URL: https://arxiv.org/pdf/2407.16931
• Copy Paste: [[2407.16931]] ScholarChemQA: Unveiling the Power of Language Models in Chemical Research Question Answering(https://arxiv.org/abs/2407.16931)
• Keywords: large language model
• Abstract: Question Answering (QA) effectively evaluates language models' reasoning and knowledge depth. While QA datasets are plentiful in areas like general domain and biomedicine, academic chemistry is less explored. Chemical QA plays a crucial role in both education and research by effectively translating complex chemical information into readily understandable format. Addressing this gap, we introduce ScholarChemQA, a large-scale QA dataset constructed from chemical papers. This dataset reflects typical real-world challenges, including an imbalanced data distribution and a substantial amount of unlabeled data that can be potentially useful. Correspondingly, we introduce a QAMatch model, specifically designed to effectively answer chemical questions by fully leveraging our collected data. We first address the issue of imbalanced label distribution by re-weighting the instance-wise loss based on the inverse frequency of each class, ensuring minority classes are not dominated by majority ones during optimization. Next, we utilize the unlabeled data to enrich the learning process, generating a variety of augmentations based on a SoftMix operation and ensuring their predictions align with the same target, i.e., pseudo-labels. To ensure the quality of the pseudo-labels, we propose a calibration procedure aimed at closely aligning the pseudo-label estimates of individual samples with a desired ground truth distribution. Experiments show that our QAMatch significantly outperforms the recent similar-scale baselines and Large Language Models (LLMs) not only on our ScholarChemQA dataset but also on four benchmark datasets. We hope our benchmark and model can facilitate and promote more research on chemical QA.

Title: Synthetic Trajectory Generation Through Convolutional Neural Networks

• Authors: Jesse Merhi, Erik Buchholz, Salil S. Kanhere
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16938
• Pdf URL: https://arxiv.org/pdf/2407.16938
• Copy Paste: [[2407.16938]] Synthetic Trajectory Generation Through Convolutional Neural Networks(https://arxiv.org/abs/2407.16938)
• Keywords: privacy, generative
• Abstract: Location trajectories provide valuable insights for applications from urban planning to pandemic control. However, mobility data can also reveal sensitive information about individuals, such as political opinions, religious beliefs, or sexual orientations. Existing privacy-preserving approaches for publishing this data face a significant utility-privacy trade-off. Releasing synthetic trajectory data generated through deep learning offers a promising solution. Due to the trajectories' sequential nature, most existing models are based on recurrent neural networks (RNNs). However, research in generative adversarial networks (GANs) largely employs convolutional neural networks (CNNs) for image generation. This discrepancy raises the question of whether advances in computer vision can be applied to trajectory generation. In this work, we introduce a Reversible Trajectory-to-CNN Transformation (RTCT) that adapts trajectories into a format suitable for CNN-based models. We integrated this transformation with the well-known DCGAN in a proof-of-concept (PoC) and evaluated its performance against an RNN-based trajectory GAN using four metrics across two datasets. The PoC was superior in capturing spatial distributions compared to the RNN model but had difficulty replicating sequential and temporal properties. Although the PoC's utility is not sufficient for practical applications, the results demonstrate the transformation's potential to facilitate the use of CNNs for trajectory generation, opening up avenues for future research. To support continued research, all source code has been made available under an open-source license.

Title: Early screening of potential breakthrough technologies with enhanced interpretability: A patent-specific hierarchical attention network model

• Authors: Jaewoong Choi, Janghyeok Yoon, Changyong Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16939
• Pdf URL: https://arxiv.org/pdf/2407.16939
• Copy Paste: [[2407.16939]] Early screening of potential breakthrough technologies with enhanced interpretability: A patent-specific hierarchical attention network model(https://arxiv.org/abs/2407.16939)
• Keywords: robust, interpretability
• Abstract: Despite the usefulness of machine learning approaches for the early screening of potential breakthrough technologies, their practicality is often hindered by opaque models. To address this, we propose an interpretable machine learning approach to predicting future citation counts from patent texts using a patent-specific hierarchical attention network (PatentHAN) model. Central to this approach are (1) a patent-specific pre-trained language model, capturing the meanings of technical words in patent claims, (2) a hierarchical network structure, enabling detailed analysis at the claim level, and (3) a claim-wise self-attention mechanism, revealing pivotal claims during the screening process. A case study of 35,376 pharmaceutical patents demonstrates the effectiveness of our approach in early screening of potential breakthrough technologies while ensuring interpretability. Furthermore, we conduct additional analyses using different language models and claim types to examine the robustness of the approach. It is expected that the proposed approach will enhance expert-machine collaboration in identifying breakthrough technologies, providing new insight derived from text mining into technological value.

Title: McGAN: Generating Manufacturable Designs by Embedding Manufacturing Rules into Conditional Generative Adversarial Network

• Authors: Zhichao Wang, Xiaoliang Yan, Shreyes Melkote, David Rosen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16943
• Pdf URL: https://arxiv.org/pdf/2407.16943
• Copy Paste: [[2407.16943]] McGAN: Generating Manufacturable Designs by Embedding Manufacturing Rules into Conditional Generative Adversarial Network(https://arxiv.org/abs/2407.16943)
• Keywords: robust, generative, segmentation
• Abstract: Generative design (GD) methods aim to automatically generate a wide variety of designs that satisfy functional or aesthetic design requirements. However, research to date generally lacks considerations of manufacturability of the generated designs. To this end, we propose a novel GD approach by using deep neural networks to encode design for manufacturing (DFM) rules, thereby modifying part designs to make them manufacturable by a given manufacturing process. Specifically, a three-step approach is proposed: first, an instance segmentation method, Mask R-CNN, is used to decompose a part design into subregions. Second, a conditional generative adversarial neural network (cGAN), Pix2Pix, transforms unmanufacturable decomposed subregions into manufacturable subregions. The transformed subregions of designs are subsequently reintegrated into a unified manufacturable design. These three steps, Mask-RCNN, Pix2Pix, and reintegration, form the basis of the proposed Manufacturable conditional GAN (McGAN) framework. Experimental results show that McGAN can transform existing unmanufacturable designs to generate their corresponding manufacturable counterparts automatically that realize the specified manufacturing rules in an efficient and robust manner. The effectiveness of McGAN is demonstrated through two-dimensional design case studies of an injection molding process.

Title: Towards Transfer Unlearning: Empirical Evidence of Cross-Domain Bias Mitigation

• Authors: Huimin Lu, Masaru Isonuma, Junichiro Mori, Ichiro Sakata
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16951
• Pdf URL: https://arxiv.org/pdf/2407.16951
• Copy Paste: [[2407.16951]] Towards Transfer Unlearning: Empirical Evidence of Cross-Domain Bias Mitigation(https://arxiv.org/abs/2407.16951)
• Keywords: large language model
• Abstract: Large language models (LLMs) often inherit biases from vast amounts of training corpora. Traditional debiasing methods, while effective to some extent, do not completely eliminate memorized biases and toxicity in LLMs. In this paper, we study an unlearning-based approach to debiasing in LLMs by performing gradient ascent on hate speech against minority groups, i.e., minimizing the likelihood of biased or toxic content. Specifically, we propose a mask language modeling unlearning technique, which unlearns the harmful part of the text. This method enables LLMs to selectively forget and disassociate from biased and harmful content. Experimental results demonstrate the effectiveness of our approach in diminishing bias while maintaining the language modeling abilities. Surprisingly, the results also unveil an unexpected potential for cross-domain transfer unlearning: debiasing in one bias form (e.g. gender) may contribute to mitigating others (e.g. race and religion).

Title: Open Challenges on Fairness of Artificial Intelligence in Medical Imaging Applications

• Authors: Enzo Ferrante, Rodrigo Echeveste
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16953
• Pdf URL: https://arxiv.org/pdf/2407.16953
• Copy Paste: [[2407.16953]] Open Challenges on Fairness of Artificial Intelligence in Medical Imaging Applications(https://arxiv.org/abs/2407.16953)
• Keywords: fair
• Abstract: Recently, the research community of computerized medical imaging has started to discuss and address potential fairness issues that may emerge when developing and deploying AI systems for medical image analysis. This chapter covers some of the pressing challenges encountered when doing research in this area, and it is intended to raise questions and provide food for thought for those aiming to enter this research field. The chapter first discusses various sources of bias, including data collection, model training, and clinical deployment, and their impact on the fairness of machine learning algorithms in medical image computing. We then turn to discussing open challenges that we believe require attention from researchers and practitioners, as well as potential pitfalls of naive application of common methods in the field. We cover a variety of topics including the impact of biased metrics when auditing for fairness, the leveling down effect, task difficulty variations among subgroups, discovering biases in unseen populations, and explaining biases beyond standard demographic attributes.

Title: DVPE: Divided View Position Embedding for Multi-View 3D Object Detection

• Authors: Jiasen Wang, Zhenglin Li, Ke Sun, Xianyuan Liu, Yang Zhou
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2407.16955
• Pdf URL: https://arxiv.org/pdf/2407.16955
• Copy Paste: [[2407.16955]] DVPE: Divided View Position Embedding for Multi-View 3D Object Detection(https://arxiv.org/abs/2407.16955)
• Keywords: transformer
• Abstract: Sparse query-based paradigms have achieved significant success in multi-view 3D detection for autonomous vehicles. Current research faces challenges in balancing between enlarging receptive fields and reducing interference when aggregating multi-view features. Moreover, different poses of cameras present challenges in training global attention models. To address these problems, this paper proposes a divided view method, in which features are modeled globally via the visibility crossattention mechanism, but interact only with partial features in a divided local virtual space. This effectively reduces interference from other irrelevant features and alleviates the training difficulties of the transformer by decoupling the position embedding from camera poses. Additionally, 2D historical RoI features are incorporated into the object-centric temporal modeling to utilize highlevel visual semantic information. The model is trained using a one-to-many assignment strategy to facilitate stability. Our framework, named DVPE, achieves state-of-the-art performance (57.2% mAP and 64.5% NDS) on the nuScenes test set. Codes will be available at this https URL.

Title: Dynamic Graph Transformer with Correlated Spatial-Temporal Positional Encoding

• Authors: Zhe Wang, Sheng Zhou, Jiawei Chen, Zhen Zhang, Binbin Hu, Yan Feng, Chun Chen, Can Wang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16959
• Pdf URL: https://arxiv.org/pdf/2407.16959
• Copy Paste: [[2407.16959]] Dynamic Graph Transformer with Correlated Spatial-Temporal Positional Encoding(https://arxiv.org/abs/2407.16959)
• Keywords: transformer
• Abstract: Learning effective representations for Continuous-Time Dynamic Graphs (CTDGs) has garnered significant research interest, largely due to its powerful capabilities in modeling complex interactions between nodes. A fundamental and crucial requirement for representation learning in CTDGs is the appropriate estimation and preservation of proximity. However, due to the sparse and evolving characteristics of CTDGs, the spatial-temporal properties inherent in high-order proximity remain largely unexplored. Despite its importance, this property presents significant challenges due to the computationally intensive nature of personalized interaction intensity estimation and the dynamic attributes of CTDGs. To this end, we propose a novel Correlated Spatial-Temporal Positional encoding that incorporates a parameter-free personalized interaction intensity estimation under the weak assumption of the Poisson Point Process. Building on this, we introduce the Dynamic Graph Transformer with \Correlated Spatial-Temporal Positional Encoding (CorDGT), which efficiently retains the evolving spatial-temporal high-order proximity for effective node representation learning in CTDGs. Extensive experiments on seven small and two large-scale datasets demonstrate the superior performance and scalability of the proposed CorDGT.

Title: When AI Defeats Password Deception! A Deep Learning Framework to Distinguish Passwords and Honeywords

• Authors: Jimmy Dani, Brandon McCulloh, Nitesh Saxena
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16964
• Pdf URL: https://arxiv.org/pdf/2407.16964
• Copy Paste: [[2407.16964]] When AI Defeats Password Deception! A Deep Learning Framework to Distinguish Passwords and Honeywords(https://arxiv.org/abs/2407.16964)
• Keywords: security, defense, attack
• Abstract: "Honeywords" have emerged as a promising defense mechanism for detecting data breaches and foiling offline dictionary attacks (ODA) by deceiving attackers with false passwords. In this paper, we propose PassFilter, a novel deep learning (DL) based attack framework, fundamental in its ability to identify passwords from a set of sweetwords associated with a user account, effectively challenging a variety of honeywords generation techniques (HGTs). The DL model in PassFilter is trained with a set of previously collected or adversarially generated passwords and honeywords, and carefully orchestrated to predict whether a sweetword is the password or a honeyword. Our model can compromise the security of state-of-the-art, heuristics-based, and representation learning-based HGTs proposed by Dionysiou et al. Specifically, our analysis with nine publicly available password datasets shows that PassFilter significantly outperforms the baseline random guessing success rate of 5%, achieving 6.10% to 52.78% on the 1st guessing attempt, considering 20 sweetwords per account. This success rate rapidly increases with additional login attempts before account lock-outs, often allowed on many real-world online services to maintain reasonable usability. For example, it ranges from 41.78% to 96.80% for five attempts, and from 72.87% to 99.00% for ten attempts, compared to 25% and 50% random guessing, respectively. We also examined PassFilter against general-purpose language models used for honeyword generation, like those proposed by Yu et al. These honeywords also proved vulnerable to our attack, with success rates of 14.19% for 1st guessing attempt, increasing to 30.23%, 41.70%, and 63.10% after 3rd, 5th, and 10th guessing attempts, respectively. Our findings demonstrate the effectiveness of DL model deployed in PassFilter in breaching state-of-the-art HGTs and compromising password security based on ODA.

Title: Case-Enhanced Vision Transformer: Improving Explanations of Image Similarity with a ViT-based Similarity Metric

• Authors: Ziwei Zhao, David Leake, Xiaomeng Ye, David Crandall
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16981
• Pdf URL: https://arxiv.org/pdf/2407.16981
• Copy Paste: [[2407.16981]] Case-Enhanced Vision Transformer: Improving Explanations of Image Similarity with a ViT-based Similarity Metric(https://arxiv.org/abs/2407.16981)
• Keywords: explainability, transformer
• Abstract: This short paper presents preliminary research on the Case-Enhanced Vision Transformer (CEViT), a similarity measurement method aimed at improving the explainability of similarity assessments for image data. Initial experimental results suggest that integrating CEViT into k-Nearest Neighbor (k-NN) classification yields classification accuracy comparable to state-of-the-art computer vision models, while adding capabilities for illustrating differences between classes. CEViT explanations can be influenced by prior cases, to illustrate aspects of similarity relevant to those cases.

Title: Diffree: Text-Guided Shape Free Object Inpainting with Diffusion Model

• Authors: Lirui Zhao, Tianshuo Yang, Wenqi Shao, Yuxin Zhang, Yu Qiao, Ping Luo, Kaipeng Zhang, Rongrong Ji
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16982
• Pdf URL: https://arxiv.org/pdf/2407.16982
• Copy Paste: [[2407.16982]] Diffree: Text-Guided Shape Free Object Inpainting with Diffusion Model(https://arxiv.org/abs/2407.16982)
• Keywords: diffusion
• Abstract: This paper addresses an important problem of object addition for images with only text guidance. It is challenging because the new object must be integrated seamlessly into the image with consistent visual context, such as lighting, texture, and spatial location. While existing text-guided image inpainting methods can add objects, they either fail to preserve the background consistency or involve cumbersome human intervention in specifying bounding boxes or user-scribbled masks. To tackle this challenge, we introduce Diffree, a Text-to-Image (T2I) model that facilitates text-guided object addition with only text control. To this end, we curate OABench, an exquisite synthetic dataset by removing objects with advanced image inpainting techniques. OABench comprises 74K real-world tuples of an original image, an inpainted image with the object removed, an object mask, and object descriptions. Trained on OABench using the Stable Diffusion model with an additional mask prediction module, Diffree uniquely predicts the position of the new object and achieves object addition with guidance from only text. Extensive experiments demonstrate that Diffree excels in adding new objects with a high success rate while maintaining background consistency, spatial appropriateness, and object relevance and quality.

Title: DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car Reconstruction

• Authors: Xiaobiao Du, Haiyang Sun, Ming Lu, Tianqing Zhu, Xin Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16988
• Pdf URL: https://arxiv.org/pdf/2407.16988
• Copy Paste: [[2407.16988]] DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car Reconstruction(https://arxiv.org/abs/2407.16988)
• Keywords: robust, generative
• Abstract: Self-driving industries usually employ professional artists to build exquisite 3D cars. However, it is expensive to craft large-scale digital assets. Since there are already numerous datasets available that contain a vast number of images of cars, we focus on reconstructing high-quality 3D car models from these datasets. However, these datasets only contain one side of cars in the forward-moving scene. We try to use the existing generative models to provide more supervision information, but they struggle to generalize well in cars since they are trained on synthetic datasets not car-specific. In addition, The reconstructed 3D car texture misaligns due to a large error in camera pose estimation when dealing with in-the-wild images. These restrictions make it challenging for previous methods to reconstruct complete 3D cars. To address these problems, we propose a novel method, named DreamCar, which can reconstruct high-quality 3D cars given a few images even a single image. To generalize the generative model, we collect a car dataset, named Car360, with over 5,600 vehicles. With this dataset, we make the generative model more robust to cars. We use this generative prior specific to the car to guide its reconstruction via Score Distillation Sampling. To further complement the supervision information, we utilize the geometric and appearance symmetry of cars. Finally, we propose a pose optimization method that rectifies poses to tackle texture misalignment. Extensive experiments demonstrate that our method significantly outperforms existing methods in reconstructing high-quality 3D cars. \href{this https URL}{Our code is available.}

Title: LoFormer: Local Frequency Transformer for Image Deblurring

• Authors: Xintian Mao, Jiansheng Wang, Xingran Xie, Qingli Li, Yan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16993
• Pdf URL: https://arxiv.org/pdf/2407.16993
• Copy Paste: [[2407.16993]] LoFormer: Local Frequency Transformer for Image Deblurring(https://arxiv.org/abs/2407.16993)
• Keywords: transformer
• Abstract: Due to the computational complexity of self-attention (SA), prevalent techniques for image deblurring often resort to either adopting localized SA or employing coarse-grained global SA methods, both of which exhibit drawbacks such as compromising global modeling or lacking fine-grained correlation. In order to address this issue by effectively modeling long-range dependencies without sacrificing fine-grained details, we introduce a novel approach termed Local Frequency Transformer (LoFormer). Within each unit of LoFormer, we incorporate a Local Channel-wise SA in the frequency domain (Freq-LC) to simultaneously capture cross-covariance within low- and high-frequency local windows. These operations offer the advantage of (1) ensuring equitable learning opportunities for both coarse-grained structures and fine-grained details, and (2) exploring a broader range of representational properties compared to coarse-grained global SA methods. Additionally, we introduce an MLP Gating mechanism complementary to Freq-LC, which serves to filter out irrelevant features while enhancing global learning capabilities. Our experiments demonstrate that LoFormer significantly improves performance in the image deblurring task, achieving a PSNR of 34.09 dB on the GoPro dataset with 126G FLOPs. this https URL

Title: Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective

• Authors: Yujian Liu, Yang Zhang, Tommi Jaakkola, Shiyu Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16997
• Pdf URL: https://arxiv.org/pdf/2407.16997
• Copy Paste: [[2407.16997]] Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective(https://arxiv.org/abs/2407.16997)
• Keywords: attack
• Abstract: This paper investigates Who's Harry Potter (WHP), a pioneering yet insufficiently understood method for LLM unlearning. We explore it in two steps. First, we introduce a new task of LLM targeted unlearning, where given an unlearning target (e.g., a person) and some unlearning documents, we aim to unlearn only the information about the target, rather than everything in the unlearning documents. We further argue that a successful unlearning should satisfy criteria such as not outputting gibberish, not fabricating facts about the unlearning target, and not releasing factual information under jailbreak attacks. Second, we construct a causal intervention framework for targeted unlearning, where the knowledge of the unlearning target is modeled as a confounder between LLM input and output, and the unlearning process as a deconfounding process. This framework justifies and extends WHP, deriving a simple unlearning algorithm that includes WHP as a special case. Experiments on existing and new datasets show that our approach, without explicitly optimizing for the aforementioned criteria, achieves competitive performance in all of them. Our code is available at this https URL.

Title: SepsisLab: Early Sepsis Prediction with Uncertainty Quantification and Active Sensing

• Authors: Changchang Yin, Pin-Yu Chen, Bingsheng Yao, Dakuo Wang, Jeffrey Caterino, Ping Zhang
• Subjects: cs.LG, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2407.16999
• Pdf URL: https://arxiv.org/pdf/2407.16999
• Copy Paste: [[2407.16999]] SepsisLab: Early Sepsis Prediction with Uncertainty Quantification and Active Sensing(https://arxiv.org/abs/2407.16999)
• Keywords: robust
• Abstract: Sepsis is the leading cause of in-hospital mortality in the USA. Early sepsis onset prediction and diagnosis could significantly improve the survival of sepsis patients. Existing predictive models are usually trained on high-quality data with few missing information, while missing values widely exist in real-world clinical scenarios (especially in the first hours of admissions to the hospital), which causes a significant decrease in accuracy and an increase in uncertainty for the predictive models. The common method to handle missing values is imputation, which replaces the unavailable variables with estimates from the observed data. The uncertainty of imputation results can be propagated to the sepsis prediction outputs, which have not been studied in existing works on either sepsis prediction or uncertainty quantification. In this study, we first define such propagated uncertainty as the variance of prediction output and then introduce uncertainty propagation methods to quantify the propagated uncertainty. Moreover, for the potential high-risk patients with low confidence due to limited observations, we propose a robust active sensing algorithm to increase confidence by actively recommending clinicians to observe the most informative variables. We validate the proposed models in both publicly available data (i.e., MIMIC-III and AmsterdamUMCdb) and proprietary data in The Ohio State University Wexner Medical Center (OSUWMC). The experimental results show that the propagated uncertainty is dominant at the beginning of admissions to hospitals and the proposed algorithm outperforms state-of-the-art active sensing methods. Finally, we implement a SepsisLab system for early sepsis prediction and active sensing based on our pre-trained models. Clinicians and potential sepsis patients can benefit from the system in early prediction and diagnosis of sepsis.

Title: Progressive Query Refinement Framework for Bird's-Eye-View Semantic Segmentation from Surrounding Images

• Authors: Dooseop Choi, Jungyu Kang, Taeghyun An, Kyounghwan Ahn, KyoungWook Min
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17003
• Pdf URL: https://arxiv.org/pdf/2407.17003
• Copy Paste: [[2407.17003]] Progressive Query Refinement Framework for Bird's-Eye-View Semantic Segmentation from Surrounding Images(https://arxiv.org/abs/2407.17003)
• Keywords: segmentation
• Abstract: Expressing images with Multi-Resolution (MR) features has been widely adopted in many computer vision tasks. In this paper, we introduce the MR concept into Bird's-Eye-View (BEV) semantic segmentation for autonomous driving. This introduction enhances our model's ability to capture both global and local characteristics of driving scenes through our proposed residual learning. Specifically, given a set of MR BEV query maps, the lowest resolution query map is initially updated using a View Transformation (VT) encoder. This updated query map is then upscaled and merged with a higher resolution query map to undergo further updates in a subsequent VT encoder. This process is repeated until the resolution of the updated query map reaches the target. Finally, the lowest resolution map is added to the target resolution to generate the final query map. During training, we enforce both the lowest and final query maps to align with the ground-truth BEV semantic map to help our model effectively capture the global and local characteristics. We also propose a visual feature interaction network that promotes interactions between features across images and across feature levels, thus highly contributing to the performance improvement. We evaluate our model on a large-scale real-world dataset. The experimental results show that our model outperforms the SOTA models in terms of IoU metric. Codes are available at this https URL

Title: Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanism

• Authors: Anhao Zhao, Fanghua Ye, Jinlan Fu, Xiaoyu Shen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17011
• Pdf URL: https://arxiv.org/pdf/2407.17011
• Copy Paste: [[2407.17011]] Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanism(https://arxiv.org/abs/2407.17011)
• Keywords: large language model
• Abstract: Large language models (LLMs) exhibit remarkable in-context learning (ICL) capabilities. However, the underlying working mechanism of ICL remains poorly understood. Recent research presents two conflicting views on ICL: One attributes it to LLMs' inherent ability of task recognition, deeming label correctness and shot numbers of demonstrations as not crucial; the other emphasizes the impact of similar examples in the demonstrations, stressing the need for label correctness and more shots. In this work, we provide a Two-Dimensional Coordinate System that unifies both views into a systematic framework. The framework explains the behavior of ICL through two orthogonal variables: whether LLMs can recognize the task and whether similar examples are presented in the demonstrations. We propose the peak inverse rank metric to detect the task recognition ability of LLMs and study LLMs' reactions to different definitions of similarity. Based on these, we conduct extensive experiments to elucidate how ICL functions across each quadrant on multiple representative classification tasks. Finally, we extend our analyses to generation tasks, showing that our coordinate system can also be used to interpret ICL for generation tasks effectively.

Title: EAFormer: Scene Text Segmentation with Edge-Aware Transformers

• Authors: Haiyang Yu, Teng Fu, Bin Li, Xiangyang Xue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17020
• Pdf URL: https://arxiv.org/pdf/2407.17020
• Copy Paste: [[2407.17020]] EAFormer: Scene Text Segmentation with Edge-Aware Transformers(https://arxiv.org/abs/2407.17020)
• Keywords: fair, transformer, generative, segmentation
• Abstract: Scene text segmentation aims at cropping texts from scene images, which is usually used to help generative models edit or remove texts. The existing text segmentation methods tend to involve various text-related supervisions for better performance. However, most of them ignore the importance of text edges, which are significant for downstream applications. In this paper, we propose Edge-Aware Transformers, termed EAFormer, to segment texts more accurately, especially at the edge of texts. Specifically, we first design a text edge extractor to detect edges and filter out edges of non-text areas. Then, we propose an edge-guided encoder to make the model focus more on text edges. Finally, an MLP-based decoder is employed to predict text masks. We have conducted extensive experiments on commonly-used benchmarks to verify the effectiveness of EAFormer. The experimental results demonstrate that the proposed method can perform better than previous methods, especially on the segmentation of text edges. Considering that the annotations of several benchmarks (e.g., COCO_TS and MLT_S) are not accurate enough to fairly evaluate our methods, we have relabeled these datasets. Through experiments, we observe that our method can achieve a higher performance improvement when more accurate annotations are used for training.

Title: Can Language Models Evaluate Human Written Text? Case Study on Korean Student Writing for Education

• Authors: Seungyoon Kim, Seungone Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17022
• Pdf URL: https://arxiv.org/pdf/2407.17022
• Copy Paste: [[2407.17022]] Can Language Models Evaluate Human Written Text? Case Study on Korean Student Writing for Education(https://arxiv.org/abs/2407.17022)
• Keywords: robust, large language model
• Abstract: Large language model (LLM)-based evaluation pipelines have demonstrated their capability to robustly evaluate machine-generated text. Extending this methodology to assess human-written text could significantly benefit educational settings by providing direct feedback to enhance writing skills, although this application is not straightforward. In this paper, we investigate whether LLMs can effectively assess human-written text for educational purposes. We collected 100 texts from 32 Korean students across 15 types of writing and employed GPT-4-Turbo to evaluate them using grammaticality, fluency, coherence, consistency, and relevance as criteria. Our analyses indicate that LLM evaluators can reliably assess grammaticality and fluency, as well as more objective types of writing, though they struggle with other criteria and types of writing. We publicly release our dataset and feedback.

Title: Enhancing Environmental Monitoring through Multispectral Imaging: The WasteMS Dataset for Semantic Segmentation of Lakeside Waste

• Authors: Qinfeng Zhu, Ningxin Weng, Lei Fan, Yuanzhi Cai
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.17028
• Pdf URL: https://arxiv.org/pdf/2407.17028
• Copy Paste: [[2407.17028]] Enhancing Environmental Monitoring through Multispectral Imaging: The WasteMS Dataset for Semantic Segmentation of Lakeside Waste(https://arxiv.org/abs/2407.17028)
• Keywords: protect, segmentation
• Abstract: Environmental monitoring of lakeside green areas is crucial for environmental protection. Compared to manual inspections, computer vision technologies offer a more efficient solution when deployed on-site. Multispectral imaging provides diverse information about objects under different spectrums, aiding in the differentiation between waste and lakeside lawn environments. This study introduces WasteMS, the first multispectral dataset established for the semantic segmentation of lakeside waste. WasteMS includes a diverse range of waste types in lawn environments, captured under various lighting conditions. We implemented a rigorous annotation process to label waste in images. Representative semantic segmentation frameworks were used to evaluate segmentation accuracy using WasteMS. Challenges encountered when using WasteMS for segmenting waste on lakeside lawns were discussed. The WasteMS dataset is available at this https URL.

Title: Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance

• Authors: Ao Shen, Qiang Wang, Zhiquan Lai, Xionglve Li, Dongsheng Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17029
• Pdf URL: https://arxiv.org/pdf/2407.17029
• Copy Paste: [[2407.17029]] Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance(https://arxiv.org/abs/2407.17029)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have demonstrated impressive performance across various domains. However, the enormous number of model parameters makes fine-tuning challenging, significantly limiting their application and deployment. Existing solutions combine parameter quantization with Low-Rank Adaptation (LoRA), greatly reducing memory usage but resulting in noticeable performance degradation. In this paper, we identify an imbalance in fine-tuning quantized pre-trained models: overly complex adapter inputs and outputs versus low effective trainability of the adaptation. We propose Quantized LLMs with Balanced-rank Adaptation (Q-BaRA), which simplifies the adapter inputs and outputs while increasing the adapter's rank to achieve a more suitable balance for fine-tuning quantized LLMs. Additionally, for scenarios where fine-tuned LLMs need to be deployed as low-precision inference models, we introduce Quantization-Aware Fine-tuning with Higher Rank Adaptation (QA-HiRA), which simplifies the adapter inputs and outputs to align with the pre-trained model's block-wise quantization while employing a single matrix to achieve a higher rank. Both Q-BaRA and QA-HiRA are easily implemented and offer the following optimizations: (i) Q-BaRA consistently achieves the highest accuracy compared to baselines and other variants, requiring the same number of trainable parameters and computational effort; (ii) QA-HiRA naturally merges adapter parameters into the block-wise quantized model after fine-tuning, achieving the highest accuracy compared to other methods. We apply our Q-BaRA and QA-HiRA to the LLaMA and LLaMA2 model families and validate their effectiveness across different fine-tuning datasets and downstream scenarios. Code will be made available at \href{this https URL}{this https URL}

Title: Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference

• Authors: Jian Xu, Delu Zeng, John Paisley
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.17033
• Pdf URL: https://arxiv.org/pdf/2407.17033
• Copy Paste: [[2407.17033]] Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference(https://arxiv.org/abs/2407.17033)
• Keywords: robust, diffusion
• Abstract: Deep Gaussian processes (DGPs) provide a robust paradigm for Bayesian deep learning. In DGPs, a set of sparse integration locations called inducing points are selected to approximate the posterior distribution of the model. This is done to reduce computational complexity and improve model efficiency. However, inferring the posterior distribution of inducing points is not straightforward. Traditional variational inference approaches to posterior approximation often lead to significant bias. To address this issue, we propose an alternative method called Denoising Diffusion Variational Inference (DDVI) that uses a denoising diffusion stochastic differential equation (SDE) to generate posterior samples of inducing variables. We rely on score matching methods for denoising diffusion model to approximate score functions with a neural network. Furthermore, by combining classical mathematical theory of SDEs with the minimization of KL divergence between the approximate and true processes, we propose a novel explicit variational lower bound for the marginal likelihood function of DGP. Through experiments on various datasets and comparisons with baseline methods, we empirically demonstrate the effectiveness of DDVI for posterior inference of inducing points for DGP models.

Title: Q-Ground: Image Quality Grounding with Large Multi-modality Models

• Authors: Chaofeng Chen, Sensen Yang, Haoning Wu, Liang Liao, Zicheng Zhang, Annan Wang, Wenxiu Sun, Qiong Yan, Weisi Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17035
• Pdf URL: https://arxiv.org/pdf/2407.17035
• Copy Paste: [[2407.17035]] Q-Ground: Image Quality Grounding with Large Multi-modality Models(https://arxiv.org/abs/2407.17035)
• Keywords: robust, segmentation
• Abstract: Recent advances of large multi-modality models (LMM) have greatly improved the ability of image quality assessment (IQA) method to evaluate and explain the quality of visual content. However, these advancements are mostly focused on overall quality assessment, and the detailed examination of local quality, which is crucial for comprehensive visual understanding, is still largely unexplored. In this work, we introduce Q-Ground, the first framework aimed at tackling fine-scale visual quality grounding by combining large multi-modality models with detailed visual quality analysis. Central to our contribution is the introduction of the QGround-100K dataset, a novel resource containing 100k triplets of (image, quality text, distortion segmentation) to facilitate deep investigations into visual quality. The dataset comprises two parts: one with human-labeled annotations for accurate quality assessment, and another labeled automatically by LMMs such as GPT4V, which helps improve the robustness of model training while also reducing the costs of data collection. With the QGround-100K dataset, we propose a LMM-based method equipped with multi-scale feature learning to learn models capable of performing both image quality answering and distortion segmentation based on text prompts. This dual-capability approach not only refines the model's understanding of region-aware image quality but also enables it to interactively respond to complex, text-based queries about image quality and specific distortions. Q-Ground takes a step towards sophisticated visual quality analysis in a finer scale, establishing a new benchmark for future research in the area. Codes and dataset are available at this https URL.

Title: High Efficiency Image Compression for Large Visual-Language Models

• Authors: Binzhe Li, Shurun Wang, Shiqi Wang, Yan Ye
• Subjects: cs.CV, cs.AI, cs.CL, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.17060
• Pdf URL: https://arxiv.org/pdf/2407.17060
• Copy Paste: [[2407.17060]] High Efficiency Image Compression for Large Visual-Language Models(https://arxiv.org/abs/2407.17060)
• Keywords: robust
• Abstract: In recent years, large visual language models (LVLMs) have shown impressive performance and promising generalization capability in multi-modal tasks, thus replacing humans as receivers of visual information in various application scenarios. In this paper, we pioneer to propose a variable bitrate image compression framework consisting of a pre-editing module and an end-to-end codec to achieve promising rate-accuracy performance for different LVLMs. In particular, instead of optimizing an adaptive pre-editing network towards a particular task or several representative tasks, we propose a new optimization strategy tailored for LVLMs, which is designed based on the representation and discrimination capability with token-level distortion and rank. The pre-editing module and the variable bitrate end-to-end image codec are jointly trained by the losses based on semantic tokens of the large model, which introduce enhanced generalization capability for various data and tasks. {Experimental results demonstrate that the proposed framework could efficiently achieve much better rate-accuracy performance compared to the state-of-the-art coding standard, Versatile Video Coding.} Meanwhile, experiments with multi-modal tasks have revealed the robustness and generalization capability of the proposed framework.

Title: AI-based Density Recognition

• Authors: Simone Müller, Daniel Kolb, Matthias Müller, Dieter Kranzlmüller
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17064
• Pdf URL: https://arxiv.org/pdf/2407.17064
• Copy Paste: [[2407.17064]] AI-based Density Recognition(https://arxiv.org/abs/2407.17064)
• Keywords: extraction
• Abstract: Learning-based analysis of images is commonly used in the fields of mobility and robotics for safe environmental motion and interaction. This requires not only object recognition but also the assignment of certain properties to them. With the help of this information, causally related actions can be adapted to different circumstances. Such logical interactions can be optimized by recognizing object-assigned properties. Density as a physical property offers the possibility to recognize how heavy an object is, which material it is made of, which forces are at work, and consequently which influence it has on its environment. Our approach introduces an AI-based concept for assigning physical properties to objects through the use of associated images. Based on synthesized data, we derive specific patterns from 2D images using a neural network to extract further information such as volume, material, or density. Accordingly, we discuss the possibilities of property-based feature extraction to improve causally related logics.

Title: Curriculum Negative Mining For Temporal Networks

• Authors: Ziyue Chen, Tongya Zheng, Mingli Song
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17070
• Pdf URL: https://arxiv.org/pdf/2407.17070
• Copy Paste: [[2407.17070]] Curriculum Negative Mining For Temporal Networks(https://arxiv.org/abs/2407.17070)
• Keywords: robust
• Abstract: Temporal networks are effective in capturing the evolving interactions of networks over time, such as social networks and e-commerce networks. In recent years, researchers have primarily concentrated on developing specific model architectures for Temporal Graph Neural Networks (TGNNs) in order to improve the representation quality of temporal nodes and edges. However, limited attention has been given to the quality of negative samples during the training of TGNNs. When compared with static networks, temporal networks present two specific challenges for negative sampling: positive sparsity and positive shift. Positive sparsity refers to the presence of a single positive sample amidst numerous negative samples at each timestamp, while positive shift relates to the variations in positive samples across different timestamps. To robustly address these challenges in training TGNNs, we introduce Curriculum Negative Mining (CurNM), a model-aware curriculum learning framework that adaptively adjusts the difficulty of negative samples. Within this framework, we first establish a dynamically updated negative pool that balances random, historical, and hard negatives to address the challenges posed by positive sparsity. Secondly, we implement a temporal-aware negative selection module that focuses on learning from the disentangled factors of recently active edges, thus accurately capturing shifting preferences. Extensive experiments on 12 datasets and 3 TGNNs demonstrate that our method outperforms baseline methods by a significant margin. Additionally, thorough ablation studies and parameter sensitivity experiments verify the usefulness and robustness of our approach. Our code is available at this https URL.

Title: SAFETY-J: Evaluating Safety with Critique

• Authors: Yixiu Liu, Yuxiang Zheng, Shijie Xia, Yuan Guo, Jiajun Li, Yi Tu, Chaoling Song, Pengfei Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17075
• Pdf URL: https://arxiv.org/pdf/2407.17075
• Copy Paste: [[2407.17075]] SAFETY-J: Evaluating Safety with Critique(https://arxiv.org/abs/2407.17075)
• Keywords: robust, interpretability, generative, large language model
• Abstract: The deployment of Large Language Models (LLMs) in content generation raises significant safety concerns, particularly regarding the transparency and interpretability of content evaluations. Current methods, primarily focused on binary safety classifications, lack mechanisms for detailed critique, limiting their utility for model improvement and user trust. To address these limitations, we introduce SAFETY-J, a bilingual generative safety evaluator for English and Chinese with critique-based judgment. SAFETY-J utilizes a robust training dataset that includes diverse dialogues and augmented query-response pairs to assess safety across various scenarios comprehensively. We establish an automated meta-evaluation benchmark that objectively assesses the quality of critiques with minimal human intervention, facilitating scalable and continuous improvement. Additionally, SAFETY-J employs an iterative preference learning technique to dynamically refine safety assessments based on meta-evaluations and critiques. Our evaluations demonstrate that SAFETY-J provides more nuanced and accurate safety evaluations, thereby enhancing both critique quality and predictive reliability in complex content scenarios. To facilitate further research and application, we will open-source SAFETY-J's training protocols, datasets, and code.

Title: A Survey Forest Diagram : Gain a Divergent Insight View on a Specific Research Topic

• Authors: Jinghong Li, Wen Gu, Koichi Ota, Shinobu Hasegawa
• Subjects: cs.CL, cs.AI, cs.DL
• Abstract URL: https://arxiv.org/abs/2407.17081
• Pdf URL: https://arxiv.org/pdf/2407.17081
• Copy Paste: [[2407.17081]] A Survey Forest Diagram : Gain a Divergent Insight View on a Specific Research Topic(https://arxiv.org/abs/2407.17081)
• Keywords: generative
• Abstract: With the exponential growth in the number of papers and the trend of AI research, the use of Generative AI for information retrieval and question-answering has become popular for conducting research surveys. However, novice researchers unfamiliar with a particular field may not significantly improve their efficiency in interacting with Generative AI because they have not developed divergent thinking in that field. This study aims to develop an in-depth Survey Forest Diagram that guides novice researchers in divergent thinking about the research topic by indicating the citation clues among multiple papers, to help expand the survey perspective for novice researchers.

Title: OVR: A Dataset for Open Vocabulary Temporal Repetition Counting in Videos

• Authors: Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Andrew Zisserman
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17085
• Pdf URL: https://arxiv.org/pdf/2407.17085
• Copy Paste: [[2407.17085]] OVR: A Dataset for Open Vocabulary Temporal Repetition Counting in Videos(https://arxiv.org/abs/2407.17085)
• Keywords: transformer
• Abstract: We introduce a dataset of annotations of temporal repetitions in videos. The dataset, OVR (pronounced as over), contains annotations for over 72K videos, with each annotation specifying the number of repetitions, the start and end time of the repetitions, and also a free-form description of what is repeating. The annotations are provided for videos sourced from Kinetics and Ego4D, and consequently cover both Exo and Ego viewing conditions, with a huge variety of actions and activities. Moreover, OVR is almost an order of magnitude larger than previous datasets for video repetition. We also propose a baseline transformer-based counting model, OVRCounter, that can localise and count repetitions in videos that are up to 320 frames long. The model is trained and evaluated on the OVR dataset, and its performance assessed with and without using text to specify the target class to count. The performance is also compared to a prior repetition counting model. The dataset is available for download at: this https URL

Title: MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models

• Authors: Chunsan Hong, Tae-Hyun Oh, Minhyuk Sung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17095
• Pdf URL: https://arxiv.org/pdf/2407.17095
• Copy Paste: [[2407.17095]] MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models(https://arxiv.org/abs/2407.17095)
• Keywords: privacy, diffusion
• Abstract: Diffusion models have achieved remarkable success in Text-to-Image generation tasks, leading to the development of many commercial models. However, recent studies have reported that diffusion models often generate replicated images in train data when triggered by specific prompts, potentially raising social issues ranging from copyright to privacy concerns. To sidestep the memorization, there have been recent studies for developing memorization mitigation methods for diffusion models. Nevertheless, the lack of benchmarks impedes the assessment of the true effectiveness of these methods. In this work, we present MemBench, the first benchmark for evaluating image memorization mitigation methods. Our benchmark includes a large number of memorized image trigger prompts in Stable Diffusion, the most popularly used model nowadays. Furthermore, in contrast to the prior work evaluating mitigation performance only on trigger prompts, we present metrics evaluating on both trigger prompts and general prompts, so that we can see whether mitigation methods address the memorization issue while maintaining performance for general prompts. This is an important development considering the practical applications which previous works have overlooked. Through evaluation on MemBench, we verify that the performance of existing image memorization mitigation methods is still insufficient for application to diffusion models.

Title: Towards Robust Knowledge Tracing Models via k-Sparse Attention

• Authors: Shuyan Huang, Zitao Liu, Xiangyu Zhao, Weiqi Luo, Jian Weng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17097
• Pdf URL: https://arxiv.org/pdf/2407.17097
• Copy Paste: [[2407.17097]] Towards Robust Knowledge Tracing Models via k-Sparse Attention(https://arxiv.org/abs/2407.17097)
• Keywords: robust
• Abstract: Knowledge tracing (KT) is the problem of predicting students' future performance based on their historical interaction sequences. With the advanced capability of capturing contextual long-term dependency, attention mechanism becomes one of the essential components in many deep learning based KT (DLKT) models. In spite of the impressive performance achieved by these attentional DLKT models, many of them are often vulnerable to run the risk of overfitting, especially on small-scale educational datasets. Therefore, in this paper, we propose \textsc{sparseKT}, a simple yet effective framework to improve the robustness and generalization of the attention based DLKT approaches. Specifically, we incorporate a k-selection module to only pick items with the highest attention scores. We propose two sparsification heuristics : (1) soft-thresholding sparse attention and (2) top-$K$ sparse attention. We show that our \textsc{sparseKT} is able to help attentional KT models get rid of irrelevant student interactions and have comparable predictive performance when compared to 11 state-of-the-art KT models on three publicly available real-world educational datasets. To encourage reproducible research, we make our data and code publicly available at \url{this https URL}\footnote{We merged our model to the \textsc{pyKT} benchmark at \url{this https URL}.}.

Title: PiPa++: Towards Unification of Domain Adaptive Semantic Segmentation via Self-supervised Learning

• Authors: Mu Chen, Zhedong Zheng, Yi Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17101
• Pdf URL: https://arxiv.org/pdf/2407.17101
• Copy Paste: [[2407.17101]] PiPa++: Towards Unification of Domain Adaptive Semantic Segmentation via Self-supervised Learning(https://arxiv.org/abs/2407.17101)
• Keywords: robust, segmentation
• Abstract: Unsupervised domain adaptive segmentation aims to improve the segmentation accuracy of models on target domains without relying on labeled data from those domains. This approach is crucial when labeled target domain data is scarce or unavailable. It seeks to align the feature representations of the source domain (where labeled data is available) and the target domain (where only unlabeled data is present), thus enabling the model to generalize well to the target domain. Current image- and video-level domain adaptation have been addressed using different and specialized frameworks, training strategies and optimizations despite their underlying connections. In this paper, we propose a unified framework PiPa++, which leverages the core idea of ``comparing'' to (1) explicitly encourage learning of discriminative pixel-wise features with intraclass compactness and inter-class separability, (2) promote the robust feature learning of the identical patch against different contexts or fluctuations, and (3) enable the learning of temporal continuity under dynamic environments. With the designed task-smart contrastive sampling strategy, PiPa++ enables the mining of more informative training samples according to the task demand. Extensive experiments demonstrate the effectiveness of our method on both image-level and video-level domain adaption benchmarks. Moreover, the proposed method is compatible with other UDA approaches to further improve the performance without introducing extra parameters.

Title: A Self-Supervised Image Registration Approach for Measuring Local Response Patterns in Metastatic Ovarian Cancer

• Authors: Inês P. Machado, Anna Reithmeir, Fryderyk Kogl, Leonardo Rundo, Gabriel Funingana, Marika Reinius, Gift Mungmeeprued, Zeyu Gao, Cathal McCague, Eric Kerfoot, Ramona Woitek, Evis Sala, Yangming Ou, James Brenton, Julia Schnabel, Mireia Crispin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17114
• Pdf URL: https://arxiv.org/pdf/2407.17114
• Copy Paste: [[2407.17114]] A Self-Supervised Image Registration Approach for Measuring Local Response Patterns in Metastatic Ovarian Cancer(https://arxiv.org/abs/2407.17114)
• Keywords: extraction
• Abstract: High-grade serous ovarian carcinoma (HGSOC) is characterised by significant spatial and temporal heterogeneity, typically manifesting at an advanced metastatic stage. A major challenge in treating advanced HGSOC is effectively monitoring localised change in tumour burden across multiple sites during neoadjuvant chemotherapy (NACT) and predicting long-term pathological response and overall patient survival. In this work, we propose a self-supervised deformable image registration algorithm that utilises a general-purpose image encoder for image feature extraction to co-register contrast-enhanced computerised tomography scan images acquired before and after neoadjuvant chemotherapy. This approach addresses challenges posed by highly complex tumour deformations and longitudinal lesion matching during treatment. Localised tumour changes are calculated using the Jacobian determinant maps of the registration deformation at multiple disease sites and their macroscopic areas, including hypo-dense (i.e., cystic/necrotic), hyper-dense (i.e., calcified), and intermediate density (i.e., soft tissue) portions. A series of experiments is conducted to understand the role of a general-purpose image encoder and its application in quantifying change in tumour burden during neoadjuvant chemotherapy in HGSOC. This work is the first to demonstrate the feasibility of a self-supervised image registration approach in quantifying NACT-induced localised tumour changes across the whole disease burden of patients with complex multi-site HGSOC, which could be used as a potential marker for ovarian cancer patient's long-term pathological response and survival.

Title: EverAdapt: Continuous Adaptation for Dynamic Machine Fault Diagnosis Environments

• Authors: Edward, Mohamed Ragab, Yuecong Xu, Min Wu, Yuecong Xu, Zhenghua Chen, Abdulla Alseiari, Xiaoli Li
• Subjects: cs.LG, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2407.17117
• Pdf URL: https://arxiv.org/pdf/2407.17117
• Copy Paste: [[2407.17117]] EverAdapt: Continuous Adaptation for Dynamic Machine Fault Diagnosis Environments(https://arxiv.org/abs/2407.17117)
• Keywords: robust
• Abstract: Unsupervised Domain Adaptation (UDA) has emerged as a key solution in data-driven fault diagnosis, addressing domain shift where models underperform in changing environments. However, under the realm of continually changing environments, UDA tends to underperform on previously seen domains when adapting to new ones - a problem known as catastrophic forgetting. To address this limitation, we introduce the EverAdapt framework, specifically designed for continuous model adaptation in dynamic environments. Central to EverAdapt is a novel Continual Batch Normalization (CBN), which leverages source domain statistics as a reference point to standardize feature representations across domains. EverAdapt not only retains statistical information from previous domains but also adapts effectively to new scenarios. Complementing CBN, we design a class-conditional domain alignment module for effective integration of target domains, and a Sample-efficient Replay strategy to reinforce memory retention. Experiments on real-world datasets demonstrate EverAdapt superiority in maintaining robust fault diagnosis in dynamic environments. Our code is available: this https URL

Title: Behavioral Testing: Can Large Language Models Implicitly Resolve Ambiguous Entities?

• Authors: Anastasiia Sedova, Robert Litschko, Diego Frassinelli, Benjamin Roth, Barbara Plank
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17125
• Pdf URL: https://arxiv.org/pdf/2407.17125
• Copy Paste: [[2407.17125]] Behavioral Testing: Can Large Language Models Implicitly Resolve Ambiguous Entities?(https://arxiv.org/abs/2407.17125)
• Keywords: large language model
• Abstract: One of the major aspects contributing to the striking performance of large language models (LLMs) is the vast amount of factual knowledge accumulated during pre-training. Yet, many LLMs suffer from self-inconsistency, which raises doubts about their trustworthiness and reliability. In this paper, we focus on entity type ambiguity and analyze current state-of-the-art LLMs for their proficiency and consistency in applying their factual knowledge when prompted for entities under ambiguity. To do so, we propose an evaluation protocol that disentangles knowing from applying knowledge, and test state-of-the-art LLMs on 49 entities. Our experiments reveal that LLMs perform poorly with ambiguous prompts, achieving only 80% accuracy. Our results further demonstrate systematic discrepancies in LLM behavior and their failure to consistently apply information, indicating that the models can exhibit knowledge without being able to utilize it, significant biases for preferred readings, as well as self inconsistencies. Our study highlights the importance of handling entity ambiguity in future for more trustworthy LLMs

Title: SDoH-GPT: Using Large Language Models to Extract Social Determinants of Health (SDoH)

• Authors: Bernardo Consoli, Xizhi Wu, Song Wang, Xinyu Zhao, Yanshan Wang, Justin Rousseau, Tom Hartvigsen, Li Shen, Huanmei Wu, Yifan Peng, Qi Long, Tianlong Chen, Ying Ding
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17126
• Pdf URL: https://arxiv.org/pdf/2407.17126
• Copy Paste: [[2407.17126]] SDoH-GPT: Using Large Language Models to Extract Social Determinants of Health (SDoH)(https://arxiv.org/abs/2407.17126)
• Keywords: robust, large language model
• Abstract: Extracting social determinants of health (SDoH) from unstructured medical notes depends heavily on labor-intensive annotations, which are typically task-specific, hampering reusability and limiting sharing. In this study we introduced SDoH-GPT, a simple and effective few-shot Large Language Model (LLM) method leveraging contrastive examples and concise instructions to extract SDoH without relying on extensive medical annotations or costly human intervention. It achieved tenfold and twentyfold reductions in time and cost respectively, and superior consistency with human annotators measured by Cohen's kappa of up to 0.92. The innovative combination of SDoH-GPT and XGBoost leverages the strengths of both, ensuring high accuracy and computational efficiency while consistently maintaining 0.90+ AUROC scores. Testing across three distinct datasets has confirmed its robustness and accuracy. This study highlights the potential of leveraging LLMs to revolutionize medical note classification, demonstrating their capability to achieve highly accurate classifications with significantly reduced time and cost.

Title: RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer

• Authors: Wenyu Lv, Yian Zhao, Qinyao Chang, Kui Huang, Guanzhong Wang, Yi Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17140
• Pdf URL: https://arxiv.org/pdf/2407.17140
• Copy Paste: [[2407.17140]] RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer(https://arxiv.org/abs/2407.17140)
• Keywords: extraction, transformer
• Abstract: In this report, we present RT-DETRv2, an improved Real-Time DEtection TRansformer (RT-DETR). RT-DETRv2 builds upon the previous state-of-the-art real-time detector, RT-DETR, and opens up a set of bag-of-freebies for flexibility and practicality, as well as optimizing the training strategy to achieve enhanced performance. To improve the flexibility, we suggest setting a distinct number of sampling points for features at different scales in the deformable attention to achieve selective multi-scale feature extraction by the decoder. To enhance practicality, we propose an optional discrete sampling operator to replace the grid_sample operator that is specific to RT-DETR compared to YOLOs. This removes the deployment constraints typically associated with DETRs. For the training strategy, we propose dynamic data augmentation and scale-adaptive hyperparameters customization to improve performance without loss of speed. Source code and pre-trained models will be available at this https URL.

Title: SimCT: A Simple Consistency Test Protocol in LLMs Development Lifecycle

• Authors: Fufangchen Zhao, Guoqiang Jin, Rui Zhao, Jiangheng Huang, Fei Tan
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2407.17150
• Pdf URL: https://arxiv.org/pdf/2407.17150
• Copy Paste: [[2407.17150]] SimCT: A Simple Consistency Test Protocol in LLMs Development Lifecycle(https://arxiv.org/abs/2407.17150)
• Keywords: large language model
• Abstract: In this work, we report our efforts to advance the standard operation procedure of developing Large Language Models (LLMs) or LLMs-based systems or services in industry. We introduce the concept of Large Language Model Development Lifecycle (LDLC) and then highlight the importance of consistency test in ensuring the delivery quality. The principled solution of consistency test, however, is usually overlooked by industrial practitioners and not urgent in academia, and current practical solutions are insufficiently rigours and labor-intensive. We thus propose a simple yet effective consistency test protocol, named SimCT. SimCT is mainly to proactively check the consistency across different development stages of "bare metal" LLMs or associated services without accessing the model artifacts, in an attempt to expedite the delivery by reducing the back-and-forth alignment communications among multiple teams involved in different development stages. Specifically, SimCT encompasses response-wise and model-wise tests. We implement the protocol with LightGBM and Student's t-test for two components respectively, and perform extensive experiments to substantiate the effectiveness of SimCT and the involved components.

Title: FIIH: Fully Invertible Image Hiding for Secure and Robust

• Authors: Lang Huang, Lin Huo, Zheng Gan, Xinrong He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17155
• Pdf URL: https://arxiv.org/pdf/2407.17155
• Copy Paste: [[2407.17155]] FIIH: Fully Invertible Image Hiding for Secure and Robust(https://arxiv.org/abs/2407.17155)
• Keywords: secure, security, robust
• Abstract: Image hiding is the study of techniques for covert storage and transmission, which embeds a secret image into a container image and generates stego image to make it similar in appearance to a normal image. However, existing image hiding methods have a serious problem that the hiding and revealing process cannot be fully invertible, which results in the revealing network not being able to recover the secret image losslessly, which makes it impossible to simultaneously achieve high fidelity and secure transmission of the secret image in an insecure network environment. To solve this problem,this paper proposes a fully invertible image hiding architecture based on invertible neural network,aiming to realize invertible hiding of secret images,which is invertible on both data and network. Based on this ingenious architecture, the method can withstand deep learning based image steganalysis. In addition, we propose a new method for enhancing the robustness of stego images after interference during transmission. Experiments demonstrate that the FIIH proposed in this paper significantly outperforms other state-of-the-art image hiding methods in hiding a single image, and also significantly outperforms other state-of-the-art methods in robustness and security.

Title: Establishing Truly Causal Relationship Between Whole Slide Image Predictions and Diagnostic Evidence Subregions in Deep Learning

• Authors: Tianhang Nan, Yong Ding, Hao Quan, Deliang Li, Mingchen Zou, Xiaoyu Cui
• Subjects: cs.CV, q-bio.TO
• Abstract URL: https://arxiv.org/abs/2407.17157
• Pdf URL: https://arxiv.org/pdf/2407.17157
• Copy Paste: [[2407.17157]] Establishing Truly Causal Relationship Between Whole Slide Image Predictions and Diagnostic Evidence Subregions in Deep Learning(https://arxiv.org/abs/2407.17157)
• Keywords: interpretability
• Abstract: In the field of deep learning-driven Whole Slide Image (WSI) classification, Multiple Instance Learning (MIL) has gained significant attention due to its ability to be trained using only slide-level diagnostic labels. Previous MIL researches have primarily focused on enhancing feature aggregators for globally analyzing WSIs, but overlook a causal relationship in diagnosis: model's prediction should ideally stem solely from regions of the image that contain diagnostic evidence (such as tumor cells), which usually occupy relatively small areas. To address this limitation and establish the truly causal relationship between model predictions and diagnostic evidence regions, we propose Causal Inference Multiple Instance Learning (CI-MIL). CI-MIL integrates feature distillation with a novel patch decorrelation mechanism, employing a two-stage causal inference approach to distill and process patches with high diagnostic value. Initially, CI-MIL leverages feature distillation to identify patches likely containing tumor cells and extracts their corresponding feature representations. These features are then mapped to random Fourier feature space, where a learnable weighting scheme is employed to minimize inter-feature correlations, effectively reducing redundancy from homogenous patches and mitigating data bias. These processes strengthen the causal relationship between model predictions and diagnostically relevant regions, making the prediction more direct and reliable. Experimental results demonstrate that CI-MIL outperforms state-of-the-art methods. Additionally, CI-MIL exhibits superior interpretability, as its selected regions demonstrate high consistency with ground truth annotations, promising more reliable diagnostic assistance for pathologists.

Title: Context-aware Multi-task Learning for Pedestrian Intent and Trajectory Prediction

• Authors: Farzeen Munir, Tomasz Piotr Kucner
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2407.17162
• Pdf URL: https://arxiv.org/pdf/2407.17162
• Copy Paste: [[2407.17162]] Context-aware Multi-task Learning for Pedestrian Intent and Trajectory Prediction(https://arxiv.org/abs/2407.17162)
• Keywords: robust
• Abstract: The advancement of socially-aware autonomous vehicles hinges on precise modeling of human behavior. Within this broad paradigm, the specific challenge lies in accurately predicting pedestrian's trajectory and intention. Traditional methodologies have leaned heavily on historical trajectory data, frequently overlooking vital contextual cues such as pedestrian-specific traits and environmental factors. Furthermore, there's a notable knowledge gap as trajectory and intention prediction have largely been approached as separate problems, despite their mutual dependence. To bridge this gap, we introduce PTINet (Pedestrian Trajectory and Intention Prediction Network), which jointly learns the trajectory and intention prediction by combining past trajectory observations, local contextual features (individual pedestrian behaviors), and global features (signs, markings etc.). The efficacy of our approach is evaluated on widely used public datasets: JAAD and PIE, where it has demonstrated superior performance over existing state-of-the-art models in trajectory and intention prediction. The results from our experiments and ablation studies robustly validate PTINet's effectiveness in jointly exploring intention and trajectory prediction for pedestrian behaviour modelling. The experimental evaluation indicates the advantage of using global and local contextual features for pedestrian trajectory and intention prediction. The effectiveness of PTINet in predicting pedestrian behavior paves the way for the development of automated systems capable of seamlessly interacting with pedestrians in urban settings.

Title: Robust Deep Hawkes Process under Label Noise of Both Event and Occurrence

• Authors: Xiaoyu Tan, Bin Li, Xihe Qiu, Jingjing Huang, Yinghui Xu, Wei Chu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17164
• Pdf URL: https://arxiv.org/pdf/2407.17164
• Copy Paste: [[2407.17164]] Robust Deep Hawkes Process under Label Noise of Both Event and Occurrence(https://arxiv.org/abs/2407.17164)
• Keywords: robust
• Abstract: Integrating deep neural networks with the Hawkes process has significantly improved predictive capabilities in finance, health informatics, and information technology. Nevertheless, these models often face challenges in real-world settings, particularly due to substantial label noise. This issue is of significant concern in the medical field, where label noise can arise from delayed updates in electronic medical records or misdiagnoses, leading to increased prediction risks. Our research indicates that deep Hawkes process models exhibit reduced robustness when dealing with label noise, particularly when it affects both event types and timing. To address these challenges, we first investigate the influence of label noise in approximated intensity functions and present a novel framework, the Robust Deep Hawkes Process (RDHP), to overcome the impact of label noise on the intensity function of Hawkes models, considering both the events and their occurrences. We tested RDHP using multiple open-source benchmarks with synthetic noise and conducted a case study on obstructive sleep apnea-hypopnea syndrome (OSAHS) in a real-world setting with inherent label noise. The results demonstrate that RDHP can effectively perform classification and regression tasks, even in the presence of noise related to events and their timing. To the best of our knowledge, this is the first study to successfully address both event and time label noise in deep Hawkes process models, offering a promising solution for medical applications, specifically in diagnosing OSAHS.

Title: Explainable Artificial Intelligence Techniques for Irregular Temporal Classification of Multidrug Resistance Acquisition in Intensive Care Unit Patients

• Authors: Óscar Escudero-Arnanz, Cristina Soguero-Ruiz, Joaquín Álvarez-Rodríguez, Antonio G. Marques
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17165
• Pdf URL: https://arxiv.org/pdf/2407.17165
• Copy Paste: [[2407.17165]] Explainable Artificial Intelligence Techniques for Irregular Temporal Classification of Multidrug Resistance Acquisition in Intensive Care Unit Patients(https://arxiv.org/abs/2407.17165)
• Keywords: interpretability
• Abstract: Antimicrobial Resistance represents a significant challenge in the Intensive Care Unit (ICU), where patients are at heightened risk of Multidrug-Resistant (MDR) infections-pathogens resistant to multiple antimicrobial agents. This study introduces a novel methodology that integrates Gated Recurrent Units (GRUs) with advanced intrinsic and post-hoc interpretability techniques for detecting the onset of MDR in patients across time. Within interpretability methods, we propose Explainable Artificial Intelligence (XAI) approaches to handle irregular Multivariate Time Series (MTS), introducing Irregular Time Shapley Additive Explanations (IT-SHAP), a modification of Shapley Additive Explanations designed for irregular MTS with Recurrent Neural Networks focused on temporal outputs. Our methodology aims to identify specific risk factors associated with MDR in ICU patients. GRU with Hadamard's attention demonstrated high initial specificity and increasing sensitivity over time, correlating with increased nosocomial infection risks during prolonged ICU stays. XAI analysis, enhanced by Hadamard attention and IT-SHAP, identified critical factors such as previous non-resistant cultures, specific antibiotic usage patterns, and hospital environment dynamics. These insights suggest that early detection of at-risk patients can inform interventions such as preventive isolation and customized treatments, significantly improving clinical outcomes. The proposed GRU model for temporal classification achieved an average Receiver Operating Characteristic Area Under the Curve of 78.27 +- 1.26 over time, indicating strong predictive performance. In summary, this study highlights the clinical utility of our methodology, which combines predictive accuracy with interpretability, thereby facilitating more effective healthcare interventions by professionals.

Title: Domain Generalized Recaptured Screen Image Identification Using SWIN Transformer

• Authors: Preeti Mehta, Aman Sagar, Suchi Kumari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17170
• Pdf URL: https://arxiv.org/pdf/2407.17170
• Copy Paste: [[2407.17170]] Domain Generalized Recaptured Screen Image Identification Using SWIN Transformer(https://arxiv.org/abs/2407.17170)
• Keywords: attack, transformer
• Abstract: An increasing number of classification approaches have been developed to address the issue of image rebroadcast and recapturing, a standard attack strategy in insurance frauds, face spoofing, and video piracy. However, most of them neglected scale variations and domain generalization scenarios, performing poorly in instances involving domain shifts, typically made worse by inter-domain and cross-domain scale variances. To overcome these issues, we propose a cascaded data augmentation and SWIN transformer domain generalization framework (DAST-DG) in the current research work Initially, we examine the disparity in dataset representation. A feature generator is trained to make authentic images from various domains indistinguishable. This process is then applied to recaptured images, creating a dual adversarial learning setup. Extensive experiments demonstrate that our approach is practical and surpasses state-of-the-art methods across different databases. Our model achieves an accuracy of approximately 82\% with a precision of 95\% on high-variance datasets.

Title: Unpaired Photo-realistic Image Deraining with Energy-informed Diffusion Model

• Authors: Yuanbo Wen, Tao Gao, Ting Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17193
• Pdf URL: https://arxiv.org/pdf/2407.17193
• Copy Paste: [[2407.17193]] Unpaired Photo-realistic Image Deraining with Energy-informed Diffusion Model(https://arxiv.org/abs/2407.17193)
• Keywords: diffusion
• Abstract: Existing unpaired image deraining approaches face challenges in accurately capture the distinguishing characteristics between the rainy and clean domains, resulting in residual degradation and color distortion within the reconstructed images. To this end, we propose an energy-informed diffusion model for unpaired photo-realistic image deraining (UPID-EDM). Initially, we delve into the intricate visual-language priors embedded within the contrastive language-image pre-training model (CLIP), and demonstrate that the CLIP priors aid in the discrimination of rainy and clean images. Furthermore, we introduce a dual-consistent energy function (DEF) that retains the rain-irrelevant characteristics while eliminating the rain-relevant features. This energy function is trained by the non-corresponding rainy and clean images. In addition, we employ the rain-relevance discarding energy function (RDEF) and the rain-irrelevance preserving energy function (RPEF) to direct the reverse sampling procedure of a pre-trained diffusion model, effectively removing the rain streaks while preserving the image contents. Extensive experiments demonstrate that our energy-informed model surpasses the existing unpaired learning approaches in terms of both supervised and no-reference metrics.

Title: ALPI: Auto-Labeller with Proxy Injection for 3D Object Detection using 2D Labels Only

• Authors: Saad Lahlali, Nicolas Granger, Hervé Le Borgne, Quoc-Cuong Pham
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17197
• Pdf URL: https://arxiv.org/pdf/2407.17197
• Copy Paste: [[2407.17197]] ALPI: Auto-Labeller with Proxy Injection for 3D Object Detection using 2D Labels Only(https://arxiv.org/abs/2407.17197)
• Keywords: robust
• Abstract: 3D object detection plays a crucial role in various applications such as autonomous vehicles, robotics and augmented reality. However, training 3D detectors requires a costly precise annotation, which is a hindrance to scaling annotation to large datasets. To address this challenge, we propose a weakly supervised 3D annotator that relies solely on 2D bounding box annotations from images, along with size priors. One major problem is that supervising a 3D detection model using only 2D boxes is not reliable due to ambiguities between different 3D poses and their identical 2D projection. We introduce a simple yet effective and generic solution: we build 3D proxy objects with annotations by construction and add them to the training dataset. Our method requires only size priors to adapt to new classes. To better align 2D supervision with 3D detection, our method ensures depth invariance with a novel expression of the 2D losses. Finally, to detect more challenging instances, our annotator follows an offline pseudo-labelling scheme which gradually improves its 3D pseudo-labels. Extensive experiments on the KITTI dataset demonstrate that our method not only performs on-par or above previous works on the Car category, but also achieves performance close to fully supervised methods on more challenging classes. We further demonstrate the effectiveness and robustness of our method by being the first to experiment on the more challenging nuScenes dataset. We additionally propose a setting where weak labels are obtained from a 2D detector pre-trained on MS-COCO instead of human annotations.

Title: Graph Neural Networks: A suitable Alternative to MLPs in Latent 3D Medical Image Classification?

• Authors: Johannes Kiechle, Daniel M. Lang, Stefan M. Fischer, Lina Felsner, Jan C. Peeken, Julia A. Schnabel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17219
• Pdf URL: https://arxiv.org/pdf/2407.17219
• Copy Paste: [[2407.17219]] Graph Neural Networks: A suitable Alternative to MLPs in Latent 3D Medical Image Classification?(https://arxiv.org/abs/2407.17219)
• Keywords: robust, transformer
• Abstract: Recent studies have underscored the capabilities of natural imaging foundation models to serve as powerful feature extractors, even in a zero-shot setting for medical imaging data. Most commonly, a shallow multi-layer perceptron (MLP) is appended to the feature extractor to facilitate end-to-end learning and downstream prediction tasks such as classification, thus representing the de facto standard. However, as graph neural networks (GNNs) have become a practicable choice for various tasks in medical research in the recent past, we direct attention to the question of how effective GNNs are compared to MLP prediction heads for the task of 3D medical image classification, proposing them as a potential alternative. In our experiments, we devise a subject-level graph for each volumetric dataset instance. Therein latent representations of all slices in the volume, encoded through a DINOv2 pretrained vision transformer (ViT), constitute the nodes and their respective node features. We use public datasets to compare the classification heads numerically and evaluate various graph construction and graph convolution methods in our experiments. Our findings show enhancements of the GNN in classification performance and substantial improvements in runtime compared to an MLP prediction head. Additional robustness evaluations further validate the promising performance of the GNN, promoting them as a suitable alternative to traditional MLP classification heads. Our code is publicly available at: this https URL

Title: Sublinear Regret for An Actor-Critic Algorithm in Continuous-Time Linear-Quadratic Reinforcement Learning

• Authors: Yilie Huang, Yanwei Jia, Xun Yu Zhou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17226
• Pdf URL: https://arxiv.org/pdf/2407.17226
• Copy Paste: [[2407.17226]] Sublinear Regret for An Actor-Critic Algorithm in Continuous-Time Linear-Quadratic Reinforcement Learning(https://arxiv.org/abs/2407.17226)
• Keywords: diffusion
• Abstract: We study reinforcement learning (RL) for a class of continuous-time linear-quadratic (LQ) control problems for diffusions where volatility of the state processes depends on both state and control variables. We apply a model-free approach that relies neither on knowledge of model parameters nor on their estimations, and devise an actor-critic algorithm to learn the optimal policy parameter directly. Our main contributions include the introduction of a novel exploration schedule and a regret analysis of the proposed algorithm. We provide the convergence rate of the policy parameter to the optimal one, and prove that the algorithm achieves a regret bound of $O(N^{\frac{3}{4}})$ up to a logarithmic factor. We conduct a simulation study to validate the theoretical results and demonstrate the effectiveness and reliability of the proposed algorithm. We also perform numerical comparisons between our method and those of the recent model-based stochastic LQ RL studies adapted to the state- and control-dependent volatility setting, demonstrating a better performance of the former in terms of regret bounds.

Title: LPGen: Enhancing High-Fidelity Landscape Painting Generation through Diffusion Model

• Authors: Wanggong Yang, Xiaona Wang, Yingrui Qiu, Yifei Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17229
• Pdf URL: https://arxiv.org/pdf/2407.17229
• Copy Paste: [[2407.17229]] LPGen: Enhancing High-Fidelity Landscape Painting Generation through Diffusion Model(https://arxiv.org/abs/2407.17229)
• Keywords: diffusion
• Abstract: Generating landscape paintings expands the possibilities of artistic creativity and imagination. Traditional landscape painting methods involve using ink or colored ink on rice paper, which requires substantial time and effort. These methods are susceptible to errors and inconsistencies and lack precise control over lines and colors. This paper presents LPGen, a high-fidelity, controllable model for landscape painting generation, introducing a novel multi-modal framework that integrates image prompts into the diffusion model. We extract its edges and contours by computing canny edges from the target landscape image. These, along with natural language text prompts and drawing style references, are fed into the latent diffusion model as conditions. We implement a decoupled cross-attention strategy to ensure compatibility between image and text prompts, facilitating multi-modal image generation. A decoder generates the final image. Quantitative and qualitative analyses demonstrate that our method outperforms existing approaches in landscape painting generation and exceeds the current state-of-the-art. The LPGen network effectively controls the composition and color of landscape paintings, generates more accurate images, and supports further research in deep learning-based landscape painting generation.

Title: Improving ICD coding using Chapter based Named Entities and Attentional Models

• Authors: Abhijith R. Beeravolu, Mirjam Jonkman, Sami Azam, Friso De Boer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17230
• Pdf URL: https://arxiv.org/pdf/2407.17230
• Copy Paste: [[2407.17230]] Improving ICD coding using Chapter based Named Entities and Attentional Models(https://arxiv.org/abs/2407.17230)
• Keywords: interpretability, transformer
• Abstract: Recent advancements in natural language processing (NLP) have led to automation in various domains. However, clinical NLP often relies on benchmark datasets that may not reflect real-world scenarios accurately. Automatic ICD coding, a vital NLP task, typically uses outdated and imbalanced datasets like MIMIC-III, with existing methods yielding micro-averaged F1 scores between 0.4 and 0.7 due to many false positives. Our research introduces an enhanced approach to ICD coding that improves F1 scores by using chapter-based named entities and attentional models. This method categorizes discharge summaries into ICD-9 Chapters and develops attentional models with chapter-specific data, eliminating the need to consider external data for code identification. For categorization, we use Chapter-IV to de-bias and influence key entities and weights without neural networks, creating accurate thresholds and providing interpretability for human validation. Post-validation, we develop attentional models for three frequent and three non-frequent codes from Chapter-IV using Bidirectional-Gated Recurrent Units (GRUs) with Attention and Transformer with Multi-head Attention architectures. The average Micro-F1 scores of 0.79 and 0.81 from these models demonstrate significant performance improvements in ICD coding.

Title: Channel-Aware Low-Rank Adaptation in Time Series Forecasting

• Authors: Tong Nie, Yuewen Mei, Guoyang Qin, Jian Sun, Wei Ma
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17246
• Pdf URL: https://arxiv.org/pdf/2407.17246
• Copy Paste: [[2407.17246]] Channel-Aware Low-Rank Adaptation in Time Series Forecasting(https://arxiv.org/abs/2407.17246)
• Keywords: robust
• Abstract: The balance between model capacity and generalization has been a key focus of recent discussions in long-term time series forecasting. Two representative channel strategies are closely associated with model expressivity and robustness, including channel independence (CI) and channel dependence (CD). The former adopts individual channel treatment and has been shown to be more robust to distribution shifts, but lacks sufficient capacity to model meaningful channel interactions. The latter is more expressive for representing complex cross-channel dependencies, but is prone to overfitting. To balance the two strategies, we present a channel-aware low-rank adaptation method to condition CD models on identity-aware individual components. As a plug-in solution, it is adaptable for a wide range of backbone architectures. Extensive experiments show that it can consistently and significantly improve the performance of both CI and CD models with demonstrated efficiency and flexibility. The code is available at this https URL.

Title: Critical Infrastructure Security: Penetration Testing and Exploit Development Perspectives

• Authors: Papa Kobina Orleans-Bosomtwe
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.17256
• Pdf URL: https://arxiv.org/pdf/2407.17256
• Copy Paste: [[2407.17256]] Critical Infrastructure Security: Penetration Testing and Exploit Development Perspectives(https://arxiv.org/abs/2407.17256)
• Keywords: security, defense, robust
• Abstract: Critical infrastructure refers to essential physical and cyber systems vital to the functioning and stability of societies and economies. These systems include key sectors such as healthcare, energy, and water supply, which are crucial for societal and economic stability and are increasingly becoming prime targets for malicious actors, including state-sponsored hackers, seeking to disrupt national security and economic stability. This paper reviews literature on critical infrastructure security, focusing on penetration testing and exploit development. It explores four main questions: the characteristics of critical infrastructure, the role and challenges of penetration testing, methodologies of exploit development, and the contribution of these practices to security and resilience. The findings of this paper reveal inherent vulnerabilities in critical infrastructure and sophisticated threats posed by cyber adversaries. Penetration testing is highlighted as a vital tool for identifying and addressing security weaknesses, allowing organizations to fortify their defenses. Additionally, understanding exploit development helps anticipate and mitigate potential threats, leading to more robust security measures. The review underscores the necessity of continuous and proactive security assessments, advocating for integrating penetration testing and exploit development into regular security protocols. By doing so, organizations can preemptively identify and mitigate risks, enhancing the overall resilience of critical infrastructure. The paper concludes by emphasizing the need for ongoing research and collaboration between the public and private sectors to develop innovative solutions for the evolving cyber threat landscape. This comprehensive review aims to provide a foundational understanding of critical infrastructure security and guide future research and practices.

Title: Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentation

• Authors: Hyunwoo Yu, Yubin Cho, Beoungwoo Kang, Seunghun Moon, Kyeongbo Kong, Suk-Ju Kang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17261
• Pdf URL: https://arxiv.org/pdf/2407.17261
• Copy Paste: [[2407.17261]] Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentation(https://arxiv.org/abs/2407.17261)
• Keywords: transformer, segmentation
• Abstract: We present an Encoder-Decoder Attention Transformer, EDAFormer, which consists of the Embedding-Free Transformer (EFT) encoder and the all-attention decoder leveraging our Embedding-Free Attention (EFA) structure. The proposed EFA is a novel global context modeling mechanism that focuses on functioning the global non-linearity, not the specific roles of the query, key and value. For the decoder, we explore the optimized structure for considering the globality, which can improve the semantic segmentation performance. In addition, we propose a novel Inference Spatial Reduction (ISR) method for the computational efficiency. Different from the previous spatial reduction attention methods, our ISR method further reduces the key-value resolution at the inference phase, which can mitigate the computation-performance trade-off gap for the efficient semantic segmentation. Our EDAFormer shows the state-of-the-art performance with the efficient computation compared to the existing transformer-based semantic segmentation models in three public benchmarks, including ADE20K, Cityscapes and COCO-Stuff. Furthermore, our ISR method reduces the computational cost by up to 61% with minimal mIoU performance degradation on Cityscapes dataset. The code is available at this https URL.

Title: SCIsegV2: A Universal Tool for Segmentation of Intramedullary Lesions in Spinal Cord Injury

• Authors: Enamundram Naga Karthik, Jan Valošek, Lynn Farner, Dario Pfyffer, Simon Schading-Sassenhausen, Anna Lebret, Gergely David, Andrew C. Smith, Kenneth A. Weber II, Maryam Seif, RHSCIR Network Imaging Group, Patrick Freund, Julien Cohen-Adad
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17265
• Pdf URL: https://arxiv.org/pdf/2407.17265
• Copy Paste: [[2407.17265]] SCIsegV2: A Universal Tool for Segmentation of Intramedullary Lesions in Spinal Cord Injury(https://arxiv.org/abs/2407.17265)
• Keywords: generative, segmentation
• Abstract: Spinal cord injury (SCI) is a devastating incidence leading to permanent paralysis and loss of sensory-motor functions potentially resulting in the formation of lesions within the spinal cord. Imaging biomarkers obtained from magnetic resonance imaging (MRI) scans can predict the functional recovery of individuals with SCI and help choose the optimal treatment strategy. Currently, most studies employ manual quantification of these MRI-derived biomarkers, which is a subjective and tedious task. In this work, we propose (i) a universal tool for the automatic segmentation of intramedullary SCI lesions, dubbed \texttt{SCIsegV2}, and (ii) a method to automatically compute the width of the tissue bridges from the segmented lesion. Tissue bridges represent the spared spinal tissue adjacent to the lesion, which is associated with functional recovery in SCI patients. The tool was trained and validated on a heterogeneous dataset from 7 sites comprising patients from different SCI phases (acute, sub-acute, and chronic) and etiologies (traumatic SCI, ischemic SCI, and degenerative cervical myelopathy). Tissue bridges quantified automatically did not significantly differ from those computed manually, suggesting that the proposed automatic tool can be used to derive relevant MRI biomarkers. \texttt{SCIsegV2} and the automatic tissue bridges computation are open-source and available in Spinal Cord Toolbox (v6.4 and above) via the \texttt{sct\_deepseg -task seg\_sc\_lesion\_t2w\_sci} and \texttt{sct\_analyze\_lesion} functions, respectively.

Title: Advanced Penetration Testing for Enhancing 5G Security

• Authors: Shari-Ann Smith-Haynes
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.17269
• Pdf URL: https://arxiv.org/pdf/2407.17269
• Copy Paste: [[2407.17269]] Advanced Penetration Testing for Enhancing 5G Security(https://arxiv.org/abs/2407.17269)
• Keywords: security, attack
• Abstract: Advances in fifth-generation (5G) networks enable unprecedented reliability, speed, and connectivity compared to previous mobile networks. These advancements can revolutionize various sectors by supporting applications requiring real-time data processing. However, the rapid deployment and integration of 5G networks bring security concerns that must be addressed to operate these infrastructures safely. This paper reviews penetration testing approaches for identifying security vulnerabilities in 5G networks. Penetration testing is an ethical hacking technique used to simulate a network's security posture in the event of cyberattacks. This review highlights the capabilities, advantages, and limitations of recent 5G-targeting security tools for penetration testing. It examines ways adversaries exploit vulnerabilities in 5G networks, covering tactics and strategies targeted at 5G features. A key topic explored is the comparison of penetration testing methods for 5G and earlier generations. The article delves into the unique characteristics of 5G, including massive MIMO, edge computing, and network slicing, and how these aspects require new penetration testing methods. Understanding these differences helps develop more effective security solutions tailored to 5G networks. Our research also indicates that 5G penetration testing should use a multithreaded approach for addressing current security challenges. Furthermore, this paper includes case studies illustrating practical challenges and limitations in real-world applications of penetration testing in 5G networks. A comparative analysis of penetration testing tools for 5G networks highlights their effectiveness in mitigating vulnerabilities, emphasizing the need for advanced security measures against evolving cyber threats in 5G deployment.

Title: Bridging Trust into the Blockchain: A Systematic Review on On-Chain Identity

• Authors: Awid Vaziry, Kaustabh Barman, Patrick Herbke
• Subjects: cs.CR, cs.CY, cs.DC
• Abstract URL: https://arxiv.org/abs/2407.17276
• Pdf URL: https://arxiv.org/pdf/2407.17276
• Copy Paste: [[2407.17276]] Bridging Trust into the Blockchain: A Systematic Review on On-Chain Identity(https://arxiv.org/abs/2407.17276)
• Keywords: privacy
• Abstract: The ongoing regulation of blockchain-based services and applications requires the identification of users who are issuing transactions on the blockchain. This systematic review explores the current status, identifies research gaps, and outlines future research directions for establishing trusted and privacy-compliant identities on the blockchain (on-chain identity). A systematic search term was applied across various scientific databases, collecting 2232 potentially relevant research papers. These papers were narrowed down in two methodologically executed steps to 98 and finally to 13 relevant sources. The relevant articles were then systematically analyzed based on a set of screening questions. The results of the selected studies have provided insightful findings on the mechanisms of on-chain identities. On-chain identities are established using zero-knowledge proofs, public key infrastructure/certificates, and web of trust approaches. The technologies and architectures used by the authors are also highlighted. Trust has emerged as a key research gap, manifesting in two ways: firstly, a gap in how to trust the digital identity representation of a physical human; secondly, a gap in how to trust identity providers that issue identity confirmations on-chain. Potential future research avenues are suggested to help fill the current gaps in establishing trust and on-chain identities.

Title: A Novel Two-Step Fine-Tuning Pipeline for Cold-Start Active Learning in Text Classification Tasks

• Authors: Fabiano Belém, Washington Cunha, Celso França, Claudio Andrade, Leonardo Rocha, Marcos André Gonçalves
• Subjects: cs.LG, cs.DB, cs.IR
• Abstract URL: https://arxiv.org/abs/2407.17284
• Pdf URL: https://arxiv.org/pdf/2407.17284
• Copy Paste: [[2407.17284]] A Novel Two-Step Fine-Tuning Pipeline for Cold-Start Active Learning in Text Classification Tasks(https://arxiv.org/abs/2407.17284)
• Keywords: robust
• Abstract: This is the first work to investigate the effectiveness of BERT-based contextual embeddings in active learning (AL) tasks on cold-start scenarios, where traditional fine-tuning is infeasible due to the absence of labeled data. Our primary contribution is the proposal of a more robust fine-tuning pipeline - DoTCAL - that diminishes the reliance on labeled data in AL using two steps: (1) fully leveraging unlabeled data through domain adaptation of the embeddings via masked language modeling and (2) further adjusting model weights using labeled data selected by AL. Our evaluation contrasts BERT-based embeddings with other prevalent text representation paradigms, including Bag of Words (BoW), Latent Semantic Indexing (LSI), and FastText, at two critical stages of the AL process: instance selection and classification. Experiments conducted on eight ATC benchmarks with varying AL budgets (number of labeled instances) and number of instances (about 5,000 to 300,000) demonstrate DoTCAL's superior effectiveness, achieving up to a 33% improvement in Macro-F1 while reducing labeling efforts by half compared to the traditional one-step method. We also found that in several tasks, BoW and LSI (due to information aggregation) produce results superior (up to 59% ) to BERT, especially in low-budget scenarios and hard-to-classify tasks, which is quite surprising.

Title: Physical Adversarial Attack on Monocular Depth Estimation via Shape-Varying Patches

• Authors: Chenxing Zhao, Yang Li, Shihao Wu, Wenyi Tan, Shuangju Zhou, Quan Pan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17312
• Pdf URL: https://arxiv.org/pdf/2407.17312
• Copy Paste: [[2407.17312]] Physical Adversarial Attack on Monocular Depth Estimation via Shape-Varying Patches(https://arxiv.org/abs/2407.17312)
• Keywords: attack
• Abstract: Adversarial attacks against monocular depth estimation (MDE) systems pose significant challenges, particularly in safety-critical applications such as autonomous driving. Existing patch-based adversarial attacks for MDE are confined to the vicinity of the patch, making it difficult to affect the entire target. To address this limitation, we propose a physics-based adversarial attack on monocular depth estimation, employing a framework called Attack with Shape-Varying Patches (ASP), aiming to optimize patch content, shape, and position to maximize effectiveness. We introduce various mask shapes, including quadrilateral, rectangular, and circular masks, to enhance the flexibility and efficiency of the attack. Furthermore, we propose a new loss function to extend the influence of the patch beyond the overlapping regions. Experimental results demonstrate that our attack method generates an average depth error of 18 meters on the target car with a patch area of 1/9, affecting over 98\% of the target area.

Title: DarSwin-Unet: Distortion Aware Encoder-Decoder Architecture

• Authors: Akshaya Athwale, Ichrak Shili, Émile Bergeron, Ola Ahmad, Jean-François Lalonde
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17328
• Pdf URL: https://arxiv.org/pdf/2407.17328
• Copy Paste: [[2407.17328]] DarSwin-Unet: Distortion Aware Encoder-Decoder Architecture(https://arxiv.org/abs/2407.17328)
• Keywords: security, transformer
• Abstract: Wide-angle fisheye images are becoming increasingly common for perception tasks in applications such as robotics, security, and mobility (e.g. drones, avionics). However, current models often either ignore the distortions in wide-angle images or are not suitable to perform pixel-level tasks. In this paper, we present an encoder-decoder model based on a radial transformer architecture that adapts to distortions in wide-angle lenses by leveraging the physical characteristics defined by the radial distortion profile. In contrast to the original model, which only performs classification tasks, we introduce a U-Net architecture, DarSwin-Unet, designed for pixel level tasks. Furthermore, we propose a novel strategy that minimizes sparsity when sampling the image for creating its input tokens. Our approach enhances the model capability to handle pixel-level tasks in wide-angle fisheye images, making it more effective for real-world applications. Compared to other baselines, DarSwin-Unet achieves the best results across different datasets, with significant gains when trained on bounded levels of distortions (very low, low, medium, and high) and tested on all, including out-of-distribution distortions. We demonstrate its performance on depth estimation and show through extensive experiments that DarSwin-Unet can perform zero-shot adaptation to unseen distortions of different wide-angle lenses.

Title: Global and Local Confidence Based Fraud Detection Graph Neural Network

• Authors: Jiaxun Liu, Yue Tian, Guanjun Liu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17333
• Pdf URL: https://arxiv.org/pdf/2407.17333
• Copy Paste: [[2407.17333]] Global and Local Confidence Based Fraud Detection Graph Neural Network(https://arxiv.org/abs/2407.17333)
• Keywords: robust
• Abstract: This paper presents the Global and Local Confidence Graph Neural Network (GLC-GNN), an innovative approach to graph-based anomaly detection that addresses the challenges of heterophily and camouflage in fraudulent activities. By introducing a prototype to encapsulate the global features of a graph and calculating a Global Confidence (GC) value for each node, GLC-GNN effectively distinguishes between benign and fraudulent nodes. The model leverages GC to generate attention values for message aggregation, enhancing its ability to capture both homophily and heterophily. Through extensive experiments on four open datasets, GLC-GNN demonstrates superior performance over state-of-the-art models in accuracy and convergence speed, while maintaining a compact model size and expedited training process. The integration of global and local confidence measures in GLC-GNN offers a robust solution for detecting anomalies in graphs, with significant implications for fraud detection across diverse domains.

Title: Preliminary study on artificial intelligence methods for cybersecurity threat detection in computer networks based on raw data packets

• Authors: Aleksander Ogonowski, Michał Żebrowski, Arkadiusz Ćwiek, Tobiasz Jarosiewicz, Konrad Klimaszewski, Adam Padee, Piotr Wasiuk, Michał Wójcik
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2407.17339
• Pdf URL: https://arxiv.org/pdf/2407.17339
• Copy Paste: [[2407.17339]] Preliminary study on artificial intelligence methods for cybersecurity threat detection in computer networks based on raw data packets(https://arxiv.org/abs/2407.17339)
• Keywords: security, attack
• Abstract: Most of the intrusion detection methods in computer networks are based on traffic flow characteristics. However, this approach may not fully exploit the potential of deep learning algorithms to directly extract features and patterns from raw packets. Moreover, it impedes real-time monitoring due to the necessity of waiting for the processing pipeline to complete and introduces dependencies on additional software components. In this paper, we investigate deep learning methodologies capable of detecting attacks in real-time directly from raw packet data within network traffic. We propose a novel approach where packets are stacked into windows and separately recognised, with a 2D image representation suitable for processing with computer vision models. Our investigation utilizes the CIC IDS-2017 dataset, which includes both benign traffic and prevalent real-world attacks, providing a comprehensive foundation for our research.

Title: Label Alignment and Reassignment with Generalist Large Language Model for Enhanced Cross-Domain Named Entity Recognition

• Authors: Ke Bao, Chonghuan Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17344
• Pdf URL: https://arxiv.org/pdf/2407.17344
• Copy Paste: [[2407.17344]] Label Alignment and Reassignment with Generalist Large Language Model for Enhanced Cross-Domain Named Entity Recognition(https://arxiv.org/abs/2407.17344)
• Keywords: large language model
• Abstract: Named entity recognition on the in-domain supervised and few-shot settings have been extensively discussed in the NLP community and made significant progress. However, cross-domain NER, a more common task in practical scenarios, still poses a challenge for most NER methods. Previous research efforts in that area primarily focus on knowledge transfer such as correlate label information from source to target domains but few works pay attention to the problem of label conflict. In this study, we introduce a label alignment and reassignment approach, namely LAR, to address this issue for enhanced cross-domain named entity recognition, which includes two core procedures: label alignment between source and target domains and label reassignment for type inference. The process of label reassignment can significantly be enhanced by integrating with an advanced large-scale language model such as ChatGPT. We conduct an extensive range of experiments on NER datasets involving both supervised and zero-shot scenarios. Empirical experimental results demonstrate the validation of our method with remarkable performance under the supervised and zero-shot out-of-domain settings compared to SOTA methods.

Title: Insider Threats Mitigation: Role of Penetration Testing

• Authors: Krutarth Chauhan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.17346
• Pdf URL: https://arxiv.org/pdf/2407.17346
• Copy Paste: [[2407.17346]] Insider Threats Mitigation: Role of Penetration Testing(https://arxiv.org/abs/2407.17346)
• Keywords: security, defense, attack
• Abstract: Conventional security solutions are insufficient to address the urgent cybersecurity challenge posed by insider attacks. While a great deal of research has been done in this area, our systematic literature analysis attempts to give readers a thorough grasp of penetration testing's role in reducing insider risks. We aim to arrange and integrate the body of knowledge on insider threat prevention by using a grounded theory approach for a thorough literature review. This analysis classifies and evaluates the approaches used in penetration testing today, including how well they uncover and mitigate insider threats and how well they work in tandem with other security procedures. Additionally, we look at how penetration testing is used in different industries, present case studies with real-world implementations, and discuss the obstacles and constraints that businesses must overcome. This study aims to improve the knowledge of penetration testing as a critical part of insider threat defense, helping to create more comprehensive and successful security policies.

Title: Boosting Large Language Models with Socratic Method for Conversational Mathematics Teaching

• Authors: Yuyang Ding, Hanglei Hu, Jie Zhou, Qin Chen, Bo Jiang, Liang He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17349
• Pdf URL: https://arxiv.org/pdf/2407.17349
• Copy Paste: [[2407.17349]] Boosting Large Language Models with Socratic Method for Conversational Mathematics Teaching(https://arxiv.org/abs/2407.17349)
• Keywords: generative, large language model
• Abstract: With the introduction of large language models (LLMs), automatic math reasoning has seen tremendous success. However, current methods primarily focus on providing solutions or using techniques like Chain-of-Thought to enhance problem-solving accuracy. In this paper, we focus on improving the capability of mathematics teaching via a Socratic teaching-based LLM (\texttt{SocraticLLM}), which guides learners toward profound thinking with clarity and self-discovery via conversation. We collect and release a high-quality mathematical teaching dataset, named \texttt{SocraticMATH}, which provides Socratic-style conversations of problems with extra knowledge. Also, we propose a knowledge-enhanced LLM as a strong baseline to generate reliable responses with review, guidance/heuristic, rectification, and summarization. Experimental results show the great advantages of \texttt{SocraticLLM} by comparing it with several strong generative models. The codes and datasets are available on \url{this https URL}.

Title: Scalify: scale propagation for efficient low-precision LLM training

• Authors: Paul Balança, Sam Hosegood, Carlo Luschi, Andrew Fitzgibbon
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17353
• Pdf URL: https://arxiv.org/pdf/2407.17353
• Copy Paste: [[2407.17353]] Scalify: scale propagation for efficient low-precision LLM training(https://arxiv.org/abs/2407.17353)
• Keywords: large language model
• Abstract: Low-precision formats such as float8 have been introduced in machine learning accelerated hardware to improve computational efficiency for large language models training and inference. Nevertheless, adoption by the ML community has been slowed down by the complex, and sometimes brittle, techniques required to match higher precision training accuracy. In this work, we present Scalify, a end-to-end scale propagation paradigm for computational graphs, generalizing and formalizing existing tensor scaling methods. Experiment results show that Scalify supports out-of-the-box float8 matrix multiplication and gradients representation, as well as float16 optimizer state storage. Our JAX implementation of Scalify is open-sourced at this https URL

Title: Deep Spherical Superpixels

• Authors: Rémi Giraud, Michaël Clément
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17354
• Pdf URL: https://arxiv.org/pdf/2407.17354
• Copy Paste: [[2407.17354]] Deep Spherical Superpixels(https://arxiv.org/abs/2407.17354)
• Keywords: segmentation
• Abstract: Over the years, the use of superpixel segmentation has become very popular in various applications, serving as a preprocessing step to reduce data size by adapting to the content of the image, regardless of its semantic content. While the superpixel segmentation of standard planar images, captured with a 90° field of view, has been extensively studied, there has been limited focus on dedicated methods to omnidirectional or spherical images, captured with a 360° field of view. In this study, we introduce the first deep learning-based superpixel segmentation approach tailored for omnidirectional images called DSS (for Deep Spherical Superpixels). Our methodology leverages on spherical CNN architectures and the differentiable K-means clustering paradigm for superpixels, to generate superpixels that follow the spherical geometry. Additionally, we propose to use data augmentation techniques specifically designed for 360° images, enabling our model to efficiently learn from a limited set of annotated omnidirectional data. Our extensive validation across two datasets demonstrates that taking into account the inherent circular geometry of such images into our framework improves the segmentation performance over traditional and deep learning-based superpixel methods. Our code is available online.

Title: MuST: Multi-Scale Transformers for Surgical Phase Recognition

• Authors: Alejandra Pérez, Santiago Rodríguez, Nicolás Ayobi, Nicolás Aparicio, Eugénie Dessevres, Pablo Arbeláez
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17361
• Pdf URL: https://arxiv.org/pdf/2407.17361
• Copy Paste: [[2407.17361]] MuST: Multi-Scale Transformers for Surgical Phase Recognition(https://arxiv.org/abs/2407.17361)
• Keywords: transformer
• Abstract: Phase recognition in surgical videos is crucial for enhancing computer-aided surgical systems as it enables automated understanding of sequential procedural stages. Existing methods often rely on fixed temporal windows for video analysis to identify dynamic surgical phases. Thus, they struggle to simultaneously capture short-, mid-, and long-term information necessary to fully understand complex surgical procedures. To address these issues, we propose Multi-Scale Transformers for Surgical Phase Recognition (MuST), a novel Transformer-based approach that combines a Multi-Term Frame encoder with a Temporal Consistency Module to capture information across multiple temporal scales of a surgical video. Our Multi-Term Frame Encoder computes interdependencies across a hierarchy of temporal scales by sampling sequences at increasing strides around the frame of interest. Furthermore, we employ a long-term Transformer encoder over the frame embeddings to further enhance long-term reasoning. MuST achieves higher performance than previous state-of-the-art methods on three different public benchmarks.

Title: ViPer: Visual Personalization of Generative Models via Individual Preference Learning

• Authors: Sogand Salehi, Mahdi Shafiei, Teresa Yeo, Roman Bachmann, Amir Zamir
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17365
• Pdf URL: https://arxiv.org/pdf/2407.17365
• Copy Paste: [[2407.17365]] ViPer: Visual Personalization of Generative Models via Individual Preference Learning(https://arxiv.org/abs/2407.17365)
• Keywords: generative, large language model
• Abstract: Different users find different images generated for the same prompt desirable. This gives rise to personalized image generation which involves creating images aligned with an individual's visual preference. Current generative models are, however, unpersonalized, as they are tuned to produce outputs that appeal to a broad audience. Using them to generate images aligned with individual users relies on iterative manual prompt engineering by the user which is inefficient and undesirable. We propose to personalize the image generation process by first capturing the generic preferences of the user in a one-time process by inviting them to comment on a small selection of images, explaining why they like or dislike each. Based on these comments, we infer a user's structured liked and disliked visual attributes, i.e., their visual preference, using a large language model. These attributes are used to guide a text-to-image model toward producing images that are tuned towards the individual user's visual preference. Through a series of user studies and large language model guided evaluations, we demonstrate that the proposed method results in generations that are well aligned with individual users' visual preferences.

Title: PrevPredMap: Exploring Temporal Modeling with Previous Predictions for Online Vectorized HD Map Construction

• Authors: Nan Peng, Xun Zhou, Mingming Wang, Xiaojun Yang, Songming Chen, Guisong Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17378
• Pdf URL: https://arxiv.org/pdf/2407.17378
• Copy Paste: [[2407.17378]] PrevPredMap: Exploring Temporal Modeling with Previous Predictions for Online Vectorized HD Map Construction(https://arxiv.org/abs/2407.17378)
• Keywords: robust
• Abstract: Temporal information is crucial for detecting occluded instances. Existing temporal representations have progressed from BEV or PV features to more compact query features. Compared to these aforementioned features, predictions offer the highest level of abstraction, providing explicit information. In the context of online vectorized HD map construction, this unique characteristic of predictions is potentially advantageous for long-term temporal modeling and the integration of map priors. This paper introduces PrevPredMap, a pioneering temporal modeling framework that leverages previous predictions for constructing online vectorized HD maps. We have meticulously crafted two essential modules for PrevPredMap: the previous-predictions-based query generator and the dynamic-position-query decoder. Specifically, the previous-predictions-based query generator is designed to separately encode different types of information from previous predictions, which are then effectively utilized by the dynamic-position-query decoder to generate current predictions. Furthermore, we have developed a dual-mode strategy to ensure PrevPredMap's robust performance across both single-frame and temporal modes. Extensive experiments demonstrate that PrevPredMap achieves state-of-the-art performance on the nuScenes and Argoverse2 datasets. Code will be available at this https URL.

Title: A Comprehensive Approach to Misspelling Correction with BERT and Levenshtein Distance

• Authors: Amirreza Naziri, Hossein Zeinali
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17383
• Pdf URL: https://arxiv.org/pdf/2407.17383
• Copy Paste: [[2407.17383]] A Comprehensive Approach to Misspelling Correction with BERT and Levenshtein Distance(https://arxiv.org/abs/2407.17383)
• Keywords: transformer
• Abstract: Writing, as an omnipresent form of human communication, permeates nearly every aspect of contemporary life. Consequently, inaccuracies or errors in written communication can lead to profound consequences, ranging from financial losses to potentially life-threatening situations. Spelling mistakes, among the most prevalent writing errors, are frequently encountered due to various factors. This research aims to identify and rectify diverse spelling errors in text using neural networks, specifically leveraging the Bidirectional Encoder Representations from Transformers (BERT) masked language model. To achieve this goal, we compiled a comprehensive dataset encompassing both non-real-word and real-word errors after categorizing different types of spelling mistakes. Subsequently, multiple pre-trained BERT models were employed. To ensure optimal performance in correcting misspelling errors, we propose a combined approach utilizing the BERT masked language model and Levenshtein distance. The results from our evaluation data demonstrate that the system presented herein exhibits remarkable capabilities in identifying and rectifying spelling mistakes, often surpassing existing systems tailored for the Persian language.

Title: PERSONA: A Reproducible Testbed for Pluralistic Alignment

• Authors: Louis Castricato, Nathan Lile, Rafael Rafailov, Jan-Philipp Fränken, Chelsea Finn
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17387
• Pdf URL: https://arxiv.org/pdf/2407.17387
• Copy Paste: [[2407.17387]] PERSONA: A Reproducible Testbed for Pluralistic Alignment(https://arxiv.org/abs/2407.17387)
• Keywords: robust
• Abstract: The rapid advancement of language models (LMs) necessitates robust alignment with diverse user values. However, current preference optimization approaches often fail to capture the plurality of user opinions, instead reinforcing majority viewpoints and marginalizing minority perspectives. We introduce PERSONA, a reproducible test bed designed to evaluate and improve pluralistic alignment of LMs. We procedurally generate diverse user profiles from US census data, resulting in 1,586 synthetic personas with varied demographic and idiosyncratic attributes. We then generate a large-scale evaluation dataset containing 3,868 prompts and 317,200 feedback pairs obtained from our synthetic personas. Leveraging this dataset, we systematically evaluate LM capabilities in role-playing diverse users, verified through human judges, and the establishment of both a benchmark, PERSONA Bench, for pluralistic alignment approaches as well as an extensive dataset to create new and future benchmarks. The full dataset and benchmarks are available here: this https URL.

Title: CovScore: Evaluation of Multi-Document Abstractive Title Set Generation

• Authors: Itamar Trainin, Omri Abend
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17390
• Pdf URL: https://arxiv.org/pdf/2407.17390
• Copy Paste: [[2407.17390]] CovScore: Evaluation of Multi-Document Abstractive Title Set Generation(https://arxiv.org/abs/2407.17390)
• Keywords: extraction
• Abstract: This paper introduces CovScore, an automatic reference-less methodology for evaluating thematic title sets, extracted from a corpus of documents. While such extraction methods are widely used, evaluating their effectiveness remains an open question. Moreover, some existing practices heavily rely on slow and laborious human annotation procedures. Inspired by recently introduced LLM-based judge methods, we propose a novel methodology that decomposes quality into five main metrics along different aspects of evaluation. This framing simplifies and expedites the manual evaluation process and enables automatic and independent LLM-based evaluation. As a test case, we apply our approach to a corpus of Holocaust survivor testimonies, motivated both by its relevance to title set extraction and by the moral significance of this pursuit. We validate the methodology by experimenting with naturalistic and synthetic title set generation systems and compare their performance with the methodology.

Title: Five reasons against assuming a data-generating distribution in Machine Learning

• Authors: Benedikt Höltgen, Robert C. Williamson
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17395
• Pdf URL: https://arxiv.org/pdf/2407.17395
• Copy Paste: [[2407.17395]] Five reasons against assuming a data-generating distribution in Machine Learning(https://arxiv.org/abs/2407.17395)
• Keywords: generative
• Abstract: Machine Learning research, as most of Statistics, heavily relies on the concept of a data-generating probability distribution. As data points are thought to be sampled from such a distribution, we can learn from observed data about this distribution and, thus, predict future data points drawn from it (with some probability of success). Drawing on scholarship across disciplines, we here argue that this framework is not always a good model. Not only do such true probability distributions not exist; the framework can also be misleading and obscure both the choices made and the goals pursued in machine learning practice. We suggest an alternative framework that focuses on finite populations rather than abstract distributions; while classical learning theory can be left almost unchanged, it opens new opportunities, especially to model sampling. We compile these considerations into five reasons for modelling machine learning -- in some settings -- with finite distributions rather than generative distributions, both to be more faithful to practice and to provide novel theoretical insights.

Title: 3D Question Answering for City Scene Understanding

• Authors: Penglei Sun, Yaoxian Song, Xiang Liu, Xiaofei Yang, Qiang Wang, Tiefeng Li, Yang Yang, Xiaowen Chu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17398
• Pdf URL: https://arxiv.org/pdf/2407.17398
• Copy Paste: [[2407.17398]] 3D Question Answering for City Scene Understanding(https://arxiv.org/abs/2407.17398)
• Keywords: robust, large language model
• Abstract: 3D multimodal question answering (MQA) plays a crucial role in scene understanding by enabling intelligent agents to comprehend their surroundings in 3D environments. While existing research has primarily focused on indoor household tasks and outdoor roadside autonomous driving tasks, there has been limited exploration of city-level scene understanding tasks. Furthermore, existing research faces challenges in understanding city scenes, due to the absence of spatial semantic information and human-environment interaction information at the city this http URL address these challenges, we investigate 3D MQA from both dataset and method perspectives. From the dataset perspective, we introduce a novel 3D MQA dataset named City-3DQA for city-level scene understanding, which is the first dataset to incorporate scene semantic and human-environment interactive tasks within the city. From the method perspective, we propose a Scene graph enhanced City-level Understanding method (Sg-CityU), which utilizes the scene graph to introduce the spatial semantic. A new benchmark is reported and our proposed Sg-CityU achieves accuracy of 63.94 % and 63.76 % in different settings of City-3DQA. Compared to indoor 3D MQA methods and zero-shot using advanced large language models (LLMs), Sg-CityU demonstrates state-of-the-art (SOTA) performance in robustness and generalization.

Title: Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models

• Authors: Yida Zhao, Chao Lou, Kewei Tu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17406
• Pdf URL: https://arxiv.org/pdf/2407.17406
• Copy Paste: [[2407.17406]] Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Models(https://arxiv.org/abs/2407.17406)
• Keywords: transformer
• Abstract: Syntactic Transformer language models aim to achieve better generalization through simultaneously modeling syntax trees and sentences. While prior work has been focusing on adding constituency-based structures to Transformers, we introduce Dependency Transformer Grammars (DTGs), a new class of Transformer language model with explicit dependency-based inductive bias. DTGs simulate dependency transition systems with constrained attention patterns by modifying attention masks, incorporate the stack information through relative positional encoding, and augment dependency arc representation with a combination of token embeddings and operation embeddings. When trained on a dataset of sentences annotated with dependency trees, DTGs achieve better generalization while maintaining comparable perplexity with Transformer language model baselines. DTGs also outperform recent constituency-based models, showing that dependency can better guide Transformer language models. Our code is released at this https URL.

Title: (PASS) Visual Prompt Locates Good Structure Sparsity through a Recurrent HyperNetwork

• Authors: Tianjin Huang, Fang Meng, Li Shen, Fan Liu, Yulong Pei, Mykola Pechenizkiy, Shiwei Liu, Tianlong Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17412
• Pdf URL: https://arxiv.org/pdf/2407.17412
• Copy Paste: [[2407.17412]] (PASS) Visual Prompt Locates Good Structure Sparsity through a Recurrent HyperNetwork(https://arxiv.org/abs/2407.17412)
• Keywords: large language model
• Abstract: Large-scale neural networks have demonstrated remarkable performance in different domains like vision and language processing, although at the cost of massive computation resources. As illustrated by compression literature, structural model pruning is a prominent algorithm to encourage model efficiency, thanks to its acceleration-friendly sparsity patterns. One of the key questions of structural pruning is how to estimate the channel significance. In parallel, work on data-centric AI has shown that prompting-based techniques enable impressive generalization of large language models across diverse downstream tasks. In this paper, we investigate a charming possibility - \textit{leveraging visual prompts to capture the channel importance and derive high-quality structural sparsity}. To this end, we propose a novel algorithmic framework, namely \texttt{PASS}. It is a tailored hyper-network to take both visual prompts and network weight statistics as input, and output layer-wise channel sparsity in a recurrent manner. Such designs consider the intrinsic channel dependency between layers. Comprehensive experiments across multiple network architectures and six datasets demonstrate the superiority of \texttt{PASS} in locating good structural sparsity. For example, at the same FLOPs level, \texttt{PASS} subnetworks achieve $1\%\sim 3\%$ better accuracy on Food101 dataset; or with a similar performance of $80\%$ accuracy, \texttt{PASS} subnetworks obtain $0.35\times$ more speedup than the baselines.

Title: Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?

• Authors: Michael-Andrei Panaitescu-Liess, Zora Che, Bang An, Yuancheng Xu, Pankayaraj Pathmanathan, Souradip Chakraborty, Sicheng Zhu, Tom Goldstein, Furong Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17417
• Pdf URL: https://arxiv.org/pdf/2407.17417
• Copy Paste: [[2407.17417]] Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?(https://arxiv.org/abs/2407.17417)
• Keywords: attack, membership infer, watermark, large language model
• Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in generating diverse and contextually rich text. However, concerns regarding copyright infringement arise as LLMs may inadvertently produce copyrighted material. In this paper, we first investigate the effectiveness of watermarking LLMs as a deterrent against the generation of copyrighted texts. Through theoretical analysis and empirical evaluation, we demonstrate that incorporating watermarks into LLMs significantly reduces the likelihood of generating copyrighted content, thereby addressing a critical concern in the deployment of LLMs. Additionally, we explore the impact of watermarking on Membership Inference Attacks (MIAs), which aim to discern whether a sample was part of the pretraining dataset and may be used to detect copyright violations. Surprisingly, we find that watermarking adversely affects the success rate of MIAs, complicating the task of detecting copyrighted text in the pretraining dataset. Finally, we propose an adaptive technique to improve the success rate of a recent MIA under watermarking. Our findings underscore the importance of developing adaptive methods to study critical problems in LLMs with potential legal implications.

Title: Vision Language Model-Empowered Contract Theory for AIGC Task Allocation in Teleoperation

• Authors: Zijun Zhan, Yaxian Dong, Yuqing Hu, Shuai Li, Shaohua Cao, Zhu Han
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17428
• Pdf URL: https://arxiv.org/pdf/2407.17428
• Copy Paste: [[2407.17428]] Vision Language Model-Empowered Contract Theory for AIGC Task Allocation in Teleoperation(https://arxiv.org/abs/2407.17428)
• Keywords: diffusion
• Abstract: Integrating low-light image enhancement techniques, in which diffusion-based AI-generated content (AIGC) models are promising, is necessary to enhance nighttime teleoperation. Remarkably, the AIGC model is computation-intensive, thus necessitating the allocation of AIGC tasks to edge servers with ample computational resources. Given the distinct cost of the AIGC model trained with varying-sized datasets and AIGC tasks possessing disparate demand, it is imperative to formulate a differential pricing strategy to optimize the utility of teleoperators and edge servers concurrently. Nonetheless, the pricing strategy formulation is under information asymmetry, i.e., the demand (e.g., the difficulty level of AIGC tasks and their distribution) of AIGC tasks is hidden information to edge servers. Additionally, manually assessing the difficulty level of AIGC tasks is tedious and unnecessary for teleoperators. To this end, we devise a framework of AIGC task allocation assisted by the Vision Language Model (VLM)-empowered contract theory, which includes two components: VLM-empowered difficulty assessment and contract theory-assisted AIGC task allocation. The first component enables automatic and accurate AIGC task difficulty assessment. The second component is capable of formulating the pricing strategy for edge servers under information asymmetry, thereby optimizing the utility of both edge servers and teleoperators. The simulation results demonstrated that our proposed framework can improve the average utility of teleoperators and edge servers by 10.88~12.43% and 1.4~2.17%, respectively. Code and data are available at this https URL.

Title: An FPGA-Based Open-Source Hardware-Software Framework for Side-Channel Security Research

• Authors: Davide Zoni, Andrea Galimberti, Davide Galli
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2407.17432
• Pdf URL: https://arxiv.org/pdf/2407.17432
• Copy Paste: [[2407.17432]] An FPGA-Based Open-Source Hardware-Software Framework for Side-Channel Security Research(https://arxiv.org/abs/2407.17432)
• Keywords: security, attack
• Abstract: Attacks based on side-channel analysis (SCA) pose a severe security threat to modern computing platforms, further exacerbated on IoT devices by their pervasiveness and handling of private and critical data. Designing SCA-resistant computing platforms requires a significant additional effort in the early stages of the IoT devices' life cycle, which is severely constrained by strict time-to-market deadlines and tight budgets. This manuscript introduces a hardware-software framework meant for SCA research on FPGA targets. It delivers an IoT-class system-on-chip (SoC) that includes a RISC-V CPU, provides observability and controllability through an ad-hoc debug infrastructure to facilitate SCA attacks and evaluate the platform's security, and streamlines the deployment of SCA countermeasures through dedicated hardware and software features such as a DFS actuator and FreeRTOS support. The open-source release of the framework includes the SoC, the scripts to configure the computing platform, compile a target application, and assess the SCA security, as well as a suite of state-of-the-art SCA attacks and countermeasures. The goal is to foster its adoption and novel developments in the field, empowering designers and researchers to focus on studying SCA countermeasures and attacks while relying on a sound and stable hardware-software platform as the foundation for their research.

Title: HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation

• Authors: Zhenzhi Wang, Yixuan Li, Yanhong Zeng, Youqing Fang, Yuwei Guo, Wenran Liu, Jing Tan, Kai Chen, Tianfan Xue, Bo Dai, Dahua Lin
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17438
• Pdf URL: https://arxiv.org/pdf/2407.17438
• Copy Paste: [[2407.17438]] HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation(https://arxiv.org/abs/2407.17438)
• Keywords: fair
• Abstract: Human image animation involves generating videos from a character photo, allowing user control and unlocking potential for video and movie production. While recent approaches yield impressive results using high-quality training data, the inaccessibility of these datasets hampers fair and transparent benchmarking. Moreover, these approaches prioritize 2D human motion and overlook the significance of camera motions in videos, leading to limited control and unstable video this http URL demystify the training data, we present HumanVid, the first large-scale high-quality dataset tailored for human image animation, which combines crafted real-world and synthetic data. For the real-world data, we compile a vast collection of copyright-free real-world videos from the internet. Through a carefully designed rule-based filtering strategy, we ensure the inclusion of high-quality videos, resulting in a collection of 20K human-centric videos in 1080P resolution. Human and camera motion annotation is accomplished using a 2D pose estimator and a SLAM-based method. For the synthetic data, we gather 2,300 copyright-free 3D avatar assets to augment existing available 3D assets. Notably, we introduce a rule-based camera trajectory generation method, enabling the synthetic pipeline to incorporate diverse and precise camera motion annotation, which can rarely be found in real-world data. To verify the effectiveness of HumanVid, we establish a baseline model named CamAnimate, short for Camera-controllable Human Animation, that considers both human and camera motions as conditions. Through extensive experimentation, we demonstrate that such simple baseline training on our HumanVid achieves state-of-the-art performance in controlling both human pose and camera motions, setting a new benchmark. Code and data will be publicly available at \url{this https URL}.

Title: Fluent Student-Teacher Redteaming

• Authors: T. Ben Thompson, Michael Sklar (Confirm Labs)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.17447
• Pdf URL: https://arxiv.org/pdf/2407.17447
• Copy Paste: [[2407.17447]] Fluent Student-Teacher Redteaming(https://arxiv.org/abs/2407.17447)
• Keywords: security, attack
• Abstract: Many publicly available language models have been safety tuned to reduce the likelihood of toxic or liability-inducing text. Users or security analysts attempt to jailbreak or redteam these models with adversarial prompts which cause compliance with requests. One attack method is to apply discrete optimization techniques to the prompt. However, the resulting attack strings are often gibberish text, easily filtered by defenders due to high measured perplexity, and may fail for unseen tasks and/or well-tuned models. In this work, we improve existing algorithms (primarily GCG and BEAST) to develop powerful and fluent attacks on safety-tuned models like Llama-2 and Phi-3. Our technique centers around a new distillation-based approach that encourages the victim model to emulate a toxified finetune, either in terms of output probabilities or internal activations. To encourage human-fluent attacks, we add a multi-model perplexity penalty and a repetition penalty to the objective. We also enhance optimizer strength by allowing token insertions, token swaps, and token deletions and by using longer attack sequences. The resulting process is able to reliably jailbreak the most difficult target models with prompts that appear similar to human-written prompts. On Advbench we achieve attack success rates $>93$% for Llama-2-7B, Llama-3-8B, and Vicuna-7B, while maintaining model-measured perplexity $<33$; we achieve $95$% attack success for Phi-3, though with higher perplexity. We also find a universally-optimized single fluent prompt that induces $>88$% compliance on previously unseen tasks across Llama-2-7B, Phi-3-mini and Vicuna-7B and transfers to other black-box models.

Title: $VILA^2$: VILA Augmented VILA

• Authors: Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jang Hyun Cho, Marco Pavone, Song Han, Hongxu Yin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17453
• Pdf URL: https://arxiv.org/pdf/2407.17453
• Copy Paste: [[2407.17453]] $VILA^2$: VILA Augmented VILA(https://arxiv.org/abs/2407.17453)
• Keywords: large language model
• Abstract: Visual language models (VLMs) have rapidly progressed, driven by the success of large language models (LLMs). While model architectures and training infrastructures advance rapidly, data curation remains under-explored. When data quantity and quality become a bottleneck, existing work either directly crawls more raw data from the Internet that does not have a guarantee of data quality or distills from black-box commercial models (e.g., GPT-4V / Gemini) causing the performance upper bounded by that model. In this work, we introduce a novel approach that includes a self-augment step and a specialist-augment step to iteratively improve data quality and model performance. In the self-augment step, a VLM recaptions its own pretraining data to enhance data quality, and then retrains from scratch using this refined dataset to improve model performance. This process can iterate for several rounds. Once self-augmentation saturates, we employ several specialist VLMs finetuned from the self-augmented VLM with domain-specific expertise, to further infuse specialist knowledge into the generalist VLM through task-oriented recaptioning and retraining. With the combined self-augmented and specialist-augmented training, we introduce $VILA^2$ (VILA-augmented-VILA), a VLM family that consistently improves the accuracy on a wide range of tasks over prior art, and achieves new state-of-the-art results on MMMU leaderboard among open-sourced models.

Title: Hidden or Inferred: Fair Learning-To-Rank with Unknown Demographics

• Authors: Oluseun Olulana, Kathleen Cachel, Fabricio Murai, Elke Rundensteiner
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.17459
• Pdf URL: https://arxiv.org/pdf/2407.17459
• Copy Paste: [[2407.17459]] Hidden or Inferred: Fair Learning-To-Rank with Unknown Demographics(https://arxiv.org/abs/2407.17459)
• Keywords: privacy, protect, robust, fair
• Abstract: As learning-to-rank models are increasingly deployed for decision-making in areas with profound life implications, the FairML community has been developing fair learning-to-rank (LTR) models. These models rely on the availability of sensitive demographic features such as race or sex. However, in practice, regulatory obstacles and privacy concerns protect this data from collection and use. As a result, practitioners may either need to promote fairness despite the absence of these features or turn to demographic inference tools to attempt to infer them. Given that these tools are fallible, this paper aims to further understand how errors in demographic inference impact the fairness performance of popular fair LTR strategies. In which cases would it be better to keep such demographic attributes hidden from models versus infer them? We examine a spectrum of fair LTR strategies ranging from fair LTR with and without demographic features hidden versus inferred to fairness-unaware LTR followed by fair re-ranking. We conduct a controlled empirical investigation modeling different levels of inference errors by systematically perturbing the inferred sensitive attribute. We also perform three case studies with real-world datasets and popular open-source inference methods. Our findings reveal that as inference noise grows, LTR-based methods that incorporate fairness considerations into the learning process may increase bias. In contrast, fair re-ranking strategies are more robust to inference errors. All source code, data, and experimental artifacts of our experimental study are available here: this https URL

Title: CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models

• Authors: Jiawei Gu, Zacc Yang, Chuanghao Ding, Rui Zhao, Fei Tan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.17467
• Pdf URL: https://arxiv.org/pdf/2407.17467
• Copy Paste: [[2407.17467]] CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models(https://arxiv.org/abs/2407.17467)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) excel in diverse tasks but often underperform in specialized fields due to limited domain-specific or proprietary corpus. Continual pre-training (CPT) enhances LLM capabilities by imbuing new domain-specific or proprietary knowledge while replaying general corpus to prevent catastrophic forgetting. The data mixture ratio of general corpus and domain-specific corpus, however, has been chosen heuristically, leading to sub-optimal training efficiency in practice. In this context, we attempt to re-visit the scaling behavior of LLMs under the hood of CPT, and discover a power-law relationship between loss, mixture ratio, and training tokens scale. We formalize the trade-off between general and domain-specific capabilities, leading to a well-defined Critical Mixture Ratio (CMR) of general and domain data. By striking the balance, CMR maintains the model's general ability and achieves the desired domain transfer, ensuring the highest utilization of available resources. Therefore, if we value the balance between efficiency and effectiveness, CMR can be consider as the optimal mixture ratio.Through extensive experiments, we ascertain the predictability of CMR, and propose CMR scaling law and have substantiated its generalization. These findings offer practical guidelines for optimizing LLM training in specialized domains, ensuring both general and domain-specific performance while efficiently managing training resources.

Title: WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queries

• Authors: Wenting Zhao, Tanya Goyal, Yu Ying Chiu, Liwei Jiang, Benjamin Newman, Abhilasha Ravichander, Khyathi Chandu, Ronan Le Bras, Claire Cardie, Yuntian Deng, Yejin Choi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17468
• Pdf URL: https://arxiv.org/pdf/2407.17468
• Copy Paste: [[2407.17468]] WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queries(https://arxiv.org/abs/2407.17468)
• Keywords: large language model
• Abstract: While hallucinations of large language models (LLMs) prevail as a major challenge, existing evaluation benchmarks on factuality do not cover the diverse domains of knowledge that the real-world users of LLMs seek information about. To bridge this gap, we introduce WildHallucinations, a benchmark that evaluates factuality. It does so by prompting LLMs to generate information about entities mined from user-chatbot conversations in the wild. These generations are then automatically fact-checked against a systematically curated knowledge source collected from web search. Notably, half of these real-world entities do not have associated Wikipedia pages. We evaluate 118,785 generations from 15 LLMs on 7,919 entities. We find that LLMs consistently hallucinate more on entities without Wikipedia pages and exhibit varying hallucination rates across different domains. Finally, given the same base models, adding a retrieval component only slightly reduces hallucinations but does not eliminate hallucinations.

Title: I Could've Asked That: Reformulating Unanswerable Questions

• Authors: Wenting Zhao, Ge Gao, Claire Cardie, Alexander M. Rush
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.17469
• Pdf URL: https://arxiv.org/pdf/2407.17469
• Copy Paste: [[2407.17469]] I Could've Asked That: Reformulating Unanswerable Questions(https://arxiv.org/abs/2407.17469)
• Keywords: large language model
• Abstract: When seeking information from unfamiliar documents, users frequently pose questions that cannot be answered by the documents. While existing large language models (LLMs) identify these unanswerable questions, they do not assist users in reformulating their questions, thereby reducing their overall utility. We curate CouldAsk, an evaluation benchmark composed of existing and new datasets for document-grounded question answering, specifically designed to study reformulating unanswerable questions. We evaluate state-of-the-art open-source and proprietary LLMs on CouldAsk. The results demonstrate the limited capabilities of these models in reformulating questions. Specifically, GPT-4 and Llama2-7B successfully reformulate questions only 26% and 12% of the time, respectively. Error analysis shows that 62% of the unsuccessful reformulations stem from the models merely rephrasing the questions or even generating identical questions. We publicly release the benchmark and the code to reproduce the experiments.

Title: SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency

• Authors: Yiming Xie, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, Varun Jampani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.17470
• Pdf URL: https://arxiv.org/pdf/2407.17470
• Copy Paste: [[2407.17470]] SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency(https://arxiv.org/abs/2407.17470)
• Keywords: diffusion, generative
• Abstract: We present Stable Video 4D (SV4D), a latent video diffusion model for multi-frame and multi-view consistent dynamic 3D content generation. Unlike previous methods that rely on separately trained generative models for video generation and novel view synthesis, we design a unified diffusion model to generate novel view videos of dynamic 3D objects. Specifically, given a monocular reference video, SV4D generates novel views for each video frame that are temporally consistent. We then use the generated novel view videos to optimize an implicit 4D representation (dynamic NeRF) efficiently, without the need for cumbersome SDS-based optimization used in most prior works. To train our unified novel view video generation model, we curated a dynamic 3D object dataset from the existing Objaverse dataset. Extensive experimental results on multiple datasets and user studies demonstrate SV4D's state-of-the-art performance on novel-view video synthesis as well as 4D generation compared to prior works.