2024-08-30

Title: Meta-Learning for Federated Face Recognition in Imbalanced Data Regimes

• Authors: Arwin Gansekoele, Emiel Hess, Sandjai Bhulai
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16003
• Pdf URL: https://arxiv.org/pdf/2408.16003
• Copy Paste: [[2408.16003]] Meta-Learning for Federated Face Recognition in Imbalanced Data Regimes(https://arxiv.org/abs/2408.16003)
• Keywords: privacy, federate, fair
• Abstract: The growing privacy concerns surrounding face image data demand new techniques that can guarantee user privacy. One such face recognition technique that claims to achieve better user privacy is Federated Face Recognition (FRR), a subfield of Federated Learning (FL). However, FFR faces challenges due to the heterogeneity of the data, given the large number of classes that need to be handled. To overcome this problem, solutions are sought in the field of personalized FL. This work introduces three new data partitions based on the CelebA dataset, each with a different form of data heterogeneity. It also proposes Hessian-Free Model Agnostic Meta-Learning (HF-MAML) in an FFR setting. We show that HF-MAML scores higher in verification tests than current FFR models on three different CelebA data partitions. In particular, the verification scores improve the most in heterogeneous data partitions. To balance personalization with the development of an effective global model, an embedding regularization term is introduced for the loss function. This term can be combined with HF-MAML and is shown to increase global model verification performance. Lastly, this work performs a fairness analysis, showing that HF-MAML and its embedding regularization extension can improve fairness by reducing the standard deviation over the client evaluation scores.

Title: Using large language models to estimate features of multi-word expressions: Concreteness, valence, arousal

• Authors: Gonzalo Martínez, Juan Diego Molero, Sandra González, Javier Conde, Marc Brysbaert, Pedro Reviriego
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16012
• Pdf URL: https://arxiv.org/pdf/2408.16012
• Copy Paste: [[2408.16012]] Using large language models to estimate features of multi-word expressions: Concreteness, valence, arousal(https://arxiv.org/abs/2408.16012)
• Keywords: large language model
• Abstract: This study investigates the potential of large language models (LLMs) to provide accurate estimates of concreteness, valence and arousal for multi-word expressions. Unlike previous artificial intelligence (AI) methods, LLMs can capture the nuanced meanings of multi-word expressions. We systematically evaluated ChatGPT-4o's ability to predict concreteness, valence and arousal. In Study 1, ChatGPT-4o showed strong correlations with human concreteness ratings (r = .8) for multi-word expressions. In Study 2, these findings were repeated for valence and arousal ratings of individual words, matching or outperforming previous AI models. Study 3 extended the prevalence and arousal analysis to multi-word expressions and showed promising results despite the lack of large-scale human benchmarks. These findings highlight the potential of LLMs for generating valuable psycholinguistic data related to multiword expressions. To help researchers with stimulus selection, we provide datasets with AI norms of concreteness, valence and arousal for 126,397 English single words and 63,680 multi-word expressions

Title: Differentially Private Publication of Electricity Time Series Data in Smart Grids

• Authors: Sina Shaham, Gabriel Ghinita, Bhaskar Krishnamachari, Cyrus Shahabi
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16017
• Pdf URL: https://arxiv.org/pdf/2408.16017
• Copy Paste: [[2408.16017]] Differentially Private Publication of Electricity Time Series Data in Smart Grids(https://arxiv.org/abs/2408.16017)
• Keywords: privacy, transformer
• Abstract: Smart grids are a valuable data source to study consumer behavior and guide energy policy decisions. In particular, time-series of power consumption over geographical areas are essential in deciding the optimal placement of expensive resources (e.g., transformers, storage elements) and their activation schedules. However, publication of such data raises significant privacy issues, as it may reveal sensitive details about personal habits and lifestyles. Differential privacy (DP) is well-suited for sanitization of individual data, but current DP techniques for time series lead to significant loss in utility, due to the existence of temporal correlation between data readings. We introduce {\em STPT (Spatio-Temporal Private Timeseries)}, a novel method for DP-compliant publication of electricity consumption data that analyzes spatio-temporal attributes and captures both micro and macro patterns by leveraging RNNs. Additionally, it employs a partitioning method for releasing electricity consumption time series based on identified patterns. We demonstrate through extensive experiments, on both real-world and synthetic datasets, that STPT significantly outperforms existing benchmarks, providing a well-balanced trade-off between data utility and user privacy.

Title: SPICED: Syntactical Bug and Trojan Pattern Identification in A/MS Circuits using LLM-Enhanced Detection

• Authors: Jayeeta Chaudhuri, Dhruv Thapar, Arjun Chaudhuri, Farshad Firouzi, Krishnendu Chakrabarty
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16018
• Pdf URL: https://arxiv.org/pdf/2408.16018
• Copy Paste: [[2408.16018]] SPICED: Syntactical Bug and Trojan Pattern Identification in A/MS Circuits using LLM-Enhanced Detection(https://arxiv.org/abs/2408.16018)
• Keywords: security, steal, watermark, large language model
• Abstract: Analog and mixed-signal (A/MS) integrated circuits (ICs) are crucial in modern electronics, playing key roles in signal processing, amplification, sensing, and power management. Many IC companies outsource manufacturing to third-party foundries, creating security risks such as stealthy analog Trojans. Traditional detection methods, including embedding circuit watermarks or conducting hardware-based monitoring, often impose significant area and power overheads, and may not effectively identify all types of Trojans. To address these shortcomings, we propose SPICED, a Large Language Model (LLM)-based framework that operates within the software domain, eliminating the need for hardware modifications for Trojan detection and localization. This is the first work using LLM-aided techniques for detecting and localizing syntactical bugs and analog Trojans in circuit netlists, requiring no explicit training and incurring zero area overhead. Our framework employs chain-of-thought reasoning and few-shot examples to teach anomaly detection rules to LLMs. With the proposed method, we achieve an average Trojan coverage of 93.32% and an average true positive rate of 93.4% in identifying Trojan-impacted nodes for the evaluated analog benchmark circuits. These experimental results validate the effectiveness of LLMs in detecting and locating both syntactical bugs and Trojans within analog netlists.

Title: XG-NID: Dual-Modality Network Intrusion Detection using a Heterogeneous Graph Neural Network and Large Language Model

• Authors: Yasir Ali Farrukh, Syed Wali, Irfan Khan, Nathaniel D. Bastian
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16021
• Pdf URL: https://arxiv.org/pdf/2408.16021
• Copy Paste: [[2408.16021]] XG-NID: Dual-Modality Network Intrusion Detection using a Heterogeneous Graph Neural Network and Large Language Model(https://arxiv.org/abs/2408.16021)
• Keywords: security, defense, robust, extraction, interpretability, large language model
• Abstract: In the rapidly evolving field of cybersecurity, the integration of flow-level and packet-level information for real-time intrusion detection remains a largely untapped area of research. This paper introduces "XG-NID," a novel framework that, to the best of our knowledge, is the first to fuse flow-level and packet-level data within a heterogeneous graph structure, offering a comprehensive analysis of network traffic. Leveraging a heterogeneous graph neural network (GNN) with graph-level classification, XG-NID uniquely enables real-time inference while effectively capturing the intricate relationships between flow and packet payload data. Unlike traditional GNN-based methodologies that predominantly analyze historical data, XG-NID is designed to accommodate the heterogeneous nature of network traffic, providing a robust and real-time defense mechanism. Our framework extends beyond mere classification; it integrates Large Language Models (LLMs) to generate detailed, human-readable explanations and suggest potential remedial actions, ensuring that the insights produced are both actionable and comprehensible. Additionally, we introduce a new set of flow features based on temporal information, further enhancing the contextual and explainable inferences provided by our model. To facilitate practical application and accessibility, we developed "GNN4ID," an open-source tool that enables the extraction and transformation of raw network traffic into the proposed heterogeneous graph structure, seamlessly integrating flow and packet-level data. Our comprehensive quantitative comparative analysis demonstrates that XG-NID achieves an F1 score of 97\% in multi-class classification, outperforming existing baseline and state-of-the-art methods. This sets a new standard in Network Intrusion Detection Systems by combining innovative data fusion with enhanced interpretability and real-time capabilities.

Title: Improving Adversarial Robustness in Android Malware Detection by Reducing the Impact of Spurious Correlations

• Authors: Hamid Bostani, Zhengyu Zhao, Veelasha Moonsamy
• Subjects: cs.CR, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.16025
• Pdf URL: https://arxiv.org/pdf/2408.16025
• Copy Paste: [[2408.16025]] Improving Adversarial Robustness in Android Malware Detection by Reducing the Impact of Spurious Correlations(https://arxiv.org/abs/2408.16025)
• Keywords: defense, attack, robust
• Abstract: Machine learning (ML) has demonstrated significant advancements in Android malware detection (AMD); however, the resilience of ML against realistic evasion attacks remains a major obstacle for AMD. One of the primary factors contributing to this challenge is the scarcity of reliable generalizations. Malware classifiers with limited generalizability tend to overfit spurious correlations derived from biased features. Consequently, adversarial examples (AEs), generated by evasion attacks, can modify these features to evade detection. In this study, we propose a domain adaptation technique to improve the generalizability of AMD by aligning the distribution of malware samples and AEs. Specifically, we utilize meaningful feature dependencies, reflecting domain constraints in the feature space, to establish a robust feature space. Training on the proposed robust feature space enables malware classifiers to learn from predefined patterns associated with app functionality rather than from individual features. This approach helps mitigate spurious correlations inherent in the initial feature space. Our experiments conducted on DREBIN, a renowned Android malware detector, demonstrate that our approach surpasses the state-of-the-art defense, Sec-SVM, when facing realistic evasion attacks. In particular, our defense can improve adversarial robustness by up to 55% against realistic evasion attacks compared to Sec-SVM.

Title: ANVIL: Anomaly-based Vulnerability Identification without Labelled Training Data

• Authors: Weizhou Wang, Eric Liu, Xiangyu Guo, David Lie
• Subjects: cs.CR, cs.LG, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.16028
• Pdf URL: https://arxiv.org/pdf/2408.16028
• Copy Paste: [[2408.16028]] ANVIL: Anomaly-based Vulnerability Identification without Labelled Training Data(https://arxiv.org/abs/2408.16028)
• Keywords: large language model
• Abstract: Supervised learning-based software vulnerability detectors often fall short due to the inadequate availability of labelled training data. In contrast, Large Language Models (LLMs) such as GPT-4, are not trained on labelled data, but when prompted to detect vulnerabilities, LLM prediction accuracy is only marginally better than random guessing. In this paper, we explore a different approach by reframing vulnerability detection as one of anomaly detection. Since the vast majority of code does not contain vulnerabilities and LLMs are trained on massive amounts of such code, vulnerable code can be viewed as an anomaly from the LLM's predicted code distribution, freeing the model from the need for labelled data to provide a learnable representation of vulnerable code. Leveraging this perspective, we demonstrate that LLMs trained for code generation exhibit a significant gap in prediction accuracy when prompted to reconstruct vulnerable versus non-vulnerable code. Using this insight, we implement ANVIL, a detector that identifies software vulnerabilities at line-level granularity. Our experiments explore the discriminating power of different anomaly scoring methods, as well as the sensitivity of ANVIL to context size. We also study the effectiveness of ANVIL on various LLM families, and conduct leakage experiments on vulnerabilities that were discovered after the knowledge cutoff of our evaluated LLMs. On a collection of vulnerabilities from the Magma benchmark, ANVIL outperforms state-of-the-art line-level vulnerability detectors, LineVul and LineVD, which have been trained with labelled data, despite ANVIL having never been trained with labelled vulnerabilities. Specifically, our approach achieves $1.62\times$ to $2.18\times$ better Top-5 accuracies and $1.02\times$ to $1.29\times$ times better ROC scores on line-level vulnerability detection tasks.

Title: An Extremely Data-efficient and Generative LLM-based Reinforcement Learning Agent for Recommenders

• Authors: Shuang Feng, Grace Feng
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.16032
• Pdf URL: https://arxiv.org/pdf/2408.16032
• Copy Paste: [[2408.16032]] An Extremely Data-efficient and Generative LLM-based Reinforcement Learning Agent for Recommenders(https://arxiv.org/abs/2408.16032)
• Keywords: generative, large language model
• Abstract: Recent advancements in large language models (LLMs) have enabled understanding webpage contexts, product details, and human instructions. Utilizing LLMs as the foundational architecture for either reward models or policies in reinforcement learning has gained popularity -- a notable achievement is the success of InstructGPT. RL algorithms have been instrumental in maximizing long-term customer satisfaction and avoiding short-term, myopic goals in industrial recommender systems, which often rely on deep learning models to predict immediate clicks or purchases. In this project, several RL methods are implemented and evaluated using the WebShop benchmark environment, data, simulator, and pre-trained model checkpoints. The goal is to train an RL agent to maximize the purchase reward given a detailed human instruction describing a desired product. The RL agents are developed by fine-tuning a pre-trained BERT model with various objectives, learning from preferences without a reward model, and employing contemporary training techniques such as Proximal Policy Optimization (PPO) as used in InstructGPT, and Direct Preference Optimization (DPO). This report also evaluates the RL agents trained using generative trajectories. Evaluations were conducted using Thompson sampling in the WebShop simulator environment. The simulated online experiments demonstrate that agents trained on generated trajectories exhibited comparable task performance to those trained using human trajectories. This has demonstrated an example of an extremely low-cost data-efficient way of training reinforcement learning agents. Also, with limited training time (<2hours), without utilizing any images, a DPO agent achieved a 19% success rate after approximately 3000 steps or 30 minutes of training on T4 GPUs, compared to a PPO agent, which reached a 15% success rate.

Title: Ethical Hacking and its role in Cybersecurity

• Authors: Fatima Asif, Fatima Sohail, Zuhaib Hussain Butt, Faiz Nasir, Nida Asgar
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16033
• Pdf URL: https://arxiv.org/pdf/2408.16033
• Copy Paste: [[2408.16033]] Ethical Hacking and its role in Cybersecurity(https://arxiv.org/abs/2408.16033)
• Keywords: security, defense
• Abstract: This review paper investigates the diverse functions of ethical hacking within modern cybersecurity. By integrating current research, it analyzes the progression of ethical hacking techniques,their use in identifying vulnerabilities and conducting penetration tests, and their influence on strengthening organizational security. Additionally, the paper discusses the ethical considerations, legal contexts and challenges that arises with ethical hacking. This review ultimately enhances the understanding of how ethical hacking can bolster cybersecurity defenses.

Title: Systematic Evaluation of Synthetic Data Augmentation for Multi-class NetFlow Traffic

• Authors: Maximilian Wolf, Dieter Landes, Andreas Hotho, Daniel Schlör
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16034
• Pdf URL: https://arxiv.org/pdf/2408.16034
• Copy Paste: [[2408.16034]] Systematic Evaluation of Synthetic Data Augmentation for Multi-class NetFlow Traffic(https://arxiv.org/abs/2408.16034)
• Keywords: attack, generative
• Abstract: The detection of cyber-attacks in computer networks is a crucial and ongoing research challenge. Machine learning-based attack classification offers a promising solution, as these models can be continuously updated with new data, enhancing the effectiveness of network intrusion detection systems (NIDS). Unlike binary classification models that simply indicate the presence of an attack, multi-class models can identify specific types of attacks, allowing for more targeted and effective incident responses. However, a significant drawback of these classification models is their sensitivity to imbalanced training data. Recent advances suggest that generative models can assist in data augmentation, claiming to offer superior solutions for imbalanced datasets. Classical balancing methods, although less novel, also provide potential remedies for this issue. Despite these claims, a comprehensive comparison of these methods within the NIDS domain is lacking. Most existing studies focus narrowly on individual methods, making it difficult to compare results due to varying experimental setups. To close this gap, we designed a systematic framework to compare classical and generative resampling methods for class balancing across multiple popular classification models in the NIDS domain, evaluated on several NIDS benchmark datasets. Our experiments indicate that resampling methods for balancing training data do not reliably improve classification performance. Although some instances show performance improvements, the majority of results indicate decreased performance, with no consistent trend in favor of a specific resampling technique enhancing a particular classifier.

Title: Fairness, Accuracy, and Unreliable Data

• Authors: Kevin Stangl
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16040
• Pdf URL: https://arxiv.org/pdf/2408.16040
• Copy Paste: [[2408.16040]] Fairness, Accuracy, and Unreliable Data(https://arxiv.org/abs/2408.16040)
• Keywords: robust, fair
• Abstract: This thesis investigates three areas targeted at improving the reliability of machine learning; fairness in machine learning, strategic classification, and algorithmic robustness. Each of these domains has special properties or structure that can complicate learning. A theme throughout this thesis is thinking about ways in which a `plain' empirical risk minimization algorithm will be misleading or ineffective because of a mis-match between classical learning theory assumptions and specific properties of some data distribution in the wild. Theoretical understanding in eachof these domains can help guide best practices and allow for the design of effective, reliable, and robust systems.

Title: Scaling Up Diffusion and Flow-based XGBoost Models

• Authors: Jesse C. Cresswell, Taewoo Kim
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16046
• Pdf URL: https://arxiv.org/pdf/2408.16046
• Copy Paste: [[2408.16046]] Scaling Up Diffusion and Flow-based XGBoost Models(https://arxiv.org/abs/2408.16046)
• Keywords: diffusion, generative
• Abstract: Novel machine learning methods for tabular data generation are often developed on small datasets which do not match the scale required for scientific applications. We investigate a recent proposal to use XGBoost as the function approximator in diffusion and flow-matching models on tabular data, which proved to be extremely memory intensive, even on tiny datasets. In this work, we conduct a critical analysis of the existing implementation from an engineering perspective, and show that these limitations are not fundamental to the method; with better implementation it can be scaled to datasets 370x larger than previously used. Our efficient implementation also unlocks scaling models to much larger sizes which we show directly leads to improved performance on benchmark tasks. We also propose algorithmic improvements that can further benefit resource usage and model performance, including multi-output trees which are well-suited to generative modeling. Finally, we present results on large-scale scientific datasets derived from experimental particle physics as part of the Fast Calorimeter Simulation Challenge. Code is available at this https URL.

Title: 3D Reconstruction with Spatial Memory

• Authors: Hengyi Wang, Lourdes Agapito
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16061
• Pdf URL: https://arxiv.org/pdf/2408.16061
• Copy Paste: [[2408.16061]] 3D Reconstruction with Spatial Memory(https://arxiv.org/abs/2408.16061)
• Keywords: transformer
• Abstract: We present Spann3R, a novel approach for dense 3D reconstruction from ordered or unordered image collections. Built on the DUSt3R paradigm, Spann3R uses a transformer-based architecture to directly regress pointmaps from images without any prior knowledge of the scene or camera parameters. Unlike DUSt3R, which predicts per image-pair pointmaps each expressed in its local coordinate frame, Spann3R can predict per-image pointmaps expressed in a global coordinate system, thus eliminating the need for optimization-based global alignment. The key idea of Spann3R is to manage an external spatial memory that learns to keep track of all previous relevant 3D information. Spann3R then queries this spatial memory to predict the 3D structure of the next frame in a global coordinate system. Taking advantage of DUSt3R's pre-trained weights, and further fine-tuning on a subset of datasets, Spann3R shows competitive performance and generalization ability on various unseen datasets and can process ordered image collections in real time. Project page: \url{this https URL}

Title: Using Large Language Models to Create AI Personas for Replication and Prediction of Media Effects: An Empirical Test of 133 Published Experimental Research Findings

• Authors: Leo Yeykelis, Kaavya Pichai, James J. Cummings, Byron Reeves
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16073
• Pdf URL: https://arxiv.org/pdf/2408.16073
• Copy Paste: [[2408.16073]] Using Large Language Models to Create AI Personas for Replication and Prediction of Media Effects: An Empirical Test of 133 Published Experimental Research Findings(https://arxiv.org/abs/2408.16073)
• Keywords: large language model
• Abstract: This report analyzes the potential for large language models (LLMs) to expedite accurate replication of published message effects studies. We tested LLM-powered participants (personas) by replicating 133 experimental findings from 14 papers containing 45 recent studies in the Journal of Marketing (January 2023-May 2024). We used a new software tool, Viewpoints AI (this https URL), that takes study designs, stimuli, and measures as input, automatically generates prompts for LLMs to act as a specified sample of unique personas, and collects their responses to produce a final output in the form of a complete dataset and statistical analysis. The underlying LLM used was Anthropic's Claude Sonnet 3.5. We generated 19,447 AI personas to replicate these studies with the exact same sample attributes, study designs, stimuli, and measures reported in the original human research. Our LLM replications successfully reproduced 76% of the original main effects (84 out of 111), demonstrating strong potential for AI-assisted replication of studies in which people respond to media stimuli. When including interaction effects, the overall replication rate was 68% (90 out of 133). The use of LLMs to replicate and accelerate marketing research on media effects is discussed with respect to the replication crisis in social science, potential solutions to generalizability problems in sampling subjects and experimental conditions, and the ability to rapidly test consumer responses to various media stimuli. We also address the limitations of this approach, particularly in replicating complex interaction effects in media response studies, and suggest areas for future research and improvement in AI-assisted experimental replication of media effects.

Title: Ensuring Equitable Financial Decisions: Leveraging Counterfactual Fairness and Deep Learning for Bias

• Authors: Saish Shinde
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16088
• Pdf URL: https://arxiv.org/pdf/2408.16088
• Copy Paste: [[2408.16088]] Ensuring Equitable Financial Decisions: Leveraging Counterfactual Fairness and Deep Learning for Bias(https://arxiv.org/abs/2408.16088)
• Keywords: fair
• Abstract: Concerns regarding fairness and bias have been raised in recent years due to the growing use of machine learning models in crucial decision-making processes, especially when it comes to delicate characteristics like gender. In order to address biases in machine learning models, this research paper investigates advanced bias mitigation techniques, with a particular focus on counterfactual fairness in conjunction with data augmentation. The study looks into how these integrated approaches can lessen gender bias in the financial industry, specifically in loan approval procedures. We show that these approaches are effective in achieving more equitable results through thorough testing and assessment on a skewed financial dataset. The findings emphasize how crucial it is to use fairness-aware techniques when creating machine learning models in order to guarantee morally righteous and impartial decision-making.

Title: Structured Event Reasoning with Large Language Models

• Authors: Li Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16098
• Pdf URL: https://arxiv.org/pdf/2408.16098
• Copy Paste: [[2408.16098]] Structured Event Reasoning with Large Language Models(https://arxiv.org/abs/2408.16098)
• Keywords: interpretability, large language model
• Abstract: Reasoning about real-life events is a unifying challenge in AI and NLP that has profound utility in a variety of domains, while fallacy in high-stake applications could be catastrophic. Able to work with diverse text in these domains, large language models (LLMs) have proven capable of answering questions and solving problems. However, I show that end-to-end LLMs still systematically fail to reason about complex events, and they lack interpretability due to their black-box nature. To address these issues, I propose three general approaches to use LLMs in conjunction with a structured representation of events. The first is a language-based representation involving relations of sub-events that can be learned by LLMs via fine-tuning. The second is a semi-symbolic representation involving states of entities that can be predicted and leveraged by LLMs via few-shot prompting. The third is a fully symbolic representation that can be predicted by LLMs trained with structured data and be executed by symbolic solvers. On a suite of event reasoning tasks spanning common-sense inference and planning, I show that each approach greatly outperforms end-to-end LLMs with more interpretability. These results suggest manners of synergy between LLMs and structured representations for event reasoning and beyond.

Title: LLMSecCode: Evaluating Large Language Models for Secure Coding

• Authors: Anton Rydén, Erik Näslund, Elad Michael Schiller, Magnus Almgren
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.16100
• Pdf URL: https://arxiv.org/pdf/2408.16100
• Copy Paste: [[2408.16100]] LLMSecCode: Evaluating Large Language Models for Secure Coding(https://arxiv.org/abs/2408.16100)
• Keywords: secure, security, large language model
• Abstract: The rapid deployment of Large Language Models (LLMs) requires careful consideration of their effect on cybersecurity. Our work aims to improve the selection process of LLMs that are suitable for facilitating Secure Coding (SC). This raises challenging research questions, such as (RQ1) Which functionality can streamline the LLM evaluation? (RQ2) What should the evaluation measure? (RQ3) How to attest that the evaluation process is impartial? To address these questions, we introduce LLMSecCode, an open-source evaluation framework designed to assess LLM SC capabilities objectively. We validate the LLMSecCode implementation through experiments. When varying parameters and prompts, we find a 10% and 9% difference in performance, respectively. We also compare some results to reliable external actors, where our results show a 5% difference. We strive to ensure the ease of use of our open-source framework and encourage further development by external actors. With LLMSecCode, we hope to encourage the standardization and benchmarking of LLMs' capabilities in security-oriented code and tasks.

Title: Variational Mode Decomposition and Linear Embeddings are What You Need For Time-Series Forecasting

• Authors: Hafizh Raihan Kurnia Putra, Novanto Yudistira, Tirana Noor Fatyanosa
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2408.16122
• Pdf URL: https://arxiv.org/pdf/2408.16122
• Copy Paste: [[2408.16122]] Variational Mode Decomposition and Linear Embeddings are What You Need For Time-Series Forecasting(https://arxiv.org/abs/2408.16122)
• Keywords: robust
• Abstract: Time-series forecasting often faces challenges due to data volatility, which can lead to inaccurate predictions. Variational Mode Decomposition (VMD) has emerged as a promising technique to mitigate volatility by decomposing data into distinct modes, thereby enhancing forecast accuracy. In this study, we integrate VMD with linear models to develop a robust forecasting framework. Our approach is evaluated on 13 diverse datasets, including ETTm2, WindTurbine, M4, and 10 air quality datasets from various Southeast Asian cities. The effectiveness of the VMD strategy is assessed by comparing Root Mean Squared Error (RMSE) values from models utilizing VMD against those without it. Additionally, we benchmark linear-based models against well-known neural network architectures such as LSTM, BLSTM, and RNN. The results demonstrate a significant reduction in RMSE across nearly all models following VMD application. Notably, the Linear + VMD model achieved the lowest average RMSE in univariate forecasting at 0.619. In multivariate forecasting, the DLinear + VMD model consistently outperformed others, attaining the lowest RMSE across all datasets with an average of 0.019. These findings underscore the effectiveness of combining VMD with linear models for superior time-series forecasting.

Title: ChartEye: A Deep Learning Framework for Chart Information Extraction

• Authors: Osama Mustafa, Muhammad Khizer Ali, Momina Moetesum, Imran Siddiqi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16123
• Pdf URL: https://arxiv.org/pdf/2408.16123
• Copy Paste: [[2408.16123]] ChartEye: A Deep Learning Framework for Chart Information Extraction(https://arxiv.org/abs/2408.16123)
• Keywords: extraction, transformer, generative
• Abstract: The widespread use of charts and infographics as a means of data visualization in various domains has inspired recent research in automated chart understanding. However, information extraction from chart images is a complex multitasked process due to style variations and, as a consequence, it is challenging to design an end-to-end system. In this study, we propose a deep learning-based framework that provides a solution for key steps in the chart information extraction pipeline. The proposed framework utilizes hierarchal vision transformers for the tasks of chart-type and text-role classification, while YOLOv7 for text detection. The detected text is then enhanced using Super Resolution Generative Adversarial Networks to improve the recognition output of the OCR. Experimental results on a benchmark dataset show that our proposed framework achieves excellent performance at every stage with F1-scores of 0.97 for chart-type classification, 0.91 for text-role classification, and a mean Average Precision of 0.95 for text detection.

Title: Development of a cyber risk assessment tool for Irish small business owners

• Authors: Miriam Curtin, Brian Sheehan, Melanie Gruben, Nikoletta Kozma, Gillian O'Carroll, Hazel Murray
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16124
• Pdf URL: https://arxiv.org/pdf/2408.16124
• Copy Paste: [[2408.16124]] Development of a cyber risk assessment tool for Irish small business owners(https://arxiv.org/abs/2408.16124)
• Keywords: security
• Abstract: Small and medium enterprises (SMEs) are increasingly vulnerable to cyber threats due to limited resources and cybersecurity expertise, in addition to an increasingly hostile cyber threat environment at national and international levels. This study aims to improve the cyber resilience amongst SMEs by developing a national risk assessment tool. This research is guided by three key questions: 1. What current international SME risk assessment tools are available and supported or endorsed by national cybersecurity centres? 2. How can a risk assessment tool be created that is accessible to SME owners with little to no cybersecurity knowledge? 3. What are the key areas of cybersecurity risks for SMEs? To answer these questions, a comprehensive review of existing risk assessment tools was carried out. Through iterative collaboration with SMEs, the development of a user-friendly tool that simplifies risk for non-expert users was made possible.

Title: Using Backbone Foundation Model for Evaluating Fairness in Chest Radiography Without Demographic Data

• Authors: Dilermando Queiroz, André Anjos, Lilian Berton
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16130
• Pdf URL: https://arxiv.org/pdf/2408.16130
• Copy Paste: [[2408.16130]] Using Backbone Foundation Model for Evaluating Fairness in Chest Radiography Without Demographic Data(https://arxiv.org/abs/2408.16130)
• Keywords: protect, robust, fair
• Abstract: Ensuring consistent performance across diverse populations and incorporating fairness into machine learning models are crucial for advancing medical image diagnostics and promoting equitable healthcare. However, many databases do not provide protected attributes or contain unbalanced representations of demographic groups, complicating the evaluation of model performance across different demographics and the application of bias mitigation techniques that rely on these attributes. This study aims to investigate the effectiveness of using the backbone of Foundation Models as an embedding extractor for creating groups that represent protected attributes, such as gender and age. We propose utilizing these groups in different stages of bias mitigation, including pre-processing, in-processing, and evaluation. Using databases in and out-of-distribution scenarios, it is possible to identify that the method can create groups that represent gender in both databases and reduce in 4.44% the difference between the gender attribute in-distribution and 6.16% in out-of-distribution. However, the model lacks robustness in handling age attributes, underscoring the need for more fundamentally fair and robust Foundation models. These findings suggest a role in promoting fairness assessment in scenarios where we lack knowledge of attributes, contributing to the development of more equitable medical diagnostics.

Title: Trustless Distributed Symmetric-key Encryption

• Authors: Florian Le Mouël, Maxime Godon, Renaud Brien, Erwan Beurier, Nora Boulahia-Cuppens, Frédéric Cuppens
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16137
• Pdf URL: https://arxiv.org/pdf/2408.16137
• Copy Paste: [[2408.16137]] Trustless Distributed Symmetric-key Encryption(https://arxiv.org/abs/2408.16137)
• Keywords: security, protect, attack
• Abstract: Threshold cryptography has gained momentum in the last decades as a mechanism to protect long term secret keys. Rather than having a single secret key, this allows to distribute the ability to perform a cryptographic operation such as signing or encrypting. Threshold cryptographic operations are shared among different parties such that a threshold number of them must participate in order to run the operation. This makes the job of an attacker strictly more difficult in the sense that they would have to corrupt at least a threshold number of parties to breach the security. Most works in this field focus on asymmetric-key schemes that allow threshold signing or decrypting. We focus on the symmetric-key setting, allowing both threshold encryption and threshold decryption. Previous work relies on the presence of a trusted third party. Such a party may not exist in some use cases, and it represents a single point of failure. We propose to remove the requirement of a trusted third party by designing a dealer-free setup in which no entity can at any point obtain full knowledge of the secret keys. We implement a proof of concept of our construction in Python. We evaluate the proof of concept with timing metrics to compare to theoretical expectations and assess the cost in complexity of not relying on a trusted third party. While the setup phase suffers moderate additional cost, the encryption and decryption phases perform the same as the original algorithm.

Title: SoK: Identifying Limitations and Bridging Gaps of Cybersecurity Capability Maturity Models (CCMMs)

• Authors: Lasini Liyanage, Nalin Asanka Gamagedara Arachchilage, Giovanni Russello
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16140
• Pdf URL: https://arxiv.org/pdf/2408.16140
• Copy Paste: [[2408.16140]] SoK: Identifying Limitations and Bridging Gaps of Cybersecurity Capability Maturity Models (CCMMs)(https://arxiv.org/abs/2408.16140)
• Keywords: security
• Abstract: In the rapidly evolving digital landscape, where organisations are increasingly vulnerable to cybersecurity threats, Cybersecurity Capability Maturity Models (CCMMs) emerge as pivotal tools in enhancing organisational cybersecurity posture. CCMMs provide a structured framework to guide organisations in assessing their current cybersecurity capabilities, identifying critical gaps, and prioritising improvements. However, the full potential of CCMMs is often not realised due to inherent limitations within the models and challenges encountered during their implementation and adoption processes. These limitations and challenges can significantly hamper the efficacy of CCMMs in improving cybersecurity. As a result, organisations remain vulnerable to cyber threats as they may fail to identify and address critical security gaps, implement necessary improvements or allocate resources effectively. To address these limitations and challenges, conducting a thorough investigation into existing models is essential. Therefore, we conducted a Systematic Literature Review (SLR) analysing 43 publications to identify existing CCMMs, their limitations, and the challenges organisations face when implementing and adopting them. By understanding these barriers, we aim to explore avenues for enhancing the efficacy of CCMMs, ensuring they more effectively meet the cybersecurity needs of organisational entities.

Title: Does Data-Efficient Generalization Exacerbate Bias in Foundation Models?

• Authors: Dilermando Queiroz, Anderson Carlos, Maíra Fatoretto, André Anjos, Lilian Berton, Luis Filipe Nakayama
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16154
• Pdf URL: https://arxiv.org/pdf/2408.16154
• Copy Paste: [[2408.16154]] Does Data-Efficient Generalization Exacerbate Bias in Foundation Models?(https://arxiv.org/abs/2408.16154)
• Keywords: robust, fair
• Abstract: Foundation models have emerged as robust models with label efficiency in diverse domains. In medical imaging, these models contribute to the advancement of medical diagnoses due to the difficulty in obtaining labeled data. However, it is unclear whether using a large amount of unlabeled data, biased by the presence of sensitive attributes during pre-training, influences the fairness of the model. This research examines the bias in the Foundation model (RetFound) when it is applied to fine-tune the Brazilian Multilabel Ophthalmological Dataset (BRSET), which has a different population than the pre-training dataset. The model evaluation, in comparison with supervised learning, shows that the Foundation Model has the potential to reduce the gap between the maximum AUC and minimum AUC evaluations across gender and age groups. However, in a data-efficient generalization, the model increases the bias when the data amount decreases. These findings suggest that when deploying a Foundation Model in real-life scenarios with limited data, the possibility of fairness issues should be considered.

Title: FRACTURED-SORRY-Bench: Framework for Revealing Attacks in Conversational Turns Undermining Refusal Efficacy and Defenses over SORRY-Bench

• Authors: Aman Priyanshu, Supriti Vijay
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16163
• Pdf URL: https://arxiv.org/pdf/2408.16163
• Copy Paste: [[2408.16163]] FRACTURED-SORRY-Bench: Framework for Revealing Attacks in Conversational Turns Undermining Refusal Efficacy and Defenses over SORRY-Bench(https://arxiv.org/abs/2408.16163)
• Keywords: defense, attack, robust, large language model
• Abstract: This paper introduces FRACTURED-SORRY-Bench, a framework for evaluating the safety of Large Language Models (LLMs) against multi-turn conversational attacks. Building upon the SORRY-Bench dataset, we propose a simple yet effective method for generating adversarial prompts by breaking down harmful queries into seemingly innocuous sub-questions. Our approach achieves a maximum increase of +46.22\% in Attack Success Rates (ASRs) across GPT-4, GPT-4o, GPT-4o-mini, and GPT-3.5-Turbo models compared to baseline methods. We demonstrate that this technique poses a challenge to current LLM safety measures and highlights the need for more robust defenses against subtle, multi-turn attacks.

Title: Free Lunch in the Forest: Functionally-Identical Pruning of Boosted Tree Ensembles

• Authors: Youssouf Emine, Alexandre Forel, Idriss Malek, Thibaut Vidal
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2408.16167
• Pdf URL: https://arxiv.org/pdf/2408.16167
• Copy Paste: [[2408.16167]] Free Lunch in the Forest: Functionally-Identical Pruning of Boosted Tree Ensembles(https://arxiv.org/abs/2408.16167)
• Keywords: interpretability
• Abstract: Tree ensembles, including boosting methods, are highly effective and widely used for tabular data. However, large ensembles lack interpretability and require longer inference times. We introduce a method to prune a tree ensemble into a reduced version that is "functionally identical" to the original model. In other words, our method guarantees that the prediction function stays unchanged for any possible input. As a consequence, this pruning algorithm is lossless for any aggregated metric. We formalize the problem of functionally identical pruning on ensembles, introduce an exact optimization model, and provide a fast yet highly effective method to prune large ensembles. Our algorithm iteratively prunes considering a finite set of points, which is incrementally augmented using an adversarial model. In multiple computational experiments, we show that our approach is a "free lunch", significantly reducing the ensemble size without altering the model's behavior. Thus, we can preserve state-of-the-art performance at a fraction of the original model's size.

Title: LeMON: Learning to Learn Multi-Operator Networks

• Authors: Jingmin Sun, Zecheng Zhang, Hayden Schaeffer
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16168
• Pdf URL: https://arxiv.org/pdf/2408.16168
• Copy Paste: [[2408.16168]] LeMON: Learning to Learn Multi-Operator Networks(https://arxiv.org/abs/2408.16168)
• Keywords: data-free
• Abstract: Single-operator learning involves training a deep neural network to learn a specific operator, whereas recent work in multi-operator learning uses an operator embedding structure to train a single neural network on data from multiple operators. Thus, multi-operator learning is capable of predicting a range of operators within one model. In this work, we propose pretraining and fine-tuning strategies for solving PDEs using multi-operator learning. One key aspect is that by increasing the number of families of operators used in pretraining, a PDE foundation model can be fine-tuned to downstream tasks involving new PDEs with a limited number of samples, thus outperforming single operator neural networks. Specifically, a multi-operator learning model pre-trained with data from diverse PDE families can predict unseen operators after fine-tuning with only a limited number of operators from the new family, enabling them to serve as a data-free PDE solver. We also show that the proposed training and fine-tuning method is able to predict new operators in zero-shot prediction without samples. Additionally, we introduce a PDE-agnostic meta-learning algorithm to improve the adaptability of the model to various PDEs by providing a better parameter initialization process. To address the needs of applications with limited computing resources, we explore low-rank adaptation methods that reduce computational costs while enhancing solver accuracy. Lastly, by examining the scaling law with respect to the number of operator families, we establish and highlight its potential for broad adaptation in PDE-solving tasks.

Title: Simulating realistic short tandem repeat capillary electrophoretic signal using a generative adversarial network

• Authors: Duncan Taylor, Melissa Humphries
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16169
• Pdf URL: https://arxiv.org/pdf/2408.16169
• Copy Paste: [[2408.16169]] Simulating realistic short tandem repeat capillary electrophoretic signal using a generative adversarial network(https://arxiv.org/abs/2408.16169)
• Keywords: robust, generative
• Abstract: DNA profiles are made up from multiple series of electrophoretic signal measuring fluorescence over time. Typically, human DNA analysts 'read' DNA profiles using their experience to distinguish instrument noise, artefactual signal, and signal corresponding to DNA fragments of interest. Recent work has developed an artificial neural network, ANN, to carry out the task of classifying fluorescence types into categories in DNA profile electrophoretic signal. But the creation of the necessarily large amount of labelled training data for the ANN is time consuming and expensive, and a limiting factor in the ability to robustly train the ANN. If realistic, prelabelled, training data could be simulated then this would remove the barrier to training an ANN with high efficacy. Here we develop a generative adversarial network, GAN, modified from the pix2pix GAN to achieve this task. With 1078 DNA profiles we train the GAN and achieve the ability to simulate DNA profile information, and then use the generator from the GAN as a 'realism filter' that applies the noise and artefact elements exhibited in typical electrophoretic signal.

Title: PolarBEVDet: Exploring Polar Representation for Multi-View 3D Object Detection in Bird's-Eye-View

• Authors: Zichen Yu, Quanli Liu, Wei Wang, Liyong Zhang, Xiaoguang Zhao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16200
• Pdf URL: https://arxiv.org/pdf/2408.16200
• Copy Paste: [[2408.16200]] PolarBEVDet: Exploring Polar Representation for Multi-View 3D Object Detection in Bird's-Eye-View(https://arxiv.org/abs/2408.16200)
• Keywords: extraction, transformer
• Abstract: Recently, LSS-based multi-view 3D object detection provides an economical and deployment-friendly solution for autonomous driving. However, all the existing LSS-based methods transform multi-view image features into a Cartesian Bird's-Eye-View(BEV) representation, which does not take into account the non-uniform image information distribution and hardly exploits the view symmetry. In this paper, in order to adapt the image information distribution and preserve the view symmetry by regular convolution, we propose to employ the polar BEV representation to substitute the Cartesian BEV representation. To achieve this, we elaborately tailor three modules: a polar view transformer to generate the polar BEV representation, a polar temporal fusion module for fusing historical polar BEV features and a polar detection head to predict the polar-parameterized representation of the object. In addition, we design a 2D auxiliary detection head and a spatial attention enhancement module to improve the quality of feature extraction in perspective view and BEV, respectively. Finally, we integrate the above improvements into a novel multi-view 3D object detector, PolarBEVDet. Experiments on nuScenes show that PolarBEVDet achieves the superior performance. The code is available at this https URL.

Title: Uni-3DAD: GAN-Inversion Aided Universal 3D Anomaly Detection on Model-free Products

• Authors: Jiayu Liu, Shancong Mou, Nathan Gaw, Yinan Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16201
• Pdf URL: https://arxiv.org/pdf/2408.16201
• Copy Paste: [[2408.16201]] Uni-3DAD: GAN-Inversion Aided Universal 3D Anomaly Detection on Model-free Products(https://arxiv.org/abs/2408.16201)
• Keywords: robust, fair
• Abstract: Anomaly detection is a long-standing challenge in manufacturing systems. Traditionally, anomaly detection has relied on human inspectors. However, 3D point clouds have gained attention due to their robustness to environmental factors and their ability to represent geometric data. Existing 3D anomaly detection methods generally fall into two categories. One compares scanned 3D point clouds with design files, assuming these files are always available. However, such assumptions are often violated in many real-world applications where model-free products exist, such as fresh produce (i.e., ``Cookie", ``Potato", etc.), dentures, bone, etc. The other category compares patches of scanned 3D point clouds with a library of normal patches named memory bank. However, those methods usually fail to detect incomplete shapes, which is a fairly common defect type (i.e., missing pieces of different products). The main challenge is that missing areas in 3D point clouds represent the absence of scanned points. This makes it infeasible to compare the missing region with existing point cloud patches in the memory bank. To address these two challenges, we proposed a unified, unsupervised 3D anomaly detection framework capable of identifying all types of defects on model-free products. Our method integrates two detection modules: a feature-based detection module and a reconstruction-based detection module. Feature-based detection covers geometric defects, such as dents, holes, and cracks, while the reconstruction-based method detects missing regions. Additionally, we employ a One-class Support Vector Machine (OCSVM) to fuse the detection results from both modules. The results demonstrate that (1) our proposed method outperforms the state-of-the-art methods in identifying incomplete shapes and (2) it still maintains comparable performance with the SOTA methods in detecting all other types of anomalies.

Title: Short-Term Electricity-Load Forecasting by Deep Learning: A Comprehensive Survey

• Authors: Qi Dong, Rubing Huang, Chenhui Cui, Dave Towey, Ling Zhou, Jinyu Tian, Jianzhou Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16202
• Pdf URL: https://arxiv.org/pdf/2408.16202
• Copy Paste: [[2408.16202]] Short-Term Electricity-Load Forecasting by Deep Learning: A Comprehensive Survey(https://arxiv.org/abs/2408.16202)
• Keywords: extraction
• Abstract: Short-Term Electricity-Load Forecasting (STELF) refers to the prediction of the immediate demand (in the next few hours to several days) for the power system. Various external factors, such as weather changes and the emergence of new electricity consumption scenarios, can impact electricity demand, causing load data to fluctuate and become non-linear, which increases the complexity and difficulty of STELF. In the past decade, deep learning has been applied to STELF, modeling and predicting electricity demand with high accuracy, and contributing significantly to the development of STELF. This paper provides a comprehensive survey on deep-learning-based STELF over the past ten years. It examines the entire forecasting process, including data pre-processing, feature extraction, deep-learning modeling and optimization, and results evaluation. This paper also identifies some research challenges and potential research directions to be further investigated in future work.

Title: ReXamine-Global: A Framework for Uncovering Inconsistencies in Radiology Report Generation Metrics

• Authors: Oishi Banerjee, Agustina Saenz, Kay Wu, Warren Clements, Adil Zia, Dominic Buensalido, Helen Kavnoudias, Alain S. Abi-Ghanem, Nour El Ghawi, Cibele Luna, Patricia Castillo, Khaled Al-Surimi, Rayyan A. Daghistani, Yuh-Min Chen, Heng-sheng Chao, Lars Heiliger, Moon Kim, Johannes Haubold, Frederic Jonske, Pranav Rajpurkar
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16208
• Pdf URL: https://arxiv.org/pdf/2408.16208
• Copy Paste: [[2408.16208]] ReXamine-Global: A Framework for Uncovering Inconsistencies in Radiology Report Generation Metrics(https://arxiv.org/abs/2408.16208)
• Keywords: robust, generative
• Abstract: Given the rapidly expanding capabilities of generative AI models for radiology, there is a need for robust metrics that can accurately measure the quality of AI-generated radiology reports across diverse hospitals. We develop ReXamine-Global, a LLM-powered, multi-site framework that tests metrics across different writing styles and patient populations, exposing gaps in their generalization. First, our method tests whether a metric is undesirably sensitive to reporting style, providing different scores depending on whether AI-generated reports are stylistically similar to ground-truth reports or not. Second, our method measures whether a metric reliably agrees with experts, or whether metric and expert scores of AI-generated report quality diverge for some sites. Using 240 reports from 6 hospitals around the world, we apply ReXamine-Global to 7 established report evaluation metrics and uncover serious gaps in their generalizability. Developers can apply ReXamine-Global when designing new report evaluation metrics, ensuring their robustness across sites. Additionally, our analysis of existing metrics can guide users of those metrics towards evaluation procedures that work reliably at their sites of interest.

Title: M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretation

• Authors: Jonggwon Park, Soobum Kim, Byungmu Yoon, Jihun Hyun, Kyoyun Choi
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16213
• Pdf URL: https://arxiv.org/pdf/2408.16213
• Copy Paste: [[2408.16213]] M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretation(https://arxiv.org/abs/2408.16213)
• Keywords: large language model
• Abstract: The rapid evolution of artificial intelligence, especially in large language models (LLMs), has significantly impacted various domains, including healthcare. In chest X-ray (CXR) analysis, previous studies have employed LLMs, but with limitations: either underutilizing the multi-tasking capabilities of LLMs or lacking clinical accuracy. This paper presents M4CXR, a multi-modal LLM designed to enhance CXR interpretation. The model is trained on a visual instruction-following dataset that integrates various task-specific datasets in a conversational format. As a result, the model supports multiple tasks such as medical report generation (MRG), visual grounding, and visual question answering (VQA). M4CXR achieves state-of-the-art clinical accuracy in MRG by employing a chain-of-thought prompting strategy, in which it identifies findings in CXR images and subsequently generates corresponding reports. The model is adaptable to various MRG scenarios depending on the available inputs, such as single-image, multi-image, and multi-study contexts. In addition to MRG, M4CXR performs visual grounding at a level comparable to specialized models and also demonstrates outstanding performance in VQA. Both quantitative and qualitative assessments reveal M4CXR's versatility in MRG, visual grounding, and VQA, while consistently maintaining clinical accuracy.

Title: Training-free Video Temporal Grounding using Large-scale Pre-trained Models

• Authors: Minghang Zheng, Xinhao Cai, Qingchao Chen, Yuxin Peng, Yang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16219
• Pdf URL: https://arxiv.org/pdf/2408.16219
• Copy Paste: [[2408.16219]] Training-free Video Temporal Grounding using Large-scale Pre-trained Models(https://arxiv.org/abs/2408.16219)
• Keywords: large language model
• Abstract: Video temporal grounding aims to identify video segments within untrimmed videos that are most relevant to a given natural language query. Existing video temporal localization models rely on specific datasets for training and have high data collection costs, but they exhibit poor generalization capability under the across-dataset and out-of-distribution (OOD) settings. In this paper, we propose a Training-Free Video Temporal Grounding (TFVTG) approach that leverages the ability of pre-trained large models. A naive baseline is to enumerate proposals in the video and use the pre-trained visual language models (VLMs) to select the best proposal according to the vision-language alignment. However, most existing VLMs are trained on image-text pairs or trimmed video clip-text pairs, making it struggle to (1) grasp the relationship and distinguish the temporal boundaries of multiple events within the same video; (2) comprehend and be sensitive to the dynamic transition of events (the transition from one event to another) in the video. To address these issues, we propose leveraging large language models (LLMs) to analyze multiple sub-events contained in the query text and analyze the temporal order and relationships between these events. Secondly, we split a sub-event into dynamic transition and static status parts and propose the dynamic and static scoring functions using VLMs to better evaluate the relevance between the event and the description. Finally, for each sub-event description, we use VLMs to locate the top-k proposals and leverage the order and relationships between sub-events provided by LLMs to filter and integrate these proposals. Our method achieves the best performance on zero-shot video temporal grounding on Charades-STA and ActivityNet Captions datasets without any training and demonstrates better generalization capabilities in cross-dataset and OOD settings.

Title: LightSLH: Provable and Low-Overhead Spectre v1 Mitigation through Targeted Instruction Hardening

• Authors: Yiming Zhu, Wenchao Huang, Yan Xiong
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16220
• Pdf URL: https://arxiv.org/pdf/2408.16220
• Copy Paste: [[2408.16220]] LightSLH: Provable and Low-Overhead Spectre v1 Mitigation through Targeted Instruction Hardening(https://arxiv.org/abs/2408.16220)
• Keywords: security, protect
• Abstract: Several software mitigations have been proposed to defend against Spectre vulnerabilities. However, these countermeasures often suffer from high performance overhead, largely due to unnecessary protections. We propose LightSLH, designed to mitigate this overhead by hardening instructions only when they are under threat from Spectre vulnerabilities. LightSLH leverages program analysis techniques based on abstract interpretation to identify all instructions that could potentially lead to Spectre vulnerabilities and provides provable protection. To enhance analysis efficiency and precision, LightSLH employs novel taint and value domains. The taint domain enables bit-level taint tracking, while the value domain allows LightSLH to analyze complex program structures such as pointers and structures. Furthermore, LightSLH uses a two-stage abstract interpretation approach to circumvent potential analysis paralysis issues. We demonstrate the security guarantees of LightSLH and evaluate its performance on cryptographic algorithm implementations from OpenSSL. LightSLH significantly reduces the overhead associated with speculative-load-hardening techniques. Our results show that LightSLH introduces no protection and thus no overhead on 4 out of the 7 studied algorithms, which contrasts with existing countermeasures that introduce additional overhead due to unnecessary hardening. Additionally, LightSLH performs, for the first time, a rigorous analysis of the security guarantees of RSA against Spectre v1, highlighting that the memory access patterns generated by the scatter-gather algorithm depend on secrets, even for observers at the cache line granularity, necessitating protection for such accesses.

Title: LLaVA-SG: Leveraging Scene Graphs as Visual Semantic Expression in Vision-Language Models

• Authors: Jingyi Wang, Jianzhong Ju, Jian Luan, Zhidong Deng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16224
• Pdf URL: https://arxiv.org/pdf/2408.16224
• Copy Paste: [[2408.16224]] LLaVA-SG: Leveraging Scene Graphs as Visual Semantic Expression in Vision-Language Models(https://arxiv.org/abs/2408.16224)
• Keywords: transformer
• Abstract: Recent advances in large vision-language models (VLMs) typically employ vision encoders based on the Vision Transformer (ViT) architecture. The division of the images into patches by ViT results in a fragmented perception, thereby hindering the visual understanding capabilities of VLMs. In this paper, we propose an innovative enhancement to address this limitation by introducing a Scene Graph Expression (SGE) module in VLMs. This module extracts and structurally expresses the complex semantic information within images, thereby improving the foundational perception and understanding abilities of VLMs. Extensive experiments demonstrate that integrating our SGE module significantly enhances the VLM's performance in vision-language tasks, indicating its effectiveness in preserving intricate semantic details and facilitating better visual understanding. Code and data would be available.

Title: Revisiting 360 Depth Estimation with PanoGabor: A New Fusion Perspective

• Authors: Zhijie Shen, Chunyu Lin, Lang Nie, Kang Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16227
• Pdf URL: https://arxiv.org/pdf/2408.16227
• Copy Paste: [[2408.16227]] Revisiting 360 Depth Estimation with PanoGabor: A New Fusion Perspective(https://arxiv.org/abs/2408.16227)
• Keywords: extraction
• Abstract: Depth estimation from a monocular 360 image is important to the perception of the entire 3D environment. However, the inherent distortion and large field of view (FoV) in 360 images pose great challenges for this task. To this end, existing mainstream solutions typically introduce additional perspective-based 360 representations (\textit{e.g.}, Cubemap) to achieve effective feature extraction. Nevertheless, regardless of the introduced representations, they eventually need to be unified into the equirectangular projection (ERP) format for the subsequent depth estimation, which inevitably reintroduces the troublesome distortions. In this work, we propose an oriented distortion-aware Gabor Fusion framework (PGFuse) to address the above challenges. First, we introduce Gabor filters that analyze texture in the frequency domain, thereby extending the receptive fields and enhancing depth cues. To address the reintroduced distortions, we design a linear latitude-aware distortion representation method to generate customized, distortion-aware Gabor filters (PanoGabor filters). Furthermore, we design a channel-wise and spatial-wise unidirectional fusion module (CS-UFM) that integrates the proposed PanoGabor filters to unify other representations into the ERP format, delivering effective and distortion-free features. Considering the orientation sensitivity of the Gabor transform, we introduce a spherical gradient constraint to stabilize this sensitivity. Experimental results on three popular indoor 360 benchmarks demonstrate the superiority of the proposed PGFuse to existing state-of-the-art solutions. Code can be available upon acceptance.

Title: Enhancing Conditional Image Generation with Explainable Latent Space Manipulation

• Authors: Kshitij Pathania
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16232
• Pdf URL: https://arxiv.org/pdf/2408.16232
• Copy Paste: [[2408.16232]] Enhancing Conditional Image Generation with Explainable Latent Space Manipulation(https://arxiv.org/abs/2408.16232)
• Keywords: diffusion
• Abstract: In the realm of image synthesis, achieving fidelity to a reference image while adhering to conditional prompts remains a significant challenge. This paper proposes a novel approach that integrates a diffusion model with latent space manipulation and gradient-based selective attention mechanisms to address this issue. Leveraging Grad-SAM (Gradient-based Selective Attention Manipulation), we analyze the cross attention maps of the cross attention layers and gradients for the denoised latent vector, deriving importance scores of elements of denoised latent vector related to the subject of interest. Using this information, we create masks at specific timesteps during denoising to preserve subjects while seamlessly integrating the reference image features. This approach ensures the faithful formation of subjects based on conditional prompts, while concurrently refining the background for a more coherent composition. Our experiments on places365 dataset demonstrate promising results, with our proposed model achieving the lowest mean and median Frechet Inception Distance (FID) scores compared to baseline models, indicating superior fidelity preservation. Furthermore, our model exhibits competitive performance in aligning the generated images with provided textual descriptions, as evidenced by high CLIP scores. These results highlight the effectiveness of our approach in both fidelity preservation and textual context preservation, offering a significant advancement in text-to-image synthesis tasks.

Title: Making the Most of your Model: Methods for Finetuning and Applying Pretrained Transformers

• Authors: Davis Yoshida
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16241
• Pdf URL: https://arxiv.org/pdf/2408.16241
• Copy Paste: [[2408.16241]] Making the Most of your Model: Methods for Finetuning and Applying Pretrained Transformers(https://arxiv.org/abs/2408.16241)
• Keywords: transformer, generative
• Abstract: This thesis provides methods and analysis of models which make progress on this goal. The techniques outlined are task agnostic, and should provide benefit when used with nearly any transformer LM. We introduce two new finetuning methods which add new capabilities to the models they are used on. The first adds a recurrence mechanism, which removes the fixed-window sized constraint and improves the efficiency of a transformer decoder. The second allows masked language models (MLMs) to be used for initialization of both the encoder and decoder of a non-autoregressive sequence-to-sequence transformer, opening up generative applications of models which were previously only used for natural language understanding tasks. We also introduce two new techniques for improving the quality of predictions of any transformer decoder without additional finetuning. One, hidden state optimization, can be applied to any transformer decoder to improve the quality of predictions at inference time, especially for few-shot classification. The other, conditional beam search, allows practitioners to search for natural language generation (NLG) model outputs with high likelihood while conditioning on the event that the output is not degenerate (e.g. empty, repetitive, etc.). Finally, we provide theoretical and empirical insights on the divergence of model-likelihood and output quality which has widely been observed in prior work. These insights apply to any model which represents a distribution over text, and apply to language models which are not transformers or even autoregressive. We argue that the NLP community has, to some extent, misunderstood the implications of these findings, and encourage a point of view which has more nuance.

Title: Large-Scale Multi-omic Biosequence Transformers for Modeling Peptide-Nucleotide Interactions

• Authors: Sully F. Chen, Robert J. Steele, Beakal Lemeneh, Shivanand P. Lad, Eric Oermann
• Subjects: cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2408.16245
• Pdf URL: https://arxiv.org/pdf/2408.16245
• Copy Paste: [[2408.16245]] Large-Scale Multi-omic Biosequence Transformers for Modeling Peptide-Nucleotide Interactions(https://arxiv.org/abs/2408.16245)
• Keywords: transformer
• Abstract: The transformer architecture has revolutionized bioinformatics and driven progress in the understanding and prediction of the properties of biomolecules. Almost all research on large-scale biosequence transformers has focused on one domain at a time (single-omic), usually nucleotides or peptides. These models have seen incredible success in downstream tasks in each domain and have achieved particularly noteworthy breakthroughs in sequences of peptides and structural modeling. However, these single-omic models are naturally incapable of modeling multi-omic tasks, one of the most biologically critical being nucleotide-peptide interactions. We present our work training the first multi-omic nucleotide-peptide foundation models. We show that these multi-omic models (MOMs) can learn joint representations between various single-omic distributions that are emergently consistent with the Central Dogma of molecular biology, despite only being trained on unlabeled biosequences. We further demonstrate that MOMs can be fine-tuned to achieve state-of-the-art results on peptide-nucleotide interaction tasks, namely predicting the change in Gibbs free energy ({\Delta}G) of the binding interaction between a given oligonucleotide and peptide, as well as the effect on this binding interaction due to mutations in the oligonucleotide sequence ({\Delta}{\Delta}G). Remarkably, we show that multi-omic biosequence transformers emergently learn useful structural information without any prior structural training, allowing us to predict which peptide residues are most involved in the peptide-nucleotide binding interaction. Lastly, we provide evidence that multi-omic biosequence models are non-inferior to foundation models trained on single-omics distributions, suggesting a more generalized or foundational approach to building these models.

Title: EvLight++: Low-Light Video Enhancement with an Event Camera: A Large-Scale Real-World Dataset, Novel Method, and More

• Authors: Kanghao Chen, Guoqiang Liang, Hangyu Li, Yunfan Lu, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16254
• Pdf URL: https://arxiv.org/pdf/2408.16254
• Copy Paste: [[2408.16254]] EvLight++: Low-Light Video Enhancement with an Event Camera: A Large-Scale Real-World Dataset, Novel Method, and More(https://arxiv.org/abs/2408.16254)
• Keywords: robust, segmentation
• Abstract: Event cameras offer significant advantages for low-light video enhancement, primarily due to their high dynamic range. Current research, however, is severely limited by the absence of large-scale, real-world, and spatio-temporally aligned event-video datasets. To address this, we introduce a large-scale dataset with over 30,000 pairs of frames and events captured under varying illumination. This dataset was curated using a robotic arm that traces a consistent non-linear trajectory, achieving spatial alignment precision under 0.03mm and temporal alignment with errors under 0.01s for 90% of the dataset. Based on the dataset, we propose \textbf{EvLight++}, a novel event-guided low-light video enhancement approach designed for robust performance in real-world scenarios. Firstly, we design a multi-scale holistic fusion branch to integrate structural and textural information from both images and events. To counteract variations in regional illumination and noise, we introduce Signal-to-Noise Ratio (SNR)-guided regional feature selection, enhancing features from high SNR regions and augmenting those from low SNR regions by extracting structural information from events. To incorporate temporal information and ensure temporal coherence, we further introduce a recurrent module and temporal loss in the whole pipeline. Extensive experiments on our and the synthetic SDSD dataset demonstrate that EvLight++ significantly outperforms both single image- and video-based methods by 1.37 dB and 3.71 dB, respectively. To further explore its potential in downstream tasks like semantic segmentation and monocular depth estimation, we extend our datasets by adding pseudo segmentation and depth labels via meticulous annotation efforts with foundation models. Experiments under diverse low-light scenes show that the enhanced results achieve a 15.97% improvement in mIoU for semantic segmentation.

Title: LoraMap: Harnessing the Power of LoRA Connections

• Authors: Hyeryun Park, Jeongwon Kwak, Dongsuk Jang, Sumin Park, Jinwook Choi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16264
• Pdf URL: https://arxiv.org/pdf/2408.16264
• Copy Paste: [[2408.16264]] LoraMap: Harnessing the Power of LoRA Connections(https://arxiv.org/abs/2408.16264)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) can benefit from mitigating hallucinations through fact-checking and overcoming substantial computational overhead with parameter-efficient techniques such as Low-Rank Adaptation (LoRA). While some studies have explored the parallel integration of multiple LoRAs, these approaches need attention to the connections between them. This paper investigates methods to establish connections among multiple LoRAs. We create three reasoning datasets tailored to fact-checking and fine-tune individual LoRAs, allowing them to view and reason from diverse perspectives. Then, we explore strategies for allocating these reasoning LoRAs and introduce LoraMap, an approach to map connections between them. The results on the fact-checking task demonstrate that the performance of LoraMap is superior to LoraHub, an existing LoRA composition method. LoraMap also outperforms with significantly fewer parameters than LoraConcat, which concatenates LoRAs and further fine-tunes them.

Title: Improving Diffusion-based Data Augmentation with Inversion Spherical Interpolation

• Authors: Yanghao Wang, Long Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16266
• Pdf URL: https://arxiv.org/pdf/2408.16266
• Copy Paste: [[2408.16266]] Improving Diffusion-based Data Augmentation with Inversion Spherical Interpolation(https://arxiv.org/abs/2408.16266)
• Keywords: diffusion
• Abstract: Data Augmentation (DA), \ie, synthesizing faithful and diverse samples to expand the original training set, is a prevalent and effective strategy to improve various visual recognition tasks. With the powerful image generation ability, diffusion-based DA has shown strong performance gains on different benchmarks. In this paper, we analyze today's diffusion-based DA methods, and argue that they cannot take account of both faithfulness and diversity, which are two critical keys for generating high-quality samples and boosting final classification performance. To this end, we propose a novel Diffusion-based Inversion Interpolation DA method: Diff-II. Specifically, Diff-II consists of three main steps: 1) Category concepts learning: Learning concept embeddings for each category. 2) Inversion interpolation: Calculating the inversion for each image, and conducting spherical interpolation for two randomly sampled inversions from the same category. 3) Two-stage denoising: Using different prompts to generate synthesized images in a coarse-to-fine manner. Extensive experiments on multiple image classification tasks (\eg, few-shot, long-tailed, and out-of-distribution classification) have demonstrated its effectiveness over state-of-the-art diffusion-based DA methods.

Title: Beyond Uncertainty: Evidential Deep Learning for Robust Video Temporal Grounding

• Authors: Kaijing Ma, Haojian Huang, Jin Chen, Haodong Chen, Pengliang Ji, Xianghao Zang, Han Fang, Chao Ban, Hao Sun, Mulin Chen, Xuelong Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16272
• Pdf URL: https://arxiv.org/pdf/2408.16272
• Copy Paste: [[2408.16272]] Beyond Uncertainty: Evidential Deep Learning for Robust Video Temporal Grounding(https://arxiv.org/abs/2408.16272)
• Keywords: robust, interpretability
• Abstract: Existing Video Temporal Grounding (VTG) models excel in accuracy but often overlook open-world challenges posed by open-vocabulary queries and untrimmed videos. This leads to unreliable predictions for noisy, corrupted, and out-of-distribution data. Adapting VTG models to dynamically estimate uncertainties based on user input can address this issue. To this end, we introduce SRAM, a robust network module that benefits from a two-stage cross-modal alignment task. More importantly, it integrates Deep Evidential Regression (DER) to explicitly and thoroughly quantify uncertainty during training, thus allowing the model to say "I do not know" in scenarios beyond its handling capacity. However, the direct application of traditional DER theory and its regularizer reveals structural flaws, leading to unintended constraints in VTG tasks. In response, we develop a simple yet effective Geom-regularizer that enhances the uncertainty learning framework from the ground up. To the best of our knowledge, this marks the first successful attempt of DER in VTG. Our extensive quantitative and qualitative results affirm the effectiveness, robustness, and interpretability of our modules and the uncertainty learning paradigm in VTG tasks. The code will be made available.

Title: SAU: A Dual-Branch Network to Enhance Long-Tailed Recognition via Generative Models

• Authors: Guangxi Li, Yinsheng Song, Mingkai Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16273
• Pdf URL: https://arxiv.org/pdf/2408.16273
• Copy Paste: [[2408.16273]] SAU: A Dual-Branch Network to Enhance Long-Tailed Recognition via Generative Models(https://arxiv.org/abs/2408.16273)
• Keywords: robust, generative
• Abstract: Long-tailed distributions in image recognition pose a considerable challenge due to the severe imbalance between a few dominant classes with numerous examples and many minority classes with few samples. Recently, the use of large generative models to create synthetic data for image classification has been realized, but utilizing synthetic data to address the challenge of long-tailed recognition remains relatively unexplored. In this work, we proposed the use of synthetic data as a complement to long-tailed datasets to eliminate the impact of data imbalance. To tackle this real-synthetic mixed dataset, we designed a two-branch model that contains Synthetic-Aware and Unaware branches (SAU). The core ideas are (1) a synthetic-unaware branch for classification that mixes real and synthetic data and treats all data equally without distinguishing between them. (2) A synthetic-aware branch for improving the robustness of the feature extractor by distinguishing between real and synthetic data and learning their discrepancies. Extensive experimental results demonstrate that our method can improve the accuracy of long-tailed image recognition. Notably, our approach achieves state-of-the-art Top-1 accuracy and significantly surpasses other methods on CIFAR-10-LT and CIFAR-100-LT datasets across various imbalance factors. Our code is available at this https URL.

Title: Enhancing AI-Driven Psychological Consultation: Layered Prompts with Large Language Models

• Authors: Rafael Souza, Jia-Hao Lim, Alexander Davis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16276
• Pdf URL: https://arxiv.org/pdf/2408.16276
• Copy Paste: [[2408.16276]] Enhancing AI-Driven Psychological Consultation: Layered Prompts with Large Language Models(https://arxiv.org/abs/2408.16276)
• Keywords: large language model
• Abstract: Psychological consultation is essential for improving mental health and well-being, yet challenges such as the shortage of qualified professionals and scalability issues limit its accessibility. To address these challenges, we explore the use of large language models (LLMs) like GPT-4 to augment psychological consultation services. Our approach introduces a novel layered prompting system that dynamically adapts to user input, enabling comprehensive and relevant information gathering. We also develop empathy-driven and scenario-based prompts to enhance the LLM's emotional intelligence and contextual understanding in therapeutic settings. We validated our approach through experiments using a newly collected dataset of psychological consultation dialogues, demonstrating significant improvements in response quality. The results highlight the potential of our prompt engineering techniques to enhance AI-driven psychological consultation, offering a scalable and accessible solution to meet the growing demand for mental health support.

Title: ART: Actually Robust Training

• Authors: Sebastian Chwilczyński, Kacper Trębacz, Karol Cyganik, Mateusz Małecki, Dariusz Brzezinski
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2408.16285
• Pdf URL: https://arxiv.org/pdf/2408.16285
• Copy Paste: [[2408.16285]] ART: Actually Robust Training(https://arxiv.org/abs/2408.16285)
• Keywords: robust, interpretability
• Abstract: Current interest in deep learning captures the attention of many programmers and researchers. Unfortunately, the lack of a unified schema for developing deep learning models results in methodological inconsistencies, unclear documentation, and problems with reproducibility. Some guidelines have been proposed, yet currently, they lack practical implementations. Furthermore, neural network training often takes on the form of trial and error, lacking a structured and thoughtful process. To alleviate these issues, in this paper, we introduce Art, a Python library designed to help automatically impose rules and standards while developing deep learning pipelines. Art divides model development into a series of smaller steps of increasing complexity, each concluded with a validation check improving the interpretability and robustness of the process. The current version of Art comes equipped with nine predefined steps inspired by Andrej Karpathy's Recipe for Training Neural Networks, a visualization dashboard, and integration with loggers such as Neptune. The code related to this paper is available at: this https URL.

Title: Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form

• Authors: Toshinori Kitamura, Tadashi Kozuno, Wataru Kumagai, Kenta Hoshino, Yohei Hosoe, Kazumi Kasaura, Masashi Hamaya, Paavo Parmas, Yutaka Matsuo
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2408.16286
• Pdf URL: https://arxiv.org/pdf/2408.16286
• Copy Paste: [[2408.16286]] Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form(https://arxiv.org/abs/2408.16286)
• Keywords: robust
• Abstract: Designing a safe policy for uncertain environments is crucial in real-world control applications. However, this challenge remains inadequately addressed within the Markov decision process (MDP) framework. This paper presents the first algorithm capable of identifying a near-optimal policy in a robust constrained MDP (RCMDP), where an optimal policy minimizes cumulative cost while satisfying constraints in the worst-case scenario across a set of environments. We first prove that the conventional Lagrangian max-min formulation with policy gradient methods can become trapped in suboptimal solutions by encountering a sum of conflicting gradients from the objective and constraint functions during its inner minimization problem. To address this, we leverage the epigraph form of the RCMDP problem, which resolves the conflict by selecting a single gradient from either the objective or the constraints. Building on the epigraph form, we propose a binary search algorithm with a policy gradient subroutine and prove that it identifies an $\varepsilon$-optimal policy in an RCMDP with $\tilde{\mathcal{O}}(\varepsilon^{-4})$ policy evaluations.

Title: OpenFGL: A Comprehensive Benchmarks for Federated Graph Learning

• Authors: Xunkai Li, Yinlin Zhu, Boyang Pang, Guochen Yan, Yeyu Yan, Zening Li, Zhengyu Wu, Wentao Zhang, Rong-Hua Li, Guoren Wang
• Subjects: cs.LG, cs.AI, cs.DB, cs.SI
• Abstract URL: https://arxiv.org/abs/2408.16288
• Pdf URL: https://arxiv.org/pdf/2408.16288
• Copy Paste: [[2408.16288]] OpenFGL: A Comprehensive Benchmarks for Federated Graph Learning(https://arxiv.org/abs/2408.16288)
• Keywords: privacy, robust, federate, fair
• Abstract: Federated graph learning (FGL) has emerged as a promising distributed training paradigm for graph neural networks across multiple local systems without direct data sharing. This approach is particularly beneficial in privacy-sensitive scenarios and offers a new perspective on addressing scalability challenges in large-scale graph learning. Despite the proliferation of FGL, the diverse motivations from practical applications, spanning various research backgrounds and experimental settings, pose a significant challenge to fair evaluation. To fill this gap, we propose OpenFGL, a unified benchmark designed for the primary FGL scenarios: Graph-FL and Subgraph-FL. Specifically, OpenFGL includes 38 graph datasets from 16 application domains, 8 federated data simulation strategies that emphasize graph properties, and 5 graph-based downstream tasks. Additionally, it offers 18 recently proposed SOTA FGL algorithms through a user-friendly API, enabling a thorough comparison and comprehensive evaluation of their effectiveness, robustness, and efficiency. Empirical results demonstrate the ability of FGL while also revealing its potential limitations, offering valuable insights for future exploration in this thriving field.

Title: Flexible framework for generating synthetic electrocardiograms and photoplethysmograms

• Authors: Katri Karhinoja, Antti Vasankari, Jukka-Pekka Sirkiä, Antti Airola, David Wong, Matti Kaisti
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2408.16291
• Pdf URL: https://arxiv.org/pdf/2408.16291
• Copy Paste: [[2408.16291]] Flexible framework for generating synthetic electrocardiograms and photoplethysmograms(https://arxiv.org/abs/2408.16291)
• Keywords: segmentation
• Abstract: By generating synthetic biosignals, the quantity and variety of health data can be increased. This is especially useful when training machine learning models by enabling data augmentation and introduction of more physiologically plausible variation to the data. For these purposes, we have developed a synthetic biosignal model for two signal modalities, electrocardiography (ECG) and photoplethysmography (PPG). The model produces realistic signals that account for physiological effects such as breathing modulation and changes in heart rate due to physical stress. Arrhythmic signals can be generated with beat intervals extracted from real measurements. The model also includes a flexible approach to adding different kinds of noise and signal artifacts. The noise is generated from power spectral densities extracted from both measured noisy signals and modeled power spectra. Importantly, the model also automatically produces labels for noise, segmentation (e.g. P and T waves, QRS complex, for electrocardiograms), and artifacts. We assessed how this comprehensive model can be used in practice to improve the performance of models trained on ECG or PPG data. For example, we trained an LSTM to detect ECG R-peaks using both real ECG signals from the MIT-BIH arrythmia set and our new generator. The F1 score of the model was 0.83 using real data, in comparison to 0.98 using our generator. In addition, the model can be used for example in signal segmentation, quality detection and bench-marking detection algorithms. The model code has been released in \url{this https URL}

Title: Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Models

• Authors: Kengo Nakata, Daisuke Miyashita, Youyang Ng, Yasuto Hoshi, Jun Deguchi
• Subjects: cs.CV, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.16296
• Pdf URL: https://arxiv.org/pdf/2408.16296
• Copy Paste: [[2408.16296]] Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Models(https://arxiv.org/abs/2408.16296)
• Keywords: large language model
• Abstract: In this paper, we rethink sparse lexical representations for image retrieval. By utilizing multi-modal large language models (M-LLMs) that support visual prompting, we can extract image features and convert them into textual data, enabling us to utilize efficient sparse retrieval algorithms employed in natural language processing for image retrieval tasks. To assist the LLM in extracting image features, we apply data augmentation techniques for key expansion and analyze the impact with a metric for relevance between images and textual data. We empirically show the superior precision and recall performance of our image retrieval method compared to conventional vision-language model-based methods on the MS-COCO, PASCAL VOC, and NUS-WIDE datasets in a keyword-based image retrieval scenario, where keywords serve as search queries. We also demonstrate that the retrieval performance can be improved by iteratively incorporating keywords into search queries.

Title: Understanding Privacy Norms through Web Forms

• Authors: Hao Cui, Rahmadi Trimananda, Athina Markopoulou
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16304
• Pdf URL: https://arxiv.org/pdf/2408.16304
• Copy Paste: [[2408.16304]] Understanding Privacy Norms through Web Forms(https://arxiv.org/abs/2408.16304)
• Keywords: privacy, large language model
• Abstract: Web forms are one of the primary ways to collect personal information online, yet they are relatively under-studied. Unlike web tracking, data collection through web forms is explicit and contextualized. Users (i) are asked to input specific personal information types, and (ii) know the specific context (i.e., on which website and for what purpose). For web forms to be trusted by users, they must meet the common sense standards of appropriate data collection practices within a particular context (i.e., privacy norms). In this paper, we extract the privacy norms embedded within web forms through a measurement study. First, we build a specialized crawler to discover web forms on websites. We run it on 11,500 popular websites, and we create a dataset of 293K web forms. Second, to process data of this scale, we develop a cost-efficient way to annotate web forms with form types and personal information types, using text classifiers trained with assistance of large language models (LLMs). Third, by analyzing the annotated dataset, we reveal common patterns of data collection practices. We find that (i) these patterns are explained by functional necessities and legal obligations, thus reflecting privacy norms, and that (ii) deviations from the observed norms often signal unnecessary data collection. In addition, we analyze the privacy policies that accompany web forms. We show that, despite their wide adoption and use, there is a disconnect between privacy policy disclosures and the observed privacy norms.

Title: Semantics-Oriented Multitask Learning for DeepFake Detection: A Joint Embedding Approach

• Authors: Mian Zou, Baosheng Yu, Yibing Zhan, Siwei Lyu, Kede Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16305
• Pdf URL: https://arxiv.org/pdf/2408.16305
• Copy Paste: [[2408.16305]] Semantics-Oriented Multitask Learning for DeepFake Detection: A Joint Embedding Approach(https://arxiv.org/abs/2408.16305)
• Keywords: security
• Abstract: In recent years, the multimedia forensics and security community has seen remarkable progress in multitask learning for DeepFake (i.e., face forgery) detection. The prevailing strategy has been to frame DeepFake detection as a binary classification problem augmented by manipulation-oriented auxiliary tasks. This strategy focuses on learning features specific to face manipulations, which exhibit limited generalizability. In this paper, we delve deeper into semantics-oriented multitask learning for DeepFake detection, leveraging the relationships among face semantics via joint embedding. We first propose an automatic dataset expansion technique that broadens current face forgery datasets to support semantics-oriented DeepFake detection tasks at both the global face attribute and local face region levels. Furthermore, we resort to joint embedding of face images and their corresponding labels (depicted by textual descriptions) for prediction. This approach eliminates the need for manually setting task-agnostic and task-specific parameters typically required when predicting labels directly from images. In addition, we employ a bi-level optimization strategy to dynamically balance the fidelity loss weightings of various tasks, making the training process fully automated. Extensive experiments on six DeepFake datasets show that our method improves the generalizability of DeepFake detection and, meanwhile, renders some degree of model interpretation by providing human-understandable explanations.

Title: Bootstrap Segmentation Foundation Model under Distribution Shift via Object-Centric Learning

• Authors: Luyao Tang, Yuxuan Yuan, Chaoqi Chen, Kunze Huang, Xinghao Ding, Yue Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16310
• Pdf URL: https://arxiv.org/pdf/2408.16310
• Copy Paste: [[2408.16310]] Bootstrap Segmentation Foundation Model under Distribution Shift via Object-Centric Learning(https://arxiv.org/abs/2408.16310)
• Keywords: segmentation
• Abstract: Foundation models have made incredible strides in achieving zero-shot or few-shot generalization, leveraging prompt engineering to mimic the problem-solving approach of human intelligence. However, when it comes to some foundation models like Segment Anything, there is still a challenge in performing well on out-of-distribution data, including camouflaged and medical images. Inconsistent prompting strategies during fine-tuning and testing further compound the issue, leading to decreased performance. Drawing inspiration from how human cognition processes new environments, we introduce SlotSAM, a method that reconstructs features from the encoder in a self-supervised manner to create object-centric representations. These representations are then integrated into the foundation model, bolstering its object-level perceptual capabilities while reducing the impact of distribution-related variables. The beauty of SlotSAM lies in its simplicity and adaptability to various tasks, making it a versatile solution that significantly enhances the generalization abilities of foundation models. Through limited parameter fine-tuning in a bootstrap manner, our approach paves the way for improved generalization in novel environments. The code is available at this http URL.

Title: BEVal: A Cross-dataset Evaluation Study of BEV Segmentation Models for Autononomous Driving

• Authors: Manuel Alejandro Diaz-Zapata (CHROMA), Wenqian Liu (CHROMA, UGA), Robin Baruffa (CHROMA), Christian Laugier (CHROMA, E-MOTION, Inria)
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.16322
• Pdf URL: https://arxiv.org/pdf/2408.16322
• Copy Paste: [[2408.16322]] BEVal: A Cross-dataset Evaluation Study of BEV Segmentation Models for Autononomous Driving(https://arxiv.org/abs/2408.16322)
• Keywords: robust, segmentation
• Abstract: Current research in semantic bird's-eye view segmentation for autonomous driving focuses solely on optimizing neural network models using a single dataset, typically nuScenes. This practice leads to the development of highly specialized models that may fail when faced with different environments or sensor setups, a problem known as domain shift. In this paper, we conduct a comprehensive cross-dataset evaluation of state-of-the-art BEV segmentation models to assess their performance across different training and testing datasets and setups, as well as different semantic categories. We investigate the influence of different sensors, such as cameras and LiDAR, on the models' ability to generalize to diverse conditions and scenarios. Additionally, we conduct multi-dataset training experiments that improve models' BEV segmentation performance compared to single-dataset training. Our work addresses the gap in evaluating BEV segmentation models under cross-dataset validation. And our findings underscore the importance of enhancing model generalizability and adaptability to ensure more robust and reliable BEV segmentation approaches for autonomous driving applications.

Title: P2P-Bridge: Diffusion Bridges for 3D Point Cloud Denoising

• Authors: Mathias Vogel, Keisuke Tateno, Marc Pollefeys, Federico Tombari, Marie-Julie Rakotosaona, Francis Engelmann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16325
• Pdf URL: https://arxiv.org/pdf/2408.16325
• Copy Paste: [[2408.16325]] P2P-Bridge: Diffusion Bridges for 3D Point Cloud Denoising(https://arxiv.org/abs/2408.16325)
• Keywords: diffusion
• Abstract: In this work, we tackle the task of point cloud denoising through a novel framework that adapts Diffusion Schrödinger bridges to points clouds. Unlike previous approaches that predict point-wise displacements from point features or learned noise distributions, our method learns an optimal transport plan between paired point clouds. Experiments on object datasets like PU-Net and real-world datasets such as ScanNet++ and ARKitScenes show that P2P-Bridge achieves significant improvements over existing methods. While our approach demonstrates strong results using only point coordinates, we also show that incorporating additional features, such as color information or point-wise DINOv2 features, further enhances the performance. Code and pretrained models are available at this https URL.

Title: Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Critic

• Authors: Xin Zheng, Jie Lou, Boxi Cao, Xueru Wen, Yuqiu Ji, Hongyu Lin, Yaojie Lu, Xianpei Han, Debing Zhang, Le Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16326
• Pdf URL: https://arxiv.org/pdf/2408.16326
• Copy Paste: [[2408.16326]] Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Critic(https://arxiv.org/abs/2408.16326)
• Keywords: large language model
• Abstract: Self-critic has become an important mechanism for enhancing the reasoning performance of LLMs. However, current approaches mainly involve basic prompts without further training, which tend to be over-simplified, leading to limited accuracy.Moreover, there is a lack of in-depth investigation of the relationship between LLM's ability to criticism and its task-solving this http URL address these issues, we propose Critic-CoT, a novel framework that pushes LLMs toward System-2-like critic capability, via step-wise CoT reasoning format and distant-supervision data construction, without the need for human annotation. Experiments on GSM8K and MATH show that via filtering out invalid solutions or iterative refinement, our enhanced model boosts task-solving performance, which demonstrates the effectiveness of our method. Further, we find that training on critique and refinement alone improves the generation. We hope our work could shed light on future research on improving the reasoning and critic ability of LLMs.

Title: Self-Improving Diffusion Models with Synthetic Data

• Authors: Sina Alemohammad, Ahmed Imtiaz Humayun, Shruti Agarwal, John Collomosse, Richard Baraniuk
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16333
• Pdf URL: https://arxiv.org/pdf/2408.16333
• Copy Paste: [[2408.16333]] Self-Improving Diffusion Models with Synthetic Data(https://arxiv.org/abs/2408.16333)
• Keywords: fair, diffusion, generative
• Abstract: The artificial intelligence (AI) world is running out of real data for training increasingly large generative models, resulting in accelerating pressure to train on synthetic data. Unfortunately, training new generative models with synthetic data from current or past generation models creates an autophagous (self-consuming) loop that degrades the quality and/or diversity of the synthetic data in what has been termed model autophagy disorder (MAD) and model collapse. Current thinking around model autophagy recommends that synthetic data is to be avoided for model training lest the system deteriorate into MADness. In this paper, we take a different tack that treats synthetic data differently from real data. Self-IMproving diffusion models with Synthetic data (SIMS) is a new training concept for diffusion models that uses self-synthesized data to provide negative guidance during the generation process to steer a model's generative process away from the non-ideal synthetic data manifold and towards the real data distribution. We demonstrate that SIMS is capable of self-improvement; it establishes new records based on the Fréchet inception distance (FID) metric for CIFAR-10 and ImageNet-64 generation and achieves competitive results on FFHQ-64 and ImageNet-512. Moreover, SIMS is, to the best of our knowledge, the first prophylactic generative AI algorithm that can be iteratively trained on self-generated synthetic data without going MAD. As a bonus, SIMS can adjust a diffusion model's synthetic data distribution to match any desired in-domain target distribution to help mitigate biases and ensure fairness.

Title: GL-TSVM: A robust and smooth twin support vector machine with guardian loss function

• Authors: Mushir Akhtar, M. Tanveer, Mohd. Arshad
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16336
• Pdf URL: https://arxiv.org/pdf/2408.16336
• Copy Paste: [[2408.16336]] GL-TSVM: A robust and smooth twin support vector machine with guardian loss function(https://arxiv.org/abs/2408.16336)
• Keywords: robust
• Abstract: Twin support vector machine (TSVM), a variant of support vector machine (SVM), has garnered significant attention due to its $3/4$ times lower computational complexity compared to SVM. However, due to the utilization of the hinge loss function, TSVM is sensitive to outliers or noise. To remedy it, we introduce the guardian loss (G-loss), a novel loss function distinguished by its asymmetric, bounded, and smooth characteristics. We then fuse the proposed G-loss function into the TSVM and yield a robust and smooth classifier termed GL-TSVM. Further, to adhere to the structural risk minimization (SRM) principle and reduce overfitting, we incorporate a regularization term into the objective function of GL-TSVM. To address the optimization challenges of GL-TSVM, we devise an efficient iterative algorithm. The experimental analysis on UCI and KEEL datasets substantiates the effectiveness of the proposed GL-TSVM in comparison to the baseline models. Moreover, to showcase the efficacy of the proposed GL-TSVM in the biomedical domain, we evaluated it on the breast cancer (BreaKHis) and schizophrenia datasets. The outcomes strongly demonstrate the competitiveness of the proposed GL-TSVM against the baseline models.

Title: Toward Robust Early Detection of Alzheimer's Disease via an Integrated Multimodal Learning Approach

• Authors: Yifei Chen, Shenghao Zhu, Zhaojie Fang, Chang Liu, Binfeng Zou, Yuhe Wang, Shuo Chang, Fan Jia, Feiwei Qin, Jin Fan, Yong Peng, Changmiao Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16343
• Pdf URL: https://arxiv.org/pdf/2408.16343
• Copy Paste: [[2408.16343]] Toward Robust Early Detection of Alzheimer's Disease via an Integrated Multimodal Learning Approach(https://arxiv.org/abs/2408.16343)
• Keywords: robust, generative
• Abstract: Alzheimer's Disease (AD) is a complex neurodegenerative disorder marked by memory loss, executive dysfunction, and personality changes. Early diagnosis is challenging due to subtle symptoms and varied presentations, often leading to misdiagnosis with traditional unimodal diagnostic methods due to their limited scope. This study introduces an advanced multimodal classification model that integrates clinical, cognitive, neuroimaging, and EEG data to enhance diagnostic accuracy. The model incorporates a feature tagger with a tabular data coding architecture and utilizes the TimesBlock module to capture intricate temporal patterns in Electroencephalograms (EEG) data. By employing Cross-modal Attention Aggregation module, the model effectively fuses Magnetic Resonance Imaging (MRI) spatial information with EEG temporal data, significantly improving the distinction between AD, Mild Cognitive Impairment, and Normal Cognition. Simultaneously, we have constructed the first AD classification dataset that includes three modalities: EEG, MRI, and tabular data. Our innovative approach aims to facilitate early diagnosis and intervention, potentially slowing the progression of AD. The source code and our private ADMC dataset are available at this https URL.

Title: The Unreasonable Ineffectiveness of Nucleus Sampling on Mitigating Text Memorization

• Authors: Luka Borec, Philipp Sadler, David Schlangen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16345
• Pdf URL: https://arxiv.org/pdf/2408.16345
• Copy Paste: [[2408.16345]] The Unreasonable Ineffectiveness of Nucleus Sampling on Mitigating Text Memorization(https://arxiv.org/abs/2408.16345)
• Keywords: large language model
• Abstract: This work analyses the text memorization behavior of large language models (LLMs) when subjected to nucleus sampling. Stochastic decoding methods like nucleus sampling are typically applied to overcome issues such as monotonous and repetitive text generation, which are often observed with maximization-based decoding techniques. We hypothesize that nucleus sampling might also reduce the occurrence of memorization patterns, because it could lead to the selection of tokens outside the memorized sequence. To test this hypothesis we create a diagnostic dataset with a known distribution of duplicates that gives us some control over the likelihood of memorization of certain parts of the training data. Our analysis of two GPT-Neo models fine-tuned on this dataset interestingly shows that (i) an increase of the nucleus size reduces memorization only modestly, and (ii) even when models do not engage in "hard" memorization -- a verbatim reproduction of training samples -- they may still display "soft" memorization whereby they generate outputs that echo the training data but without a complete one-by-one resemblance.

Title: Law of Vision Representation in MLLMs

• Authors: Shijia Yang, Bohan Zhai, Quanzeng You, Jianbo Yuan, Hongxia Yang, Chenfeng Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16357
• Pdf URL: https://arxiv.org/pdf/2408.16357
• Copy Paste: [[2408.16357]] Law of Vision Representation in MLLMs(https://arxiv.org/abs/2408.16357)
• Keywords: large language model
• Abstract: We present the "Law of Vision Representation" in multimodal large language models (MLLMs). It reveals a strong correlation between the combination of cross-modal alignment, correspondence in vision representation, and MLLM performance. We quantify the two factors using the cross-modal Alignment and Correspondence score (AC score). Through extensive experiments involving thirteen different vision representation settings and evaluations across eight benchmarks, we find that the AC score is linearly correlated to model performance. By leveraging this relationship, we are able to identify and train the optimal vision representation only, which does not require finetuning the language model every time, resulting in a 99.7% reduction in computational cost.

Title: Enhancing MOTION2NX for Efficient, Scalable and Secure Image Inference using Convolutional Neural Networks

• Authors: Haritha K, Ramya Burra, Srishti Mittal, Sarthak Sharma, Abhilash Venkatesh, Anshoo Tandon
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16387
• Pdf URL: https://arxiv.org/pdf/2408.16387
• Copy Paste: [[2408.16387]] Enhancing MOTION2NX for Efficient, Scalable and Secure Image Inference using Convolutional Neural Networks(https://arxiv.org/abs/2408.16387)
• Keywords: secure, security
• Abstract: This work contributes towards the development of an efficient and scalable open-source Secure Multi-Party Computation (SMPC) protocol on machines with moderate computational resources. We use the ABY2.0 SMPC protocol implemented on the C++ based MOTION2NX framework for secure convolutional neural network (CNN) inference application with semi-honest security. Our list of contributions are as follows. Firstly, we enhance MOTION2NX by providing a tensorized version of several primitive functions including the Hadamard product, indicator function and argmax function. Our design of secure indicator function based on a novel approach that uses secure Relu function available in the baseline MOTION2NX implementation. The secure indicator function is used, in turn, as a building block for a novel implementation of secure argmax. Secondly, we also develop a novel splitting of the computations at each CNN layer into multiple configurable chunks thereby resulting in significant reduction in RAM usage. Thirdly, we adapt an existing Helper node algorithm, working in tandem with the ABY2.0 protocol, for efficient convolution computation. This algorithm not only reduces execution time but also reduces the RAM usage required to execute CNN models, but comes at a cost of an additional compute server. Moreover, the ideas presented in this paper can also be applied to secure neural network training.

Title: IBO: Inpainting-Based Occlusion to Enhance Explainable Artificial Intelligence Evaluation in Histopathology

• Authors: Pardis Afshar, Sajjad Hashembeiki, Pouya Khani, Emad Fatemizadeh, Mohammad Hossein Rohban
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16395
• Pdf URL: https://arxiv.org/pdf/2408.16395
• Copy Paste: [[2408.16395]] IBO: Inpainting-Based Occlusion to Enhance Explainable Artificial Intelligence Evaluation in Histopathology(https://arxiv.org/abs/2408.16395)
• Keywords: interpretability, diffusion
• Abstract: Histopathological image analysis is crucial for accurate cancer diagnosis and treatment planning. While deep learning models, especially convolutional neural networks, have advanced this field, their "black-box" nature raises concerns about interpretability and trustworthiness. Explainable Artificial Intelligence (XAI) techniques aim to address these concerns, but evaluating their effectiveness remains challenging. A significant issue with current occlusion-based XAI methods is that they often generate Out-of-Distribution (OoD) samples, leading to inaccurate evaluations. In this paper, we introduce Inpainting-Based Occlusion (IBO), a novel occlusion strategy that utilizes a Denoising Diffusion Probabilistic Model to inpaint occluded regions in histopathological images. By replacing cancerous areas with realistic, non-cancerous tissue, IBO minimizes OoD artifacts and preserves data integrity. We evaluate our method on the CAMELYON16 dataset through two phases: first, by assessing perceptual similarity using the Learned Perceptual Image Patch Similarity (LPIPS) metric, and second, by quantifying the impact on model predictions through Area Under the Curve (AUC) analysis. Our results demonstrate that IBO significantly improves perceptual fidelity, achieving nearly twice the improvement in LPIPS scores compared to the best existing occlusion strategy. Additionally, IBO increased the precision of XAI performance prediction from 42% to 71% compared to traditional methods. These results demonstrate IBO's potential to provide more reliable evaluations of XAI techniques, benefiting histopathology and other applications. The source code for this study is available at this https URL.

Title: Outside the Comfort Zone: Analysing LLM Capabilities in Software Vulnerability Detection

• Authors: Yuejun Guo, Constantinos Patsakis, Qiang Hu, Qiang Tang, Fran Casino
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16400
• Pdf URL: https://arxiv.org/pdf/2408.16400
• Copy Paste: [[2408.16400]] Outside the Comfort Zone: Analysing LLM Capabilities in Software Vulnerability Detection(https://arxiv.org/abs/2408.16400)
• Keywords: security, large language model
• Abstract: The significant increase in software production driven by automation and faster development lifecycles has resulted in a corresponding surge in software vulnerabilities. In parallel, the evolving landscape of software vulnerability detection, highlighting the shift from traditional methods to machine learning and large language models (LLMs), provides massive opportunities at the cost of resource-demanding computations. This paper thoroughly analyses LLMs' capabilities in detecting vulnerabilities within source code by testing models beyond their usual applications to study their potential in cybersecurity tasks. We evaluate the performance of six open-source models that are specifically trained for vulnerability detection against six general-purpose LLMs, three of which were further fine-tuned on a dataset that we compiled. Our dataset, alongside five state-of-the-art benchmark datasets, were used to create a pipeline to leverage a binary classification task, namely classifying code into vulnerable and non-vulnerable. The findings highlight significant variations in classification accuracy across benchmarks, revealing the critical influence of fine-tuning in enhancing the detection capabilities of small LLMs over their larger counterparts, yet only in the specific scenarios in which they were trained. Further experiments and analysis also underscore the issues with current benchmark datasets, particularly around mislabeling and their impact on model training and performance, which raises concerns about the current state of practice. We also discuss the road ahead in the field suggesting strategies for improved model training and dataset curation.

Title: COIN: Control-Inpainting Diffusion Prior for Human and Camera Motion Estimation

• Authors: Jiefeng Li, Ye Yuan, Davis Rempe, Haotian Zhang, Pavlo Molchanov, Cewu Lu, Jan Kautz, Umar Iqbal
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16426
• Pdf URL: https://arxiv.org/pdf/2408.16426
• Copy Paste: [[2408.16426]] COIN: Control-Inpainting Diffusion Prior for Human and Camera Motion Estimation(https://arxiv.org/abs/2408.16426)
• Keywords: diffusion
• Abstract: Estimating global human motion from moving cameras is challenging due to the entanglement of human and camera motions. To mitigate the ambiguity, existing methods leverage learned human motion priors, which however often result in oversmoothed motions with misaligned 2D projections. To tackle this problem, we propose COIN, a control-inpainting motion diffusion prior that enables fine-grained control to disentangle human and camera motions. Although pre-trained motion diffusion models encode rich motion priors, we find it non-trivial to leverage such knowledge to guide global motion estimation from RGB videos. COIN introduces a novel control-inpainting score distillation sampling method to ensure well-aligned, consistent, and high-quality motion from the diffusion prior within a joint optimization framework. Furthermore, we introduce a new human-scene relation loss to alleviate the scale ambiguity by enforcing consistency among the humans, camera, and scene. Experiments on three challenging benchmarks demonstrate the effectiveness of COIN, which outperforms the state-of-the-art methods in terms of global human motion estimation and camera motion estimation. As an illustrative example, COIN outperforms the state-of-the-art method by 33% in world joint position error (W-MPJPE) on the RICH dataset.

Title: Gradient-free variational learning with conditional mixture networks

• Authors: Conor Heins, Hao Wu, Dimitrije Markovic, Alexander Tschantz, Jeff Beck, Christopher Buckley
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.16429
• Pdf URL: https://arxiv.org/pdf/2408.16429
• Copy Paste: [[2408.16429]] Gradient-free variational learning with conditional mixture networks(https://arxiv.org/abs/2408.16429)
• Keywords: robust
• Abstract: Balancing computational efficiency with robust predictive performance is crucial in supervised learning, especially for critical applications. Standard deep learning models, while accurate and scalable, often lack probabilistic features like calibrated predictions and uncertainty quantification. Bayesian methods address these issues but can be computationally expensive as model and data complexity increase. Previous work shows that fast variational methods can reduce the compute requirements of Bayesian methods by eliminating the need for gradient computation or sampling, but are often limited to simple models. We demonstrate that conditional mixture networks (CMNs), a probabilistic variant of the mixture-of-experts (MoE) model, are suitable for fast, gradient-free inference and can solve complex classification tasks. CMNs employ linear experts and a softmax gating network. By exploiting conditional conjugacy and Pólya-Gamma augmentation, we furnish Gaussian likelihoods for the weights of both the linear experts and the gating network. This enables efficient variational updates using coordinate ascent variational inference (CAVI), avoiding traditional gradient-based optimization. We validate this approach by training two-layer CMNs on standard benchmarks from the UCI repository. Our method, CAVI-CMN, achieves competitive and often superior predictive accuracy compared to maximum likelihood estimation (MLE) with backpropagation, while maintaining competitive runtime and full posterior distributions over all model parameters. Moreover, as input size or the number of experts increases, computation time scales competitively with MLE and other gradient-based solutions like black-box variational inference (BBVI), making CAVI-CMN a promising tool for deep, fast, and gradient-free Bayesian networks.

Title: Discriminative Spatial-Semantic VOS Solution: 1st Place Solution for 6th LSVOS

• Authors: Deshui Miao, Yameng Gu, Xin Li, Zhenyu He, Yaowei Wang, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16431
• Pdf URL: https://arxiv.org/pdf/2408.16431
• Copy Paste: [[2408.16431]] Discriminative Spatial-Semantic VOS Solution: 1st Place Solution for 6th LSVOS(https://arxiv.org/abs/2408.16431)
• Keywords: segmentation
• Abstract: Video object segmentation (VOS) is a crucial task in computer vision, but current VOS methods struggle with complex scenes and prolonged object motions. To address these challenges, the MOSE dataset aims to enhance object recognition and differentiation in complex environments, while the LVOS dataset focuses on segmenting objects exhibiting long-term, intricate movements. This report introduces a discriminative spatial-temporal VOS model that utilizes discriminative object features as query representations. The semantic understanding of spatial-semantic modules enables it to recognize object parts, while salient features highlight more distinctive object characteristics. Our model, trained on extensive VOS datasets, achieved first place (\textbf{80.90\%} $\mathcal{J \& F}$) on the test set of the 6th LSVOS challenge in the VOS Track, demonstrating its effectiveness in tackling the aforementioned challenges. The code will be available at \href{this https URL}{code}.

Title: Instruction-tuned Large Language Models for Machine Translation in the Medical Domain

• Authors: Miguel Rios
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16440
• Pdf URL: https://arxiv.org/pdf/2408.16440
• Copy Paste: [[2408.16440]] Instruction-tuned Large Language Models for Machine Translation in the Medical Domain(https://arxiv.org/abs/2408.16440)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown promising results on machine translation for high resource language pairs and domains. However, in specialised domains (e.g. medical) LLMs have shown lower performance compared to standard neural machine translation models. The consistency in the machine translation of terminology is crucial for users, researchers, and translators in specialised domains. In this study, we compare the performance between baseline LLMs and instruction-tuned LLMs in the medical domain. In addition, we introduce terminology from specialised medical dictionaries into the instruction formatted datasets for fine-tuning LLMs. The instruction-tuned LLMs significantly outperform the baseline models with automatic metrics.

Title: Integrating Features for Recognizing Human Activities through Optimized Parameters in Graph Convolutional Networks and Transformer Architectures

• Authors: Mohammad Belal (1), Taimur Hassan (2), Abdelfatah Hassan (1), Nael Alsheikh (1), Noureldin Elhendawi (1), Irfan Hussain (1) ((1) Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates, (2) Abu Dhabi University, Abu Dhabi, United Arab Emirates)
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.16442
• Pdf URL: https://arxiv.org/pdf/2408.16442
• Copy Paste: [[2408.16442]] Integrating Features for Recognizing Human Activities through Optimized Parameters in Graph Convolutional Networks and Transformer Architectures(https://arxiv.org/abs/2408.16442)
• Keywords: transformer
• Abstract: Human activity recognition is a major field of study that employs computer vision, machine vision, and deep learning techniques to categorize human actions. The field of deep learning has made significant progress, with architectures that are extremely effective at capturing human dynamics. This study emphasizes the influence of feature fusion on the accuracy of activity recognition. This technique addresses the limitation of conventional models, which face difficulties in identifying activities because of their limited capacity to understand spatial and temporal features. The technique employs sensory data obtained from four publicly available datasets: HuGaDB, PKU-MMD, LARa, and TUG. The accuracy and F1-score of two deep learning models, specifically a Transformer model and a Parameter-Optimized Graph Convolutional Network (PO-GCN), were evaluated using these datasets. The feature fusion technique integrated the final layer features from both models and inputted them into a classifier. Empirical evidence demonstrates that PO-GCN outperforms standard models in activity recognition. HuGaDB demonstrated a 2.3% improvement in accuracy and a 2.2% increase in F1-score. TUG showed a 5% increase in accuracy and a 0.5% rise in F1-score. On the other hand, LARa and PKU-MMD achieved lower accuracies of 64% and 69% respectively. This indicates that the integration of features enhanced the performance of both the Transformer model and PO-GCN.

Title: Mismatched: Evaluating the Limits of Image Matching Approaches and Benchmarks

• Authors: Sierra Bonilla, Chiara Di Vece, Rema Daher, Xinwei Ju, Danail Stoyanov, Francisco Vasconcelos, Sophia Bano
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16445
• Pdf URL: https://arxiv.org/pdf/2408.16445
• Copy Paste: [[2408.16445]] Mismatched: Evaluating the Limits of Image Matching Approaches and Benchmarks(https://arxiv.org/abs/2408.16445)
• Keywords: segmentation
• Abstract: Three-dimensional (3D) reconstruction from two-dimensional images is an active research field in computer vision, with applications ranging from navigation and object tracking to segmentation and three-dimensional modeling. Traditionally, parametric techniques have been employed for this task. However, recent advancements have seen a shift towards learning-based methods. Given the rapid pace of research and the frequent introduction of new image matching methods, it is essential to evaluate them. In this paper, we present a comprehensive evaluation of various image matching methods using a structure-from-motion pipeline. We assess the performance of these methods on both in-domain and out-of-domain datasets, identifying key limitations in both the methods and benchmarks. We also investigate the impact of edge detection as a pre-processing step. Our analysis reveals that image matching for 3D reconstruction remains an open challenge, necessitating careful selection and tuning of models for specific scenarios, while also highlighting mismatches in how metrics currently represent method performance.

Title: Is text normalization relevant for classifying medieval charters?

• Authors: Florian Atzenhofer-Baumgartner, Tamás Kovács
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.16446
• Pdf URL: https://arxiv.org/pdf/2408.16446
• Copy Paste: [[2408.16446]] Is text normalization relevant for classifying medieval charters?(https://arxiv.org/abs/2408.16446)
• Keywords: transformer
• Abstract: This study examines the impact of historical text normalization on the classification of medieval charters, specifically focusing on document dating and locating. Using a data set of Middle High German charters from a digital archive, we evaluate various classifiers, including traditional and transformer-based models, with and without normalization. Our results indicate that the given normalization minimally improves locating tasks but reduces accuracy for dating, implying that original texts contain crucial features that normalization may obscure. We find that support vector machines and gradient boosting outperform other models, questioning the efficiency of transformers for this use case. Results suggest a selective approach to historical text normalization, emphasizing the significance of preserving some textual characteristics that are critical for classification tasks in document analysis.

Title: What to Preserve and What to Transfer: Faithful, Identity-Preserving Diffusion-based Hairstyle Transfer

• Authors: Chaeyeon Chung, Sunghyun Park, Jeongho Kim, Jaegul Choo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16450
• Pdf URL: https://arxiv.org/pdf/2408.16450
• Copy Paste: [[2408.16450]] What to Preserve and What to Transfer: Faithful, Identity-Preserving Diffusion-based Hairstyle Transfer(https://arxiv.org/abs/2408.16450)
• Keywords: diffusion
• Abstract: Hairstyle transfer is a challenging task in the image editing field that modifies the hairstyle of a given face image while preserving its other appearance and background features. The existing hairstyle transfer approaches heavily rely on StyleGAN, which is pre-trained on cropped and aligned face images. Hence, they struggle to generalize under challenging conditions such as extreme variations of head poses or focal lengths. To address this issue, we propose a one-stage hairstyle transfer diffusion model, HairFusion, that applies to real-world scenarios. Specifically, we carefully design a hair-agnostic representation as the input of the model, where the original hair information is thoroughly eliminated. Next, we introduce a hair align cross-attention (Align-CA) to accurately align the reference hairstyle with the face image while considering the difference in their face shape. To enhance the preservation of the face image's original features, we leverage adaptive hair blending during the inference, where the output's hair regions are estimated by the cross-attention map in Align-CA and blended with non-hair areas of the face image. Our experimental results show that our method achieves state-of-the-art performance compared to the existing methods in preserving the integrity of both the transferred hairstyle and the surrounding features. The codes are available at this https URL.

Title: Weakly Supervised Object Detection for Automatic Tooth-marked Tongue Recognition

• Authors: Yongcun Zhang, Jiajun Xu, Yina He, Shaozi Li, Zhiming Luo, Huangwei Lei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16451
• Pdf URL: https://arxiv.org/pdf/2408.16451
• Copy Paste: [[2408.16451]] Weakly Supervised Object Detection for Automatic Tooth-marked Tongue Recognition(https://arxiv.org/abs/2408.16451)
• Keywords: extraction, transformer
• Abstract: Tongue diagnosis in Traditional Chinese Medicine (TCM) is a crucial diagnostic method that can reflect an individual's health status. Traditional methods for identifying tooth-marked tongues are subjective and inconsistent because they rely on practitioner experience. We propose a novel fully automated Weakly Supervised method using Vision transformer and Multiple instance learning WSVM for tongue extraction and tooth-marked tongue recognition. Our approach first accurately detects and extracts the tongue region from clinical images, removing any irrelevant background information. Then, we implement an end-to-end weakly supervised object detection method. We utilize Vision Transformer (ViT) to process tongue images in patches and employ multiple instance loss to identify tooth-marked regions with only image-level annotations. WSVM achieves high accuracy in tooth-marked tongue classification, and visualization experiments demonstrate its effectiveness in pinpointing these regions. This automated approach enhances the objectivity and accuracy of tooth-marked tongue diagnosis. It provides significant clinical value by assisting TCM practitioners in making precise diagnoses and treatment recommendations. Code is available at this https URL.

Title: HYGENE: A Diffusion-based Hypergraph Generation Method

• Authors: Dorian Gailhard, Enzo Tartaglione, Lirida Naviner De Barros, Jhony H. Giraldo
• Subjects: cs.LG, cs.DM
• Abstract URL: https://arxiv.org/abs/2408.16457
• Pdf URL: https://arxiv.org/pdf/2408.16457
• Copy Paste: [[2408.16457]] HYGENE: A Diffusion-based Hypergraph Generation Method(https://arxiv.org/abs/2408.16457)
• Keywords: diffusion, generative
• Abstract: Hypergraphs are powerful mathematical structures that can model complex, high-order relationships in various domains, including social networks, bioinformatics, and recommender systems. However, generating realistic and diverse hypergraphs remains challenging due to their inherent complexity and lack of effective generative models. In this paper, we introduce a diffusion-based Hypergraph Generation (HYGENE) method that addresses these challenges through a progressive local expansion approach. HYGENE works on the bipartite representation of hypergraphs, starting with a single pair of connected nodes and iteratively expanding it to form the target hypergraph. At each step, nodes and hyperedges are added in a localized manner using a denoising diffusion process, which allows for the construction of the global structure before refining local details. Our experiments demonstrated the effectiveness of HYGENE, proving its ability to closely mimic a variety of properties in hypergraphs. To the best of our knowledge, this is the first attempt to employ deep learning models for hypergraph generation, and our work aims to lay the groundwork for future research in this area.

Title: An Exploratory Deep Learning Approach for Predicting Subsequent Suicidal Acts in Chinese Psychological Support Hotlines

• Authors: Changwei Song, Qing Zhao, Jianqiang Li, Yining Chen, Yongsheng Tong, Guanghui Fu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16463
• Pdf URL: https://arxiv.org/pdf/2408.16463
• Copy Paste: [[2408.16463]] An Exploratory Deep Learning Approach for Predicting Subsequent Suicidal Acts in Chinese Psychological Support Hotlines(https://arxiv.org/abs/2408.16463)
• Keywords: extraction
• Abstract: Psychological support hotlines are an effective suicide prevention measure that typically relies on professionals using suicide risk assessment scales to predict individual risk scores. However, the accuracy of scale-based predictive methods for suicide risk assessment can vary widely depending on the expertise of the operator. This limitation underscores the need for more reliable methods, prompting this research's innovative exploration of the use of artificial intelligence to improve the accuracy and efficiency of suicide risk prediction within the context of psychological support hotlines. The study included data from 1,549 subjects from 2015-2017 in China who contacted a psychological support hotline. Each participant was followed for 12 months to identify instances of suicidal behavior. We proposed a novel multi-task learning method that uses the large-scale pre-trained model Whisper for feature extraction and fits psychological scales while predicting the risk of suicide. The proposed method yields a 2.4\% points improvement in F1-score compared to the traditional manual approach based on the psychological scales. Our model demonstrated superior performance compared to the other eight popular models. To our knowledge, this study is the first to apply deep learning to long-term speech data to predict suicide risk in China, indicating grate potential for clinical applications. The source code is publicly available at: \url{this https URL}.

Title: Multi-source Domain Adaptation for Panoramic Semantic Segmentation

• Authors: Jing Jiang, Sicheng Zhao, Jiankun Zhu, Wenbo Tang, Zhaopan Xu, Jidong Yang, Pengfei Xu, Hongxun Yao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16469
• Pdf URL: https://arxiv.org/pdf/2408.16469
• Copy Paste: [[2408.16469]] Multi-source Domain Adaptation for Panoramic Semantic Segmentation(https://arxiv.org/abs/2408.16469)
• Keywords: segmentation
• Abstract: Panoramic semantic segmentation has received widespread attention recently due to its comprehensive 360\degree field of view. However, labeling such images demands greater resources compared to pinhole images. As a result, many unsupervised domain adaptation methods for panoramic semantic segmentation have emerged, utilizing real pinhole images or low-cost synthetic panoramic images. But, the segmentation model lacks understanding of the panoramic structure when only utilizing real pinhole images, and it lacks perception of real-world scenes when only adopting synthetic panoramic images. Therefore, in this paper, we propose a new task of multi-source domain adaptation for panoramic semantic segmentation, aiming to utilize both real pinhole and synthetic panoramic images in the source domains, enabling the segmentation model to perform well on unlabeled real panoramic images in the target domain. Further, we propose Deformation Transform Aligner for Panoramic Semantic Segmentation (DTA4PASS), which converts all pinhole images in the source domains into panoramic-like images, and then aligns the converted source domains with the target domain. Specifically, DTA4PASS consists of two main components: Unpaired Semantic Morphing (USM) and Distortion Gating Alignment (DGA). Firstly, in USM, the Semantic Dual-view Discriminator (SDD) assists in training the diffeomorphic deformation network, enabling the effective transformation of pinhole images without paired panoramic views. Secondly, DGA assigns pinhole-like and panoramic-like features to each image by gating, and aligns these two features through uncertainty estimation. DTA4PASS outperforms the previous state-of-the-art methods by 1.92% and 2.19% on the outdoor and indoor multi-source domain adaptation scenarios, respectively. The source code will be released.

Title: Creating a Segmented Pointcloud of Grapevines by Combining Multiple Viewpoints Through Visual Odometry

• Authors: Michael Adlerstein, Angelo Bratta, João Carlos Virgolino Soares, Giovanni Dessy, Miguel Fernandes, Matteo Gatti, Claudio Semini
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16472
• Pdf URL: https://arxiv.org/pdf/2408.16472
• Copy Paste: [[2408.16472]] Creating a Segmented Pointcloud of Grapevines by Combining Multiple Viewpoints Through Visual Odometry(https://arxiv.org/abs/2408.16472)
• Keywords: segmentation
• Abstract: Grapevine winter pruning is a labor-intensive and repetitive process that significantly influences the quality and quantity of the grape harvest and produced wine of the following season. It requires a careful and expert detection of the point to be cut. Because of its complexity, repetitive nature and time constraint, the task requires skilled labor that needs to be trained. This extended abstract presents the computer vision pipeline employed in project Vinum, using detectron2 as a segmentation network and keypoint visual odometry to merge different observation into a single pointcloud used to make informed pruning decisions.

Title: MICDrop: Masking Image and Depth Features via Complementary Dropout for Domain-Adaptive Semantic Segmentation

• Authors: Linyan Yang, Lukas Hoyer, Mark Weber, Tobias Fischer, Dengxin Dai, Laura Leal-Taixé, Marc Pollefeys, Daniel Cremers, Luc Van Gool
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16478
• Pdf URL: https://arxiv.org/pdf/2408.16478
• Copy Paste: [[2408.16478]] MICDrop: Masking Image and Depth Features via Complementary Dropout for Domain-Adaptive Semantic Segmentation(https://arxiv.org/abs/2408.16478)
• Keywords: robust, segmentation
• Abstract: Unsupervised Domain Adaptation (UDA) is the task of bridging the domain gap between a labeled source domain, e.g., synthetic data, and an unlabeled target domain. We observe that current UDA methods show inferior results on fine structures and tend to oversegment objects with ambiguous appearance. To address these shortcomings, we propose to leverage geometric information, i.e., depth predictions, as depth discontinuities often coincide with segmentation boundaries. We show that naively incorporating depth into current UDA methods does not fully exploit the potential of this complementary information. To this end, we present MICDrop, which learns a joint feature representation by masking image encoder features while inversely masking depth encoder features. With this simple yet effective complementary masking strategy, we enforce the use of both modalities when learning the joint feature representation. To aid this process, we propose a feature fusion module to improve both global as well as local information sharing while being robust to errors in the depth predictions. We show that our method can be plugged into various recent UDA methods and consistently improve results across standard UDA benchmarks, obtaining new state-of-the-art performances.

Title: Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learning

• Authors: Rochelle Choenni, Ekaterina Shutova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16482
• Pdf URL: https://arxiv.org/pdf/2408.16482
• Copy Paste: [[2408.16482]] Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learning(https://arxiv.org/abs/2408.16482)
• Keywords: large language model
• Abstract: Improving the alignment of Large Language Models (LLMs) with respect to the cultural values that they encode has become an increasingly important topic. In this work, we study whether we can exploit existing knowledge about cultural values at inference time to adjust model responses to cultural value probes. We present a simple and inexpensive method that uses a combination of in-context learning (ICL) and human survey data, and show that we can improve the alignment to cultural values across 5 models that include both English-centric and multilingual LLMs. Importantly, we show that our method could prove useful in test languages other than English and can improve alignment to the cultural values that correspond to a range of culturally diverse countries.

Title: Learning from Negative Samples in Generative Biomedical Entity Linking

• Authors: Chanhwi Kim, Hyunjae Kim, Sihyeon Park, Jiwoo Lee, Mujeen Sung, Jaewoo Kang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16493
• Pdf URL: https://arxiv.org/pdf/2408.16493
• Copy Paste: [[2408.16493]] Learning from Negative Samples in Generative Biomedical Entity Linking(https://arxiv.org/abs/2408.16493)
• Keywords: generative
• Abstract: Generative models have become widely used in biomedical entity linking (BioEL) due to their excellent performance and efficient memory usage. However, these models are usually trained only with positive samples--entities that match the input mention's identifier--and do not explicitly learn from hard negative samples, which are entities that look similar but have different meanings. To address this limitation, we introduce ANGEL (Learning from Negative Samples in Generative Biomedical Entity Linking), the first framework that trains generative BioEL models using negative samples. Specifically, a generative model is initially trained to generate positive samples from the knowledge base for given input entities. Subsequently, both correct and incorrect outputs are gathered from the model's top-k predictions. The model is then updated to prioritize the correct predictions through direct preference optimization. Our models fine-tuned with ANGEL outperform the previous best baseline models by up to an average top-1 accuracy of 1.4% on five benchmarks. When incorporating our framework into pre-training, the performance improvement further increases to 1.7%, demonstrating its effectiveness in both the pre-training and fine-tuning stages. Our code is available at this https URL.

Title: On-device AI: Quantization-aware Training of Transformers in Time-Series

• Authors: Tianheng Ling, Gregor Schiele
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16495
• Pdf URL: https://arxiv.org/pdf/2408.16495
• Copy Paste: [[2408.16495]] On-device AI: Quantization-aware Training of Transformers in Time-Series(https://arxiv.org/abs/2408.16495)
• Keywords: transformer
• Abstract: Artificial Intelligence (AI) models for time-series in pervasive computing keep getting larger and more complicated. The Transformer model is by far the most compelling of these AI models. However, it is difficult to obtain the desired performance when deploying such a massive model on a sensor device with limited resources. My research focuses on optimizing the Transformer model for time-series forecasting tasks. The optimized model will be deployed as hardware accelerators on embedded Field Programmable Gate Arrays (FPGAs). I will investigate the impact of applying Quantization-aware Training to the Transformer model to reduce its size and runtime memory footprint while maximizing the advantages of FPGAs.

Title: LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs?

• Authors: Jan Cegin, Jakub Simko, Peter Brusilovsky
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16502
• Pdf URL: https://arxiv.org/pdf/2408.16502
• Copy Paste: [[2408.16502]] LLMs vs Established Text Augmentation Techniques for Classification: When do the Benefits Outweight the Costs?(https://arxiv.org/abs/2408.16502)
• Keywords: generative, large language model
• Abstract: The generative large language models (LLMs) are increasingly being used for data augmentation tasks, where text samples are LLM-paraphrased and then used for classifier fine-tuning. However, a research that would confirm a clear cost-benefit advantage of LLMs over more established augmentation methods is largely missing. To study if (and when) is the LLM-based augmentation advantageous, we compared the effects of recent LLM augmentation methods with established ones on 6 datasets, 3 classifiers and 2 fine-tuning methods. We also varied the number of seeds and collected samples to better explore the downstream model accuracy space. Finally, we performed a cost-benefit analysis and show that LLM-based methods are worthy of deployment only when very small number of seeds is used. Moreover, in many cases, established methods lead to similar or better model accuracies.

Title: A Simple and Generalist Approach for Panoptic Segmentation

• Authors: Nedyalko Prisadnikov, Wouter Van Gansbeke, Danda Pani Paudel, Luc Van Gool
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16504
• Pdf URL: https://arxiv.org/pdf/2408.16504
• Copy Paste: [[2408.16504]] A Simple and Generalist Approach for Panoptic Segmentation(https://arxiv.org/abs/2408.16504)
• Keywords: diffusion, segmentation
• Abstract: Generalist vision models aim for one and the same architecture for a variety of vision tasks. While such shared architecture may seem attractive, generalist models tend to be outperformed by their bespoken counterparts, especially in the case of panoptic segmentation. We address this problem by introducing two key contributions, without compromising the desirable properties of generalist models. These contributions are: (i) a positional-embedding (PE) based loss for improved centroid regressions; (ii) Edge Distance Sampling (EDS) for the better separation of instance boundaries. The PE-based loss facilitates a better per-pixel regression of the associated instance's centroid, whereas EDS contributes by carefully handling the void regions (caused by missing labels) and smaller instances. These two simple yet effective modifications significantly improve established baselines, while achieving state-of-the-art results among all generalist solutions. More specifically, our method achieves a panoptic quality(PQ) of 52.5 on the COCO dataset, which is an improvement of 10 points over the best model with similar approach (Painter), and is superior by 2 to the best performing diffusion-based method Pix2Seq-$\mathcal{D}$. Furthermore, we provide insights into and an in-depth analysis of our contributions through exhaustive experiments. Our source code and model weights will be made publicly available.

Title: CanCal: Towards Real-time and Lightweight Ransomware Detection and Response in Industrial Environments

• Authors: Shenao Wang, Feng Dong, Hangfeng Yang, Jingheng Xu, Haoyu Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16515
• Pdf URL: https://arxiv.org/pdf/2408.16515
• Copy Paste: [[2408.16515]] CanCal: Towards Real-time and Lightweight Ransomware Detection and Response in Industrial Environments(https://arxiv.org/abs/2408.16515)
• Keywords: security, defense, attack
• Abstract: Ransomware attacks have emerged as one of the most significant cybersecurity threats. Despite numerous proposed detection and defense methods, existing approaches face two fundamental limitations in large-scale industrial applications: intolerable system overheads and notorious alert fatigue. To address these challenges, we propose CanCal, a real-time and lightweight ransomware detection system. Specifically, CanCal selectively filters suspicious processes by the monitoring layers and then performs in-depth behavioral analysis to isolate ransomware activities from benign operations, minimizing alert fatigue while ensuring lightweight computational and storage overhead. The experimental results on a large-scale industrial environment~(1,761 ransomware, ~3 million events, continuous test over 5 months) indicate that CanCal is as effective as state-of-the-art techniques while enabling rapid inference within 30ms and real-time response within a maximum of 3 seconds. CanCal dramatically reduces average CPU utilization by 91.04% (from 6.7% to 0.6%) and peak CPU utilization by 76.69% (from 26.6% to 6.2%), while avoiding 76.50% (from 3,192 to 750) of the inspection efforts from security analysts. By the time of this writing, CanCal has been integrated into a commercial product and successfully deployed on 3.32 million endpoints for over a year. From March 2023 to April 2024, CanCal successfully detected and thwarted 61 ransomware attacks, demonstrating the effectiveness of CanCal in combating sophisticated ransomware threats in real-world scenarios.

Title: CNIMA: A Universal Evaluation Framework and Automated Approach for Assessing Second Language Dialogues

• Authors: Rena Gao, Jingxuan Wu, Carsten Roever, Xuetong Wu, Jing Wu, Long Lv, Jey Han Lau
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16518
• Pdf URL: https://arxiv.org/pdf/2408.16518
• Copy Paste: [[2408.16518]] CNIMA: A Universal Evaluation Framework and Automated Approach for Assessing Second Language Dialogues(https://arxiv.org/abs/2408.16518)
• Keywords: robust, large language model
• Abstract: We develop CNIMA (Chinese Non-Native Interactivity Measurement and Automation), a Chinese-as-a-second-language labelled dataset with 10K dialogues. We annotate CNIMA using an evaluation framework -- originally introduced for English-as-a-second-language dialogues -- that assesses micro-level features (e.g.\ backchannels) and macro-level interactivity labels (e.g.\ topic management) and test the framework's transferability from English to Chinese. We found the framework robust across languages and revealed universal and language-specific relationships between micro-level and macro-level features. Next, we propose an approach to automate the evaluation and find strong performance, creating a new tool for automated second language assessment. Our system can be adapted to other languages easily as it uses large language models and as such does not require large-scale annotated training data.

Title: Towards Modality-agnostic Label-efficient Segmentation with Entropy-Regularized Distribution Alignment

• Authors: Liyao Tang, Zhe Chen, Shanshan Zhao, Chaoyue Wang, Dacheng Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16520
• Pdf URL: https://arxiv.org/pdf/2408.16520
• Copy Paste: [[2408.16520]] Towards Modality-agnostic Label-efficient Segmentation with Entropy-Regularized Distribution Alignment(https://arxiv.org/abs/2408.16520)
• Keywords: segmentation
• Abstract: Label-efficient segmentation aims to perform effective segmentation on input data using only sparse and limited ground-truth labels for training. This topic is widely studied in 3D point cloud segmentation due to the difficulty of annotating point clouds densely, while it is also essential for cost-effective segmentation on 2D images. Until recently, pseudo-labels have been widely employed to facilitate training with limited ground-truth labels, and promising progress has been witnessed in both the 2D and 3D segmentation. However, existing pseudo-labeling approaches could suffer heavily from the noises and variations in unlabelled data, which would result in significant discrepancies between generated pseudo-labels and current model predictions during training. We analyze that this can further confuse and affect the model learning process, which shows to be a shared problem in label-efficient learning across both 2D and 3D modalities. To address this issue, we propose a novel learning strategy to regularize the pseudo-labels generated for training, thus effectively narrowing the gaps between pseudo-labels and model predictions. More specifically, our method introduces an Entropy Regularization loss and a Distribution Alignment loss for label-efficient learning, resulting in an ERDA learning strategy. Interestingly, by using KL distance to formulate the distribution alignment loss, ERDA reduces to a deceptively simple cross-entropy-based loss which optimizes both the pseudo-label generation module and the segmentation model simultaneously. In addition, we innovate in the pseudo-label generation to make our ERDA consistently effective across both 2D and 3D data modalities for segmentation. Enjoying simplicity and more modality-agnostic pseudo-label generation, our method has shown outstanding performance in fully utilizing all unlabeled data points for training across ...

Title: Multitask learning for improved scour detection: A dynamic wave tank study

• Authors: Simon M. Brealy, Aidan J. Hughes, Tina A. Dardeno, Lawrence A. Bull, Robin S. Mills, Nikolaos Dervilis, Keith Worden
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16527
• Pdf URL: https://arxiv.org/pdf/2408.16527
• Copy Paste: [[2408.16527]] Multitask learning for improved scour detection: A dynamic wave tank study(https://arxiv.org/abs/2408.16527)
• Keywords: robust
• Abstract: Population-based structural health monitoring (PBSHM), aims to share information between members of a population. An offshore wind (OW) farm could be considered as a population of nominally-identical wind-turbine structures. However, benign variations exist among members, such as geometry, sea-bed conditions and temperature differences. These factors could influence structural properties and therefore the dynamic response, making it more difficult to detect structural problems via traditional SHM techniques. This paper explores the use of a Bayesian hierarchical model as a means of multitask learning, to infer foundation stiffness distribution parameters at both population and local levels. To do this, observations of natural frequency from populations of structures were first generated from both numerical and experimental models. These observations were then used in a partially-pooled Bayesian hierarchical model in tandem with surrogate FE models of the structures to infer foundation stiffness parameters. Finally, it is demonstrated how the learned parameters may be used as a basis to perform more robust anomaly detection (as compared to a no-pooling approach) e.g. as a result of scour.

Title: S3C2 Summit 2023-11: Industry Secure Supply Chain Summit

• Authors: Nusrat Zahan, Yasemin Acar, Michel Cukier, William Enck, Christian Kästner, Alexandros Kapravelos, Dominik Wermke, Laurie Williams
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16529
• Pdf URL: https://arxiv.org/pdf/2408.16529
• Copy Paste: [[2408.16529]] S3C2 Summit 2023-11: Industry Secure Supply Chain Summit(https://arxiv.org/abs/2408.16529)
• Keywords: secure, attack
• Abstract: Cyber attacks leveraging or targeting the software supply chain, such as the SolarWinds and the Log4j incidents, affected thousands of businesses and their customers, drawing attention from both industry and government stakeholders. To foster open dialogue, facilitate mutual sharing, and discuss shared challenges encountered by stakeholders in securing their software supply chain, researchers from the NSF-supported Secure Software Supply Chain Center (S3C2) organize Secure Supply Chain Summits with stakeholders. This paper summarizes the Industry Secure Supply Chain Summit held on November 16, 2023, which consisted of \panels{} panel discussions with a diverse set of \participants{} practitioners from the industry. The individual panels were framed with open-ended questions and included the topics of Software Bills of Materials (SBOMs), vulnerable dependencies, malicious commits, build and deploy infrastructure, reducing entire classes of vulnerabilities at scale, and supporting a company culture conductive to securing the software supply chain. The goal of this summit was to enable open discussions, mutual sharing, and shedding light on common challenges that industry practitioners with practical experience face when securing their software supply chain.

Title: A Comprehensive Review of 3D Object Detection in Autonomous Driving: Technological Advances and Future Directions

• Authors: Yu Wang, Shaohua Wang, Yicheng Li, Mingchun Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16530
• Pdf URL: https://arxiv.org/pdf/2408.16530
• Copy Paste: [[2408.16530]] A Comprehensive Review of 3D Object Detection in Autonomous Driving: Technological Advances and Future Directions(https://arxiv.org/abs/2408.16530)
• Keywords: robust
• Abstract: In recent years, 3D object perception has become a crucial component in the development of autonomous driving systems, providing essential environmental awareness. However, as perception tasks in autonomous driving evolve, their variants have increased, leading to diverse insights from industry and academia. Currently, there is a lack of comprehensive surveys that collect and summarize these perception tasks and their developments from a broader perspective. This review extensively summarizes traditional 3D object detection methods, focusing on camera-based, LiDAR-based, and fusion detection techniques. We provide a comprehensive analysis of the strengths and limitations of each approach, highlighting advancements in accuracy and robustness. Furthermore, we discuss future directions, including methods to improve accuracy such as temporal perception, occupancy grids, and end-to-end learning frameworks. We also explore cooperative perception methods that extend the perception range through collaborative communication. By providing a holistic view of the current state and future developments in 3D object perception, we aim to offer a more comprehensive understanding of perception tasks for autonomous driving. Additionally, we have established an active repository to provide continuous updates on the latest advancements in this field, accessible at: this https URL.

Title: SFR-GNN: Simple and Fast Robust GNNs against Structural Attacks

• Authors: Xing Ai, Guanyu Zhu, Yulin Zhu, Yu Zheng, Gaolei Li, Jianhua Li, Kai Zhou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16537
• Pdf URL: https://arxiv.org/pdf/2408.16537
• Copy Paste: [[2408.16537]] SFR-GNN: Simple and Fast Robust GNNs against Structural Attacks(https://arxiv.org/abs/2408.16537)
• Keywords: defense, attack, robust
• Abstract: Graph Neural Networks (GNNs) have demonstrated commendable performance for graph-structured data. Yet, GNNs are often vulnerable to adversarial structural attacks as embedding generation relies on graph topology. Existing efforts are dedicated to purifying the maliciously modified structure or applying adaptive aggregation, thereby enhancing the robustness against adversarial structural attacks. It is inevitable for a defender to consume heavy computational costs due to lacking prior knowledge about modified structures. To this end, we propose an efficient defense method, called Simple and Fast Robust Graph Neural Network (SFR-GNN), supported by mutual information theory. The SFR-GNN first pre-trains a GNN model using node attributes and then fine-tunes it over the modified graph in the manner of contrastive learning, which is free of purifying modified structures and adaptive aggregation, thus achieving great efficiency gains. Consequently, SFR-GNN exhibits a 24%--162% speedup compared to advanced robust models, demonstrating superior robustness for node classification tasks.

Title: GRPose: Learning Graph Relations for Human Image Generation with Pose Priors

• Authors: Xiangchen Yin, Donglin Di, Lei Fan, Hao Li, Chen Wei, Xiaofei Gou, Yang Song, Xiao Sun, Xun Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16540
• Pdf URL: https://arxiv.org/pdf/2408.16540
• Copy Paste: [[2408.16540]] GRPose: Learning Graph Relations for Human Image Generation with Pose Priors(https://arxiv.org/abs/2408.16540)
• Keywords: diffusion
• Abstract: Recent methods using diffusion models have made significant progress in human image generation with various additional controls such as pose priors. However, existing approaches still struggle to generate high-quality images with consistent pose alignment, resulting in unsatisfactory outputs. In this paper, we propose a framework delving into the graph relations of pose priors to provide control information for human image generation. The main idea is to establish a graph topological structure between the pose priors and latent representation of diffusion models to capture the intrinsic associations between different pose parts. A Progressive Graph Integrator (PGI) is designed to learn the spatial relationships of the pose priors with the graph structure, adopting a hierarchical strategy within an Adapter to gradually propagate information across different pose parts. A pose perception loss is further introduced based on a pretrained pose estimation network to minimize the pose differences. Extensive qualitative and quantitative experiments conducted on the Human-Art and LAION-Human datasets demonstrate that our model achieves superior performance, with a 9.98% increase in pose average precision compared to the latest benchmark model. The code is released on *******.

Title: Android Malware Detection Based on RGB Images and Multi-feature Fusion

• Authors: Zhiqiang Wang, Qiulong Yu, Sicheng Yuan
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16555
• Pdf URL: https://arxiv.org/pdf/2408.16555
• Copy Paste: [[2408.16555]] Android Malware Detection Based on RGB Images and Multi-feature Fusion(https://arxiv.org/abs/2408.16555)
• Keywords: security, extraction
• Abstract: With the widespread adoption of smartphones, Android malware has become a significant challenge in the field of mobile device security. Current Android malware detection methods often rely on feature engineering to construct dynamic or static features, which are then used for learning. However, static feature-based methods struggle to counter code obfuscation, packing, and signing techniques, while dynamic feature-based methods involve time-consuming feature extraction. Image-based methods for Android malware detection offer better resilience against malware variants and polymorphic malware. This paper proposes an end-to-end Android malware detection technique based on RGB images and multi-feature fusion. The approach involves extracting Dalvik Executable (DEX) files, AndroidManifest.xml files, and API calls from APK files, converting them into grayscale images, and enhancing their texture features using Canny edge detection, histogram equalization, and adaptive thresholding techniques. These grayscale images are then combined into an RGB image containing multi-feature fusion information, which is analyzed using mainstream image classification models for Android malware detection. Extensive experiments demonstrate that the proposed method effectively captures Android malware characteristics, achieving an accuracy of up to 97.25%, outperforming existing detection methods that rely solely on DEX files as classification features. Additionally, ablation experiments confirm the effectiveness of using the three key files for feature representation in the proposed approach.

Title: MST-KD: Multiple Specialized Teachers Knowledge Distillation for Fair Face Recognition

• Authors: Eduarda Caldeira, Jaime S. Cardoso, Ana F. Sequeira, Pedro C. Neto
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16563
• Pdf URL: https://arxiv.org/pdf/2408.16563
• Copy Paste: [[2408.16563]] MST-KD: Multiple Specialized Teachers Knowledge Distillation for Fair Face Recognition(https://arxiv.org/abs/2408.16563)
• Keywords: robust, fair
• Abstract: As in school, one teacher to cover all subjects is insufficient to distill equally robust information to a student. Hence, each subject is taught by a highly specialised teacher. Following a similar philosophy, we propose a multiple specialized teacher framework to distill knowledge to a student network. In our approach, directed at face recognition use cases, we train four teachers on one specific ethnicity, leading to four highly specialized and biased teachers. Our strategy learns a project of these four teachers into a common space and distill that information to a student network. Our results highlighted increased performance and reduced bias for all our experiments. In addition, we further show that having biased/specialized teachers is crucial by showing that our approach achieves better results than when knowledge is distilled from four teachers trained on balanced datasets. Our approach represents a step forward to the understanding of the importance of ethnicity-specific features.

Title: FastForensics: Efficient Two-Stream Design for Real-Time Image Manipulation Detection

• Authors: Yangxiang Zhang, Yuezun Li, Ao Luo, Jiaran Zhou, Junyu Dong
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16582
• Pdf URL: https://arxiv.org/pdf/2408.16582
• Copy Paste: [[2408.16582]] FastForensics: Efficient Two-Stream Design for Real-Time Image Manipulation Detection(https://arxiv.org/abs/2408.16582)
• Keywords: transformer
• Abstract: With the rise in popularity of portable devices, the spread of falsified media on social platforms has become rampant. This necessitates the timely identification of authentic content. However, most advanced detection methods are computationally heavy, hindering their real-time application. In this paper, we describe an efficient two-stream architecture for real-time image manipulation detection. Our method consists of two-stream branches targeting the cognitive and inspective perspectives. In the cognitive branch, we propose efficient wavelet-guided Transformer blocks to capture the global manipulation traces related to frequency. This block contains an interactive wavelet-guided self-attention module that integrates wavelet transformation with efficient attention design, interacting with the knowledge from the inspective branch. The inspective branch consists of simple convolutions that capture fine-grained traces and interact bidirectionally with Transformer blocks to provide mutual support. Our method is lightweight ($\sim$ 8M) but achieves competitive performance compared to many other counterparts, demonstrating its efficacy in image manipulation detection and its potential for portable integration.

Title: Enhancing Dialogue Generation in Werewolf Game Through Situation Analysis and Persuasion Strategies

• Authors: Zhiyang Qi, Michimasa Inaba
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16586
• Pdf URL: https://arxiv.org/pdf/2408.16586
• Copy Paste: [[2408.16586]] Enhancing Dialogue Generation in Werewolf Game Through Situation Analysis and Persuasion Strategies(https://arxiv.org/abs/2408.16586)
• Keywords: large language model
• Abstract: Recent advancements in natural language processing, particularly with large language models (LLMs) like GPT-4, have significantly enhanced dialogue systems, enabling them to generate more natural and fluent conversations. Despite these improvements, challenges persist, such as managing continuous dialogues, memory retention, and minimizing hallucinations. The AIWolfDial2024 addresses these challenges by employing the Werewolf Game, an incomplete information game, to test the capabilities of LLMs in complex interactive environments. This paper introduces a LLM-based Werewolf Game AI, where each role is supported by situation analysis to aid response generation. Additionally, for the werewolf role, various persuasion strategies, including logical appeal, credibility appeal, and emotional appeal, are employed to effectively persuade other players to align with its actions.

Title: CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions

• Authors: Laurin Wagner, Bernhard Thallinger, Mario Zusag
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16589
• Pdf URL: https://arxiv.org/pdf/2408.16589
• Copy Paste: [[2408.16589]] CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions(https://arxiv.org/abs/2408.16589)
• Keywords: robust, segmentation
• Abstract: We demonstrate that carefully adjusting the tokenizer of the Whisper speech recognition model significantly improves the precision of word-level timestamps when applying dynamic time warping to the decoder's cross-attention scores. We fine-tune the model to produce more verbatim speech transcriptions and employ several techniques to increase robustness against multiple speakers and background noise. These adjustments achieve state-of-the-art performance on benchmarks for verbatim speech transcription, word segmentation, and the timed detection of filler events, and can further mitigate transcription hallucinations. The code is available open this https URL.

Title: Data Quality Monitoring through Transfer Learning on Anomaly Detection for the Hadron Calorimeters

• Authors: Mulugeta Weldezgina Asres, Christian Walter Omlin, Long Wang, Pavel Parygin, David Yu, Jay Dittmann, The CMS-HCAL Collaboration
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16612
• Pdf URL: https://arxiv.org/pdf/2408.16612
• Copy Paste: [[2408.16612]] Data Quality Monitoring through Transfer Learning on Anomaly Detection for the Hadron Calorimeters(https://arxiv.org/abs/2408.16612)
• Keywords: robust
• Abstract: The proliferation of sensors brings an immense volume of spatio-temporal (ST) data in many domains for various purposes, including monitoring, diagnostics, and prognostics applications. Data curation is a time-consuming process for a large volume of data, making it challenging and expensive to deploy data analytics platforms in new environments. Transfer learning (TL) mechanisms promise to mitigate data sparsity and model complexity by utilizing pre-trained models for a new task. Despite the triumph of TL in fields like computer vision and natural language processing, efforts on complex ST models for anomaly detection (AD) applications are limited. In this study, we present the potential of TL within the context of AD for the Hadron Calorimeter of the Compact Muon Solenoid experiment at CERN. We have transferred the ST AD models trained on data collected from one part of a calorimeter to another. We have investigated different configurations of TL on semi-supervised autoencoders of the ST AD models -- transferring convolutional, graph, and recurrent neural networks of both the encoder and decoder networks. The experiment results demonstrate that TL effectively enhances the model learning accuracy on a target subdetector. The TL achieves promising data reconstruction and AD performance while substantially reducing the trainable parameters of the AD models. It also improves robustness against anomaly contamination in the training data sets of the semi-supervised AD models.

Title: Towards Infusing Auxiliary Knowledge for Distracted Driver Detection

• Authors: Ishwar B Balappanawar, Ashmit Chamoli, Ruwan Wickramarachchi, Aditya Mishra, Ponnurangam Kumaraguru, Amit P. Sheth
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16621
• Pdf URL: https://arxiv.org/pdf/2408.16621
• Copy Paste: [[2408.16621]] Towards Infusing Auxiliary Knowledge for Distracted Driver Detection(https://arxiv.org/abs/2408.16621)
• Keywords: robust
• Abstract: Distracted driving is a leading cause of road accidents globally. Identification of distracted driving involves reliably detecting and classifying various forms of driver distraction (e.g., texting, eating, or using in-car devices) from in-vehicle camera feeds to enhance road safety. This task is challenging due to the need for robust models that can generalize to a diverse set of driver behaviors without requiring extensive annotated datasets. In this paper, we propose KiD3, a novel method for distracted driver detection (DDD) by infusing auxiliary knowledge about semantic relations between entities in a scene and the structural configuration of the driver's pose. Specifically, we construct a unified framework that integrates the scene graphs, and driver pose information with the visual cues in video frames to create a holistic representation of the driver's actions.Our results indicate that KiD3 achieves a 13.64% accuracy improvement over the vision-only baseline by incorporating such auxiliary knowledge with visual information.

Title: 3D Pose-Based Temporal Action Segmentation for Figure Skating: A Fine-Grained and Jump Procedure-Aware Annotation Approach

• Authors: Ryota Tanaka, Tomohiro Suzuki, Keisuke Fujii
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16638
• Pdf URL: https://arxiv.org/pdf/2408.16638
• Copy Paste: [[2408.16638]] 3D Pose-Based Temporal Action Segmentation for Figure Skating: A Fine-Grained and Jump Procedure-Aware Annotation Approach(https://arxiv.org/abs/2408.16638)
• Keywords: segmentation
• Abstract: Understanding human actions from videos is essential in many domains, including sports. In figure skating, technical judgments are performed by watching skaters' 3D movements, and its part of the judging procedure can be regarded as a Temporal Action Segmentation (TAS) task. TAS tasks in figure skating that automatically assign temporal semantics to video are actively researched. However, there is a lack of datasets and effective methods for TAS tasks requiring 3D pose data. In this study, we first created the FS-Jump3D dataset of complex and dynamic figure skating jumps using optical markerless motion capture. We also propose a new fine-grained figure skating jump TAS dataset annotation method with which TAS models can learn jump procedures. In the experimental results, we validated the usefulness of 3D pose features as input and the fine-grained dataset for the TAS model in figure skating. FS-Jump3D Dataset is available at this https URL.

Title: SODAWideNet++: Combining Attention and Convolutions for Salient Object Detection

• Authors: Rohit Venkata Sai Dulam, Chandra Kambhamettu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16645
• Pdf URL: https://arxiv.org/pdf/2408.16645
• Copy Paste: [[2408.16645]] SODAWideNet++: Combining Attention and Convolutions for Salient Object Detection(https://arxiv.org/abs/2408.16645)
• Keywords: extraction, transformer, segmentation
• Abstract: Salient Object Detection (SOD) has traditionally relied on feature refinement modules that utilize the features of an ImageNet pre-trained backbone. However, this approach limits the possibility of pre-training the entire network because of the distinct nature of SOD and image classification. Additionally, the architecture of these backbones originally built for Image classification is sub-optimal for a dense prediction task like SOD. To address these issues, we propose a novel encoder-decoder-style neural network called SODAWideNet++ that is designed explicitly for SOD. Inspired by the vision transformers ability to attain a global receptive field from the initial stages, we introduce the Attention Guided Long Range Feature Extraction (AGLRFE) module, which combines large dilated convolutions and self-attention. Specifically, we use attention features to guide long-range information extracted by multiple dilated convolutions, thus taking advantage of the inductive biases of a convolution operation and the input dependency brought by self-attention. In contrast to the current paradigm of ImageNet pre-training, we modify 118K annotated images from the COCO semantic segmentation dataset by binarizing the annotations to pre-train the proposed model end-to-end. Further, we supervise the background predictions along with the foreground to push our model to generate accurate saliency predictions. SODAWideNet++ performs competitively on five different datasets while only containing 35% of the trainable parameters compared to the state-of-the-art models. The code and pre-computed saliency maps are provided at this https URL.

Title: DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving

• Authors: Yongjie Fu, Anmol Jain, Xuan Di, Xu Chen, Zhaobin Mo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16647
• Pdf URL: https://arxiv.org/pdf/2408.16647
• Copy Paste: [[2408.16647]] DriveGenVLM: Real-world Video Generation for Vision Language Model based Autonomous Driving(https://arxiv.org/abs/2408.16647)
• Keywords: diffusion
• Abstract: The advancement of autonomous driving technologies necessitates increasingly sophisticated methods for understanding and predicting real-world scenarios. Vision language models (VLMs) are emerging as revolutionary tools with significant potential to influence autonomous driving. In this paper, we propose the DriveGenVLM framework to generate driving videos and use VLMs to understand them. To achieve this, we employ a video generation framework grounded in denoising diffusion probabilistic models (DDPM) aimed at predicting real-world video sequences. We then explore the adequacy of our generated videos for use in VLMs by employing a pre-trained model known as Efficient In-context Learning on Egocentric Videos (EILEV). The diffusion model is trained with the Waymo open dataset and evaluated using the Fréchet Video Distance (FVD) score to ensure the quality and realism of the generated videos. Corresponding narrations are provided by EILEV for these generated videos, which may be beneficial in the autonomous driving domain. These narrations can enhance traffic scene understanding, aid in navigation, and improve planning capabilities. The integration of video generation with VLMs in the DriveGenVLM framework represents a significant step forward in leveraging advanced AI models to address complex challenges in autonomous driving.

Title: Eigen-Cluster VIS: Improving Weakly-supervised Video Instance Segmentation by Leveraging Spatio-temporal Consistency

• Authors: Farnoosh Arefi, Amir M. Mansourian, Shohreh Kasaei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16661
• Pdf URL: https://arxiv.org/pdf/2408.16661
• Copy Paste: [[2408.16661]] Eigen-Cluster VIS: Improving Weakly-supervised Video Instance Segmentation by Leveraging Spatio-temporal Consistency(https://arxiv.org/abs/2408.16661)
• Keywords: robust, transformer, segmentation
• Abstract: The performance of Video Instance Segmentation (VIS) methods has improved significantly with the advent of transformer networks. However, these networks often face challenges in training due to the high annotation cost. To address this, unsupervised and weakly-supervised methods have been developed to reduce the dependency on annotations. This work introduces a novel weakly-supervised method called Eigen-cluster VIS that, without requiring any mask annotations, achieves competitive accuracy compared to other VIS approaches. This method is based on two key innovations: a Temporal Eigenvalue Loss (TEL) and a clip-level Quality Cluster Coefficient (QCC). The TEL ensures temporal coherence by leveraging the eigenvalues of the Laplacian matrix derived from graph adjacency matrices. By minimizing the mean absolute error (MAE) between the eigenvalues of adjacent frames, this loss function promotes smooth transitions and stable segmentation boundaries over time, reducing temporal discontinuities and improving overall segmentation quality. The QCC employs the K-means method to ensure the quality of spatio-temporal clusters without relying on ground truth masks. Using the Davies-Bouldin score, the QCC provides an unsupervised measure of feature discrimination, allowing the model to self-evaluate and adapt to varying object distributions, enhancing robustness during the testing phase. These enhancements are computationally efficient and straightforward, offering significant performance gains without additional annotated data. The proposed Eigen-Cluster VIS method is evaluated on the YouTube-VIS 2019/2021 and OVIS datasets, demonstrating that it effectively narrows the performance gap between the fully-supervised and weakly-supervised VIS approaches. The code is available on: this https URL

Title: Entropic Distribution Matching in Supervised Fine-tuning of LLMs: Less Overfitting and Better Diversity

• Authors: Ziniu Li, Congliang Chen, Tian Xu, Zeyu Qin, Jiancong Xiao, Ruoyu Sun, Zhi-Quan Luo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16673
• Pdf URL: https://arxiv.org/pdf/2408.16673
• Copy Paste: [[2408.16673]] Entropic Distribution Matching in Supervised Fine-tuning of LLMs: Less Overfitting and Better Diversity(https://arxiv.org/abs/2408.16673)
• Keywords: large language model
• Abstract: Large language models rely on Supervised Fine-Tuning (SFT) to specialize in downstream tasks. Cross Entropy (CE) loss is the de facto choice in SFT, but it often leads to overfitting and limited output diversity due to its aggressive updates to the data distribution. This paper aim to address these issues by introducing the maximum entropy principle, which favors models with flatter distributions that still effectively capture the data. Specifically, we develop a new distribution matching method called GEM, which solves reverse Kullback-Leibler divergence minimization with an entropy regularizer. For the SFT of Llama-3-8B models, GEM outperforms CE in several aspects. First, when applied to the UltraFeedback dataset to develop general instruction-following abilities, GEM exhibits reduced overfitting, evidenced by lower perplexity and better performance on the IFEval benchmark. Furthermore, GEM enhances output diversity, leading to performance gains of up to 7 points on math reasoning and code generation tasks using best-of-n sampling, even without domain-specific data. Second, when fine-tuning with domain-specific datasets for math reasoning and code generation, GEM also shows less overfitting and improvements of up to 10 points compared with CE.

Title: PartFormer: Awakening Latent Diverse Representation from Vision Transformer for Object Re-Identification

• Authors: Lei Tan, Pingyang Dai, Jie Chen, Liujuan Cao, Yongjian Wu, Rongrong Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16684
• Pdf URL: https://arxiv.org/pdf/2408.16684
• Copy Paste: [[2408.16684]] PartFormer: Awakening Latent Diverse Representation from Vision Transformer for Object Re-Identification(https://arxiv.org/abs/2408.16684)
• Keywords: robust, transformer
• Abstract: Extracting robust feature representation is critical for object re-identification to accurately identify objects across non-overlapping cameras. Although having a strong representation ability, the Vision Transformer (ViT) tends to overfit on most distinct regions of training data, limiting its generalizability and attention to holistic object features. Meanwhile, due to the structural difference between CNN and ViT, fine-grained strategies that effectively address this issue in CNN do not continue to be successful in ViT. To address this issue, by observing the latent diverse representation hidden behind the multi-head attention, we present PartFormer, an innovative adaptation of ViT designed to overcome the granularity limitations in object Re-ID tasks. The PartFormer integrates a Head Disentangling Block (HDB) that awakens the diverse representation of multi-head self-attention without the typical loss of feature richness induced by concatenation and FFN layers post-attention. To avoid the homogenization of attention heads and promote robust part-based feature learning, two head diversity constraints are imposed: attention diversity constraint and correlation diversity constraint. These constraints enable the model to exploit diverse and discriminative feature representations from different attention heads. Comprehensive experiments on various object Re-ID benchmarks demonstrate the superiority of the PartFormer. Specifically, our framework significantly outperforms state-of-the-art by 2.4\% mAP scores on the most challenging MSMT17 dataset.

Title: GradBias: Unveiling Word Influence on Bias in Text-to-Image Generative Models

• Authors: Moreno D'Incà, Elia Peruzzo, Massimiliano Mancini, Xingqian Xu, Humphrey Shi, Nicu Sebe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16700
• Pdf URL: https://arxiv.org/pdf/2408.16700
• Copy Paste: [[2408.16700]] GradBias: Unveiling Word Influence on Bias in Text-to-Image Generative Models(https://arxiv.org/abs/2408.16700)
• Keywords: fair, generative, large language model
• Abstract: Recent progress in Text-to-Image (T2I) generative models has enabled high-quality image generation. As performance and accessibility increase, these models are gaining significant attraction and popularity: ensuring their fairness and safety is a priority to prevent the dissemination and perpetuation of biases. However, existing studies in bias detection focus on closed sets of predefined biases (e.g., gender, ethnicity). In this paper, we propose a general framework to identify, quantify, and explain biases in an open set setting, i.e. without requiring a predefined set. This pipeline leverages a Large Language Model (LLM) to propose biases starting from a set of captions. Next, these captions are used by the target generative model for generating a set of images. Finally, Vision Question Answering (VQA) is leveraged for bias evaluation. We show two variations of this framework: OpenBias and GradBias. OpenBias detects and quantifies biases, while GradBias determines the contribution of individual prompt words on biases. OpenBias effectively detects both well-known and novel biases related to people, objects, and animals and highly aligns with existing closed-set bias detection methods and human judgment. GradBias shows that neutral words can significantly influence biases and it outperforms several baselines, including state-of-the-art foundation models. Code available here: this https URL.

Title: One-Shot Learning Meets Depth Diffusion in Multi-Object Videos

• Authors: Anisha Jain
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16704
• Pdf URL: https://arxiv.org/pdf/2408.16704
• Copy Paste: [[2408.16704]] One-Shot Learning Meets Depth Diffusion in Multi-Object Videos(https://arxiv.org/abs/2408.16704)
• Keywords: diffusion
• Abstract: Creating editable videos that depict complex interactions between multiple objects in various artistic styles has long been a challenging task in filmmaking. Progress is often hampered by the scarcity of data sets that contain paired text descriptions and corresponding videos that showcase these interactions. This paper introduces a novel depth-conditioning approach that significantly advances this field by enabling the generation of coherent and diverse videos from just a single text-video pair using a pre-trained depth-aware Text-to-Image (T2I) model. Our method fine-tunes the pre-trained model to capture continuous motion by employing custom-designed spatial and temporal attention mechanisms. During inference, we use the DDIM inversion to provide structural guidance for video generation. This innovative technique allows for continuously controllable depth in videos, facilitating the generation of multiobject interactions while maintaining the concept generation and compositional strengths of the original T2I model across various artistic styles, such as photorealism, animation, and impressionism.

Title: Enhanced forecasting of stock prices based on variational mode decomposition, PatchTST, and adaptive scale-weighted layer

• Authors: Xiaorui Xue, Shaofang Li, Xiaonan Wang
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2408.16707
• Pdf URL: https://arxiv.org/pdf/2408.16707
• Copy Paste: [[2408.16707]] Enhanced forecasting of stock prices based on variational mode decomposition, PatchTST, and adaptive scale-weighted layer(https://arxiv.org/abs/2408.16707)
• Keywords: robust
• Abstract: The significant fluctuations in stock index prices in recent years highlight the critical need for accurate forecasting to guide investment and financial strategies. This study introduces a novel composite forecasting framework that integrates variational mode decomposition (VMD), PatchTST, and adaptive scale-weighted layer (ASWL) to address these challenges. Utilizing datasets of four major stock indices--SP500, DJI, SSEC, and FTSE--from 2000 to 2024, the proposed method first decomposes the raw price series into intrinsic mode functions (IMFs) using VMD. Each IMF is then modeled with PatchTST to capture temporal patterns effectively. The ASWL module is applied to incorporate scale information, enhancing prediction accuracy. The final forecast is derived by aggregating predictions from all IMFs. The VMD-PatchTST-ASWL framework demonstrates significant improvements in forecasting accuracy compared to traditional models, showing robust performance across different indices. This innovative approach provides a powerful tool for stock index price forecasting, with potential applications in various financial analysis and investment decision-making contexts.

Title: ARINC 429 Cyber-vulnerabilities and Voltage Data in a Hardware-in-the-Loop Simulator

• Authors: Connor Trask, Rosene Clark, Justace Clutter, Mark Herrera, Steve Movit, Kelly Tran
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16714
• Pdf URL: https://arxiv.org/pdf/2408.16714
• Copy Paste: [[2408.16714]] ARINC 429 Cyber-vulnerabilities and Voltage Data in a Hardware-in-the-Loop Simulator(https://arxiv.org/abs/2408.16714)
• Keywords: secure, attack
• Abstract: ARINC 429 is a ubiquitous data bus for civil avionics, enabling reliable communication between devices from disparate manufacturers. However, ARINC 429 lacks any form of encryption or authentication, making it an inherently insecure communication protocol and rendering any connected avionics vulnerable to a range of attacks. We constructed a hardware-in-the-loop simulator with ARINC 429 buses, explored these vulnerabilities, and identified their potential to deny, degrade, or disrupt aircraft capabilities. We performed a denial-of-service attack against a multi-function display via a compromised ARINC 429 bus using commercially available tools, which succeeded in disabling important navigational aids. This proven attack on physical avionics illustrates the risk inherent in ARINC 429 and the need for the ability to detect these attacks. One potential mitigation is an intrusion detection system (IDS) trained on data collected from the electrical properties of the physical bus. Although previous research has demonstrated the feasibility of an IDS on an ARINC 429 bus, no IDS has been trained on data generated by avionics hardware. To facilitate this, we recorded voltage traces and message history generated by avionics and adversarial devices on the ARINC 429 bus. To the best of our knowledge, this is the first publicly available collection of hardware-generated ARINC 429 signal data.

Title: A GREAT Architecture for Edge-Based Graph Problems Like TSP

• Authors: Attila Lischka, Jiaming Wu, Morteza Haghir Chehreghani, Balázs Kulcsár
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16717
• Pdf URL: https://arxiv.org/pdf/2408.16717
• Copy Paste: [[2408.16717]] A GREAT Architecture for Edge-Based Graph Problems Like TSP(https://arxiv.org/abs/2408.16717)
• Keywords: transformer
• Abstract: In the last years, many neural network-based approaches have been proposed to tackle combinatorial optimization problems such as routing problems. Many of these approaches are based on graph neural networks (GNNs) or related transformers, operating on the Euclidean coordinates representing the routing problems. However, GNNs are inherently not well suited to operate on dense graphs, such as in routing problems. Furthermore, models operating on Euclidean coordinates cannot be applied to non-Euclidean versions of routing problems that are often found in real-world settings. To overcome these limitations, we propose a novel GNN-related edge-based neural model called Graph Edge Attention Network (GREAT). We evaluate the performance of GREAT in the edge-classification task to predict optimal edges in the Traveling Salesman Problem (TSP). We can use such a trained GREAT model to produce sparse TSP graph instances, keeping only the edges GREAT finds promising. Compared to other, non-learning-based methods to sparsify TSP graphs, GREAT can produce very sparse graphs while keeping most of the optimal edges. Furthermore, we build a reinforcement learning-based GREAT framework which we apply to Euclidean and non-Euclidean asymmetric TSP. This framework achieves state-of-the-art results.

Title: H-SGANet: Hybrid Sparse Graph Attention Network for Deformable Medical Image Registration

• Authors: Yufeng Zhou, Wenming Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16719
• Pdf URL: https://arxiv.org/pdf/2408.16719
• Copy Paste: [[2408.16719]] H-SGANet: Hybrid Sparse Graph Attention Network for Deformable Medical Image Registration(https://arxiv.org/abs/2408.16719)
• Keywords: transformer
• Abstract: The integration of Convolutional Neural Network (ConvNet) and Transformer has emerged as a strong candidate for image registration, leveraging the strengths of both models and a large parameter space. However, this hybrid model, treating brain MRI volumes as grid or sequence structures, faces challenges in accurately representing anatomical connectivity, diverse brain regions, and vital connections contributing to the brain's internal architecture. Concerns also arise regarding the computational expense and GPU memory usage associated with this model. To tackle these issues, a lightweight hybrid sparse graph attention network (H-SGANet) has been developed. This network incorporates a central mechanism, Sparse Graph Attention (SGA), based on a Vision Graph Neural Network (ViG) with predetermined anatomical connections. The SGA module expands the model's receptive field and seamlessly integrates into the network. To further amplify the advantages of the hybrid network, the Separable Self-Attention (SSA) is employed as an enhanced token mixer, integrated with depth-wise convolution to constitute SSAFormer. This strategic integration is designed to more effectively extract long-range dependencies. As a hybrid ConvNet-ViG-Transformer model, H-SGANet offers threefold benefits for volumetric medical image registration. It optimizes fixed and moving images concurrently through a hybrid feature fusion layer and an end-to-end learning framework. Compared to VoxelMorph, a model with a similar parameter count, H-SGANet demonstrates significant performance enhancements of 3.5% and 1.5% in Dice score on the OASIS dataset and LPBA40 dataset, respectively.

Title: Prediction-Feedback DETR for Temporal Action Detection

• Authors: Jihwan Kim, Miso Lee, Cheol-Ho Cho, Jihyun Lee, Jae-Pil Heo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16729
• Pdf URL: https://arxiv.org/pdf/2408.16729
• Copy Paste: [[2408.16729]] Prediction-Feedback DETR for Temporal Action Detection(https://arxiv.org/abs/2408.16729)
• Keywords: transformer
• Abstract: Temporal Action Detection (TAD) is fundamental yet challenging for real-world video applications. Leveraging the unique benefits of transformers, various DETR-based approaches have been adopted in TAD. However, it has recently been identified that the attention collapse in self-attention causes the performance degradation of DETR for TAD. Building upon previous research, this paper newly addresses the attention collapse problem in cross-attention within DETR-based TAD methods. Moreover, our findings reveal that cross-attention exhibits patterns distinct from predictions, indicating a short-cut phenomenon. To resolve this, we propose a new framework, Prediction-Feedback DETR (Pred-DETR), which utilizes predictions to restore the collapse and align the cross- and self-attention with predictions. Specifically, we devise novel prediction-feedback objectives using guidance from the relations of the predictions. As a result, Pred-DETR significantly alleviates the collapse and achieves state-of-the-art performance among DETR-based methods on various challenging benchmarks including THUMOS14, ActivityNet-v1.3, HACS, and FineAction.

Title: VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation

• Authors: Shiwei Wu, Joya Chen, Kevin Qinghong Lin, Qimeng Wang, Yan Gao, Qianli Xu, Tong Xu, Yao Hu, Enhong Chen, Mike Zheng Shou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16730
• Pdf URL: https://arxiv.org/pdf/2408.16730
• Copy Paste: [[2408.16730]] VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation(https://arxiv.org/abs/2408.16730)
• Keywords: transformer
• Abstract: A well-known dilemma in large vision-language models (e.g., GPT-4, LLaVA) is that while increasing the number of vision tokens generally enhances visual understanding, it also significantly raises memory and computational costs, especially in long-term, dense video frame streaming scenarios. Although learnable approaches like Q-Former and Perceiver Resampler have been developed to reduce the vision token burden, they overlook the context causally modeled by LLMs (i.e., key-value cache), potentially leading to missed visual cues when addressing user queries. In this paper, we introduce a novel approach to reduce vision compute by leveraging redundant vision tokens "skipping layers" rather than decreasing the number of vision tokens. Our method, VideoLLM-MoD, is inspired by mixture-of-depths LLMs and addresses the challenge of numerous vision tokens in long-term or streaming video. Specifically, for each transformer layer, we learn to skip the computation for a high proportion (e.g., 80\%) of vision tokens, passing them directly to the next layer. This approach significantly enhances model efficiency, achieving approximately \textasciitilde42\% time and \textasciitilde30\% memory savings for the entire training. Moreover, our method reduces the computation in the context and avoid decreasing the vision tokens, thus preserving or even improving performance compared to the vanilla model. We conduct extensive experiments to demonstrate the effectiveness of VideoLLM-MoD, showing its state-of-the-art results on multiple benchmarks, including narration, forecasting, and summarization tasks in COIN, Ego4D, and Ego-Exo4D datasets.

Title: Theoretical and Methodological Framework for Studying Texts Produced by Large Language Models

• Authors: Jiří Milička
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16740
• Pdf URL: https://arxiv.org/pdf/2408.16740
• Copy Paste: [[2408.16740]] Theoretical and Methodological Framework for Studying Texts Produced by Large Language Models(https://arxiv.org/abs/2408.16740)
• Keywords: robust, large language model
• Abstract: This paper addresses the conceptual, methodological and technical challenges in studying large language models (LLMs) and the texts they produce from a quantitative linguistics perspective. It builds on a theoretical framework that distinguishes between the LLM as a substrate and the entities the model simulates. The paper advocates for a strictly non-anthropomorphic approach to models while cautiously applying methodologies used in studying human linguistic behavior to the simulated entities. While natural language processing researchers focus on the models themselves, their architecture, evaluation, and methods for improving performance, we as quantitative linguists should strive to build a robust theory concerning the characteristics of texts produced by LLMs, how they differ from human-produced texts, and the properties of simulated entities. Additionally, we should explore the potential of LLMs as an instrument for studying human culture, of which language is an integral part.

Title: Assessing Large Language Models for Online Extremism Research: Identification, Explanation, and New Knowledge

• Authors: Beidi Dong, Jin R. Lee, Ziwei Zhu, Balassubramanian Srinivasan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16749
• Pdf URL: https://arxiv.org/pdf/2408.16749
• Copy Paste: [[2408.16749]] Assessing Large Language Models for Online Extremism Research: Identification, Explanation, and New Knowledge(https://arxiv.org/abs/2408.16749)
• Keywords: transformer, generative, large language model
• Abstract: The United States has experienced a significant increase in violent extremism, prompting the need for automated tools to detect and limit the spread of extremist ideology online. This study evaluates the performance of Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-Trained Transformers (GPT) in detecting and classifying online domestic extremist posts. We collected social media posts containing "far-right" and "far-left" ideological keywords and manually labeled them as extremist or non-extremist. Extremist posts were further classified into one or more of five contributing elements of extremism based on a working definitional framework. The BERT model's performance was evaluated based on training data size and knowledge transfer between categories. We also compared the performance of GPT 3.5 and GPT 4 models using different prompts: naïve, layperson-definition, role-playing, and professional-definition. Results showed that the best performing GPT models outperformed the best performing BERT models, with more detailed prompts generally yielding better results. However, overly complex prompts may impair performance. Different versions of GPT have unique sensitives to what they consider extremist. GPT 3.5 performed better at classifying far-left extremist posts, while GPT 4 performed better at classifying far-right extremist posts. Large language models, represented by GPT models, hold significant potential for online extremism classification tasks, surpassing traditional BERT models in a zero-shot setting. Future research should explore human-computer interactions in optimizing GPT models for extremist detection and classification tasks to develop more efficient (e.g., quicker, less effort) and effective (e.g., fewer errors or mistakes) methods for identifying extremist content.

Title: A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models

• Authors: Yi-Lin Tuan, William Yang Wang
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.16751
• Pdf URL: https://arxiv.org/pdf/2408.16751
• Copy Paste: [[2408.16751]] A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models(https://arxiv.org/abs/2408.16751)
• Keywords: generative
• Abstract: Beyond maximum likelihood estimation (MLE), the standard objective of a language model (LM) that optimizes good examples probabilities, many studies have explored ways that also penalize bad examples for enhancing the quality of output distribution, including unlikelihood training, exponential maximizing average treatment effect (ExMATE), and direct preference optimization (DPO). To systematically compare these methods and further provide a unified recipe for LM optimization, in this paper, we present a unique angle of gradient analysis of loss functions that simultaneously reward good examples and penalize bad ones in LMs. Through both mathematical results and experiments on CausalDialogue and Anthropic HH-RLHF datasets, we identify distinct functional characteristics among these methods. We find that ExMATE serves as a superior surrogate for MLE, and that combining DPO with ExMATE instead of MLE further enhances both the statistical (5-7%) and generative (+18% win rate) performance.

Title: Reinforcement Learning without Human Feedback for Last Mile Fine-Tuning of Large Language Models

• Authors: Alec Solway
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16753
• Pdf URL: https://arxiv.org/pdf/2408.16753
• Copy Paste: [[2408.16753]] Reinforcement Learning without Human Feedback for Last Mile Fine-Tuning of Large Language Models(https://arxiv.org/abs/2408.16753)
• Keywords: large language model
• Abstract: Reinforcement learning is used to align language models with human preference signals after first pre-training the model to predict the next token of text within a large corpus using likelihood maximization. Before being deployed in a specific domain, models are often further fine-tuned on task specific data. Since human preferences are often unavailable for the last step, it is performed using likelihood maximization as that is the typical default method. However, reinforcement learning has other advantages besides facilitating alignment to a human derived reward function. For one, whereas likelihood maximization is a form of imitation learning in which the model is trained on what to do under ideal conditions, reinforcement learning is not limited to demonstrating actions just for optimally reached states and trains a model what to do under a range of scenarios as it explores the policy space. In addition, it also trains a model what not to do, suppressing competitive but poor actions. This work develops a framework for last-mile fine-tuning using reinforcement learning and tests whether it garners performance gains. The experiments center on abstractive summarization, but the framework is general and broadly applicable. Use of the procedure produced significantly better results than likelihood maximization when comparing raw predictions. For the specific data tested, the gap could be bridged by employing post-processing of the maximum likelihood outputs. Nonetheless, the framework offers a new avenue for model optimization in situations where post-processing may be less straightforward or effective, and it can be extended to include more complex classes of undesirable outputs to penalize and train against, such as hallucinations.

Title: How Far Can Cantonese NLP Go? Benchmarking Cantonese Capabilities of Large Language Models

• Authors: Jiyue Jiang, Liheng Chen, Pengan Chen, Sheng Wang, Qinghang Bao, Lingpeng Kong, Yu Li, Chuan Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16756
• Pdf URL: https://arxiv.org/pdf/2408.16756
• Copy Paste: [[2408.16756]] How Far Can Cantonese NLP Go? Benchmarking Cantonese Capabilities of Large Language Models(https://arxiv.org/abs/2408.16756)
• Keywords: large language model
• Abstract: The rapid evolution of large language models (LLMs) has transformed the competitive landscape in natural language processing (NLP), particularly for English and other data-rich languages. However, underrepresented languages like Cantonese, spoken by over 85 million people, face significant development gaps, which is particularly concerning given the economic significance of the Guangdong-Hong Kong-Macau Greater Bay Area, and in substantial Cantonese-speaking populations in places like Singapore and North America. Despite its wide use, Cantonese has scant representation in NLP research, especially compared to other languages from similarly developed regions. To bridge these gaps, we outline current Cantonese NLP methods and introduce new benchmarks designed to evaluate LLM performance in factual generation, mathematical logic, complex reasoning, and general knowledge in Cantonese, which aim to advance open-source Cantonese LLM technology. We also propose future research directions and recommended models to enhance Cantonese LLM development.

Title: UV-free Texture Generation with Denoising and Geodesic Heat Diffusions

• Authors: Simone Foti, Stefanos Zafeiriou, Tolga Birdal
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16762
• Pdf URL: https://arxiv.org/pdf/2408.16762
• Copy Paste: [[2408.16762]] UV-free Texture Generation with Denoising and Geodesic Heat Diffusions(https://arxiv.org/abs/2408.16762)
• Keywords: diffusion, generative
• Abstract: Seams, distortions, wasted UV space, vertex-duplication, and varying resolution over the surface are the most prominent issues of the standard UV-based texturing of meshes. These issues are particularly acute when automatic UV-unwrapping techniques are used. For this reason, instead of generating textures in automatically generated UV-planes like most state-of-the-art methods, we propose to represent textures as coloured point-clouds whose colours are generated by a denoising diffusion probabilistic model constrained to operate on the surface of 3D objects. Our sampling and resolution agnostic generative model heavily relies on heat diffusion over the surface of the meshes for spatial communication between points. To enable processing of arbitrarily sampled point-cloud textures and ensure long-distance texture consistency we introduce a fast re-sampling of the mesh spectral properties used during the heat diffusion and introduce a novel heat-diffusion-based self-attention mechanism. Our code and pre-trained models are available at this http URL.

Title: A Score-Based Density Formula, with Applications in Diffusion Generative Models

• Authors: Gen Li, Yuling Yan
• Subjects: cs.LG, cs.AI, math.PR, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.16765
• Pdf URL: https://arxiv.org/pdf/2408.16765
• Copy Paste: [[2408.16765]] A Score-Based Density Formula, with Applications in Diffusion Generative Models(https://arxiv.org/abs/2408.16765)
• Keywords: diffusion, generative
• Abstract: Score-based generative models (SGMs) have revolutionized the field of generative modeling, achieving unprecedented success in generating realistic and diverse content. Despite empirical advances, the theoretical basis for why optimizing the evidence lower bound (ELBO) on the log-likelihood is effective for training diffusion generative models, such as DDPMs, remains largely unexplored. In this paper, we address this question by establishing a density formula for a continuous-time diffusion process, which can be viewed as the continuous-time limit of the forward process in an SGM. This formula reveals the connection between the target density and the score function associated with each step of the forward process. Building on this, we demonstrate that the minimizer of the optimization objective for training DDPMs nearly coincides with that of the true objective, providing a theoretical foundation for optimizing DDPMs using the ELBO. Furthermore, we offer new insights into the role of score-matching regularization in training GANs, the use of ELBO in diffusion classifiers, and the recently proposed diffusion loss.

Title: CSGO: Content-Style Composition in Text-to-Image Generation

• Authors: Peng Xing, Haofan Wang, Yanpeng Sun, Qixun Wang, Xu Bai, Hao Ai, Renyuan Huang, Zechao Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.16766
• Pdf URL: https://arxiv.org/pdf/2408.16766
• Copy Paste: [[2408.16766]] CSGO: Content-Style Composition in Text-to-Image Generation(https://arxiv.org/abs/2408.16766)
• Keywords: diffusion
• Abstract: The diffusion model has shown exceptional capabilities in controlled image generation, which has further fueled interest in image style transfer. Existing works mainly focus on training free-based methods (e.g., image inversion) due to the scarcity of specific data. In this study, we present a data construction pipeline for content-style-stylized image triplets that generates and automatically cleanses stylized data triplets. Based on this pipeline, we construct a dataset IMAGStyle, the first large-scale style transfer dataset containing 210k image triplets, available for the community to explore and research. Equipped with IMAGStyle, we propose CSGO, a style transfer model based on end-to-end training, which explicitly decouples content and style features employing independent feature injection. The unified CSGO implements image-driven style transfer, text-driven stylized synthesis, and text editing-driven stylized synthesis. Extensive experiments demonstrate the effectiveness of our approach in enhancing style control capabilities in image generation. Additional visualization and access to the source code can be located on the project page: \url{this https URL}.

Title: ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model

• Authors: Fangfu Liu, Wenqiang Sun, Hanyang Wang, Yikai Wang, Haowen Sun, Junliang Ye, Jun Zhang, Yueqi Duan
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.16767
• Pdf URL: https://arxiv.org/pdf/2408.16767
• Copy Paste: [[2408.16767]] ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model(https://arxiv.org/abs/2408.16767)
• Keywords: diffusion, generative
• Abstract: Advancements in 3D scene reconstruction have transformed 2D images from the real world into 3D models, producing realistic 3D results from hundreds of input photos. Despite great success in dense-view reconstruction scenarios, rendering a detailed scene from insufficient captured views is still an ill-posed optimization problem, often resulting in artifacts and distortions in unseen areas. In this paper, we propose ReconX, a novel 3D scene reconstruction paradigm that reframes the ambiguous reconstruction challenge as a temporal generation task. The key insight is to unleash the strong generative prior of large pre-trained video diffusion models for sparse-view reconstruction. However, 3D view consistency struggles to be accurately preserved in directly generated video frames from pre-trained models. To address this, given limited input views, the proposed ReconX first constructs a global point cloud and encodes it into a contextual space as the 3D structure condition. Guided by the condition, the video diffusion model then synthesizes video frames that are both detail-preserved and exhibit a high degree of 3D consistency, ensuring the coherence of the scene from various perspectives. Finally, we recover the 3D scene from the generated video through a confidence-aware 3D Gaussian Splatting optimization scheme. Extensive experiments on various real-world datasets show the superiority of our ReconX over state-of-the-art methods in terms of quality and generalizability.

Title: SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Manners

• Authors: Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Chengzhuo Tong, Peng Gao, Chunyuan Li, Pheng-Ann Heng
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16768
• Pdf URL: https://arxiv.org/pdf/2408.16768
• Copy Paste: [[2408.16768]] SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Manners(https://arxiv.org/abs/2408.16768)
• Keywords: robust, segmentation
• Abstract: We introduce SAM2Point, a preliminary exploration adapting Segment Anything Model 2 (SAM 2) for zero-shot and promptable 3D segmentation. SAM2Point interprets any 3D data as a series of multi-directional videos, and leverages SAM 2 for 3D-space segmentation, without further training or 2D-3D projection. Our framework supports various prompt types, including 3D points, boxes, and masks, and can generalize across diverse scenarios, such as 3D objects, indoor scenes, outdoor environments, and raw sparse LiDAR. Demonstrations on multiple 3D datasets, e.g., Objaverse, S3DIS, ScanNet, Semantic3D, and KITTI, highlight the robust generalization capabilities of SAM2Point. To our best knowledge, we present the most faithful implementation of SAM in 3D, which may serve as a starting point for future research in promptable 3D segmentation. Online Demo: this https URL . Code: this https URL .

Title: PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learning

• Authors: Noor Hussein, Fahad Shamshad, Muzammal Naseer, Karthik Nandakumar
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.16769
• Pdf URL: https://arxiv.org/pdf/2408.16769
• Copy Paste: [[2408.16769]] PromptSmooth: Certifying Robustness of Medical Vision-Language Models via Prompt Learning(https://arxiv.org/abs/2408.16769)
• Keywords: attack, robust
• Abstract: Medical vision-language models (Med-VLMs) trained on large datasets of medical image-text pairs and later fine-tuned for specific tasks have emerged as a mainstream paradigm in medical image analysis. However, recent studies have highlighted the susceptibility of these Med-VLMs to adversarial attacks, raising concerns about their safety and robustness. Randomized smoothing is a well-known technique for turning any classifier into a model that is certifiably robust to adversarial perturbations. However, this approach requires retraining the Med-VLM-based classifier so that it classifies well under Gaussian noise, which is often infeasible in practice. In this paper, we propose a novel framework called PromptSmooth to achieve efficient certified robustness of Med-VLMs by leveraging the concept of prompt learning. Given any pre-trained Med-VLM, PromptSmooth adapts it to handle Gaussian noise by learning textual prompts in a zero-shot or few-shot manner, achieving a delicate balance between accuracy and robustness, while minimizing the computational overhead. Moreover, PromptSmooth requires only a single model to handle multiple noise levels, which substantially reduces the computational cost compared to traditional methods that rely on training a separate model for each noise level. Comprehensive experiments based on three Med-VLMs and across six downstream datasets of various imaging modalities demonstrate the efficacy of PromptSmooth. Our code and models are available at this https URL.