2024-08-28

Title: Evolvable Psychology Informed Neural Network for Memory Behavior Modeling

• Authors: Xiaoxuan Shen, Zhihai Hu, Qirong Chen, Shengyingjie Liu, Ruxia Liang, Jianwen Sun
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.14492
• Pdf URL: https://arxiv.org/pdf/2408.14492
• Copy Paste: [[2408.14492]] Evolvable Psychology Informed Neural Network for Memory Behavior Modeling(https://arxiv.org/abs/2408.14492)
• Keywords: interpretability
• Abstract: Memory behavior modeling is a core issue in cognitive psychology and education. Classical psychological theories typically use memory equations to describe memory behavior, which exhibits insufficient accuracy and controversy, while data-driven memory modeling methods often require large amounts of training data and lack interpretability. Knowledge-informed neural network models have shown excellent performance in fields like physics, but there have been few attempts in the domain of behavior modeling. This paper proposed a psychology theory informed neural networks for memory behavior modeling named PsyINN, where it constructs a framework that combines neural network with differentiating sparse regression, achieving joint optimization. Specifically, to address the controversies and ambiguity of descriptors in memory equations, a descriptor evolution method based on differentiating operators is proposed to achieve precise characterization of descriptors and the evolution of memory theoretical equations. Additionally, a buffering mechanism for the sparse regression and a multi-module alternating iterative optimization method are proposed, effectively mitigating gradient instability and local optima issues. On four large-scale real-world memory behavior datasets, the proposed method surpasses the state-of-the-art methods in prediction accuracy. Ablation study demonstrates the effectiveness of the proposed refinements, and application experiments showcase its potential in inspiring psychological research.

Title: Extraction of Typical Operating Scenarios of New Power System Based on Deep Time Series Aggregation

• Authors: Zhaoyang Qu, Zhenming Zhang, Nan Qu, Yuguang Zhou, Yang Li, Tao Jiang, Min Li, Chao Long
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2408.14493
• Pdf URL: https://arxiv.org/pdf/2408.14493
• Copy Paste: [[2408.14493]] Extraction of Typical Operating Scenarios of New Power System Based on Deep Time Series Aggregation(https://arxiv.org/abs/2408.14493)
• Keywords: robust, extraction
• Abstract: Extracting typical operational scenarios is essential for making flexible decisions in the dispatch of a new power system. This study proposed a novel deep time series aggregation scheme (DTSAs) to generate typical operational scenarios, considering the large amount of historical operational snapshot data. Specifically, DTSAs analyze the intrinsic mechanisms of different scheduling operational scenario switching to mathematically represent typical operational scenarios. A gramian angular summation field (GASF) based operational scenario image encoder was designed to convert operational scenario sequences into high-dimensional spaces. This enables DTSAs to fully capture the spatiotemporal characteristics of new power systems using deep feature iterative aggregation models. The encoder also facilitates the generation of typical operational scenarios that conform to historical data distributions while ensuring the integrity of grid operational snapshots. Case studies demonstrate that the proposed method extracted new fine-grained power system dispatch schemes and outperformed the latest high-dimensional featurescreening methods. In addition, experiments with different new energy access ratios were conducted to verify the robustness of the proposed method. DTSAs enables dispatchers to master the operation experience of the power system in advance, and actively respond to the dynamic changes of the operation scenarios under the high access rate of new energy.

Title: Knowledge Graph Modeling-Driven Large Language Model Operating System (LLM OS) for Task Automation in Process Engineering Problem-Solving

• Authors: Sakhinana Sagar Srinivas, Vijay Sri Vaikunth, Venkataramana Runkana
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14494
• Pdf URL: https://arxiv.org/pdf/2408.14494
• Copy Paste: [[2408.14494]] Knowledge Graph Modeling-Driven Large Language Model Operating System (LLM OS) for Task Automation in Process Engineering Problem-Solving(https://arxiv.org/abs/2408.14494)
• Keywords: large language model
• Abstract: We present the Process Engineering Operations Assistant (PEOA), an AI-driven framework designed to solve complex problems in the chemical and process industries. The framework employs a modular architecture orchestrated by a meta-agent, which serves as the central coordinator, managing an action generator and instruction-tuned small-scale language models (expert models). The action generator decomposes complex problems into sub-tasks and identifies suitable expert models to execute each, delivering precise solutions for multi-step problem-solving. Key techniques include advanced knowledge modeling using property graphs for improved information retrieval, facilitating more accurate and contextually relevant solutions. Additionally, the framework utilizes a teacher-student transfer-learning approach with GPT-4 (Omni) to fine-tune the action generator and expert models for domain adaptation, alongside an iterative problem-solving mechanism with sophisticated error handling. Custom datasets were developed to evaluate the framework against leading proprietary language models on various engineering tasks. The results demonstrate the framework effectiveness in automating calculations, accelerating prototyping, and providing AI-augmented decision support for industrial processes, marking a significant advancement in process engineering capabilities.

Title: Empowering Pre-Trained Language Models for Spatio-Temporal Forecasting via Decoupling Enhanced Discrete Reprogramming

• Authors: Hao Wang, Jindong Han, Wei Fan, Hao Liu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14505
• Pdf URL: https://arxiv.org/pdf/2408.14505
• Copy Paste: [[2408.14505]] Empowering Pre-Trained Language Models for Spatio-Temporal Forecasting via Decoupling Enhanced Discrete Reprogramming(https://arxiv.org/abs/2408.14505)
• Keywords: diffusion
• Abstract: Spatio-temporal time series forecasting plays a critical role in various real-world applications, such as transportation optimization, energy management, and climate analysis. The recent advancements in Pre-trained Language Models (PLMs) have inspired efforts to reprogram these models for time series forecasting tasks, by leveraging their superior reasoning and generalization capabilities. However, existing approaches fall short in handling complex spatial inter-series dependencies and intrinsic intra-series frequency components, limiting their spatio-temporal forecasting performance. Moreover, the linear mapping of continuous time series to a compressed subset vocabulary in reprogramming constrains the spatio-temporal semantic expressivity of PLMs and may lead to potential information bottleneck. To overcome the above limitations, we propose \textsc{RePST}, a tailored PLM reprogramming framework for spatio-temporal forecasting. The key insight of \textsc{RePST} is to decouple the spatio-temporal dynamics in the frequency domain, allowing better alignment with the PLM text space. Specifically, we first decouple spatio-temporal data in Fourier space and devise a structural diffusion operator to obtain temporal intrinsic and spatial diffusion signals, making the dynamics more comprehensible and predictable for PLMs. To avoid information bottleneck from a limited vocabulary, we further propose a discrete reprogramming strategy that selects relevant discrete textual information from an expanded vocabulary space in a differentiable manner. Extensive experiments on four real-world datasets show that our proposed approach significantly outperforms state-of-the-art spatio-temporal forecasting models, particularly in data-scarce scenarios.

Title: LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings

• Authors: Duo Wang, Yuan Zuo, Fengzhi Li, Junjie Wu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14512
• Pdf URL: https://arxiv.org/pdf/2408.14512
• Copy Paste: [[2408.14512]] LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings(https://arxiv.org/abs/2408.14512)
• Keywords: large language model
• Abstract: Zero-shot graph machine learning, especially with graph neural networks (GNNs), has garnered significant interest due to the challenge of scarce labeled data. While methods like self-supervised learning and graph prompt learning have been extensively explored, they often rely on fine-tuning with task-specific labels, limiting their effectiveness in zero-shot scenarios. Inspired by the zero-shot capabilities of instruction-fine-tuned large language models (LLMs), we introduce a novel framework named Token Embedding-Aligned Graph Language Model (TEA-GLM) that leverages LLMs as cross-dataset and cross-task zero-shot learners for graph machine learning. Concretely, we pretrain a GNN, aligning its representations with token embeddings of an LLM. We then train a linear projector that transforms the GNN's representations into a fixed number of graph token embeddings without tuning the LLM. A unified instruction is designed for various graph tasks at different levels, such as node classification (node-level) and link prediction (edge-level). These design choices collectively enhance our method's effectiveness in zero-shot learning, setting it apart from existing methods. Experiments show that our graph token embeddings help the LLM predictor achieve state-of-the-art performance on unseen datasets and tasks compared to other methods using LLMs as predictors.

Title: Variational autoencoder-based neural network model compression

• Authors: Liang Cheng, Peiyuan Guan, Amir Taherkordi, Lei Liu, Dapeng Lan
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14513
• Pdf URL: https://arxiv.org/pdf/2408.14513
• Copy Paste: [[2408.14513]] Variational autoencoder-based neural network model compression(https://arxiv.org/abs/2408.14513)
• Keywords: generative
• Abstract: Variational Autoencoders (VAEs), as a form of deep generative model, have been widely used in recent years, and shown great great peformance in a number of different domains, including image generation and anomaly detection, etc.. This paper aims to explore neural network model compression method based on VAE. The experiment uses different neural network models for MNIST recognition as compression targets, including Feedforward Neural Network (FNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM). These models are the most basic models in deep learning, and other more complex and advanced models are based on them or inherit their features and evolve. In the experiment, the first step is to train the models mentioned above, each trained model will have different accuracy and number of total parameters. And then the variants of parameters for each model are processed as training data in VAEs separately, and the trained VAEs are tested by the true model parameters. The experimental results show that using the latent space as a representation of the model compression can improve the compression rate compared to some traditional methods such as pruning and quantization, meanwhile the accuracy is not greatly affected using the model parameters reconstructed based on the latent space. In the future, a variety of different large-scale deep learning models will be used more widely, so exploring different ways to save time and space on saving or transferring models will become necessary, and the use of VAE in this paper can provide a basis for these further explorations.

Title: Retrieval Augmented Generation for Dynamic Graph Modeling

• Authors: Yuxia Wu, Yuan Fang, Lizi Liao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14523
• Pdf URL: https://arxiv.org/pdf/2408.14523
• Copy Paste: [[2408.14523]] Retrieval Augmented Generation for Dynamic Graph Modeling(https://arxiv.org/abs/2408.14523)
• Keywords: generative
• Abstract: Dynamic graph modeling is crucial for analyzing evolving patterns in various applications. Existing approaches often integrate graph neural networks with temporal modules or redefine dynamic graph modeling as a generative sequence task. However, these methods typically rely on isolated historical contexts of the target nodes from a narrow perspective, neglecting occurrences of similar patterns or relevant cases associated with other nodes. In this work, we introduce the Retrieval-Augmented Generation for Dynamic Graph Modeling (RAG4DyG) framework, which leverages guidance from contextually and temporally analogous examples to broaden the perspective of each node. This approach presents two critical challenges: (1) How to identify and retrieve high-quality demonstrations that are contextually and temporally analogous to dynamic graph samples? (2) How can these demonstrations be effectively integrated to improve dynamic graph modeling? To address these challenges, we propose RAG4DyG, which enriches the understanding of historical contexts by retrieving and learning from contextually and temporally pertinent demonstrations. Specifically, we employ a time- and context-aware contrastive learning module to identify and retrieve relevant cases for each query sequence. Moreover, we design a graph fusion strategy to integrate the retrieved cases, thereby augmenting the inherent historical contexts for improved prediction. Extensive experiments on real-world datasets across different domains demonstrate the effectiveness of RAG4DyG for dynamic graph modeling.

Title: Behavior-Based Detection of GPU Cryptojacking

• Authors: Dmitry Tanana
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.14554
• Pdf URL: https://arxiv.org/pdf/2408.14554
• Copy Paste: [[2408.14554]] Behavior-Based Detection of GPU Cryptojacking(https://arxiv.org/abs/2408.14554)
• Keywords: attack
• Abstract: With the surge in blockchain-based cryptocurrencies, illegal mining for cryptocurrency has become a popular cyberthreat. Host-based cryptojacking, where malicious actors exploit victims systems to mine cryptocurrency without their knowledge, is on the rise. Regular cryptojacking is relatively well-known and well-studied threat, however, recently attackers started switching to GPU cryptojacking, which promises greater profits due to high GPU hash rates and lower detection chance. Additionally, GPU cryptojackers can easily propagate using, for example, modified graphic card drivers. This article considers question of GPU cryptojacking detection. First, we discuss brief history and definition of GPU cryptojacking as well as previous attempts to design a detection technique for such threats. We also propose complex exposure mechanism based on GPU load by an application and graphic card RAM consumption, which can be used to detect both browser-based and host-based cryptojacking samples. Then we design a prototype decision tree detection program based on our technique. It was tested in a controlled virtual machine environment with 80% successful detection rate against selected set of GPU cryptojacking samples and 20% false positive rate against selected number of legitimate GPU-heavy applications.

Title: A Survey of Camouflaged Object Detection and Beyond

• Authors: Fengyang Xiao, Sujie Hu, Yuqi Shen, Chengyu Fang, Jinfa Huang, Chunming He, Longxiang Tang, Ziyun Yang, Xiu Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14562
• Pdf URL: https://arxiv.org/pdf/2408.14562
• Copy Paste: [[2408.14562]] A Survey of Camouflaged Object Detection and Beyond(https://arxiv.org/abs/2408.14562)
• Keywords: segmentation
• Abstract: Camouflaged Object Detection (COD) refers to the task of identifying and segmenting objects that blend seamlessly into their surroundings, posing a significant challenge for computer vision systems. In recent years, COD has garnered widespread attention due to its potential applications in surveillance, wildlife conservation, autonomous systems, and more. While several surveys on COD exist, they often have limitations in terms of the number and scope of papers covered, particularly regarding the rapid advancements made in the field since mid-2023. To address this void, we present the most comprehensive review of COD to date, encompassing both theoretical frameworks and practical contributions to the field. This paper explores various COD methods across four domains, including both image-level and video-level solutions, from the perspectives of traditional and deep learning approaches. We thoroughly investigate the correlations between COD and other camouflaged scenario methods, thereby laying the theoretical foundation for subsequent analyses. Beyond object-level detection, we also summarize extended methods for instance-level tasks, including camouflaged instance segmentation, counting, and ranking. Additionally, we provide an overview of commonly used benchmarks and evaluation metrics in COD tasks, conducting a comprehensive evaluation of deep learning-based techniques in both image and video domains, considering both qualitative and quantitative performance. Finally, we discuss the limitations of current COD models and propose 9 promising directions for future research, focusing on addressing inherent challenges and exploring novel, meaningful technologies. For those interested, a curated list of COD-related techniques, datasets, and additional resources can be found at this https URL

Title: Improving Clinical Note Generation from Complex Doctor-Patient Conversation

• Authors: Yizhan Li, Sifan Wu, Christopher Smith, Thomas Lo, Bang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14568
• Pdf URL: https://arxiv.org/pdf/2408.14568
• Copy Paste: [[2408.14568]] Improving Clinical Note Generation from Complex Doctor-Patient Conversation(https://arxiv.org/abs/2408.14568)
• Keywords: large language model
• Abstract: Writing clinical notes and documenting medical exams is a critical task for healthcare professionals, serving as a vital component of patient care documentation. However, manually writing these notes is time-consuming and can impact the amount of time clinicians can spend on direct patient interaction and other tasks. Consequently, the development of automated clinical note generation systems has emerged as a clinically meaningful area of research within AI for health. In this paper, we present three key contributions to the field of clinical note generation using large language models (LLMs). First, we introduce CliniKnote, a comprehensive dataset consisting of 1,200 complex doctor-patient conversations paired with their full clinical notes. This dataset, created and curated by medical experts with the help of modern neural networks, provides a valuable resource for training and evaluating models in clinical note generation tasks. Second, we propose the K-SOAP (Keyword, Subjective, Objective, Assessment, and Plan) note format, which enhances traditional SOAP~\cite{podder2023soap} (Subjective, Objective, Assessment, and Plan) notes by adding a keyword section at the top, allowing for quick identification of essential information. Third, we develop an automatic pipeline to generate K-SOAP notes from doctor-patient conversations and benchmark various modern LLMs using various metrics. Our results demonstrate significant improvements in efficiency and performance compared to standard LLM finetuning methods.

Title: CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation

• Authors: Muhammad Fawi
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14572
• Pdf URL: https://arxiv.org/pdf/2408.14572
• Copy Paste: [[2408.14572]] CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation(https://arxiv.org/abs/2408.14572)
• Keywords: large language model
• Abstract: This paper introduces CURLoRA, a novel approach to fine-tuning large language models (LLMs) that leverages CUR matrix decomposition in the context of Low-Rank Adaptation (LoRA). Our method addresses two critical challenges in LLM fine-tuning: mitigating catastrophic forgetting during continual learning and reducing the number of trainable parameters. We propose a unique modification to the CUR decomposition process, utilizing inverted probabilities for column and row selection which acts as an implicit regularization, and initializing the $U$ matrix as a zero matrix, and only fine-tuning it. We demonstrate through experiments on multiple datasets that CURLoRA outperforms standard LoRA in mitigating catastrophic forgetting. It maintains model stability and performance across tasks while significantly reducing the number of trainable parameters. Our results show that CURLoRA achieves very good and stable task accuracy while maintaining base model's perplexity scores fixed compared to LoRA upon continual fine-tuning, particularly in scenarios with limited data.

Title: DIAGen: Diverse Image Augmentation with Generative Models

• Authors: Tobias Lingenberg, Markus Reuter, Gopika Sudhakaran, Dominik Gojny, Stefan Roth, Simone Schaub-Meyer
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14584
• Pdf URL: https://arxiv.org/pdf/2408.14584
• Copy Paste: [[2408.14584]] DIAGen: Diverse Image Augmentation with Generative Models(https://arxiv.org/abs/2408.14584)
• Keywords: diffusion, generative
• Abstract: Simple data augmentation techniques, such as rotations and flips, are widely used to enhance the generalization power of computer vision models. However, these techniques often fail to modify high-level semantic attributes of a class. To address this limitation, researchers have explored generative augmentation methods like the recently proposed DA-Fusion. Despite some progress, the variations are still largely limited to textural changes, thus falling short on aspects like varied viewpoints, environment, weather conditions, or even class-level semantic attributes (eg, variations in a dog's breed). To overcome this challenge, we propose DIAGen, building upon DA-Fusion. First, we apply Gaussian noise to the embeddings of an object learned with Textual Inversion to diversify generations using a pre-trained diffusion model's knowledge. Second, we exploit the general knowledge of a text-to-text generative model to guide the image generation of the diffusion model with varied class-specific prompts. Finally, we introduce a weighting mechanism to mitigate the impact of poorly generated samples. Experimental results across various datasets show that DIAGen not only enhances semantic diversity but also improves the performance of subsequent classifiers. The advantages of DIAGen over standard augmentations and the DA-Fusion baseline are particularly pronounced with out-of-distribution samples.

Title: Global-Local Distillation Network-Based Audio-Visual Speaker Tracking with Incomplete Modalities

• Authors: Yidi Li, Yihan Li, Yixin Guo, Bin Ren, Zhenhuan Xu, Hao Guo, Hong Liu, Nicu Sebe
• Subjects: cs.CV, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.14585
• Pdf URL: https://arxiv.org/pdf/2408.14585
• Copy Paste: [[2408.14585]] Global-Local Distillation Network-Based Audio-Visual Speaker Tracking with Incomplete Modalities(https://arxiv.org/abs/2408.14585)
• Keywords: robust, generative
• Abstract: In speaker tracking research, integrating and complementing multi-modal data is a crucial strategy for improving the accuracy and robustness of tracking systems. However, tracking with incomplete modalities remains a challenging issue due to noisy observations caused by occlusion, acoustic noise, and sensor failures. Especially when there is missing data in multiple modalities, the performance of existing multi-modal fusion methods tends to decrease. To this end, we propose a Global-Local Distillation-based Tracker (GLDTracker) for robust audio-visual speaker tracking. GLDTracker is driven by a teacher-student distillation model, enabling the flexible fusion of incomplete information from each modality. The teacher network processes global signals captured by camera and microphone arrays, and the student network handles local information subject to visual occlusion and missing audio channels. By transferring knowledge from teacher to student, the student network can better adapt to complex dynamic scenes with incomplete observations. In the student network, a global feature reconstruction module based on the generative adversarial network is constructed to reconstruct global features from feature embedding with missing local information. Furthermore, a multi-modal multi-level fusion attention is introduced to integrate the incomplete feature and the reconstructed feature, leveraging the complementarity and consistency of audio-visual and global-local features. Experimental results on the AV16.3 dataset demonstrate that the proposed GLDTracker outperforms existing state-of-the-art audio-visual trackers and achieves leading performance on both standard and incomplete modalities datasets, highlighting its superiority and robustness in complex conditions. The code and models will be available.

Title: MMR: Evaluating Reading Ability of Large Multimodal Models

• Authors: Jian Chen, Ruiyi Zhang, Yufan Zhou, Ryan Rossi, Jiuxiang Gu, Changyou Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14594
• Pdf URL: https://arxiv.org/pdf/2408.14594
• Copy Paste: [[2408.14594]] MMR: Evaluating Reading Ability of Large Multimodal Models(https://arxiv.org/abs/2408.14594)
• Keywords: extraction
• Abstract: Large multimodal models (LMMs) have demonstrated impressive capabilities in understanding various types of image, including text-rich images. Most existing text-rich image benchmarks are simple extraction-based question answering, and many LMMs now easily achieve high scores. This means that current benchmarks fail to accurately reflect performance of different models, and a natural idea is to build a new benchmark to evaluate their complex reasoning and spatial understanding abilities. In this work, we propose the Multi-Modal Reading (MMR) benchmark in 11 diverse tasks to evaluate LMMs for text-rich image understanding. MMR is the first text-rich image benchmark built on human annotations with the help of language models. By evaluating several state-of-the-art LMMs, including GPT-4o, it reveals the limited capabilities of existing LMMs underscoring the value of our benchmark.

Title: PVAFN: Point-Voxel Attention Fusion Network with Multi-Pooling Enhancing for 3D Object Detection

• Authors: Yidi Li, Jiahao Wen, Bin Ren, Wenhao Li, Zhenhuan Xu, Hao Guo, Hong Liu, Nicu Sebe
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14600
• Pdf URL: https://arxiv.org/pdf/2408.14600
• Copy Paste: [[2408.14600]] PVAFN: Point-Voxel Attention Fusion Network with Multi-Pooling Enhancing for 3D Object Detection(https://arxiv.org/abs/2408.14600)
• Keywords: extraction
• Abstract: The integration of point and voxel representations is becoming more common in LiDAR-based 3D object detection. However, this combination often struggles with capturing semantic information effectively. Moreover, relying solely on point features within regions of interest can lead to information loss and limitations in local feature representation. To tackle these challenges, we propose a novel two-stage 3D object detector, called Point-Voxel Attention Fusion Network (PVAFN). PVAFN leverages an attention mechanism to improve multi-modal feature fusion during the feature extraction phase. In the refinement stage, it utilizes a multi-pooling strategy to integrate both multi-scale and region-specific information effectively. The point-voxel attention mechanism adaptively combines point cloud and voxel-based Bird's-Eye-View (BEV) features, resulting in richer object representations that help to reduce false detections. Additionally, a multi-pooling enhancement module is introduced to boost the model's perception capabilities. This module employs cluster pooling and pyramid pooling techniques to efficiently capture key geometric details and fine-grained shape structures, thereby enhancing the integration of local and global features. Extensive experiments on the KITTI and Waymo datasets demonstrate that the proposed PVAFN achieves competitive performance. The code and models will be available.

Title: Securing Biometric Data: Fully Homomorphic Encryption in Multimodal Iris and Face Recognition

• Authors: Surendra Singh, Lambert Igene, Stephanie Schuckers
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.14609
• Pdf URL: https://arxiv.org/pdf/2408.14609
• Copy Paste: [[2408.14609]] Securing Biometric Data: Fully Homomorphic Encryption in Multimodal Iris and Face Recognition(https://arxiv.org/abs/2408.14609)
• Keywords: security, privacy, protect, attack, robust, biometric
• Abstract: Multimodal biometric systems have gained popularity for their enhanced recognition accuracy and resistance to attacks like spoofing. This research explores methods for fusing iris and face feature vectors and implements robust security measures to protect fused databases and conduct matching operations on encrypted templates using fully homomorphic encryption (FHE). Evaluations on the QFIRE-I database demonstrate that our method effectively balances user privacy and accuracy while maintaining a high level of precision. Through experimentation, we demonstrate the effectiveness of employing FHE for template protection and matching within the encrypted domain, achieving notable results: a 96.41% True Acceptance Rate (TAR) for iris recognition, 81.19% TAR for face recognition, 98.81% TAR for iris fusion (left and right), and achieving a 100% TAR at 0.1% false acceptance rate (FAR) for face and iris fusion. The application of FHE presents a promising solution for ensuring accurate template matching while safeguarding user privacy and mitigating information leakage.

Title: Security Concerns in IoT Light Bulbs: Investigating Covert Channels

• Authors: Ravisha Rohilla, Janvi Panwar
• Subjects: cs.CR, cs.ET
• Abstract URL: https://arxiv.org/abs/2408.14613
• Pdf URL: https://arxiv.org/pdf/2408.14613
• Copy Paste: [[2408.14613]] Security Concerns in IoT Light Bulbs: Investigating Covert Channels(https://arxiv.org/abs/2408.14613)
• Keywords: security, attack, robust
• Abstract: The proliferation of Internet of Things (IoT) devices has raised significant concerns regarding their security vulnerabilities. This paper explores the security risks associated with smart light systems, focusing on covert communication channels. Drawing upon previous re-search highlighting vulnerabilities in communication protocols and en-cryption flaws, the study investigates the potential for exploiting smart light systems for covert data transmission. Specifically, the paper repli-cates and analyzes an attack method introduced by Ronen and Shamir, which utilizes the Philips Hue White lighting system to create a covert channel through visible light communication (VLC). Experimental re-sults demonstrate the feasibility of transmitting data covertly through subtle variations in brightness levels, leveraging the inherent functional-ity of smart light bulbs. Despite limit. ations imposed by device constraints and communication protocols, the study underscores the need for heightened awareness and security measures in IoT environment. Ultimately, the findings emphasize the importance of implementing robust security practices and exercising caution when deploying networked IoT devices in sensitive environment.

Title: What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation

• Authors: Dingyi Yang, Qin Jin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14622
• Pdf URL: https://arxiv.org/pdf/2408.14622
• Copy Paste: [[2408.14622]] What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation(https://arxiv.org/abs/2408.14622)
• Keywords: generative, large language model
• Abstract: With the development of artificial intelligence, particularly the success of Large Language Models (LLMs), the quantity and quality of automatically generated stories have significantly increased. This has led to the need for automatic story evaluation to assess the generative capabilities of computing systems and analyze the quality of both automatic-generated and human-written stories. Evaluating a story can be more challenging than other generation evaluation tasks. While tasks like machine translation primarily focus on assessing the aspects of fluency and accuracy, story evaluation demands complex additional measures such as overall coherence, character development, interestingness, etc. This requires a thorough review of relevant research. In this survey, we first summarize existing storytelling tasks, including text-to-text, visual-to-text, and text-to-visual. We highlight their evaluation challenges, identify various human criteria to measure stories, and present existing benchmark datasets. Then, we propose a taxonomy to organize evaluation metrics that have been developed or can be adopted for story evaluation. We also provide descriptions of these metrics, along with the discussion of their merits and limitations. Later, we discuss the human-AI collaboration for story evaluation and generation. Finally, we suggest potential future research directions, extending from story evaluation to general evaluations.

Title: Hybrid Deep Convolutional Neural Networks Combined with Autoencoders And Augmented Data To Predict The Look-Up Table 2006

• Authors: Messaoud Djeddou, Aouatef Hellal, Ibrahim A. Hameed, Xingang Zhao, Djehad Al Dallal
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14626
• Pdf URL: https://arxiv.org/pdf/2408.14626
• Copy Paste: [[2408.14626]] Hybrid Deep Convolutional Neural Networks Combined with Autoencoders And Augmented Data To Predict The Look-Up Table 2006(https://arxiv.org/abs/2408.14626)
• Keywords: robust
• Abstract: This study explores the development of a hybrid deep convolutional neural network (DCNN) model enhanced by autoencoders and data augmentation techniques to predict critical heat flux (CHF) with high accuracy. By augmenting the original input features using three different autoencoder configurations, the model's predictive capabilities were significantly improved. The hybrid models were trained and tested on a dataset of 7225 samples, with performance metrics including the coefficient of determination (R2), Nash-Sutcliffe efficiency (NSE), mean absolute error (MAE), and normalized root-mean-squared error (NRMSE) used for evaluation. Among the tested models, the DCNN_3F-A2 configuration demonstrated the highest accuracy, achieving an R2 of 0.9908 during training and 0.9826 during testing, outperforming the base model and other augmented versions. These results suggest that the proposed hybrid approach, combining deep learning with feature augmentation, offers a robust solution for CHF prediction, with the potential to generalize across a wider range of conditions.

Title: ParTEETor: A System for Partial Deployments of TEEs within Tor

• Authors: Rachel King, Quinn Burke, Yohan Beugin, Blaine Hoak, Kunyang Li, Eric Pauley, Ryan Sheatsley, Patrick McDaniel
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.14646
• Pdf URL: https://arxiv.org/pdf/2408.14646
• Copy Paste: [[2408.14646]] ParTEETor: A System for Partial Deployments of TEEs within Tor(https://arxiv.org/abs/2408.14646)
• Keywords: security, privacy, protect, attack
• Abstract: The Tor anonymity network allows users such as political activists and those under repressive governments to protect their privacy when communicating over the internet. At the same time, Tor has been demonstrated to be vulnerable to several classes of deanonymizing attacks that expose user behavior and identities. Prior work has shown that these threats can be mitigated by leveraging trusted execution environments (TEEs). However, previous proposals assume that all relays in the network will be TEE-based-which as a practical matter is unrealistic. In this work, we introduce ParTEETor, a Tor-variant system, which leverages partial deployments of TEEs to thwart known attacks. We study two modes of operation: non-policy and policy. Non-policy mode uses the existing Tor relay selection algorithm to provide users incident security. Policy mode extends the relay selection algorithm to address the classes of attacks by enforcing a specific TEE circuit configuration. We evaluate ParTEETor for security, performance, and privacy. Our evaluation demonstrates that at even a small TEE penetration (e.g., 10% of relays are TEE-based), users can reach performance of Tor today while enforcing a security policy to guarantee protection from at least two classes of attacks. Overall, we find that partial deployments of TEEs can substantially improve the security of Tor, without a significant impact on performance or privacy.

Title: Russian Cyber Onslaught was Blunted by Ukrainian Cyber Resilience, not Merely Security

• Authors: Alexander Kott, George (Yegor)Dubynskyi, Andrii Paziuk, Stephanie E. Galaitsi, Benjamin D. Trump, Igor Linkov
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.14667
• Pdf URL: https://arxiv.org/pdf/2408.14667
• Copy Paste: [[2408.14667]] Russian Cyber Onslaught was Blunted by Ukrainian Cyber Resilience, not Merely Security(https://arxiv.org/abs/2408.14667)
• Keywords: security, defense, attack, robust
• Abstract: Russian cyberattacks on Ukraine largely failed to produce meaningful outcomes not merely due to robust Ukrainian cyber defenses but were instead primarily a result of Ukraine's effective cyber resilience.

Title: Physically Feasible Semantic Segmentation

• Authors: Shamik Basu, Christos Sakaridis, Luc Van Gool
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14672
• Pdf URL: https://arxiv.org/pdf/2408.14672
• Copy Paste: [[2408.14672]] Physically Feasible Semantic Segmentation(https://arxiv.org/abs/2408.14672)
• Keywords: segmentation
• Abstract: State-of-the-art semantic segmentation models are typically optimized in a data-driven fashion, minimizing solely per-pixel classification objectives on their training data. This purely data-driven paradigm often leads to absurd segmentations, especially when the domain of input images is shifted from the one encountered during training. For instance, state-of-the-art models may assign the label ``road'' to a segment which is located above a segment that is respectively labeled as ``sky'', although our knowledge of the physical world dictates that such a configuration is not feasible for images captured by forward-facing upright cameras. Our method, Physically Feasible Semantic Segmentation (PhyFea), extracts explicit physical constraints that govern spatial class relations from the training sets of semantic segmentation datasets and enforces a differentiable loss function that penalizes violations of these constraints to promote prediction feasibility. PhyFea yields significant performance improvements in mIoU over each state-of-the-art network we use as baseline across ADE20K, Cityscapes and ACDC, notably a $1.5\%$ improvement on ADE20K and a $2.1\%$ improvement on ACDC.

Title: Enhancing Neural Network Interpretability Through Conductance-Based Information Plane Analysis

• Authors: Jaouad Dabounou, Amine Baazzouz
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14681
• Pdf URL: https://arxiv.org/pdf/2408.14681
• Copy Paste: [[2408.14681]] Enhancing Neural Network Interpretability Through Conductance-Based Information Plane Analysis(https://arxiv.org/abs/2408.14681)
• Keywords: robust, interpretability
• Abstract: The Information Plane is a conceptual framework used to analyze the flow of information in neural networks, but traditional methods based on activations may not fully capture the dynamics of information processing. This paper introduces a new approach that uses layer conductance, a measure of sensitivity to input features, to enhance the Information Plane analysis. By incorporating gradient-based contributions, we provide a more precise characterization of information dynamics within the network. The proposed conductance-based Information Plane and a new Information Transformation Efficiency (ITE) metric are evaluated on pretrained ResNet50 and VGG16 models using the ImageNet dataset. Our results demonstrate the ability to identify critical hidden layers that contribute significantly to model performance and interpretability, giving insights into information compression, preservation, and utilization across layers. The conductance-based approach offers a granular perspective on feature attribution, enhancing our understanding of the decision-making processes within neural networks. Furthermore, our empirical findings challenge certain theoretical predictions of the Information Bottleneck theory, highlighting the complexities of information dynamics in real-world data scenarios. The proposed method not only advances our understanding of information dynamics in neural networks but also has the potential to significantly impact the broader field of Artificial Intelligence by enabling the development of more interpretable, efficient, and robust models.

Title: Detecting Interpretable Subgroup Drifts

• Authors: Flavio Giobergia, Eliana Pastor, Luca de Alfaro, Elena Baralis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.14682
• Pdf URL: https://arxiv.org/pdf/2408.14682
• Copy Paste: [[2408.14682]] Detecting Interpretable Subgroup Drifts(https://arxiv.org/abs/2408.14682)
• Keywords: robust
• Abstract: The ability to detect and adapt to changes in data distributions is crucial to maintain the accuracy and reliability of machine learning models. Detection is generally approached by observing the drift of model performance from a global point of view. However, drifts occurring in (fine-grained) data subgroups may go unnoticed when monitoring global drift. We take a different perspective, and introduce methods for observing drift at the finer granularity of subgroups. Relevant data subgroups are identified during training and monitored efficiently throughout the model's life. Performance drifts in any subgroup are detected, quantified and characterized so as to provide an interpretable summary of the model behavior over time. Experimental results confirm that our subgroup-level drift analysis identifies drifts that do not show at the (coarser) global dataset level. The proposed approach provides a valuable tool for monitoring model performance in dynamic real-world applications, offering insights into the evolving nature of data and ultimately contributing to more robust and adaptive models.

Title: Training-Free Activation Sparsity in Large Language Models

• Authors: James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, Ben Athiwaratkun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14690
• Pdf URL: https://arxiv.org/pdf/2408.14690
• Copy Paste: [[2408.14690]] Training-Free Activation Sparsity in Large Language Models(https://arxiv.org/abs/2408.14690)
• Keywords: large language model
• Abstract: Activation sparsity can enable practical inference speedups in large language models (LLMs) by reducing the compute and memory-movement required for matrix multiplications during the forward pass. However, existing methods face limitations that inhibit widespread adoption. Some approaches are tailored towards older models with ReLU-based sparsity, while others require extensive continued pre-training on up to hundreds of billions of tokens. This paper describes TEAL, a simple training-free method that applies magnitude-based activation sparsity to hidden states throughout the entire model. TEAL achieves 40-50% model-wide sparsity with minimal performance degradation across Llama-2, Llama-3, and Mistral families, with sizes varying from 7B to 70B. We improve existing sparse kernels and demonstrate wall-clock decoding speed-ups of up to 1.53$\times$ and 1.8$\times$ at 40% and 50% model-wide sparsity. TEAL is compatible with weight quantization, enabling further efficiency gains.

Title: PAT: Pruning-Aware Tuning for Large Language Models

• Authors: Yijiang Liu, Huanrui Yang, Youxin Chen, Rongyu Zhang, Miao Wang, Yuan Du, Li Du
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14721
• Pdf URL: https://arxiv.org/pdf/2408.14721
• Copy Paste: [[2408.14721]] PAT: Pruning-Aware Tuning for Large Language Models(https://arxiv.org/abs/2408.14721)
• Keywords: robust, large language model
• Abstract: Large language models (LLMs) excel in language tasks, especially with supervised fine-tuning after pre-training. However, their substantial memory and computational requirements hinder practical applications. Structural pruning, which reduces less significant weight dimensions, is one solution. Yet, traditional post-hoc pruning often leads to significant performance loss, with limited recovery from further fine-tuning due to reduced capacity. Since the model fine-tuning refines the general and chaotic knowledge in pre-trained models, we aim to incorporate structural pruning with the fine-tuning, and propose the Pruning-Aware Tuning (PAT) paradigm to eliminate model redundancy while preserving the model performance to the maximum extend. Specifically, we insert the innovative Hybrid Sparsification Modules (HSMs) between the Attention and FFN components to accordingly sparsify the upstream and downstream linear modules. The HSM comprises a lightweight operator and a globally shared trainable mask. The lightweight operator maintains a training overhead comparable to that of LoRA, while the trainable mask unifies the channels to be sparsified, ensuring structural pruning. Additionally, we propose the Identity Loss which decouples the transformation and scaling properties of the HSMs to enhance training robustness. Extensive experiments demonstrate that PAT excels in both performance and efficiency. For example, our Llama2-7b model with a 25\% pruning ratio achieves 1.33$\times$ speedup while outperforming the LoRA-finetuned model by up to 1.26\% in accuracy with a similar training cost. Code: this https URL

Title: TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training

• Authors: Bongsoo Yi, Rongjie Lai, Yao Li
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.14728
• Pdf URL: https://arxiv.org/pdf/2408.14728
• Copy Paste: [[2408.14728]] TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training(https://arxiv.org/abs/2408.14728)
• Keywords: defense, attack, robust
• Abstract: Adversarial training has been shown to be successful in enhancing the robustness of deep neural networks against adversarial attacks. However, this robustness is accompanied by a significant decline in accuracy on clean data. In this paper, we propose a novel method, called Tangent Direction Guided Adversarial Training (TART), that leverages the tangent space of the data manifold to ameliorate the existing adversarial defense algorithms. We argue that training with adversarial examples having large normal components significantly alters the decision boundary and hurts accuracy. TART mitigates this issue by estimating the tangent direction of adversarial examples and allocating an adaptive perturbation limit according to the norm of their tangential component. To the best of our knowledge, our paper is the first work to consider the concept of tangent space and direction in the context of adversarial defense. We validate the effectiveness of TART through extensive experiments on both simulated and benchmark datasets. The results demonstrate that TART consistently boosts clean accuracy while retaining a high level of robustness against adversarial attacks. Our findings suggest that incorporating the geometric properties of data can lead to more effective and efficient adversarial training methods.

Title: OctFusion: Octree-based Diffusion Models for 3D Shape Generation

• Authors: Bojun Xiong, Si-Tong Wei, Xin-Yang Zheng, Yan-Pei Cao, Zhouhui Lian, Peng-Shuai Wang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.14732
• Pdf URL: https://arxiv.org/pdf/2408.14732
• Copy Paste: [[2408.14732]] OctFusion: Octree-based Diffusion Models for 3D Shape Generation(https://arxiv.org/abs/2408.14732)
• Keywords: diffusion
• Abstract: Diffusion models have emerged as a popular method for 3D generation. However, it is still challenging for diffusion models to efficiently generate diverse and high-quality 3D shapes. In this paper, we introduce OctFusion, which can generate 3D shapes with arbitrary resolutions in 2.5 seconds on a single Nvidia 4090 GPU, and the extracted meshes are guaranteed to be continuous and manifold. The key components of OctFusion are the octree-based latent representation and the accompanying diffusion models. The representation combines the benefits of both implicit neural representations and explicit spatial octrees and is learned with an octree-based variational autoencoder. The proposed diffusion model is a unified multi-scale U-Net that enables weights and computation sharing across different octree levels and avoids the complexity of widely used cascaded diffusion schemes. We verify the effectiveness of OctFusion on the ShapeNet and Objaverse datasets and achieve state-of-the-art performances on shape generation tasks. We demonstrate that OctFusion is extendable and flexible by generating high-quality color fields for textured mesh generation and high-quality 3D shapes conditioned on text prompts, sketches, or category labels. Our code and pre-trained models are available at \url{this https URL}.

Title: Learning Differentially Private Diffusion Models via Stochastic Adversarial Distillation

• Authors: Bochao Liu, Pengju Wang, Shiming Ge
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14738
• Pdf URL: https://arxiv.org/pdf/2408.14738
• Copy Paste: [[2408.14738]] Learning Differentially Private Diffusion Models via Stochastic Adversarial Distillation(https://arxiv.org/abs/2408.14738)
• Keywords: privacy, diffusion, generative
• Abstract: While the success of deep learning relies on large amounts of training datasets, data is often limited in privacy-sensitive domains. To address this challenge, generative model learning with differential privacy has emerged as a solution to train private generative models for desensitized data generation. However, the quality of the images generated by existing methods is limited due to the complexity of modeling data distribution. We build on the success of diffusion models and introduce DP-SAD, which trains a private diffusion model by a stochastic adversarial distillation method. Specifically, we first train a diffusion model as a teacher and then train a student by distillation, in which we achieve differential privacy by adding noise to the gradients from other models to the student. For better generation quality, we introduce a discriminator to distinguish whether an image is from the teacher or the student, which forms the adversarial training. Extensive experiments and analysis clearly demonstrate the effectiveness of our proposed method.

Title: RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models

• Authors: Junyao Ge, Yang Zheng, Kaitai Guo, Jimin Liang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14744
• Pdf URL: https://arxiv.org/pdf/2408.14744
• Copy Paste: [[2408.14744]] RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models(https://arxiv.org/abs/2408.14744)
• Keywords: large language model
• Abstract: Abundant, well-annotated multimodal data in remote sensing are pivotal for aligning complex visual remote sensing (RS) scenes with human language, enabling the development of specialized vision language models across diverse RS interpretation tasks. However, annotating RS images with rich linguistic semantics at scale demands expertise in RS and substantial human labor, making it costly and often impractical. In this study, we propose a workflow that leverages large language models (LLMs) to generate multimodal datasets with semantically rich captions at scale from plain OpenStreetMap (OSM) data for images sourced from the Google Earth Engine (GEE) platform. This approach facilitates the generation of paired remote sensing data and can be readily scaled up using openly available data. Within this framework, we present RSTeller, a multimodal dataset comprising over 1 million RS images, each accompanied by multiple descriptive captions. Extensive experiments demonstrate that RSTeller enhances the performance of multiple existing vision language models for RS scene understanding through continual pre-training. Our methodology significantly reduces the manual effort and expertise needed for annotating remote sensing imagery while democratizing access to high-quality annotated data. This advancement fosters progress in visual language modeling and encourages broader participation in remote sensing research and applications. The RSTeller dataset is available at this https URL.

Title: LyCon: Lyrics Reconstruction from the Bag-of-Words Using Large Language Models

• Authors: Haven Kim, Kahyun Choi
• Subjects: cs.CL, cs.DL
• Abstract URL: https://arxiv.org/abs/2408.14750
• Pdf URL: https://arxiv.org/pdf/2408.14750
• Copy Paste: [[2408.14750]] LyCon: Lyrics Reconstruction from the Bag-of-Words Using Large Language Models(https://arxiv.org/abs/2408.14750)
• Keywords: protect, large language model
• Abstract: This paper addresses the unique challenge of conducting research in lyric studies, where direct use of lyrics is often restricted due to copyright concerns. Unlike typical data, internet-sourced lyrics are frequently protected under copyright law, necessitating alternative approaches. Our study introduces a novel method for generating copyright-free lyrics from publicly available Bag-of-Words (BoW) datasets, which contain the vocabulary of lyrics but not the lyrics themselves. Utilizing metadata associated with BoW datasets and large language models, we successfully reconstructed lyrics. We have compiled and made available a dataset of reconstructed lyrics, LyCon, aligned with metadata from renowned sources including the Million Song Dataset, Deezer Mood Detection Dataset, and AllMusic Genre Dataset, available for public access. We believe that the integration of metadata such as mood annotations or genres enables a variety of academic experiments on lyrics, such as conditional lyric generation.

Title: Channel-wise Influence: Estimating Data Influence for Multivariate Time Series

• Authors: Muyao Wang, Zeke Xie, Bo Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.14763
• Pdf URL: https://arxiv.org/pdf/2408.14763
• Copy Paste: [[2408.14763]] Channel-wise Influence: Estimating Data Influence for Multivariate Time Series(https://arxiv.org/abs/2408.14763)
• Keywords: robust, interpretability
• Abstract: The influence function, a technique from robust statistics, measures the impact on model parameters or related functions when training data is removed or modified. This effective and valuable post-hoc method allows for studying the interpretability of machine learning models without requiring costly model retraining. It would provide extensions like increasing model performance, improving model generalization, and offering interpretability. Recently, Multivariate Time Series (MTS) analysis has become an important yet challenging task, attracting significant attention. However, there is no preceding research on the influence functions of MTS to shed light on the effects of modifying the channel of training MTS. Given that each channel in an MTS plays a crucial role in its analysis, it is essential to characterize the influence of different channels. To fill this gap, we propose a channel-wise influence function, which is the first method that can estimate the influence of different channels in MTS, utilizing a first-order gradient approximation that leverages the more informative average gradient of the data set. Additionally, we demonstrate how this influence function can be used to estimate the impact of a channel in MTS. Finally, we validated the accuracy and effectiveness of our influence estimation function in critical MTS analysis tasks, such as MTS anomaly detection and MTS forecasting. According to abundant experiments on real-world dataset, the original influence function performs worse than our method and even fail for the channel pruning problem, which demonstrate the superiority and necessity of channel-wise influence function in MTS analysis tasks.

Title: SynthDoc: Bilingual Documents Synthesis for Visual Document Understanding

• Authors: Chuanghao Ding, Xuejing Liu, Wei Tang, Juan Li, Xiaoliang Wang, Rui Zhao, Cam-Tu Nguyen, Fei Tan
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.14764
• Pdf URL: https://arxiv.org/pdf/2408.14764
• Copy Paste: [[2408.14764]] SynthDoc: Bilingual Documents Synthesis for Visual Document Understanding(https://arxiv.org/abs/2408.14764)
• Keywords: robust
• Abstract: This paper introduces SynthDoc, a novel synthetic document generation pipeline designed to enhance Visual Document Understanding (VDU) by generating high-quality, diverse datasets that include text, images, tables, and charts. Addressing the challenges of data acquisition and the limitations of existing datasets, SynthDoc leverages publicly available corpora and advanced rendering tools to create a comprehensive and versatile dataset. Our experiments, conducted using the Donut model, demonstrate that models trained with SynthDoc's data achieve superior performance in pre-training read tasks and maintain robustness in downstream tasks, despite language inconsistencies. The release of a benchmark dataset comprising 5,000 image-text pairs not only showcases the pipeline's capabilities but also provides a valuable resource for the VDU community to advance research and development in document image recognition. This work significantly contributes to the field by offering a scalable solution to data scarcity and by validating the efficacy of end-to-end models in parsing complex, real-world documents.

Title: CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View Synthesis

• Authors: Weijia Li, Jun He, Junyan Ye, Huaping Zhong, Zhimeng Zheng, Zilong Huang, Dahua Lin, Conghui He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14765
• Pdf URL: https://arxiv.org/pdf/2408.14765
• Copy Paste: [[2408.14765]] CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View Synthesis(https://arxiv.org/abs/2408.14765)
• Keywords: diffusion
• Abstract: Satellite-to-street view synthesis aims at generating a realistic street-view image from its corresponding satellite-view image. Although stable diffusion models have exhibit remarkable performance in a variety of image generation applications, their reliance on similar-view inputs to control the generated structure or texture restricts their application to the challenging cross-view synthesis task. In this work, we propose CrossViewDiff, a cross-view diffusion model for satellite-to-street view synthesis. To address the challenges posed by the large discrepancy across views, we design the satellite scene structure estimation and cross-view texture mapping modules to construct the structural and textural controls for street-view image synthesis. We further design a cross-view control guided denoising process that incorporates the above controls via an enhanced cross-view attention module. To achieve a more comprehensive evaluation of the synthesis results, we additionally design a GPT-based scoring method as a supplement to standard evaluation metrics. We also explore the effect of different data sources (e.g., text, maps, building heights, and multi-temporal satellite imagery) on this task. Results on three public cross-view datasets show that CrossViewDiff outperforms current state-of-the-art on both standard and GPT-based evaluation metrics, generating high-quality street-view panoramas with more realistic structures and textures across rural, suburban, and urban scenes. The code and models of this work will be released at this https URL.

Title: A global AI community requires language-diverse publishing

• Authors: Haley Lepp, Parth Sarin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14772
• Pdf URL: https://arxiv.org/pdf/2408.14772
• Copy Paste: [[2408.14772]] A global AI community requires language-diverse publishing(https://arxiv.org/abs/2408.14772)
• Keywords: extraction, large language model
• Abstract: In this provocation, we discuss the English dominance of the AI research community, arguing that the requirement for English language publishing upholds and reinforces broader regimes of extraction in AI. While large language models and machine translation have been celebrated as a way to break down barriers, we regard their use as a symptom of linguistic exclusion of scientists and potential readers. We propose alternative futures for a healthier publishing culture, organized around three themes: administering conferences in the languages of the country in which they are held, instructing peer reviewers not to adjudicate the language appropriateness of papers, and offering opportunities to publish and present in multiple languages. We welcome new translations of this piece. Please contact the authors if you would like to contribute one.

Title: Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning

• Authors: Simran Kaur, Simon Park, Anirudh Goyal, Sanjeev Arora
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14774
• Pdf URL: https://arxiv.org/pdf/2408.14774
• Copy Paste: [[2408.14774]] Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning(https://arxiv.org/abs/2408.14774)
• Keywords: extraction
• Abstract: We introduce Instruct-SkillMix, an automated approach for creating diverse, high quality SFT data. The Instruct-SkillMix pipeline involves two stages, each leveraging an existing powerful LLM: (1) Skill extraction: uses the LLM to extract core "skills" for instruction-following, either from existing datasets, or by directly prompting the model; (2) Data generation: uses the powerful LLM to generate (instruction, response) data that exhibit a randomly chosen pair of these skills. Here, the use of random skill combinations promotes diversity and difficulty. Vanilla SFT (i.e., no PPO, DPO, or RL methods) on data generated from Instruct-SkillMix leads to strong gains on instruction following benchmarks such as AlpacaEval 2.0, MT-Bench, and WildBench. With just $4$K examples, LLaMA-3-8B-Base achieves 42.76% length-controlled win rate on AlpacaEval 2.0. To our knowledge, this achieves state-of-the-art performance among all models that have only undergone SFT (no RL methods) and competes with proprietary models such as Claude 3 Opus and LLaMA-3.1-405B-Instruct. Ablation studies also suggest plausible reasons for why creating open instruction-tuning datasets via naive crowd-sourcing has proved difficult. Introducing low quality answers ("shirkers") in $20\%$ of Instruct-SkillMix examples causes performance to plummet, sometimes catastrophically. The Instruct-SkillMix pipeline is flexible and is adaptable to other settings.

Title: MROVSeg: Breaking the Resolution Curse of Vision-Language Models in Open-Vocabulary Semantic Segmentation

• Authors: Yuanbing Zhu, Bingke Zhu, Zhen Chen, Huan Xu, Ming Tang, Jinqiao Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14776
• Pdf URL: https://arxiv.org/pdf/2408.14776
• Copy Paste: [[2408.14776]] MROVSeg: Breaking the Resolution Curse of Vision-Language Models in Open-Vocabulary Semantic Segmentation(https://arxiv.org/abs/2408.14776)
• Keywords: segmentation
• Abstract: Open-vocabulary semantic segmentation aims to segment and recognize semantically meaningful regions based on text-based descriptions during inference. A typical solution to address this task is to leverage powerful vision-language models (VLMs), such as CLIP, to bridge the gap between open- and close-vocabulary recognition. As VLMs are usually pretrained with low-resolution images (e.g. $224\times224$), most previous methods operate only on downscaled images. We question this design as low resolution features often fail to preserve fine details. Although employing additional image backbones for high-resolution inputs can mitigate this issue, it may also introduce significant computation overhead. Therefore, we propose MROVSeg, a multi-resolution training framework for open-vocabulary semantic segmentation with a single pretrained CLIP backbone, that uses sliding windows to slice the high-resolution input into uniform patches, each matching the input size of the well-trained image encoder. Its key components include a Multi-Res Adapter, which restores the spatial geometry and grasps local-global correspondences across patches by learnable convolutional and scale attention layers. To achieve accurate segmentation, we introduce Multi-grained Masked Attention scheme to aggregate multi-grained semantics by performing cross-attention between object queries and multi-resolution CLIP features within the region of interests. Through comprehensive experiments, we demonstrate the superiority of MROVSeg on well-established open-vocabulary semantic segmentation benchmarks, particularly for high-resolution inputs, establishing new standards for open-vocabulary semantic segmentation.

Title: GINN-KAN: Interpretability pipelining with applications in Physics Informed Neural Networks

• Authors: Nisal Ranasinghe, Yu Xia, Sachith Seneviratne, Saman Halgamuge
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14780
• Pdf URL: https://arxiv.org/pdf/2408.14780
• Copy Paste: [[2408.14780]] GINN-KAN: Interpretability pipelining with applications in Physics Informed Neural Networks(https://arxiv.org/abs/2408.14780)
• Keywords: interpretability
• Abstract: Neural networks are powerful function approximators, yet their ``black-box" nature often renders them opaque and difficult to interpret. While many post-hoc explanation methods exist, they typically fail to capture the underlying reasoning processes of the networks. A truly interpretable neural network would be trained similarly to conventional models using techniques such as backpropagation, but additionally provide insights into the learned input-output relationships. In this work, we introduce the concept of interpretability pipelineing, to incorporate multiple interpretability techniques to outperform each individual technique. To this end, we first evaluate several architectures that promise such interpretability, with a particular focus on two recent models selected for their potential to incorporate interpretability into standard neural network architectures while still leveraging backpropagation: the Growing Interpretable Neural Network (GINN) and Kolmogorov Arnold Networks (KAN). We analyze the limitations and strengths of each and introduce a novel interpretable neural network GINN-KAN that synthesizes the advantages of both models. When tested on the Feynman symbolic regression benchmark datasets, GINN-KAN outperforms both GINN and KAN. To highlight the capabilities and the generalizability of this approach, we position GINN-KAN as an alternative to conventional black-box networks in Physics-Informed Neural Networks (PINNs). We expect this to have far-reaching implications in the application of deep learning pipelines in the natural sciences. Our experiments with this interpretable PINN on 15 different partial differential equations demonstrate that GINN-KAN augmented PINNs outperform PINNs with black-box networks in solving differential equations and surpass the capabilities of both GINN and KAN.

Title: Learning from Complementary Features

• Authors: Kosuke Sugiyama, Masato Uchida
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.14788
• Pdf URL: https://arxiv.org/pdf/2408.14788
• Copy Paste: [[2408.14788]] Learning from Complementary Features(https://arxiv.org/abs/2408.14788)
• Keywords: privacy
• Abstract: While precise data observation is essential for the learning processes of predictive models, it can be challenging owing to factors such as insufficient observation accuracy, high collection costs, and privacy constraints. In this paper, we examines cases where some qualitative features are unavailable as precise information indicating "what it is," but rather as complementary information indicating "what it is not." We refer to features defined by precise information as ordinary features (OFs) and those defined by complementary information as complementary features (CFs). We then formulate a new learning scenario termed Complementary Feature Learning (CFL), where predictive models are constructed using instances consisting of OFs and CFs. The simplest formalization of CFL applies conventional supervised learning directly using the observed values of CFs. However, this approach does not resolve the ambiguity associated with CFs, making learning challenging and complicating the interpretation of the predictive model's specific predictions. Therefore, we derive an objective function from an information-theoretic perspective to estimate the OF values corresponding to CFs and to predict output labels based on these estimations. Based on this objective function, we propose a theoretically guaranteed graph-based estimation method along with its practical approximation, for estimating OF values corresponding to CFs. The results of numerical experiments conducted with real-world data demonstrate that our proposed method effectively estimates OF values corresponding to CFs and predicts output labels.

Title: Revisiting Surgical Instrument Segmentation Without Human Intervention: A Graph Partitioning View

• Authors: Mingyu Sheng, Jianan Fan, Dongnan Liu, Ron Kikinis, Weidong Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14789
• Pdf URL: https://arxiv.org/pdf/2408.14789
• Copy Paste: [[2408.14789]] Revisiting Surgical Instrument Segmentation Without Human Intervention: A Graph Partitioning View(https://arxiv.org/abs/2408.14789)
• Keywords: robust, segmentation
• Abstract: Surgical instrument segmentation (SIS) on endoscopic images stands as a long-standing and essential task in the context of computer-assisted interventions for boosting minimally invasive surgery. Given the recent surge of deep learning methodologies and their data-hungry nature, training a neural predictive model based on massive expert-curated annotations has been dominating and served as an off-the-shelf approach in the field, which could, however, impose prohibitive burden to clinicians for preparing fine-grained pixel-wise labels corresponding to the collected surgical video frames. In this work, we propose an unsupervised method by reframing the video frame segmentation as a graph partitioning problem and regarding image pixels as graph nodes, which is significantly different from the previous efforts. A self-supervised pre-trained model is firstly leveraged as a feature extractor to capture high-level semantic features. Then, Laplacian matrixs are computed from the features and are eigendecomposed for graph partitioning. On the "deep" eigenvectors, a surgical video frame is meaningfully segmented into different modules such as tools and tissues, providing distinguishable semantic information like locations, classes, and relations. The segmentation problem can then be naturally tackled by applying clustering or threshold on the eigenvectors. Extensive experiments are conducted on various datasets (e.g., EndoVis2017, EndoVis2018, UCL, etc.) for different clinical endpoints. Across all the challenging scenarios, our method demonstrates outstanding performance and robustness higher than unsupervised state-of-the-art (SOTA) methods. The code is released at this https URL.

Title: GSIFN: A Graph-Structured and Interlaced-Masked Multimodal Transformer Based Fusion Network for Multimodal Sentiment Analysis

• Authors: Yijie Jin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14809
• Pdf URL: https://arxiv.org/pdf/2408.14809
• Copy Paste: [[2408.14809]] GSIFN: A Graph-Structured and Interlaced-Masked Multimodal Transformer Based Fusion Network for Multimodal Sentiment Analysis(https://arxiv.org/abs/2408.14809)
• Keywords: robust, transformer
• Abstract: Multimodal Sentiment Analysis (MSA) leverages multiple modals to analyze sentiments. Typically, advanced fusion methods and representation learning-based methods are designed to tackle it. Our proposed GSIFN solves two key problems to be solved in MSA: (i) In multimodal fusion, the decoupling of modal combinations and tremendous parameter redundancy in existing fusion methods, which lead to poor fusion performance and efficiency. (ii) The trade-off between representation capability and computation overhead of the unimodal feature extractors and enhancers. GSIFN incorporates two main components to solve these problems: (i) Graph-Structured and Interlaced-Masked Multimodal Transformer. It adopts the Interlaced Mask mechanism to construct robust multimodal graph embedding, achieve all-modal-in-one Transformer-based fusion, and greatly reduce the computation overhead. (ii) A self-supervised learning framework with low computation overhead and high performance, which utilizes a parallelized LSTM with matrix memory to enhance non-verbal modal feature for unimodal label generation. Evaluated on the MSA datasets CMU-MOSI, CMU-MOSEI, and CH-SIMS, GSIFN demonstrates superior performance with significantly lower computation overhead compared with state-of-the-art methods.

Title: HPT++: Hierarchically Prompting Vision-Language Models with Multi-Granularity Knowledge Generation and Improved Structure Modeling

• Authors: Yubin Wang, Xinyang Jiang, De Cheng, Wenli Sun, Dongsheng Li, Cairong Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14812
• Pdf URL: https://arxiv.org/pdf/2408.14812
• Copy Paste: [[2408.14812]] HPT++: Hierarchically Prompting Vision-Language Models with Multi-Granularity Knowledge Generation and Improved Structure Modeling(https://arxiv.org/abs/2408.14812)
• Keywords: large language model
• Abstract: Prompt learning has become a prevalent strategy for adapting vision-language foundation models (VLMs) such as CLIP to downstream tasks. With the emergence of large language models (LLMs), recent studies have explored the potential of using category-related descriptions to enhance prompt effectiveness. However, conventional descriptions lack explicit structured information necessary to represent the interconnections among key elements like entities or attributes with relation to a particular category. Since existing prompt tuning methods give little consideration to managing structured knowledge, this paper advocates leveraging LLMs to construct a graph for each description to prioritize such structured knowledge. Consequently, we propose a novel approach called Hierarchical Prompt Tuning (HPT), enabling simultaneous modeling of both structured and conventional linguistic knowledge. Specifically, we introduce a relationship-guided attention module to capture pair-wise associations among entities and attributes for low-level prompt learning. In addition, by incorporating high-level and global-level prompts modeling overall semantics, the proposed hierarchical structure forges cross-level interlinks and empowers the model to handle more complex and long-term relationships. Finally, by enhancing multi-granularity knowledge generation, redesigning the relationship-driven attention re-weighting module, and incorporating consistent constraints on the hierarchical text encoder, we propose HPT++, which further improves the performance of HPT. Our experiments are conducted across a wide range of evaluation settings, including base-to-new generalization, cross-dataset evaluation, and domain generalization. Extensive results and ablation studies demonstrate the effectiveness of our methods, which consistently outperform existing SOTA methods.

Title: Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation

• Authors: Abdelrahman Eldesokey, Peter Wonka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14819
• Pdf URL: https://arxiv.org/pdf/2408.14819
• Copy Paste: [[2408.14819]] Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation(https://arxiv.org/abs/2408.14819)
• Keywords: diffusion
• Abstract: We propose a diffusion-based approach for Text-to-Image (T2I) generation with interactive 3D layout control. Layout control has been widely studied to alleviate the shortcomings of T2I diffusion models in understanding objects' placement and relationships from text descriptions. Nevertheless, existing approaches for layout control are limited to 2D layouts, require the user to provide a static layout beforehand, and fail to preserve generated images under layout changes. This makes these approaches unsuitable for applications that require 3D object-wise control and iterative refinements, e.g., interior design and complex scene generation. To this end, we leverage the recent advancements in depth-conditioned T2I models and propose a novel approach for interactive 3D layout control. We replace the traditional 2D boxes used in layout control with 3D boxes. Furthermore, we revamp the T2I task as a multi-stage generation process, where at each stage, the user can insert, change, and move an object in 3D while preserving objects from earlier stages. We achieve this through our proposed Dynamic Self-Attention (DSA) module and the consistent 3D object translation strategy. Experiments show that our approach can generate complicated scenes based on 3D layouts, boosting the object generation success rate over the standard depth-conditioned T2I methods by 2x. Moreover, it outperforms other methods in comparison in preserving objects under layout changes. Project Page: \url{this https URL}

Title: Data-driven Effective Modeling of Multiscale Stochastic Dynamical Systems

• Authors: Yuan Chen, Dongbin Xiu
• Subjects: cs.LG, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.14821
• Pdf URL: https://arxiv.org/pdf/2408.14821
• Copy Paste: [[2408.14821]] Data-driven Effective Modeling of Multiscale Stochastic Dynamical Systems(https://arxiv.org/abs/2408.14821)
• Keywords: generative
• Abstract: We present a numerical method for learning the dynamics of slow components of unknown multiscale stochastic dynamical systems. While the governing equations of the systems are unknown, bursts of observation data of the slow variables are available. By utilizing the observation data, our proposed method is capable of constructing a generative stochastic model that can accurately capture the effective dynamics of the slow variables in distribution. We present a comprehensive set of numerical examples to demonstrate the performance of the proposed method.

Title: Alfie: Democratising RGBA Image Generation With No $$$

• Authors: Fabio Quattrini, Vittorio Pippi, Silvia Cascianelli, Rita Cucchiara
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.14826
• Pdf URL: https://arxiv.org/pdf/2408.14826
• Copy Paste: [[2408.14826]] Alfie: Democratising RGBA Image Generation With No $$$(https://arxiv.org/abs/2408.14826)
• Keywords: diffusion, transformer
• Abstract: Designs and artworks are ubiquitous across various creative fields, requiring graphic design skills and dedicated software to create compositions that include many graphical elements, such as logos, icons, symbols, and art scenes, which are integral to visual storytelling. Automating the generation of such visual elements improves graphic designers' productivity, democratizes and innovates the creative industry, and helps generate more realistic synthetic data for related tasks. These illustration elements are mostly RGBA images with irregular shapes and cutouts, facilitating blending and scene composition. However, most image generation models are incapable of generating such images and achieving this capability requires expensive computational resources, specific training recipes, or post-processing solutions. In this work, we propose a fully-automated approach for obtaining RGBA illustrations by modifying the inference-time behavior of a pre-trained Diffusion Transformer model, exploiting the prompt-guided controllability and visual quality offered by such models with no additional computational cost. We force the generation of entire subjects without sharp croppings, whose background is easily removed for seamless integration into design projects or artistic scenes. We show with a user study that, in most cases, users prefer our solution over generating and then matting an image, and we show that our generated illustrations yield good results when used as inputs for composite scene generation pipelines. We release the code at this https URL.

Title: Time-Aware Face Anti-Spoofing with Rotation Invariant Local Binary Patterns and Deep Learning

• Authors: Moritz Finke, Alexandra Dmitrienko
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14829
• Pdf URL: https://arxiv.org/pdf/2408.14829
• Copy Paste: [[2408.14829]] Time-Aware Face Anti-Spoofing with Rotation Invariant Local Binary Patterns and Deep Learning(https://arxiv.org/abs/2408.14829)
• Keywords: attack
• Abstract: Facial recognition systems have become an integral part of the modern world. These methods accomplish the task of human identification in an automatic, fast, and non-interfering way. Past research has uncovered high vulnerability to simple imitation attacks that could lead to erroneous identification and subsequent authentication of attackers. Similar to face recognition, imitation attacks can also be detected with Machine Learning. Attack detection systems use a variety of facial features and advanced machine learning models for uncovering the presence of attacks. In this work, we assess existing work on liveness detection and propose a novel approach that promises high classification accuracy by combining previously unused features with time-aware deep learning strategies.

Title: PolicyLR: A Logic Representation For Privacy Policies

• Authors: Ashish Hooda, Rishabh Khandelwal, Prasad Chalasani, Kassem Fawaz, Somesh Jha
• Subjects: cs.CR, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14830
• Pdf URL: https://arxiv.org/pdf/2408.14830
• Copy Paste: [[2408.14830]] PolicyLR: A Logic Representation For Privacy Policies(https://arxiv.org/abs/2408.14830)
• Keywords: privacy, large language model
• Abstract: Privacy policies are crucial in the online ecosystem, defining how services handle user data and adhere to regulations such as GDPR and CCPA. However, their complexity and frequent updates often make them difficult for stakeholders to understand and analyze. Current automated analysis methods, which utilize natural language processing, have limitations. They typically focus on individual tasks and fail to capture the full context of the policies. We propose PolicyLR, a new paradigm that offers a comprehensive machine-readable representation of privacy policies, serving as an all-in-one solution for multiple downstream tasks. PolicyLR converts privacy policies into a machine-readable format using valuations of atomic formulae, allowing for formal definitions of tasks like compliance and consistency. We have developed a compiler that transforms unstructured policy text into this format using off-the-shelf Large Language Models (LLMs). This compiler breaks down the transformation task into a two-stage translation and entailment procedure. This procedure considers the full context of the privacy policy to infer a complex formula, where each formula consists of simpler atomic formulae. The advantage of this model is that PolicyLR is interpretable by design and grounded in segments of the privacy policy. We evaluated the compiler using ToS;DR, a community-annotated privacy policy entailment dataset. Utilizing open-source LLMs, our compiler achieves precision and recall values of 0.91 and 0.88, respectively. Finally, we demonstrate the utility of PolicyLR in three privacy tasks: Policy Compliance, Inconsistency Detection, and Privacy Comparison Shopping.

Title: DRL-Based Federated Self-Supervised Learning for Task Offloading and Resource Allocation in ISAC-Enabled Vehicle Edge Computing

• Authors: Xueying Gu, Qiong Wu, Pingyi Fan, Nan Cheng, Wen Chen, Khaled B. Letaief
• Subjects: cs.LG, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2408.14831
• Pdf URL: https://arxiv.org/pdf/2408.14831
• Copy Paste: [[2408.14831]] DRL-Based Federated Self-Supervised Learning for Task Offloading and Resource Allocation in ISAC-Enabled Vehicle Edge Computing(https://arxiv.org/abs/2408.14831)
• Keywords: federate
• Abstract: Intelligent Transportation Systems (ITS) leverage Integrated Sensing and Communications (ISAC) to enhance data exchange between vehicles and infrastructure in the Internet of Vehicles (IoV). This integration inevitably increases computing demands, risking real-time system stability. Vehicle Edge Computing (VEC) addresses this by offloading tasks to Road Side Unit (RSU), ensuring timely services. Our previous work FLSimCo algorithm, which uses local resources for Federated Self-Supervised Learning (SSL), though vehicles often can't complete all iterations task. Our improved algorithm offloads partial task to RSU and optimizes energy consumption by adjusting transmission power, CPU frequency, and task assignment ratios, balancing local and RSU-based training. Meanwhile, setting an offloading threshold further prevents inefficiencies. Simulation results show that the enhanced algorithm reduces energy consumption, improves offloading efficiency and the accuracy of Federated SSL.

Title: Diffusion Models Are Real-Time Game Engines

• Authors: Dani Valevski, Yaniv Leviathan, Moab Arar, Shlomi Fruchter
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14837
• Pdf URL: https://arxiv.org/pdf/2408.14837
• Copy Paste: [[2408.14837]] Diffusion Models Are Real-Time Game Engines(https://arxiv.org/abs/2408.14837)
• Keywords: diffusion
• Abstract: We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories at high quality. GameNGen can interactively simulate the classic game DOOM at over 20 frames per second on a single TPU. Next frame prediction achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation. GameNGen is trained in two phases: (1) an RL-agent learns to play the game and the training sessions are recorded, and (2) a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions. Conditioning augmentations enable stable auto-regressive generation over long trajectories.

Title: Diffusion based Semantic Outlier Generation via Nuisance Awareness for Out-of-Distribution Detection

• Authors: Suhee Yoon, Sanghyu Yoon, Hankook Lee, Ye Seul Sim, Sungik Choi, Kyungeun Lee, Hye-Seung Cho, Woohyung Lim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14841
• Pdf URL: https://arxiv.org/pdf/2408.14841
• Copy Paste: [[2408.14841]] Diffusion based Semantic Outlier Generation via Nuisance Awareness for Out-of-Distribution Detection(https://arxiv.org/abs/2408.14841)
• Keywords: diffusion
• Abstract: Out-of-distribution (OOD) detection, which determines whether a given sample is part of the in-distribution (ID), has recently shown promising results through training with synthetic OOD datasets. Nonetheless, existing methods often produce outliers that are considerably distant from the ID, showing limited efficacy for capturing subtle distinctions between ID and OOD. To address these issues, we propose a novel framework, Semantic Outlier generation via Nuisance Awareness (SONA), which notably produces challenging outliers by directly leveraging pixel-space ID samples through diffusion models. Our approach incorporates SONA guidance, providing separate control over semantic and nuisance regions of ID samples. Thereby, the generated outliers achieve two crucial properties: (i) they present explicit semantic-discrepant information, while (ii) maintaining various levels of nuisance resemblance with ID. Furthermore, the improved OOD detector training with SONA outliers facilitates learning with a focus on semantic distinctions. Extensive experiments demonstrate the effectiveness of our framework, achieving an impressive AUROC of 88% on near-OOD datasets, which surpasses the performance of baseline methods by a significant margin of approximately 6%.

Title: From Bias to Balance: Detecting Facial Expression Recognition Biases in Large Multimodal Foundation Models

• Authors: Kaylee Chhua, Zhoujinyi Wen, Vedant Hathalia, Kevin Zhu, Sean O'Brien
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.14842
• Pdf URL: https://arxiv.org/pdf/2408.14842
• Copy Paste: [[2408.14842]] From Bias to Balance: Detecting Facial Expression Recognition Biases in Large Multimodal Foundation Models(https://arxiv.org/abs/2408.14842)
• Keywords: fair
• Abstract: This study addresses the racial biases in facial expression recognition (FER) systems within Large Multimodal Foundation Models (LMFMs). Despite advances in deep learning and the availability of diverse datasets, FER systems often exhibit higher error rates for individuals with darker skin tones. Existing research predominantly focuses on traditional FER models (CNNs, RNNs, ViTs), leaving a gap in understanding racial biases in LMFMs. We benchmark four leading LMFMs: GPT-4o, PaliGemma, Gemini, and CLIP to assess their performance in facial emotion detection across different racial demographics. A linear classifier trained on CLIP embeddings obtains accuracies of 95.9\% for RADIATE, 90.3\% for Tarr, and 99.5\% for Chicago Face. Furthermore, we identify that Anger is misclassified as Disgust 2.1 times more often in Black Females than White Females. This study highlights the need for fairer FER systems and establishes a foundation for developing unbiased, accurate FER technologies. Visit this https URL for further information regarding the biases within facial expression recognition.

Title: Correntropy-Based Improper Likelihood Model for Robust Electrophysiological Source Imaging

• Authors: Yuanhao Li, Badong Chen, Zhongxu Hu, Keita Suzuki, Wenjun Bai, Yasuharu Koike, Okito Yamashita
• Subjects: cs.LG, cs.NE, eess.SP
• Abstract URL: https://arxiv.org/abs/2408.14843
• Pdf URL: https://arxiv.org/pdf/2408.14843
• Copy Paste: [[2408.14843]] Correntropy-Based Improper Likelihood Model for Robust Electrophysiological Source Imaging(https://arxiv.org/abs/2408.14843)
• Keywords: robust
• Abstract: Bayesian learning provides a unified skeleton to solve the electrophysiological source imaging task. From this perspective, existing source imaging algorithms utilize the Gaussian assumption for the observation noise to build the likelihood function for Bayesian inference. However, the electromagnetic measurements of brain activity are usually affected by miscellaneous artifacts, leading to a potentially non-Gaussian distribution for the observation noise. Hence the conventional Gaussian likelihood model is a suboptimal choice for the real-world source imaging task. In this study, we aim to solve this problem by proposing a new likelihood model which is robust with respect to non-Gaussian noises. Motivated by the robust maximum correntropy criterion, we propose a new improper distribution model concerning the noise assumption. This new noise distribution is leveraged to structure a robust likelihood function and integrated with hierarchical prior distributions to estimate source activities by variational inference. In particular, the score matching is adopted to determine the hyperparameters for the improper likelihood model. A comprehensive performance evaluation is performed to compare the proposed noise assumption to the conventional Gaussian model. Simulation results show that, the proposed method can realize more precise source reconstruction by designing known ground-truth. The real-world dataset also demonstrates the superiority of our new method with the visual perception task. This study provides a new backbone for Bayesian source imaging, which would facilitate its application using real-world noisy brain signal.

Title: AAVENUE: Detecting LLM Biases on NLU Tasks in AAVE via a Novel Benchmark

• Authors: Abhay Gupta, Philip Meng, Ece Yurtseven, Sean O'Brien, Kevin Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14845
• Pdf URL: https://arxiv.org/pdf/2408.14845
• Copy Paste: [[2408.14845]] AAVENUE: Detecting LLM Biases on NLU Tasks in AAVE via a Novel Benchmark(https://arxiv.org/abs/2408.14845)
• Keywords: large language model
• Abstract: Detecting biases in natural language understanding (NLU) for African American Vernacular English (AAVE) is crucial to developing inclusive natural language processing (NLP) systems. To address dialect-induced performance discrepancies, we introduce AAVENUE ({AAVE} {N}atural Language {U}nderstanding {E}valuation), a benchmark for evaluating large language model (LLM) performance on NLU tasks in AAVE and Standard American English (SAE). AAVENUE builds upon and extends existing benchmarks like VALUE, replacing deterministic syntactic and morphological transformations with a more flexible methodology leveraging LLM-based translation with few-shot prompting, improving performance across our evaluation metrics when translating key tasks from the GLUE and SuperGLUE benchmarks. We compare AAVENUE and VALUE translations using five popular LLMs and a comprehensive set of metrics including fluency, BARTScore, quality, coherence, and understandability. Additionally, we recruit fluent AAVE speakers to validate our translations for authenticity. Our evaluations reveal that LLMs consistently perform better on SAE tasks than AAVE-translated versions, underscoring inherent biases and highlighting the need for more inclusive NLP models. We have open-sourced our source code on GitHub and created a website to showcase our work at https://aavenue.live.

Title: Diffusion-Occ: 3D Point Cloud Completion via Occupancy Diffusion

• Authors: Guoqing Zhang, Jian Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14846
• Pdf URL: https://arxiv.org/pdf/2408.14846
• Copy Paste: [[2408.14846]] Diffusion-Occ: 3D Point Cloud Completion via Occupancy Diffusion(https://arxiv.org/abs/2408.14846)
• Keywords: extraction, diffusion, transformer, generative
• Abstract: Point clouds are crucial for capturing three-dimensional data but often suffer from incompleteness due to limitations such as resolution and occlusion. Traditional methods typically rely on point-based approaches within discriminative frameworks for point cloud completion. In this paper, we introduce \textbf{Diffusion-Occ}, a novel framework for Diffusion Point Cloud Completion. Diffusion-Occ utilizes a two-stage coarse-to-fine approach. In the first stage, the Coarse Density Voxel Prediction Network (CDNet) processes partial points to predict coarse density voxels, streamlining global feature extraction through voxel classification, as opposed to previous regression-based methods. In the second stage, we introduce the Occupancy Generation Network (OccGen), a conditional occupancy diffusion model based on a transformer architecture and enhanced by our Point-Voxel Fuse (PVF) block. This block integrates coarse density voxels with partial points to leverage both global and local features for comprehensive completion. By thresholding the occupancy field, we convert it into a complete point cloud. Additionally, our method employs diverse training mixtures and efficient diffusion parameterization to enable effective one-step sampling during both training and inference. Experimental results demonstrate that Diffusion-Occ outperforms existing discriminative and generative methods.

Title: Detecting AI Flaws: Target-Driven Attacks on Internal Faults in Language Models

• Authors: Yuhao Du, Zhuo Li, Pengyu Cheng, Xiang Wan, Anningzhe Gao
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.14853
• Pdf URL: https://arxiv.org/pdf/2408.14853
• Copy Paste: [[2408.14853]] Detecting AI Flaws: Target-Driven Attacks on Internal Faults in Language Models(https://arxiv.org/abs/2408.14853)
• Keywords: security, attack, large language model
• Abstract: Large Language Models (LLMs) have become a focal point in the rapidly evolving field of artificial intelligence. However, a critical concern is the presence of toxic content within the pre-training corpus of these models, which can lead to the generation of inappropriate outputs. Investigating methods for detecting internal faults in LLMs can help us understand their limitations and improve their security. Existing methods primarily focus on jailbreaking attacks, which involve manually or automatically constructing adversarial content to prompt the target LLM to generate unexpected responses. These methods rely heavily on prompt engineering, which is time-consuming and usually requires specially designed questions. To address these challenges, this paper proposes a target-driven attack paradigm that focuses on directly eliciting the target response instead of optimizing the prompts. We introduce the use of another LLM as the detector for toxic content, referred to as ToxDet. Given a target toxic response, ToxDet can generate a possible question and a preliminary answer to provoke the target model into producing desired toxic responses with meanings equivalent to the provided one. ToxDet is trained by interacting with the target LLM and receiving reward signals from it, utilizing reinforcement learning for the optimization process. While the primary focus of the target models is on open-source LLMs, the fine-tuned ToxDet can also be transferred to attack black-box models such as GPT-4o, achieving notable results. Experimental results on AdvBench and HH-Harmless datasets demonstrate the effectiveness of our methods in detecting the tendencies of target LLMs to generate harmful responses. This algorithm not only exposes vulnerabilities but also provides a valuable resource for researchers to strengthen their models against such attacks.

Title: DiffSurf: A Transformer-based Diffusion Model for Generating and Reconstructing 3D Surfaces in Pose

• Authors: Yusuke Yoshiyasu, Leyuan Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14860
• Pdf URL: https://arxiv.org/pdf/2408.14860
• Copy Paste: [[2408.14860]] DiffSurf: A Transformer-based Diffusion Model for Generating and Reconstructing 3D Surfaces in Pose(https://arxiv.org/abs/2408.14860)
• Keywords: diffusion, transformer, generative
• Abstract: This paper presents DiffSurf, a transformer-based denoising diffusion model for generating and reconstructing 3D surfaces. Specifically, we design a diffusion transformer architecture that predicts noise from noisy 3D surface vertices and normals. With this architecture, DiffSurf is able to generate 3D surfaces in various poses and shapes, such as human bodies, hands, animals and man-made objects. Further, DiffSurf is versatile in that it can address various 3D downstream tasks including morphing, body shape variation and 3D human mesh fitting to 2D keypoints. Experimental results on 3D human model benchmarks demonstrate that DiffSurf can generate shapes with greater diversity and higher quality than previous generative models. Furthermore, when applied to the task of single-image 3D human mesh recovery, DiffSurf achieves accuracy comparable to prior techniques at a near real-time rate.

Title: Advancing Adversarial Suffix Transfer Learning on Aligned Large Language Models

• Authors: Hongfu Liu, Yuxi Xie, Ye Wang, Michael Shieh
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.14866
• Pdf URL: https://arxiv.org/pdf/2408.14866
• Copy Paste: [[2408.14866]] Advancing Adversarial Suffix Transfer Learning on Aligned Large Language Models(https://arxiv.org/abs/2408.14866)
• Keywords: large language model
• Abstract: Language Language Models (LLMs) face safety concerns due to potential misuse by malicious users. Recent red-teaming efforts have identified adversarial suffixes capable of jailbreaking LLMs using the gradient-based search algorithm Greedy Coordinate Gradient (GCG). However, GCG struggles with computational inefficiency, limiting further investigations regarding suffix transferability and scalability across models and data. In this work, we bridge the connection between search efficiency and suffix transferability. We propose a two-stage transfer learning framework, DeGCG, which decouples the search process into behavior-agnostic pre-searching and behavior-relevant post-searching. Specifically, we employ direct first target token optimization in pre-searching to facilitate the search process. We apply our approach to cross-model, cross-data, and self-transfer scenarios. Furthermore, we introduce an interleaved variant of our approach, i-DeGCG, which iteratively leverages self-transferability to accelerate the search process. Experiments on HarmBench demonstrate the efficiency of our approach across various models and domains. Notably, our i-DeGCG outperforms the baseline on Llama2-chat-7b with ASRs of $43.9$ ($+22.2$) and $39.0$ ($+19.5$) on valid and test sets, respectively. Further analysis on cross-model transfer indicates the pivotal role of first target token optimization in leveraging suffix transferability for efficient searching.

Title: ZeroMamba: Exploring Visual State Space Model for Zero-Shot Learning

• Authors: Wenjin Hou, Dingjie Fu, Kun Li, Shiming Chen, Hehe Fan, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14868
• Pdf URL: https://arxiv.org/pdf/2408.14868
• Copy Paste: [[2408.14868]] ZeroMamba: Exploring Visual State Space Model for Zero-Shot Learning(https://arxiv.org/abs/2408.14868)
• Keywords: transformer
• Abstract: Zero-shot learning (ZSL) aims to recognize unseen classes by transferring semantic knowledge from seen classes to unseen ones, guided by semantic information. To this end, existing works have demonstrated remarkable performance by utilizing global visual features from Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs) for visual-semantic interactions. Due to the limited receptive fields of CNNs and the quadratic complexity of ViTs, however, these visual backbones achieve suboptimal visual-semantic interactions. In this paper, motivated by the visual state space model (i.e., Vision Mamba), which is capable of capturing long-range dependencies and modeling complex visual dynamics, we propose a parameter-efficient ZSL framework called ZeroMamba to advance ZSL. Our ZeroMamba comprises three key components: Semantic-aware Local Projection (SLP), Global Representation Learning (GRL), and Semantic Fusion (SeF). Specifically, SLP integrates semantic embeddings to map visual features to local semantic-related representations, while GRL encourages the model to learn global semantic representations. SeF combines these two semantic representations to enhance the discriminability of semantic features. We incorporate these designs into Vision Mamba, forming an end-to-end ZSL framework. As a result, the learned semantic representations are better suited for classification. Through extensive experiments on four prominent ZSL benchmarks, ZeroMamba demonstrates superior performance, significantly outperforming the state-of-the-art (i.e., CNN-based and ViT-based) methods under both conventional ZSL (CZSL) and generalized ZSL (GZSL) settings. Code is available at: https://anonymous.4open.science/r/ZeroMamba.

Title: Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data

• Authors: Han Xia, Songyang Gao, Qiming Ge, Zhiheng Xi, Qi Zhang, Xuanjing Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14874
• Pdf URL: https://arxiv.org/pdf/2408.14874
• Copy Paste: [[2408.14874]] Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data(https://arxiv.org/abs/2408.14874)
• Keywords: robust, large language model
• Abstract: Reinforcement Learning from Human Feedback (RLHF) has proven effective in aligning large language models with human intentions, yet it often relies on complex methodologies like Proximal Policy Optimization (PPO) that require extensive hyper-parameter tuning and present challenges in sample efficiency and stability. In this paper, we introduce Inverse-Q*, an innovative framework that transcends traditional RL methods by optimizing token-level reinforcement learning without the need for additional reward or value models. Inverse-Q* leverages direct preference optimization techniques but extends them by estimating the conditionally optimal policy directly from the model's responses, facilitating more granular and flexible policy shaping. Our approach reduces reliance on human annotation and external supervision, making it especially suitable for low-resource settings. We present extensive experimental results demonstrating that Inverse-Q* not only matches but potentially exceeds the effectiveness of PPO in terms of convergence speed and the alignment of model responses with human preferences. Our findings suggest that Inverse-Q* offers a practical and robust alternative to conventional RLHF approaches, paving the way for more efficient and adaptable model training approaches.

Title: Adversarial Attacks and Defenses in Multivariate Time-Series Forecasting for Smart and Connected Infrastructures

• Authors: Pooja Krishan, Rohan Mohapatra, Saptarshi Sengupta
• Subjects: cs.LG, cs.AI, cs.CR, cs.PF
• Abstract URL: https://arxiv.org/abs/2408.14875
• Pdf URL: https://arxiv.org/pdf/2408.14875
• Copy Paste: [[2408.14875]] Adversarial Attacks and Defenses in Multivariate Time-Series Forecasting for Smart and Connected Infrastructures(https://arxiv.org/abs/2408.14875)
• Keywords: security, defense, attack, robust
• Abstract: The emergence of deep learning models has revolutionized various industries over the last decade, leading to a surge in connected devices and infrastructures. However, these models can be tricked into making incorrect predictions with high confidence, leading to disastrous failures and security concerns. To this end, we explore the impact of adversarial attacks on multivariate time-series forecasting and investigate methods to counter them. Specifically, we employ untargeted white-box attacks, namely the Fast Gradient Sign Method (FGSM) and the Basic Iterative Method (BIM), to poison the inputs to the training process, effectively misleading the model. We also illustrate the subtle modifications to the inputs after the attack, which makes detecting the attack using the naked eye quite difficult. Having demonstrated the feasibility of these attacks, we develop robust models through adversarial training and model hardening. We are among the first to showcase the transferability of these attacks and defenses by extrapolating our work from the benchmark electricity data to a larger, 10-year real-world data used for predicting the time-to-failure of hard disks. Our experimental results confirm that the attacks and defenses achieve the desired security thresholds, leading to a 72.41% and 94.81% decrease in RMSE for the electricity and hard disk datasets respectively after implementing the adversarial defenses.

Title: Adversarial Manhole: Challenging Monocular Depth Estimation and Semantic Segmentation Models with Patch Attack

• Authors: Naufal Suryanto, Andro Aprila Adiputra, Ahmada Yusril Kadiptya, Yongsu Kim, Howon Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14879
• Pdf URL: https://arxiv.org/pdf/2408.14879
• Copy Paste: [[2408.14879]] Adversarial Manhole: Challenging Monocular Depth Estimation and Semantic Segmentation Models with Patch Attack(https://arxiv.org/abs/2408.14879)
• Keywords: attack, segmentation
• Abstract: Monocular depth estimation (MDE) and semantic segmentation (SS) are crucial for the navigation and environmental interpretation of many autonomous driving systems. However, their vulnerability to practical adversarial attacks is a significant concern. This paper presents a novel adversarial attack using practical patches that mimic manhole covers to deceive MDE and SS models. The goal is to cause these systems to misinterpret scenes, leading to false detections of near obstacles or non-passable objects. We use Depth Planar Mapping to precisely position these patches on road surfaces, enhancing the attack's effectiveness. Our experiments show that these adversarial patches cause a 43% relative error in MDE and achieve a 96% attack success rate in SS. These patches create affected error regions over twice their size in MDE and approximately equal to their size in SS. Our studies also confirm the patch's effectiveness in physical simulations, the adaptability of the patches across different target models, and the effectiveness of our proposed modules, highlighting their practical implications.

Title: User-level Social Multimedia Traffic Anomaly Detection with Meta-Learning

• Authors: Tongtong Feng
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.14884
• Pdf URL: https://arxiv.org/pdf/2408.14884
• Copy Paste: [[2408.14884]] User-level Social Multimedia Traffic Anomaly Detection with Meta-Learning(https://arxiv.org/abs/2408.14884)
• Keywords: security, privacy, generative
• Abstract: Accuracy anomaly detection in user-level social multimedia traffic is crucial for privacy security. Compared with existing models that passively detect specific anomaly classes with large labeled training samples, user-level social multimedia traffic contains sizeable new anomaly classes with few labeled samples and has an imbalance, self-similar, and data-hungry nature. Recent advances, such as Generative Adversarial Networks (GAN), solve it by learning a sample generator only from seen class samples to synthesize new samples. However, if we detect many new classes, the number of synthesizing samples would be unfeasibly estimated, and this operation will drastically increase computational complexity and energy consumption. Motivation on these limitations, in this paper, we propose \textit{Meta-UAD}, a Meta-learning scheme for User-level social multimedia traffic Anomaly Detection. This scheme relies on the episodic training paradigm and learns from the collection of K-way-M-shot classification tasks, which can use the pre-trained model to adapt any new class with few samples by going through few iteration steps. Since user-level social multimedia traffic emerges from a complex interaction process of users and social applications, we further develop a feature extractor to improve scheme performance. It extracts statistical features using cumulative importance ranking and time-series features using an LSTM-based AutoEncoder. We evaluate our scheme on two public datasets and the results further demonstrate the superiority of Meta-UAD.

Title: MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation

• Authors: Hyunwoo Kim, Itai Lang, Noam Aigerman, Thibault Groueix, Vladimir G. Kim, Rana Hanocka
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.14899
• Pdf URL: https://arxiv.org/pdf/2408.14899
• Copy Paste: [[2408.14899]] MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation(https://arxiv.org/abs/2408.14899)
• Keywords: diffusion
• Abstract: We propose MeshUp, a technique that deforms a 3D mesh towards multiple target concepts, and intuitively controls the region where each concept is expressed. Conveniently, the concepts can be defined as either text queries, e.g., "a dog" and "a turtle," or inspirational images, and the local regions can be selected as any number of vertices on the mesh. We can effectively control the influence of the concepts and mix them together using a novel score distillation approach, referred to as the Blended Score Distillation (BSD). BSD operates on each attention layer of the denoising U-Net of a diffusion model as it extracts and injects the per-objective activations into a unified denoising pipeline from which the deformation gradients are calculated. To localize the expression of these activations, we create a probabilistic Region of Interest (ROI) map on the surface of the mesh, and turn it into 3D-consistent masks that we use to control the expression of these activations. We demonstrate the effectiveness of BSD empirically and show that it can deform various meshes towards multiple objectives.

Title: Writing in the Margins: Better Inference Pattern for Long Context Retrieval

• Authors: Melisa Russak, Umar Jamil, Christopher Bryant, Kiran Kamble, Axel Magnuson, Mateusz Russak, Waseem AlShikh
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.14906
• Pdf URL: https://arxiv.org/pdf/2408.14906
• Copy Paste: [[2408.14906]] Writing in the Margins: Better Inference Pattern for Long Context Retrieval(https://arxiv.org/abs/2408.14906)
• Keywords: transformer, large language model
• Abstract: In this paper, we introduce Writing in the Margins (WiM), a new inference pattern for Large Language Models designed to optimize the handling of long input sequences in retrieval-oriented tasks. This approach leverages the chunked prefill of the key-value cache to perform segment-wise inference, which enables efficient processing of extensive contexts along with the generation and classification of intermediate information ("margins") that guide the model towards specific tasks. This method increases computational overhead marginally while significantly enhancing the performance of off-the-shelf models without the need for fine-tuning. Specifically, we observe that WiM provides an average enhancement of 7.5% in accuracy for reasoning skills (HotpotQA, MultiHop-RAG) and more than a 30.0% increase in the F1-score for aggregation tasks (CWE). Additionally, we show how the proposed pattern fits into an interactive retrieval design that provides end-users with ongoing updates about the progress of context processing, and pinpoints the integration of relevant information into the final response. We release our implementation of WiM using Hugging Face Transformers library at this https URL.

Title: SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models

• Authors: Shuaijie Shen, Chao Wang, Renzhuo Huang, Yan Zhong, Qinghai Guo, Zhichao Lu, Jianguo Zhang, Luziwei Leng
• Subjects: cs.CL, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2408.14909
• Pdf URL: https://arxiv.org/pdf/2408.14909
• Copy Paste: [[2408.14909]] SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models(https://arxiv.org/abs/2408.14909)
• Keywords: large language model
• Abstract: Known as low energy consumption networks, spiking neural networks (SNNs) have gained a lot of attention within the past decades. While SNNs are increasing competitive with artificial neural networks (ANNs) for vision tasks, they are rarely used for long sequence tasks, despite their intrinsic temporal dynamics. In this work, we develop spiking state space models (SpikingSSMs) for long sequence learning by leveraging on the sequence learning abilities of state space models (SSMs). Inspired by dendritic neuron structure, we hierarchically integrate neuronal dynamics with the original SSM block, meanwhile realizing sparse synaptic computation. Furthermore, to solve the conflict of event-driven neuronal dynamics with parallel computing, we propose a light-weight surrogate dynamic network which accurately predicts the after-reset membrane potential and compatible to learnable thresholds, enabling orders of acceleration in training speed compared with conventional iterative methods. On the long range arena benchmark task, SpikingSSM achieves competitive performance to state-of-the-art SSMs meanwhile realizing on average 90\% of network sparsity. On language modeling, our network significantly surpasses existing spiking large language models (spikingLLMs) on the WikiText-103 dataset with only a third of the model size, demonstrating its potential as backbone architecture for low computation cost LLMs.

Title: Can Transformers Do Enumerative Geometry?

• Authors: Baran Hashemi, Roderic G. Corominas, Alessandro Giacchetto
• Subjects: cs.LG, math.AG
• Abstract URL: https://arxiv.org/abs/2408.14915
• Pdf URL: https://arxiv.org/pdf/2408.14915
• Copy Paste: [[2408.14915]] Can Transformers Do Enumerative Geometry?(https://arxiv.org/abs/2408.14915)
• Keywords: robust, interpretability, transformer
• Abstract: How can Transformers model and learn enumerative geometry? What is a robust procedure for using Transformers in abductive knowledge discovery within a mathematician-machine collaboration? In this work, we introduce a new paradigm in computational enumerative geometry in analyzing the $\psi$-class intersection numbers on the moduli space of curves. By formulating the enumerative problem as a continuous optimization task, we develop a Transformer-based model for computing $\psi$-class intersection numbers based on the underlying quantum Airy structure. For a finite range of genera, our model is capable of regressing intersection numbers that span an extremely wide range of values, from $10^{-45}$ to $10^{45}$. To provide a proper inductive bias for capturing the recursive behavior of intersection numbers, we propose a new activation function, Dynamic Range Activator (DRA). Moreover, given the severe heteroscedasticity of $\psi$-class intersections and the required precision, we quantify the uncertainty of the predictions using Conformal Prediction with a dynamic sliding window that is aware of the number of marked points. Next, we go beyond merely computing intersection numbers and explore the enumerative "world-model" of the Transformers. Through a series of causal inference and correlational interpretability analyses, we demonstrate that Transformers are actually modeling Virasoro constraints in a purely data-driven manner. Additionally, we provide evidence for the comprehension of several values appearing in the large genus asymptotic of $\psi$-class intersection numbers through abductive hypothesis testing.

Title: From Chaos to Consistency: The Role of CSAF in Streamlining Security Advisories

• Authors: Julia Wunder, Janik Aurich, Zinaida Benenson
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.14937
• Pdf URL: https://arxiv.org/pdf/2408.14937
• Copy Paste: [[2408.14937]] From Chaos to Consistency: The Role of CSAF in Streamlining Security Advisories(https://arxiv.org/abs/2408.14937)
• Keywords: security
• Abstract: Security advisories have become an important part of vulnerability management. They can be used to gather and distribute valuable information about vulnerabilities. Although there is a predefined broad format for advisories, it is not really standardized. As a result, their content and form vary greatly depending on the vendor. Thus, it is cumbersome and resource-intensive for security analysts to extract the relevant information. The Common Security Advisory Format (CSAF) aims to bring security advisories into a standardized format which is intended to solve existing problems and to enable automated processing of the advisories. However, a new standard only makes sense if it can benefit users. Hence the questions arise: Do security advisories cause issues in their current state? Which of these issues is CSAF able to resolve? What is the current state of automation? To investigate these questions, we interviewed three security experts, and then conducted an online survey with 197 participants. The results show that problems exist and can often be traced back to confusing and inconsistent structures and formats. CSAF attempts to solve precisely these problems. However, our results show that CSAF is currently rarely used. Although users perceive automation as necessary to improve the processing of security advisories, many are at the same time skeptical. One of the main reasons is that systems are not yet designed for automation and a migration would require vast amounts of resources.

Title: BOX3D: Lightweight Camera-LiDAR Fusion for 3D Object Detection and Localization

• Authors: Mario A.V. Saucedo, Nikolaos Stathoulopoulos, Vidya Sumathy, Christoforos Kanellakis, George Nikolakopoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14941
• Pdf URL: https://arxiv.org/pdf/2408.14941
• Copy Paste: [[2408.14941]] BOX3D: Lightweight Camera-LiDAR Fusion for 3D Object Detection and Localization(https://arxiv.org/abs/2408.14941)
• Keywords: extraction
• Abstract: Object detection and global localization play a crucial role in robotics, spanning across a great spectrum of applications from autonomous cars to multi-layered 3D Scene Graphs for semantic scene understanding. This article proposes BOX3D, a novel multi-modal and lightweight scheme for localizing objects of interest by fusing the information from RGB camera and 3D LiDAR. BOX3D is structured around a three-layered architecture, building up from the local perception of the incoming sequential sensor data to the global perception refinement that covers for outliers and the general consistency of each object's observation. More specifically, the first layer handles the low-level fusion of camera and LiDAR data for initial 3D bounding box extraction. The second layer converts each LiDAR's scan 3D bounding boxes to the world coordinate frame and applies a spatial pairing and merging mechanism to maintain the uniqueness of objects observed from different viewpoints. Finally, BOX3D integrates the third layer that supervises the consistency of the results on the global map iteratively, using a point-to-voxel comparison for identifying all points in the global map that belong to the object. Benchmarking results of the proposed novel architecture are showcased in multiple experimental trials on public state-of-the-art large-scale dataset of urban environments.

Title: NeuralOOD: Improving Out-of-Distribution Generalization Performance with Brain-machine Fusion Learning Framework

• Authors: Shuangchen Zhao, Changde Du, Hui Li, Huiguang He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14950
• Pdf URL: https://arxiv.org/pdf/2408.14950
• Copy Paste: [[2408.14950]] NeuralOOD: Improving Out-of-Distribution Generalization Performance with Brain-machine Fusion Learning Framework(https://arxiv.org/abs/2408.14950)
• Keywords: extraction, transformer
• Abstract: Deep Neural Networks (DNNs) have demonstrated exceptional recognition capabilities in traditional computer vision (CV) tasks. However, existing CV models often suffer a significant decrease in accuracy when confronted with out-of-distribution (OOD) data. In contrast to these DNN models, human can maintain a consistently low error rate when facing OOD scenes, partly attributed to the rich prior cognitive knowledge stored in the human brain. Previous OOD generalization researches only focus on the single modal, overlooking the advantages of multimodal learning method. In this paper, we utilize the multimodal learning method to improve the OOD generalization and propose a novel Brain-machine Fusion Learning (BMFL) framework. We adopt the cross-attention mechanism to fuse the visual knowledge from CV model and prior cognitive knowledge from the human brain. Specially, we employ a pre-trained visual neural encoding model to predict the functional Magnetic Resonance Imaging (fMRI) from visual features which eliminates the need for the fMRI data collection and pre-processing, effectively reduces the workload associated with conventional BMFL methods. Furthermore, we construct a brain transformer to facilitate the extraction of knowledge inside the fMRI data. Moreover, we introduce the Pearson correlation coefficient maximization regularization method into the training process, which improves the fusion capability with better constrains. Our model outperforms the DINOv2 and baseline models on the ImageNet-1k validation dataset as well as six curated OOD datasets, showcasing its superior performance in diverse scenarios.

Title: Applying ViT in Generalized Few-shot Semantic Segmentation

• Authors: Liyuan Geng, Jinhong Xia, Yuanhe Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14957
• Pdf URL: https://arxiv.org/pdf/2408.14957
• Copy Paste: [[2408.14957]] Applying ViT in Generalized Few-shot Semantic Segmentation(https://arxiv.org/abs/2408.14957)
• Keywords: transformer, segmentation
• Abstract: This paper explores the capability of ViT-based models under the generalized few-shot semantic segmentation (GFSS) framework. We conduct experiments with various combinations of backbone models, including ResNets and pretrained Vision Transformer (ViT)-based models, along with decoders featuring a linear classifier, UPerNet, and Mask Transformer. The structure made of DINOv2 and linear classifier takes the lead on popular few-shot segmentation bench mark PASCAL-$5^i$, substantially outperforming the best of ResNet structure by 116% in one-shot scenario. We demonstrate the great potential of large pretrained ViT-based model on GFSS task, and expect further improvement on testing benchmarks. However, a potential caveat is that when applying pure ViT-based model and large scale ViT decoder, the model is easy to overfit.

Title: Cross-Modal Learning for Chemistry Property Prediction: Large Language Models Meet Graph Machine Learning

• Authors: Sakhinana Sagar Srinivas, Venkataramana Runkana
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.14964
• Pdf URL: https://arxiv.org/pdf/2408.14964
• Copy Paste: [[2408.14964]] Cross-Modal Learning for Chemistry Property Prediction: Large Language Models Meet Graph Machine Learning(https://arxiv.org/abs/2408.14964)
• Keywords: robust, generative, large language model
• Abstract: In the field of chemistry, the objective is to create novel molecules with desired properties, facilitating accurate property predictions for applications such as material design and drug screening. However, existing graph deep learning methods face limitations that curb their expressive power. To address this, we explore the integration of vast molecular domain knowledge from Large Language Models (LLMs) with the complementary strengths of Graph Neural Networks (GNNs) to enhance performance in property prediction tasks. We introduce a Multi-Modal Fusion (MMF) framework that synergistically harnesses the analytical prowess of GNNs and the linguistic generative and predictive abilities of LLMs, thereby improving accuracy and robustness in predicting molecular properties. Our framework combines the effectiveness of GNNs in modeling graph-structured data with the zero-shot and few-shot learning capabilities of LLMs, enabling improved predictions while reducing the risk of overfitting. Furthermore, our approach effectively addresses distributional shifts, a common challenge in real-world applications, and showcases the efficacy of learning cross-modal representations, surpassing state-of-the-art baselines on benchmark datasets for property prediction tasks.

Title: AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systems

• Authors: Chi-Min Chan, Jianxuan Yu, Weize Chen, Chunyang Jiang, Xinyu Liu, Weijie Shi, Zhiyuan Liu, Wei Xue, Yike Guo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14972
• Pdf URL: https://arxiv.org/pdf/2408.14972
• Copy Paste: [[2408.14972]] AgentMonitor: A Plug-and-Play Framework for Predictive and Secure Multi-Agent Systems(https://arxiv.org/abs/2408.14972)
• Keywords: secure, security, large language model
• Abstract: The rapid advancement of large language models (LLMs) has led to the rise of LLM-based agents. Recent research shows that multi-agent systems (MAS), where each agent plays a specific role, can outperform individual LLMs. However, configuring an MAS for a task remains challenging, with performance only observable post-execution. Inspired by scaling laws in LLM development, we investigate whether MAS performance can be predicted beforehand. We introduce AgentMonitor, a framework that integrates at the agent level to capture inputs and outputs, transforming them into statistics for training a regression model to predict task performance. Additionally, it can further apply real-time corrections to address security risks posed by malicious agents, mitigating negative impacts and enhancing MAS security. Experiments demonstrate that an XGBoost model achieves a Spearman correlation of 0.89 in-domain and 0.58 in more challenging scenarios. Furthermore, using AgentMonitor reduces harmful content by 6.2% and increases helpful content by 1.8% on average, enhancing safety and reliability. Code is available at \url{this https URL}.

Title: MegActor-$\Sigma$: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformer

• Authors: Shurong Yang, Huadong Li, Juhao Wu, Minhao Jing, Linze Li, Renhe Ji, Jiajun Liang, Haoqiang Fan, Jin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14975
• Pdf URL: https://arxiv.org/pdf/2408.14975
• Copy Paste: [[2408.14975]] MegActor-$\Sigma$: Unlocking Flexible Mixed-Modal Control in Portrait Animation with Diffusion Transformer(https://arxiv.org/abs/2408.14975)
• Keywords: diffusion, transformer
• Abstract: Diffusion models have demonstrated superior performance in the field of portrait animation. However, current approaches relied on either visual or audio modality to control character movements, failing to exploit the potential of mixed-modal control. This challenge arises from the difficulty in balancing the weak control strength of audio modality and the strong control strength of visual modality. To address this issue, we introduce MegActor-$\Sigma$: a mixed-modal conditional diffusion transformer (DiT), which can flexibly inject audio and visual modality control signals into portrait animation. Specifically, we make substantial advancements over its predecessor, MegActor, by leveraging the promising model structure of DiT and integrating audio and visual conditions through advanced modules within the DiT framework. To further achieve flexible combinations of mixed-modal control signals, we propose a ``Modality Decoupling Control" training strategy to balance the control strength between visual and audio modalities, along with the ``Amplitude Adjustment" inference strategy to freely regulate the motion amplitude of each modality. Finally, to facilitate extensive studies in this field, we design several dataset evaluation metrics to filter out public datasets and solely use this filtered dataset to train MegActor-$\Sigma$. Extensive experiments demonstrate the superiority of our approach in generating vivid portrait animations, outperforming previous methods trained on private dataset.

Title: Speech Recognition Transformers: Topological-lingualism Perspective

• Authors: Shruti Singh, Muskaan Singh, Virender Kadyan
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.14991
• Pdf URL: https://arxiv.org/pdf/2408.14991
• Copy Paste: [[2408.14991]] Speech Recognition Transformers: Topological-lingualism Perspective(https://arxiv.org/abs/2408.14991)
• Keywords: transformer
• Abstract: Transformers have evolved with great success in various artificial intelligence tasks. Thanks to our recent prevalence of self-attention mechanisms, which capture long-term dependency, phenomenal outcomes in speech processing and recognition tasks have been produced. The paper presents a comprehensive survey of transformer techniques oriented in speech modality. The main contents of this survey include (1) background of traditional ASR, end-to-end transformer ecosystem, and speech transformers (2) foundational models in a speech via lingualism paradigm, i.e., monolingual, bilingual, multilingual, and cross-lingual (3) dataset and languages, acoustic features, architecture, decoding, and evaluation metric from a specific topological lingualism perspective (4) popular speech transformer toolkit for building end-to-end ASR systems. Finally, highlight the discussion of open challenges and potential research directions for the community to conduct further research in this domain.

Title: FastTextSpotter: A High-Efficiency Transformer for Multilingual Scene Text Spotting

• Authors: Alloy Das, Sanket Biswas, Umapada Pal, Josep Lladós, Saumik Bhattacharya
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.14998
• Pdf URL: https://arxiv.org/pdf/2408.14998
• Copy Paste: [[2408.14998]] FastTextSpotter: A High-Efficiency Transformer for Multilingual Scene Text Spotting(https://arxiv.org/abs/2408.14998)
• Keywords: robust, transformer
• Abstract: The proliferation of scene text in both structured and unstructured environments presents significant challenges in optical character recognition (OCR), necessitating more efficient and robust text spotting solutions. This paper presents FastTextSpotter, a framework that integrates a Swin Transformer visual backbone with a Transformer Encoder-Decoder architecture, enhanced by a novel, faster self-attention unit, SAC2, to improve processing speeds while maintaining accuracy. FastTextSpotter has been validated across multiple datasets, including ICDAR2015 for regular texts and CTW1500 and TotalText for arbitrary-shaped texts, benchmarking against current state-of-the-art models. Our results indicate that FastTextSpotter not only achieves superior accuracy in detecting and recognizing multilingual scene text (English and Vietnamese) but also improves model efficiency, thereby setting new benchmarks in the field. This study underscores the potential of advanced transformer architectures in improving the adaptability and speed of text spotting applications in diverse real-world settings. The dataset, code, and pre-trained models have been released in our Github.

Title: Hierarchical Graph Interaction Transformer with Dynamic Token Clustering for Camouflaged Object Detection

• Authors: Siyuan Yao, Hao Sun, Tian-Zhu Xiang, Xiao Wang, Xiaochun Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15020
• Pdf URL: https://arxiv.org/pdf/2408.15020
• Copy Paste: [[2408.15020]] Hierarchical Graph Interaction Transformer with Dynamic Token Clustering for Camouflaged Object Detection(https://arxiv.org/abs/2408.15020)
• Keywords: transformer
• Abstract: Camouflaged object detection (COD) aims to identify the objects that seamlessly blend into the surrounding backgrounds. Due to the intrinsic similarity between the camouflaged objects and the background region, it is extremely challenging to precisely distinguish the camouflaged objects by existing approaches. In this paper, we propose a hierarchical graph interaction network termed HGINet for camouflaged object detection, which is capable of discovering imperceptible objects via effective graph interaction among the hierarchical tokenized features. Specifically, we first design a region-aware token focusing attention (RTFA) with dynamic token clustering to excavate the potentially distinguishable tokens in the local region. Afterwards, a hierarchical graph interaction transformer (HGIT) is proposed to construct bi-directional aligned communication between hierarchical features in the latent interaction space for visual semantics enhancement. Furthermore, we propose a decoder network with confidence aggregated feature fusion (CAFF) modules, which progressively fuses the hierarchical interacted features to refine the local detail in ambiguous regions. Extensive experiments conducted on the prevalent datasets, i.e. COD10K, CAMO, NC4K and CHAMELEON demonstrate the superior performance of HGINet compared to existing state-of-the-art methods. Our code is available at this https URL.

Title: Evidence-Enhanced Triplet Generation Framework for Hallucination Alleviation in Generative Question Answering

• Authors: Haowei Du, Huishuai Zhang, Dongyan Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15037
• Pdf URL: https://arxiv.org/pdf/2408.15037
• Copy Paste: [[2408.15037]] Evidence-Enhanced Triplet Generation Framework for Hallucination Alleviation in Generative Question Answering(https://arxiv.org/abs/2408.15037)
• Keywords: generative
• Abstract: To address the hallucination in generative question answering (GQA) where the answer can not be derived from the document, we propose a novel evidence-enhanced triplet generation framework, EATQA, encouraging the model to predict all the combinations of (Question, Evidence, Answer) triplet by flipping the source pair and the target label to understand their logical relationships, i.e., predict Answer(A), Question(Q), and Evidence(E) given a QE, EA, and QA pairs, respectively. Furthermore, we bridge the distribution gap to distill the knowledge from evidence in inference stage. Our framework ensures the model to learn the logical relation between query, evidence and answer, which simultaneously improves the evidence generation and query answering. In this paper, we apply EATQA to LLama and it outperforms other LLMs-based methods and hallucination mitigation approaches on two challenging GQA benchmarks. Further analysis shows that our method not only keeps prior knowledge within LLM, but also mitigates hallucination and generates faithful answers.

Title: Interactive Occlusion Boundary Estimation through Exploitation of Synthetic Data

• Authors: Lintao Xu, Chaohui Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15038
• Pdf URL: https://arxiv.org/pdf/2408.15038
• Copy Paste: [[2408.15038]] Interactive Occlusion Boundary Estimation through Exploitation of Synthetic Data(https://arxiv.org/abs/2408.15038)
• Keywords: robust
• Abstract: Occlusion boundaries (OBs) geometrically localize the occlusion events in a 2D image, and contain useful information for addressing various scene understanding problems. To advance their study, we have led the investigation in the following three aspects. Firstly, we have studied interactive estimation of OBs, which is the first in the literature, and proposed an efficient deep-network-based method using multiple-scribble intervention, named DNMMSI, which significantly improves the performance over the state-of-the-art fully-automatic methods. Secondly, we propose to exploit the synthetic benchmark for the training process, thanks to the particularity that OBs are determined geometrically and unambiguously from the 3D scene. To this end, we have developed an efficient tool, named Mesh2OB, for the automatic generation of 2D images together with their ground-truth OBs, using which we have constructed a synthetic benchmark, named OB-FUTURE. Abundant experimental results demonstrate that leveraging such a synthetic benchmark for training achieves promising performance, even without the use of domain adaptation techniques. Finally, to achieve a more compelling and robust evaluation in OB-related research, we have created a real benchmark, named OB-LabName, consisting of 120 high-resolution images together with their ground-truth OBs, with precision surpassing that of previous benchmarks. We will release DNMMSI with pre-trained parameters, Mesh2OB, OB-FUTURE, and OB-LabName to support further research.

Title: A Survey of Large Language Models for European Languages

• Authors: Wazir Ali, Sampo Pyysalo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15040
• Pdf URL: https://arxiv.org/pdf/2408.15040
• Copy Paste: [[2408.15040]] A Survey of Large Language Models for European Languages(https://arxiv.org/abs/2408.15040)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have gained significant attention due to their high performance on a wide range of natural language tasks since the release of ChatGPT. The LLMs learn to understand and generate language by training billions of model parameters on vast volumes of text data. Despite being a relatively new field, LLM research is rapidly advancing in various directions. In this paper, we present an overview of LLM families, including LLaMA, PaLM, GPT, and MoE, and the methods developed to create and enhance LLMs for official European Union (EU) languages. We provide a comprehensive summary of common monolingual and multilingual datasets used for pretraining LLMs.

Title: Enabling Efficient and Scalable DRAM Read Disturbance Mitigation via New Experimental Insights into Modern DRAM Chips

• Authors: Abdullah Giray Yağlıkçı
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2408.15044
• Pdf URL: https://arxiv.org/pdf/2408.15044
• Copy Paste: [[2408.15044]] Enabling Efficient and Scalable DRAM Read Disturbance Mitigation via New Experimental Insights into Modern DRAM Chips(https://arxiv.org/abs/2408.15044)
• Keywords: secure, protect, defense, attack, robust
• Abstract: Increasing storage density exacerbates DRAM read disturbance, a circuit-level vulnerability exploited by system-level attacks. Unfortunately, existing defenses are either ineffective or prohibitively expensive. Efficient mitigation is critical to ensure robust (reliable, secure, and safe) execution in future DRAM-based systems. This dissertation tackles two problems: 1) protecting DRAM-based systems becomes more expensive as technology scaling increases read disturbance vulnerability, and 2) many existing solutions depend on proprietary knowledge of DRAM internals. First, we build a detailed understanding of DRAM read disturbance by rigorously characterizing off-the-shelf modern DRAM chips under varying 1) temperatures, 2) memory access patterns, 3) in-chip locations, and 4) voltage. Our novel observations demystify the implications of large DRAM read disturbance variation on future DRAM read disturbance attacks and solutions. Second, we propose new mechanisms that mitigate read disturbance bitflips efficiently and scalably by leveraging insights into DRAM chip design: 1) subarray-level parallelism and 2) variation in read disturbance across DRAM rows in off-the-shelf DRAM chips. Third, we propose a novel solution that mitigates DRAM read disturbance by selectively throttling unsafe memory accesses that might otherwise cause read disturbance bitflips without proprietary knowledge of DRAM chip internals. We demonstrate that it is possible to mitigate DRAM read disturbance efficiently and scalably with worsening DRAM read disturbance by 1) building a detailed understanding of DRAM read disturbance, 2) leveraging insights into DRAM chips, and 3) devising novel solutions that do not require proprietary knowledge of DRAM chip internals. Our experimental insights and solutions enable future works targeting robust memory systems.

Title: DocLayLLM: An Efficient and Effective Multi-modal Extension of Large Language Models for Text-rich Document Understanding

• Authors: Wenhui Liao, Jiapeng Wang, Hongliang Li, Chengyu Wang, Jun Huang, Lianwen Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15045
• Pdf URL: https://arxiv.org/pdf/2408.15045
• Copy Paste: [[2408.15045]] DocLayLLM: An Efficient and Effective Multi-modal Extension of Large Language Models for Text-rich Document Understanding(https://arxiv.org/abs/2408.15045)
• Keywords: large language model
• Abstract: Text-rich document understanding (TDU) refers to analyzing and comprehending documents containing substantial textual content. With the rapid evolution of large language models (LLMs), they have been widely leveraged for TDU due to their remarkable versatility and generalization. In this paper, we introduce DocLayLLM, an efficient and effective multi-modal extension of LLMs specifically designed for TDU. By integrating visual patch tokens and 2D positional tokens into LLMs and encoding the document content using the LLMs themselves, we fully take advantage of the document comprehension capability of LLMs and enhance their perception of OCR information. We have also deeply considered the role of the chain-of-thought (CoT) and innovatively proposed the techniques of CoT Pre-training and CoT Annealing. Our DocLayLLM can achieve remarkable performances with lightweight training settings, showcasing its efficiency and effectiveness. Experimental results demonstrate that our DocLayLLM surpasses existing OCR-dependent methods and also outperforms OCR-free competitors.

Title: Causal Rule Forest: Toward Interpretable and Precise Treatment Effect Estimation

• Authors: Chan Hsu, Jun-Ting Wu, Yihuang Kang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15055
• Pdf URL: https://arxiv.org/pdf/2408.15055
• Copy Paste: [[2408.15055]] Causal Rule Forest: Toward Interpretable and Precise Treatment Effect Estimation(https://arxiv.org/abs/2408.15055)
• Keywords: interpretability
• Abstract: Understanding and inferencing Heterogeneous Treatment Effects (HTE) and Conditional Average Treatment Effects (CATE) are vital for developing personalized treatment recommendations. Many state-of-the-art approaches achieve inspiring performance in estimating HTE on benchmark datasets or simulation studies. However, the indirect predicting manner and complex model architecture reduce the interpretability of these approaches. To mitigate the gap between predictive performance and heterogeneity interpretability, we introduce the Causal Rule Forest (CRF), a novel approach to learning hidden patterns from data and transforming the patterns into interpretable multi-level Boolean rules. By training the other interpretable causal inference models with data representation learned by CRF, we can reduce the predictive errors of these models in estimating HTE and CATE, while keeping their interpretability for identifying subgroups that a treatment is more effective. Our experiments underscore the potential of CRF to advance personalized interventions and policies, paving the way for future research to enhance its scalability and application across complex causal inference challenges.

Title: Subgroup Analysis via Model-based Rule Forest

• Authors: I-Ling Cheng, Chan Hsu, Chantung Ku, Pei-Ju Lee, Yihuang Kang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15057
• Pdf URL: https://arxiv.org/pdf/2408.15057
• Copy Paste: [[2408.15057]] Subgroup Analysis via Model-based Rule Forest(https://arxiv.org/abs/2408.15057)
• Keywords: interpretability
• Abstract: Machine learning models are often criticized for their black-box nature, raising concerns about their applicability in critical decision-making scenarios. Consequently, there is a growing demand for interpretable models in such contexts. In this study, we introduce Model-based Deep Rule Forests (mobDRF), an interpretable representation learning algorithm designed to extract transparent models from data. By leveraging IF-THEN rules with multi-level logic expressions, mobDRF enhances the interpretability of existing models without compromising accuracy. We apply mobDRF to identify key risk factors for cognitive decline in an elderly population, demonstrating its effectiveness in subgroup analysis and local model optimization. Our method offers a promising solution for developing trustworthy and interpretable machine learning models, particularly valuable in fields like healthcare, where understanding differential effects across patient subgroups can lead to more personalized and effective treatments.

Title: Adapting Segment Anything Model to Multi-modal Salient Object Detection with Semantic Feature Fusion Guidance

• Authors: Kunpeng Wang, Keke Chen, Chenglong Li, Zhengzheng Tu, Bin Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15063
• Pdf URL: https://arxiv.org/pdf/2408.15063
• Copy Paste: [[2408.15063]] Adapting Segment Anything Model to Multi-modal Salient Object Detection with Semantic Feature Fusion Guidance(https://arxiv.org/abs/2408.15063)
• Keywords: robust
• Abstract: Although most existing multi-modal salient object detection (SOD) methods demonstrate effectiveness through training models from scratch, the limited multi-modal data hinders these methods from reaching optimality. In this paper, we propose a novel framework to explore and exploit the powerful feature representation and zero-shot generalization ability of the pre-trained Segment Anything Model (SAM) for multi-modal SOD. Despite serving as a recent vision fundamental model, driving the class-agnostic SAM to comprehend and detect salient objects accurately is non-trivial, especially in challenging scenes. To this end, we develop \underline{SAM} with se\underline{m}antic f\underline{e}ature fu\underline{s}ion guidanc\underline{e} (Sammese), which incorporates multi-modal saliency-specific knowledge into SAM to adapt SAM to multi-modal SOD tasks. However, it is difficult for SAM trained on single-modal data to directly mine the complementary benefits of multi-modal inputs and comprehensively utilize them to achieve accurate saliency this http URL address these issues, we first design a multi-modal complementary fusion module to extract robust multi-modal semantic features by integrating information from visible and thermal or depth image pairs. Then, we feed the extracted multi-modal semantic features into both the SAM image encoder and mask decoder for fine-tuning and prompting, respectively. Specifically, in the image encoder, a multi-modal adapter is proposed to adapt the single-modal SAM to multi-modal information. In the mask decoder, a semantic-geometric prompt generation strategy is proposed to produce corresponding embeddings with various saliency cues. Extensive experiments on both RGB-D and RGB-T SOD benchmarks show the effectiveness of the proposed framework.

Title: MMASD+: A Novel Dataset for Privacy-Preserving Behavior Analysis of Children with Autism Spectrum Disorder

• Authors: Pavan Uttej Ravva, Behdokht Kiafar, Pinar Kullu, Jicheng Li, Anjana Bhat, Roghayeh Leila Barmaki
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15077
• Pdf URL: https://arxiv.org/pdf/2408.15077
• Copy Paste: [[2408.15077]] MMASD+: A Novel Dataset for Privacy-Preserving Behavior Analysis of Children with Autism Spectrum Disorder(https://arxiv.org/abs/2408.15077)
• Keywords: privacy, transformer
• Abstract: Autism spectrum disorder (ASD) is characterized by significant challenges in social interaction and comprehending communication signals. Recently, therapeutic interventions for ASD have increasingly utilized Deep learning powered-computer vision techniques to monitor individual progress over time. These models are trained on private, non-public datasets from the autism community, creating challenges in comparing results across different models due to privacy-preserving data-sharing issues. This work introduces MMASD+. MMASD+ consists of diverse data modalities, including 3D-Skeleton, 3D Body Mesh, and Optical Flow data. It integrates the capabilities of Yolov8 and Deep SORT algorithms to distinguish between the therapist and children, addressing a significant barrier in the original dataset. Additionally, a Multimodal Transformer framework is proposed to predict 11 action types and the presence of ASD. This framework achieves an accuracy of 95.03% for predicting action types and 96.42% for predicting ASD presence, demonstrating over a 10% improvement compared to models trained on single data modalities. These findings highlight the advantages of integrating multiple data modalities within the Multimodal Transformer framework.

Title: BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline

• Authors: Guosheng Dong, Da Pan, Yiding Sun, Shusen Zhang, Zheng Liang, Xin Wu, Yanjun Shen, Fan Yang, Haoze Sun, Tianpeng Li, Mingan Lin, Jianhua Xu, Yufan Zhang, Xiaonan Nie, Lei Su, Bingning Wang, Wentao Zhang, Jiaxin Mao, Zenan Zhou, Weipeng Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15079
• Pdf URL: https://arxiv.org/pdf/2408.15079
• Copy Paste: [[2408.15079]] BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and Deduplication by Introducing a Competitive Large Language Model Baseline(https://arxiv.org/abs/2408.15079)
• Keywords: large language model
• Abstract: The general capabilities of Large Language Models (LLM) highly rely on the composition and selection on extensive pretraining datasets, treated as commercial secrets by several institutions. To mitigate this issue, we open-source the details of a universally applicable data processing pipeline and validate its effectiveness and potential by introducing a competitive LLM baseline. Specifically, the data processing pipeline consists of broad collection to scale up and reweighting to improve quality. We then pretrain a 7B model BaichuanSEED with 3T tokens processed by our pipeline without any deliberate downstream task-related optimization, followed by an easy but effective supervised fine-tuning stage. BaichuanSEED demonstrates consistency and predictability throughout training and achieves comparable performance on comprehensive benchmarks with several commercial advanced large language models, such as Qwen1.5 and Llama3. We also conduct several heuristic experiments to discuss the potential for further optimization of downstream tasks, such as mathematics and coding.

Title: Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models

• Authors: Xiyu Liu, Zhengxiao Liu, Naibin Gu, Zheng Lin, Wanli Ma, Ji Xiang, Weiping Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15091
• Pdf URL: https://arxiv.org/pdf/2408.15091
• Copy Paste: [[2408.15091]] Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models(https://arxiv.org/abs/2408.15091)
• Keywords: transformer
• Abstract: The storage and recall of factual associations in auto-regressive transformer language models (LMs) have drawn a great deal of attention, inspiring knowledge editing by directly modifying the located model weights. Most editing works achieve knowledge editing under the guidance of existing interpretations of knowledge recall that mainly focus on subject knowledge. However, these interpretations are seriously flawed, neglecting relation information and leading to the over-generalizing problem for editing. In this work, we discover a novel relation-focused perspective to interpret the knowledge recall of transformer LMs during inference and apply it on knowledge editing to avoid over-generalizing. Experimental results on the dataset supplemented with a new R-Specificity criterion demonstrate that our editing approach significantly alleviates over-generalizing while remaining competitive on other criteria, breaking the domination of subject-focused editing for future research.

Title: Constrained Diffusion Models via Dual Training

• Authors: Shervin Khalafi, Dongsheng Ding, Alejandro Ribeiro
• Subjects: cs.LG, cs.CV, math.OC
• Abstract URL: https://arxiv.org/abs/2408.15094
• Pdf URL: https://arxiv.org/pdf/2408.15094
• Copy Paste: [[2408.15094]] Constrained Diffusion Models via Dual Training(https://arxiv.org/abs/2408.15094)
• Keywords: fair, diffusion
• Abstract: Diffusion models have attained prominence for their ability to synthesize a probability distribution for a given dataset via a diffusion process, enabling the generation of new data points with high fidelity. However, diffusion processes are prone to generating biased data based on the training dataset. To address this issue, we develop constrained diffusion models by imposing diffusion constraints based on desired distributions that are informed by requirements. Specifically, we cast the training of diffusion models under requirements as a constrained distribution optimization problem that aims to reduce the distribution difference between original and generated data while obeying constraints on the distribution of generated data. We show that our constrained diffusion models generate new data from a mixture data distribution that achieves the optimal trade-off among objective and constraints. To train constrained diffusion models, we develop a dual training algorithm and characterize the optimality of the trained constrained diffusion model. We empirically demonstrate the effectiveness of our constrained models in two constrained generation tasks: (i) we consider a dataset with one or more underrepresented classes where we train the model with constraints to ensure fairly sampling from all classes during inference; (ii) we fine-tune a pre-trained diffusion model to sample from a new dataset while avoiding overfitting.

Title: Post-processing fairness with minimal changes

• Authors: Federico Di Gennaro, Thibault Laugel, Vincent Grari, Xavier Renard, Marcin Detyniecki
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15096
• Pdf URL: https://arxiv.org/pdf/2408.15096
• Copy Paste: [[2408.15096]] Post-processing fairness with minimal changes(https://arxiv.org/abs/2408.15096)
• Keywords: fair
• Abstract: In this paper, we introduce a novel post-processing algorithm that is both model-agnostic and does not require the sensitive attribute at test time. In addition, our algorithm is explicitly designed to enforce minimal changes between biased and debiased predictions; a property that, while highly desirable, is rarely prioritized as an explicit objective in fairness literature. Our approach leverages a multiplicative factor applied to the logit value of probability scores produced by a black-box classifier. We demonstrate the efficacy of our method through empirical evaluations, comparing its performance against other four debiasing algorithms on two widely used datasets in fairness research.

Title: CLIP-AGIQA: Boosting the Performance of AI-Generated Image Quality Assessment with CLIP

• Authors: Zhenchen Tang, Zichuan Wang, Bo Peng, Jing Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15098
• Pdf URL: https://arxiv.org/pdf/2408.15098
• Copy Paste: [[2408.15098]] CLIP-AGIQA: Boosting the Performance of AI-Generated Image Quality Assessment with CLIP(https://arxiv.org/abs/2408.15098)
• Keywords: generative
• Abstract: With the rapid development of generative technologies, AI-Generated Images (AIGIs) have been widely applied in various aspects of daily life. However, due to the immaturity of the technology, the quality of the generated images varies, so it is important to develop quality assessment techniques for the generated images. Although some models have been proposed to assess the quality of generated images, they are inadequate when faced with the ever-increasing and diverse categories of generated images. Consequently, the development of more advanced and effective models for evaluating the quality of generated images is urgently needed. Recent research has explored the significant potential of the visual language model CLIP in image quality assessment, finding that it performs well in evaluating the quality of natural images. However, its application to generated images has not been thoroughly investigated. In this paper, we build on this idea and further explore the potential of CLIP in evaluating the quality of generated images. We design CLIP-AGIQA, a CLIP-based regression model for quality assessment of generated images, leveraging rich visual and textual knowledge encapsulated in CLIP. Particularly, we implement multi-category learnable prompts to fully utilize the textual knowledge in CLIP for quality assessment. Extensive experiments on several generated image quality assessment benchmarks, including AGIQA-3K and AIGCIQA2023, demonstrate that CLIP-AGIQA outperforms existing IQA models, achieving excellent results in evaluating the quality of generated images.

Title: No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery

• Authors: Alexander Rutherford, Michael Beukman, Timon Willi, Bruno Lacerda, Nick Hawes, Jakob Foerster
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.15099
• Pdf URL: https://arxiv.org/pdf/2408.15099
• Copy Paste: [[2408.15099]] No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery(https://arxiv.org/abs/2408.15099)
• Keywords: robust
• Abstract: What data or environments to use for training to improve downstream performance is a longstanding and very topical question in reinforcement learning. In particular, Unsupervised Environment Design (UED) methods have gained recent attention as their adaptive curricula enable agents to be robust to in- and out-of-distribution tasks. We ask to what extent these methods are themselves robust when applied to a novel setting, closely inspired by a real-world robotics problem. Surprisingly, we find that the state-of-the-art UED methods either do not improve upon the naïve baseline of Domain Randomisation (DR), or require substantial hyperparameter tuning to do so. Our analysis shows that this is due to their underlying scoring functions failing to predict intuitive measures of ``learnability'', i.e., in finding the settings that the agent sometimes solves, but not always. Based on this, we instead directly train on levels with high learnability and find that this simple and intuitive approach outperforms UED methods and DR in several binary-outcome environments, including on our domain and the standard UED domain of Minigrid. We further introduce a new adversarial evaluation procedure for directly measuring robustness, closely mirroring the conditional value at risk (CVaR). We open-source all our code and present visualisations of final policies here: this https URL.

Title: MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoders

• Authors: Baijiong Lin, Weisen Jiang, Pengguang Chen, Shu Liu, Ying-Cong Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15101
• Pdf URL: https://arxiv.org/pdf/2408.15101
• Copy Paste: [[2408.15101]] MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoders(https://arxiv.org/abs/2408.15101)
• Keywords: transformer
• Abstract: Multi-task dense scene understanding, which trains a model for multiple dense prediction tasks, has a wide range of application scenarios. Capturing long-range dependency and enhancing cross-task interactions are crucial to multi-task dense prediction. In this paper, we propose MTMamba++, a novel architecture for multi-task scene understanding featuring with a Mamba-based decoder. It contains two types of core blocks: self-task Mamba (STM) block and cross-task Mamba (CTM) block. STM handles long-range dependency by leveraging state-space models, while CTM explicitly models task interactions to facilitate information exchange across tasks. We design two types of CTM block, namely F-CTM and S-CTM, to enhance cross-task interaction from feature and semantic perspectives, respectively. Experiments on NYUDv2, PASCAL-Context, and Cityscapes datasets demonstrate the superior performance of MTMamba++ over CNN-based and Transformer-based methods. The code is available at this https URL.

Title: The Illusion of Randomness: An Empirical Analysis of Address Space Layout Randomization Implementations

• Authors: Lorenzo Binosi, Gregorio Barzasi, Michele Carminati, Mario Polino, Stefano Zanero
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.15107
• Pdf URL: https://arxiv.org/pdf/2408.15107
• Copy Paste: [[2408.15107]] The Illusion of Randomness: An Empirical Analysis of Address Space Layout Randomization Implementations(https://arxiv.org/abs/2408.15107)
• Keywords: defense, attack, robust
• Abstract: Address Space Layout Randomization (ASLR) is a crucial defense mechanism employed by modern operating systems to mitigate exploitation by randomizing processes' memory layouts. However, the stark reality is that real-world implementations of ASLR are imperfect and subject to weaknesses that attackers can exploit. This work evaluates the effectiveness of ASLR on major desktop platforms, including Linux, MacOS, and Windows, by examining the variability in the placement of memory objects across various processes, threads, and system restarts. In particular, we collect samples of memory object locations, conduct statistical analyses to measure the randomness of these placements and examine the memory layout to find any patterns among objects that could decrease this randomness. The results show that while some systems, like Linux distributions, provide robust randomization, others, like Windows and MacOS, often fail to adequately randomize key areas like executable code and libraries. Moreover, we find a significant entropy reduction in the entropy of libraries after the Linux 5.18 version and identify correlation paths that an attacker could leverage to reduce exploitation complexity significantly. Ultimately, we rank the identified weaknesses based on severity and validate our entropy estimates with a proof-of-concept attack. In brief, this paper provides the first comprehensive evaluation of ASLR effectiveness across different operating systems and highlights opportunities for Operating System (OS) vendors to strengthen ASLR implementations.

Title: Urdu Digital Text Word Optical Character Recognition Using Permuted Auto Regressive Sequence Modeling

• Authors: Ahmed Mustafa, Ijlal Baig, Hasan Sajid
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15119
• Pdf URL: https://arxiv.org/pdf/2408.15119
• Copy Paste: [[2408.15119]] Urdu Digital Text Word Optical Character Recognition Using Permuted Auto Regressive Sequence Modeling(https://arxiv.org/abs/2408.15119)
• Keywords: transformer
• Abstract: This research paper introduces an innovative word-level Optical Character Recognition (OCR) model specifically designed for digital Urdu text recognition. Utilizing transformer-based architectures and attention mechanisms, the model was trained on a comprehensive dataset of approximately 160,000 Urdu text images, achieving a character error rate (CER) of 0.178, which highlights its superior accuracy in recognizing Urdu characters. The model's strength lies in its unique architecture, incorporating the permuted autoregressive sequence (PARSeq) model, which allows for context-aware inference and iterative refinement by leveraging bidirectional context information to enhance recognition accuracy. Furthermore, its capability to handle a diverse range of Urdu text styles, fonts, and variations enhances its applicability in real-world scenarios. Despite its promising results, the model has some limitations, such as difficulty with blurred images, non-horizontal orientations, and overlays of patterns, lines, or other text, which can occasionally lead to suboptimal performance. Additionally, trailing or following punctuation marks can introduce noise into the recognition process. Addressing these challenges will be a focus of future research, aiming to refine the model further, explore data augmentation techniques, optimize hyperparameters, and integrate contextual improvements for more accurate and efficient Urdu text recognition.

Title: Machine Learning for Methane Detection and Quantification from Space -- A survey

• Authors: Enno Tiemann, Shanyu Zhou, Alexander Kläser, Konrad Heidler, Rochelle Schneider, Xiao Xiang Zhu
• Subjects: cs.CV, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2408.15122
• Pdf URL: https://arxiv.org/pdf/2408.15122
• Copy Paste: [[2408.15122]] Machine Learning for Methane Detection and Quantification from Space -- A survey(https://arxiv.org/abs/2408.15122)
• Keywords: transformer, segmentation
• Abstract: Methane (CH_4) is a potent anthropogenic greenhouse gas, contributing 86 times more to global warming than Carbon Dioxide (CO_2) over 20 years, and it also acts as an air pollutant. Given its high radiative forcing potential and relatively short atmospheric lifetime (9\textpm1 years), methane has important implications for climate change, therefore, cutting methane emissions is crucial for effective climate change mitigation. This work expands existing information on operational methane point source detection sensors in the Short-Wave Infrared (SWIR) bands. It reviews the state-of-the-art for traditional as well as Machine Learning (ML) approaches. The architecture and data used in such ML models will be discussed separately for methane plume segmentation and emission rate estimation. Traditionally, experts rely on labor-intensive manually adjusted methods for methane detection. However, ML approaches offer greater scalability. Our analysis reveals that ML models outperform traditional methods, particularly those based on convolutional neural networks (CNN), which are based on the U-net and transformer architectures. These ML models extract valuable information from methane-sensitive spectral data, enabling a more accurate detection. Challenges arise when comparing these methods due to variations in data, sensor specifications, and evaluation metrics. To address this, we discuss existing datasets and metrics, providing an overview of available resources and identifying open research problems. Finally, we explore potential future advances in ML, emphasizing approaches for model comparability, large dataset creation, and the European Union's forthcoming methane strategy.

Title: T-FAKE: Synthesizing Thermal Images for Facial Landmarking

• Authors: Philipp Flotho (1), Moritz Piening (2), Anna Kukleva (3), Gabriele Steidl (2) ((1) Systems Neuroscience & Neurotechnology Unit, Faculty of Medicine, Saarland University & htw saar, (2) Institute of Mathematics, Technische Universität Berlin, (3) Max Planck Institute for Informatics, Saarland Informatics Campus)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15127
• Pdf URL: https://arxiv.org/pdf/2408.15127
• Copy Paste: [[2408.15127]] T-FAKE: Synthesizing Thermal Images for Facial Landmarking(https://arxiv.org/abs/2408.15127)
• Keywords: security, biometric
• Abstract: Facial analysis is a key component in a wide range of applications such as security, autonomous driving, entertainment, and healthcare. Despite the availability of various facial RGB datasets, the thermal modality, which plays a crucial role in life sciences, medicine, and biometrics, has been largely overlooked. To address this gap, we introduce the T-FAKE dataset, a new large-scale synthetic thermal dataset with sparse and dense landmarks. To facilitate the creation of the dataset, we propose a novel RGB2Thermal loss function, which enables the transfer of thermal style to RGB faces. By utilizing the Wasserstein distance between thermal and RGB patches and the statistical analysis of clinical temperature distributions on faces, we ensure that the generated thermal images closely resemble real samples. Using RGB2Thermal style transfer based on our RGB2Thermal loss function, we create the T-FAKE dataset, a large-scale synthetic thermal dataset of faces. Leveraging our novel T-FAKE dataset, probabilistic landmark prediction, and label adaptation networks, we demonstrate significant improvements in landmark detection methods on thermal images across different landmark conventions. Our models show excellent performance with both sparse 70-point landmarks and dense 478-point landmark annotations. Our code and models are available at this https URL.

Title: How transformers learn structured data: insights from hierarchical filtering

• Authors: Jerome Garnier-Brun, Marc Mézard, Emanuele Moscato, Luca Saglietti
• Subjects: cs.LG, cond-mat.dis-nn, cond-mat.stat-mech, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15138
• Pdf URL: https://arxiv.org/pdf/2408.15138
• Copy Paste: [[2408.15138]] How transformers learn structured data: insights from hierarchical filtering(https://arxiv.org/abs/2408.15138)
• Keywords: transformer, generative
• Abstract: We introduce a hierarchical filtering procedure for generative models of sequences on trees, enabling control over the range of positional correlations in the data. Leveraging this controlled setting, we provide evidence that vanilla encoder-only transformer architectures can implement the optimal Belief Propagation algorithm on both root classification and masked language modeling tasks. Correlations at larger distances corresponding to increasing layers of the hierarchy are sequentially included as the network is trained. We analyze how the transformer layers succeed by focusing on attention maps from models trained with varying degrees of filtering. These attention maps show clear evidence for iterative hierarchical reconstruction of correlations, and we can relate these observations to a plausible implementation of the exact inference algorithm for the network sizes considered.

Title: Measuring text summarization factuality using atomic facts entailment metrics in the context of retrieval augmented generation

• Authors: N. E. Kriman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15171
• Pdf URL: https://arxiv.org/pdf/2408.15171
• Copy Paste: [[2408.15171]] Measuring text summarization factuality using atomic facts entailment metrics in the context of retrieval augmented generation(https://arxiv.org/abs/2408.15171)
• Keywords: large language model
• Abstract: The use of large language models (LLMs) has significantly increased since the introduction of ChatGPT in 2022, demonstrating their value across various applications. However, a major challenge for enterprise and commercial adoption of LLMs is their tendency to generate inaccurate information, a phenomenon known as "hallucination." This project proposes a method for estimating the factuality of a summary generated by LLMs when compared to a source text. Our approach utilizes Naive Bayes classification to assess the accuracy of the content produced.

Title: A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationships

• Authors: Gracile Astlin Pereira, Muhammad Hussain
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15178
• Pdf URL: https://arxiv.org/pdf/2408.15178
• Copy Paste: [[2408.15178]] A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationships(https://arxiv.org/abs/2408.15178)
• Keywords: transformer, segmentation
• Abstract: Transformer-based models have transformed the landscape of natural language processing (NLP) and are increasingly applied to computer vision tasks with remarkable success. These models, renowned for their ability to capture long-range dependencies and contextual information, offer a promising alternative to traditional convolutional neural networks (CNNs) in computer vision. In this review paper, we provide an extensive overview of various transformer architectures adapted for computer vision tasks. We delve into how these models capture global context and spatial relationships in images, empowering them to excel in tasks such as image classification, object detection, and segmentation. Analyzing the key components, training methodologies, and performance metrics of transformer-based models, we highlight their strengths, limitations, and recent advancements. Additionally, we discuss potential research directions and applications of transformer-based models in computer vision, offering insights into their implications for future advancements in the field.

Title: PoseWatch: A Transformer-based Architecture for Human-centric Video Anomaly Detection Using Spatio-temporal Pose Tokenization

• Authors: Ghazal Alinezhad Noghre, Armin Danesh Pazho, Hamed Tabkhi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15185
• Pdf URL: https://arxiv.org/pdf/2408.15185
• Copy Paste: [[2408.15185]] PoseWatch: A Transformer-based Architecture for Human-centric Video Anomaly Detection Using Spatio-temporal Pose Tokenization(https://arxiv.org/abs/2408.15185)
• Keywords: privacy, robust, transformer
• Abstract: Video Anomaly Detection (VAD) presents a significant challenge in computer vision, particularly due to the unpredictable and infrequent nature of anomalous events, coupled with the diverse and dynamic environments in which they occur. Human-centric VAD, a specialized area within this domain, faces additional complexities, including variations in human behavior, potential biases in data, and substantial privacy concerns related to human subjects. These issues complicate the development of models that are both robust and generalizable. To address these challenges, recent advancements have focused on pose-based VAD, which leverages human pose as a high-level feature to mitigate privacy concerns, reduce appearance biases, and minimize background interference. In this paper, we introduce PoseWatch, a novel transformer-based architecture designed specifically for human-centric pose-based VAD. PoseWatch features an innovative Spatio-Temporal Pose and Relative Pose (ST-PRP) tokenization method that enhances the representation of human motion over time, which is also beneficial for broader human behavior analysis tasks. The architecture's core, a Unified Encoder Twin Decoders (UETD) transformer, significantly improves the detection of anomalous behaviors in video data. Extensive evaluations across multiple benchmark datasets demonstrate that PoseWatch consistently outperforms existing methods, establishing a new state-of-the-art in pose-based VAD. This work not only demonstrates the efficacy of PoseWatch but also highlights the potential of integrating Natural Language Processing techniques with computer vision to advance human behavior analysis.

Title: An Investigation on The Position Encoding in Vision-Based Dynamics Prediction

• Authors: Jiageng Zhu, Hanchen Xie, Jiazhi Li, Mahyar Khayatkhoei, Wael AbdAlmageed
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15201
• Pdf URL: https://arxiv.org/pdf/2408.15201
• Copy Paste: [[2408.15201]] An Investigation on The Position Encoding in Vision-Based Dynamics Prediction(https://arxiv.org/abs/2408.15201)
• Keywords: segmentation
• Abstract: Despite the success of vision-based dynamics prediction models, which predict object states by utilizing RGB images and simple object descriptions, they were challenged by environment misalignments. Although the literature has demonstrated that unifying visual domains with both environment context and object abstract, such as semantic segmentation and bounding boxes, can effectively mitigate the visual domain misalignment challenge, discussions were focused on the abstract of environment context, and the insight of using bounding box as the object abstract is under-explored. Furthermore, we notice that, as empirical results shown in the literature, even when the visual appearance of objects is removed, object bounding boxes alone, instead of being directly fed into the network, can indirectly provide sufficient position information via the Region of Interest Pooling operation for dynamics prediction. However, previous literature overlooked discussions regarding how such position information is implicitly encoded in the dynamics prediction model. Thus, in this paper, we provide detailed studies to investigate the process and necessary conditions for encoding position information via using the bounding box as the object abstract into output features. Furthermore, we study the limitation of solely using object abstracts, such that the dynamics prediction performance will be jeopardized when the environment context varies.

Title: Can Unconfident LLM Annotations Be Used for Confident Conclusions?

• Authors: Kristina Gligorić, Tijana Zrnic, Cinoo Lee, Emmanuel J. Candès, Dan Jurafsky
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.15204
• Pdf URL: https://arxiv.org/pdf/2408.15204
• Copy Paste: [[2408.15204]] Can Unconfident LLM Annotations Be Used for Confident Conclusions?(https://arxiv.org/abs/2408.15204)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown high agreement with human raters across a variety of tasks, demonstrating potential to ease the challenges of human data collection. In computational social science (CSS), researchers are increasingly leveraging LLM annotations to complement slow and expensive human annotations. Still, guidelines for collecting and using LLM annotations, without compromising the validity of downstream conclusions, remain limited. We introduce Confidence-Driven Inference: a method that combines LLM annotations and LLM confidence indicators to strategically select which human annotations should be collected, with the goal of producing accurate statistical estimates and provably valid confidence intervals while reducing the number of human annotations needed. Our approach comes with safeguards against LLM annotations of poor quality, guaranteeing that the conclusions will be both valid and no less accurate than if we only relied on human annotations. We demonstrate the effectiveness of Confidence-Driven Inference over baselines in statistical estimation tasks across three CSS settings--text politeness, stance, and bias--reducing the needed number of human annotations by over 25% in each. Although we use CSS settings for demonstration, Confidence-Driven Inference can be used to estimate most standard quantities across a broad range of NLP problems.

Title: Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation

• Authors: Jian Hu, Jiayi Lin, Junchi Yan, Shaogang Gong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15205
• Pdf URL: https://arxiv.org/pdf/2408.15205
• Copy Paste: [[2408.15205]] Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation(https://arxiv.org/abs/2408.15205)
• Keywords: large language model, segmentation
• Abstract: Promptable segmentation typically requires instance-specific manual prompts to guide the segmentation of each desired object. To minimize such a need, task-generic promptable segmentation has been introduced, which employs a single task-generic prompt to segment various images of different objects in the same task. Current methods use Multimodal Large Language Models (MLLMs) to reason detailed instance-specific prompts from a task-generic prompt for improving segmentation accuracy. The effectiveness of this segmentation heavily depends on the precision of these derived prompts. However, MLLMs often suffer hallucinations during reasoning, resulting in inaccurate prompting. While existing methods focus on eliminating hallucinations to improve a model, we argue that MLLM hallucinations can reveal valuable contextual insights when leveraged correctly, as they represent pre-trained large-scale knowledge beyond individual images. In this paper, we utilize hallucinations to mine task-related information from images and verify its accuracy for enhancing precision of the generated prompts. Specifically, we introduce an iterative Prompt-Mask Cycle generation framework (ProMaC) with a prompt generator and a mask generator.The prompt generator uses a multi-scale chain of thought prompting, initially exploring hallucinations for extracting extended contextual knowledge on a test image.These hallucinations are then reduced to formulate precise instance-specific prompts, directing the mask generator to produce masks that are consistent with task semantics by mask semantic alignment. The generated masks iteratively induce the prompt generator to focus more on task-relevant image areas and reduce irrelevant hallucinations, resulting jointly in better prompts and masks. Experiments on 5 benchmarks demonstrate the effectiveness of ProMaC. Code given in this https URL.

Title: FRAMER/Miu: Tagged Pointer-based Capability and Fundamental Cost of Memory Safety & Coherence (Position Paper)

• Authors: Myoung Jin Nam
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.15219
• Pdf URL: https://arxiv.org/pdf/2408.15219
• Copy Paste: [[2408.15219]] FRAMER/Miu: Tagged Pointer-based Capability and Fundamental Cost of Memory Safety & Coherence (Position Paper)(https://arxiv.org/abs/2408.15219)
• Keywords: security, protect, attack
• Abstract: Ensuring system correctness, such as memory safety, can eliminate security vulnerabilities that attackers could exploit in the first place. However, high and unpredictable performance degradation remains a primary challenge. Recognizing that it is extremely difficult to achieve complete system correctness for production deployment, researchers make trade-offs between performance, detection coverage, interoperability, precision, and detection timing. This research strikes a balance between comprehensive system protection and the costs required to obtain it, identifies the desirable roles of software and hardware, and presents a tagged pointer-based capability system as a stand-alone software solution and a prototype for future hardware design. This paper presents follow-up plans for the FRAMER/Miu generic framework to achieve these goals.

Title: LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet

• Authors: Nathaniel Li, Ziwen Han, Ian Steneker, Willow Primack, Riley Goodside, Hugh Zhang, Zifan Wang, Cristina Menghini, Summer Yue
• Subjects: cs.LG, cs.CL, cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.15221
• Pdf URL: https://arxiv.org/pdf/2408.15221
• Copy Paste: [[2408.15221]] LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet(https://arxiv.org/abs/2408.15221)
• Keywords: security, defense, attack, robust, large language model
• Abstract: Recent large language model (LLM) defenses have greatly improved models' ability to refuse harmful queries, even when adversarially attacked. However, LLM defenses are primarily evaluated against automated adversarial attacks in a single turn of conversation, an insufficient threat model for real-world malicious use. We demonstrate that multi-turn human jailbreaks uncover significant vulnerabilities, exceeding 70% attack success rate (ASR) on HarmBench against defenses that report single-digit ASRs with automated single-turn attacks. Human jailbreaks also reveal vulnerabilities in machine unlearning defenses, successfully recovering dual-use biosecurity knowledge from unlearned models. We compile these results into Multi-Turn Human Jailbreaks (MHJ), a dataset of 2,912 prompts across 537 multi-turn jailbreaks. We publicly release MHJ alongside a compendium of jailbreak tactics developed across dozens of commercial red teaming engagements, supporting research towards stronger LLM defenses.

Title: DCT-CryptoNets: Scaling Private Inference in the Frequency Domain

• Authors: Arjun Roy, Kaushik Roy
• Subjects: cs.CR, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15231
• Pdf URL: https://arxiv.org/pdf/2408.15231
• Copy Paste: [[2408.15231]] DCT-CryptoNets: Scaling Private Inference in the Frequency Domain(https://arxiv.org/abs/2408.15231)
• Keywords: privacy
• Abstract: The convergence of fully homomorphic encryption (FHE) and machine learning offers unprecedented opportunities for private inference of sensitive data. FHE enables computation directly on encrypted data, safeguarding the entire machine learning pipeline, including data and model confidentiality. However, existing FHE-based implementations for deep neural networks face significant challenges in computational cost, latency, and scalability, limiting their practical deployment. This paper introduces DCT-CryptoNets, a novel approach that leverages frequency-domain learning to tackle these issues. Our method operates directly in the frequency domain, utilizing the discrete cosine transform (DCT) commonly employed in JPEG compression. This approach is inherently compatible with remote computing services, where images are usually transmitted and stored in compressed formats. DCT-CryptoNets reduces the computational burden of homomorphic operations by focusing on perceptually relevant low-frequency components. This is demonstrated by substantial latency reduction of up to 5.3$\times$ compared to prior work on image classification tasks, including a novel demonstration of ImageNet inference within 2.5 hours, down from 12.5 hours compared to prior work on equivalent compute resources. Moreover, DCT-CryptoNets improves the reliability of encrypted accuracy by reducing variability (e.g., from $\pm$2.5\% to $\pm$1.0\% on ImageNet). This study demonstrates a promising avenue for achieving efficient and practical privacy-preserving deep learning on high resolution images seen in real-world applications.

Title: Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations

• Authors: Yucheng Jiang, Yijia Shao, Dekun Ma, Sina J. Semnani, Monica S. Lam
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.15232
• Pdf URL: https://arxiv.org/pdf/2408.15232
• Copy Paste: [[2408.15232]] Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations(https://arxiv.org/abs/2408.15232)
• Keywords: generative
• Abstract: While language model (LM)-powered chatbots and generative search engines excel at answering concrete queries, discovering information in the terrain of unknown unknowns remains challenging for users. To emulate the common educational scenario where children/students learn by listening to and participating in conversations of their parents/teachers, we create Collaborative STORM (Co-STORM). Unlike QA systems that require users to ask all the questions, Co-STORM lets users observe and occasionally steer the discourse among several LM agents. The agents ask questions on the user's behalf, allowing the user to discover unknown unknowns serendipitously. To facilitate user interaction, Co-STORM assists users in tracking the discourse by organizing the uncovered information into a dynamic mind map, ultimately generating a comprehensive report as takeaways. For automatic evaluation, we construct the WildSeek dataset by collecting real information-seeking records with user goals. Co-STORM outperforms baseline methods on both discourse trace and report quality. In a further human evaluation, 70% of participants prefer Co-STORM over a search engine, and 78% favor it over a RAG chatbot.

Title: The Mamba in the Llama: Distilling and Accelerating Hybrid Models

• Authors: Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, Tri Dao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15237
• Pdf URL: https://arxiv.org/pdf/2408.15237
• Copy Paste: [[2408.15237]] The Mamba in the Llama: Distilling and Accelerating Hybrid Models(https://arxiv.org/abs/2408.15237)
• Keywords: transformer
• Abstract: Linear RNN architectures, like Mamba, can be competitive with Transformer models in language modeling while having advantageous deployment characteristics. Given the focus on training large-scale Transformer models, we consider the challenge of converting these pretrained models for deployment. We demonstrate that it is feasible to distill large Transformers into linear RNNs by reusing the linear projection weights from attention layers with academic GPU resources. The resulting hybrid model, which incorporates a quarter of the attention layers, achieves performance comparable to the original Transformer in chat benchmarks and outperforms open-source hybrid Mamba models trained from scratch with trillions of tokens in both chat benchmarks and general benchmarks. Moreover, we introduce a hardware-aware speculative decoding algorithm that accelerates the inference speed of Mamba and hybrid models. Overall we show how, with limited computation resources, we can remove many of the original attention layers and generate from the resulting model more efficiently. Our top-performing model, distilled from Llama3-8B-Instruct, achieves a 29.61 length-controlled win rate on AlpacaEval 2 against GPT-4 and 7.35 on MT-Bench, surpassing the best instruction-tuned linear RNN model.

Title: Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation

• Authors: Xiaojuan Wang, Boyang Zhou, Brian Curless, Ira Kemelmacher-Shlizerman, Aleksander Holynski, Steven M. Seitz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15239
• Pdf URL: https://arxiv.org/pdf/2408.15239
• Copy Paste: [[2408.15239]] Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation(https://arxiv.org/abs/2408.15239)
• Keywords: diffusion, generative
• Abstract: We present a method for generating video sequences with coherent motion between a pair of input key frames. We adapt a pretrained large-scale image-to-video diffusion model (originally trained to generate videos moving forward in time from a single input image) for key frame interpolation, i.e., to produce a video in between two input frames. We accomplish this adaptation through a lightweight fine-tuning technique that produces a version of the model that instead predicts videos moving backwards in time from a single input image. This model (along with the original forward-moving model) is subsequently used in a dual-directional diffusion sampling process that combines the overlapping model estimates starting from each of the two keyframes. Our experiments show that our method outperforms both existing diffusion-based methods and traditional frame interpolation techniques.

Title: Generative Verifiers: Reward Modeling as Next-Token Prediction

• Authors: Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, Rishabh Agarwal
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15240
• Pdf URL: https://arxiv.org/pdf/2408.15240
• Copy Paste: [[2408.15240]] Generative Verifiers: Reward Modeling as Next-Token Prediction(https://arxiv.org/abs/2408.15240)
• Keywords: generative, large language model
• Abstract: Verifiers or reward models are often used to enhance the reasoning performance of large language models (LLMs). A common approach is the Best-of-N method, where N candidate solutions generated by the LLM are ranked by a verifier, and the best one is selected. While LLM-based verifiers are typically trained as discriminative classifiers to score solutions, they do not utilize the text generation capabilities of pretrained LLMs. To overcome this limitation, we instead propose training verifiers using the ubiquitous next-token prediction objective, jointly on verification and solution generation. Compared to standard verifiers, such generative verifiers (GenRM) can benefit from several advantages of LLMs: they integrate seamlessly with instruction tuning, enable chain-of-thought reasoning, and can utilize additional inference-time compute via majority voting for better verification. We demonstrate that when using Gemma-based verifiers on algorithmic and grade-school math reasoning tasks, GenRM outperforms discriminative verifiers and LLM-as-a-Judge, showing a 16-64% improvement in the percentage of problems solved with Best-of-N. Furthermore, we show that GenRM scales favorably across dataset size, model capacity, and inference-time compute.

Title: GenRec: Unifying Video Generation and Recognition with Diffusion Models

• Authors: Zejia Weng, Xitong Yang, Zhen Xing, Zuxuan Wu, Yu-Gang Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15241
• Pdf URL: https://arxiv.org/pdf/2408.15241
• Copy Paste: [[2408.15241]] GenRec: Unifying Video Generation and Recognition with Diffusion Models(https://arxiv.org/abs/2408.15241)
• Keywords: robust, diffusion, generative
• Abstract: Video diffusion models are able to generate high-quality videos by learning strong spatial-temporal priors on large-scale datasets. In this paper, we aim to investigate whether such priors derived from a generative process are suitable for video recognition, and eventually joint optimization of generation and recognition. Building upon Stable Video Diffusion, we introduce GenRec, the first unified framework trained with a random-frame conditioning process so as to learn generalized spatial-temporal representations. The resulting framework can naturally supports generation and recognition, and more importantly is robust even when visual inputs contain limited information. Extensive experiments demonstrate the efficacy of GenRec for both recognition and generation. In particular, GenRec achieves competitive recognition performance, offering 75.8% and 87.2% accuracy on SSV2 and K400, respectively. GenRec also performs the best class-conditioned image-to-video generation results, achieving 46.5 and 49.3 FVD scores on SSV2 and EK-100 datasets. Furthermore, GenRec demonstrates extraordinary robustness in scenarios that only limited frames can be observed.

Title: Drone-assisted Road Gaussian Splatting with Cross-view Uncertainty

• Authors: Saining Zhang, Baijun Ye, Xiaoxue Chen, Yuantao Chen, Zongzheng Zhang, Cheng Peng, Yongliang Shi, Hao Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15242
• Pdf URL: https://arxiv.org/pdf/2408.15242
• Copy Paste: [[2408.15242]] Drone-assisted Road Gaussian Splatting with Cross-view Uncertainty(https://arxiv.org/abs/2408.15242)
• Keywords: robust
• Abstract: Robust and realistic rendering for large-scale road scenes is essential in autonomous driving simulation. Recently, 3D Gaussian Splatting (3D-GS) has made groundbreaking progress in neural rendering, but the general fidelity of large-scale road scene renderings is often limited by the input imagery, which usually has a narrow field of view and focuses mainly on the street-level local area. Intuitively, the data from the drone's perspective can provide a complementary viewpoint for the data from the ground vehicle's perspective, enhancing the completeness of scene reconstruction and rendering. However, training naively with aerial and ground images, which exhibit large view disparity, poses a significant convergence challenge for 3D-GS, and does not demonstrate remarkable improvements in performance on road views. In order to enhance the novel view synthesis of road views and to effectively use the aerial information, we design an uncertainty-aware training method that allows aerial images to assist in the synthesis of areas where ground images have poor learning outcomes instead of weighting all pixels equally in 3D-GS training like prior work did. We are the first to introduce the cross-view uncertainty to 3D-GS by matching the car-view ensemble-based rendering uncertainty to aerial images, weighting the contribution of each pixel to the training process. Additionally, to systematically quantify evaluation metrics, we assemble a high-quality synthesized dataset comprising both aerial and ground images for road scenes.