2024-07-24

Title: A Survey on Trustworthiness in Foundation Models for Medical Image Analysis

• Authors: Congzhen Shi, Ryan Rezai, Jiaxi Yang, Qi Dou, Xiaoxiao Li
• Subjects: cs.CV, cs.AI, cs.CY, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15851
• Pdf URL: https://arxiv.org/pdf/2407.15851
• Copy Paste: [[2407.15851]] A Survey on Trustworthiness in Foundation Models for Medical Image Analysis(https://arxiv.org/abs/2407.15851)
• Keywords: privacy, robust, fair, explainability, segmentation
• Abstract: The rapid advancement of foundation models in medical imaging represents a significant leap toward enhancing diagnostic accuracy and personalized treatment. However, the deployment of foundation models in healthcare necessitates a rigorous examination of their trustworthiness, encompassing privacy, robustness, reliability, explainability, and fairness. The current body of survey literature on foundation models in medical imaging reveals considerable gaps, particularly in the area of trustworthiness. Additionally, extant surveys on the trustworthiness of foundation models fail to address their specific variations and applications within the medical imaging domain. This survey paper reviews the current research on foundation models in the major medical imaging applications, with a focus on segmentation, medical report generation, medical question and answering (Q&A), and disease diagnosis, which includes trustworthiness discussion in their manuscripts. We explore the complex challenges of making foundation models for medical image analysis trustworthy, associated with each application, and summarize the current concerns and strategies to enhance trustworthiness. Furthermore, we explore the future promises of these models in revolutionizing patient care. Our analysis underscores the imperative for advancing towards trustworthy AI in medical image analysis, advocating for a balanced approach that fosters innovation while ensuring ethical and equitable healthcare delivery.

Title: Data Poisoning Attacks in Intelligent Transportation Systems: A Survey

• Authors: Feilong Wang, Xin Wang, Xuegang Ban
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15855
• Pdf URL: https://arxiv.org/pdf/2407.15855
• Copy Paste: [[2407.15855]] Data Poisoning Attacks in Intelligent Transportation Systems: A Survey(https://arxiv.org/abs/2407.15855)
• Keywords: security, attack
• Abstract: Emerging technologies drive the ongoing transformation of Intelligent Transportation Systems (ITS). This transformation has given rise to cybersecurity concerns, among which data poisoning attack emerges as a new threat as ITS increasingly relies on data. In data poisoning attacks, attackers inject malicious perturbations into datasets, potentially leading to inaccurate results in offline learning and real-time decision-making processes. This paper concentrates on data poisoning attack models against ITS. We identify the main ITS data sources vulnerable to poisoning attacks and application scenarios that enable staging such attacks. A general framework is developed following rigorous study process from cybersecurity but also considering specific ITS application needs. Data poisoning attacks against ITS are reviewed and categorized following the framework. We then discuss the current limitations of these attack models and the future research directions. Our work can serve as a guideline to better understand the threat of data poisoning attacks against ITS applications, while also giving a perspective on the future development of trustworthy ITS.

Title: BoRA: Bayesian Hierarchical Low-Rank Adaption for Multi-task Large Language Models

• Authors: Simen Eide, Arnoldo Frigessi
• Subjects: cs.LG, cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.15857
• Pdf URL: https://arxiv.org/pdf/2407.15857
• Copy Paste: [[2407.15857]] BoRA: Bayesian Hierarchical Low-Rank Adaption for Multi-task Large Language Models(https://arxiv.org/abs/2407.15857)
• Keywords: large language model
• Abstract: This paper introduces Bayesian Hierarchical Low-Rank Adaption (BoRA), a novel method for finetuning multi-task Large Language Models (LLMs). Current finetuning approaches, such as Low-Rank Adaption (LoRA), perform exeptionally well in reducing training parameters and memory usage but face limitations when applied to multiple similar tasks. Practitioners usually have to choose between training separate models for each task or a single model for all tasks, both of which come with trade-offs in specialization and data utilization. BoRA addresses these trade-offs by leveraging a Bayesian hierarchical model that allows tasks to share information through global hierarchical priors. This enables tasks with limited data to benefit from the overall structure derived from related tasks while allowing tasks with more data to specialize. Our experimental results show that BoRA outperforms both individual and unified model approaches, achieving lower perplexity and better generalization across tasks. This method provides a scalable and efficient solution for multi-task LLM finetuning, with significant practical implications for diverse applications.

Title: Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Survey

• Authors: Chenyu Zhang, Mingwang Hu, Wenhui Li, Lanjun Wang
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.15861
• Pdf URL: https://arxiv.org/pdf/2407.15861
• Copy Paste: [[2407.15861]] Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Survey(https://arxiv.org/abs/2407.15861)
• Keywords: defense, attack, robust, diffusion
• Abstract: Recently, the text-to-image diffusion model has gained considerable attention from the community due to its exceptional image generation capability. A representative model, Stable Diffusion, amassed more than 10 million users within just two months of its release. This surge in popularity has facilitated studies on the robustness and safety of the model, leading to the proposal of various adversarial attack methods. Simultaneously, there has been a marked increase in research focused on defense methods to improve the robustness and safety of these models. In this survey, we provide a comprehensive review of the literature on adversarial attacks and defenses targeting text-to-image diffusion models. We begin with an overview of text-to-image diffusion models, followed by an introduction to a taxonomy of adversarial attacks and an in-depth review of existing attack methods. We then present a detailed analysis of current defense methods that improve model robustness and safety. Finally, we discuss ongoing challenges and explore promising future research directions. For a complete list of the adversarial attack and defense methods covered in this survey, please refer to our curated repository at this https URL.

Title: Performance Evaluation of Lightweight Open-source Large Language Models in Pediatric Consultations: A Comparative Analysis

• Authors: Qiuhong Wei, Ying Cui, Mengwei Ding, Yanqin Wang, Lingling Xiang, Zhengxiong Yao, Ceran Chen, Ying Long, Zhezhen Jin, Ximing Xu
• Subjects: cs.LG, cs.AI, cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.15862
• Pdf URL: https://arxiv.org/pdf/2407.15862
• Copy Paste: [[2407.15862]] Performance Evaluation of Lightweight Open-source Large Language Models in Pediatric Consultations: A Comparative Analysis(https://arxiv.org/abs/2407.15862)
• Keywords: privacy, large language model
• Abstract: Large language models (LLMs) have demonstrated potential applications in medicine, yet data privacy and computational burden limit their deployment in healthcare institutions. Open-source and lightweight versions of LLMs emerge as potential solutions, but their performance, particularly in pediatric settings remains underexplored. In this cross-sectional study, 250 patient consultation questions were randomly selected from a public online medical forum, with 10 questions from each of 25 pediatric departments, spanning from December 1, 2022, to October 30, 2023. Two lightweight open-source LLMs, ChatGLM3-6B and Vicuna-7B, along with a larger-scale model, Vicuna-13B, and the widely-used proprietary ChatGPT-3.5, independently answered these questions in Chinese between November 1, 2023, and November 7, 2023. To assess reproducibility, each inquiry was replicated once. We found that ChatGLM3-6B demonstrated higher accuracy and completeness than Vicuna-13B and Vicuna-7B (P < .001), but all were outperformed by ChatGPT-3.5. ChatGPT-3.5 received the highest ratings in accuracy (65.2%) compared to ChatGLM3-6B (41.2%), Vicuna-13B (11.2%), and Vicuna-7B (4.4%). Similarly, in completeness, ChatGPT-3.5 led (78.4%), followed by ChatGLM3-6B (76.0%), Vicuna-13B (34.8%), and Vicuna-7B (22.0%) in highest ratings. ChatGLM3-6B matched ChatGPT-3.5 in readability, both outperforming Vicuna models (P < .001). In terms of empathy, ChatGPT-3.5 outperformed the lightweight LLMs (P < .001). In safety, all models performed comparably well (P > .05), with over 98.4% of responses being rated as safe. Repetition of inquiries confirmed these findings. In conclusion, Lightweight LLMs demonstrate promising application in pediatric healthcare. However, the observed gap between lightweight and large-scale proprietary LLMs underscores the need for continued development efforts.

Title: SmartQuant: CXL-based AI Model Store in Support of Runtime Configurable Weight Quantization

• Authors: Rui Xie, Asad Ul Haq, Linsen Ma, Krystal Sun, Sanchari Sen, Swagath Venkataramani, Liu Liu, Tong Zhang
• Subjects: cs.LG, cs.AI, cs.AR
• Abstract URL: https://arxiv.org/abs/2407.15866
• Pdf URL: https://arxiv.org/pdf/2407.15866
• Copy Paste: [[2407.15866]] SmartQuant: CXL-based AI Model Store in Support of Runtime Configurable Weight Quantization(https://arxiv.org/abs/2407.15866)
• Keywords: transformer, generative
• Abstract: Recent studies have revealed that, during the inference on generative AI models such as transformer, the importance of different weights exhibits substantial context-dependent variations. This naturally manifests a promising potential of adaptively configuring weight quantization to improve the generative AI inference efficiency. Although configurable weight quantization can readily leverage the hardware support of variable-precision arithmetics in modern GPU and AI accelerators, little prior research has studied how one could exploit variable weight quantization to proportionally improve the AI model memory access speed and energy efficiency. Motivated by the rapidly maturing CXL ecosystem, this work develops a CXL-based design solution to fill this gap. The key is to allow CXL memory controllers play an active role in supporting and exploiting runtime configurable weight quantization. Using transformer as a representative generative AI model, we carried out experiments that well demonstrate the effectiveness of the proposed design solution.

Title: A Survey on Differential Privacy for SpatioTemporal Data in Transportation Research

• Authors: Rahul Bhadani
• Subjects: cs.CR, cs.CY, cs.LG, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.15868
• Pdf URL: https://arxiv.org/pdf/2407.15868
• Copy Paste: [[2407.15868]] A Survey on Differential Privacy for SpatioTemporal Data in Transportation Research(https://arxiv.org/abs/2407.15868)
• Keywords: privacy
• Abstract: With low-cost computing devices, improved sensor technology, and the proliferation of data-driven algorithms, we have more data than we know what to do with. In transportation, we are seeing a surge in spatiotemporal data collection. At the same time, concerns over user privacy have led to research on differential privacy in applied settings. In this paper, we look at some recent developments in differential privacy in the context of spatiotemporal data. Spatiotemporal data contain not only features about users but also the geographical locations of their frequent visits. Hence, the public release of such data carries extreme risks. To address the need for such data in research and inference without exposing private information, significant work has been proposed. This survey paper aims to summarize these efforts and provide a review of differential privacy mechanisms and related software. We also discuss related work in transportation where such mechanisms have been applied. Furthermore, we address the challenges in the deployment and mass adoption of differential privacy in transportation spatiotemporal data for downstream analyses.

Title: Long Input Sequence Network for Long Time Series Forecasting

• Authors: Chao Ma, Yikai Hou, Xiang Li, Yinggang Sun, Haining Yu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15869
• Pdf URL: https://arxiv.org/pdf/2407.15869
• Copy Paste: [[2407.15869]] Long Input Sequence Network for Long Time Series Forecasting(https://arxiv.org/abs/2407.15869)
• Keywords: interpretability
• Abstract: Short fixed-length inputs are the main bottleneck of deep learning methods in long time-series forecasting tasks. Prolonging input length causes overfitting, rapidly deteriorating accuracy. Our research indicates that the overfitting is a combination reaction of the multi-scale pattern coupling in time series and the fixed focusing scale of current models. First, we find that the patterns exhibited by a time series across various scales are reflective of its multi-periodic nature, where each scale corresponds to specific period length. Second, We find that the token size predominantly dictates model behavior, as it determines the scale at which the model focuses and the context size it can accommodate. Our idea is to decouple the multi-scale temporal patterns of time series and to model each pattern with its corresponding period length as token size. We introduced a novel series-decomposition module(MPSD), and a Multi-Token Pattern Recognition neural network(MTPR), enabling the model to handle \textit{inputs up to $10\times$ longer}. Sufficient context enhances performance(\textit{38% maximum precision improvement}), and the decoupling approach offers \textit{Low complexity($0.22\times$ cost)} and \textit{high interpretability}.

Title: Semantic Prototypes: Enhancing Transparency Without Black Boxes

• Authors: Orfeas Menis-Mastromichalakis, Giorgos Filandrianos, Jason Liartis, Edmund Dervakos, Giorgos Stamou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15871
• Pdf URL: https://arxiv.org/pdf/2407.15871
• Copy Paste: [[2407.15871]] Semantic Prototypes: Enhancing Transparency Without Black Boxes(https://arxiv.org/abs/2407.15871)
• Keywords: interpretability, explainability
• Abstract: As machine learning (ML) models and datasets increase in complexity, the demand for methods that enhance explainability and interpretability becomes paramount. Prototypes, by encapsulating essential characteristics within data, offer insights that enable tactical decision-making and enhance transparency. Traditional prototype methods often rely on sub-symbolic raw data and opaque latent spaces, reducing explainability and increasing the risk of misinterpretations. This paper presents a novel framework that utilizes semantic descriptions to define prototypes and provide clear explanations, effectively addressing the shortcomings of conventional methods. Our approach leverages concept-based descriptions to cluster data on the semantic level, ensuring that prototypes not only represent underlying properties intuitively but are also straightforward to interpret. Our method simplifies the interpretative process and effectively bridges the gap between complex data structures and human cognitive processes, thereby enhancing transparency and fostering trust. Our approach outperforms existing widely-used prototype methods in facilitating human understanding and informativeness, as validated through a user survey.

Title: A reinforcement learning strategy to automate and accelerate h/p-multigrid solvers

• Authors: David Huergo, Laura Alonso, Saumitra Joshi, Adrian Juanicoteca, Gonzalo Rubio, Esteban Ferrer
• Subjects: cs.LG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2407.15872
• Pdf URL: https://arxiv.org/pdf/2407.15872
• Copy Paste: [[2407.15872]] A reinforcement learning strategy to automate and accelerate h/p-multigrid solvers(https://arxiv.org/abs/2407.15872)
• Keywords: robust, diffusion
• Abstract: We explore a reinforcement learning strategy to automate and accelerate h/p-multigrid methods in high-order solvers. Multigrid methods are very efficient but require fine-tuning of numerical parameters, such as the number of smoothing sweeps per level and the correction fraction (i.e., proportion of the corrected solution that is transferred from a coarser grid to a finer grid). The objective of this paper is to use a proximal policy optimization algorithm to automatically tune the multigrid parameters and, by doing so, improve stability and efficiency of the h/p-multigrid strategy. Our findings reveal that the proposed reinforcement learning h/p-multigrid approach significantly accelerates and improves the robustness of steady-state simulations for one dimensional advection-diffusion and nonlinear Burgers' equations, when discretized using high-order h/p methods, on uniform and nonuniform grids.

Title: CRMSP: A Semi-supervised Approach for Key Information Extraction with Class-Rebalancing and Merged Semantic Pseudo-Labeling

• Authors: Qi Zhang, Yonghong Song, Pengcheng Guo, Yangyang Hui
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15873
• Pdf URL: https://arxiv.org/pdf/2407.15873
• Copy Paste: [[2407.15873]] CRMSP: A Semi-supervised Approach for Key Information Extraction with Class-Rebalancing and Merged Semantic Pseudo-Labeling(https://arxiv.org/abs/2407.15873)
• Keywords: extraction
• Abstract: There is a growing demand in the field of KIE (Key Information Extraction) to apply semi-supervised learning to save manpower and costs, as training document data using fully-supervised methods requires labor-intensive manual annotation. The main challenges of applying SSL in the KIE are (1) underestimation of the confidence of tail classes in the long-tailed distribution and (2) difficulty in achieving intra-class compactness and inter-class separability of tail features. To address these challenges, we propose a novel semi-supervised approach for KIE with Class-Rebalancing and Merged Semantic Pseudo-Labeling (CRMSP). Firstly, the Class-Rebalancing Pseudo-Labeling (CRP) module introduces a reweighting factor to rebalance pseudo-labels, increasing attention to tail classes. Secondly, we propose the Merged Semantic Pseudo-Labeling (MSP) module to cluster tail features of unlabeled data by assigning samples to Merged Prototypes (MP). Additionally, we designed a new contrastive loss specifically for MSP. Extensive experimental results on three well-known benchmarks demonstrate that CRMSP achieves state-of-the-art performance. Remarkably, CRMSP achieves 3.24% f1-score improvement over state-of-the-art on the CORD.

Title: Blockchain in Healthcare: Implementing Hyperledger Fabric for Electronic Health Records at Frere Provincial Hospital

• Authors: Abayomi Agbeyangi, Olukayode Oki, Aphelele Mgidi
• Subjects: cs.CR, cs.ET
• Abstract URL: https://arxiv.org/abs/2407.15876
• Pdf URL: https://arxiv.org/pdf/2407.15876
• Copy Paste: [[2407.15876]] Blockchain in Healthcare: Implementing Hyperledger Fabric for Electronic Health Records at Frere Provincial Hospital(https://arxiv.org/abs/2407.15876)
• Keywords: secure, security
• Abstract: As healthcare systems worldwide continue to grapple with the challenges of interoperability, data security, and accessibility, integrating emerging technologies becomes imperative. This paper investigates the implementation of blockchain technology, specifically Hyperledger Fabric, for Electronic Health Records (EHR) management at Frere Hospital in the Eastern Cape province of South Africa. The paper examines the benefits and challenges of integrating blockchain into healthcare information systems. Hyperledger Fabric's modular architecture is harnessed to create a secure, transparent, and decentralized platform for storing, managing, and sharing EHRs among stakeholders. The study used a mixed-methods approach, integrating case studies and data collection methods through observation and informal questions, with the specific goal of understanding current record management methods and challenges. This method offers practical insights and validates the approach. The result demonstrates the role of blockchain in transforming healthcare, framed within a rigorous exploration and analysis. The findings of this study have broader implications for healthcare institutions seeking advanced solutions to address the persistent challenges in electronic health record management. Ultimately, the research underscores the transformative potential of blockchain technology in healthcare settings, fostering trust, security, and efficiency in the management of sensitive patient data.

Title: Decentralized Federated Anomaly Detection in Smart Grids: A P2P Gossip Approach

• Authors: Muhammad Akbar Husnoo, Adnan Anwar, Md Enamul Haque, A. N. Mahmood
• Subjects: cs.CR, cs.AI, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15879
• Pdf URL: https://arxiv.org/pdf/2407.15879
• Copy Paste: [[2407.15879]] Decentralized Federated Anomaly Detection in Smart Grids: A P2P Gossip Approach(https://arxiv.org/abs/2407.15879)
• Keywords: security, privacy, attack, robust, federate
• Abstract: The increasing security and privacy concerns in the Smart Grid sector have led to a significant demand for robust intrusion detection systems within critical smart grid infrastructure. To address the challenges posed by privacy preservation and decentralized power system zones with distinct data ownership, Federated Learning (FL) has emerged as a promising privacy-preserving solution which facilitates collaborative training of attack detection models without necessitating the sharing of raw data. However, FL presents several implementation limitations in the power system domain due to its heavy reliance on a centralized aggregator and the risks of privacy leakage during model update transmission. To overcome these technical bottlenecks, this paper introduces a novel decentralized federated anomaly detection scheme based on two main gossip protocols namely Random Walk and Epidemic. Our findings indicate that the Random Walk protocol exhibits superior performance compared to the Epidemic protocol, highlighting its efficacy in decentralized federated learning environments. Experimental validation of the proposed framework utilizing publicly available industrial control systems datasets demonstrates superior attack detection accuracy while safeguarding data confidentiality and mitigating the impact of communication latency and stragglers. Furthermore, our approach yields a notable 35% improvement in training time compared to conventional FL, underscoring the efficacy and robustness of our decentralized learning method.

Title: Diff4VS: HIV-inhibiting Molecules Generation with Classifier Guidance Diffusion for Virtual Screening

• Authors: Jiaqing Lyu, Changjie Chen, Bing Liang, Yijia Zhang
• Subjects: cs.LG, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2407.15880
• Pdf URL: https://arxiv.org/pdf/2407.15880
• Copy Paste: [[2407.15880]] Diff4VS: HIV-inhibiting Molecules Generation with Classifier Guidance Diffusion for Virtual Screening(https://arxiv.org/abs/2407.15880)
• Keywords: diffusion, generative
• Abstract: The AIDS epidemic has killed 40 million people and caused serious global problems. The identification of new HIV-inhibiting molecules is of great importance for combating the AIDS epidemic. Here, the Classifier Guidance Diffusion model and ligand-based virtual screening strategy are combined to discover potential HIV-inhibiting molecules for the first time. We call it Diff4VS. An extra classifier is trained using the HIV molecule dataset, and the gradient of the classifier is used to guide the Diffusion to generate HIV-inhibiting molecules. Experiments show that Diff4VS can generate more candidate HIV-inhibiting molecules than other methods. Inspired by ligand-based virtual screening, a new metric DrugIndex is proposed. The DrugIndex is the ratio of the proportion of candidate drug molecules in the generated molecule to the proportion of candidate drug molecules in the training set. DrugIndex provides a new evaluation method for evolving molecular generative models from a pharmaceutical perspective. Besides, we report a new phenomenon observed when using molecule generation models for virtual screening. Compared to real molecules, the generated molecules have a lower proportion that is highly similar to known drug molecules. We call it Degradation in molecule generation. Based on the data analysis, the Degradation may result from the difficulty of generating molecules with a specific structure in the generative model. Our research contributes to the application of generative models in drug design from method, metric, and phenomenon analysis.

Title: CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models

• Authors: Zheng Chong, Xiao Dong, Haoxiang Li, Shiyue Zhang, Wenqing Zhang, Xujie Zhang, Hanqing Zhao, Xiaodan Liang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15886
• Pdf URL: https://arxiv.org/pdf/2407.15886
• Copy Paste: [[2407.15886]] CatVTON: Concatenation Is All You Need for Virtual Try-On with Diffusion Models(https://arxiv.org/abs/2407.15886)
• Keywords: diffusion
• Abstract: Virtual try-on methods based on diffusion models achieve realistic try-on effects but often replicate the backbone network as a ReferenceNet or use additional image encoders to process condition inputs, leading to high training and inference costs. In this work, we rethink the necessity of ReferenceNet and image encoders and innovate the interaction between garment and person by proposing CatVTON, a simple and efficient virtual try-on diffusion model. CatVTON facilitates the seamless transfer of in-shop or worn garments of any category to target persons by simply concatenating them in spatial dimensions as inputs. The efficiency of our model is demonstrated in three aspects: (1) Lightweight network: Only the original diffusion modules are used, without additional network modules. The text encoder and cross-attentions for text injection in the backbone are removed, reducing the parameters by 167.02M. (2) Parameter-efficient training: We identified the try-on relevant modules through experiments and achieved high-quality try-on effects by training only 49.57M parameters, approximately 5.51 percent of the backbone network's parameters. (3) Simplified inference: CatVTON eliminates all unnecessary conditions and preprocessing steps, including pose estimation, human parsing, and text input, requiring only a garment reference, target person image, and mask for the virtual try-on process. Extensive experiments demonstrate that CatVTON achieves superior qualitative and quantitative results with fewer prerequisites and trainable parameters than baseline methods. Furthermore, CatVTON shows good generalization in in-the-wild scenarios despite using open-source datasets with only 73K samples.

Title: RazorAttention: Efficient KV Cache Compression Through Retrieval Heads

• Authors: Hanlin Tang, Yang Lin, Jing Lin, Qingsen Han, Shikuan Hong, Yiwu Yao, Gongyi Wang
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15891
• Pdf URL: https://arxiv.org/pdf/2407.15891
• Copy Paste: [[2407.15891]] RazorAttention: Efficient KV Cache Compression Through Retrieval Heads(https://arxiv.org/abs/2407.15891)
• Keywords: large language model
• Abstract: The memory and computational demands of Key-Value (KV) cache present significant challenges for deploying long-context language models. Previous approaches attempt to mitigate this issue by selectively dropping tokens, which irreversibly erases critical information that might be needed for future queries. In this paper, we propose a novel compression technique for KV cache that preserves all token information. Our investigation reveals that: i) Most attention heads primarily focus on the local context; ii) Only a few heads, denoted as retrieval heads, can essentially pay attention to all input tokens. These key observations motivate us to use separate caching strategy for attention heads. Therefore, we propose RazorAttention, a training-free KV cache compression algorithm, which maintains a full cache for these crucial retrieval heads and discards the remote tokens in non-retrieval heads. Furthermore, we introduce a novel mechanism involving a "compensation token" to further recover the information in the dropped tokens. Extensive evaluations across a diverse set of large language models (LLMs) demonstrate that RazorAttention achieves a reduction in KV cache size by over 70% without noticeable impacts on performance. Additionally, RazorAttention is compatible with FlashAttention, rendering it an efficient and plug-and-play solution that enhances LLM inference efficiency without overhead or retraining of the original model.

Title: MINI-SEQUENCE TRANSFORMER: Optimizing Intermediate Memory for Long Sequences Training

• Authors: Cheng Luo, Jiawei Zhao, Zhuoming Chen, Beidi Chen, Anima Anandkumar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15892
• Pdf URL: https://arxiv.org/pdf/2407.15892
• Copy Paste: [[2407.15892]] MINI-SEQUENCE TRANSFORMER: Optimizing Intermediate Memory for Long Sequences Training(https://arxiv.org/abs/2407.15892)
• Keywords: transformer
• Abstract: We introduce Mini-Sequence Transformer (MsT), a simple and effective methodology for highly efficient and accurate LLM training with extremely long sequences. MsT partitions input sequences and iteratively processes mini-sequences to reduce intermediate memory usage. Integrated with activation recomputation, it enables significant memory savings in both forward and backward passes. In experiments with the Llama3-8B model, with MsT, we measure no degradation in throughput or convergence even with 12x longer sequences than standard implementations due to our careful memory optimizations. MsT is fully general, implementation-agnostic, and requires minimal code changes to integrate with existing LLM training frameworks.

Title: Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning

• Authors: Jingchen Sun, Rohan Sharma, Vishnu Suresh Lokhande, Changyou Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.15894
• Pdf URL: https://arxiv.org/pdf/2407.15894
• Copy Paste: [[2407.15894]] Craft: Cross-modal Aligned Features Improve Robustness of Prompt Tuning(https://arxiv.org/abs/2407.15894)
• Keywords: robust
• Abstract: Prompt Tuning has emerged as a prominent research paradigm for adapting vision-language models to various downstream tasks. However, recent research indicates that prompt tuning methods often lead to overfitting due to limited training samples. In this paper, we propose a \textbf{Cr}oss-modal \textbf{a}ligned \textbf{f}eature \textbf{t}uning (\textbf{Craft}) method to address this issue. Cross-modal alignment is conducted by first selecting anchors from the alternative domain and deriving relative representations of the embeddings for the selected anchors. Optimizing for a feature alignment loss over anchor-aligned text and image modalities creates a more unified text-image common space. Overfitting in prompt tuning also deteriorates model performance on out-of-distribution samples. To further improve the prompt model's robustness, we propose minimizing Maximum Mean Discrepancy (MMD) over the anchor-aligned feature spaces to mitigate domain shift. The experiment on four different prompt tuning structures consistently shows the improvement of our method, with increases of up to $6.1\%$ in the Base-to-Novel generalization task, $5.8\%$ in the group robustness task, and $2.7\%$ in the out-of-distribution tasks. The code will be available at \href{this https URL}

Title: Revisiting the Robust Alignment of Circuit Breakers

• Authors: Leo Schwinn, Simon Geisler
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.15902
• Pdf URL: https://arxiv.org/pdf/2407.15902
• Copy Paste: [[2407.15902]] Revisiting the Robust Alignment of Circuit Breakers(https://arxiv.org/abs/2407.15902)
• Keywords: defense, attack, robust
• Abstract: Over the past decade, adversarial training has emerged as one of the few reliable methods for enhancing model robustness against adversarial attacks [Szegedy et al., 2014, Madry et al., 2018, Xhonneux et al., 2024], while many alternative approaches have failed to withstand rigorous subsequent evaluations. Recently, an alternative defense mechanism, namely "circuit breakers" [Zou et al., 2024], has shown promising results for aligning LLMs. In this report, we show that the robustness claims of "Improving Alignment and Robustness with Circuit Breakers" against unconstraint continuous attacks in the embedding space of the input tokens may be overestimated [Zou et al., 2024]. Specifically, we demonstrate that by implementing a few simple changes to embedding space attacks [Schwinn et al., 2024a,b], we achieve 100% attack success rate (ASR) against circuit breaker models. Without conducting any further hyperparameter tuning, these adjustments increase the ASR by more than 80% compared to the original evaluation. Code is accessible at: this https URL

Title: Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Models

• Authors: Aayush Saxena, Arit Kumar Bishwas, Ayush Ashok Mishra, Ryan Armstrong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.15904
• Pdf URL: https://arxiv.org/pdf/2407.15904
• Copy Paste: [[2407.15904]] Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Models(https://arxiv.org/abs/2407.15904)
• Keywords: generative, large language model
• Abstract: Deep learning models have achieved tremendous success in most of the industries in recent years. The evolution of these models has also led to an increase in the model size and energy requirement, making it difficult to deploy in production on low compute devices. An increase in the number of connected devices around the world warrants compressed models that can be easily deployed at the local devices with low compute capacity and power accessibility. A wide range of solutions have been proposed by different researchers to reduce the size and complexity of such models, prominent among them are, Weight Quantization, Parameter Pruning, Network Pruning, low-rank representation, weights sharing, neural architecture search, knowledge distillation etc. In this research work, we investigate the performance impacts on various trained deep learning models, compressed using quantization and pruning techniques. We implemented both, quantization and pruning, compression techniques on popular deep learning models used in the image classification, object detection, language models and generative models-based problem statements. We also explored performance of various large language models (LLMs) after quantization and low rank adaptation. We used the standard evaluation metrics (model's size, accuracy, and inference time) for all the related problem statements and concluded this paper by discussing the challenges and future work.

Title: A Survey of Explainable Artificial Intelligence (XAI) in Financial Time Series Forecasting

• Authors: Pierre-Daniel Arsenault, Shengrui Wang, Jean-Marc Patenande
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.15909
• Pdf URL: https://arxiv.org/pdf/2407.15909
• Copy Paste: [[2407.15909]] A Survey of Explainable Artificial Intelligence (XAI) in Financial Time Series Forecasting(https://arxiv.org/abs/2407.15909)
• Keywords: interpretability, explainability
• Abstract: Artificial Intelligence (AI) models have reached a very significant level of accuracy. While their superior performance offers considerable benefits, their inherent complexity often decreases human trust, which slows their application in high-risk decision-making domains, such as finance. The field of eXplainable AI (XAI) seeks to bridge this gap, aiming to make AI models more understandable. This survey, focusing on published work from the past five years, categorizes XAI approaches that predict financial time series. In this paper, explainability and interpretability are distinguished, emphasizing the need to treat these concepts separately as they are not applied the same way in practice. Through clear definitions, a rigorous taxonomy of XAI approaches, a complementary characterization, and examples of XAI's application in the finance industry, this paper provides a comprehensive view of XAI's current role in finance. It can also serve as a guide for selecting the most appropriate XAI approach for future applications.

Title: Development of Multistage Machine Learning Classifier using Decision Trees and Boosting Algorithms over Darknet Network Traffic

• Authors: Anjali Sureshkumar Nair, Dr. Prashant Nitnaware
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2407.15910
• Pdf URL: https://arxiv.org/pdf/2407.15910
• Copy Paste: [[2407.15910]] Development of Multistage Machine Learning Classifier using Decision Trees and Boosting Algorithms over Darknet Network Traffic(https://arxiv.org/abs/2407.15910)
• Keywords: security, robust
• Abstract: In recent years, the clandestine nature of darknet activities has presented an escalating challenge to cybersecurity efforts, necessitating sophisticated methods for the detection and classification of network traffic associated with these covert operations. The system addresses the significant challenge of class imbalance within Darknet traffic datasets, where malicious traffic constitutes a minority, hindering effective discrimination between normal and malicious behavior. By leveraging boosting algorithms like AdaBoost and Gradient Boosting coupled with decision trees, this study proposes a robust solution for network traffic classification. Boosting algorithms ensemble learning corrects errors iteratively and assigns higher weights to minority class instances, complemented by the hierarchical structure of decision trees. The additional Feature Selection which is a preprocessing method by utilizing Information Gain metrics, Fisher's Score, and Chi-Square test selection for features is employed. Rigorous experimentation with diverse Darknet traffic datasets validates the efficacy of the proposed multistage classifier, evaluated through various performance metrics such as accuracy, precision, recall, and F1-score, offering a comprehensive solution for accurate detection and classification of Darknet activities.

Title: The Shadow of Fraud: The Emerging Danger of AI-powered Social Engineering and its Possible Cure

• Authors: Jingru Yu, Yi Yu, Xuhong Wang, Yilun Lin, Manzhi Yang, Yu Qiao, Fei-Yue Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.15912
• Pdf URL: https://arxiv.org/pdf/2407.15912
• Copy Paste: [[2407.15912]] The Shadow of Fraud: The Emerging Danger of AI-powered Social Engineering and its Possible Cure(https://arxiv.org/abs/2407.15912)
• Keywords: defense, attack, robust, diffusion, large language model
• Abstract: Social engineering (SE) attacks remain a significant threat to both individuals and organizations. The advancement of Artificial Intelligence (AI), including diffusion models and large language models (LLMs), has potentially intensified these threats by enabling more personalized and convincing attacks. This survey paper categorizes SE attack mechanisms, analyzes their evolution, and explores methods for measuring these threats. It highlights the challenges in raising awareness about the risks of AI-enhanced SE attacks and offers insights into developing proactive and adaptable defense strategies. Additionally, we introduce a categorization of the evolving nature of AI-powered social engineering attacks into "3E phases": Enlarging, wherein the magnitude of attacks expands through the leverage of digital media; Enriching, introducing novel attack vectors and techniques; and Emerging, signifying the advent of novel threats and methods. Moreover, we emphasize the necessity for a robust framework to assess the risk of AI-powered SE attacks. By identifying and addressing gaps in existing research, we aim to guide future studies and encourage the development of more effective defenses against the growing threat of AI-powered social engineering.

Title: Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models

• Authors: Raza Imam, Hanan Gani, Muhammad Huzaifa, Karthik Nandakumar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.15913
• Pdf URL: https://arxiv.org/pdf/2407.15913
• Copy Paste: [[2407.15913]] Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models(https://arxiv.org/abs/2407.15913)
• Keywords: transformer, large language model
• Abstract: The conventional modus operandi for adapting pre-trained vision-language models (VLMs) during test-time involves tuning learnable prompts, ie, test-time prompt tuning. This paper introduces Test-Time Low-rank adaptation (TTL) as an alternative to prompt tuning for zero-shot generalization of large-scale VLMs. Taking inspiration from recent advancements in efficiently fine-tuning large language models, TTL offers a test-time parameter-efficient adaptation approach that updates the attention weights of the transformer encoder by maximizing prediction confidence. The self-supervised confidence maximization objective is specified using a weighted entropy loss that enforces consistency among predictions of augmented samples. TTL introduces only a small amount of trainable parameters for low-rank adapters in the model space while keeping the prompts and backbone frozen. Extensive experiments on a variety of natural distribution and cross-domain tasks show that TTL can outperform other techniques for test-time optimization of VLMs in strict zero-shot settings. Specifically, TTL outperforms test-time prompt tuning baselines with a significant improvement on average. Our code is available at at this https URL.

Title: Multilingual Fine-Grained News Headline Hallucination Detection

• Authors: Jiaming Shen, Tianqi Liu, Jialu Liu, Zhen Qin, Jay Pavagadhi, Simon Baumgartner, Michael Bendersky
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.15975
• Pdf URL: https://arxiv.org/pdf/2407.15975
• Copy Paste: [[2407.15975]] Multilingual Fine-Grained News Headline Hallucination Detection(https://arxiv.org/abs/2407.15975)
• Keywords: large language model
• Abstract: The popularity of automated news headline generation has surged with advancements in pre-trained language models. However, these models often suffer from the ``hallucination'' problem, where the generated headline is not fully supported by its source article. Efforts to address this issue have predominantly focused on English, using over-simplistic classification schemes that overlook nuanced hallucination types. In this study, we introduce the first multilingual, fine-grained news headline hallucination detection dataset that contains over 11 thousand pairs in 5 languages, each annotated with detailed hallucination types by experts. We conduct extensive experiments on this dataset under two settings. First, we implement several supervised fine-tuning approaches as preparatory solutions and demonstrate this dataset's challenges and utilities. Second, we test various large language models' in-context learning abilities and propose two novel techniques, language-dependent demonstration selection and coarse-to-fine prompting, to boost the few-shot hallucination detection performance in terms of the example-F1 metric. We release this dataset to foster further research in multilingual, fine-grained headline hallucination detection.

Title: Virtual Reality and Augmented Reality Security: A Reconnaissance and Vulnerability Assessment Approach

• Authors: Sarina Dastgerdy
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.15984
• Pdf URL: https://arxiv.org/pdf/2407.15984
• Copy Paste: [[2407.15984]] Virtual Reality and Augmented Reality Security: A Reconnaissance and Vulnerability Assessment Approach(https://arxiv.org/abs/2407.15984)
• Keywords: security, attack
• Abstract: Various industries have widely adopted Virtual Reality (VR) and Augmented Reality (AR) technologies to enhance productivity and user experiences. However, their integration introduces significant security challenges. This systematic literature review focuses on identifying devices used in AR and VR technologies and specifies the associated vulnerabilities, particularly during the reconnaissance phase and vulnerability assessment, which are critical steps in penetration testing. Following Kitchenham and Charters' guidelines, we systematically selected and analyzed primary studies. The reconnaissance phase involves gathering detailed information about AR and VR systems to identify potential attack vectors. In the vulnerability assessment phase, these vectors are analyzed to pinpoint weaknesses that malicious actors could exploit. Our findings reveal that AR and VR devices, such as headsets (e.g., HTC Vive, Oculus Quest), development platforms (e.g., Unity Framework, Google Cardboard SDK), and applications (e.g., Bigscreen VR, VRChat), are susceptible to various attacks, including remote code execution, cross-site scripting (XSS), eavesdropping, and man-in-the-room attacks. Specifically, the Bigscreen VR application exhibited severe vulnerabilities like remote code execution (RCE) via the 'Application.OpenURL' API, XSS in user inputs, and botnet propagation. Similarly, the Oculus Quest demonstrated susceptibility to side-channel attacks and ransomware. This paper provides a detailed overview of specific device vulnerabilities and emphasizes the importance of the initial steps in penetration testing to identify security weaknesses in AR and VR systems. By highlighting these vulnerabilities, we aim to assist researchers in exploring and mitigating these security challenges, ensuring the safe deployment and use of AR and VR technologies across various sectors.

Title: EfficientCD: A New Strategy For Change Detection Based With Bi-temporal Layers Exchanged

• Authors: Sijun Dong, Yuwei Zhu, Geng Chen, Xiaoliang Meng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.15999
• Pdf URL: https://arxiv.org/pdf/2407.15999
• Copy Paste: [[2407.15999]] EfficientCD: A New Strategy For Change Detection Based With Bi-temporal Layers Exchanged(https://arxiv.org/abs/2407.15999)
• Keywords: extraction
• Abstract: With the widespread application of remote sensing technology in environmental monitoring, the demand for efficient and accurate remote sensing image change detection (CD) for natural environments is growing. We propose a novel deep learning framework named EfficientCD, specifically designed for remote sensing image change detection. The framework employs EfficientNet as its backbone network for feature extraction. To enhance the information exchange between bi-temporal image feature maps, we have designed a new Feature Pyramid Network module targeted at remote sensing change detection, named ChangeFPN. Additionally, to make full use of the multi-level feature maps in the decoding stage, we have developed a layer-by-layer feature upsampling module combined with Euclidean distance to improve feature fusion and reconstruction during the decoding stage. The EfficientCD has been experimentally validated on four remote sensing datasets: LEVIR-CD, SYSU-CD, CLCD, and WHUCD. The experimental results demonstrate that EfficientCD exhibits outstanding performance in change detection accuracy. The code and pretrained models will be released at this https URL.

Title: ImPress: Securing DRAM Against Data-Disturbance Errors via Implicit Row-Press Mitigation

• Authors: Moinuddin Qureshi, Anish Saxena, Aamer Jaleel
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2407.16006
• Pdf URL: https://arxiv.org/pdf/2407.16006
• Copy Paste: [[2407.16006]] ImPress: Securing DRAM Against Data-Disturbance Errors via Implicit Row-Press Mitigation(https://arxiv.org/abs/2407.16006)
• Keywords: security, attack
• Abstract: DRAM cells are susceptible to Data-Disturbance Errors (DDE), which can be exploited by an attacker to compromise system security. Rowhammer is a well-known DDE vulnerability that occurs when a row is repeatedly activated. Rowhammer can be mitigated by tracking aggressor rows inside DRAM (in-DRAM) or at the Memory Controller (MC). Row-Press (RP) is a new DDE vulnerability that occurs when a row is kept open for a long time. RP significantly reduces the number of activations required to induce an error, thus breaking existing RH solutions. Prior work on Explicit Row-Press mitigation, ExPress, requires the memory controller to limit the maximum row-open-time, and redesign existing Rowhammer solutions with reduced Rowhammer threshold. Unfortunately, ExPress incurs significant performance and storage overheads, and being a memory controller-based solution, it is incompatible with in-DRAM trackers. In this paper, we propose Implicit Row-Press mitigation (ImPress), which does not restrict row-open-time, is compatible with memory controller-based and in-DRAM solutions and does not reduce the tolerated Rowhammer threshold. ImPress treats a row open for a specified time as equivalent to an activation. We design ImPress by developing a Unified Charge-Loss Model, which combines the net effect of both Rowhammer and Row-Press for arbitrary patterns. We analyze both controller-based (Graphene and PARA) and in-DRAM trackers (Mithril and MINT). We show that ImPress makes Rowhammer solutions resilient to Row-Press transparently, without affecting the Rowhammer threshold.

Title: Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation

• Authors: Jiaming Shen, Ran Xu, Yennie Jun, Zhen Qin, Tianqi Liu, Carl Yang, Yi Liang, Simon Baumgartner, Michael Bendersky
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16008
• Pdf URL: https://arxiv.org/pdf/2407.16008
• Copy Paste: [[2407.16008]] Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation(https://arxiv.org/abs/2407.16008)
• Keywords: large language model
• Abstract: Reward models (RMs) are crucial for aligning large language models (LLMs) with human preferences. They are trained using preference datasets where each example consists of one input prompt, two responses, and a preference label. As curating a high-quality human labeled preference dataset is both time-consuming and expensive, people often rely on existing powerful LLMs for preference label generation. This can potentially introduce noise and impede RM training. In this work, we present RMBoost, a novel synthetic preference data generation paradigm to boost reward model quality. Unlike traditional methods, which generate two responses before obtaining the preference label, RMBoost first generates one response and selects a preference label, followed by generating the second more (or less) preferred response conditioned on the pre-selected preference label and the first response. This approach offers two main advantages. First, RMBoost reduces labeling noise since preference pairs are constructed intentionally. Second, RMBoost facilitates the creation of more diverse responses by incorporating various quality aspects (e.g., helpfulness, relevance, completeness) into the prompts. We conduct extensive experiments across three diverse datasets and demonstrate that RMBoost outperforms other synthetic preference data generation techniques and significantly boosts the performance of four distinct reward models.

Title: AIDE: Antithetical, Intent-based, and Diverse Example-Based Explanations

• Authors: Ikhtiyor Nematov, Dimitris Sacharidis, Tomer Sagi, Katja Hose
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16010
• Pdf URL: https://arxiv.org/pdf/2407.16010
• Copy Paste: [[2407.16010]] AIDE: Antithetical, Intent-based, and Diverse Example-Based Explanations(https://arxiv.org/abs/2407.16010)
• Keywords: explainability
• Abstract: For many use-cases, it is often important to explain the prediction of a black-box model by identifying the most influential training data samples. Existing approaches lack customization for user intent and often provide a homogeneous set of explanation samples, failing to reveal the model's reasoning from different angles. In this paper, we propose AIDE, an approach for providing antithetical (i.e., contrastive), intent-based, diverse explanations for opaque and complex models. AIDE distinguishes three types of explainability intents: interpreting a correct, investigating a wrong, and clarifying an ambiguous prediction. For each intent, AIDE selects an appropriate set of influential training samples that support or oppose the prediction either directly or by contrast. To provide a succinct summary, AIDE uses diversity-aware sampling to avoid redundancy and increase coverage of the training data. We demonstrate the effectiveness of AIDE on image and text classification tasks, in three ways: quantitatively, assessing correctness and continuity; qualitatively, comparing anecdotal evidence from AIDE and other example-based approaches; and via a user study, evaluating multiple aspects of AIDE. The results show that AIDE addresses the limitations of existing methods and exhibits desirable traits for an explainability method.

Title: Enhancing Temporal Understanding in LLMs for Semi-structured Tables

• Authors: Irwin Deng, Kushagra Dixit, Vivek Gupta, Dan Roth
• Subjects: cs.CL, cs.AI, cs.DB, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16030
• Pdf URL: https://arxiv.org/pdf/2407.16030
• Copy Paste: [[2407.16030]] Enhancing Temporal Understanding in LLMs for Semi-structured Tables(https://arxiv.org/abs/2407.16030)
• Keywords: large language model
• Abstract: Temporal reasoning over tabular data presents substantial challenges for large language models (LLMs), as evidenced by recent research. In this study, we conduct a comprehensive analysis of temporal datasets to pinpoint the specific limitations of LLMs. Our investigation leads to enhancements in TempTabQA, a dataset specifically designed for tabular temporal question answering. We provide critical insights for improving LLM performance in temporal reasoning tasks with tabular data. Furthermore, we introduce a novel approach, C.L.E.A.R to strengthen LLM capabilities in this domain. Our findings demonstrate that our method significantly improves evidence-based reasoning across various models. Additionally, our experimental results reveal that indirect supervision with auxiliary data substantially boosts model performance in these tasks. This work contributes to a deeper understanding of LLMs' temporal reasoning abilities over tabular data and promotes advancements in their application across diverse fields.

Title: Transformer-based Capacity Prediction for Lithium-ion Batteries with Data Augmentation

• Authors: Gift Modekwe, Saif Al-Wahaibi, Qiugang Lu
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2407.16036
• Pdf URL: https://arxiv.org/pdf/2407.16036
• Copy Paste: [[2407.16036]] Transformer-based Capacity Prediction for Lithium-ion Batteries with Data Augmentation(https://arxiv.org/abs/2407.16036)
• Keywords: robust, transformer
• Abstract: Lithium-ion batteries are pivotal to technological advancements in transportation, electronics, and clean energy storage. The optimal operation and safety of these batteries require proper and reliable estimation of battery capacities to monitor the state of health. Current methods for estimating the capacities fail to adequately account for long-term temporal dependencies of key variables (e.g., voltage, current, and temperature) associated with battery aging and degradation. In this study, we explore the usage of transformer networks to enhance the estimation of battery capacity. We develop a transformer-based battery capacity prediction model that accounts for both long-term and short-term patterns in battery data. Further, to tackle the data scarcity issue, data augmentation is used to increase the data size, which helps to improve the performance of the model. Our proposed method is validated with benchmark datasets. Simulation results show the effectiveness of data augmentation and the transformer network in improving the accuracy and robustness of battery capacity prediction.

Title: MINT: Securely Mitigating Rowhammer with a Minimalist In-DRAM Tracker

• Authors: Moinuddin Qureshi, Salman Qazi, Aamer Jaleel
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2407.16038
• Pdf URL: https://arxiv.org/pdf/2407.16038
• Copy Paste: [[2407.16038]] MINT: Securely Mitigating Rowhammer with a Minimalist In-DRAM Tracker(https://arxiv.org/abs/2407.16038)
• Keywords: secure, protect, attack
• Abstract: This paper investigates secure low-cost in-DRAM trackers for mitigating Rowhammer (RH). In-DRAM solutions have the advantage that they can solve the RH problem within the DRAM chip, without relying on other parts of the system. However, in-DRAM mitigation suffers from two key challenges: First, the mitigations are synchronized with refresh, which means we cannot mitigate at arbitrary times. Second, the SRAM area available for aggressor tracking is severely limited, to only a few bytes. Existing low-cost in-DRAM trackers (such as TRR) have been broken by well-crafted access patterns, whereas prior counter-based schemes require impractical overheads of hundreds or thousands of entries per bank. The goal of our paper is to develop an ultra low-cost secure in-DRAM tracker. Our solution is based on a simple observation: if only one row can be mitigated at refresh, then we should ideally need to track only one row. We propose a Minimalist In-DRAM Tracker (MINT), which provides secure mitigation with just a single entry. At each refresh, MINT probabilistically decides which activation in the upcoming interval will be selected for mitigation at the next refresh. MINT provides guaranteed protection against classic single and double-sided attacks. We also derive the minimum RH threshold (MinTRH) tolerated by MINT across all patterns. MINT has a MinTRH of 1482 which can be lowered to 356 with RFM. The MinTRH of MINT is lower than a prior counter-based design with 677 entries per bank, and is within 2x of the MinTRH of an idealized design that stores one-counter-per-row. We also analyze the impact of refresh postponement on the MinTRH of low-cost in-DRAM trackers, and propose an efficient solution to make such trackers compatible with refresh postponement.

Title: Leveraging Large Language Models to Geolocate Linguistic Variations in Social Media Posts

• Authors: Davide Savarro, Davide Zago, Stefano Zoia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16047
• Pdf URL: https://arxiv.org/pdf/2407.16047
• Copy Paste: [[2407.16047]] Leveraging Large Language Models to Geolocate Linguistic Variations in Social Media Posts(https://arxiv.org/abs/2407.16047)
• Keywords: large language model
• Abstract: Geolocalization of social media content is the task of determining the geographical location of a user based on textual data, that may show linguistic variations and informal language. In this project, we address the GeoLingIt challenge of geolocalizing tweets written in Italian by leveraging large language models (LLMs). GeoLingIt requires the prediction of both the region and the precise coordinates of the tweet. Our approach involves fine-tuning pre-trained LLMs to simultaneously predict these geolocalization aspects. By integrating innovative methodologies, we enhance the models' ability to understand the nuances of Italian social media text to improve the state-of-the-art in this domain. This work is conducted as part of the Large Language Models course at the Bertinoro International Spring School 2024. We make our code publicly available on GitHub this https URL.

Title: HIERVAR: A Hierarchical Feature Selection Method for Time Series Analysis

• Authors: Alireza Keshavarzian, Shahrokh Valaee
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2407.16048
• Pdf URL: https://arxiv.org/pdf/2407.16048
• Copy Paste: [[2407.16048]] HIERVAR: A Hierarchical Feature Selection Method for Time Series Analysis(https://arxiv.org/abs/2407.16048)
• Keywords: extraction
• Abstract: Time series classification stands as a pivotal and intricate challenge across various domains, including finance, healthcare, and industrial systems. In contemporary research, there has been a notable upsurge in exploring feature extraction through random sampling. Unlike deep convolutional networks, these methods sidestep elaborate training procedures, yet they often necessitate generating a surplus of features to comprehensively encapsulate time series nuances. Consequently, some features may lack relevance to labels or exhibit multi-collinearity with others. In this paper, we propose a novel hierarchical feature selection method aided by ANOVA variance analysis to address this challenge. Through meticulous experimentation, we demonstrate that our method substantially reduces features by over 94% while preserving accuracy -- a significant advancement in the field of time series analysis and feature selection.

Title: LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies

• Authors: Jia Shi, Gautam Gare, Jinjin Tian, Siqi Chai, Zhiqiu Lin, Arun Vasudevan, Di Feng, Francesco Ferroni, Shu Kong
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16067
• Pdf URL: https://arxiv.org/pdf/2407.16067
• Copy Paste: [[2407.16067]] LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies(https://arxiv.org/abs/2407.16067)
• Keywords: robust
• Abstract: We tackle the challenge of predicting models' Out-of-Distribution (OOD) performance using in-distribution (ID) measurements without requiring OOD data. Existing evaluations with "Effective Robustness", which use ID accuracy as an indicator of OOD accuracy, encounter limitations when models are trained with diverse supervision and distributions, such as class labels (Vision Models, VMs, on ImageNet) and textual descriptions (Visual-Language Models, VLMs, on LAION). VLMs often generalize better to OOD data than VMs despite having similar or lower ID performance. To improve the prediction of models' OOD performance from ID measurements, we introduce the Lowest Common Ancestor (LCA)-on-the-Line framework. This approach revisits the established concept of LCA distance, which measures the hierarchical distance between labels and predictions within a predefined class hierarchy, such as WordNet. We assess 75 models using ImageNet as the ID dataset and five significantly shifted OOD variants, uncovering a strong linear correlation between ID LCA distance and OOD top-1 accuracy. Our method provides a compelling alternative for understanding why VLMs tend to generalize better. Additionally, we propose a technique to construct a taxonomic hierarchy on any dataset using K-means clustering, demonstrating that LCA distance is robust to the constructed taxonomic hierarchy. Moreover, we demonstrate that aligning model predictions with class taxonomies, through soft labels or prompt engineering, can enhance model generalization. Open source code in our Project Page: this https URL.

Title: KaPQA: Knowledge-Augmented Product Question-Answering

• Authors: Swetha Eppalapally, Daksh Dangi, Chaithra Bhat, Ankita Gupta, Ruiyi Zhang, Shubham Agarwal, Karishma Bagga, Seunghyun Yoon, Nedim Lipka, Ryan A. Rossi, Franck Dernoncourt
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16073
• Pdf URL: https://arxiv.org/pdf/2407.16073
• Copy Paste: [[2407.16073]] KaPQA: Knowledge-Augmented Product Question-Answering(https://arxiv.org/abs/2407.16073)
• Keywords: generative, large language model
• Abstract: Question-answering for domain-specific applications has recently attracted much interest due to the latest advancements in large language models (LLMs). However, accurately assessing the performance of these applications remains a challenge, mainly due to the lack of suitable benchmarks that effectively simulate real-world scenarios. To address this challenge, we introduce two product question-answering (QA) datasets focused on Adobe Acrobat and Photoshop products to help evaluate the performance of existing models on domain-specific product QA tasks. Additionally, we propose a novel knowledge-driven RAG-QA framework to enhance the performance of the models in the product QA task. Our experiments demonstrated that inducing domain knowledge through query reformulation allowed for increased retrieval and generative performance when compared to standard RAG-QA methods. This improvement, however, is slight, and thus illustrates the challenge posed by the datasets introduced.

Title: Universal Spectral Transfer with Physical Prior-Informed Deep Generative Learning

• Authors: Yanmin Zhu, Loza F. Tadesse
• Subjects: cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.16094
• Pdf URL: https://arxiv.org/pdf/2407.16094
• Copy Paste: [[2407.16094]] Universal Spectral Transfer with Physical Prior-Informed Deep Generative Learning(https://arxiv.org/abs/2407.16094)
• Keywords: generative
• Abstract: Spectroscopy is a powerful analytical technique for characterizing matter across physical and biological realms1-5. However, its fundamental principle necessitates specialized instrumentation per physical phenomena probed, limiting broad adoption and use in all relevant research. In this study, we introduce SpectroGen, a novel physical prior-informed deep generative model for generating relevant spectral signatures across modalities using experimentally collected spectral input only from a single modality. We achieve this by reimagining the representation of spectral data as mathematical constructs of distributions instead of their traditional physical and molecular state representations. The results from 319 standard mineral samples tested demonstrate generating with 99% correlation and 0.01 root mean square error with superior resolution than experimentally acquired ground truth spectra. We showed transferring capability across Raman, Infrared, and X-ray Diffraction modalities with Gaussian, Lorentzian, and Voigt distribution priors respectively6-10. This approach however is globally generalizable for any spectral input that can be represented by a distribution prior, making it universally applicable. We believe our work revolutionizes the application sphere of spectroscopy, which has traditionally been limited by access to the required sophisticated and often expensive equipment towards accelerating material, pharmaceutical, and biological discoveries.

Title: Augmented Efficiency: Reducing Memory Footprint and Accelerating Inference for 3D Semantic Segmentation through Hybrid Vision

• Authors: Aditya Krishnan, Jayneel Vora, Prasant Mohapatra
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16102
• Pdf URL: https://arxiv.org/pdf/2407.16102
• Copy Paste: [[2407.16102]] Augmented Efficiency: Reducing Memory Footprint and Accelerating Inference for 3D Semantic Segmentation through Hybrid Vision(https://arxiv.org/abs/2407.16102)
• Keywords: segmentation
• Abstract: Semantic segmentation has emerged as a pivotal area of study in computer vision, offering profound implications for scene understanding and elevating human-machine interactions across various domains. While 2D semantic segmentation has witnessed significant strides in the form of lightweight, high-precision models, transitioning to 3D semantic segmentation poses distinct challenges. Our research focuses on achieving efficiency and lightweight design for 3D semantic segmentation models, similar to those achieved for 2D models. Such a design impacts applications of 3D semantic segmentation where memory and latency are of concern. This paper introduces a novel approach to 3D semantic segmentation, distinguished by incorporating a hybrid blend of 2D and 3D computer vision techniques, enabling a streamlined, efficient process. We conduct 2D semantic segmentation on RGB images linked to 3D point clouds and extend the results to 3D using an extrusion technique for specific class labels, reducing the point cloud subspace. We perform rigorous evaluations with the DeepViewAgg model on the complete point cloud as our baseline by measuring the Intersection over Union (IoU) accuracy, inference time latency, and memory consumption. This model serves as the current state-of-the-art 3D semantic segmentation model on the KITTI-360 dataset. We can achieve heightened accuracy outcomes, surpassing the baseline for 6 out of the 15 classes while maintaining a marginal 1% deviation below the baseline for the remaining class labels. Our segmentation approach demonstrates a 1.347x speedup and about a 43% reduced memory usage compared to the baseline.

Title: Transformer-based Graph Neural Networks for Battery Range Prediction in AIoT Battery-Swap Services

• Authors: Zhao Li, Yang Liu, Chuan Zhou, Xuanwu Liu, Xuming Pan, Buqing Cao, Xindong Wu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16115
• Pdf URL: https://arxiv.org/pdf/2407.16115
• Copy Paste: [[2407.16115]] Transformer-based Graph Neural Networks for Battery Range Prediction in AIoT Battery-Swap Services(https://arxiv.org/abs/2407.16115)
• Keywords: transformer
• Abstract: The concept of the sharing economy has gained broad recognition, and within this context, Sharing E-Bike Battery (SEB) have emerged as a focal point of societal interest. Despite the popularity, a notable discrepancy remains between user expectations regarding the remaining battery range of SEBs and the reality, leading to a pronounced inclination among users to find an available SEB during emergency situations. In response to this challenge, the integration of Artificial Intelligence of Things (AIoT) and battery-swap services has surfaced as a viable solution. In this paper, we propose a novel structural Transformer-based model, referred to as the SEB-Transformer, designed specifically for predicting the battery range of SEBs. The scenario is conceptualized as a dynamic heterogeneous graph that encapsulates the interactions between users and bicycles, providing a comprehensive framework for analysis. Furthermore, we incorporate the graph structure into the SEB-Transformer to facilitate the estimation of the remaining e-bike battery range, in conjunction with mean structural similarity, enhancing the prediction accuracy. By employing the predictions made by our model, we are able to dynamically adjust the optimal cycling routes for users in real-time, while also considering the strategic locations of charging stations, thereby optimizing the user experience. Empirically our results on real-world datasets demonstrate the superiority of our model against nine competitive baselines. These innovations, powered by AIoT, not only bridge the gap between user expectations and the physical limitations of battery range but also significantly improve the operational efficiency and sustainability of SEB services. Through these advancements, the shared electric bicycle ecosystem is evolving, making strides towards a more reliable, user-friendly, and sustainable mode of transportation.

Title: Towards Effective Fusion and Forecasting of Multimodal Spatio-temporal Data for Smart Mobility

• Authors: Chenxing Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16123
• Pdf URL: https://arxiv.org/pdf/2407.16123
• Copy Paste: [[2407.16123]] Towards Effective Fusion and Forecasting of Multimodal Spatio-temporal Data for Smart Mobility(https://arxiv.org/abs/2407.16123)
• Keywords: privacy
• Abstract: With the rapid development of location based services, multimodal spatio-temporal (ST) data including trajectories, transportation modes, traffic flow and social check-ins are being collected for deep learning based methods. These deep learning based methods learn ST correlations to support the downstream tasks in the fields such as smart mobility, smart city and other intelligent transportation systems. Despite their effectiveness, ST data fusion and forecasting methods face practical challenges in real-world scenarios. First, forecasting performance for ST data-insufficient area is inferior, making it necessary to transfer meta knowledge from heterogeneous area to enhance the sparse representations. Second, it is nontrivial to accurately forecast in multi-transportation-mode scenarios due to the fine-grained ST features of similar transportation modes, making it necessary to distinguish and measure the ST correlations to alleviate the influence caused by entangled ST features. At last, partial data modalities (e.g., transportation mode) are lost due to privacy or technical issues in certain scenarios, making it necessary to effectively fuse the multimodal sparse ST features and enrich the ST representations. To tackle these challenges, our research work aim to develop effective fusion and forecasting methods for multimodal ST data in smart mobility scenario. In this paper, we will introduce our recent works that investigates the challenges in terms of various real-world applications and establish the open challenges in this field for future work.

Title: Fr\'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos

• Authors: Jiahe Liu, Youran Qu, Qi Yan, Xiaohui Zeng, Lele Wang, Renjie Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16124
• Pdf URL: https://arxiv.org/pdf/2407.16124
• Copy Paste: [[2407.16124]] Fr\'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos(https://arxiv.org/abs/2407.16124)
• Keywords: generative
• Abstract: Significant advancements have been made in video generative models recently. Unlike image generation, video generation presents greater challenges, requiring not only generating high-quality frames but also ensuring temporal consistency across these frames. Despite the impressive progress, research on metrics for evaluating the quality of generated videos, especially concerning temporal and motion consistency, remains underexplored. To bridge this research gap, we propose Fréchet Video Motion Distance (FVMD) metric, which focuses on evaluating motion consistency in video generation. Specifically, we design explicit motion features based on key point tracking, and then measure the similarity between these features via the Fréchet distance. We conduct sensitivity analysis by injecting noise into real videos to verify the effectiveness of FVMD. Further, we carry out a large-scale human study, demonstrating that our metric effectively detects temporal noise and aligns better with human perceptions of generated video quality than existing metrics. Additionally, our motion features can consistently improve the performance of Video Quality Assessment (VQA) models, indicating that our approach is also applicable to unary video quality evaluation. Code is available at this https URL.

Title: Diffusion Prior-Based Amortized Variational Inference for Noisy Inverse Problems

• Authors: Sojin Lee, Dogyun Park, Inho Kong, Hyunwoo J. Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16125
• Pdf URL: https://arxiv.org/pdf/2407.16125
• Copy Paste: [[2407.16125]] Diffusion Prior-Based Amortized Variational Inference for Noisy Inverse Problems(https://arxiv.org/abs/2407.16125)
• Keywords: diffusion
• Abstract: Recent studies on inverse problems have proposed posterior samplers that leverage the pre-trained diffusion models as powerful priors. These attempts have paved the way for using diffusion models in a wide range of inverse problems. However, the existing methods entail computationally demanding iterative sampling procedures and optimize a separate solution for each measurement, which leads to limited scalability and lack of generalization capability across unseen samples. To address these limitations, we propose a novel approach, Diffusion prior-based Amortized Variational Inference (DAVI) that solves inverse problems with a diffusion prior from an amortized variational inference perspective. Specifically, instead of separate measurement-wise optimization, our amortized inference learns a function that directly maps measurements to the implicit posterior distributions of corresponding clean data, enabling a single-step posterior sampling even for unseen measurements. Extensive experiments on image restoration tasks, e.g., Gaussian deblur, 4$\times$ super-resolution, and box inpainting with two benchmark datasets, demonstrate our approach's superior performance over strong baselines. Code is available at this https URL.

Title: MxT: Mamba x Transformer for Image Inpainting

• Authors: Shuang Chen, Amir Atapour-Abarghouei, Haozheng Zhang, Hubert P. H. Shum
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16126
• Pdf URL: https://arxiv.org/pdf/2407.16126
• Copy Paste: [[2407.16126]] MxT: Mamba x Transformer for Image Inpainting(https://arxiv.org/abs/2407.16126)
• Keywords: transformer
• Abstract: Image inpainting, or image completion, is a crucial task in computer vision that aims to restore missing or damaged regions of images with semantically coherent content. This technique requires a precise balance of local texture replication and global contextual understanding to ensure the restored image integrates seamlessly with its surroundings. Traditional methods using Convolutional Neural Networks (CNNs) are effective at capturing local patterns but often struggle with broader contextual relationships due to the limited receptive fields. Recent advancements have incorporated transformers, leveraging their ability to understand global interactions. However, these methods face computational inefficiencies and struggle to maintain fine-grained details. To overcome these challenges, we introduce MxT composed of the proposed Hybrid Module (HM), which combines Mamba with the transformer in a synergistic manner. Mamba is adept at efficiently processing long sequences with linear computational costs, making it an ideal complement to the transformer for handling long-scale data interactions. Our HM facilitates dual-level interaction learning at both pixel and patch levels, greatly enhancing the model to reconstruct images with high quality and contextual accuracy. We evaluate MxT on the widely-used CelebA-HQ and Places2-standard datasets, where it consistently outperformed existing state-of-the-art methods.

Title: Finetuning Generative Large Language Models with Discrimination Instructions for Knowledge Graph Completion

• Authors: Yang Liu, Xiaobin Tian, Zequn Sun, Wei Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16127
• Pdf URL: https://arxiv.org/pdf/2407.16127
• Copy Paste: [[2407.16127]] Finetuning Generative Large Language Models with Discrimination Instructions for Knowledge Graph Completion(https://arxiv.org/abs/2407.16127)
• Keywords: generative, large language model
• Abstract: Traditional knowledge graph (KG) completion models learn embeddings to predict missing facts. Recent works attempt to complete KGs in a text-generation manner with large language models (LLMs). However, they need to ground the output of LLMs to KG entities, which inevitably brings errors. In this paper, we present a finetuning framework, DIFT, aiming to unleash the KG completion ability of LLMs and avoid grounding errors. Given an incomplete fact, DIFT employs a lightweight model to obtain candidate entities and finetunes an LLM with discrimination instructions to select the correct one from the given candidates. To improve performance while reducing instruction data, DIFT uses a truncated sampling method to select useful facts for finetuning and injects KG embeddings into the LLM. Extensive experiments on benchmark datasets demonstrate the effectiveness of our proposed framework.

Title: Advancing Brain Imaging Analysis Step-by-step via Progressive Self-paced Learning

• Authors: Yanwu Yang, Hairui Chen, Jiesi Hu, Xutao Guo, Ting Ma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16128
• Pdf URL: https://arxiv.org/pdf/2407.16128
• Copy Paste: [[2407.16128]] Advancing Brain Imaging Analysis Step-by-step via Progressive Self-paced Learning(https://arxiv.org/abs/2407.16128)
• Keywords: robust
• Abstract: Recent advancements in deep learning have shifted the development of brain imaging analysis. However, several challenges remain, such as heterogeneity, individual variations, and the contradiction between the high dimensionality and small size of brain imaging datasets. These issues complicate the learning process, preventing models from capturing intrinsic, meaningful patterns and potentially leading to suboptimal performance due to biases and overfitting. Curriculum learning (CL) presents a promising solution by organizing training examples from simple to complex, mimicking the human learning process, and potentially fostering the development of more robust and accurate models. Despite its potential, the inherent limitations posed by small initial training datasets present significant challenges, including overfitting and poor generalization. In this paper, we introduce the Progressive Self-Paced Distillation (PSPD) framework, employing an adaptive and progressive pacing and distillation mechanism. This allows for dynamic curriculum adjustments based on the states of both past and present models. The past model serves as a teacher, guiding the current model with gradually refined curriculum knowledge and helping prevent the loss of previously acquired knowledge. We validate PSPD's efficacy and adaptability across various convolutional neural networks using the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, underscoring its superiority in enhancing model performance and generalization capabilities. The source code for this approach will be released at this https URL.

Title: FoRA: Low-Rank Adaptation Model beyond Multimodal Siamese Network

• Authors: Weiying Xie, Yusi Zhang, Tianlin Hui, Jiaqing Zhang, Jie Lei, Yunsong Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16129
• Pdf URL: https://arxiv.org/pdf/2407.16129
• Copy Paste: [[2407.16129]] FoRA: Low-Rank Adaptation Model beyond Multimodal Siamese Network(https://arxiv.org/abs/2407.16129)
• Keywords: robust
• Abstract: Multimodal object detection offers a promising prospect to facilitate robust detection in various visual conditions. However, existing two-stream backbone networks are challenged by complex fusion and substantial parameter increments. This is primarily due to large data distribution biases of multimodal homogeneous information. In this paper, we propose a novel multimodal object detector, named Low-rank Modal Adaptors (LMA) with a shared backbone. The shared parameters enhance the consistency of homogeneous information, while lightweight modal adaptors focus on modality unique features. Furthermore, we design an adaptive rank allocation strategy to adapt to the varying heterogeneity at different feature levels. When applied to two multimodal object detection datasets, experiments validate the effectiveness of our method. Notably, on DroneVehicle, LMA attains a 10.4% accuracy improvement over the state-of-the-art method with a 149M-parameters reduction. The code is available at this https URL. Our work was submitted to ACM MM in April 2024, but was rejected. We will continue to refine our work and paper writing next, mainly including proof of theory and multi-task applications of FoRA.

Title: Users Feel Guilty: Measurement of Illegal Software Installation Guide Videos on YouTube for Malware Distribution

• Authors: Rei Yamagishi, Shota Fujii, Tatsuya Mori
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.16132
• Pdf URL: https://arxiv.org/pdf/2407.16132
• Copy Paste: [[2407.16132]] Users Feel Guilty: Measurement of Illegal Software Installation Guide Videos on YouTube for Malware Distribution(https://arxiv.org/abs/2407.16132)
• Keywords: attack, robust
• Abstract: This study introduces and examines a sophisticated malware distribution technique that exploits popular video sharing platforms. In this attack, threat actors distribute malware through deceptive content that promises free versions of premium software and game cheats. Throughout this paper, we call this attack MalTube. MalTube is particularly insidious because it exploits the guilt feelings of users for engaging in potentially illegal activity, making them less likely to report the infection or ask for a help. To investigate this emerging threat, we developed video platform exploitation reconnaissance VIPER, a novel monitoring system designed to detect, monitor, and analyze MalTube activity at scale. Over a four-month data collection period, VIPER processed and analyzed 14,363 videos, 8,671 associated channels, and 1,269 unique fully qualified domain names associated with malware downloads. Our findings reveal that MalTube attackers primarily target young gamers, using the lure of free software and game cheats as infection vectors. The attackers employ various sophisticated social engineering techniques to maximize user engagement and ensure successful malware propagation. These techniques include the strategic use of platform-specific features such as trending keywords, emoticons, and eye-catching thumbnails. These tactics closely mimic legitimate content creation strategies while providing detailed instructions for malware infection. Based on our in-depth analysis, we propose a set of robust detection and mitigation strategies that exploit the invariant characteristics of MalTube videos, offering the potential for automated threat detection and prevention.

Title: Open-Set Biometrics: Beyond Good Closed-Set Models

• Authors: Yiyang Su, Minchul Kim, Feng Liu, Anil Jain, Xiaoming Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16133
• Pdf URL: https://arxiv.org/pdf/2407.16133
• Copy Paste: [[2407.16133]] Open-Set Biometrics: Beyond Good Closed-Set Models(https://arxiv.org/abs/2407.16133)
• Keywords: biometric
• Abstract: Biometric recognition has primarily addressed closed-set identification, assuming all probe subjects are in the gallery. However, most practical applications involve open-set biometrics, where probe subjects may or may not be present in the gallery. This poses distinct challenges in effectively distinguishing individuals in the gallery while minimizing false detections. While it is commonly believed that powerful biometric models can excel in both closed- and open-set scenarios, existing loss functions are inconsistent with open-set evaluation. They treat genuine (mated) and imposter (non-mated) similarity scores symmetrically and neglect the relative magnitudes of imposter scores. To address these issues, we simulate open-set evaluation using minibatches during training and introduce novel loss functions: (1) the identification-detection loss optimized for open-set performance under selective thresholds and (2) relative threshold minimization to reduce the maximum negative score for each probe. Across diverse biometric tasks, including face recognition, gait recognition, and person re-identification, our experiments demonstrate the effectiveness of the proposed loss functions, significantly enhancing open-set performance while positively impacting closed-set performance. Our code and models are available at this https URL.

Title: Diffusion Transformer Captures Spatial-Temporal Dependencies: A Theory for Gaussian Process Data

• Authors: Hengyu Fu, Zehao Dou, Jiawei Guo, Mengdi Wang, Minshuo Chen
• Subjects: cs.LG, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.16134
• Pdf URL: https://arxiv.org/pdf/2407.16134
• Copy Paste: [[2407.16134]] Diffusion Transformer Captures Spatial-Temporal Dependencies: A Theory for Gaussian Process Data(https://arxiv.org/abs/2407.16134)
• Keywords: diffusion, transformer
• Abstract: Diffusion Transformer, the backbone of Sora for video generation, successfully scales the capacity of diffusion models, pioneering new avenues for high-fidelity sequential data generation. Unlike static data such as images, sequential data consists of consecutive data frames indexed by time, exhibiting rich spatial and temporal dependencies. These dependencies represent the underlying dynamic model and are critical to validate the generated data. In this paper, we make the first theoretical step towards bridging diffusion transformers for capturing spatial-temporal dependencies. Specifically, we establish score approximation and distribution estimation guarantees of diffusion transformers for learning Gaussian process data with covariance functions of various decay patterns. We highlight how the spatial-temporal dependencies are captured and affect learning efficiency. Our study proposes a novel transformer approximation theory, where the transformer acts to unroll an algorithm. We support our theoretical results by numerical experiments, providing strong evidence that spatial-temporal dependencies are captured within attention layers, aligning with our approximation theory.

Title: 3D-UGCN: A Unified Graph Convolutional Network for Robust 3D Human Pose Estimation from Monocular RGB Images

• Authors: Jie Zhao, Jianing Li, Weihan Chen, Wentong Wang, Pengfei Yuan, Xu Zhang, Deshu Peng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16137
• Pdf URL: https://arxiv.org/pdf/2407.16137
• Copy Paste: [[2407.16137]] 3D-UGCN: A Unified Graph Convolutional Network for Robust 3D Human Pose Estimation from Monocular RGB Images(https://arxiv.org/abs/2407.16137)
• Keywords: robust
• Abstract: Human pose estimation remains a multifaceted challenge in computer vision, pivotal across diverse domains such as behavior recognition, human-computer interaction, and pedestrian tracking. This paper proposes an improved method based on the spatial-temporal graph convolution net-work (UGCN) to address the issue of missing human posture skeleton sequences in single-view videos. We present the improved UGCN, which allows the network to process 3D human pose data and improves the 3D human pose skeleton sequence, thereby resolving the occlusion issue.

Title: Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformation

• Authors: Xinghao Wu, Jianwei Niu, Xuefeng Liu, Mingjia Shi, Guogang Zhu, Shaojie Tang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16139
• Pdf URL: https://arxiv.org/pdf/2407.16139
• Copy Paste: [[2407.16139]] Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformation(https://arxiv.org/abs/2407.16139)
• Keywords: federate
• Abstract: In traditional Federated Learning approaches like FedAvg, the global model underperforms when faced with data heterogeneity. Personalized Federated Learning (PFL) enables clients to train personalized models to fit their local data distribution better. However, we surprisingly find that the feature extractor in FedAvg is superior to those in most PFL methods. More interestingly, by applying a linear transformation on local features extracted by the feature extractor to align with the classifier, FedAvg can surpass the majority of PFL methods. This suggests that the primary cause of FedAvg's inadequate performance stems from the mismatch between the locally extracted features and the classifier. While current PFL methods mitigate this issue to some extent, their designs compromise the quality of the feature extractor, thus limiting the full potential of PFL. In this paper, we propose a new PFL framework called FedPFT to address the mismatch problem while enhancing the quality of the feature extractor. FedPFT integrates a feature transformation module, driven by personalized prompts, between the global feature extractor and classifier. In each round, clients first train prompts to transform local features to match the global classifier, followed by training model parameters. This approach can also align the training objectives of clients, reducing the impact of data heterogeneity on model collaboration. Moreover, FedPFT's feature transformation module is highly scalable, allowing for the use of different prompts to tailor local features to various tasks. Leveraging this, we introduce a collaborative contrastive learning task to further refine feature extractor quality. Our experiments demonstrate that FedPFT outperforms state-of-the-art methods by up to 7.08%.

Title: Diffusion Models as Optimizers for Efficient Planning in Offline RL

• Authors: Renming Huang, Yunqiang Pei, Guoqing Wang, Yangming Zhang, Yang Yang, Peng Wang, Hengtao Shen
• Subjects: cs.LG, cs.AI, cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2407.16142
• Pdf URL: https://arxiv.org/pdf/2407.16142
• Copy Paste: [[2407.16142]] Diffusion Models as Optimizers for Efficient Planning in Offline RL(https://arxiv.org/abs/2407.16142)
• Keywords: diffusion
• Abstract: Diffusion models have shown strong competitiveness in offline reinforcement learning tasks by formulating decision-making as sequential generation. However, the practicality of these methods is limited due to the lengthy inference processes they require. In this paper, we address this problem by decomposing the sampling process of diffusion models into two decoupled subprocesses: 1) generating a feasible trajectory, which is a time-consuming process, and 2) optimizing the trajectory. With this decomposition approach, we are able to partially separate efficiency and quality factors, enabling us to simultaneously gain efficiency advantages and ensure quality assurance. We propose the Trajectory Diffuser, which utilizes a faster autoregressive model to handle the generation of feasible trajectories while retaining the trajectory optimization process of diffusion models. This allows us to achieve more efficient planning without sacrificing capability. To evaluate the effectiveness and efficiency of the Trajectory Diffuser, we conduct experiments on the D4RL benchmarks. The results demonstrate that our method achieves $\it 3$-$\it 10 \times$ faster inference speed compared to previous sequence modeling methods, while also outperforming them in terms of overall performance. this https URL Keywords: Reinforcement Learning and Efficient Planning and Diffusion Model

Title: On the Benefits of Rank in Attention Layers

• Authors: Noah Amsel, Gilad Yehudai, Joan Bruna
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2407.16153
• Pdf URL: https://arxiv.org/pdf/2407.16153
• Copy Paste: [[2407.16153]] On the Benefits of Rank in Attention Layers(https://arxiv.org/abs/2407.16153)
• Keywords: transformer
• Abstract: Attention-based mechanisms are widely used in machine learning, most prominently in transformers. However, hyperparameters such as the rank of the attention matrices and the number of heads are scaled nearly the same way in all realizations of this architecture, without theoretical justification. In this work we show that there are dramatic trade-offs between the rank and number of heads of the attention mechanism. Specifically, we present a simple and natural target function that can be represented using a single full-rank attention head for any context length, but that cannot be approximated by low-rank attention unless the number of heads is exponential in the embedding dimension, even for short context lengths. Moreover, we prove that, for short context lengths, adding depth allows the target to be approximated by low-rank attention. For long contexts, we conjecture that full-rank attention is necessary. Finally, we present experiments with off-the-shelf transformers that validate our theoretical findings.

Title: DDK: Distilling Domain Knowledge for Efficient Large Language Models

• Authors: Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanxing Zhang, Haoran Que, Ken Deng, Zhiqi Bai, Jie Liu, Ge Zhang, Jiakai Wang, Yanan Wu, Congnan Liu, Wenbo Su, Jiamang Wang, Lin Qu, Bo Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16154
• Pdf URL: https://arxiv.org/pdf/2407.16154
• Copy Paste: [[2407.16154]] DDK: Distilling Domain Knowledge for Efficient Large Language Models(https://arxiv.org/abs/2407.16154)
• Keywords: large language model
• Abstract: Despite the advanced intelligence abilities of large language models (LLMs) in various applications, they still face significant computational and storage demands. Knowledge Distillation (KD) has emerged as an effective strategy to improve the performance of a smaller LLM (i.e., the student model) by transferring knowledge from a high-performing LLM (i.e., the teacher model). Prevailing techniques in LLM distillation typically use a black-box model API to generate high-quality pretrained and aligned datasets, or utilize white-box distillation by altering the loss function to better transfer knowledge from the teacher LLM. However, these methods ignore the knowledge differences between the student and teacher LLMs across domains. This results in excessive focus on domains with minimal performance gaps and insufficient attention to domains with large gaps, reducing overall performance. In this paper, we introduce a new LLM distillation framework called DDK, which dynamically adjusts the composition of the distillation dataset in a smooth manner according to the domain performance differences between the teacher and student models, making the distillation process more stable and effective. Extensive evaluations show that DDK significantly improves the performance of student models, outperforming both continuously pretrained baselines and existing knowledge distillation methods by a large margin.

Title: TransFeat-TPP: An Interpretable Deep Covariate Temporal Point Processes

• Authors: Zizhuo Meng, Boyu Li, Xuhui Fan, Zhidong Li, Yang Wang, Fang Chen, Feng Zhou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16161
• Pdf URL: https://arxiv.org/pdf/2407.16161
• Copy Paste: [[2407.16161]] TransFeat-TPP: An Interpretable Deep Covariate Temporal Point Processes(https://arxiv.org/abs/2407.16161)
• Keywords: interpretability, transformer
• Abstract: The classical temporal point process (TPP) constructs an intensity function by taking the occurrence times into account. Nevertheless, occurrence time may not be the only relevant factor, other contextual data, termed covariates, may also impact the event evolution. Incorporating such covariates into the model is beneficial, while distinguishing their relevance to the event dynamics is of great practical significance. In this work, we propose a Transformer-based covariate temporal point process (TransFeat-TPP) model to improve the interpretability of deep covariate-TPPs while maintaining powerful expressiveness. TransFeat-TPP can effectively model complex relationships between events and covariates, and provide enhanced interpretability by discerning the importance of various covariates. Experimental results on synthetic and real datasets demonstrate improved prediction accuracy and consistently interpretable feature importance when compared to existing deep covariate-TPPs.

Title: Representation Magnitude has a Liability to Privacy Vulnerability

• Authors: Xingli Fang, Jung-Eun Kim
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16164
• Pdf URL: https://arxiv.org/pdf/2407.16164
• Copy Paste: [[2407.16164]] Representation Magnitude has a Liability to Privacy Vulnerability(https://arxiv.org/abs/2407.16164)
• Keywords: privacy
• Abstract: The privacy-preserving approaches to machine learning (ML) models have made substantial progress in recent years. However, it is still opaque in which circumstances and conditions the model becomes privacy-vulnerable, leading to a challenge for ML models to maintain both performance and privacy. In this paper, we first explore the disparity between member and non-member data in the representation of models under common training frameworks. We identify how the representation magnitude disparity correlates with privacy vulnerability and address how this correlation impacts privacy vulnerability. Based on the observations, we propose Saturn Ring Classifier Module (SRCM), a plug-in model-level solution to mitigate membership privacy leakage. Through a confined yet effective representation space, our approach ameliorates models' privacy vulnerability while maintaining generalizability. The code of this work can be found here: \url{this https URL}

Title: Robust Privacy Amidst Innovation with Large Language Models Through a Critical Assessment of the Risks

• Authors: Yao-Shun Chuang, Atiquer Rahman Sarkar, Noman Mohammed, Xiaoqian Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16166
• Pdf URL: https://arxiv.org/pdf/2407.16166
• Copy Paste: [[2407.16166]] Robust Privacy Amidst Innovation with Large Language Models Through a Critical Assessment of the Risks(https://arxiv.org/abs/2407.16166)
• Keywords: secure, privacy, protect, attack, robust, extraction, large language model
• Abstract: This study examines integrating EHRs and NLP with large language models (LLMs) to improve healthcare data management and patient care. It focuses on using advanced models to create secure, HIPAA-compliant synthetic patient notes for biomedical research. The study used de-identified and re-identified MIMIC III datasets with GPT-3.5, GPT-4, and Mistral 7B to generate synthetic notes. Text generation employed templates and keyword extraction for contextually relevant notes, with one-shot generation for comparison. Privacy assessment checked PHI occurrence, while text utility was tested using an ICD-9 coding task. Text quality was evaluated with ROUGE and cosine similarity metrics to measure semantic similarity with source notes. Analysis of PHI occurrence and text utility via the ICD-9 coding task showed that the keyword-based method had low risk and good performance. One-shot generation showed the highest PHI exposure and PHI co-occurrence, especially in geographic location and date categories. The Normalized One-shot method achieved the highest classification accuracy. Privacy analysis revealed a critical balance between data utility and privacy protection, influencing future data use and sharing. Re-identified data consistently outperformed de-identified data. This study demonstrates the effectiveness of keyword-based methods in generating privacy-protecting synthetic clinical notes that retain data usability, potentially transforming clinical data-sharing practices. The superior performance of re-identified over de-identified data suggests a shift towards methods that enhance utility and privacy by using dummy PHIs to perplex privacy attacks.

Title: Learning Trimodal Relation for AVQA with Missing Modality

• Authors: Kyu Ri Park, Hong Joo Lee, Jung Uk Kim
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.16171
• Pdf URL: https://arxiv.org/pdf/2407.16171
• Copy Paste: [[2407.16171]] Learning Trimodal Relation for AVQA with Missing Modality(https://arxiv.org/abs/2407.16171)
• Keywords: robust, diffusion
• Abstract: Recent Audio-Visual Question Answering (AVQA) methods rely on complete visual and audio input to answer questions accurately. However, in real-world scenarios, issues such as device malfunctions and data transmission errors frequently result in missing audio or visual modality. In such cases, existing AVQA methods suffer significant performance degradation. In this paper, we propose a framework that ensures robust AVQA performance even when a modality is missing. First, we propose a Relation-aware Missing Modal (RMM) generator with Relation-aware Missing Modal Recalling (RMMR) loss to enhance the ability of the generator to recall missing modal information by understanding the relationships and context among the available modalities. Second, we design an Audio-Visual Relation-aware (AVR) diffusion model with Audio-Visual Enhancing (AVE) loss to further enhance audio-visual features by leveraging the relationships and shared cues between the audio-visual modalities. As a result, our method can provide accurate answers by effectively utilizing available information even when input modalities are missing. We believe our method holds potential applications not only in AVQA research but also in various multi-modal scenarios.

Title: Structural Optimization Ambiguity and Simplicity Bias in Unsupervised Neural Grammar Induction

• Authors: Jinwook Park, Kangil Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16181
• Pdf URL: https://arxiv.org/pdf/2407.16181
• Copy Paste: [[2407.16181]] Structural Optimization Ambiguity and Simplicity Bias in Unsupervised Neural Grammar Induction(https://arxiv.org/abs/2407.16181)
• Keywords: interpretability
• Abstract: Neural parameterization has significantly advanced unsupervised grammar induction. However, training these models with a traditional likelihood loss for all possible parses exacerbates two issues: 1) $\textit{structural optimization ambiguity}$ that arbitrarily selects one among structurally ambiguous optimal grammars despite the specific preference of gold parses, and 2) $\textit{structural simplicity bias}$ that leads a model to underutilize rules to compose parse trees. These challenges subject unsupervised neural grammar induction (UNGI) to inevitable prediction errors, high variance, and the necessity for extensive grammars to achieve accurate predictions. This paper tackles these issues, offering a comprehensive analysis of their origins. As a solution, we introduce $\textit{sentence-wise parse-focusing}$ to reduce the parse pool per sentence for loss evaluation, using the structural bias from pre-trained parsers on the same dataset. In unsupervised parsing benchmark tests, our method significantly improves performance while effectively reducing variance and bias toward overly simplistic parses. Our research promotes learning more compact, accurate, and consistent explicit grammars, facilitating better interpretability.

Title: No Re-Train, More Gain: Upgrading Backbones with Diffusion Model for Few-Shot Segmentation

• Authors: Shuai Chen, Fanman Meng, Chenhao Wu, Haoran Wei, Runtong Zhang, Qingbo Wu, Linfeng Xu, Hongliang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16182
• Pdf URL: https://arxiv.org/pdf/2407.16182
• Copy Paste: [[2407.16182]] No Re-Train, More Gain: Upgrading Backbones with Diffusion Model for Few-Shot Segmentation(https://arxiv.org/abs/2407.16182)
• Keywords: diffusion, generative, segmentation
• Abstract: Few-Shot Segmentation (FSS) aims to segment novel classes using only a few annotated images. Despite considerable process under pixel-wise support annotation, current FSS methods still face three issues: the inflexibility of backbone upgrade without re-training, the inability to uniformly handle various types of annotations (e.g., scribble, bounding box, mask and text), and the difficulty in accommodating different annotation quantity. To address these issues simultaneously, we propose DiffUp, a novel FSS method that conceptualizes the FSS task as a conditional generative problem using a diffusion process. For the first issue, we introduce a backbone-agnostic feature transformation module that converts different segmentation cues into unified coarse priors, facilitating seamless backbone upgrade without re-training. For the second issue, due to the varying granularity of transformed priors from diverse annotation types, we conceptualize these multi-granular transformed priors as analogous to noisy intermediates at different steps of a diffusion model. This is implemented via a self-conditioned modulation block coupled with a dual-level quality modulation branch. For the third issue, we incorporates an uncertainty-aware information fusion module that harmonizing the variability across zero-shot, one-shot and many-shot scenarios. Evaluated through rigorous benchmarks, DiffUp significantly outperforms existing FSS models in terms of flexibility and accuracy.

Title: CloudFixer: Test-Time Adaptation for 3D Point Clouds via Diffusion-Guided Geometric Transformation

• Authors: Hajin Shim, Changhun Kim, Eunho Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16193
• Pdf URL: https://arxiv.org/pdf/2407.16193
• Copy Paste: [[2407.16193]] CloudFixer: Test-Time Adaptation for 3D Point Clouds via Diffusion-Guided Geometric Transformation(https://arxiv.org/abs/2407.16193)
• Keywords: robust, diffusion
• Abstract: 3D point clouds captured from real-world sensors frequently encompass noisy points due to various obstacles, such as occlusion, limited resolution, and variations in scale. These challenges hinder the deployment of pre-trained point cloud recognition models trained on clean point clouds, leading to significant performance degradation. While test-time adaptation (TTA) strategies have shown promising results on this issue in the 2D domain, their application to 3D point clouds remains under-explored. Among TTA methods, an input adaptation approach, which directly converts test instances to the source domain using a pre-trained diffusion model, has been proposed in the 2D domain. Despite its robust TTA performance in practical situations, naively adopting this into the 3D domain may be suboptimal due to the neglect of inherent properties of point clouds, and its prohibitive computational cost. Motivated by these limitations, we propose CloudFixer, a test-time input adaptation method tailored for 3D point clouds, employing a pre-trained diffusion model. Specifically, CloudFixer optimizes geometric transformation parameters with carefully designed objectives that leverage the geometric properties of point clouds. We also substantially improve computational efficiency by avoiding backpropagation through the diffusion model and a prohibitive generation process. Furthermore, we propose an online model adaptation strategy by aligning the original model prediction with that of the adapted input. Extensive experiments showcase the superiority of CloudFixer over various TTA baselines, excelling in handling common corruptions and natural distribution shifts across diverse real-world scenarios. Our code is available at this https URL

Title: LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera

• Authors: Yukai Ma, Jianbiao Mei, Xuemeng Yang, Licheng Wen, Weihua Xu, Jiangning Zhang, Botian Shi, Yong Liu, Xingxing Zuo
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2407.16197
• Pdf URL: https://arxiv.org/pdf/2407.16197
• Copy Paste: [[2407.16197]] LiCROcc: Teach Radar for Accurate Semantic Occupancy Prediction using LiDAR and Camera(https://arxiv.org/abs/2407.16197)
• Keywords: robust
• Abstract: Semantic Scene Completion (SSC) is pivotal in autonomous driving perception, frequently confronted with the complexities of weather and illumination changes. The long-term strategy involves fusing multi-modal information to bolster the system's robustness. Radar, increasingly utilized for 3D target detection, is gradually replacing LiDAR in autonomous driving applications, offering a robust sensing alternative. In this paper, we focus on the potential of 3D radar in semantic scene completion, pioneering cross-modal refinement techniques for improved robustness against weather and illumination changes, and enhancing SSC performance.Regarding model architecture, we propose a three-stage tight fusion approach on BEV to realize a fusion framework for point clouds and images. Based on this foundation, we designed three cross-modal distillation modules-CMRD, BRD, and PDD. Our approach enhances the performance in both radar-only (R-LiCROcc) and radar-camera (RC-LiCROcc) settings by distilling to them the rich semantic and structural information of the fused features of LiDAR and camera. Finally, our LC-Fusion (teacher model), R-LiCROcc and RC-LiCROcc achieve the best performance on the nuScenes-Occupancy dataset, with mIOU exceeding the baseline by 22.9%, 44.1%, and 15.5%, respectively. The project page is available at this https URL.

Title: INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Model

• Authors: Yiwei Ma, Zhibin Wang, Xiaoshuai Sun, Weihuang Lin, Qiang Zhou, Jiayi Ji, Rongrong Ji
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16198
• Pdf URL: https://arxiv.org/pdf/2407.16198
• Copy Paste: [[2407.16198]] INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Model(https://arxiv.org/abs/2407.16198)
• Keywords: large language model
• Abstract: With advancements in data availability and computing resources, Multimodal Large Language Models (MLLMs) have showcased capabilities across various fields. However, the quadratic complexity of the vision encoder in MLLMs constrains the resolution of input images. Most current approaches mitigate this issue by cropping high-resolution images into smaller sub-images, which are then processed independently by the vision encoder. Despite capturing sufficient local details, these sub-images lack global context and fail to interact with one another. To address this limitation, we propose a novel MLLM, INF-LLaVA, designed for effective high-resolution image perception. INF-LLaVA incorporates two innovative components. First, we introduce a Dual-perspective Cropping Module (DCM), which ensures that each sub-image contains continuous details from a local perspective and comprehensive information from a global perspective. Second, we introduce Dual-perspective Enhancement Module (DEM) to enable the mutual enhancement of global and local features, allowing INF-LLaVA to effectively process high-resolution images by simultaneously capturing detailed local information and comprehensive global context. Extensive ablation studies validate the effectiveness of these components, and experiments on a diverse set of benchmarks demonstrate that INF-LLaVA outperforms existing MLLMs. Code and pretrained model are available at this https URL.

Title: CLII: Visual-Text Inpainting via Cross-Modal Predictive Interaction

• Authors: Liang Zhao, Qing Guo, Xiaoguang Li, Song Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16204
• Pdf URL: https://arxiv.org/pdf/2407.16204
• Copy Paste: [[2407.16204]] CLII: Visual-Text Inpainting via Cross-Modal Predictive Interaction(https://arxiv.org/abs/2407.16204)
• Keywords: robust
• Abstract: Image inpainting aims to fill missing pixels in damaged images and has achieved significant progress with cut-edging learning techniques. Nevertheless, state-of-the-art inpainting methods are mainly designed for nature images and cannot correctly recover text within scene text images, and training existing models on the scene text images cannot fix the issues. In this work, we identify the visual-text inpainting task to achieve high-quality scene text image restoration and text completion: Given a scene text image with unknown missing regions and the corresponding text with unknown missing characters, we aim to complete the missing information in both images and text by leveraging their complementary information. Intuitively, the input text, even if damaged, contains language priors of the contents within the images and can guide the image inpainting. Meanwhile, the scene text image includes the appearance cues of the characters that could benefit text recovery. To this end, we design the cross-modal predictive interaction (CLII) model containing two branches, i.e., ImgBranch and TxtBranch, for scene text inpainting and text completion, respectively while leveraging their complementary effectively. Moreover, we propose to embed our model into the SOTA scene text spotting method and significantly enhance its robustness against missing pixels, which demonstrates the practicality of the newly developed task. To validate the effectiveness of our method, we construct three real datasets based on existing text-related datasets, containing 1838 images and covering three scenarios with curved, incidental, and styled texts, and conduct extensive experiments to show that our method outperforms baselines significantly.

Title: Figure it Out: Analyzing-based Jailbreak Attack on Large Language Models

• Authors: Shi Lin, Rongchang Li, Xun Wang, Changting Lin, Wenpeng Xing, Meng Han
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16205
• Pdf URL: https://arxiv.org/pdf/2407.16205
• Copy Paste: [[2407.16205]] Figure it Out: Analyzing-based Jailbreak Attack on Large Language Models(https://arxiv.org/abs/2407.16205)
• Keywords: security, defense, attack, robust, generative, large language model
• Abstract: The rapid development of Large Language Models (LLMs) has brought remarkable generative capabilities across diverse tasks. However, despite the impressive achievements, these models still have numerous security vulnerabilities, particularly when faced with jailbreak attacks. Therefore, by investigating jailbreak attacks, we can uncover hidden weaknesses in LLMs and guide us in developing more robust defense mechanisms to fortify their security. In this paper, we further explore the boundary of jailbreak attacks on LLMs and propose Analyzing-based Jailbreak (ABJ). This effective jailbreak attack method takes advantage of LLMs' growing analyzing and reasoning capability and reveals their underlying vulnerabilities when facing analysis-based tasks. We conduct a detailed evaluation of ABJ across various open-source and closed-source LLMs, which achieves 94.8% Attack Success Rate (ASR) and 1.06 Attack Efficiency (AE) on GPT-4-turbo-0409, demonstrating state-of-the-art attack effectiveness and efficiency. Our research highlights the importance of prioritizing and enhancing the safety of LLMs to mitigate the risks of misuse.

Title: Graph-Structured Speculative Decoding

• Authors: Zhuocheng Gong, Jiahao Liu, Ziyue Wang, Pengfei Wu, Jingang Wang, Xunliang Cai, Dongyan Zhao, Rui Yan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16207
• Pdf URL: https://arxiv.org/pdf/2407.16207
• Copy Paste: [[2407.16207]] Graph-Structured Speculative Decoding(https://arxiv.org/abs/2407.16207)
• Keywords: large language model
• Abstract: Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models (LLMs) by employing a small language model to draft a hypothesis sequence, which is then validated by the LLM. The effectiveness of this approach heavily relies on the balance between performance and efficiency of the draft model. In our research, we focus on enhancing the proportion of draft tokens that are accepted to the final output by generating multiple hypotheses instead of just one. This allows the LLM more options to choose from and select the longest sequence that meets its standards. Our analysis reveals that hypotheses produced by the draft model share many common token sequences, suggesting a potential for optimizing computation. Leveraging this observation, we introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. This structure enables us to efficiently predict and merge recurring token sequences, vastly reducing the computational demands of the draft model. We term this approach Graph-structured Speculative Decoding (GSD). We apply GSD across a range of LLMs, including a 70-billion parameter LLaMA-2 model, and observe a remarkable speedup of 1.73$\times$ to 1.96$\times$, significantly surpassing standard speculative decoding.

Title: VidyaRANG: Conversational Learning Based Platform powered by Large Language Model

• Authors: Chitranshu Harbola, Anupam Purwar
• Subjects: cs.CR, cs.ET
• Abstract URL: https://arxiv.org/abs/2407.16209
• Pdf URL: https://arxiv.org/pdf/2407.16209
• Copy Paste: [[2407.16209]] VidyaRANG: Conversational Learning Based Platform powered by Large Language Model(https://arxiv.org/abs/2407.16209)
• Keywords: secure, security, privacy, large language model
• Abstract: Providing authoritative information tailored to a student's specific doubt is a hurdle in this era where search engines return an overwhelming number of article links. Large Language Models such as GPTs fail to provide answers to questions that were derived from sensitive confidential information. This information which is specific to some organisations is not available to LLMs due to privacy constraints. This is where knowledge-augmented retrieval techniques become particularly useful. The proposed platform is designed to cater to the needs of learners from divergent fields. Today, the most common format of learning is video and books, which our proposed platform allows learners to interact and ask questions. This increases learners' focus time exponentially by restricting access to pertinent content and, at the same time allowing personalized access and freedom to gain in-depth knowledge. Instructor's roles and responsibilities are significantly simplified allowing them to train a larger audience. To preserve privacy, instructors can grant course access to specific individuals, enabling personalized conversation on the provided content. This work includes an extensive spectrum of software development and product management skills, which also circumscribe knowledge of cloud computing for running Large Language Models and maintaining the application. For Frontend development, which is responsible for user interaction and user experience, Streamlit and React framework have been utilized. To improve security and privacy, the server is routed to a domain with an SSL certificate, and all the API key/s are stored securely on an AWS EC2 instance, to enhance user experience, web connectivity to an Android Studio-based mobile app has been established, and in-process to publish the app on play store, thus addressing all major software engineering disciplines

Title: Diff-Shadow: Global-guided Diffusion Model for Shadow Removal

• Authors: Jinting Luo, Ru Li, Chengzhi Jiang, Mingyan Han, Xiaoming Zhang, Ting Jiang, Haoqiang Fan, Shuaicheng Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16214
• Pdf URL: https://arxiv.org/pdf/2407.16214
• Copy Paste: [[2407.16214]] Diff-Shadow: Global-guided Diffusion Model for Shadow Removal(https://arxiv.org/abs/2407.16214)
• Keywords: diffusion, transformer
• Abstract: We propose Diff-Shadow, a global-guided diffusion model for high-quality shadow removal. Previous transformer-based approaches can utilize global information to relate shadow and non-shadow regions but are limited in their synthesis ability and recover images with obvious boundaries. In contrast, diffusion-based methods can generate better content but ignore global information, resulting in inconsistent illumination. In this work, we combine the advantages of diffusion models and global guidance to realize shadow-free restoration. Specifically, we propose a parallel UNets architecture: 1) the local branch performs the patch-based noise estimation in the diffusion process, and 2) the global branch recovers the low-resolution shadow-free images. A Reweight Cross Attention (RCA) module is designed to integrate global contextural information of non-shadow regions into the local branch. We further design a Global-guided Sampling Strategy (GSS) that mitigates patch boundary issues and ensures consistent illumination across shaded and unshaded regions in the recovered image. Comprehensive experiments on three publicly standard datasets ISTD, ISTD+, and SRD have demonstrated the effectiveness of Diff-Shadow. Compared to state-of-the-art methods, our method achieves a significant improvement in terms of PSNR, increasing from 32.33dB to 33.69dB on the SRD dataset. Codes will be released.

Title: A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More

• Authors: Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Zixu (James)Zhu, Xiang-Bo Mao, Sitaram Asur, Na (Claire)Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16216
• Pdf URL: https://arxiv.org/pdf/2407.16216
• Copy Paste: [[2407.16216]] A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More(https://arxiv.org/abs/2407.16216)
• Keywords: transformer, large language model
• Abstract: With advancements in self-supervised learning, the availability of trillions tokens in a pre-training corpus, instruction fine-tuning, and the development of large Transformers with billions of parameters, large language models (LLMs) are now capable of generating factual and coherent responses to human queries. However, the mixed quality of training data can lead to the generation of undesired responses, presenting a significant challenge. Over the past two years, various methods have been proposed from different perspectives to enhance LLMs, particularly in aligning them with human expectation. Despite these efforts, there has not been a comprehensive survey paper that categorizes and details these approaches. In this work, we aim to address this gap by categorizing these papers into distinct topics and providing detailed explanations of each alignment method, thereby helping readers gain a thorough understanding of the current state of the field.

Title: Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models

• Authors: Nishanth Madhusudhan, Sathwik Tejaswi Madhusudhan, Vikas Yadav, Masoud Hashemi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16221
• Pdf URL: https://arxiv.org/pdf/2407.16221
• Copy Paste: [[2407.16221]] Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models(https://arxiv.org/abs/2407.16221)
• Keywords: large language model
• Abstract: As Large Language Models (LLMs) achieve remarkable performance across various NLP tasks, their reliability becomes essential for widespread adoption. This paper focuses on Abstention Ability (AA), a critical yet under explored aspect of reliability - the ability of LLMs to refrain from answering questions when they are uncertain or when definitive answer is not possible, while maintaining question-answering (QA) task performance. While previous works have focused on understanding the recollection abilities of LLMs or their ability to identify imponderable/unanswerable questions, we believe there is a need for an effective AA evaluation method. Therefore, we propose a black-box evaluation methodology to examine and understand the AA of LLMs across a variety of multiple-choice QA tasks. We measure AA by rewarding models for abstaining from answering when their predictions are incorrect or when the questions are inherently unanswerable. We investigate three strategies, Strict Prompting, Verbal Confidence Thresholding, and Chain-of-Thought (CoT), to understand their impact on abstention across different LLMs. Our findings reveal that while even state-of-the-art LLMs like GPT-4 struggle with abstention, strategic prompting such as CoT, can significantly enhance this ability. Furthermore, we demonstrate that improving AA also leads to better overall QA task performance, underscoring the importance of evaluating AA in LLMs.

Title: PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment

• Authors: Jiahuan Li, Shujian Huang, Xinyu Dai, Jiajun Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16222
• Pdf URL: https://arxiv.org/pdf/2407.16222
• Copy Paste: [[2407.16222]] PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment(https://arxiv.org/abs/2407.16222)
• Keywords: large language model
• Abstract: Large language models demonstrate reasonable multilingual abilities, despite predominantly English-centric pretraining. However, the spontaneous multilingual alignment in these models is shown to be weak, leading to unsatisfactory cross-lingual transfer and knowledge sharing. Previous works attempt to address this issue by explicitly injecting multilingual alignment information during or after pretraining. Thus for the early stage in pretraining, the alignment is weak for sharing information or knowledge across languages. In this paper, we propose PreAlign, a framework that establishes multilingual alignment prior to language model pretraining. PreAlign injects multilingual alignment by initializing the model to generate similar representations of aligned words and preserves this alignment using a code-switching strategy during pretraining. Extensive experiments in a synthetic English to English-Clone setting demonstrate that PreAlign significantly outperforms standard multilingual joint training in language modeling, zero-shot cross-lingual transfer, and cross-lingual knowledge application. Further experiments in real-world scenarios further validate PreAlign's effectiveness across various model sizes.

Title: OutfitAnyone: Ultra-high Quality Virtual Try-On for Any Clothing and Any Person

• Authors: Ke Sun, Jian Cao, Qi Wang, Linrui Tian, Xindi Zhang, Lian Zhuo, Bang Zhang, Liefeng Bo, Wenbo Zhou, Weiming Zhang, Daiheng Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16224
• Pdf URL: https://arxiv.org/pdf/2407.16224
• Copy Paste: [[2407.16224]] OutfitAnyone: Ultra-high Quality Virtual Try-On for Any Clothing and Any Person(https://arxiv.org/abs/2407.16224)
• Keywords: diffusion
• Abstract: Virtual Try-On (VTON) has become a transformative technology, empowering users to experiment with fashion without ever having to physically try on clothing. However, existing methods often struggle with generating high-fidelity and detail-consistent results. While diffusion models, such as Stable Diffusion series, have shown their capability in creating high-quality and photorealistic images, they encounter formidable challenges in conditional generation scenarios like VTON. Specifically, these models struggle to maintain a balance between control and consistency when generating images for virtual clothing trials. OutfitAnyone addresses these limitations by leveraging a two-stream conditional diffusion model, enabling it to adeptly handle garment deformation for more lifelike results. It distinguishes itself with scalability-modulating factors such as pose, body shape and broad applicability, extending from anime to in-the-wild images. OutfitAnyone's performance in diverse scenarios underscores its utility and readiness for real-world deployment. For more details and animated results, please see \url{this https URL}.

Title: Hooked: A Real-World Study on QR Code Phishing

• Authors: Marvin Geisler, Daniela Pöhn
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.16230
• Pdf URL: https://arxiv.org/pdf/2407.16230
• Copy Paste: [[2407.16230]] Hooked: A Real-World Study on QR Code Phishing(https://arxiv.org/abs/2407.16230)
• Keywords: attack
• Abstract: The usage of quick response (QR) codes was limited in the pre-era of the COVID-19 pandemic. Due to the widespread and frequent application since then, this opened up an attractive phishing opportunity for malicious actors. They trick users into scanning the codes and redirecting them to malicious websites. In order to explore whether phishing with QR codes is another successful attack vector, we conducted a real-world phishing campaign with two different QR code variants at a research campus. The first version was rather plain, whereas the second version was more professionally designed and included the possibility to win a voucher. After the study was completed, a qualitative survey on phishing and QR codes was conducted to verify the results of the phishing campaign. Both, the phishing campaign and the survey, show that a professional design receives more attention. They also illustrate that QR codes are used more frequently by curious users because of their easy functionality. Although the results confirm that technical-savvy users are more aware of the risks, they also underpin the malicious potential for non-technical-savvy users and suggest further work regarding countermeasures.

Title: Channel-Partitioned Windowed Attention And Frequency Learning for Single Image Super-Resolution

• Authors: Dinh Phu Tran, Dao Duy Hung, Daeyoung Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16232
• Pdf URL: https://arxiv.org/pdf/2407.16232
• Copy Paste: [[2407.16232]] Channel-Partitioned Windowed Attention And Frequency Learning for Single Image Super-Resolution(https://arxiv.org/abs/2407.16232)
• Keywords: transformer
• Abstract: Recently, window-based attention methods have shown great potential for computer vision tasks, particularly in Single Image Super-Resolution (SISR). However, it may fall short in capturing long-range dependencies and relationships between distant tokens. Additionally, we find that learning on spatial domain does not convey the frequency content of the image, which is a crucial aspect in SISR. To tackle these issues, we propose a new Channel-Partitioned Attention Transformer (CPAT) to better capture long-range dependencies by sequentially expanding windows along the height and width of feature maps. In addition, we propose a novel Spatial-Frequency Interaction Module (SFIM), which incorporates information from spatial and frequency domains to provide a more comprehensive information from feature maps. This includes information about the frequency content and enhances the receptive field across the entire image. Experimental findings demonstrate the effectiveness of our proposed modules and architecture. In particular, CPAT surpasses current state-of-the-art methods by up to 0.31dB.

Title: Algebraic Adversarial Attacks on Integrated Gradients

• Authors: Lachlan Simpson, Federico Costanza, Kyle Millar, Adriel Cheng, Cheng-Chew Lim, Hong Gunn Chew
• Subjects: cs.LG, math.GR
• Abstract URL: https://arxiv.org/abs/2407.16233
• Pdf URL: https://arxiv.org/pdf/2407.16233
• Copy Paste: [[2407.16233]] Algebraic Adversarial Attacks on Integrated Gradients(https://arxiv.org/abs/2407.16233)
• Keywords: attack, explainability
• Abstract: Adversarial attacks on explainability models have drastic consequences when explanations are used to understand the reasoning of neural networks in safety critical systems. Path methods are one such class of attribution methods susceptible to adversarial attacks. Adversarial learning is typically phrased as a constrained optimisation problem. In this work, we propose algebraic adversarial examples and study the conditions under which one can generate adversarial examples for integrated gradients. Algebraic adversarial examples provide a mathematically tractable approach to adversarial examples.

Title: A Multi-view Mask Contrastive Learning Graph Convolutional Neural Network for Age Estimation

• Authors: Yiping Zhang, Yuntao Shou, Tao Meng, Wei Ai, Keqin Li
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16234
• Pdf URL: https://arxiv.org/pdf/2407.16234
• Copy Paste: [[2407.16234]] A Multi-view Mask Contrastive Learning Graph Convolutional Neural Network for Age Estimation(https://arxiv.org/abs/2407.16234)
• Keywords: security, extraction, transformer
• Abstract: The age estimation task aims to use facial features to predict the age of people and is widely used in public security, marketing, identification, and other fields. However, the features are mainly concentrated in facial keypoints, and existing CNN and Transformer-based methods have inflexibility and redundancy for modeling complex irregular structures. Therefore, this paper proposes a Multi-view Mask Contrastive Learning Graph Convolutional Neural Network (MMCL-GCN) for age estimation. Specifically, the overall structure of the MMCL-GCN network contains a feature extraction stage and an age estimation stage. In the feature extraction stage, we introduce a graph structure to construct face images as input and then design a Multi-view Mask Contrastive Learning (MMCL) mechanism to learn complex structural and semantic information about face images. The learning mechanism employs an asymmetric siamese network architecture, which utilizes an online encoder-decoder structure to reconstruct the missing information from the original graph and utilizes the target encoder to learn latent representations for contrastive learning. Furthermore, to promote the two learning mechanisms better compatible and complementary, we adopt two augmentation strategies and optimize the joint losses. In the age estimation stage, we design a Multi-layer Extreme Learning Machine (ML-IELM) with identity mapping to fully use the features extracted by the online encoder. Then, a classifier and a regressor were constructed based on ML-IELM, which were used to identify the age grouping interval and accurately estimate the final age. Extensive experiments show that MMCL-GCN can effectively reduce the error of age estimation on benchmark datasets such as Adience, MORPH-II, and LAP-2016.

Title: How to Design a Blue Team Scenario for Beginners on the Example of Brute-Force Attacks on Authentications

• Authors: Andreas Eipper, Daniela Pöhn
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.16238
• Pdf URL: https://arxiv.org/pdf/2407.16238
• Copy Paste: [[2407.16238]] How to Design a Blue Team Scenario for Beginners on the Example of Brute-Force Attacks on Authentications(https://arxiv.org/abs/2407.16238)
• Keywords: protect, attack
• Abstract: Cyber attacks are ubiquitous and a constantly growing threat in the age of digitization. In order to protect important data, developers and system administrators must be trained and made aware of possible threats. Practical training can be used for students alike to introduce them to the topic. A constant threat to websites that require user authentication is so-called brute-force attacks, which attempt to crack a password by systematically trying every possible combination. As this is a typical threat, but comparably easy to detect, it is ideal for beginners. Therefore, three open-source blue team scenarios are designed and systematically described. They are contiguous to maximize the learning effect.

Title: Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities

• Authors: Muhammad Irzam Liaqat, Shah Nawaz, Muhammad Zaigham Zaheer, Muhammad Saad Saeed, Hassan Sajjad, Tom De Schepper, Karthik Nandakumar, Muhammad Haris Khan Markus Schedl
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16243
• Pdf URL: https://arxiv.org/pdf/2407.16243
• Copy Paste: [[2407.16243]] Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalities(https://arxiv.org/abs/2407.16243)
• Keywords: robust
• Abstract: Multimodal learning has demonstrated remarkable performance improvements over unimodal architectures. However, multimodal learning methods often exhibit deteriorated performances if one or more modalities are missing. This may be attributed to the commonly used multi-branch design containing modality-specific streams making the models reliant on the availability of a complete set of modalities. In this work, we propose a robust textual-visual multimodal learning method, Chameleon, that completely deviates from the conventional multi-branch design. To enable this, we present the unification of input modalities into one format by encoding textual modality into visual representations. As a result, our approach does not require modality-specific branches to learn modality-independent multimodal representations making it robust to missing modalities. Extensive experiments are performed on four popular challenging datasets including Hateful Memes, UPMC Food-101, MM-IMDb, and Ferramenta. Chameleon not only achieves superior performance when all modalities are present at train/test time but also demonstrates notable resilience in the case of missing modalities.

Title: HSVLT: Hierarchical Scale-Aware Vision-Language Transformer for Multi-Label Image Classification

• Authors: Shuyi Ouyang, Hongyi Wang, Ziwei Niu, Zhenjia Bai, Shiao Xie, Yingying Xu, Ruofeng Tong, Yen-Wei Chen, Lanfen Lin
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.16244
• Pdf URL: https://arxiv.org/pdf/2407.16244
• Copy Paste: [[2407.16244]] HSVLT: Hierarchical Scale-Aware Vision-Language Transformer for Multi-Label Image Classification(https://arxiv.org/abs/2407.16244)
• Keywords: extraction, transformer
• Abstract: The task of multi-label image classification involves recognizing multiple objects within a single image. Considering both valuable semantic information contained in the labels and essential visual features presented in the image, tight visual-linguistic interactions play a vital role in improving classification performance. Moreover, given the potential variance in object size and appearance within a single image, attention to features of different scales can help to discover possible objects in the image. Recently, Transformer-based methods have achieved great success in multi-label image classification by leveraging the advantage of modeling long-range dependencies, but they have several limitations. Firstly, existing methods treat visual feature extraction and cross-modal fusion as separate steps, resulting in insufficient visual-linguistic alignment in the joint semantic space. Additionally, they only extract visual features and perform cross-modal fusion at a single scale, neglecting objects with different characteristics. To address these issues, we propose a Hierarchical Scale-Aware Vision-Language Transformer (HSVLT) with two appealing designs: (1)~A hierarchical multi-scale architecture that involves a Cross-Scale Aggregation module, which leverages joint multi-modal features extracted from multiple scales to recognize objects of varying sizes and appearances in images. (2)~Interactive Visual-Linguistic Attention, a novel attention mechanism module that tightly integrates cross-modal interaction, enabling the joint updating of visual, linguistic and multi-modal features. We have evaluated our method on three benchmark datasets. The experimental results demonstrate that HSVLT surpasses state-of-the-art methods with lower computational cost.

Title: Evaluation Scheme to Analyze Keystroke Dynamics Methods

• Authors: Anastasia Dimaratos, Daniela Pöhn
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.16247
• Pdf URL: https://arxiv.org/pdf/2407.16247
• Copy Paste: [[2407.16247]] Evaluation Scheme to Analyze Keystroke Dynamics Methods(https://arxiv.org/abs/2407.16247)
• Keywords: security, biometric
• Abstract: Password authentication is a weak point for security as passwords are easily stolen and a user may ignore the security by using a simple password. Therefore, services increasingly demand a second factor. While this may enhance security, it comes with a lower level of usability and another factor to be forgotten. A smartphone is an important device in daily life. With the growing number of sensors and features in a smartphone, keystroke dynamics may provide an easy-to-use method. In this paper, we introduce requirements for biometric authentication and keystroke dynamics. This results in an evaluation scheme, which is applied to three selected approaches. Based on the comparison, keystroke dynamics and the evaluation scheme are discussed. The obtained results indicate that keystroke dynamics can be used as another authentication method but can be bypassed by stronger adversaries. For further research, a common data set would improve the comparability.

Title: Systematically Searching for Identity-Related Information in the Internet with OSINT Tools

• Authors: Marcus Walkow, Daniela Pöhn
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.16251
• Pdf URL: https://arxiv.org/pdf/2407.16251
• Copy Paste: [[2407.16251]] Systematically Searching for Identity-Related Information in the Internet with OSINT Tools(https://arxiv.org/abs/2407.16251)
• Keywords: attack
• Abstract: The increase of Internet services has not only created several digital identities but also more information available about the persons behind them. The data can be collected and used for attacks on digital identities as well as on identity management systems, which manage digital identities. In order to identify possible attack vectors and take countermeasures at an early stage, it is important for individuals and organizations to systematically search for and analyze the data. This paper proposes a classification of data and open-source intelligence (OSINT) tools related to identities. This classification helps to systematically search for data. In the next step, the data can be analyzed and countermeasures can be taken. Last but not least, an OSINT framework approach applying this classification for searching and analyzing data is presented and discussed.

Title: LawLuo: A Chinese Law Firm Co-run by LLM Agents

• Authors: Jingyun Sun, Chengxiao Dai, Zhongze Luo, Yangbo Chang, Yang Li
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16252
• Pdf URL: https://arxiv.org/pdf/2407.16252
• Copy Paste: [[2407.16252]] LawLuo: A Chinese Law Firm Co-run by LLM Agents(https://arxiv.org/abs/2407.16252)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) demonstrate substantial potential in delivering legal consultation services to users without a legal background, attributed to their superior text comprehension and generation capabilities. Nonetheless, existing Chinese legal LLMs limit interaction to a single model-user dialogue, unlike the collaborative consultations typical of law firms, where multiple staff members contribute to a single consultation. This limitation prevents an authentic consultation experience. Additionally, extant Chinese legal LLMs suffer from critical limitations: (1) insufficient control over the quality of instruction fine-tuning data; (2) increased model hallucination resulting from users' ambiguous queries; and (3) a reduction in the model's ability to follow instructions over multiple dialogue turns. In response to these challenges, we propose a novel legal dialogue framework that leverages the collaborative capabilities of multiple LLM agents, termed LawLuo. This framework encompasses four agents: a receptionist, a lawyer, a secretary, and a boss, each responsible for different functionalities, collaboratively providing a comprehensive legal consultation to users. Additionally, we constructed two high-quality legal dialogue datasets, KINLED and MURLED, and fine-tuned ChatGLM-3-6b using these datasets. We propose a legal query clarification algorithm called ToLC. Experimental results demonstrate that LawLuo outperforms baseline LLMs, including GPT-4, across three dimensions: lawyer-like language style, the usefulness of legal advice, and the accuracy of legal knowledge. Our code and datasets are available at this https URL.

Title: Self-Reasoning Assistant Learning for non-Abelian Gauge Fields Design

• Authors: Jinyang Sun, Xi Chen, Xiumei Wang, Dandan Zhu, Xingping Zhou
• Subjects: cs.LG, cond-mat.mes-hall, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16255
• Pdf URL: https://arxiv.org/pdf/2407.16255
• Copy Paste: [[2407.16255]] Self-Reasoning Assistant Learning for non-Abelian Gauge Fields Design(https://arxiv.org/abs/2407.16255)
• Keywords: diffusion
• Abstract: Non-Abelian braiding has attracted substantial attention because of its pivotal role in describing the exchange behaviour of anyons, in which the input and outcome of non-Abelian braiding are connected by a unitary matrix. Implementing braiding in a classical system can assist the experimental investigation of non-Abelian physics. However, the design of non-Abelian gauge fields faces numerous challenges stemmed from the intricate interplay of group structures, Lie algebra properties, representation theory, topology, and symmetry breaking. The extreme diversity makes it a powerful tool for the study of condensed matter physics. Whereas the widely used artificial intelligence with data-driven approaches has greatly promoted the development of physics, most works are limited on the data-to-data design. Here we propose a self-reasoning assistant learning framework capable of directly generating non-Abelian gauge fields. This framework utilizes the forward diffusion process to capture and reproduce the complex patterns and details inherent in the target distribution through continuous transformation. Then the reverse diffusion process is used to make the generated data closer to the distribution of the original situation. Thus, it owns strong self-reasoning capabilities, allowing to automatically discover the feature representation and capture more subtle relationships from the dataset. Moreover, the self-reasoning eliminates the need for manual feature engineering and simplifies the process of model building. Our framework offers a disruptive paradigm shift to parse complex physical processes, automatically uncovering patterns from massive datasets.

Title: DreamDissector: Learning Disentangled Text-to-3D Generation from 2D Diffusion Priors

• Authors: Zizheng Yan, Jiapeng Zhou, Fanpeng Meng, Yushuang Wu, Lingteng Qiu, Zisheng Ye, Shuguang Cui, Guanying Chen, Xiaoguang Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16260
• Pdf URL: https://arxiv.org/pdf/2407.16260
• Copy Paste: [[2407.16260]] DreamDissector: Learning Disentangled Text-to-3D Generation from 2D Diffusion Priors(https://arxiv.org/abs/2407.16260)
• Keywords: diffusion
• Abstract: Text-to-3D generation has recently seen significant progress. To enhance its practicality in real-world applications, it is crucial to generate multiple independent objects with interactions, similar to layer-compositing in 2D image editing. However, existing text-to-3D methods struggle with this task, as they are designed to generate either non-independent objects or independent objects lacking spatially plausible interactions. Addressing this, we propose DreamDissector, a text-to-3D method capable of generating multiple independent objects with interactions. DreamDissector accepts a multi-object text-to-3D NeRF as input and produces independent textured meshes. To achieve this, we introduce the Neural Category Field (NeCF) for disentangling the input NeRF. Additionally, we present the Category Score Distillation Sampling (CSDS), facilitated by a Deep Concept Mining (DCM) module, to tackle the concept gap issue in diffusion models. By leveraging NeCF and CSDS, we can effectively derive sub-NeRFs from the original scene. Further refinement enhances geometry and texture. Our experimental results validate the effectiveness of DreamDissector, providing users with novel means to control 3D synthesis at the object level and potentially opening avenues for various creative applications in the future.

Title: Image Classification using Fuzzy Pooling in Convolutional Kolmogorov-Arnold Networks

• Authors: Ayan Igali, Pakizar Shamoi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16268
• Pdf URL: https://arxiv.org/pdf/2407.16268
• Copy Paste: [[2407.16268]] Image Classification using Fuzzy Pooling in Convolutional Kolmogorov-Arnold Networks(https://arxiv.org/abs/2407.16268)
• Keywords: interpretability
• Abstract: Nowadays, deep learning models are increasingly required to be both interpretable and highly accurate. We present an approach that integrates Kolmogorov-Arnold Network (KAN) classification heads and Fuzzy Pooling into convolutional neural networks (CNNs). By utilizing the interpretability of KAN and the uncertainty handling capabilities of fuzzy logic, the integration shows potential for improved performance in image classification tasks. Our comparative analysis demonstrates that the modified CNN architecture with KAN and Fuzzy Pooling achieves comparable or higher accuracy than traditional models. The findings highlight the effectiveness of combining fuzzy logic and KAN to develop more interpretable and efficient deep learning models. Future work will aim to expand this approach across larger datasets.

Title: HyTAS: A Hyperspectral Image Transformer Architecture Search Benchmark and Analysis

• Authors: Fangqin Zhou, Mert Kilickaya, Joaquin Vanschoren, Ran Piao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16269
• Pdf URL: https://arxiv.org/pdf/2407.16269
• Copy Paste: [[2407.16269]] HyTAS: A Hyperspectral Image Transformer Architecture Search Benchmark and Analysis(https://arxiv.org/abs/2407.16269)
• Keywords: transformer
• Abstract: Hyperspectral Imaging (HSI) plays an increasingly critical role in precise vision tasks within remote sensing, capturing a wide spectrum of visual data. Transformer architectures have significantly enhanced HSI task performance, while advancements in Transformer Architecture Search (TAS) have improved model discovery. To harness these advancements for HSI classification, we make the following contributions: i) We propose HyTAS, the first benchmark on transformer architecture search for Hyperspectral imaging, ii) We comprehensively evaluate 12 different methods to identify the optimal transformer over 5 different datasets, iii) We perform an extensive factor analysis on the Hyperspectral transformer search performance, greatly motivating future research in this direction. All benchmark materials are available at HyTAS.

Title: Backdoor Attacks against Hybrid Classical-Quantum Neural Networks

• Authors: Ji Guo, Wenbo Jiang, Rui Zhang, Wenshu Fan, Jiachen Li, Guoming Lu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.16273
• Pdf URL: https://arxiv.org/pdf/2407.16273
• Copy Paste: [[2407.16273]] Backdoor Attacks against Hybrid Classical-Quantum Neural Networks(https://arxiv.org/abs/2407.16273)
• Keywords: security, attack, robust, steal
• Abstract: Hybrid Quantum Neural Networks (HQNNs) represent a promising advancement in Quantum Machine Learning (QML), yet their security has been rarely explored. In this paper, we present the first systematic study of backdoor attacks on HQNNs. We begin by proposing an attack framework and providing a theoretical analysis of the generalization bounds and minimum perturbation requirements for backdoor attacks on HQNNs. Next, we employ two classic backdoor attack methods on HQNNs and Convolutional Neural Networks (CNNs) to further investigate the robustness of HQNNs. Our experimental results demonstrate that HQNNs are more robust than CNNs, requiring more significant image modifications for successful attacks. Additionally, we introduce the Qcolor backdoor, which utilizes color shifts as triggers and employs the Non-dominated Sorting Genetic Algorithm II (NSGA-II) to optimize hyperparameters. Through extensive experiments, we demonstrate the effectiveness, stealthiness, and robustness of the Qcolor backdoor.

Title: Comparative Analysis of AES, Blowfish, Twofish, Salsa20, and ChaCha20 for Image Encryption

• Authors: Rebwar Khalid Muhammed, Ribwar Rashid Aziz, Alla Ahmad Hassan, Aso Mohammed Aladdin, Shaida Jumaah Saydah, Tarik Ahmed. Rashid, Bryar Ahmad Hassan
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16274
• Pdf URL: https://arxiv.org/pdf/2407.16274
• Copy Paste: [[2407.16274]] Comparative Analysis of AES, Blowfish, Twofish, Salsa20, and ChaCha20 for Image Encryption(https://arxiv.org/abs/2407.16274)
• Keywords: security, privacy, protect, attack
• Abstract: Nowadays, cybersecurity has grown into a more significant and difficult scientific issue. The recog-nition of threats and attacks meant for knowledge and safety on the internet is growing harder to detect. Since cybersecurity guarantees the privacy and security of data sent via the Internet, it is essential, while also providing protection against malicious attacks. Encrypt has grown into an an-swer that has become an essential element of information security systems. To ensure the security of shared data, including text, images, or videos, it is essential to employ various methods and strategies. This study delves into the prevalent cryptographic methods and algorithms utilized for prevention and stream encryption, examining their encoding techniques such as advanced encryp-tion standard (AES), Blowfish, Twofish, Salsa20, and ChaCha20. The primary objective of this re-search is to identify the optimal times and throughputs (speeds) for data encryption and decryption processes. The methodology of this study involved selecting five distinct types of images to com-pare the outcomes of the techniques evaluated in this research. The assessment focused on pro-cessing time and speed parameters, examining visual encoding and decoding using Java as the pri-mary platform. A comparative analysis of several symmetric key ciphers was performed, focusing on handling large datasets. Despite this limitation, comparing different images helped evaluate the techniques' novelty. The results showed that ChaCha20 had the best average time for both encryp-tion and decryption, being over 50% faster than some other algorithms. However, the Twofish algo-rithm had lower throughput during testing. The paper concludes with findings and suggestions for future improvements.

Title: When, Where, and What? An Novel Benchmark for Accident Anticipation and Localization with Large Language Models

• Authors: Haicheng Liao, Yongkang Li, Chengyue Wang, Yanchen Guan, KaHou Tam, Chunlin Tian, Li Li, Chengzhong Xu, Zhenning Li
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2407.16277
• Pdf URL: https://arxiv.org/pdf/2407.16277
• Copy Paste: [[2407.16277]] When, Where, and What? An Novel Benchmark for Accident Anticipation and Localization with Large Language Models(https://arxiv.org/abs/2407.16277)
• Keywords: large language model
• Abstract: As autonomous driving systems increasingly become part of daily transportation, the ability to accurately anticipate and mitigate potential traffic accidents is paramount. Traditional accident anticipation models primarily utilizing dashcam videos are adept at predicting when an accident may occur but fall short in localizing the incident and identifying involved entities. Addressing this gap, this study introduces a novel framework that integrates Large Language Models (LLMs) to enhance predictive capabilities across multiple dimensions--what, when, and where accidents might occur. We develop an innovative chain-based attention mechanism that dynamically adjusts to prioritize high-risk elements within complex driving scenes. This mechanism is complemented by a three-stage model that processes outputs from smaller models into detailed multimodal inputs for LLMs, thus enabling a more nuanced understanding of traffic dynamics. Empirical validation on the DAD, CCD, and A3D datasets demonstrates superior performance in Average Precision (AP) and Mean Time-To-Accident (mTTA), establishing new benchmarks for accident prediction technology. Our approach not only advances the technological framework for autonomous driving safety but also enhances human-AI interaction, making predictive insights generated by autonomous systems more intuitive and actionable.

Title: A deeper look at depth pruning of LLMs

• Authors: Shoaib Ahmed Siddiqui, Xin Dong, Greg Heinrich, Thomas Breuel, Jan Kautz, David Krueger, Pavlo Molchanov
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16286
• Pdf URL: https://arxiv.org/pdf/2407.16286
• Copy Paste: [[2407.16286]] A deeper look at depth pruning of LLMs(https://arxiv.org/abs/2407.16286)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are not only resource-intensive to train but even more costly to deploy in production. Therefore, recent work has attempted to prune blocks of LLMs based on cheap proxies for estimating block importance, effectively removing 10% of blocks in well-trained LLaMa-2 and Mistral 7b models without any significant degradation of downstream metrics. In this paper, we explore different block importance metrics by considering adaptive metrics such as Shapley value in addition to static ones explored in prior work. We show that adaptive metrics exhibit a trade-off in performance between tasks i.e., improvement on one task may degrade performance on the other due to differences in the computed block influences. Furthermore, we extend this analysis from a complete block to individual self-attention and feed-forward layers, highlighting the propensity of the self-attention layers to be more amendable to pruning, even allowing removal of upto 33% of the self-attention layers without incurring any performance degradation on MMLU for Mistral 7b (significant reduction in costly maintenance of KV-cache). Finally, we look at simple performance recovery techniques to emulate the pruned layers by training lightweight additive bias or low-rank linear adapters. Performance recovery using emulated updates avoids performance degradation for the initial blocks (up to 5% absolute improvement on MMLU), which is either competitive or superior to the learning-based technique.

Title: Federated Learning for Face Recognition via Intra-subject Self-supervised Learning

• Authors: Hansol Kim, Hoyeol Choi, Youngjun Kwak
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16289
• Pdf URL: https://arxiv.org/pdf/2407.16289
• Copy Paste: [[2407.16289]] Federated Learning for Face Recognition via Intra-subject Self-supervised Learning(https://arxiv.org/abs/2407.16289)
• Keywords: robust, federate
• Abstract: Federated Learning (FL) for face recognition aggregates locally optimized models from individual clients to construct a generalized face recognition model. However, previous studies present two major challenges: insufficient incorporation of self-supervised learning and the necessity for clients to accommodate multiple subjects. To tackle these limitations, we propose FedFS (Federated Learning for personalized Face recognition via intra-subject Self-supervised learning framework), a novel federated learning architecture tailored to train personalized face recognition models without imposing subjects. Our proposed FedFS comprises two crucial components that leverage aggregated features of the local and global models to cooperate with representations of an off-the-shelf model. These components are (1) adaptive soft label construction, utilizing dot product operations to reformat labels within intra-instances, and (2) intra-subject self-supervised learning, employing cosine similarity operations to strengthen robust intra-subject representations. Additionally, we introduce a regularization loss to prevent overfitting and ensure the stability of the optimized model. To assess the effectiveness of FedFS, we conduct comprehensive experiments on the DigiFace-1M and VGGFace datasets, demonstrating superior performance compared to previous methods.

Title: TAPTRv2: Attention-based Position Update Improves Tracking Any Point

• Authors: Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Feng Li, Tianhe Ren, Bohan Li, Lei Zhang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2407.16291
• Pdf URL: https://arxiv.org/pdf/2407.16291
• Copy Paste: [[2407.16291]] TAPTRv2: Attention-based Position Update Improves Tracking Any Point(https://arxiv.org/abs/2407.16291)
• Keywords: transformer
• Abstract: In this paper, we present TAPTRv2, a Transformer-based approach built upon TAPTR for solving the Tracking Any Point (TAP) task. TAPTR borrows designs from DEtection TRansformer (DETR) and formulates each tracking point as a point query, making it possible to leverage well-studied operations in DETR-like algorithms. TAPTRv2 improves TAPTR by addressing a critical issue regarding its reliance on cost-volume,which contaminates the point queryś content feature and negatively impacts both visibility prediction and cost-volume computation. In TAPTRv2, we propose a novel attention-based position update (APU) operation and use key-aware deformable attention to realize. For each query, this operation uses key-aware attention weights to combine their corresponding deformable sampling positions to predict a new query position. This design is based on the observation that local attention is essentially the same as cost-volume, both of which are computed by dot-production between a query and its surrounding features. By introducing this new operation, TAPTRv2 not only removes the extra burden of cost-volume computation, but also leads to a substantial performance improvement. TAPTRv2 surpasses TAPTR and achieves state-of-the-art performance on many challenging datasets, demonstrating the superiority

Title: Manifoldchain: Maximizing Blockchain Throughput via Bandwidth-Clustered Sharding

• Authors: Chunjiang Che, Songze Li, Xuechao Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.16295
• Pdf URL: https://arxiv.org/pdf/2407.16295
• Copy Paste: [[2407.16295]] Manifoldchain: Maximizing Blockchain Throughput via Bandwidth-Clustered Sharding(https://arxiv.org/abs/2407.16295)
• Keywords: secure, security
• Abstract: Bandwidth limitation is the major bottleneck that hinders scaling throughput of proof-of-work blockchains. To guarantee security, the mining rate of the blockchain is determined by the miners with the lowest bandwidth, resulting in an inefficient bandwidth utilization among fast miners. We propose Manifoldchain, an innovative blockchain sharding protocol that alleviates the impact of slow miners to maximize blockchain throughput. Manifoldchain utilizes a bandwidth-clustered shard formation mechanism that groups miners with similar bandwidths into the same shard. Consequently, this approach enables us to set an optimal mining rate for each shard based on its bandwidth, effectively reducing the waiting time caused by slow miners. Nevertheless, the adversary could corrupt miners with similar bandwidths, thereby concentrating hashing power and potentially creating an adversarial majority within a single shard. To counter this adversarial strategy, we introduce sharing mining, allowing the honest mining power of the entire network to participate in the secure ledger formation of each shard, thereby achieving the same level of security as an unsharded blockchain. Additionally, we introduce an asynchronous atomic commitment mechanism to ensure transaction atomicity across shards with various mining rates. Our theoretical analysis demonstrates that Manifoldchain scales linearly in throughput with the increase in shard numbers and inversely with network delay in each shard. We implement a full system prototype of Manifoldchain, comprehensively evaluated on both simulated and real-world testbeds. These experiments validate its vertical scalability with network bandwidth and horizontal scalability with network size, achieving a substantial improvement of 186% in throughput over baseline sharding protocols, for scenarios where bandwidths of miners range from 5Mbps to 60Mbps.

Title: Understanding Impacts of Electromagnetic Signal Injection Attacks on Object Detection

• Authors: Youqian Zhang, Chunxi Yang, Eugene Y. Fu, Qinhong Jiang, Chen Yan, Sze-Yiu Chau, Grace Ngai, Hong-Va Leong, Xiapu Luo, Wenyuan Xu
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16327
• Pdf URL: https://arxiv.org/pdf/2407.16327
• Copy Paste: [[2407.16327]] Understanding Impacts of Electromagnetic Signal Injection Attacks on Object Detection(https://arxiv.org/abs/2407.16327)
• Keywords: security, attack
• Abstract: Object detection can localize and identify objects in images, and it is extensively employed in critical multimedia applications such as security surveillance and autonomous driving. Despite the success of existing object detection models, they are often evaluated in ideal scenarios where captured images guarantee the accurate and complete representation of the detecting scenes. However, images captured by image sensors may be affected by different factors in real applications, including cyber-physical attacks. In particular, attackers can exploit hardware properties within the systems to inject electromagnetic interference so as to manipulate the images. Such attacks can cause noisy or incomplete information about the captured scene, leading to incorrect detection results, potentially granting attackers malicious control over critical functions of the systems. This paper presents a research work that comprehensively quantifies and analyzes the impacts of such attacks on state-of-the-art object detection models in practice. It also sheds light on the underlying reasons for the incorrect detection outcomes.

Title: STATE: A Robust ATE Estimator of Heavy-Tailed Metrics for Variance Reduction in Online Controlled Experiments

• Authors: Hao Zhou, Kun Sun, Shaoming Li, Yangfeng Fan, Guibin Jiang, Jiaqi Zheng, Tao Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16337
• Pdf URL: https://arxiv.org/pdf/2407.16337
• Copy Paste: [[2407.16337]] STATE: A Robust ATE Estimator of Heavy-Tailed Metrics for Variance Reduction in Online Controlled Experiments(https://arxiv.org/abs/2407.16337)
• Keywords: robust
• Abstract: Online controlled experiments play a crucial role in enabling data-driven decisions across a wide range of companies. Variance reduction is an effective technique to improve the sensitivity of experiments, achieving higher statistical power while using fewer samples and shorter experimental periods. However, typical variance reduction methods (e.g., regression-adjusted estimators) are built upon the intuitional assumption of Gaussian distributions and cannot properly characterize the real business metrics with heavy-tailed distributions. Furthermore, outliers diminish the correlation between pre-experiment covariates and outcome metrics, greatly limiting the effectiveness of variance reduction. In this paper, we develop a novel framework that integrates the Student's t-distribution with machine learning tools to fit heavy-tailed metrics and construct a robust average treatment effect estimator in online controlled experiments, which we call STATE. By adopting a variational EM method to optimize the loglikehood function, we can infer a robust solution that greatly eliminates the negative impact of outliers and achieves significant variance reduction. Moreover, we extend the STATE method from count metrics to ratio metrics by utilizing linear transformation that preserves unbiased estimation, whose variance reduction is more complex but less investigated in existing works. Finally, both simulations on synthetic data and long-term empirical results on Meituan experiment platform demonstrate the effectiveness of our method. Compared with the state-of-the-art estimators (CUPAC/MLRATE), STATE achieves over 50% variance reduction, indicating it can reach the same statistical power with only half of the observations, or half the experimental duration.

Title: SOAP: Enhancing Spatio-Temporal Relation and Motion Information Capturing for Few-Shot Action Recognition

• Authors: Wenbo Huang, Jinghui Zhang, Xuwei Qian, Zhen Wu, Meng Wang, Lei Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16344
• Pdf URL: https://arxiv.org/pdf/2407.16344
• Copy Paste: [[2407.16344]] SOAP: Enhancing Spatio-Temporal Relation and Motion Information Capturing for Few-Shot Action Recognition(https://arxiv.org/abs/2407.16344)
• Keywords: robust, extraction
• Abstract: High frame-rate (HFR) videos of action recognition improve fine-grained expression while reducing the spatio-temporal relation and motion information density. Thus, large amounts of video samples are continuously required for traditional data-driven training. However, samples are not always sufficient in real-world scenarios, promoting few-shot action recognition (FSAR) research. We observe that most recent FSAR works build spatio-temporal relation of video samples via temporal alignment after spatial feature extraction, cutting apart spatial and temporal features within samples. They also capture motion information via narrow perspectives between adjacent frames without considering density, leading to insufficient motion information capturing. Therefore, we propose a novel plug-and-play architecture for FSAR called Spatio-tempOral frAme tuPle enhancer (SOAP) in this paper. The model we designed with such architecture refers to SOAP-Net. Temporal connections between different feature channels and spatio-temporal relation of features are considered instead of simple feature extraction. Comprehensive motion information is also captured, using frame tuples with multiple frames containing more motion information than adjacent frames. Combining frame tuples of diverse frame counts further provides a broader perspective. SOAP-Net achieves new state-of-the-art performance across well-known benchmarks such as SthSthV2, Kinetics, UCF101, and HMDB51. Extensive empirical evaluations underscore the competitiveness, pluggability, generalization, and robustness of SOAP. The code is released at this https URL.

Title: FACTTRACK: Time-Aware World State Tracking in Story Outlines

• Authors: Zhiheng Lyu, Kevin Yang, Lingpeng Kong, Daniel Klein
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16347
• Pdf URL: https://arxiv.org/pdf/2407.16347
• Copy Paste: [[2407.16347]] FACTTRACK: Time-Aware World State Tracking in Story Outlines(https://arxiv.org/abs/2407.16347)
• Keywords: fair
• Abstract: While accurately detecting and correcting factual contradictions in language model outputs has become increasingly important as their capabilities improve, doing so is highly challenging. We propose a novel method, FACTTRACK, for tracking atomic facts and addressing factual contradictions. Crucially, FACTTRACK also maintains time-aware validity intervals for each fact, allowing for change over time. At a high level, FACTTRACK consists of a four-step pipeline to update a world state data structure for each new event: (1) decompose the event into directional atomic facts; (2) determine the validity interval of each atomic fact using the world state; (3) detect contradictions with existing facts in the world state; and finally (4) add new facts to the world state and update existing atomic facts. When we apply FACTTRACK to contradiction detection on structured story outlines, we find that FACTTRACK using LLaMA2-7B-Chat substantially outperforms a fair baseline using LLaMA2-7B-Chat, and achieves performance comparable to a GPT4 baseline. Moreover, when using GPT4, FACTTRACK significantly outperforms the GPT4 baseline.

Title: Strike a Balance in Continual Panoptic Segmentation

• Authors: Jinpeng Chen, Runmin Cong, Yuxuan Luo, Horace Ho Shing Ip, Sam Kwong
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16354
• Pdf URL: https://arxiv.org/pdf/2407.16354
• Copy Paste: [[2407.16354]] Strike a Balance in Continual Panoptic Segmentation(https://arxiv.org/abs/2407.16354)
• Keywords: segmentation
• Abstract: This study explores the emerging area of continual panoptic segmentation, highlighting three key balances. First, we introduce past-class backtrace distillation to balance the stability of existing knowledge with the adaptability to new information. This technique retraces the features associated with past classes based on the final label assignment results, performing knowledge distillation targeting these specific features from the previous model while allowing other features to flexibly adapt to new information. Additionally, we introduce a class-proportional memory strategy, which aligns the class distribution in the replay sample set with that of the historical training data. This strategy maintains a balanced class representation during replay, enhancing the utility of the limited-capacity replay sample set in recalling prior classes. Moreover, recognizing that replay samples are annotated only for the classes of their original step, we devise balanced anti-misguidance losses, which combat the impact of incomplete annotations without incurring classification bias. Building upon these innovations, we present a new method named Balanced Continual Panoptic Segmentation (BalConpas). Our evaluation on the challenging ADE20K dataset demonstrates its superior performance compared to existing state-of-the-art methods. The official code is available at this https URL.

Title: Harmonizing Visual Text Comprehension and Generation

• Authors: Zhen Zhao, Jingqun Tang, Binghong Wu, Chunhui Lin, Shu Wei, Hao Liu, Xin Tan, Zhizhong Zhang, Can Huang, Yuan Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16364
• Pdf URL: https://arxiv.org/pdf/2407.16364
• Copy Paste: [[2407.16364]] Harmonizing Visual Text Comprehension and Generation(https://arxiv.org/abs/2407.16364)
• Keywords: generative
• Abstract: In this work, we present TextHarmony, a unified and versatile multimodal generative model proficient in comprehending and generating visual text. Simultaneously generating images and texts typically results in performance degradation due to the inherent inconsistency between vision and language modalities. To overcome this challenge, existing approaches resort to modality-specific data for supervised fine-tuning, necessitating distinct model instances. We propose Slide-LoRA, which dynamically aggregates modality-specific and modality-agnostic LoRA experts, partially decoupling the multimodal generation space. Slide-LoRA harmonizes the generation of vision and language within a singular model instance, thereby facilitating a more unified generative process. Additionally, we develop a high-quality image caption dataset, DetailedTextCaps-100K, synthesized with a sophisticated closed-source MLLM to enhance visual text generation capabilities further. Comprehensive experiments across various benchmarks demonstrate the effectiveness of the proposed approach. Empowered by Slide-LoRA, TextHarmony achieves comparable performance to modality-specific fine-tuning results with only a 2% increase in parameters and shows an average improvement of 2.5% in visual text comprehension tasks and 4.0% in visual text generation tasks. Our work delineates the viability of an integrated approach to multimodal generation within the visual text domain, setting a foundation for subsequent inquiries.

Title: Navigating Uncertainty in Medical Image Segmentation

• Authors: Kilian Zepf, Jes Frellsen, Aasa Feragen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16367
• Pdf URL: https://arxiv.org/pdf/2407.16367
• Copy Paste: [[2407.16367]] Navigating Uncertainty in Medical Image Segmentation(https://arxiv.org/abs/2407.16367)
• Keywords: segmentation
• Abstract: We address the selection and evaluation of uncertain segmentation methods in medical imaging and present two case studies: prostate segmentation, illustrating that for minimal annotator variation simple deterministic models can suffice, and lung lesion segmentation, highlighting the limitations of the Generalized Energy Distance (GED) in model selection. Our findings lead to guidelines for accurately choosing and developing uncertain segmentation models, that integrate aleatoric and epistemic components. These guidelines are designed to aid researchers and practitioners in better developing, selecting, and evaluating uncertain segmentation methods, thereby facilitating enhanced adoption and effective application of segmentation uncertainty in practice.

Title: Evolutionary Prompt Design for LLM-Based Post-ASR Error Correction

• Authors: Rithik Sachdev, Zhong-Qiu Wang, Chao-Han Huck Yang
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.16370
• Pdf URL: https://arxiv.org/pdf/2407.16370
• Copy Paste: [[2407.16370]] Evolutionary Prompt Design for LLM-Based Post-ASR Error Correction(https://arxiv.org/abs/2407.16370)
• Keywords: generative, large language model
• Abstract: Building upon the strength of modern large language models (LLMs), generative error correction (GEC) has emerged as a promising paradigm that can elevate the performance of modern automatic speech recognition (ASR) systems. One representative approach is to leverage in-context learning to prompt LLMs so that a better hypothesis can be generated by the LLMs based on a carefully-designed prompt and an $N$-best list of hypotheses produced by ASR systems. However, it is yet unknown whether the existing prompts are the most effective ones for the task of post-ASR error correction. In this context, this paper first explores alternative prompts to identify an initial set of effective prompts, and then proposes to employ an evolutionary prompt optimization algorithm to refine the initial prompts. Evaluations results on the CHiME-4 subset of the Task $1$ of the SLT $2024$ GenSEC challenge show the effectiveness and potential of the proposed algorithms.

Title: A Multitask Deep Learning Model for Classification and Regression of Hyperspectral Images: Application to the large-scale dataset

• Authors: Koushikey Chhapariya, Alexandre Benoit, Krishna Mohan Buddhiraju, Anil Kumar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16384
• Pdf URL: https://arxiv.org/pdf/2407.16384
• Copy Paste: [[2407.16384]] A Multitask Deep Learning Model for Classification and Regression of Hyperspectral Images: Application to the large-scale dataset(https://arxiv.org/abs/2407.16384)
• Keywords: robust
• Abstract: Multitask learning is a widely recognized technique in the field of computer vision and deep learning domain. However, it is still a research question in remote sensing, particularly for hyperspectral imaging. Moreover, most of the research in the remote sensing domain focuses on small and single-task-based annotated datasets, which limits the generalizability and scalability of the developed models to more diverse and complex real-world scenarios. Thus, in this study, we propose a multitask deep learning model designed to perform multiple classification and regression tasks simultaneously on hyperspectral images. We validated our approach on a large hyperspectral dataset called TAIGA, which contains 13 forest variables, including three categorical variables and ten continuous variables with different biophysical parameters. We design a sharing encoder and task-specific decoder network to streamline feature learning while allowing each task-specific decoder to focus on the unique aspects of its respective task. Additionally, a dense atrous pyramid pooling layer and attention network were integrated to extract multi-scale contextual information and enable selective information processing by prioritizing task-specific features. Further, we computed multitask loss and optimized its parameters for the proposed framework to improve the model performance and efficiency across diverse tasks. A comprehensive qualitative and quantitative analysis of the results shows that the proposed method significantly outperforms other state-of-the-art methods. We trained our model across 10 seeds/trials to ensure robustness. Our proposed model demonstrates higher mean performance while maintaining lower or equivalent variability. To make the work reproducible, the codes will be available at this https URL.

Title: Prisec II -- A Comprehensive Model for IoT Security: Cryptographic Algorithms and Cloud Integration

• Authors: Pedro Costa, Valderi Leithardt
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2407.16395
• Pdf URL: https://arxiv.org/pdf/2407.16395
• Copy Paste: [[2407.16395]] Prisec II -- A Comprehensive Model for IoT Security: Cryptographic Algorithms and Cloud Integration(https://arxiv.org/abs/2407.16395)
• Keywords: security
• Abstract: This study addresses the critical issue of ensuring data security and efficiency in interconnected devices, especially in IoT environments. The objective is to design and implement a model using cryptographic algorithms to enhance data security in 5G networks. Challenges arise from the limited computational capabilities of IoT devices, which require the analysis and selection of cryptographic algorithms to achieve efficient data transmission. This study proposes a model that includes four levels of security, each employing different levels of encryption to provide better data security. Finally, cloud computing optimizes processing efficiency and resource utilization to improve data transmission.

Title: Learning Unsigned Distance Functions from Multi-view Images with Volume Rendering Priors

• Authors: Wenyuan Zhang, Kanle Shi, Yu-Shen Liu, Zhizhong Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16396
• Pdf URL: https://arxiv.org/pdf/2407.16396
• Copy Paste: [[2407.16396]] Learning Unsigned Distance Functions from Multi-view Images with Volume Rendering Priors(https://arxiv.org/abs/2407.16396)
• Keywords: robust
• Abstract: Unsigned distance functions (UDFs) have been a vital representation for open surfaces. With different differentiable renderers, current methods are able to train neural networks to infer a UDF by minimizing the rendering errors on the UDF to the multi-view ground truth. However, these differentiable renderers are mainly handcrafted, which makes them either biased on ray-surface intersections, or sensitive to unsigned distance outliers, or not scalable to large scale scenes. To resolve these issues, we present a novel differentiable renderer to infer UDFs more accurately. Instead of using handcrafted equations, our differentiable renderer is a neural network which is pre-trained in a data-driven manner. It learns how to render unsigned distances into depth images, leading to a prior knowledge, dubbed volume rendering priors. To infer a UDF for an unseen scene from multiple RGB images, we generalize the learned volume rendering priors to map inferred unsigned distances in alpha blending for RGB image rendering. Our results show that the learned volume rendering priors are unbiased, robust, scalable, 3D aware, and more importantly, easy to learn. We evaluate our method on both widely used benchmarks and real scenes, and report superior performance over the state-of-the-art methods.

Title: On ADMM in Heterogeneous Federated Learning: Personalization, Robustness, and Fairness

• Authors: Shengkun Zhu, Jinshan Zeng, Sheng Wang, Yuan Sun, Xiaodong Li, Yuan Yao, Zhiyong Peng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16397
• Pdf URL: https://arxiv.org/pdf/2407.16397
• Copy Paste: [[2407.16397]] On ADMM in Heterogeneous Federated Learning: Personalization, Robustness, and Fairness(https://arxiv.org/abs/2407.16397)
• Keywords: attack, robust, federate, fair
• Abstract: Statistical heterogeneity is a root cause of tension among accuracy, fairness, and robustness of federated learning (FL), and is key in paving a path forward. Personalized FL (PFL) is an approach that aims to reduce the impact of statistical heterogeneity by developing personalized models for individual users, while also inherently providing benefits in terms of fairness and robustness. However, existing PFL frameworks focus on improving the performance of personalized models while neglecting the global model. Moreover, these frameworks achieve sublinear convergence rates and rely on strong assumptions. In this paper, we propose FLAME, an optimization framework by utilizing the alternating direction method of multipliers (ADMM) to train personalized and global models. We propose a model selection strategy to improve performance in situations where clients have different types of heterogeneous data. Our theoretical analysis establishes the global convergence and two kinds of convergence rates for FLAME under mild assumptions. We theoretically demonstrate that FLAME is more robust and fair than the state-of-the-art methods on a class of linear problems. Our experimental findings show that FLAME outperforms state-of-the-art methods in convergence and accuracy, and it achieves higher test accuracy under various attacks and performs more uniformly across clients.

Title: Securing Tomorrow's Smart Cities: Investigating Software Security in Internet of Vehicles and Deep Learning Technologies

• Authors: Ridhi Jain, Norbert Tihanyi, Mohamed Amine Ferrag
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2407.16410
• Pdf URL: https://arxiv.org/pdf/2407.16410
• Copy Paste: [[2407.16410]] Securing Tomorrow's Smart Cities: Investigating Software Security in Internet of Vehicles and Deep Learning Technologies(https://arxiv.org/abs/2407.16410)
• Keywords: secure, security, privacy, attack, robust
• Abstract: Integrating Deep Learning (DL) techniques in the Internet of Vehicles (IoV) introduces many security challenges and issues that require thorough examination. This literature review delves into the inherent vulnerabilities and risks associated with DL in IoV systems, shedding light on the multifaceted nature of security threats. Through an extensive analysis of existing research, we explore potential threats posed by DL algorithms, including adversarial attacks, data privacy breaches, and model poisoning. Additionally, we investigate the impact of DL on critical aspects of IoV security, such as intrusion detection, anomaly detection, and secure communication protocols. Our review emphasizes the complexities of ensuring the robustness, reliability, and trustworthiness of DL-based IoV systems, given the dynamic and interconnected nature of vehicular networks. Furthermore, we discuss the need for novel security solutions tailored to address these challenges effectively and enhance the security posture of DL-enabled IoV environments. By offering insights into these critical issues, this chapter aims to stimulate further research, innovation, and collaboration in securing DL techniques within the context of the IoV, thereby fostering a safer and more resilient future for vehicular communication and connectivity.

Title: ESOD: Efficient Small Object Detection on High-Resolution Images

• Authors: Kai Liu, Zhihang Fu, Sheng Jin, Ze Chen, Fan Zhou, Rongxin Jiang, Yaowu Chen, Jieping Ye
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16424
• Pdf URL: https://arxiv.org/pdf/2407.16424
• Copy Paste: [[2407.16424]] ESOD: Efficient Small Object Detection on High-Resolution Images(https://arxiv.org/abs/2407.16424)
• Keywords: extraction
• Abstract: Enlarging input images is a straightforward and effective approach to promote small object detection. However, simple image enlargement is significantly expensive on both computations and GPU memory. In fact, small objects are usually sparsely distributed and locally clustered. Therefore, massive feature extraction computations are wasted on the non-target background area of images. Recent works have tried to pick out target-containing regions using an extra network and perform conventional object detection, but the newly introduced computation limits their final performance. In this paper, we propose to reuse the detector's backbone to conduct feature-level object-seeking and patch-slicing, which can avoid redundant feature extraction and reduce the computation cost. Incorporating a sparse detection head, we are able to detect small objects on high-resolution inputs (e.g., 1080P or larger) for superior performance. The resulting Efficient Small Object Detection (ESOD) approach is a generic framework, which can be applied to both CNN- and ViT-based detectors to save the computation and GPU memory costs. Extensive experiments demonstrate the efficacy and efficiency of our method. In particular, our method consistently surpasses the SOTA detectors by a large margin (e.g., 8% gains on AP) on the representative VisDrone, UAVDT, and TinyPerson datasets. Code will be made public soon.

Title: FairFlow: An Automated Approach to Model-based Counterfactual Data Augmentation For NLP

• Authors: Ewoenam Kwaku Tokpo, Toon Calders
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16431
• Pdf URL: https://arxiv.org/pdf/2407.16431
• Copy Paste: [[2407.16431]] FairFlow: An Automated Approach to Model-based Counterfactual Data Augmentation For NLP(https://arxiv.org/abs/2407.16431)
• Keywords: fair
• Abstract: Despite the evolution of language models, they continue to portray harmful societal biases and stereotypes inadvertently learned from training data. These inherent biases often result in detrimental effects in various applications. Counterfactual Data Augmentation (CDA), which seeks to balance demographic attributes in training data, has been a widely adopted approach to mitigate bias in natural language processing. However, many existing CDA approaches rely on word substitution techniques using manually compiled word-pair dictionaries. These techniques often lead to out-of-context substitutions, resulting in potential quality issues. The advancement of model-based techniques, on the other hand, has been challenged by the need for parallel training data. Works in this area resort to manually generated parallel data that are expensive to collect and are consequently limited in scale. This paper proposes FairFlow, an automated approach to generating parallel data for training counterfactual text generator models that limits the need for human intervention. Furthermore, we show that FairFlow significantly overcomes the limitations of dictionary-based word-substitution approaches whilst maintaining good performance.

Title: Enhancing LLM's Cognition via Structurization

• Authors: Kai Liu, Zhihang Fu, Chao Chen, Wei Zhang, Rongxin Jiang, Fan Zhou, Yaowu Chen, Yue Wu, Jieping Ye
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16434
• Pdf URL: https://arxiv.org/pdf/2407.16434
• Copy Paste: [[2407.16434]] Enhancing LLM's Cognition via Structurization(https://arxiv.org/abs/2407.16434)
• Keywords: large language model
• Abstract: When reading long-form text, human cognition is complex and structurized. While large language models (LLMs) process input contexts through a causal and sequential perspective, this approach can potentially limit their ability to handle intricate and complex inputs effectively. To enhance LLM's cognition capability, this paper presents a novel concept of context structurization. Specifically, we transform the plain, unordered contextual sentences into well-ordered and hierarchically structurized elements. By doing so, LLMs can better grasp intricate and extended contexts through precise attention and information-seeking along the organized structures. Extensive evaluations are conducted across various model architectures and sizes (including several 7B- to 72B-size auto-regressive LLMs as well as BERT-like masking models) on a diverse set of NLP tasks (e.g., context-based question-answering, exhaustive hallucination evaluation, and passage-level dense retrieval). Empirical results show consistent and significant performance gains afforded by a single-round structurization. In particular, we boost a 72B-parameter open-source model to achieve comparable performance against GPT-3.5-Turbo as the hallucination evaluator. Besides, we show the feasibility of distilling advanced LLMs' language processing abilities to a smaller yet effective StruXGPT-7B to execute structurization, addressing the practicality of our approach. Code will be made public soon.

Title: Can time series forecasting be automated? A benchmark and analysis

• Authors: Anvitha Thirthapura Sreedhara
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16445
• Pdf URL: https://arxiv.org/pdf/2407.16445
• Copy Paste: [[2407.16445]] Can time series forecasting be automated? A benchmark and analysis(https://arxiv.org/abs/2407.16445)
• Keywords: robust
• Abstract: In the field of machine learning and artificial intelligence, time series forecasting plays a pivotal role across various domains such as finance, healthcare, and weather. However, the task of selecting the most suitable forecasting method for a given dataset is a complex task due to the diversity of data patterns and characteristics. This research aims to address this challenge by proposing a comprehensive benchmark for evaluating and ranking time series forecasting methods across a wide range of datasets. This study investigates the comparative performance of many methods from two prominent time series forecasting frameworks, AutoGluon-Timeseries, and sktime to shed light on their applicability in different real-world scenarios. This research contributes to the field of time series forecasting by providing a robust benchmarking methodology and facilitating informed decision-making when choosing forecasting methods for achieving optimal prediction.

Title: MonoWAD: Weather-Adaptive Diffusion Model for Robust Monocular 3D Object Detection

• Authors: Youngmin Oh, Hyung-Il Kim, Seong Tae Kim, Jung Uk Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16448
• Pdf URL: https://arxiv.org/pdf/2407.16448
• Copy Paste: [[2407.16448]] MonoWAD: Weather-Adaptive Diffusion Model for Robust Monocular 3D Object Detection(https://arxiv.org/abs/2407.16448)
• Keywords: robust, diffusion
• Abstract: Monocular 3D object detection is an important challenging task in autonomous driving. Existing methods mainly focus on performing 3D detection in ideal weather conditions, characterized by scenarios with clear and optimal visibility. However, the challenge of autonomous driving requires the ability to handle changes in weather conditions, such as foggy weather, not just clear weather. We introduce MonoWAD, a novel weather-robust monocular 3D object detector with a weather-adaptive diffusion model. It contains two components: (1) the weather codebook to memorize the knowledge of the clear weather and generate a weather-reference feature for any input, and (2) the weather-adaptive diffusion model to enhance the feature representation of the input feature by incorporating a weather-reference feature. This serves an attention role in indicating how much improvement is needed for the input feature according to the weather conditions. To achieve this goal, we introduce a weather-adaptive enhancement loss to enhance the feature representation under both clear and foggy weather conditions. Extensive experiments under various weather conditions demonstrate that MonoWAD achieves weather-robust monocular 3D object detection. The code and dataset are released at this https URL.

Title: Lymphoid Infiltration Assessment of the Tumor Margins in H&E Slides

• Authors: Zhuxian Guo, Amine Marzouki, Jean-François Emile, Henning Müller, Camille Kurtz, Nicolas Loménie
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16464
• Pdf URL: https://arxiv.org/pdf/2407.16464
• Copy Paste: [[2407.16464]] Lymphoid Infiltration Assessment of the Tumor Margins in H&E Slides(https://arxiv.org/abs/2407.16464)
• Keywords: segmentation
• Abstract: Lymphoid infiltration at tumor margins is a key prognostic marker in solid tumors, playing a crucial role in guiding immunotherapy decisions. Current assessment methods, heavily reliant on immunohistochemistry (IHC), face challenges in tumor margin delineation and are affected by tissue preservation conditions. In contrast, we propose a Hematoxylin and Eosin (H&E) staining-based approach, underpinned by an advanced lymphocyte segmentation model trained on a public dataset for the precise detection of CD3+ and CD20+ lymphocytes. In our colorectal cancer study, we demonstrate that our H&E-based method offers a compelling alternative to traditional IHC, achieving comparable results in many cases. Our method's validity is further explored through a Turing test, involving blinded assessments by a pathologist of anonymized curves from H&E and IHC slides. This approach invites the medical community to consider Turing tests as a standard for evaluating medical applications involving expert human evaluation, thereby opening new avenues for enhancing cancer management and immunotherapy planning.

Title: Side-Channel Analysis of OpenVINO-based Neural Network Models

• Authors: Dirmanto Jap, Jakub Breier, Zdenko Lehocký, Shivam Bhasin, Xiaolu Hou
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16467
• Pdf URL: https://arxiv.org/pdf/2407.16467
• Copy Paste: [[2407.16467]] Side-Channel Analysis of OpenVINO-based Neural Network Models(https://arxiv.org/abs/2407.16467)
• Keywords: security, attack
• Abstract: Embedded devices with neural network accelerators offer great versatility for their users, reducing the need to use cloud-based services. At the same time, they introduce new security challenges in the area of hardware attacks, the most prominent being side-channel analysis (SCA). It was shown that SCA can recover model parameters with a high accuracy, posing a threat to entities that wish to keep their models confidential. In this paper, we explore the susceptibility of quantized models implemented in OpenVINO, an embedded framework for deploying neural networks on embedded and Edge devices. We show that it is possible to recover model parameters with high precision, allowing the recovered model to perform very close to the original one. Our experiments on GoogleNet v1 show only a 1% difference in the Top 1 and a 0.64% difference in the Top 5 accuracies.

Title: Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Models

• Authors: Kenza Benkirane (1), Laura Gongas (1), Shahar Pelles (1), Naomi Fuchs (1), Joshua Darmon (1), Pontus Stenetorp (1), David Ifeoluwa Adelani (1), Eduardo Sanchez (1 and 2) ((1) University College London, (2) Meta)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16470
• Pdf URL: https://arxiv.org/pdf/2407.16470
• Copy Paste: [[2407.16470]] Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Models(https://arxiv.org/abs/2407.16470)
• Keywords: large language model
• Abstract: Recent advancements in massively multilingual machine translation systems have significantly enhanced translation accuracy; however, even the best performing systems still generate hallucinations, severely impacting user trust. Detecting hallucinations in Machine Translation (MT) remains a critical challenge, particularly since existing methods excel with High-Resource Languages (HRLs) but exhibit substantial limitations when applied to Low-Resource Languages (LRLs). This paper evaluates hallucination detection approaches using Large Language Models (LLMs) and semantic similarity within massively multilingual embeddings. Our study spans 16 language directions, covering HRLs, LRLs, with diverse scripts. We find that the choice of model is essential for performance. On average, for HRLs, Llama3-70B outperforms the previous state of the art by as much as 0.16 MCC (Matthews Correlation Coefficient). However, for LRLs we observe that Claude Sonnet outperforms other LLMs on average by 0.03 MCC. The key takeaway from our study is that LLMs can achieve performance comparable or even better than previously proposed models, despite not being explicitly trained for any machine translation task. However, their advantage is less significant for LRLs.

Title: CrudiTEE: A Stick-and-Carrot Approach to Building Trustworthy Cryptocurrency Wallets with TEEs

• Authors: Lulu Zhou, Zeyu Liu, Fan Zhang, Michael K. Reiter
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.16473
• Pdf URL: https://arxiv.org/pdf/2407.16473
• Copy Paste: [[2407.16473]] CrudiTEE: A Stick-and-Carrot Approach to Building Trustworthy Cryptocurrency Wallets with TEEs(https://arxiv.org/abs/2407.16473)
• Keywords: attack
• Abstract: Cryptocurrency introduces usability challenges by requiring users to manage signing keys. Popular signing key management services (e.g., custodial wallets), however, either introduce a trusted party or burden users with managing signing key shares, posing the same usability challenges. TEEs (Trusted Execution Environments) are a promising technology to avoid both, but practical implementations of TEEs suffer from various side-channel attacks that have proven hard to eliminate. This paper explores a new approach to side-channel mitigation through economic incentives for TEE-based cryptocurrency wallet solutions. By taking the cost and profit of side-channel attacks into consideration, we designed a Stick-and-Carrot-based cryptocurrency wallet, CrudiTEE, that leverages penalties (the stick) and rewards (the carrot) to disincentivize attackers from exfiltrating signing keys in the first place. We model the attacker's behavior using a Markov Decision Process (MDP) to evaluate the effectiveness of the bounty and enable the service provider to adjust the parameters of the bounty's reward function accordingly.

Title: qMRI Diffusor: Quantitative T1 Mapping of the Brain using a Denoising Diffusion Probabilistic Model

• Authors: Shishuai Wang, Hua Ma, Juan A. Hernandez-Tamames, Stefan Klein, Dirk H.J. Poot
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16477
• Pdf URL: https://arxiv.org/pdf/2407.16477
• Copy Paste: [[2407.16477]] qMRI Diffusor: Quantitative T1 Mapping of the Brain using a Denoising Diffusion Probabilistic Model(https://arxiv.org/abs/2407.16477)
• Keywords: diffusion, generative
• Abstract: Quantitative MRI (qMRI) offers significant advantages over weighted images by providing objective parameters related to tissue properties. Deep learning-based methods have demonstrated effectiveness in estimating quantitative maps from series of weighted images. In this study, we present qMRI Diffusor, a novel approach to qMRI utilising deep generative models. Specifically, we implemented denoising diffusion probabilistic models (DDPM) for T1 quantification in the brain, framing the estimation of quantitative maps as a conditional generation task. The proposed method is compared with the residual neural network (ResNet) and the recurrent inference machine (RIM) on both phantom and in vivo data. The results indicate that our method achieves improved accuracy and precision in parameter estimation, along with superior visual performance. Moreover, our method inherently incorporates stochasticity, enabling straightforward quantification of uncertainty. Hence, the proposed method holds significant promise for quantitative MR mapping.

Title: Dynamic Retraining-Updating Mean Teacher for Source-Free Object Detection

• Authors: Trinh Le Ba Khanh, Huy-Hung Nguyen, Long Hoang Pham, Duong Nguyen-Ngoc Tran, Jae Wook Jeon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16497
• Pdf URL: https://arxiv.org/pdf/2407.16497
• Copy Paste: [[2407.16497]] Dynamic Retraining-Updating Mean Teacher for Source-Free Object Detection(https://arxiv.org/abs/2407.16497)
• Keywords: privacy
• Abstract: In object detection, unsupervised domain adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. However, UDA's reliance on labeled source data restricts its adaptability in privacy-related scenarios. This study focuses on source-free object detection (SFOD), which adapts a source-trained detector to an unlabeled target domain without using labeled source data. Recent advancements in self-training, particularly with the Mean Teacher (MT) framework, show promise for SFOD deployment. However, the absence of source supervision significantly compromises the stability of these approaches. We identify two primary issues, (1) uncontrollable degradation of the teacher model due to inopportune updates from the student model, and (2) the student model's tendency to replicate errors from incorrect pseudo labels, leading to it being trapped in a local optimum. Both factors contribute to a detrimental circular dependency, resulting in rapid performance degradation in recent self-training frameworks. To tackle these challenges, we propose the Dynamic Retraining-Updating (DRU) mechanism, which actively manages the student training and teacher updating processes to achieve co-evolutionary training. Additionally, we introduce Historical Student Loss to mitigate the influence of incorrect pseudo labels. Our method achieves state-of-the-art performance in the SFOD setting on multiple domain adaptation benchmarks, comparable to or even surpassing advanced UDA methods. The code will be released at this https URL

Title: HDRSplat: Gaussian Splatting for High Dynamic Range 3D Scene Reconstruction from Raw Images

• Authors: Shreyas Singh, Aryan Garg, Kaushik Mitra
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2407.16503
• Pdf URL: https://arxiv.org/pdf/2407.16503
• Copy Paste: [[2407.16503]] HDRSplat: Gaussian Splatting for High Dynamic Range 3D Scene Reconstruction from Raw Images(https://arxiv.org/abs/2407.16503)
• Keywords: extraction
• Abstract: The recent advent of 3D Gaussian Splatting (3DGS) has revolutionized the 3D scene reconstruction space enabling high-fidelity novel view synthesis in real-time. However, with the exception of RawNeRF, all prior 3DGS and NeRF-based methods rely on 8-bit tone-mapped Low Dynamic Range (LDR) images for scene reconstruction. Such methods struggle to achieve accurate reconstructions in scenes that require a higher dynamic range. Examples include scenes captured in nighttime or poorly lit indoor spaces having a low signal-to-noise ratio, as well as daylight scenes with shadow regions exhibiting extreme contrast. Our proposed method HDRSplat tailors 3DGS to train directly on 14-bit linear raw images in near darkness which preserves the scenes' full dynamic range and content. Our key contributions are two-fold: Firstly, we propose a linear HDR space-suited loss that effectively extracts scene information from noisy dark regions and nearly saturated bright regions simultaneously, while also handling view-dependent colors without increasing the degree of spherical harmonics. Secondly, through careful rasterization tuning, we implicitly overcome the heavy reliance and sensitivity of 3DGS on point cloud initialization. This is critical for accurate reconstruction in regions of low texture, high depth of field, and low illumination. HDRSplat is the fastest method to date that does 14-bit (HDR) 3D scene reconstruction in $\le$15 minutes/scene ($\sim$30x faster than prior state-of-the-art RawNeRF). It also boasts the fastest inference speed at $\ge$120fps. We further demonstrate the applicability of our HDR scene reconstruction by showcasing various applications like synthetic defocus, dense depth map extraction, and post-capture control of exposure, tone-mapping and view-point.

Title: Language-Based Security for Low-Level MPC

• Authors: Christian Skalka, Joseph P. Near
• Subjects: cs.CR, cs.PL
• Abstract URL: https://arxiv.org/abs/2407.16504
• Pdf URL: https://arxiv.org/pdf/2407.16504
• Copy Paste: [[2407.16504]] Language-Based Security for Low-Level MPC(https://arxiv.org/abs/2407.16504)
• Keywords: secure, security, privacy, robust
• Abstract: Secure Multi-Party Computation (MPC) is an important enabling technology for data privacy in modern distributed applications. Currently, proof methods for low-level MPC protocols are primarily manual and thus tedious and error-prone, and are also non-standardized and unfamiliar to most PL theorists. As a step towards better language support and language-based enforcement, we develop a new staged PL for defining a variety of low-level probabilistic MPC protocols. We also formulate a collection of confidentiality and integrity hyperproperties for our language model that are familiar from information flow, including conditional noninterference, gradual release, and robust declassification. We demonstrate their relation to standard MPC threat models of passive and malicious security, and how they can be leveraged in security verification of protocols. To prove these properties we develop automated tactics in $\mathbb{F}_2$ that can be integrated with separation logic-style reasoning.

Title: DreamVTON: Customizing 3D Virtual Try-on with Personalized Diffusion Models

• Authors: Zhenyu Xie, Haoye Dong, Yufei Gao, Zehua Ma, Xiaodan Liang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16511
• Pdf URL: https://arxiv.org/pdf/2407.16511
• Copy Paste: [[2407.16511]] DreamVTON: Customizing 3D Virtual Try-on with Personalized Diffusion Models(https://arxiv.org/abs/2407.16511)
• Keywords: diffusion, generative
• Abstract: Image-based 3D Virtual Try-ON (VTON) aims to sculpt the 3D human according to person and clothes images, which is data-efficient (i.e., getting rid of expensive 3D data) but challenging. Recent text-to-3D methods achieve remarkable improvement in high-fidelity 3D human generation, demonstrating its potential for 3D virtual try-on. Inspired by the impressive success of personalized diffusion models (e.g., Dreambooth and LoRA) for 2D VTON, it is straightforward to achieve 3D VTON by integrating the personalization technique into the diffusion-based text-to-3D framework. However, employing the personalized module in a pre-trained diffusion model (e.g., StableDiffusion (SD)) would degrade the model's capability for multi-view or multi-domain synthesis, which is detrimental to the geometry and texture optimization guided by Score Distillation Sampling (SDS) loss. In this work, we propose a novel customizing 3D human try-on model, named \textbf{DreamVTON}, to separately optimize the geometry and texture of the 3D human. Specifically, a personalized SD with multi-concept LoRA is proposed to provide the generative prior about the specific person and clothes, while a Densepose-guided ControlNet is exploited to guarantee consistent prior about body pose across various camera views. Besides, to avoid the inconsistent multi-view priors from the personalized SD dominating the optimization, DreamVTON introduces a template-based optimization mechanism, which employs mask templates for geometry shape learning and normal/RGB templates for geometry/texture details learning. Furthermore, for the geometry optimization phase, DreamVTON integrates a normal-style LoRA into personalized SD to enhance normal map generative prior, facilitating smooth geometry modeling.

Title: AMONGAGENTS: Evaluating Large Language Models in the Interactive Text-Based Social Deduction Game

• Authors: Yizhou Chi, Lingjun Mao, Zineng Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16521
• Pdf URL: https://arxiv.org/pdf/2407.16521
• Copy Paste: [[2407.16521]] AMONGAGENTS: Evaluating Large Language Models in the Interactive Text-Based Social Deduction Game(https://arxiv.org/abs/2407.16521)
• Keywords: large language model
• Abstract: Strategic social deduction games serve as valuable testbeds for evaluating the understanding and inference skills of language models, offering crucial insights into social science, artificial intelligence, and strategic gaming. This paper focuses on creating proxies of human behavior in simulated environments, with \textit{Among Us} utilized as a tool for studying simulated human behavior. The study introduces a text-based game environment, named AmongAgent, that mirrors the dynamics of \textit{Among Us}. Players act as crew members aboard a spaceship, tasked with identifying impostors who are sabotaging the ship and eliminating the crew. Within this environment, the behavior of simulated language agents is analyzed. The experiments involve diverse game sequences featuring different configurations of Crewmates and Impostor personality archetypes. Our work demonstrates that state-of-the-art large language models (LLMs) can effectively grasp the game rules and make decisions based on the current context. This work aims to promote further exploration of LLMs in goal-oriented games with incomplete information and complex action spaces, as these settings offer valuable opportunities to assess language model performance in socially driven scenarios.

Title: Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Models

• Authors: Aristeidis Panos, Rahaf Aljundi, Daniel Olmeda Reino, Richard E Turner
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16526
• Pdf URL: https://arxiv.org/pdf/2407.16526
• Copy Paste: [[2407.16526]] Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Models(https://arxiv.org/abs/2407.16526)
• Keywords: robust
• Abstract: Vision language models (VLMs) demonstrate impressive capabilities in visual question answering and image captioning, acting as a crucial link between visual and language models. However, existing open-source VLMs heavily rely on pretrained and frozen vision encoders (such as CLIP). Despite CLIP's robustness across diverse domains, it still exhibits non-negligible image understanding errors. These errors propagate to the VLM responses, resulting in sub-optimal performance. In our work, we propose an efficient and robust method for updating vision encoders within VLMs. Our approach selectively and locally updates encoders, leading to substantial performance improvements on data where previous mistakes occurred, while maintaining overall robustness. Furthermore, we demonstrate the effectiveness of our method during continual few-shot updates. Theoretical grounding, generality, and computational efficiency characterize our approach.

Title: Enhancing Encrypted Internet Traffic Classification Through Advanced Data Augmentation Techniques

• Authors: Yehonatan Zion, Porat Aharon, Ran Dubin, Amit Dvir, Chen Hajaj
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16539
• Pdf URL: https://arxiv.org/pdf/2407.16539
• Copy Paste: [[2407.16539]] Enhancing Encrypted Internet Traffic Classification Through Advanced Data Augmentation Techniques(https://arxiv.org/abs/2407.16539)
• Keywords: robust
• Abstract: The increasing popularity of online services has made Internet Traffic Classification a critical field of study. However, the rapid development of internet protocols and encryption limits usable data availability. This paper addresses the challenges of classifying encrypted internet traffic, focusing on the scarcity of open-source datasets and limitations of existing ones. We propose two Data Augmentation (DA) techniques to synthetically generate data based on real samples: Average augmentation and MTU augmentation. Both augmentations are aimed to improve the performance of the classifier, each from a different perspective: The Average augmentation aims to increase dataset size by generating new synthetic samples, while the MTU augmentation enhances classifier robustness to varying Maximum Transmission Units (MTUs). Our experiments, conducted on two well-known academic datasets and a commercial dataset, demonstrate the effectiveness of these approaches in improving model performance and mitigating constraints associated with limited and homogeneous datasets. Our findings underscore the potential of data augmentation in addressing the challenges of modern internet traffic classification. Specifically, we show that our augmentation techniques significantly enhance encrypted traffic classification models. This improvement can positively impact user Quality of Experience (QoE) by more accurately classifying traffic as video streaming (e.g., YouTube) or chat (e.g., Google Chat). Additionally, it can enhance Quality of Service (QoS) for file downloading activities (e.g., Google Docs).

Title: MicroEmo: Time-Sensitive Multimodal Emotion Recognition with Micro-Expression Dynamics in Video Dialogues

• Authors: Liyun Zhang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2407.16552
• Pdf URL: https://arxiv.org/pdf/2407.16552
• Copy Paste: [[2407.16552]] MicroEmo: Time-Sensitive Multimodal Emotion Recognition with Micro-Expression Dynamics in Video Dialogues(https://arxiv.org/abs/2407.16552)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable multimodal emotion recognition capabilities, integrating multimodal cues from visual, acoustic, and linguistic contexts in the video to recognize human emotional states. However, existing methods ignore capturing local facial features of temporal dynamics of micro-expressions and do not leverage the contextual dependencies of the utterance-aware temporal segments in the video, thereby limiting their expected effectiveness to a certain extent. In this work, we propose MicroEmo, a time-sensitive MLLM aimed at directing attention to the local facial micro-expression dynamics and the contextual dependencies of utterance-aware video clips. Our model incorporates two key architectural contributions: (1) a global-local attention visual encoder that integrates global frame-level timestamp-bound image features with local facial features of temporal dynamics of micro-expressions; (2) an utterance-aware video Q-Former that captures multi-scale and contextual dependencies by generating visual token sequences for each utterance segment and for the entire video then combining them. Preliminary qualitative experiments demonstrate that in a new Explainable Multimodal Emotion Recognition (EMER) task that exploits multi-modal and multi-faceted clues to predict emotions in an open-vocabulary (OV) manner, MicroEmo demonstrates its effectiveness compared with the latest methods.

Title: COALA: A Practical and Vision-Centric Federated Learning Platform

• Authors: Weiming Zhuang, Jian Xu, Chen Chen, Jingtao Li, Lingjuan Lyu
• Subjects: cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2407.16560
• Pdf URL: https://arxiv.org/pdf/2407.16560
• Copy Paste: [[2407.16560]] COALA: A Practical and Vision-Centric Federated Learning Platform(https://arxiv.org/abs/2407.16560)
• Keywords: federate, segmentation
• Abstract: We present COALA, a vision-centric Federated Learning (FL) platform, and a suite of benchmarks for practical FL scenarios, which we categorize into three levels: task, data, and model. At the task level, COALA extends support from simple classification to 15 computer vision tasks, including object detection, segmentation, pose estimation, and more. It also facilitates federated multiple-task learning, allowing clients to tackle multiple tasks simultaneously. At the data level, COALA goes beyond supervised FL to benchmark both semi-supervised FL and unsupervised FL. It also benchmarks feature distribution shifts other than commonly considered label distribution shifts. In addition to dealing with static data, it supports federated continual learning for continuously changing data in real-world scenarios. At the model level, COALA benchmarks FL with split models and different models in different clients. COALA platform offers three degrees of customization for these practical FL scenarios, including configuration customization, components customization, and workflow customization. We conduct systematic benchmarking experiments for the practical FL scenarios and highlight potential opportunities for further advancements in FL. Codes are open sourced at this https URL.

Title: Retrieve, Generate, Evaluate: A Case Study for Medical Paraphrases Generation with Small Language Models

• Authors: Ioana Buhnila, Aman Sinha, Mathieu Constant
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16565
• Pdf URL: https://arxiv.org/pdf/2407.16565
• Copy Paste: [[2407.16565]] Retrieve, Generate, Evaluate: A Case Study for Medical Paraphrases Generation with Small Language Models(https://arxiv.org/abs/2407.16565)
• Keywords: large language model
• Abstract: Recent surge in the accessibility of large language models (LLMs) to the general population can lead to untrackable use of such models for medical-related recommendations. Language generation via LLMs models has two key problems: firstly, they are prone to hallucination and therefore, for any medical purpose they require scientific and factual grounding; secondly, LLMs pose tremendous challenge to computational resources due to their gigantic model size. In this work, we introduce pRAGe, a pipeline for Retrieval Augmented Generation and evaluation of medical paraphrases generation using Small Language Models (SLM). We study the effectiveness of SLMs and the impact of external knowledge base for medical paraphrase generation in French.

Title: Exploring Automatic Cryptographic API Misuse Detection in the Era of LLMs

• Authors: Yifan Xia, Zichen Xie, Peiyu Liu, Kangjie Lu, Yan Liu, Wenhai Wang, Shouling Ji
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.16576
• Pdf URL: https://arxiv.org/pdf/2407.16576
• Copy Paste: [[2407.16576]] Exploring Automatic Cryptographic API Misuse Detection in the Era of LLMs(https://arxiv.org/abs/2407.16576)
• Keywords: security, large language model
• Abstract: While the automated detection of cryptographic API misuses has progressed significantly, its precision diminishes for intricate targets due to the reliance on manually defined patterns. Large Language Models (LLMs), renowned for their contextual understanding, offer a promising avenue to address existing shortcomings. However, applying LLMs in this security-critical domain presents challenges, particularly due to the unreliability stemming from LLMs' stochastic nature and the well-known issue of hallucination. To explore the prevalence of LLMs' unreliable analysis and potential solutions, this paper introduces a systematic evaluation framework to assess LLMs in detecting cryptographic misuses, utilizing a comprehensive dataset encompassing both manually-crafted samples and real-world projects. Our in-depth analysis of 11,940 LLM-generated reports highlights that the inherent instabilities in LLMs can lead to over half of the reports being false positives. Nevertheless, we demonstrate how a constrained problem scope, coupled with LLMs' self-correction capability, significantly enhances the reliability of the detection. The optimized approach achieves a remarkable detection rate of nearly 90%, surpassing traditional methods and uncovering previously unknown misuses in established benchmarks. Moreover, we identify the failure patterns that persistently hinder LLMs' reliability, including both cryptographic knowledge deficiency and code semantics misinterpretation. Guided by these insights, we develop an LLM-based workflow to examine open-source repositories, leading to the discovery of 63 real-world cryptographic misuses. Of these, 46 have been acknowledged by the development community, with 23 currently being addressed and 6 resolved. Reflecting on developers' feedback, we offer recommendations for future research and the development of LLM-based security tools.

Title: Shared Imagination: LLMs Hallucinate Alike

• Authors: Yilun Zhou, Caiming Xiong, Silvio Savarese, Chien-Sheng Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16604
• Pdf URL: https://arxiv.org/pdf/2407.16604
• Copy Paste: [[2407.16604]] Shared Imagination: LLMs Hallucinate Alike(https://arxiv.org/abs/2407.16604)
• Keywords: large language model
• Abstract: Despite the recent proliferation of large language models (LLMs), their training recipes -- model architecture, pre-training data and optimization algorithm -- are often very similar. This naturally raises the question of the similarity among the resulting models. In this paper, we propose a novel setting, imaginary question answering (IQA), to better understand model similarity. In IQA, we ask one model to generate purely imaginary questions (e.g., on completely made-up concepts in physics) and prompt another model to answer. Surprisingly, despite the total fictionality of these questions, all models can answer each other's questions with remarkable success, suggesting a "shared imagination space" in which these models operate during such hallucinations. We conduct a series of investigations into this phenomenon and discuss implications on model homogeneity, hallucination, and computational creativity.

Title: Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?

• Authors: Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, Noah A. Smith
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16607
• Pdf URL: https://arxiv.org/pdf/2407.16607
• Copy Paste: [[2407.16607]] Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?(https://arxiv.org/abs/2407.16607)
• Keywords: attack
• Abstract: The pretraining data of today's strongest language models is opaque. In particular, little is known about the proportions of various domains or languages represented. In this work, we tackle a task which we call data mixture inference, which aims to uncover the distributional make-up of training data. We introduce a novel attack based on a previously overlooked source of information -- byte-pair encoding (BPE) tokenizers, used by the vast majority of modern language models. Our key insight is that the ordered list of merge rules learned by a BPE tokenizer naturally reveals information about the token frequencies in its training data: the first merge is the most common byte pair, the second is the most common pair after merging the first token, and so on. Given a tokenizer's merge list along with data samples for each category of interest, we formulate a linear program that solves for the proportion of each category in the tokenizer's training set. Importantly, to the extent to which tokenizer training data is representative of the pretraining data, we indirectly learn about the pretraining data. In controlled experiments, we show that our attack recovers mixture ratios with high precision for tokenizers trained on known mixtures of natural languages, programming languages, and data sources. We then apply our approach to off-the-shelf tokenizers released with recent LMs. We confirm much publicly disclosed information about these models, and also make several new inferences: GPT-4o's tokenizer is much more multilingual than its predecessors, training on 39% non-English data; Llama3 extends GPT-3.5's tokenizer primarily for multilingual (48%) use; GPT-3.5's and Claude's tokenizers are trained on predominantly code (~60%). We hope our work sheds light on current design practices for pretraining data, and inspires continued research into data mixture inference for LMs.

Title: Lawma: The Power of Specialization for Legal Tasks

• Authors: Ricardo Dominguez-Olmedo, Vedant Nanda, Rediet Abebe, Stefan Bechtold, Christoph Engel, Jens Frankenreiter, Krishna Gummadi, Moritz Hardt, Michael Livermore
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16615
• Pdf URL: https://arxiv.org/pdf/2407.16615
• Copy Paste: [[2407.16615]] Lawma: The Power of Specialization for Legal Tasks(https://arxiv.org/abs/2407.16615)
• Keywords: large language model
• Abstract: Annotation and classification of legal text are central components of empirical legal research. Traditionally, these tasks are often delegated to trained research assistants. Motivated by the advances in language modeling, empirical legal scholars are increasingly turning to prompting commercial models, hoping that it will alleviate the significant cost of human annotation. Despite growing use, our understanding of how to best utilize large language models for legal tasks remains limited. We conduct a comprehensive study of 260 legal text classification tasks, nearly all new to the machine learning community. Starting from GPT-4 as a baseline, we show that it has non-trivial but highly varied zero-shot accuracy, often exhibiting performance that may be insufficient for legal work. We then demonstrate that a lightly fine-tuned Llama 3 model vastly outperforms GPT-4 on almost all tasks, typically by double-digit percentage points. We find that larger models respond better to fine-tuning than smaller models. A few tens to hundreds of examples suffice to achieve high classification accuracy. Notably, we can fine-tune a single model on all 260 tasks simultaneously at a small loss in accuracy relative to having a separate model for each task. Our work points to a viable alternative to the predominant practice of prompting commercial models. For concrete legal tasks with some available labeled data, researchers are better off using a fine-tuned open-source model.

Title: Course-Correction: Safety Alignment Using Synthetic Preferences

• Authors: Rongwu Xu, Yishuo Cai, Zhenhong Zhou, Renjie Gu, Haiqin Weng, Yan Liu, Tianwei Zhang, Wei Xu, Han Qiu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16637
• Pdf URL: https://arxiv.org/pdf/2407.16637
• Copy Paste: [[2407.16637]] Course-Correction: Safety Alignment Using Synthetic Preferences(https://arxiv.org/abs/2407.16637)
• Keywords: attack, large language model
• Abstract: The risk of harmful content generated by large language models (LLMs) becomes a critical concern. This paper presents a systematic study on assessing and improving LLMs' capability to perform the task of \textbf{course-correction}, \ie, the model can steer away from generating harmful content autonomously. To start with, we introduce the \textsc{C$^2$-Eval} benchmark for quantitative assessment and analyze 10 popular LLMs, revealing varying proficiency of current safety-tuned LLMs in course-correction. To improve, we propose fine-tuning LLMs with preference learning, emphasizing the preference for timely course-correction. Using an automated pipeline, we create \textsc{C$^2$-Syn}, a synthetic dataset with 750K pairwise preferences, to teach models the concept of timely course-correction through data-driven preference learning. Experiments on 2 LLMs, \textsc{Llama2-Chat 7B} and \textsc{Qwen2 7B}, show that our method effectively enhances course-correction skills without affecting general performance. Additionally, it effectively improves LLMs' safety, particularly in resisting jailbreak attacks.

Title: Unveiling and Mitigating Bias in Audio Visual Segmentation

• Authors: Peiwen Sun, Honggang Zhang, Di Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16638
• Pdf URL: https://arxiv.org/pdf/2407.16638
• Copy Paste: [[2407.16638]] Unveiling and Mitigating Bias in Audio Visual Segmentation(https://arxiv.org/abs/2407.16638)
• Keywords: transformer, segmentation
• Abstract: Community researchers have developed a range of advanced audio-visual segmentation models aimed at improving the quality of sounding objects' masks. While masks created by these models may initially appear plausible, they occasionally exhibit anomalies with incorrect grounding logic. We attribute this to real-world inherent preferences and distributions as a simpler signal for learning than the complex audio-visual grounding, which leads to the disregard of important modality information. Generally, the anomalous phenomena are often complex and cannot be directly observed systematically. In this study, we made a pioneering effort with the proper synthetic data to categorize and analyze phenomena as two types "audio priming bias" and "visual prior" according to the source of anomalies. For audio priming bias, to enhance audio sensitivity to different intensities and semantics, a perception module specifically for audio perceives the latent semantic information and incorporates information into a limited set of queries, namely active queries. Moreover, the interaction mechanism related to such active queries in the transformer decoder is customized to adapt to the need for interaction regulating among audio semantics. For visual prior, multiple contrastive training strategies are explored to optimize the model by incorporating a biased branch, without even changing the structure of the model. During experiments, observation demonstrates the presence and the impact that has been produced by the biases of the existing model. Finally, through experimental evaluation of AVS benchmarks, we demonstrate the effectiveness of our methods in handling both types of biases, achieving competitive performance across all three subsets.

Title: Deformable Convolution Based Road Scene Semantic Segmentation of Fisheye Images in Autonomous Driving

• Authors: Anam Manzoor, Aryan Singh, Ganesh Sistu, Reenu Mohandas, Eoin Grua, Anthony Scanlan, Ciarán Eising
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16647
• Pdf URL: https://arxiv.org/pdf/2407.16647
• Copy Paste: [[2407.16647]] Deformable Convolution Based Road Scene Semantic Segmentation of Fisheye Images in Autonomous Driving(https://arxiv.org/abs/2407.16647)
• Keywords: segmentation
• Abstract: This study investigates the effectiveness of modern Deformable Convolutional Neural Networks (DCNNs) for semantic segmentation tasks, particularly in autonomous driving scenarios with fisheye images. These images, providing a wide field of view, pose unique challenges for extracting spatial and geometric information due to dynamic changes in object attributes. Our experiments focus on segmenting the WoodScape fisheye image dataset into ten distinct classes, assessing the Deformable Networks' ability to capture intricate spatial relationships and improve segmentation accuracy. Additionally, we explore different loss functions to address class imbalance issues and compare the performance of conventional CNN architectures with Deformable Convolution-based CNNs, including Vanilla U-Net and Residual U-Net architectures. The significant improvement in mIoU score resulting from integrating Deformable CNNs demonstrates their effectiveness in handling the geometric distortions present in fisheye imagery, exceeding the performance of traditional CNN architectures. This underscores the significant role of Deformable convolution in enhancing semantic segmentation performance for fisheye imagery.

Title: Aggregated Attributions for Explanatory Analysis of 3D Segmentation Models

• Authors: Maciej Chrabaszcz, Hubert Baniecki, Piotr Komorowski, Szymon Płotka, Przemyslaw Biecek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16653
• Pdf URL: https://arxiv.org/pdf/2407.16653
• Copy Paste: [[2407.16653]] Aggregated Attributions for Explanatory Analysis of 3D Segmentation Models(https://arxiv.org/abs/2407.16653)
• Keywords: explainability, transformer, segmentation
• Abstract: Analysis of 3D segmentation models, especially in the context of medical imaging, is often limited to segmentation performance metrics that overlook the crucial aspect of explainability and bias. Currently, effectively explaining these models with saliency maps is challenging due to the high dimensions of input images multiplied by the ever-growing number of segmented class labels. To this end, we introduce Agg^2Exp, a methodology for aggregating fine-grained voxel attributions of the segmentation model's predictions. Unlike classical explanation methods that primarily focus on the local feature attribution, Agg^2Exp enables a more comprehensive global view on the importance of predicted segments in 3D images. Our benchmarking experiments show that gradient-based voxel attributions are more faithful to the model's predictions than perturbation-based explanations. As a concrete use-case, we apply Agg^2Exp to discover knowledge acquired by the Swin UNEt TRansformer model trained on the TotalSegmentator v2 dataset for segmenting anatomical structures in computed tomography medical images. Agg^2Exp facilitates the explanatory analysis of large segmentation models beyond their predictive performance.

Title: MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence

• Authors: Canyu Zhao, Mingyu Liu, Wen Wang, Jianlong Yuan, Hao Chen, Bo Zhang, Chunhua Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16655
• Pdf URL: https://arxiv.org/pdf/2407.16655
• Copy Paste: [[2407.16655]] MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequence(https://arxiv.org/abs/2407.16655)
• Keywords: diffusion
• Abstract: Recent advancements in video generation have primarily leveraged diffusion models for short-duration content. However, these approaches often fall short in modeling complex narratives and maintaining character consistency over extended periods, which is essential for long-form video production like movies. We propose MovieDreamer, a novel hierarchical framework that integrates the strengths of autoregressive models with diffusion-based rendering to pioneer long-duration video generation with intricate plot progressions and high visual fidelity. Our approach utilizes autoregressive models for global narrative coherence, predicting sequences of visual tokens that are subsequently transformed into high-quality video frames through diffusion rendering. This method is akin to traditional movie production processes, where complex stories are factorized down into manageable scene capturing. Further, we employ a multimodal script that enriches scene descriptions with detailed character information and visual style, enhancing continuity and character identity across scenes. We present extensive experiments across various movie genres, demonstrating that our approach not only achieves superior visual and narrative quality but also effectively extends the duration of generated content significantly beyond current capabilities. Homepage: this https URL.

Title: Towards scalable efficient on-device ASR with transfer learning

• Authors: Laxmi Pandey, Ke Li, Jinxi Guo, Debjyoti Paul, Arthur Guo, Jay Mahadeokar, Xuedong Zhang
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2407.16664
• Pdf URL: https://arxiv.org/pdf/2407.16664
• Copy Paste: [[2407.16664]] Towards scalable efficient on-device ASR with transfer learning(https://arxiv.org/abs/2407.16664)
• Keywords: robust
• Abstract: Multilingual pretraining for transfer learning significantly boosts the robustness of low-resource monolingual ASR models. This study systematically investigates three main aspects: (a) the impact of transfer learning on model performance during initial training or fine-tuning, (b) the influence of transfer learning across dataset domains and languages, and (c) the effect on rare-word recognition compared to non-rare words. Our finding suggests that RNNT-loss pretraining, followed by monolingual fine-tuning with Minimum Word Error Rate (MinWER) loss, consistently reduces Word Error Rates (WER) across languages like Italian and French. WER Reductions (WERR) reach 36.2% and 42.8% compared to monolingual baselines for MLS and in-house datasets. Out-of-domain pretraining leads to 28% higher WERR than in-domain pretraining. Both rare and non-rare words benefit, with rare words showing greater improvements with out-of-domain pretraining, and non-rare words with in-domain pretraining.

Title: RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent

• Authors: Huiyu Xu, Wenhui Zhang, Zhibo Wang, Feng Xiao, Rui Zheng, Yunhe Feng, Zhongjie Ba, Kui Ren
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16667
• Pdf URL: https://arxiv.org/pdf/2407.16667
• Copy Paste: [[2407.16667]] RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent(https://arxiv.org/abs/2407.16667)
• Keywords: attack, large language model
• Abstract: Recently, advanced Large Language Models (LLMs) such as GPT-4 have been integrated into many real-world applications like Code Copilot. These applications have significantly expanded the attack surface of LLMs, exposing them to a variety of threats. Among them, jailbreak attacks that induce toxic responses through jailbreak prompts have raised critical safety concerns. To identify these threats, a growing number of red teaming approaches simulate potential adversarial scenarios by crafting jailbreak prompts to test the target LLM. However, existing red teaming methods do not consider the unique vulnerabilities of LLM in different scenarios, making it difficult to adjust the jailbreak prompts to find context-specific vulnerabilities. Meanwhile, these methods are limited to refining jailbreak templates using a few mutation operations, lacking the automation and scalability to adapt to different scenarios. To enable context-aware and efficient red teaming, we abstract and model existing attacks into a coherent concept called "jailbreak strategy" and propose a multi-agent LLM system named RedAgent that leverages these strategies to generate context-aware jailbreak prompts. By self-reflecting on contextual feedback in an additional memory buffer, RedAgent continuously learns how to leverage these strategies to achieve effective jailbreaks in specific contexts. Extensive experiments demonstrate that our system can jailbreak most black-box LLMs in just five queries, improving the efficiency of existing red teaming methods by two times. Additionally, RedAgent can jailbreak customized LLM applications more efficiently. By generating context-aware jailbreak prompts towards applications on GPTs, we discover 60 severe vulnerabilities of these real-world applications with only two queries per vulnerability. We have reported all found issues and communicated with OpenAI and Meta for bug fixes.

Title: KAN or MLP: A Fairer Comparison

• Authors: Runpeng Yu, Weihao Yu, Xinchao Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.16674
• Pdf URL: https://arxiv.org/pdf/2407.16674
• Copy Paste: [[2407.16674]] KAN or MLP: A Fairer Comparison(https://arxiv.org/abs/2407.16674)
• Keywords: fair
• Abstract: This paper does not introduce a novel method. Instead, it offers a fairer and more comprehensive comparison of KAN and MLP models across various tasks, including machine learning, computer vision, audio processing, natural language processing, and symbolic formula representation. Specifically, we control the number of parameters and FLOPs to compare the performance of KAN and MLP. Our main observation is that, except for symbolic formula representation tasks, MLP generally outperforms KAN. We also conduct ablation studies on KAN and find that its advantage in symbolic formula representation mainly stems from its B-spline activation function. When B-spline is applied to MLP, performance in symbolic formula representation significantly improves, surpassing or matching that of KAN. However, in other tasks where MLP already excels over KAN, B-spline does not substantially enhance MLP's performance. Furthermore, we find that KAN's forgetting issue is more severe than that of MLP in a standard class-incremental continual learning setting, which differs from the findings reported in the KAN paper. We hope these results provide insights for future research on KAN and other MLP alternatives. Project link: this https URL

Title: SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation

• Authors: Pengfei Chen, Lingxi Xie, Xinyue Huo, Xuehui Yu, Xiaopeng Zhang, Yingfei Sun, Zhenjun Han, Qi Tian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16682
• Pdf URL: https://arxiv.org/pdf/2407.16682
• Copy Paste: [[2407.16682]] SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation(https://arxiv.org/abs/2407.16682)
• Keywords: segmentation
• Abstract: The Segment Anything model (SAM) has shown a generalized ability to group image pixels into patches, but applying it to semantic-aware segmentation still faces major challenges. This paper presents SAM-CP, a simple approach that establishes two types of composable prompts beyond SAM and composes them for versatile segmentation. Specifically, given a set of classes (in texts) and a set of SAM patches, the Type-I prompt judges whether a SAM patch aligns with a text label, and the Type-II prompt judges whether two SAM patches with the same text label also belong to the same instance. To decrease the complexity in dealing with a large number of semantic classes and patches, we establish a unified framework that calculates the affinity between (semantic and instance) queries and SAM patches and merges patches with high affinity to the query. Experiments show that SAM-CP achieves semantic, instance, and panoptic segmentation in both open and closed domains. In particular, it achieves state-of-the-art performance in open-vocabulary segmentation. Our research offers a novel and generalized methodology for equipping vision foundation models like SAM with multi-grained semantic perception abilities.

Title: Can Large Language Models Automatically Jailbreak GPT-4V?

• Authors: Yuanwei Wu, Yue Huang, Yixin Liu, Xiang Li, Pan Zhou, Lichao Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16686
• Pdf URL: https://arxiv.org/pdf/2407.16686
• Copy Paste: [[2407.16686]] Can Large Language Models Automatically Jailbreak GPT-4V?(https://arxiv.org/abs/2407.16686)
• Keywords: security, privacy, attack, large language model
• Abstract: GPT-4V has attracted considerable attention due to its extraordinary capacity for integrating and processing multimodal information. At the same time, its ability of face recognition raises new safety concerns of privacy leakage. Despite researchers' efforts in safety alignment through RLHF or preprocessing filters, vulnerabilities might still be exploited. In our study, we introduce AutoJailbreak, an innovative automatic jailbreak technique inspired by prompt optimization. We leverage Large Language Models (LLMs) for red-teaming to refine the jailbreak prompt and employ weak-to-strong in-context learning prompts to boost efficiency. Furthermore, we present an effective search method that incorporates early stopping to minimize optimization time and token expenditure. Our experiments demonstrate that AutoJailbreak significantly surpasses conventional methods, achieving an Attack Success Rate (ASR) exceeding 95.3\%. This research sheds light on strengthening GPT-4V security, underscoring the potential for LLMs to be exploited in compromising GPT-4V integrity.

Title: Explanation Regularisation through the Lens of Attributions

• Authors: Pedro Ferreira, Wilker Aziz, Ivan Titov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.16693
• Pdf URL: https://arxiv.org/pdf/2407.16693
• Copy Paste: [[2407.16693]] Explanation Regularisation through the Lens of Attributions(https://arxiv.org/abs/2407.16693)
• Keywords: robust
• Abstract: Explanation regularisation (ER) has been introduced as a way to guide models to make their predictions in a manner more akin to humans, i.e., making their attributions "plausible". This is achieved by introducing an auxiliary explanation loss, that measures how well the output of an input attribution technique for the model agrees with relevant human-annotated rationales. One positive outcome of using ER appears to be improved performance in out-of-domain (OOD) settings, presumably due to an increased reliance on "plausible" tokens. However, previous work has under-explored the impact of the ER objective on model attributions, in particular when obtained with techniques other than the one used to train ER. In this work, we contribute a study of ER's effectiveness at informing classification decisions on plausible tokens, and the relationship between increased plausibility and robustness to OOD conditions. Through a series of analyses, we find that the connection between ER and the ability of a classifier to rely on plausible features has been overstated and that a stronger reliance on plausible tokens does not seem to be the cause for any perceived OOD improvements.

Title: Aster: Fixing the Android TEE Ecosystem with Arm CCA

• Authors: Mark Kuhne, Supraja Sridhara, Andrin Bertschi, Nicolas Dutly, Srdjan Capkun, Shweta Shinde
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2407.16694
• Pdf URL: https://arxiv.org/pdf/2407.16694
• Copy Paste: [[2407.16694]] Aster: Fixing the Android TEE Ecosystem with Arm CCA(https://arxiv.org/abs/2407.16694)
• Keywords: secure, security, protect
• Abstract: The Android ecosystem relies on either TrustZone (e.g., OP-TEE, QTEE, Trusty) or trusted hypervisors (pKVM, Gunyah) to isolate security-sensitive services from malicious apps and Android bugs. TrustZone allows any secure world code to access the normal world that runs Android. Similarly, a trusted hypervisor has full access to Android running in one VM and security services in other VMs. In this paper, we motivate the need for mutual isolation, wherein Android, hypervisors, and the secure world are isolated from each other. Then, we propose a sandboxed service abstraction, such that a sandboxed execution cannot access any other sandbox, Android, hypervisor, or secure world memory. We present Aster which achieves these goals while ensuring that sandboxed execution can still communicate with Android to get inputs and provide outputs securely. Our main insight is to leverage the hardware isolation offered by Arm Confidential Computing Architecture (CCA). However, since CCA does not satisfy our sandboxing and mutual isolation requirements, Aster repurposes its hardware enforcement to meet its goals while addressing challenges such as secure interfaces, virtio, and protection against interrupts. We implement Aster to demonstrate its feasibility and assess its compatibility. We take three case studies, including one currently deployed on Android phones and insufficiently secured using a trusted hypervisor, to demonstrate that they can be protected by Aster.

Title: Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack

• Authors: Xiaoyue Xu, Qinyuan Ye, Xiang Ren
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.16695
• Pdf URL: https://arxiv.org/pdf/2407.16695
• Copy Paste: [[2407.16695]] Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack(https://arxiv.org/abs/2407.16695)
• Keywords: robust
• Abstract: We introduce Lifelong ICL, a problem setting that challenges long-context language models (LMs) to learn from a sequence of language tasks through in-context learning (ICL). We further introduce Task Haystack, an evaluation suite dedicated to assessing and diagnosing how long-context LMs utilizes contexts in Lifelong ICL. When given a task instruction and test inputs, long-context LMs are expected to leverage the relevant demonstrations in the Lifelong ICL prompt, avoid distraction and interference from other tasks, and achieve test accuracies that are not significantly worse than the Single-task ICL baseline. Task Haystack draws inspiration from the widely-adopted "needle-in-a-haystack" (NIAH) evaluation, but presents new and unique challenges. It demands that models (1) utilize the contexts with deeper understanding, rather than resorting to simple copying and pasting; (2) navigate through long streams of evolving topics and tasks, which closely approximates the complexities of real-world usage of long-context LMs. Additionally, Task Haystack inherits the controllability aspect of NIAH, providing model developers with tools and visualizations to identify model vulnerabilities effectively. We benchmark 12 long-context LMs using Task Haystack. We find that state-of-the-art closed models such as GPT-4o still struggle in this setting, failing 15% of the cases on average, while all open-weight models we evaluate further lack behind by a large margin, failing up to 61% of the cases. In our controlled analysis, we identify factors such as distraction and recency bias as contributors to these failure cases. Further, we observe declines in performance when task instructions are paraphrased at test time or when ICL demonstrations are repeated excessively, raising concerns about the robustness, instruction understanding, and true context utilization of current long-context LMs.

Title: PartGLEE: A Foundation Model for Recognizing and Parsing Any Objects

• Authors: Junyi Li, Junfeng Wu, Weizhi Zhao, Song Bai, Xiang Bai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16696
• Pdf URL: https://arxiv.org/pdf/2407.16696
• Copy Paste: [[2407.16696]] PartGLEE: A Foundation Model for Recognizing and Parsing Any Objects(https://arxiv.org/abs/2407.16696)
• Keywords: segmentation
• Abstract: We present PartGLEE, a part-level foundation model for locating and identifying both objects and parts in images. Through a unified framework, PartGLEE accomplishes detection, segmentation, and grounding of instances at any granularity in the open world scenario. Specifically, we propose a Q-Former to construct the hierarchical relationship between objects and parts, parsing every object into corresponding semantic parts. By incorporating a large amount of object-level data, the hierarchical relationships can be extended, enabling PartGLEE to recognize a rich variety of parts. We conduct comprehensive studies to validate the effectiveness of our method, PartGLEE achieves the state-of-the-art performance across various part-level tasks and obtain competitive results on object-level tasks. The proposed PartGLEE significantly enhances hierarchical modeling capabilities and part-level perception over our previous GLEE model. Further analysis indicates that the hierarchical cognitive ability of PartGLEE is able to facilitate a detailed comprehension in images for mLLMs. The model and code will be released at this https URL .

Title: AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarking

• Authors: Wenxuan Li, Chongyu Qu, Xiaoxi Chen, Pedro R. A. S. Bassi, Yijia Shi, Yuxiang Lai, Qian Yu, Huimin Xue, Yixiong Chen, Xiaorui Lin, Yutong Tang, Yining Cao, Haoqi Han, Zheyuan Zhang, Jiawei Liu, Tiezheng Zhang, Yujiu Ma, Jincheng Wang, Guang Zhang, Alan Yuille, Zongwei Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16697
• Pdf URL: https://arxiv.org/pdf/2407.16697
• Copy Paste: [[2407.16697]] AbdomenAtlas: A Large-Scale, Detailed-Annotated, & Multi-Center Dataset for Efficient Transfer Learning and Open Algorithmic Benchmarking(https://arxiv.org/abs/2407.16697)
• Keywords: segmentation
• Abstract: We introduce the largest abdominal CT dataset (termed AbdomenAtlas) of 20,460 three-dimensional CT volumes sourced from 112 hospitals across diverse populations, geographies, and facilities. AbdomenAtlas provides 673K high-quality masks of anatomical structures in the abdominal region annotated by a team of 10 radiologists with the help of AI algorithms. We start by having expert radiologists manually annotate 22 anatomical structures in 5,246 CT volumes. Following this, a semi-automatic annotation procedure is performed for the remaining CT volumes, where radiologists revise the annotations predicted by AI, and in turn, AI improves its predictions by learning from revised annotations. Such a large-scale, detailed-annotated, and multi-center dataset is needed for two reasons. Firstly, AbdomenAtlas provides important resources for AI development at scale, branded as large pre-trained models, which can alleviate the annotation workload of expert radiologists to transfer to broader clinical applications. Secondly, AbdomenAtlas establishes a large-scale benchmark for evaluating AI algorithms -- the more data we use to test the algorithms, the better we can guarantee reliable performance in complex clinical scenarios. An ISBI & MICCAI challenge named BodyMaps: Towards 3D Atlas of Human Body was launched using a subset of our AbdomenAtlas, aiming to stimulate AI innovation and to benchmark segmentation accuracy, inference efficiency, and domain generalizability. We hope our AbdomenAtlas can set the stage for larger-scale clinical trials and offer exceptional opportunities to practitioners in the medical imaging community. Codes, models, and datasets are available at this https URL

Title: Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions

• Authors: Fabio Tosi, Pierluigi Zama Ramirez, Matteo Poggi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2407.16698
• Pdf URL: https://arxiv.org/pdf/2407.16698
• Copy Paste: [[2407.16698]] Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions(https://arxiv.org/abs/2407.16698)
• Keywords: diffusion
• Abstract: We present a novel approach designed to address the complexities posed by challenging, out-of-distribution data in the single-image depth estimation task. Starting with images that facilitate depth prediction due to the absence of unfavorable factors, we systematically generate new, user-defined scenes with a comprehensive set of challenges and associated depth information. This is achieved by leveraging cutting-edge text-to-image diffusion models with depth-aware control, known for synthesizing high-quality image content from textual prompts while preserving the coherence of 3D structure between generated and source imagery. Subsequent fine-tuning of any monocular depth network is carried out through a self-distillation protocol that takes into account images generated using our strategy and its own depth predictions on simple, unchallenging scenes. Experiments on benchmarks tailored for our purposes demonstrate the effectiveness and versatility of our proposal.