2024-09-12

Title: HSR-KAN: Efficient Hyperspectral Image Super-Resolution via Kolmogorov-Arnold Networks

• Authors: Baisong Li, Xingwang Wang, Haixiao Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06705
• Pdf URL: https://arxiv.org/pdf/2409.06705
• Copy Paste: [[2409.06705]] HSR-KAN: Efficient Hyperspectral Image Super-Resolution via Kolmogorov-Arnold Networks(https://arxiv.org/abs/2409.06705)
• Keywords: extraction
• Abstract: Hyperspectral images (HSIs) have great potential in various visual tasks due to their rich spectral information. However, obtaining high-resolution hyperspectral images remains challenging due to limitations of physical imaging. Inspired by Kolmogorov-Arnold Networks (KANs), we propose an efficient HSI super-resolution (HSI-SR) model to fuse a low-resolution HSI (LR-HSI) and a high-resolution multispectral image (HR-MSI), yielding a high-resolution HSI (HR-HSI). To achieve the effective integration of spatial information from HR-MSI, we design a fusion module based on KANs, called KAN-Fusion. Further inspired by the channel attention mechanism, we design a spectral channel attention module called KAN Channel Attention Block (KAN-CAB) for post-fusion feature extraction. As a channel attention module integrated with KANs, KAN-CAB not only enhances the fine-grained adjustment ability of deep networks, enabling networks to accurately simulate details of spectral sequences and spatial textures, but also effectively avoid Curse of Dimensionality (COD). Extensive experiments show that, compared to current state-of-the-art (SOTA) HSI-SR methods, proposed HSR-KAN achieves the best performance in terms of both qualitative and quantitative assessments. Our code is available at: this https URL.

Title: McGrids: Monte Carlo-Driven Adaptive Grids for Iso-Surface Extraction

• Authors: Daxuan Renınst, Hezi Shiınst, Jianmin Zheng, Jianfei Cai
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.06710
• Pdf URL: https://arxiv.org/pdf/2409.06710
• Copy Paste: [[2409.06710]] McGrids: Monte Carlo-Driven Adaptive Grids for Iso-Surface Extraction(https://arxiv.org/abs/2409.06710)
• Keywords: extraction
• Abstract: Iso-surface extraction from an implicit field is a fundamental process in various applications of computer vision and graphics. When dealing with geometric shapes with complicated geometric details, many existing algorithms suffer from high computational costs and memory usage. This paper proposes McGrids, a novel approach to improve the efficiency of iso-surface extraction. The key idea is to construct adaptive grids for iso-surface extraction rather than using a simple uniform grid as prior art does. Specifically, we formulate the problem of constructing adaptive grids as a probability sampling problem, which is then solved by Monte Carlo process. We demonstrate McGrids' capability with extensive experiments from both analytical SDFs computed from surface meshes and learned implicit fields from real multiview images. The experiment results show that our McGrids can significantly reduce the number of implicit field queries, resulting in significant memory reduction, while producing high-quality meshes with rich geometric details.

Title: Feedback-based Modal Mutual Search for Attacking Vision-Language Pre-training Models

• Authors: Renhua Ding, Xinze Zhang, Xiao Yang, Kun He
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06726
• Pdf URL: https://arxiv.org/pdf/2409.06726
• Copy Paste: [[2409.06726]] Feedback-based Modal Mutual Search for Attacking Vision-Language Pre-training Models(https://arxiv.org/abs/2409.06726)
• Keywords: attack
• Abstract: Although vision-language pre-training (VLP) models have achieved remarkable progress on cross-modal tasks, they remain vulnerable to adversarial attacks. Using data augmentation and cross-modal interactions to generate transferable adversarial examples on surrogate models, transfer-based black-box attacks have become the mainstream methods in attacking VLP models, as they are more practical in real-world scenarios. However, their transferability may be limited due to the differences on feature representation across different models. To this end, we propose a new attack paradigm called Feedback-based Modal Mutual Search (FMMS). FMMS introduces a novel modal mutual loss (MML), aiming to push away the matched image-text pairs while randomly drawing mismatched pairs closer in feature space, guiding the update directions of the adversarial examples. Additionally, FMMS leverages the target model feedback to iteratively refine adversarial examples, driving them into the adversarial region. To our knowledge, this is the first work to exploit target model feedback to explore multi-modality adversarial boundaries. Extensive empirical evaluations on Flickr30K and MSCOCO datasets for image-text matching tasks show that FMMS significantly outperforms the state-of-the-art baselines.

Title: Data-efficient and Interpretable Inverse Materials Design using a Disentangled Variational Autoencoder

• Authors: Cheng Zeng, Zulqarnain Khan, Nathan L. Post
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2409.06740
• Pdf URL: https://arxiv.org/pdf/2409.06740
• Copy Paste: [[2409.06740]] Data-efficient and Interpretable Inverse Materials Design using a Disentangled Variational Autoencoder(https://arxiv.org/abs/2409.06740)
• Keywords: robust, interpretability
• Abstract: Inverse materials design has proven successful in accelerating novel material discovery. Many inverse materials design methods use unsupervised learning where a latent space is learned to offer a compact description of materials representations. A latent space learned this way is likely to be entangled, in terms of the target property and other properties of the materials. This makes the inverse design process ambiguous. Here, we present a semi-supervised learning approach based on a disentangled variational autoencoder to learn a probabilistic relationship between features, latent variables and target properties. This approach is data efficient because it combines all labelled and unlabelled data in a coherent manner, and it uses expert-informed prior distributions to improve model robustness even with limited labelled data. It is in essence interpretable, as the learnable target property is disentangled out of the other properties of the materials, and an extra layer of interpretability can be provided by a post-hoc analysis of the classification head of the model. We demonstrate this new approach on an experimental high-entropy alloy dataset with chemical compositions as input and single-phase formation as the single target property. While single property is used in this work, the disentangled model can be extended to customize for inverse design of materials with multiple target properties.

Title: Distributed Cooperative AI for Large-Scale Eigenvalue Computations Using Neural Networks

• Authors: Ronald Katende
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2409.06746
• Pdf URL: https://arxiv.org/pdf/2409.06746
• Copy Paste: [[2409.06746]] Distributed Cooperative AI for Large-Scale Eigenvalue Computations Using Neural Networks(https://arxiv.org/abs/2409.06746)
• Keywords: robust
• Abstract: This paper presents a novel method for eigenvalue computation using a distributed cooperative neural network framework. Unlike traditional techniques that struggle with scalability in large systems, our decentralized algorithm enables multiple autonomous agents to collaboratively estimate the smallest eigenvalue of large matrices. Each agent uses a localized neural network model, refining its estimates through inter-agent communication. Our approach guarantees convergence to the true eigenvalue, even with communication failures or network disruptions. Theoretical analysis confirms the robustness and accuracy of the method, while empirical results demonstrate its better performance compared to some traditional centralized algorithms

Title: EasyST: A Simple Framework for Spatio-Temporal Prediction

• Authors: Jiabin Tang, Wei Wei, Lianghao Xia, Chao Huang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06748
• Pdf URL: https://arxiv.org/pdf/2409.06748
• Copy Paste: [[2409.06748]] EasyST: A Simple Framework for Spatio-Temporal Prediction(https://arxiv.org/abs/2409.06748)
• Keywords: robust
• Abstract: Spatio-temporal prediction is a crucial research area in data-driven urban computing, with implications for transportation, public safety, and environmental monitoring. However, scalability and generalization challenges remain significant obstacles. Advanced models often rely on Graph Neural Networks to encode spatial and temporal correlations, but struggle with the increased complexity of large-scale datasets. The recursive GNN-based message passing schemes used in these models hinder their training and deployment in real-life urban sensing scenarios. Moreover, long-spanning large-scale spatio-temporal data introduce distribution shifts, necessitating improved generalization performance. To address these challenges, we propose a simple framework for spatio-temporal prediction - EasyST paradigm. It learns lightweight and robust Multi-Layer Perceptrons (MLPs) by effectively distilling knowledge from complex spatio-temporal GNNs. We ensure robust knowledge distillation by integrating the spatio-temporal information bottleneck with teacher-bounded regression loss, filtering out task-irrelevant noise and avoiding erroneous guidance. We further enhance the generalization ability of the student model by incorporating spatial and temporal prompts to provide downstream task contexts. Evaluation on three spatio-temporal datasets for urban computing tasks demonstrates that EasyST surpasses state-of-the-art approaches in terms of efficiency and accuracy. The implementation code is available at: this https URL.

Title: The Weak Form Is Stronger Than You Think

• Authors: Daniel A. Messenger, April Tran, Vanja Dukic, David M. Bortz
• Subjects: cs.LG, cs.CE, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.06751
• Pdf URL: https://arxiv.org/pdf/2409.06751
• Copy Paste: [[2409.06751]] The Weak Form Is Stronger Than You Think(https://arxiv.org/abs/2409.06751)
• Keywords: robust
• Abstract: The weak form is a ubiquitous, well-studied, and widely-utilized mathematical tool in modern computational and applied mathematics. In this work we provide a survey of both the history and recent developments for several fields in which the weak form can play a critical role. In particular, we highlight several recent advances in weak form versions of equation learning, parameter estimation, and coarse graining, which offer surprising noise robustness, accuracy, and computational efficiency. We note that this manuscript is a companion piece to our October 2024 SIAM News article of the same name. Here we provide more detailed explanations of mathematical developments as well as a more complete list of references. Lastly, we note that the software with which to reproduce the results in this manuscript is also available on our group's GitHub website this https URL .

Title: Beyond designer's knowledge: Generating materials design hypotheses via large language models

• Authors: Quanliang Liu, Maciej P. Polak, So Yeon Kim, MD Al Amin Shuvo, Hrishikesh Shridhar Deodhar, Jeongsoo Han, Dane Morgan, Hyunseok Oh
• Subjects: cs.LG, cond-mat.mtrl-sci, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06756
• Pdf URL: https://arxiv.org/pdf/2409.06756
• Copy Paste: [[2409.06756]] Beyond designer's knowledge: Generating materials design hypotheses via large language models(https://arxiv.org/abs/2409.06756)
• Keywords: large language model
• Abstract: Materials design often relies on human-generated hypotheses, a process inherently limited by cognitive constraints such as knowledge gaps and limited ability to integrate and extract knowledge implications, particularly when multidisciplinary expertise is required. This work demonstrates that large language models (LLMs), coupled with prompt engineering, can effectively generate non-trivial materials hypotheses by integrating scientific principles from diverse sources without explicit design guidance by human experts. These include design ideas for high-entropy alloys with superior cryogenic properties and halide solid electrolytes with enhanced ionic conductivity and formability. These design ideas have been experimentally validated in high-impact publications in 2023 not available in the LLM training data, demonstrating the LLM's ability to generate highly valuable and realizable innovative ideas not established in the literature. Our approach primarily leverages materials system charts encoding processing-structure-property relationships, enabling more effective data integration by condensing key information from numerous papers, and evaluation and categorization of numerous hypotheses for human cognition, both through the LLM. This LLM-driven approach opens the door to new avenues of artificial intelligence-driven materials discovery by accelerating design, democratizing innovation, and expanding capabilities beyond the designer's direct knowledge.

Title: Human Motion Synthesis_ A Diffusion Approach for Motion Stitching and In-Betweening

• Authors: Michael Adewole, Oluwaseyi Giwa, Favour Nerrise, Martins Osifeko, Ajibola Oyedeji
• Subjects: cs.CV, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06791
• Pdf URL: https://arxiv.org/pdf/2409.06791
• Copy Paste: [[2409.06791]] Human Motion Synthesis_ A Diffusion Approach for Motion Stitching and In-Betweening(https://arxiv.org/abs/2409.06791)
• Keywords: diffusion, transformer
• Abstract: Human motion generation is an important area of research in many fields. In this work, we tackle the problem of motion stitching and in-betweening. Current methods either require manual efforts, or are incapable of handling longer sequences. To address these challenges, we propose a diffusion model with a transformer-based denoiser to generate realistic human motion. Our method demonstrated strong performance in generating in-betweening sequences, transforming a variable number of input poses into smooth and realistic motion sequences consisting of 75 frames at 15 fps, resulting in a total duration of 5 seconds. We present the performance evaluation of our method using quantitative metrics such as Frechet Inception Distance (FID), Diversity, and Multimodality, along with visual assessments of the generated outputs.

Title: Adversarial Attacks to Multi-Modal Models

• Authors: Zhihao Dou, Xin Hu, Haibo Yang, Zhuqing Liu, Minghong Fang
• Subjects: cs.CR, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06793
• Pdf URL: https://arxiv.org/pdf/2409.06793
• Copy Paste: [[2409.06793]] Adversarial Attacks to Multi-Modal Models(https://arxiv.org/abs/2409.06793)
• Keywords: defense, attack
• Abstract: Multi-modal models have gained significant attention due to their powerful capabilities. These models effectively align embeddings across diverse data modalities, showcasing superior performance in downstream tasks compared to their unimodal counterparts. Recent study showed that the attacker can manipulate an image or audio file by altering it in such a way that its embedding matches that of an attacker-chosen targeted input, thereby deceiving downstream models. However, this method often underperforms due to inherent disparities in data from different modalities. In this paper, we introduce CrossFire, an innovative approach to attack multi-modal models. CrossFire begins by transforming the targeted input chosen by the attacker into a format that matches the modality of the original image or audio file. We then formulate our attack as an optimization problem, aiming to minimize the angular deviation between the embeddings of the transformed input and the modified image or audio file. Solving this problem determines the perturbations to be added to the original media. Our extensive experiments on six real-world benchmark datasets reveal that CrossFire can significantly manipulate downstream tasks, surpassing existing attacks. Additionally, we evaluate six defensive strategies against CrossFire, finding that current defenses are insufficient to counteract our CrossFire.

Title: Adaptive Meta-Domain Transfer Learning (AMDTL): A Novel Approach for Knowledge Transfer in AI

• Authors: Michele Laurelli
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06800
• Pdf URL: https://arxiv.org/pdf/2409.06800
• Copy Paste: [[2409.06800]] Adaptive Meta-Domain Transfer Learning (AMDTL): A Novel Approach for Knowledge Transfer in AI(https://arxiv.org/abs/2409.06800)
• Keywords: robust
• Abstract: This paper presents Adaptive Meta-Domain Transfer Learning (AMDTL), a novel methodology that combines principles of meta-learning with domain-specific adaptations to enhance the transferability of artificial intelligence models across diverse and unknown domains. AMDTL aims to address the main challenges of transfer learning, such as domain misalignment, negative transfer, and catastrophic forgetting, through a hybrid framework that emphasizes both generalization and contextual specialization. The framework integrates a meta-learner trained on a diverse distribution of tasks, adversarial training techniques for aligning domain feature distributions, and dynamic feature regulation mechanisms based on contextual domain embeddings. Experimental results on benchmark datasets demonstrate that AMDTL outperforms existing transfer learning methodologies in terms of accuracy, adaptation efficiency, and robustness. This research provides a solid theoretical and practical foundation for the application of AMDTL in various fields, opening new perspectives for the development of more adaptable and inclusive AI systems.

Title: Personalized Federated Learning Techniques: Empirical Analysis

• Authors: Azal Ahmad Khan, Ahmad Faraz Khan, Haider Ali, Ali Anwar
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.06805
• Pdf URL: https://arxiv.org/pdf/2409.06805
• Copy Paste: [[2409.06805]] Personalized Federated Learning Techniques: Empirical Analysis(https://arxiv.org/abs/2409.06805)
• Keywords: privacy, attack, federate
• Abstract: Personalized Federated Learning (pFL) holds immense promise for tailoring machine learning models to individual users while preserving data privacy. However, achieving optimal performance in pFL often requires a careful balancing act between memory overhead costs and model accuracy. This paper delves into the trade-offs inherent in pFL, offering valuable insights for selecting the right algorithms for diverse real-world scenarios. We empirically evaluate ten prominent pFL techniques across various datasets and data splits, uncovering significant differences in their performance. Our study reveals interesting insights into how pFL methods that utilize personalized (local) aggregation exhibit the fastest convergence due to their efficiency in communication and computation. Conversely, fine-tuning methods face limitations in handling data heterogeneity and potential adversarial attacks while multi-objective learning methods achieve higher accuracy at the cost of additional training and resource consumption. Our study emphasizes the critical role of communication efficiency in scaling pFL, demonstrating how it can significantly affect resource usage in real-world deployments.

Title: DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks

• Authors: Amin Karimi Monsefi, Kishore Prakash Sailaja, Ali Alilooee, Ser-Nam Lim, Rajiv Ramnath
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06809
• Pdf URL: https://arxiv.org/pdf/2409.06809
• Copy Paste: [[2409.06809]] DetailCLIP: Detail-Oriented CLIP for Fine-Grained Tasks(https://arxiv.org/abs/2409.06809)
• Keywords: robust, extraction, segmentation
• Abstract: In this paper, we introduce DetailCLIP: A Detail-Oriented CLIP to address the limitations of contrastive learning-based vision-language models, particularly CLIP, in handling detail-oriented and fine-grained tasks like segmentation. While CLIP and its variants excel in the global alignment of image and text representations, they often struggle to capture the fine-grained details necessary for precise segmentation. To overcome these challenges, we propose a novel framework that employs patch-level comparison of self-distillation and pixel-level reconstruction losses, enhanced with an attention-based token removal mechanism. This approach selectively retains semantically relevant tokens, enabling the model to focus on the image's critical regions aligned with the specific functions of our model, including textual information processing, patch comparison, and image reconstruction, ensuring that the model learns high-level semantics and detailed visual features. Our experiments demonstrate that DetailCLIP surpasses existing CLIP-based and traditional self-supervised learning (SSL) models in segmentation accuracy and exhibits superior generalization across diverse datasets. DetailCLIP represents a significant advancement in vision-language modeling, offering a robust solution for tasks that demand high-level semantic understanding and detailed feature extraction. this https URL.

Title: LLM-Enhanced Software Patch Localization

• Authors: Jinhong Yu, Yi Chen, Di Tang, Xiaozhong Liu, XiaoFeng Wang, Chen Wu, Haixu Tang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.06816
• Pdf URL: https://arxiv.org/pdf/2409.06816
• Copy Paste: [[2409.06816]] LLM-Enhanced Software Patch Localization(https://arxiv.org/abs/2409.06816)
• Keywords: security, large language model
• Abstract: Open source software (OSS) is integral to modern product development, and any vulnerability within it potentially compromises numerous products. While developers strive to apply security patches, pinpointing these patches among extensive OSS updates remains a challenge. Security patch localization (SPL) recommendation methods are leading approaches to address this. However, existing SPL models often falter when a commit lacks a clear association with its corresponding CVE, and do not consider a scenario that a vulnerability has multiple patches proposed over time before it has been fully resolved. To address these challenges, we introduce LLM-SPL, a recommendation-based SPL approach that leverages the capabilities of the Large Language Model (LLM) to locate the security patch commit for a given CVE. More specifically, we propose a joint learning framework, in which the outputs of LLM serves as additional features to aid our recommendation model in prioritizing security patches. Our evaluation on a dataset of 1,915 CVEs associated with 2,461 patches demonstrates that LLM-SPL excels in ranking patch commits, surpassing the state-of-the-art method in terms of Recall, while significantly reducing manual effort. Notably, for vulnerabilities requiring multiple patches, LLM-SPL significantly improves Recall by 22.83\%, NDCG by 19.41\%, and reduces manual effort by over 25\% when checking up to the top 10 rankings. The dataset and source code are available at \url{https://anonymous.4open.science/r/LLM-SPL-91F8}.

Title: PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation

• Authors: Ilya Gusev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06820
• Pdf URL: https://arxiv.org/pdf/2409.06820
• Copy Paste: [[2409.06820]] PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation(https://arxiv.org/abs/2409.06820)
• Keywords: robust
• Abstract: We introduce a novel benchmark for evaluating the role-playing capabilities of language models. Our approach leverages language models themselves to emulate users in dynamic, multi-turn conversations and to assess the resulting dialogues. The framework consists of three main components: a player model assuming a specific character role, an interrogator model simulating user behavior, and a judge model evaluating conversation quality. We conducted experiments comparing automated evaluations with human annotations to validate our approach, demonstrating strong correlations across multiple criteria. This work provides a foundation for a robust and dynamic evaluation of model capabilities in interactive scenarios.

Title: Sam2Rad: A Segmentation Model for Medical Images with Learnable Prompts

• Authors: Assefa Seyoum Wahd, Banafshe Felfeliyan, Yuyue Zhou, Shrimanti Ghosh, Adam McArthur, Jiechen Zhang, Jacob L. Jaremko, Abhilash Hareendranathan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06821
• Pdf URL: https://arxiv.org/pdf/2409.06821
• Copy Paste: [[2409.06821]] Sam2Rad: A Segmentation Model for Medical Images with Learnable Prompts(https://arxiv.org/abs/2409.06821)
• Keywords: segmentation
• Abstract: Foundation models like the segment anything model require high-quality manual prompts for medical image segmentation, which is time-consuming and requires expertise. SAM and its variants often fail to segment structures in ultrasound (US) images due to domain shift. We propose Sam2Rad, a prompt learning approach to adapt SAM and its variants for US bone segmentation without human prompts. It introduces a prompt predictor network (PPN) with a cross-attention module to predict prompt embeddings from image encoder features. PPN outputs bounding box and mask prompts, and 256-dimensional embeddings for regions of interest. The framework allows optional manual prompting and can be trained end-to-end using parameter-efficient fine-tuning (PEFT). Sam2Rad was tested on 3 musculoskeletal US datasets: wrist (3822 images), rotator cuff (1605 images), and hip (4849 images). It improved performance across all datasets without manual prompts, increasing Dice scores by 2-7% for hip/wrist and up to 33% for shoulder data. Sam2Rad can be trained with as few as 10 labeled images and is compatible with any SAM architecture for automatic segmentation.

Title: Cross-Modal Self-Supervised Learning with Effective Contrastive Units for LiDAR Point Clouds

• Authors: Mu Cai, Chenxu Luo, Yong Jae Lee, Xiaodong Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06827
• Pdf URL: https://arxiv.org/pdf/2409.06827
• Copy Paste: [[2409.06827]] Cross-Modal Self-Supervised Learning with Effective Contrastive Units for LiDAR Point Clouds(https://arxiv.org/abs/2409.06827)
• Keywords: segmentation
• Abstract: 3D perception in LiDAR point clouds is crucial for a self-driving vehicle to properly act in 3D environment. However, manually labeling point clouds is hard and costly. There has been a growing interest in self-supervised pre-training of 3D perception models. Following the success of contrastive learning in images, current methods mostly conduct contrastive pre-training on point clouds only. Yet an autonomous driving vehicle is typically supplied with multiple sensors including cameras and LiDAR. In this context, we systematically study single modality, cross-modality, and multi-modality for contrastive learning of point clouds, and show that cross-modality wins over other alternatives. In addition, considering the huge difference between the training sources in 2D images and 3D point clouds, it remains unclear how to design more effective contrastive units for LiDAR. We therefore propose the instance-aware and similarity-balanced contrastive units that are tailored for self-driving point clouds. Extensive experiments reveal that our approach achieves remarkable performance gains over various point cloud models across the downstream perception tasks of LiDAR based 3D object detection and 3D semantic segmentation on the four popular benchmarks including Waymo Open Dataset, nuScenes, SemanticKITTI and ONCE.

Title: Noisy Early Stopping for Noisy Labels

• Authors: William Toner, Amos Storkey
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06830
• Pdf URL: https://arxiv.org/pdf/2409.06830
• Copy Paste: [[2409.06830]] Noisy Early Stopping for Noisy Labels(https://arxiv.org/abs/2409.06830)
• Keywords: robust
• Abstract: Training neural network classifiers on datasets contaminated with noisy labels significantly increases the risk of overfitting. Thus, effectively implementing Early Stopping in noisy label environments is crucial. Under ideal circumstances, Early Stopping utilises a validation set uncorrupted by label noise to effectively monitor generalisation during training. However, obtaining a noise-free validation dataset can be costly and challenging to obtain. This study establishes that, in many typical learning environments, a noise-free validation set is not necessary for effective Early Stopping. Instead, near-optimal results can be achieved by monitoring accuracy on a noisy dataset - drawn from the same distribution as the noisy training set. Referred to as `Noisy Early Stopping' (NES), this method simplifies and reduces the cost of implementing Early Stopping. We provide theoretical insights into the conditions under which this method is effective and empirically demonstrate its robust performance across standard benchmarks using common loss functions.

Title: Few-Shot Learning: Expanding ID Cards Presentation Attack Detection to Unknown ID Countries

• Authors: Alvaro S. Rocamora, Juan M. Espin, Juan E. Tapia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06842
• Pdf URL: https://arxiv.org/pdf/2409.06842
• Copy Paste: [[2409.06842]] Few-Shot Learning: Expanding ID Cards Presentation Attack Detection to Unknown ID Countries(https://arxiv.org/abs/2409.06842)
• Keywords: attack
• Abstract: This paper proposes a Few-shot Learning (FSL) approach for detecting Presentation Attacks on ID Cards deployed in a remote verification system and its extension to new countries. Our research analyses the performance of Prototypical Networks across documents from Spain and Chile as a baseline and measures the extension of generalisation capabilities of new ID Card countries such as Argentina and Costa Rica. Specifically targeting the challenge of screen display presentation attacks. By leveraging convolutional architectures and meta-learning principles embodied in Prototypical Networks, we have crafted a model that demonstrates high efficacy with Few-shot examples. This research reveals that competitive performance can be achieved with as Few-shots as five unique identities and with under 100 images per new country added. This opens a new insight for novel generalised Presentation Attack Detection on ID cards to unknown attacks.

Title: Face Mask Removal with Region-attentive Face Inpainting

• Authors: Minmin Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06845
• Pdf URL: https://arxiv.org/pdf/2409.06845
• Copy Paste: [[2409.06845]] Face Mask Removal with Region-attentive Face Inpainting(https://arxiv.org/abs/2409.06845)
• Keywords: generative
• Abstract: During the COVID-19 pandemic, face masks have become ubiquitous in our lives. Face masks can cause some face recognition models to fail since they cover significant portion of a face. In addition, removing face masks from captured images or videos can be desirable, e.g., for better social interaction and for image/video editing and enhancement purposes. Hence, we propose a generative face inpainting method to effectively recover/reconstruct the masked part of a face. Face inpainting is more challenging compared to traditional inpainting, since it requires high fidelity while maintaining the identity at the same time. Our proposed method includes a Multi-scale Channel-Spatial Attention Module (M-CSAM) to mitigate the spatial information loss and learn the inter- and intra-channel correlation. In addition, we introduce an approach enforcing the supervised signal to focus on masked regions instead of the whole image. We also synthesize our own Masked-Faces dataset from the CelebA dataset by incorporating five different types of face masks, including surgical mask, regular mask and scarves, which also cover the neck area. The experimental results show that our proposed method outperforms different baselines in terms of structural similarity index measure, peak signal-to-noise ratio and l1 loss, while also providing better outputs qualitatively. The code will be made publicly available. Code is available at GitHub.

Title: Shadow Removal Refinement via Material-Consistent Shadow Edges

• Authors: Shilin Hu, Hieu Le, ShahRukh Athar, Sagnik Das, Dimitris Samaras
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06848
• Pdf URL: https://arxiv.org/pdf/2409.06848
• Copy Paste: [[2409.06848]] Shadow Removal Refinement via Material-Consistent Shadow Edges(https://arxiv.org/abs/2409.06848)
• Keywords: segmentation
• Abstract: Shadow boundaries can be confused with material boundaries as both exhibit sharp changes in luminance or contrast within a scene. However, shadows do not modify the intrinsic color or texture of surfaces. Therefore, on both sides of shadow edges traversing regions with the same material, the original color and textures should be the same if the shadow is removed properly. These shadow/shadow-free pairs are very useful but hard-to-collect supervision signals. The crucial contribution of this paper is to learn how to identify those shadow edges that traverse material-consistent regions and how to use them as self-supervision for shadow removal refinement during test time. To achieve this, we fine-tune SAM, an image segmentation foundation model, to produce a shadow-invariant segmentation and then extract material-consistent shadow edges by comparing the SAM segmentation with the shadow mask. Utilizing these shadow edges, we introduce color and texture-consistency losses to enhance the shadow removal process. We demonstrate the effectiveness of our method in improving shadow removal results on more challenging, in-the-wild images, outperforming the state-of-the-art shadow removal methods. Additionally, we propose a new metric and an annotated dataset for evaluating the performance of shadow removal methods without the need for paired shadow/shadow-free data.

Title: LIME-M: Less Is More for Evaluation of MLLMs

• Authors: Kang Zhu, Qianbo Zang, Shian Jia, Siwei Wu, Feiteng Fang, Yizhi Li, Shuyue Guo, Tianyu Zheng, Bo Li, Haoning Wu, Xingwei Qu, Jian Yang, Zachary Liu, Xiang Yue, J.H. Liu, Chenghua Lin, Min Yang, Shiwen Ni, Wenhao Huang, Ge Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06851
• Pdf URL: https://arxiv.org/pdf/2409.06851
• Copy Paste: [[2409.06851]] LIME-M: Less Is More for Evaluation of MLLMs(https://arxiv.org/abs/2409.06851)
• Keywords: large language model
• Abstract: With the remarkable success achieved by Multimodal Large Language Models (MLLMs), numerous benchmarks have been designed to assess MLLMs' ability to guide their development in image perception tasks (e.g., image captioning and visual question answering). However, the existence of numerous benchmarks results in a substantial computational burden when evaluating model performance across all of them. Moreover, these benchmarks contain many overly simple problems or challenging samples, which do not effectively differentiate the capabilities among various MLLMs. To address these challenges, we propose a pipeline to process the existing benchmarks, which consists of two modules: (1) Semi-Automated Screening Process and (2) Eliminating Answer Leakage. The Semi-Automated Screening Process filters out samples that cannot distinguish the model's capabilities by synthesizing various MLLMs and manually evaluating them. The Eliminate Answer Leakage module filters samples whose answers can be inferred without images. Finally, we curate the LIME-M: Less Is More for Evaluation of Multimodal LLMs, a lightweight Multimodal benchmark that can more effectively evaluate the performance of different models. Our experiments demonstrate that: LIME-M can better distinguish the performance of different MLLMs with fewer samples (24% of the original) and reduced time (23% of the original); LIME-M eliminates answer leakage, focusing mainly on the information within images; The current automatic metric (i.e., CIDEr) is insufficient for evaluating MLLMs' capabilities in captioning. Moreover, removing the caption task score when calculating the overall score provides a more accurate reflection of model performance differences. All our codes and data are released at this https URL.

Title: ExIQA: Explainable Image Quality Assessment Using Distortion Attributes

• Authors: Sepehr Kazemi Ranjbar, Emad Fatemizadeh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06853
• Pdf URL: https://arxiv.org/pdf/2409.06853
• Copy Paste: [[2409.06853]] ExIQA: Explainable Image Quality Assessment Using Distortion Attributes(https://arxiv.org/abs/2409.06853)
• Keywords: explainability
• Abstract: Blind Image Quality Assessment (BIQA) aims to develop methods that estimate the quality scores of images in the absence of a reference image. In this paper, we approach BIQA from a distortion identification perspective, where our primary goal is to predict distortion types and strengths using Vision-Language Models (VLMs), such as CLIP, due to their extensive knowledge and generalizability. Based on these predicted distortions, we then estimate the quality score of the image. To achieve this, we propose an explainable approach for distortion identification based on attribute learning. Instead of prompting VLMs with the names of distortions, we prompt them with the attributes or effects of distortions and aggregate this information to infer the distortion strength. Additionally, we consider multiple distortions per image, making our method more scalable. To support this, we generate a dataset consisting of 100,000 images for efficient training. Finally, attribute probabilities are retrieved and fed into a regressor to predict the image quality score. The results show that our approach, besides its explainability and transparency, achieves state-of-the-art (SOTA) performance across multiple datasets in both PLCC and SRCC metrics. Moreover, the zero-shot results demonstrate the generalizability of the proposed approach.

Title: AssistTaxi: A Comprehensive Dataset for Taxiway Analysis and Autonomous Operations

• Authors: Parth Ganeriwala, Siddhartha Bhattacharyya, Sean Gunther, Brian Kish, Mohammed Abdul Hafeez Khan, Ankur Dhadoti, Natasha Neogi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06856
• Pdf URL: https://arxiv.org/pdf/2409.06856
• Copy Paste: [[2409.06856]] AssistTaxi: A Comprehensive Dataset for Taxiway Analysis and Autonomous Operations(https://arxiv.org/abs/2409.06856)
• Keywords: extraction
• Abstract: The availability of high-quality datasets play a crucial role in advancing research and development especially, for safety critical and autonomous systems. In this paper, we present AssistTaxi, a comprehensive novel dataset which is a collection of images for runway and taxiway analysis. The dataset comprises of more than 300,000 frames of diverse and carefully collected data, gathered from Melbourne (MLB) and Grant-Valkaria (X59) general aviation airports. The importance of AssistTaxi lies in its potential to advance autonomous operations, enabling researchers and developers to train and evaluate algorithms for efficient and safe taxiing. Researchers can utilize AssistTaxi to benchmark their algorithms, assess performance, and explore novel approaches for runway and taxiway analysis. Addition-ally, the dataset serves as a valuable resource for validating and enhancing existing algorithms, facilitating innovation in autonomous operations for aviation. We also propose an initial approach to label the dataset using a contour based detection and line extraction technique.

Title: What is the Role of Small Models in the LLM Era: A Survey

• Authors: Lihu Chen, Gaël Varoquaux
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06857
• Pdf URL: https://arxiv.org/pdf/2409.06857
• Copy Paste: [[2409.06857]] What is the Role of Small Models in the LLM Era: A Survey(https://arxiv.org/abs/2409.06857)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have made significant progress in advancing artificial general intelligence (AGI), leading to the development of increasingly large models such as GPT-4 and LLaMA-405B. However, scaling up model sizes results in exponentially higher computational costs and energy consumption, making these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior research. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition. We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources. The code is available at this https URL

Title: A Dataset for Evaluating LLM-based Evaluation Functions for Research Question Extraction Task

• Authors: Yuya Fujisaki, Shiro Takagi, Hideki Asoh, Wataru Kumagai
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06883
• Pdf URL: https://arxiv.org/pdf/2409.06883
• Copy Paste: [[2409.06883]] A Dataset for Evaluating LLM-based Evaluation Functions for Research Question Extraction Task(https://arxiv.org/abs/2409.06883)
• Keywords: extraction
• Abstract: The progress in text summarization techniques has been remarkable. However the task of accurately extracting and summarizing necessary information from highly specialized documents such as research papers has not been sufficiently investigated. We are focusing on the task of extracting research questions (RQ) from research papers and construct a new dataset consisting of machine learning papers, RQ extracted from these papers by GPT-4, and human evaluations of the extracted RQ from multiple perspectives. Using this dataset, we systematically compared recently proposed LLM-based evaluation functions for summarizations, and found that none of the functions showed sufficiently high correlations with human evaluations. We expect our dataset provides a foundation for further research on developing better evaluation functions tailored to the RQ extraction task, and contribute to enhance the performance of the task. The dataset is available at this https URL.

Title: Semi-Supervised Reward Modeling via Iterative Self-Training

• Authors: Yifei He, Haoxiang Wang, Ziyan Jiang, Alexandros Papangelis, Han Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06903
• Pdf URL: https://arxiv.org/pdf/2409.06903
• Copy Paste: [[2409.06903]] Semi-Supervised Reward Modeling via Iterative Self-Training(https://arxiv.org/abs/2409.06903)
• Keywords: large language model
• Abstract: Reward models (RM) capture the values and preferences of humans and play a central role in Reinforcement Learning with Human Feedback (RLHF) to align pretrained large language models (LLMs). Traditionally, training these models relies on extensive human-annotated preference data, which poses significant challenges in terms of scalability and cost. To overcome these limitations, we propose Semi-Supervised Reward Modeling (SSRM), an approach that enhances RM training using unlabeled data. Given an unlabeled dataset, SSRM involves three key iterative steps: pseudo-labeling unlabeled examples, selecting high-confidence examples through a confidence threshold, and supervised finetuning on the refined dataset. Across extensive experiments on various model configurations, we demonstrate that SSRM significantly improves reward models without incurring additional labeling costs. Notably, SSRM can achieve performance comparable to models trained entirely on labeled data of equivalent volumes. Overall, SSRM substantially reduces the dependency on large volumes of human-annotated data, thereby decreasing the overall cost and time involved in training effective reward models.

Title: Applied Federated Model Personalisation in the Industrial Domain: A Comparative Study

• Authors: Ilias Siniosoglou, Vasileios Argyriou, George Fragulis, Panagiotis Fouliras, Georgios Th. Papadopoulos, Anastasios Lytos, Panagiotis Sarigiannidis
• Subjects: cs.LG, cs.AI, cs.PF
• Abstract URL: https://arxiv.org/abs/2409.06904
• Pdf URL: https://arxiv.org/pdf/2409.06904
• Copy Paste: [[2409.06904]] Applied Federated Model Personalisation in the Industrial Domain: A Comparative Study(https://arxiv.org/abs/2409.06904)
• Keywords: federate
• Abstract: The time-consuming nature of training and deploying complicated Machine and Deep Learning (DL) models for a variety of applications continues to pose significant challenges in the field of Machine Learning (ML). These challenges are particularly pronounced in the federated domain, where optimizing models for individual nodes poses significant difficulty. Many methods have been developed to tackle this problem, aiming to reduce training expenses and time while maintaining efficient optimisation. Three suggested strategies to tackle this challenge include Active Learning, Knowledge Distillation, and Local Memorization. These methods enable the adoption of smaller models that require fewer computational resources and allow for model personalization with local insights, thereby improving the effectiveness of current models. The present study delves into the fundamental principles of these three approaches and proposes an advanced Federated Learning System that utilises different Personalisation methods towards improving the accuracy of AI models and enhancing user experience in real-time NG-IoT applications, investigating the efficacy of these techniques in the local and federated domain. The results of the original and optimised models are then compared in both local and federated contexts using a comparison analysis. The post-analysis shows encouraging outcomes when it comes to optimising and personalising the models with the suggested techniques.

Title: Representation Tuning

• Authors: Christopher M. Ackerman
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.06927
• Pdf URL: https://arxiv.org/pdf/2409.06927
• Copy Paste: [[2409.06927]] Representation Tuning(https://arxiv.org/abs/2409.06927)
• Keywords: large language model
• Abstract: Activation engineering is becoming increasingly popular as a means of online control of large language models (LLMs). In this work, I extend the idea of active steering with vectors that represent a behavioral direction of interest to tuning those vectors directly into the model, obviating the need for online control. First, I identify activation vectors related to honesty in an open-source LLM (Llama- 2-13b-chat). Next, I demonstrate that model output can be made more or less honest by adding positive or negative multiples of these vectors to residual stream activations during generation. Then, I show that a similar effect can be achieved by fine-tuning the vectors directly into the model, by use of a dual loss function based on the cosine similarity of residual stream activations to the vectors combined with a standard token-based loss ("representation tuning"). Finally, I compare the generations in response to honesty-probing prompts from the resulting models to those from models fine-tuned with a token-based loss alone, and to those from the untuned model subjected to online steering. Overall, fine-tuning the vectors into the models using the cosine similarity plus token loss showed a stronger effect than online steering, and generalized better than using the standard loss, suggesting the potential utility of this approach as a safety measure. Code and data are available at this https URL tuned models are available at this https URL representation-tuning-66da1e5ab41cd1b824687d9f.

Title: Intrapartum Ultrasound Image Segmentation of Pubic Symphysis and Fetal Head Using Dual Student-Teacher Framework with CNN-ViT Collaborative Learning

• Authors: Jianmei Jiang, Huijin Wang, Jieyun Bai, Shun Long, Shuangping Chen, Victor M. Campello, Karim Lekadir
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06928
• Pdf URL: https://arxiv.org/pdf/2409.06928
• Copy Paste: [[2409.06928]] Intrapartum Ultrasound Image Segmentation of Pubic Symphysis and Fetal Head Using Dual Student-Teacher Framework with CNN-ViT Collaborative Learning(https://arxiv.org/abs/2409.06928)
• Keywords: transformer, segmentation
• Abstract: The segmentation of the pubic symphysis and fetal head (PSFH) constitutes a pivotal step in monitoring labor progression and identifying potential delivery complications. Despite the advances in deep learning, the lack of annotated medical images hinders the training of segmentation. Traditional semi-supervised learning approaches primarily utilize a unified network model based on Convolutional Neural Networks (CNNs) and apply consistency regularization to mitigate the reliance on extensive annotated data. However, these methods often fall short in capturing the discriminative features of unlabeled data and in delineating the long-range dependencies inherent in the ambiguous boundaries of PSFH within ultrasound images. To address these limitations, we introduce a novel framework, the Dual-Student and Teacher Combining CNN and Transformer (DSTCT), which synergistically integrates the capabilities of CNNs and Transformers. Our framework comprises a Vision Transformer (ViT) as the teacher and two student mod ls one ViT and one CNN. This dual-student setup enables mutual supervision through the generation of both hard and soft pseudo-labels, with the consistency in their predictions being refined by minimizing the classifier determinacy discrepancy. The teacher model further reinforces learning within this architecture through the imposition of consistency regularization constraints. To augment the generalization abilities of our approach, we employ a blend of data and model perturbation techniques. Comprehensive evaluations on the benchmark dataset of the PSFH Segmentation Grand Challenge at MICCAI 2023 demonstrate our DSTCT framework outperformed ten contemporary semi-supervised segmentation methods. Code available at this https URL.

Title: Automated Body Composition Analysis Using DAFS Express on 2D MRI Slices at L3 Vertebral Level

• Authors: Varun Akella, Razeyeh Bagherinasab, Jia Ming Li, Long Nguyen, Vincent Tze Yang Chow, Hyunwoo Lee, Karteek Popuri, Mirza Faisal Beg
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06942
• Pdf URL: https://arxiv.org/pdf/2409.06942
• Copy Paste: [[2409.06942]] Automated Body Composition Analysis Using DAFS Express on 2D MRI Slices at L3 Vertebral Level(https://arxiv.org/abs/2409.06942)
• Keywords: segmentation
• Abstract: Body composition analysis is vital in assessing health conditions such as obesity, sarcopenia, and metabolic syndromes. MRI provides detailed images of skeletal muscle (SKM), visceral adipose tissue (VAT), and subcutaneous adipose tissue (SAT), but their manual segmentation is labor-intensive and limits clinical applicability. This study validates an automated tool for MRI-based 2D body composition analysis- (Data Analysis Facilitation Suite (DAFS) Express), comparing its automated measurements with expert manual segmentations using UK Biobank data. A cohort of 399 participants from the UK Biobank dataset was selected, yielding 423 single L3 slices for analysis. DAFS Express performed automated segmentations of SKM, VAT, and SAT, which were then manually corrected by expert raters for validation. Evaluation metrics included Jaccard coefficients, Dice scores, Intraclass Correlation Coefficients (ICCs), and Bland-Altman Plots to assess segmentation agreement and reliability. High agreements were observed between automated and manual segmentations with mean Jaccard scores: SKM 99.03%, VAT 95.25%, and SAT 99.57%; and mean Dice scores: SKM 99.51%, VAT 97.41%, and SAT 99.78%. Cross-sectional area comparisons showed consistent measurements with automated methods closely matching manual measurements for SKM and SAT, and slightly higher values for VAT (SKM: Auto 132.51 cm^2, Manual 132.36 cm^2; VAT: Auto 137.07 cm^2, Manual 134.46 cm^2; SAT: Auto 203.39 cm^2, Manual 202.85 cm^2). ICCs confirmed strong reliability (SKM: 0.998, VAT: 0.994, SAT: 0.994). Bland-Altman plots revealed minimal biases, and boxplots illustrated distribution similarities across SKM, VAT, and SAT areas. On average DAFS Express took 18 seconds per DICOM. This underscores its potential to streamline image analysis processes in research and clinical settings, enhancing diagnostic accuracy and efficiency.

Title: FSMDet: Vision-guided feature diffusion for fully sparse 3D detector

• Authors: Tianran Liu, Morteza Mousa Pasandi, Robert Laganiere
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06945
• Pdf URL: https://arxiv.org/pdf/2409.06945
• Copy Paste: [[2409.06945]] FSMDet: Vision-guided feature diffusion for fully sparse 3D detector(https://arxiv.org/abs/2409.06945)
• Keywords: diffusion
• Abstract: Fully sparse 3D detection has attracted an increasing interest in the recent years. However, the sparsity of the features in these frameworks challenges the generation of proposals because of the limited diffusion process. In addition, the quest for efficiency has led to only few work on vision-assisted fully sparse models. In this paper, we propose FSMDet (Fully Sparse Multi-modal Detection), which use visual information to guide the LiDAR feature diffusion process while still maintaining the efficiency of the pipeline. Specifically, most of fully sparse works focus on complex customized center fusion diffusion/regression operators. However, we observed that if the adequate object completion is performed, even the simplest interpolation operator leads to satisfactory results. Inspired by this observation, we split the vision-guided diffusion process into two modules: a Shape Recover Layer (SRLayer) and a Self Diffusion Layer (SDLayer). The former uses RGB information to recover the shape of the visible part of an object, and the latter uses a visual prior to further spread the features to the center region. Experiments demonstrate that our approach successfully improves the performance of previous fully sparse models that use LiDAR only and reaches SOTA performance in multimodal models. At the same time, thanks to the sparse architecture, our method can be up to 5 times more efficient than previous SOTA methods in the inference process.

Title: You Have Thirteen Hours in Which to Solve the Labyrinth: Enhancing AI Game Masters with Function Calling

• Authors: Jaewoo Song, Andrew Zhu, Chris Callison-Burch
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06949
• Pdf URL: https://arxiv.org/pdf/2409.06949
• Copy Paste: [[2409.06949]] You Have Thirteen Hours in Which to Solve the Labyrinth: Enhancing AI Game Masters with Function Calling(https://arxiv.org/abs/2409.06949)
• Keywords: large language model
• Abstract: Developing a consistent and reliable AI game master for text-based games is a challenging task due to the limitations of large language models (LLMs) and the complexity of the game master's role. This paper presents a novel approach to enhance AI game masters by leveraging function calling in the context of the table-top role-playing game "Jim Henson's Labyrinth: The Adventure Game." Our methodology involves integrating game-specific controls through functions, which we show improves the narrative quality and state update consistency of the AI game master. The experimental results, based on human evaluations and unit tests, demonstrate the effectiveness of our approach in enhancing gameplay experience and maintaining coherence with the game state. This work contributes to the advancement of game AI and interactive storytelling, offering insights into the design of more engaging and consistent AI-driven game masters.

Title: Privacy-Preserving Federated Learning with Consistency via Knowledge Distillation Using Conditional Generator

• Authors: Kangyang Luo, Shuai Wang, Xiang Li, Yunshi Lan, Ming Gao, Jinlong Shu
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.06955
• Pdf URL: https://arxiv.org/pdf/2409.06955
• Copy Paste: [[2409.06955]] Privacy-Preserving Federated Learning with Consistency via Knowledge Distillation Using Conditional Generator(https://arxiv.org/abs/2409.06955)
• Keywords: privacy, attack, robust, federate
• Abstract: Federated Learning (FL) is gaining popularity as a distributed learning framework that only shares model parameters or gradient updates and keeps private data locally. However, FL is at risk of privacy leakage caused by privacy inference attacks. And most existing privacy-preserving mechanisms in FL conflict with achieving high performance and efficiency. Therefore, we propose FedMD-CG, a novel FL method with highly competitive performance and high-level privacy preservation, which decouples each client's local model into a feature extractor and a classifier, and utilizes a conditional generator instead of the feature extractor to perform server-side model aggregation. To ensure the consistency of local generators and classifiers, FedMD-CG leverages knowledge distillation to train local models and generators at both the latent feature level and the logit level. Also, we construct additional classification losses and design new diversity losses to enhance client-side training. FedMD-CG is robust to data heterogeneity and does not require training extra discriminators (like cGAN). We conduct extensive experiments on various image classification tasks to validate the superiority of FedMD-CG.

Title: Policy Filtration in RLHF to Fine-Tune LLM for Code Generation

• Authors: Wei Shen, Chuheng Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.06957
• Pdf URL: https://arxiv.org/pdf/2409.06957
• Copy Paste: [[2409.06957]] Policy Filtration in RLHF to Fine-Tune LLM for Code Generation(https://arxiv.org/abs/2409.06957)
• Keywords: large language model
• Abstract: Reinforcement learning from human feedback (RLHF) is one of the key techniques that helps large language models (LLMs) to follow instructions and provide helpful and harmless responses. While direct policy optimization methods exist, state-of-the-art LLMs adopt RL-based methods (usually PPO) in RLHF to train the policy to generate good responses guided by a reward model learned from preference data. The main challenge of these methods is the inaccuracy of the intermediate reward model, especially in code generation tasks that require long and complex reasoning to score a response. We find that the reliability of the reward model varies across responses assigned with different rewards. This motivates us to filter the samples whose rewards may be unreliable to improve signal-to-noise ratio during policy learning, resulting in Policy Filtration for Proximal Policy Optimization (PF-PPO). To choose a proper policy filtration strategy for a given reward model, the coefficient of determination ($R^2$) between rewards and actual scores on filtered samples serves as a good metrics and helps us find several promising strategies. We provide extensive experiments to validate the effectiveness of PF-PPO in code generation tasks, and find that some variants of PF-PPO are highly effective and achieve new state-of-the-art performance across 7-billion-parameter models on HumanEval, MBPP, and a new and more challenging LeetCode Contest benchmark.

Title: Brain-Inspired Stepwise Patch Merging for Vision Transformers

• Authors: Yonghao Yu, Dongcheng Zhao, Guobin Shen, Yiting Dong, Yi Zeng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06963
• Pdf URL: https://arxiv.org/pdf/2409.06963
• Copy Paste: [[2409.06963]] Brain-Inspired Stepwise Patch Merging for Vision Transformers(https://arxiv.org/abs/2409.06963)
• Keywords: robust, extraction, transformer, segmentation
• Abstract: The hierarchical architecture has become a mainstream design paradigm for Vision Transformers (ViTs), with Patch Merging serving as the pivotal component that transforms a columnar architecture into a hierarchical one. Drawing inspiration from the brain's ability to integrate global and local information for comprehensive visual understanding, we propose a novel technique called Stepwise Patch Merging (SPM), which enhances the subsequent attention mechanism's ability to 'see' better. SPM comprises two critical modules: Multi-Scale Aggregation (MSA) and Guided Local Enhancement (GLE). The MSA module integrates multi-scale features to enrich feature representation, while the GLE module focuses on refining local detail extraction, thus achieving an optimal balance between long-range dependency modeling and local feature enhancement. Extensive experiments conducted on benchmark datasets, including ImageNet-1K, COCO, and ADE20K, demonstrate that SPM significantly improves the performance of various models, particularly in dense prediction tasks such as object detection and semantic segmentation. These results underscore the efficacy of SPM in enhancing model accuracy and robustness across a wide range of computer vision tasks.

Title: PanAdapter: Two-Stage Fine-Tuning with Spatial-Spectral Priors Injecting for Pansharpening

• Authors: RuoCheng Wu, ZiEn Zhang, ShangQi Deng, YuLe Duan, LiangJian Deng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06980
• Pdf URL: https://arxiv.org/pdf/2409.06980
• Copy Paste: [[2409.06980]] PanAdapter: Two-Stage Fine-Tuning with Spatial-Spectral Priors Injecting for Pansharpening(https://arxiv.org/abs/2409.06980)
• Keywords: extraction, transformer
• Abstract: Pansharpening is a challenging image fusion task that involves restoring images using two different modalities: low-resolution multispectral images (LRMS) and high-resolution panchromatic (PAN). Many end-to-end specialized models based on deep learning (DL) have been proposed, yet the scale and performance of these models are limited by the size of dataset. Given the superior parameter scales and feature representations of pre-trained models, they exhibit outstanding performance when transferred to downstream tasks with small datasets. Therefore, we propose an efficient fine-tuning method, namely PanAdapter, which utilizes additional advanced semantic information from pre-trained models to alleviate the issue of small-scale datasets in pansharpening tasks. Specifically, targeting the large domain discrepancy between image restoration and pansharpening tasks, the PanAdapter adopts a two-stage training strategy for progressively adapting to the downstream task. In the first stage, we fine-tune the pre-trained CNN model and extract task-specific priors at two scales by proposed Local Prior Extraction (LPE) module. In the second stage, we feed the extracted two-scale priors into two branches of cascaded adapters respectively. At each adapter, we design two parameter-efficient modules for allowing the two branches to interact and be injected into the frozen pre-trained VisionTransformer (ViT) blocks. We demonstrate that by only training the proposed LPE modules and adapters with a small number of parameters, our approach can benefit from pre-trained image restoration models and achieve state-of-the-art performance in several benchmark pansharpening datasets. The code will be available soon.

Title: Enhancing Cross-domain Pre-Trained Decision Transformers with Adaptive Attention

• Authors: Wenhao Zhao, Qiushui Xu, Linjie Xu, Lei Song, Jinyu Wang, Chunlai Zhou, Jiang Bian
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.06985
• Pdf URL: https://arxiv.org/pdf/2409.06985
• Copy Paste: [[2409.06985]] Enhancing Cross-domain Pre-Trained Decision Transformers with Adaptive Attention(https://arxiv.org/abs/2409.06985)
• Keywords: extraction, transformer
• Abstract: Recently, the pre-training of decision transformers (DT) using a different domain, such as natural language text, has generated significant attention in offline reinforcement learning (Offline RL). Although this cross-domain pre-training approach achieves superior performance compared to training from scratch in environments required short-term planning ability, the mechanisms by which pre-training benefits the fine-tuning phase remain unclear. Furthermore, we point out that the cross-domain pre-training approach hinders the extraction of distant information in environments like PointMaze that require long-term planning ability, leading to performance that is much worse than training DT from scratch. This work first analyzes these issues and found that Markov Matrix, a component that exists in pre-trained attention heads, is the key to explain the significant performance disparity of pre-trained models in different planning abilities. Inspired by our analysis, we propose a general method GPT-DTMA, which equips a pre-trained DT with Mixture of Attention (MoA), to enable adaptive learning and accommodating diverse attention requirements during fine-tuning. Extensive experiments demonstrate that the effectiveness of GPT-DTMA: it achieves superior performance in short-term environments compared to baselines, and in long-term environments, it mitigates the negative impact caused by Markov Matrix, achieving results comparable to those of DT trained from scratch.

Title: 1M-Deepfakes Detection Challenge

• Authors: Zhixi Cai, Abhinav Dhall, Shreya Ghosh, Munawar Hayat, Dimitrios Kollias, Kalin Stefanov, Usman Tariq
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.06991
• Pdf URL: https://arxiv.org/pdf/2409.06991
• Copy Paste: [[2409.06991]] 1M-Deepfakes Detection Challenge(https://arxiv.org/abs/2409.06991)
• Keywords: security
• Abstract: The detection and localization of deepfake content, particularly when small fake segments are seamlessly mixed with real videos, remains a significant challenge in the field of digital media security. Based on the recently released AV-Deepfake1M dataset, which contains more than 1 million manipulated videos across more than 2,000 subjects, we introduce the 1M-Deepfakes Detection Challenge. This challenge is designed to engage the research community in developing advanced methods for detecting and localizing deepfake manipulations within the large-scale high-realistic audio-visual dataset. The participants can access the AV-Deepfake1M dataset and are required to submit their inference results for evaluation across the metrics for detection or localization tasks. The methodologies developed through the challenge will contribute to the development of next-generation deepfake detection and localization systems. Evaluation scripts, baseline models, and accompanying code will be available on this https URL.

Title: AdvLogo: Adversarial Patch Attack against Object Detectors based on Diffusion Models

• Authors: Boming Miao, Chunxiao Li, Yao Zhu, Weixiang Sun, Zizhe Wang, Xiaoyi Wang, Chuanlong Xie
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07002
• Pdf URL: https://arxiv.org/pdf/2409.07002
• Copy Paste: [[2409.07002]] AdvLogo: Adversarial Patch Attack against Object Detectors based on Diffusion Models(https://arxiv.org/abs/2409.07002)
• Keywords: attack, diffusion
• Abstract: With the rapid development of deep learning, object detectors have demonstrated impressive performance; however, vulnerabilities still exist in certain scenarios. Current research exploring the vulnerabilities using adversarial patches often struggles to balance the trade-off between attack effectiveness and visual quality. To address this problem, we propose a novel framework of patch attack from semantic perspective, which we refer to as AdvLogo. Based on the hypothesis that every semantic space contains an adversarial subspace where images can cause detectors to fail in recognizing objects, we leverage the semantic understanding of the diffusion denoising process and drive the process to adversarial subareas by perturbing the latent and unconditional embeddings at the last timestep. To mitigate the distribution shift that exposes a negative impact on image quality, we apply perturbation to the latent in frequency domain with the Fourier Transform. Experimental results demonstrate that AdvLogo achieves strong attack performance while maintaining high visual quality.

Title: ODYSSEE: Oyster Detection Yielded by Sensor Systems on Edge Electronics

• Authors: Xiaomin Lin, Vivek Mange, Arjun Suresh, Bernhard Neuberger, Aadi Palnitkar, Brendan Campbell, Alan Williams, Kleio Baxevani, Jeremy Mallette, Alhim Vera, Markus Vincze, Ioannis Rekleitis, Herbert G. Tanner, Yiannis Aloimonos
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.07003
• Pdf URL: https://arxiv.org/pdf/2409.07003
• Copy Paste: [[2409.07003]] ODYSSEE: Oyster Detection Yielded by Sensor Systems on Edge Electronics(https://arxiv.org/abs/2409.07003)
• Keywords: diffusion, segmentation
• Abstract: Oysters are a keystone species in coastal ecosystems, offering significant economic, environmental, and cultural benefits. However, current monitoring systems are often destructive, typically involving dredging to physically collect and count oysters. A nondestructive alternative is manual identification from video footage collected by divers, which is time-consuming and labor-intensive with expert input. An alternative to human monitoring is the deployment of a system with trained object detection models that performs real-time, on edge oyster detection in the field. One such platform is the Aqua2 robot. Effective training of these models requires extensive high-quality data, which is difficult to obtain in marine settings. To address these complications, we introduce a novel method that leverages stable diffusion to generate high-quality synthetic data for the marine domain. We exploit diffusion models to create photorealistic marine imagery, using ControlNet inputs to ensure consistency with the segmentation ground-truth mask, the geometry of the scene, and the target domain of real underwater images for oysters. The resulting dataset is used to train a YOLOv10-based vision model, achieving a state-of-the-art 0.657 mAP@50 for oyster detection on the Aqua2 platform. The system we introduce not only improves oyster habitat monitoring, but also paves the way to autonomous surveillance for various tasks in marine contexts, improving aquaculture and conservation efforts.

Title: Insight Any Instance: Promptable Instance Segmentation for Remote Sensing Images

• Authors: Xuexue Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07022
• Pdf URL: https://arxiv.org/pdf/2409.07022
• Copy Paste: [[2409.07022]] Insight Any Instance: Promptable Instance Segmentation for Remote Sensing Images(https://arxiv.org/abs/2409.07022)
• Keywords: segmentation
• Abstract: Instance segmentation of remote sensing images (RSIs) is an essential task for a wide range of applications such as land planning and intelligent transport. Instance segmentation of RSIs is constantly plagued by the unbalanced ratio of foreground and background and limited instance size. And most of the instance segmentation models are based on deep feature learning and contain operations such as multiple downsampling, which is harmful to instance segmentation of RSIs, and thus the performance is still limited. Inspired by the recent superior performance of prompt learning in visual tasks, we propose a new prompt paradigm to address the above issues. Based on the existing instance segmentation model, firstly, a local prompt module is designed to mine local prompt information from original local tokens for specific instances; secondly, a global-to-local prompt module is designed to model the contextual information from the global tokens to the local tokens where the instances are located for specific instances. Finally, a proposal's area loss function is designed to add a decoupling dimension for proposals on the scale to better exploit the potential of the above two prompt modules. It is worth mentioning that our proposed approach can extend the instance segmentation model to a promptable instance segmentation model, i.e., to segment the instances with the specific boxes prompt. The time consumption for each promptable instance segmentation process is only 40 ms. The paper evaluates the effectiveness of our proposed approach based on several existing models in four instance segmentation datasets of RSIs, and thorough experiments prove that our proposed approach is effective for addressing the above issues and is a competitive model for instance segmentation of RSIs.

Title: SCLNet: A Scale-Robust Complementary Learning Network for Object Detection in UAV Images

• Authors: Xuexue Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07024
• Pdf URL: https://arxiv.org/pdf/2409.07024
• Copy Paste: [[2409.07024]] SCLNet: A Scale-Robust Complementary Learning Network for Object Detection in UAV Images(https://arxiv.org/abs/2409.07024)
• Keywords: robust
• Abstract: Most recent UAV (Unmanned Aerial Vehicle) detectors focus primarily on general challenge such as uneven distribution and occlusion. However, the neglect of scale challenges, which encompass scale variation and small objects, continues to hinder object detection in UAV images. Although existing works propose solutions, they are implicitly modeled and have redundant steps, so detection performance remains limited. And one specific work addressing the above scale challenges can help improve the performance of UAV image detectors. Compared to natural scenes, scale challenges in UAV images happen with problems of limited perception in comprehensive scales and poor robustness to small objects. We found that complementary learning is beneficial for the detection model to address the scale challenges. Therefore, the paper introduces it to form our scale-robust complementary learning network (SCLNet) in conjunction with the object detection model. The SCLNet consists of two implementations and a cooperation method. In detail, one implementation is based on our proposed scale-complementary decoder and scale-complementary loss function to explicitly extract complementary information as complement, named comprehensive-scale complementary learning (CSCL). Another implementation is based on our proposed contrastive complement network and contrastive complement loss function to explicitly guide the learning of small objects with the rich texture detail information of the large objects, named inter-scale contrastive complementary learning (ICCL). In addition, an end-to-end cooperation (ECoop) between two implementations and with the detection model is proposed to exploit each potential.

Title: CPSample: Classifier Protected Sampling for Guarding Training Data During Diffusion

• Authors: Joshua Kazdan, Hao Sun, Jiaqi Han, Felix Petersen, Stefano Ermon
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07025
• Pdf URL: https://arxiv.org/pdf/2409.07025
• Copy Paste: [[2409.07025]] CPSample: Classifier Protected Sampling for Guarding Training Data During Diffusion(https://arxiv.org/abs/2409.07025)
• Keywords: privacy, protect, attack, robust, membership infer, diffusion
• Abstract: Diffusion models have a tendency to exactly replicate their training data, especially when trained on small datasets. Most prior work has sought to mitigate this problem by imposing differential privacy constraints or masking parts of the training data, resulting in a notable substantial decrease in image quality. We present CPSample, a method that modifies the sampling process to prevent training data replication while preserving image quality. CPSample utilizes a classifier that is trained to overfit on random binary labels attached to the training data. CPSample then uses classifier guidance to steer the generation process away from the set of points that can be classified with high certainty, a set that includes the training data. CPSample achieves FID scores of 4.97 and 2.97 on CIFAR-10 and CelebA-64, respectively, without producing exact replicates of the training data. Unlike prior methods intended to guard the training images, CPSample only requires training a classifier rather than retraining a diffusion model, which is computationally cheaper. Moreover, our technique provides diffusion models with greater robustness against membership inference attacks, wherein an adversary attempts to discern which images were in the model's training dataset. We show that CPSample behaves like a built-in rejection sampler, and we demonstrate its capabilities to prevent mode collapse in Stable Diffusion.

Title: Dynamic Error-Bounded Hierarchical Matrices in Neural Network Compression

• Authors: John Mango, Ronald Katende
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2409.07028
• Pdf URL: https://arxiv.org/pdf/2409.07028
• Copy Paste: [[2409.07028]] Dynamic Error-Bounded Hierarchical Matrices in Neural Network Compression(https://arxiv.org/abs/2409.07028)
• Keywords: robust
• Abstract: This paper presents an innovative framework that integrates hierarchical matrix (H-matrix) compression techniques into the structure and training of Physics-Informed Neural Networks (PINNs). By leveraging the low-rank properties of matrix sub-blocks, the proposed dynamic, error-bounded H-matrix compression method significantly reduces computational complexity and storage requirements without compromising accuracy. This approach is rigorously compared to traditional compression techniques, such as Singular Value Decomposition (SVD), pruning, and quantization, demonstrating superior performance, particularly in maintaining the Neural Tangent Kernel (NTK) properties critical for the stability and convergence of neural networks. The findings reveal that H-matrix compression not only enhances training efficiency but also ensures the scalability and robustness of PINNs for complex, large-scale applications in physics-based modeling. This work offers a substantial contribution to the optimization of deep learning models, paving the way for more efficient and practical implementations of PINNs in real-world scenarios.

Title: Beyond IID: Optimizing Instruction Learning from the Perspective of Instruction Interaction and Dependency

• Authors: Hanyu Zhao, Li Du, Yiming Ju, Chengwei Wu, Tengfei Pan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07045
• Pdf URL: https://arxiv.org/pdf/2409.07045
• Copy Paste: [[2409.07045]] Beyond IID: Optimizing Instruction Learning from the Perspective of Instruction Interaction and Dependency(https://arxiv.org/abs/2409.07045)
• Keywords: large language model
• Abstract: With the availability of various instruction datasets, a pivotal challenge is how to effectively select and integrate these instructions to fine-tune large language models (LLMs). Previous research mainly focuses on selecting individual high-quality instructions. However, these works overlooked the joint interactions and dependencies between different categories of instructions, leading to suboptimal selection strategies. Moreover, the nature of these interaction patterns remains largely unexplored, let alone optimize the instruction set with regard to them. To fill these gaps, in this paper, we: (1) systemically investigate interaction and dependency patterns between different categories of instructions, (2) manage to optimize the instruction set concerning the interaction patterns using a linear programming-based method, and optimize the learning schema of SFT using an instruction dependency taxonomy guided curriculum learning. Experimental results across different LLMs demonstrate improved performance over strong baselines on widely adopted benchmarks.

Title: Pushing the Limits of Vision-Language Models in Remote Sensing without Human Annotations

• Authors: Keumgang Cha, Donggeun Yu, Junghoon Seo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07048
• Pdf URL: https://arxiv.org/pdf/2409.07048
• Copy Paste: [[2409.07048]] Pushing the Limits of Vision-Language Models in Remote Sensing without Human Annotations(https://arxiv.org/abs/2409.07048)
• Keywords: robust
• Abstract: The prominence of generalized foundation models in vision-language integration has witnessed a surge, given their multifarious applications. Within the natural domain, the procurement of vision-language datasets to construct these foundation models is facilitated by their abundant availability and the ease of web crawling. Conversely, in the remote sensing domain, although vision-language datasets exist, their volume is suboptimal for constructing robust foundation models. This study introduces an approach to curate vision-language datasets by employing an image decoding machine learning model, negating the need for human-annotated labels. Utilizing this methodology, we amassed approximately 9.6 million vision-language paired datasets in VHR imagery. The resultant model outperformed counterparts that did not leverage publicly available vision-language datasets, particularly in downstream tasks such as zero-shot classification, semantic localization, and image-text retrieval. Moreover, in tasks exclusively employing vision encoders, such as linear probing and k-NN classification, our model demonstrated superior efficacy compared to those relying on domain-specific vision-language datasets.

Title: Native vs Non-Native Language Prompting: A Comparative Analysis

• Authors: Mohamed Bayan Kmainasi, Rakif Khan, Ali Ezzat Shahroor, Boushra Bendou, Maram Hasanain, Firoj Alam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07054
• Pdf URL: https://arxiv.org/pdf/2409.07054
• Copy Paste: [[2409.07054]] Native vs Non-Native Language Prompting: A Comparative Analysis(https://arxiv.org/abs/2409.07054)
• Keywords: large language model
• Abstract: Large language models (LLMs) have shown remarkable abilities in different fields, including standard Natural Language Processing (NLP) tasks. To elicit knowledge from LLMs, prompts play a key role, consisting of natural language instructions. Most open and closed source LLMs are trained on available labeled and unlabeled resources--digital content such as text, images, audio, and videos. Hence, these models have better knowledge for high-resourced languages but struggle with low-resourced languages. Since prompts play a crucial role in understanding their capabilities, the language used for prompts remains an important research question. Although there has been significant research in this area, it is still limited, and less has been explored for medium to low-resourced languages. In this study, we investigate different prompting strategies (native vs. non-native) on 11 different NLP tasks associated with 12 different Arabic datasets (9.7K data points). In total, we conducted 197 experiments involving 3 LLMs, 12 datasets, and 3 prompting strategies. Our findings suggest that, on average, the non-native prompt performs the best, followed by mixed and native prompts.

Title: Multimodal Emotion Recognition with Vision-language Prompting and Modality Dropout

• Authors: Anbin QI, Zhongliang Liu, Xinyong Zhou, Jinba Xiao, Fengrun Zhang, Qi Gan, Ming Tao, Gaozheng Zhang, Lu Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07078
• Pdf URL: https://arxiv.org/pdf/2409.07078
• Copy Paste: [[2409.07078]] Multimodal Emotion Recognition with Vision-language Prompting and Modality Dropout(https://arxiv.org/abs/2409.07078)
• Keywords: robust
• Abstract: In this paper, we present our solution for the Second Multimodal Emotion Recognition Challenge Track 1(MER2024-SEMI). To enhance the accuracy and generalization performance of emotion recognition, we propose several methods for Multimodal Emotion Recognition. Firstly, we introduce EmoVCLIP, a model fine-tuned based on CLIP using vision-language prompt learning, designed for video-based emotion recognition tasks. By leveraging prompt learning on CLIP, EmoVCLIP improves the performance of pre-trained CLIP on emotional videos. Additionally, to address the issue of modality dependence in multimodal fusion, we employ modality dropout for robust information fusion. Furthermore, to aid Baichuan in better extracting emotional information, we suggest using GPT-4 as the prompt for Baichuan. Lastly, we utilize a self-training strategy to leverage unlabeled videos. In this process, we use unlabeled videos with high-confidence pseudo-labels generated by our model and incorporate them into the training set. Experimental results demonstrate that our model ranks 1st in the MER2024-SEMI track, achieving an accuracy of 90.15% on the test set.

Title: Understanding Knowledge Drift in LLMs through Misinformation

• Authors: Alina Fastowski, Gjergji Kasneci
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07085
• Pdf URL: https://arxiv.org/pdf/2409.07085
• Copy Paste: [[2409.07085]] Understanding Knowledge Drift in LLMs through Misinformation(https://arxiv.org/abs/2409.07085)
• Keywords: robust, large language model
• Abstract: Large Language Models (LLMs) have revolutionized numerous applications, making them an integral part of our digital ecosystem. However, their reliability becomes critical, especially when these models are exposed to misinformation. We primarily analyze the susceptibility of state-of-the-art LLMs to factual inaccuracies when they encounter false information in a QnA scenario, an issue that can lead to a phenomenon we refer to as *knowledge drift*, which significantly undermines the trustworthiness of these models. We evaluate the factuality and the uncertainty of the models' responses relying on Entropy, Perplexity, and Token Probability metrics. Our experiments reveal that an LLM's uncertainty can increase up to 56.6% when the question is answered incorrectly due to the exposure to false information. At the same time, repeated exposure to the same false information can decrease the models uncertainty again (-52.8% w.r.t. the answers on the untainted prompts), potentially manipulating the underlying model's beliefs and introducing a drift from its original knowledge. These findings provide insights into LLMs' robustness and vulnerability to adversarial inputs, paving the way for developing more reliable LLM applications across various domains. The code is available at this https URL.

Title: Ontology-Free General-Domain Knowledge Graph-to-Text Generation Dataset Synthesis using Large Language Model

• Authors: Daehee Kim, Deokhyung Kang, Sangwon Ryu, Gary Geunbae Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07088
• Pdf URL: https://arxiv.org/pdf/2409.07088
• Copy Paste: [[2409.07088]] Ontology-Free General-Domain Knowledge Graph-to-Text Generation Dataset Synthesis using Large Language Model(https://arxiv.org/abs/2409.07088)
• Keywords: large language model
• Abstract: Knowledge Graph-to-Text (G2T) generation involves verbalizing structured knowledge graphs into natural language text. Recent advancements in Pretrained Language Models (PLMs) have improved G2T performance, but their effectiveness depends on datasets with precise graph-text alignment. However, the scarcity of high-quality, general-domain G2T generation datasets restricts progress in the general-domain G2T generation research. To address this issue, we introduce Wikipedia Ontology-Free Graph-text dataset (WikiOFGraph), a new large-scale G2T dataset generated using a novel method that leverages Large Language Model (LLM) and Data-QuestEval. Our new dataset, which contains 5.85M general-domain graph-text pairs, offers high graph-text consistency without relying on external ontologies. Experimental results demonstrate that PLM fine-tuned on WikiOFGraph outperforms those trained on other datasets across various evaluation metrics. Our method proves to be a scalable and effective solution for generating high-quality G2T data, significantly advancing the field of G2T generation.

Title: TrialSynth: Generation of Synthetic Sequential Clinical Trial Data

• Authors: Chufan Gao, Mandis Beigi, Afrah Shafquat, Jacob Aptekar, Jimeng Sun
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07089
• Pdf URL: https://arxiv.org/pdf/2409.07089
• Copy Paste: [[2409.07089]] TrialSynth: Generation of Synthetic Sequential Clinical Trial Data(https://arxiv.org/abs/2409.07089)
• Keywords: privacy
• Abstract: Analyzing data from past clinical trials is part of the ongoing effort to optimize the design, implementation, and execution of new clinical trials and more efficiently bring life-saving interventions to market. While there have been recent advances in the generation of static context synthetic clinical trial data, due to both limited patient availability and constraints imposed by patient privacy needs, the generation of fine-grained synthetic time-sequential clinical trial data has been challenging. Given that patient trajectories over an entire clinical trial are of high importance for optimizing trial design and efforts to prevent harmful adverse events, there is a significant need for the generation of high-fidelity time-sequence clinical trial data. Here we introduce TrialSynth, a Variational Autoencoder (VAE) designed to address the specific challenges of generating synthetic time-sequence clinical trial data. Distinct from related clinical data VAE methods, the core of our method leverages Hawkes Processes (HP), which are particularly well-suited for modeling event-type and time gap prediction needed to capture the structure of sequential clinical trial data. Our experiments demonstrate that TrialSynth surpasses the performance of other comparable methods that can generate sequential clinical trial data, in terms of both fidelity and in enabling the generation of highly accurate event sequences across multiple real-world sequential event datasets with small patient source populations when using minimal external information. Notably, our empirical findings highlight that TrialSynth not only outperforms existing clinical sequence-generating methods but also produces data with superior utility while empirically preserving patient privacy.

Title: Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandem

• Authors: Qianli Wang, Tatiana Anikina, Nils Feldhus, Simon Ostermann, Sebastian Möller, Vera Schmitt
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07123
• Pdf URL: https://arxiv.org/pdf/2409.07123
• Copy Paste: [[2409.07123]] Cross-Refine: Improving Natural Language Explanation Generation by Learning in Tandem(https://arxiv.org/abs/2409.07123)
• Keywords: large language model
• Abstract: Natural language explanations (NLEs) are vital for elucidating the reasoning behind large language model (LLM) decisions. Many techniques have been developed to generate NLEs using LLMs. However, like humans, LLMs might not always produce optimal NLEs on first attempt. Inspired by human learning processes, we introduce Cross-Refine, which employs role modeling by deploying two LLMs as generator and critic, respectively. The generator outputs a first NLE and then refines this initial explanation using feedback and suggestions provided by the critic. Cross-Refine does not require any supervised training data or additional training. We validate Cross-Refine across three NLP tasks using three state-of-the-art open-source LLMs through automatic and human evaluation. We select Self-Refine (Madaan et al., 2023) as the baseline, which only utilizes self-feedback to refine the explanations. Our findings from automatic evaluation and a user study indicate that Cross-Refine outperforms Self-Refine. Meanwhile, Cross-Refine can perform effectively with less powerful LLMs, whereas Self-Refine only yields strong results with ChatGPT. Additionally, we conduct an ablation study to assess the importance of feedback and suggestions. Both of them play an important role in refining explanations. We further evaluate Cross-Refine on a bilingual dataset in English and German.

Title: MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View Synthesis

• Authors: Hanyu Jiang, Jian Xue, Xing Lan, Guohong Hu, Ke Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07129
• Pdf URL: https://arxiv.org/pdf/2409.07129
• Copy Paste: [[2409.07129]] MVLLaVA: An Intelligent Agent for Unified and Flexible Novel View Synthesis(https://arxiv.org/abs/2409.07129)
• Keywords: robust, diffusion
• Abstract: This paper introduces MVLLaVA, an intelligent agent designed for novel view synthesis tasks. MVLLaVA integrates multiple multi-view diffusion models with a large multimodal model, LLaVA, enabling it to handle a wide range of tasks efficiently. MVLLaVA represents a versatile and unified platform that adapts to diverse input types, including a single image, a descriptive caption, or a specific change in viewing azimuth, guided by language instructions for viewpoint generation. We carefully craft task-specific instruction templates, which are subsequently used to fine-tune LLaVA. As a result, MVLLaVA acquires the capability to generate novel view images based on user instructions, demonstrating its flexibility across diverse tasks. Experiments are conducted to validate the effectiveness of MVLLaVA, demonstrating its robust performance and versatility in tackling diverse novel view synthesis challenges.

Title: Reranking Laws for Language Generation: A Communication-Theoretic Perspective

• Authors: António Farinhas, Haau-Sing Li, André F. T. Martins
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.07131
• Pdf URL: https://arxiv.org/pdf/2409.07131
• Copy Paste: [[2409.07131]] Reranking Laws for Language Generation: A Communication-Theoretic Perspective(https://arxiv.org/abs/2409.07131)
• Keywords: large language model
• Abstract: To ensure large language models (LLMs) are used safely, one must reduce their propensity to hallucinate or to generate unacceptable answers. A simple and often used strategy is to first let the LLM generate multiple hypotheses and then employ a reranker to choose the best one. In this paper, we draw a parallel between this strategy and the use of redundancy to decrease the error rate in noisy communication channels. We conceptualize the generator as a sender transmitting multiple descriptions of a message through parallel noisy channels. The receiver decodes the message by ranking the (potentially corrupted) descriptions and selecting the one found to be most reliable. We provide conditions under which this protocol is asymptotically error-free (i.e., yields an acceptable answer almost surely) even in scenarios where the reranker is imperfect (governed by Mallows or Zipf-Mandelbrot models) and the channel distributions are statistically dependent. We use our framework to obtain reranking laws which we validate empirically on two real-world tasks using LLMs: text-to-code generation with DeepSeek-Coder 7B and machine translation of medical data with TowerInstruct 13B.

Title: LLM-based feature generation from text for interpretable machine learning

• Authors: Vojtěch Balek, Lukáš Sýkora, Vilém Sklenák, Tomáš Kliegr
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07132
• Pdf URL: https://arxiv.org/pdf/2409.07132
• Copy Paste: [[2409.07132]] LLM-based feature generation from text for interpretable machine learning(https://arxiv.org/abs/2409.07132)
• Keywords: interpretability, large language model
• Abstract: Existing text representations such as embeddings and bag-of-words are not suitable for rule learning due to their high dimensionality and absent or questionable feature-level interpretability. This article explores whether large language models (LLMs) could address this by extracting a small number of interpretable features from text. We demonstrate this process on two datasets (CORD-19 and M17+) containing several thousand scientific articles from multiple disciplines and a target being a proxy for research impact. An evaluation based on testing for the statistically significant correlation with research impact has shown that LLama 2-generated features are semantically meaningful. We consequently used these generated features in text classification to predict the binary target variable representing the citation rate for the CORD-19 dataset and the ordinal 5-class target representing an expert-awarded grade in the M17+ dataset. Machine-learning models trained on the LLM-generated features provided similar predictive performance to the state-of-the-art embedding model SciBERT for scientific text. The LLM used only 62 features compared to 768 features in SciBERT embeddings, and these features were directly interpretable, corresponding to notions such as article methodological rigor, novelty, or grammatical correctness. As the final step, we extract a small number of well-interpretable action rules. Consistently competitive results obtained with the same LLM feature set across both thematically diverse datasets show that this approach generalizes across domains.

Title: Unsupervised Novelty Detection Methods Benchmarking with Wavelet Decomposition

• Authors: Ariel Priarone, Umberto Albertin, Carlo Cena, Mauro Martini, Marcello Chiaberge
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07135
• Pdf URL: https://arxiv.org/pdf/2409.07135
• Copy Paste: [[2409.07135]] Unsupervised Novelty Detection Methods Benchmarking with Wavelet Decomposition(https://arxiv.org/abs/2409.07135)
• Keywords: robust
• Abstract: Novelty detection is a critical task in various engineering fields. Numerous approaches to novelty detection rely on supervised or semi-supervised learning, which requires labelled datasets for training. However, acquiring labelled data, when feasible, can be expensive and time-consuming. For these reasons, unsupervised learning is a powerful alternative that allows performing novelty detection without needing labelled samples. In this study, numerous unsupervised machine learning algorithms for novelty detection are compared, highlighting their strengths and weaknesses in the context of vibration sensing. The proposed framework uses a continuous metric, unlike most traditional methods that merely flag anomalous samples without quantifying the degree of anomaly. Moreover, a new dataset is gathered from an actuator vibrating at specific frequencies to benchmark the algorithms and evaluate the framework. Novel conditions are introduced by altering the input wave signal. Our findings offer valuable insights into the adaptability and robustness of unsupervised learning techniques for real-world novelty detection applications.

Title: Leveraging Unstructured Text Data for Federated Instruction Tuning of Large Language Models

• Authors: Rui Ye, Rui Ge, Yuchi Fengting, Jingyi Chai, Yanfeng Wang, Siheng Chen
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2409.07136
• Pdf URL: https://arxiv.org/pdf/2409.07136
• Copy Paste: [[2409.07136]] Leveraging Unstructured Text Data for Federated Instruction Tuning of Large Language Models(https://arxiv.org/abs/2409.07136)
• Keywords: federate, large language model
• Abstract: Federated instruction tuning enables multiple clients to collaboratively fine-tune a shared large language model (LLM) that can follow humans' instructions without directly sharing raw data. However, existing literature impractically requires that all the clients readily hold instruction-tuning data (i.e., structured instruction-response pairs), which necessitates massive human annotations since clients' data is usually unstructured text instead. Addressing this, we propose a novel and flexible framework FedIT-U2S, which can automatically transform unstructured corpus into structured data for federated instruction tuning. FedIT-U2S consists two key steps: (1) few-shot instruction-tuning data generation, where each unstructured data piece together with several examples is combined to prompt an LLM in generating an instruction-response pair. To further enhance the flexibility, a retrieval-based example selection technique is proposed, where the examples are automatically selected based on the relatedness between the client's data piece and example pool, bypassing the need of determining examples in advance. (2) A typical federated instruction tuning process based on the generated data. Overall, FedIT-U2S can be applied to diverse scenarios as long as the client holds valuable text corpus, broadening the application scope of federated instruction tuning. We conduct a series of experiments on three domains (medicine, knowledge, and math), showing that our proposed FedIT-U2S can consistently and significantly brings improvement over the base LLM.

Title: Combined Optimization of Dynamics and Assimilation with End-to-End Learning on Sparse Observations

• Authors: Vadim Zinchenko, David S. Greenberg
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2409.07137
• Pdf URL: https://arxiv.org/pdf/2409.07137
• Copy Paste: [[2409.07137]] Combined Optimization of Dynamics and Assimilation with End-to-End Learning on Sparse Observations(https://arxiv.org/abs/2409.07137)
• Keywords: robust
• Abstract: Fitting nonlinear dynamical models to sparse and noisy observations is fundamentally challenging. Identifying dynamics requires data assimilation (DA) to estimate system states, but DA requires an accurate dynamical model. To break this deadlock we present CODA, an end-to-end optimization scheme for jointly learning dynamics and DA directly from sparse and noisy observations. A neural network is trained to carry out data accurate, efficient and parallel-in-time DA, while free parameters of the dynamical system are simultaneously optimized. We carry out end-to-end learning directly on observation data, introducing a novel learning objective that combines unrolled auto-regressive dynamics with the data- and self-consistency terms of weak-constraint 4Dvar DA. By taking into account interactions between new and existing simulation components over multiple time steps, CODA can recover initial conditions, fit unknown dynamical parameters and learn neural network-based PDE terms to match both available observations and self-consistency constraints. In addition to facilitating end-to-end learning of dynamics and providing fast, amortized, non-sequential DA, CODA provides greater robustness to model misspecification than classical DA approaches.

Title: Improving Encrypted Transport Protocol Designs: Deep Dive on the QUIC Case

• Authors: Florentin Rochet
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07138
• Pdf URL: https://arxiv.org/pdf/2409.07138
• Copy Paste: [[2409.07138]] Improving Encrypted Transport Protocol Designs: Deep Dive on the QUIC Case(https://arxiv.org/abs/2409.07138)
• Keywords: security
• Abstract: We propose in this paper to revisit the design of existing encrypted transport protocols to improve their efficiency. We call the methodology "Reverso" from reversing the order of field elements within a protocol specification. We detail how such a benign-looking change within the specifications may unlock implementation optimizations for encrypted protocols. To demonstrate our findings, we release quiceh, a QUIC implementation of QUIC VReverso, an extension of the QUIC V1 standard (RFC9000). Our methodology applied to the QUIC protocol reports ~30% of CPU efficiency improvement for processing packets at no added cost on the sender side and without relaxing any security guarantee from QUIC V1. We also implement a fork of Cloudflare's HTTP/3 module and client/server demonstrator using quiceh and show our optimizations to directly transfer to HTTP/3 as well, resulting in our new HTTP/3 to be ~ 38% more efficient than the baseline implementation using QUIC V1. We argue that Reverso applies to any modern encrypted protocol and its implementations and that similar efficiency improvement can also be unlocked for them, independently of the layer in which they operate.

Title: Gated Slot Attention for Efficient Linear-Time Sequence Modeling

• Authors: Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, Peng Zhou, Guohong Fu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07146
• Pdf URL: https://arxiv.org/pdf/2409.07146
• Copy Paste: [[2409.07146]] Gated Slot Attention for Efficient Linear-Time Sequence Modeling(https://arxiv.org/abs/2409.07146)
• Keywords: transformer
• Abstract: Linear attention Transformers and their gated variants, celebrated for enabling parallel training and efficient recurrent inference, still fall short in recall-intensive tasks compared to traditional Transformers and demand significant resources for training from scratch. This paper introduces Gated Slot Attention (GSA), which enhances Attention with Bounded-memory-Control (ABC) by incorporating a gating mechanism inspired by Gated Linear Attention (GLA). Essentially, GSA comprises a two-layer GLA linked via softmax, utilizing context-aware memory reading and adaptive forgetting to improve memory capacity while maintaining compact recurrent state size. This design greatly enhances both training and inference efficiency through GLA's hardware-efficient training algorithm and reduced state size. Additionally, retaining the softmax operation is particularly beneficial in "finetuning pretrained Transformers to RNNs" (T2R) settings, reducing the need for extensive training from scratch. Extensive experiments confirm GSA's superior performance in scenarios requiring in-context recall and in T2R settings.

Title: Ciphertext Policy Attribute Based Encryption with Intel SGX

• Authors: Vivek Suryawanshi (Indian Institute of Technology Kharagpur), Shamik Sural (Indian Institute of Technology Kharagpur)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07149
• Pdf URL: https://arxiv.org/pdf/2409.07149
• Copy Paste: [[2409.07149]] Ciphertext Policy Attribute Based Encryption with Intel SGX(https://arxiv.org/abs/2409.07149)
• Keywords: secure, security, protect, robust
• Abstract: Modern computing environments demand robust security measures to protect sensitive data and resources. Ciphertext-Policy Attribute-Based Encryption (CP-ABE) is a well-established encryption technique known for its fine-grained access control capabilities. However, as the digital landscape evolves, there is a growing need to enhance the security of CP-ABE operations. We propose an approach that utilizes CP-ABE with Intel SGX. It allows data to be encrypted and decrypted securely within the SGX enclave based on the rules in policy by ensuring that only authorized users gain access. We evaluate its performance through different experiments by focusing on key parameters such as the number of rules, attributes and file size. Our results demonstrate the performance and scalability of integrating SGX with CP-ABE in enhancing data security with only minimal increase in execution time due to enclave overhead.

Title: ZKFault: Fault attack analysis on zero-knowledge based post-quantum digital signature schemes

• Authors: Puja Mondal, Supriya Adhikary, Suparna Kundu, Angshuman Karmakar
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07150
• Pdf URL: https://arxiv.org/pdf/2409.07150
• Copy Paste: [[2409.07150]] ZKFault: Fault attack analysis on zero-knowledge based post-quantum digital signature schemes(https://arxiv.org/abs/2409.07150)
• Keywords: secure, security, attack
• Abstract: Computationally hard problems based on coding theory, such as the syndrome decoding problem, have been used for constructing secure cryptographic schemes for a long time. Schemes based on these problems are also assumed to be secure against quantum computers. However, these schemes are often considered impractical for real-world deployment due to large key sizes and inefficient computation time. In the recent call for standardization of additional post-quantum digital signatures by the National Institute of Standards and Technology, several code-based candidates have been proposed, including LESS, CROSS, and MEDS. These schemes are designed on the relatively new zero-knowledge framework. Although several works analyze the hardness of these schemes, there is hardly any work that examines the security of these schemes in the presence of physical attacks. In this work, we analyze these signature schemes from the perspective of fault attacks. All these schemes use a similar tree-based construction to compress the signature size. We attack this component of these schemes. Therefore, our attack is applicable to all of these schemes. In this work, we first analyze the LESS signature scheme and devise our attack. Furthermore, we showed how this attack can be extended to the CROSS signature scheme. Our attacks are built on very simple fault assumptions. Our results show that we can recover the entire secret key of LESS and CROSS using as little as a single fault. Finally, we propose various countermeasures to prevent these kinds of attacks and discuss their efficiency and shortcomings.

Title: A Fine-grained Sentiment Analysis of App Reviews using Large Language Models: An Evaluation Study

• Authors: Faiz Ali Shah, Ahmed Sabir, Rajesh Sharma
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.07162
• Pdf URL: https://arxiv.org/pdf/2409.07162
• Copy Paste: [[2409.07162]] A Fine-grained Sentiment Analysis of App Reviews using Large Language Models: An Evaluation Study(https://arxiv.org/abs/2409.07162)
• Keywords: extraction, large language model
• Abstract: Analyzing user reviews for sentiment towards app features can provide valuable insights into users' perceptions of app functionality and their evolving needs. Given the volume of user reviews received daily, an automated mechanism to generate feature-level sentiment summaries of user reviews is needed. Recent advances in Large Language Models (LLMs) such as ChatGPT have shown impressive performance on several new tasks without updating the model's parameters i.e. using zero or a few labeled examples. Despite these advancements, LLMs' capabilities to perform feature-specific sentiment analysis of user reviews remain unexplored. This study compares the performance of state-of-the-art LLMs, including GPT-4, ChatGPT, and LLama-2-chat variants, for extracting app features and associated sentiments under 0-shot, 1-shot, and 5-shot scenarios. Results indicate the best-performing GPT-4 model outperforms rule-based approaches by 23.6% in f1-score with zero-shot feature extraction; 5-shot further improving it by 6%. GPT-4 achieves a 74% f1-score for predicting positive sentiment towards correctly predicted app features, with 5-shot enhancing it by 7%. Our study suggests that LLM models are promising for generating feature-specific sentiment summaries of user reviews.

Title: H$_2$O$_2$RAM: A High-Performance Hierarchical Doubly Oblivious RAM

• Authors: Leqian Zheng, Zheng Zhang, Wentao Dong, Yao Zhang, Ye Wu, Cong Wang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07167
• Pdf URL: https://arxiv.org/pdf/2409.07167
• Copy Paste: [[2409.07167]] H$_2$O$_2$RAM: A High-Performance Hierarchical Doubly Oblivious RAM(https://arxiv.org/abs/2409.07167)
• Keywords: protect, attack
• Abstract: The combination of Oblivious RAM (ORAM) with Trusted Execution Environments (TEE) has found numerous real-world applications due to their complementary nature. TEEs alleviate the performance bottlenecks of ORAM, such as network bandwidth and roundtrip latency, and ORAM provides general-purpose protection for TEE applications against attacks exploiting memory access patterns. The defining property of this combination, which sets it apart from traditional ORAM designs, is its ability to ensure that memory accesses, both inside and outside of TEEs, are made oblivious, thus termed doubly oblivious RAM (O$_2$RAM). Efforts to develop O$_2$RAM with enhanced performance are ongoing. In this work, we propose H$_2$O$_2$RAM, a high-performance doubly oblivious RAM construction. The distinguishing feature of our approach, compared to the existing tree-based doubly oblivious designs, is its first adoption of the hierarchical framework that enjoys inherently better data locality and parallelization. While the latest hierarchical solution, FutORAMa, achieves concrete efficiency in the classic client-server model by leveraging a relaxed assumption of sublinear-sized client-side private memory, adapting it to our scenario poses challenges due to the conflict between this relaxed assumption and our doubly oblivious requirement. To this end, we introduce several new efficient oblivious components to build a high-performance hierarchical O$_2$RAM (H$_2$O$_2$RAM). We implement our design and evaluate it on various scenarios. The results indicate that H$_2$O$_2$RAM reduces execution time by up to $\sim 10^3$ times and saves memory usage by $5\sim44$ times compared to state-of-the-art solutions.

Title: Swin-LiteMedSAM: A Lightweight Box-Based Segment Anything Model for Large-Scale Medical Image Datasets

• Authors: Ruochen Gao, Donghang Lyu, Marius Staring
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07172
• Pdf URL: https://arxiv.org/pdf/2409.07172
• Copy Paste: [[2409.07172]] Swin-LiteMedSAM: A Lightweight Box-Based Segment Anything Model for Large-Scale Medical Image Datasets(https://arxiv.org/abs/2409.07172)
• Keywords: transformer, segmentation
• Abstract: Medical imaging is essential for the diagnosis and treatment of diseases, with medical image segmentation as a subtask receiving high attention. However, automatic medical image segmentation models are typically task-specific and struggle to handle multiple scenarios, such as different imaging modalities and regions of interest. With the introduction of the Segment Anything Model (SAM), training a universal model for various clinical scenarios has become feasible. Recently, several Medical SAM (MedSAM) methods have been proposed, but these models often rely on heavy image encoders to achieve high performance, which may not be practical for real-world applications due to their high computational demands and slow inference speed. To address this issue, a lightweight version of the MedSAM (LiteMedSAM) can provide a viable solution, achieving high performance while requiring fewer resources and less time. In this work, we introduce Swin-LiteMedSAM, a new variant of LiteMedSAM. This model integrates the tiny Swin Transformer as the image encoder, incorporates multiple types of prompts, including box-based points and scribble generated from a given bounding box, and establishes skip connections between the image encoder and the mask decoder. In the \textit{Segment Anything in Medical Images on Laptop} challenge (CVPR 2024), our approach strikes a good balance between segmentation performance and speed, demonstrating significantly improved overall results across multiple modalities compared to the LiteMedSAM baseline provided by the challenge organizers. Our proposed model achieved a DSC score of \textbf{0.8678} and an NSD score of \textbf{0.8844} on the validation set. On the final test set, it attained a DSC score of \textbf{0.8193} and an NSD score of \textbf{0.8461}, securing fourth place in the challenge.

Title: Phy124: Fast Physics-Driven 4D Content Generation from a Single Image

• Authors: Jiajing Lin, Zhenzhong Wang, Yongjie Hou, Yuzhou Tang, Min Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07179
• Pdf URL: https://arxiv.org/pdf/2409.07179
• Copy Paste: [[2409.07179]] Phy124: Fast Physics-Driven 4D Content Generation from a Single Image(https://arxiv.org/abs/2409.07179)
• Keywords: diffusion
• Abstract: 4D content generation focuses on creating dynamic 3D objects that change over time. Existing methods primarily rely on pre-trained video diffusion models, utilizing sampling processes or reference videos. However, these approaches face significant challenges. Firstly, the generated 4D content often fails to adhere to real-world physics since video diffusion models do not incorporate physical priors. Secondly, the extensive sampling process and the large number of parameters in diffusion models result in exceedingly time-consuming generation processes. To address these issues, we introduce Phy124, a novel, fast, and physics-driven method for controllable 4D content generation from a single image. Phy124 integrates physical simulation directly into the 4D generation process, ensuring that the resulting 4D content adheres to natural physical laws. Phy124 also eliminates the use of diffusion models during the 4D dynamics generation phase, significantly speeding up the process. Phy124 allows for the control of 4D dynamics, including movement speed and direction, by manipulating external forces. Extensive experiments demonstrate that Phy124 generates high-fidelity 4D content with significantly reduced inference times, achieving stateof-the-art performance. The code and generated 4D content are available at the provided link: https://anonymous.4open.science/r/BBF2/.

Title: Enhancing Angular Resolution via Directionality Encoding and Geometric Constraints in Brain Diffusion Tensor Imaging

• Authors: Sheng Chen, Zihao Tang, Mariano Cabezas, Xinyi Wang, Arkiev D'Souza, Michael Barnett, Fernando Calamante, Weidong Cai, Chenyu Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07186
• Pdf URL: https://arxiv.org/pdf/2409.07186
• Copy Paste: [[2409.07186]] Enhancing Angular Resolution via Directionality Encoding and Geometric Constraints in Brain Diffusion Tensor Imaging(https://arxiv.org/abs/2409.07186)
• Keywords: robust, diffusion
• Abstract: Diffusion-weighted imaging (DWI) is a type of Magnetic Resonance Imaging (MRI) technique sensitised to the diffusivity of water molecules, offering the capability to inspect tissue microstructures and is the only in-vivo method to reconstruct white matter fiber tracts non-invasively. The DWI signal can be analysed with the diffusion tensor imaging (DTI) model to estimate the directionality of water diffusion within voxels. Several scalar metrics, including axial diffusivity (AD), mean diffusivity (MD), radial diffusivity (RD), and fractional anisotropy (FA), can be further derived from DTI to quantitatively summarise the microstructural integrity of brain tissue. These scalar metrics have played an important role in understanding the organisation and health of brain tissue at a microscopic level in clinical studies. However, reliable DTI metrics rely on DWI acquisitions with high gradient directions, which often go beyond the commonly used clinical protocols. To enhance the utility of clinically acquired DWI and save scanning time for robust DTI analysis, this work proposes DirGeo-DTI, a deep learning-based method to estimate reliable DTI metrics even from a set of DWIs acquired with the minimum theoretical number (6) of gradient directions. DirGeo-DTI leverages directional encoding and geometric constraints to facilitate the training process. Two public DWI datasets were used for evaluation, demonstrating the effectiveness of the proposed method. Extensive experimental results show that the proposed method achieves the best performance compared to existing DTI enhancement methods and potentially reveals further clinical insights with routine clinical DWI scans.

Title: Cyber Deception: State of the art, Trends and Open challenges

• Authors: Pedro Beltrán López, Manuel Gil Pérez, Pantaleone Nespoli
• Subjects: cs.CR, cs.AI, cs.GT
• Abstract URL: https://arxiv.org/abs/2409.07194
• Pdf URL: https://arxiv.org/pdf/2409.07194
• Copy Paste: [[2409.07194]] Cyber Deception: State of the art, Trends and Open challenges(https://arxiv.org/abs/2409.07194)
• Keywords: security, defense, attack
• Abstract: The growing interest in cybersecurity has significantly increased articles designing and implementing various Cyber Deception (CYDEC) mechanisms. This trend reflects the urgent need for new strategies to address cyber threats effectively. Since its emergence, CYDEC has established itself as an innovative defense against attackers, thanks to its proactive and reactive capabilities, finding applications in numerous real-life scenarios. Despite the considerable work devoted to CYDEC, the literature still presents significant gaps. In particular, there has not been (i) a comprehensive analysis of the main components characterizing CYDEC, (ii) a generic classification covering all types of solutions, nor (iii) a survey of the current state of the literature in various contexts. This article aims to fill these gaps through a detailed review of the main features that comprise CYDEC, developing a comprehensive classification taxonomy. In addition, the different frameworks used to generate CYDEC are reviewed, presenting a more comprehensive one. Existing solutions in the literature using CYDEC, both without Artificial Intelligence (AI) and with AI, are studied and compared. Finally, the most salient trends of the current state of the art are discussed, offering a list of pending challenges for future research.

Title: Heterogeneity-Aware Coordination for Federated Learning via Stitching Pre-trained blocks

• Authors: Shichen Zhan, Yebo Wu, Chunlin Tian, Yan Zhao, Li Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07202
• Pdf URL: https://arxiv.org/pdf/2409.07202
• Copy Paste: [[2409.07202]] Heterogeneity-Aware Coordination for Federated Learning via Stitching Pre-trained blocks(https://arxiv.org/abs/2409.07202)
• Keywords: privacy, federate
• Abstract: Federated learning (FL) coordinates multiple devices to collaboratively train a shared model while preserving data privacy. However, large memory footprint and high energy consumption during the training process excludes the low-end devices from contributing to the global model with their own data, which severely deteriorates the model performance in real-world scenarios. In this paper, we propose FedStitch, a hierarchical coordination framework for heterogeneous federated learning with pre-trained blocks. Unlike the traditional approaches that train the global model from scratch, for a new task, FedStitch composes the global model via stitching pre-trained blocks. Specifically, each participating client selects the most suitable block based on their local data from the candidate pool composed of blocks from pre-trained models. The server then aggregates the optimal block for stitching. This process iterates until a new stitched network is generated. Except for the new training paradigm, FedStitch consists of the following three core components: 1) an RL-weighted aggregator, 2) a search space optimizer deployed on the server side, and 3) a local energy optimizer deployed on each participating client. The RL-weighted aggregator helps to select the right block in the non-IID scenario, while the search space optimizer continuously reduces the size of the candidate block pool during stitching. Meanwhile, the local energy optimizer is designed to minimize energy consumption of each client while guaranteeing the overall training progress. The results demonstrate that compared to existing approaches, FedStitch improves the model accuracy up to 20.93%. At the same time, it achieves up to 8.12% speedup, reduces the memory footprint up to 79.5%, and achieves 89.41% energy saving at most during the learning procedure.

Title: Watchlist Challenge: 3rd Open-set Face Detection and Identification

• Authors: Furkan Kasım, Terrance E. Boult, Rensso Mora, Bernardo Biesseck, Rafael Ribeiro, Jan Schlueter, Tomáš Repák, Rafael Henrique Vareto, David Menotti, William Robson Schwartz, Manuel Günther
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07220
• Pdf URL: https://arxiv.org/pdf/2409.07220
• Copy Paste: [[2409.07220]] Watchlist Challenge: 3rd Open-set Face Detection and Identification(https://arxiv.org/abs/2409.07220)
• Keywords: robust, biometric
• Abstract: In the current landscape of biometrics and surveillance, the ability to accurately recognize faces in uncontrolled settings is paramount. The Watchlist Challenge addresses this critical need by focusing on face detection and open-set identification in real-world surveillance scenarios. This paper presents a comprehensive evaluation of participating algorithms, using the enhanced UnConstrained College Students (UCCS) dataset with new evaluation protocols. In total, four participants submitted four face detection and nine open-set face recognition systems. The evaluation demonstrates that while detection capabilities are generally robust, closed-set identification performance varies significantly, with models pre-trained on large-scale datasets showing superior performance. However, open-set scenarios require further improvement, especially at higher true positive identification rates, i.e., lower thresholds.

Title: Riemannian Federated Learning via Averaging Gradient Stream

• Authors: Zhenwei Huang, Wen Huang, Pratik Jawanpuria, Bamdev Mishra
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2409.07223
• Pdf URL: https://arxiv.org/pdf/2409.07223
• Copy Paste: [[2409.07223]] Riemannian Federated Learning via Averaging Gradient Stream(https://arxiv.org/abs/2409.07223)
• Keywords: privacy, federate
• Abstract: In recent years, federated learning has garnered significant attention as an efficient and privacy-preserving distributed learning paradigm. In the Euclidean setting, Federated Averaging (FedAvg) and its variants are a class of efficient algorithms for expected (empirical) risk minimization. This paper develops and analyzes a Riemannian Federated Averaging Gradient Stream (RFedAGS) algorithm, which is a generalization of FedAvg, to problems defined on a Riemannian manifold. Under standard assumptions, the convergence rate of RFedAGS with fixed step sizes is proven to be sublinear for an approximate stationary solution. If decaying step sizes are used, the global convergence is established. Furthermore, assuming that the objective obeys the Riemannian Polyak-Łojasiewicz property, the optimal gaps generated by RFedAGS with fixed step size are linearly decreasing up to a tiny upper bound, meanwhile, if decaying step sizes are used, then the gaps sublinearly vanish. Numerical simulations conducted on synthetic and real-world data demonstrate the performance of the proposed RFedAGS.

Title: Diff-VPS: Video Polyp Segmentation via a Multi-task Diffusion Network with Adversarial Temporal Reasoning

• Authors: Yingling Lu, Yijun Yang, Zhaohu Xing, Qiong Wang, Lei Zhu
• Subjects: cs.CV, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.07238
• Pdf URL: https://arxiv.org/pdf/2409.07238
• Copy Paste: [[2409.07238]] Diff-VPS: Video Polyp Segmentation via a Multi-task Diffusion Network with Adversarial Temporal Reasoning(https://arxiv.org/abs/2409.07238)
• Keywords: diffusion, generative, segmentation
• Abstract: Diffusion Probabilistic Models have recently attracted significant attention in the community of computer vision due to their outstanding performance. However, while a substantial amount of diffusion-based research has focused on generative tasks, no work introduces diffusion models to advance the results of polyp segmentation in videos, which is frequently challenged by polyps' high camouflage and redundant temporal this http URL this paper, we present a novel diffusion-based network for video polyp segmentation task, dubbed as Diff-VPS. We incorporate multi-task supervision into diffusion models to promote the discrimination of diffusion models on pixel-by-pixel segmentation. This integrates the contextual high-level information achieved by the joint classification and detection tasks. To explore the temporal dependency, Temporal Reasoning Module (TRM) is devised via reasoning and reconstructing the target frame from the previous frames. We further equip TRM with a generative adversarial self-supervised strategy to produce more realistic frames and thus capture better dynamic cues. Extensive experiments are conducted on SUN-SEG, and the results indicate that our proposed Diff-VPS significantly achieves state-of-the-art performance. Code is available at this https URL.

Title: PiTe: Pixel-Temporal Alignment for Large Video-Language Model

• Authors: Yang Liu, Pengxiang Ding, Siteng Huang, Min Zhang, Han Zhao, Donglin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07239
• Pdf URL: https://arxiv.org/pdf/2409.07239
• Copy Paste: [[2409.07239]] PiTe: Pixel-Temporal Alignment for Large Video-Language Model(https://arxiv.org/abs/2409.07239)
• Keywords: large language model
• Abstract: Fueled by the Large Language Models (LLMs) wave, Large Visual-Language Models (LVLMs) have emerged as a pivotal advancement, bridging the gap between image and text. However, video making it challenging for LVLMs to perform adequately due to the complexity of the relationship between language and spatial-temporal data structure. Recent Large Video-Language Models (LVidLMs) align feature of static visual data like image into latent space of language feature, by general multi-modal tasks to leverage abilities of LLMs sufficiently. In this paper, we explore fine-grained alignment approach via object trajectory for different modalities across both spatial and temporal dimensions simultaneously. Thus, we propose a novel LVidLM by trajectory-guided Pixel-Temporal Alignment, dubbed PiTe, that exhibits promising applicable model property. To achieve fine-grained video-language alignment, we curate a multi-modal pre-training dataset PiTe-143k, the dataset provision of moving trajectories in pixel level for all individual objects, that appear and mention in the video and caption both, by our automatic annotation pipeline. Meanwhile, PiTe demonstrates astounding capabilities on myriad video-related multi-modal tasks through beat the state-of-the-art methods by a large margin.

Title: Single-View 3D Reconstruction via SO(2)-Equivariant Gaussian Sculpting Networks

• Authors: Ruihan Xu, Anthony Opipari, Joshua Mah, Stanley Lewis, Haoran Zhang, Hanzhe Guo, Odest Chadwicke Jenkins
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.07245
• Pdf URL: https://arxiv.org/pdf/2409.07245
• Copy Paste: [[2409.07245]] Single-View 3D Reconstruction via SO(2)-Equivariant Gaussian Sculpting Networks(https://arxiv.org/abs/2409.07245)
• Keywords: diffusion
• Abstract: This paper introduces SO(2)-Equivariant Gaussian Sculpting Networks (GSNs) as an approach for SO(2)-Equivariant 3D object reconstruction from single-view image observations. GSNs take a single observation as input to generate a Gaussian splat representation describing the observed object's geometry and texture. By using a shared feature extractor before decoding Gaussian colors, covariances, positions, and opacities, GSNs achieve extremely high throughput (>150FPS). Experiments demonstrate that GSNs can be trained efficiently using a multi-view rendering loss and are competitive, in quality, with expensive diffusion-based reconstruction algorithms. The GSN model is validated on multiple benchmark experiments. Moreover, we demonstrate the potential for GSNs to be used within a robotic manipulation pipeline for object-centric grasping.

Title: Alignment of Diffusion Models: Fundamentals, Challenges, and Future

• Authors: Buhua Liu, Shitong Shao, Bao Li, Lichen Bai, Haoyi Xiong, James Kwok, Sumi Helal, Zeke Xie
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07253
• Pdf URL: https://arxiv.org/pdf/2409.07253
• Copy Paste: [[2409.07253]] Alignment of Diffusion Models: Fundamentals, Challenges, and Future(https://arxiv.org/abs/2409.07253)
• Keywords: diffusion, generative, large language model
• Abstract: Diffusion models have emerged as the leading paradigm in generative modeling, excelling in various applications. Despite their success, these models often misalign with human intentions, generating outputs that may not match text prompts or possess desired properties. Inspired by the success of alignment in tuning large language models, recent studies have investigated aligning diffusion models with human expectations and preferences. This work mainly reviews alignment of diffusion models, covering advancements in fundamentals of alignment, alignment techniques of diffusion models, preference benchmarks, and evaluation for diffusion models. Moreover, we discuss key perspectives on current challenges and promising future directions on solving the remaining challenges in alignment of diffusion models. To the best of our knowledge, our work is the first comprehensive review paper for researchers and engineers to comprehend, practice, and research alignment of diffusion models.

Title: EMOdiffhead: Continuously Emotional Control in Talking Head Generation via Diffusion

• Authors: Jian Zhang, Weijian Mai, Zhijun Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07255
• Pdf URL: https://arxiv.org/pdf/2409.07255
• Copy Paste: [[2409.07255]] EMOdiffhead: Continuously Emotional Control in Talking Head Generation via Diffusion(https://arxiv.org/abs/2409.07255)
• Keywords: diffusion
• Abstract: The task of audio-driven portrait animation involves generating a talking head video using an identity image and an audio track of speech. While many existing approaches focus on lip synchronization and video quality, few tackle the challenge of generating emotion-driven talking head videos. The ability to control and edit emotions is essential for producing expressive and realistic animations. In response to this challenge, we propose EMOdiffhead, a novel method for emotional talking head video generation that not only enables fine-grained control of emotion categories and intensities but also enables one-shot generation. Given the FLAME 3D model's linearity in expression modeling, we utilize the DECA method to extract expression vectors, that are combined with audio to guide a diffusion model in generating videos with precise lip synchronization and rich emotional expressiveness. This approach not only enables the learning of rich facial information from emotion-irrelevant data but also facilitates the generation of emotional videos. It effectively overcomes the limitations of emotional data, such as the lack of diversity in facial and background information, and addresses the absence of emotional details in emotion-irrelevant data. Extensive experiments and user studies demonstrate that our approach achieves state-of-the-art performance compared to other emotion portrait animation methods.

Title: MRAC Track 1: 2nd Workshop on Multimodal, Generative and Responsible Affective Computing

• Authors: Shreya Ghosh, Zhixi Cai, Abhinav Dhall, Dimitrios Kollias, Roland Goecke, Tom Gedeon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07256
• Pdf URL: https://arxiv.org/pdf/2409.07256
• Copy Paste: [[2409.07256]] MRAC Track 1: 2nd Workshop on Multimodal, Generative and Responsible Affective Computing(https://arxiv.org/abs/2409.07256)
• Keywords: generative
• Abstract: With the rapid advancements in multimodal generative technology, Affective Computing research has provoked discussion about the potential consequences of AI systems equipped with emotional intelligence. Affective Computing involves the design, evaluation, and implementation of Emotion AI and related technologies aimed at improving people's lives. Designing a computational model in affective computing requires vast amounts of multimodal data, including RGB images, video, audio, text, and physiological signals. Moreover, Affective Computing research is deeply engaged with ethical considerations at various stages-from training emotionally intelligent models on large-scale human data to deploying these models in specific applications. Fundamentally, the development of any AI system must prioritize its impact on humans, aiming to augment and enhance human abilities rather than replace them, while drawing inspiration from human intelligence in a safe and responsible manner. The MRAC 2024 Track 1 workshop seeks to extend these principles from controlled, small-scale lab environments to real-world, large-scale contexts, emphasizing responsible development. The workshop also aims to highlight the potential implications of generative technology, along with the ethical consequences of its use, to researchers and industry professionals. To the best of our knowledge, this is the first workshop series to comprehensively address the full spectrum of multimodal, generative affective computing from a responsible AI perspective, and this is the second iteration of this workshop. Webpage: this https URL

Title: MiniDrive: More Efficient Vision-Language Models with Multi-Level 2D Features as Text Tokens for Autonomous Driving

• Authors: Enming Zhang, Xingyuan Dai, Yisheng Lv, Qianghai Miao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07267
• Pdf URL: https://arxiv.org/pdf/2409.07267
• Copy Paste: [[2409.07267]] MiniDrive: More Efficient Vision-Language Models with Multi-Level 2D Features as Text Tokens for Autonomous Driving(https://arxiv.org/abs/2409.07267)
• Keywords: large language model
• Abstract: Vision-language models (VLMs) serve as general-purpose end-to-end models in autonomous driving, performing subtasks such as prediction, planning, and perception through question-and-answer interactions. However, most existing methods rely on computationally expensive visual encoders and large language models (LLMs), making them difficult to deploy in real-world scenarios and real-time applications. Meanwhile, most existing VLMs lack the ability to process multiple images, making it difficult to adapt to multi-camera perception in autonomous driving. To address these issues, we propose a novel framework called MiniDrive, which incorporates our proposed Feature Engineering Mixture of Experts (FE-MoE) module and Dynamic Instruction Adapter (DI-Adapter). The FE-MoE effectively maps 2D features into visual token embeddings before being input into the language model. The DI-Adapter enables the visual token embeddings to dynamically change with the instruction text embeddings, resolving the issue of static visual token embeddings for the same image in previous approaches. Compared to previous works, MiniDrive achieves state-of-the-art performance in terms of parameter size, floating point operations, and response efficiency, with the smallest version containing only 83M parameters.

Title: Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models

• Authors: Sanoojan Baliah, Qinliang Lin, Shengcai Liao, Xiaodan Liang, Muhammad Haris Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07269
• Pdf URL: https://arxiv.org/pdf/2409.07269
• Copy Paste: [[2409.07269]] Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models(https://arxiv.org/abs/2409.07269)
• Keywords: robust, diffusion
• Abstract: Despite promising progress in face swapping task, realistic swapped images remain elusive, often marred by artifacts, particularly in scenarios involving high pose variation, color differences, and occlusion. To address these issues, we propose a novel approach that better harnesses diffusion models for face-swapping by making following core contributions. (a) We propose to re-frame the face-swapping task as a self-supervised, train-time inpainting problem, enhancing the identity transfer while blending with the target image. (b) We introduce a multi-step Denoising Diffusion Implicit Model (DDIM) sampling during training, reinforcing identity and perceptual similarities. (c) Third, we introduce CLIP feature disentanglement to extract pose, expression, and lighting information from the target image, improving fidelity. (d) Further, we introduce a mask shuffling technique during inpainting training, which allows us to create a so-called universal model for swapping, with an additional feature of head swapping. Ours can swap hair and even accessories, beyond traditional face swapping. Unlike prior works reliant on multiple off-the-shelf models, ours is a relatively unified approach and so it is resilient to errors in other off-the-shelf models. Extensive experiments on FFHQ and CelebA datasets validate the efficacy and robustness of our approach, showcasing high-fidelity, realistic face-swapping with minimal inference time. Our code is available at this https URL.

Title: CCFExp: Facial Image Synthesis with Cycle Cross-Fusion Diffusion Model for Facial Paralysis Individuals

• Authors: Weixiang Gao, Yifan Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07271
• Pdf URL: https://arxiv.org/pdf/2409.07271
• Copy Paste: [[2409.07271]] CCFExp: Facial Image Synthesis with Cycle Cross-Fusion Diffusion Model for Facial Paralysis Individuals(https://arxiv.org/abs/2409.07271)
• Keywords: robust, diffusion, generative
• Abstract: Facial paralysis is a debilitating condition that affects the movement of facial muscles, leading to a significant loss of facial expressions. Currently, the diagnosis of facial paralysis remains a challenging task, often relying heavily on the subjective judgment and experience of clinicians, which can introduce variability and uncertainty in the assessment process. One promising application in real-life situations is the automatic estimation of facial paralysis. However, the scarcity of facial paralysis datasets limits the development of robust machine learning models for automated diagnosis and therapeutic interventions. To this end, this study aims to synthesize a high-quality facial paralysis dataset to address this gap, enabling more accurate and efficient algorithm training. Specifically, a novel Cycle Cross-Fusion Expression Generative Model (CCFExp) based on the diffusion model is proposed to combine different features of facial information and enhance the visual details of facial appearance and texture in facial regions, thus creating synthetic facial images that accurately represent various degrees and types of facial paralysis. We have qualitatively and quantitatively evaluated the proposed method on the commonly used public clinical datasets of facial paralysis to demonstrate its effectiveness. Experimental results indicate that the proposed method surpasses state-of-the-art methods, generating more realistic facial images and maintaining identity consistency.

Title: Tuning-Free Online Robust Principal Component Analysis through Implicit Regularization

• Authors: Lakshmi Jayalal, Gokularam Muthukrishnan, Sheetal Kalyani
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.07275
• Pdf URL: https://arxiv.org/pdf/2409.07275
• Copy Paste: [[2409.07275]] Tuning-Free Online Robust Principal Component Analysis through Implicit Regularization(https://arxiv.org/abs/2409.07275)
• Keywords: robust
• Abstract: The performance of the standard Online Robust Principal Component Analysis (OR-PCA) technique depends on the optimum tuning of the explicit regularizers and this tuning is dataset sensitive. We aim to remove the dependency on these tuning parameters by using implicit regularization. We propose to use the implicit regularization effect of various modified gradient descents to make OR-PCA tuning free. Our method incorporates three different versions of modified gradient descent that separately but naturally encourage sparsity and low-rank structures in the data. The proposed method performs comparable or better than the tuned OR-PCA for both simulated and real-world datasets. Tuning-free ORPCA makes it more scalable for large datasets since we do not require dataset-dependent parameter tuning.

Title: TLD-READY: Traffic Light Detection -- Relevance Estimation and Deployment Analysis

• Authors: Nikolai Polley, Svetlana Pavlitska, Yacin Boualili, Patrick Rohrbeck, Paul Stiller, Ashok Kumar Bangaru, J. Marius Zöllner
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07284
• Pdf URL: https://arxiv.org/pdf/2409.07284
• Copy Paste: [[2409.07284]] TLD-READY: Traffic Light Detection -- Relevance Estimation and Deployment Analysis(https://arxiv.org/abs/2409.07284)
• Keywords: robust
• Abstract: Effective traffic light detection is a critical component of the perception stack in autonomous vehicles. This work introduces a novel deep-learning detection system while addressing the challenges of previous work. Utilizing a comprehensive dataset amalgamation, including the Bosch Small Traffic Lights Dataset, LISA, the DriveU Traffic Light Dataset, and a proprietary dataset from Karlsruhe, we ensure a robust evaluation across varied scenarios. Furthermore, we propose a relevance estimation system that innovatively uses directional arrow markings on the road, eliminating the need for prior map creation. On the DriveU dataset, this approach results in 96% accuracy in relevance estimation. Finally, a real-world evaluation is performed to evaluate the deployment and generalizing abilities of these models. For reproducibility and to facilitate further research, we provide the model weights and code: this https URL.

Title: Exploring User-level Gradient Inversion with a Diffusion Prior

• Authors: Zhuohang Li, Andrew Lowy, Jing Liu, Toshiaki Koike-Akino, Bradley Malin, Kieran Parsons, Ye Wang
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.07291
• Pdf URL: https://arxiv.org/pdf/2409.07291
• Copy Paste: [[2409.07291]] Exploring User-level Gradient Inversion with a Diffusion Prior(https://arxiv.org/abs/2409.07291)
• Keywords: attack, diffusion
• Abstract: We explore user-level gradient inversion as a new attack surface in distributed learning. We first investigate existing attacks on their ability to make inferences about private information beyond training data reconstruction. Motivated by the low reconstruction quality of existing methods, we propose a novel gradient inversion attack that applies a denoising diffusion model as a strong image prior in order to enhance recovery in the large batch setting. Unlike traditional attacks, which aim to reconstruct individual samples and suffer at large batch and image sizes, our approach instead aims to recover a representative image that captures the sensitive shared semantic information corresponding to the underlying user. Our experiments with face images demonstrate the ability of our methods to recover realistic facial images along with private user attributes.

Title: PaveSAM Segment Anything for Pavement Distress

• Authors: Neema Jakisa Owor, Yaw Adu-Gyamfi, Armstrong Aboah, Mark Amo-Boateng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07295
• Pdf URL: https://arxiv.org/pdf/2409.07295
• Copy Paste: [[2409.07295]] PaveSAM Segment Anything for Pavement Distress(https://arxiv.org/abs/2409.07295)
• Keywords: segmentation
• Abstract: Automated pavement monitoring using computer vision can analyze pavement conditions more efficiently and accurately than manual methods. Accurate segmentation is essential for quantifying the severity and extent of pavement defects and consequently, the overall condition index used for prioritizing rehabilitation and maintenance activities. Deep learning-based segmentation models are however, often supervised and require pixel-level annotations, which can be costly and time-consuming. While the recent evolution of zero-shot segmentation models can generate pixel-wise labels for unseen classes without any training data, they struggle with irregularities of cracks and textured pavement backgrounds. This research proposes a zero-shot segmentation model, PaveSAM, that can segment pavement distresses using bounding box prompts. By retraining SAM's mask decoder with just 180 images, pavement distress segmentation is revolutionized, enabling efficient distress segmentation using bounding box prompts, a capability not found in current segmentation models. This not only drastically reduces labeling efforts and costs but also showcases our model's high performance with minimal input, establishing the pioneering use of SAM in pavement distress segmentation. Furthermore, researchers can use existing open-source pavement distress images annotated with bounding boxes to create segmentation masks, which increases the availability and diversity of segmentation pavement distress datasets.

Title: Data Augmentation via Latent Diffusion for Saliency Prediction

• Authors: Bahar Aydemir, Deblina Bhattacharjee, Tong Zhang, Mathieu Salzmann, Sabine Süsstrunk
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07307
• Pdf URL: https://arxiv.org/pdf/2409.07307
• Copy Paste: [[2409.07307]] Data Augmentation via Latent Diffusion for Saliency Prediction(https://arxiv.org/abs/2409.07307)
• Keywords: diffusion
• Abstract: Saliency prediction models are constrained by the limited diversity and quantity of labeled data. Standard data augmentation techniques such as rotating and cropping alter scene composition, affecting saliency. We propose a novel data augmentation method for deep saliency prediction that edits natural images while preserving the complexity and variability of real-world scenes. Since saliency depends on high-level and low-level features, our approach involves learning both by incorporating photometric and semantic attributes such as color, contrast, brightness, and class. To that end, we introduce a saliency-guided cross-attention mechanism that enables targeted edits on the photometric properties, thereby enhancing saliency within specific image regions. Experimental results show that our data augmentation method consistently improves the performance of various saliency models. Moreover, leveraging the augmentation features for saliency prediction yields superior performance on publicly available saliency benchmarks. Our predictions align closely with human visual attention patterns in the edited images, as validated by a user study.

Title: Optimizing Neural Network Performance and Interpretability with Diophantine Equation Encoding

• Authors: Ronald Katende
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2409.07310
• Pdf URL: https://arxiv.org/pdf/2409.07310
• Copy Paste: [[2409.07310]] Optimizing Neural Network Performance and Interpretability with Diophantine Equation Encoding(https://arxiv.org/abs/2409.07310)
• Keywords: attack, robust, interpretability
• Abstract: This paper explores the integration of Diophantine equations into neural network (NN) architectures to improve model interpretability, stability, and efficiency. By encoding and decoding neural network parameters as integer solutions to Diophantine equations, we introduce a novel approach that enhances both the precision and robustness of deep learning models. Our method integrates a custom loss function that enforces Diophantine constraints during training, leading to better generalization, reduced error bounds, and enhanced resilience against adversarial attacks. We demonstrate the efficacy of this approach through several tasks, including image classification and natural language processing, where improvements in accuracy, convergence, and robustness are observed. This study offers a new perspective on combining mathematical theory and machine learning to create more interpretable and efficient models.

Title: MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications

• Authors: Praveen K Kanithi, Clément Christophe, Marco AF Pimentel, Tathagata Raha, Nada Saadi, Hamza Javed, Svetlana Maslenkova, Nasir Hayat, Ronnie Rajan, Shadab Khan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07314
• Pdf URL: https://arxiv.org/pdf/2409.07314
• Copy Paste: [[2409.07314]] MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical Applications(https://arxiv.org/abs/2409.07314)
• Keywords: large language model
• Abstract: The rapid development of Large Language Models (LLMs) for healthcare applications has spurred calls for holistic evaluation beyond frequently-cited benchmarks like USMLE, to better reflect real-world performance. While real-world assessments are valuable indicators of utility, they often lag behind the pace of LLM evolution, likely rendering findings obsolete upon deployment. This temporal disconnect necessitates a comprehensive upfront evaluation that can guide model selection for specific clinical applications. We introduce MEDIC, a framework assessing LLMs across five critical dimensions of clinical competence: medical reasoning, ethics and bias, data and language understanding, in-context learning, and clinical safety. MEDIC features a novel cross-examination framework quantifying LLM performance across areas like coverage and hallucination detection, without requiring reference outputs. We apply MEDIC to evaluate LLMs on medical question-answering, safety, summarization, note generation, and other tasks. Our results show performance disparities across model sizes, baseline vs medically finetuned models, and have implications on model selection for applications requiring specific model strengths, such as low hallucination or lower cost of inference. MEDIC's multifaceted evaluation reveals these performance trade-offs, bridging the gap between theoretical capabilities and practical implementation in healthcare settings, ensuring that the most promising models are identified and adapted for diverse healthcare applications.

Title: Module-wise Adaptive Adversarial Training for End-to-end Autonomous Driving

• Authors: Tianyuan Zhang, Lu Wang, Jiaqi Kang, Xinwei Zhang, Siyuan Liang, Yuwei Chen, Aishan Liu, Xianglong Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07321
• Pdf URL: https://arxiv.org/pdf/2409.07321
• Copy Paste: [[2409.07321]] Module-wise Adaptive Adversarial Training for End-to-end Autonomous Driving(https://arxiv.org/abs/2409.07321)
• Keywords: defense, attack, robust
• Abstract: Recent advances in deep learning have markedly improved autonomous driving (AD) models, particularly end-to-end systems that integrate perception, prediction, and planning stages, achieving state-of-the-art performance. However, these models remain vulnerable to adversarial attacks, where human-imperceptible perturbations can disrupt decision-making processes. While adversarial training is an effective method for enhancing model robustness against such attacks, no prior studies have focused on its application to end-to-end AD models. In this paper, we take the first step in adversarial training for end-to-end AD models and present a novel Module-wise Adaptive Adversarial Training (MA2T). However, extending conventional adversarial training to this context is highly non-trivial, as different stages within the model have distinct objectives and are strongly interconnected. To address these challenges, MA2T first introduces Module-wise Noise Injection, which injects noise before the input of different modules, targeting training models with the guidance of overall objectives rather than each independent module loss. Additionally, we introduce Dynamic Weight Accumulation Adaptation, which incorporates accumulated weight changes to adaptively learn and adjust the loss weights of each module based on their contributions (accumulated reduction rates) for better balance and robust training. To demonstrate the efficacy of our defense, we conduct extensive experiments on the widely-used nuScenes dataset across several end-to-end AD models under both white-box and black-box attacks, where our method outperforms other baselines by large margins (+5-10%). Moreover, we validate the robustness of our defense through closed-loop evaluation in the CARLA simulation environment, showing improved resilience even against natural corruption.

Title: Efficient and Unbiased Sampling of Boltzmann Distributions via Consistency Models

• Authors: Fengzhe Zhang, Jiajun He, Laurence I. Midgley, Javier Antorán, José Miguel Hernández-Lobato
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.07323
• Pdf URL: https://arxiv.org/pdf/2409.07323
• Copy Paste: [[2409.07323]] Efficient and Unbiased Sampling of Boltzmann Distributions via Consistency Models(https://arxiv.org/abs/2409.07323)
• Keywords: diffusion
• Abstract: Diffusion models have shown promising potential for advancing Boltzmann Generators. However, two critical challenges persist: (1) inherent errors in samples due to model imperfections, and (2) the requirement of hundreds of functional evaluations (NFEs) to achieve high-quality samples. While existing solutions like importance sampling and distillation address these issues separately, they are often incompatible, as most distillation models lack the necessary density information for importance sampling. This paper introduces a novel sampling method that effectively combines Consistency Models (CMs) with importance sampling. We evaluate our approach on both synthetic energy functions and equivariant n-body particle systems. Our method produces unbiased samples using only 6-25 NFEs while achieving a comparable Effective Sample Size (ESS) to Denoising Diffusion Probabilistic Models (DDPMs) that require approximately 100 NFEs.

Title: Current Symmetry Group Equivariant Convolution Frameworks for Representation Learning

• Authors: Ramzan Basheer, Deepak Mishra
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07327
• Pdf URL: https://arxiv.org/pdf/2409.07327
• Copy Paste: [[2409.07327]] Current Symmetry Group Equivariant Convolution Frameworks for Representation Learning(https://arxiv.org/abs/2409.07327)
• Keywords: robust
• Abstract: Euclidean deep learning is often inadequate for addressing real-world signals where the representation space is irregular and curved with complex topologies. Interpreting the geometric properties of such feature spaces has become paramount in obtaining robust and compact feature representations that remain unaffected by nontrivial geometric transformations, which vanilla CNNs cannot effectively handle. Recognizing rotation, translation, permutation, or scale symmetries can lead to equivariance properties in the learned representations. This has led to notable advancements in computer vision and machine learning tasks under the framework of geometric deep learning, as compared to their invariant counterparts. In this report, we emphasize the importance of symmetry group equivariant deep learning models and their realization of convolution-like operations on graphs, 3D shapes, and non-Euclidean spaces by leveraging group theory and symmetry. We categorize them as regular, steerable, and PDE-based convolutions and thoroughly examine the inherent symmetries of their input spaces and ensuing representations. We also outline the mathematical link between group convolutions or message aggregation operations and the concept of equivariance. The report also highlights various datasets, their application scopes, limitations, and insightful observations on future directions to serve as a valuable reference and stimulate further research in this emerging discipline.

Title: Learning to Compress Contexts for Efficient Knowledge-based Visual Question Answering

• Authors: Weixi Weng, Jieming Zhu, Hao Zhang, Xiaojun Meng, Rui Zhang, Chun Yuan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07331
• Pdf URL: https://arxiv.org/pdf/2409.07331
• Copy Paste: [[2409.07331]] Learning to Compress Contexts for Efficient Knowledge-based Visual Question Answering(https://arxiv.org/abs/2409.07331)
• Keywords: large language model
• Abstract: Multimodal Large Language Models (MLLMs) have demonstrated great zero-shot performance on visual question answering (VQA). However, when it comes to knowledge-based VQA (KB-VQA), MLLMs may lack human commonsense or specialized domain knowledge to answer such questions and require obtaining necessary information from external knowledge sources. Previous works like Retrival-Augmented VQA-v2 (RAVQA-v2) focus on utilizing as much input information, such as image-based textual descriptions and retrieved knowledge, as possible to improve performance, but they all overlook the issue that with the number of input tokens increasing, inference efficiency significantly decreases, which contradicts the demands of practical applications. To address this issue, we propose Retrieval-Augmented MLLM with Compressed Contexts (RACC). RACC learns to compress and aggregate retrieved contexts, from which it generates a compact modulation in the form of Key-Value (KV) cache. This modulation is then used to adapt the downstream frozen MLLM, thereby achieving effective and efficient inference. RACC achieves a state-of-the-art (SOTA) performance of 62.9% on OK-VQA. Moreover, it significantly reduces inference latency by 22.0%-59.7% compared to the prominent RAVQA-v2. Abundant experiments show RACC's broad applicability. It is compatible with various off-the-shelf MLLMs and can also handle different knowledge sources including textual and multimodal documents.

Title: Online Decision MetaMorphFormer: A Casual Transformer-Based Reinforcement Learning Framework of Universal Embodied Intelligence

• Authors: Luo Ji, Runji Lin
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.07341
• Pdf URL: https://arxiv.org/pdf/2409.07341
• Copy Paste: [[2409.07341]] Online Decision MetaMorphFormer: A Casual Transformer-Based Reinforcement Learning Framework of Universal Embodied Intelligence(https://arxiv.org/abs/2409.07341)
• Keywords: transformer
• Abstract: Interactive artificial intelligence in the motion control field is an interesting topic, especially when universal knowledge is adaptive to multiple tasks and universal environments. Despite there being increasing efforts in the field of Reinforcement Learning (RL) with the aid of transformers, most of them might be limited by the offline training pipeline, which prohibits exploration and generalization abilities. To address this limitation, we propose the framework of Online Decision MetaMorphFormer (ODM) which aims to achieve self-awareness, environment recognition, and action planning through a unified model architecture. Motivated by cognitive and behavioral psychology, an ODM agent is able to learn from others, recognize the world, and practice itself based on its own experience. ODM can also be applied to any arbitrary agent with a multi-joint body, located in different environments, and trained with different types of tasks using large-scale pre-trained datasets. Through the use of pre-trained datasets, ODM can quickly warm up and learn the necessary knowledge to perform the desired task, while the target environment continues to reinforce the universal policy. Extensive online experiments as well as few-shot and zero-shot environmental tests are used to verify ODM's performance and generalization ability. The results of our study contribute to the study of general artificial intelligence in embodied and cognitive fields. Code, results, and video examples can be found on the website \url{this https URL}.

Title: Federated Impression for Learning with Distributed Heterogeneous Data

• Authors: Sana Ayromlou, Atrin Arya, Armin Saadat, Purang Abolmaesumi, Xiaoxiao Li
• Subjects: cs.LG, cs.AI, cs.CV, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.07351
• Pdf URL: https://arxiv.org/pdf/2409.07351
• Copy Paste: [[2409.07351]] Federated Impression for Learning with Distributed Heterogeneous Data(https://arxiv.org/abs/2409.07351)
• Keywords: privacy, federate
• Abstract: Standard deep learning-based classification approaches may not always be practical in real-world clinical applications, as they require a centralized collection of all samples. Federated learning (FL) provides a paradigm that can learn from distributed datasets across clients without requiring them to share data, which can help mitigate privacy and data ownership issues. In FL, sub-optimal convergence caused by data heterogeneity is common among data from different health centers due to the variety in data collection protocols and patient demographics across centers. Through experimentation in this study, we show that data heterogeneity leads to the phenomenon of catastrophic forgetting during local training. We propose FedImpres which alleviates catastrophic forgetting by restoring synthetic data that represents the global information as federated impression. To achieve this, we distill the global model resulting from each communication round. Subsequently, we use the synthetic data alongside the local data to enhance the generalization of local training. Extensive experiments show that the proposed method achieves state-of-the-art performance on both the BloodMNIST and Retina datasets, which contain label imbalance and domain shift, with an improvement in classification accuracy of up to 20%.

Title: Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks

• Authors: Md Zarif Hossain, Ahmed Imteaj
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07353
• Pdf URL: https://arxiv.org/pdf/2409.07353
• Copy Paste: [[2409.07353]] Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks(https://arxiv.org/abs/2409.07353)
• Keywords: defense, attack, robust
• Abstract: Large Vision-Language Models (LVLMs), trained on multimodal big datasets, have significantly advanced AI by excelling in vision-language tasks. However, these models remain vulnerable to adversarial attacks, particularly jailbreak attacks, which bypass safety protocols and cause the model to generate misleading or harmful responses. This vulnerability stems from both the inherent susceptibilities of LLMs and the expanded attack surface introduced by the visual modality. We propose Sim-CLIP+, a novel defense mechanism that adversarially fine-tunes the CLIP vision encoder by leveraging a Siamese architecture. This approach maximizes cosine similarity between perturbed and clean samples, facilitating resilience against adversarial manipulations. Sim-CLIP+ offers a plug-and-play solution, allowing seamless integration into existing LVLM architectures as a robust vision encoder. Unlike previous defenses, our method requires no structural modifications to the LVLM and incurs minimal computational overhead. Sim-CLIP+ demonstrates effectiveness against both gradient-based adversarial attacks and various jailbreak techniques. We evaluate Sim-CLIP+ against three distinct jailbreak attack strategies and perform clean evaluations using standard downstream datasets, including COCO for image captioning and OKVQA for visual question answering. Extensive experiments demonstrate that Sim-CLIP+ maintains high clean accuracy while substantially improving robustness against both gradient-based adversarial attacks and jailbreak techniques. Our code and robust vision encoders are available at this https URL.

Title: Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluation

• Authors: SeongYeub Chu, JongWoo Kim, MunYong Yi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07355
• Pdf URL: https://arxiv.org/pdf/2409.07355
• Copy Paste: [[2409.07355]] Think Together and Work Better: Combining Humans' and LLMs' Think-Aloud Outcomes for Effective Text Evaluation(https://arxiv.org/abs/2409.07355)
• Keywords: large language model
• Abstract: This study introduces \textbf{InteractEval}, a framework that integrates human expertise and Large Language Models (LLMs) using the Think-Aloud (TA) method to generate attributes for checklist-based text evaluation. By combining human flexibility and reasoning with LLM consistency, InteractEval outperforms traditional non-LLM-based and LLM-based baselines across four distinct dimensions, consisting of Coherence, Fluency, Consistency, and Relevance. The experiment also investigates the effectiveness of the TA method, showing that it promotes divergent thinking in both humans and LLMs, leading to the generation of a wider range of relevant attributes and enhance text evaluation performance. Comparative analysis reveals that humans excel at identifying attributes related to internal quality (Coherence and Fluency), but LLMs perform better at those attributes related to external alignment (Consistency and Relevance). Consequently, leveraging both humans and LLMs together produces the best evaluation outcomes. In other words, this study emphasizes the necessity of effectively combining humans and LLMs in an automated checklist-based text evaluation framework. The code is available at \textbf{\url{this https URL}}.

Title: Demo: SGCode: A Flexible Prompt-Optimizing System for Secure Generation of Code

• Authors: Khiem Ton, Nhi Nguyen, Mahmoud Nazzal, Abdallah Khreishah, Cristian Borcea, NhatHai Phan, Ruoming Jin, Issa Khalil, Yelong Shen
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07368
• Pdf URL: https://arxiv.org/pdf/2409.07368
• Copy Paste: [[2409.07368]] Demo: SGCode: A Flexible Prompt-Optimizing System for Secure Generation of Code(https://arxiv.org/abs/2409.07368)
• Keywords: secure, security, generative, large language model
• Abstract: This paper introduces SGCode, a flexible prompt-optimizing system to generate secure code with large language models (LLMs). SGCode integrates recent prompt-optimization approaches with LLMs in a unified system accessible through front-end and back-end APIs, enabling users to 1) generate secure code, which is free of vulnerabilities, 2) review and share security analysis, and 3) easily switch from one prompt optimization approach to another, while providing insights on model and system performance. We populated SGCode on an AWS server with PromSec, an approach that optimizes prompts by combining an LLM and security tools with a lightweight generative adversarial graph neural network to detect and fix security vulnerabilities in the generated code. Extensive experiments show that SGCode is practical as a public tool to gain insights into the trade-offs between model utility, secure code generation, and system cost. SGCode has only a marginal cost compared with prompting LLMs. SGCode is available at: this http URL.

Title: D-CAPTCHA++: A Study of Resilience of Deepfake CAPTCHA under Transferable Imperceptible Adversarial Attack

• Authors: Hong-Hanh Nguyen-Le, Van-Tuan Tran, Dinh-Thuc Nguyen, Nhien-An Le-Khac
• Subjects: cs.CR, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.07390
• Pdf URL: https://arxiv.org/pdf/2409.07390
• Copy Paste: [[2409.07390]] D-CAPTCHA++: A Study of Resilience of Deepfake CAPTCHA under Transferable Imperceptible Adversarial Attack(https://arxiv.org/abs/2409.07390)
• Keywords: security, attack, robust, generative
• Abstract: The advancements in generative AI have enabled the improvement of audio synthesis models, including text-to-speech and voice conversion. This raises concerns about its potential misuse in social manipulation and political interference, as synthetic speech has become indistinguishable from natural human speech. Several speech-generation programs are utilized for malicious purposes, especially impersonating individuals through phone calls. Therefore, detecting fake audio is crucial to maintain social security and safeguard the integrity of information. Recent research has proposed a D-CAPTCHA system based on the challenge-response protocol to differentiate fake phone calls from real ones. In this work, we study the resilience of this system and introduce a more robust version, D-CAPTCHA++, to defend against fake calls. Specifically, we first expose the vulnerability of the D-CAPTCHA system under transferable imperceptible adversarial attack. Secondly, we mitigate such vulnerability by improving the robustness of the system by using adversarial training in D-CAPTCHA deepfake detectors and task classifiers.

Title: A Scalable Algorithm for Active Learning

• Authors: Youguang Chen, Zheyu Wen, George Biros
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.07392
• Pdf URL: https://arxiv.org/pdf/2409.07392
• Copy Paste: [[2409.07392]] A Scalable Algorithm for Active Learning(https://arxiv.org/abs/2409.07392)
• Keywords: robust
• Abstract: FIRAL is a recently proposed deterministic active learning algorithm for multiclass classification using logistic regression. It was shown to outperform the state-of-the-art in terms of accuracy and robustness and comes with theoretical performance guarantees. However, its scalability suffers when dealing with datasets featuring a large number of points $n$, dimensions $d$, and classes $c$, due to its $\mathcal{O}(c^2d^2+nc^2d)$ storage and $\mathcal{O}(c^3(nd^2 + bd^3 + bn))$ computational complexity where $b$ is the number of points to select in active learning. To address these challenges, we propose an approximate algorithm with storage requirements reduced to $\mathcal{O}(n(d+c) + cd^2)$ and a computational complexity of $\mathcal{O}(bncd^2)$. Additionally, we present a parallel implementation on GPUs. We demonstrate the accuracy and scalability of our approach using MNIST, CIFAR-10, Caltech101, and ImageNet. The accuracy tests reveal no deterioration in accuracy compared to FIRAL. We report strong and weak scaling tests on up to 12 GPUs, for three million point synthetic dataset.

Title: AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge

• Authors: Han Wang, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07394
• Pdf URL: https://arxiv.org/pdf/2409.07394
• Copy Paste: [[2409.07394]] AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge(https://arxiv.org/abs/2409.07394)
• Keywords: large language model
• Abstract: Knowledge conflict arises from discrepancies between information in the context of a large language model (LLM) and the knowledge stored in its parameters. This can hurt performance when using standard decoding techniques, which tend to ignore the context. Existing test-time contrastive methods seek to address this by comparing the LLM's output distribution with and without the context and adjust the model according to the contrast between them. However, we find that these methods frequently misjudge the degree of conflict and struggle to handle instances that vary in their amount of conflict, with static methods over-adjusting when conflict is absent. We propose a fine-grained, instance-level approach called AdaCAD, which dynamically infers the weight of adjustment based on the degree of conflict, as measured by the Jensen-Shannon divergence between distributions representing contextual and parametric knowledge. Our experiments across four models on six diverse question-answering (QA) datasets and three summarization tasks demonstrate that our training-free adaptive method consistently outperforms other decoding methods on QA, with average accuracy gains of 14.21% (absolute) over a static contrastive baseline, and improves the factuality of summaries by 5.59 (AlignScore). Furthermore, our analysis shows that while decoding with contrastive baselines hurts performance when conflict is absent, AdaCAD mitigates these losses, making it more applicable to real-world datasets in which some examples have conflict and others do not.

Title: Revisiting Static Feature-Based Android Malware Detection

• Authors: Md Tanvirul Alam, Dipkamal Bhusal, Nidhi Rastogi
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07397
• Pdf URL: https://arxiv.org/pdf/2409.07397
• Copy Paste: [[2409.07397]] Revisiting Static Feature-Based Android Malware Detection(https://arxiv.org/abs/2409.07397)
• Keywords: security, fair
• Abstract: The increasing reliance on machine learning (ML) in computer security, particularly for malware classification, has driven significant advancements. However, the replicability and reproducibility of these results are often overlooked, leading to challenges in verifying research findings. This paper highlights critical pitfalls that undermine the validity of ML research in Android malware detection, focusing on dataset and methodological issues. We comprehensively analyze Android malware detection using two datasets and assess offline and continual learning settings with six widely used ML models. Our study reveals that when properly tuned, simpler baseline methods can often outperform more complex models. To address reproducibility challenges, we propose solutions for improving datasets and methodological practices, enabling fairer model comparisons. Additionally, we open-source our code to facilitate malware analysis, making it extensible for new models and datasets. Our paper aims to support future research in Android malware detection and other security domains, enhancing the reliability and reproducibility of published results.

Title: CLNX: Bridging Code and Natural Language for C/C++ Vulnerability-Contributing Commits Identification

• Authors: Zeqing Qin, Yiwei Wu, Lansheng Han
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07407
• Pdf URL: https://arxiv.org/pdf/2409.07407
• Copy Paste: [[2409.07407]] CLNX: Bridging Code and Natural Language for C/C++ Vulnerability-Contributing Commits Identification(https://arxiv.org/abs/2409.07407)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have shown great promise in vulnerability identification. As C/C++ comprises half of the Open-Source Software (OSS) vulnerabilities over the past decade and updates in OSS mainly occur through commits, enhancing LLMs' ability to identify C/C++ Vulnerability-Contributing Commits (VCCs) is essential. However, current studies primarily focus on further pre-training LLMs on massive code datasets, which is resource-intensive and poses efficiency challenges. In this paper, we enhance the ability of BERT-based LLMs to identify C/C++ VCCs in a lightweight manner. We propose CodeLinguaNexus (CLNX) as a bridge facilitating communication between C/C++ programs and LLMs. Based on commits, CLNX efficiently converts the source code into a more natural representation while preserving key details. Specifically, CLNX first applies structure-level naturalization to decompose complex programs, followed by token-level naturalization to interpret complex symbols. We evaluate CLNX on public datasets of 25,872 C/C++ functions with their commits. The results show that CLNX significantly enhances the performance of LLMs on identifying C/C++ VCCs. Moreover, CLNX-equipped CodeBERT achieves new state-of-the-art and identifies 38 OSS vulnerabilities in the real world.

Title: SoK: Security and Privacy Risks of Medical AI

• Authors: Yuanhaur Chang, Han Liu, Evin Jaff, Chenyang Lu, Ning Zhang
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07415
• Pdf URL: https://arxiv.org/pdf/2409.07415
• Copy Paste: [[2409.07415]] SoK: Security and Privacy Risks of Medical AI(https://arxiv.org/abs/2409.07415)
• Keywords: security, privacy, attack
• Abstract: The integration of technology and healthcare has ushered in a new era where software systems, powered by artificial intelligence and machine learning, have become essential components of medical products and services. While these advancements hold great promise for enhancing patient care and healthcare delivery efficiency, they also expose sensitive medical data and system integrity to potential cyberattacks. This paper explores the security and privacy threats posed by AI/ML applications in healthcare. Through a thorough examination of existing research across a range of medical domains, we have identified significant gaps in understanding the adversarial attacks targeting medical AI systems. By outlining specific adversarial threat models for medical settings and identifying vulnerable application domains, we lay the groundwork for future research that investigates the security and resilience of AI-driven medical systems. Through our analysis of different threat models and feasibility studies on adversarial attacks in different medical domains, we provide compelling insights into the pressing need for cybersecurity research in the rapidly evolving field of AI healthcare technology.

Title: XDC Staking and Tokenomics -- Improvement Proposal: Enhancing Sustainability and Decentralization on the Eve of XDC 2.0

• Authors: Van Khanh Nguyen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07420
• Pdf URL: https://arxiv.org/pdf/2409.07420
• Copy Paste: [[2409.07420]] XDC Staking and Tokenomics -- Improvement Proposal: Enhancing Sustainability and Decentralization on the Eve of XDC 2.0(https://arxiv.org/abs/2409.07420)
• Keywords: robust
• Abstract: As the XDC network celebrates five years of stable mainnet operation and prepares for the highly anticipated launch of XDC 2.0, this research proposes a comprehensive improvement plan for the network's staking and tokenomics mechanisms. Our analysis reveals opportunities to optimize the current model, ensuring a more sustainable, decentralized, and resilient ecosystem. We introduce novel concepts, including validator NFTs, decentralized governance, and utility-based tokenomics, to increase validator node liquidity and promote staking participation. Our proposal aims to establish a robust foundation for XDC 2.0, fostering a thriving ecosystem that rewards validators, stakeholders, and users alike. By addressing the intricacies of staking and tokenomics, this research paves the way for XDC to solidify its position as a leading decentralized network, poised for long-term success and growth.

Title: Enhancing adversarial robustness in Natural Language Inference using explanations

• Authors: Alexandros Koulakos, Maria Lymperaiou, Giorgos Filandrianos, Giorgos Stamou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07423
• Pdf URL: https://arxiv.org/pdf/2409.07423
• Copy Paste: [[2409.07423]] Enhancing adversarial robustness in Natural Language Inference using explanations(https://arxiv.org/abs/2409.07423)
• Keywords: attack, robust, transformer
• Abstract: The surge of state-of-the-art Transformer-based models has undoubtedly pushed the limits of NLP model performance, excelling in a variety of tasks. We cast the spotlight on the underexplored task of Natural Language Inference (NLI), since models trained on popular well-suited datasets are susceptible to adversarial attacks, allowing subtle input interventions to mislead the model. In this work, we validate the usage of natural language explanation as a model-agnostic defence strategy through extensive experimentation: only by fine-tuning a classifier on the explanation rather than premise-hypothesis inputs, robustness under various adversarial attacks is achieved in comparison to explanation-free baselines. Moreover, since there is no standard strategy of testing the semantic validity of the generated explanations, we research the correlation of widely used language generation metrics with human perception, in order for them to serve as a proxy towards robust NLI models. Our approach is resource-efficient and reproducible without significant computational limitations.

Title: Towards Fairer Health Recommendations: finding informative unbiased samples via Word Sense Disambiguation

• Authors: Gavin Butts, Pegah Emdad, Jethro Lee, Shannon Song, Chiman Salavati, Willmar Sosa Diaz, Shiri Dori-Hacohen, Fabricio Murai
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07424
• Pdf URL: https://arxiv.org/pdf/2409.07424
• Copy Paste: [[2409.07424]] Towards Fairer Health Recommendations: finding informative unbiased samples via Word Sense Disambiguation(https://arxiv.org/abs/2409.07424)
• Keywords: fair
• Abstract: There have been growing concerns around high-stake applications that rely on models trained with biased data, which consequently produce biased predictions, often harming the most vulnerable. In particular, biased medical data could cause health-related applications and recommender systems to create outputs that jeopardize patient care and widen disparities in health outcomes. A recent framework titled Fairness via AI posits that, instead of attempting to correct model biases, researchers must focus on their root causes by using AI to debias data. Inspired by this framework, we tackle bias detection in medical curricula using NLP models, including LLMs, and evaluate them on a gold standard dataset containing 4,105 excerpts annotated by medical experts for bias from a large corpus. We build on previous work by coauthors which augments the set of negative samples with non-annotated text containing social identifier terms. However, some of these terms, especially those related to race and ethnicity, can carry different meanings (e.g., "white matter of spinal cord"). To address this issue, we propose the use of Word Sense Disambiguation models to refine dataset quality by removing irrelevant sentences. We then evaluate fine-tuned variations of BERT models as well as GPT models with zero- and few-shot prompting. We found LLMs, considered SOTA on many NLP tasks, unsuitable for bias detection, while fine-tuned BERT models generally perform well across all evaluated metrics.

Title: Deep Neural Network-Based Sign Language Recognition: A Comprehensive Approach Using Transfer Learning with Explainability

• Authors: A. E. M Ridwan, Mushfiqul Islam Chowdhury, Mekhala Mariam Mary, Md Tahmid Chowdhury Abir
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07426
• Pdf URL: https://arxiv.org/pdf/2409.07426
• Copy Paste: [[2409.07426]] Deep Neural Network-Based Sign Language Recognition: A Comprehensive Approach Using Transfer Learning with Explainability(https://arxiv.org/abs/2409.07426)
• Keywords: robust, explainability
• Abstract: To promote inclusion and ensuring effective communication for those who rely on sign language as their main form of communication, sign language recognition (SLR) is crucial. Sign language recognition (SLR) seamlessly incorporates with diverse technology, enhancing accessibility for the deaf community by facilitating their use of digital platforms, video calls, and communication devices. To effectively solve this problem, we suggest a novel solution that uses a deep neural network to fully automate sign language recognition. This methodology integrates sophisticated preprocessing methodologies to optimise the overall performance. The architectures resnet, inception, xception, and vgg are utilised to selectively categorise images of sign language. We prepared a DNN architecture and merged it with the pre-processing architectures. In the post-processing phase, we utilised the SHAP deep explainer, which is based on cooperative game theory, to quantify the influence of specific features on the output of a machine learning model. Bhutanese-Sign-Language (BSL) dataset was used for training and testing the suggested technique. While training on Bhutanese-Sign-Language (BSL) dataset, overall ResNet50 with the DNN model performed better accuracy which is 98.90%. Our model's ability to provide informational clarity was assessed using the SHAP (SHapley Additive exPlanations) method. In part to its considerable robustness and reliability, the proposed methodological approach can be used to develop a fully automated system for sign language recognition.

Title: Agent Workflow Memory

• Authors: Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, Graham Neubig
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07429
• Pdf URL: https://arxiv.org/pdf/2409.07429
• Copy Paste: [[2409.07429]] Agent Workflow Memory(https://arxiv.org/abs/2409.07429)
• Keywords: robust
• Abstract: Despite the potential of language model-based agents to solve real-world tasks such as web navigation, current methods still struggle with long-horizon tasks with complex action trajectories. In contrast, humans can flexibly solve complex tasks by learning reusable task workflows from past experiences and using them to guide future actions. To build agents that can similarly benefit from this process, we introduce Agent Workflow Memory (AWM), a method for inducing commonly reused routines, i.e., workflows, and selectively providing workflows to the agent to guide subsequent generations. AWM flexibly applies to both offline and online scenarios, where agents induce workflows from training examples beforehand or from test queries on the fly. We experiment on two major web navigation benchmarks -- Mind2Web and WebArena -- that collectively cover 1000+ tasks from 200+ domains across travel, shopping, and social media, among others. AWM substantially improves the baseline results by 24.6% and 51.1% relative success rate on Mind2Web and WebArena while reducing the number of steps taken to solve WebArena tasks successfully. Furthermore, online AWM robustly generalizes in cross-task, website, and domain evaluations, surpassing baselines from 8.9 to 14.0 absolute points as train-test task distribution gaps widen.

Title: StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videos

• Authors: Sijie Zhao, Wenbo Hu, Xiaodong Cun, Yong Zhang, Xiaoyu Li, Zhe Kong, Xiangjun Gao, Muyao Niu, Ying Shan
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.07447
• Pdf URL: https://arxiv.org/pdf/2409.07447
• Copy Paste: [[2409.07447]] StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videos(https://arxiv.org/abs/2409.07447)
• Keywords: diffusion
• Abstract: This paper presents a novel framework for converting 2D videos to immersive stereoscopic 3D, addressing the growing demand for 3D content in immersive experience. Leveraging foundation models as priors, our approach overcomes the limitations of traditional methods and boosts the performance to ensure the high-fidelity generation required by the display devices. The proposed system consists of two main steps: depth-based video splatting for warping and extracting occlusion mask, and stereo video inpainting. We utilize pre-trained stable video diffusion as the backbone and introduce a fine-tuning protocol for the stereo video inpainting task. To handle input video with varying lengths and resolutions, we explore auto-regressive strategies and tiled processing. Finally, a sophisticated data processing pipeline has been developed to reconstruct a large-scale and high-quality dataset to support our training. Our framework demonstrates significant improvements in 2D-to-3D video conversion, offering a practical solution for creating immersive content for 3D devices like Apple Vision Pro and 3D displays. In summary, this work contributes to the field by presenting an effective method for generating high-quality stereoscopic videos from monocular input, potentially transforming how we experience digital media.

Title: Introducing Perturb-ability Score (PS) to Enhance Robustness Against Evasion Adversarial Attacks on ML-NIDS

• Authors: Mohamed elShehaby, Ashraf Matrawy
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07448
• Pdf URL: https://arxiv.org/pdf/2409.07448
• Copy Paste: [[2409.07448]] Introducing Perturb-ability Score (PS) to Enhance Robustness Against Evasion Adversarial Attacks on ML-NIDS(https://arxiv.org/abs/2409.07448)
• Keywords: attack, robust
• Abstract: This paper proposes a novel Perturb-ability Score (PS) that can be used to identify Network Intrusion Detection Systems (NIDS) features that can be easily manipulated by attackers in the problem-space. We demonstrate that using PS to select only non-perturb-able features for ML-based NIDS maintains detection performance while enhancing robustness against adversarial attacks.

Title: FreeEnhance: Tuning-Free Image Enhancement via Content-Consistent Noising-and-Denoising Process

• Authors: Yang Luo, Yiheng Zhang, Zhaofan Qiu, Ting Yao, Zhineng Chen, Yu-Gang Jiang, Tao Mei
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.07451
• Pdf URL: https://arxiv.org/pdf/2409.07451
• Copy Paste: [[2409.07451]] FreeEnhance: Tuning-Free Image Enhancement via Content-Consistent Noising-and-Denoising Process(https://arxiv.org/abs/2409.07451)
• Keywords: diffusion
• Abstract: The emergence of text-to-image generation models has led to the recognition that image enhancement, performed as post-processing, would significantly improve the visual quality of the generated images. Exploring diffusion models to enhance the generated images nevertheless is not trivial and necessitates to delicately enrich plentiful details while preserving the visual appearance of key content in the original image. In this paper, we propose a novel framework, namely FreeEnhance, for content-consistent image enhancement using the off-the-shelf image diffusion models. Technically, FreeEnhance is a two-stage process that firstly adds random noise to the input image and then capitalizes on a pre-trained image diffusion model (i.e., Latent Diffusion Models) to denoise and enhance the image details. In the noising stage, FreeEnhance is devised to add lighter noise to the region with higher frequency to preserve the high-frequent patterns (e.g., edge, corner) in the original image. In the denoising stage, we present three target properties as constraints to regularize the predicted noise, enhancing images with high acutance and high visual quality. Extensive experiments conducted on the HPDv2 dataset demonstrate that our FreeEnhance outperforms the state-of-the-art image enhancement models in terms of quantitative metrics and human preference. More remarkably, FreeEnhance also shows higher human preference compared to the commercial image enhancement solution of Magnific AI.

Title: Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models

• Authors: Haibo Yang, Yang Chen, Yingwei Pan, Ting Yao, Zhineng Chen, Chong-Wah Ngo, Tao Mei
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.07452
• Pdf URL: https://arxiv.org/pdf/2409.07452
• Copy Paste: [[2409.07452]] Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models(https://arxiv.org/abs/2409.07452)
• Keywords: diffusion
• Abstract: Despite having tremendous progress in image-to-3D generation, existing methods still struggle to produce multi-view consistent images with high-resolution textures in detail, especially in the paradigm of 2D diffusion that lacks 3D awareness. In this work, we present High-resolution Image-to-3D model (Hi3D), a new video diffusion based paradigm that redefines a single image to multi-view images as 3D-aware sequential image generation (i.e., orbital video generation). This methodology delves into the underlying temporal consistency knowledge in video diffusion model that generalizes well to geometry consistency across multiple views in 3D generation. Technically, Hi3D first empowers the pre-trained video diffusion model with 3D-aware prior (camera pose condition), yielding multi-view images with low-resolution texture details. A 3D-aware video-to-video refiner is learnt to further scale up the multi-view images with high-resolution texture details. Such high-resolution multi-view images are further augmented with novel views through 3D Gaussian Splatting, which are finally leveraged to obtain high-fidelity meshes via 3D reconstruction. Extensive experiments on both novel view synthesis and single view reconstruction demonstrate that our Hi3D manages to produce superior multi-view consistency images with highly-detailed textures. Source code and data are available at \url{this https URL}.

Title: DreamMesh: Jointly Manipulating and Texturing Triangle Meshes for Text-to-3D Generation

• Authors: Haibo Yang, Yang Chen, Yingwei Pan, Ting Yao, Zhineng Chen, Zuxuan Wu, Yu-Gang Jiang, Tao Mei
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.07454
• Pdf URL: https://arxiv.org/pdf/2409.07454
• Copy Paste: [[2409.07454]] DreamMesh: Jointly Manipulating and Texturing Triangle Meshes for Text-to-3D Generation(https://arxiv.org/abs/2409.07454)
• Keywords: diffusion
• Abstract: Learning radiance fields (NeRF) with powerful 2D diffusion models has garnered popularity for text-to-3D generation. Nevertheless, the implicit 3D representations of NeRF lack explicit modeling of meshes and textures over surfaces, and such surface-undefined way may suffer from the issues, e.g., noisy surfaces with ambiguous texture details or cross-view inconsistency. To alleviate this, we present DreamMesh, a novel text-to-3D architecture that pivots on well-defined surfaces (triangle meshes) to generate high-fidelity explicit 3D model. Technically, DreamMesh capitalizes on a distinctive coarse-to-fine scheme. In the coarse stage, the mesh is first deformed by text-guided Jacobians and then DreamMesh textures the mesh with an interlaced use of 2D diffusion models in a tuning free manner from multiple viewpoints. In the fine stage, DreamMesh jointly manipulates the mesh and refines the texture map, leading to high-quality triangle meshes with high-fidelity textured materials. Extensive experiments demonstrate that DreamMesh significantly outperforms state-of-the-art text-to-3D methods in faithfully generating 3D content with richer textual details and enhanced geometry. Our project page is available at this https URL.