2024-09-10

Title: ARSecure: A Novel End-to-End Encryption Messaging System Using Augmented Reality

• Authors: Hamish Alsop, Douglas Alsop, Joseph Solomon, Liam Aumento, Mark Butters, Cameron Millar, Yagmur Yigit, Leandros Maglaras, Naghmeh Moradpoor
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.04457
• Pdf URL: https://arxiv.org/pdf/2409.04457
• Copy Paste: [[2409.04457]] ARSecure: A Novel End-to-End Encryption Messaging System Using Augmented Reality(https://arxiv.org/abs/2409.04457)
• Keywords: secure
• Abstract: End-to-End Encryption (E2EE) ensures that only the intended recipient(s) can read messages. Popular instant messaging (IM) applications such as Signal, WhatsApp, Apple's iMessage, and Telegram claim to offer E2EE. However, client-side scanning (CSS) undermines these claims by scanning all messages, including text, images, audio, and video files, on both sending and receiving ends. Industry and government parties support CSS to combat harmful content such as child pornography, terrorism, and other illegal activities. In this paper, we introduce ARSecure, a novel end-to-end encryption messaging solution utilizing augmented reality glasses. ARSecure allows users to encrypt and decrypt their messages before they reach their phone devices, effectively countering the CSS technology in E2EE systems.

Title: WET: Overcoming Paraphrasing Vulnerabilities in Embeddings-as-a-Service with Linear Transformation Watermarks

• Authors: Anudeex Shetty, Qiongkai Xu, Jey Han Lau
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04459
• Pdf URL: https://arxiv.org/pdf/2409.04459
• Copy Paste: [[2409.04459]] WET: Overcoming Paraphrasing Vulnerabilities in Embeddings-as-a-Service with Linear Transformation Watermarks(https://arxiv.org/abs/2409.04459)
• Keywords: protect, attack, robust, watermark, large language model
• Abstract: Embeddings-as-a-Service (EaaS) is a service offered by large language model (LLM) developers to supply embeddings generated by LLMs. Previous research suggests that EaaS is prone to imitation attacks -- attacks that clone the underlying EaaS model by training another model on the queried embeddings. As a result, EaaS watermarks are introduced to protect the intellectual property of EaaS providers. In this paper, we first show that existing EaaS watermarks can be removed by paraphrasing when attackers clone the model. Subsequently, we propose a novel watermarking technique that involves linearly transforming the embeddings, and show that it is empirically and theoretically robust against paraphrasing.

Title: Leveraging Large Language Models for Solving Rare MIP Challenges

• Authors: Teng Wang, Wing-Yin Yu, Ruifeng She, Wenhan Yang, Taijie Chen, Jianping Zhang
• Subjects: cs.CL, cs.AI, cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2409.04464
• Pdf URL: https://arxiv.org/pdf/2409.04464
• Copy Paste: [[2409.04464]] Leveraging Large Language Models for Solving Rare MIP Challenges(https://arxiv.org/abs/2409.04464)
• Keywords: large language model
• Abstract: Mixed Integer Programming (MIP) has been extensively applied in areas requiring mathematical solvers to address complex instances within tight time constraints. However, as the problem scale increases, the complexity of model formulation and finding feasible solutions escalates significantly. In contrast, the model-building cost for end-to-end models, such as large language models (LLMs), remains largely unaffected by problem scale due to their pattern recognition capabilities. While LLMs, like GPT-4, without fine-tuning, can handle some traditional medium-scale MIP problems, they struggle with uncommon or highly specialized MIP scenarios. Fine-tuning LLMs can yield some feasible solutions for medium-scale MIP instances, but these models typically fail to explore diverse solutions when constrained by a low and constant temperature, limiting their performance. In this paper, we propose and evaluate a recursively dynamic temperature method integrated with a chain-of-thought approach. Our findings show that starting with a high temperature and gradually lowering it leads to better feasible solutions compared to other dynamic temperature strategies. Additionally, by comparing results generated by the LLM with those from Gurobi, we demonstrate that the LLM can produce solutions that complement traditional solvers by accelerating the pruning process and improving overall efficiency.

Title: Towards Understanding and Applying Security Assurance Cases for Automotive Systems

• Authors: Mazen Mohamad
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.04474
• Pdf URL: https://arxiv.org/pdf/2409.04474
• Copy Paste: [[2409.04474]] Towards Understanding and Applying Security Assurance Cases for Automotive Systems(https://arxiv.org/abs/2409.04474)
• Keywords: security
• Abstract: Security Assurance Cases (SAC) are structured bodies of arguments and evidence used to reason about security properties of a certain artefact. SAC are gaining focus in the automotive domain as the need for security assurance is growing due to software becoming a main part of vehicles. Market demands for new services and products in the domain require connectivity, and hence, raise security concerns. Regulators and standardisation bodies started recently to require a structured for security assurance of products in the automotive domain, and automotive companies started, hence, to study ways to create and maintain these cases, as well as adopting them in their current way of working. In order to facilitate the adoption of SAC in the automotive domain, we created CASCADE, an approach for creating SAC which have integrated quality assurance and are compliant with the requirements of ISO/SAE-21434, the upcoming cybersecurity standard for automotive systems. CASCADE was created by conducting design science research study in two iterative cycles. The design decisions of CASCADE are based on insights from a qualitative research study which includes a workshop, a survey, and one-to-one interviews, done in collaboration with our industrial partners about the needs and drivers of work in SAC in industry, and a systematic literature review in which we identified gaps between the industrial needs and the state of the art. The evaluation of CASCADE was done with help of security experts from a large automotive OEM. It showed that CASCADE is suitable for integration in industrial product development processes. Additionally, our results show that the elements of CASCADE align well with respect to the way of working at the company, and has the potential to scale to cover the requirements and needs of the company with its large organization and complex products

Title: Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small

• Authors: Maheep Chaudhary, Atticus Geiger
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2409.04478
• Pdf URL: https://arxiv.org/pdf/2409.04478
• Copy Paste: [[2409.04478]] Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small(https://arxiv.org/abs/2409.04478)
• Keywords: interpretability
• Abstract: A popular new method in mechanistic interpretability is to train high-dimensional sparse autoencoders (SAEs) on neuron activations and use SAE features as the atomic units of analysis. However, the body of evidence on whether SAE feature spaces are useful for causal analysis is underdeveloped. In this work, we use the RAVEL benchmark to evaluate whether SAEs trained on hidden representations of GPT-2 small have sets of features that separately mediate knowledge of which country a city is in and which continent it is in. We evaluate four open-source SAEs for GPT-2 small against each other, with neurons serving as a baseline, and linear features learned via distributed alignment search (DAS) serving as a skyline. For each, we learn a binary mask to select features that will be patched to change the country of a city without changing the continent, or vice versa. Our results show that SAEs struggle to reach the neuron baseline, and none come close to the DAS skyline. We release code here: this https URL

Title: The HitchHiker's Guide to High-Assurance System Observability Protection with Efficient Permission Switches

• Authors: Chuqi Zhang, Jun Zeng, Yiming Zhang, Adil Ahmad, Fengwei Zhang, Hai Jin, Zhenkai Liang
• Subjects: cs.CR, cs.OS
• Abstract URL: https://arxiv.org/abs/2409.04484
• Pdf URL: https://arxiv.org/pdf/2409.04484
• Copy Paste: [[2409.04484]] The HitchHiker's Guide to High-Assurance System Observability Protection with Efficient Permission Switches(https://arxiv.org/abs/2409.04484)
• Keywords: secure, security, protect, attack
• Abstract: Protecting system observability records (logs) from compromised OSs has gained significant traction in recent times, with several note-worthy approaches proposed. Unfortunately, none of the proposed approaches achieve high performance with tiny log protection delays. They also leverage risky environments for protection (\eg many use general-purpose hypervisors or TrustZone, which have large TCB and attack surfaces). HitchHiker is an attempt to rectify this problem. The system is designed to ensure (a) in-memory protection of batched logs within a short and configurable real-time deadline by efficient hardware permission switching, and (b) an end-to-end high-assurance environment built upon hardware protection primitives with debloating strategies for secure log protection, persistence, and management. Security evaluations and validations show that HitchHiker reduces log protection delay by 93.3--99.3% compared to the state-of-the-art, while reducing TCB by 9.4--26.9X. Performance evaluations show HitchHiker incurs a geometric mean of less than 6% overhead on diverse real-world programs, improving on the state-of-the-art approach by 61.9--77.5%.

Title: Comment on Revisiting Neural Program Smoothing for Fuzzing

• Authors: Dongdong She, Kexin Pei, Junfeng Yang, Baishakhi Ray, Suman Jana
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.04504
• Pdf URL: https://arxiv.org/pdf/2409.04504
• Copy Paste: [[2409.04504]] Comment on Revisiting Neural Program Smoothing for Fuzzing(https://arxiv.org/abs/2409.04504)
• Keywords: fair
• Abstract: MLFuzz, a work accepted at ACM FSE 2023, revisits the performance of a machine learning-based fuzzer, NEUZZ. We demonstrate that its main conclusion is entirely wrong due to several fatal bugs in the implementation and wrong evaluation setups, including an initialization bug in persistent mode, a program crash, an error in training dataset collection, and a mistake in fuzzing result collection. Additionally, MLFuzz uses noisy training datasets without sufficient data cleaning and preprocessing, which contributes to a drastic performance drop in NEUZZ. We address these issues and provide a corrected implementation and evaluation setup, showing that NEUZZ consistently performs well over AFL on the FuzzBench dataset. Finally, we reflect on the evaluation methods used in MLFuzz and offer practical advice on fair and scientific fuzzing evaluations.

Title: Operator Learning with Gaussian Processes

• Authors: Carlos Mora, Amin Yousefpour, Shirin Hosseinmardi, Houman Owhadi, Ramin Bostanabad
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.04538
• Pdf URL: https://arxiv.org/pdf/2409.04538
• Copy Paste: [[2409.04538]] Operator Learning with Gaussian Processes(https://arxiv.org/abs/2409.04538)
• Keywords: robust, interpretability
• Abstract: Operator learning focuses on approximating mappings $\mathcal{G}^\dagger:\mathcal{U} \rightarrow\mathcal{V}$ between infinite-dimensional spaces of functions, such as $u: \Omega_u\rightarrow\mathbb{R}$ and $v: \Omega_v\rightarrow\mathbb{R}$. This makes it particularly suitable for solving parametric nonlinear partial differential equations (PDEs). While most machine learning methods for operator learning rely on variants of deep neural networks (NNs), recent studies have shown that Gaussian Processes (GPs) are also competitive while offering interpretability and theoretical guarantees. In this paper, we introduce a hybrid GP/NN-based framework for operator learning that leverages the strengths of both methods. Instead of approximating the function-valued operator $\mathcal{G}^\dagger$, we use a GP to approximate its associated real-valued bilinear form $\widetilde{\mathcal{G}}^\dagger: \mathcal{U}\times\mathcal{V}^*\rightarrow\mathbb{R}.$ This bilinear form is defined by $\widetilde{\mathcal{G}}^\dagger(u,\varphi) := [\varphi,\mathcal{G}^\dagger(u)],$ which allows us to recover the operator $\mathcal{G}^\dagger$ through $\mathcal{G}^\dagger(u)(y)=\widetilde{\mathcal{G}}^\dagger(u,\delta_y).$ The GP mean function can be zero or parameterized by a neural operator and for each setting we develop a robust training mechanism based on maximum likelihood estimation (MLE) that can optionally leverage the physics involved. Numerical benchmarks show that (1) it improves the performance of a base neural operator by using it as the mean function of a GP, and (2) it enables zero-shot data-driven models for accurate predictions without prior training. Our framework also handles multi-output operators where $\mathcal{G}^\dagger:\mathcal{U} \rightarrow\prod_{s=1}^S\mathcal{V}^s$, and benefits from computational speed-ups via product kernel structures and Kronecker product matrix representations.

Title: How Does Code Pretraining Affect Language Model Task Performance?

• Authors: Jackson Petty, Sjoerd van Steenkiste, Tal Linzen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04556
• Pdf URL: https://arxiv.org/pdf/2409.04556
• Copy Paste: [[2409.04556]] How Does Code Pretraining Affect Language Model Task Performance?(https://arxiv.org/abs/2409.04556)
• Keywords: large language model
• Abstract: Large language models are increasingly trained on corpora containing both natural language and non-linguistic data like source code. Aside from aiding programming-related tasks, anecdotal evidence suggests that including code in pretraining corpora may improve performance on other, unrelated tasks, yet to date no work has been able to establish a causal connection by controlling between language and code data. Here we do just this. We pretrain language models on datasets which interleave natural language and code in two different settings: additive, in which the total volume of data seen during pretraining is held constant; and competitive, in which the volume of language data is held constant. We study how the pretraining mixture affects performance on (a) a diverse collection of tasks included in the BigBench benchmark, and (b) compositionality, measured by generalization accuracy on semantic parsing and syntactic transformations. We find that pretraining on higher proportions of code improves performance on compositional tasks involving structured output (like semantic parsing), and mathematics. Conversely, increase code mixture can harm performance on other tasks, including on tasks that requires sensitivity to linguistic structure such as syntax or morphology, and tasks measuring real-world knowledge.

Title: Thinking Outside the BBox: Unconstrained Generative Object Compositing

• Authors: Gemma Canet Tarrés, Zhe Lin, Zhifei Zhang, Jianming Zhang, Yizhi Song, Dan Ruta, Andrew Gilbert, John Collomosse, Soo Ye Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04559
• Pdf URL: https://arxiv.org/pdf/2409.04559
• Copy Paste: [[2409.04559]] Thinking Outside the BBox: Unconstrained Generative Object Compositing(https://arxiv.org/abs/2409.04559)
• Keywords: diffusion, generative
• Abstract: Compositing an object into an image involves multiple non-trivial sub-tasks such as object placement and scaling, color/lighting harmonization, viewpoint/geometry adjustment, and shadow/reflection generation. Recent generative image compositing methods leverage diffusion models to handle multiple sub-tasks at once. However, existing models face limitations due to their reliance on masking the original object during training, which constrains their generation to the input mask. Furthermore, obtaining an accurate input mask specifying the location and scale of the object in a new image can be highly challenging. To overcome such limitations, we define a novel problem of unconstrained generative object compositing, i.e., the generation is not bounded by the mask, and train a diffusion-based model on a synthesized paired dataset. Our first-of-its-kind model is able to generate object effects such as shadows and reflections that go beyond the mask, enhancing image realism. Additionally, if an empty mask is provided, our model automatically places the object in diverse natural locations and scales, accelerating the compositing workflow. Our model outperforms existing object placement and compositing models in various quality metrics and user studies.

Title: Multi-Modal Diffusion for Hand-Object Grasp Generation

• Authors: Jinkun Cao, Jingyuan Liu, Kris Kitani, Yi Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04560
• Pdf URL: https://arxiv.org/pdf/2409.04560
• Copy Paste: [[2409.04560]] Multi-Modal Diffusion for Hand-Object Grasp Generation(https://arxiv.org/abs/2409.04560)
• Keywords: diffusion
• Abstract: In this work, we focus on generating hand grasp over objects. Compared to previous works of generating hand poses with a given object, we aim to allow the generalization of both hand and object shapes by a single model. Our proposed method Multi-modal Grasp Diffusion (MGD) learns the prior and conditional posterior distribution of both modalities from heterogeneous data sources. Therefore it relieves the limitation of hand-object grasp datasets by leveraging the large-scale 3D object datasets. According to both qualitative and quantitative experiments, both conditional and unconditional generation of hand grasp achieve good visual plausibility and diversity. The proposed method also generalizes well to unseen object shapes. The code and weights will be available at \url{this https URL}.

Title: Influence of Early through Late Fusion on Pancreas Segmentation from Imperfectly Registered Multimodal MRI

• Authors: Lucas W. Remedios, Han Liu, Samuel W. Remedios, Lianrui Zuo, Adam M. Saunders, Shunxing Bao, Yuankai Huo, Alvin C. Powers, John Virostko, Bennett A. Landman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04563
• Pdf URL: https://arxiv.org/pdf/2409.04563
• Copy Paste: [[2409.04563]] Influence of Early through Late Fusion on Pancreas Segmentation from Imperfectly Registered Multimodal MRI(https://arxiv.org/abs/2409.04563)
• Keywords: segmentation
• Abstract: Multimodal fusion promises better pancreas segmentation. However, where to perform fusion in models is still an open question. It is unclear if there is a best location to fuse information when analyzing pairs of imperfectly aligned images. Two main alignment challenges in this pancreas segmentation study are 1) the pancreas is deformable and 2) breathing deforms the abdomen. Even after image registration, relevant deformations are often not corrected. We examine how early through late fusion impacts pancreas segmentation. We used 353 pairs of T2-weighted (T2w) and T1-weighted (T1w) abdominal MR images from 163 subjects with accompanying pancreas labels. We used image registration (deeds) to align the image pairs. We trained a collection of basic UNets with different fusion points, spanning from early to late, to assess how early through late fusion influenced segmentation performance on imperfectly aligned images. We assessed generalization of fusion points on nnUNet. The single-modality T2w baseline using a basic UNet model had a Dice score of 0.73, while the same baseline on the nnUNet model achieved 0.80. For the basic UNet, the best fusion approach occurred in the middle of the encoder (early/mid fusion), which led to a statistically significant improvement of 0.0125 on Dice score compared to the baseline. For the nnUNet, the best fusion approach was naïve image concatenation before the model (early fusion), which resulted in a statistically significant Dice score increase of 0.0021 compared to baseline. Fusion in specific blocks can improve performance, but the best blocks for fusion are model specific, and the gains are small. In imperfectly registered datasets, fusion is a nuanced problem, with the art of design remaining vital for uncovering potential insights. Future innovation is needed to better address fusion in cases of imperfect alignment of abdominal image pairs.

Title: Customizing Large Language Model Generation Style using Parameter-Efficient Finetuning

• Authors: Xinyue Liu, Harshita Diddee, Daphne Ippolito
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.04574
• Pdf URL: https://arxiv.org/pdf/2409.04574
• Copy Paste: [[2409.04574]] Customizing Large Language Model Generation Style using Parameter-Efficient Finetuning(https://arxiv.org/abs/2409.04574)
• Keywords: large language model
• Abstract: One-size-fits-all large language models (LLMs) are increasingly being used to help people with their writing. However, the style these models are trained to write in may not suit all users or use cases. LLMs would be more useful as writing assistants if their idiolect could be customized to match each user. In this paper, we explore whether parameter-efficient finetuning (PEFT) with Low-Rank Adaptation can effectively guide the style of LLM generations. We use this method to customize LLaMA-2 to ten different authors and show that the generated text has lexical, syntactic, and surface alignment with the target author but struggles with content memorization. Our findings highlight the potential of PEFT to support efficient, user-level customization of LLMs.

Title: CubicML: Automated ML for Distributed ML Systems Co-design with ML Prediction of Performance

• Authors: Wei Wen, Quanyu Zhu, Weiwei Chu, Wen-Yen Chen, Jiyan Yang
• Subjects: cs.LG, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2409.04585
• Pdf URL: https://arxiv.org/pdf/2409.04585
• Copy Paste: [[2409.04585]] CubicML: Automated ML for Distributed ML Systems Co-design with ML Prediction of Performance(https://arxiv.org/abs/2409.04585)
• Keywords: large language model
• Abstract: Scaling up deep learning models has been proven effective to improve intelligence of machine learning (ML) models, especially for industry recommendation models and large language models. The co-design of distributed ML systems and algorithms (to maximize training performance) plays a pivotal role for its success. As it scales, the number of co-design hyper-parameters grows rapidly which brings challenges to feasibly find the optimal setup for system performance maximization. In this paper, we propose CubicML which uses ML to automatically optimize training performance of distributed ML systems. In CubicML, we use a ML model as a proxy to predict the training performance for search efficiency and performance modeling flexibility. We proved that CubicML can effectively optimize training speed of in-house ads recommendation models and large language models at Meta.

Title: Self-Supervised Contrastive Learning for Videos using Differentiable Local Alignment

• Authors: Keyne Oei, Amr Gomaa, Anna Maria Feit, João Belo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04607
• Pdf URL: https://arxiv.org/pdf/2409.04607
• Copy Paste: [[2409.04607]] Self-Supervised Contrastive Learning for Videos using Differentiable Local Alignment(https://arxiv.org/abs/2409.04607)
• Keywords: robust, transformer
• Abstract: Robust frame-wise embeddings are essential to perform video analysis and understanding tasks. We present a self-supervised method for representation learning based on aligning temporal video sequences. Our framework uses a transformer-based encoder to extract frame-level features and leverages them to find the optimal alignment path between video sequences. We introduce the novel Local-Alignment Contrastive (LAC) loss, which combines a differentiable local alignment loss to capture local temporal dependencies with a contrastive loss to enhance discriminative learning. Prior works on video alignment have focused on using global temporal ordering across sequence pairs, whereas our loss encourages identifying the best-scoring subsequence alignment. LAC uses the differentiable Smith-Waterman (SW) affine method, which features a flexible parameterization learned through the training phase, enabling the model to adjust the temporal gap penalty length dynamically. Evaluations show that our learned representations outperform existing state-of-the-art approaches on action recognition tasks.

Title: Detection of False Data Injection Attacks (FDIA) on Power Dynamical Systems With a State Prediction Method

• Authors: Abhijeet Sahu, Truc Nguyen, Kejun Chen, Xiangyu Zhang, Malik Hassanaly
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04609
• Pdf URL: https://arxiv.org/pdf/2409.04609
• Copy Paste: [[2409.04609]] Detection of False Data Injection Attacks (FDIA) on Power Dynamical Systems With a State Prediction Method(https://arxiv.org/abs/2409.04609)
• Keywords: security, protect, attack
• Abstract: With the deeper penetration of inverter-based resources in power systems, false data injection attacks (FDIA) are a growing cyber-security concern. They have the potential to disrupt the system's stability like frequency stability, thereby leading to catastrophic failures. Therefore, an FDIA detection method would be valuable to protect power systems. FDIAs typically induce a discrepancy between the desired and the effective behavior of the power system dynamics. A suitable detection method can leverage power dynamics predictions to identify whether such a discrepancy was induced by an FDIA. This work investigates the efficacy of temporal and spatio-temporal state prediction models, such as Long Short-Term Memory (LSTM) and a combination of Graph Neural Networks (GNN) with LSTM, for predicting frequency dynamics in the absence of an FDIA but with noisy measurements, and thereby identify FDIA events. For demonstration purposes, the IEEE 39 New England Kron-reduced model simulated with a swing equation is considered. It is shown that the proposed state prediction models can be used as a building block for developing an effective FDIA detection method that can maintain high detection accuracy across various attack and deployment settings. It is also shown how the FDIA detection should be deployed to limit its exposure to detection inaccuracies and mitigate its computational burden.

Title: Sparse Rewards Can Self-Train Dialogue Agents

• Authors: Barrett Martin Lattimer, Varun Gangal, Ryan McDonald, Yi Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.04617
• Pdf URL: https://arxiv.org/pdf/2409.04617
• Copy Paste: [[2409.04617]] Sparse Rewards Can Self-Train Dialogue Agents(https://arxiv.org/abs/2409.04617)
• Keywords: large language model
• Abstract: Recent advancements in state-of-the-art (SOTA) Large Language Model (LLM) agents, especially in multi-turn dialogue tasks, have been primarily driven by supervised fine-tuning and high-quality human feedback. However, as base LLM models continue to improve, acquiring meaningful human feedback has become increasingly challenging and costly. In certain domains, base LLM agents may eventually exceed human capabilities, making traditional feedback-driven methods impractical. In this paper, we introduce a novel self-improvement paradigm that empowers LLM agents to autonomously enhance their performance without external human feedback. Our method, Juxtaposed Outcomes for Simulation Harvesting (JOSH), is a self-alignment algorithm that leverages a sparse reward simulation environment to extract ideal behaviors and further train the LLM on its own outputs. We present ToolWOZ, a sparse reward tool-calling simulation environment derived from MultiWOZ. We demonstrate that models trained with JOSH, both small and frontier, significantly improve tool-based interactions while preserving general model capabilities across diverse benchmarks. Our code and data are publicly available on GitHub.

Title: Notes on Sampled Gaussian Mechanism

• Authors: Nikita P. Kalinin
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.04636
• Pdf URL: https://arxiv.org/pdf/2409.04636
• Copy Paste: [[2409.04636]] Notes on Sampled Gaussian Mechanism(https://arxiv.org/abs/2409.04636)
• Keywords: privacy
• Abstract: In these notes, we prove a recent conjecture posed in the paper by Räisä, O. et al. [Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimization (2024)]. Theorem 6.2 of the paper asserts that for the Sampled Gaussian Mechanism - a composition of subsampling and additive Gaussian noise, the effective noise level, $\sigma_{\text{eff}} = \frac{\sigma(q)}{q}$, decreases as a function of the subsampling rate $q$. Consequently, larger subsampling rates are preferred for better privacy-utility trade-offs. Our notes provide a rigorous proof of Conjecture 6.3, which was left unresolved in the original paper, thereby completing the proof of Theorem 6.2.

Title: The Kubernetes Security Landscape: AI-Driven Insights from Developer Discussions

• Authors: J. Alexander Curtis, Nasir U. Eisty
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.04647
• Pdf URL: https://arxiv.org/pdf/2409.04647
• Copy Paste: [[2409.04647]] The Kubernetes Security Landscape: AI-Driven Insights from Developer Discussions(https://arxiv.org/abs/2409.04647)
• Keywords: secure, security
• Abstract: Kubernetes, the go-to container orchestration solution, has swiftly become the industry standard for managing containers at scale in production environments. Its widespread adoption, particularly in large organizations, has elevated its profile and made it a prime target for security concerns. This study aims to understand how prevalent security concerns are among Kubernetes practitioners by analyzing all Kubernetes posts made on Stack Overflow over the past four years. We gathered security insights from Kubernetes practitioners and transformed the data through machine learning algorithms for cleaning and topic clustering. Subsequently, we used advanced AI tools to automatically generate topic descriptions, thereby reducing the analysis process. In our analysis, security-related posts ranked as the fourth most prevalent topic in these forums, comprising 12.3% of the overall discussions. Furthermore, the findings indicated that although the frequency of security discussions has remained constant, their popularity and influence have experienced significant growth. Kubernetes users consistently prioritize security topics, and the rising popularity of security posts reflects a growing interest and concern for maintaining secure Kubernetes clusters. The findings underscore key security issues that warrant further research and the development of additional tools to resolve them.

Title: Privacy-Preserving Race/Ethnicity Estimation for Algorithmic Bias Measurement in the U.S

• Authors: Saikrishna Badrinarayanan, Osonde Osoba, Miao Cheng, Ryan Rogers, Sakshi Jain, Rahul Tandra, Natesh S. Pillai
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.04652
• Pdf URL: https://arxiv.org/pdf/2409.04652
• Copy Paste: [[2409.04652]] Privacy-Preserving Race/Ethnicity Estimation for Algorithmic Bias Measurement in the U.S(https://arxiv.org/abs/2409.04652)
• Keywords: secure, privacy, fair
• Abstract: AI fairness measurements, including tests for equal treatment, often take the form of disaggregated evaluations of AI systems. Such measurements are an important part of Responsible AI operations. These measurements compare system performance across demographic groups or sub-populations and typically require member-level demographic signals such as gender, race, ethnicity, and location. However, sensitive member-level demographic attributes like race and ethnicity can be challenging to obtain and use due to platform choices, legal constraints, and cultural norms. In this paper, we focus on the task of enabling AI fairness measurements on race/ethnicity for \emph{U.S. LinkedIn members} in a privacy-preserving manner. We present the Privacy-Preserving Probabilistic Race/Ethnicity Estimation (PPRE) method for performing this task. PPRE combines the Bayesian Improved Surname Geocoding (BISG) model, a sparse LinkedIn survey sample of self-reported demographics, and privacy-enhancing technologies like secure two-party computation and differential privacy to enable meaningful fairness measurements while preserving member privacy. We provide details of the PPRE method and its privacy guarantees. We then illustrate sample measurement operations. We conclude with a review of open research and engineering challenges for expanding our privacy-preserving fairness measurement capabilities.

Title: Generalization vs. Memorization in the Presence of Statistical Biases in Transformers

• Authors: John Mitros, Damien Teney
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.04654
• Pdf URL: https://arxiv.org/pdf/2409.04654
• Copy Paste: [[2409.04654]] Generalization vs. Memorization in the Presence of Statistical Biases in Transformers(https://arxiv.org/abs/2409.04654)
• Keywords: transformer
• Abstract: This study aims to understand how statistical biases affect the model's ability to generalize to in-distribution and out-of-distribution data on algorithmic tasks. Prior research indicates that transformers may inadvertently learn to rely on these spurious correlations, leading to an overestimation of their generalization capabilities. To investigate this, we evaluate transformer models on several synthetic algorithmic tasks, systematically introducing and varying the presence of these biases. We also analyze how different components of the transformer models impact their generalization. Our findings suggest that statistical biases impair the model's performance on out-of-distribution data, providing a overestimation of its generalization capabilities. The models rely heavily on these spurious correlations for inference, as indicated by their performance on tasks including such biases.

Title: Multi-Conditioned Denoising Diffusion Probabilistic Model (mDDPM) for Medical Image Synthesis

• Authors: Arjun Krishna, Ge Wang, Klaus Mueller
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04670
• Pdf URL: https://arxiv.org/pdf/2409.04670
• Copy Paste: [[2409.04670]] Multi-Conditioned Denoising Diffusion Probabilistic Model (mDDPM) for Medical Image Synthesis(https://arxiv.org/abs/2409.04670)
• Keywords: diffusion, generative
• Abstract: Medical imaging applications are highly specialized in terms of human anatomy, pathology, and imaging domains. Therefore, annotated training datasets for training deep learning applications in medical imaging not only need to be highly accurate but also diverse and large enough to encompass almost all plausible examples with respect to those specifications. We argue that achieving this goal can be facilitated through a controlled generation framework for synthetic images with annotations, requiring multiple conditional specifications as input to provide control. We employ a Denoising Diffusion Probabilistic Model (DDPM) to train a large-scale generative model in the lung CT domain and expand upon a classifier-free sampling strategy to showcase one such generation framework. We show that our approach can produce annotated lung CT images that can faithfully represent anatomy, convincingly fooling experts into perceiving them as real. Our experiments demonstrate that controlled generative frameworks of this nature can surpass nearly every state-of-the-art image generative model in achieving anatomical consistency in generated medical images when trained on comparable large medical datasets.

Title: PANTS: Practical Adversarial Network Traffic Samples against ML-powered Networking Classifiers

• Authors: Minhao Jin, Maria Apostolaki
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.04691
• Pdf URL: https://arxiv.org/pdf/2409.04691
• Copy Paste: [[2409.04691]] PANTS: Practical Adversarial Network Traffic Samples against ML-powered Networking Classifiers(https://arxiv.org/abs/2409.04691)
• Keywords: security, attack, robust
• Abstract: Multiple network management tasks, from resource allocation to intrusion detection, rely on some form of ML-based network-traffic classification (MNC). Despite their potential, MNCs are vulnerable to adversarial inputs, which can lead to outages, poor decision-making, and security violations, among other issues. The goal of this paper is to help network operators assess and enhance the robustness of their MNC against adversarial inputs. The most critical step for this is generating inputs that can fool the MNC while being realizable under various threat models. Compared to other ML models, finding adversarial inputs against MNCs is more challenging due to the existence of non-differentiable components e.g., traffic engineering and the need to constrain inputs to preserve semantics and ensure reliability. These factors prevent the direct use of well-established gradient-based methods developed in adversarial ML (AML). To address these challenges, we introduce PANTS, a practical white-box framework that uniquely integrates AML techniques with Satisfiability Modulo Theories (SMT) solvers to generate adversarial inputs for MNCs. We also embed PANTS into an iterative adversarial training process that enhances the robustness of MNCs against adversarial inputs. PANTS is 70% and 2x more likely in median to find adversarial inputs against target MNCs compared to two state-of-the-art baselines, namely Amoeba and BAP. Integrating PANTS into the adversarial training process enhances the robustness of the target MNCs by 52.7% without sacrificing their accuracy. Critically, these PANTS-robustified MNCs are more robust than their vanilla counterparts against distinct attack-generation methodologies.

Title: Hierarchical Sparse Representation Clustering for High-Dimensional Data Streams

• Authors: Jie Chen, Hua Mao, Yuanbiao Gou, Xi Peng
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04698
• Pdf URL: https://arxiv.org/pdf/2409.04698
• Copy Paste: [[2409.04698]] Hierarchical Sparse Representation Clustering for High-Dimensional Data Streams(https://arxiv.org/abs/2409.04698)
• Keywords: robust
• Abstract: Data stream clustering reveals patterns within continuously arriving, potentially unbounded data sequences. Numerous data stream algorithms have been proposed to cluster data streams. The existing data stream clustering algorithms still face significant challenges when addressing high-dimensional data streams. First, it is intractable to measure the similarities among high-dimensional data objects via Euclidean distances when constructing and merging microclusters. Second, these algorithms are highly sensitive to the noise contained in high-dimensional data streams. In this paper, we propose a hierarchical sparse representation clustering (HSRC) method for clustering high-dimensional data streams. HSRC first employs an $l_1$-minimization technique to learn an affinity matrix for data objects in individual landmark windows with fixed sizes, where the number of neighboring data objects is automatically selected. This approach ensures that highly correlated data samples within clusters are grouped together. Then, HSRC applies a spectral clustering technique to the affinity matrix to generate microclusters. These microclusters are subsequently merged into macroclusters based on their sparse similarity degrees (SSDs). Additionally, HSRC introduces sparsity residual values (SRVs) to adaptively select representative data objects from the current landmark window. These representatives serve as dictionary samples for the next landmark window. Finally, HSRC refines each macrocluster through fine-tuning. In particular, HSRC enables the detection of outliers in high-dimensional data streams via the associated SRVs. The experimental results obtained on several benchmark datasets demonstrate the effectiveness and robustness of HSRC.

Title: Dual-stream Feature Augmentation for Domain Generalization

• Authors: Shanshan Wang, ALuSi, Xun Yang, Ke Xu, Huibin Tan, Xingyi Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04699
• Pdf URL: https://arxiv.org/pdf/2409.04699
• Copy Paste: [[2409.04699]] Dual-stream Feature Augmentation for Domain Generalization(https://arxiv.org/abs/2409.04699)
• Keywords: robust
• Abstract: Domain generalization (DG) task aims to learn a robust model from source domains that could handle the out-of-distribution (OOD) issue. In order to improve the generalization ability of the model in unseen domains, increasing the diversity of training samples is an effective solution. However, existing augmentation approaches always have some limitations. On the one hand, the augmentation manner in most DG methods is not enough as the model may not see the perturbed features in approximate the worst case due to the randomness, thus the transferability in features could not be fully explored. On the other hand, the causality in discriminative features is not involved in these methods, which harms the generalization ability of model due to the spurious correlations. To address these issues, we propose a Dual-stream Feature Augmentation~(DFA) method by constructing some hard features from two perspectives. Firstly, to improve the transferability, we construct some targeted features with domain related augmentation manner. Through the guidance of uncertainty, some hard cross-domain fictitious features are generated to simulate domain shift. Secondly, to take the causality into consideration, the spurious correlated non-causal information is disentangled by an adversarial mask, then the more discriminative features can be extracted through these hard causal related information. Different from previous fixed synthesizing strategy, the two augmentations are integrated into a unified learnable feature disentangle model. Based on these hard features, contrastive learning is employed to keep the semantic consistency and improve the robustness of the model. Extensive experiments on several datasets demonstrated that our approach could achieve state-of-the-art performance for domain generalization. Our code is available at: this https URL.

Title: Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models

• Authors: Michael Günther, Isabelle Mohr, Bo Wang, Han Xiao
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.04701
• Pdf URL: https://arxiv.org/pdf/2409.04701
• Copy Paste: [[2409.04701]] Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models(https://arxiv.org/abs/2409.04701)
• Keywords: transformer
• Abstract: Many use cases require retrieving smaller portions of text, and dense vector-based retrieval systems often perform better with shorter text segments, as the semantics are less likely to be "over-compressed" in the embeddings. Consequently, practitioners often split text documents into smaller chunks and encode them separately. However, chunk embeddings created in this way can lose contextual information from surrounding chunks, resulting in suboptimal representations. In this paper, we introduce a novel method called "late chunking," which leverages long context embedding models to first embed all tokens of the long text, with chunking applied after the transformer model and just before mean pooling. The resulting chunk embeddings capture the full contextual information, leading to superior results across various retrieval tasks without the need for additional training. Moreover, our method is generic enough to be applied to any long-context embedding model.

Title: A Multi-scenario Attention-based Generative Model for Personalized Blood Pressure Time Series Forecasting

• Authors: Cheng Wan, Chenjie Xie, Longfei Liu, Dan Wu, Ye Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04704
• Pdf URL: https://arxiv.org/pdf/2409.04704
• Copy Paste: [[2409.04704]] A Multi-scenario Attention-based Generative Model for Personalized Blood Pressure Time Series Forecasting(https://arxiv.org/abs/2409.04704)
• Keywords: robust, generative
• Abstract: Continuous blood pressure (BP) monitoring is essential for timely diagnosis and intervention in critical care settings. However, BP varies significantly across individuals, this inter-patient variability motivates the development of personalized models tailored to each patient's physiology. In this work, we propose a personalized BP forecasting model mainly using electrocardiogram (ECG) and photoplethysmogram (PPG) signals. This time-series model incorporates 2D representation learning to capture complex physiological relationships. Experiments are conducted on datasets collected from three diverse scenarios with BP measurements from 60 subjects total. Results demonstrate that the model achieves accurate and robust BP forecasts across scenarios within the Association for the Advancement of Medical Instrumentation (AAMI) standard criteria. This reliable early detection of abnormal fluctuations in BP is crucial for at-risk patients undergoing surgery or intensive care. The proposed model provides a valuable addition for continuous BP tracking to reduce mortality and improve prognosis.

Title: Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training

• Authors: Yuhan Ma, Dan Sun, Erdi Gao, Ningjing Sang, Iris Li, Guanming Huang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04707
• Pdf URL: https://arxiv.org/pdf/2409.04707
• Copy Paste: [[2409.04707]] Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Training(https://arxiv.org/abs/2409.04707)
• Keywords: interpretability
• Abstract: Optimization theory serves as a pivotal scientific instrument for achieving optimal system performance, with its origins in economic applications to identify the best investment strategies for maximizing benefits. Over the centuries, from the geometric inquiries of ancient Greece to the calculus contributions by Newton and Leibniz, optimization theory has significantly advanced. The persistent work of scientists like Lagrange, Cauchy, and von Neumann has fortified its progress. The modern era has seen an unprecedented expansion of optimization theory applications, particularly with the growth of computer science, enabling more sophisticated computational practices and widespread utilization across engineering, decision analysis, and operations research. This paper delves into the profound relationship between optimization theory and deep learning, highlighting the omnipresence of optimization problems in the latter. We explore the gradient descent algorithm and its variants, which are the cornerstone of optimizing neural networks. The chapter introduces an enhancement to the SGD optimizer, drawing inspiration from numerical optimization methods, aiming to enhance interpretability and accuracy. Our experiments on diverse deep learning tasks substantiate the improved algorithm's efficacy. The paper concludes by emphasizing the continuous development of optimization theory and its expanding role in solving intricate problems, enhancing computational capabilities, and informing better policy decisions.

Title: Unleashing the Power of Generic Segmentation Models: A Simple Baseline for Infrared Small Target Detection

• Authors: Mingjin Zhang, Chi Zhang, Qiming Zhang, Yunsong Li, Xinbo Gao, Jing Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04714
• Pdf URL: https://arxiv.org/pdf/2409.04714
• Copy Paste: [[2409.04714]] Unleashing the Power of Generic Segmentation Models: A Simple Baseline for Infrared Small Target Detection(https://arxiv.org/abs/2409.04714)
• Keywords: segmentation
• Abstract: Recent advancements in deep learning have greatly advanced the field of infrared small object detection (IRSTD). Despite their remarkable success, a notable gap persists between these IRSTD methods and generic segmentation approaches in natural image domains. This gap primarily arises from the significant modality differences and the limited availability of infrared data. In this study, we aim to bridge this divergence by investigating the adaptation of generic segmentation models, such as the Segment Anything Model (SAM), to IRSTD tasks. Our investigation reveals that many generic segmentation models can achieve comparable performance to state-of-the-art IRSTD methods. However, their full potential in IRSTD remains untapped. To address this, we propose a simple, lightweight, yet effective baseline model for segmenting small infrared objects. Through appropriate distillation strategies, we empower smaller student models to outperform state-of-the-art methods, even surpassing fine-tuned teacher results. Furthermore, we enhance the model's performance by introducing a novel query design comprising dense and sparse queries to effectively encode multi-scale features. Through extensive experimentation across four popular IRSTD datasets, our model demonstrates significantly improved performance in both accuracy and throughput compared to existing approaches, surpassing SAM and Semantic-SAM by over 14 IoU on NUDT and 4 IoU on IRSTD1k. The source code and models will be released at this https URL.

Title: Cross-Organ Domain Adaptive Neural Network for Pancreatic Endoscopic Ultrasound Image Segmentation

• Authors: ZhiChao Yan, Hui Xue, Yi Zhu, Bin Xiao, Hao Yuan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04718
• Pdf URL: https://arxiv.org/pdf/2409.04718
• Copy Paste: [[2409.04718]] Cross-Organ Domain Adaptive Neural Network for Pancreatic Endoscopic Ultrasound Image Segmentation(https://arxiv.org/abs/2409.04718)
• Keywords: segmentation
• Abstract: Accurate segmentation of lesions in pancreatic endoscopic ultrasound (EUS) images is crucial for effective diagnosis and treatment. However, the collection of enough crisp EUS images for effective diagnosis is arduous. Recently, domain adaptation (DA) has been employed to address these challenges by leveraging related knowledge from other domains. Most DA methods only focus on multi-view representations of the same organ, which makes it still tough to clearly depict the tumor lesion area with limited semantic information. Although transferring homogeneous similarity from different organs could benefit the issue, there is a lack of relevant work due to the enormous domain gap between them. To address these challenges, we propose the Cross-Organ Tumor Segmentation Networks (COTS-Nets), consisting of a universal network and an auxiliary network. The universal network utilizes boundary loss to learn common boundary information of different tumors, enabling accurate delineation of tumors in EUS despite limited and low-quality data. Simultaneously, we incorporate consistency loss in the universal network to align the prediction of pancreatic EUS with tumor boundaries from other organs to mitigate the domain gap. To further reduce the cross-organ domain gap, the auxiliary network integrates multi-scale features from different organs, aiding the universal network in acquiring domain-invariant knowledge. Systematic experiments demonstrate that COTS-Nets significantly improves the accuracy of pancreatic cancer diagnosis. Additionally, we developed the Pancreatic Cancer Endoscopic Ultrasound (PCEUS) dataset, comprising 501 pathologically confirmed pancreatic EUS images, to facilitate model development.

Title: VidLPRO: A $\underline{Vid}$eo-$\underline{L}$anguage $\underline{P}$re-training Framework for $\underline{Ro}$botic and Laparoscopic Surgery

• Authors: Mohammadmahdi Honarmand, Muhammad Abdullah Jamal, Omid Mohareri
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04732
• Pdf URL: https://arxiv.org/pdf/2409.04732
• Copy Paste: [[2409.04732]] VidLPRO: A $\underline{Vid}$eo-$\underline{L}$anguage $\underline{P}$re-training Framework for $\underline{Ro}$botic and Laparoscopic Surgery(https://arxiv.org/abs/2409.04732)
• Keywords: robust
• Abstract: We introduce VidLPRO, a novel video-language (VL) pre-training framework designed specifically for robotic and laparoscopic surgery. While existing surgical VL models primarily rely on contrastive learning, we propose a more comprehensive approach to capture the intricate temporal dynamics and align video with language. VidLPRO integrates video-text contrastive learning, video-text matching, and masked language modeling objectives to learn rich VL representations. To support this framework, we present GenSurg+, a carefully curated dataset derived from GenSurgery, comprising 17k surgical video clips paired with captions generated by GPT-4 using transcripts extracted by the Whisper model. This dataset addresses the need for large-scale, high-quality VL data in the surgical domain. Extensive experiments on benchmark datasets, including Cholec80 and AutoLaparo, demonstrate the efficacy of our approach. VidLPRO achieves state-of-the-art performance in zero-shot surgical phase recognition, significantly outperforming existing surgical VL models such as SurgVLP and HecVL. Our model demonstrates improvements of up to 21.5\% in accuracy and 15.7% in F1 score, setting a new benchmark in the field. Notably, VidLPRO exhibits robust performance even with single-frame inference, while effectively scaling with increased temporal context. Ablation studies reveal the impact of frame sampling strategies on model performance and computational efficiency. These results underscore VidLPRO's potential as a foundation model for surgical video understanding.

Title: A Sample Efficient Alternating Minimization-based Algorithm For Robust Phase Retrieval

• Authors: Adarsh Barik, Anand Krishna, Vincent Y. F. Tan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04733
• Pdf URL: https://arxiv.org/pdf/2409.04733
• Copy Paste: [[2409.04733]] A Sample Efficient Alternating Minimization-based Algorithm For Robust Phase Retrieval(https://arxiv.org/abs/2409.04733)
• Keywords: robust
• Abstract: In this work, we study the robust phase retrieval problem where the task is to recover an unknown signal $\theta^* \in \mathbb{R}^d$ in the presence of potentially arbitrarily corrupted magnitude-only linear measurements. We propose an alternating minimization approach that incorporates an oracle solver for a non-convex optimization problem as a subroutine. Our algorithm guarantees convergence to $\theta^*$ and provides an explicit polynomial dependence of the convergence rate on the fraction of corrupted measurements. We then provide an efficient construction of the aforementioned oracle under a sparse arbitrary outliers model and offer valuable insights into the geometric properties of the loss landscape in phase retrieval with corrupted measurements. Our proposed oracle avoids the need for computationally intensive spectral initialization, using a simple gradient descent algorithm with a constant step size and random initialization instead. Additionally, our overall algorithm achieves nearly linear sample complexity, $\mathcal{O}(d \, \mathrm{polylog}(d))$.

Title: Swin Transformer for Robust Differentiation of Real and Synthetic Images: Intra- and Inter-Dataset Analysis

• Authors: Preetu Mehta, Aman Sagar, Suchi Kumari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04734
• Pdf URL: https://arxiv.org/pdf/2409.04734
• Copy Paste: [[2409.04734]] Swin Transformer for Robust Differentiation of Real and Synthetic Images: Intra- and Inter-Dataset Analysis(https://arxiv.org/abs/2409.04734)
• Keywords: robust, transformer
• Abstract: \textbf{Purpose} This study aims to address the growing challenge of distinguishing computer-generated imagery (CGI) from authentic digital images in the RGB color space. Given the limitations of existing classification methods in handling the complexity and variability of CGI, this research proposes a Swin Transformer-based model for accurate differentiation between natural and synthetic images. \textbf{Methods} The proposed model leverages the Swin Transformer's hierarchical architecture to capture local and global features crucial for distinguishing CGI from natural images. The model's performance was evaluated through intra-dataset and inter-dataset testing across three distinct datasets: CiFAKE, JSSSTU, and Columbia. The datasets were tested individually (D1, D2, D3) and in combination (D1+D2+D3) to assess the model's robustness and domain generalization capabilities. \textbf{Results} The Swin Transformer-based model demonstrated high accuracy, consistently achieving a range of 97-99\% across all datasets and testing scenarios. These results confirm the model's effectiveness in detecting CGI, showcasing its robustness and reliability in both intra-dataset and inter-dataset evaluations. \textbf{Conclusion} The findings of this study highlight the Swin Transformer model's potential as an advanced tool for digital image forensics, particularly in distinguishing CGI from natural images. The model's strong performance across multiple datasets indicates its capability for domain generalization, making it a valuable asset in scenarios requiring precise and reliable image classification.

Title: LiTelFuzz : Swarms Fuzzing Based on Linear Temporal Logic Constraints

• Authors: Zhiwei Zhang, Ruoyu Zhou, Haocheng Han, Xiaodong Zhang, Yulong Shen
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.04736
• Pdf URL: https://arxiv.org/pdf/2409.04736
• Copy Paste: [[2409.04736]] LiTelFuzz : Swarms Fuzzing Based on Linear Temporal Logic Constraints(https://arxiv.org/abs/2409.04736)
• Keywords: security, attack, robust
• Abstract: Multi-robot swarms utilize swarm intelligence to collaborate on tasks and play an increasingly significant role in a variety of practical scenarios. However, due to the complex design, multi-robot swarm systems often have vulnerabilities caused by logical errors, which can severely disrupt the normal operations of multi-robot swarms. Despite the significant security threats that logical vulnerabilities pose to multi-robot swarms, there are still considerable challenges in testing and identifying these vulnerabilities, and related research still faces two major challenges: 1) the explosion of input space for testing, 2) the lack of effective test-guidance strategies. Therefore, in this paper, we overcome the two major challenges mentioned above, and propose a formal verification method to discover logical flaws in multi-robot swarms. Specifically, we abstract linear temporal logic constraints of the swarm and compute swarm robustness based on these constraints thus guiding fuzzing, we call this approach LiTelFuzz (Fuzzing based on Linear Temporal Logic Constraints). The core idea of LiTelFuzz is to design a metric based on behavioral constraints to assess the state of the multi-robot swarm at different moments, and guide fuzz testing based on the assessment results. Based on this idea, we overcome the two challenges of excessive test case input space and the lack of fuzzing guidance. Consequently, we implement a single attack drone fuzzing scheme and a multiple attack drones scheme based on LiTelFuzz. These are named SA-Fuzzing and MA-Fuzzing, respectively. Finally, we tested three popular swarm algorithms using LiTelFuzz with an average success rate of 87.35% for SA-Fuzzing and 91.73% for MA-Fuzzing to find vulnerabilities. The success rate and efficiency are better than the existing state-of-the-art fuzzer SWARMFLAWFINDER.

Title: Enhancing Image Authenticity Detection: Swin Transformers and Color Frame Analysis for CGI vs. Real Images

• Authors: Preeti Mehta, Aman Sagar, Suchi Kumari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04742
• Pdf URL: https://arxiv.org/pdf/2409.04742
• Copy Paste: [[2409.04742]] Enhancing Image Authenticity Detection: Swin Transformers and Color Frame Analysis for CGI vs. Real Images(https://arxiv.org/abs/2409.04742)
• Keywords: robust, transformer
• Abstract: The rapid advancements in computer graphics have greatly enhanced the quality of computer-generated images (CGI), making them increasingly indistinguishable from authentic images captured by digital cameras (ADI). This indistinguishability poses significant challenges, especially in an era of widespread misinformation and digitally fabricated content. This research proposes a novel approach to classify CGI and ADI using Swin Transformers and preprocessing techniques involving RGB and CbCrY color frame analysis. By harnessing the capabilities of Swin Transformers, our method foregoes handcrafted features instead of relying on raw pixel data for model training. This approach achieves state-of-the-art accuracy while offering substantial improvements in processing speed and robustness against joint image manipulations such as noise addition, blurring, and JPEG compression. Our findings highlight the potential of Swin Transformers combined with advanced color frame analysis for effective and efficient image authenticity detection.

Title: LMGT: Optimizing Exploration-Exploitation Balance in Reinforcement Learning through Language Model Guided Trade-offs

• Authors: Yongxin Deng, Xihe Qiu, Xiaoyu Tan, Wei Chu, Yinghui Xu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04744
• Pdf URL: https://arxiv.org/pdf/2409.04744
• Copy Paste: [[2409.04744]] LMGT: Optimizing Exploration-Exploitation Balance in Reinforcement Learning through Language Model Guided Trade-offs(https://arxiv.org/abs/2409.04744)
• Keywords: large language model
• Abstract: The uncertainty inherent in the environmental transition model of Reinforcement Learning (RL) necessitates a careful balance between exploration and exploitation to optimize the use of computational resources for accurately estimating an agent's expected reward. Achieving balance in control systems is particularly challenging in scenarios with sparse rewards. However, given the extensive prior knowledge available for many environments, it is redundant to begin learning from scratch in such settings. To address this, we introduce \textbf{L}anguage \textbf{M}odel \textbf{G}uided \textbf{T}rade-offs (i.e., \textbf{LMGT}), a novel, sample-efficient framework that leverages the comprehensive prior knowledge embedded in Large Language Models (LLMs) and their adeptness at processing non-standard data forms, such as wiki tutorials. LMGT proficiently manages the exploration-exploitation trade-off by employing reward shifts guided by LLMs, which direct agents' exploration endeavors, thereby improving sample efficiency. We have thoroughly tested LMGT across various RL tasks and deployed it in industrial-grade RL recommendation systems, where it consistently outperforms baseline methods. The results indicate that our framework can significantly reduce the time cost required during the training phase in RL.

Title: Training-Free Style Consistent Image Synthesis with Condition and Mask Guidance in E-Commerce

• Authors: Guandong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04750
• Pdf URL: https://arxiv.org/pdf/2409.04750
• Copy Paste: [[2409.04750]] Training-Free Style Consistent Image Synthesis with Condition and Mask Guidance in E-Commerce(https://arxiv.org/abs/2409.04750)
• Keywords: diffusion
• Abstract: Generating style-consistent images is a common task in the e-commerce field, and current methods are largely based on diffusion models, which have achieved excellent results. This paper introduces the concept of the QKV (query/key/value) level, referring to modifications in the attention maps (self-attention and cross-attention) when integrating UNet with image conditions. Without disrupting the product's main composition in e-commerce images, we aim to use a train-free method guided by pre-set conditions. This involves using shared KV to enhance similarity in cross-attention and generating mask guidance from the attention map to cleverly direct the generation of style-consistent images. Our method has shown promising results in practical applications.

Title: SGSeg: Enabling Text-free Inference in Language-guided Segmentation of Chest X-rays via Self-guidance

• Authors: Shuchang Ye, Mingyuan Meng, Mingjian Li, Dagan Feng, Jinman Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04758
• Pdf URL: https://arxiv.org/pdf/2409.04758
• Copy Paste: [[2409.04758]] SGSeg: Enabling Text-free Inference in Language-guided Segmentation of Chest X-rays via Self-guidance(https://arxiv.org/abs/2409.04758)
• Keywords: extraction, segmentation
• Abstract: Segmentation of infected areas in chest X-rays is pivotal for facilitating the accurate delineation of pulmonary structures and pathological anomalies. Recently, multi-modal language-guided image segmentation methods have emerged as a promising solution for chest X-rays where the clinical text reports, depicting the assessment of the images, are used as guidance. Nevertheless, existing language-guided methods require clinical reports alongside the images, and hence, they are not applicable for use in image segmentation in a decision support context, but rather limited to retrospective image analysis after clinical reporting has been completed. In this study, we propose a self-guided segmentation framework (SGSeg) that leverages language guidance for training (multi-modal) while enabling text-free inference (uni-modal), which is the first that enables text-free inference in language-guided segmentation. We exploit the critical location information of both pulmonary and pathological structures depicted in the text reports and introduce a novel localization-enhanced report generation (LERG) module to generate clinical reports for self-guidance. Our LERG integrates an object detector and a location-based attention aggregator, weakly-supervised by a location-aware pseudo-label extraction module. Extensive experiments on a well-benchmarked QaTa-COV19 dataset demonstrate that our SGSeg achieved superior performance than existing uni-modal segmentation methods and closely matched the state-of-the-art performance of multi-modal language-guided segmentation methods.

Title: Medical Image Segmentation via Single-Source Domain Generalization with Random Amplitude Spectrum Synthesis

• Authors: Qiang Qiao, Wenyu Wang, Meixia Qu, Kun Su, Bin Jiang, Qiang Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04768
• Pdf URL: https://arxiv.org/pdf/2409.04768
• Copy Paste: [[2409.04768]] Medical Image Segmentation via Single-Source Domain Generalization with Random Amplitude Spectrum Synthesis(https://arxiv.org/abs/2409.04768)
• Keywords: privacy, segmentation
• Abstract: The field of medical image segmentation is challenged by domain generalization (DG) due to domain shifts in clinical datasets. The DG challenge is exacerbated by the scarcity of medical data and privacy concerns. Traditional single-source domain generalization (SSDG) methods primarily rely on stacking data augmentation techniques to minimize domain discrepancies. In this paper, we propose Random Amplitude Spectrum Synthesis (RASS) as a training augmentation for medical images. RASS enhances model generalization by simulating distribution changes from a frequency perspective. This strategy introduces variability by applying amplitude-dependent perturbations to ensure broad coverage of potential domain variations. Furthermore, we propose random mask shuffle and reconstruction components, which can enhance the ability of the backbone to process structural information and increase resilience intra- and cross-domain changes. The proposed Random Amplitude Spectrum Synthesis for Single-Source Domain Generalization (RAS^4DG) is validated on 3D fetal brain images and 2D fundus photography, and achieves an improved DG segmentation performance compared to other SSDG models.

Title: Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models

• Authors: Junfeng Tian, Da Zheng, Yang Cheng, Rui Wang, Colin Zhang, Debing Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04774
• Pdf URL: https://arxiv.org/pdf/2409.04774
• Copy Paste: [[2409.04774]] Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models(https://arxiv.org/abs/2409.04774)
• Keywords: large language model
• Abstract: Large language models (LLM) have prioritized expanding the context window from which models can incorporate more information. However, training models to handle long contexts presents significant challenges. These include the scarcity of high-quality natural long-context data, the potential for performance degradation on short-context tasks, and the reduced training efficiency associated with attention mechanisms. In this paper, we introduce Untie the Knots (\textbf{UtK}), a novel data augmentation strategy employed during the continue pre-training phase, designed to efficiently enable LLMs to gain long-context capabilities without the need to modify the existing data mixture. In particular, we chunk the documents, shuffle the chunks, and create a complex and knotted structure of long texts; LLMs are then trained to untie these knots and identify relevant segments within seemingly chaotic token sequences. This approach greatly improves the model's performance by accurately attending to relevant information in long context and the training efficiency is also largely increased. We conduct extensive experiments on models with 7B and 72B parameters, trained on 20 billion tokens, demonstrating that UtK achieves 75\% and 84.5\% accurracy on RULER at 128K context length, significantly outperforming other long context strategies. The trained models will open-source for further research.

Title: Optimization Hyper-parameter Laws for Large Language Models

• Authors: Xingyu Xie, Kuangyu Ding, Shuicheng Yan, Kim-Chuan Toh, Tianwen Wei
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2409.04777
• Pdf URL: https://arxiv.org/pdf/2409.04777
• Copy Paste: [[2409.04777]] Optimization Hyper-parameter Laws for Large Language Models(https://arxiv.org/abs/2409.04777)
• Keywords: robust, interpretability, large language model
• Abstract: Large Language Models have driven significant AI advancements, yet their training is resource-intensive and highly sensitive to hyper-parameter selection. While scaling laws provide valuable guidance on model size and data requirements, they fall short in choosing dynamic hyper-parameters, such as learning-rate (LR) schedules, that evolve during training. To bridge this gap, we present Optimization Hyper-parameter Laws (Opt-Laws), a framework that effectively captures the relationship between hyper-parameters and training outcomes, enabling the pre-selection of potential optimal schedules. Grounded in stochastic differential equations, Opt-Laws introduce novel mathematical interpretability and offer a robust theoretical foundation for some popular LR schedules. Our extensive validation across diverse model sizes and data scales demonstrates Opt-Laws' ability to accurately predict training loss and identify optimal LR schedule candidates in pre-training, continual training, and fine-tuning scenarios. This approach significantly reduces computational costs while enhancing overall model performance.

Title: Selective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Models

• Authors: Sonam Gupta, Yatin Nandwani, Asaf Yehudai, Mayank Mishra, Gaurav Pandey, Dinesh Raghu, Sachindra Joshi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04787
• Pdf URL: https://arxiv.org/pdf/2409.04787
• Copy Paste: [[2409.04787]] Selective Self-Rehearsal: A Fine-Tuning Approach to Improve Generalization in Large Language Models(https://arxiv.org/abs/2409.04787)
• Keywords: large language model
• Abstract: Fine-tuning Large Language Models (LLMs) on specific datasets is a common practice to improve performance on target tasks. However, this performance gain often leads to overfitting, where the model becomes too specialized in either the task or the characteristics of the training data, resulting in a loss of generalization. This paper introduces Selective Self-Rehearsal (SSR), a fine-tuning approach that achieves performance comparable to the standard supervised fine-tuning (SFT) while improving generalization. SSR leverages the fact that there can be multiple valid responses to a query. By utilizing the model's correct responses, SSR reduces model specialization during the fine-tuning stage. SSR first identifies the correct model responses from the training set by deploying an appropriate LLM as a judge. Then, it fine-tunes the model using the correct model responses and the gold response for the remaining samples. The effectiveness of SSR is demonstrated through experiments on the task of identifying unanswerable queries across various datasets. The results show that standard SFT can lead to an average performance drop of up to $16.7\%$ on multiple benchmarks, such as MMLU and TruthfulQA. In contrast, SSR results in close to $2\%$ drop on average, indicating better generalization capabilities compared to standard SFT.

Title: Beyond One-Time Validation: A Framework for Adaptive Validation of Prognostic and Diagnostic AI-based Medical Devices

• Authors: Florian Hellmeier, Kay Brosien, Carsten Eickhoff, Alexander Meyer
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.04794
• Pdf URL: https://arxiv.org/pdf/2409.04794
• Copy Paste: [[2409.04794]] Beyond One-Time Validation: A Framework for Adaptive Validation of Prognostic and Diagnostic AI-based Medical Devices(https://arxiv.org/abs/2409.04794)
• Keywords: robust
• Abstract: Prognostic and diagnostic AI-based medical devices hold immense promise for advancing healthcare, yet their rapid development has outpaced the establishment of appropriate validation methods. Existing approaches often fall short in addressing the complexity of practically deploying these devices and ensuring their effective, continued operation in real-world settings. Building on recent discussions around the validation of AI models in medicine and drawing from validation practices in other fields, a framework to address this gap is presented. It offers a structured, robust approach to validation that helps ensure device reliability across differing clinical environments. The primary challenges to device performance upon deployment are discussed while highlighting the impact of changes related to individual healthcare institutions and operational processes. The presented framework emphasizes the importance of repeating validation and fine-tuning during deployment, aiming to mitigate these issues while being adaptable to challenges unforeseen during device development. The framework is also positioned within the current US and EU regulatory landscapes, underscoring its practical viability and relevance considering regulatory requirements. Additionally, a practical example demonstrating potential benefits of the framework is presented. Lastly, guidance on assessing model performance is offered and the importance of involving clinical stakeholders in the validation and fine-tuning process is discussed.

Title: Phrase-Level Adversarial Training for Mitigating Bias in Neural Network-based Automatic Essay Scoring

• Authors: Haddad Philip, Tsegaye Misikir Tashu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04795
• Pdf URL: https://arxiv.org/pdf/2409.04795
• Copy Paste: [[2409.04795]] Phrase-Level Adversarial Training for Mitigating Bias in Neural Network-based Automatic Essay Scoring(https://arxiv.org/abs/2409.04795)
• Keywords: attack, robust
• Abstract: Automatic Essay Scoring (AES) is widely used to evaluate candidates for educational purposes. However, due to the lack of representative data, most existing AES systems are not robust, and their scoring predictions are biased towards the most represented data samples. In this study, we propose a model-agnostic phrase-level method to generate an adversarial essay set to address the biases and robustness of AES models. Specifically, we construct an attack test set comprising samples from the original test set and adversarially generated samples using our proposed method. To evaluate the effectiveness of the attack strategy and data augmentation, we conducted a comprehensive analysis utilizing various neural network scoring models. Experimental results show that the proposed approach significantly improves AES model performance in the presence of adversarial examples and scenarios without such attacks.

Title: SpotActor: Training-Free Layout-Controlled Consistent Image Generation

• Authors: Jiahao Wang, Caixia Yan, Weizhan Zhang, Haonan Lin, Mengmeng Wang, Guang Dai, Tieliang Gong, Hao Sun, Jingdong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04801
• Pdf URL: https://arxiv.org/pdf/2409.04801
• Copy Paste: [[2409.04801]] SpotActor: Training-Free Layout-Controlled Consistent Image Generation(https://arxiv.org/abs/2409.04801)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models significantly enhance the efficiency of artistic creation with high-fidelity image generation. However, in typical application scenarios like comic book production, they can neither place each subject into its expected spot nor maintain the consistent appearance of each subject across images. For these issues, we pioneer a novel task, Layout-to-Consistent-Image (L2CI) generation, which produces consistent and compositional images in accordance with the given layout conditions and text prompts. To accomplish this challenging task, we present a new formalization of dual energy guidance with optimization in a dual semantic-latent space and thus propose a training-free pipeline, SpotActor, which features a layout-conditioned backward update stage and a consistent forward sampling stage. In the backward stage, we innovate a nuanced layout energy function to mimic the attention activations with a sigmoid-like objective. While in the forward stage, we design Regional Interconnection Self-Attention (RISA) and Semantic Fusion Cross-Attention (SFCA) mechanisms that allow mutual interactions across images. To evaluate the performance, we present ActorBench, a specified benchmark with hundreds of reasonable prompt-box pairs stemming from object detection datasets. Comprehensive experiments are conducted to demonstrate the effectiveness of our method. The results prove that SpotActor fulfills the expectations of this task and showcases the potential for practical applications with superior layout alignment, subject consistency, prompt conformity and background diversity.

Title: SSFam: Scribble Supervised Salient Object Detection Family

• Authors: Zhengyi Liu, Sheng Deng, Xinrui Wang, Linbo Wang, Xianyong Fang, Bin Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04817
• Pdf URL: https://arxiv.org/pdf/2409.04817
• Copy Paste: [[2409.04817]] SSFam: Scribble Supervised Salient Object Detection Family(https://arxiv.org/abs/2409.04817)
• Keywords: extraction, segmentation
• Abstract: Scribble supervised salient object detection (SSSOD) constructs segmentation ability of attractive objects from surroundings under the supervision of sparse scribble labels. For the better segmentation, depth and thermal infrared modalities serve as the supplement to RGB images in the complex scenes. Existing methods specifically design various feature extraction and multi-modal fusion strategies for RGB, RGB-Depth, RGB-Thermal, and Visual-Depth-Thermal image input respectively, leading to similar model flood. As the recently proposed Segment Anything Model (SAM) possesses extraordinary segmentation and prompt interactive capability, we propose an SSSOD family based on SAM, named SSFam, for the combination input with different modalities. Firstly, different modal-aware modulators are designed to attain modal-specific knowledge which cooperates with modal-agnostic information extracted from the frozen SAM encoder for the better feature ensemble. Secondly, a siamese decoder is tailored to bridge the gap between the training with scribble prompt and the testing with no prompt for the stronger decoding ability. Our model demonstrates the remarkable performance among combinations of different modalities and refreshes the highest level of scribble supervised methods and comes close to the ones of fully supervised methods. this https URL

Title: Top-GAP: Integrating Size Priors in CNNs for more Interpretability, Robustness, and Bias Mitigation

• Authors: Lars Nieradzik, Henrike Stephani, Janis Keuper
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04819
• Pdf URL: https://arxiv.org/pdf/2409.04819
• Copy Paste: [[2409.04819]] Top-GAP: Integrating Size Priors in CNNs for more Interpretability, Robustness, and Bias Mitigation(https://arxiv.org/abs/2409.04819)
• Keywords: attack, robust, interpretability, explainability
• Abstract: This paper introduces Top-GAP, a novel regularization technique that enhances the explainability and robustness of convolutional neural networks. By constraining the spatial size of the learned feature representation, our method forces the network to focus on the most salient image regions, effectively reducing background influence. Using adversarial attacks and the Effective Receptive Field, we show that Top-GAP directs more attention towards object pixels rather than the background. This leads to enhanced interpretability and robustness. We achieve over 50% robust accuracy on CIFAR-10 with PGD $\epsilon=\frac{8}{255}$ and $20$ iterations while maintaining the original clean accuracy. Furthermore, we see increases of up to 5% accuracy against distribution shifts. Our approach also yields more precise object localization, as evidenced by up to 25% improvement in Intersection over Union (IOU) compared to methods like GradCAM and Recipro-CAM.

Title: Exploring Straightforward Conversational Red-Teaming

• Authors: George Kour, Naama Zwerdling, Marcel Zalmanovici, Ateret Anaby-Tavor, Ora Nova Fandina, Eitan Farchi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04822
• Pdf URL: https://arxiv.org/pdf/2409.04822
• Copy Paste: [[2409.04822]] Exploring Straightforward Conversational Red-Teaming(https://arxiv.org/abs/2409.04822)
• Keywords: security, attack, large language model
• Abstract: Large language models (LLMs) are increasingly used in business dialogue systems but they pose security and ethical risks. Multi-turn conversations, where context influences the model's behavior, can be exploited to produce undesired responses. In this paper, we examine the effectiveness of utilizing off-the-shelf LLMs in straightforward red-teaming approaches, where an attacker LLM aims to elicit undesired output from a target LLM, comparing both single-turn and conversational red-teaming tactics. Our experiments offer insights into various usage strategies that significantly affect their performance as red teamers. They suggest that off-the-shelf models can act as effective red teamers and even adjust their attack strategy based on past attempts, although their effectiveness decreases with greater alignment.

Title: POINTS: Improving Your Vision-language Model with Affordable Strategies

• Authors: Yuan Liu, Zhongyin Zhao, Ziyuan Zhuang, Le Tian, Xiao Zhou, Jie Zhou
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.04828
• Pdf URL: https://arxiv.org/pdf/2409.04828
• Copy Paste: [[2409.04828]] POINTS: Improving Your Vision-language Model with Affordable Strategies(https://arxiv.org/abs/2409.04828)
• Keywords: robust, large language model
• Abstract: In recent years, vision-language models have made significant strides, excelling in tasks like optical character recognition and geometric problem-solving. However, several critical issues remain: 1) Proprietary models often lack transparency about their architectures, while open-source models need more detailed ablations of their training strategies. 2) Pre-training data in open-source works is under-explored, with datasets added empirically, making the process cumbersome. 3) Fine-tuning often focuses on adding datasets, leading to diminishing returns. To address these issues, we propose the following contributions: 1) We trained a robust baseline model using the latest advancements in vision-language models, introducing effective improvements and conducting comprehensive ablation and validation for each technique. 2) Inspired by recent work on large language models, we filtered pre-training data using perplexity, selecting the lowest perplexity data for training. This approach allowed us to train on a curated 1M dataset, achieving competitive performance. 3) During visual instruction tuning, we used model soup on different datasets when adding more datasets yielded marginal improvements. These innovations resulted in a 9B parameter model that performs competitively with state-of-the-art models. Our strategies are efficient and lightweight, making them easily adoptable by the community.

Title: Reward-Directed Score-Based Diffusion Models via q-Learning

• Authors: Xuefeng Gao, Jiale Zha, Xun Yu Zhou
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2409.04832
• Pdf URL: https://arxiv.org/pdf/2409.04832
• Copy Paste: [[2409.04832]] Reward-Directed Score-Based Diffusion Models via q-Learning(https://arxiv.org/abs/2409.04832)
• Keywords: diffusion, generative
• Abstract: We propose a new reinforcement learning (RL) formulation for training continuous-time score-based diffusion models for generative AI to generate samples that maximize reward functions while keeping the generated distributions close to the unknown target data distributions. Different from most existing studies, our formulation does not involve any pretrained model for the unknown score functions of the noise-perturbed data distributions. We present an entropy-regularized continuous-time RL problem and show that the optimal stochastic policy has a Gaussian distribution with a known covariance matrix. Based on this result, we parameterize the mean of Gaussian policies and develop an actor-critic type (little) q-learning algorithm to solve the RL problem. A key ingredient in our algorithm design is to obtain noisy observations from the unknown score function via a ratio estimator. Numerically, we show the effectiveness of our approach by comparing its performance with two state-of-the-art RL methods that fine-tune pretrained models. Finally, we discuss extensions of our RL formulation to probability flow ODE implementation of diffusion models and to conditional diffusion models.

Title: Achieving Peak Performance for Large Language Models: A Systematic Review

• Authors: Zhyar Rzgar K Rostam, Sándor Szénási, Gábor Kertész
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04833
• Pdf URL: https://arxiv.org/pdf/2409.04833
• Copy Paste: [[2409.04833]] Achieving Peak Performance for Large Language Models: A Systematic Review(https://arxiv.org/abs/2409.04833)
• Keywords: large language model
• Abstract: In recent years, large language models (LLMs) have achieved remarkable success in natural language processing (NLP). LLMs require an extreme amount of parameters to attain high performance. As models grow into the trillion-parameter range, computational and memory costs increase significantly. This makes it difficult for many researchers to access the resources needed to train or apply these models. Optimizing LLM performance involves two main approaches: fine-tuning pre-trained models for specific tasks to achieve state-of-the-art performance, and reducing costs or improving training time while maintaining similar performance. This paper presents a systematic literature review (SLR) following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. We reviewed 65 publications out of 983 from 2017 to December 2023, retrieved from 5 databases. The study presents methods to optimize and accelerate LLMs while achieving cutting-edge results without sacrificing accuracy. We begin with an overview of the development of language modeling, followed by a detailed explanation of commonly used frameworks and libraries, and a taxonomy for improving and speeding up LLMs based on three classes: LLM training, LLM inference, and system serving. We then delve into recent optimization and acceleration strategies such as training optimization, hardware optimization, scalability and reliability, accompanied by the taxonomy and categorization of these strategies. Finally, we provide an in-depth comparison of each class and strategy, with two case studies on optimizing model training and enhancing inference efficiency. These case studies showcase practical approaches to address LLM resource limitations while maintaining performance.

Title: Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation

• Authors: Jiaxin Cheng, Zixu Zhao, Tong He, Tianjun Xiao, Yicong Zhou, Zheng Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04847
• Pdf URL: https://arxiv.org/pdf/2409.04847
• Copy Paste: [[2409.04847]] Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation(https://arxiv.org/abs/2409.04847)
• Keywords: generative
• Abstract: Recent advancements in generative models have significantly enhanced their capacity for image generation, enabling a wide range of applications such as image editing, completion and video editing. A specialized area within generative modeling is layout-to-image (L2I) generation, where predefined layouts of objects guide the generative process. In this study, we introduce a novel regional cross-attention module tailored to enrich layout-to-image generation. This module notably improves the representation of layout regions, particularly in scenarios where existing methods struggle with highly complex and detailed textual descriptions. Moreover, while current open-vocabulary L2I methods are trained in an open-set setting, their evaluations often occur in closed-set environments. To bridge this gap, we propose two metrics to assess L2I performance in open-vocabulary scenarios. Additionally, we conduct a comprehensive user study to validate the consistency of these metrics with human preferences.

Title: FedModule: A Modular Federated Learning Framework

• Authors: Chuyi Chen, Zhe Zhang, Yanchao Zhao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04849
• Pdf URL: https://arxiv.org/pdf/2409.04849
• Copy Paste: [[2409.04849]] FedModule: A Modular Federated Learning Framework(https://arxiv.org/abs/2409.04849)
• Keywords: robust, federate
• Abstract: Federated learning (FL) has been widely adopted across various applications, such as healthcare, finance, and smart cities. However, as experimental scenarios become more complex, existing FL frameworks and benchmarks have struggled to keep pace. This paper introduces FedModule, a flexible and extensible FL experimental framework that has been open-sourced to support diverse FL paradigms and provide comprehensive benchmarks for complex experimental scenarios. FedModule adheres to the "one code, all scenarios" principle and employs a modular design that breaks the FL process into individual components, allowing for the seamless integration of different FL paradigms. The framework supports synchronous, asynchronous, and personalized federated learning, with over 20 implemented algorithms. Experiments conducted on public datasets demonstrate the flexibility and extensibility of FedModule. The framework offers multiple execution modes-including linear, threaded, process-based, and distributed-enabling users to tailor their setups to various experimental needs. Additionally, FedModule provides extensive logging and testing capabilities, which facilitate detailed performance analysis of FL algorithms. Comparative evaluations against existing FL toolkits, such as TensorFlow Federated, PySyft, Flower, and FLGo, highlight FedModule's superior scalability, flexibility, and comprehensive benchmark support. By addressing the limitations of current FL frameworks, FedModule marks a significant advancement in FL experimentation, providing researchers and practitioners with a robust tool for developing and evaluating FL algorithms across a wide range of scenarios.

Title: AdaptiveFusion: Adaptive Multi-Modal Multi-View Fusion for 3D Human Body Reconstruction

• Authors: Anjun Chen, Xiangyu Wang, Zhi Xu, Kun Shi, Yan Qin, Yuchi Huo, Jiming Chen, Qi Ye
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04851
• Pdf URL: https://arxiv.org/pdf/2409.04851
• Copy Paste: [[2409.04851]] AdaptiveFusion: Adaptive Multi-Modal Multi-View Fusion for 3D Human Body Reconstruction(https://arxiv.org/abs/2409.04851)
• Keywords: robust, transformer
• Abstract: Recent advancements in sensor technology and deep learning have led to significant progress in 3D human body reconstruction. However, most existing approaches rely on data from a specific sensor, which can be unreliable due to the inherent limitations of individual sensing modalities. On the other hand, existing multi-modal fusion methods generally require customized designs based on the specific sensor combinations or setups, which limits the flexibility and generality of these methods. Furthermore, conventional point-image projection-based and Transformer-based fusion networks are susceptible to the influence of noisy modalities and sensor poses. To address these limitations and achieve robust 3D human body reconstruction in various conditions, we propose AdaptiveFusion, a generic adaptive multi-modal multi-view fusion framework that can effectively incorporate arbitrary combinations of uncalibrated sensor inputs. By treating different modalities from various viewpoints as equal tokens, and our handcrafted modality sampling module by leveraging the inherent flexibility of Transformer models, AdaptiveFusion is able to cope with arbitrary numbers of inputs and accommodate noisy modalities with only a single training network. Extensive experiments on large-scale human datasets demonstrate the effectiveness of AdaptiveFusion in achieving high-quality 3D human body reconstruction in various environments. In addition, our method achieves superior accuracy compared to state-of-the-art fusion methods.

Title: Contrastive Disentangling: Fine-grained representation learning through multi-level contrastive learning without class priors

• Authors: Houwang Jiang, Zhuxian Liu, Guodong Liu, Xiaolong Liu, Shihua Zhan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04867
• Pdf URL: https://arxiv.org/pdf/2409.04867
• Copy Paste: [[2409.04867]] Contrastive Disentangling: Fine-grained representation learning through multi-level contrastive learning without class priors(https://arxiv.org/abs/2409.04867)
• Keywords: extraction
• Abstract: Recent advancements in unsupervised representation learning often leverage class information to enhance feature extraction and clustering performance. However, this reliance on class priors limits the applicability of such methods in real-world scenarios where class information is unavailable or ambiguous. In this paper, we propose Contrastive Disentangling (CD), a simple and effective framework that learns representations without any reliance on class priors. Our framework employs a multi-level contrastive learning strategy that combines instance-level and feature-level losses with a normalized entropy loss to learn semantically rich and fine-grained representations. Specifically, (1) the instance-level contrastive loss encourages the separation of feature representations for different samples, (2) the feature-level contrastive loss promotes independence among the feature head predictions, and (3) the normalized entropy loss encourages the feature heads to capture meaningful and prevalent attributes from the data. These components work together to enable CD to significantly outperform existing methods, as demonstrated by extensive experiments on benchmark datasets including CIFAR-10, CIFAR-100, STL-10, and ImageNet-10, particularly in scenarios where class priors are absent. The code is available at this https URL.

Title: Strong Privacy-Preserving Universally Composable AKA Protocol with Seamless Handover Support for Mobile Virtual Network Operator

• Authors: Rabiah Alnashwan, Yang Yang, Yilu Dong, Prosanta Gope, Behzad Abdolmaleki, Syed Rafiul Hussain
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.04877
• Pdf URL: https://arxiv.org/pdf/2409.04877
• Copy Paste: [[2409.04877]] Strong Privacy-Preserving Universally Composable AKA Protocol with Seamless Handover Support for Mobile Virtual Network Operator(https://arxiv.org/abs/2409.04877)
• Keywords: security, privacy
• Abstract: Consumers seeking a new mobile plan have many choices in the present mobile landscape. The Mobile Virtual Network Operator (MVNO) has recently gained considerable attention among these options. MVNOs offer various benefits, making them an appealing choice for a majority of consumers. These advantages encompass flexibility, access to cutting-edge technologies, enhanced coverage, superior customer service, and substantial cost savings. Even though MVNO offers several advantages, it also creates some security and privacy concerns for the customer simultaneously. For instance, in the existing solution, MVNO needs to hand over all the sensitive details, including the users' identities and master secret keys of their customers, to a mobile operator (MNO) to validate the customers while offering any services. This allows MNOs to have unrestricted access to the MVNO subscribers' location and mobile data, including voice calls, SMS, and Internet, which the MNOs frequently sell to third parties (e.g., advertisement companies and surveillance agencies) for more profit. Although critical for mass users, such privacy loss has been historically ignored due to the lack of practical and privacy-preserving solutions for registration and handover procedures in cellular networks. In this paper, we propose a universally composable authentication and handover scheme with strong user privacy support, where each MVNO user can validate a mobile operator (MNO) and vice-versa without compromising user anonymity and unlinkability support. Here, we anticipate that our proposed solution will most likely be deployed by the MVNO(s) to ensure enhanced privacy support to their customer(s).

Title: Plug-and-Hide: Provable and Adjustable Diffusion Generative Steganography

• Authors: Jiahao Zhu, Zixuan Chen, Lingxiao Yang, Xiaohua Xie, Yi Zhou
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.04878
• Pdf URL: https://arxiv.org/pdf/2409.04878
• Copy Paste: [[2409.04878]] Plug-and-Hide: Provable and Adjustable Diffusion Generative Steganography(https://arxiv.org/abs/2409.04878)
• Keywords: security, extraction, diffusion, generative
• Abstract: Generative Steganography (GS) is a novel technique that utilizes generative models to conceal messages without relying on cover images. Contemporary GS algorithms leverage the powerful generative capabilities of Diffusion Models (DMs) to create high-fidelity stego images. However, these algorithms, while yielding relatively satisfactory generation outcomes and message extraction accuracy, significantly alter modifications to the initial Gaussian noise of DMs, thereby compromising steganographic security. In this paper, we rethink the trade-off among image quality, steganographic security, and message extraction accuracy within Diffusion Generative Steganography (DGS) settings. Our findings reveal that the normality of initial noise of DMs is crucial to these factors and can offer theoretically grounded guidance for DGS design. Based on this insight, we propose a Provable and Adjustable Message Mapping (PA-B2G) approach. It can, on one hand, theoretically guarantee reversible encoding of bit messages from arbitrary distributions into standard Gaussian noise for DMs. On the other hand, its adjustability provides a more natural and fine-grained way to trade off image quality, steganographic security, and message extraction accuracy. By integrating PA-B2G with a probability flow ordinary differential equation, we establish an invertible mapping between secret messages and stego images. PA-B2G can be seamlessly incorporated with most mainstream DMs, such as the Stable Diffusion, without necessitating additional training or fine-tuning. Comprehensive experiments corroborate our theoretical insights regarding the trade-off in DGS settings and demonstrate the effectiveness of our DGS algorithm in producing high-quality stego images while preserving desired levels of steganographic security and extraction accuracy.

Title: A Quantitative Approach for Evaluating Disease Focus and Interpretability of Deep Learning Models for Alzheimer's Disease Classification

• Authors: Thomas Yu Chow Tam, Litian Liang, Ke Chen, Haohan Wang, Wei Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04888
• Pdf URL: https://arxiv.org/pdf/2409.04888
• Copy Paste: [[2409.04888]] A Quantitative Approach for Evaluating Disease Focus and Interpretability of Deep Learning Models for Alzheimer's Disease Classification(https://arxiv.org/abs/2409.04888)
• Keywords: interpretability, segmentation
• Abstract: Deep learning (DL) models have shown significant potential in Alzheimer's Disease (AD) classification. However, understanding and interpreting these models remains challenging, which hinders the adoption of these models in clinical practice. Techniques such as saliency maps have been proven effective in providing visual and empirical clues about how these models work, but there still remains a gap in understanding which specific brain regions DL models focus on and whether these brain regions are pathologically associated with AD. To bridge such gap, in this study, we developed a quantitative disease-focusing strategy to first enhance the interpretability of DL models using saliency maps and brain segmentations; then we propose a disease-focus (DF) score that quantifies how much a DL model focuses on brain areas relevant to AD pathology based on clinically known MRI-based pathological regions of AD. Using this strategy, we compared several state-of-the-art DL models, including a baseline 3D ResNet model, a pretrained MedicalNet model, and a MedicalNet with data augmentation to classify patients with AD vs. cognitive normal patients using MRI data; then we evaluated these models in terms of their abilities to focus on disease-relevant regions. Our results show interesting disease-focusing patterns with different models, particularly characteristic patterns with the pretrained models and data augmentation, and also provide insight into their classification performance. These results suggest that the approach we developed for quantitatively assessing the abilities of DL models to focus on disease-relevant regions may help improve interpretability of these models for AD classification and facilitate their adoption for AD diagnosis in clinical practice. The code is publicly available at this https URL.

Title: Unlocking the Potential of Model Calibration in Federated Learning

• Authors: Yun-Wei Chu, Dong-Jun Han, Seyyedali Hosseinalipour, Christopher Brinton
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04901
• Pdf URL: https://arxiv.org/pdf/2409.04901
• Copy Paste: [[2409.04901]] Unlocking the Potential of Model Calibration in Federated Learning(https://arxiv.org/abs/2409.04901)
• Keywords: federate
• Abstract: Over the past several years, various federated learning (FL) methodologies have been developed to improve model accuracy, a primary performance metric in machine learning. However, to utilize FL in practical decision-making scenarios, beyond considering accuracy, the trained model must also have a reliable confidence in each of its predictions, an aspect that has been largely overlooked in existing FL research. Motivated by this gap, we propose Non-Uniform Calibration for Federated Learning (NUCFL), a generic framework that integrates FL with the concept of model calibration. The inherent data heterogeneity in FL environments makes model calibration particularly difficult, as it must ensure reliability across diverse data distributions and client conditions. Our NUCFL addresses this challenge by dynamically adjusting the model calibration objectives based on statistical relationships between each client's local model and the global model in FL. In particular, NUCFL assesses the similarity between local and global model relationships, and controls the penalty term for the calibration loss during client-side local training. By doing so, NUCFL effectively aligns calibration needs for the global model in heterogeneous FL settings while not sacrificing accuracy. Extensive experiments show that NUCFL offers flexibility and effectiveness across various FL algorithms, enhancing accuracy as well as model calibration.

Title: Activation Function Optimization Scheme for Image Classification

• Authors: Abdur Rahman, Lu He, Haifeng Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04915
• Pdf URL: https://arxiv.org/pdf/2409.04915
• Copy Paste: [[2409.04915]] Activation Function Optimization Scheme for Image Classification(https://arxiv.org/abs/2409.04915)
• Keywords: transformer
• Abstract: Activation function has a significant impact on the dynamics, convergence, and performance of deep neural networks. The search for a consistent and high-performing activation function has always been a pursuit during deep learning model development. Existing state-of-the-art activation functions are manually designed with human expertise except for Swish. Swish was developed using a reinforcement learning-based search strategy. In this study, we propose an evolutionary approach for optimizing activation functions specifically for image classification tasks, aiming to discover functions that outperform current state-of-the-art options. Through this optimization framework, we obtain a series of high-performing activation functions denoted as Exponential Error Linear Unit (EELU). The developed activation functions are evaluated for image classification tasks from two perspectives: (1) five state-of-the-art neural network architectures, such as ResNet50, AlexNet, VGG16, MobileNet, and Compact Convolutional Transformer which cover computationally heavy to light neural networks, and (2) eight standard datasets, including CIFAR10, Imagenette, MNIST, Fashion MNIST, Beans, Colorectal Histology, CottonWeedID15, and TinyImageNet which cover from typical machine vision benchmark, agricultural image applications to medical image applications. Finally, we statistically investigate the generalization of the resultant activation functions developed through the optimization scheme. With a Friedman test, we conclude that the optimization scheme is able to generate activation functions that outperform the existing standard ones in 92.8% cases among 28 different cases studied, and $-x\cdot erf(e^{-x})$ is found to be the best activation function for image classification generated by the optimization scheme.

Title: Training-free ZS-CIR via Weighted Modality Fusion and Similarity

• Authors: Ren-Di Wu, Yu-Yen Lin, Huei-Fang Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04918
• Pdf URL: https://arxiv.org/pdf/2409.04918
• Copy Paste: [[2409.04918]] Training-free ZS-CIR via Weighted Modality Fusion and Similarity(https://arxiv.org/abs/2409.04918)
• Keywords: large language model
• Abstract: Composed image retrieval (CIR), which formulates the query as a combination of a reference image and modified text, has emerged as a new form of image search due to its enhanced ability to capture users' intentions. However, training a CIR model in a supervised manner typically requires labor-intensive collection of (reference image, text modifier, target image) triplets. While existing zero-shot CIR (ZS-CIR) methods eliminate the need for training on specific downstream datasets, they still require additional pretraining with large-scale image-text pairs. In this paper, we introduce a training-free approach for ZS-CIR. Our approach, \textbf{Wei}ghted \textbf{Mo}dality fusion and similarity for \textbf{CIR} (WeiMoCIR), operates under the assumption that image and text modalities can be effectively combined using a simple weighted average. This allows the query representation to be constructed directly from the reference image and text modifier. To further enhance retrieval performance, we employ multimodal large language models (MLLMs) to generate image captions for the database images and incorporate these textual captions into the similarity computation by combining them with image information using a weighted average. Our approach is simple, easy to implement, and its effectiveness is validated through experiments on the FashionIQ and CIRR datasets.

Title: Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue

• Authors: Junkai Wu, Xulin Fan, Bo-Ru Lu, Xilin Jiang, Nima Mesgarani, Mark Hasegawa-Johnson, Mari Ostendorf
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.04927
• Pdf URL: https://arxiv.org/pdf/2409.04927
• Copy Paste: [[2409.04927]] Just ASR + LLM? A Study on Speech Large Language Models' Ability to Identify and Understand Speaker in Spoken Dialogue(https://arxiv.org/abs/2409.04927)
• Keywords: large language model
• Abstract: In recent years, we have observed a rapid advancement in speech language models (SpeechLLMs), catching up with humans' listening and reasoning abilities. Remarkably, SpeechLLMs have demonstrated impressive spoken dialogue question-answering (SQA) performance in benchmarks like Gaokao, the English listening test of the college entrance exam in China, which seemingly requires understanding both the spoken content and voice characteristics of speakers in a conversation. However, after carefully examining Gaokao's questions, we find the correct answers to many questions can be inferred from the conversation context alone without identifying the speaker asked in the question. Our evaluation of state-of-the-art models Qwen-Audio and WavLLM in both Gaokao and our proposed "What Do You Like?" dataset shows a significantly higher accuracy in these context-based questions than in identity-critical questions, which can only be answered correctly with correct speaker identification. Our results and analysis suggest that when solving SQA, the current SpeechLLMs exhibit limited speaker awareness from the audio and behave similarly to an LLM reasoning from the conversation transcription without sound. We propose that our definitions and automated classification of context-based and identity-critical questions could offer a more accurate evaluation framework of SpeechLLMs in SQA tasks.

Title: PIXHELL Attack: Leaking Sensitive Information from Air-Gap Computers via `Singing Pixels'

• Authors: Mordechai Guri
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.04930
• Pdf URL: https://arxiv.org/pdf/2409.04930
• Copy Paste: [[2409.04930]] PIXHELL Attack: Leaking Sensitive Information from Air-Gap Computers via `Singing Pixels'(https://arxiv.org/abs/2409.04930)
• Keywords: defense, attack, steal
• Abstract: Air-gapped systems are disconnected from the Internet and other networks because they contain or process sensitive data. However, it is known that attackers can use computer speakers to leak data via sound to circumvent the air-gap defense. To cope with this threat, when highly sensitive data is involved, the prohibition of loudspeakers or audio hardware might be enforced. This measure is known as an `audio gap'. In this paper, we present PIXHELL, a new type of covert channel attack allowing hackers to leak information via noise generated by the pixels on the screen. No audio hardware or loudspeakers is required. Malware in the air-gap and audio-gap computers generates crafted pixel patterns that produce noise in the frequency range of 0 - 22 kHz. The malicious code exploits the sound generated by coils and capacitors to control the frequencies emanating from the screen. Acoustic signals can encode and transmit sensitive information. We present the adversarial attack model, cover related work, and provide technical background. We discuss bitmap generation and correlated acoustic signals and provide implementation details on the modulation and demodulation process. We evaluated the covert channel on various screens and tested it with different types of information. We also discuss \textit{evasion and stealth} using low-brightness patterns that appear like black, turned-off screens. Finally, we propose a set of countermeasures. Our test shows that with a PIXHELL attack, textual and binary data can be exfiltrated from air-gapped, audio-gapped computers at a distance of 2m via sound modulated from LCD screens.

Title: Noise-Based Authentication: Is It Secure?

• Authors: Sarah A. Flanery, Christiana Chamon
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.04931
• Pdf URL: https://arxiv.org/pdf/2409.04931
• Copy Paste: [[2409.04931]] Noise-Based Authentication: Is It Secure?(https://arxiv.org/abs/2409.04931)
• Keywords: secure, robust, biometric
• Abstract: This paper introduces a three-point biometric authentication system for a blockchain-based decentralized identity network. We use existing biometric authentication systems to demonstrate the unique noise fingerprints that belong to each individual human and the respective information leak from the biological characteristics. We then propose the concept of using unique thermal noise amplitudes generated by each user and explore the open questions regarding the robustness of unconditionally secure authentication.

Title: Maximizing Relation Extraction Potential: A Data-Centric Study to Unveil Challenges and Opportunities

• Authors: Anushka Swarup, Avanti Bhandarkar, Olivia P. Dizon-Paradis, Ronald Wilson, Damon L. Woodard
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04934
• Pdf URL: https://arxiv.org/pdf/2409.04934
• Copy Paste: [[2409.04934]] Maximizing Relation Extraction Potential: A Data-Centric Study to Unveil Challenges and Opportunities(https://arxiv.org/abs/2409.04934)
• Keywords: robust, extraction, large language model
• Abstract: Relation extraction is a Natural Language Processing task aiming to extract relationships from textual data. It is a critical step for information extraction. Due to its wide-scale applicability, research in relation extraction has rapidly scaled to using highly advanced neural networks. Despite their computational superiority, modern relation extractors fail to handle complicated extraction scenarios. However, a comprehensive performance analysis of the state-of-the-art relation extractors that compile these challenges has been missing from the literature, and this paper aims to bridge this gap. The goal has been to investigate the possible data-centric characteristics that impede neural relation extraction. Based on extensive experiments conducted using 15 state-of-the-art relation extraction algorithms ranging from recurrent architectures to large language models and seven large-scale datasets, this research suggests that modern relation extractors are not robust to complex data and relation characteristics. It emphasizes pivotal issues, such as contextual ambiguity, correlating relations, long-tail data, and fine-grained relation distributions. In addition, it sets a marker for future directions to alleviate these issues, thereby proving to be a critical resource for novice and advanced researchers. Efficient handling of the challenges described can have significant implications for the field of information extraction, which is a critical part of popular systems such as search engines and chatbots. Data and relevant code can be found at this https URL.

Title: Fast Deep Predictive Coding Networks for Videos Feature Extraction without Labels

• Authors: Wenqian Xue, Chi Ding, Jose Principe
• Subjects: cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.04945
• Pdf URL: https://arxiv.org/pdf/2409.04945
• Copy Paste: [[2409.04945]] Fast Deep Predictive Coding Networks for Videos Feature Extraction without Labels(https://arxiv.org/abs/2409.04945)
• Keywords: robust, extraction, explainability
• Abstract: Brain-inspired deep predictive coding networks (DPCNs) effectively model and capture video features through a bi-directional information flow, even without labels. They are based on an overcomplete description of video scenes, and one of the bottlenecks has been the lack of effective sparsification techniques to find discriminative and robust dictionaries. FISTA has been the best alternative. This paper proposes a DPCN with a fast inference of internal model variables (states and causes) that achieves high sparsity and accuracy of feature clustering. The proposed unsupervised learning procedure, inspired by adaptive dynamic programming with a majorization-minimization framework, and its convergence are rigorously analyzed. Experiments in the data sets CIFAR-10, Super Mario Bros video game, and Coil-100 validate the approach, which outperforms previous versions of DPCNs on learning rate, sparsity ratio, and feature clustering accuracy. Because of DCPN's solid foundation and explainability, this advance opens the door for general applications in object recognition in video without labels.

Title: DDNet: Deformable Convolution and Dense FPN for Surface Defect Detection in Recycled Books

• Authors: Jun Yu, WenJian Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04958
• Pdf URL: https://arxiv.org/pdf/2409.04958
• Copy Paste: [[2409.04958]] DDNet: Deformable Convolution and Dense FPN for Surface Defect Detection in Recycled Books(https://arxiv.org/abs/2409.04958)
• Keywords: robust, extraction
• Abstract: Recycled and recirculated books, such as ancient texts and reused textbooks, hold significant value in the second-hand goods market, with their worth largely dependent on surface preservation. However, accurately assessing surface defects is challenging due to the wide variations in shape, size, and the often imprecise detection of defects. To address these issues, we propose DDNet, an innovative detection model designed to enhance defect localization and classification. DDNet introduces a surface defect feature extraction module based on a deformable convolution operator (DC) and a densely connected FPN module (DFPN). The DC module dynamically adjusts the convolution grid to better align with object contours, capturing subtle shape variations and improving boundary delineation and prediction accuracy. Meanwhile, DFPN leverages dense skip connections to enhance feature fusion, constructing a hierarchical structure that generates multi-resolution, high-fidelity feature maps, thus effectively detecting defects of various sizes. In addition to the model, we present a comprehensive dataset specifically curated for surface defect detection in recycled and recirculated books. This dataset encompasses a diverse range of defect types, shapes, and sizes, making it ideal for evaluating the robustness and effectiveness of defect detection models. Through extensive evaluations, DDNet achieves precise localization and classification of surface defects, recording a mAP value of 46.7% on our proprietary dataset - an improvement of 14.2% over the baseline model - demonstrating its superior detection capabilities.

Title: GS-PT: Exploiting 3D Gaussian Splatting for Comprehensive Point Cloud Understanding via Self-supervised Learning

• Authors: Keyi Liu, Yeqi Luo, Weidong Yang, Jingyi Xu, Zhijun Li, Wen-Ming Chen, Ben Fei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04963
• Pdf URL: https://arxiv.org/pdf/2409.04963
• Copy Paste: [[2409.04963]] GS-PT: Exploiting 3D Gaussian Splatting for Comprehensive Point Cloud Understanding via Self-supervised Learning(https://arxiv.org/abs/2409.04963)
• Keywords: transformer, segmentation
• Abstract: Self-supervised learning of point cloud aims to leverage unlabeled 3D data to learn meaningful representations without reliance on manual annotations. However, current approaches face challenges such as limited data diversity and inadequate augmentation for effective feature learning. To address these challenges, we propose GS-PT, which integrates 3D Gaussian Splatting (3DGS) into point cloud self-supervised learning for the first time. Our pipeline utilizes transformers as the backbone for self-supervised pre-training and introduces novel contrastive learning tasks through 3DGS. Specifically, the transformers aim to reconstruct the masked point cloud. 3DGS utilizes multi-view rendered images as input to generate enhanced point cloud distributions and novel view images, facilitating data augmentation and cross-modal contrastive learning. Additionally, we incorporate features from depth maps. By optimizing these tasks collectively, our method enriches the tri-modal self-supervised learning process, enabling the model to leverage the correlation across 3D point clouds and 2D images from various modalities. We freeze the encoder after pre-training and test the model's performance on multiple downstream tasks. Experimental results indicate that GS-PT outperforms the off-the-shelf self-supervised learning methods on various downstream tasks including 3D object classification, real-world classifications, and few-shot learning and segmentation.

Title: Evaluation of Google Translate for Mandarin Chinese translation using sentiment and semantic analysis

• Authors: Xuechun Wang, Rodney Beard, Rohitash Chandra
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04964
• Pdf URL: https://arxiv.org/pdf/2409.04964
• Copy Paste: [[2409.04964]] Evaluation of Google Translate for Mandarin Chinese translation using sentiment and semantic analysis(https://arxiv.org/abs/2409.04964)
• Keywords: large language model
• Abstract: Machine translation using large language models (LLMs) is having a significant global impact, making communication easier. Mandarin Chinese is the official language used for communication by the government, education institutes, and media in China. In this study, we provide an automated assessment of machine translation models with human experts using sentiment and semantic analysis. In order to demonstrate our framework, we select classic early twentieth-century novel 'The True Story of Ah Q' with selected Mandarin Chinese to English translations. We also us Google Translate to generate the given text into English and then conduct a chapter-wise sentiment analysis and semantic analysis to compare the extracted sentiments across the different translations. We utilise LLMs for semantic and sentiment analysis. Our results indicate that the precision of Google Translate differs both in terms of semantic and sentiment analysis when compared to human expert translations. We find that Google Translate is unable to translate some of the specific words or phrases in Chinese, such as Chinese traditional allusions. The mistranslations have to its lack of contextual significance and historical knowledge of China. Thus, this framework brought us some new insights about machine translation for Chinese Mandarin. The future work can explore other languages or types of texts with this framework.

Title: Natias: Neuron Attribution based Transferable Image Adversarial Steganography

• Authors: Zexin Fan, Kejiang Chen, Kai Zeng, Jiansong Zhang, Weiming Zhang, Nenghai Yu
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.04968
• Pdf URL: https://arxiv.org/pdf/2409.04968
• Copy Paste: [[2409.04968]] Natias: Neuron Attribution based Transferable Image Adversarial Steganography(https://arxiv.org/abs/2409.04968)
• Keywords: security
• Abstract: Image steganography is a technique to conceal secret messages within digital images. Steganalysis, on the contrary, aims to detect the presence of secret messages within images. Recently, deep-learning-based steganalysis methods have achieved excellent detection performance. As a countermeasure, adversarial steganography has garnered considerable attention due to its ability to effectively deceive deep-learning-based steganalysis. However, steganalysts often employ unknown steganalytic models for detection. Therefore, the ability of adversarial steganography to deceive non-target steganalytic models, known as transferability, becomes especially important. Nevertheless, existing adversarial steganographic methods do not consider how to enhance transferability. To address this issue, we propose a novel adversarial steganographic scheme named Natias. Specifically, we first attribute the output of a steganalytic model to each neuron in the target middle layer to identify critical features. Next, we corrupt these critical features that may be adopted by diverse steganalytic models. Consequently, it can promote the transferability of adversarial steganography. Our proposed method can be seamlessly integrated with existing adversarial steganography frameworks. Thorough experimental analyses affirm that our proposed technique possesses improved transferability when contrasted with former approaches, and it attains heightened security in retraining scenarios.

Title: Balancing Security and Accuracy: A Novel Federated Learning Approach for Cyberattack Detection in Blockchain Networks

• Authors: Tran Viet Khoa, Mohammad Abu Alsheikh, Yibeltal Alem, Dinh Thai Hoang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04972
• Pdf URL: https://arxiv.org/pdf/2409.04972
• Copy Paste: [[2409.04972]] Balancing Security and Accuracy: A Novel Federated Learning Approach for Cyberattack Detection in Blockchain Networks(https://arxiv.org/abs/2409.04972)
• Keywords: secure, security, privacy, protect, attack, federate
• Abstract: This paper presents a novel Collaborative Cyberattack Detection (CCD) system aimed at enhancing the security of blockchain-based data-sharing networks by addressing the complex challenges associated with noise addition in federated learning models. Leveraging the theoretical principles of differential privacy, our approach strategically integrates noise into trained sub-models before reconstructing the global model through transmission. We systematically explore the effects of various noise types, i.e., Gaussian, Laplace, and Moment Accountant, on key performance metrics, including attack detection accuracy, deep learning model convergence time, and the overall runtime of global model generation. Our findings reveal the intricate trade-offs between ensuring data privacy and maintaining system performance, offering valuable insights into optimizing these parameters for diverse CCD environments. Through extensive simulations, we provide actionable recommendations for achieving an optimal balance between data protection and system efficiency, contributing to the advancement of secure and reliable blockchain networks.

Title: PatchAlign:Fair and Accurate Skin Disease Image Classification by Alignment with Clinical Labels

• Authors: Aayushman, Hemanth Gaddey, Vidhi Mittal, Manisha Chawla, Gagan Raj Gupta
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04975
• Pdf URL: https://arxiv.org/pdf/2409.04975
• Copy Paste: [[2409.04975]] PatchAlign:Fair and Accurate Skin Disease Image Classification by Alignment with Clinical Labels(https://arxiv.org/abs/2409.04975)
• Keywords: robust, fair
• Abstract: Deep learning models have achieved great success in automating skin lesion diagnosis. However, the ethnic disparity in these models' predictions needs to be addressed before deploying them. We introduce a novel approach, PatchAlign, to enhance skin condition image classification accuracy and fairness by aligning with clinical text representations of skin conditions. PatchAlign uses Graph Optimal Transport (GOT) Loss as a regularizer to perform cross-domain alignment. The representations obtained are robust and generalize well across skin tones, even with limited training samples. To reduce the effect of noise and artifacts in clinical dermatology images, we propose a learnable Masked Graph Optimal Transport for cross-domain alignment that further improves fairness metrics. We compare our model to the state-of-the-art FairDisCo on two skin lesion datasets with different skin types: Fitzpatrick17k and Diverse Dermatology Images (DDI). PatchAlign enhances the accuracy of skin condition image classification by 2.8% (in-domain) and 6.2% (out-domain) on Fitzpatrick17k, and 4.2% (in-domain) on DDI compared to FairDisCo. Additionally, it consistently improves the fairness of true positive rates across skin tones. The source code for the implementation is available at the following GitHub repository: this https URL, enabling easy reproduction and further experimentation.

Title: RCBEVDet++: Toward High-accuracy Radar-Camera Fusion 3D Perception Network

• Authors: Zhiwei Lin, Zhe Liu, Yongtao Wang, Le Zhang, Ce Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04979
• Pdf URL: https://arxiv.org/pdf/2409.04979
• Copy Paste: [[2409.04979]] RCBEVDet++: Toward High-accuracy Radar-Camera Fusion 3D Perception Network(https://arxiv.org/abs/2409.04979)
• Keywords: segmentation
• Abstract: Perceiving the surrounding environment is a fundamental task in autonomous driving. To obtain highly accurate perception results, modern autonomous driving systems typically employ multi-modal sensors to collect comprehensive environmental data. Among these, the radar-camera multi-modal perception system is especially favored for its excellent sensing capabilities and cost-effectiveness. However, the substantial modality differences between radar and camera sensors pose challenges in fusing information. To address this problem, this paper presents RCBEVDet, a radar-camera fusion 3D object detection framework. Specifically, RCBEVDet is developed from an existing camera-based 3D object detector, supplemented by a specially designed radar feature extractor, RadarBEVNet, and a Cross-Attention Multi-layer Fusion (CAMF) module. Firstly, RadarBEVNet encodes sparse radar points into a dense bird's-eye-view (BEV) feature using a dual-stream radar backbone and a Radar Cross Section aware BEV encoder. Secondly, the CAMF module utilizes a deformable attention mechanism to align radar and camera BEV features and adopts channel and spatial fusion layers to fuse them. To further enhance RCBEVDet's capabilities, we introduce RCBEVDet++, which advances the CAMF through sparse fusion, supports query-based multi-view camera perception models, and adapts to a broader range of perception tasks. Extensive experiments on the nuScenes show that our method integrates seamlessly with existing camera-based 3D perception models and improves their performance across various perception tasks. Furthermore, our method achieves state-of-the-art radar-camera fusion results in 3D object detection, BEV semantic segmentation, and 3D multi-object tracking tasks. Notably, with ViT-L as the image backbone, RCBEVDet++ achieves 72.73 NDS and 67.34 mAP in 3D object detection without test-time augmentation or model ensembling.

Title: 2DSig-Detect: a semi-supervised framework for anomaly detection on image data using 2D-signatures

• Authors: Xinheng Xie, Kureha Yamaguchi, Margaux Leblanc, Simon Malzard, Varun Chhabra, Victoria Nockles, Yue Wu
• Subjects: cs.CV, math.PR, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.04982
• Pdf URL: https://arxiv.org/pdf/2409.04982
• Copy Paste: [[2409.04982]] 2DSig-Detect: a semi-supervised framework for anomaly detection on image data using 2D-signatures(https://arxiv.org/abs/2409.04982)
• Keywords: security, attack
• Abstract: The rapid advancement of machine learning technologies raises questions about the security of machine learning models, with respect to both training-time (poisoning) and test-time (evasion, impersonation, and inversion) attacks. Models performing image-related tasks, e.g. detection, and classification, are vulnerable to adversarial attacks that can degrade their performance and produce undesirable outcomes. This paper introduces a novel technique for anomaly detection in images called 2DSig-Detect, which uses a 2D-signature-embedded semi-supervised framework rooted in rough path theory. We demonstrate our method in adversarial settings for training-time and test-time attacks, and benchmark our framework against other state of the art methods. Using 2DSig-Detect for anomaly detection, we show both superior performance and a reduction in the computation time to detect the presence of adversarial perturbations in images.

Title: DynamicFL: Federated Learning with Dynamic Communication Resource Allocation

• Authors: Qi Le, Enmao Diao, Xinran Wang, Vahid Tarokh, Jie Ding, Ali Anwar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04986
• Pdf URL: https://arxiv.org/pdf/2409.04986
• Copy Paste: [[2409.04986]] DynamicFL: Federated Learning with Dynamic Communication Resource Allocation(https://arxiv.org/abs/2409.04986)
• Keywords: federate
• Abstract: Federated Learning (FL) is a collaborative machine learning framework that allows multiple users to train models utilizing their local data in a distributed manner. However, considerable statistical heterogeneity in local data across devices often leads to suboptimal model performance compared with independently and identically distributed (IID) data scenarios. In this paper, we introduce DynamicFL, a new FL framework that investigates the trade-offs between global model performance and communication costs for two widely adopted FL methods: Federated Stochastic Gradient Descent (FedSGD) and Federated Averaging (FedAvg). Our approach allocates diverse communication resources to clients based on their data statistical heterogeneity, considering communication resource constraints, and attains substantial performance enhancements compared to uniform communication resource allocation. Notably, our method bridges the gap between FedSGD and FedAvg, providing a flexible framework leveraging communication heterogeneity to address statistical heterogeneity in FL. Through extensive experiments, we demonstrate that DynamicFL surpasses current state-of-the-art methods with up to a 10% increase in model accuracy, demonstrating its adaptability and effectiveness in tackling data statistical heterogeneity challenges.

Title: Vision-fused Attack: Advancing Aggressive and Stealthy Adversarial Text against Neural Machine Translation

• Authors: Yanni Xue, Haojie Hao, Jiakai Wang, Qiang Sheng, Renshuai Tao, Yu Liang, Pu Feng, Xianglong Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05021
• Pdf URL: https://arxiv.org/pdf/2409.05021
• Copy Paste: [[2409.05021]] Vision-fused Attack: Advancing Aggressive and Stealthy Adversarial Text against Neural Machine Translation(https://arxiv.org/abs/2409.05021)
• Keywords: attack, steal, large language model
• Abstract: While neural machine translation (NMT) models achieve success in our daily lives, they show vulnerability to adversarial attacks. Despite being harmful, these attacks also offer benefits for interpreting and enhancing NMT models, thus drawing increased research attention. However, existing studies on adversarial attacks are insufficient in both attacking ability and human imperceptibility due to their sole focus on the scope of language. This paper proposes a novel vision-fused attack (VFA) framework to acquire powerful adversarial text, i.e., more aggressive and stealthy. Regarding the attacking ability, we design the vision-merged solution space enhancement strategy to enlarge the limited semantic solution space, which enables us to search for adversarial candidates with higher attacking ability. For human imperceptibility, we propose the perception-retained adversarial text selection strategy to align the human text-reading mechanism. Thus, the finally selected adversarial text could be more deceptive. Extensive experiments on various models, including large language models (LLMs) like LLaMA and GPT-3.5, strongly support that VFA outperforms the comparisons by large margins (up to 81%/14% improvements on ASR/SSIM).

Title: Deep Self-cleansing for Medical Image Segmentation with Noisy Labels

• Authors: Jiahua Dong, Yue Zhang, Qiuli Wang, Ruofeng Tong, Shihong Ying, Shaolin Gong, Xuanpu Zhang, Lanfen Lin, Yen-Wei Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05024
• Pdf URL: https://arxiv.org/pdf/2409.05024
• Copy Paste: [[2409.05024]] Deep Self-cleansing for Medical Image Segmentation with Noisy Labels(https://arxiv.org/abs/2409.05024)
• Keywords: segmentation
• Abstract: Medical image segmentation is crucial in the field of medical imaging, aiding in disease diagnosis and surgical planning. Most established segmentation methods rely on supervised deep learning, in which clean and precise labels are essential for supervision and significantly impact the performance of models. However, manually delineated labels often contain noise, such as missing labels and inaccurate boundary delineation, which can hinder networks from correctly modeling target characteristics. In this paper, we propose a deep self-cleansing segmentation framework that can preserve clean labels while cleansing noisy ones in the training phase. To achieve this, we devise a gaussian mixture model-based label filtering module that distinguishes noisy labels from clean labels. Additionally, we develop a label cleansing module to generate pseudo low-noise labels for identified noisy samples. The preserved clean labels and pseudo-labels are then used jointly to supervise the network. Validated on a clinical liver tumor dataset and a public cardiac diagnosis dataset, our method can effectively suppress the interference from noisy labels and achieve prominent segmentation performance.

Title: Some Results on Neural Network Stability, Consistency, and Convergence: Insights into Non-IID Data, High-Dimensional Settings, and Physics-Informed Neural Networks

• Authors: Ronald Katende, Henry Kasumba, Godwin Kakuba, John M. Mango
• Subjects: cs.LG, math.NA
• Abstract URL: https://arxiv.org/abs/2409.05030
• Pdf URL: https://arxiv.org/pdf/2409.05030
• Copy Paste: [[2409.05030]] Some Results on Neural Network Stability, Consistency, and Convergence: Insights into Non-IID Data, High-Dimensional Settings, and Physics-Informed Neural Networks(https://arxiv.org/abs/2409.05030)
• Keywords: robust, federate
• Abstract: This paper addresses critical challenges in machine learning, particularly the stability, consistency, and convergence of neural networks under non-IID data, distribution shifts, and high-dimensional settings. We provide new theoretical results on uniform stability for neural networks with dynamic learning rates in non-convex settings. Further, we establish consistency bounds for federated learning models in non-Euclidean spaces, accounting for distribution shifts and curvature effects. For Physics-Informed Neural Networks (PINNs), we derive stability, consistency, and convergence guarantees for solving Partial Differential Equations (PDEs) in noisy environments. These results fill significant gaps in understanding model behavior in complex, non-ideal conditions, paving the way for more robust and reliable machine learning applications.

Title: Using Large Language Models for Template Detection from Security Event Logs

• Authors: Risto Vaarandi, Hayretdin Bahsi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.05045
• Pdf URL: https://arxiv.org/pdf/2409.05045
• Copy Paste: [[2409.05045]] Using Large Language Models for Template Detection from Security Event Logs(https://arxiv.org/abs/2409.05045)
• Keywords: security, attack, large language model
• Abstract: In modern IT systems and computer networks, real-time and offline event log analysis is a crucial part of cyber security monitoring. In particular, event log analysis techniques are essential for the timely detection of cyber attacks and for assisting security experts with the analysis of past security incidents. The detection of line patterns or templates from unstructured textual event logs has been identified as an important task of event log analysis since detected templates represent event types in the event log and prepare the logs for downstream online or offline security monitoring tasks. During the last two decades, a number of template mining algorithms have been proposed. However, many proposed algorithms rely on traditional data mining techniques, and the usage of Large Language Models (LLMs) has received less attention so far. Also, most approaches that harness LLMs are supervised, and unsupervised LLM-based template mining remains an understudied area. The current paper addresses this research gap and investigates the application of LLMs for unsupervised detection of templates from unstructured security event logs.

Title: Sight View Constraint for Robust Point Cloud Registration

• Authors: Yaojie Zhang, Weijun Wang, Tianlun Huang, Zhiyong Wang, Wei Feng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05065
• Pdf URL: https://arxiv.org/pdf/2409.05065
• Copy Paste: [[2409.05065]] Sight View Constraint for Robust Point Cloud Registration(https://arxiv.org/abs/2409.05065)
• Keywords: robust
• Abstract: Partial to Partial Point Cloud Registration (partial PCR) remains a challenging task, particularly when dealing with a low overlap rate. In comparison to the full-to-full registration task, we find that the objective of partial PCR is still not well-defined, indicating no metric can reliably identify the true transformation. We identify this as the most fundamental challenge in partial PCR tasks. In this paper, instead of directly seeking the optimal transformation, we propose a novel and general Sight View Constraint (SVC) to conclusively identify incorrect transformations, thereby enhancing the robustness of existing PCR methods. Extensive experiments validate the effectiveness of SVC on both indoor and outdoor scenes. On the challenging 3DLoMatch dataset, our approach increases the registration recall from 78\% to 82\%, achieving the state-of-the-art result. This research also highlights the significance of the decision version problem of partial PCR, which has the potential to provide novel insights into the partial PCR problem.

Title: Lepskii Principle for Distributed Kernel Ridge Regression

• Authors: Shao-Bo Lin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05070
• Pdf URL: https://arxiv.org/pdf/2409.05070
• Copy Paste: [[2409.05070]] Lepskii Principle for Distributed Kernel Ridge Regression(https://arxiv.org/abs/2409.05070)
• Keywords: privacy
• Abstract: Parameter selection without communicating local data is quite challenging in distributed learning, exhibing an inconsistency between theoretical analysis and practical application of it in tackling distributively stored data. Motivated by the recently developed Lepskii principle and non-privacy communication protocol for kernel learning, we propose a Lepskii principle to equip distributed kernel ridge regression (DKRR) and consequently develop an adaptive DKRR with Lepskii principle (Lep-AdaDKRR for short) by using a double weighted averaging synthesization scheme. We deduce optimal learning rates for Lep-AdaDKRR and theoretically show that Lep-AdaDKRR succeeds in adapting to the regularity of regression functions, effective dimension decaying rate of kernels and different metrics of generalization, which fills the gap of the mentioned inconsistency between theory and application.

Title: PIP: Detecting Adversarial Examples in Large Vision-Language Models via Attention Patterns of Irrelevant Probe Questions

• Authors: Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, Yu Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05076
• Pdf URL: https://arxiv.org/pdf/2409.05076
• Copy Paste: [[2409.05076]] PIP: Detecting Adversarial Examples in Large Vision-Language Models via Attention Patterns of Irrelevant Probe Questions(https://arxiv.org/abs/2409.05076)
• Keywords: attack, robust
• Abstract: Large Vision-Language Models (LVLMs) have demonstrated their powerful multimodal capabilities. However, they also face serious safety problems, as adversaries can induce robustness issues in LVLMs through the use of well-designed adversarial examples. Therefore, LVLMs are in urgent need of detection tools for adversarial examples to prevent incorrect responses. In this work, we first discover that LVLMs exhibit regular attention patterns for clean images when presented with probe questions. We propose an unconventional method named PIP, which utilizes the attention patterns of one randomly selected irrelevant probe question (e.g., "Is there a clock?") to distinguish adversarial examples from clean examples. Regardless of the image to be tested and its corresponding question, PIP only needs to perform one additional inference of the image to be tested and the probe question, and then achieves successful detection of adversarial examples. Even under black-box attacks and open dataset scenarios, our PIP, coupled with a simple SVM, still achieves more than 98% recall and a precision of over 90%. Our PIP is the first attempt to detect adversarial attacks on LVLMs via simple irrelevant probe questions, shedding light on deeper understanding and introspection within LVLMs. The code is available at this https URL.

Title: Adaptive $k$-nearest neighbor classifier based on the local estimation of the shape operator

• Authors: Alexandre Luís Magalhães Levada, Frank Nielsen, Michel Ferreira Cardia Haddad
• Subjects: cs.LG, cs.AI, cs.IT
• Abstract URL: https://arxiv.org/abs/2409.05084
• Pdf URL: https://arxiv.org/pdf/2409.05084
• Copy Paste: [[2409.05084]] Adaptive $k$-nearest neighbor classifier based on the local estimation of the shape operator(https://arxiv.org/abs/2409.05084)
• Keywords: robust
• Abstract: The $k$-nearest neighbor ($k$-NN) algorithm is one of the most popular methods for nonparametric classification. However, a relevant limitation concerns the definition of the number of neighbors $k$. This parameter exerts a direct impact on several properties of the classifier, such as the bias-variance tradeoff, smoothness of decision boundaries, robustness to noise, and class imbalance handling. In the present paper, we introduce a new adaptive $k$-nearest neighbours ($kK$-NN) algorithm that explores the local curvature at a sample to adaptively defining the neighborhood size. The rationale is that points with low curvature could have larger neighborhoods (locally, the tangent space approximates well the underlying data shape), whereas points with high curvature could have smaller neighborhoods (locally, the tangent space is a loose approximation). We estimate the local Gaussian curvature by computing an approximation to the local shape operator in terms of the local covariance matrix as well as the local Hessian matrix. Results on many real-world datasets indicate that the new $kK$-NN algorithm yields superior balanced accuracy compared to the established $k$-NN method and also another adaptive $k$-NN algorithm. This is particularly evident when the number of samples in the training data is limited, suggesting that the $kK$-NN is capable of learning more discriminant functions with less data considering many relevant cases.

Title: Transformer with Leveraged Masked Autoencoder for video-based Pain Assessment

• Authors: Minh-Duc Nguyen, Hyung-Jeong Yang, Soo-Hyung Kim, Ji-Eun Shin, Seung-Won Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05088
• Pdf URL: https://arxiv.org/pdf/2409.05088
• Copy Paste: [[2409.05088]] Transformer with Leveraged Masked Autoencoder for video-based Pain Assessment(https://arxiv.org/abs/2409.05088)
• Keywords: transformer
• Abstract: Accurate pain assessment is crucial in healthcare for effective diagnosis and treatment; however, traditional methods relying on self-reporting are inadequate for populations unable to communicate their pain. Cutting-edge AI is promising for supporting clinicians in pain recognition using facial video data. In this paper, we enhance pain recognition by employing facial video analysis within a Transformer-based deep learning model. By combining a powerful Masked Autoencoder with a Transformers-based classifier, our model effectively captures pain level indicators through both expressions and micro-expressions. We conducted our experiment on the AI4Pain dataset, which produced promising results that pave the way for innovative healthcare solutions that are both comprehensive and objective.

Title: DreamMapping: High-Fidelity Text-to-3D Generation via Variational Distribution Mapping

• Authors: Zeyu Cai, Duotun Wang, Yixun Liang, Zhijing Shao, Ying-Cong Chen, Xiaohang Zhan, Zeyu Wang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.05099
• Pdf URL: https://arxiv.org/pdf/2409.05099
• Copy Paste: [[2409.05099]] DreamMapping: High-Fidelity Text-to-3D Generation via Variational Distribution Mapping(https://arxiv.org/abs/2409.05099)
• Keywords: diffusion
• Abstract: Score Distillation Sampling (SDS) has emerged as a prevalent technique for text-to-3D generation, enabling 3D content creation by distilling view-dependent information from text-to-2D guidance. However, they frequently exhibit shortcomings such as over-saturated color and excess smoothness. In this paper, we conduct a thorough analysis of SDS and refine its formulation, finding that the core design is to model the distribution of rendered images. Following this insight, we introduce a novel strategy called Variational Distribution Mapping (VDM), which expedites the distribution modeling process by regarding the rendered images as instances of degradation from diffusion-based generation. This special design enables the efficient training of variational distribution by skipping the calculations of the Jacobians in the diffusion U-Net. We also introduce timestep-dependent Distribution Coefficient Annealing (DCA) to further improve distilling precision. Leveraging VDM and DCA, we use Gaussian Splatting as the 3D representation and build a text-to-3D generation framework. Extensive experiments and evaluations demonstrate the capability of VDM and DCA to generate high-fidelity and realistic assets with optimization efficiency.

Title: MaxCutPool: differentiable feature-aware Maxcut for pooling in graph neural networks

• Authors: Carlo Abate, Filippo Maria Bianchi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05100
• Pdf URL: https://arxiv.org/pdf/2409.05100
• Copy Paste: [[2409.05100]] MaxCutPool: differentiable feature-aware Maxcut for pooling in graph neural networks(https://arxiv.org/abs/2409.05100)
• Keywords: robust
• Abstract: We propose a novel approach to compute the \texttt{MAXCUT} in attributed graphs, \textit{i.e.}, graphs with features associated with nodes and edges. Our approach is robust to the underlying graph topology and is fully differentiable, making it possible to find solutions that jointly optimize the \texttt{MAXCUT} along with other objectives. Based on the obtained \texttt{MAXCUT} partition, we implement a hierarchical graph pooling layer for Graph Neural Networks, which is sparse, differentiable, and particularly suitable for downstream tasks on heterophilic graphs.

Title: WaterSeeker: Efficient Detection of Watermarked Segments in Large Documents

• Authors: Leyi Pan, Aiwei Liu, Yijian Lu, Zitian Gao, Yichen Di, Lijie Wen, Irwin King, Philip S. Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05112
• Pdf URL: https://arxiv.org/pdf/2409.05112
• Copy Paste: [[2409.05112]] WaterSeeker: Efficient Detection of Watermarked Segments in Large Documents(https://arxiv.org/abs/2409.05112)
• Keywords: extraction, watermark, large language model
• Abstract: Watermarking algorithms for large language models (LLMs) have attained high accuracy in detecting LLM-generated text. However, existing methods primarily focus on distinguishing fully watermarked text from non-watermarked text, overlooking real-world scenarios where LLMs generate only small sections within large documents. In this scenario, balancing time complexity and detection performance poses significant challenges. This paper presents WaterSeeker, a novel approach to efficiently detect and locate watermarked segments amid extensive natural text. It first applies an efficient anomaly extraction method to preliminarily locate suspicious watermarked regions. Following this, it conducts a local traversal and performs full-text detection for more precise verification. Theoretical analysis and experimental results demonstrate that WaterSeeker achieves a superior balance between detection accuracy and computational efficiency. Moreover, WaterSeeker's localization ability supports the development of interpretable AI detection systems. This work pioneers a new direction in watermarked segment detection, facilitating more reliable AI-generated content identification.

Title: PMT: Progressive Mean Teacher via Exploring Temporal Consistency for Semi-Supervised Medical Image Segmentation

• Authors: Ning Gao, Sanping Zhou, Le Wang, Nanning Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05122
• Pdf URL: https://arxiv.org/pdf/2409.05122
• Copy Paste: [[2409.05122]] PMT: Progressive Mean Teacher via Exploring Temporal Consistency for Semi-Supervised Medical Image Segmentation(https://arxiv.org/abs/2409.05122)
• Keywords: robust, segmentation
• Abstract: Semi-supervised learning has emerged as a widely adopted technique in the field of medical image segmentation. The existing works either focuses on the construction of consistency constraints or the generation of pseudo labels to provide high-quality supervisory signals, whose main challenge mainly comes from how to keep the continuous improvement of model capabilities. In this paper, we propose a simple yet effective semi-supervised learning framework, termed Progressive Mean Teachers (PMT), for medical image segmentation, whose goal is to generate high-fidelity pseudo labels by learning robust and diverse features in the training process. Specifically, our PMT employs a standard mean teacher to penalize the consistency of the current state and utilizes two sets of MT architectures for co-training. The two sets of MT architectures are individually updated for prolonged periods to maintain stable model diversity established through performance gaps generated by iteration differences. Additionally, a difference-driven alignment regularizer is employed to expedite the alignment of lagging models with the representation capabilities of leading models. Furthermore, a simple yet effective pseudo-label filtering algorithm is employed for facile evaluation of models and selection of high-fidelity pseudo-labels outputted when models are operating at high performance for co-training purposes. Experimental results on two datasets with different modalities, i.e., CT and MRI, demonstrate that our method outperforms the state-of-the-art medical image segmentation approaches across various dimensions. The code is available at this https URL.

Title: PdfTable: A Unified Toolkit for Deep Learning-Based Table Extraction

• Authors: Lei Sheng, Shuai-Shuai Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05125
• Pdf URL: https://arxiv.org/pdf/2409.05125
• Copy Paste: [[2409.05125]] PdfTable: A Unified Toolkit for Deep Learning-Based Table Extraction(https://arxiv.org/abs/2409.05125)
• Keywords: extraction
• Abstract: Currently, a substantial volume of document data exists in an unstructured format, encompassing Portable Document Format (PDF) files and images. Extracting information from these documents presents formidable challenges due to diverse table styles, complex forms, and the inclusion of different languages. Several open-source toolkits, such as Camelot, Plumb a PDF (pdfnumber), and Paddle Paddle Structure V2 (PP-StructureV2), have been developed to facilitate table extraction from PDFs or images. However, each toolkit has its limitations. Camelot and pdfnumber can solely extract tables from digital PDFs and cannot handle image-based PDFs and pictures. On the other hand, PP-StructureV2 can comprehensively extract image-based PDFs and tables from pictures. Nevertheless, it lacks the ability to differentiate between diverse application scenarios, such as wired tables and wireless tables, digital PDFs, and image-based PDFs. To address these issues, we have introduced the PDF table extraction (PdfTable) toolkit. This toolkit integrates numerous open-source models, including seven table recognition models, four Optical character recognition (OCR) recognition tools, and three layout analysis models. By refining the PDF table extraction process, PdfTable achieves adaptability across various application scenarios. We substantiate the efficacy of the PdfTable toolkit through verification on a self-labeled wired table dataset and the open-source wireless Publicly Table Reconition Dataset (PubTabNet). The PdfTable code will available on Github: this https URL.

Title: From Struggle to Simplicity with a Usable and Secure API for Encryption in Java

• Authors: Ehsan Firouzi, Ammar Mansuri, Mohammad Ghafari, Maziar Kaveh
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.05128
• Pdf URL: https://arxiv.org/pdf/2409.05128
• Copy Paste: [[2409.05128]] From Struggle to Simplicity with a Usable and Secure API for Encryption in Java(https://arxiv.org/abs/2409.05128)
• Keywords: secure
• Abstract: Cryptography misuses are prevalent in the wild. Crypto APIs are hard to use for developers, and static analysis tools do not detect every misuse. We developed SafEncrypt, an API that streamlines encryption tasks for Java developers. It is built on top of the native Java Cryptography Architecture, and it shields developers from crypto complexities and erroneous low-level details. Experiments showed that SafEncrypt is suitable for developers with varying levels of experience.

Title: MHS-STMA: Multimodal Hate Speech Detection via Scalable Transformer-Based Multilevel Attention Framework

• Authors: Anusha Chhabra, Dinesh Kumar Vishwakarma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05136
• Pdf URL: https://arxiv.org/pdf/2409.05136
• Copy Paste: [[2409.05136]] MHS-STMA: Multimodal Hate Speech Detection via Scalable Transformer-Based Multilevel Attention Framework(https://arxiv.org/abs/2409.05136)
• Keywords: transformer
• Abstract: Social media has a significant impact on people's lives. Hate speech on social media has emerged as one of society's most serious issues recently. Text and pictures are two forms of multimodal data distributed within articles. Unimodal analysis has been the primary emphasis of earlier approaches. Additionally, when doing multimodal analysis, researchers neglect to preserve the distinctive qualities associated with each modality. The present article suggests a scalable architecture for multimodal hate content detection called transformer-based multilevel attention (STMA) to address these shortcomings. This architecture consists of three main parts: a combined attention-based deep learning mechanism, a vision attention mechanism encoder, and a caption attention-mechanism encoder. To identify hate content, each component uses various attention processes and uniquely handles multimodal data. Several studies employing multiple assessment criteria on three hate speech datasets: Hateful memes, MultiOff, and MMHS150K, validate the suggested architecture's efficacy. The outcomes demonstrate that on all three datasets, the suggested strategy performs better than the baseline approaches.

Title: READoc: A Unified Benchmark for Realistic Document Structured Extraction

• Authors: Zichao Li, Aizier Abulaiti, Yaojie Lu, Xuanang Chen, Jia Zheng, Hongyu Lin, Xianpei Han, Le Sun
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05137
• Pdf URL: https://arxiv.org/pdf/2409.05137
• Copy Paste: [[2409.05137]] READoc: A Unified Benchmark for Realistic Document Structured Extraction(https://arxiv.org/abs/2409.05137)
• Keywords: extraction, segmentation
• Abstract: Document Structured Extraction (DSE) aims to extract structured content from raw documents. Despite the emergence of numerous DSE systems, their unified evaluation remains inadequate, significantly hindering the field's advancement. This problem is largely attributed to existing benchmark paradigms, which exhibit fragmented and localized characteristics. To address these limitations and offer a thorough evaluation of DSE systems, we introduce a novel benchmark named READoc, which defines DSE as a realistic task of converting unstructured PDFs into semantically rich Markdown. The READoc dataset is derived from 2,233 diverse and real-world documents from arXiv and GitHub. In addition, we develop a DSE Evaluation S$^3$uite comprising Standardization, Segmentation and Scoring modules, to conduct a unified evaluation of state-of-the-art DSE approaches. By evaluating a range of pipeline tools, expert visual models, and general VLMs, we identify the gap between current work and the unified, realistic DSE objective for the first time. We aspire that READoc will catalyze future research in DSE, fostering more comprehensive and practical solutions.

Title: OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs

• Authors: Jintian Zhang, Cheng Peng, Mengshu Sun, Xiang Chen, Lei Liang, Zhiqiang Zhang, Jun Zhou, Huajun Chen, Ningyu Zhang
• Subjects: cs.CL, cs.AI, cs.DB, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05152
• Pdf URL: https://arxiv.org/pdf/2409.05152
• Copy Paste: [[2409.05152]] OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs(https://arxiv.org/abs/2409.05152)
• Keywords: generative, large language model
• Abstract: Despite the recent advancements in Large Language Models (LLMs), which have significantly enhanced the generative capabilities for various NLP tasks, LLMs still face limitations in directly handling retrieval tasks. However, many practical applications demand the seamless integration of both retrieval and generation. This paper introduces a novel and efficient One-pass Generation and retrieval framework (OneGen), designed to improve LLMs' performance on tasks that require both generation and retrieval. The proposed framework bridges the traditionally separate training approaches for generation and retrieval by incorporating retrieval tokens generated autoregressively. This enables a single LLM to handle both tasks simultaneously in a unified forward pass. We conduct experiments on two distinct types of composite tasks, RAG and Entity Linking, to validate the pluggability, effectiveness, and efficiency of OneGen in training and inference. Furthermore, our results show that integrating generation and retrieval within the same context preserves the generative capabilities of LLMs while improving retrieval performance. To the best of our knowledge, OneGen is the first to enable LLMs to conduct vector retrieval during the generation.

Title: Can OOD Object Detectors Learn from Foundation Models?

• Authors: Jiahui Liu, Xin Wen, Shizhen Zhao, Yingxian Chen, Xiaojuan Qi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05162
• Pdf URL: https://arxiv.org/pdf/2409.05162
• Copy Paste: [[2409.05162]] Can OOD Object Detectors Learn from Foundation Models?(https://arxiv.org/abs/2409.05162)
• Keywords: diffusion, generative
• Abstract: Out-of-distribution (OOD) object detection is a challenging task due to the absence of open-set OOD data. Inspired by recent advancements in text-to-image generative models, such as Stable Diffusion, we study the potential of generative models trained on large-scale open-set data to synthesize OOD samples, thereby enhancing OOD object detection. We introduce SyncOOD, a simple data curation method that capitalizes on the capabilities of large foundation models to automatically extract meaningful OOD data from text-to-image generative models. This offers the model access to open-world knowledge encapsulated within off-the-shelf foundation models. The synthetic OOD samples are then employed to augment the training of a lightweight, plug-and-play OOD detector, thus effectively optimizing the in-distribution (ID)/OOD decision boundaries. Extensive experiments across multiple benchmarks demonstrate that SyncOOD significantly outperforms existing methods, establishing new state-of-the-art performance with minimal synthetic data usage.

Title: Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?

• Authors: Neeladri Bhuiya, Viktor Schlegel, Stefan Winkler
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05197
• Pdf URL: https://arxiv.org/pdf/2409.05197
• Copy Paste: [[2409.05197]] Seemingly Plausible Distractors in Multi-Hop Reasoning: Are Large Language Models Attentive Readers?(https://arxiv.org/abs/2409.05197)
• Keywords: large language model
• Abstract: State-of-the-art Large Language Models (LLMs) are accredited with an increasing number of different capabilities, ranging from reading comprehension, over advanced mathematical and reasoning skills to possessing scientific knowledge. In this paper we focus on their multi-hop reasoning capability: the ability to identify and integrate information from multiple textual sources. Given the concerns with the presence of simplifying cues in existing multi-hop reasoning benchmarks, which allow models to circumvent the reasoning requirement, we set out to investigate, whether LLMs are prone to exploiting such simplifying cues. We find evidence that they indeed circumvent the requirement to perform multi-hop reasoning, but they do so in more subtle ways than what was reported about their fine-tuned pre-trained language model (PLM) predecessors. Motivated by this finding, we propose a challenging multi-hop reasoning benchmark, by generating seemingly plausible multi-hop reasoning chains, which ultimately lead to incorrect answers. We evaluate multiple open and proprietary state-of-the-art LLMs, and find that their performance to perform multi-hop reasoning is affected, as indicated by up to 45% relative decrease in F1 score when presented with such seemingly plausible alternatives. We conduct a deeper analysis and find evidence that while LLMs tend to ignore misleading lexical cues, misleading reasoning paths indeed present a significant challenge.

Title: Lung-DETR: Deformable Detection Transformer for Sparse Lung Nodule Anomaly Detection

• Authors: Hooman Ramezani, Dionne Aleman, Daniel Létourneau
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05200
• Pdf URL: https://arxiv.org/pdf/2409.05200
• Copy Paste: [[2409.05200]] Lung-DETR: Deformable Detection Transformer for Sparse Lung Nodule Anomaly Detection(https://arxiv.org/abs/2409.05200)
• Keywords: transformer
• Abstract: Accurate lung nodule detection for computed tomography (CT) scan imagery is challenging in real-world settings due to the sparse occurrence of nodules and similarity to other anatomical structures. In a typical positive case, nodules may appear in as few as 3% of CT slices, complicating detection. To address this, we reframe the problem as an anomaly detection task, targeting rare nodule occurrences in a predominantly normal dataset. We introduce a novel solution leveraging custom data preprocessing and Deformable Detection Transformer (Deformable- DETR). A 7.5mm Maximum Intensity Projection (MIP) is utilized to combine adjacent lung slices into single images, reducing the slice count and decreasing nodule sparsity. This enhances spatial context, allowing for better differentiation between nodules and other structures such as complex vascular structures and bronchioles. Deformable-DETR is employed to detect nodules, with a custom focal loss function to better handle the imbalanced dataset. Our model achieves state-of-the-art performance on the LUNA16 dataset with an F1 score of 94.2% (95.2% recall, 93.3% precision) on a dataset sparsely populated with lung nodules that is reflective of real-world clinical data.

Title: Efficient Homomorphically Encrypted Convolutional Neural Network Without Rotation

• Authors: Sajjad Akherati, Xinmiao Zhang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.05205
• Pdf URL: https://arxiv.org/pdf/2409.05205
• Copy Paste: [[2409.05205]] Efficient Homomorphically Encrypted Convolutional Neural Network Without Rotation(https://arxiv.org/abs/2409.05205)
• Keywords: security, privacy
• Abstract: Privacy-preserving neural network (NN) inference can be achieved by utilizing homomorphic encryption (HE), which allows computations to be directly carried out over ciphertexts. Popular HE schemes are built over large polynomial rings. To allow simultaneous multiplications in the convolutional (Conv) and fully-connected (FC) layers, multiple input data are mapped to coefficients in the same polynomial, so are the weights of NNs. However, ciphertext rotations are necessary to compute the sums of products and/or incorporate the outputs of different channels into the same polynomials. Ciphertext rotations have much higher complexity than ciphertext multiplications and contribute to the majority of the latency of HE-evaluated Conv and FC layers. This paper proposes a novel reformulated server-client joint computation procedure and a new filter coefficient packing scheme to eliminate ciphertext rotations without affecting the security of the HE scheme. Our proposed scheme also leads to substantial reductions on the number of coefficient multiplications needed and the communication cost between the server and client. For various plain-20 classifiers over the CIFAR-10/100 datasets, our design reduces the running time of the Conv and FC layers by 15.5% and the communication cost between client and server by more than 50%, compared to the best prior design.

Title: SEF: A Method for Computing Prediction Intervals by Shifting the Error Function in Neural Networks

• Authors: E. V. Aretos, D. G. Sotiropoulos
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.05206
• Pdf URL: https://arxiv.org/pdf/2409.05206
• Copy Paste: [[2409.05206]] SEF: A Method for Computing Prediction Intervals by Shifting the Error Function in Neural Networks(https://arxiv.org/abs/2409.05206)
• Keywords: robust
• Abstract: In today's era, Neural Networks (NN) are applied in various scientific fields such as robotics, medicine, engineering, etc. However, the predictions of neural networks themselves contain a degree of uncertainty that must always be taken into account before any decision is made. This is why many researchers have focused on developing different ways to quantify the uncertainty of neural network predictions. Some of these methods are based on generating prediction intervals (PI) via neural networks for the requested target values. The SEF (Shifting the Error Function) method presented in this paper is a new method that belongs to this category of methods. The proposed approach involves training a single neural network three times, thus generating an estimate along with the corresponding upper and lower bounds for a given problem. A pivotal aspect of the method is the calculation of a parameter from the initial network's estimates, which is then integrated into the loss functions of the other two networks. This innovative process effectively produces PIs, resulting in a robust and efficient technique for uncertainty quantification. To evaluate the effectiveness of our method, a comparison in terms of successful PI generation between the SEF, PI3NN and PIVEN methods was made using two synthetic datasets.

Title: Low Latency Transformer Inference on FPGAs for Physics Applications with hls4ml

• Authors: Zhixing Jiang, Dennis Yin, Yihui Chen, Elham E Khoda, Scott Hauck, Shih-Chieh Hsu, Ekaterina Govorkova, Philip Harris, Vladimir Loncar, Eric A. Moreno
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05207
• Pdf URL: https://arxiv.org/pdf/2409.05207
• Copy Paste: [[2409.05207]] Low Latency Transformer Inference on FPGAs for Physics Applications with hls4ml(https://arxiv.org/abs/2409.05207)
• Keywords: transformer
• Abstract: This study presents an efficient implementation of transformer architectures in Field-Programmable Gate Arrays(FPGAs) using hls4ml. We demonstrate the strategy for implementing the multi-head attention, softmax, and normalization layer and evaluate three distinct models. Their deployment on VU13P FPGA chip achieved latency less than 2us, demonstrating the potential for real-time applications. HLS4ML compatibility with any TensorFlow-built transformer model further enhances the scalability and applicability of this work. Index Terms: FPGAs, machine learning, transformers, high energy physics, LIGO

Title: Influence-based Attributions can be Manipulated

• Authors: Chhavi Yadav, Ruihan Wu, Kamalika Chaudhuri
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05208
• Pdf URL: https://arxiv.org/pdf/2409.05208
• Copy Paste: [[2409.05208]] Influence-based Attributions can be Manipulated(https://arxiv.org/abs/2409.05208)
• Keywords: attack, fair
• Abstract: Influence Functions are a standard tool for attributing predictions to training data in a principled manner and are widely used in applications such as data valuation and fairness. In this work, we present realistic incentives to manipulate influencebased attributions and investigate whether these attributions can be systematically tampered by an adversary. We show that this is indeed possible and provide efficient attacks with backward-friendly implementations. Our work raises questions on the reliability of influence-based attributions under adversarial circumstances.

Title: ICML Topological Deep Learning Challenge 2024: Beyond the Graph Domain

• Authors: Guillermo Bernárdez, Lev Telyatnikov, Marco Montagna, Federica Baccini, Mathilde Papillon, Miquel Ferriol-Galmés, Mustafa Hajij, Theodore Papamarkou, Maria Sofia Bucarelli, Olga Zaghen, Johan Mathe, Audun Myers, Scott Mahan, Hansen Lillemark, Sharvaree Vadgama, Erik Bekkers, Tim Doster, Tegan Emerson, Henry Kvinge, Katrina Agate, Nesreen K Ahmed, Pengfei Bai, Michael Banf, Claudio Battiloro, Maxim Beketov, Paul Bogdan, Martin Carrasco, Andrea Cavallo, Yun Young Choi, George Dasoulas, Matouš Elphick, Giordan Escalona, Dominik Filipiak, Halley Fritze, Thomas Gebhart, Manel Gil-Sorribes, Salvish Goomanee, Victor Guallar, Liliya Imasheva, Andrei Irimia, Hongwei Jin, Graham Johnson, Nikos Kanakaris, Boshko Koloski, Veljko Kovač, Manuel Lecha, Minho Lee, Pierrick Leroy, Theodore Long, German Magai, Alvaro Martinez, Marissa Masden, Sebastian Mežnar, Bertran Miquel-Oliver, Alexis Molina, Alexander Nikitin, Marco Nurisso, Matt Piekenbrock, Yu Qin, Patryk Rygiel, Alessandro Salatiello, Max Schattauer, Pavel Snopov, Julian Suk, Valentina Sánchez, Mauricio Tec, Francesco Vaccarino, Jonas Verhellen, Frederic Wantiez, Alexander Weers, Patrik Zajec, Blaž Škrlj, Nina Miolane
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05211
• Pdf URL: https://arxiv.org/pdf/2409.05211
• Copy Paste: [[2409.05211]] ICML Topological Deep Learning Challenge 2024: Beyond the Graph Domain(https://arxiv.org/abs/2409.05211)
• Keywords: generative
• Abstract: This paper describes the 2nd edition of the ICML Topological Deep Learning Challenge that was hosted within the ICML 2024 ELLIS Workshop on Geometry-grounded Representation Learning and Generative Modeling (GRaM). The challenge focused on the problem of representing data in different discrete topological domains in order to bridge the gap between Topological Deep Learning (TDL) and other types of structured datasets (e.g. point clouds, graphs). Specifically, participants were asked to design and implement topological liftings, i.e. mappings between different data structures and topological domains --like hypergraphs, or simplicial/cell/combinatorial complexes. The challenge received 52 submissions satisfying all the requirements. This paper introduces the main scope of the challenge, and summarizes the main results and findings.

Title: Synthetic Tabular Data Generation for Class Imbalance and Fairness: A Comparative Study

• Authors: Emmanouil Panagiotou, Arjun Roy, Eirini Ntoutsi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05215
• Pdf URL: https://arxiv.org/pdf/2409.05215
• Copy Paste: [[2409.05215]] Synthetic Tabular Data Generation for Class Imbalance and Fairness: A Comparative Study(https://arxiv.org/abs/2409.05215)
• Keywords: protect, fair, generative
• Abstract: Due to their data-driven nature, Machine Learning (ML) models are susceptible to bias inherited from data, especially in classification problems where class and group imbalances are prevalent. Class imbalance (in the classification target) and group imbalance (in protected attributes like sex or race) can undermine both ML utility and fairness. Although class and group imbalances commonly coincide in real-world tabular datasets, limited methods address this scenario. While most methods use oversampling techniques, like interpolation, to mitigate imbalances, recent advancements in synthetic tabular data generation offer promise but have not been adequately explored for this purpose. To this end, this paper conducts a comparative analysis to address class and group imbalances using state-of-the-art models for synthetic tabular data generation and various sampling strategies. Experimental results on four datasets, demonstrate the effectiveness of generative models for bias mitigation, creating opportunities for further exploration in this direction.

Title: Socially Responsible Data for Large Multilingual Language Models

• Authors: Andrew Smart, Ben Hutchinson, Lameck Mbangula Amugongo, Suzanne Dikker, Alex Zito, Amber Ebinama, Zara Wudiri, Ding Wang, Erin van Liemt, João Sedoc, Seyi Olojo, Stanley Uwakwe, Edem Wornyo, Sonja Schmer-Galunder, Jamila Smith-Loud
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05247
• Pdf URL: https://arxiv.org/pdf/2409.05247
• Copy Paste: [[2409.05247]] Socially Responsible Data for Large Multilingual Language Models(https://arxiv.org/abs/2409.05247)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) have rapidly increased in size and apparent capabilities in the last three years, but their training data is largely English text. There is growing interest in multilingual LLMs, and various efforts are striving for models to accommodate languages of communities outside of the Global North, which include many languages that have been historically underrepresented in digital realms. These languages have been coined as "low resource languages" or "long-tail languages", and LLMs performance on these languages is generally poor. While expanding the use of LLMs to more languages may bring many potential benefits, such as assisting cross-community communication and language preservation, great care must be taken to ensure that data collection on these languages is not extractive and that it does not reproduce exploitative practices of the past. Collecting data from languages spoken by previously colonized people, indigenous people, and non-Western languages raises many complex sociopolitical and ethical questions, e.g., around consent, cultural safety, and data sovereignty. Furthermore, linguistic complexity and cultural nuances are often lost in LLMs. This position paper builds on recent scholarship, and our own work, and outlines several relevant social, cultural, and ethical considerations and potential ways to mitigate them through qualitative research, community partnerships, and participatory design approaches. We provide twelve recommendations for consideration when collecting language data on underrepresented language communities outside of the Global North.

Title: NetDPSyn: Synthesizing Network Traces under Differential Privacy

• Authors: Danyu Sun, Joann Qiongna Chen, Chen Gong, Tianhao Wang, Zhou Li
• Subjects: cs.CR, cs.DB, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.05249
• Pdf URL: https://arxiv.org/pdf/2409.05249
• Copy Paste: [[2409.05249]] NetDPSyn: Synthesizing Network Traces under Differential Privacy(https://arxiv.org/abs/2409.05249)
• Keywords: privacy, attack, generative
• Abstract: As the utilization of network traces for the network measurement research becomes increasingly prevalent, concerns regarding privacy leakage from network traces have garnered the public's attention. To safeguard network traces, researchers have proposed the trace synthesis that retains the essential properties of the raw data. However, previous works also show that synthesis traces with generative models are vulnerable under linkage attacks. This paper introduces NetDPSyn, the first system to synthesize high-fidelity network traces under privacy guarantees. NetDPSyn is built with the Differential Privacy (DP) framework as its core, which is significantly different from prior works that apply DP when training the generative model. The experiments conducted on three flow and two packet datasets indicate that NetDPSyn achieves much better data utility in downstream tasks like anomaly detection. NetDPSyn is also 2.5 times faster than the other methods on average in data synthesis.

Title: MRStyle: A Unified Framework for Color Style Transfer with Multi-Modality Reference

• Authors: Jiancheng Huang, Yu Gao, Zequn Jie, Yujie Zhong, Xintong Han, Lin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05250
• Pdf URL: https://arxiv.org/pdf/2409.05250
• Copy Paste: [[2409.05250]] MRStyle: A Unified Framework for Color Style Transfer with Multi-Modality Reference(https://arxiv.org/abs/2409.05250)
• Keywords: diffusion
• Abstract: In this paper, we introduce MRStyle, a comprehensive framework that enables color style transfer using multi-modality reference, including image and text. To achieve a unified style feature space for both modalities, we first develop a neural network called IRStyle, which generates stylized 3D lookup tables for image reference. This is accomplished by integrating an interaction dual-mapping network with a combined supervised learning pipeline, resulting in three key benefits: elimination of visual artifacts, efficient handling of high-resolution images with low memory usage, and maintenance of style consistency even in situations with significant color style variations. For text reference, we align the text feature of stable diffusion priors with the style feature of our IRStyle to perform text-guided color style transfer (TRStyle). Our TRStyle method is highly efficient in both training and inference, producing notable open-set text-guided transfer results. Extensive experiments in both image and text settings demonstrate that our proposed method outperforms the state-of-the-art in both qualitative and quantitative evaluations.

Title: UPCS: Unbiased Persona Construction for Dialogue Generation

• Authors: Kuiyun Chen, Yanbin Wei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05257
• Pdf URL: https://arxiv.org/pdf/2409.05257
• Copy Paste: [[2409.05257]] UPCS: Unbiased Persona Construction for Dialogue Generation(https://arxiv.org/abs/2409.05257)
• Keywords: fair
• Abstract: Narrative systems, such as dialogue and storytelling systems, often utilize persona profiles to enhance personalized interactions. Existing persona profiles frequently exhibit biases, posing risks to system integrity and fairness. To address this, we introduce the UPCS framework, which categorizes character descriptions into eight dimensions, including bias mitigation strategies. Experimental results demonstrate UPCS's superiority in accuracy, diversity, bias elimination, and user satisfaction, marking a significant advancement in persona construction for reliable narrative systems.

Title: Towards Automated Machine Learning Research

• Authors: Shervin Ardeshir
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05258
• Pdf URL: https://arxiv.org/pdf/2409.05258
• Copy Paste: [[2409.05258]] Towards Automated Machine Learning Research(https://arxiv.org/abs/2409.05258)
• Keywords: large language model
• Abstract: This paper explores a top-down approach to automating incremental advances in machine learning research through component-level innovation, facilitated by Large Language Models (LLMs). Our framework systematically generates novel components, validates their feasibility, and evaluates their performance against existing baselines. A key distinction of this approach lies in how these novel components are generated. Unlike traditional AutoML and NAS methods, which often rely on a bottom-up combinatorial search over predefined, hardcoded base components, our method leverages the cross-domain knowledge embedded in LLMs to propose new components that may not be confined to any hard-coded predefined set. By incorporating a reward model to prioritize promising hypotheses, we aim to improve the efficiency of the hypothesis generation and evaluation process. We hope this approach offers a new avenue for exploration and contributes to the ongoing dialogue in the field.

Title: RexUniNLU: Recursive Method with Explicit Schema Instructor for Universal NLU

• Authors: Chengyuan Liu, Shihang Wang, Fubang Zhao, Kun Kuang, Yangyang Kang, Weiming Lu, Changlong Sun, Fei Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05275
• Pdf URL: https://arxiv.org/pdf/2409.05275
• Copy Paste: [[2409.05275]] RexUniNLU: Recursive Method with Explicit Schema Instructor for Universal NLU(https://arxiv.org/abs/2409.05275)
• Keywords: extraction
• Abstract: Information Extraction (IE) and Text Classification (CLS) serve as the fundamental pillars of NLU, with both disciplines relying on analyzing input sequences to categorize outputs into pre-established schemas. However, there is no existing encoder-based model that can unify IE and CLS tasks from this perspective. To fully explore the foundation shared within NLU tasks, we have proposed a Recursive Method with Explicit Schema Instructor for Universal NLU. Specifically, we firstly redefine the true universal information extraction (UIE) with a formal formulation that covers almost all extraction schemas, including quadruples and quintuples which remain unsolved for previous UIE models. Then, we expands the formulation to all CLS and multi-modal NLU tasks. Based on that, we introduce RexUniNLU, an universal NLU solution that employs explicit schema constraints for IE and CLS, which encompasses all IE and CLS tasks and prevent incorrect connections between schema and input sequence. To avoid interference between different schemas, we reset the position ids and attention mask matrices. Extensive experiments are conducted on IE, CLS in both English and Chinese, and multi-modality, revealing the effectiveness and superiority. Our codes are publicly released.

Title: Disentangled Representations for Short-Term and Long-Term Person Re-Identification

• Authors: Chanho Eom, Wonkyung Lee, Geon Lee, Bumsub Ham
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05277
• Pdf URL: https://arxiv.org/pdf/2409.05277
• Copy Paste: [[2409.05277]] Disentangled Representations for Short-Term and Long-Term Person Re-Identification(https://arxiv.org/abs/2409.05277)
• Keywords: robust, generative
• Abstract: We address the problem of person re-identification (reID), that is, retrieving person images from a large dataset, given a query image of the person of interest. A key challenge is to learn person representations robust to intra-class variations, as different persons could have the same attribute, and persons' appearances look different, e.g., with viewpoint changes. Recent reID methods focus on learning person features discriminative only for a particular factor of variations (e.g., human pose), which also requires corresponding supervisory signals (e.g., pose annotations). To tackle this problem, we propose to factorize person images into identity-related and unrelated features. Identity-related features contain information useful for specifying a particular person (e.g., clothing), while identity-unrelated ones hold other factors (e.g., human pose). To this end, we propose a new generative adversarial network, dubbed identity shuffle GAN (IS-GAN). It disentangles identity-related and unrelated features from person images through an identity-shuffling technique that exploits identification labels alone without any auxiliary supervisory signals. We restrict the distribution of identity-unrelated features or encourage the identity-related and unrelated features to be uncorrelated, facilitating the disentanglement process. Experimental results validate the effectiveness of IS-GAN, showing state-of-the-art performance on standard reID benchmarks, including Market-1501, CUHK03, and DukeMTMC-reID. We further demonstrate the advantages of disentangling person representations on a long-term reID task, setting a new state of the art on a Celeb-reID dataset.

Title: BrainDecoder: Style-Based Visual Decoding of EEG Signals

• Authors: Minsuk Choi, Hiroshi Ishikawa
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05279
• Pdf URL: https://arxiv.org/pdf/2409.05279
• Copy Paste: [[2409.05279]] BrainDecoder: Style-Based Visual Decoding of EEG Signals(https://arxiv.org/abs/2409.05279)
• Keywords: extraction
• Abstract: Decoding neural representations of visual stimuli from electroencephalography (EEG) offers valuable insights into brain activity and cognition. Recent advancements in deep learning have significantly enhanced the field of visual decoding of EEG, primarily focusing on reconstructing the semantic content of visual stimuli. In this paper, we present a novel visual decoding pipeline that, in addition to recovering the content, emphasizes the reconstruction of the style, such as color and texture, of images viewed by the subject. Unlike previous methods, this ``style-based'' approach learns in the CLIP spaces of image and text separately, facilitating a more nuanced extraction of information from EEG signals. We also use captions for text alignment simpler than previously employed, which we find work better. Both quantitative and qualitative evaluations show that our method better preserves the style of visual stimuli and extracts more fine-grained semantic information from neural signals. Notably, it achieves significant improvements in quantitative results and sets a new state-of-the-art on the popular Brain2Image dataset.

Title: RotCAtt-TransUNet++: Novel Deep Neural Network for Sophisticated Cardiac Segmentation

• Authors: Quoc-Bao Nguyen-Le, Tuan-Hy Le, Anh-Triet Do, Quoc-Huy Trinh
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05280
• Pdf URL: https://arxiv.org/pdf/2409.05280
• Copy Paste: [[2409.05280]] RotCAtt-TransUNet++: Novel Deep Neural Network for Sophisticated Cardiac Segmentation(https://arxiv.org/abs/2409.05280)
• Keywords: robust, transformer, segmentation
• Abstract: Cardiovascular disease is a major global health concern, contributing significantly to global mortality. Accurately segmenting cardiac medical imaging data is crucial for reducing fatality rates associated with these conditions. However, current state-of-the-art (SOTA) neural networks, including CNN-based and Transformer-based approaches, face challenges in capturing both inter-slice connections and intra-slice details, especially in datasets featuring intricate, long-range details along the z-axis like coronary arteries. Existing methods also struggle with differentiating non-cardiac components from the myocardium, resulting in segmentation inaccuracies and the "spraying" phenomenon. To address these issues, we introduce RotCAtt-TransUNet++, a novel architecture designed for robust segmentation of intricate cardiac structures. Our approach enhances global context modeling through multiscale feature aggregation and nested skip connections in the encoder. Transformer layers facilitate capturing intra-slice interactions, while a rotatory attention mechanism handles inter-slice connectivity. A channel-wise cross-attention gate integrates multiscale information and decoder features, effectively bridging semantic gaps. Experimental results across multiple datasets demonstrate superior performance over current methods, achieving near-perfect annotation of coronary arteries and myocardium. Ablation studies confirm that our rotatory attention mechanism significantly improves segmentation accuracy by transforming embedded vectorized patches in semantic dimensional space.

Title: Seek and Solve Reasoning for Table Question Answering

• Authors: Ruya Jiang, Chun Wang, Weihong Deng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05286
• Pdf URL: https://arxiv.org/pdf/2409.05286
• Copy Paste: [[2409.05286]] Seek and Solve Reasoning for Table Question Answering(https://arxiv.org/abs/2409.05286)
• Keywords: large language model
• Abstract: Table-based Question Answering (TQA) involves answering questions based on tabular data. The complexity of table structures and question logic makes this task difficult even for Large Language Models (LLMs). This paper improves TQA performance by leveraging LLMs' reasoning capabilities. Inspired by how humans solve TQA tasks, we propose a Seek-and-Solve pipeline that instructs the LLM to first seek relevant information and then answer questions. The two stages are integrated at the reasoning level, and their Chain of Thought (CoT) paths are integrated into a coherent Seek-and-Solve CoT (SS-CoT). Furthermore, we present a compact single-stage TQA-solving prompt distilled from the pipeline. Experiments demonstrate that under In-Context Learning settings, using samples with SS-CoT paths as demonstrations, the TQA-solving prompt can effectively guide the LLM to solve complex TQA tasks, resulting in improved performance and reliability. Our results highlight the importance of properly eliciting LLMs' reasoning capabilities in solving complex TQA tasks.

Title: Towards Fast Rates for Federated and Multi-Task Reinforcement Learning

• Authors: Feng Zhu, Robert W. Heath Jr., Aritra Mitra
• Subjects: cs.LG, eess.SY, math.OC
• Abstract URL: https://arxiv.org/abs/2409.05291
• Pdf URL: https://arxiv.org/pdf/2409.05291
• Copy Paste: [[2409.05291]] Towards Fast Rates for Federated and Multi-Task Reinforcement Learning(https://arxiv.org/abs/2409.05291)
• Keywords: federate
• Abstract: We consider a setting involving $N$ agents, where each agent interacts with an environment modeled as a Markov Decision Process (MDP). The agents' MDPs differ in their reward functions, capturing heterogeneous objectives/tasks. The collective goal of the agents is to communicate intermittently via a central server to find a policy that maximizes the average of long-term cumulative rewards across environments. The limited existing work on this topic either only provide asymptotic rates, or generate biased policies, or fail to establish any benefits of collaboration. In response, we propose Fast-FedPG - a novel federated policy gradient algorithm with a carefully designed bias-correction mechanism. Under a gradient-domination condition, we prove that our algorithm guarantees (i) fast linear convergence with exact gradients, and (ii) sub-linear rates that enjoy a linear speedup w.r.t. the number of agents with noisy, truncated policy gradients. Notably, in each case, the convergence is to a globally optimal policy with no heterogeneity-induced bias. In the absence of gradient-domination, we establish convergence to a first-order stationary point at a rate that continues to benefit from collaboration.

Title: TERD: A Unified Framework for Safeguarding Diffusion Models Against Backdoors

• Authors: Yichuan Mo, Hui Huang, Mingjie Li, Ang Li, Yisen Wang
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05294
• Pdf URL: https://arxiv.org/pdf/2409.05294
• Copy Paste: [[2409.05294]] TERD: A Unified Framework for Safeguarding Diffusion Models Against Backdoors(https://arxiv.org/abs/2409.05294)
• Keywords: secure, protect, defense, attack, diffusion
• Abstract: Diffusion models have achieved notable success in image generation, but they remain highly vulnerable to backdoor attacks, which compromise their integrity by producing specific undesirable outputs when presented with a pre-defined trigger. In this paper, we investigate how to protect diffusion models from this dangerous threat. Specifically, we propose TERD, a backdoor defense framework that builds unified modeling for current attacks, which enables us to derive an accessible reversed loss. A trigger reversion strategy is further employed: an initial approximation of the trigger through noise sampled from a prior distribution, followed by refinement through differential multi-step samplers. Additionally, with the reversed trigger, we propose backdoor detection from the noise space, introducing the first backdoor input detection approach for diffusion models and a novel model detection algorithm that calculates the KL divergence between reversed and benign distributions. Extensive evaluations demonstrate that TERD secures a 100% True Positive Rate (TPR) and True Negative Rate (TNR) across datasets of varying resolutions. TERD also demonstrates nice adaptability to other Stochastic Differential Equation (SDE)-based models. Our code is available at this https URL.

Title: Resource-Efficient Generative AI Model Deployment in Mobile Edge Networks

• Authors: Yuxin Liang, Peng Yang, Yuanyuan He, Feng Lyu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05303
• Pdf URL: https://arxiv.org/pdf/2409.05303
• Copy Paste: [[2409.05303]] Resource-Efficient Generative AI Model Deployment in Mobile Edge Networks(https://arxiv.org/abs/2409.05303)
• Keywords: generative
• Abstract: The surging development of Artificial Intelligence-Generated Content (AIGC) marks a transformative era of the content creation and production. Edge servers promise attractive benefits, e.g., reduced service delay and backhaul traffic load, for hosting AIGC services compared to cloud-based solutions. However, the scarcity of available resources on the edge pose significant challenges in deploying generative AI models. In this paper, by characterizing the resource and delay demands of typical generative AI models, we find that the consumption of storage and GPU memory, as well as the model switching delay represented by I/O delay during the preloading phase, are significant and vary across models. These multidimensional coupling factors render it difficult to make efficient edge model deployment decisions. Hence, we present a collaborative edge-cloud framework aiming to properly manage generative AI model deployment on the edge. Specifically, we formulate edge model deployment problem considering heterogeneous features of models as an optimization problem, and propose a model-level decision selection algorithm to solve it. It enables pooled resource sharing and optimizes the trade-off between resource consumption and delay in edge generative AI model deployment. Simulation results validate the efficacy of the proposed algorithm compared with baselines, demonstrating its potential to reduce overall costs by providing feature-aware model deployment decisions.

Title: Fitting Skeletal Models via Graph-based Learning

• Authors: Nicolás Gaggion, Enzo Ferrante, Beatriz Paniagua, Jared Vicory
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05311
• Pdf URL: https://arxiv.org/pdf/2409.05311
• Copy Paste: [[2409.05311]] Fitting Skeletal Models via Graph-based Learning(https://arxiv.org/abs/2409.05311)
• Keywords: segmentation
• Abstract: Skeletonization is a popular shape analysis technique that models an object's interior as opposed to just its boundary. Fitting template-based skeletal models is a time-consuming process requiring much manual parameter tuning. Recently, machine learning-based methods have shown promise for generating s-reps from object boundaries. In this work, we propose a new skeletonization method which leverages graph convolutional networks to produce skeletal representations (s-reps) from dense segmentation masks. The method is evaluated on both synthetic data and real hippocampus segmentations, achieving promising results and fast inference.

Title: FIF-UNet: An Efficient UNet Using Feature Interaction and Fusion for Medical Image Segmentation

• Authors: Xiaolin Gou, Chuanlin Liao, Jizhe Zhou, Fengshuo Ye, Yi Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05324
• Pdf URL: https://arxiv.org/pdf/2409.05324
• Copy Paste: [[2409.05324]] FIF-UNet: An Efficient UNet Using Feature Interaction and Fusion for Medical Image Segmentation(https://arxiv.org/abs/2409.05324)
• Keywords: robust, segmentation
• Abstract: Nowadays, pre-trained encoders are widely used in medical image segmentation because of their ability to capture complex feature representations. However, the existing models fail to effectively utilize the rich features obtained by the pre-trained encoder, resulting in suboptimal segmentation results. In this work, a novel U-shaped model, called FIF-UNet, is proposed to address the above issue, including three plug-and-play modules. A channel spatial interaction module (CSI) is proposed to obtain informative features by establishing the interaction between encoder stages and corresponding decoder stages. A cascaded conv-SE module (CoSE) is designed to enhance the representation of critical features by adaptively assigning importance weights on different feature channels. A multi-level fusion module (MLF) is proposed to fuse the multi-scale features from the decoder stages, ensuring accurate and robust final segmentation. Comprehensive experiments on the Synapse and ACDC datasets demonstrate that the proposed FIF-UNet outperforms existing state-of-the-art methods, which achieves the highest average DICE of 86.05% and 92.58%, respectively.

Title: ICPR 2024 Competition on Safe Segmentation of Drive Scenes in Unstructured Traffic and Adverse Weather Conditions

• Authors: Furqan Ahmed Shaik, Sandeep Nagar, Aiswarya Maturi, Harshit Kumar Sankhla, Dibyendu Ghosh, Anshuman Majumdar, Srikanth Vidapanakal, Kunal Chaudhary, Sunny Manchanda, Girish Varma
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05327
• Pdf URL: https://arxiv.org/pdf/2409.05327
• Copy Paste: [[2409.05327]] ICPR 2024 Competition on Safe Segmentation of Drive Scenes in Unstructured Traffic and Adverse Weather Conditions(https://arxiv.org/abs/2409.05327)
• Keywords: robust, segmentation
• Abstract: The ICPR 2024 Competition on Safe Segmentation of Drive Scenes in Unstructured Traffic and Adverse Weather Conditions served as a rigorous platform to evaluate and benchmark state-of-the-art semantic segmentation models under challenging conditions for autonomous driving. Over several months, participants were provided with the IDD-AW dataset, consisting of 5000 high-quality RGB-NIR image pairs, each annotated at the pixel level and captured under adverse weather conditions such as rain, fog, low light, and snow. A key aspect of the competition was the use and improvement of the Safe mean Intersection over Union (Safe mIoU) metric, designed to penalize unsafe incorrect predictions that could be overlooked by traditional mIoU. This innovative metric emphasized the importance of safety in developing autonomous driving systems. The competition showed significant advancements in the field, with participants demonstrating models that excelled in semantic segmentation and prioritized safety and robustness in unstructured and adverse conditions. The results of the competition set new benchmarks in the domain, highlighting the critical role of safety in deploying autonomous vehicles in real-world scenarios. The contributions from this competition are expected to drive further innovation in autonomous driving technology, addressing the critical challenges of operating in diverse and unpredictable environments.

Title: KAN-Based Fusion of Dual-Domain for Audio-Driven Facial Landmarks Generation

• Authors: Hoang-Son Vo-Thanh, Quang-Vinh Nguyen, Soo-Hyung Kim
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.05330
• Pdf URL: https://arxiv.org/pdf/2409.05330
• Copy Paste: [[2409.05330]] KAN-Based Fusion of Dual-Domain for Audio-Driven Facial Landmarks Generation(https://arxiv.org/abs/2409.05330)
• Keywords: robust
• Abstract: Audio-driven talking face generation is a widely researched topic due to its high applicability. Reconstructing a talking face using audio significantly contributes to fields such as education, healthcare, online conversations, virtual assistants, and virtual reality. Early studies often focused solely on changing the mouth movements, which resulted in outcomes with limited practical applications. Recently, researchers have proposed a new approach of constructing the entire face, including face pose, neck, and shoulders. To achieve this, they need to generate through landmarks. However, creating stable landmarks that align well with the audio is a challenge. In this paper, we propose the KFusion of Dual-Domain model, a robust model that generates landmarks from audio. We separate the audio into two distinct domains to learn emotional information and facial context, then use a fusion mechanism based on the KAN model. Our model demonstrates high efficiency compared to recent models. This will lay the groundwork for the development of the audio-driven talking face generation problem in the future.

Title: TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiency

• Authors: Ahmed Imteaj, Md Zarif Hossain, Saika Zaman, Abdur R. Shahid
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05347
• Pdf URL: https://arxiv.org/pdf/2409.05347
• Copy Paste: [[2409.05347]] TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiency(https://arxiv.org/abs/2409.05347)
• Keywords: privacy, federate, fair
• Abstract: The rapid advancement and increasing complexity of pretrained models, exemplified by CLIP, offer significant opportunities as well as challenges for Federated Learning (FL), a critical component of privacy-preserving artificial intelligence. This research delves into the intricacies of integrating large foundation models like CLIP within FL frameworks to enhance privacy, efficiency, and adaptability across heterogeneous data landscapes. It specifically addresses the challenges posed by non-IID data distributions, the computational and communication overheads of leveraging such complex models, and the skewed representation of classes within datasets. We propose TriplePlay, a framework that integrates CLIP as an adapter to enhance FL's adaptability and performance across diverse data distributions. This approach addresses the long-tail distribution challenge to ensure fairness while reducing resource demands through quantization and low-rank adaptation techniques.Our simulation results demonstrate that TriplePlay effectively decreases GPU usage costs and speeds up the learning process, achieving convergence with reduced communication overhead.

Title: On the Convergence Analysis of Over-Parameterized Variational Autoencoders: A Neural Tangent Kernel Perspective

• Authors: Li Wang, Wei Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05349
• Pdf URL: https://arxiv.org/pdf/2409.05349
• Copy Paste: [[2409.05349]] On the Convergence Analysis of Over-Parameterized Variational Autoencoders: A Neural Tangent Kernel Perspective(https://arxiv.org/abs/2409.05349)
• Keywords: generative
• Abstract: Variational Auto-Encoders (VAEs) have emerged as powerful probabilistic models for generative tasks. However, their convergence properties have not been rigorously proven. The challenge of proving convergence is inherently difficult due to the highly non-convex nature of the training objective and the implementation of a Stochastic Neural Network (SNN) within VAE architectures. This paper addresses these challenges by characterizing the optimization trajectory of SNNs utilized in VAEs through the lens of Neural Tangent Kernel (NTK) techniques. These techniques govern the optimization and generalization behaviors of ultra-wide neural networks. We provide a mathematical proof of VAE convergence under mild assumptions, thus advancing the theoretical understanding of VAE optimization dynamics. Furthermore, we establish a novel connection between the optimization problem faced by over-parameterized SNNs and the Kernel Ridge Regression (KRR) problem. Our findings not only contribute to the theoretical foundation of VAEs but also open new avenues for investigating the optimization of generative models using advanced kernel methods. Our theoretical claims are verified by experimental simulations.

Title: Driving with Prior Maps: Unified Vector Prior Encoding for Autonomous Vehicle Mapping

• Authors: Shuang Zeng, Xinyuan Chang, Xinran Liu, Zheng Pan, Xing Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05352
• Pdf URL: https://arxiv.org/pdf/2409.05352
• Copy Paste: [[2409.05352]] Driving with Prior Maps: Unified Vector Prior Encoding for Autonomous Vehicle Mapping(https://arxiv.org/abs/2409.05352)
• Keywords: robust
• Abstract: High-Definition Maps (HD maps) are essential for the precise navigation and decision-making of autonomous vehicles, yet their creation and upkeep present significant cost and timeliness challenges. The online construction of HD maps using on-board sensors has emerged as a promising solution; however, these methods can be impeded by incomplete data due to occlusions and inclement weather. This paper proposes the PriorDrive framework to addresses these limitations by harnessing the power of prior maps, significantly enhancing the robustness and accuracy of online HD map construction. Our approach integrates a variety of prior maps, such as OpenStreetMap's Standard Definition Maps (SD maps), outdated HD maps from vendors, and locally constructed maps from historical vehicle data. To effectively encode this prior information into online mapping models, we introduce a Hybrid Prior Representation (HPQuery) that standardizes the representation of diverse map elements. At the core of PriorDrive is the Unified Vector Encoder (UVE), which employs a dual encoding mechanism to process vector data. The intra-vector encoder captures fine-grained local features, while the inter-vector encoder integrates global context. Furthermore, we propose a segment-level and point-level pre-training strategy that enables the UVE to learn the prior distribution of vector data, thereby improving the encoder's generalizability and performance. Through extensive testing on the nuScenes dataset, we demonstrate that PriorDrive is highly compatible with various online mapping models and substantially improves map prediction capabilities. The integration of prior maps through the PriorDrive framework offers a robust solution to the challenges of single-perception data, paving the way for more reliable autonomous vehicle navigation.

Title: FedBrain-Distill: Communication-Efficient Federated Brain Tumor Classification Using Ensemble Knowledge Distillation on Non-IID Data

• Authors: Rasoul Jafari Gohari, Laya Aliahmadipour, Ezat Valipour
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05359
• Pdf URL: https://arxiv.org/pdf/2409.05359
• Copy Paste: [[2409.05359]] FedBrain-Distill: Communication-Efficient Federated Brain Tumor Classification Using Ensemble Knowledge Distillation on Non-IID Data(https://arxiv.org/abs/2409.05359)
• Keywords: privacy, federate
• Abstract: Brain is one the most complex organs in the human body. Due to its complexity, classification of brain tumors still poses a significant challenge, making brain tumors a particularly serious medical issue. Techniques such as Machine Learning (ML) coupled with Magnetic Resonance Imaging (MRI) have paved the way for doctors and medical institutions to classify different types of tumors. However, these techniques suffer from limitations that violate patients privacy. Federated Learning (FL) has recently been introduced to solve such an issue, but the FL itself suffers from limitations like communication costs and dependencies on model architecture, forcing all models to have identical architectures. In this paper, we propose FedBrain-Distill, an approach that leverages Knowledge Distillation (KD) in an FL setting that maintains the users privacy and ensures the independence of FL clients in terms of model architecture. FedBrain-Distill uses an ensemble of teachers that distill their knowledge to a simple student model. The evaluation of FedBrain-Distill demonstrated high-accuracy results for both Independent and Identically Distributed (IID) and non-IID data with substantial low communication costs on the real-world Figshare brain tumor dataset. It is worth mentioning that we used Dirichlet distribution to partition the data into IID and non-IID data. All the implementation details are accessible through our Github repository.

Title: KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Models

• Authors: Yingshu Li, Zhanyu Wang, Yunyi Liu, Lei Wang, Lingqiao Liu, Luping Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05370
• Pdf URL: https://arxiv.org/pdf/2409.05370
• Copy Paste: [[2409.05370]] KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Models(https://arxiv.org/abs/2409.05370)
• Keywords: robust, large language model
• Abstract: Harnessing the robust capabilities of Large Language Models (LLMs) for narrative generation, logical reasoning, and common-sense knowledge integration, this study delves into utilizing LLMs to enhance automated radiology report generation (R2Gen). Despite the wealth of knowledge within LLMs, efficiently triggering relevant knowledge within these large models for specific tasks like R2Gen poses a critical research challenge. This paper presents KARGEN, a Knowledge-enhanced Automated radiology Report GENeration framework based on LLMs. Utilizing a frozen LLM to generate reports, the framework integrates a knowledge graph to unlock chest disease-related knowledge within the LLM to enhance the clinical utility of generated reports. This is achieved by leveraging the knowledge graph to distill disease-related features in a designed way. Since a radiology report encompasses both normal and disease-related findings, the extracted graph-enhanced disease-related features are integrated with regional image features, attending to both aspects. We explore two fusion methods to automatically prioritize and select the most relevant features. The fused features are employed by LLM to generate reports that are more sensitive to diseases and of improved quality. Our approach demonstrates promising results on the MIMIC-CXR and IU-Xray datasets.

Title: Towards Building a Robust Knowledge Intensive Question Answering Model with Large Language Models

• Authors: Hong Xingyun Hong, Shao Yan Shao, Wang Zhilin Wang, Duan Manni Duan, Jin Xiongnan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05385
• Pdf URL: https://arxiv.org/pdf/2409.05385
• Copy Paste: [[2409.05385]] Towards Building a Robust Knowledge Intensive Question Answering Model with Large Language Models(https://arxiv.org/abs/2409.05385)
• Keywords: robust, large language model
• Abstract: The development of LLMs has greatly enhanced the intelligence and fluency of question answering, while the emergence of retrieval enhancement has enabled models to better utilize external information. However, the presence of noise and errors in retrieved information poses challenges to the robustness of LLMs. In this work, to evaluate the model's performance under multiple interferences, we first construct a dataset based on machine reading comprehension datasets simulating various scenarios, including critical information absence, noise, and conflicts. To address the issue of model accuracy decline caused by noisy external information, we propose a data augmentation-based fine-tuning method to enhance LLM's robustness against noise. Additionally, contrastive learning approach is utilized to preserve the model's discrimination capability of external information. We have conducted experiments on both existing LLMs and our approach, the results are evaluated by GPT-4, which indicates that our proposed methods improve model robustness while strengthening the model's discrimination capability.

Title: Decoupling Contact for Fine-Grained Motion Style Transfer

• Authors: Xiangjun Tang, Linjun Wu, He Wang, Yiqian Wu, Bo Hu, Songnan Li, Xu Gong, Yuchen Liao, Qilong Kou, Xiaogang Jin
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.05387
• Pdf URL: https://arxiv.org/pdf/2409.05387
• Copy Paste: [[2409.05387]] Decoupling Contact for Fine-Grained Motion Style Transfer(https://arxiv.org/abs/2409.05387)
• Keywords: transformer
• Abstract: Motion style transfer changes the style of a motion while retaining its content and is useful in computer animations and games. Contact is an essential component of motion style transfer that should be controlled explicitly in order to express the style vividly while enhancing motion naturalness and quality. However, it is unknown how to decouple and control contact to achieve fine-grained control in motion style transfer. In this paper, we present a novel style transfer method for fine-grained control over contacts while achieving both motion naturalness and spatial-temporal variations of style. Based on our empirical evidence, we propose controlling contact indirectly through the hip velocity, which can be further decomposed into the trajectory and contact timing, respectively. To this end, we propose a new model that explicitly models the correlations between motions and trajectory/contact timing/style, allowing us to decouple and control each separately. Our approach is built around a motion manifold, where hip controls can be easily integrated into a Transformer-based decoder. It is versatile in that it can generate motions directly as well as be used as post-processing for existing methods to improve quality and contact controllability. In addition, we propose a new metric that measures a correlation pattern of motions based on our empirical evidence, aligning well with human perception in terms of motion naturalness. Based on extensive evaluation, our method outperforms existing methods in terms of style expressivity and motion quality.

Title: TAVP: Task-Adaptive Visual Prompt for Cross-domain Few-shot Segmentation

• Authors: Jiaqi Yang, Ye Huang, Xiangjian He, Linlin Shen, Guoping Qiu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05393
• Pdf URL: https://arxiv.org/pdf/2409.05393
• Copy Paste: [[2409.05393]] TAVP: Task-Adaptive Visual Prompt for Cross-domain Few-shot Segmentation(https://arxiv.org/abs/2409.05393)
• Keywords: extraction, generative, segmentation
• Abstract: Under the backdrop of large-scale pre-training, large visual models (LVM) have demonstrated significant potential in image understanding. The recent emergence of the Segment Anything Model (SAM) has brought a qualitative shift in the field of image segmentation, supporting flexible interactive cues and strong learning capabilities. However, its performance often falls short in cross-domain and few-shot applications. Transferring prior knowledge from foundation models to new applications while preserving learning capabilities is worth exploring. This work proposes a task-adaptive prompt framework based on SAM, a new paradigm for Cross-dominan few-shot segmentation (CD-FSS). First, a Multi-level Feature Fusion (MFF) was used for integrated feature extraction. Besides, an additional Class Domain Task-Adaptive Auto-Prompt (CDTAP) module was combined with the segmentation branch for class-domain agnostic feature extraction and high-quality learnable prompt production. This significant advancement uses a unique generative approach to prompts alongside a comprehensive model structure and specialized prototype computation. While ensuring that the prior knowledge of SAM is not discarded, the new branch disentangles category and domain information through prototypes, guiding it in adapting the CD-FSS. We have achieved the best results on three benchmarks compared to the recent state-of-the-art (SOTA) methods. Comprehensive experiments showed that after task-specific and weighted guidance, the abundant feature information of SAM can be better learned for CD-FSS.

Title: Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling

• Authors: Georgios Pantazopoulos, Malvina Nikandrou, Alessandro Suglia, Oliver Lemon, Arash Eshghi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05395
• Pdf URL: https://arxiv.org/pdf/2409.05395
• Copy Paste: [[2409.05395]] Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling(https://arxiv.org/abs/2409.05395)
• Keywords: transformer
• Abstract: This study explores replacing Transformers in Visual Language Models (VLMs) with Mamba, a recent structured state space model (SSM) that demonstrates promising performance in sequence modeling. We test models up to 3B parameters under controlled conditions, showing that Mamba-based VLMs outperforms Transformers-based VLMs in captioning, question answering, and reading comprehension. However, we find that Transformers achieve greater performance in visual grounding and the performance gap widens with scale. We explore two hypotheses to explain this phenomenon: 1) the effect of task-agnostic visual encoding on the updates of the hidden states, and 2) the difficulty in performing visual grounding from the perspective of in-context multimodal retrieval. Our results indicate that a task-aware encoding yields minimal performance gains on grounding, however, Transformers significantly outperform Mamba at in-context multimodal retrieval. Overall, Mamba shows promising performance on tasks where the correct output relies on a summary of the image but struggles when retrieval of explicit information from the context is required.

Title: Sequential Posterior Sampling with Diffusion Models

• Authors: Tristan S.W. Stevens, Oisín Nolan, Jean-Luc Robert, Ruud J.G. van Sloun
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05399
• Pdf URL: https://arxiv.org/pdf/2409.05399
• Copy Paste: [[2409.05399]] Sequential Posterior Sampling with Diffusion Models(https://arxiv.org/abs/2409.05399)
• Keywords: diffusion, transformer, generative
• Abstract: Diffusion models have quickly risen in popularity for their ability to model complex distributions and perform effective posterior sampling. Unfortunately, the iterative nature of these generative models makes them computationally expensive and unsuitable for real-time sequential inverse problems such as ultrasound imaging. Considering the strong temporal structure across sequences of frames, we propose a novel approach that models the transition dynamics to improve the efficiency of sequential diffusion posterior sampling in conditional image synthesis. Through modeling sequence data using a video vision transformer (ViViT) transition model based on previous diffusion outputs, we can initialize the reverse diffusion trajectory at a lower noise scale, greatly reducing the number of iterations required for convergence. We demonstrate the effectiveness of our approach on a real-world dataset of high frame rate cardiac ultrasound images and show that it achieves the same performance as a full diffusion trajectory while accelerating inference 25$\times$, enabling real-time posterior sampling. Furthermore, we show that the addition of a transition model improves the PSNR up to 8\% in cases with severe motion. Our method opens up new possibilities for real-time applications of diffusion models in imaging and other domains requiring real-time inference.

Title: A Survey of Multimodal Composite Editing and Retrieval

• Authors: Suyan Li, Fuxiang Huang, Lei Zhang
• Subjects: cs.CV, cs.AI, cs.IR, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.05405
• Pdf URL: https://arxiv.org/pdf/2409.05405
• Copy Paste: [[2409.05405]] A Survey of Multimodal Composite Editing and Retrieval(https://arxiv.org/abs/2409.05405)
• Keywords: transformer
• Abstract: In the real world, where information is abundant and diverse across different modalities, understanding and utilizing various data types to improve retrieval systems is a key focus of research. Multimodal composite retrieval integrates diverse modalities such as text, image and audio, etc. to provide more accurate, personalized, and contextually relevant results. To facilitate a deeper understanding of this promising direction, this survey explores multimodal composite editing and retrieval in depth, covering image-text composite editing, image-text composite retrieval, and other multimodal composite retrieval. In this survey, we systematically organize the application scenarios, methods, benchmarks, experiments, and future directions. Multimodal learning is a hot topic in large model era, and have also witnessed some surveys in multimodal learning and vision-language models with transformers published in the PAMI journal. To the best of our knowledge, this survey is the first comprehensive review of the literature on multimodal composite retrieval, which is a timely complement of multimodal fusion to existing reviews. To help readers' quickly track this field, we build the project page for this survey, which can be found at this https URL.

Title: KRONC: Keypoint-based Robust Camera Optimization for 3D Car Reconstruction

• Authors: Davide Di Nucci, Alessandro Simoni, Matteo Tomei, Luca Ciuffreda, Roberto Vezzani, Rita Cucchiara
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05407
• Pdf URL: https://arxiv.org/pdf/2409.05407
• Copy Paste: [[2409.05407]] KRONC: Keypoint-based Robust Camera Optimization for 3D Car Reconstruction(https://arxiv.org/abs/2409.05407)
• Keywords: robust, diffusion
• Abstract: The three-dimensional representation of objects or scenes starting from a set of images has been a widely discussed topic for years and has gained additional attention after the diffusion of NeRF-based approaches. However, an underestimated prerequisite is the knowledge of camera poses or, more specifically, the estimation of the extrinsic calibration parameters. Although excellent general-purpose Structure-from-Motion methods are available as a pre-processing step, their computational load is high and they require a lot of frames to guarantee sufficient overlapping among the views. This paper introduces KRONC, a novel approach aimed at inferring view poses by leveraging prior knowledge about the object to reconstruct and its representation through semantic keypoints. With a focus on vehicle scenes, KRONC is able to estimate the position of the views as a solution to a light optimization problem targeting the convergence of keypoints' back-projections to a singular point. To validate the method, a specific dataset of real-world car scenes has been collected. Experiments confirm KRONC's ability to generate excellent estimates of camera poses starting from very coarse initialization. Results are comparable with Structure-from-Motion methods with huge savings in computation. Code and data will be made publicly available.

Title: CipherDM: Secure Three-Party Inference for Diffusion Model Sampling

• Authors: Xin Zhao, Xiaojun Chen, Xudong Chen, He Li, Tingyu Fan, Zhendong Zhao
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05414
• Pdf URL: https://arxiv.org/pdf/2409.05414
• Copy Paste: [[2409.05414]] CipherDM: Secure Three-Party Inference for Diffusion Model Sampling(https://arxiv.org/abs/2409.05414)
• Keywords: secure, privacy, protect, diffusion
• Abstract: Diffusion Models (DMs) achieve state-of-the-art synthesis results in image generation and have been applied to various fields. However, DMs sometimes seriously violate user privacy during usage, making the protection of privacy an urgent issue. Using traditional privacy computing schemes like Secure Multi-Party Computation (MPC) directly in DMs faces significant computation and communication challenges. To address these issues, we propose CipherDM, the first novel, versatile and universal framework applying MPC technology to DMs for secure sampling, which can be widely implemented on multiple DM based tasks. We thoroughly analyze sampling latency breakdown, find time-consuming parts and design corresponding secure MPC protocols for computing nonlinear activations including SoftMax, SiLU and Mish. CipherDM is evaluated on popular architectures (DDPM, DDIM) using MNIST dataset and on SD deployed by diffusers. Compared to direct implementation on SPU, our approach improves running time by approximately 1.084\times \sim 2.328\times, and reduces communication costs by approximately 1.212\times \sim 1.791\times.

Title: AD-Net: Attention-based dilated convolutional residual network with guided decoder for robust skin lesion segmentation

• Authors: Asim Naveed, Syed S. Naqvi, Tariq M. Khan, Shahzaib Iqbal, M. Yaqoob Wani, Haroon Ahmed Khan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05420
• Pdf URL: https://arxiv.org/pdf/2409.05420
• Copy Paste: [[2409.05420]] AD-Net: Attention-based dilated convolutional residual network with guided decoder for robust skin lesion segmentation(https://arxiv.org/abs/2409.05420)
• Keywords: robust, segmentation
• Abstract: In computer-aided diagnosis tools employed for skin cancer treatment and early diagnosis, skin lesion segmentation is important. However, achieving precise segmentation is challenging due to inherent variations in appearance, contrast, texture, and blurry lesion boundaries. This research presents a robust approach utilizing a dilated convolutional residual network, which incorporates an attention-based spatial feature enhancement block (ASFEB) and employs a guided decoder strategy. In each dilated convolutional residual block, dilated convolution is employed to broaden the receptive field with varying dilation rates. To improve the spatial feature information of the encoder, we employed an attention-based spatial feature enhancement block in the skip connections. The ASFEB in our proposed method combines feature maps obtained from average and maximum-pooling operations. These combined features are then weighted using the active outcome of global average pooling and convolution operations. Additionally, we have incorporated a guided decoder strategy, where each decoder block is optimized using an individual loss function to enhance the feature learning process in the proposed AD-Net. The proposed AD-Net presents a significant benefit by necessitating fewer model parameters compared to its peer methods. This reduction in parameters directly impacts the number of labeled data required for training, facilitating faster convergence during the training process. The effectiveness of the proposed AD-Net was evaluated using four public benchmark datasets. We conducted a Wilcoxon signed-rank test to verify the efficiency of the AD-Net. The outcomes suggest that our method surpasses other cutting-edge methods in performance, even without the implementation of data augmentation strategies.

Title: Distribution Discrepancy and Feature Heterogeneity for Active 3D Object Detection

• Authors: Huang-Yu Chen, Jia-Fong Yeh, Jia-Wei Liao, Pin-Hsuan Peng, Winston H. Hsu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05425
• Pdf URL: https://arxiv.org/pdf/2409.05425
• Copy Paste: [[2409.05425]] Distribution Discrepancy and Feature Heterogeneity for Active 3D Object Detection(https://arxiv.org/abs/2409.05425)
• Keywords: robust
• Abstract: LiDAR-based 3D object detection is a critical technology for the development of autonomous driving and robotics. However, the high cost of data annotation limits its advancement. We propose a novel and effective active learning (AL) method called Distribution Discrepancy and Feature Heterogeneity (DDFH), which simultaneously considers geometric features and model embeddings, assessing information from both the instance-level and frame-level perspectives. Distribution Discrepancy evaluates the difference and novelty of instances within the unlabeled and labeled distributions, enabling the model to learn efficiently with limited data. Feature Heterogeneity ensures the heterogeneity of intra-frame instance features, maintaining feature diversity while avoiding redundant or similar instances, thus minimizing annotation costs. Finally, multiple indicators are efficiently aggregated using Quantile Transform, providing a unified measure of informativeness. Extensive experiments demonstrate that DDFH outperforms the current state-of-the-art (SOTA) methods on the KITTI and Waymo datasets, effectively reducing the bounding box annotation cost by 56.3% and showing robustness when working with both one-stage and two-stage models.

Title: TextToucher: Fine-Grained Text-to-Touch Generation

• Authors: Jiahang Tu, Hao Fu, Fengyu Yang, Hanbin Zhao, Chao Zhang, Hui Qian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05427
• Pdf URL: https://arxiv.org/pdf/2409.05427
• Copy Paste: [[2409.05427]] TextToucher: Fine-Grained Text-to-Touch Generation(https://arxiv.org/abs/2409.05427)
• Keywords: diffusion, transformer, large language model
• Abstract: Tactile sensation plays a crucial role in the development of multi-modal large models and embodied intelligence. To collect tactile data with minimal cost as possible, a series of studies have attempted to generate tactile images by vision-to-touch image translation. However, compared to text modality, visual modality-driven tactile generation cannot accurately depict human tactile sensation. In this work, we analyze the characteristics of tactile images in detail from two granularities: object-level (tactile texture, tactile shape), and sensor-level (gel status). We model these granularities of information through text descriptions and propose a fine-grained Text-to-Touch generation method (TextToucher) to generate high-quality tactile samples. Specifically, we introduce a multimodal large language model to build the text sentences about object-level tactile information and employ a set of learnable text prompts to represent the sensor-level tactile information. To better guide the tactile generation process with the built text information, we fuse the dual grains of text information and explore various dual-grain text conditioning methods within the diffusion transformer architecture. Furthermore, we propose a Contrastive Text-Touch Pre-training (CTTP) metric to precisely evaluate the quality of text-driven generated tactile data. Extensive experiments demonstrate the superiority of our TextToucher method. The source codes will be available at \url{this https URL}.

Title: EndoOmni: Zero-Shot Cross-Dataset Depth Estimation in Endoscopy by Robust Self-Learning from Noisy Labels

• Authors: Qingyao Tian, Zhen Chen, Huai Liao, Xinyan Huang, Lujie Li, Sebastien Ourselin, Hongbin Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05442
• Pdf URL: https://arxiv.org/pdf/2409.05442
• Copy Paste: [[2409.05442]] EndoOmni: Zero-Shot Cross-Dataset Depth Estimation in Endoscopy by Robust Self-Learning from Noisy Labels(https://arxiv.org/abs/2409.05442)
• Keywords: robust
• Abstract: Single-image depth estimation is essential for endoscopy tasks such as localization, reconstruction, and augmented reality. Most existing methods in surgical scenes focus on in-domain depth estimation, limiting their real-world applicability. This constraint stems from the scarcity and inferior labeling quality of medical data for training. In this work, we present EndoOmni, the first foundation model for zero-shot cross-domain depth estimation for endoscopy. To harness the potential of diverse training data, we refine the advanced self-learning paradigm that employs a teacher model to generate pseudo-labels, guiding a student model trained on large-scale labeled and unlabeled data. To address training disturbance caused by inherent noise in depth labels, we propose a robust training framework that leverages both depth labels and estimated confidence from the teacher model to jointly guide the student model training. Moreover, we propose a weighted scale-and-shift invariant loss to adaptively adjust learning weights based on label confidence, thus imposing learning bias towards cleaner label pixels while reducing the influence of highly noisy pixels. Experiments on zero-shot relative depth estimation show that our EndoOmni improves state-of-the-art methods in medical imaging for 41\% and existing foundation models for 25\% in terms of absolute relative error on specific dataset. Furthermore, our model provides strong initialization for fine-tuning to metric depth estimation, maintaining superior performance in both in-domain and out-of-domain scenarios. The source code will be publicly available.

Title: Representational Analysis of Binding in Large Language Models

• Authors: Qin Dai, Benjamin Heinzerling, Kentaro Inui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05448
• Pdf URL: https://arxiv.org/pdf/2409.05448
• Copy Paste: [[2409.05448]] Representational Analysis of Binding in Large Language Models(https://arxiv.org/abs/2409.05448)
• Keywords: large language model
• Abstract: Entity tracking is essential for complex reasoning. To perform in-context entity tracking, language models (LMs) must bind an entity to its attribute (e.g., bind a container to its content) to recall attribute for a given entity. For example, given a context mentioning ``The coffee is in Box Z, the stone is in Box M, the map is in Box H'', to infer ``Box Z contains the coffee'' later, LMs must bind ``Box Z'' to ``coffee''. To explain the binding behaviour of LMs, Feng and Steinhardt (2023) introduce a Binding ID mechanism and state that LMs use a abstract concept called Binding ID (BI) to internally mark entity-attribute pairs. However, they have not directly captured the BI determinant information from entity activations. In this work, we provide a novel view of the Binding ID mechanism by localizing the prototype of BI information. Specifically, we discover that there exists a low-rank subspace in the hidden state (or activation) of LMs, that primarily encodes the order of entity and attribute and which is used as the prototype of BI to causally determine the binding. To identify this subspace, we choose principle component analysis as our first attempt and it is empirically proven to be effective. Moreover, we also discover that when editing representations along directions in the subspace, LMs tend to bind a given entity to other attributes accordingly. For example, by patching activations along the BI encoding direction we can make the LM to infer ``Box Z contains the stone'' and ``Box Z contains the map''.

Title: DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation

• Authors: Wei Wu, Xi Guo, Weixuan Tang, Tingxuan Huang, Chiyu Wang, Dongyue Chen, Chenjing Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05463
• Pdf URL: https://arxiv.org/pdf/2409.05463
• Copy Paste: [[2409.05463]] DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation(https://arxiv.org/abs/2409.05463)
• Keywords: robust, transformer, generative
• Abstract: Recent advancements in generative models have provided promising solutions for synthesizing realistic driving videos, which are crucial for training autonomous driving perception models. However, existing approaches often struggle with multi-view video generation due to the challenges of integrating 3D information while maintaining spatial-temporal consistency and effectively learning from a unified model. In this paper, we propose an end-to-end framework named DriveScape for multi-view, 3D condition-guided video generation. DriveScape not only streamlines the process by integrating camera data to ensure comprehensive spatial-temporal coverage, but also introduces a Bi-Directional Modulated Transformer module to effectively align 3D road structural information. As a result, our approach enables precise control over video generation, significantly enhancing realism and providing a robust solution for generating multi-view driving videos. Our framework achieves state-of-the-art results on the nuScenes dataset, demonstrating impressive generative quality metrics with an FID score of 8.34 and an FVD score of 76.39, as well as superior performance across various perception tasks. This paves the way for more accurate environmental simulations in autonomous driving. Code will be available at our project homepage.

Title: Retrofitting Temporal Graph Neural Networks with Transformer

• Authors: Qiang Huang, Xiao Yan, Xin Wang, Susie Xi Rao, Zhichao Han, Fangcheng Fu, Wentao Zhang, Jiawei Jiang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05477
• Pdf URL: https://arxiv.org/pdf/2409.05477
• Copy Paste: [[2409.05477]] Retrofitting Temporal Graph Neural Networks with Transformer(https://arxiv.org/abs/2409.05477)
• Keywords: transformer
• Abstract: Temporal graph neural networks (TGNNs) outperform regular GNNs by incorporating time information into graph-based operations. However, TGNNs adopt specialized models (e.g., TGN, TGAT, and APAN ) and require tailored training frameworks (e.g., TGL and ETC). In this paper, we propose TF-TGN, which uses Transformer decoder as the backbone model for TGNN to enjoy Transformer's codebase for efficient training. In particular, Transformer achieves tremendous success for language modeling, and thus the community developed high-performance kernels (e.g., flash-attention and memory-efficient attention) and efficient distributed training schemes (e.g., PyTorch FSDP, DeepSpeed, and Megatron-LM). We observe that TGNN resembles language modeling, i.e., the message aggregation operation between chronologically occurring nodes and their temporal neighbors in TGNNs can be structured as sequence modeling. Beside this similarity, we also incorporate a series of algorithm designs including suffix infilling, temporal graph attention with self-loop, and causal masking self-attention to make TF-TGN work. During training, existing systems are slow in transforming the graph topology and conducting graph sampling. As such, we propose methods to parallelize the CSR format conversion and graph sampling. We also adapt Transformer codebase to train TF-TGN efficiently with multiple GPUs. We experiment with 9 graphs and compare with 2 state-of-the-art TGNN training frameworks. The results show that TF-TGN can accelerate training by over 2.20 while providing comparable or even superior accuracy to existing SOTA TGNNs. TF-TGN is available at this https URL.

Title: CRADLE-VAE: Enhancing Single-Cell Gene Perturbation Modeling with Counterfactual Reasoning-based Artifact Disentanglement

• Authors: Seungheun Baek, Soyon Park, Yan Ting Chok, Junhyun Lee, Jueon Park, Mogan Gim, Jaewoo Kang
• Subjects: cs.LG, cs.AI, q-bio.GN, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2409.05484
• Pdf URL: https://arxiv.org/pdf/2409.05484
• Copy Paste: [[2409.05484]] CRADLE-VAE: Enhancing Single-Cell Gene Perturbation Modeling with Counterfactual Reasoning-based Artifact Disentanglement(https://arxiv.org/abs/2409.05484)
• Keywords: robust, generative
• Abstract: Predicting cellular responses to various perturbations is a critical focus in drug discovery and personalized therapeutics, with deep learning models playing a significant role in this endeavor. Single-cell datasets contain technical artifacts that may hinder the predictability of such models, which poses quality control issues highly regarded in this area. To address this, we propose CRADLE-VAE, a causal generative framework tailored for single-cell gene perturbation modeling, enhanced with counterfactual reasoning-based artifact disentanglement. Throughout training, CRADLE-VAE models the underlying latent distribution of technical artifacts and perturbation effects present in single-cell datasets. It employs counterfactual reasoning to effectively disentangle such artifacts by modulating the latent basal spaces and learns robust features for generating cellular response data with improved quality. Experimental results demonstrate that this approach improves not only treatment effect estimation performance but also generative quality as well. The CRADLE-VAE codebase is publicly available at this https URL.

Title: Elsevier Arena: Human Evaluation of Chemistry/Biology/Health Foundational Large Language Models

• Authors: Camilo Thorne, Christian Druckenbrodt, Kinga Szarkowska, Deepika Goyal, Pranita Marajan, Vijay Somanath, Corey Harper, Mao Yan, Tony Scerri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05486
• Pdf URL: https://arxiv.org/pdf/2409.05486
• Copy Paste: [[2409.05486]] Elsevier Arena: Human Evaluation of Chemistry/Biology/Health Foundational Large Language Models(https://arxiv.org/abs/2409.05486)
• Keywords: transformer, large language model
• Abstract: The quality and capabilities of large language models cannot be currently fully assessed with automated, benchmark evaluations. Instead, human evaluations that expand on traditional qualitative techniques from natural language generation literature are required. One recent best-practice consists in using A/B-testing frameworks, which capture preferences of human evaluators for specific models. In this paper we describe a human evaluation experiment focused on the biomedical domain (health, biology, chemistry/pharmacology) carried out at Elsevier. In it a large but not massive (8.8B parameter) decoder-only foundational transformer trained on a relatively small (135B tokens) but highly curated collection of Elsevier datasets is compared to OpenAI's GPT-3.5-turbo and Meta's foundational 7B parameter Llama 2 model against multiple criteria. Results indicate -- even if IRR scores were generally low -- a preference towards GPT-3.5-turbo, and hence towards models that possess conversational abilities, are very large and were trained on very large datasets. But at the same time, indicate that for less massive models training on smaller but well-curated training sets can potentially give rise to viable alternatives in the biomedical domain.

Title: A Taxonomy of Miscompressions: Preparing Image Forensics for Neural Compression

• Authors: Nora Hofer, Rainer Böhme
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05490
• Pdf URL: https://arxiv.org/pdf/2409.05490
• Copy Paste: [[2409.05490]] A Taxonomy of Miscompressions: Preparing Image Forensics for Neural Compression(https://arxiv.org/abs/2409.05490)
• Keywords: generative
• Abstract: Neural compression has the potential to revolutionize lossy image compression. Based on generative models, recent schemes achieve unprecedented compression rates at high perceptual quality but compromise semantic fidelity. Details of decompressed images may appear optically flawless but semantically different from the originals, making compression errors difficult or impossible to detect. We explore the problem space and propose a provisional taxonomy of miscompressions. It defines three types of 'what happens' and has a binary 'high impact' flag indicating miscompressions that alter symbols. We discuss how the taxonomy can facilitate risk communication and research into mitigations.

Title: An Atmospheric Correction Integrated LULC Segmentation Model for High-Resolution Satellite Imagery

• Authors: Soham Mukherjee, Yash Dixit, Naman Srivastava, Joel D Joy, Rohan Olikara, Koesha Sinha, Swarup E, Rakshit Ramesh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05494
• Pdf URL: https://arxiv.org/pdf/2409.05494
• Copy Paste: [[2409.05494]] An Atmospheric Correction Integrated LULC Segmentation Model for High-Resolution Satellite Imagery(https://arxiv.org/abs/2409.05494)
• Keywords: segmentation
• Abstract: The integration of fine-scale multispectral imagery with deep learning models has revolutionized land use and land cover (LULC) classification. However, the atmospheric effects present in Top-of-Atmosphere sensor measured Digital Number values must be corrected to retrieve accurate Bottom-of-Atmosphere surface reflectance for reliable analysis. This study employs look-up-table-based radiative transfer simulations to estimate the atmospheric path reflectance and transmittance for atmospherically correcting high-resolution CARTOSAT-3 Multispectral (MX) imagery for several Indian cities. The corrected surface reflectance data were subsequently used in supervised and semi-supervised segmentation models, demonstrating stability in multi-class (buildings, roads, trees and water bodies) LULC segmentation accuracy, particularly in scenarios with sparsely labelled data.

Title: Optimizing VarLiNGAM for Scalable and Efficient Time Series Causal Discovery

• Authors: Ziyang Jiao, Ce Guo, Wayne Luk
• Subjects: cs.LG, cs.DC, cs.PF, stat.CO
• Abstract URL: https://arxiv.org/abs/2409.05500
• Pdf URL: https://arxiv.org/pdf/2409.05500
• Copy Paste: [[2409.05500]] Optimizing VarLiNGAM for Scalable and Efficient Time Series Causal Discovery(https://arxiv.org/abs/2409.05500)
• Keywords: robust
• Abstract: Causal discovery is designed to identify causal relationships in data, a task that has become increasingly complex due to the computational demands of traditional methods such as VarLiNGAM, which combines Vector Autoregressive Model with Linear Non-Gaussian Acyclic Model for time series data. This study is dedicated to optimising causal discovery specifically for time series data, which is common in practical applications. Time series causal discovery is particularly challenging due to the need to account for temporal dependencies and potential time lag effects. By designing a specialised dataset generator and reducing the computational complexity of the VarLiNGAM model from \( O(m^3 \cdot n) \) to \( O(m^3 + m^2 \cdot n) \), this study significantly improves the feasibility of processing large datasets. The proposed methods have been validated on advanced computational platforms and tested across simulated, real-world, and large-scale datasets, showcasing enhanced efficiency and performance. The optimised algorithm achieved 7 to 13 times speedup compared with the original algorithm and around 4.5 times speedup compared with the GPU-accelerated version on large-scale datasets with feature sizes between 200 and 400. Our methods aim to push the boundaries of current causal discovery capabilities, making them more robust, scalable, and applicable to real-world scenarios, thus facilitating breakthroughs in various fields such as healthcare and finance.

Title: Harmonic Reasoning in Large Language Models

• Authors: Anna Kruspe
• Subjects: cs.CL, cs.AI, cs.SD
• Abstract URL: https://arxiv.org/abs/2409.05521
• Pdf URL: https://arxiv.org/pdf/2409.05521
• Copy Paste: [[2409.05521]] Harmonic Reasoning in Large Language Models(https://arxiv.org/abs/2409.05521)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) are becoming very popular and are used for many different purposes, including creative tasks in the arts. However, these models sometimes have trouble with specific reasoning tasks, especially those that involve logical thinking and counting. This paper looks at how well LLMs understand and reason when dealing with musical tasks like figuring out notes from intervals and identifying chords and scales. We tested GPT-3.5 and GPT-4o to see how they handle these tasks. Our results show that while LLMs do well with note intervals, they struggle with more complicated tasks like recognizing chords and scales. This points out clear limits in current LLM abilities and shows where we need to make them better, which could help improve how they think and work in both artistic and other complex areas. We also provide an automatically generated benchmark data set for the described tasks.

Title: Exploring Rich Subjective Quality Information for Image Quality Assessment in the Wild

• Authors: Xiongkuo Min, Yixuan Gao, Yuqin Cao, Guangtao Zhai, Wenjun Zhang, Huifang Sun, Chang Wen Chen
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.05540
• Pdf URL: https://arxiv.org/pdf/2409.05540
• Copy Paste: [[2409.05540]] Exploring Rich Subjective Quality Information for Image Quality Assessment in the Wild(https://arxiv.org/abs/2409.05540)
• Keywords: transformer
• Abstract: Traditional in the wild image quality assessment (IQA) models are generally trained with the quality labels of mean opinion score (MOS), while missing the rich subjective quality information contained in the quality ratings, for example, the standard deviation of opinion scores (SOS) or even distribution of opinion scores (DOS). In this paper, we propose a novel IQA method named RichIQA to explore the rich subjective rating information beyond MOS to predict image quality in the wild. RichIQA is characterized by two key novel designs: (1) a three-stage image quality prediction network which exploits the powerful feature representation capability of the Convolutional vision Transformer (CvT) and mimics the short-term and long-term memory mechanisms of human brain; (2) a multi-label training strategy in which rich subjective quality information like MOS, SOS and DOS are concurrently used to train the quality prediction network. Powered by these two novel designs, RichIQA is able to predict the image quality in terms of a distribution, from which the mean image quality can be subsequently obtained. Extensive experimental results verify that the three-stage network is tailored to predict rich quality information, while the multi-label training strategy can fully exploit the potentials within subjective quality rating and enhance the prediction performance and generalizability of the network. RichIQA outperforms state-of-the-art competitors on multiple large-scale in the wild IQA databases with rich subjective rating labels. The code of RichIQA will be made publicly available on GitHub.

Title: Efficient Quality Estimation of True Random Bit-streams

• Authors: Cesare Caratozzolo, Valeria Rossi, Kamil Witek, Alberto Trombetta, Massimo Caccia
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.05543
• Pdf URL: https://arxiv.org/pdf/2409.05543
• Copy Paste: [[2409.05543]] Efficient Quality Estimation of True Random Bit-streams(https://arxiv.org/abs/2409.05543)
• Keywords: security, robust
• Abstract: Generating random bit streams is required in various applications, most notably cyber-security. Ensuring high-quality and robust randomness is crucial to mitigate risks associated with predictability and system compromise. True random numbers provide the highest unpredictability levels. However, potential biases in the processes exploited for the random number generation must be carefully monitored. This paper reports the implementation and characterization of an on-line procedure for the detection of anomalies in a true random bit stream. It is based on the NIST Adaptive Proportion and Repetition Count tests, complemented by statistical analysis relying on the Monobit and RUNS. The procedure is firmware implemented and performed simultaneously with the bit stream generation, and providing as well an estimate of the entropy of the source. The experimental validation of the approach is performed upon the bit streams generated by a quantum, silicon-based entropy source.

Title: Seeing Through the Mask: Rethinking Adversarial Examples for CAPTCHAs

• Authors: Yahya Jabary, Andreas Plesner, Turlan Kuzhagaliyev, Roger Wattenhofer
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05558
• Pdf URL: https://arxiv.org/pdf/2409.05558
• Copy Paste: [[2409.05558]] Seeing Through the Mask: Rethinking Adversarial Examples for CAPTCHAs(https://arxiv.org/abs/2409.05558)
• Keywords: robust, transformer
• Abstract: Modern CAPTCHAs rely heavily on vision tasks that are supposedly hard for computers but easy for humans. However, advances in image recognition models pose a significant threat to such CAPTCHAs. These models can easily be fooled by generating some well-hidden "random" noise and adding it to the image, or hiding objects in the image. However, these methods are model-specific and thus can not aid CAPTCHAs in fooling all models. We show in this work that by allowing for more significant changes to the images while preserving the semantic information and keeping it solvable by humans, we can fool many state-of-the-art models. Specifically, we demonstrate that by adding masks of various intensities the Accuracy @ 1 (Acc@1) drops by more than 50%-points for all models, and supposedly robust models such as vision transformers see an Acc@1 drop of 80%-points. These masks can therefore effectively fool modern image classifiers, thus showing that machines have not caught up with humans -- yet.

Title: LEROjD: Lidar Extended Radar-Only Object Detection

• Authors: Patrick Palmer, Martin Krüger, Stefan Schütte, Richard Altendorfer, Ganesh Adam, Torsten Bertram
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.05564
• Pdf URL: https://arxiv.org/pdf/2409.05564
• Copy Paste: [[2409.05564]] LEROjD: Lidar Extended Radar-Only Object Detection(https://arxiv.org/abs/2409.05564)
• Keywords: robust
• Abstract: Accurate 3D object detection is vital for automated driving. While lidar sensors are well suited for this task, they are expensive and have limitations in adverse weather conditions. 3+1D imaging radar sensors offer a cost-effective, robust alternative but face challenges due to their low resolution and high measurement noise. Existing 3+1D imaging radar datasets include radar and lidar data, enabling cross-modal model improvements. Although lidar should not be used during inference, it can aid the training of radar-only object detectors. We explore two strategies to transfer knowledge from the lidar to the radar domain and radar-only object detectors: 1. multi-stage training with sequential lidar point cloud thin-out, and 2. cross-modal knowledge distillation. In the multi-stage process, three thin-out methods are examined. Our results show significant performance gains of up to 4.2 percentage points in mean Average Precision with multi-stage training and up to 3.9 percentage points with knowledge distillation by initializing the student with the teacher's weights. The main benefit of these approaches is their applicability to other 3D object detection networks without altering their architecture, as we show by analyzing it on two different object detectors. Our code is available at this https URL

Title: Learning to Model Graph Structural Information on MLPs via Graph Structure Self-Contrasting

• Authors: Lirong Wu, Haitao Lin, Guojiang Zhao, Cheng Tan, Stan Z. Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05573
• Pdf URL: https://arxiv.org/pdf/2409.05573
• Copy Paste: [[2409.05573]] Learning to Model Graph Structural Information on MLPs via Graph Structure Self-Contrasting(https://arxiv.org/abs/2409.05573)
• Keywords: robust
• Abstract: Recent years have witnessed great success in handling graph-related tasks with Graph Neural Networks (GNNs). However, most existing GNNs are based on message passing to perform feature aggregation and transformation, where the structural information is explicitly involved in the forward propagation by coupling with node features through graph convolution at each layer. As a result, subtle feature noise or structure perturbation may cause severe error propagation, resulting in extremely poor robustness. In this paper, we rethink the roles played by graph structural information in graph data training and identify that message passing is not the only path to modeling structural information. Inspired by this, we propose a simple but effective Graph Structure Self-Contrasting (GSSC) framework that learns graph structural information without message passing. The proposed framework is based purely on Multi-Layer Perceptrons (MLPs), where the structural information is only implicitly incorporated as prior knowledge to guide the computation of supervision signals, substituting the explicit message propagation as in GNNs. Specifically, it first applies structural sparsification to remove potentially uninformative or noisy edges in the neighborhood, and then performs structural self-contrasting in the sparsified neighborhood to learn robust node representations. Finally, structural sparsification and self-contrasting are formulated as a bi-level optimization problem and solved in a unified framework. Extensive experiments have qualitatively and quantitatively demonstrated that the GSSC framework can produce truly encouraging performance with better generalization and robustness than other leading competitors.

Title: Latent 3D Brain MRI Counterfactual

• Authors: Wei Peng, Tian Xia, Fabio De Sousa Ribeiro, Tomas Bosschieter, Ehsan Adeli, Qingyu Zhao, Ben Glocker, Kilian M. Pohl
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05585
• Pdf URL: https://arxiv.org/pdf/2409.05585
• Copy Paste: [[2409.05585]] Latent 3D Brain MRI Counterfactual(https://arxiv.org/abs/2409.05585)
• Keywords: generative
• Abstract: The number of samples in structural brain MRI studies is often too small to properly train deep learning models. Generative models show promise in addressing this issue by effectively learning the data distribution and generating high-fidelity MRI. However, they struggle to produce diverse, high-quality data outside the distribution defined by the training data. One way to address the issue is using causal models developed for 3D volume counterfactuals. However, accurately modeling causality in high-dimensional spaces is a challenge so that these models generally generate 3D brain MRIS of lower quality. To address these challenges, we propose a two-stage method that constructs a Structural Causal Model (SCM) within the latent space. In the first stage, we employ a VQ-VAE to learn a compact embedding of the MRI volume. Subsequently, we integrate our causal model into this latent space and execute a three-step counterfactual procedure using a closed-form Generalized Linear Model (GLM). Our experiments conducted on real-world high-resolution MRI data (1mm) demonstrate that our method can generate high-quality 3D MRI counterfactuals.

Title: DSDFormer: An Innovative Transformer-Mamba Framework for Robust High-Precision Driver Distraction Identification

• Authors: Junzhou Chen, Zirui Zhang, Jing Yu, Heqiang Huang, Ronghui Zhang, Xuemiao Xu, Bin Sheng, Hong Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05587
• Pdf URL: https://arxiv.org/pdf/2409.05587
• Copy Paste: [[2409.05587]] DSDFormer: An Innovative Transformer-Mamba Framework for Robust High-Precision Driver Distraction Identification(https://arxiv.org/abs/2409.05587)
• Keywords: robust, extraction, transformer
• Abstract: Driver distraction remains a leading cause of traffic accidents, posing a critical threat to road safety globally. As intelligent transportation systems evolve, accurate and real-time identification of driver distraction has become essential. However, existing methods struggle to capture both global contextual and fine-grained local features while contending with noisy labels in training datasets. To address these challenges, we propose DSDFormer, a novel framework that integrates the strengths of Transformer and Mamba architectures through a Dual State Domain Attention (DSDA) mechanism, enabling a balance between long-range dependencies and detailed feature extraction for robust driver behavior recognition. Additionally, we introduce Temporal Reasoning Confident Learning (TRCL), an unsupervised approach that refines noisy labels by leveraging spatiotemporal correlations in video sequences. Our model achieves state-of-the-art performance on the AUC-V1, AUC-V2, and 100-Driver datasets and demonstrates real-time processing efficiency on the NVIDIA Jetson AGX Orin platform. Extensive experimental results confirm that DSDFormer and TRCL significantly improve both the accuracy and robustness of driver distraction detection, offering a scalable solution to enhance road safety.

Title: MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery

• Authors: Hongjin Qian, Peitian Zhang, Zheng Liu, Kelong Mao, Zhicheng Dou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05591
• Pdf URL: https://arxiv.org/pdf/2409.05591
• Copy Paste: [[2409.05591]] MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery(https://arxiv.org/abs/2409.05591)
• Keywords: large language model
• Abstract: Retrieval-Augmented Generation (RAG) leverages retrieval tools to access external databases, thereby enhancing the generation quality of large language models (LLMs) through optimized context. However, the existing retrieval methods are constrained inherently, as they can only perform relevance matching between explicitly stated queries and well-formed knowledge, but unable to handle tasks involving ambiguous information needs or unstructured knowledge. Consequently, existing RAG systems are primarily effective for straightforward question-answering tasks. In this work, we propose \textbf{MemoRAG}, a novel retrieval-augmented generation paradigm empowered by long-term memory. MemoRAG adopts a dual-system architecture. On the one hand, it employs a \textit{light but long-range} LLM to form the global memory of database. Once a task is presented, it generates draft answers, cluing the retrieval tools to locate useful information within the database. On the other hand, it leverages an \textit{expensive but expressive} LLM, which generates the ultimate answer based on the retrieved information. Building on this general framework, we further optimize MemoRAG's performance by enhancing its cluing mechanism and memorization capacity. In our experiment, MemoRAG achieves superior performance across a variety of evaluation tasks, including both complex ones where conventional RAG fails and straightforward ones where RAG is commonly applied.

Title: SynMorph: Generating Synthetic Face Morphing Dataset with Mated Samples

• Authors: Haoyu Zhang, Raghavendra Ramachandra, Kiran Raja, Christoph Busch
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05595
• Pdf URL: https://arxiv.org/pdf/2409.05595
• Copy Paste: [[2409.05595]] SynMorph: Generating Synthetic Face Morphing Dataset with Mated Samples(https://arxiv.org/abs/2409.05595)
• Keywords: privacy, attack, biometric
• Abstract: Face morphing attack detection (MAD) algorithms have become essential to overcome the vulnerability of face recognition systems. To solve the lack of large-scale and public-available datasets due to privacy concerns and restrictions, in this work we propose a new method to generate a synthetic face morphing dataset with 2450 identities and more than 100k morphs. The proposed synthetic face morphing dataset is unique for its high-quality samples, different types of morphing algorithms, and the generalization for both single and differential morphing attack detection algorithms. For experiments, we apply face image quality assessment and vulnerability analysis to evaluate the proposed synthetic face morphing dataset from the perspective of biometric sample quality and morphing attack potential on face recognition systems. The results are benchmarked with an existing SOTA synthetic dataset and a representative non-synthetic and indicate improvement compared with the SOTA. Additionally, we design different protocols and study the applicability of using the proposed synthetic dataset on training morphing attack detection algorithms.

Title: Normalizing Energy Consumption for Hardware-Independent Evaluation

• Authors: Constance Douwes, Romain Serizel
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05602
• Pdf URL: https://arxiv.org/pdf/2409.05602
• Copy Paste: [[2409.05602]] Normalizing Energy Consumption for Hardware-Independent Evaluation(https://arxiv.org/abs/2409.05602)
• Keywords: robust, fair
• Abstract: The increasing use of machine learning (ML) models in signal processing has raised concerns about their environmental impact, particularly during resource-intensive training phases. In this study, we present a novel methodology for normalizing energy consumption across different hardware platforms to facilitate fair and consistent comparisons. We evaluate different normalization strategies by measuring the energy used to train different ML architectures on different GPUs, focusing on audio tagging tasks. Our approach shows that the number of reference points, the type of regression and the inclusion of computational metrics significantly influences the normalization process. We find that the appropriate selection of two reference points provides robust normalization, while incorporating the number of floating-point operations and parameters improves the accuracy of energy consumption predictions. By supporting more accurate energy consumption evaluation, our methodology promotes the development of environmentally sustainable ML practices.

Title: Forward KL Regularized Preference Optimization for Aligning Diffusion Policies

• Authors: Zhao Shan, Chenyou Fan, Shuang Qiu, Jiyuan Shi, Chenjia Bai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05622
• Pdf URL: https://arxiv.org/pdf/2409.05622
• Copy Paste: [[2409.05622]] Forward KL Regularized Preference Optimization for Aligning Diffusion Policies(https://arxiv.org/abs/2409.05622)
• Keywords: diffusion
• Abstract: Diffusion models have achieved remarkable success in sequential decision-making by leveraging the highly expressive model capabilities in policy learning. A central problem for learning diffusion policies is to align the policy output with human intents in various tasks. To achieve this, previous methods conduct return-conditioned policy generation or Reinforcement Learning (RL)-based policy optimization, while they both rely on pre-defined reward functions. In this work, we propose a novel framework, Forward KL regularized Preference optimization for aligning Diffusion policies, to align the diffusion policy with preferences directly. We first train a diffusion policy from the offline dataset without considering the preference, and then align the policy to the preference data via direct preference optimization. During the alignment phase, we formulate direct preference learning in a diffusion policy, where the forward KL regularization is employed in preference optimization to avoid generating out-of-distribution actions. We conduct extensive experiments for MetaWorld manipulation and D4RL tasks. The results show our method exhibits superior alignment with preferences and outperforms previous state-of-the-art algorithms.

Title: A Framework for Differential Privacy Against Timing Attacks

• Authors: Zachary Ratliff, Salil Vadhan
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.05623
• Pdf URL: https://arxiv.org/pdf/2409.05623
• Copy Paste: [[2409.05623]] A Framework for Differential Privacy Against Timing Attacks(https://arxiv.org/abs/2409.05623)
• Keywords: privacy, attack
• Abstract: The standard definition of differential privacy (DP) ensures that a mechanism's output distribution on adjacent datasets is indistinguishable. However, real-world implementations of DP can, and often do, reveal information through their runtime distributions, making them susceptible to timing attacks. In this work, we establish a general framework for ensuring differential privacy in the presence of timing side channels. We define a new notion of timing privacy, which captures programs that remain differentially private to an adversary that observes the program's runtime in addition to the output. Our framework enables chaining together component programs that are timing-stable followed by a random delay to obtain DP programs that achieve timing privacy. Importantly, our definitions allow for measuring timing privacy and output privacy using different privacy measures. We illustrate how to instantiate our framework by giving programs for standard DP computations in the RAM and Word RAM models of computation. Furthermore, we show how our framework can be realized in code through a natural extension of the OpenDP Programming Framework.

Title: Renormalized Connection for Scale-preferred Object Detection in Satellite Imagery

• Authors: Fan Zhang, Lingling Li, Licheng Jiao, Xu Liu, Fang Liu, Shuyuan Yang, Biao Hou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05624
• Pdf URL: https://arxiv.org/pdf/2409.05624
• Copy Paste: [[2409.05624]] Renormalized Connection for Scale-preferred Object Detection in Satellite Imagery(https://arxiv.org/abs/2409.05624)
• Keywords: extraction
• Abstract: Satellite imagery, due to its long-range imaging, brings with it a variety of scale-preferred tasks, such as the detection of tiny/small objects, making the precise localization and detection of small objects of interest a challenging task. In this article, we design a Knowledge Discovery Network (KDN) to implement the renormalization group theory in terms of efficient feature extraction. Renormalized connection (RC) on the KDN enables ``synergistic focusing'' of multi-scale features. Based on our observations of KDN, we abstract a class of RCs with different connection strengths, called n21C, and generalize it to FPN-based multi-branch detectors. In a series of FPN experiments on the scale-preferred tasks, we found that the ``divide-and-conquer'' idea of FPN severely hampers the detector's learning in the right direction due to the large number of large-scale negative samples and interference from background noise. Moreover, these negative samples cannot be eliminated by the focal loss function. The RCs extends the multi-level feature's ``divide-and-conquer'' mechanism of the FPN-based detectors to a wide range of scale-preferred tasks, and enables synergistic effects of multi-level features on the specific learning goal. In addition, interference activations in two aspects are greatly reduced and the detector learns in a more correct direction. Extensive experiments of 17 well-designed detection architectures embedded with n21s on five different levels of scale-preferred tasks validate the effectiveness and efficiency of the RCs. Especially the simplest linear form of RC, E421C performs well in all tasks and it satisfies the scaling property of RGT. We hope that our approach will transfer a large number of well-designed detectors from the computer vision community to the remote sensing community.

Title: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case

• Authors: Vagrant Gautam, Julius Steuer, Eileen Bingert, Ray Johns, Anne Lauscher, Dietrich Klakow
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05653
• Pdf URL: https://arxiv.org/pdf/2409.05653
• Copy Paste: [[2409.05653]] Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case(https://arxiv.org/abs/2409.05653)
• Keywords: robust
• Abstract: While measuring bias and robustness in coreference resolution are important goals, such measurements are only as good as the tools we use to measure them with. Winogender schemas (Rudinger et al., 2018) are an influential dataset proposed to evaluate gender bias in coreference resolution, but a closer look at the data reveals issues with the instances that compromise their use for reliable evaluation, including treating different grammatical cases of pronouns in the same way, violations of template constraints, and typographical errors. We identify these issues and fix them, contributing a new dataset: Winogender 2.0. Our changes affect performance with state-of-the-art supervised coreference resolution systems as well as all model sizes of the language model FLAN-T5, with F1 dropping on average 0.1 points. We also propose a new method to evaluate pronominal bias in coreference resolution that goes beyond the binary. With this method and our new dataset which is balanced for grammatical case, we empirically demonstrate that bias characteristics vary not just across pronoun sets, but also across surface forms of those sets.

Title: Adversarial Attacks on Data Attribution

• Authors: Xinhe Wang, Pingbang Hu, Junwei Deng, Jiaqi W. Ma
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05657
• Pdf URL: https://arxiv.org/pdf/2409.05657
• Copy Paste: [[2409.05657]] Adversarial Attacks on Data Attribution(https://arxiv.org/abs/2409.05657)
• Keywords: attack, robust, membership infer
• Abstract: Data attribution aims to quantify the contribution of individual training data points to the outputs of an AI model, which has been used to measure the value of training data and compensate data providers. Given the impact on financial decisions and compensation mechanisms, a critical question arises concerning the adversarial robustness of data attribution methods. However, there has been little to no systematic research addressing this issue. In this work, we aim to bridge this gap by detailing a threat model with clear assumptions about the adversary's goal and capabilities, and by proposing principled adversarial attack methods on data attribution. We present two such methods, Shadow Attack and Outlier Attack, both of which generate manipulated datasets to adversarially inflate the compensation. The Shadow Attack leverages knowledge about the data distribution in the AI applications, and derives adversarial perturbations through "shadow training", a technique commonly used in membership inference attacks. In contrast, the Outlier Attack does not assume any knowledge about the data distribution and relies solely on black-box queries to the target model's predictions. It exploits an inductive bias present in many data attribution methods - outlier data points are more likely to be influential - and employs adversarial examples to generate manipulated datasets. Empirically, in image classification and text generation tasks, the Shadow Attack can inflate the data-attribution-based compensation by at least 200%, while the Outlier Attack achieves compensation inflation ranging from 185% to as much as 643%.

Title: Real-Time Human Action Recognition on Embedded Platforms

• Authors: Ruiqi Wang, Zichen Wang, Peiqi Gao, Mingzhen Li, Jaehwan Jeong, Yihang Xu, Yejin Lee, Lisa Connor, Chenyang Lu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05662
• Pdf URL: https://arxiv.org/pdf/2409.05662
• Copy Paste: [[2409.05662]] Real-Time Human Action Recognition on Embedded Platforms(https://arxiv.org/abs/2409.05662)
• Keywords: extraction
• Abstract: With advancements in computer vision and deep learning, video-based human action recognition (HAR) has become practical. However, due to the complexity of the computation pipeline, running HAR on live video streams incurs excessive delays on embedded platforms. This work tackles the real-time performance challenges of HAR with four contributions: 1) an experimental study identifying a standard Optical Flow (OF) extraction technique as the latency bottleneck in a state-of-the-art HAR pipeline, 2) an exploration of the latency-accuracy tradeoff between the standard and deep learning approaches to OF extraction, which highlights the need for a novel, efficient motion feature extractor, 3) the design of Integrated Motion Feature Extractor (IMFE), a novel single-shot neural network architecture for motion feature extraction with drastic improvement in latency, 4) the development of RT-HARE, a real-time HAR system tailored for embedded platforms. Experimental results on an Nvidia Jetson Xavier NX platform demonstrated that RT-HARE realizes real-time HAR at a video frame rate of 30 frames per second while delivering high levels of recognition accuracy.

Title: Unlearning or Concealment? A Critical Analysis and Evaluation Metrics for Unlearning in Diffusion Models

• Authors: Aakash Sen Sharma, Niladri Sarkar, Vikram Chundawat, Ankur A Mali, Murari Mandal
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05668
• Pdf URL: https://arxiv.org/pdf/2409.05668
• Copy Paste: [[2409.05668]] Unlearning or Concealment? A Critical Analysis and Evaluation Metrics for Unlearning in Diffusion Models(https://arxiv.org/abs/2409.05668)
• Keywords: attack, diffusion
• Abstract: Recent research has seen significant interest in methods for concept removal and targeted forgetting in diffusion models. In this paper, we conduct a comprehensive white-box analysis to expose significant vulnerabilities in existing diffusion model unlearning methods. We show that the objective functions used for unlearning in the existing methods lead to decoupling of the targeted concepts (meant to be forgotten) for the corresponding prompts. This is concealment and not actual unlearning, which was the original goal. The ineffectiveness of current methods stems primarily from their narrow focus on reducing generation probabilities for specific prompt sets, neglecting the diverse modalities of intermediate guidance employed during the inference process. The paper presents a rigorous theoretical and empirical examination of four commonly used techniques for unlearning in diffusion models. We introduce two new evaluation metrics: Concept Retrieval Score (CRS) and Concept Confidence Score (CCS). These metrics are based on a successful adversarial attack setup that can recover forgotten concepts from unlearned diffusion models. The CRS measures the similarity between the latent representations of the unlearned and fully trained models after unlearning. It reports the extent of retrieval of the forgotten concepts with increasing amount of guidance. The CCS quantifies the confidence of the model in assigning the target concept to the manipulated data. It reports the probability of the unlearned model's generations to be aligned with the original domain knowledge with increasing amount of guidance. Evaluating existing unlearning methods with our proposed stringent metrics for diffusion models reveals significant shortcomings in their ability to truly unlearn concepts. Source Code: this https URL

Title: Zero-shot Outlier Detection via Prior-data Fitted Networks: Model Selection Bygone!

• Authors: Yuchen Shen, Haomin Wen, Leman Akoglu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05672
• Pdf URL: https://arxiv.org/pdf/2409.05672
• Copy Paste: [[2409.05672]] Zero-shot Outlier Detection via Prior-data Fitted Networks: Model Selection Bygone!(https://arxiv.org/abs/2409.05672)
• Keywords: security, transformer
• Abstract: Outlier detection (OD) has a vast literature as it finds numerous applications in environmental monitoring, cybersecurity, finance, and medicine to name a few. Being an inherently unsupervised task, model selection is a key bottleneck for OD (both algorithm and hyperparameter selection) without label supervision. There is a long list of techniques to choose from -- both classical algorithms and deep neural architectures -- and while several studies report their hyperparameter sensitivity, the literature is quite slim on unsupervised model selection -- limiting the effective use of OD in practice. In this paper we present FoMo-0D, for zero/0-shot OD exploring a transformative new direction that bypasses the hurdle of model selection altogether (!), thus breaking new ground. The fundamental idea behind FoMo-0D is the Prior-data Fitted Networks, recently introduced by Muller et al.(2022), which trains a Transformer model on a large body of synthetically generated data from a prior data distribution. In essence, FoMo-0D is a pretrained Foundation Model for zero/0-shot OD on tabular data, which can directly predict the (outlier/inlier) label of any test data at inference time, by merely a single forward pass -- making obsolete the need for choosing an algorithm/architecture, tuning its associated hyperparameters, and even training any model parameters when given a new OD dataset. Extensive experiments on 57 public benchmark datasets against 26 baseline methods show that FoMo-0D performs statistically no different from the top 2nd baseline, while significantly outperforming the majority of the baselines, with an average inference time of 7.7 ms per test sample.

Title: SX-Stitch: An Efficient VMS-UNet Based Framework for Intraoperative Scoliosis X-Ray Image Stitching

• Authors: Yi Li, Heting Gao, Mingde He, Jinqian Liang, Jason Gu, Wei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05681
• Pdf URL: https://arxiv.org/pdf/2409.05681
• Copy Paste: [[2409.05681]] SX-Stitch: An Efficient VMS-UNet Based Framework for Intraoperative Scoliosis X-Ray Image Stitching(https://arxiv.org/abs/2409.05681)
• Keywords: robust, segmentation
• Abstract: In scoliosis surgery, the limited field of view of the C-arm X-ray machine restricts the surgeons' holistic analysis of spinal structures .This paper presents an end-to-end efficient and robust intraoperative X-ray image stitching method for scoliosis surgery,named SX-Stitch. The method is divided into two stages:segmentation and stitching. In the segmentation stage, We propose a medical image segmentation model named Vision Mamba of Spine-UNet (VMS-UNet), which utilizes the state space Mamba to capture long-distance contextual information while maintaining linear computational complexity, and incorporates the SimAM attention mechanism, significantly improving the segmentation this http URL the stitching stage, we simplify the alignment process between images to the minimization of a registration energy function. The total energy function is then optimized to order unordered images, and a hybrid energy function is introduced to optimize the best seam, effectively eliminating parallax artifacts. On the clinical dataset, Sx-Stitch demonstrates superiority over SOTA schemes both qualitatively and quantitatively.

Title: Segmentation by Factorization: Unsupervised Semantic Segmentation for Pathology by Factorizing Foundation Model Features

• Authors: Jacob Gildenblat, Ofir Hadar
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05697
• Pdf URL: https://arxiv.org/pdf/2409.05697
• Copy Paste: [[2409.05697]] Segmentation by Factorization: Unsupervised Semantic Segmentation for Pathology by Factorizing Foundation Model Features(https://arxiv.org/abs/2409.05697)
• Keywords: robust, segmentation
• Abstract: We introduce Segmentation by Factorization (F-SEG), an unsupervised segmentation method for pathology that generates segmentation masks from pre-trained deep learning models. F-SEG allows the use of pre-trained deep neural networks, including recently developed pathology foundation models, for semantic segmentation. It achieves this without requiring additional training or finetuning, by factorizing the spatial features extracted by the models into segmentation masks and their associated concept features. We create generic tissue phenotypes for H&E images by training clustering models for multiple numbers of clusters on features extracted from several deep learning models on The Cancer Genome Atlas Program (TCGA), and then show how the clusters can be used for factorizing corresponding segmentation masks using off-the-shelf deep learning models. Our results show that F-SEG provides robust unsupervised segmentation capabilities for H&E pathology images, and that the segmentation quality is greatly improved by utilizing pathology foundation models. We discuss and propose methods for evaluating the performance of unsupervised segmentation in pathology.

Title: Boosting CNN-based Handwriting Recognition Systems with Learnable Relaxation Labeling

• Authors: Sara Ferro, Alessandro Torcinovich, Arianna Traviglia, Marcello Pelillo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05699
• Pdf URL: https://arxiv.org/pdf/2409.05699
• Copy Paste: [[2409.05699]] Boosting CNN-based Handwriting Recognition Systems with Learnable Relaxation Labeling(https://arxiv.org/abs/2409.05699)
• Keywords: transformer
• Abstract: The primary challenge for handwriting recognition systems lies in managing long-range contextual dependencies, an issue that traditional models often struggle with. To mitigate it, attention mechanisms have recently been employed to enhance context-aware labelling, thereby achieving state-of-the-art performance. In the field of pattern recognition and image analysis, however, the use of contextual information in labelling problems has a long history and goes back at least to the early 1970's. Among the various approaches developed in those years, Relaxation Labelling (RL) processes have played a prominent role and have been the method of choice in the field for more than a decade. Contrary to recent transformer-based architectures, RL processes offer a principled approach to the use of contextual constraints, having a solid theoretic foundation grounded on variational inequality and game theory, as well as effective algorithms with convergence guarantees. In this paper, we propose a novel approach to handwriting recognition that integrates the strengths of two distinct methodologies. In particular, we propose integrating (trainable) RL processes with various well-established neural architectures and we introduce a sparsification technique that accelerates the convergence of the algorithm and enhances the overall system's performance. Experiments over several benchmark datasets show that RL processes can improve the generalisation ability, even surpassing in some cases transformer-based architectures.

Title: pFedGPA: Diffusion-based Generative Parameter Aggregation for Personalized Federated Learning

• Authors: Jiahao Lai, Jiaqi Li, Jian Xu, Yanru Wu, Boshi Tang, Siqi Chen, Yongfeng Huang, Wenbo Ding, Yang Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05701
• Pdf URL: https://arxiv.org/pdf/2409.05701
• Copy Paste: [[2409.05701]] pFedGPA: Diffusion-based Generative Parameter Aggregation for Personalized Federated Learning(https://arxiv.org/abs/2409.05701)
• Keywords: federate, diffusion, generative
• Abstract: Federated Learning (FL) offers a decentralized approach to model training, where data remains local and only model parameters are shared between the clients and the central server. Traditional methods, such as Federated Averaging (FedAvg), linearly aggregate these parameters which are usually trained on heterogeneous data distributions, potentially overlooking the complex, high-dimensional nature of the parameter space. This can result in degraded performance of the aggregated model. While personalized FL approaches can mitigate the heterogeneous data issue to some extent, the limitation of linear aggregation remains unresolved. To alleviate this issue, we investigate the generative approach of diffusion model and propose a novel generative parameter aggregation framework for personalized FL, \texttt{pFedGPA}. In this framework, we deploy a diffusion model on the server to integrate the diverse parameter distributions and propose a parameter inversion method to efficiently generate a set of personalized parameters for each client. This inversion method transforms the uploaded parameters into a latent code, which is then aggregated through denoising sampling to produce the final personalized parameters. By encoding the dependence of a client's model parameters on the specific data distribution using the high-capacity diffusion model, \texttt{pFedGPA} can effectively decouple the complexity of the overall distribution of all clients' model parameters from the complexity of each individual client's parameter distribution. Our experimental results consistently demonstrate the superior performance of the proposed method across multiple datasets, surpassing baseline approaches.

Title: Towards Democratizing Multilingual Large Language Models For Medicine Through A Two-Stage Instruction Fine-tuning Approach

• Authors: Meng Zhou, Surajsinh Parmar, Anubhav Bhatti
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05732
• Pdf URL: https://arxiv.org/pdf/2409.05732
• Copy Paste: [[2409.05732]] Towards Democratizing Multilingual Large Language Models For Medicine Through A Two-Stage Instruction Fine-tuning Approach(https://arxiv.org/abs/2409.05732)
• Keywords: large language model
• Abstract: Open-source, multilingual medical large language models (LLMs) have the potential to serve linguistically diverse populations across different regions. Adapting generic LLMs for healthcare often requires continual pretraining, but this approach is computationally expensive and sometimes impractical. Instruction fine-tuning on a specific task may not always guarantee optimal performance due to the lack of broader domain knowledge that the model needs to understand and reason effectively in diverse scenarios. To address these challenges, we introduce two multilingual instruction fine-tuning datasets, MMed-IFT and MMed-IFT-MC, containing over 200k high-quality medical samples in six languages. We propose a two-stage training paradigm: the first stage injects general medical knowledge using MMed-IFT, while the second stage fine-tunes task-specific multiple-choice questions with MMed-IFT-MC. Our method achieves competitive results on both English and multilingual benchmarks, striking a balance between computational efficiency and performance. We plan to make our dataset and model weights public at \url{this https URL} in the future.

Title: ReL-SAR: Representation Learning for Skeleton Action Recognition with Convolutional Transformers and BYOL

• Authors: Safwen Naimi, Wassim Bouachir, Guillaume-Alexandre Bilodeau
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05749
• Pdf URL: https://arxiv.org/pdf/2409.05749
• Copy Paste: [[2409.05749]] ReL-SAR: Representation Learning for Skeleton Action Recognition with Convolutional Transformers and BYOL(https://arxiv.org/abs/2409.05749)
• Keywords: robust, transformer
• Abstract: To extract robust and generalizable skeleton action recognition features, large amounts of well-curated data are typically required, which is a challenging task hindered by annotation and computation costs. Therefore, unsupervised representation learning is of prime importance to leverage unlabeled skeleton data. In this work, we investigate unsupervised representation learning for skeleton action recognition. For this purpose, we designed a lightweight convolutional transformer framework, named ReL-SAR, exploiting the complementarity of convolutional and attention layers for jointly modeling spatial and temporal cues in skeleton sequences. We also use a Selection-Permutation strategy for skeleton joints to ensure more informative descriptions from skeletal data. Finally, we capitalize on Bootstrap Your Own Latent (BYOL) to learn robust representations from unlabeled skeleton sequence data. We achieved very competitive results on limited-size datasets: MCAD, IXMAS, JHMDB, and NW-UCLA, showing the effectiveness of our proposed method against state-of-the-art methods in terms of both performance and computational efficiency. To ensure reproducibility and reusability, the source code including all implementation parameters is provided at: this https URL

Title: Evidence from fMRI Supports a Two-Phase Abstraction Process in Language Models

• Authors: Emily Cheng, Richard J. Antonello
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05771
• Pdf URL: https://arxiv.org/pdf/2409.05771
• Copy Paste: [[2409.05771]] Evidence from fMRI Supports a Two-Phase Abstraction Process in Language Models(https://arxiv.org/abs/2409.05771)
• Keywords: large language model
• Abstract: Research has repeatedly demonstrated that intermediate hidden states extracted from large language models are able to predict measured brain response to natural language stimuli. Yet, very little is known about the representation properties that enable this high prediction performance. Why is it the intermediate layers, and not the output layers, that are most capable for this unique and highly general transfer task? In this work, we show that evidence from language encoding models in fMRI supports the existence of a two-phase abstraction process within LLMs. We use manifold learning methods to show that this abstraction process naturally arises over the course of training a language model and that the first "composition" phase of this abstraction process is compressed into fewer layers as training continues. Finally, we demonstrate a strong correspondence between layerwise encoding performance and the intrinsic dimensionality of representations from LLMs. We give initial evidence that this correspondence primarily derives from the inherent compositionality of LLMs and not their next-word prediction properties.

Title: Predicting Critical Heat Flux with Uncertainty Quantification and Domain Generalization Using Conditional Variational Autoencoders and Deep Neural Networks

• Authors: Farah Alsafadi, Aidan Furlong, Xu Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05790
• Pdf URL: https://arxiv.org/pdf/2409.05790
• Copy Paste: [[2409.05790]] Predicting Critical Heat Flux with Uncertainty Quantification and Domain Generalization Using Conditional Variational Autoencoders and Deep Neural Networks(https://arxiv.org/abs/2409.05790)
• Keywords: generative
• Abstract: Deep generative models (DGMs) have proven to be powerful in generating realistic data samples. Their capability to learn the underlying distribution of a dataset enable them to generate synthetic data samples that closely resemble the original training dataset, thus addressing the challenge of data scarcity. In this work, we investigated the capabilities of DGMs by developing a conditional variational autoencoder (CVAE) model to augment the critical heat flux (CHF) measurement data that was used to generate the 2006 Groeneveld lookup table. To determine how this approach compared to traditional methods, a fine-tuned deep neural network (DNN) regression model was created and evaluated with the same dataset. Both the CVAE and DNN models achieved small mean absolute relative errors, with the CVAE model maintaining more favorable results. To quantify the uncertainty in the model's predictions, uncertainty quantification (UQ) was performed with repeated sampling of the CVAE model and ensembling of the DNN model. Following UQ, the DNN ensemble notably improved performance when compared to the baseline DNN model, while the CVAE model achieved similar results to its non-UQ results. The CVAE model was shown to have significantly less variability and a higher confidence after assessment of the prediction-wise relative standard deviations. Evaluating domain generalization, both models achieved small mean error values when predicting both inside and outside the training domain, with predictions outside the training domain showing slightly larger errors. Overall, the CVAE model was comparable to the DNN regression model in predicting CHF values but with better uncertainty behavior.

Title: Enhancing Preference-based Linear Bandits via Human Response Time

• Authors: Shen Li, Yuyang Zhang, Zhaolin Ren, Claire Liang, Na Li, Julie A. Shah
• Subjects: cs.LG, cs.AI, cs.HC, econ.EM, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.05798
• Pdf URL: https://arxiv.org/pdf/2409.05798
• Copy Paste: [[2409.05798]] Enhancing Preference-based Linear Bandits via Human Response Time(https://arxiv.org/abs/2409.05798)
• Keywords: diffusion
• Abstract: Binary human choice feedback is widely used in interactive preference learning for its simplicity, but it provides limited information about preference strength. To overcome this limitation, we leverage human response times, which inversely correlate with preference strength, as complementary information. Our work integrates the EZ-diffusion model, which jointly models human choices and response times, into preference-based linear bandits. We introduce a computationally efficient utility estimator that reformulates the utility estimation problem using both choices and response times as a linear regression problem. Theoretical and empirical comparisons with traditional choice-only estimators reveal that for queries with strong preferences ("easy" queries), choices alone provide limited information, while response times offer valuable complementary information about preference strength. As a result, incorporating response times makes easy queries more useful. We demonstrate this advantage in the fixed-budget best-arm identification problem, with simulations based on three real-world datasets, consistently showing accelerated learning when response times are incorporated.

Title: Input Space Mode Connectivity in Deep Neural Networks

• Authors: Jakub Vrabel, Ori Shem-Ur, Yaron Oz, David Krueger
• Subjects: cs.LG, cond-mat.stat-mech, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05800
• Pdf URL: https://arxiv.org/pdf/2409.05800
• Copy Paste: [[2409.05800]] Input Space Mode Connectivity in Deep Neural Networks(https://arxiv.org/abs/2409.05800)
• Keywords: interpretability
• Abstract: We extend the concept of loss landscape mode connectivity to the input space of deep neural networks. Mode connectivity was originally studied within parameter space, where it describes the existence of low-loss paths between different solutions (loss minimizers) obtained through gradient descent. We present theoretical and empirical evidence of its presence in the input space of deep networks, thereby highlighting the broader nature of the phenomenon. We observe that different input images with similar predictions are generally connected, and for trained models, the path tends to be simple, with only a small deviation from being a linear path. Our methodology utilizes real, interpolated, and synthetic inputs created using the input optimization technique for feature visualization. We conjecture that input space mode connectivity in high-dimensional spaces is a geometric effect that takes place even in untrained models and can be explained through percolation theory. We exploit mode connectivity to obtain new insights about adversarial examples and demonstrate its potential for adversarial detection. Additionally, we discuss applications for the interpretability of deep networks.

Title: Celcomen: spatial causal disentanglement for single-cell and tissue perturbation modeling

• Authors: Stathis Megas, Daniel G. Chen, Krzysztof Polanski, Moshe Eliasof, Carola-Bibiane Schonlieb, Sarah A. Teichmann
• Subjects: cs.LG, q-bio.TO
• Abstract URL: https://arxiv.org/abs/2409.05804
• Pdf URL: https://arxiv.org/pdf/2409.05804
• Copy Paste: [[2409.05804]] Celcomen: spatial causal disentanglement for single-cell and tissue perturbation modeling(https://arxiv.org/abs/2409.05804)
• Keywords: generative
• Abstract: Celcomen leverages a mathematical causality framework to disentangle intra- and inter- cellular gene regulation programs in spatial transcriptomics and single-cell data through a generative graph neural network. It can learn gene-gene interactions, as well as generate post-perturbation counterfactual spatial transcriptomics, thereby offering access to experimentally inaccessible samples. We validated its disentanglement, identifiability, and counterfactual prediction capabilities through simulations and in clinically relevant human glioblastoma, human fetal spleen, and mouse lung cancer samples. Celcomen provides the means to model disease and therapy induced changes allowing for new insights into single-cell spatially resolved tissue responses relevant to human health.

Title: Benchmarking Chinese Knowledge Rectification in Large Language Models

• Authors: Tianhe Lu, Jizhan Fang, Yunzhi Yao, Xin Xu, Ningyu Zhang, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05806
• Pdf URL: https://arxiv.org/pdf/2409.05806
• Copy Paste: [[2409.05806]] Benchmarking Chinese Knowledge Rectification in Large Language Models(https://arxiv.org/abs/2409.05806)
• Keywords: generative, large language model
• Abstract: While Large Language Models (LLMs) exhibit remarkable generative capabilities, they are not without flaws, particularly in the form of hallucinations. This issue is even more pronounced when LLMs are applied to specific languages and domains. For example, LLMs may generate nonsense information when handling Chinese ancient poetry, proverbs, or idioms, owing to the lack of specific knowledge. To this end, this paper introduces a benchmark for rectifying Chinese knowledge in LLMs via knowledge editing. Specifically, we introduce a new Chinese dataset, CKnowEdit, by collecting seven type of knowledge from various sources, including classical texts, idioms, and content from Baidu Tieba Ruozhiba, thereby accounting for the unique polyphony, antithesis, and logical constructs inherent in the Chinese language. Through the analysis of this dataset, we uncover the challenges faced by current LLMs in mastering Chinese. Furthermore, our evaluation of state-of-the-art knowledge editing techniques on this dataset unveil the substantial scope for advancement in the rectification of Chinese knowledge. Code and dataset are available at this https URL.

Title: VFA: Vision Frequency Analysis of Foundation Models and Human

• Authors: Mohammad-Javad Darvishi-Bayazi, Md Rifat Arefin, Jocelyn Faubert, Irina Rish
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.05817
• Pdf URL: https://arxiv.org/pdf/2409.05817
• Copy Paste: [[2409.05817]] VFA: Vision Frequency Analysis of Foundation Models and Human(https://arxiv.org/abs/2409.05817)
• Keywords: robust
• Abstract: Machine learning models often struggle with distribution shifts in real-world scenarios, whereas humans exhibit robust adaptation. Models that better align with human perception may achieve higher out-of-distribution generalization. In this study, we investigate how various characteristics of large-scale computer vision models influence their alignment with human capabilities and robustness. Our findings indicate that increasing model and data size and incorporating rich semantic information and multiple modalities enhance models' alignment with human perception and their overall robustness. Our empirical analysis demonstrates a strong correlation between out-of-distribution accuracy and human alignment.

Title: The Quest to Build Trust Earlier in Digital Design

• Authors: Benjamin Tan
• Subjects: cs.CR, cs.AR
• Abstract URL: https://arxiv.org/abs/2409.05832
• Pdf URL: https://arxiv.org/pdf/2409.05832
• Copy Paste: [[2409.05832]] The Quest to Build Trust Earlier in Digital Design(https://arxiv.org/abs/2409.05832)
• Keywords: secure, security
• Abstract: The ever-rising complexity of computer systems presents challenges for maintaining security and trust throughout their lifetime. As hardware forms the foundation of a secure system, we need tools and techniques that support computer hardware engineers to improve trust and help them address security concerns. This paper highlights a vision for tools and techniques to enhance the security of digital hardware in earlier stages of the digital design process, especially during design with hardware description languages. We discuss the challenges that design teams face and explore some recent literature on understanding, identifying, and mitigating hardware security weaknesses as early as possible. We highlight the opportunities that emerge with open-source hardware development and sketch some open questions that guide ongoing research in this domain.

Title: MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct

• Authors: Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, Heng Tao Shen, Yunshui Li, Xiaobo Xia, Fei Huang, Jingkuan Song, Yongbin Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05840
• Pdf URL: https://arxiv.org/pdf/2409.05840
• Copy Paste: [[2409.05840]] MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct(https://arxiv.org/abs/2409.05840)
• Keywords: robust, large language model
• Abstract: The development of Multimodal Large Language Models (MLLMs) has seen significant advancements. However, the quantity and quality of multimodal instruction data have emerged as significant bottlenecks in their progress. Manually creating multimodal instruction data is both time-consuming and inefficient, posing challenges in producing instructions of high complexity. Moreover, distilling instruction data from black-box commercial models (e.g., GPT-4o, GPT-4V) often results in simplistic instruction data, which constrains performance to that of these models. The challenge of curating diverse and complex instruction data remains substantial. We propose MMEvol, a novel multimodal instruction data evolution framework that combines fine-grained perception evolution, cognitive reasoning evolution, and interaction evolution. This iterative approach breaks through data quality bottlenecks to generate a complex and diverse image-text instruction dataset, thereby empowering MLLMs with enhanced capabilities. Beginning with an initial set of instructions, SEED-163K, we utilize MMEvol to systematically broadens the diversity of instruction types, integrates reasoning steps to enhance cognitive capabilities, and extracts detailed information from images to improve visual understanding and robustness. To comprehensively evaluate the effectiveness of our data, we train LLaVA-NeXT using the evolved data and conduct experiments across 13 vision-language tasks. Compared to the baseline trained with seed data, our approach achieves an average accuracy improvement of 3.1 points and reaches state-of-the-art (SOTA) performance on 9 of these tasks.

Title: LSVOS Challenge Report: Large-scale Complex and Long Video Object Segmentation

• Authors: Henghui Ding, Lingyi Hong, Chang Liu, Ning Xu, Linjie Yang, Yuchen Fan, Deshui Miao, Yameng Gu, Xin Li, Zhenyu He, Yaowei Wang, Ming-Hsuan Yang, Jinming Chai, Qin Ma, Junpei Zhang, Licheng Jiao, Fang Liu, Xinyu Liu, Jing Zhang, Kexin Zhang, Xu Liu, LingLing Li, Hao Fang, Feiyu Pan, Xiankai Lu, Wei Zhang, Runmin Cong, Tuyen Tran, Bin Cao, Yisi Zhang, Hanyi Wang, Xingjian He, Jing Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05847
• Pdf URL: https://arxiv.org/pdf/2409.05847
• Copy Paste: [[2409.05847]] LSVOS Challenge Report: Large-scale Complex and Long Video Object Segmentation(https://arxiv.org/abs/2409.05847)
• Keywords: segmentation
• Abstract: Despite the promising performance of current video segmentation models on existing benchmarks, these models still struggle with complex scenes. In this paper, we introduce the 6th Large-scale Video Object Segmentation (LSVOS) challenge in conjunction with ECCV 2024 workshop. This year's challenge includes two tasks: Video Object Segmentation (VOS) and Referring Video Object Segmentation (RVOS). In this year, we replace the classic YouTube-VOS and YouTube-RVOS benchmark with latest datasets MOSE, LVOS, and MeViS to assess VOS under more challenging complex environments. This year's challenge attracted 129 registered teams from more than 20 institutes across over 8 countries. This report include the challenge and dataset introduction, and the methods used by top 7 teams in two tracks. More details can be found in our homepage this https URL.