2024-08-05

Title: CATD: Unified Representation Learning for EEG-to-fMRI Cross-Modal Generation

• Authors: Weiheng Yao, Shuqiang Wang
• Subjects: cs.CV, eess.SP, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2408.00777
• Pdf URL: https://arxiv.org/pdf/2408.00777
• Copy Paste: [[2408.00777]] CATD: Unified Representation Learning for EEG-to-fMRI Cross-Modal Generation(https://arxiv.org/abs/2408.00777)
• Keywords: diffusion, segmentation
• Abstract: Multi-modal neuroimaging analysis is crucial for a comprehensive understanding of brain function and pathology, as it allows for the integration of different imaging techniques, thus overcoming the limitations of individual modalities. However, the high costs and limited availability of certain modalities pose significant challenges. To address these issues, this paper proposed the Condition-Aligned Temporal Diffusion (CATD) framework for end-to-end cross-modal synthesis of neuroimaging, enabling the generation of functional magnetic resonance imaging (fMRI)-detected Blood Oxygen Level Dependent (BOLD) signals from more accessible Electroencephalography (EEG) signals. By constructing Conditionally Aligned Block (CAB), heterogeneous neuroimages are aligned into a potential space, achieving a unified representation that provides the foundation for cross-modal transformation in neuroimaging. The combination with the constructed Dynamic Time-Frequency Segmentation (DTFS) module also enables the use of EEG signals to improve the temporal resolution of BOLD signals, thus augmenting the capture of the dynamic details of the brain. Experimental validation demonstrated the effectiveness of the framework in improving the accuracy of neural activity prediction, identifying abnormal brain regions, and enhancing the temporal resolution of BOLD signals. The proposed framework establishes a new paradigm for cross-modal synthesis of neuroimaging by unifying heterogeneous neuroimaging data into a potential representation space, showing promise in medical applications such as improving Parkinson's disease prediction and identifying abnormal brain regions.

Title: Data-driven Verification of DNNs for Object Recognition

• Authors: Clemens Otte, Yinchong Yang, Danny Benlin Oswan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00783
• Pdf URL: https://arxiv.org/pdf/2408.00783
• Copy Paste: [[2408.00783]] Data-driven Verification of DNNs for Object Recognition(https://arxiv.org/abs/2408.00783)
• Keywords: segmentation
• Abstract: The paper proposes a new testing approach for Deep Neural Networks (DNN) using gradient-free optimization to find perturbation chains that successfully falsify the tested DNN, going beyond existing grid-based or combinatorial testing. Applying it to an image segmentation task of detecting railway tracks in images, we demonstrate that the approach can successfully identify weaknesses of the tested DNN regarding particular combinations of common perturbations (e.g., rain, fog, blur, noise) on specific clusters of test images.

Title: A Scalable and Generalized Deep Learning Framework for Anomaly Detection in Surveillance Videos

• Authors: Sabah Abdulazeez Jebur, Khalid A. Hussein, Haider Kadhim Hoomod, Laith Alzubaidi, Ahmed Ali Saihood, YuanTong Gu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00792
• Pdf URL: https://arxiv.org/pdf/2408.00792
• Copy Paste: [[2408.00792]] A Scalable and Generalized Deep Learning Framework for Anomaly Detection in Surveillance Videos(https://arxiv.org/abs/2408.00792)
• Keywords: robust, fair, explainability
• Abstract: Anomaly detection in videos is challenging due to the complexity, noise, and diverse nature of activities such as violence, shoplifting, and vandalism. While deep learning (DL) has shown excellent performance in this area, existing approaches have struggled to apply DL models across different anomaly tasks without extensive retraining. This repeated retraining is time-consuming, computationally intensive, and unfair. To address this limitation, a new DL framework is introduced in this study, consisting of three key components: transfer learning to enhance feature generalization, model fusion to improve feature representation, and multi-task classification to generalize the classifier across multiple tasks without training from scratch when new task is introduced. The framework's main advantage is its ability to generalize without requiring retraining from scratch for each new task. Empirical evaluations demonstrate the framework's effectiveness, achieving an accuracy of 97.99% on the RLVS dataset (violence detection), 83.59% on the UCF dataset (shoplifting detection), and 88.37% across both datasets using a single classifier without retraining. Additionally, when tested on an unseen dataset, the framework achieved an accuracy of 87.25%. The study also utilizes two explainability tools to identify potential biases, ensuring robustness and fairness. This research represents the first successful resolution of the generalization issue in anomaly detection, marking a significant advancement in the field.

Title: Calibrating Bayesian Generative Machine Learning for Bayesiamplification

• Authors: Sebastian Bieringer, Sascha Diefenbacher, Gregor Kasieczka, Mathias Trabs
• Subjects: cs.LG, cs.AI, hep-ph
• Abstract URL: https://arxiv.org/abs/2408.00838
• Pdf URL: https://arxiv.org/pdf/2408.00838
• Copy Paste: [[2408.00838]] Calibrating Bayesian Generative Machine Learning for Bayesiamplification(https://arxiv.org/abs/2408.00838)
• Keywords: generative
• Abstract: Recently, combinations of generative and Bayesian machine learning have been introduced in particle physics for both fast detector simulation and inference tasks. These neural networks aim to quantify the uncertainty on the generated distribution originating from limited training statistics. The interpretation of a distribution-wide uncertainty however remains ill-defined. We show a clear scheme for quantifying the calibration of Bayesian generative machine learning models. For a Continuous Normalizing Flow applied to a low-dimensional toy example, we evaluate the calibration of Bayesian uncertainties from either a mean-field Gaussian weight posterior, or Monte Carlo sampling network weights, to gauge their behaviour on unsteady distribution edges. Well calibrated uncertainties can then be used to roughly estimate the number of uncorrelated truth samples that are equivalent to the generated sample and clearly indicate data amplification for smooth features of the distribution.

Title: UniMoT: Unified Molecule-Text Language Model with Discrete Token Representation

• Authors: Juzheng Zhang, Yatao Bian, Yongqiang Chen, Quanming Yao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00863
• Pdf URL: https://arxiv.org/pdf/2408.00863
• Copy Paste: [[2408.00863]] UniMoT: Unified Molecule-Text Language Model with Discrete Token Representation(https://arxiv.org/abs/2408.00863)
• Keywords: large language model
• Abstract: The remarkable success of Large Language Models (LLMs) across diverse tasks has driven the research community to extend their capabilities to molecular applications. However, most molecular LLMs employ adapter-based architectures that do not treat molecule and text modalities equally and lack a supervision signal for the molecule modality. To address these issues, we introduce UniMoT, a Unified Molecule-Text LLM adopting a tokenizer-based architecture that expands the vocabulary of LLM with molecule tokens. Specifically, we introduce a Vector Quantization-driven tokenizer that incorporates a Q-Former to bridge the modality gap between molecule and text. This tokenizer transforms molecules into sequences of molecule tokens with causal dependency, encapsulating high-level molecular and textual information. Equipped with this tokenizer, UniMoT can unify molecule and text modalities under a shared token representation and an autoregressive training paradigm, enabling it to interpret molecules as a foreign language and generate them as text. Following a four-stage training scheme, UniMoT emerges as a multi-modal generalist capable of performing both molecule-to-text and text-to-molecule tasks. Extensive experiments demonstrate that UniMoT achieves state-of-the-art performance across a wide range of molecule comprehension and generation tasks.

Title: Medical SAM 2: Segment medical images as video via Segment Anything Model 2

• Authors: Jiayuan Zhu, Yunli Qi, Junde Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00874
• Pdf URL: https://arxiv.org/pdf/2408.00874
• Copy Paste: [[2408.00874]] Medical SAM 2: Segment medical images as video via Segment Anything Model 2(https://arxiv.org/abs/2408.00874)
• Keywords: segmentation
• Abstract: In this paper, we introduce Medical SAM 2 (MedSAM-2), an advanced segmentation model that utilizes the SAM 2 framework to address both 2D and 3D medical image segmentation tasks. By adopting the philosophy of taking medical images as videos, MedSAM-2 not only applies to 3D medical images but also unlocks new One-prompt Segmentation capability. That allows users to provide a prompt for just one or a specific image targeting an object, after which the model can autonomously segment the same type of object in all subsequent images, regardless of temporal relationships between the images. We evaluated MedSAM-2 across a variety of medical imaging modalities, including abdominal organs, optic discs, brain tumors, thyroid nodules, and skin lesions, comparing it against state-of-the-art models in both traditional and interactive segmentation settings. Our findings show that MedSAM-2 not only surpasses existing models in performance but also exhibits superior generalization across a range of medical image segmentation tasks. Our code will be released at: this https URL

Title: Benchmarking Attacks on Learning with Errors

• Authors: Emily Wenger, Eshika Saxena, Mohamed Malhou, Ellie Thieu, Kristin Lauter
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.00882
• Pdf URL: https://arxiv.org/pdf/2408.00882
• Copy Paste: [[2408.00882]] Benchmarking Attacks on Learning with Errors(https://arxiv.org/abs/2408.00882)
• Keywords: security, attack
• Abstract: Lattice cryptography schemes based on the learning with errors (LWE) hardness assumption have been standardized by NIST for use as post-quantum cryptosystems, and by this http URL for encrypted compute on sensitive data. Thus, understanding their concrete security is critical. Most work on LWE security focuses on theoretical estimates of attack performance, which is important but may overlook attack nuances arising in real-world implementations. The sole existing concrete benchmarking effort, the Darmstadt Lattice Challenge, does not include benchmarks relevant to the standardized LWE parameter choices - such as small secret and small error distributions, and Ring-LWE (RLWE) and Module-LWE (MLWE) variants. To improve our understanding of concrete LWE security, we provide the first benchmarks for LWE secret recovery on standardized parameters, for small and low-weight (sparse) secrets. We evaluate four LWE attacks in these settings to serve as a baseline: the Search-LWE attacks uSVP, SALSA, and Cool & Cruel, and the Decision-LWE attack: Dual Hybrid Meet-in-the-Middle (MitM). We extend the SALSA and Cool & Cruel attacks in significant ways, and implement and scale up MitM attacks for the first time. For example, we recover hamming weight $9-11$ binomial secrets for KYBER ($\kappa=2$) parameters in $28-36$ hours with SALSA and Cool\&Cruel, while we find that MitM can solve Decision-LWE instances for hamming weights up to $4$ in under an hour for Kyber parameters, while uSVP attacks do not recover any secrets after running for more than $1100$ hours. We also compare concrete performance against theoretical estimates. Finally, we open source the code to enable future research.

Title: Discrete Randomized Smoothing Meets Quantum Computing

• Authors: Tom Wollschläger, Aman Saxena, Nicola Franco, Jeanette Miriam Lorenz, Stephan Günnemann
• Subjects: cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2408.00895
• Pdf URL: https://arxiv.org/pdf/2408.00895
• Copy Paste: [[2408.00895]] Discrete Randomized Smoothing Meets Quantum Computing(https://arxiv.org/abs/2408.00895)
• Keywords: attack
• Abstract: Breakthroughs in machine learning (ML) and advances in quantum computing (QC) drive the interdisciplinary field of quantum machine learning to new levels. However, due to the susceptibility of ML models to adversarial attacks, practical use raises safety-critical concerns. Existing Randomized Smoothing (RS) certification methods for classical machine learning models are computationally intensive. In this paper, we propose the combination of QC and the concept of discrete randomized smoothing to speed up the stochastic certification of ML models for discrete data. We show how to encode all the perturbations of the input binary data in superposition and use Quantum Amplitude Estimation (QAE) to obtain a quadratic reduction in the number of calls to the model that are required compared to traditional randomized smoothing techniques. In addition, we propose a new binary threat model to allow for an extensive evaluation of our approach on images, graphs, and text.

Title: Parkinson's Disease Detection from Resting State EEG using Multi-Head Graph Structure Learning with Gradient Weighted Graph Attention Explanations

• Authors: Christopher Neves, Yong Zeng, Yiming Xiao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00906
• Pdf URL: https://arxiv.org/pdf/2408.00906
• Copy Paste: [[2408.00906]] Parkinson's Disease Detection from Resting State EEG using Multi-Head Graph Structure Learning with Gradient Weighted Graph Attention Explanations(https://arxiv.org/abs/2408.00906)
• Keywords: robust, explainability, generative
• Abstract: Parkinson's disease (PD) is a debilitating neurodegenerative disease that has severe impacts on an individual's quality of life. Compared with structural and functional MRI-based biomarkers for the disease, electroencephalography (EEG) can provide more accessible alternatives for clinical insights. While deep learning (DL) techniques have provided excellent outcomes, many techniques fail to model spatial information and dynamic brain connectivity, and face challenges in robust feature learning, limited data sizes, and poor explainability. To address these issues, we proposed a novel graph neural network (GNN) technique for explainable PD detection using resting state EEG. Specifically, we employ structured global convolutions with contrastive learning to better model complex features with limited data, a novel multi-head graph structure learner to capture the non-Euclidean structure of EEG data, and a head-wise gradient-weighted graph attention explainer to offer neural connectivity insights. We developed and evaluated our method using the UC San Diego Parkinson's disease EEG dataset, and achieved 69.40% detection accuracy in subject-wise leave-one-out cross-validation while generating intuitive explanations for the learnt graph topology.

Title: Distance-Preserving Generative Modeling of Spatial Transcriptomics

• Authors: Wenbin Zhou, Jin-Hong Du
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.00911
• Pdf URL: https://arxiv.org/pdf/2408.00911
• Copy Paste: [[2408.00911]] Distance-Preserving Generative Modeling of Spatial Transcriptomics(https://arxiv.org/abs/2408.00911)
• Keywords: generative
• Abstract: Spatial transcriptomics data is invaluable for understanding the spatial organization of gene expression in tissues. There have been consistent efforts in studying how to effectively utilize the associated spatial information for refining gene expression modeling. We introduce a class of distance-preserving generative models for spatial transcriptomics, which utilizes the provided spatial information to regularize the learned representation space of gene expressions to have a similar pair-wise distance structure. This helps the latent space to capture meaningful encodings of genes in spatial proximity. We carry out theoretical analysis over a tractable loss function for this purpose and formalize the overall learning objective as a regularized evidence lower bound. Our framework grants compatibility with any variational-inference-based generative models for gene expression modeling. Empirically, we validate our proposed method on the mouse brain tissues Visium dataset and observe improved performance with variational autoencoders and scVI used as backbone models.

Title: WHITE PAPER: A Brief Exploration of Data Exfiltration using GCG Suffixes

• Authors: Victor Valbuena
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00925
• Pdf URL: https://arxiv.org/pdf/2408.00925
• Copy Paste: [[2408.00925]] WHITE PAPER: A Brief Exploration of Data Exfiltration using GCG Suffixes(https://arxiv.org/abs/2408.00925)
• Keywords: attack
• Abstract: The cross-prompt injection attack (XPIA) is an effective technique that can be used for data exfiltration, and that has seen increasing use. In this attack, the attacker injects a malicious instruction into third party data which an LLM is likely to consume when assisting a user, who is the victim. XPIA is often used as a means for data exfiltration, and the estimated cost of the average data breach for a business is nearly $4.5 million, which includes breaches such as compromised enterprise credentials. With the rise of gradient-based attacks such as the GCG suffix attack, the odds of an XPIA occurring which uses a GCG suffix are worryingly high. As part of my work in Microsoft's AI Red Team, I demonstrated a viable attack model using a GCG suffix paired with an injection in a simulated XPIA scenario. The results indicate that the presence of a GCG suffix can increase the odds of successful data exfiltration by nearly 20%, with some caveats.

Title: Verification of Machine Unlearning is Fragile

• Authors: Binchi Zhang, Zihan Chen, Cong Shen, Jundong Li
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2408.00929
• Pdf URL: https://arxiv.org/pdf/2408.00929
• Copy Paste: [[2408.00929]] Verification of Machine Unlearning is Fragile(https://arxiv.org/abs/2408.00929)
• Keywords: privacy
• Abstract: As privacy concerns escalate in the realm of machine learning, data owners now have the option to utilize machine unlearning to remove their data from machine learning models, following recent legislation. To enhance transparency in machine unlearning and avoid potential dishonesty by model providers, various verification strategies have been proposed. These strategies enable data owners to ascertain whether their target data has been effectively unlearned from the model. However, our understanding of the safety issues of machine unlearning verification remains nascent. In this paper, we explore the novel research question of whether model providers can circumvent verification strategies while retaining the information of data supposedly unlearned. Our investigation leads to a pessimistic answer: \textit{the verification of machine unlearning is fragile}. Specifically, we categorize the current verification strategies regarding potential dishonesty among model providers into two types. Subsequently, we introduce two novel adversarial unlearning processes capable of circumventing both types. We validate the efficacy of our methods through theoretical analysis and empirical experiments using real-world datasets. This study highlights the vulnerabilities and limitations in machine unlearning verification, paving the way for further research into the safety of machine unlearning.

Title: Towards Zero-Shot Annotation of the Built Environment with Vision-Language Models (Vision Paper)

• Authors: Bin Han, Yiwei Yang, Anat Caspi, Bill Howe
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00932
• Pdf URL: https://arxiv.org/pdf/2408.00932
• Copy Paste: [[2408.00932]] Towards Zero-Shot Annotation of the Built Environment with Vision-Language Models (Vision Paper)(https://arxiv.org/abs/2408.00932)
• Keywords: segmentation
• Abstract: Equitable urban transportation applications require high-fidelity digital representations of the built environment: not just streets and sidewalks, but bike lanes, marked and unmarked crossings, curb ramps and cuts, obstructions, traffic signals, signage, street markings, potholes, and more. Direct inspections and manual annotations are prohibitively expensive at scale. Conventional machine learning methods require substantial annotated training data for adequate performance. In this paper, we consider vision language models as a mechanism for annotating diverse urban features from satellite images, reducing the dependence on human annotation to produce large training sets. While these models have achieved impressive results in describing common objects in images captured from a human perspective, their training sets are less likely to include strong signals for esoteric features in the built environment, and their performance in these settings is therefore unclear. We demonstrate proof-of-concept combining a state-of-the-art vision language model and variants of a prompting strategy that asks the model to consider segmented elements independently of the original image. Experiments on two urban features -- stop lines and raised tables -- show that while direct zero-shot prompting correctly annotates nearly zero images, the pre-segmentation strategies can annotate images with near 40% intersection-over-union accuracy. We describe how these results inform a new research agenda in automatic annotation of the built environment to improve equity, accessibility, and safety at broad scale and in diverse environments.

Title: Data-Driven Traffic Simulation for an Intersection in a Metropolis

• Authors: Chengbo Zang, Mehmet Kerem Turkcan, Gil Zussman, Javad Ghaderi, Zoran Kostic
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00943
• Pdf URL: https://arxiv.org/pdf/2408.00943
• Copy Paste: [[2408.00943]] Data-Driven Traffic Simulation for an Intersection in a Metropolis(https://arxiv.org/abs/2408.00943)
• Keywords: generative
• Abstract: We present a novel data-driven simulation environment for modeling traffic in metropolitan street intersections. Using real-world tracking data collected over an extended period of time, we train trajectory forecasting models to learn agent interactions and environmental constraints that are difficult to capture conventionally. Trajectories of new agents are first coarsely generated by sampling from the spatial and temporal generative distributions, then refined using state-of-the-art trajectory forecasting models. The simulation can run either autonomously, or under explicit human control conditioned on the generative distributions. We present the experiments for a variety of model configurations. Under an iterative prediction scheme, the way-point-supervised TrajNet++ model obtained 0.36 Final Displacement Error (FDE) in 20 FPS on an NVIDIA A100 GPU.

Title: Leveraging Large Language Models (LLMs) for Traffic Management at Urban Intersections: The Case of Mixed Traffic Scenarios

• Authors: Sari Masri, Huthaifa I. Ashqar, Mohammed Elhenawy
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.00948
• Pdf URL: https://arxiv.org/pdf/2408.00948
• Copy Paste: [[2408.00948]] Leveraging Large Language Models (LLMs) for Traffic Management at Urban Intersections: The Case of Mixed Traffic Scenarios(https://arxiv.org/abs/2408.00948)
• Keywords: large language model
• Abstract: Urban traffic management faces significant challenges due to the dynamic environments, and traditional algorithms fail to quickly adapt to this environment in real-time and predict possible conflicts. This study explores the ability of a Large Language Model (LLM), specifically, GPT-4o-mini to improve traffic management at urban intersections. We recruited GPT-4o-mini to analyze, predict position, detect and resolve the conflicts at an intersection in real-time for various basic scenarios. The key findings of this study to investigate whether LLMs can logically reason and understand the scenarios to enhance the traffic efficiency and safety by providing real-time analysis. The study highlights the potential of LLMs in urban traffic management creating more intelligent and more adaptive systems. Results showed the GPT-4o-mini was effectively able to detect and resolve conflicts in heavy traffic, congestion, and mixed-speed conditions. The complex scenario of multiple intersections with obstacles and pedestrians saw successful conflict management as well. Results show that the integration of LLMs promises to improve the effectiveness of traffic control for safer and more efficient urban intersection management.

Title: PrivateGaze: Preserving User Privacy in Black-box Mobile Gaze Tracking Services

• Authors: Lingyu Du, Jinyuan Jia, Xucong Zhang, Guohao Lan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00950
• Pdf URL: https://arxiv.org/pdf/2408.00950
• Copy Paste: [[2408.00950]] PrivateGaze: Preserving User Privacy in Black-box Mobile Gaze Tracking Services(https://arxiv.org/abs/2408.00950)
• Keywords: privacy, protect
• Abstract: Eye gaze contains rich information about human attention and cognitive processes. This capability makes the underlying technology, known as gaze tracking, a critical enabler for many ubiquitous applications and has triggered the development of easy-to-use gaze estimation services. Indeed, by utilizing the ubiquitous cameras on tablets and smartphones, users can readily access many gaze estimation services. In using these services, users must provide their full-face images to the gaze estimator, which is often a black box. This poses significant privacy threats to the users, especially when a malicious service provider gathers a large collection of face images to classify sensitive user attributes. In this work, we present PrivateGaze, the first approach that can effectively preserve users' privacy in black-box gaze tracking services without compromising gaze estimation performance. Specifically, we proposed a novel framework to train a privacy preserver that converts full-face images into obfuscated counterparts, which are effective for gaze estimation while containing no privacy information. Evaluation on four datasets shows that the obfuscated image can protect users' private information, such as identity and gender, against unauthorized attribute classification. Meanwhile, when used directly by the black-box gaze estimator as inputs, the obfuscated images lead to comparable tracking performance to the conventional, unprotected full-face images.

Title: PERSOMA: PERsonalized SOft ProMpt Adapter Architecture for Personalized Language Prompting

• Authors: Liam Hebert, Krishna Sayana, Ambarish Jash, Alexandros Karatzoglou, Sukhdeep Sodhi, Sumanth Doddapaneni, Yanli Cai, Dima Kuzmin
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.00960
• Pdf URL: https://arxiv.org/pdf/2408.00960
• Copy Paste: [[2408.00960]] PERSOMA: PERsonalized SOft ProMpt Adapter Architecture for Personalized Language Prompting(https://arxiv.org/abs/2408.00960)
• Keywords: large language model
• Abstract: Understanding the nuances of a user's extensive interaction history is key to building accurate and personalized natural language systems that can adapt to evolving user preferences. To address this, we introduce PERSOMA, Personalized Soft Prompt Adapter architecture. Unlike previous personalized prompting methods for large language models, PERSOMA offers a novel approach to efficiently capture user history. It achieves this by resampling and compressing interactions as free form text into expressive soft prompt embeddings, building upon recent research utilizing embedding representations as input for LLMs. We rigorously validate our approach by evaluating various adapter architectures, first-stage sampling strategies, parameter-efficient tuning techniques like LoRA, and other personalization methods. Our results demonstrate PERSOMA's superior ability to handle large and complex user histories compared to existing embedding-based and text-prompt-based techniques.

Title: Automatic Extraction of Relationships among Motivations, Emotions and Actions from Natural Language Texts

• Authors: Fei Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.00966
• Pdf URL: https://arxiv.org/pdf/2408.00966
• Copy Paste: [[2408.00966]] Automatic Extraction of Relationships among Motivations, Emotions and Actions from Natural Language Texts(https://arxiv.org/abs/2408.00966)
• Keywords: extraction, large language model
• Abstract: We propose a new graph-based framework to reveal relationships among motivations, emotions and actions explicitly given natural language texts. A directed acyclic graph is designed to describe human's nature. Nurture beliefs are incorporated to connect outside events and the human's nature graph. No annotation resources are required due to the power of large language models. Amazon Fine Foods Reviews dataset is used as corpus and food-related motivations are focused. Totally 92,990 relationship graphs are generated, of which 63% make logical sense. We make further analysis to investigate error types for optimization direction in future research.

Title: Extracting Object Heights From LiDAR & Aerial Imagery

• Authors: Jesus Guerrero
• Subjects: cs.CV, cs.ET
• Abstract URL: https://arxiv.org/abs/2408.00967
• Pdf URL: https://arxiv.org/pdf/2408.00967
• Copy Paste: [[2408.00967]] Extracting Object Heights From LiDAR & Aerial Imagery(https://arxiv.org/abs/2408.00967)
• Keywords: generative, segmentation
• Abstract: This work shows a procedural method for extracting object heights from LiDAR and aerial imagery. We discuss how to get heights and the future of LiDAR and imagery processing. SOTA object segmentation allows us to take get object heights with no deep learning background. Engineers will be keeping track of world data across generations and reprocessing them. They will be using older procedural methods like this paper and newer ones discussed here. SOTA methods are going beyond analysis and into generative AI. We cover both a procedural methodology and the newer ones performed with language models. These include point cloud, imagery and text encoding allowing for spatially aware AI.

Title: DNSSEC+: An Enhanced DNS Scheme Motivated by Benefits and Pitfalls of DNSSEC

• Authors: Ali Sadeghi Jahromi, AbdelRahman Abdou, Paul C. van Oorschot
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.00968
• Pdf URL: https://arxiv.org/pdf/2408.00968
• Copy Paste: [[2408.00968]] DNSSEC+: An Enhanced DNS Scheme Motivated by Benefits and Pitfalls of DNSSEC(https://arxiv.org/abs/2408.00968)
• Keywords: secure, security, privacy, attack
• Abstract: The absence of security measures between DNS recursive resolvers and authoritative nameservers has been exploited by both inline and off-path attacks. While many security proposals have been made in practice and previous literature, they typically suffer from deployability barriers and/or inadequate security properties. The absence of a broadly adopted security solution between resolvers and nameservers motivates a new scheme that mitigates these issues in previous proposals. We present DNSSEC+, which addresses security and deployability downsides of DNSSEC, while retaining its benefits. DNSSEC+ takes advantage of the existent DNSSEC trust model and authorizes the nameservers within a zone for short intervals to serve the zone data securely, facilitating real-time security properties for DNS responses, without requiring long-term private keys to be duplicated (thus put at risk) on authoritative nameservers. Regarding name resolution latency, DNSSEC+ offers a performance comparable to less secure schemes. We define nine security, privacy, and deployability properties for name resolution, and show how DNSSEC+ fulfills these properties.

Title: Visible-Thermal Multiple Object Tracking: Large-scale Video Dataset and Progressive Fusion Approach

• Authors: Yabin Zhu, Qianwu Wang, Chenglong Li, Jin Tang, Zhixiang Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.00969
• Pdf URL: https://arxiv.org/pdf/2408.00969
• Copy Paste: [[2408.00969]] Visible-Thermal Multiple Object Tracking: Large-scale Video Dataset and Progressive Fusion Approach(https://arxiv.org/abs/2408.00969)
• Keywords: robust, segmentation
• Abstract: The complementary benefits from visible and thermal infrared data are widely utilized in various computer vision task, such as visual tracking, semantic segmentation and object detection, but rarely explored in Multiple Object Tracking (MOT). In this work, we contribute a large-scale Visible-Thermal video benchmark for MOT, called VT-MOT. VT-MOT has the following main advantages. 1) The data is large scale and high diversity. VT-MOT includes 582 video sequence pairs, 401k frame pairs from surveillance, drone, and handheld platforms. 2) The cross-modal alignment is highly accurate. We invite several professionals to perform both spatial and temporal alignment frame by frame. 3) The annotation is dense and high-quality. VT-MOT has 3.99 million annotation boxes annotated and double-checked by professionals, including heavy occlusion and object re-acquisition (object disappear and reappear) challenges. To provide a strong baseline, we design a simple yet effective tracking framework, which effectively fuses temporal information and complementary information of two modalities in a progressive manner, for robust visible-thermal MOT. A comprehensive experiment are conducted on VT-MOT and the results prove the superiority and effectiveness of the proposed method compared with state-of-the-art methods. From the evaluation results and analysis, we specify several potential future directions for visible-thermal MOT. The project is released in this https URL.

Title: Reconstructing Richtmyer-Meshkov instabilities from noisy radiographs using low dimensional features and attention-based neural networks

• Authors: Daniel A. Serino, Marc L. Klasky, Balasubramanya T. Nadiga, Xiaojian Xu, Trevor Wilcox
• Subjects: cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.00985
• Pdf URL: https://arxiv.org/pdf/2408.00985
• Copy Paste: [[2408.00985]] Reconstructing Richtmyer-Meshkov instabilities from noisy radiographs using low dimensional features and attention-based neural networks(https://arxiv.org/abs/2408.00985)
• Keywords: robust, transformer
• Abstract: A trained attention-based transformer network can robustly recover the complex topologies given by the Richtmyer-Meshkoff instability from a sequence of hydrodynamic features derived from radiographic images corrupted with blur, scatter, and noise. This approach is demonstrated on ICF-like double shell hydrodynamic simulations. The key component of this network is a transformer encoder that acts on a sequence of features extracted from noisy radiographs. This encoder includes numerous self-attention layers that act to learn temporal dependencies in the input sequences and increase the expressiveness of the model. This approach is demonstrated to exhibit an excellent ability to accurately recover the Richtmyer-Meshkov instability growth rates, even despite the gas-metal interface being greatly obscured by radiographic noise.

Title: Fairness in Large Language Models in Three Hour

• Authors: Thang Doan Viet, Zichong Wang, Minh Nhat Nguyen, Wenbin Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.00992
• Pdf URL: https://arxiv.org/pdf/2408.00992
• Copy Paste: [[2408.00992]] Fairness in Large Language Models in Three Hour(https://arxiv.org/abs/2408.00992)
• Keywords: fair, large language model
• Abstract: Large Language Models (LLMs) have demonstrated remarkable success across various domains but often lack fairness considerations, potentially leading to discriminatory outcomes against marginalized populations. Unlike fairness in traditional machine learning, fairness in LLMs involves unique backgrounds, taxonomies, and fulfillment techniques. This tutorial provides a systematic overview of recent advances in the literature concerning fair LLMs, beginning with real-world case studies to introduce LLMs, followed by an analysis of bias causes therein. The concept of fairness in LLMs is then explored, summarizing the strategies for evaluating bias and the algorithms designed to promote fairness. Additionally, resources for assessing bias in LLMs, including toolkits and datasets, are compiled, and current research challenges and open questions in the field are discussed. The repository is available at \url{this https URL}.

Title: FBSDiff: Plug-and-Play Frequency Band Substitution of Diffusion Features for Highly Controllable Text-Driven Image Translation

• Authors: Xiang Gao, Jiaying Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.00998
• Pdf URL: https://arxiv.org/pdf/2408.00998
• Copy Paste: [[2408.00998]] FBSDiff: Plug-and-Play Frequency Band Substitution of Diffusion Features for Highly Controllable Text-Driven Image Translation(https://arxiv.org/abs/2408.00998)
• Keywords: diffusion, generative
• Abstract: Large-scale text-to-image diffusion models have been a revolutionary milestone in the evolution of generative AI and multimodal technology, allowing extraordinary image generation based on natural-language text prompts. However, the issue of lacking controllability of such models restricts their practical applicability for real-life content creation, for which attention has been focused on leveraging a reference image to control text-to-image synthesis. Due to the close correlation between the reference image and the generated image, this problem can also be regarded as the task of manipulating (or editing) the reference image as per the text, namely text-driven image-to-image translation. This paper contributes a novel, concise, and efficient approach that adapts the pre-trained large-scale text-to-image (T2I) diffusion model to the image-to-image (I2I) paradigm in a plug-and-play manner, realizing high-quality and versatile text-driven I2I translation without any model training, model fine-tuning, or online optimization process. To guide T2I generation with a reference image, we propose to model diverse guiding factors with correspondingly different frequency bands of diffusion features in the DCT spectral space, and accordingly devise a novel frequency band substitution layer that dynamically substitutes a certain DCT frequency band of the diffusion features with the corresponding counterpart of the reference image along the reverse sampling process. We demonstrate that our method flexibly enables highly controllable text-driven I2I translation both in the guiding factor and guiding intensity of the reference image, simply by tuning the type and bandwidth of the substituted frequency band, respectively. Extensive qualitative and quantitative experiments verify the superiority of our approach over related methods in I2I translation visual quality, versatility, and controllability.

Title: Adaptive Two-Stage Cloud Resource Scaling via Hierarchical Multi-Indicator Forecasting and Bayesian Decision-Making

• Authors: Yang Luo, Shiyu Wang, Zhemeng Yu, Wei Lu, Xiaofeng Gao, Lintao Ma, Guihai Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.01000
• Pdf URL: https://arxiv.org/pdf/2408.01000
• Copy Paste: [[2408.01000]] Adaptive Two-Stage Cloud Resource Scaling via Hierarchical Multi-Indicator Forecasting and Bayesian Decision-Making(https://arxiv.org/abs/2408.01000)
• Keywords: robust
• Abstract: The surging demand for cloud computing resources, driven by the rapid growth of sophisticated large-scale models and data centers, underscores the critical importance of efficient and adaptive resource allocation. As major tech enterprises deploy massive infrastructures with thousands of GPUs, existing cloud platforms still struggle with low resource utilization due to key challenges: capturing hierarchical indicator structures, modeling non-Gaussian distributions, and decision-making under uncertainty. To address these challenges, we propose HRAMONY, an adaptive Hierarchical Attention-based Resource Modeling and Decision-Making System. HARMONY combines hierarchical multi-indicator distribution forecasting and uncertainty-aware Bayesian decision-making. It introduces a novel hierarchical attention mechanism that comprehensively models complex inter-indicator dependencies, enabling accurate predictions that can adapt to evolving environment states. By transforming Gaussian projections into adaptive non-Gaussian distributions via Normalizing Flows. Crucially, HARMONY leverages the full predictive distributions in an adaptive Bayesian process, proactively incorporating uncertainties to optimize resource allocation while robustly meeting SLA constraints under varying conditions. Extensive evaluations across four large-scale cloud datasets demonstrate HARMONY's state-of-the-art performance, significantly outperforming nine established methods. A month-long real-world deployment validated HARMONY's substantial practical impact, realizing over 35,000 GPU hours in savings and translating to $100K+ in cost reduction, showcasing its remarkable economic value through adaptive, uncertainty-aware scaling. Our code is available at this https URL.

Title: Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMs

• Authors: Afia Anjum, Maksim E. Eren, Ismael Boureima, Boian Alexandrov, Manish Bhattarai
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01008
• Pdf URL: https://arxiv.org/pdf/2408.01008
• Copy Paste: [[2408.01008]] Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMs(https://arxiv.org/abs/2408.01008)
• Keywords: large language model
• Abstract: In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing (NLP) tasks, such as question-answering, sentiment analysis, text summarization, and machine translation. However, the ever-growing complexity of LLMs demands immense computational resources, hindering the broader research and application of these models. To address this, various parameter-efficient fine-tuning strategies, such as Low-Rank Approximation (LoRA) and Adapters, have been developed. Despite their potential, these methods often face limitations in compressibility. Specifically, LoRA struggles to scale effectively with the increasing number of trainable parameters in modern large scale LLMs. Additionally, Low-Rank Economic Tensor-Train Adaptation (LoRETTA), which utilizes tensor train decomposition, has not yet achieved the level of compression necessary for fine-tuning very large scale models with limited resources. This paper introduces Tensor Train Low-Rank Approximation (TT-LoRA), a novel parameter-efficient fine-tuning (PEFT) approach that extends LoRETTA with optimized tensor train (TT) decomposition integration. By eliminating Adapters and traditional LoRA-based structures, TT-LoRA achieves greater model compression without compromising downstream task performance, along with reduced inference latency and computational overhead. We conduct an exhaustive parameter search to establish benchmarks that highlight the trade-off between model compression and performance. Our results demonstrate significant compression of LLMs while maintaining comparable performance to larger models, facilitating their deployment on resource-constraint platforms.

Title: EIUP: A Training-Free Approach to Erase Non-Compliant Concepts Conditioned on Implicit Unsafe Prompts

• Authors: Die Chen, Zhiwen Li, Mingyuan Fan, Cen Chen, Wenmeng Zhou, Yaliang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01014
• Pdf URL: https://arxiv.org/pdf/2408.01014
• Copy Paste: [[2408.01014]] EIUP: A Training-Free Approach to Erase Non-Compliant Concepts Conditioned on Implicit Unsafe Prompts(https://arxiv.org/abs/2408.01014)
• Keywords: security, diffusion
• Abstract: Text-to-image diffusion models have shown the ability to learn a diverse range of concepts. However, it is worth noting that they may also generate undesirable outputs, consequently giving rise to significant security concerns. Specifically, issues such as Not Safe for Work (NSFW) content and potential violations of style copyright may be encountered. Since image generation is conditioned on text, prompt purification serves as a straightforward solution for content safety. Similar to the approach taken by LLM, some efforts have been made to control the generation of safe outputs by purifying prompts. However, it is also important to note that even with these efforts, non-toxic text still carries a risk of generating non-compliant images, which is referred to as implicit unsafe prompts. Furthermore, some existing works fine-tune the models to erase undesired concepts from model weights. This type of method necessitates multiple training iterations whenever the concept is updated, which can be time-consuming and may potentially lead to catastrophic forgetting. To address these challenges, we propose a simple yet effective approach that incorporates non-compliant concepts into an erasure prompt. This erasure prompt proactively participates in the fusion of image spatial features and text embeddings. Through attention mechanisms, our method is capable of identifying feature representations of non-compliant concepts in the image space. We re-weight these features to effectively suppress the generation of unsafe images conditioned on original implicit unsafe prompts. Our method exhibits superior erasure effectiveness while achieving high scores in image fidelity compared to the state-of-the-art baselines. WARNING: This paper contains model outputs that may be offensive.

Title: GNN-MolKAN: Harnessing the Power of KAN to Advance Molecular Representation Learning with GNNs

• Authors: Ruifeng Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01018
• Pdf URL: https://arxiv.org/pdf/2408.01018
• Copy Paste: [[2408.01018]] GNN-MolKAN: Harnessing the Power of KAN to Advance Molecular Representation Learning with GNNs(https://arxiv.org/abs/2408.01018)
• Keywords: robust
• Abstract: Effective molecular representation learning is crucial for molecular property prediction and drug design. However, existing approaches struggle with limitations in insufficient annotations and suboptimal architecture design. For instance, Graph Neural Networks (GNNs) suffer from over-squashing, causing the loss of important structural details in molecules, thus impairing molecular representations. In this work, we propose a new class of GNNs, GNN-MolKAN and its augmented variant, GNN-MolKAN+, that integrate the Kolmogorov-Arnold Networks (KAN) architecture from AI + Science into GNNs to address these challenges. Additionally, we introduce Adaptive FastKAN (AdFastKAN), an advanced KAN that offers increased stability and speed, further enhancing the performance of standard GNNs. Notably, our approach holds three key benefits: 1) Superior Performance: GNN-MolKAN and GNN-MolKAN+ demonstrate superior prediction ability, robust generalization to unseen scaffolds, and versatile transferability across different GNN architectures. 2) Efficiency: These models require less computational time and fewer parameters while matching or surpassing the state-of-the-art (SOTA) self-supervised methods. 3) Few-shot Learning Ability: GNN-MolKAN demonstrates great potential in few-shot learning scenarios, achieving an average improvement of 6.97% across few-shot benchmarks. Overall, we validate our architecture on 6 classification datasets, 6 regression datasets, and 4 few-shot learning datasets, consistently achieving highly competitive results across all of them.

Title: POA: Pre-training Once for Models of All Sizes

• Authors: Yingying Zhang, Xin Guo, Jiangwei Lao, Lei Yu, Lixiang Ru, Jian Wang, Guo Ye, Huimei He, Jingdong Chen, Ming Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01031
• Pdf URL: https://arxiv.org/pdf/2408.01031
• Copy Paste: [[2408.01031]] POA: Pre-training Once for Models of All Sizes(https://arxiv.org/abs/2408.01031)
• Keywords: extraction, transformer
• Abstract: Large-scale self-supervised pre-training has paved the way for one foundation model to handle many different vision tasks. Most pre-training methodologies train a single model of a certain size at one time. Nevertheless, various computation or storage constraints in real-world scenarios require substantial efforts to develop a series of models with different sizes to deploy. Thus, in this study, we propose a novel tri-branch self-supervised training framework, termed as POA (Pre-training Once for All), to tackle this aforementioned issue. Our approach introduces an innovative elastic student branch into a modern self-distillation paradigm. At each pre-training step, we randomly sample a sub-network from the original student to form the elastic student and train all branches in a self-distilling fashion. Once pre-trained, POA allows the extraction of pre-trained models of diverse sizes for downstream tasks. Remarkably, the elastic student facilitates the simultaneous pre-training of multiple models with different sizes, which also acts as an additional ensemble of models of various sizes to enhance representation learning. Extensive experiments, including k-nearest neighbors, linear probing evaluation and assessments on multiple downstream tasks demonstrate the effectiveness and advantages of our POA. It achieves state-of-the-art performance using ViT, Swin Transformer and ResNet backbones, producing around a hundred models with different sizes through a single pre-training session. The code is available at: this https URL.

Title: MambaST: A Plug-and-Play Cross-Spectral Spatial-Temporal Fuser for Efficient Pedestrian Detection

• Authors: Xiangbo Gao, Asiegbu Miracle Kanu-Asiegbu, Xiaoxiao Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01037
• Pdf URL: https://arxiv.org/pdf/2408.01037
• Copy Paste: [[2408.01037]] MambaST: A Plug-and-Play Cross-Spectral Spatial-Temporal Fuser for Efficient Pedestrian Detection(https://arxiv.org/abs/2408.01037)
• Keywords: robust, transformer
• Abstract: This paper proposes MambaST, a plug-and-play cross-spectral spatial-temporal fusion pipeline for efficient pedestrian detection. Several challenges exist for pedestrian detection in autonomous driving applications. First, it is difficult to perform accurate detection using RGB cameras under dark or low-light conditions. Cross-spectral systems must be developed to integrate complementary information from multiple sensor modalities, such as thermal and visible cameras, to improve the robustness of the detections. Second, pedestrian detection models are latency-sensitive. Efficient and easy-to-scale detection models with fewer parameters are highly desirable for real-time applications such as autonomous driving. Third, pedestrian video data provides spatial-temporal correlations of pedestrian movement. It is beneficial to incorporate temporal as well as spatial information to enhance pedestrian detection. This work leverages recent advances in the state space model (Mamba) and proposes a novel Multi-head Hierarchical Patching and Aggregation (MHHPA) structure to extract both fine-grained and coarse-grained information from both RGB and thermal imagery. Experimental results show that the proposed MHHPA is an effective and efficient alternative to a Transformer model for cross-spectral pedestrian detection. Our proposed model also achieves superior performance on small-scale pedestrian detection. The code is available at this https URL}{this https URL.

Title: UNER: A Unified Prediction Head for Named Entity Recognition in Visually-rich Documents

• Authors: Yi Tu, Chong Zhang, Ya Guo, Huan Chen, Jinyang Tang, Huijia Zhu, Qi Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01038
• Pdf URL: https://arxiv.org/pdf/2408.01038
• Copy Paste: [[2408.01038]] UNER: A Unified Prediction Head for Named Entity Recognition in Visually-rich Documents(https://arxiv.org/abs/2408.01038)
• Keywords: robust, extraction, transformer
• Abstract: The recognition of named entities in visually-rich documents (VrD-NER) plays a critical role in various real-world scenarios and applications. However, the research in VrD-NER faces three major challenges: complex document layouts, incorrect reading orders, and unsuitable task formulations. To address these challenges, we propose a query-aware entity extraction head, namely UNER, to collaborate with existing multi-modal document transformers to develop more robust VrD-NER models. The UNER head considers the VrD-NER task as a combination of sequence labeling and reading order prediction, effectively addressing the issues of discontinuous entities in documents. Experimental evaluations on diverse datasets demonstrate the effectiveness of UNER in improving entity extraction performance. Moreover, the UNER head enables a supervised pre-training stage on various VrD-NER datasets to enhance the document transformer backbones and exhibits substantial knowledge transfer from the pre-training stage to the fine-tuning stage. By incorporating universal layout understanding, a pre-trained UNER-based model demonstrates significant advantages in few-shot and cross-linguistic scenarios and exhibits zero-shot entity extraction abilities.

Title: Boosting Gaze Object Prediction via Pixel-level Supervision from Vision Foundation Model

• Authors: Yang Jin, Lei Zhang, Shi Yan, Bin Fan, Binglu Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01044
• Pdf URL: https://arxiv.org/pdf/2408.01044
• Copy Paste: [[2408.01044]] Boosting Gaze Object Prediction via Pixel-level Supervision from Vision Foundation Model(https://arxiv.org/abs/2408.01044)
• Keywords: segmentation
• Abstract: Gaze object prediction (GOP) aims to predict the category and location of the object that a human is looking at. Previous methods utilized box-level supervision to identify the object that a person is looking at, but struggled with semantic ambiguity, ie, a single box may contain several items since objects are close together. The Vision foundation model (VFM) has improved in object segmentation using box prompts, which can reduce confusion by more precisely locating objects, offering advantages for fine-grained prediction of gaze objects. This paper presents a more challenging gaze object segmentation (GOS) task, which involves inferring the pixel-level mask corresponding to the object captured by human gaze behavior. In particular, we propose that the pixel-level supervision provided by VFM can be integrated into gaze object prediction to mitigate semantic ambiguity. This leads to our gaze object detection and segmentation framework that enables accurate pixel-level predictions. Different from previous methods that require additional head input or ignore head features, we propose to automatically obtain head features from scene features to ensure the model's inference efficiency and flexibility in the real world. Moreover, rather than directly fuse features to predict gaze heatmap as in existing methods, which may overlook spatial location and subtle details of the object, we develop a space-to-object gaze regression method to facilitate human-object gaze interaction. Specifically, it first constructs an initial human-object spatial connection, then refines this connection by interacting with semantically clear features in the segmentation branch, ultimately predicting a gaze heatmap for precise localization. Extensive experiments on GOO-Synth and GOO-Real datasets demonstrate the effectiveness of our method.

Title: Leveraging Large Language Models for Mobile App Review Feature Extraction

• Authors: Quim Motger, Alessio Miaschi, Felice Dell'Orletta, Xavier Franch, Jordi Marco
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.01063
• Pdf URL: https://arxiv.org/pdf/2408.01063
• Copy Paste: [[2408.01063]] Leveraging Large Language Models for Mobile App Review Feature Extraction(https://arxiv.org/abs/2408.01063)
• Keywords: extraction, transformer, large language model
• Abstract: Mobile app review analysis presents unique challenges due to the low quality, subjective bias, and noisy content of user-generated documents. Extracting features from these reviews is essential for tasks such as feature prioritization and sentiment analysis, but it remains a challenging task. Meanwhile, encoder-only models based on the Transformer architecture have shown promising results for classification and information extraction tasks for multiple software engineering processes. This study explores the hypothesis that encoder-only large language models can enhance feature extraction from mobile app reviews. By leveraging crowdsourced annotations from an industrial context, we redefine feature extraction as a supervised token classification task. Our approach includes extending the pre-training of these models with a large corpus of user reviews to improve contextual understanding and employing instance selection techniques to optimize model fine-tuning. Empirical evaluations demonstrate that this method improves the precision and recall of extracted features and enhances performance efficiency. Key contributions include a novel approach to feature extraction, annotated datasets, extended pre-trained models, and an instance selection mechanism for cost-effective fine-tuning. This research provides practical methods and empirical evidence in applying large language models to natural language processing tasks within mobile app reviews, offering improved performance in feature extraction.

Title: Amodal Segmentation for Laparoscopic Surgery Video Instruments

• Authors: Ruohua Shi, Zhaochen Liu, Lingyu Duan, Tingting Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01067
• Pdf URL: https://arxiv.org/pdf/2408.01067
• Copy Paste: [[2408.01067]] Amodal Segmentation for Laparoscopic Surgery Video Instruments(https://arxiv.org/abs/2408.01067)
• Keywords: segmentation
• Abstract: Segmentation of surgical instruments is crucial for enhancing surgeon performance and ensuring patient safety. Conventional techniques such as binary, semantic, and instance segmentation share a common drawback: they do not accommodate the parts of instruments obscured by tissues or other instruments. Precisely predicting the full extent of these occluded instruments can significantly improve laparoscopic surgeries by providing critical guidance during operations and assisting in the analysis of potential surgical errors, as well as serving educational purposes. In this paper, we introduce Amodal Segmentation to the realm of surgical instruments in the medical field. This technique identifies both the visible and occluded parts of an object. To achieve this, we introduce a new Amoal Instruments Segmentation (AIS) dataset, which was developed by reannotating each instrument with its complete mask, utilizing the 2017 MICCAI EndoVis Robotic Instrument Segmentation Challenge dataset. Additionally, we evaluate several leading amodal segmentation methods to establish a benchmark for this new dataset.

Title: PhysMamba: Leveraging Dual-Stream Cross-Attention SSD for Remote Physiological Measurement

• Authors: Zhixin Yan, Yan Zhong, Wenjun Zhang, Lin Shu, Hongbin Xu, Wenxiong Kang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01077
• Pdf URL: https://arxiv.org/pdf/2408.01077
• Copy Paste: [[2408.01077]] PhysMamba: Leveraging Dual-Stream Cross-Attention SSD for Remote Physiological Measurement(https://arxiv.org/abs/2408.01077)
• Keywords: robust
• Abstract: Remote Photoplethysmography (rPPG) is a non-contact technique for extracting physiological signals from facial videos, used in applications like emotion monitoring, medical assistance, and anti-face spoofing. Unlike controlled laboratory settings, real-world environments often contain motion artifacts and noise, affecting the performance of existing methods. To address this, we propose PhysMamba, a dual-stream time-frequency interactive model based on Mamba. PhysMamba integrates the state-of-the-art Mamba-2 model and employs a dual-stream architecture to learn diverse rPPG features, enhancing robustness in noisy conditions. Additionally, we designed the Cross-Attention State Space Duality (CASSD) module to improve information exchange and feature complementarity between the two streams. We validated PhysMamba using PURE, UBFC-rPPG and MMPD. Experimental results show that PhysMamba achieves state-of-the-art performance across various scenarios, particularly in complex environments, demonstrating its potential in practical remote heart rate monitoring applications.

Title: Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts

• Authors: Youna Kim, Hyuhng Joon Kim, Cheonbok Park, Choonghyun Park, Hyunsoo Cho, Junyeob Kim, Kang Min Yoo, Sang-goo Lee, Taeuk Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01084
• Pdf URL: https://arxiv.org/pdf/2408.01084
• Copy Paste: [[2408.01084]] Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts(https://arxiv.org/abs/2408.01084)
• Keywords: robust, large language model
• Abstract: When using large language models (LLMs) in knowledge-intensive tasks, such as open-domain question answering, external context can bridge a gap between external knowledge and LLM's parametric knowledge. Recent research has been developed to amplify contextual knowledge over the parametric knowledge of LLM with contrastive decoding approaches. While these approaches could yield truthful responses when relevant context is provided, they are prone to vulnerabilities when faced with noisy contexts. We extend the scope of previous studies to encompass noisy contexts and propose adaptive contrastive decoding (ACD) to leverage contextual influence effectively. ACD demonstrates improvements in open-domain question answering tasks compared to baselines, especially in robustness by remaining undistracted by noisy contexts in retrieval-augmented generation.

Title: Bridging Information Gaps in Dialogues With Grounded Exchanges Using Knowledge Graphs

• Authors: Phillip Schneider, Nektarios Machner, Kristiina Jokinen, Florian Matthes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01088
• Pdf URL: https://arxiv.org/pdf/2408.01088
• Copy Paste: [[2408.01088]] Bridging Information Gaps in Dialogues With Grounded Exchanges Using Knowledge Graphs(https://arxiv.org/abs/2408.01088)
• Keywords: large language model
• Abstract: Knowledge models are fundamental to dialogue systems for enabling conversational interactions, which require handling domain-specific knowledge. Ensuring effective communication in information-providing conversations entails aligning user understanding with the knowledge available to the system. However, dialogue systems often face challenges arising from semantic inconsistencies in how information is expressed in natural language compared to how it is represented within the system's internal knowledge. To address this problem, we study the potential of large language models for conversational grounding, a mechanism to bridge information gaps by establishing shared knowledge between dialogue participants. Our approach involves annotating human conversations across five knowledge domains to create a new dialogue corpus called BridgeKG. Through a series of experiments on this dataset, we empirically evaluate the capabilities of large language models in classifying grounding acts and identifying grounded information items within a knowledge graph structure. Our findings offer insights into how these models use in-context learning for conversational grounding tasks and common prediction errors, which we illustrate with examples from challenging dialogues. We discuss how the models handle knowledge graphs as a semantic layer between unstructured dialogue utterances and structured information items.

Title: Contribution-based Low-Rank Adaptation with Pre-training Model for Real Image Restoration

• Authors: Donwon Park, Hayeon Kim, Se Young Chun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01099
• Pdf URL: https://arxiv.org/pdf/2408.01099
• Copy Paste: [[2408.01099]] Contribution-based Low-Rank Adaptation with Pre-training Model for Real Image Restoration(https://arxiv.org/abs/2408.01099)
• Keywords: robust
• Abstract: Recently, pre-trained model and efficient parameter tuning have achieved remarkable success in natural language processing and high-level computer vision with the aid of masked modeling and prompt tuning. In low-level computer vision, however, there have been limited investigations on pre-trained models and even efficient fine-tuning strategy has not yet been explored despite its importance and benefit in various real-world tasks such as alleviating memory inflation issue when integrating new tasks on AI edge devices. Here, we propose a novel efficient parameter tuning approach dubbed contribution-based low-rank adaptation (CoLoRA) for multiple image restorations along with effective pre-training method with random order degradations (PROD). Unlike prior arts that tune all network parameters, our CoLoRA effectively fine-tunes small amount of parameters by leveraging LoRA (low-rank adaptation) for each new vision task with our contribution-based method to adaptively determine layer by layer capacity for that task to yield comparable performance to full tuning. Furthermore, our PROD strategy allows to extend the capability of pre-trained models with improved performance as well as robustness to bridge synthetic pre-training and real-world fine-tuning. Our CoLoRA with PROD has demonstrated its superior performance in various image restoration tasks across diverse degradation types on both synthetic and real-world datasets for known and novel tasks.

Title: BioRAG: A RAG-LLM Framework for Biological Question Reasoning

• Authors: Chengrui Wang, Qingqing Long, Xiao Meng, Xunxin Cai, Chengjun Wu, Zhen Meng, Xuezhi Wang, Yuanchun Zhou
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.01107
• Pdf URL: https://arxiv.org/pdf/2408.01107
• Copy Paste: [[2408.01107]] BioRAG: A RAG-LLM Framework for Biological Question Reasoning(https://arxiv.org/abs/2408.01107)
• Keywords: large language model
• Abstract: The question-answering system for Life science research, which is characterized by the rapid pace of discovery, evolving insights, and complex interactions among knowledge entities, presents unique challenges in maintaining a comprehensive knowledge warehouse and accurate information retrieval. To address these issues, we introduce BioRAG, a novel Retrieval-Augmented Generation (RAG) with the Large Language Models (LLMs) framework. Our approach starts with parsing, indexing, and segmenting an extensive collection of 22 million scientific papers as the basic knowledge, followed by training a specialized embedding model tailored to this domain. Additionally, we enhance the vector retrieval process by incorporating a domain-specific knowledge hierarchy, which aids in modeling the intricate interrelationships among each query and context. For queries requiring the most current information, BioRAG deconstructs the question and employs an iterative retrieval process incorporated with the search engine for step-by-step reasoning. Rigorous experiments have demonstrated that our model outperforms fine-tuned LLM, LLM with search engines, and other scientific RAG frameworks across multiple life science question-answering tasks.

Title: IAI Group at CheckThat! 2024: Transformer Models and Data Augmentation for Checkworthy Claim Detection

• Authors: Peter Røysland Aarnes, Vinay Setty, Petra Galuščáková
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01118
• Pdf URL: https://arxiv.org/pdf/2408.01118
• Copy Paste: [[2408.01118]] IAI Group at CheckThat! 2024: Transformer Models and Data Augmentation for Checkworthy Claim Detection(https://arxiv.org/abs/2408.01118)
• Keywords: transformer, generative
• Abstract: This paper describes IAI group's participation for automated check-worthiness estimation for claims, within the framework of the 2024 CheckThat! Lab "Task 1: Check-Worthiness Estimation". The task involves the automated detection of check-worthy claims in English, Dutch, and Arabic political debates and Twitter data. We utilized various pre-trained generative decoder and encoder transformer models, employing methods such as few-shot chain-of-thought reasoning, fine-tuning, data augmentation, and transfer learning from one language to another. Despite variable success in terms of performance, our models achieved notable placements on the organizer's leaderboard: ninth-best in English, third-best in Dutch, and the top placement in Arabic, utilizing multilingual datasets for enhancing the generalizability of check-worthiness detection. Despite a significant drop in performance on the unlabeled test dataset compared to the development test dataset, our findings contribute to the ongoing efforts in claim detection research, highlighting the challenges and potential of language-specific adaptations in claim verification systems.

Title: Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer

• Authors: Robert Belanec, Simon Ostermann, Ivan Srba, Maria Bielikova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01119
• Pdf URL: https://arxiv.org/pdf/2408.01119
• Copy Paste: [[2408.01119]] Task Prompt Vectors: Effective Initialization through Multi-Task Soft-Prompt Transfer(https://arxiv.org/abs/2408.01119)
• Keywords: large language model
• Abstract: Prompt tuning is a modular and efficient solution for training large language models (LLMs). One of its main advantages is task modularity, making it suitable for multi-task problems. However, current soft-prompt-based methods often sacrifice multi-task modularity, requiring the training process to be fully or partially repeated for each newly added task. While recent work on task vectors applied arithmetic operations on full model weights to achieve the desired multi-task performance, a similar approach for soft-prompts is still missing. To this end, we introduce Task Prompt Vectors, created by element-wise difference between weights of tuned soft-prompts and their random initialization. Experimental results on 12 NLU datasets show that task prompt vectors can be used in low-resource settings to effectively initialize prompt tuning on similar tasks. In addition, we show that task prompt vectors are independent of the random initialization of prompt tuning. This allows prompt arithmetics with the pre-trained vectors from different tasks. In this way, by arithmetic addition of task prompt vectors from multiple tasks, we are able to outperform a state-of-the-art baseline in some cases.

Title: An Efficient and Effective Transformer Decoder-Based Framework for Multi-Task Visual Grounding

• Authors: Wei Chen, Long Chen, Yu Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01120
• Pdf URL: https://arxiv.org/pdf/2408.01120
• Copy Paste: [[2408.01120]] An Efficient and Effective Transformer Decoder-Based Framework for Multi-Task Visual Grounding(https://arxiv.org/abs/2408.01120)
• Keywords: transformer, segmentation
• Abstract: Most advanced visual grounding methods rely on Transformers for visual-linguistic feature fusion. However, these Transformer-based approaches encounter a significant drawback: the computational costs escalate quadratically due to the self-attention mechanism in the Transformer Encoder, particularly when dealing with high-resolution images or long context sentences. This quadratic increase in computational burden restricts the applicability of visual grounding to more intricate scenes, such as conversation-based reasoning segmentation, which involves lengthy language expressions. In this paper, we propose an efficient and effective multi-task visual grounding (EEVG) framework based on Transformer Decoder to address this issue, which reduces the cost in both language and visual aspects. In the language aspect, we employ the Transformer Decoder to fuse visual and linguistic features, where linguistic features are input as memory and visual features as queries. This allows fusion to scale linearly with language expression length. In the visual aspect, we introduce a parameter-free approach to reduce computation by eliminating background visual tokens based on attention scores. We then design a light mask head to directly predict segmentation masks from the remaining sparse feature maps. Extensive results and ablation studies on benchmarks demonstrate the efficiency and effectiveness of our approach. Code is available in this https URL.

Title: CFBench: A Comprehensive Constraints-Following Benchmark for LLMs

• Authors: Tao Zhang, Yanjun Shen, Wenjing Luo, Yan Zhang, Hao Liang, Tao Zhang, Fan Yang, Mingan Lin, Yujing Qiao, Weipeng Chen, Bin Cui, Wentao Zhang, Zenan Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01122
• Pdf URL: https://arxiv.org/pdf/2408.01122
• Copy Paste: [[2408.01122]] CFBench: A Comprehensive Constraints-Following Benchmark for LLMs(https://arxiv.org/abs/2408.01122)
• Keywords: large language model
• Abstract: The adeptness of Large Language Models (LLMs) in comprehending and following natural language instructions is critical for their deployment in sophisticated real-world applications. Existing evaluations mainly focus on fragmented constraints or narrow scenarios, but they overlook the comprehensiveness and authenticity of constraints from the user's perspective. To bridge this gap, we propose CFBench, a large-scale Comprehensive Constraints Following Benchmark for LLMs, featuring 1,000 curated samples that cover more than 200 real-life scenarios and over 50 NLP tasks. CFBench meticulously compiles constraints from real-world instructions and constructs an innovative systematic framework for constraint types, which includes 10 primary categories and over 25 subcategories, and ensures each constraint is seamlessly integrated within the instructions. To make certain that the evaluation of LLM outputs aligns with user perceptions, we propose an advanced methodology that integrates multi-dimensional assessment criteria with requirement prioritization, covering various perspectives of constraints, instructions, and requirement fulfillment. Evaluating current leading LLMs on CFBench reveals substantial room for improvement in constraints following, and we further investigate influencing factors and enhancement strategies. The data and code are publicly available at this https URL

Title: IG-SLAM: Instant Gaussian SLAM

• Authors: Furkan Aykut Sarikamis, Abdullah Aydin Alatan
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.01126
• Pdf URL: https://arxiv.org/pdf/2408.01126
• Copy Paste: [[2408.01126]] IG-SLAM: Instant Gaussian SLAM(https://arxiv.org/abs/2408.01126)
• Keywords: robust
• Abstract: 3D Gaussian Splatting has recently shown promising results as an alternative scene representation in SLAM systems to neural implicit representations. However, current methods either lack dense depth maps to supervise the mapping process or detailed training designs that consider the scale of the environment. To address these drawbacks, we present IG-SLAM, a dense RGB-only SLAM system that employs robust Dense-SLAM methods for tracking and combines them with Gaussian Splatting. A 3D map of the environment is constructed using accurate pose and dense depth provided by tracking. Additionally, we utilize depth uncertainty in map optimization to improve 3D reconstruction. Our decay strategy in map optimization enhances convergence and allows the system to run at 10 fps in a single process. We demonstrate competitive performance with state-of-the-art RGB-only SLAM systems while achieving faster operation speeds. We present our experiments on the Replica, TUM-RGBD, ScanNet, and EuRoC datasets. The system achieves photo-realistic 3D reconstruction in large-scale sequences, particularly in the EuRoC dataset.

Title: A Survey of Mamba

• Authors: Haohao Qu, Liangbo Ning, Rui An, Wenqi Fan, Tyler Derr, Xin Xu, Qing Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01129
• Pdf URL: https://arxiv.org/pdf/2408.01129
• Copy Paste: [[2408.01129]] A Survey of Mamba(https://arxiv.org/abs/2408.01129)
• Keywords: transformer, large language model
• Abstract: Deep learning, as a vital technique, has sparked a notable revolution in artificial intelligence. As the most representative architecture, Transformers have empowered numerous advanced models, especially the large language models that comprise billions of parameters, becoming a cornerstone in deep learning. Despite the impressive achievements, Transformers still face inherent limitations, particularly the time-consuming inference resulting from the quadratic computation complexity of attention calculation. Recently, a novel architecture named Mamba, drawing inspiration from classical state space models, has emerged as a promising alternative for building foundation models, delivering comparable modeling abilities to Transformers while preserving near-linear scalability concerning sequence length. This has sparked an increasing number of studies actively exploring Mamba's potential to achieve impressive performance across diverse domains. Given such rapid evolution, there is a critical need for a systematic review that consolidates existing Mamba-empowered models, offering a comprehensive understanding of this emerging model architecture. In this survey, we therefore conduct an in-depth investigation of recent Mamba-associated studies, covering from three main aspects: the advancements of Mamba-based models, the techniques of adapting Mamba to diverse data, and the applications where Mamba can excel. Specifically, we first recall the foundational knowledge of various representative deep learning models and the details of Mamba as preliminaries. Then, to showcase the significance of Mamba, we comprehensively review the related studies focusing on Mamba models' architecture design, data adaptability, and applications. Finally, we present an discussion of current limitations and explore various promising research directions to provide deeper insights for future investigations.

Title: PGNeXt: High-Resolution Salient Object Detection via Pyramid Grafting Network

• Authors: Changqun Xia, Chenxi Xie, Zhentao He, Tianshu Yu, Jia Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01137
• Pdf URL: https://arxiv.org/pdf/2408.01137
• Copy Paste: [[2408.01137]] PGNeXt: High-Resolution Salient Object Detection via Pyramid Grafting Network(https://arxiv.org/abs/2408.01137)
• Keywords: transformer
• Abstract: We present an advanced study on more challenging high-resolution salient object detection (HRSOD) from both dataset and network framework perspectives. To compensate for the lack of HRSOD dataset, we thoughtfully collect a large-scale high resolution salient object detection dataset, called UHRSD, containing 5,920 images from real-world complex scenarios at 4K-8K resolutions. All the images are finely annotated in pixel-level, far exceeding previous low-resolution SOD datasets. Aiming at overcoming the contradiction between the sampling depth and the receptive field size in the past methods, we propose a novel one-stage framework for HR-SOD task using pyramid grafting mechanism. In general, transformer-based and CNN-based backbones are adopted to extract features from different resolution images independently and then these features are grafted from transformer branch to CNN branch. An attention-based Cross-Model Grafting Module (CMGM) is proposed to enable CNN branch to combine broken detailed information more holistically, guided by different source feature during decoding process. Moreover, we design an Attention Guided Loss (AGL) to explicitly supervise the attention matrix generated by CMGM to help the network better interact with the attention from different branches. Comprehensive experiments on UHRSD and widely-used SOD datasets demonstrate that our method can simultaneously locate salient object and preserve rich details, outperforming state-of-the-art methods. To verify the generalization ability of the proposed framework, we apply it to the camouflaged object detection (COD) task. Notably, our method performs superior to most state-of-the-art COD methods without bells and whistles.

Title: TCR-GPT: Integrating Autoregressive Model and Reinforcement Learning for T-Cell Receptor Repertoires Generation

• Authors: Yicheng Lin, Dandan Zhang, Yun Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01156
• Pdf URL: https://arxiv.org/pdf/2408.01156
• Copy Paste: [[2408.01156]] TCR-GPT: Integrating Autoregressive Model and Reinforcement Learning for T-Cell Receptor Repertoires Generation(https://arxiv.org/abs/2408.01156)
• Keywords: transformer
• Abstract: T-cell receptors (TCRs) play a crucial role in the immune system by recognizing and binding to specific antigens presented by infected or cancerous cells. Understanding the sequence patterns of TCRs is essential for developing targeted immune therapies and designing effective vaccines. Language models, such as auto-regressive transformers, offer a powerful solution to this problem by learning the probability distributions of TCR repertoires, enabling the generation of new TCR sequences that inherit the underlying patterns of the repertoire. We introduce TCR-GPT, a probabilistic model built on a decoder-only transformer architecture, designed to uncover and replicate sequence patterns in TCR repertoires. TCR-GPT demonstrates an accuracy of 0.953 in inferring sequence probability distributions measured by Pearson correlation coefficient. Furthermore, by leveraging Reinforcement Learning(RL), we adapted the distribution of TCR sequences to generate TCRs capable of recognizing specific peptides, offering significant potential for advancing targeted immune therapies and vaccine development. With the efficacy of RL, fine-tuned pretrained TCR-GPT models demonstrated the ability to produce TCR repertoires likely to bind specific peptides, illustrating RL's efficiency in enhancing the model's adaptability to the probability distributions of biologically relevant TCR sequences.

Title: Robust Curve Detection in Volumetric Medical Imaging via Attraction Field

• Authors: Farukh Yaushev, Daria Nogina, Valentin Samokhin, Mariya Dugova, Ekaterina Petrash, Dmitry Sevryukov, Mikhail Belyaev, Maxim Pisov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01159
• Pdf URL: https://arxiv.org/pdf/2408.01159
• Copy Paste: [[2408.01159]] Robust Curve Detection in Volumetric Medical Imaging via Attraction Field(https://arxiv.org/abs/2408.01159)
• Keywords: robust
• Abstract: Understanding body part geometry is crucial for precise medical diagnostics. Curves effectively describe anatomical structures and are widely used in medical imaging applications related to cardiovascular, respiratory, and skeletal diseases. Traditional curve detection methods are often task-specific, relying heavily on domain-specific features, limiting their broader applicability. This paper introduces a novel approach for detecting non-branching curves, which does not require prior knowledge of the object's orientation, shape, or position. Our method uses neural networks to predict (1) an attraction field, which offers subpixel accuracy, and (2) a closeness map, which limits the region of interest and essentially eliminates outliers far from the desired curve. We tested our curve detector on several clinically relevant tasks with diverse morphologies and achieved impressive subpixel-level accuracy results that surpass existing methods, highlighting its versatility and robustness. Additionally, to support further advancements in this field, we provide our private annotations of aortic centerlines and masks, which can serve as a benchmark for future research. The dataset can be found at this https URL.

Title: PreMix: Boosting Multiple Instance Learning in Digital Histopathology through Pre-training with Intra-Batch Slide Mixing

• Authors: Bryan Wong, Mun Yong Yi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01162
• Pdf URL: https://arxiv.org/pdf/2408.01162
• Copy Paste: [[2408.01162]] PreMix: Boosting Multiple Instance Learning in Digital Histopathology through Pre-training with Intra-Batch Slide Mixing(https://arxiv.org/abs/2408.01162)
• Keywords: extraction, transformer
• Abstract: The classification of gigapixel-sized whole slide images (WSIs), digital representations of histological slides obtained via a high-resolution scanner, faces significant challenges associated with the meticulous and time-consuming nature of fine-grained labeling. While weakly-supervised multiple instance learning (MIL) has emerged as a promising approach, current MIL methods are constrained by their limited ability to leverage the wealth of information embedded within unlabeled WSIs. This limitation often necessitates training MIL feature aggregators from scratch after the feature extraction process, hindering efficiency and accuracy. PreMix extends the general MIL framework by pre-training the MIL aggregator with an intra-batch slide mixing approach. Specifically, PreMix incorporates Barlow Twins Slide Mixing during pre-training, enhancing its ability to handle diverse WSI sizes and maximizing the utility of unlabeled WSIs. Combined with Mixup and Manifold Mixup during fine-tuning, PreMix achieves a mean of 4.7% performance improvement over the baseline MIL framework, the hierarchical image pyramid transformer (HIPT), on the Camelyon16 dataset. The observed improvement across a range of active learning acquisition functions and WSI-labeled training budgets highlights the framework's adaptability to diverse datasets and varying resource constraints. Ultimately, PreMix paves the way for more efficient and accurate WSI classification under limited WSI-labeled datasets, encouraging the broader adoption of unlabeled WSI data in histopathological research. The code is available at https://anonymous.4open.science/r/PreMix

Title: Rethinking Pre-trained Feature Extractor Selection in Multiple Instance Learning for Whole Slide Image Classification

• Authors: Bryan Wong, Mun Yong Yi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01167
• Pdf URL: https://arxiv.org/pdf/2408.01167
• Copy Paste: [[2408.01167]] Rethinking Pre-trained Feature Extractor Selection in Multiple Instance Learning for Whole Slide Image Classification(https://arxiv.org/abs/2408.01167)
• Keywords: transformer
• Abstract: Multiple instance learning (MIL) has become a preferred method for classifying gigapixel whole slide images (WSIs), without requiring patch label annotation. The focus of the current MIL research stream is on the embedding-based MIL approach, which involves extracting feature vectors from patches using a pre-trained feature extractor. These feature vectors are then fed into an MIL aggregator for slide-level prediction. Despite prior research suggestions on enhancing the most commonly used ResNet50 supervised model pre-trained on ImageNet-1K, there remains a lack of clear guidance on selecting the optimal feature extractor to maximize WSI performance. This study aims at addressing this gap by examining MIL feature extractors across three dimensions: pre-training dataset, backbone model, and pre-training method. Extensive experiments were carried out on the two public WSI datasets (TCGA-NSCLC and Camelyon16) using four SOTA MIL models. The main findings indicate the following: 1) Performance significantly improves with larger and more varied pre-training datasets in both CNN and Transformer backbones. 2) `Modern and deeper' backbones greatly outperform `standard' backbones (ResNet and ViT), with performance improvements more guaranteed in Transformer-based backbones. 3) The choice of self-supervised learning (SSL) method is crucial, with the most significant benefits observed when applied to the Transformer (ViT) backbone. The study findings have practical implications, including designing more effective pathological foundation models. Our code is available at: https://anonymous.4open.science/r/MIL-Feature-Extractor-Selection

Title: Misinforming LLMs: vulnerabilities, challenges and opportunities

• Authors: Bo Zhou, Daniel Geißler, Paul Lukowicz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01168
• Pdf URL: https://arxiv.org/pdf/2408.01168
• Copy Paste: [[2408.01168]] Misinforming LLMs: vulnerabilities, challenges and opportunities(https://arxiv.org/abs/2408.01168)
• Keywords: transformer, generative, large language model
• Abstract: Large Language Models (LLMs) have made significant advances in natural language processing, but their underlying mechanisms are often misunderstood. Despite exhibiting coherent answers and apparent reasoning behaviors, LLMs rely on statistical patterns in word embeddings rather than true cognitive processes. This leads to vulnerabilities such as "hallucination" and misinformation. The paper argues that current LLM architectures are inherently untrustworthy due to their reliance on correlations of sequential patterns of word embedding vectors. However, ongoing research into combining generative transformer-based models with fact bases and logic programming languages may lead to the development of trustworthy LLMs capable of generating statements based on given truth and explaining their self-reasoning process.

Title: EmoBack: Backdoor Attacks Against Speaker Identification Using Emotional Prosody

• Authors: Coen Schoof, Stefanos Koffas, Mauro Conti, Stjepan Picek
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.01178
• Pdf URL: https://arxiv.org/pdf/2408.01178
• Copy Paste: [[2408.01178]] EmoBack: Backdoor Attacks Against Speaker Identification Using Emotional Prosody(https://arxiv.org/abs/2408.01178)
• Keywords: defense, attack, robust
• Abstract: Speaker identification (SI) determines a speaker's identity based on their spoken utterances. Previous work indicates that SI deep neural networks (DNNs) are vulnerable to backdoor attacks. Backdoor attacks involve embedding hidden triggers in DNNs' training data, causing the DNN to produce incorrect output when these triggers are present during inference. This is the first work that explores SI DNNs' vulnerability to backdoor attacks using speakers' emotional prosody, resulting in dynamic, inconspicuous triggers. %Such an attack could have real-world implications in forensics, authentication, and surveillance. We conducted a parameter study using three different datasets and DNN architectures to determine the impact of emotions as backdoor triggers on the accuracy of SI systems. Additionally, we have explored the robustness of our attacks by applying defenses like pruning, STRIP-ViTA, and three popular preprocessing techniques: quantization, median filtering, and squeezing. Our findings show that the aforementioned models are prone to our attack, indicating that emotional triggers (sad and neutral prosody) can be effectively used to compromise the integrity of SI systems. However, the results of our pruning experiments suggest potential solutions for reinforcing the models against our attacks, decreasing the attack success rate up to 40%.

Title: VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling

• Authors: Qian Zhang, Xiangzi Dai, Ninghua Yang, Xiang An, Ziyong Feng, Xingyu Ren
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01181
• Pdf URL: https://arxiv.org/pdf/2408.01181
• Copy Paste: [[2408.01181]] VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling(https://arxiv.org/abs/2408.01181)
• Keywords: robust, transformer
• Abstract: VAR is a new generation paradigm that employs 'next-scale prediction' as opposed to 'next-token prediction'. This innovative transformation enables auto-regressive (AR) transformers to rapidly learn visual distributions and achieve robust generalization. However, the original VAR model is constrained to class-conditioned synthesis, relying solely on textual captions for guidance. In this paper, we introduce VAR-CLIP, a novel text-to-image model that integrates Visual Auto-Regressive techniques with the capabilities of CLIP. The VAR-CLIP framework encodes captions into text embeddings, which are then utilized as textual conditions for image generation. To facilitate training on extensive datasets, such as ImageNet, we have constructed a substantial image-text dataset leveraging BLIP2. Furthermore, we delve into the significance of word positioning within CLIP for the purpose of caption guidance. Extensive experiments confirm VAR-CLIP's proficiency in generating fantasy images with high fidelity, textual congruence, and aesthetic excellence. Our project page are this https URL

Title: A Weakly Supervised and Globally Explainable Learning Framework for Brain Tumor Segmentation

• Authors: Ruitao Xie, Limai Jiang, Xiaoxi He, Yi Pan, Yunpeng Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01191
• Pdf URL: https://arxiv.org/pdf/2408.01191
• Copy Paste: [[2408.01191]] A Weakly Supervised and Globally Explainable Learning Framework for Brain Tumor Segmentation(https://arxiv.org/abs/2408.01191)
• Keywords: explainability, segmentation
• Abstract: Machine-based brain tumor segmentation can help doctors make better diagnoses. However, the complex structure of brain tumors and expensive pixel-level annotations present challenges for automatic tumor segmentation. In this paper, we propose a counterfactual generation framework that not only achieves exceptional brain tumor segmentation performance without the need for pixel-level annotations, but also provides explainability. Our framework effectively separates class-related features from class-unrelated features of the samples, and generate new samples that preserve identity features while altering class attributes by embedding different class-related features. We perform topological data analysis on the extracted class-related features and obtain a globally explainable manifold, and for each abnormal sample to be segmented, a meaningful normal sample could be effectively generated with the guidance of the rule-based paths designed within the manifold for comparison for identifying the tumor regions. We evaluate our proposed method on two datasets, which demonstrates superior performance of brain tumor segmentation. The code is available at this https URL.

Title: High-Throughput Phenotyping of Clinical Text Using Large Language Models

• Authors: Daniel B. Hier, S. Ilyas Munzir, Anne Stahlfeld, Tayo Obafemi-Ajayi, Michael D. Carrithers
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01214
• Pdf URL: https://arxiv.org/pdf/2408.01214
• Copy Paste: [[2408.01214]] High-Throughput Phenotyping of Clinical Text Using Large Language Models(https://arxiv.org/abs/2408.01214)
• Keywords: large language model
• Abstract: High-throughput phenotyping automates the mapping of patient signs to standardized ontology concepts and is essential for precision medicine. This study evaluates the automation of phenotyping of clinical summaries from the Online Mendelian Inheritance in Man (OMIM) database using large language models. Due to their rich phenotype data, these summaries can be surrogates for physician notes. We conduct a performance comparison of GPT-4 and GPT-3.5-Turbo. Our results indicate that GPT-4 surpasses GPT-3.5-Turbo in identifying, categorizing, and normalizing signs, achieving concordance with manual annotators comparable to inter-rater agreement. Despite some limitations in sign normalization, the extensive pre-training of GPT-4 results in high performance and generalizability across several phenotyping tasks while obviating the need for manually annotated training data. Large language models are expected to be the dominant method for automating high-throughput phenotyping of clinical text.

Title: ZNorm: Z-Score Gradient Normalization for Accelerating Neural Network Training

• Authors: Juyoung Yun, Hoyoung Kim, Suin Cho, Hangil Kang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.01215
• Pdf URL: https://arxiv.org/pdf/2408.01215
• Copy Paste: [[2408.01215]] ZNorm: Z-Score Gradient Normalization for Accelerating Neural Network Training(https://arxiv.org/abs/2408.01215)
• Keywords: robust, segmentation
• Abstract: The rapid advancements in deep learning necessitate efficient training methods for deep neural networks (DNNs). As models grow in complexity, vanishing and exploding gradients impede convergence and performance. We propose Z-Score Normalization for Gradient Descent (ZNorm), an innovative technique that adjusts only the gradients to enhance training efficiency and improve model performance. ZNorm normalizes the overall gradients, providing consistent gradient scaling across layers, thereby reducing the risks of vanishing and exploding gradients. Our extensive experiments on CIFAR-10 and medical datasets demonstrate that ZNorm not only accelerates convergence but also enhances performance metrics. ZNorm consistently outperforms existing methods, achieving superior results using the same computational settings. In medical imaging applications, ZNorm improves tumor prediction and segmentation performances, underscoring its practical utility. These findings highlight ZNorm's potential as a robust and versatile tool for improving the efficiency and effectiveness of deep neural network training across a wide range of architectures and applications.

Title: Multi-head Spatial-Spectral Mamba for Hyperspectral Image Classification

• Authors: Muhammad Ahmad, Muhammad Hassaan Farooq Butt, Muhammad Usama, Hamad Ahmed Altuwaijri, Manual Mazzara, Salvatore Distenano
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01224
• Pdf URL: https://arxiv.org/pdf/2408.01224
• Copy Paste: [[2408.01224]] Multi-head Spatial-Spectral Mamba for Hyperspectral Image Classification(https://arxiv.org/abs/2408.01224)
• Keywords: transformer
• Abstract: Spatial-Spectral Mamba (SSM) improves computational efficiency and captures long-range dependencies, addressing Transformer limitations. However, traditional Mamba models overlook rich spectral information in HSIs and struggle with high dimensionality and sequential data. To address these issues, we propose the SSM with multi-head self-attention and token enhancement (MHSSMamba). This model integrates spectral and spatial information by enhancing spectral tokens and using multi-head attention to capture complex relationships between spectral bands and spatial locations. It also manages long-range dependencies and the sequential nature of HSI data, preserving contextual information across spectral bands. MHSSMamba achieved remarkable classification accuracies of 97.62\% on Pavia University, 96.92\% on the University of Houston, 96.85\% on Salinas, and 99.49\% on Wuhan-longKou datasets.

Title: The Phantom Menace: Unmasking Privacy Leakages in Vision-Language Models

• Authors: Simone Caldarella, Massimiliano Mancini, Elisa Ricci, Rahaf Aljundi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01228
• Pdf URL: https://arxiv.org/pdf/2408.01228
• Copy Paste: [[2408.01228]] The Phantom Menace: Unmasking Privacy Leakages in Vision-Language Models(https://arxiv.org/abs/2408.01228)
• Keywords: privacy, protect, robust
• Abstract: Vision-Language Models (VLMs) combine visual and textual understanding, rendering them well-suited for diverse tasks like generating image captions and answering visual questions across various domains. However, these capabilities are built upon training on large amount of uncurated data crawled from the web. The latter may include sensitive information that VLMs could memorize and leak, raising significant privacy concerns. In this paper, we assess whether these vulnerabilities exist, focusing on identity leakage. Our study leads to three key findings: (i) VLMs leak identity information, even when the vision-language alignment and the fine-tuning use anonymized data; (ii) context has little influence on identity leakage; (iii) simple, widely used anonymization techniques, like blurring, are not sufficient to address the problem. These findings underscore the urgent need for robust privacy protection strategies when deploying VLMs. Ethical awareness and responsible development practices are essential to mitigate these risks.

Title: WaveMamba: Spatial-Spectral Wavelet Mamba for Hyperspectral Image Classification

• Authors: Muhammad Ahmad, Muhammad Usama, Manual Mazzara
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.01231
• Pdf URL: https://arxiv.org/pdf/2408.01231
• Copy Paste: [[2408.01231]] WaveMamba: Spatial-Spectral Wavelet Mamba for Hyperspectral Image Classification(https://arxiv.org/abs/2408.01231)
• Keywords: transformer
• Abstract: Hyperspectral Imaging (HSI) has proven to be a powerful tool for capturing detailed spectral and spatial information across diverse applications. Despite the advancements in Deep Learning (DL) and Transformer architectures for HSI Classification (HSIC), challenges such as computational efficiency and the need for extensive labeled data persist. This paper introduces WaveMamba, a novel approach that integrates wavelet transformation with the Spatial-Spectral Mamba architecture to enhance HSIC. WaveMamba captures both local texture patterns and global contextual relationships in an end-to-end trainable model. The Wavelet-based enhanced features are then processed through the state-space architecture to model spatial-spectral relationships and temporal dependencies. The experimental results indicate that WaveMamba surpasses existing models, achieving an accuracy improvement of 4.5\% on the University of Houston dataset and a 2.0\% increase on the Pavia University dataset. These findings validate its effectiveness in addressing the complex data interactions inherent in HSIs.

Title: CLIP4Sketch: Enhancing Sketch to Mugshot Matching through Dataset Augmentation using Diffusion Models

• Authors: Kushal Kumar Jain, Steve Grosz, Anoop M. Namboodiri, Anil K. Jain
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01233
• Pdf URL: https://arxiv.org/pdf/2408.01233
• Copy Paste: [[2408.01233]] CLIP4Sketch: Enhancing Sketch to Mugshot Matching through Dataset Augmentation using Diffusion Models(https://arxiv.org/abs/2408.01233)
• Keywords: diffusion
• Abstract: Forensic sketch-to-mugshot matching is a challenging task in face recognition, primarily hindered by the scarcity of annotated forensic sketches and the modality gap between sketches and photographs. To address this, we propose CLIP4Sketch, a novel approach that leverages diffusion models to generate a large and diverse set of sketch images, which helps in enhancing the performance of face recognition systems in sketch-to-mugshot matching. Our method utilizes Denoising Diffusion Probabilistic Models (DDPMs) to generate sketches with explicit control over identity and style. We combine CLIP and Adaface embeddings of a reference mugshot, along with textual descriptions of style, as the conditions to the diffusion model. We demonstrate the efficacy of our approach by generating a comprehensive dataset of sketches corresponding to mugshots and training a face recognition model on our synthetic data. Our results show significant improvements in sketch-to-mugshot matching accuracy over training on an existing, limited amount of real face sketch data, validating the potential of diffusion models in enhancing the performance of face recognition systems across modalities. We also compare our dataset with datasets generated using GAN-based methods to show its superiority.

Title: Automated Classification of Dry Bean Varieties Using XGBoost and SVM Models

• Authors: Ramtin Ardeshirifar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.01244
• Pdf URL: https://arxiv.org/pdf/2408.01244
• Copy Paste: [[2408.01244]] Automated Classification of Dry Bean Varieties Using XGBoost and SVM Models(https://arxiv.org/abs/2408.01244)
• Keywords: robust, extraction
• Abstract: This paper presents a comparative study on the automated classification of seven different varieties of dry beans using machine learning models. Leveraging a dataset of 12,909 dry bean samples, reduced from an initial 13,611 through outlier removal and feature extraction, we applied Principal Component Analysis (PCA) for dimensionality reduction and trained two multiclass classifiers: XGBoost and Support Vector Machine (SVM). The models were evaluated using nested cross-validation to ensure robust performance assessment and hyperparameter tuning. The XGBoost and SVM models achieved overall correct classification rates of 94.00% and 94.39%, respectively. The results underscore the efficacy of these machine learning approaches in agricultural applications, particularly in enhancing the uniformity and efficiency of seed classification. This study contributes to the growing body of work on precision agriculture, demonstrating that automated systems can significantly support seed quality control and crop yield optimization. Future work will explore incorporating more diverse datasets and advanced algorithms to further improve classification accuracy.

Title: MapComp: A Secure View-based Collaborative Analytics Framework for Join-Group-Aggregation

• Authors: Xinyu Peng, Feng Han, Li Peng, Weiran Liu, Zheng Yan, Kai Kang, Xinyuan Zhang, Guoxing Wei, Jianling Sun, Jinfei Liu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.01246
• Pdf URL: https://arxiv.org/pdf/2408.01246
• Copy Paste: [[2408.01246]] MapComp: A Secure View-based Collaborative Analytics Framework for Join-Group-Aggregation(https://arxiv.org/abs/2408.01246)
• Keywords: secure
• Abstract: This paper introduces MapComp, a novel view-based framework to facilitate join-group-aggregation (JGA) queries for collaborative analytics. Through specially crafted materialized view for join and novel design of group-aggregation (GA) protocols, MapComp removes duplicated join workload and expedites subsequent GA, improving the efficiency of JGA query execution. To support continuous data updates, our materialized view offers payload-independence feature and brings in significant efficiency improvement of view refreshing with free MPC overhead. This feature also allows further acceleration for GA, where we devised multiple novel protocols that outperform prior works. Notably, our work represents the first endeavor to expedite secure collaborative JGA queries using materialized views. Our experiments demonstrate a significant advantage of MapComp, achieving up to a 2189.9x efficiency improvement compared to the non-view based baseline when executing queries eight times.

Title: RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework

• Authors: Kunlun Zhu, Yifan Luo, Dingling Xu, Ruobing Wang, Shi Yu, Shuo Wang, Yukun Yan, Zhenghao Liu, Xu Han, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.01262
• Pdf URL: https://arxiv.org/pdf/2408.01262
• Copy Paste: [[2408.01262]] RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework(https://arxiv.org/abs/2408.01262)
• Keywords: large language model
• Abstract: Retrieval-Augmented Generation (RAG) systems have demonstrated their advantages in alleviating the hallucination of Large Language Models (LLMs). Existing RAG benchmarks mainly focus on evaluating whether LLMs can correctly answer the general knowledge. However, they are unable to evaluate the effectiveness of the RAG system in dealing with the data from different vertical domains. This paper introduces RAGEval, a framework for automatically generating evaluation datasets to evaluate the knowledge usage ability of different LLMs in different scenarios. Specifically, RAGEval summarizes a schema from seed documents, applies the configurations to generate diverse documents, and constructs question-answering pairs according to both articles and configurations. We propose three novel metrics, Completeness, Hallucination, and Irrelevance, to carefully evaluate the responses generated by LLMs. By benchmarking RAG models in vertical domains, RAGEval has the ability to better evaluate the knowledge usage ability of LLMs, which avoids the confusion regarding the source of knowledge in answering question in existing QA datasets--whether it comes from parameterized memory or retrieval.

Title: A General Framework to Boost 3D GS Initialization for Text-to-3D Generation by Lexical Richness

• Authors: Lutao Jiang, Hangyu Li, Lin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01269
• Pdf URL: https://arxiv.org/pdf/2408.01269
• Copy Paste: [[2408.01269]] A General Framework to Boost 3D GS Initialization for Text-to-3D Generation by Lexical Richness(https://arxiv.org/abs/2408.01269)
• Keywords: diffusion
• Abstract: Text-to-3D content creation has recently received much attention, especially with the prevalence of 3D Gaussians Splatting. In general, GS-based methods comprise two key stages: initialization and rendering optimization. To achieve initialization, existing works directly apply random sphere initialization or 3D diffusion models, e.g., Point-E, to derive the initial shapes. However, such strategies suffer from two critical yet challenging problems: 1) the final shapes are still similar to the initial ones even after training; 2) shapes can be produced only from simple texts, e.g., "a dog", not for lexically richer texts, e.g., "a dog is sitting on the top of the airplane". To address these problems, this paper proposes a novel general framework to boost the 3D GS Initialization for text-to-3D generation upon the lexical richness. Our key idea is to aggregate 3D Gaussians into spatially uniform voxels to represent complex shapes while enabling the spatial interaction among the 3D Gaussians and semantic interaction between Gaussians and texts. Specifically, we first construct a voxelized representation, where each voxel holds a 3D Gaussian with its position, scale, and rotation fixed while setting opacity as the sole factor to determine a position's occupancy. We then design an initialization network mainly consisting of two novel components: 1) Global Information Perception (GIP) block and 2) Gaussians-Text Fusion (GTF) block. Such a design enables each 3D Gaussian to assimilate the spatial information from other areas and semantic information from texts. Extensive experiments show the superiority of our framework of high-quality 3D GS initialization against the existing methods, e.g., Shap-E, by taking lexically simple, medium, and hard texts. Also, our framework can be seamlessly plugged into SoTA training frameworks, e.g., LucidDreamer, for semantically consistent text-to-3D generation.

Title: Certified Robust Invariant Polytope Training in Neural Controlled ODEs

• Authors: Akash Harapanahalli, Samuel Coogan
• Subjects: cs.LG, eess.SY, math.OC
• Abstract URL: https://arxiv.org/abs/2408.01273
• Pdf URL: https://arxiv.org/pdf/2408.01273
• Copy Paste: [[2408.01273]] Certified Robust Invariant Polytope Training in Neural Controlled ODEs(https://arxiv.org/abs/2408.01273)
• Keywords: robust
• Abstract: We consider a nonlinear control system modeled as an ordinary differential equation subject to disturbance, with a state feedback controller parameterized as a feedforward neural network. We propose a framework for training controllers with certified robust forward invariant polytopes, where any trajectory initialized inside the polytope remains within the polytope, regardless of the disturbance. First, we parameterize a family of lifted control systems in a higher dimensional space, where the original neural controlled system evolves on an invariant subspace of each lifted system. We use interval analysis and neural network verifiers to further construct a family of lifted embedding systems, carefully capturing the knowledge of this invariant subspace. If the vector field of any lifted embedding system satisfies a sign constraint at a single point, then a certain convex polytope of the original system is robustly forward invariant. Treating the neural network controller and the lifted system parameters as variables, we propose an algorithm to train controllers with certified forward invariant polytopes in the closed-loop control system. Through two examples, we demonstrate how the simplicity of the sign constraint allows our approach to scale with system dimension to over $50$ states, and outperform state-of-the-art Lyapunov-based sampling approaches in runtime.

Title: The Mismeasure of Man and Models: Evaluating Allocational Harms in Large Language Models

• Authors: Hannah Chen, Yangfeng Ji, David Evans
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.01285
• Pdf URL: https://arxiv.org/pdf/2408.01285
• Copy Paste: [[2408.01285]] The Mismeasure of Man and Models: Evaluating Allocational Harms in Large Language Models(https://arxiv.org/abs/2408.01285)
• Keywords: large language model
• Abstract: Large language models (LLMs) are now being considered and even deployed for applications that support high-stakes decision-making, such as recruitment and clinical decisions. While several methods have been proposed for measuring bias, there remains a gap between predictions, which are what the proposed methods consider, and how they are used to make decisions. In this work, we introduce Rank-Allocational-Based Bias Index (RABBI), a model-agnostic bias measure that assesses potential allocational harms arising from biases in LLM predictions. We compare RABBI and current bias metrics on two allocation decision tasks. We evaluate their predictive validity across ten LLMs and utility for model selection. Our results reveal that commonly-used bias metrics based on average performance gap and distribution distance fail to reliably capture group disparities in allocation outcomes, whereas RABBI exhibits a strong correlation with allocation disparities. Our work highlights the need to account for how models are used in contexts with limited resource constraints.

Title: Deep Learning based Visually Rich Document Content Understanding: A Survey

• Authors: Yihao Ding, Jean Lee, Soyeon Caren Han
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01287
• Pdf URL: https://arxiv.org/pdf/2408.01287
• Copy Paste: [[2408.01287]] Deep Learning based Visually Rich Document Content Understanding: A Survey(https://arxiv.org/abs/2408.01287)
• Keywords: extraction
• Abstract: Visually Rich Documents (VRDs) are essential in academia, finance, medical fields, and marketing due to their multimodal information content. Traditional methods for extracting information from VRDs depend on expert knowledge and manual labor, making them costly and inefficient. The advent of deep learning has revolutionized this process, introducing models that leverage multimodal information vision, text, and layout along with pretraining tasks to develop comprehensive document representations. These models have achieved state-of-the-art performance across various downstream tasks, significantly enhancing the efficiency and accuracy of information extraction from VRDs. In response to the growing demands and rapid developments in Visually Rich Document Understanding (VRDU), this paper provides a comprehensive review of deep learning-based VRDU frameworks. We systematically survey and analyze existing methods and benchmark datasets, categorizing them based on adopted strategies and downstream tasks. Furthermore, we compare different techniques used in VRDU models, focusing on feature representation and fusion, model architecture, and pretraining methods, while highlighting their strengths, limitations, and appropriate scenarios. Finally, we identify emerging trends and challenges in VRDU, offering insights into future research directions and practical applications. This survey aims to provide a thorough understanding of VRDU advancements, benefiting both academic and industrial sectors.

Title: TexGen: Text-Guided 3D Texture Generation with Multi-view Sampling and Resampling

• Authors: Dong Huo, Zixin Guo, Xinxin Zuo, Zhihao Shi, Juwei Lu, Peng Dai, Songcen Xu, Li Cheng, Yee-Hong Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01291
• Pdf URL: https://arxiv.org/pdf/2408.01291
• Copy Paste: [[2408.01291]] TexGen: Text-Guided 3D Texture Generation with Multi-view Sampling and Resampling(https://arxiv.org/abs/2408.01291)
• Keywords: diffusion
• Abstract: Given a 3D mesh, we aim to synthesize 3D textures that correspond to arbitrary textual descriptions. Current methods for generating and assembling textures from sampled views often result in prominent seams or excessive smoothing. To tackle these issues, we present TexGen, a novel multi-view sampling and resampling framework for texture generation leveraging a pre-trained text-to-image diffusion model. For view consistent sampling, first of all we maintain a texture map in RGB space that is parameterized by the denoising step and updated after each sampling step of the diffusion model to progressively reduce the view discrepancy. An attention-guided multi-view sampling strategy is exploited to broadcast the appearance information across views. To preserve texture details, we develop a noise resampling technique that aids in the estimation of noise, generating inputs for subsequent denoising steps, as directed by the text prompt and current texture map. Through an extensive amount of qualitative and quantitative evaluations, we demonstrate that our proposed method produces significantly better texture quality for diverse 3D objects with a high degree of view consistency and rich appearance details, outperforming current state-of-the-art methods. Furthermore, our proposed texture generation technique can also be applied to texture editing while preserving the original identity. More experimental results are available at this https URL

Title: Underwater Object Detection Enhancement via Channel Stabilization

• Authors: Muhammad Ali, Salman Khan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01293
• Pdf URL: https://arxiv.org/pdf/2408.01293
• Copy Paste: [[2408.01293]] Underwater Object Detection Enhancement via Channel Stabilization(https://arxiv.org/abs/2408.01293)
• Keywords: transformer
• Abstract: The complex marine environment exacerbates the challenges of object detection manifold. Marine trash endangers the aquatic ecosystem, presenting a persistent challenge. Accurate detection of marine deposits is crucial for mitigating this harm. Our work addresses underwater object detection by enhancing image quality and evaluating detection methods. We use Detectron2's backbone with various base models and configurations for this task. We propose a novel channel stabilization technique alongside a simplified image enhancement model to reduce haze and color cast in training images, improving multi-scale object detection. Following image processing, we test different Detectron2 backbones for optimal detection accuracy. Additionally, we apply a sharpening filter with augmentation techniques to highlight object profiles for easier recognition. Results are demonstrated on the TrashCan Dataset, both instance and material versions. The best-performing backbone method incorporates our channel stabilization and augmentation techniques. We also compare our Detectron2 detection results with the Deformable Transformer. In the instance version of TrashCan 1.0, our method achieves a 9.53% absolute increase in average precision for small objects and a 7% absolute gain in bounding box detection compared to the baseline. The code will be available on Code: this https URL Object-Detection-via-Channel-Stablization

Title: Feature Clock: High-Dimensional Effects in Two-Dimensional Plots

• Authors: Olga Ovcharenko, Rita Sevastjanova, Valentina Boeva
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.01294
• Pdf URL: https://arxiv.org/pdf/2408.01294
• Copy Paste: [[2408.01294]] Feature Clock: High-Dimensional Effects in Two-Dimensional Plots(https://arxiv.org/abs/2408.01294)
• Keywords: explainability
• Abstract: Humans struggle to perceive and interpret high-dimensional data. Therefore, high-dimensional data are often projected into two dimensions for visualization. Many applications benefit from complex nonlinear dimensionality reduction techniques, but the effects of individual high-dimensional features are hard to explain in the two-dimensional space. Most visualization solutions use multiple two-dimensional plots, each showing the effect of one high-dimensional feature in two dimensions; this approach creates a need for a visual inspection of k plots for a k-dimensional input space. Our solution, Feature Clock, provides a novel approach that eliminates the need to inspect these k plots to grasp the influence of original features on the data structure depicted in two dimensions. Feature Clock enhances the explainability and compactness of visualizations of embedded data and is available in an open-source Python library.

Title: Optimal Mixed Integer Linear Optimization Trained Multivariate Classification Trees

• Authors: Brandon Alston, Illya V. Hicks
• Subjects: cs.LG, cs.DM, math.CO
• Abstract URL: https://arxiv.org/abs/2408.01297
• Pdf URL: https://arxiv.org/pdf/2408.01297
• Copy Paste: [[2408.01297]] Optimal Mixed Integer Linear Optimization Trained Multivariate Classification Trees(https://arxiv.org/abs/2408.01297)
• Keywords: robust
• Abstract: Multivariate decision trees are powerful machine learning tools for classification and regression that attract many researchers and industry professionals. An optimal binary tree has two types of vertices, (i) branching vertices which have exactly two children and where datapoints are assessed on a set of discrete features and (ii) leaf vertices at which datapoints are given a prediction, and can be obtained by solving a biobjective optimization problem that seeks to (i) maximize the number of correctly classified datapoints and (ii) minimize the number of branching vertices. Branching vertices are linear combinations of training features and therefore can be thought of as hyperplanes. In this paper, we propose two cut-based mixed integer linear optimization (MILO) formulations for designing optimal binary classification trees (leaf vertices assign discrete classes). Our models leverage on-the-fly identification of minimal infeasible subsystems (MISs) from which we derive cutting planes that hold the form of packing constraints. We show theoretical improvements on the strongest flow-based MILO formulation currently in the literature and conduct experiments on publicly available datasets to show our models' ability to scale, strength against traditional branch and bound approaches, and robustness in out-of-sample test performance. Our code and data are available on GitHub.

Title: A Systematic Mapping Study on SDN Controllers for Enhancing Security in IoT Networks

• Authors: Charles Oredola, Adnan Ashraf
• Subjects: cs.CR, cs.NI, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.01303
• Pdf URL: https://arxiv.org/pdf/2408.01303
• Copy Paste: [[2408.01303]] A Systematic Mapping Study on SDN Controllers for Enhancing Security in IoT Networks(https://arxiv.org/abs/2408.01303)
• Keywords: security
• Abstract: Context: The increase in Internet of Things (IoT) devices gives rise to an increase in deceptive manipulations by malicious actors. These actors should be prevented from targeting the IoT networks. Cybersecurity threats have evolved and become dynamically sophisticated, such that they could exploit any vulnerability found in IoT networks. However, with the introduction of the Software Defined Network (SDN) in the IoT networks as the central monitoring unit, IoT networks are less vulnerable and less prone to threats. %Although, the SDN itself is vulnerable to several threats. Objective: To present a comprehensive and unbiased overview of the state-of-the-art on IoT networks security enhancement using SDN controllers. Method: We review the current body of knowledge on enhancing the security of IoT networks using SDN with a Systematic Mapping Study (SMS) following the established guidelines. Results: The SMS result comprises 33 primary studies analyzed against four major research questions. The SMS highlights current research trends and identifies gaps in the SDN-IoT network security. Conclusion: We conclude that the SDN controller architecture commonly used for securing IoT networks is the centralized controller architecture. However, this architecture is not without its limitations. Additionally, the predominant technique utilized for risk mitigation is machine learning.

Title: Reconsidering Token Embeddings with the Definitions for Pre-trained Language Models

• Authors: Ying Zhang, Dongyuan Li, Manabu Okumura
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01308
• Pdf URL: https://arxiv.org/pdf/2408.01308
• Copy Paste: [[2408.01308]] Reconsidering Token Embeddings with the Definitions for Pre-trained Language Models(https://arxiv.org/abs/2408.01308)
• Keywords: robust
• Abstract: Learning token embeddings based on token co-occurrence statistics has proven effective for both pre-training and fine-tuning in natural language processing. However, recent studies have pointed out the distribution of learned embeddings degenerates into anisotropy, and even pre-trained language models (PLMs) suffer from a loss of semantics-related information in embeddings for low-frequency tokens. This study first analyzes fine-tuning dynamics of a PLM, BART-large, and demonstrates its robustness against degeneration. On the basis of this finding, we propose DefinitionEMB, a method that utilizes definitions to construct isotropically distributed and semantics-related token embeddings for PLMs while maintaining original robustness during fine-tuning. Our experiments demonstrate the effectiveness of leveraging definitions from Wiktionary to construct such embeddings for RoBERTa-base and BART-large. Furthermore, the constructed embeddings for low-frequency tokens improve the performance of these models across various GLUE and four text summarization datasets.

Title: PsybORG+: Cognitive Modeling for Triggering and Detection of Cognitive Biases of Advanced Persistent Threats

• Authors: Shuo Huang, Quanyan Zhu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.01310
• Pdf URL: https://arxiv.org/pdf/2408.01310
• Copy Paste: [[2408.01310]] PsybORG+: Cognitive Modeling for Triggering and Detection of Cognitive Biases of Advanced Persistent Threats(https://arxiv.org/abs/2408.01310)
• Keywords: security, attack, steal
• Abstract: Advanced Persistent Threats (APTs) bring significant challenge to cybersecurity due to their sophisticated and stealthy nature. Traditional cybersecurity measures fail to defend against APTs. Cognitive vulnerabilities can significantly influence attackers' decision-making processes, which presents an opportunity for defenders to exploit these weaknesses. This paper introduces PsybORG, a multi-agent cybersecurity simulation environment designed to model APT behaviors influenced by cognitive vulnerabilities. PsybORG uses a Hidden Markov Model (HMM) to simulate attacker behaviors. We use Bayesian inference and decision tree analysis of action sequences to do cognitive vulnerabilities inference. In addition, a system called PsybORG+ is built for generating synthetic data. We also design a trigger to stimulate the sunk cost fallacy in attackers. Our contributions include the mathematical modeling of APTs, the development of PsybORG, and the implementation of techniques to infer attackers' cognitive vulnerabilities.

Title: A Robotics-Inspired Scanpath Model Reveals the Importance of Uncertainty and Semantic Object Cues for Gaze Guidance in Dynamic Scenes

• Authors: Vito Mengers, Nicolas Roth, Oliver Brock, Klaus Obermayer, Martin Rolfs
• Subjects: cs.CV, cs.AI, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2408.01322
• Pdf URL: https://arxiv.org/pdf/2408.01322
• Copy Paste: [[2408.01322]] A Robotics-Inspired Scanpath Model Reveals the Importance of Uncertainty and Semantic Object Cues for Gaze Guidance in Dynamic Scenes(https://arxiv.org/abs/2408.01322)
• Keywords: segmentation
• Abstract: How we perceive objects around us depends on what we actively attend to, yet our eye movements depend on the perceived objects. Still, object segmentation and gaze behavior are typically treated as two independent processes. Drawing on an information processing pattern from robotics, we present a mechanistic model that simulates these processes for dynamic real-world scenes. Our image-computable model uses the current scene segmentation for object-based saccadic decision-making while using the foveated object to refine its scene segmentation recursively. To model this refinement, we use a Bayesian filter, which also provides an uncertainty estimate for the segmentation that we use to guide active scene exploration. We demonstrate that this model closely resembles observers' free viewing behavior, measured by scanpath statistics, including foveation duration and saccade amplitude distributions used for parameter fitting and higher-level statistics not used for fitting. These include how object detections, inspections, and returns are balanced and a delay of returning saccades without an explicit implementation of such temporal inhibition of return. Extensive simulations and ablation studies show that uncertainty promotes balanced exploration and that semantic object cues are crucial to form the perceptual units used in object-based attention. Moreover, we show how our model's modular design allows for extensions, such as incorporating saccadic momentum or pre-saccadic attention, to further align its output with human scanpaths.

Title: FANNO: Augmenting High-Quality Instruction Data with Open-Sourced LLMs Only

• Authors: He Zhu, Junyou Su, Tianle Lun, Yicheng Tao, Wenjia Zhang, Zipei Fan, Guanhua Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01323
• Pdf URL: https://arxiv.org/pdf/2408.01323
• Copy Paste: [[2408.01323]] FANNO: Augmenting High-Quality Instruction Data with Open-Sourced LLMs Only(https://arxiv.org/abs/2408.01323)
• Keywords: large language model
• Abstract: Instruction fine-tuning stands as a crucial advancement in leveraging large language models (LLMs) for enhanced task performance. However, the annotation of instruction datasets has traditionally been expensive and laborious, often relying on manual annotations or costly API calls of proprietary LLMs. To address these challenges, we introduce FANNO, a fully autonomous, open-sourced framework that revolutionizes the annotation process without the need for pre-existing annotated data. Utilizing a Mistral-7b-instruct model, FANNO efficiently produces diverse and high-quality datasets through a structured process involving document pre-screening, instruction generation, and response generation. Experiments on Open LLM Leaderboard and AlpacaEval benchmark show that the FANNO can generate high-quality data with diversity and complexity for free, comparable to human-annotated or cleaned datasets like Alpaca-GPT4-Cleaned.

Title: StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentation

• Authors: Bingyu Li, Da Zhang, Zhiyuan Zhao, Junyu Gao, Xuelong Li
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.01343
• Pdf URL: https://arxiv.org/pdf/2408.01343
• Copy Paste: [[2408.01343]] StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentation(https://arxiv.org/abs/2408.01343)
• Keywords: segmentation
• Abstract: Multimodal semantic segmentation shows significant potential for enhancing segmentation accuracy in complex scenes. However, current methods often incorporate specialized feature fusion modules tailored to specific modalities, thereby restricting input flexibility and increasing the number of training parameters. To address these challenges, we propose StitchFusion, a straightforward yet effective modal fusion framework that integrates large-scale pre-trained models directly as encoders and feature fusers. This approach facilitates comprehensive multi-modal and multi-scale feature fusion, accommodating any visual modal inputs. Specifically, Our framework achieves modal integration during encoding by sharing multi-modal visual information. To enhance information exchange across modalities, we introduce a multi-directional adapter module (MultiAdapter) to enable cross-modal information transfer during encoding. By leveraging MultiAdapter to propagate multi-scale information across pre-trained encoders during the encoding process, StitchFusion achieves multi-modal visual information integration during encoding. Extensive comparative experiments demonstrate that our model achieves state-of-the-art performance on four multi-modal segmentation datasets with minimal additional parameters. Furthermore, the experimental integration of MultiAdapter with existing Feature Fusion Modules (FFMs) highlights their complementary nature. Our code is available at StitchFusion_repo.

Title: MCGMark: An Encodable and Robust Online Watermark for LLM-Generated Malicious Code

• Authors: Kaiwen Ning, Jiachi Chen, Qingyuan Zhong, Tao Zhang, Yanlin Wang, Wei Li, Yu Zhang, Weizhe Zhang, Zibin Zheng
• Subjects: cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2408.01354
• Pdf URL: https://arxiv.org/pdf/2408.01354
• Copy Paste: [[2408.01354]] MCGMark: An Encodable and Robust Online Watermark for LLM-Generated Malicious Code(https://arxiv.org/abs/2408.01354)
• Keywords: attack, robust, watermark, large language model
• Abstract: With the advent of large language models (LLMs), numerous software service providers (SSPs) are dedicated to developing LLMs customized for code generation tasks, such as CodeLlama and Copilot. However, these LLMs can be leveraged by attackers to create malicious software, which may pose potential threats to the software ecosystem. For example, they can automate the creation of advanced phishing malware. To address this issue, we first conduct an empirical study and design a prompt dataset, MCGTest, which involves approximately 400 person-hours of work and consists of 406 malicious code generation tasks. Utilizing this dataset, we propose MCGMark, the first robust, code structure-aware, and encodable watermarking approach to trace LLM-generated code. We embed encodable information by controlling the token selection and ensuring the output quality based on probabilistic outliers. Additionally, we enhance the robustness of the watermark by considering the structural features of malicious code, preventing the embedding of the watermark in easily modified positions, such as comments. We validate the effectiveness and robustness of MCGMark on the DeepSeek-Coder. MCGMark achieves an embedding success rate of 88.9% within a maximum output limit of 400 tokens. Furthermore, it also demonstrates strong robustness and has minimal impact on the quality of the output code. Our approach assists SSPs in tracing and holding responsible parties accountable for malicious code generated by LLMs.

Title: Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs

• Authors: Peng Ding, Jingyu Wu, Jun Kuang, Dan Ma, Xuezhi Cao, Xunliang Cai, Shi Chen, Jiajun Chen, Shujian Huang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.01355
• Pdf URL: https://arxiv.org/pdf/2408.01355
• Copy Paste: [[2408.01355]] Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs(https://arxiv.org/abs/2408.01355)
• Keywords: generative, large language model
• Abstract: Multi-modal Large Language Models (MLLMs) have demonstrated remarkable performance on various visual-language understanding and generation tasks. However, MLLMs occasionally generate content inconsistent with the given images, which is known as "hallucination". Prior works primarily center on evaluating hallucination using standard, unperturbed benchmarks, which overlook the prevalent occurrence of perturbed inputs in real-world scenarios-such as image cropping or blurring-that are critical for a comprehensive assessment of MLLMs' hallucination. In this paper, to bridge this gap, we propose Hallu-PI, the first benchmark designed to evaluate Hallucination in MLLMs within Perturbed Inputs. Specifically, Hallu-PI consists of seven perturbed scenarios, containing 1,260 perturbed images from 11 object types. Each image is accompanied by detailed annotations, which include fine-grained hallucination types, such as existence, attribute, and relation. We equip these annotations with a rich set of questions, making Hallu-PI suitable for both discriminative and generative tasks. Extensive experiments on 12 mainstream MLLMs, such as GPT-4V and Gemini-Pro Vision, demonstrate that these models exhibit significant hallucinations on Hallu-PI, which is not observed in unperturbed scenarios. Furthermore, our research reveals a severe bias in MLLMs' ability to handle different types of hallucinations. We also design two baselines specifically for perturbed scenarios, namely Perturbed-Reminder and Perturbed-ICL. We hope that our study will bring researchers' attention to the limitations of MLLMs when dealing with perturbed inputs, and spur further investigations to address this issue. Our code and datasets are publicly available at this https URL.

Title: Balanced Residual Distillation Learning for 3D Point Cloud Class-Incremental Semantic Segmentation

• Authors: Yuanzhi Su, Siyuan Chen, Yuan-Gen Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01356
• Pdf URL: https://arxiv.org/pdf/2408.01356
• Copy Paste: [[2408.01356]] Balanced Residual Distillation Learning for 3D Point Cloud Class-Incremental Semantic Segmentation(https://arxiv.org/abs/2408.01356)
• Keywords: segmentation
• Abstract: Class-incremental learning (CIL) thrives due to its success in processing the influx of information by learning from continuously added new classes while preventing catastrophic forgetting about the old ones. It is essential for the performance breakthrough of CIL to effectively refine past knowledge from the base model and balance it with new learning. However, such an issue has not yet been considered in current research. In this work, we explore the potential of CIL from these perspectives and propose a novel balanced residual distillation framework (BRD-CIL) to push the performance bar of CIL to a new higher level. Specifically, BRD-CIL designs a residual distillation learning strategy, which can dynamically expand the network structure to capture the residuals between the base and target models, effectively refining the past knowledge. Furthermore, BRD-CIL designs a balanced pseudo-label learning strategy by generating a guidance mask to reduce the preference for old classes, ensuring balanced learning from new and old classes. We apply the proposed BRD-CIL to a challenging 3D point cloud semantic segmentation task where the data are unordered and unstructured. Extensive experimental results demonstrate that BRD-CIL sets a new benchmark with an outstanding balance capability in class-biased scenarios.

Title: Transformers are Universal In-context Learners

• Authors: Takashi Furuya, Maarten V. de Hoop, Gabriel Peyré
• Subjects: cs.CL, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.01367
• Pdf URL: https://arxiv.org/pdf/2408.01367
• Copy Paste: [[2408.01367]] Transformers are Universal In-context Learners(https://arxiv.org/abs/2408.01367)
• Keywords: transformer
• Abstract: Transformers are deep architectures that define "in-context mappings" which enable predicting new tokens based on a given set of tokens (such as a prompt in NLP applications or a set of patches for vision transformers). This work studies in particular the ability of these architectures to handle an arbitrarily large number of context tokens. To mathematically and uniformly address the expressivity of these architectures, we consider the case that the mappings are conditioned on a context represented by a probability distribution of tokens (discrete for a finite number of tokens). The related notion of smoothness corresponds to continuity in terms of the Wasserstein distance between these contexts. We demonstrate that deep transformers are universal and can approximate continuous in-context mappings to arbitrary precision, uniformly over compact token domains. A key aspect of our results, compared to existing findings, is that for a fixed precision, a single transformer can operate on an arbitrary (even infinite) number of tokens. Additionally, it operates with a fixed embedding dimension of tokens (this dimension does not increase with precision) and a fixed number of heads (proportional to the dimension). The use of MLP layers between multi-head attention layers is also explicitly controlled.

Title: EVIT: Event-based Visual-Inertial Tracking in Semi-Dense Maps Using Windowed Nonlinear Optimization

• Authors: Runze Yuan, Tao Liu, Zijia Dai, Yi-Fan Zuo, Laurent Kneip
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2408.01370
• Pdf URL: https://arxiv.org/pdf/2408.01370
• Copy Paste: [[2408.01370]] EVIT: Event-based Visual-Inertial Tracking in Semi-Dense Maps Using Windowed Nonlinear Optimization(https://arxiv.org/abs/2408.01370)
• Keywords: robust
• Abstract: Event cameras are an interesting visual exteroceptive sensor that reacts to brightness changes rather than integrating absolute image intensities. Owing to this design, the sensor exhibits strong performance in situations of challenging dynamics and illumination conditions. While event-based simultaneous tracking and mapping remains a challenging problem, a number of recent works have pointed out the sensor's suitability for prior map-based tracking. By making use of cross-modal registration paradigms, the camera's ego-motion can be tracked across a large spectrum of illumination and dynamics conditions on top of accurate maps that have been created a priori by more traditional sensors. The present paper follows up on a recently introduced event-based geometric semi-dense tracking paradigm, and proposes the addition of inertial signals in order to robustify the estimation. More specifically, the added signals provide strong cues for pose initialization as well as regularization during windowed, multi-frame tracking. As a result, the proposed framework achieves increased performance under challenging illumination conditions as well as a reduction of the rate at which intermediate event representations need to be registered in order to maintain stable tracking across highly dynamic sequences. Our evaluation focuses on a diverse set of real world sequences and comprises a comparison of our proposed method against a purely event-based alternative running at different rates.

Title: Spatial-Spectral Morphological Mamba for Hyperspectral Image Classification

• Authors: Muhammad Ahmad, Muhammad Hassaan Farooq Butt, Muhammad Usama, Adil Mehmood Khan, Manual Mazzara, Salvatore Distenano
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.01372
• Pdf URL: https://arxiv.org/pdf/2408.01372
• Copy Paste: [[2408.01372]] Spatial-Spectral Morphological Mamba for Hyperspectral Image Classification(https://arxiv.org/abs/2408.01372)
• Keywords: transformer
• Abstract: In recent years, Transformers have garnered significant attention for Hyperspectral Image Classification (HSIC) due to their self-attention mechanism, which provides strong classification performance. However, these models face major challenges in computational efficiency, as their complexity increases quadratically with the sequence length. The Mamba architecture, leveraging a State Space Model, offers a more efficient alternative to Transformers. This paper introduces the Spatial-Spectral Morphological Mamba (MorpMamba) model. In the MorpMamba model, a token generation module first converts the Hyperspectral Image (HSI) patch into spatial-spectral tokens. These tokens are then processed by a morphology block, which computes structural and shape information using depthwise separable convolutional operations. The extracted information is enhanced in a feature enhancement module that adjusts the spatial and spectral tokens based on the center region of the HSI sample, allowing for effective information fusion within each block. Subsequently, the tokens are refined in a multi-head self-attention block to further improve the feature space. Finally, the combined information is fed into the state space block for classification and the creation of the ground truth map. Experiments on widely used Hyperspectral (HS) datasets demonstrate that the MorpMamba model outperforms (parametric efficiency) both CNN and Transformer models.

Title: Coalitions of Large Language Models Increase the Robustness of AI Agents

• Authors: Prattyush Mangal, Carol Mak, Theo Kanakis, Timothy Donovan, Dave Braines, Edward Pyzer-Knapp
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01380
• Pdf URL: https://arxiv.org/pdf/2408.01380
• Copy Paste: [[2408.01380]] Coalitions of Large Language Models Increase the Robustness of AI Agents(https://arxiv.org/abs/2408.01380)
• Keywords: robust, large language model
• Abstract: The emergence of Large Language Models (LLMs) have fundamentally altered the way we interact with digital systems and have led to the pursuit of LLM powered AI agents to assist in daily workflows. LLMs, whilst powerful and capable of demonstrating some emergent properties, are not logical reasoners and often struggle to perform well at all sub-tasks carried out by an AI agent to plan and execute a workflow. While existing studies tackle this lack of proficiency by generalised pretraining at a huge scale or by specialised fine-tuning for tool use, we assess if a system comprising of a coalition of pretrained LLMs, each exhibiting specialised performance at individual sub-tasks, can match the performance of single model agents. The coalition of models approach showcases its potential for building robustness and reducing the operational costs of these AI agents by leveraging traits exhibited by specific models. Our findings demonstrate that fine-tuning can be mitigated by considering a coalition of pretrained models and believe that this approach can be applied to other non-agentic systems which utilise LLMs.

Title: NOLO: Navigate Only Look Once

• Authors: Bohan Zhou, Jiangxing Wang, Zongqing Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01384
• Pdf URL: https://arxiv.org/pdf/2408.01384
• Copy Paste: [[2408.01384]] NOLO: Navigate Only Look Once(https://arxiv.org/abs/2408.01384)
• Keywords: transformer
• Abstract: The in-context learning ability of Transformer models has brought new possibilities to visual navigation. In this paper, we focus on the video navigation setting, where an in-context navigation policy needs to be learned purely from videos in an offline manner, without access to the actual environment. For this setting, we propose Navigate Only Look Once (NOLO), a method for learning a navigation policy that possesses the in-context ability and adapts to new scenes by taking corresponding context videos as input without finetuning or re-training. To enable learning from videos, we first propose a pseudo action labeling procedure using optical flow to recover the action label from egocentric videos. Then, offline reinforcement learning is applied to learn the navigation policy. Through extensive experiments on different scenes, we show that our algorithm outperforms baselines by a large margin, which demonstrates the in-context learning ability of the learned policy.

Title: Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer

• Authors: Yu Yang, Pan Xu
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01402
• Pdf URL: https://arxiv.org/pdf/2408.01402
• Copy Paste: [[2408.01402]] Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer(https://arxiv.org/abs/2408.01402)
• Keywords: transformer
• Abstract: Decision Transformer (DT) has emerged as a promising class of algorithms in offline reinforcement learning (RL) tasks, leveraging pre-collected datasets and Transformer's capability to model long sequences. Recent works have demonstrated that using parts of trajectories from training tasks as prompts in DT enhances its performance on unseen tasks, giving rise to Prompt-DT methods. However, collecting data from specific environments can be both costly and unsafe in many scenarios, leading to suboptimal performance and limited few-shot prompt abilities due to the data-hungry nature of Transformer-based models. Additionally, the limited datasets used in pre-training make it challenging for Prompt-DT type of methods to distinguish between various RL tasks through prompts alone. To address these challenges, we introduce the Language model-initialized Prompt Decision Transformer (LPDT), which leverages pre-trained language models for meta-RL tasks and fine-tunes the model using Low-rank Adaptation (LoRA). We further incorporate prompt regularization to effectively differentiate between tasks based on prompt feature representations. Our approach integrates pre-trained language model and RL tasks seamlessly. Extensive empirical studies demonstrate that initializing with a pre-trained language model significantly enhances the performance of Prompt-DT on unseen tasks compared to baseline methods.

Title: The Quest for the Right Mediator: A History, Survey, and Theoretical Grounding of Causal Interpretability

• Authors: Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, Yonatan Belinkov
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01416
• Pdf URL: https://arxiv.org/pdf/2408.01416
• Copy Paste: [[2408.01416]] The Quest for the Right Mediator: A History, Survey, and Theoretical Grounding of Causal Interpretability(https://arxiv.org/abs/2408.01416)
• Keywords: interpretability
• Abstract: Interpretability provides a toolset for understanding how and why neural networks behave in certain ways. However, there is little unity in the field: most studies employ ad-hoc evaluations and do not share theoretical foundations, making it difficult to measure progress and compare the pros and cons of different techniques. Furthermore, while mechanistic understanding is frequently discussed, the basic causal units underlying these mechanisms are often not explicitly defined. In this paper, we propose a perspective on interpretability research grounded in causal mediation analysis. Specifically, we describe the history and current state of interpretability taxonomized according to the types of causal units (mediators) employed, as well as methods used to search over mediators. We discuss the pros and cons of each mediator, providing insights as to when particular kinds of mediators and search methods are most appropriate depending on the goals of a given study. We argue that this framing yields a more cohesive narrative of the field, as well as actionable insights for future work. Specifically, we recommend a focus on discovering new mediators with better trade-offs between human-interpretability and compute-efficiency, and which can uncover more sophisticated abstractions from neural networks than the primarily linear mediators employed in current work. We also argue for more standardized evaluations that enable principled comparisons across mediator types, such that we can better understand when particular causal units are better suited to particular use cases.

Title: Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs

• Authors: Yilun Hua, Yoav Artzi
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.01417
• Pdf URL: https://arxiv.org/pdf/2408.01417
• Copy Paste: [[2408.01417]] Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs(https://arxiv.org/abs/2408.01417)
• Keywords: large language model
• Abstract: Humans spontaneously use increasingly efficient language as interactions progress, by adapting and forming ad-hoc conventions. This phenomenon has been studied extensively using reference games, showing properties of human language that go beyond relaying intents. It remains unexplored whether multimodal large language models (MLLMs) similarly increase communication efficiency during interactions, and what mechanisms they may adopt for this purpose. We introduce ICCA, an automated framework to evaluate such conversational adaptation as an in-context behavior in MLLMs. We evaluate several state-of-the-art MLLMs, and observe that while they may understand the increasingly efficient language of their interlocutor, they do not spontaneously make their own language more efficient over time. This latter ability can only be elicited in some models (e.g., GPT-4) with heavy-handed prompting. This shows that this property of linguistic interaction does not arise from current training regimes, even though it is a common hallmark of human language. ICCA is available at this https URL.

Title: DebateQA: Evaluating Question Answering on Debatable Knowledge

• Authors: Rongwu Xu, Xuan Qi, Zehan Qi, Wei Xu, Zhijiang Guo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01419
• Pdf URL: https://arxiv.org/pdf/2408.01419
• Copy Paste: [[2408.01419]] DebateQA: Evaluating Question Answering on Debatable Knowledge(https://arxiv.org/abs/2408.01419)
• Keywords: large language model
• Abstract: The rise of large language models (LLMs) has enabled us to seek answers to inherently debatable questions on LLM chatbots, necessitating a reliable way to evaluate their ability. However, traditional QA benchmarks assume fixed answers are inadequate for this purpose. To address this, we introduce DebateQA, a dataset of 2,941 debatable questions, each accompanied by multiple human-annotated partial answers that capture a variety of perspectives. We develop two metrics: Perspective Diversity, which evaluates the comprehensiveness of perspectives, and Dispute Awareness, which assesses if the LLM acknowledges the question's debatable nature. Experiments demonstrate that both metrics align with human preferences and are stable across different underlying models. Using DebateQA with two metrics, we assess 12 popular LLMs and retrieval-augmented generation methods. Our findings reveal that while LLMs generally excel at recognizing debatable issues, their ability to provide comprehensive answers encompassing diverse perspectives varies considerably.

Title: Mission Impossible: A Statistical Perspective on Jailbreaking LLMs

• Authors: Jingtong Su, Julia Kempe, Karen Ullrich
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01420
• Pdf URL: https://arxiv.org/pdf/2408.01420
• Copy Paste: [[2408.01420]] Mission Impossible: A Statistical Perspective on Jailbreaking LLMs(https://arxiv.org/abs/2408.01420)
• Keywords: large language model
• Abstract: Large language models (LLMs) are trained on a deluge of text data with limited quality control. As a result, LLMs can exhibit unintended or even harmful behaviours, such as leaking information, fake news or hate speech. Countermeasures, commonly referred to as preference alignment, include fine-tuning the pretrained LLMs with carefully crafted text examples of desired behaviour. Even then, empirical evidence shows preference aligned LLMs can be enticed to harmful behaviour. This so called jailbreaking of LLMs is typically achieved by adversarially modifying the input prompt to the LLM. Our paper provides theoretical insights into the phenomenon of preference alignment and jailbreaking from a statistical perspective. Under our framework, we first show that pretrained LLMs will mimic harmful behaviour if present in the training corpus. Under that same framework, we then introduce a statistical notion of alignment, and lower-bound the jailbreaking probability, showing that it is unpreventable under reasonable assumptions. Based on our insights, we propose an alteration to the currently prevalent alignment strategy RLHF. Specifically, we introduce a simple modification to the RLHF objective, we call E-RLHF, that aims to increase the likelihood of safe responses. E-RLHF brings no additional training cost, and is compatible with other methods. Empirically, we demonstrate that E-RLHF outperforms RLHF on all alignment problems put forward by the AdvBench and HarmBench project without sacrificing model performance as measured by the MT-Bench project.

Title: Prompt Recursive Search: A Living Framework with Adaptive Growth in LLM Auto-Prompting

• Authors: Xiangyu Zhao, Chengqian Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01423
• Pdf URL: https://arxiv.org/pdf/2408.01423
• Copy Paste: [[2408.01423]] Prompt Recursive Search: A Living Framework with Adaptive Growth in LLM Auto-Prompting(https://arxiv.org/abs/2408.01423)
• Keywords: large language model
• Abstract: Large Language Models (LLMs) exhibit remarkable proficiency in addressing a diverse array of tasks within the Natural Language Processing (NLP) domain, with various prompt design strategies significantly augmenting their capabilities. However, these prompts, while beneficial, each possess inherent limitations. The primary prompt design methodologies are twofold: The first, exemplified by the Chain of Thought (CoT), involves manually crafting prompts specific to individual datasets, hence termed Expert-Designed Prompts (EDPs). Once these prompts are established, they are unalterable, and their effectiveness is capped by the expertise of the human designers. When applied to LLMs, the static nature of EDPs results in a uniform approach to both simple and complex problems within the same dataset, leading to the inefficient use of tokens for straightforward issues. The second method involves prompts autonomously generated by the LLM, known as LLM-Derived Prompts (LDPs), which provide tailored solutions to specific problems, mitigating the limitations of EDPs. However, LDPs may encounter a decline in performance when tackling complex problems due to the potential for error accumulation during the solution planning process. To address these challenges, we have conceived a novel Prompt Recursive Search (PRS) framework that leverages the LLM to generate solutions specific to the problem, thereby conserving tokens. The framework incorporates an assessment of problem complexity and an adjustable structure, ensuring a reduction in the likelihood of errors. We have substantiated the efficacy of PRS framework through extensive experiments using LLMs with different numbers of parameters across a spectrum of datasets in various domains. Compared to the CoT method, the PRS method has increased the accuracy on the BBH dataset by 8% using Llama3-7B model, achieving a 22% improvement.